Orionnnn/hub-stats huggingface dataset:
M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
Authors: name: Mingsheng Li, name: Xin Chen, name: Chi Zhang, name: Sijin Chen, name: Hongyuan Zhu, name: Fukun Yin, name: Gang Yu, name: Tao Chen
Published: 2023-12-17 16:53:30

Recently, 3D understanding has become popular to facilitate autonomous agents
to perform further decisionmaking. However, existing 3D datasets and methods
are often limited to specific tasks. On the other hand, recent progress in
Large Language Models (LLMs) and Multimodal Language Models (MLMs) have
demonstrated exceptional general language and imagery tasking performance.
Therefore, it is interesting to unlock MLM's potential to be 3D generalist for
wider tasks. However, current MLMs' research has been less focused on 3D tasks
due to a lack of large-scale 3D instruction-following datasets. In this work,
we introduce a comprehensive 3D instructionfollowing dataset called M3DBench,
which possesses the following characteristics: 1) It supports general
multimodal instructions interleaved with text, images, 3D objects, and other
visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels,
covering a variety of fundamental abilities in real-world 3D environments. 3)
It is a large-scale 3D instruction-following dataset with over 320k
instruction-response pairs. Furthermore, we establish a new benchmark for
assessing the performance of large models in understanding multi-modal 3D
prompts. Extensive experiments demonstrate the effectiveness of our dataset and
baseline, supporting general 3D-centric tasks, which can inspire future
research.
Paloma: A Benchmark for Evaluating Language Model Fit
Authors: name: Ian Magnusson, name: Akshita Bhagia, name: Valentin Hofmann, name: Luca Soldaini, name: Ananya Harsh Jha, name: Oyvind Tafjord, name: Dustin Schwenk, name: Evan Pete Walsh, name: Yanai Elazar, name: Kyle Lo, name: Dirk Groeneveld, name: Iz Beltagy, name: Hannaneh Hajishirzi, name: Noah A. Smith, name: Kyle Richardson, name: Jesse Dodge
Published: 2023-12-16 19:12:45

Language models (LMs) commonly report perplexity on monolithic data held out
from training. Implicitly or explicitly, this data is composed of
domainsx2013varying distributions of language. Rather than assuming
perplexity on one distribution extrapolates to others, Perplexity Analysis for
Language Model Assessment (Paloma), measures LM fit to 585 text domains,
ranging from nytimes.com to r/depression on Reddit. We invite submissions to
our benchmark and organize results by comparability based on compliance with
guidelines such as removal of benchmark contamination from pretraining.
Submissions can also record parameter and training token count to make
comparisons of Pareto efficiency for performance as a function of these
measures of cost. We populate our benchmark with results from 6 baselines
pretrained on popular corpora. In case studies, we demonstrate analyses that
are possible with Paloma, such as finding that pretraining without data beyond
Common Crawl leads to inconsistent fit to many domains.
Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
Authors: name: Dirk Groeneveld, name: Anas Awadalla, name: Iz Beltagy, name: Akshita Bhagia, name: Ian Magnusson, name: Hao Peng, name: Oyvind Tafjord, name: Pete Walsh, name: Kyle Richardson, name: Jesse Dodge
Published: 2023-12-15 23:11:45

The success of large language models has shifted the evaluation paradigms in
natural language processing (NLP). The community's interest has drifted towards
comparing NLP models across many tasks, domains, and datasets, often at an
extreme scale. This imposes new engineering challenges: efforts in constructing
datasets and models have been fragmented, and their formats and interfaces are
incompatible. As a result, it often takes extensive (re)implementation efforts
to make fair and controlled comparisons at scale.
  Catwalk aims to address these issues. Catwalk provides a unified interface to
a broad range of existing NLP datasets and models, ranging from both canonical
supervised training and fine-tuning, to more modern paradigms like in-context
learning. Its carefully-designed abstractions allow for easy extensions to many
others. Catwalk substantially lowers the barriers to conducting controlled
experiments at scale. For example, we finetuned and evaluated over 64 models on
over 86 datasets with a single command, without writing any code. Maintained by
the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2),
Catwalk is an ongoing open-source effort: https://github.com/allenai/catwalk.
ProTIP: Progressive Tool Retrieval Improves Planning
Authors: name: Raviteja Anantha, name: Bortik Bandyopadhyay, name: Anirudh Kashi, name: Sayantan Mahinder, name: Andrew W Hill, name: Srinivas Chappidi
Published: 2023-12-16 05:43:11

Large language models (LLMs) are increasingly employed for complex multi-step
planning tasks, where the tool retrieval (TR) step is crucial for achieving
successful outcomes. Two prevalent approaches for TR are single-step retrieval,
which utilizes the complete query, and sequential retrieval using task
decomposition (TD), where a full query is segmented into discrete atomic
subtasks. While single-step retrieval lacks the flexibility to handle
"inter-tool dependency," the TD approach necessitates maintaining "subtask-tool
atomicity alignment," as the toolbox can evolve dynamically. To address these
limitations, we introduce the Progressive Tool retrieval to Improve Planning
(ProTIP) framework. ProTIP is a lightweight, contrastive learning-based
framework that implicitly performs TD without the explicit requirement of
subtask labels, while simultaneously maintaining subtask-tool atomicity. On the
ToolBench dataset, ProTIP outperforms the ChatGPT task decomposition-based
approach by a remarkable margin, achieving a 24% improvement in Recall@K=10 for
TR and a 41% enhancement in tool accuracy for plan generation.
G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
Authors: name: Jiahui Gao, name: Renjie Pi, name: Jipeng Zhang, name: Jiacheng Ye, name: Wanjun Zhong, name: Yufei Wang, name: Lanqing Hong, name: Jianhua Han, name: Hang Xu, name: Zhenguo Li, name: Lingpeng Kong
Published: 2023-12-18 17:36:20

Large language models (LLMs) have shown remarkable proficiency in human-level
reasoning and generation capabilities, which encourages extensive research on
their application in mathematical problem solving. However, current work has
been largely focused on text-based mathematical problems, with limited
investigation in problems involving geometric information. Addressing this gap,
we aim to enable LLMs to solve geometric problems by understanding image input.
We first analyze the limitations of current Multimodal Large Language Models
(MLLMs) in this area: they struggle to accurately comprehending basic geometric
elements and their relationships. To overcome these challenges, we take
advantage of the unique characteristics of geometric problems (such as unique
geometric logical form, and geometric scalability) and the capacity of the
textual LLMs to build an enriched multimodal geometry dataset based on existing
data. The augmented dataset, Geo170K, contains more than 170K geometric
image-caption and question-answer pairs. Utilizing our constructed Geo170K
dataset, we develop G-LLaVA, which demonstrates exceptional performance in
solving geometric problems, significantly outperforming GPT-4-V on the
MathVista benchmark with only 7B parameters.
Silkie: Preference Distillation for Large Visual Language Models
Authors: name: Lei Li, name: Zhihui Xie, name: Mukai Li, name: Shunian Chen, name: Peiyi Wang, name: Liang Chen, name: Yazheng Yang, name: Benyou Wang, name: Lingpeng Kong
Published: 2023-12-17 09:44:27

This paper explores preference distillation for large vision language models
(LVLMs), improving their ability to generate helpful and faithful responses
anchoring the visual context. We first build a vision-language feedback
(VLFeedback) dataset utilizing AI annotation. Specifically, responses are
generated by models sampled from 12 LVLMs, conditioned on multi-modal
instructions sourced from various datasets. We adopt GPT-4V to assess the
generated outputs regarding helpfulness, visual faithfulness, and ethical
considerations. Furthermore, the preference supervision is distilled into
Qwen-VL-Chat through the direct preference optimization (DPO) method. The
resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME
benchmark regarding the perception and cognition capabilities, respectively.
Silkie also demonstrates reduced hallucination by setting a new
state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis
shows that DPO with our VLFeedback dataset mainly boosts the fine-grained
perception and complex cognition abilities of LVLMs, leading to more
comprehensive improvements compared to human-annotated preference datasets.
Rich Human Feedback for Text-to-Image Generation
Authors: name: Youwei Liang, name: Junfeng He, name: Gang Li, name: Peizhao Li, name: Arseniy Klimovskiy, name: Nicholas Carolan, name: Jiao Sun, name: Jordi Pont-Tuset, name: Sarah Young, name: Feng Yang, name: Junjie Ke, name: Krishnamurthy Dj Dvijotham, name: Katie Collins, name: Yiwen Luo, name: Yang Li, name: Kai J Kohlhoff, name: Deepak Ramachandran, name: Vidhya Navalpakkam
Published: 2023-12-15 22:18:38

Recent Text-to-Image (T2I) generation models such as Stable Diffusion and
Imagen have made significant progress in generating high-resolution images
based on text descriptions. However, many generated images still suffer from
issues such as artifacts/implausibility, misalignment with text descriptions,
and low aesthetic quality. Inspired by the success of Reinforcement Learning
with Human Feedback (RLHF) for large language models, prior works collected
human-provided scores as feedback on generated images and trained a reward
model to improve the T2I generation. In this paper, we enrich the feedback
signal by (i) marking image regions that are implausible or misaligned with the
text, and (ii) annotating which words in the text prompt are misrepresented or
missing on the image. We collect such rich human feedback on 18K generated
images and train a multimodal transformer to predict the rich feedback
automatically. We show that the predicted rich human feedback can be leveraged
to improve image generation, for example, by selecting high-quality training
data to finetune and improve the generative models, or by creating masks with
predicted heatmaps to inpaint the problematic regions. Notably, the
improvements generalize to models (Muse) beyond those used to generate the
images on which human feedback data were collected (Stable Diffusion variants).
VidToMe: Video Token Merging for Zero-Shot Video Editing
Authors: name: Xirui Li, name: Chao Ma, name: Xiaokang Yang, name: Ming-Hsuan Yang
Published: 2023-12-17 09:05:56

Diffusion models have made significant advances in generating high-quality
images, but their application to video generation has remained challenging due
to the complexity of temporal motion. Zero-shot video editing offers a solution
by utilizing pre-trained image diffusion models to translate source videos into
new ones. Nevertheless, existing methods struggle to maintain strict temporal
consistency and efficient memory consumption. In this work, we propose a novel
approach to enhance temporal consistency in generated videos by merging
self-attention tokens across frames. By aligning and compressing temporally
redundant tokens across frames, our method improves temporal coherence and
reduces memory consumption in self-attention computations. The merging strategy
matches and aligns tokens according to the temporal correspondence between
frames, facilitating natural temporal consistency in generated video frames. To
manage the complexity of video processing, we divide videos into chunks and
develop intra-chunk local token merging and inter-chunk global token merging,
ensuring both short-term video continuity and long-term content consistency.
Our video editing approach seamlessly extends the advancements in image editing
to video editing, rendering favorable results in temporal consistency over
state-of-the-art methods.
MobileSAMv2: Faster Segment Anything to Everything
Authors: name: Chaoning Zhang, name: Dongshen Han, name: Sheng Zheng, name: Jinwoo Choi, name: Tae-Ho Kim, name: Choong Seon Hong
Published: 2023-12-15 07:21:12

Segment anything model (SAM) addresses two practical yet challenging
segmentation tasks: segment anything (SegAny), which utilizes a
certain point to predict the mask for a single object of interest, and
segment everything (SegEvery), which predicts the masks for all
objects on the image. What makes SegAny slow for SAM is its heavyweight image
encoder, which has been addressed by MobileSAM via decoupled knowledge
distillation. The efficiency bottleneck of SegEvery with SAM, however, lies in
its mask decoder because it needs to first generate numerous masks with
redundant grid-search prompts and then perform filtering to obtain the final
valid masks. We propose to improve its efficiency by directly generating the
final masks with only valid prompts, which can be obtained through object
discovery. Our proposed approach not only helps reduce the total time on the
mask decoder by at least 16 times but also achieves superior performance.
Specifically, our approach yields an average performance boost of 3.6\% (42.5\%
v.s. 38.9\%) for zero-shot object proposal on the LVIS dataset with
the mask AR@K metric. Qualitative results show that our approach generates
fine-grained masks while avoiding over-segmenting things. This project
targeting faster SegEvery than the original SAM is termed MobileSAMv2 to
differentiate from MobileSAM which targets faster SegAny. Moreover, we
demonstrate that our new prompt sampling is also compatible with the distilled
image encoders in MobileSAM, contributing to a unified framework for efficient
SegAny and SegEvery. The code is available at the same link as MobileSAM
Project
https://github.com/ChaoningZhang/MobileSAM{red{https://github.com/ChaoningZhang/MobileSAM}}.
abstract
Point Transformer V3: Simpler, Faster, Stronger
Authors: name: Xiaoyang Wu, name: Li Jiang, name: Peng-Shuai Wang, name: Zhijian Liu, name: Xihui Liu, name: Yu Qiao, name: Wanli Ouyang, name: Tong He, name: Hengshuang Zhao
Published: 2023-12-15 18:59:59

This paper is not motivated to seek innovation within the attention
mechanism. Instead, it focuses on overcoming the existing trade-offs between
accuracy and efficiency within the context of point cloud processing,
leveraging the power of scale. Drawing inspiration from recent advances in 3D
large-scale representation learning, we recognize that model performance is
more influenced by scale than by intricate design. Therefore, we present Point
Transformer V3 (PTv3), which prioritizes simplicity and efficiency over the
accuracy of certain mechanisms that are minor to the overall performance after
scaling, such as replacing the precise neighbor search by KNN with an efficient
serialized neighbor mapping of point clouds organized with specific patterns.
This principle enables significant scaling, expanding the receptive field from
16 to 1024 points while remaining efficient (a 3x increase in processing speed
and a 10x improvement in memory efficiency compared with its predecessor,
PTv2). PTv3 attains state-of-the-art results on over 20 downstream tasks that
span both indoor and outdoor scenarios. Further enhanced with multi-dataset
joint training, PTv3 pushes these results to a higher level.
Extending Context Window of Large Language Models via Semantic
  Compression
Authors: name: Weizhi Fei, name: Xueyan Niu, name: Pingyi Zhou, name: Lu Hou, name: Bo Bai, name: Lei Deng, name: Wei Han
Published: 2023-12-15 07:04:33

Transformer-based Large Language Models (LLMs) often impose limitations on
the length of the text input to ensure the generation of fluent and relevant
responses. This constraint restricts their applicability in scenarios involving
long texts. We propose a novel semantic compression method that enables
generalization to texts that are 6-8 times longer, without incurring
significant computational costs or requiring fine-tuning. Our proposed
framework draws inspiration from source coding in information theory and
employs a pre-trained model to reduce the semantic redundancy of long inputs
before passing them to the LLMs for downstream tasks. Experimental results
demonstrate that our method effectively extends the context window of LLMs
across a range of tasks including question answering, summarization, few-shot
learning, and information retrieval. Furthermore, the proposed semantic
compression method exhibits consistent fluency in text generation while
reducing the associated computational overhead.
DreamTalk: When Expressive Talking Head Generation Meets Diffusion
  Probabilistic Models
Authors: name: Yifeng Ma, name: Shiwei Zhang, name: Jiayu Wang, name: Xiang Wang, name: Yingya Zhang, name: Zhidong Deng
Published: 2023-12-15 13:15:42

Diffusion models have shown remarkable success in a variety of downstream
generative tasks, yet remain under-explored in the important and challenging
expressive talking head generation. In this work, we propose a DreamTalk
framework to fulfill this gap, which employs meticulous design to unlock the
potential of diffusion models in generating expressive talking heads.
Specifically, DreamTalk consists of three crucial components: a denoising
network, a style-aware lip expert, and a style predictor. The diffusion-based
denoising network is able to consistently synthesize high-quality audio-driven
face motions across diverse expressions. To enhance the expressiveness and
accuracy of lip motions, we introduce a style-aware lip expert that can guide
lip-sync while being mindful of the speaking styles. To eliminate the need for
expression reference video or text, an extra diffusion-based style predictor is
utilized to predict the target expression directly from the audio. By this
means, DreamTalk can harness powerful diffusion models to generate expressive
faces effectively and reduce the reliance on expensive style references.
Experimental results demonstrate that DreamTalk is capable of generating
photo-realistic talking faces with diverse speaking styles and achieving
accurate lip motions, surpassing existing state-of-the-art counterparts.
SlimmeRF: Slimmable Radiance Fields
Authors: name: Shiran Yuan, name: Hao Zhao
Published: 2023-12-15 18:59:55

Neural Radiance Field (NeRF) and its variants have recently emerged as
successful methods for novel view synthesis and 3D scene reconstruction.
However, most current NeRF models either achieve high accuracy using large
model sizes, or achieve high memory-efficiency by trading off accuracy. This
limits the applicable scope of any single model, since high-accuracy models
might not fit in low-memory devices, and memory-efficient models might not
satisfy high-quality requirements. To this end, we present SlimmeRF, a model
that allows for instant test-time trade-offs between model size and accuracy
through slimming, thus making the model simultaneously suitable for scenarios
with different computing budgets. We achieve this through a newly proposed
algorithm named Tensorial Rank Incrementation (TRaIn) which increases the rank
of the model's tensorial representation gradually during training. We also
observe that our model allows for more effective trade-offs in sparse-view
scenarios, at times even achieving higher accuracy after being slimmed. We
credit this to the fact that erroneous information such as floaters tend to be
stored in components corresponding to higher ranks. Our implementation is
available at https://github.com/Shiran-Yuan/SlimmeRF.
Faster Diffusion: Rethinking the Role of UNet Encoder in Diffusion
  Models
Authors: name: Senmao Li, name: Taihang Hu, name: Fahad Shahbaz Khan, name: Linxuan Li, name: Shiqi Yang, name: Yaxing Wang, name: Ming-Ming Cheng, name: Jian Yang
Published: 2023-12-15 08:46:43

One of the key components within diffusion models is the UNet for noise
prediction. While several works have explored basic properties of the UNet
decoder, its encoder largely remains unexplored. In this work, we conduct the
first comprehensive study of the UNet encoder. We empirically analyze the
encoder features and provide insights to important questions regarding their
changes at the inference process. In particular, we find that encoder features
change gently, whereas the decoder features exhibit substantial variations
across different time-steps. This finding inspired us to omit the encoder at
certain adjacent time-steps and reuse cyclically the encoder features in the
previous time-steps for the decoder. Further based on this observation, we
introduce a simple yet effective encoder propagation scheme to accelerate the
diffusion sampling for a diverse set of tasks. By benefiting from our
propagation scheme, we are able to perform in parallel the decoder at certain
adjacent time-steps. Additionally, we introduce a prior noise injection method
to improve the texture details in the generated image. Besides the standard
text-to-image task, we also validate our approach on other tasks:
text-to-video, personalized generation and reference-guided generation. Without
utilizing any knowledge distillation technique, our approach accelerates both
the Stable Diffusion (SD) and the DeepFloyd-IF models sampling by 41% and
24% respectively, while maintaining high-quality generation performance. Our
code is available in
https://github.com/hutaiHang/Faster-Diffusion{FasterDiffusion}.
Perspectives on the State and Future of Deep Learning - 2023
Authors: name: Micah Goldblum, name: Anima Anandkumar, name: Richard Baraniuk, name: Tom Goldstein, name: Kyunghyun Cho, name: Zachary C Lipton, name: Melanie Mitchell, name: Preetum Nakkiran, name: Max Welling, name: Andrew Gordon Wilson
Published: 2023-12-07 19:58:37

The goal of this series is to chronicle opinions and issues in the field of
machine learning as they stand today and as they change over time. The plan is
to host this survey periodically until the AI singularity
paperclip-frenzy-driven doomsday, keeping an updated list of topical questions
and interviewing new community members for each edition. In this issue, we
probed people's opinions on interpretable AI, the value of benchmarking in
modern NLP, the state of progress towards understanding deep learning, and the
future of academia.
Amphion: An Open-Source Audio, Music and Speech Generation Toolkit
Authors: name: Xueyao Zhang, name: Liumeng Xue, name: Yuancheng Wang, name: Yicheng Gu, name: Xi Chen, name: Zihao Fang, name: Haopeng Chen, name: Lexiao Zou, name: Chaoren Wang, name: Jun Han, name: Kai Chen, name: Haizhou Li, name: Zhizheng Wu
Published: 2023-12-15 16:23:21

Amphion is a toolkit for Audio, Music, and Speech Generation. Its purpose is
to support reproducible research and help junior researchers and engineers get
started in the field of audio, music, and speech generation research and
development. Amphion offers a unique feature: visualizations of classic models
or architectures. We believe that these visualizations are beneficial for
junior researchers and engineers who wish to gain a better understanding of the
model. The North-Star objective of Amphion is to offer a platform for studying
the conversion of any inputs into general audio. Amphion is designed to support
individual generation tasks. In addition to the specific generation tasks,
Amphion also includes several vocoders and evaluation metrics. A vocoder is an
important module for producing high-quality audio signals, while evaluation
metrics are critical for ensuring consistent metrics in generation tasks. In
this paper, we provide a high-level overview of Amphion.