Orionnnn/hub-stats huggingface dataset:
LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
Authors: name: Kaichen Zhang, name: Bo Li, name: Peiyuan Zhang, name: Fanyi Pu, name: Joshua Adrian Cahyono, name: Kairui Hu, name: Shuai Liu, name: Yuanhan Zhang, name: Jingkang Yang, name: Chunyuan Li, name: Ziwei Liu
Published: 2024-07-17 17:51:53

The advances of large foundation models necessitate wide-coverage, low-cost,
and zero-contamination benchmarks. Despite continuous exploration of language
model evaluations, comprehensive studies on the evaluation of Large Multi-modal
Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified
and standardized multimodal benchmark framework with over 50 tasks and more
than 10 models to promote transparent and reproducible evaluations. Although
LMMS-EVAL offers comprehensive coverage, we find it still falls short in
achieving low cost and zero contamination. To approach this evaluation
trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that
emphasizes both coverage and efficiency. Additionally, we present Multimodal
LIVEBENCH that utilizes continuously updating news and online forums to assess
models' generalization abilities in the wild, featuring a low-cost and
zero-contamination evaluation approach. In summary, our work highlights the
importance of considering the evaluation trilemma and provides practical
solutions to navigate the trade-offs in evaluating large multi-modal models,
paving the way for more effective and reliable benchmarking of LMMs. We
opensource our codebase and maintain leaderboard of LIVEBENCH at
https://github.com/EvolvingLMMs-Lab/lmms-eval and
https://huggingface.co/spaces/lmms-lab/LiveBench.
Audio Conditioning for Music Generation via Discrete Bottleneck Features
Authors: name: Simon Rouard, name: Yossi Adi, name: Jade Copet, name: Axel Roebel, name: Alexandre Défossez
Published: 2024-07-17 13:47:17

While most music generation models use textual or parametric conditioning
(e.g. tempo, harmony, musical genre), we propose to condition a language model
based music generation system with audio input. Our exploration involves two
distinct strategies. The first strategy, termed textual inversion, leverages a
pre-trained text-to-music model to map audio input to corresponding
"pseudowords" in the textual embedding space. For the second model we train a
music language model from scratch jointly with a text conditioner and a
quantized audio feature extractor. At inference time, we can mix textual and
audio conditioning and balance them thanks to a novel double classifier free
guidance method. We conduct automatic and human studies that validates our
approach. We will release the code and we provide music samples on
https://musicgenstyle.github.io in order to show the quality of our model.
Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for
  Unconstrained Photo Collections
Authors: name: Congrong Xu, name: Justin Kerr, name: Angjoo Kanazawa
Published: 2024-07-17 04:02:54

Novel view synthesis from unconstrained in-the-wild image collections remains
a significant yet challenging task due to photometric variations and transient
occluders that complicate accurate scene reconstruction. Previous methods have
approached these issues by integrating per-image appearance features embeddings
in Neural Radiance Fields (NeRFs). Although 3D Gaussian Splatting (3DGS) offers
faster training and real-time rendering, adapting it for unconstrained image
collections is non-trivial due to the substantially different architecture. In
this paper, we introduce Splatfacto-W, an approach that integrates per-Gaussian
neural color features and per-image appearance embeddings into the
rasterization process, along with a spherical harmonics-based background model
to represent varying photometric appearances and better depict backgrounds. Our
key contributions include latent appearance modeling, efficient transient
object handling, and precise background modeling. Splatfacto-W delivers
high-quality, real-time novel view synthesis with improved scene consistency in
in-the-wild scenarios. Our method improves the Peak Signal-to-Noise Ratio
(PSNR) by an average of 5.3 dB compared to 3DGS, enhances training speed by 150
times compared to NeRF-based methods, and achieves a similar rendering speed to
3DGS. Additional video results and code integrated into Nerfstudio are
available at https://kevinxu02.github.io/splatfactow/.
VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
Authors: name: Sherwin Bahmani, name: Ivan Skorokhodov, name: Aliaksandr Siarohin, name: Willi Menapace, name: Guocheng Qian, name: Michael Vasilkovsky, name: Hsin-Ying Lee, name: Chaoyang Wang, name: Jiaxu Zou, name: Andrea Tagliasacchi, name: David B. Lindell, name: Sergey Tulyakov
Published: 2024-07-17 17:59:05

Modern text-to-video synthesis models demonstrate coherent, photorealistic
generation of complex videos from a text description. However, most existing
models lack fine-grained control over camera movement, which is critical for
downstream applications related to content creation, visual effects, and 3D
vision. Recently, new methods demonstrate the ability to generate videos with
controllable camera poses these techniques leverage pre-trained U-Net-based
diffusion models that explicitly disentangle spatial and temporal generation.
Still, no existing approach enables camera control for new, transformer-based
video diffusion models that process spatial and temporal information jointly.
Here, we propose to tame video transformers for 3D camera control using a
ControlNet-like conditioning mechanism that incorporates spatiotemporal camera
embeddings based on Plucker coordinates. The approach demonstrates
state-of-the-art performance for controllable video generation after
fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our
work is the first to enable camera control for transformer-based video
diffusion models.
EfficientQAT: Efficient Quantization-Aware Training for Large Language
  Models
Authors: name: Mengzhao Chen, name: Wenqi Shao, name: Peng Xu, name: Jiahao Wang, name: Peng Gao, name: Kaipeng Zhang, name: Yu Qiao, name: Ping Luo
Published: 2024-07-10 17:53:30

Large language models (LLMs) are integral to modern natural language
processing and artificial intelligence. However, they face challenges in
managing their significant memory requirements. Although quantization-aware
training (QAT) offers a solution by reducing memory consumption through low-bit
representations with minimal accuracy loss, it demands substantial training
resources to optimize model weights and quantization parameters. To address
this, we propose Efficient Quantization-Aware Training (EfficientQAT), a novel
quantization technique for compressing LLMs. EfficientQAT involves two
consecutive phases: Block-wise training of all parameters (Block-AP) and
end-to-end training of quantization parameters (E2E-QP). Block-AP sequentially
conducts quantization-aware training for all parameters in each transformer
block with block-wise reconstruction, maintaining efficiency by avoiding
training the entire LLM. Initialized with quantized model, E2E-QP then trains
only quantization parameters (step sizes) end-to-end, enhancing efficiency with
a fixed quantized backbone and reduced trainable parameter count. Extensive
experiments demonstrate that EfficientQAT outperforms previous quantization
methods across a range of models, including base LLMs, instruction-tuned LLMs,
and multimodal LLMs, with scales from 7B to 70B parameters at various
quantization bits. For instance, EfficientQAT obtains a 2-bit Llama-2-70B model
on a single A100-80GB GPU in 41 hours, with less than 3\% accuracy degradation
compared to the full precision (69.48 vs. 72.41). Notably, this INT2 quantized
70B model obtains a 1.67 accuracy gain over the Llama-2-13B model (69.48 vs.
67.81) while requiring less memory (19.2GB vs. 24.2GB). Code is available at
https://github.com/OpenGVLab/EfficientQAT.
Uncertainty is Fragile: Manipulating Uncertainty in Large Language
  Models
Authors: name: Qingcheng Zeng, name: Mingyu Jin, name: Qinkai Yu, name: Zhenting Wang, name: Wenyue Hua, name: Zihao Zhou, name: Guangyan Sun, name: Yanda Meng, name: Shiqing Ma, name: Qifan Wang, name: Felix Juefei-Xu, name: Kaize Ding, name: Fan Yang, name: Ruixiang Tang, name: Yongfeng Zhang
Published: 2024-07-15 23:41:11

Large Language Models (LLMs) are employed across various high-stakes domains,
where the reliability of their outputs is crucial. One commonly used method to
assess the reliability of LLMs' responses is uncertainty estimation, which
gauges the likelihood of their answers being correct. While many studies focus
on improving the accuracy of uncertainty estimations for LLMs, our research
investigates the fragility of uncertainty estimation and explores potential
attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which,
when activated by a specific trigger in the input, manipulates the model's
uncertainty without affecting the final output. Specifically, the proposed
backdoor attack method can alter an LLM's output probability distribution,
causing the probability distribution to converge towards an attacker-predefined
distribution while ensuring that the top-1 prediction remains unchanged. Our
experimental results demonstrate that this attack effectively undermines the
model's self-evaluation reliability in multiple-choice questions. For instance,
we achieved a 100 attack success rate (ASR) across three different triggering
strategies in four models. Further, we investigate whether this manipulation
generalizes across different prompts and domains. This work highlights a
significant threat to the reliability of LLMs and underscores the need for
future defenses against such attacks. The code is available at
https://github.com/qcznlp/uncertainty_attack.
Vibravox: A Dataset of French Speech Captured with Body-conduction Audio
  Sensors
Authors: name: Julien Hauret, name: Malo Olivier, name: Thomas Joubaud, name: Christophe Langrenne, name: Sarah Poirée, name: Véronique Zimpfer, name: Éric Bavu
Published: 2024-07-16 15:16:10

Vibravox is a dataset compliant with the General Data Protection Regulation
(GDPR) containing audio recordings using five different body-conduction audio
sensors : two in-ear microphones, two bone conduction vibration pickups and a
laryngophone. The data set also includes audio data from an airborne microphone
used as a reference. The Vibravox corpus contains 38 hours of speech samples
and physiological sounds recorded by 188 participants under different acoustic
conditions imposed by an high order ambisonics 3D spatializer. Annotations
about the recording conditions and linguistic transcriptions are also included
in the corpus. We conducted a series of experiments on various speech-related
tasks, including speech recognition, speech enhancement and speaker
verification. These experiments were carried out using state-of-the-art models
to evaluate and compare their performances on signals captured by the different
audio sensors offered by the Vibravox dataset, with the aim of gaining a better
grasp of their individual characteristics.
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of
  Multimodal Models
Authors: name: Pengxiang Li, name: Zhi Gao, name: Bofei Zhang, name: Tao Yuan, name: Yuwei Wu, name: Mehrtash Harandi, name: Yunde Jia, name: Song-Chun Zhu, name: Qing Li
Published: 2024-07-16 09:00:45

Vision language models (VLMs) have achieved impressive progress in diverse
applications, becoming a prevalent research direction. In this paper, we build
FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn
conversations that are derived from 27 source datasets, empowering VLMs to
spontaneously refine their responses based on user feedback across diverse
tasks. To scale up the data collection, FIRE is collected in two components:
FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is
freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a
benchmark to comprehensively evaluate the feedback-refining capability of VLMs,
which contains 11K feedback-refinement conversations as the test data, two
evaluation settings, and a model to provide feedback for VLMs. We develop the
FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows
remarkable feedback-refining capability on FIRE-Bench and outperforms untrained
VLMs by 50%, making more efficient user-agent interactions and underscoring the
significance of the FIRE dataset.
Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
Authors: name: Yaoting Wang, name: Peiwen Sun, name: Dongzhan Zhou, name: Guangyao Li, name: Honggang Zhang, name: Di Hu
Published: 2024-07-15 17:54:45

Traditional reference segmentation tasks have predominantly focused on silent
visual scenes, neglecting the integral role of multimodal perception and
interaction in human experiences. In this work, we introduce a novel task
called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment
objects within the visual domain based on expressions containing multimodal
cues. Such expressions are articulated in natural language forms but are
enriched with multimodal cues, including audio and visual descriptions. To
facilitate this research, we construct the first Ref-AVS benchmark, which
provides pixel-level annotations for objects described in corresponding
multimodal-cue expressions. To tackle the Ref-AVS task, we propose a new method
that adequately utilizes multimodal cues to offer precise segmentation
guidance. Finally, we conduct quantitative and qualitative experiments on three
test subsets to compare our approach with existing methods from related tasks.
The results demonstrate the effectiveness of our method, highlighting its
capability to precisely segment objects using multimodal-cue expressions.
Dataset is available at
https://gewu-lab.github.io/Ref-AVS{https://gewu-lab.github.io/Ref-AVS}.
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality
  Models
Authors: name: Haodong Duan, name: Junming Yang, name: Yuxuan Qiao, name: Xinyu Fang, name: Lin Chen, name: Yuan Liu, name: Xiaoyi Dong, name: Yuhang Zang, name: Pan Zhang, name: Jiaqi Wang, name: Dahua Lin, name: Kai Chen
Published: 2024-07-16 13:06:15

We present VLMEvalKit: an open-source toolkit for evaluating large
multi-modality models based on PyTorch. The toolkit aims to provide a
user-friendly and comprehensive framework for researchers and developers to
evaluate existing multi-modality models and publish reproducible evaluation
results. In VLMEvalKit, we implement over 70 different large multi-modality
models, including both proprietary APIs and open-source models, as well as more
than 20 different multi-modal benchmarks. By implementing a single interface,
new models can be easily added to the toolkit, while the toolkit automatically
handles the remaining workloads, including data preparation, distributed
inference, prediction post-processing, and metric calculation. Although the
toolkit is currently mainly used for evaluating large vision-language models,
its design is compatible with future updates that incorporate additional
modalities, such as audio and video. Based on the evaluation results obtained
with the toolkit, we host OpenVLM Leaderboard, a comprehensive leaderboard to
track the progress of multi-modality learning research. The toolkit is released
at https://github.com/open-compass/VLMEvalKit and is actively maintained.
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Authors: name: Zehan Wang, name: Ziang Zhang, name: Hang Zhang, name: Luping Liu, name: Rongjie Huang, name: Xize Cheng, name: Hengshuang Zhao, name: Zhou Zhao
Published: 2024-07-16 16:24:31

Recently, human-computer interaction with various modalities has shown
promising applications, like GPT-4o and Gemini. Given the foundational role of
multimodal joint representation in understanding and generation pipelines,
high-quality omni joint representations would be a step toward co-processing
more diverse multimodal information. In this work, we present OmniBind,
large-scale multimodal joint representation models ranging in scale from 7
billion to 30 billion parameters, which support 3D, audio, image, and language
inputs. Due to the scarcity of data pairs across all modalities, instead of
training large models from scratch, we propose remapping and binding the spaces
of various pre-trained specialist models together. This approach enables
"scaling up" by indirectly increasing the model parameters and the amount of
seen data. To effectively integrate various spaces, we dynamically assign
weights to different spaces by learning routers with two objectives:
cross-modal overall alignment and language representation decoupling. Notably,
since binding and routing spaces both only require lightweight networks,
OmniBind is extremely training-efficient. Learning the largest 30B model
requires merely unpaired unimodal data and approximately 3 days on a single
8-4090 node. Extensive experiments demonstrate the versatility and superiority
of OmniBind as an omni representation model, highlighting its great potential
for diverse applications, such as any-query and composable multimodal
understanding.
Sibyl: Simple yet Effective Agent Framework for Complex Real-world
  Reasoning
Authors: name: Yulong Wang, name: Tianhao Shen, name: Lifeng Liu, name: Jian Xie
Published: 2024-07-15 13:45:40

Existing agents based on large language models (LLMs) demonstrate robust
problem-solving capabilities by integrating LLMs' inherent knowledge, strong
in-context learning and zero-shot capabilities, and the use of tools combined
with intricately designed LLM invocation workflows by humans. However, these
agents still exhibit shortcomings in long-term reasoning and under-use the
potential of existing tools, leading to noticeable deficiencies in complex
real-world reasoning scenarios. To address these limitations, we introduce
Sibyl, a simple yet powerful LLM-based agent framework designed to tackle
complex reasoning tasks by efficiently leveraging a minimal set of tools.
Drawing inspiration from Global Workspace Theory, Sibyl incorporates a global
workspace to enhance the management and sharing of knowledge and conversation
history throughout the system. Furthermore, guided by Society of Mind Theory,
Sibyl implements a multi-agent debate-based jury to self-refine the final
answers, ensuring a comprehensive and balanced approach. This approach aims to
reduce system complexity while expanding the scope of problems solvable-from
matters typically resolved by humans in minutes to those requiring hours or
even days, thus facilitating a shift from System-1 to System-2 thinking. Sibyl
has been designed with a focus on scalability and ease of debugging by
incorporating the concept of reentrancy from functional programming from its
inception, with the aim of seamless and low effort integration in other LLM
applications to improve capabilities. Our experimental results on the GAIA
benchmark test set reveal that the Sibyl agent instantiated with GPT-4 achieves
state-of-the-art performance with an average score of 34.55%, compared to other
agents based on GPT-4. We hope that Sibyl can inspire more reliable and
reusable LLM-based agent solutions to address complex real-world reasoning
tasks.
Efficient Training with Denoised Neural Weights
Authors: name: Yifan Gong, name: Zheng Zhan, name: Yanyu Li, name: Yerlan Idelbayev, name: Andrey Zharkov, name: Kfir Aberman, name: Sergey Tulyakov, name: Yanzhi Wang, name: Jian Ren
Published: 2024-07-16 17:59:42

Good weight initialization serves as an effective measure to reduce the
training cost of a deep neural network (DNN) model. The choice of how to
initialize parameters is challenging and may require manual tuning, which can
be time-consuming and prone to human error. To overcome such limitations, this
work takes a novel step towards building a weight generator to synthesize the
neural weights for initialization. We use the image-to-image translation task
with generative adversarial networks (GANs) as an example due to the ease of
collecting model weights spanning a wide range. Specifically, we first collect
a dataset with various image editing concepts and their corresponding trained
weights, which are later used for the training of the weight generator. To
address the different characteristics among layers and the substantial number
of weights to be predicted, we divide the weights into equal-sized blocks and
assign each block an index. Subsequently, a diffusion model is trained with
such a dataset using both text conditions of the concept and the block indexes.
By initializing the image translation model with the denoised weights predicted
by our diffusion model, the training requires only 43.3 seconds. Compared to
training from scratch (i.e., Pix2pix), we achieve a 15x training time
acceleration for a new concept while obtaining even better image generation
quality.
NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context
  Window?
Authors: name: Mo Li, name: Songyang Zhang, name: Yunxin Liu, name: Kai Chen
Published: 2024-07-16 17:59:06

In evaluating the long-context capabilities of large language models (LLMs),
identifying content relevant to a user's query from original long documents is
a crucial prerequisite for any LLM to answer questions based on long text. We
present NeedleBench, a framework consisting of a series of progressively more
challenging tasks for assessing bilingual long-context capabilities, spanning
multiple length intervals (4k, 8k, 32k, 128k, 200k, 1000k, and beyond) and
different depth ranges, allowing the strategic insertion of critical data
points in different text depth zones to rigorously test the retrieval and
reasoning capabilities of models in diverse contexts. We use the NeedleBench
framework to assess how well the leading open-source models can identify key
information relevant to the question and apply that information to reasoning in
bilingual long texts. Furthermore, we propose the Ancestral Trace Challenge
(ATC) to mimic the complexity of logical reasoning challenges that are likely
to be present in real-world long-context tasks, providing a simple method for
evaluating LLMs in dealing with complex long-context situations. Our results
suggest that current LLMs have significant room for improvement in practical
long-context applications, as they struggle with the complexity of logical
reasoning challenges that are likely to be present in real-world long-context
tasks. All codes and resources are available at OpenCompass:
https://github.com/open-compass/opencompass.
Data-Juicer Sandbox: A Comprehensive Suite for Multimodal Data-Model
  Co-development
Authors: name: Daoyuan Chen, name: Haibin Wang, name: Yilun Huang, name: Ce Ge, name: Yaliang Li, name: Bolin Ding, name: Jingren Zhou
Published: 2024-07-16 14:40:07

The emergence of large-scale multi-modal generative models has drastically
advanced artificial intelligence, introducing unprecedented levels of
performance and functionality. However, optimizing these models remains
challenging due to historically isolated paths of model-centric and
data-centric developments, leading to suboptimal outcomes and inefficient
resource utilization. In response, we present a novel sandbox suite tailored
for integrated data-model co-development. This sandbox provides a comprehensive
experimental platform, enabling rapid iteration and insight-driven refinement
of both data and models. Our proposed "Probe-Analyze-Refine" workflow,
validated through applications on state-of-the-art LLaVA-like and DiT based
models, yields significant performance boosts, such as topping the VBench
leaderboard. We also uncover fruitful insights gleaned from exhaustive
benchmarks, shedding light on the critical interplay between data quality,
diversity, and model behavior. With the hope of fostering deeper understanding
and future progress in multi-modal data and generative modeling, our codes,
datasets, and models are maintained and accessible at
https://github.com/modelscope/data-juicer/blob/main/docs/Sandbox.md.