Orionnnn/hub-stats huggingface dataset:
ZipVL: Efficient Large Vision-Language Models with Dynamic Token
  Sparsification and KV Cache Compression
Authors: name: Yefei He, name: Feng Chen, name: Jing Liu, name: Wenqi Shao, name: Hong Zhou, name: Kaipeng Zhang, name: Bohan Zhuang
Published: 2024-10-11 07:24:21

The efficiency of large vision-language models (LVLMs) is constrained by the
computational bottleneck of the attention mechanism during the prefill phase
and the memory bottleneck of fetching the key-value (KV) cache in the decoding
phase, particularly in scenarios involving high-resolution images or videos.
Visual content often exhibits substantial redundancy, resulting in highly
sparse attention maps within LVLMs. This sparsity can be leveraged to
accelerate attention computation or compress the KV cache through various
approaches. However, most studies focus on addressing only one of these
bottlenecks and do not adequately support dynamic adjustment of sparsity
concerning distinct layers or tasks. In this paper, we present ZipVL, an
efficient inference framework designed for LVLMs that resolves both computation
and memory bottlenecks through a dynamic ratio allocation strategy of important
tokens. This ratio is adaptively determined based on the layer-specific
distribution of attention scores, rather than fixed hyper-parameters, thereby
improving efficiency for less complex tasks while maintaining high performance
for more challenging ones. Then we select important tokens based on their
normalized attention scores and perform attention mechanism solely on those
important tokens to accelerate the prefill phase. To mitigate the memory
bottleneck in the decoding phase, we employ mixed-precision quantization to the
KV cache, where high-bit quantization is used for caches of important tokens,
while low-bit quantization is applied to those of less importance. Our
experiments demonstrate that ZipVL can accelerate the prefill phase by
2.6times and reduce GPU memory usage by 50.0%, with a minimal accuracy
reduction of only 0.2% on Video-MME benchmark over LongVA-7B model, effectively
enhancing the generation efficiency of LVLMs.
The Curse of Multi-Modalities: Evaluating Hallucinations of Large
  Multimodal Models across Language, Visual, and Audio
Authors: name: Sicong Leng, name: Yun Xing, name: Zesen Cheng, name: Yang Zhou, name: Hang Zhang, name: Xin Li, name: Deli Zhao, name: Shijian Lu, name: Chunyan Miao, name: Lidong Bing
Published: 2024-10-16 17:59:02

Recent advancements in large multimodal models (LMMs) have significantly
enhanced performance across diverse tasks, with ongoing efforts to further
integrate additional modalities such as video and audio. However, most existing
LMMs remain vulnerable to hallucinations, the discrepancy between the factual
multimodal input and the generated textual output, which has limited their
applicability in various real-world scenarios. This paper presents the first
systematic investigation of hallucinations in LMMs involving the three most
common modalities: language, visual, and audio. Our study reveals two key
contributors to hallucinations: overreliance on unimodal priors and spurious
inter-modality correlations. To address these challenges, we introduce the
benchmark The Curse of Multi-Modalities (CMM), which comprehensively evaluates
hallucinations in LMMs, providing a detailed analysis of their underlying
issues. Our findings highlight key vulnerabilities, including imbalances in
modality integration and biases from training data, underscoring the need for
balanced cross-modal learning and enhanced hallucination mitigation strategies.
Based on our observations and findings, we suggest potential research
directions that could enhance the reliability of LMMs.
HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex
  Diagrams in Coding Tasks
Authors: name: Fengji Zhang, name: Linquan Wu, name: Huiyu Bai, name: Guancheng Lin, name: Xiao Li, name: Xiao Yu, name: Yue Wang, name: Bei Chen, name: Jacky Keung
Published: 2024-10-16 09:04:57

Understanding and reasoning over diagrams is a fundamental aspect of human
intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive
capabilities across various tasks, existing benchmarks lack comprehensive
evaluation of their diagram interpretation and reasoning abilities,
particularly in coding contexts. We present HumanEval-V, a rigorous benchmark
of human-annotated coding tasks that spans six task types and evaluates diverse
visual reasoning capabilities. Each task features carefully crafted diagrams
paired with function signatures and test cases, employing novel code generation
tasks to thoroughly assess models' diagram comprehension. Through extensive
experiments with 22 LMMs, we find that even top-performing models achieve
modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1,
highlighting substantial room for improvement. Our analysis reveals that
current LMMs struggle with spatial transformations, topological relationships,
and dynamic patterns that humans find intuitive. These findings provide
valuable insights for advancing LMMs' visual reasoning abilities. We have
open-sourced our code and benchmark at
https://github.com/HumanEval-V/HumanEval-V-Benchmark.
OMCAT: Omni Context Aware Transformer
Authors: name: Arushi Goel, name: Karan Sapra, name: Matthieu Le, name: Rafael Valle, name: Andrew Tao, name: Bryan Catanzaro
Published: 2024-10-15 23:16:28

Large Language Models (LLMs) have made significant strides in text generation
and comprehension, with recent advancements extending into multimodal LLMs that
integrate visual and audio inputs. However, these models continue to struggle
with fine-grained, cross-modal temporal understanding, particularly when
correlating events across audio and video streams. We address these challenges
with two key contributions: a new dataset and model, called OCTAV and OMCAT
respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset
designed to capture event transitions across audio and video. Second, OMCAT
(Omni Context Aware Transformer) is a powerful model that leverages RoTE
(Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal
grounding and computational efficiency in time-anchored tasks. Through a robust
three-stage training pipeline-feature alignment, instruction tuning, and
OCTAV-specific training-OMCAT excels in cross-modal temporal understanding. Our
model demonstrates state-of-the-art performance on Audio-Visual Question
Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in
temporal reasoning and cross-modal alignment, as validated through
comprehensive experiments and ablation studies. Our dataset and code will be
made publicly available. The link to our demo page is https://om-cat.github.io.
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Authors: name: Jingming Zhuo, name: Songyang Zhang, name: Xinyu Fang, name: Haodong Duan, name: Dahua Lin, name: Kai Chen
Published: 2024-10-16 09:38:13

Large language models (LLMs) have demonstrated impressive capabilities across
various tasks, but their performance is highly sensitive to the prompts
utilized. This variability poses challenges for accurate assessment and user
satisfaction. Current research frequently overlooks instance-level prompt
variations and their implications on subjective evaluations. To address these
shortcomings, we introduce ProSA, a framework designed to evaluate and
comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity
metric, PromptSensiScore, and leverages decoding confidence to elucidate
underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers
that prompt sensitivity fluctuates across datasets and models, with larger
models exhibiting enhanced robustness. We observe that few-shot examples can
alleviate this sensitivity issue, and subjective evaluations are also
susceptible to prompt sensitivities, particularly in complex,
reasoning-oriented tasks. Furthermore, our findings indicate that higher model
confidence correlates with increased prompt robustness. We believe this work
will serve as a helpful tool in studying prompt sensitivity of LLMs. The
project is released at: https://github.com/open-compass/ProSA .
Exploring Model Kinship for Merging Large Language Models
Authors: name: Yedi Hu, name: Yunzhi Yao, name: Ningyu Zhang, name: Shumin Deng, name: Huajun Chen
Published: 2024-10-16 14:29:29

Model merging has become one of the key technologies for enhancing the
capabilities and efficiency of Large Language Models (LLMs). However, our
understanding of the expected performance gains and principles when merging any
two models remains limited. In this work, we introduce model kinship, the
degree of similarity or relatedness between LLMs, analogous to biological
evolution. With comprehensive empirical analysis, we find that there is a
certain relationship between model kinship and the performance gains after
model merging, which can help guide our selection of candidate models. Inspired
by this, we propose a new model merging strategy: Top-k Greedy Merging with
Model Kinship, which can yield better performance on benchmark datasets.
Specifically, we discover that using model kinship as a criterion can assist us
in continuously performing model merging, alleviating the degradation (local
optima) in model evolution, whereas model kinship can serve as a guide to
escape these traps. Code is available at
https://github.com/zjunlp/ModelKinship.
DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse
  Synthetic Data and Global-to-Local Adaptive Perception
Authors: name: Zhiyuan Zhao, name: Hengrui Kang, name: Bin Wang, name: Conghui He
Published: 2024-10-16 14:50:47

Document Layout Analysis is crucial for real-world document understanding
systems, but it encounters a challenging trade-off between speed and accuracy:
multimodal methods leveraging both text and visual features achieve higher
accuracy but suffer from significant latency, whereas unimodal methods relying
solely on visual features offer faster processing speeds at the expense of
accuracy. To address this dilemma, we introduce DocLayout-YOLO, a novel
approach that enhances accuracy while maintaining speed advantages through
document-specific optimizations in both pre-training and model design. For
robust document pre-training, we introduce the Mesh-candidate BestFit
algorithm, which frames document synthesis as a two-dimensional bin packing
problem, generating the large-scale, diverse DocSynth-300K dataset.
Pre-training on the resulting DocSynth-300K dataset significantly improves
fine-tuning performance across various document types. In terms of model
optimization, we propose a Global-to-Local Controllable Receptive Module that
is capable of better handling multi-scale variations of document elements.
Furthermore, to validate performance across different document types, we
introduce a complex and challenging benchmark named DocStructBench. Extensive
experiments on downstream datasets demonstrate that DocLayout-YOLO excels in
both speed and accuracy. Code, data, and models are available at
https://github.com/opendatalab/DocLayout-YOLO.
Improving Long-Text Alignment for Text-to-Image Diffusion Models
Authors: name: Luping Liu, name: Chao Du, name: Tianyu Pang, name: Zehan Wang, name: Chongxuan Li, name: Dong Xu
Published: 2024-10-15 17:46:31

The rapid advancement of text-to-image (T2I) diffusion models has enabled
them to generate unprecedented results from given texts. However, as text
inputs become longer, existing encoding methods like CLIP face limitations, and
aligning the generated images with long texts becomes challenging. To tackle
these issues, we propose LongAlign, which includes a segment-level encoding
method for processing long texts and a decomposed preference optimization
method for effective alignment training. For segment-level encoding, long texts
are divided into multiple segments and processed separately. This method
overcomes the maximum input length limits of pretrained encoding models. For
preference optimization, we provide decomposed CLIP-based preference models to
fine-tune diffusion models. Specifically, to utilize CLIP-based preference
models for T2I alignment, we delve into their scoring mechanisms and find that
the preference scores can be decomposed into two components: a text-relevant
part that measures T2I alignment and a text-irrelevant part that assesses other
visual aspects of human preference. Additionally, we find that the
text-irrelevant part contributes to a common overfitting problem during
fine-tuning. To address this, we propose a reweighting strategy that assigns
different weights to these two components, thereby reducing overfitting and
enhancing alignment. After fine-tuning 512 times 512 Stable Diffusion (SD)
v1.5 for about 20 hours using our method, the fine-tuned SD outperforms
stronger foundation models in T2I alignment, such as PixArt-alpha and
Kandinsky v2.2. The code is available at
https://github.com/luping-liu/LongAlign.
Revealing the Barriers of Language Agents in Planning
Authors: name: Jian Xie, name: Kexun Zhang, name: Jiangjie Chen, name: Siyu Yuan, name: Kai Zhang, name: Yikai Zhang, name: Lei Li, name: Yanghua Xiao
Published: 2024-10-16 09:44:38

Autonomous planning has been an ongoing pursuit since the inception of
artificial intelligence. Based on curated problem solvers, early planning
agents could deliver precise solutions for specific tasks but lacked
generalization. The emergence of large language models (LLMs) and their
powerful reasoning capabilities has reignited interest in autonomous planning
by automatically generating reasonable solutions for given tasks. However,
prior research and our experiments show that current language agents still lack
human-level planning abilities. Even the state-of-the-art reasoning model,
OpenAI o1, achieves only 15.6% on one of the complex real-world planning
benchmarks. This highlights a critical question: What hinders language agents
from achieving human-level planning? Although existing studies have highlighted
weak performance in agent planning, the deeper underlying issues and the
mechanisms and limitations of the strategies proposed to address them remain
insufficiently understood. In this work, we apply the feature attribution study
and identify two key factors that hinder agent planning: the limited role of
constraints and the diminishing influence of questions. We also find that
although current strategies help mitigate these challenges, they do not fully
resolve them, indicating that agents still have a long way to go before
reaching human-level intelligence.
MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video
  Retrieval
Authors: name: Reno Kriz, name: Kate Sanders, name: David Etter, name: Kenton Murray, name: Cameron Carpenter, name: Kelly Van Ochten, name: Hannah Recknor, name: Jimena Guallar-Blasco, name: Alexander Martin, name: Ronald Colaianni, name: Nolan King, name: Eugene Yang, name: Benjamin Van Durme
Published: 2024-10-15 13:56:34

Efficiently retrieving and synthesizing information from large-scale
multimodal collections has become a critical challenge. However, existing video
retrieval datasets suffer from scope limitations, primarily focusing on
matching descriptive but vague queries with small collections of professionally
edited, English-centric videos. To address this gap, we introduce
MultiVENT 2.0, a large-scale, multilingual event-centric video
retrieval benchmark featuring a collection of more than 218,000 news videos and
3,906 queries targeting specific world events. These queries specifically
target information found in the visual content, audio, embedded text, and text
metadata of the videos, requiring systems leverage all these sources to succeed
at the task. Preliminary results show that state-of-the-art vision-language
models struggle significantly with this task, and while alternative approaches
show promise, they are still insufficient to adequately address this problem.
These findings underscore the need for more robust multimodal retrieval
systems, as effective video retrieval is a crucial step towards multimodal
content understanding and generation tasks.
EchoPrime: A Multi-Video View-Informed Vision-Language Model for
  Comprehensive Echocardiography Interpretation
Authors: name: Milos Vukadinovic, name: Xiu Tang, name: Neal Yuan, name: Paul Cheng, name: Debiao Li, name: Susan Cheng, name: Bryan He, name: David Ouyang
Published: 2024-10-13 03:04:22

Echocardiography is the most widely used cardiac imaging modality, capturing
ultrasound video data to assess cardiac structure and function. Artificial
intelligence (AI) in echocardiography has the potential to streamline manual
tasks and improve reproducibility and precision. However, most echocardiography
AI models are single-view, single-task systems that do not synthesize
complementary information from multiple views captured during a full exam, and
thus lead to limited performance and scope of applications. To address this
problem, we introduce EchoPrime, a multi-view, view-informed, video-based
vision-language foundation model trained on over 12 million video-report pairs.
EchoPrime uses contrastive learning to train a unified embedding model for all
standard views in a comprehensive echocardiogram study with representation of
both rare and common diseases and diagnoses. EchoPrime then utilizes
view-classification and a view-informed anatomic attention model to weight
video-specific interpretations that accurately maps the relationship between
echocardiographic views and anatomical structures. With retrieval-augmented
interpretation, EchoPrime integrates information from all echocardiogram videos
in a comprehensive study and performs holistic comprehensive clinical
echocardiography interpretation. In datasets from two independent healthcare
systems, EchoPrime achieves state-of-the art performance on 23 diverse
benchmarks of cardiac form and function, surpassing the performance of both
task-specific approaches and prior foundation models. Following rigorous
clinical evaluation, EchoPrime can assist physicians in the automated
preliminary assessment of comprehensive echocardiography.
NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of
  Large Language Models
Authors: name: Han Han, name: Tong Zhu, name: Xiang Zhang, name: Mengsong Wu, name: Hao Xiong, name: Wenliang Chen
Published: 2024-10-15 17:33:43

Large language models (LLMs) combined with tool learning have gained
impressive results in real-world applications. During tool learning, LLMs may
call multiple tools in nested orders, where the latter tool call may take the
former response as its input parameters. However, current research on the
nested tool learning capabilities is still under-explored, since the existing
benchmarks lack of relevant data instances. To address this problem, we
introduce NesTools to bridge the current gap in comprehensive nested tool
learning evaluations. NesTools comprises a novel automatic data generation
method to construct large-scale nested tool calls with different nesting
structures. With manual review and refinement, the dataset is in high quality
and closely aligned with real-world scenarios. Therefore, NesTools can serve as
a new benchmark to evaluate the nested tool learning abilities of LLMs. We
conduct extensive experiments on 22 LLMs, and provide in-depth analyses with
NesTools, which shows that current LLMs still suffer from the complex nested
tool learning task.
Agent-as-a-Judge: Evaluate Agents with Agents
Authors: name: Mingchen Zhuge, name: Changsheng Zhao, name: Dylan Ashley, name: Wenyi Wang, name: Dmitrii Khizbullin, name: Yunyang Xiong, name: Zechun Liu, name: Ernie Chang, name: Raghuraman Krishnamoorthi, name: Yuandong Tian, name: Yangyang Shi, name: Vikas Chandra, name: Jürgen Schmidhuber
Published: 2024-10-14 17:57:02

Contemporary evaluation techniques are inadequate for agentic systems. These
approaches either focus exclusively on final outcomes -- ignoring the
step-by-step nature of agentic systems, or require excessive manual labour. To
address this, we introduce the Agent-as-a-Judge framework, wherein agentic
systems are used to evaluate agentic systems. This is an organic extension of
the LLM-as-a-Judge framework, incorporating agentic features that enable
intermediate feedback for the entire task-solving process. We apply the
Agent-as-a-Judge to the task of code generation. To overcome issues with
existing benchmarks and provide a proof-of-concept testbed for
Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated
AI development tasks. It includes rich manual annotations, like a total of 365
hierarchical user requirements. We benchmark three of the popular agentic
systems using Agent-as-a-Judge and find it dramatically outperforms
LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether,
we believe that Agent-as-a-Judge marks a concrete step forward for modern
agentic systems -- by providing rich and reliable reward signals necessary for
dynamic and scalable self-improvement.
Towards Natural Image Matting in the Wild via Real-Scenario Prior
Authors: name: Ruihao Xia, name: Yu Liang, name: Peng-Tao Jiang, name: Hao Zhang, name: Qianru Sun, name: Yang Tang, name: Bo Li, name: Pan Zhou
Published: 2024-10-09 06:43:19

Recent approaches attempt to adapt powerful interactive segmentation models,
such as SAM, to interactive matting and fine-tune the models based on synthetic
matting datasets. However, models trained on synthetic data fail to generalize
to complex and occlusion scenes. We address this challenge by proposing a new
matting dataset based on the COCO dataset, namely COCO-Matting. Specifically,
the construction of our COCO-Matting includes accessory fusion and
mask-to-matte, which selects real-world complex images from COCO and converts
semantic segmentation masks to matting labels. The built COCO-Matting comprises
an extensive collection of 38,251 human instance-level alpha mattes in complex
natural scenarios. Furthermore, existing SAM-based matting methods extract
intermediate features and masks from a frozen SAM and only train a lightweight
matting decoder by end-to-end matting losses, which do not fully exploit the
potential of the pre-trained SAM. Thus, we propose SEMat which revamps the
network architecture and training objectives. For network architecture, the
proposed feature-aligned transformer learns to extract fine-grained edge and
transparency features. The proposed matte-aligned decoder aims to segment
matting-specific objects and convert coarse masks into high-precision mattes.
For training objectives, the proposed regularization and trimap loss aim to
retain the prior from the pre-trained model and push the matting logits
extracted from the mask decoder to contain trimap-based semantic information.
Extensive experiments across seven diverse datasets demonstrate the superior
performance of our method, proving its efficacy in interactive natural image
matting. We open-source our code, models, and dataset at
https://github.com/XiaRho/SEMat.