Orionnnn/hub-stats huggingface dataset:
DuoAttention: Efficient Long-Context LLM Inference with Retrieval and
  Streaming Heads
Authors: name: Guangxuan Xiao, name: Jiaming Tang, name: Jingwei Zuo, name: Junxian Guo, name: Shang Yang, name: Haotian Tang, name: Yao Fu, name: Song Han
Published: 2024-10-14 17:59:58

Deploying long-context large language models (LLMs) is essential but poses
significant computational and memory challenges. Caching all Key and Value (KV)
states across all attention heads consumes substantial memory. Existing KV
cache pruning methods either damage the long-context capabilities of LLMs or
offer only limited efficiency improvements. In this paper, we identify that
only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for
processing long contexts and require full attention across all tokens. In
contrast, all other heads, which primarily focus on recent tokens and attention
sinks--referred to as Streaming Heads--do not require full attention. Based on
this insight, we introduce DuoAttention, a framework that only applies a full
KV cache to retrieval heads while using a light-weight, constant-length KV
cache for streaming heads, which reduces both LLM's decoding and pre-filling
memory and latency without compromising its long-context abilities.
DuoAttention uses a lightweight, optimization-based algorithm with synthetic
data to identify retrieval heads accurately. Our method significantly reduces
long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models
while speeding up decoding by up to 2.18x and 1.50x and accelerating
pre-filling by up to 1.73x and 1.63x for MHA and GQA models, respectively, with
minimal accuracy loss compared to full attention. Notably, combined with
quantization, DuoAttention enables Llama-3-8B decoding with 3.3 million context
length on a single A100 GPU. Code is provided in
https://github.com/mit-han-lab/duo-attention.
ReLU's Revival: On the Entropic Overload in Normalization-Free Large
  Language Models
Authors: name: Nandan Kumar Jha, name: Brandon Reagen
Published: 2024-10-12 20:26:01

LayerNorm is a critical component in modern large language models (LLMs) for
stabilizing training and ensuring smooth optimization. However, it introduces
significant challenges in mechanistic interpretability, outlier feature
suppression, faithful signal propagation, and computational and communication
complexity of private inference. This work explores desirable activation
functions in normalization-free decoder-only LLMs. Contrary to the conventional
preference for the GELU in transformer-based models, our empirical findings
demonstrate an {\em opposite trend} -- ReLU significantly outperforms GELU in
LayerNorm-free models, leading to an {\bf 8.2\%} perplexity improvement. We
discover a key issue with GELU, where early layers experience entropic
overload, leading to the under-utilization of the representational capacity of
attention heads. This highlights that smoother activations like GELU are {\em
ill-suited} for LayerNorm-free architectures, whereas ReLU's geometrical
properties -- specialization in input space and intra-class selectivity -- lead
to improved learning dynamics and better information retention in the absence
of LayerNorm. This study offers key insights for optimizing transformer
architectures where LayerNorm introduces significant challenges.
The Same But Different: Structural Similarities and Differences in
  Multilingual Language Modeling
Authors: name: Ruochen Zhang, name: Qinan Yu, name: Matianyu Zang, name: Carsten Eickhoff, name: Ellie Pavlick
Published: 2024-10-11 19:57:55

We employ new tools from mechanistic interpretability in order to ask whether
the internal structure of large language models (LLMs) shows correspondence to
the linguistic structures which underlie the languages on which they are
trained. In particular, we ask (1) when two languages employ the same
morphosyntactic processes, do LLMs handle them using shared internal circuitry?
and (2) when two languages require different morphosyntactic processes, do LLMs
handle them using different internal circuitry? Using English and Chinese
multilingual and monolingual models, we analyze the internal circuitry involved
in two tasks. We find evidence that models employ the same circuit to handle
the same syntactic process independently of the language in which it occurs,
and that this is the case even for monolingual models trained completely
independently. Moreover, we show that multilingual models employ
language-specific components (attention heads and feed-forward networks) when
needed to handle linguistic processes (e.g., morphological marking) that only
exist in some languages. Together, our results provide new insights into how
LLMs trade off between exploiting common structures and preserving linguistic
differences when tasked with modeling multiple languages simultaneously.
VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality
  Documents
Authors: name: Shi Yu, name: Chaoyue Tang, name: Bokai Xu, name: Junbo Cui, name: Junhao Ran, name: Yukun Yan, name: Zhenghao Liu, name: Shuo Wang, name: Xu Han, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-10-14 15:04:18

Retrieval-augmented generation (RAG) is an effective technique that enables
large language models (LLMs) to utilize external knowledge sources for
generation. However, current RAG systems are solely based on text, rendering it
impossible to utilize vision information like layout and images that play
crucial roles in real-world multi-modality documents. In this paper, we
introduce VisRAG, which tackles this issue by establishing a vision-language
model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the
document to obtain text, the document is directly embedded using a VLM as an
image and then retrieved to enhance the generation of a VLM. Compared to
traditional text-based RAG, VisRAG maximizes the retention and utilization of
the data information in the original documents, eliminating the information
loss introduced during the parsing process. We collect both open-source and
synthetic data to train the retriever in VisRAG and explore a variety of
generation methods. Experiments demonstrate that VisRAG outperforms traditional
RAG in both the retrieval and generation stages, achieving a 25--39\%
end-to-end performance gain over traditional text-based RAG pipeline. Further
analysis reveals that VisRAG is effective in utilizing training data and
demonstrates strong generalization capability, positioning it as a promising
solution for RAG on multi-modality documents. Our code and data are available
at https://github.com/openbmb/visrag .
Thinking LLMs: General Instruction Following with Thought Generation
Authors: name: Tianhao Wu, name: Janice Lan, name: Weizhe Yuan, name: Jiantao Jiao, name: Jason Weston, name: Sainbayar Sukhbaatar
Published: 2024-10-14 15:38:56

LLMs are typically trained to answer user questions or follow instructions
similarly to how human experts respond. However, in the standard alignment
framework they lack the basic ability of explicit thinking before answering.
Thinking is important for complex questions that require reasoning and planning
-- but can be applied to any task. We propose a training method for equipping
existing LLMs with such thinking abilities for general instruction following
without use of additional human data. We achieve this by an iterative search
and optimization procedure that explores the space of possible thought
generations, allowing the model to learn how to think without direct
supervision. For each instruction, the thought candidates are scored using a
judge model to evaluate their responses only, and then optimized via preference
optimization. We show that this procedure leads to superior performance on
AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning
categories such as marketing, health and general knowledge, in addition to more
traditional reasoning & problem-solving tasks.
MMCOMPOSITION: Revisiting the Compositionality of Pre-trained
  Vision-Language Models
Authors: name: Hang Hua, name: Yunlong Tang, name: Ziyun Zeng, name: Liangliang Cao, name: Zhengyuan Yang, name: Hangfeng He, name: Chenliang Xu, name: Jiebo Luo
Published: 2024-10-13 05:35:09

The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal understanding, enabling more sophisticated and accurate integration
of visual and textual information across various tasks, including image and
video captioning, visual question answering, and cross-modal retrieval. Despite
VLMs' superior capabilities, researchers lack a comprehensive understanding of
their compositionality -- the ability to understand and produce novel
combinations of known visual and textual components. Prior benchmarks provide
only a relatively rough compositionality evaluation from the perspectives of
objects, relations, and attributes while neglecting deeper reasoning about
object interactions, counting, and complex compositions. However,
compositionality is a critical ability that facilitates coherent reasoning and
understanding across modalities for VLMs. To address this limitation, we
propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively
and accurately evaluating VLMs' compositionality. Our proposed benchmark serves
as a complement to these earlier works. With MMCOMPOSITION, we can quantify and
explore the compositionality of the mainstream VLMs. Surprisingly, we find
GPT-4o's compositionality inferior to the best open-source model, and we
analyze the underlying reasons. Our experimental analysis reveals the
limitations of VLMs in fine-grained compositional perception and reasoning, and
points to areas for improvement in VLM design and training. Resources available
at: https://hanghuacs.github.io/MMComposition/
LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
Authors: name: Nimrod Shabtay, name: Felipe Maia Polo, name: Sivan Doveh, name: Wei Lin, name: M. Jehanzeb Mirza, name: Leshem Chosen, name: Mikhail Yurochkin, name: Yuekai Sun, name: Assaf Arbelle, name: Leonid Karlinsky, name: Raja Giryes
Published: 2024-10-14 17:51:23

The large-scale training of multi-modal models on data scraped from the web
has shown outstanding utility in infusing these models with the required world
knowledge to perform effectively on multiple downstream tasks. However, one
downside of scraping data from the web can be the potential sacrifice of the
benchmarks on which the abilities of these models are often evaluated. To
safeguard against test data contamination and to truly test the abilities of
these foundation models we propose LiveXiv: A scalable evolving live benchmark
based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts
at any given timestamp and proposes to automatically generate visual
question-answer pairs (VQA). This is done without any human-in-the-loop, using
the multi-modal content in the manuscripts, like graphs, charts, and tables.
Moreover, we introduce an efficient evaluation approach that estimates the
performance of all models on the evolving benchmark using evaluations of only a
subset of models. This significantly reduces the overall evaluation cost. We
benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the
first version of our benchmark, showing its challenging nature and exposing the
models true abilities, avoiding contamination. Lastly, in our commitment to
high quality, we have collected and evaluated a manually verified subset. By
comparing its overall results to our automatic annotations, we have found that
the performance variance is indeed minimal (<2.5%). Our dataset is available
online on HuggingFace, and our code will be available here.
Tree of Problems: Improving structured problem solving with
  compositionality
Authors: name: Armel Zebaze, name: Benoît Sagot, name: Rachel Bawden
Published: 2024-10-09 07:35:46

Large Language Models (LLMs) have demonstrated remarkable performance across
multiple tasks through in-context learning. For complex reasoning tasks that
require step-by-step thinking, Chain-of-Thought (CoT) prompting has given
impressive results, especially when combined with self-consistency.
Nonetheless, some tasks remain particularly difficult for LLMs to solve. Tree
of Thoughts (ToT) and Graph of Thoughts (GoT) emerged as alternatives, dividing
the complex problem into paths of subproblems. In this paper, we propose Tree
of Problems (ToP), a simpler version of ToT, which we hypothesise can work
better for complex tasks that can be divided into identical subtasks. Our
empirical results show that our approach outperforms ToT and GoT, and in
addition performs better than CoT on complex reasoning tasks. All code for this
paper is publicly available here:
https://github.com/ArmelRandy/tree-of-problems.
Semantic Image Inversion and Editing using Rectified Stochastic
  Differential Equations
Authors: name: Litu Rout, name: Yujia Chen, name: Nataniel Ruiz, name: Constantine Caramanis, name: Sanjay Shakkottai, name: Wen-Sheng Chu
Published: 2024-10-14 17:56:24

Generative models transform random noise into images; their inversion aims to
transform images back to structured noise for recovery and editing. This paper
addresses two key tasks: (i) inversion and (ii) editing of a real image using
stochastic equivalents of rectified flow models (such as Flux). Although
Diffusion Models (DMs) have recently dominated the field of generative modeling
for images, their inversion presents faithfulness and editability challenges
due to nonlinearities in drift and diffusion. Existing state-of-the-art DM
inversion approaches rely on training of additional parameters or test-time
optimization of latent variables; both are expensive in practice. Rectified
Flows (RFs) offer a promising alternative to diffusion models, yet their
inversion has been underexplored. We propose RF inversion using dynamic optimal
control derived via a linear quadratic regulator. We prove that the resulting
vector field is equivalent to a rectified stochastic differential equation.
Additionally, we extend our framework to design a stochastic sampler for Flux.
Our inversion method allows for state-of-the-art performance in zero-shot
inversion and editing, outperforming prior works in stroke-to-image synthesis
and semantic image editing, with large-scale human evaluations confirming user
preference.
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive
  Memory
Authors: name: Di Wu, name: Hongwei Wang, name: Wenhao Yu, name: Yuwei Zhang, name: Kai-Wei Chang, name: Dong Yu
Published: 2024-10-14 17:59:44

Recent large language model (LLM)-driven chat assistant systems have
integrated memory components to track user-assistant chat histories, enabling
more accurate and personalized responses. However, their long-term memory
capabilities in sustained interactions remain underexplored. This paper
introduces LongMemEval, a comprehensive benchmark designed to evaluate five
core long-term memory abilities of chat assistants: information extraction,
multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
With 500 meticulously curated questions embedded within freely scalable
user-assistant chat histories, LongMemEval presents a significant challenge to
existing long-term memory systems, with commercial chat assistants and
long-context LLMs showing 30% accuracy drop on memorizing information across
sustained interactions. We then present a unified framework that breaks down
the long-term memory design into four design choices across the indexing,
retrieval, and reading stages. Built upon key experimental insights, we propose
several memory designs including session decomposition for optimizing value
granularity, fact-augmented key expansion for enhancing the index structure,
and time-aware query expansion for refining the search scope. Experiment
results show that these optimizations greatly improve both memory recall and
downstream question answering on LongMemEval. Overall, our study provides
valuable resources and guidance for advancing the long-term memory capabilities
of LLM-based chat assistants, paving the way toward more personalized and
reliable conversational AI.
TemporalBench: Benchmarking Fine-grained Temporal Understanding for
  Multimodal Video Models
Authors: name: Mu Cai, name: Reuben Tan, name: Jianrui Zhang, name: Bocheng Zou, name: Kai Zhang, name: Feng Yao, name: Fangrui Zhu, name: Jing Gu, name: Yiwu Zhong, name: Yuzhang Shang, name: Yao Dou, name: Jaden Park, name: Jianfeng Gao, name: Yong Jae Lee, name: Jianwei Yang
Published: 2024-10-14 17:59:58

Understanding fine-grained temporal dynamics is crucial for multimodal video
comprehension and generation. Due to the lack of fine-grained temporal
annotations, existing video benchmarks mostly resemble static image benchmarks
and are incompetent at evaluating models for temporal understanding. In this
paper, we introduce TemporalBench, a new benchmark dedicated to evaluating
fine-grained temporal understanding in videos. TemporalBench consists of ~10K
video question-answer pairs, derived from ~2K high-quality human annotations
detailing the temporal dynamics in video clips. As a result, our benchmark
provides a unique testbed for evaluating various temporal understanding and
reasoning abilities such as action frequency, motion magnitude, event order,
etc. Moreover, it enables evaluations on various tasks like both video question
answering and captioning, both short and long video understanding, as well as
different models such as multimodal video embedding models and text generation
models. Results show that state-of-the-art models like GPT-4o achieve only
38.5% question answering accuracy on TemporalBench, demonstrating a significant
gap (~30%) between humans and AI in temporal understanding. Furthermore, we
notice a critical pitfall for multi-choice QA where LLMs can detect the subtle
changes in negative captions and find a centralized description as a cue for
its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such
bias. We hope that TemporalBench can foster research on improving models'
temporal reasoning capabilities. Both dataset and evaluation code will be made
available.
Generalizable Humanoid Manipulation with Improved 3D Diffusion Policies
Authors: name: Yanjie Ze, name: Zixuan Chen, name: Wenhao Wang, name: Tianyi Chen, name: Xialin He, name: Ying Yuan, name: Xue Bin Peng, name: Jiajun Wu
Published: 2024-10-14 17:59:00

Humanoid robots capable of autonomous operation in diverse environments have
long been a goal for roboticists. However, autonomous manipulation by humanoid
robots has largely been restricted to one specific scene, primarily due to the
difficulty of acquiring generalizable skills. Recent advances in 3D visuomotor
policies, such as the 3D Diffusion Policy (DP3), have shown promise in
extending these capabilities to wilder environments. However, 3D visuomotor
policies often rely on camera calibration and point-cloud segmentation, which
present challenges for deployment on mobile robots like humanoids. In this
work, we introduce the Improved 3D Diffusion Policy (iDP3), a novel 3D
visuomotor policy that eliminates these constraints by leveraging egocentric 3D
visual representations. We demonstrate that iDP3 enables a full-sized humanoid
robot to autonomously perform skills in diverse real-world scenarios, using
only data collected in the lab. Videos are available at:
https://humanoid-manipulation.github.io
Cavia: Camera-controllable Multi-view Video Diffusion with
  View-Integrated Attention
Authors: name: Dejia Xu, name: Yifan Jiang, name: Chen Huang, name: Liangchen Song, name: Thorsten Gernoth, name: Liangliang Cao, name: Zhangyang Wang, name: Hao Tang
Published: 2024-10-14 17:46:32

In recent years there have been remarkable breakthroughs in image-to-video
generation. However, the 3D consistency and camera controllability of generated
frames have remained unsolved. Recent studies have attempted to incorporate
camera control into the generation process, but their results are often limited
to simple trajectories or lack the ability to generate consistent videos from
multiple distinct camera paths for the same scene. To address these
limitations, we introduce Cavia, a novel framework for camera-controllable,
multi-view video generation, capable of converting an input image into multiple
spatiotemporally consistent videos. Our framework extends the spatial and
temporal attention modules into view-integrated attention modules, improving
both viewpoint and temporal consistency. This flexible design allows for joint
training with diverse curated data sources, including scene-level static
videos, object-level synthetic multi-view dynamic videos, and real-world
monocular dynamic videos. To our best knowledge, Cavia is the first of its kind
that allows the user to precisely specify camera motion while obtaining object
motion. Extensive experiments demonstrate that Cavia surpasses state-of-the-art
methods in terms of geometric consistency and perceptual quality. Project Page:
https://ir1d.github.io/Cavia/
Animate-X: Universal Character Image Animation with Enhanced Motion
  Representation
Authors: name: Shuai Tan, name: Biao Gong, name: Xiang Wang, name: Shiwei Zhang, name: Dandan Zheng, name: Ruobing Zheng, name: Kecheng Zheng, name: Jingdong Chen, name: Ming Yang
Published: 2024-10-14 09:06:55

Character image animation, which generates high-quality videos from a
reference image and target pose sequence, has seen significant progress in
recent years. However, most existing methods only apply to human figures, which
usually do not generalize well on anthropomorphic characters commonly used in
industries like gaming and entertainment. Our in-depth analysis suggests to
attribute this limitation to their insufficient modeling of motion, which is
unable to comprehend the movement pattern of the driving video, thus imposing a
pose sequence rigidly onto the target character. To this end, this paper
proposes Animate-X, a universal animation framework based on LDM for various
character types (collectively named X), including anthropomorphic characters.
To enhance motion representation, we introduce the Pose Indicator, which
captures comprehensive motion pattern from the driving video through both
implicit and explicit manner. The former leverages CLIP visual features of a
driving video to extract its gist of motion, like the overall movement pattern
and temporal relations among motions, while the latter strengthens the
generalization of LDM by simulating possible inputs in advance that may arise
during inference. Moreover, we introduce a new Animated Anthropomorphic
Benchmark (A^2Bench) to evaluate the performance of Animate-X on universal and
widely applicable animation images. Extensive experiments demonstrate the
superiority and effectiveness of Animate-X compared to state-of-the-art
methods.