Orionnnn/hub-stats huggingface dataset:
Mitigating Object Hallucination via Concentric Causal Attention
Authors: name: Yun Xing, name: Yiheng Li, name: Ivan Laptev, name: Shijian Lu
Published: 2024-10-21 11:54:53

Recent Large Vision Language Models (LVLMs) present remarkable zero-shot
conversational and reasoning capabilities given multimodal queries.
Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs
are prone to generate textual responses not factually aligned with image
inputs. Our pilot study reveals that object hallucination is closely tied with
Rotary Position Encoding (RoPE), a widely adopted positional dependency
modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs
tend to hallucinate more when relevant visual cues are distant from instruction
tokens in the multimodal input sequence. Additionally, we observe a similar
effect when reversing the sequential order of visual tokens during multimodal
alignment. Our tests indicate that long-term decay in RoPE poses challenges to
LVLMs while capturing visual-instruction interactions across long distances. We
propose Concentric Causal Attention (CCA), a simple yet effective positional
alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs
by naturally reducing relative distance between visual and instruction tokens.
With CCA, visual tokens can better interact with instruction tokens, thereby
enhancing model's perception capability and alleviating object hallucination.
Without bells and whistles, our positional alignment method surpasses existing
hallucination mitigation strategies by large margins on multiple object
hallucination benchmarks.
Aligning Large Language Models via Self-Steering Optimization
Authors: name: Hao Xiang, name: Bowen Yu, name: Hongyu Lin, name: Keming Lu, name: Yaojie Lu, name: Xianpei Han, name: Le Sun, name: Jingren Zhou, name: Junyang Lin
Published: 2024-10-22 16:04:03

Automated alignment develops alignment systems with minimal human
intervention. The key to automated alignment lies in providing learnable and
accurate preference signals for preference learning without human annotation.
In this paper, we introduce Self-Steering Optimization (SSO), an algorithm
that autonomously generates high-quality preference signals based on predefined
principles during iterative training, eliminating the need for manual
annotation. SSO maintains the accuracy of signals by ensuring a consistent
gap between chosen and rejected responses while keeping them both on-policy to
suit the current policy model's learning capacity. SSO can benefit the online
and offline training of the policy model, as well as enhance the training of
reward models. We validate the effectiveness of SSO with two foundation
models, Qwen2 and Llama3.1, indicating that it provides accurate, on-policy
preference signals throughout iterative training. Without any manual annotation
or external models, SSO leads to significant performance improvements across
six subjective or objective benchmarks. Besides, the preference data generated
by SSO significantly enhanced the performance of the reward model on
Rewardbench. Our work presents a scalable approach to preference optimization,
paving the way for more efficient and effective automated alignment.
SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes
Authors: name: Cheng-De Fan, name: Chen-Wei Chang, name: Yi-Ruei Liu, name: Jie-Ying Lee, name: Jiun-Long Huang, name: Yu-Chee Tseng, name: Yu-Lun Liu
Published: 2024-10-22 17:59:56

We present SpectroMotion, a novel approach that combines 3D Gaussian
Splatting (3DGS) with physically-based rendering (PBR) and deformation fields
to reconstruct dynamic specular scenes. Previous methods extending 3DGS to
model dynamic scenes have struggled to accurately represent specular surfaces.
Our method addresses this limitation by introducing a residual correction
technique for accurate surface normal computation during deformation,
complemented by a deformable environment map that adapts to time-varying
lighting conditions. We implement a coarse-to-fine training strategy that
significantly enhances both scene geometry and specular color prediction. We
demonstrate that our model outperforms prior methods for view synthesis of
scenes containing dynamic specular objects and that it is the only existing
3DGS method capable of synthesizing photorealistic real-world dynamic specular
scenes, outperforming state-of-the-art methods in rendering complex, dynamic,
and specular scenes.
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid
  Visual Redundancy Reduction
Authors: name: Long Xing, name: Qidong Huang, name: Xiaoyi Dong, name: Jiajie Lu, name: Pan Zhang, name: Yuhang Zang, name: Yuhang Cao, name: Conghui He, name: Jiaqi Wang, name: Feng Wu, name: Dahua Lin
Published: 2024-10-22 17:59:53

In large vision-language models (LVLMs), images serve as inputs that carry a
wealth of information. As the idiom "A picture is worth a thousand words"
implies, representing a single image in current LVLMs can require hundreds or
even thousands of tokens. This results in significant computational costs,
which grow quadratically as input image resolution increases, thereby severely
impacting the efficiency of both training and inference. Previous approaches
have attempted to reduce the number of image tokens either before or within the
early layers of LVLMs. However, these strategies inevitably result in the loss
of crucial image information, ultimately diminishing model performance. To
address this challenge, we conduct an empirical study revealing that all visual
tokens are necessary for LVLMs in the shallow layers, and token redundancy
progressively increases in the deeper layers of the model. To this end, we
propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost
their efficiency in both training and inference with neglectable performance
loss. Specifically, we partition the LVLM into several stages and drop part of
the image tokens at the end of each stage with a pre-defined ratio, creating
pyramid-like visual tokens across model layers. The dropping is based on a
lightweight similarity calculation with a negligible time overhead. Extensive
experiments demonstrate that PyramidDrop can achieve a 40% training time and
55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance.
Besides, the PyramidDrop could also serve as a plug-and-play strategy for
inference acceleration without training, with better performance and lower
inference cost than counterparts. We hope that the insights and approach
introduced by PyramidDrop will inspire future research to further investigate
the role of image tokens in LVLMs.
JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding
  Benchmark for Culture-aware Evaluation
Authors: name: Shota Onohara, name: Atsuyuki Miyai, name: Yuki Imajuku, name: Kazuki Egashira, name: Jeonghun Baek, name: Xiang Yue, name: Graham Neubig, name: Kiyoharu Aizawa
Published: 2024-10-22 17:59:56

Accelerating research on Large Multimodal Models (LMMs) in non-English
languages is crucial for enhancing user experiences across broader populations.
In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale
Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the
Japanese cultural context. To facilitate comprehensive culture-aware
evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA)
subset, where the culture-independent subjects (e.g., Math) are selected and
translated into Japanese, enabling one-to-one comparison with its English
counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly
crafted subjects that reflect Japanese cultural context. Using the CA subset,
we observe performance drop in many LMMs when evaluated in Japanese, which is
purely attributable to language variation. Using the CS subset, we reveal their
inadequate Japanese cultural understanding. Further, by combining both subsets,
we identify that some LMMs perform well on the CA subset but not on the CS
subset, exposing a shallow understanding of the Japanese language that lacks
depth in cultural understanding. We hope this work will not only help advance
LMM performance in Japanese but also serve as a guideline to create
high-standard, culturally diverse benchmarks for multilingual LMM development.
The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.
EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary
  Search
Authors: name: Oliver Sieberling, name: Denis Kuznedelev, name: Eldar Kurtic, name: Dan Alistarh
Published: 2024-10-18 17:46:37

The high computational costs of large language models (LLMs) have led to a
flurry of research on LLM compression, via methods such as quantization,
sparsification, or structured pruning. A new frontier in this area is given by
dynamic, non-uniform compression methods, which adjust the compression
levels (e.g., sparsity) per-block or even per-layer in order to minimize
accuracy loss, while guaranteeing a global compression threshold. Yet, current
methods rely on heuristics for identifying the "importance" of a given layer
towards the loss, based on assumptions such as error monotonicity, i.e.
that the end-to-end model compression error is proportional to the sum of
layer-wise errors. In this paper, we revisit this area, and propose a new and
general approach for dynamic compression that is provably optimal in a given
input range. We begin from the motivating observation that, in general,
error monotonicity does not hold for LLMs: compressed models with lower
sum of per-layer errors can perform worse than models with higher error
sums. To address this, we propose a new general evolutionary framework for
dynamic LLM compression called EvoPress, which has provable convergence, and
low sample and evaluation complexity. We show that these theoretical guarantees
lead to highly competitive practical performance for dynamic compression of
Llama, Mistral and Phi models. Via EvoPress, we set new state-of-the-art
results across all compression approaches: structural pruning (block/layer
dropping), unstructured sparsity, as well as quantization with dynamic
bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress.
MiniPLM: Knowledge Distillation for Pre-Training Language Models
Authors: name: Yuxian Gu, name: Hao Zhou, name: Fandong Meng, name: Jie Zhou, name: Minlie Huang
Published: 2024-10-22 17:40:32

Knowledge distillation (KD) is widely used to train small, high-performing
student language models (LMs) using large teacher LMs. While effective in
fine-tuning, KD during pre-training faces challenges in efficiency,
flexibility, and effectiveness. Existing methods either incur high
computational costs due to online teacher inference, require tokenization
matching between teacher and student LMs, or risk losing the difficulty and
diversity of the teacher-generated training data. To address these issues, we
propose MiniPLM, a KD framework for pre-training LMs by refining the training
data distribution with the teacher's knowledge. For efficiency, MiniPLM
performs offline teacher LM inference, allowing KD for multiple student LMs
without adding training-time costs. For flexibility, MiniPLM operates solely on
the training corpus, enabling KD across model families. For effectiveness,
MiniPLM leverages the differences between large and small LMs to enhance the
difficulty and diversity of the training data, helping student LMs acquire
versatile and sophisticated knowledge. Extensive experiments demonstrate that
MiniPLM boosts the student LMs' performance on 9 widely used downstream tasks,
improves the language modeling capabilities, and reduces pre-training
computation. The benefit of MiniPLM extends to large pre-training scales,
evidenced by the extrapolation of the scaling curves. Further analysis reveals
that MiniPLM supports KD across model families and enhances the utilization of
pre-training data. Our model, code, and data are available at
https://github.com/thu-coai/MiniPLM.
How Many Van Goghs Does It Take to Van Gogh? Finding the Imitation
  Threshold
Authors: name: Sahil Verma, name: Royi Rassin, name: Arnav Das, name: Gantavya Bhatt, name: Preethi Seshadri, name: Chirag Shah, name: Jeff Bilmes, name: Hannaneh Hajishirzi, name: Yanai Elazar
Published: 2024-10-19 06:28:14

Text-to-image models are trained using large datasets collected by scraping
image-text pairs from the internet. These datasets often include private,
copyrighted, and licensed material. Training models on such datasets enables
them to generate images with such content, which might violate copyright laws
and individual privacy. This phenomenon is termed imitation -- generation of
images with content that has recognizable similarity to its training images. In
this work we study the relationship between a concept's frequency in the
training dataset and the ability of a model to imitate it. We seek to determine
the point at which a model was trained on enough instances to imitate a concept
-- the imitation threshold. We posit this question as a new problem: Finding
the Imitation Threshold (FIT) and propose an efficient approach that estimates
the imitation threshold without incurring the colossal cost of training
multiple models from scratch. We experiment with two domains -- human faces and
art styles -- for which we create four datasets, and evaluate three
text-to-image models which were trained on two pretraining datasets. Our
results reveal that the imitation threshold of these models is in the range of
200-600 images, depending on the domain and the model. The imitation threshold
can provide an empirical basis for copyright violation claims and acts as a
guiding principle for text-to-image model developers that aim to comply with
copyright and privacy laws. We release the code and data at
https://github.com/vsahil/MIMETIC-2.git and the project's website is
hosted at https://how-many-van-goghs-does-it-take.github.io.
Agent-to-Sim: Learning Interactive Behavior Models from Casual
  Longitudinal Videos
Authors: name: Gengshan Yang, name: Andrea Bajcsy, name: Shunsuke Saito, name: Angjoo Kanazawa
Published: 2024-10-21 17:57:50

We present Agent-to-Sim (ATS), a framework for learning interactive behavior
models of 3D agents from casual longitudinal video collections. Different from
prior works that rely on marker-based tracking and multiview cameras, ATS
learns natural behaviors of animal and human agents non-invasively through
video observations recorded over a long time-span (e.g., a month) in a single
environment. Modeling 3D behavior of an agent requires persistent 3D tracking
(e.g., knowing which point corresponds to which) over a long time period. To
obtain such data, we develop a coarse-to-fine registration method that tracks
the agent and the camera over time through a canonical 3D space, resulting in a
complete and persistent spacetime 4D representation. We then train a generative
model of agent behaviors using paired data of perception and motion of an agent
queried from the 4D reconstruction. ATS enables real-to-sim transfer from video
recordings of an agent to an interactive behavior simulator. We demonstrate
results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos
captured by a smartphone.
In-context learning and Occam's razor
Authors: name: Eric Elmoznino, name: Tom Marty, name: Tejas Kasetty, name: Leo Gagnon, name: Sarthak Mittal, name: Mahan Fathi, name: Dhanya Sridhar, name: Guillaume Lajoie
Published: 2024-10-17 23:37:34

The goal of machine learning is generalization. While the No Free Lunch
Theorem states that we cannot obtain theoretical guarantees for generalization
without further assumptions, in practice we observe that simple models which
explain the training data generalize best: a principle called Occam's razor.
Despite the need for simple models, most current approaches in machine learning
only minimize the training error, and at best indirectly promote simplicity
through regularization or architecture design. Here, we draw a connection
between Occam's razor and in-context learning: an emergent ability of certain
sequence models like Transformers to learn at inference time from past
observations in a sequence. In particular, we show that the next-token
prediction loss used to train in-context learners is directly equivalent to a
data compression technique called prequential coding, and that minimizing this
loss amounts to jointly minimizing both the training error and the complexity
of the model that was implicitly learned from context. Our theory and the
empirical experiments we use to support it not only provide a normative account
of in-context learning, but also elucidate the shortcomings of current
in-context learning methods, suggesting ways in which they can be improved. We
make our code available at https://github.com/3rdCore/PrequentialCode.
Router-Tuning: A Simple and Effective Approach for Enabling
  Dynamic-Depth in Transformers
Authors: name: Shwai He, name: Tao Ge, name: Guoheng Sun, name: Bowei Tian, name: Xiaoyang Wang, name: Ang Li, name: Dong Yu
Published: 2024-10-17 03:23:50

Traditional transformer models often allocate a fixed amount of computational
resources to every input token, leading to inefficient and unnecessary
computation. To address this, the Mixture of Depths (MoD) was introduced to
dynamically adjust the computational depth by skipping less important layers.
Despite its promise, current MoD approaches remain under-explored and face two
main challenges: (1) high training costs due to the need to train the
entire model along with the routers that determine which layers to skip, and
(2) the risk of performance degradation when important layers are
bypassed. In response to the first issue, we propose Router-Tuning, a method
that fine-tunes only the router on a small dataset, drastically reducing the
computational overhead associated with full model training. For the second
challenge, we propose MindSkip, which deploys Attention with Dynamic
Depths. This method preserves the model's performance while significantly
enhancing computational and memory efficiency. Extensive experiments
demonstrate that our approach delivers competitive results while dramatically
improving the computation efficiency, e.g., 21\% speedup and only a 0.2\%
performance drop. The code is released at
https://github.com/CASE-Lab-UMD/Router-Tuning.
Hallucination Detox: Sensitive Neuron Dropout (SeND) for Large Language
  Model Training
Authors: name: Shahrad Mohammadzadeh, name: Juan David Guerra, name: Marco Bonizzato, name: Reihaneh Rabbany, name: Golnoosh Farnadi
Published: 2024-10-20 18:18:23

As large language models (LLMs) become increasingly deployed across various
industries, concerns regarding their reliability, particularly due to
hallucinations-outputs that are factually inaccurate or irrelevant to user
input-have grown. Our research investigates the relationship between the
training process and the emergence of hallucinations to address a key gap in
existing research that focuses primarily on post hoc detection and mitigation
strategies. Using models from the Pythia suite (70M-12B parameters) and several
hallucination detection metrics, we analyze hallucination trends throughout
training and explore LLM internal dynamics. We introduce SEnsitive Neuron
Dropout (SeND), a novel training protocol designed to mitigate hallucinations
by reducing variance during training. SeND achieves this by deterministically
dropping neurons with significant variability on a dataset, referred to as
Sensitive Neurons. In addition, we develop an unsupervised hallucination
detection metric, Efficient EigenScore (EES), which approximates the
traditional EigenScore in 2x speed. This efficient metric is integrated into
our protocol, allowing SeND to be both computationally scalable and effective
at reducing hallucinations. Our empirical evaluation demonstrates that our
approach improves LLM reliability at test time by up to 40% compared to normal
training while also providing an efficient method to improve factual accuracy
when adapting LLMs to domains such as Wikipedia and Medical datasets.
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant
Authors: name: Alan Dao, name: Dinh Bach Vu, name: Huy Hoang Ha
Published: 2024-10-20 07:03:49

Large Language Models (LLMs) have revolutionized natural language processing,
but their application to speech-based tasks remains challenging due to the
complexities of integrating audio and text modalities. This paper introduces
Ichigo, a mixed-modal model that seamlessly processes interleaved sequences of
speech and text. Utilizing a tokenized early-fusion approach, Ichigo quantizes
speech into discrete tokens and employs a uniform transformer-based
architecture for both speech and text modalities. This method enables joint
reasoning and generation across modalities without the need for separate
adapters. We present a comprehensive training methodology, including
pre-training on multilingual speech recognition datasets and fine-tuning on a
curated instruction dataset. Ichigo demonstrates state-of-the-art performance
on speech question-answering benchmarks, outperforming existing open-source
speech language models and achieving comparable results to cascaded systems.
Notably, Ichigo exhibits a latency of just 111 ms to first token generation,
significantly lower than current models. Our approach not only advances the
field of multimodal AI but also provides a framework for smaller research teams
to contribute effectively to open-source speech-language models.
Zero-shot Model-based Reinforcement Learning using Large Language Models
Authors: name: Abdelhakim Benechehab, name: Youssef Attia El Hili, name: Ambroise Odonnat, name: Oussama Zekri, name: Albert Thomas, name: Giuseppe Paolo, name: Maurizio Filippone, name: Ievgen Redko, name: Balázs Kégl
Published: 2024-10-15 15:46:53

The emerging zero-shot capabilities of Large Language Models (LLMs) have led
to their applications in areas extending well beyond natural language
processing tasks. In reinforcement learning, while LLMs have been extensively
used in text-based environments, their integration with continuous state spaces
remains understudied. In this paper, we investigate how pre-trained LLMs can be
leveraged to predict in context the dynamics of continuous Markov decision
processes. We identify handling multivariate data and incorporating the control
signal as key challenges that limit the potential of LLMs' deployment in this
setup and propose Disentangled In-Context Learning (DICL) to address them. We
present proof-of-concept applications in two reinforcement learning settings:
model-based policy evaluation and data-augmented off-policy reinforcement
learning, supported by theoretical analysis of the proposed methods. Our
experiments further demonstrate that our approach produces well-calibrated
uncertainty estimates. We release the code at
https://github.com/abenechehab/dicl.