Orionnnn/hub-stats huggingface dataset:
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language
  Models
Authors: name: Michael Noukhovitch, name: Shengyi Huang, name: Sophie Xhonneux, name: Arian Hosseini, name: Rishabh Agarwal, name: Aaron Courville
Published: 2024-10-23 19:59:50

The dominant paradigm for RLHF is online and on-policy RL: synchronously
generating from the large language model (LLM) policy, labelling with a reward
model, and learning using feedback on the LLM's own outputs. While performant,
this paradigm is computationally inefficient. Inspired by classical deep RL
literature, we propose separating generation and learning in RLHF. This enables
asynchronous generation of new samples while simultaneously training on old
samples, leading to faster training and more compute-optimal scaling. However,
asynchronous training relies on an underexplored regime, online but off-policy
RLHF: learning on samples from previous iterations of our model. To understand
the challenges in this regime, we investigate a fundamental question: how much
off-policyness can we tolerate for asynchronous training to speed up learning
but maintain performance? Among several RLHF algorithms we tested, we find that
online DPO is most robust to off-policy data, and robustness increases with the
scale of the policy model. We study further compute optimizations for
asynchronous RLHF but find that they come at a performance cost, giving rise to
a trade-off. Finally, we verify the scalability of asynchronous RLHF by
training LLaMA 3.1 8B on an instruction-following task 40% faster than a
synchronous run while matching final performance.
SMITE: Segment Me In TimE
Authors: name: Amirhossein Alimohammadi, name: Sauradip Nag, name: Saeid Asgari Taghanaki, name: Andrea Tagliasacchi, name: Ghassan Hamarneh, name: Ali Mahdavi Amiri
Published: 2024-10-24 08:38:20

Segmenting an object in a video presents significant challenges. Each pixel
must be accurately labelled, and these labels must remain consistent across
frames. The difficulty increases when the segmentation is with arbitrary
granularity, meaning the number of segments can vary arbitrarily, and masks are
defined based on only one or a few sample images. In this paper, we address
this issue by employing a pre-trained text to image diffusion model
supplemented with an additional tracking mechanism. We demonstrate that our
approach can effectively manage various segmentation scenarios and outperforms
state-of-the-art alternatives.
Steering Knowledge Selection Behaviours in LLMs via SAE-Based
  Representation Engineering
Authors: name: Yu Zhao, name: Alessio Devoto, name: Giwon Hong, name: Xiaotang Du, name: Aryo Pradipta Gema, name: Hongru Wang, name: Kam-Fai Wong, name: Pasquale Minervini
Published: 2024-10-21 13:30:47

Large language models (LLMs) can store a significant amount of factual
knowledge in their parameters. However, their parametric knowledge may conflict
with the information provided in the context -- this phenomenon, known as
context-memory knowledge conflicts, can lead to undesirable model
behaviour, such as reliance on outdated or incorrect information. Analysing the
internal activations of LLMs, we find that they can internally register the
signals of knowledge conflict at mid-layers. Such signals allow us to detect
whether a knowledge conflict occurs and use inference-time intervention
strategies to resolve it. In this work, we propose SpARE, a
training-free representation engineering method that uses pre-trained
sparse auto-encoders (SAEs) to control the knowledge selection behaviour of
LLMs. SpARE identifies the functional features that control the
knowledge selection behaviours and applies them to edit the internal
activations of LLMs at inference time. Our experimental results show that
SpARE can effectively control the usage of either knowledge source to
resolve knowledge conflict in open-domain question-answering tasks, surpassing
existing representation engineering methods (+10%) as well as contrastive
decoding methods (+15%).
DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate
  Hallucinations
Authors: name: Aryo Pradipta Gema, name: Chen Jin, name: Ahmed Abdulaal, name: Tom Diethe, name: Philip Teare, name: Beatrice Alex, name: Pasquale Minervini, name: Amrutha Saseendran
Published: 2024-10-24 15:44:33

Large Language Models (LLMs) often hallucinate, producing unfaithful or
factually incorrect outputs by misrepresenting the provided context or
incorrectly recalling internal knowledge. Recent studies have identified
specific attention heads within the Transformer architecture, known as
retrieval heads, responsible for extracting relevant contextual information. We
hypothesise that masking these retrieval heads can induce hallucinations and
that contrasting the outputs of the base LLM and the masked LLM can reduce
hallucinations. To this end, we propose Decoding by Contrasting Retrieval Heads
(DeCoRe), a novel training-free decoding strategy that amplifies information
found in the context and model parameters. DeCoRe mitigates potentially
hallucinated responses by dynamically contrasting the outputs of the base LLM
and the masked LLM, using conditional entropy as a guide. Our extensive
experiments confirm that DeCoRe significantly improves performance on tasks
requiring high contextual faithfulness, such as summarisation (XSum by 18.6%),
instruction following (MemoTrap by 10.9%), and open-book question answering
(NQ-Open by 2.4% and NQ-Swap by 5.5%).
Data Scaling Laws in Imitation Learning for Robotic Manipulation
Authors: name: Fanqi Lin, name: Yingdong Hu, name: Pingyue Sheng, name: Chuan Wen, name: Jiacheng You, name: Yang Gao
Published: 2024-10-24 11:19:30

Data scaling has revolutionized fields like natural language processing and
computer vision, providing models with remarkable generalization capabilities.
In this paper, we investigate whether similar data scaling laws exist in
robotics, particularly in robotic manipulation, and whether appropriate data
scaling can yield single-task robot policies that can be deployed zero-shot for
any object within the same category in any environment. To this end, we conduct
a comprehensive empirical study on data scaling in imitation learning. By
collecting data across numerous environments and objects, we study how a
policy's generalization performance changes with the number of training
environments, objects, and demonstrations. Throughout our research, we collect
over 40,000 demonstrations and execute more than 15,000 real-world robot
rollouts under a rigorous evaluation protocol. Our findings reveal several
intriguing results: the generalization performance of the policy follows a
roughly power-law relationship with the number of environments and objects. The
diversity of environments and objects is far more important than the absolute
number of demonstrations; once the number of demonstrations per environment or
object reaches a certain threshold, additional demonstrations have minimal
effect. Based on these insights, we propose an efficient data collection
strategy. With four data collectors working for one afternoon, we collect
sufficient data to enable the policies for two tasks to achieve approximately
90% success rates in novel environments with unseen objects.
Taipan: Efficient and Expressive State Space Language Models with
  Selective Attention
Authors: name: Chien Van Nguyen, name: Huy Huu Nguyen, name: Thang M. Pham, name: Ruiyi Zhang, name: Hanieh Deilamsalehy, name: Puneet Mathur, name: Ryan A. Rossi, name: Trung Bui, name: Viet Dac Lai, name: Franck Dernoncourt, name: Thien Huu Nguyen
Published: 2024-10-24 09:25:37

Efficient long-context language modeling remains a significant challenge in
Natural Language Processing (NLP). While Transformers dominate language tasks,
they struggle with long sequences due to quadratic computational complexity in
training and linearly scaling memory costs during inference. Recent State Space
Models (SSMs) such as Mamba offer alternatives with constant memory usage, but
they underperform in tasks requiring extensive in-context retrieval. We
introduce Taipan, a novel hybrid architecture that combines Mamba-2 with
Selective Attention Layers (SALs). These SALs identify tokens requiring
long-range interactions, remove less important features, and then augment their
representations using the attention module. This approach balances Mamba's
efficiency with Transformer-like performance in memory-intensive tasks. By
constraining the attention budget, Taipan extends accurate predictions to
context lengths of up to 1 million tokens while preserving computational
efficiency. Our experiments demonstrate Taipan's superior performance across
various scales and tasks, offering a promising solution for efficient
long-context language modeling.
Value Residual Learning For Alleviating Attention Concentration In
  Transformers
Authors: name: Zhanchao Zhou, name: Tianyi Wu, name: Zhiyun Jiang, name: Zhenzhong Lan
Published: 2024-10-23 14:15:07

Transformers can capture long-range dependencies using self-attention,
allowing tokens to attend to all others directly. However, stacking multiple
attention layers leads to attention concentration. One natural way to address
this issue is to use cross-layer attention, allowing information from earlier
layers to be directly accessible to later layers. However, this approach is
computationally expensive. To address this problem, we propose Transformer with
residual value (ResFormer) which approximates cross-layer attention through
adding a residual connection from the values of the the first layer to all
subsequent layers. Based on this method, one variant is the Transformer with
single layer value (SVFormer), where all layers share the same value embedding
from first layer, reducing the KV cache by nearly 50%. Comprehensive empirical
evidence demonstrates that ResFormer mitigates attention concentration problem
in deeper layers and enhances representation across most layers, outperforming
the vanilla Transformer, DenseFormer, and NeuTRENO in training error as well as
downstream tasks. SVFormer trains significantly faster than the vanilla
Transformer and performs better than other methods like GQA and CLA, with
performance influenced by sequence length and cumulative learning rate.
ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language
  Tuning
Authors: name: Zhiwei Hao, name: Jianyuan Guo, name: Li Shen, name: Yong Luo, name: Han Hu, name: Yonggang Wen
Published: 2024-10-23 11:31:06

Recent advancements in multimodal fusion have witnessed the remarkable
success of vision-language (VL) models, which excel in various multimodal
applications such as image captioning and visual question answering. However,
building VL models requires substantial hardware resources, where efficiency is
restricted by two key factors: the extended input sequence of the language
model with vision features demands more computational operations, and a large
number of additional learnable parameters increase memory complexity. These
challenges significantly restrict the broader applicability of such models. To
bridge this gap, we propose ADEM-VL, an efficient vision-language method that
tunes VL models based on pretrained large language models (LLMs) by adopting a
parameter-free cross-attention mechanism for similarity measurements in
multimodal fusion. This approach only requires embedding vision features into
the language space, significantly reducing the number of trainable parameters
and accelerating both training and inference speeds. To enhance representation
learning in fusion module, we introduce an efficient multiscale feature
generation scheme that requires only a single forward pass through the vision
encoder. Moreover, we propose an adaptive fusion scheme that dynamically
discards less relevant visual information for each text token based on its
attention score. This ensures that the fusion process prioritizes the most
pertinent visual features. With experiments on various tasks including visual
question answering, image captioning, and instruction-following, we demonstrate
that our framework outperforms existing approaches. Specifically, our method
surpasses existing methods by an average accuracy of 0.77% on ScienceQA
dataset, with reduced training and inference latency, demonstrating the
superiority of our framework. The code is available at
https://github.com/Hao840/ADEM-VL.
CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
Authors: name: Sara Ghaboura, name: Ahmed Heakl, name: Omkar Thawakar, name: Ali Alharthi, name: Ines Riahi, name: Abduljalil Saif, name: Jorma Laaksonen, name: Fahad S. Khan, name: Salman Khan, name: Rao M. Anwer
Published: 2024-10-24 17:59:38

Recent years have witnessed a significant interest in developing large
multimodal models (LMMs) capable of performing various visual reasoning and
understanding tasks. This has led to the introduction of multiple LMM
benchmarks to evaluate LMMs on different tasks. However, most existing LMM
evaluation benchmarks are predominantly English-centric. In this work, we
develop a comprehensive LMM evaluation benchmark for the Arabic language to
represent a large population of over 400 million speakers. The proposed
benchmark, named CAMEL-Bench, comprises eight diverse domains and 38
sub-domains including, multi-image understanding, complex visual perception,
handwritten document understanding, video understanding, medical imaging, plant
diseases, and remote sensing-based land use understanding to evaluate broad
scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions
that are filtered from a larger pool of samples, where the quality is manually
verified by native speakers to ensure reliable model assessment. We conduct
evaluations of both closed-source, including GPT-4 series, and open-source
LMMs. Our analysis reveals the need for substantial improvement, especially
among the best open-source models, with even the closed-source GPT-4o achieving
an overall score of 62%. Our benchmark and evaluation scripts are open-sourced.
Distill Visual Chart Reasoning Ability from LLMs to MLLMs
Authors: name: Wei He, name: Zhiheng Xi, name: Wanxu Zhao, name: Xiaoran Fan, name: Yiwen Ding, name: Zifei Shan, name: Tao Gui, name: Qi Zhang, name: Xuanjing Huang
Published: 2024-10-24 14:50:42

Solving complex chart Q&A tasks requires advanced visual reasoning abilities
in multimodal large language models (MLLMs). Recent studies highlight that
these abilities consist of two main parts: recognizing key information from
visual inputs and conducting reasoning over it. Thus, a promising approach to
enhance MLLMs is to construct relevant training data focusing on the two
aspects. However, collecting and annotating complex charts and questions is
costly and time-consuming, and ensuring the quality of annotated answers
remains a challenge. In this paper, we propose Code-as-Intermediary Translation
(CIT), a cost-effective, efficient and easily scalable data synthesis method
for distilling visual reasoning abilities from LLMs to MLLMs. The code serves
as an intermediary that translates visual chart representations into textual
representations, enabling LLMs to understand cross-modal information.
Specifically, we employ text-based synthesizing techniques to construct
chart-plotting code and produce ReachQA, a dataset containing 3k
reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and
reasoning abilities. Experiments show that when fine-tuned with our data,
models not only perform well on chart-related benchmarks, but also demonstrate
improved multimodal reasoning abilities on general mathematical benchmarks like
MathVista. The code and dataset are publicly available at
https://github.com/hewei2001/ReachQA.
WAFFLE: Multi-Modal Model for Automated Front-End Development
Authors: name: Shanchao Liang, name: Nan Jiang, name: Shangshu Qian, name: Lin Tan
Published: 2024-10-24 01:49:49

Web development involves turning UI designs into functional webpages, which
can be difficult for both beginners and experienced developers due to the
complexity of HTML's hierarchical structures and styles. While Large Language
Models (LLMs) have shown promise in generating source code, two major
challenges persist in UI-to-HTML code generation: (1) effectively representing
HTML's hierarchical structure for LLMs, and (2) bridging the gap between the
visual nature of UI designs and the text-based format of HTML code. To tackle
these challenges, we introduce Waffle, a new fine-tuning strategy that uses a
structure-aware attention mechanism to improve LLMs' understanding of HTML's
structure and a contrastive fine-tuning approach to align LLMs' understanding
of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp
(percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP,
and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing
benchmark Design2Code, outperforming current fine-tuning methods.
Can Knowledge Editing Really Correct Hallucinations?
Authors: name: Baixiang Huang, name: Canyu Chen, name: Xiongxiao Xu, name: Ali Payani, name: Kai Shu
Published: 2024-10-21 17:55:54

Large Language Models (LLMs) suffer from hallucinations, referring to the
non-factual information in generated content, despite their superior capacities
across tasks. Meanwhile, knowledge editing has been developed as a new popular
paradigm to correct the erroneous factual knowledge encoded in LLMs with the
advantage of avoiding retraining from scratch. However, one common issue of
existing evaluation datasets for knowledge editing is that they do not ensure
LLMs actually generate hallucinated answers to the evaluation questions before
editing. When LLMs are evaluated on such datasets after being edited by
different techniques, it is hard to directly adopt the performance to assess
the effectiveness of different knowledge editing methods in correcting
hallucinations. Thus, the fundamental question remains insufficiently
validated: Can knowledge editing really correct hallucinations in LLMs? We
proposed HalluEditBench to holistically benchmark knowledge editing methods in
correcting real-world hallucinations. First, we rigorously construct a massive
hallucination dataset with 9 domains, 26 topics and more than 6,000
hallucinations. Then, we assess the performance of knowledge editing methods in
a holistic way on five dimensions including Efficacy, Generalization,
Portability, Locality, and Robustness. Through HalluEditBench, we have provided
new insights into the potentials and limitations of different knowledge editing
methods in correcting hallucinations, which could inspire future improvements
and facilitate the progress in the field of knowledge editing.
MotionCLR: Motion Generation and Training-free Editing via Understanding
  Attention Mechanisms
Authors: name: Ling-Hao Chen, name: Wenxun Dai, name: Xuan Ju, name: Shunlin Lu, name: Lei Zhang
Published: 2024-10-24 17:59:45

This research delves into the problem of interactive editing of human motion
generation. Previous motion diffusion models lack explicit modeling of the
word-level text-motion correspondence and good explainability, hence
restricting their fine-grained editing ability. To address this issue, we
propose an attention-based motion diffusion model, namely MotionCLR, with CLeaR
modeling of attention mechanisms. Technically, MotionCLR models the in-modality
and cross-modality interactions with self-attention and cross-attention,
respectively. More specifically, the self-attention mechanism aims to measure
the sequential similarity between frames and impacts the order of motion
features. By contrast, the cross-attention mechanism works to find the
fine-grained word-sequence correspondence and activate the corresponding
timesteps in the motion sequence. Based on these key properties, we develop a
versatile set of simple yet effective motion editing methods via manipulating
attention maps, such as motion (de-)emphasizing, in-place motion replacement,
and example-based motion generation, etc. For further verification of the
explainability of the attention mechanism, we additionally explore the
potential of action-counting and grounded motion generation ability via
attention maps. Our experimental results show that our method enjoys good
generation and editing ability with good explainability.
Language Models are Symbolic Learners in Arithmetic
Authors: name: Chunyuan Deng, name: Zhiqi Li, name: Roy Xie, name: Ruidi Chang, name: Hanjie Chen
Published: 2024-10-21 01:57:16

Large Language Models (LLMs) are thought to struggle with arithmetic learning
due to the inherent differences between language modeling and numerical
computation, but concrete evidence has been lacking. This work responds to this
claim through a two-side experiment. We first investigate whether LLMs leverage
partial products during arithmetic learning. We find that although LLMs can
identify some partial products after learning, they fail to leverage them for
arithmetic tasks, conversely. We then explore how LLMs approach arithmetic
symbolically by breaking tasks into subgroups, hypothesizing that difficulties
arise from subgroup complexity and selection. Our results show that when
subgroup complexity is fixed, LLMs treat a collection of different arithmetic
operations similarly. By analyzing position-level accuracy across different
training sizes, we further observe that it follows a U-shaped pattern: LLMs
quickly learn the easiest patterns at the first and last positions, while
progressively learning the more difficult patterns in the middle positions.
This suggests that LLMs select subgroup following an easy-to-hard paradigm
during learning. Our work confirms that LLMs are pure symbolic learners in
arithmetic tasks and underscores the importance of understanding them deeply
through subgroup-level quantification.
Robust Watermarking Using Generative Priors Against Image Editing: From
  Benchmarking to Advances
Authors: name: Shilin Lu, name: Zihan Zhou, name: Jiayou Lu, name: Yuanzhi Zhu, name: Adams Wai-Kin Kong
Published: 2024-10-24 14:28:32

Current image watermarking methods are vulnerable to advanced image editing
techniques enabled by large-scale text-to-image models. These models can
distort embedded watermarks during editing, posing significant challenges to
copyright protection. In this work, we introduce W-Bench, the first
comprehensive benchmark designed to evaluate the robustness of watermarking
methods against a wide range of image editing techniques, including image
regeneration, global editing, local editing, and image-to-video generation.
Through extensive evaluations of eleven representative watermarking methods
against prevalent editing techniques, we demonstrate that most methods fail to
detect watermarks after such edits. To address this limitation, we propose
VINE, a watermarking method that significantly enhances robustness against
various image editing techniques while maintaining high image quality. Our
approach involves two key innovations: (1) we analyze the frequency
characteristics of image editing and identify that blurring distortions exhibit
similar frequency properties, which allows us to use them as surrogate attacks
during training to bolster watermark robustness; (2) we leverage a large-scale
pretrained diffusion model SDXL-Turbo, adapting it for the watermarking task to
achieve more imperceptible and robust watermark embedding. Experimental results
show that our method achieves outstanding watermarking performance under
various image editing techniques, outperforming existing methods in both image
quality and robustness. Code is available at https://github.com/Shilin-LU/VINE.