Orionnnn/hub-stats huggingface dataset:
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse
  Gradients
Authors: name: Aashiq Muhamed, name: Oscar Li, name: David Woodruff, name: Mona Diab, name: Virginia Smith
Published: 2024-06-25 15:50:32

Large language model (LLM) training and finetuning are often bottlenecked by
limited GPU memory. While existing projection-based optimization methods
address this by projecting gradients into a lower-dimensional subspace to
reduce optimizer state memory, they typically rely on dense projection
matrices, which can introduce computational and memory overheads. In this work,
we propose Grass (GRAdient Stuctured Sparsification), a novel approach that
leverages sparse projections to transform gradients into structured sparse
updates. This design not only significantly reduces memory usage for optimizer
states but also minimizes gradient memory footprint, computation, and
communication costs, leading to substantial throughput improvements. Extensive
experiments on pretraining and finetuning tasks demonstrate that Grass achieves
competitive performance to full-rank training and existing projection-based
methods. Notably, Grass enables half-precision pretraining of a 13B parameter
LLaMA model on a single 40GB A100 GPU--a feat infeasible for previous
methods--and yields up to a 2times throughput improvement on an 8-GPU
system. Code can be found at https://github.com/aashiqmuhamed/GRASS .
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended
  Multi-Doc QA
Authors: name: Minzheng Wang, name: Longze Chen, name: Cheng Fu, name: Shengyi Liao, name: Xinghua Zhang, name: Bingli Wu, name: Haiyang Yu, name: Nan Xu, name: Lei Zhang, name: Run Luo, name: Yunshui Li, name: Min Yang, name: Fei Huang, name: Yongbin Li
Published: 2024-06-25 09:42:56

Long-context modeling capabilities have garnered widespread attention,
leading to the emergence of Large Language Models (LLMs) with ultra-context
windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually
catching up. However, existing benchmarks employ irrelevant noise texts to
artificially extend the length of test cases, diverging from the real-world
scenarios of long-context applications. To bridge this gap, we propose a novel
long-context benchmark, Loong, aligning with realistic scenarios through
extended multi-document question answering (QA). Unlike typical document QA, in
Loong's test cases, each document is relevant to the final answer, ignoring any
document will lead to the failure of the answer. Furthermore, Loong introduces
four types of tasks with a range of context lengths: Spotlight Locating,
Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic
and comprehensive evaluation of long-context understanding. Extensive
experiments indicate that existing long-context language models still exhibit
considerable potential for enhancement. Retrieval augmented generation (RAG)
achieves poor performance, demonstrating that Loong can reliably assess the
model's long-context modeling capabilities.
Multi-property Steering of Large Language Models with Dynamic Activation
  Composition
Authors: name: Daniel Scalena, name: Gabriele Sarti, name: Malvina Nissim
Published: 2024-06-25 14:00:42

Activation steering methods were shown to be effective in conditioning
language model generation by additively intervening over models' intermediate
representations. However, the evaluation of these techniques has so far been
limited to single conditioning properties and synthetic settings. In this work,
we conduct a comprehensive evaluation of various activation steering
strategies, highlighting the property-dependent nature of optimal parameters to
ensure a robust effect throughout generation. To address this issue, we propose
Dynamic Activation Composition, an information-theoretic approach to modulate
the steering intensity of one or more properties throughout generation. Our
experiments on multi-property steering show that our method successfully
maintains high conditioning while minimizing the impact of conditioning on
generation fluency.
Segment Any Text: A Universal Approach for Robust, Efficient and
  Adaptable Sentence Segmentation
Authors: name: Markus Frohmann, name: Igor Sterner, name: Ivan Vulić, name: Benjamin Minixhofer, name: Markus Schedl
Published: 2024-06-24 14:36:11

Segmenting text into sentences plays an early and crucial role in many NLP
systems. This is commonly achieved by using rule-based or statistical methods
relying on lexical features such as punctuation. Although some recent works no
longer exclusively rely on punctuation, we find that no prior method achieves
all of (i) robustness to missing punctuation, (ii) effective adaptability to
new domains, and (iii) high efficiency. We introduce a new model - Segment any
Text (SaT) - to solve this problem. To enhance robustness, we propose a new
pretraining scheme that ensures less reliance on punctuation. To address
adaptability, we introduce an extra stage of parameter-efficient fine-tuning,
establishing state-of-the-art performance in distinct domains such as verses
from lyrics and legal documents. Along the way, we introduce architectural
modifications that result in a threefold gain in speed over the previous state
of the art and solve spurious reliance on context far in the future. Finally,
we introduce a variant of our model with fine-tuning on a diverse, multilingual
mixture of sentence-segmented data, acting as a drop-in replacement and
enhancement for existing segmentation tools. Overall, our contributions provide
a universal approach for segmenting any text. Our method outperforms all
baselines - including strong LLMs - across 8 corpora spanning diverse domains
and languages, especially in practically relevant situations where text is
poorly formatted. Our models and code, including documentation, are available
at https://huggingface.co/segment-any-text under the MIT license.
DialSim: A Real-Time Simulator for Evaluating Long-Term Dialogue
  Understanding of Conversational Agents
Authors: name: Jiho Kim, name: Woosog Chay, name: Hyeonji Hwang, name: Daeun Kyung, name: Hyunseung Chung, name: Eunbyeol Cho, name: Yohan Jo, name: Edward Choi
Published: 2024-06-19 01:37:10

Recent advancements in Large Language Models (LLMs) have significantly
enhanced the capabilities of conversational agents, making them applicable to
various fields (e.g., education). Despite their progress, the evaluation of the
agents often overlooks the complexities of real-world conversations, such as
real-time interactions, multi-party dialogues, and extended contextual
dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue
simulator. In this simulator, an agent is assigned the role of a character from
popular TV shows, requiring it to respond to spontaneous questions using past
dialogue information and to distinguish between known and unknown information.
Key features of DialSim include evaluating the agent's ability to respond
within a reasonable time limit, handling long-term multi-party dialogues, and
managing adversarial settings (e.g., swap character names) to challenge the
agent's reliance on pre-trained knowledge. We utilized this simulator to
evaluate the latest conversational agents and analyze their limitations. Our
experiments highlight both the strengths and weaknesses of these agents,
providing valuable insights for future improvements in the field of
conversational AI. DialSim is available at
https://github.com/jiho283/Simulator.
Image Conductor: Precision Control for Interactive Video Synthesis
Authors: name: Yaowei Li, name: Xintao Wang, name: Zhaoyang Zhang, name: Zhouxia Wang, name: Ziyang Yuan, name: Liangbin Xie, name: Yuexian Zou, name: Ying Shan
Published: 2024-06-21 17:55:05

Filmmaking and animation production often require sophisticated techniques
for coordinating camera transitions and object movements, typically involving
labor-intensive real-world capturing. Despite advancements in generative AI for
video creation, achieving precise control over motion for interactive video
asset generation remains challenging. To this end, we propose Image Conductor,
a method for precise control of camera transitions and object movements to
generate video assets from a single image. An well-cultivated training strategy
is proposed to separate distinct camera and object motion by camera LoRA
weights and object LoRA weights. To further address cinematographic variations
from ill-posed trajectories, we introduce a camera-free guidance technique
during inference, enhancing object movements while eliminating camera
transitions. Additionally, we develop a trajectory-oriented video motion data
curation pipeline for training. Quantitative and qualitative experiments
demonstrate our method's precision and fine-grained control in generating
motion-controllable videos from images, advancing the practical application of
interactive video synthesis. Project webpage available at
https://liyaowei-stu.github.io/project/ImageConductor/
On the Transformations across Reward Model, Parameter Update, and
  In-Context Prompt
Authors: name: Deng Cai, name: Huayang Li, name: Tingchen Fu, name: Siheng Li, name: Weiwen Xu, name: Shuaiyi Li, name: Bowen Cao, name: Zhisong Zhang, name: Xinting Huang, name: Leyang Cui, name: Yan Wang, name: Lemao Liu, name: Taro Watanabe, name: Shuming Shi
Published: 2024-06-24 07:42:32

Despite the general capabilities of pre-trained large language models (LLMs),
they still need further adaptation to better serve practical applications. In
this paper, we demonstrate the interchangeability of three popular and distinct
adaptation tools: parameter updating, reward modeling, and in-context
prompting. This interchangeability establishes a triangular framework with six
transformation directions, each of which facilitates a variety of applications.
Our work offers a holistic view that unifies numerous existing studies and
suggests potential research directions. We envision our work as a useful
roadmap for future research on LLMs.
Aligning Diffusion Models with Noise-Conditioned Perception
Authors: name: Alexander Gambashidze, name: Anton Kulikov, name: Yuriy Sosnin, name: Ilya Makarov
Published: 2024-06-25 15:21:50

Recent advancements in human preference optimization, initially developed for
Language Models (LMs), have shown promise for text-to-image Diffusion Models,
enhancing prompt alignment, visual appeal, and user preference. Unlike LMs,
Diffusion Models typically optimize in pixel or VAE space, which does not align
well with human perception, leading to slower and less efficient training
during the preference alignment stage. We propose using a perceptual objective
in the U-Net embedding space of the diffusion model to address these issues.
Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct
Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and
supervised fine-tuning (SFT) within this embedding space. This method
significantly outperforms standard latent-space implementations across various
metrics, including quality and computational cost. For SDXL, our approach
provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt
following against original open-sourced SDXL-DPO on the PartiPrompts dataset,
while significantly reducing compute. Our approach not only improves the
efficiency and quality of human preference alignment for diffusion models but
is also easily integrable with other optimization techniques. The training code
and LoRA weights will be available here:
https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
The FineWeb Datasets: Decanting the Web for the Finest Text Data at
  Scale
Authors: name: Guilherme Penedo, name: Hynek Kydlíček, name: Loubna Ben allal, name: Anton Lozhkov, name: Margaret Mitchell, name: Colin Raffel, name: Leandro Von Werra, name: Thomas Wolf
Published: 2024-06-25 13:50:56

The performance of a large language model (LLM) depends heavily on the
quality and size of its pretraining dataset. However, the pretraining datasets
for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly
available and very little is known about how they were created. In this work,
we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl
snapshots that produces better-performing LLMs than other open pretraining
datasets. To advance the understanding of how best to curate high-quality
pretraining datasets, we carefully document and ablate all of the design
choices used in FineWeb, including in-depth investigations of deduplication and
filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion
token collection of educational text filtered from FineWeb. LLMs pretrained on
FineWeb-Edu exhibit dramatically better performance on knowledge- and
reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we
publicly release our data curation codebase and all of the models trained
during our ablation experiments.
MotionBooth: Motion-Aware Customized Text-to-Video Generation
Authors: name: Jianzong Wu, name: Xiangtai Li, name: Yanhong Zeng, name: Jiangning Zhang, name: Qianyu Zhou, name: Yining Li, name: Yunhai Tong, name: Kai Chen
Published: 2024-06-25 17:42:25

In this work, we present MotionBooth, an innovative framework designed for
animating customized subjects with precise control over both object and camera
movements. By leveraging a few images of a specific object, we efficiently
fine-tune a text-to-video model to capture the object's shape and attributes
accurately. Our approach presents subject region loss and video preservation
loss to enhance the subject's learning performance, along with a subject token
cross-attention loss to integrate the customized subject with motion control
signals. Additionally, we propose training-free techniques for managing subject
and camera motions during inference. In particular, we utilize cross-attention
map manipulation to govern subject motion and introduce a novel latent shift
module for camera movement control as well. MotionBooth excels in preserving
the appearance of subjects while simultaneously controlling the motions in
generated videos. Extensive quantitative and qualitative evaluations
demonstrate the superiority and effectiveness of our method. Our project page
is at https://jianzongwu.github.io/projects/motionbooth
LongIns: A Challenging Long-context Instruction-based Exam for LLMs
Authors: name: Shawn Gavin, name: Tuney Zheng, name: Jiaheng Liu, name: Quehry Que, name: Noah Wang, name: Jian Yang, name: Chenchen Zhang, name: Wenhao Huang, name: Wenhu Chen, name: Ge Zhang
Published: 2024-06-25 14:31:26

The long-context capabilities of large language models (LLMs) have been a hot
topic in recent years. To evaluate the performance of LLMs in different
scenarios, various assessment benchmarks have emerged. However, as most of
these benchmarks focus on identifying key information to answer questions,
which mainly requires the retrieval ability of LLMs, these benchmarks can
partially represent the reasoning performance of LLMs from large amounts of
information. Meanwhile, although LLMs often claim to have context windows of
32k, 128k, 200k, or even longer, these benchmarks fail to reveal the actual
supported length of these LLMs. To address these issues, we propose the LongIns
benchmark dataset, a challenging long-context instruction-based exam for LLMs,
which is built based on the existing instruction datasets. Specifically, in our
LongIns, we introduce three evaluation settings: Global Instruction & Single
Task (GIST), Local Instruction & Single Task (LIST), and Local Instruction &
Multiple Tasks (LIMT). Based on LongIns, we perform comprehensive evaluations
on existing LLMs and have the following important findings: (1). The
top-performing GPT-4 with 128k context length performs poorly on the evaluation
context window of 16k in our LongIns. (2). For the multi-hop reasoning ability
of many existing LLMs, significant efforts are still needed under short context
windows (less than 4k).
DiffusionPDE: Generative PDE-Solving Under Partial Observation
Authors: name: Jiahe Huang, name: Guandao Yang, name: Zichen Wang, name: Jeong Joon Park
Published: 2024-06-25 17:48:24

We introduce a general framework for solving partial differential equations
(PDEs) using generative diffusion models. In particular, we focus on the
scenarios where we do not have the full knowledge of the scene necessary to
apply classical solvers. Most existing forward or inverse PDE approaches
perform poorly when the observations on the data or the underlying coefficients
are incomplete, which is a common assumption for real-world measurements. In
this work, we propose DiffusionPDE that can simultaneously fill in the missing
information and solve a PDE by modeling the joint distribution of the solution
and coefficient spaces. We show that the learned generative priors lead to a
versatile framework for accurately solving a wide range of PDEs under partial
observation, significantly outperforming the state-of-the-art methods for both
forward and inverse directions.
FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
Authors: name: Haonan Qiu, name: Zhaoxi Chen, name: Zhouxia Wang, name: Yingqing He, name: Menghan Xia, name: Ziwei Liu
Published: 2024-06-24 17:59:56

Diffusion model has demonstrated remarkable capability in video generation,
which further sparks interest in introducing trajectory control into the
generation process. While existing works mainly focus on training-based methods
(e.g., conditional adapter), we argue that diffusion model itself allows decent
control over the generated content without requiring any training. In this
study, we introduce a tuning-free framework to achieve trajectory-controllable
video generation, by imposing guidance on both noise construction and attention
computation. Specifically, 1) we first show several instructive phenomenons and
analyze how initial noises influence the motion trajectory of generated
content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that
enables trajectory control by modifying noise sampling and attention
mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger
video generation with controllable trajectories. Equipped with these designs,
users have the flexibility to provide trajectories manually or opt for
trajectories automatically generated by the LLM trajectory planner. Extensive
experiments validate the efficacy of our approach in enhancing the trajectory
controllability of video diffusion models.
YouDream: Generating Anatomically Controllable Consistent Text-to-3D
  Animals
Authors: name: Sandeep Mishra, name: Oindrila Saha, name: Alan C. Bovik
Published: 2024-06-24 02:40:26

3D generation guided by text-to-image diffusion models enables the creation
of visually compelling assets. However previous methods explore generation
based on image or text. The boundaries of creativity are limited by what can be
expressed through words or the images that can be sourced. We present YouDream,
a method to generate high-quality anatomically controllable animals. YouDream
is guided using a text-to-image diffusion model controlled by 2D views of a 3D
pose prior. Our method generates 3D animals that are not possible to create
using previous text-to-3D generative methods. Additionally, our method is
capable of preserving anatomic consistency in the generated animals, an area
where prior text-to-3D approaches often struggle. Moreover, we design a fully
automated pipeline for generating commonly found animals. To circumvent the
need for human intervention to create a 3D pose, we propose a multi-agent LLM
that adapts poses from a limited library of animal 3D poses to represent the
desired animal. A user study conducted on the outcomes of YouDream demonstrates
the preference of the animal models generated by our method over others.
Turntable results and code are released at https://youdream3d.github.io/
Unlocking Continual Learning Abilities in Language Models
Authors: name: Wenyu Du, name: Shuang Cheng, name: Tongxu Luo, name: Zihan Qiu, name: Zeyu Huang, name: Ka Chun Cheung, name: Reynold Cheng, name: Jie Fu
Published: 2024-06-25 03:24:06

Language models (LMs) exhibit impressive performance and generalization
capabilities. However, LMs struggle with the persistent challenge of
catastrophic forgetting, which undermines their long-term sustainability in
continual learning (CL). Existing approaches usually address the issue by
incorporating old task data or task-wise inductive bias into LMs. However, old
data and accurate task information are often unavailable or costly to collect,
hindering the availability of current CL approaches for LMs. To address this
limitation, we introduce MIGU (MagnItude-based
Gradient Updating for continual learning), a
rehearsal-free and task-label-free method that only updates the model
parameters with large magnitudes of output in LMs' linear layers. MIGU is based
on our observation that the L1-normalized magnitude distribution of the output
in LMs' linear layers is different when the LM models deal with different task
data. By imposing this simple constraint on the gradient update process, we can
leverage the inherent behaviors of LMs, thereby unlocking their innate CL
abilities. Our experiments demonstrate that MIGU is universally applicable to
all three LM architectures (T5, RoBERTa, and Llama2), delivering
state-of-the-art or on-par performance across continual finetuning and
continual pre-training settings on four CL benchmarks. For example, MIGU brings
a 15.2% average accuracy improvement over conventional parameter-efficient
finetuning baselines in a 15-task CL benchmark. MIGU can also seamlessly
integrate with all three existing CL types to further enhance performance. Code
is available at https://github.com/wenyudu/MIGU{this https URL}.
MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
Authors: name: Xiangyu Zhao, name: Xiangtai Li, name: Haodong Duan, name: Haian Huang, name: Yining Li, name: Kai Chen, name: Hua Yang
Published: 2024-06-25 17:55:11

Multi-modal large language models (MLLMs) have made significant strides in
various visual understanding tasks. However, the majority of these models are
constrained to process low-resolution images, which limits their effectiveness
in perception tasks that necessitate detailed visual information. In our study,
we present MG-LLaVA, an innovative MLLM that enhances the model's visual
processing capabilities by incorporating a multi-granularity vision flow, which
includes low-resolution, high-resolution, and object-centric features. We
propose the integration of an additional high-resolution visual encoder to
capture fine-grained details, which are then fused with base visual features
through a Conv-Gate fusion network. To further refine the model's object
recognition abilities, we incorporate object-level features derived from
bounding boxes identified by offline detectors. Being trained solely on
publicly available multimodal data through instruction tuning, MG-LLaVA
demonstrates exceptional perception skills. We instantiate MG-LLaVA with a wide
variety of language encoders, ranging from 3.8B to 34B, to evaluate the model's
performance comprehensively. Extensive evaluations across multiple benchmarks
demonstrate that MG-LLaVA outperforms existing MLLMs of comparable parameter
sizes, showcasing its remarkable efficacy. The code will be available at
https://github.com/PhoenixZ810/MG-LLaVA.