Orionnnn/hub-stats huggingface dataset:
BitStack: Fine-Grained Size Control for Compressed Large Language Models
  in Variable Memory Environments
Authors: name: Xinghao Wang, name: Pengyu Wang, name: Bo Wang, name: Dong Zhang, name: Yunhua Zhou, name: Xipeng Qiu
Published: 2024-10-31 13:26:11

Large language models (LLMs) have revolutionized numerous applications, yet
their deployment remains challenged by memory constraints on local devices.
While scaling laws have enhanced LLM capabilities, the primary bottleneck has
shifted from capability to availability, emphasizing the need
for efficient memory management. Traditional compression methods, such as
quantization, often require predefined compression ratios and separate
compression processes for each setting, complicating deployment in variable
memory environments. In this paper, we introduce BitStack, a novel,
training-free weight compression approach that enables megabyte-level
trade-offs between memory usage and model performance. By leveraging weight
decomposition, BitStack can dynamically adjust the model size with minimal
transmission between running memory and storage devices. Our approach
iteratively decomposes weight matrices while considering the significance of
each parameter, resulting in an approximately 1-bit per parameter residual
block in each decomposition iteration. These blocks are sorted and stacked in
storage as basic transmission units, with different quantities loaded based on
current memory availability. Extensive experiments across a wide range of tasks
demonstrate that, despite offering fine-grained size control, BitStack
consistently matches or surpasses strong quantization baselines, particularly
at extreme compression ratios. To the best of our knowledge, this is the first
decomposition-based method that effectively bridges the gap to practical
compression techniques like quantization. Code is available at
https://github.com/xinghaow99/BitStack.
A Pointer Network-based Approach for Joint Extraction and Detection of
  Multi-Label Multi-Class Intents
Authors: name: Ankan Mullick, name: Sombit Bose, name: Abhilash Nandy, name: Gajula Sai Chaitanya, name: Pawan Goyal
Published: 2024-10-29 19:10:12

In task-oriented dialogue systems, intent detection is crucial for
interpreting user queries and providing appropriate responses. Existing
research primarily addresses simple queries with a single intent, lacking
effective systems for handling complex queries with multiple intents and
extracting different intent spans. Additionally, there is a notable absence of
multilingual, multi-intent datasets. This study addresses three critical tasks:
extracting multiple intent spans from queries, detecting multiple intents, and
developing a multi-lingual multi-label intent dataset. We introduce a novel
multi-label multi-class intent detection dataset (MLMCID-dataset) curated from
existing benchmark datasets. We also propose a pointer network-based
architecture (MLMCID) to extract intent spans and detect multiple intents with
coarse and fine-grained labels in the form of sextuplets. Comprehensive
analysis demonstrates the superiority of our pointer network-based system over
baseline approaches in terms of accuracy and F1-score across various datasets.
Navigating the Unknown: A Chat-Based Collaborative Interface for
  Personalized Exploratory Tasks
Authors: name: Yingzhe Peng, name: Xiaoting Qin, name: Zhiyang Zhang, name: Jue Zhang, name: Qingwei Lin, name: Xu Yang, name: Dongmei Zhang, name: Saravan Rajmohan, name: Qi Zhang
Published: 2024-10-31 15:30:55

The rise of large language models (LLMs) has revolutionized user interactions
with knowledge-based systems, enabling chatbots to synthesize vast amounts of
information and assist with complex, exploratory tasks. However, LLM-based
chatbots often struggle to provide personalized support, particularly when
users start with vague queries or lack sufficient contextual information. This
paper introduces the Collaborative Assistant for Personalized Exploration
(CARE), a system designed to enhance personalization in exploratory tasks by
combining a multi-agent LLM framework with a structured user interface. CARE's
interface consists of a Chat Panel, Solution Panel, and Needs Panel, enabling
iterative query refinement and dynamic solution generation. The multi-agent
framework collaborates to identify both explicit and implicit user needs,
delivering tailored, actionable solutions. In a within-subject user study with
22 participants, CARE was consistently preferred over a baseline LLM chatbot,
with users praising its ability to reduce cognitive load, inspire creativity,
and provide more tailored solutions. Our findings highlight CARE's potential to
transform LLM-based systems from passive information retrievers to proactive
partners in personalized problem-solving and exploration.
Unpacking SDXL Turbo: Interpreting Text-to-Image Models with Sparse
  Autoencoders
Authors: name: Viacheslav Surkov, name: Chris Wendler, name: Mikhail Terekhov, name: Justin Deschenaux, name: Robert West, name: Caglar Gulcehre
Published: 2024-10-28 19:01:18

Sparse autoencoders (SAEs) have become a core ingredient in the reverse
engineering of large-language models (LLMs). For LLMs, they have been shown to
decompose intermediate representations that often are not interpretable
directly into sparse sums of interpretable features, facilitating better
control and subsequent analysis. However, similar analyses and approaches have
been lacking for text-to-image models. We investigated the possibility of using
SAEs to learn interpretable features for a few-step text-to-image diffusion
models, such as SDXL Turbo. To this end, we train SAEs on the updates performed
by transformer blocks within SDXL Turbo's denoising U-net. We find that their
learned features are interpretable, causally influence the generation process,
and reveal specialization among the blocks. In particular, we find one block
that deals mainly with image composition, one that is mainly responsible for
adding local details, and one for color, illumination, and style. Therefore,
our work is an important first step towards better understanding the internals
of generative text-to-image models like SDXL Turbo and showcases the potential
of features learned by SAEs for the visual domain.
  Code is available at https://github.com/surkovv/sdxl-unbox
What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A
  Gradient Perspective
Authors: name: Ming Li, name: Yanhong Li, name: Tianyi Zhou
Published: 2024-10-31 08:58:06

What makes a difference in the post-training of LLMs? We investigate the
training patterns of different layers in large language models (LLMs), through
the lens of gradient, when training with different responses and initial
models. We are specifically interested in how fast vs. slow thinking affects
the layer-wise gradients, given the recent popularity of training LLMs on
reasoning paths such as chain-of-thoughts (CoT) and process rewards. In our
study, fast thinking without CoT leads to larger gradients and larger
differences of gradients across layers than slow thinking (Detailed CoT),
indicating the learning stability brought by the latter. Moreover, pre-trained
LLMs are less affected by the instability of fast thinking than
instruction-tuned LLMs. Additionally, we study whether the gradient patterns
can reflect the correctness of responses when training different LLMs using
slow vs. fast thinking paths. The results show that the gradients of slow
thinking can distinguish correct and irrelevant reasoning paths. As a
comparison, we conduct similar gradient analyses on non-reasoning knowledge
learning tasks, on which, however, trivially increasing the response length
does not lead to similar behaviors of slow thinking. Our study strengthens
fundamental understandings of LLM training and sheds novel insights on its
efficiency and stability, which pave the way towards building a generalizable
System-2 agent. Our code, data, and gradient statistics can be found in:
https://github.com/MingLiiii/Layer_Gradient.
DELTA: Dense Efficient Long-range 3D Tracking for any video
Authors: name: Tuan Duc Ngo, name: Peiye Zhuang, name: Chuang Gan, name: Evangelos Kalogerakis, name: Sergey Tulyakov, name: Hsin-Ying Lee, name: Chaoyang Wang
Published: 2024-10-31 17:59:01

Tracking dense 3D motion from monocular videos remains challenging,
particularly when aiming for pixel-level precision over long sequences. We
introduce \Approach, a novel method that efficiently tracks every pixel in 3D
space, enabling accurate motion estimation across entire videos. Our approach
leverages a joint global-local attention mechanism for reduced-resolution
tracking, followed by a transformer-based upsampler to achieve high-resolution
predictions. Unlike existing methods, which are limited by computational
inefficiency or sparse tracking, \Approach delivers dense 3D tracking at scale,
running over 8x faster than previous methods while achieving state-of-the-art
accuracy. Furthermore, we explore the impact of depth representation on
tracking performance and identify log-depth as the optimal choice. Extensive
experiments demonstrate the superiority of \Approach on multiple benchmarks,
achieving new state-of-the-art results in both 2D and 3D dense tracking tasks.
Our method provides a robust solution for applications requiring fine-grained,
long-term motion tracking in 3D space.
Language Models can Self-Lengthen to Generate Long Texts
Authors: name: Shanghaoran Quan, name: Tianyi Tang, name: Bowen Yu, name: An Yang, name: Dayiheng Liu, name: Bofei Gao, name: Jianhong Tu, name: Yichang Zhang, name: Jingren Zhou, name: Junyang Lin
Published: 2024-10-31 13:47:10

Recent advancements in Large Language Models (LLMs) have significantly
enhanced their ability to process long contexts, yet a notable gap remains in
generating long, aligned outputs. This limitation stems from a training gap
where pre-training lacks effective instructions for long-text generation, and
post-training data primarily consists of short query-response pairs. Current
approaches, such as instruction backtranslation and behavior imitation, face
challenges including data quality, copyright issues, and constraints on
proprietary model usage. In this paper, we introduce an innovative iterative
training framework called Self-Lengthen that leverages only the intrinsic
knowledge and skills of LLMs without the need for auxiliary data or proprietary
models. The framework consists of two roles: the Generator and the Extender.
The Generator produces the initial response, which is then split and expanded
by the Extender. This process results in a new, longer response, which is used
to train both the Generator and the Extender iteratively. Through this process,
the models are progressively trained to handle increasingly longer responses.
Experiments on benchmarks and human evaluations show that Self-Lengthen
outperforms existing methods in long-text generation, when applied to top
open-source LLMs such as Qwen2 and LLaMA3. Our code is publicly available at
https://github.com/QwenLM/Self-Lengthen.
Constraint Back-translation Improves Complex Instruction Following of
  Large Language Models
Authors: name: Yunjia Qi, name: Hao Peng, name: Xiaozhi Wang, name: Bin Xu, name: Lei Hou, name: Juanzi Li
Published: 2024-10-31 17:42:26

Large language models (LLMs) struggle to follow instructions with complex
constraints in format, length, etc. Following the conventional
instruction-tuning practice, previous works conduct post-training on complex
instruction-response pairs generated by feeding complex instructions to
advanced LLMs. However, even advanced LLMs cannot follow complex instructions
well, thus limiting the quality of generated data. In this work, we find that
existing datasets inherently contain implicit complex constraints and propose a
novel data generation technique, constraint back-translation. Specifically, we
take the high-quality instruction-response pairs in existing datasets and only
adopt advanced LLMs to add complex constraints already met by the responses to
the instructions, which naturally reduces costs and data noise. In the
experiments, we adopt Llama3-70B-Instruct to back-translate constraints and
create a high-quality complex instruction-response dataset, named CRAB. We
present that post-training on CRAB improves multiple backbone LLMs' complex
instruction-following ability, evaluated on extensive instruction-following
benchmarks. We further find that constraint back-translation also serves as a
useful auxiliary training objective in post-training. Our code, data, and
models will be released to facilitate future research.
BenchX: A Unified Benchmark Framework for Medical Vision-Language
  Pretraining on Chest X-Rays
Authors: name: Yang Zhou, name: Tan Li Hui Faith, name: Yanyu Xu, name: Sicong Leng, name: Xinxing Xu, name: Yong Liu, name: Rick Siow Mong Goh
Published: 2024-10-29 11:53:18

Medical Vision-Language Pretraining (MedVLP) shows promise in learning
generalizable and transferable visual representations from paired and unpaired
medical images and reports. MedVLP can provide useful features to downstream
tasks and facilitate adapting task-specific models to new setups using fewer
examples. However, existing MedVLP methods often differ in terms of datasets,
preprocessing, and finetuning implementations. This pose great challenges in
evaluating how well a MedVLP method generalizes to various clinically-relevant
tasks due to the lack of unified, standardized, and comprehensive benchmark. To
fill this gap, we propose BenchX, a unified benchmark framework that enables
head-to-head comparison and systematical analysis between MedVLP methods using
public chest X-ray datasets. Specifically, BenchX is composed of three
components: 1) Comprehensive datasets covering nine datasets and four medical
tasks; 2) Benchmark suites to standardize data preprocessing, train-test
splits, and parameter selection; 3) Unified finetuning protocols that
accommodate heterogeneous MedVLP methods for consistent task adaptation in
classification, segmentation, and report generation, respectively. Utilizing
BenchX, we establish baselines for nine state-of-the-art MedVLP methods and
found that the performance of some early MedVLP methods can be enhanced to
surpass more recent ones, prompting a revisiting of the developments and
conclusions from prior works in MedVLP. Our code are available at
https://github.com/yangzhou12/BenchX.
Learning Video Representations without Natural Videos
Authors: name: Xueyang Yu, name: Xinlei Chen, name: Yossi Gandelsman
Published: 2024-10-31 17:59:30

In this paper, we show that useful video representations can be learned from
synthetic videos and natural images, without incorporating natural videos in
the training. We propose a progression of video datasets synthesized by simple
generative processes, that model a growing set of natural video properties
(e.g. motion, acceleration, and shape transformations). The downstream
performance of video models pre-trained on these generated datasets gradually
increases with the dataset progression. A VideoMAE model pre-trained on our
synthetic videos closes 97.2% of the performance gap on UCF101 action
classification between training from scratch and self-supervised pre-training
from natural videos, and outperforms the pre-trained model on HMDB51.
Introducing crops of static images to the pre-training stage results in similar
performance to UCF101 pre-training and outperforms the UCF101 pre-trained model
on 11 out of 14 out-of-distribution datasets of UCF101-P. Analyzing the
low-level properties of the datasets, we identify correlations between frame
diversity, frame similarity to natural data, and downstream performance. Our
approach provides a more controllable and transparent alternative to video data
curation processes for pre-training.
Can Models Help Us Create Better Models? Evaluating LLMs as Data
  Scientists
Authors: name: Michał Pietruszka, name: Łukasz Borchmann, name: Aleksander Jędrosz, name: Paweł Morawiecki
Published: 2024-10-30 17:59:01

We present a benchmark for large language models designed to tackle one of
the most knowledge-intensive tasks in data science: writing feature engineering
code, which requires domain knowledge in addition to a deep understanding of
the underlying problem and data structure. The model is provided with a dataset
description in a prompt and asked to generate code transforming it. The
evaluation score is derived from the improvement achieved by an XGBoost model
fit on the modified dataset compared to the original data. By an extensive
evaluation of state-of-the-art models and comparison to well-established
benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and
efficiently assess the broad capabilities of LLMs, in contrast to the existing
methods.
SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video
  Generation
Authors: name: Yining Hong, name: Beide Liu, name: Maxine Wu, name: Yuanhao Zhai, name: Kai-Wei Chang, name: Lingjie Li, name: Kevin Lin, name: Chung-Ching Lin, name: Jianfeng Wang, name: Zhengyuan Yang, name: Yingnian Wu, name: Lijuan Wang
Published: 2024-10-30 17:55:52

Human beings are endowed with a complementary learning system, which bridges
the slow learning of general world dynamics with fast storage of episodic
memory from a new experience. Previous video generation models, however,
primarily focus on slow learning by pre-training on vast amounts of data,
overlooking the fast learning phase crucial for episodic memory storage. This
oversight leads to inconsistencies across temporally distant frames when
generating longer videos, as these frames fall beyond the model's context
window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning
system for action-driven long video generation. Our approach incorporates a
masked conditional video diffusion model for the slow learning of world
dynamics, alongside an inference-time fast learning strategy based on a
temporal LoRA module. Specifically, the fast learning process updates its
temporal LoRA parameters based on local inputs and outputs, thereby efficiently
storing episodic memory in its parameters. We further propose a slow-fast
learning loop algorithm that seamlessly integrates the inner fast learning loop
into the outer slow learning loop, enabling the recall of prior multi-episode
experiences for context-aware skill learning. To facilitate the slow learning
of an approximate world model, we collect a large-scale dataset of 200k videos
with language action annotations, covering a wide range of scenarios. Extensive
experiments show that SlowFast-VGen outperforms baselines across various
metrics for action-driven video generation, achieving an FVD score of 514
compared to 782, and maintaining consistency in longer videos, with an average
of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm
significantly enhances performances on long-horizon planning tasks as well.
Project Website: https://slowfast-vgen.github.io
On Memorization of Large Language Models in Logical Reasoning
Authors: name: Chulin Xie, name: Yangsibo Huang, name: Chiyuan Zhang, name: Da Yu, name: Xinyun Chen, name: Bill Yuchen Lin, name: Bo Li, name: Badih Ghazi, name: Ravi Kumar
Published: 2024-10-30 15:31:54

Large language models (LLMs) achieve good performance on challenging
reasoning benchmarks, yet could also make basic reasoning mistakes. This
contrasting behavior is puzzling when it comes to understanding the mechanisms
behind LLMs' reasoning capabilities. One hypothesis is that the increasingly
high and nearly saturated performance on common reasoning benchmarks could be
due to the memorization of similar problems. In this paper, we systematically
investigate this hypothesis with a quantitative measurement of memorization in
reasoning tasks, using a dynamically generated logical reasoning benchmark
based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate
the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet
fail when those puzzles are slightly perturbed, suggesting that the models
heavily rely on memorization to solve those training puzzles. On the other
hand, we show that while fine-tuning leads to heavy memorization, it also
consistently improves generalization performance. In-depth analyses with
perturbation tests, cross difficulty-level transferability, probing model
internals, and fine-tuning with wrong answers suggest that the LLMs learn to
reason on K&K puzzles despite training data memorization. This phenomenon
indicates that LLMs exhibit a complex interplay between memorization and
genuine reasoning abilities. Finally, our analysis with per-sample memorization
score sheds light on how LLMs switch between reasoning and memorization in
solving logical puzzles. Our code and data are available at
https://memkklogic.github.io.
Decoding Reading Goals from Eye Movements
Authors: name: Omer Shubi, name: Cfir Avraham Hadar, name: Yevgeni Berzak
Published: 2024-10-28 06:40:03

Readers can have different goals with respect to the text they are reading.
Can these goals be decoded from the pattern of their eye movements over the
text? In this work, we examine for the first time whether it is possible to
decode two types of reading goals that are common in daily life: information
seeking and ordinary reading. Using large scale eye-tracking data, we apply to
this task a wide range of state-of-the-art models for eye movements and text
that cover different architectural and data representation strategies, and
further introduce a new model ensemble. We systematically evaluate these models
at three levels of generalization: new textual item, new participant, and the
combination of both. We find that eye movements contain highly valuable signals
for this task. We further perform an error analysis which builds on prior
empirical findings on differences between ordinary reading and information
seeking and leverages rich textual annotations. This analysis reveals key
properties of textual items and participant eye movements that contribute to
the difficulty of the task.
ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
Authors: name: Anurag Bagchi, name: Zhipeng Bao, name: Yu-Xiong Wang, name: Pavel Tokmakov, name: Martial Hebert
Published: 2024-10-30 17:59:26

We present REM, a framework for segmenting a wide range of concepts in video
that can be described through natural language. Our method capitalizes on
visual-language representations learned by video diffusion models on
Internet-scale datasets. A key insight of our approach is preserving as much of
the generative model's original representation as possible, while fine-tuning
it on narrow-domain Referral Object Segmentation datasets. As a result, our
framework can accurately segment and track rare and unseen objects, despite
being trained on object masks from a limited set of categories. Additionally,
it can generalize to non-object dynamic concepts, such as waves crashing in the
ocean, as demonstrated in our newly introduced benchmark for Referral Video
Process Segmentation (Ref-VPS). Our experiments show that REM performs on par
with state-of-the-art approaches on in-domain datasets, like Ref-DAVIS, while
outperforming them by up to twelve points in terms of region similarity on
out-of-domain data, leveraging the power of Internet-scale pre-training.