Orionnnn/hub-stats huggingface dataset:
Open Problems in Mechanistic Interpretability
Authors: name: Lee Sharkey, name: Bilal Chughtai, name: Joshua Batson, name: Jack Lindsey, name: Jeff Wu, name: Lucius Bushnaq, name: Nicholas Goldowsky-Dill, name: Stefan Heimersheim, name: Alejandro Ortega, name: Joseph Bloom, name: Stella Biderman, name: Adria Garriga-Alonso, name: Arthur Conmy, name: Neel Nanda, name: Jessica Rumbelow, name: Martin Wattenberg, name: Nandi Schoots, name: Joseph Miller, name: Eric J. Michaud, name: Stephen Casper, name: Max Tegmark, name: William Saunders, name: David Bau, name: Eric Todd, name: Atticus Geiger, name: Mor Geva, name: Jesse Hoogland, name: Daniel Murfet, name: Tom McGrath
Published: 2025-01-27 20:57:18

Mechanistic interpretability aims to understand the computational mechanisms
underlying neural networks' capabilities in order to accomplish concrete
scientific and engineering goals. Progress in this field thus promises to
provide greater assurance over AI system behavior and shed light on exciting
scientific questions about the nature of intelligence. Despite recent progress
toward these goals, there are many open problems in the field that require
solutions before many scientific and practical benefits can be realized: Our
methods require both conceptual and practical improvements to reveal deeper
insights; we must figure out how best to apply our methods in pursuit of
specific goals; and the field must grapple with socio-technical challenges that
influence and are influenced by our work. This forward-facing review discusses
the current frontier of mechanistic interpretability and the open problems that
the field may benefit from prioritizing.
Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
Authors: name: J. Pablo Muñoz, name: Jinjie Yuan, name: Nilesh Jain
Published: 2025-01-23 02:14:08

The rapid expansion of Large Language Models (LLMs) has posed significant
challenges regarding the computational resources required for fine-tuning and
deployment. Recent advancements in low-rank adapters have demonstrated their
efficacy in parameter-efficient fine-tuning (PEFT) of these models. This
retrospective paper comprehensively discusses innovative approaches that
synergize low-rank representations with Neural Architecture Search (NAS)
techniques, particularly weight-sharing super-networks. Robust solutions for
compressing and fine-tuning large pre-trained models are developed by
integrating these methodologies. Our analysis highlights the potential of these
combined strategies to democratize the use of LLMs, making them more accessible
for deployment in resource-constrained environments. The resulting models
exhibit reduced memory footprints and faster inference times, paving the way
for more practical and scalable applications of LLMs. Models and code are
available at
https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task
  Language Understanding
Authors: name: Sankalp KJ, name: Ashutosh Kumar, name: Laxmaan Balaji, name: Nikunj Kotecha, name: Vinija Jain, name: Aman Chadha, name: Sreyoshi Bhaduri
Published: 2025-01-27 03:19:03

Known by more than 1.5 billion people in the Indian subcontinent, Indic
languages present unique challenges and opportunities for natural language
processing (NLP) research due to their rich cultural heritage, linguistic
diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark
designed to evaluate Large Language Models (LLMs) across Indic languages,
building upon the MMLU Pro (Massive Multitask Language Understanding)
framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi,
Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique
challenges and opportunities presented by the linguistic diversity of the
Indian subcontinent. This benchmark encompasses a wide range of tasks in
language comprehension, reasoning, and generation, meticulously crafted to
capture the intricacies of Indian languages. IndicMMLU-Pro provides a
standardized evaluation framework to push the research boundaries in Indic
language AI, facilitating the development of more accurate, efficient, and
culturally sensitive models. This paper outlines the benchmarks' design
principles, task taxonomy, and data collection methodology, and presents
baseline results from state-of-the-art multilingual models.
CodeMonkeys: Scaling Test-Time Compute for Software Engineering
Authors: name: Ryan Ehrlich, name: Bradley Brown, name: Jordan Juravsky, name: Ronald Clark, name: Christopher Ré, name: Azalia Mirhoseini
Published: 2025-01-24 18:58:40

Scaling test-time compute is a promising axis for improving LLM capabilities.
However, test-time compute can be scaled in a variety of ways, and effectively
combining different approaches remains an active area of research. Here, we
explore this problem in the context of solving real-world GitHub issues from
the SWE-bench dataset. Our system, named CodeMonkeys, allows models to
iteratively edit a codebase by jointly generating and running a testing script
alongside their draft edit. We sample many of these multi-turn trajectories for
every issue to generate a collection of candidate edits. This approach lets us
scale "serial" test-time compute by increasing the number of iterations per
trajectory and "parallel" test-time compute by increasing the number of
trajectories per problem. With parallel scaling, we can amortize up-front costs
across multiple downstream samples, allowing us to identify relevant codebase
context using the simple method of letting an LLM read every file. In order to
select between candidate edits, we combine voting using model-generated tests
with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys
resolves 57.4% of issues from SWE-bench Verified using a budget of
approximately 2300 USD. Our selection method can also be used to combine
candidates from different sources. Selecting over an ensemble of edits from
existing top SWE-bench Verified submissions obtains a score of 66.2% and
outperforms the best member of the ensemble on its own. We fully release our
code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.
Visual Generation Without Guidance
Authors: name: Huayu Chen, name: Kai Jiang, name: Kaiwen Zheng, name: Jianfei Chen, name: Hang Su, name: Jun Zhu
Published: 2025-01-26 06:48:05

Classifier-Free Guidance (CFG) has been a default technique in various visual
generative models, yet it requires inference from both conditional and
unconditional models during sampling. We propose to build visual models that
are free from guided sampling. The resulting algorithm, Guidance-Free Training
(GFT), matches the performance of CFG while reducing sampling to a single
model, halving the computational cost. Unlike previous distillation-based
approaches that rely on pretrained CFG networks, GFT enables training directly
from scratch. GFT is simple to implement. It retains the same maximum
likelihood objective as CFG and differs mainly in the parameterization of
conditional models. Implementing GFT requires only minimal modifications to
existing codebases, as most design choices and hyperparameters are directly
inherited from CFG. Our extensive experiments across five distinct visual
models demonstrate the effectiveness and versatility of GFT. Across domains of
diffusion, autoregressive, and masked-prediction modeling, GFT consistently
achieves comparable or even lower FID scores, with similar diversity-fidelity
trade-offs compared with CFG baselines, all while being guidance-free. Code
will be available at https://github.com/thu-ml/GFT.
Are Vision Language Models Texture or Shape Biased and Can We Steer
  Them?
Authors: name: Paul Gavrikov, name: Jovita Lukasik, name: Steffen Jung, name: Robert Geirhos, name: Bianca Lamm, name: Muhammad Jehanzeb Mirza, name: Margret Keuper, name: Janis Keuper
Published: 2024-03-14 09:07:14

Vision language models (VLMs) have drastically changed the computer vision
model landscape in only a few years, opening an exciting array of new
applications from zero-shot image classification, over to image captioning, and
visual question answering. Unlike pure vision models, they offer an intuitive
way to access visual content through language prompting. The wide applicability
of such models encourages us to ask whether they also align with human vision -
specifically, how far they adopt human-induced visual biases through multimodal
fusion, or whether they simply inherit biases from pure vision models. One
important visual bias is the texture vs. shape bias, or the dominance of local
over global information. In this paper, we study this bias in a wide range of
popular VLMs. Interestingly, we find that VLMs are often more shape-biased than
their vision encoders, indicating that visual biases are modulated to some
extent through text in multimodal models. If text does indeed influence visual
biases, this suggests that we may be able to steer visual biases not just
through visual input but also through language: a hypothesis that we confirm
through extensive experiments. For instance, we are able to steer shape bias
from as low as 49% to as high as 72% through prompting alone. For now, the
strong human bias towards shape (96%) remains out of reach for all tested VLMs.
OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale
  Synthetic Personas
Authors: name: Xiaoyang Wang, name: Hongming Zhang, name: Tao Ge, name: Wenhao Yu, name: Dian Yu, name: Dong Yu
Published: 2025-01-26 07:07:01

Customizable role-playing in large language models (LLMs), also known as
character generalization, is gaining increasing attention for its versatility
and cost-efficiency in developing and deploying role-playing dialogue agents.
This study explores a large-scale data synthesis approach to equip LLMs with
character generalization capabilities. We begin by synthesizing large-scale
character profiles using personas from Persona Hub and then explore two
strategies: response rewriting and response generation, to create
character-aligned instructional responses. To validate the effectiveness of our
synthetic instruction tuning data for character generalization, we perform
supervised fine-tuning (SFT) using the LLaMA-3 8B model. Our best-performing
model strengthens the original LLaMA-3 8B Instruct model and achieves
performance comparable to GPT-4o models on role-playing dialogue. We release
our synthetic characters and instruction-tuning dialogues to support public
research.
Return of the Encoder: Maximizing Parameter Efficiency for SLMs
Authors: name: Mohamed Elfeki, name: Rui Liu, name: Chad Voegele
Published: 2025-01-27 18:06:36

The dominance of large decoder-only language models has overshadowed
encoder-decoder architectures, despite their fundamental efficiency advantages
in sequence processing. For small language models (SLMs) - those with 1 billion
parameters or fewer - our systematic analysis across GPU, CPU, and NPU
platforms reveals that encoder-decoder architectures achieve 47% lower
first-token latency and 4.7x higher throughput compared to decoder-only models
on edge devices. These gains may be attributed to encoder-decoder's one-time
input processing and efficient separation of understanding and generation
phases.
  We introduce a novel knowledge distillation framework that enables
encoder-decoder models to leverage capabilities from large scalable
decoder-only teachers while preserving their architectural advantages,
achieving up to 6 average performance points improvement across diverse tasks,
with significant gains in asymmetric sequence tasks where input and output
distributions can benefit from different processing approaches.
  When combined with modern advances like Rotary Positional Embeddings (RoPE)
and Vision encoders, our systematic investigation demonstrates that
encoder-decoder architectures provide a more practical path toward deploying
capable language models in resource-constrained environments. Our findings
challenge the prevailing trend toward decoder-only scaling, showing that
architectural choices become increasingly crucial as parameter budgets
decrease, particularly for on-device and edge deployments where computational
efficiency is paramount.
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for
  Mixture-of-Experts Language Models
Authors: name: Samira Abnar, name: Harshay Shah, name: Dan Busbridge, name: Alaaeldin Mohamed Elnouby Ali, name: Josh Susskind, name: Vimal Thilak
Published: 2025-01-21 18:51:15

Scaling the capacity of language models has consistently proven to be a
reliable approach for improving performance and unlocking new capabilities.
Capacity can be primarily defined by two dimensions: the number of model
parameters and the compute per example. While scaling typically involves
increasing both, the precise interplay between these factors and their combined
contribution to overall capacity remains not fully understood. We explore this
relationship in the context of sparse Mixture-of-Experts (MoEs), which allow
scaling the number of parameters without proportionally increasing the FLOPs
per example. We investigate how varying the sparsity level, i.e., the fraction
of inactive parameters, impacts model's performance during pretraining and
downstream few-shot evaluation. We find that under different constraints (e.g.,
parameter size and total training compute), there is an optimal level of
sparsity that improves both training efficiency and model performance. These
results provide a better understanding of the impact of sparsity in scaling
laws for MoEs and complement existing works in this area, offering insights for
designing more efficient architectures.
Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for
  Speech Generation
Authors: name: Haorui He, name: Zengqiang Shang, name: Chaoren Wang, name: Xuyuan Li, name: Yicheng Gu, name: Hua Hua, name: Liwei Liu, name: Chen Yang, name: Jiaqi Li, name: Peiyang Shi, name: Yuancheng Wang, name: Kai Chen, name: Pengyuan Zhang, name: Zhizheng Wu
Published: 2025-01-27 09:59:20

Recent advancements in speech generation have been driven by the large-scale
training datasets. However, current models fall short of capturing the
spontaneity and variability inherent in real-world human speech, due to their
reliance on audiobook datasets limited to formal read-aloud speech styles. To
bridge this gap, we introduce Emilia-Pipe, an open-source preprocessing
pipeline to extract high-quality training data from valuable yet underexplored
in-the-wild data that capture spontaneous human speech in real-world contexts.
By leveraging Emilia-Pipe, we construct Emilia, the first multilingual speech
generation dataset derived from in-the-wild speech data. This dataset comprises
over 101k hours of speech across six languages: English, Chinese, German,
French, Japanese, and Korean. Besides, we expand Emilia to Emilia-Large, a
dataset exceeding 216k hours, making it the largest open-source speech
generation dataset available. Extensive experiments demonstrate that Emilia
significantly outperforms traditional audiobook datasets in generating
spontaneous and human-like speech, showcasing superior performance in capturing
diverse speaker timbre and speaking styles of real-world human speech.
Furthermore, this work underscores the importance of scaling dataset size to
advance speech generation research and validates the effectiveness of Emilia
for both multilingual and crosslingual speech generation.
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language
  Model Born from Transformer
Authors: name: Lin Yueyu, name: Li Zhiyuan, name: Peter Yue, name: Liu Xiao
Published: 2025-01-26 15:56:56

As is known, hybrid quadratic and subquadratic attention models in multi-head
architectures have surpassed both Transformer and Linear RNN models , with
these works primarily focusing on reducing KV complexity and improving
efficiency. For further research on expressiveness, we introduce our series of
models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which
aims to make RNN more expressive and demonstrates state tracking ability beyond
transformers. We work with QRWK 32B based on RWKV-6 architecture, another
approach that reduces the entire knowledge processing time to just 8 hours
using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the
distillation process can utilize any LLM, not just Qwen, and enables knowledge
transfer from larger LLMs to smaller ones with more fewer tokens. We will
explain the detailed process and share our insights on building more powerful
foundation models. Please note that this is an ongoing work that will be
updated continuously. The model checkpoints and source code are available at
https://github.com/yynil/RWKVInside{https://github.com/yynil/RWKVInside},
https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
iFormer: Integrating ConvNet and Transformer for Mobile Application
Authors: name: Chuanyang Zheng
Published: 2025-01-26 02:34:58

We present a new family of mobile hybrid vision networks, called iFormer,
with a focus on optimizing latency and accuracy on mobile applications. iFormer
effectively integrates the fast local representation capacity of convolution
with the efficient global modeling ability of self-attention. The local
interactions are derived from transforming a standard convolutional network,
i.e., ConvNeXt, to design a more lightweight mobile network. Our newly
introduced mobile modulation attention removes memory-intensive operations in
MHA and employs an efficient modulation mechanism to boost dynamic global
representational capacity. We conduct comprehensive experiments demonstrating
that iFormer outperforms existing lightweight networks across various tasks.
Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k
with a latency of only 1.10 ms on an iPhone 13, surpassing the recently
proposed MobileNetV4 under similar latency constraints. Additionally, our
method shows significant improvements in downstream tasks, including COCO
object detection, instance segmentation, and ADE20k semantic segmentation,
while still maintaining low latency on mobile devices for high-resolution
inputs in these scenarios.
Feasible Learning
Authors: name: Juan Ramirez, name: Ignacio Hounie, name: Juan Elenter, name: Jose Gallego-Posada, name: Meraj Hashemizadeh, name: Alejandro Ribeiro, name: Simon Lacoste-Julien
Published: 2025-01-24 20:39:38

We introduce Feasible Learning (FL), a sample-centric learning paradigm where
models are trained by solving a feasibility problem that bounds the loss for
each training sample. In contrast to the ubiquitous Empirical Risk Minimization
(ERM) framework, which optimizes for average performance, FL demands
satisfactory performance on every individual data point. Since any model that
meets the prescribed performance threshold is a valid FL solution, the choice
of optimization algorithm and its dynamics play a crucial role in shaping the
properties of the resulting solutions. In particular, we study a primal-dual
approach which dynamically re-weights the importance of each sample during
training. To address the challenge of setting a meaningful threshold in
practice, we introduce a relaxation of FL that incorporates slack variables of
minimal norm. Our empirical analysis, spanning image classification, age
regression, and preference optimization in large language models, demonstrates
that models trained via FL can learn from data while displaying improved tail
behavior compared to ERM, with only a marginal impact on average performance.
Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with
  Modality-Aware Sparsity
Authors: name: Weixin Liang, name: Junhong Shen, name: Genghan Zhang, name: Ning Dong, name: Luke Zettlemoyer, name: Lili Yu
Published: 2025-01-27 18:35:05

State Space Models (SSMs) have emerged as efficient alternatives to
Transformers for sequential modeling, but their inability to leverage
modality-specific features limits their performance in multi-modal pretraining.
Here, we propose Mixture-of-Mamba, a novel SSM architecture that introduces
modality-aware sparsity through modality-specific parameterization of the Mamba
block. Building on Mixture-of-Transformers (W. Liang et al. arXiv:2411.04996;
2024), we extend the benefits of modality-aware sparsity to SSMs while
preserving their computational efficiency. We evaluate Mixture-of-Mamba across
three multi-modal pretraining settings: Transfusion (interleaved text and
continuous image tokens with diffusion loss), Chameleon (interleaved text and
discrete image tokens), and an extended three-modality framework incorporating
speech. Mixture-of-Mamba consistently reaches the same loss values at earlier
training steps with significantly reduced computational costs. In the
Transfusion setting, Mixture-of-Mamba achieves equivalent image loss using only
34.76% of the training FLOPs at the 1.4B scale. In the Chameleon setting,
Mixture-of-Mamba reaches similar image loss with just 42.50% of the FLOPs at
the 1.4B scale, and similar text loss with just 65.40% of the FLOPs. In the
three-modality setting, MoM matches speech loss at 24.80% of the FLOPs at the
1.4B scale. Our ablation study highlights the synergistic effects of decoupling
projection components, where joint decoupling yields greater gains than
individual modifications. These results establish modality-aware sparsity as a
versatile and effective design principle, extending its impact from
Transformers to SSMs and setting new benchmarks in multi-modal pretraining. Our
code can be accessed at https://github.com/Weixin-Liang/Mixture-of-Mamba
Towards General-Purpose Model-Free Reinforcement Learning
Authors: name: Scott Fujimoto, name: Pierluca D'Oro, name: Amy Zhang, name: Yuandong Tian, name: Michael Rabbat
Published: 2025-01-27 15:36:37

Reinforcement learning (RL) promises a framework for near-universal
problem-solving. In practice however, RL algorithms are often tailored to
specific benchmarks, relying on carefully tuned hyperparameters and algorithmic
choices. Recently, powerful model-based RL methods have shown impressive
general results across benchmarks but come at the cost of increased complexity
and slow run times, limiting their broader applicability. In this paper, we
attempt to find a unifying model-free deep RL algorithm that can address a
diverse class of domains and problem settings. To achieve this, we leverage
model-based representations that approximately linearize the value function,
taking advantage of the denser task objectives used by model-based RL while
avoiding the costs associated with planning or simulated trajectories. We
evaluate our algorithm, MR.Q, on a variety of common RL benchmarks with a
single set of hyperparameters and show a competitive performance against
domain-specific and general baselines, providing a concrete step towards
building general-purpose model-free deep RL algorithms.