Orionnnn/hub-stats huggingface dataset:
Guaranteed Guess: A Language Modeling Approach for CISC-to-RISC
  Transpilation with Testing Guarantees
Authors: name: Ahmed Heakl, name: Sarim Hashmi, name: Chaimaa Abi, name: Celine Lee, name: Abdulrahman Mahmoud
Published: 2025-06-17 15:06:54

The hardware ecosystem is rapidly evolving, with increasing interest in
translating low-level programs across different instruction set architectures
(ISAs) in a quick, flexible, and correct way to enhance the portability and
longevity of existing code. A particularly challenging class of this
transpilation problem is translating between complex- (CISC) and reduced-
(RISC) hardware architectures, due to fundamental differences in instruction
complexity, memory models, and execution paradigms. In this work, we introduce
GG (Guaranteed Guess), an ISA-centric transpilation pipeline that combines the
translation power of pre-trained large language models (LLMs) with the rigor of
established software testing constructs. Our method generates candidate
translations using an LLM from one ISA to another, and embeds such translations
within a software-testing framework to build quantifiable confidence in the
translation. We evaluate our GG approach over two diverse datasets, enforce
high code coverage (>98%) across unit tests, and achieve functional/semantic
correctness of 99% on HumanEval programs and 49% on BringupBench programs,
respectively. Further, we compare our approach to the state-of-the-art Rosetta
2 framework on Apple Silicon, showcasing 1.73x faster runtime performance,
1.47x better energy efficiency, and 2.41x better memory usage for our
transpiled code, demonstrating the effectiveness of GG for real-world
CISC-to-RISC translation tasks. We will open-source our codes, data, models,
and benchmarks to establish a common foundation for ISA-level code translation
research.
LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
Authors: name: Xiaoran Liu, name: Zhigeng Liu, name: Zengfeng Huang, name: Qipeng Guo, name: Ziwei He, name: Xipeng Qiu
Published: 2025-06-17 11:45:37

Large Language Diffusion Models, or diffusion LLMs, have emerged as a
significant focus in NLP research, with substantial effort directed toward
understanding their scalability and downstream task performance. However, their
long-context capabilities remain unexplored, lacking systematic analysis or
methods for context extension. In this work, we present the first systematic
investigation comparing the long-context performance of diffusion LLMs and
traditional auto-regressive LLMs. We first identify a unique characteristic of
diffusion LLMs, unlike auto-regressive LLMs, they maintain remarkably
\textit{stable perplexity} during direct context extrapolation.
Furthermore, where auto-regressive models fail outright during the
Needle-In-A-Haystack task with context exceeding their pretrained length, we
discover diffusion LLMs exhibit a distinct \textit{local perception}
phenomenon, enabling successful retrieval from recent context segments. We
explain both phenomena through the lens of Rotary Position Embedding (RoPE)
scaling theory. Building on these observations, we propose LongLLaDA, a
training-free method that integrates LLaDA with the NTK-based RoPE
extrapolation. Our results validate that established extrapolation scaling laws
remain effective for extending the context windows of diffusion LLMs.
Furthermore, we identify long-context tasks where diffusion LLMs outperform
auto-regressive LLMs and others where they fall short. Consequently, this study
establishes the first context extrapolation method for diffusion LLMs while
providing essential theoretical insights and empirical benchmarks critical for
advancing future research on long-context diffusion LLMs.
Stream-Omni: Simultaneous Multimodal Interactions with Large
  Language-Vision-Speech Model
Authors: name: Shaolei Zhang, name: Shoutao Guo, name: Qingkai Fang, name: Yan Zhou, name: Yang Feng
Published: 2025-06-16 16:06:45

The emergence of GPT-4o-like large multimodal models (LMMs) has raised the
exploration of integrating text, vision, and speech modalities to support more
flexible multimodal interaction. Existing LMMs typically concatenate
representation of modalities along the sequence dimension and feed them into a
large language model (LLM) backbone. While sequence-dimension concatenation is
straightforward for modality integration, it often relies heavily on
large-scale data to learn modality alignments. In this paper, we aim to model
the relationships between modalities more purposefully, thereby achieving more
efficient and flexible modality alignments. To this end, we propose
Stream-Omni, a large language-vision-speech model with efficient modality
alignments, which can simultaneously support interactions under various
modality combinations. Stream-Omni employs LLM as the backbone and aligns the
vision and speech to the text based on their relationships. For vision that is
semantically complementary to text, Stream-Omni uses sequence-dimension
concatenation to achieve vision-text alignment. For speech that is semantically
consistent with text, Stream-Omni introduces a CTC-based layer-dimension
mapping to achieve speech-text alignment. In this way, Stream-Omni can achieve
modality alignments with less data (especially speech), enabling the transfer
of text capabilities to other modalities. Experiments on various benchmarks
demonstrate that Stream-Omni achieves strong performance on visual
understanding, speech interaction, and vision-grounded speech interaction
tasks. Owing to the layer-dimensional mapping, Stream-Omni can simultaneously
provide intermediate text outputs (such as ASR transcriptions and model
responses) during speech interaction, offering users a comprehensive multimodal
experience.
Taming Polysemanticity in LLMs: Provable Feature Recovery via Sparse
  Autoencoders
Authors: name: Siyu Chen, name: Heejune Sheen, name: Xuyuan Xiong, name: Tianhao Wang, name: Zhuoran Yang
Published: 2025-06-16 20:58:05

We study the challenge of achieving theoretically grounded feature recovery
using Sparse Autoencoders (SAEs) for the interpretation of Large Language
Models. Existing SAE training algorithms often lack rigorous mathematical
guarantees and suffer from practical limitations such as hyperparameter
sensitivity and instability. To address these issues, we first propose a novel
statistical framework for the feature recovery problem, which includes a new
notion of feature identifiability by modeling polysemantic features as sparse
mixtures of underlying monosemantic concepts. Building on this framework, we
introduce a new SAE training algorithm based on ``bias adaptation'', a
technique that adaptively adjusts neural network bias parameters to ensure
appropriate activation sparsity. We theoretically prove that this
algorithm correctly recovers all monosemantic features when input data is
sampled from our proposed statistical model. Furthermore, we develop an
improved empirical variant, Group Bias Adaptation (GBA), and
demonstrate its superior performance against benchmark methods when
applied to LLMs with up to 1.5 billion parameters. This work represents a
foundational step in demystifying SAE training by providing the first SAE
algorithm with theoretical recovery guarantees, thereby advancing the
development of more transparent and trustworthy AI systems through enhanced
mechanistic interpretability.
Infini-gram mini: Exact n-gram Search at the Internet Scale with
  FM-Index
Authors: name: Hao Xu, name: Jiacheng Liu, name: Yejin Choi, name: Noah A. Smith, name: Hannaneh Hajishirzi
Published: 2025-06-13 21:13:57

Language models are trained mainly on massive text data from the Internet,
and it becomes increasingly important to understand this data source.
Exact-match search engines enable searching in large text corpora -- counting
string appearances and retrieving the enclosing documents -- yet the high
storage overhead hinders their application on Internet-scale data. We present
Infini-gram mini, an efficient and scalable system that can make petabyte-level
text corpora searchable. Based on the FM-index data structure (Ferragina and
Manzini, 2000), which simultaneously indexes and compresses text, our system
creates indexes with size only 44% of the corpus. Infini-gram mini greatly
improves upon the best existing implementation of FM-index in terms of indexing
speed (18times) and memory use during both indexing (3.2times reduction)
and querying (down to a negligible amount). We index 46TB of Internet text in
50 days with a single 128-core CPU node (or 19 hours if using 75 such nodes).
We show one important use case of Infini-gram mini in a large-scale analysis of
benchmark contamination. We find several core LM evaluation benchmarks to be
heavily contaminated in Internet crawls (up to 40% in SQuAD), which could lead
to overestimating the capabilities of language models if trained on such data.
We host a benchmark contamination bulletin to share the contamination rate of
many core and community-contributed benchmarks. We also release a web interface
and an API endpoint to serve general search queries on Infini-gram mini
indexes.
Personalizable Long-Context Symbolic Music Infilling with MIDI-RWKV
Authors: name: Christian Zhou-Zheng, name: Philippe Pasquier
Published: 2025-06-16 00:04:01

Existing work in automatic music generation has primarily focused on
end-to-end systems that produce complete compositions or continuations.
However, because musical composition is typically an iterative process, such
systems make it difficult to engage in the back-and-forth between human and
machine that is essential to computer-assisted creativity. In this study, we
address the task of personalizable, multi-track, long-context, and controllable
symbolic music infilling to enhance the process of computer-assisted
composition. We present MIDI-RWKV, a novel model based on the RWKV-7 linear
architecture, to enable efficient and coherent musical cocreation on edge
devices. We also demonstrate that MIDI-RWKV admits an effective method of
finetuning its initial state for personalization in the very-low-sample regime.
We evaluate MIDI-RWKV and its state tuning on several quantitative and
qualitative metrics, and release model weights and code at
https://github.com/christianazinn/MIDI-RWKV.
DiffusionBlocks: Blockwise Training for Generative Models via
  Score-Based Diffusion
Authors: name: Makoto Shing, name: Takuya Akiba
Published: 2025-06-17 05:44:18

Training large neural networks with end-to-end backpropagation creates
significant memory bottlenecks, limiting accessibility to state-of-the-art AI
research. We propose DiffusionBlocks, a novel training framework
that interprets neural network blocks as performing denoising operations in a
continuous-time diffusion process. By partitioning the network into
independently trainable blocks and optimizing noise level assignments based on
equal cumulative probability mass, our approach achieves significant memory
efficiency while maintaining competitive performance compared to traditional
backpropagation in generative tasks. Experiments on image generation and
language modeling tasks demonstrate memory reduction proportional to the number
of blocks while achieving superior performance. DiffusionBlocks provides a
promising pathway for democratizing access to large-scale neural network
training with limited computational resources.
Essential-Web v1.0: 24T tokens of organized web data
Authors: name: Essential AI, name: Andrew Hojel, name: Michael Pust, name: Tim Romanski, name: Yash Vanjani, name: Ritvik Kapila, name: Mohit Parmar, name: Adarsh Chaluvaraju, name: Alok Tripathy, name: Anil Thomas, name: Ashish Tanwer, name: Darsh J Shah, name: Ishaan Shah, name: Karl Stratos, name: Khoi Nguyen, name: Kurt Smith, name: Michael Callahan, name: Peter Rushton, name: Philip Monk, name: Platon Mazarakis, name: Saad Jamal, name: Saurabh Srivastava, name: Somanshu Singla, name: Ashish Vaswani
Published: 2025-06-17 02:03:36

Data plays the most prominent role in how language models acquire skills and
knowledge. The lack of massive, well-organized pre-training datasets results in
costly and inaccessible data pipelines. We present Essential-Web v1.0, a
24-trillion-token dataset in which every document is annotated with a
twelve-category taxonomy covering topic, format, content complexity, and
quality. Taxonomy labels are produced by EAI-Distill-0.5b, a fine-tuned
0.5b-parameter model that achieves an annotator agreement within 3% of
Qwen2.5-32B-Instruct. With nothing more than SQL-style filters, we obtain
competitive web-curated datasets in math (-8.0% relative to SOTA), web code
(+14.3%), STEM (+24.5%) and medical (+8.6%). Essential-Web v1.0 is available on
HuggingFace: https://huggingface.co/datasets/EssentialAI/essential-web-v1.0
BOW: Bottlenecked Next Word Exploration
Authors: name: Ming Shen, name: Zhikun Xu, name: Xiao Ye, name: Jacob Dineen, name: Ben Zhou
Published: 2025-06-16 13:58:54

Large language models (LLMs) are typically trained via next-word prediction
(NWP), which provides strong surface-level fluency but often lacks support for
robust reasoning. We propose BOttlenecked next Word exploration (BOW), a novel
RL framework that rethinks NWP by introducing a reasoning bottleneck where a
policy model first generates a reasoning path rather than predicting the next
token directly, after which a frozen judge model predicts the next token
distribution based solely on this reasoning path. We train the policy model
using GRPO with rewards that quantify how effectively the reasoning path
facilitates next-word recovery. Compared with other continual pretraining
baselines, we show that BOW improves both the general and next-word reasoning
capabilities of the base model, evaluated on various benchmarks. Our findings
show that BOW can serve as an effective and scalable alternative to vanilla
NWP.
Provably Learning from Language Feedback
Authors: name: Wanqiao Xu, name: Allen Nie, name: Ruijie Zheng, name: Aditya Modi, name: Adith Swaminathan, name: Ching-An Cheng
Published: 2025-06-12 04:35:02

Interactively learning from observation and language feedback is an
increasingly studied area driven by the emergence of large language model (LLM)
agents. While impressive empirical demonstrations have been shown, so far a
principled framing of these decision problems remains lacking. In this paper,
we formalize the Learning from Language Feedback (LLF) problem, assert
sufficient assumptions to enable learning despite latent rewards, and introduce
transfer eluder dimension as a complexity measure to characterize
the hardness of LLF problems. We show that transfer eluder dimension captures
the intuition that information in the feedback changes the learning complexity
of the LLF problem. We demonstrate cases where learning from rich language
feedback can be exponentially faster than learning from reward. We develop a
no-regret algorithm, called HELiX, that provably solves LLF problems
through sequential interactions, with performance guarantees that scale with
the transfer eluder dimension of the problem. Across several empirical domains,
we show that HELiX performs well even when repeatedly prompting LLMs
does not work reliably. Our contributions mark a first step towards designing
principled interactive learning algorithms from generic language feedback.
AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT
  and RL Synergy
Authors: name: Zihan Liu, name: Zhuolin Yang, name: Yang Chen, name: Chankyu Lee, name: Mohammad Shoeybi, name: Bryan Catanzaro, name: Wei Ping
Published: 2025-06-16 09:27:48

In this work, we investigate the synergy between supervised fine-tuning (SFT)
and reinforcement learning (RL) in developing strong reasoning models. We begin
by curating the SFT training data through two scaling strategies: increasing
the number of collected prompts and the number of generated responses per
prompt. Both approaches yield notable improvements in reasoning performance,
with scaling the number of prompts resulting in more substantial gains. We then
explore the following questions regarding the synergy between SFT and RL: (i)
Does a stronger SFT model consistently lead to better final performance after
large-scale RL training? (ii) How can we determine an appropriate sampling
temperature during RL training to effectively balance exploration and
exploitation for a given SFT initialization? Our findings suggest that (i)
holds true, provided effective RL training is conducted, particularly when the
sampling temperature is carefully chosen to maintain the temperature-adjusted
entropy around 0.3, a setting that strikes a good balance between exploration
and exploitation. Notably, the performance gap between initial SFT models
narrows significantly throughout the RL process. Leveraging a strong SFT
foundation and insights into the synergistic interplay between SFT and RL, our
AceReason-Nemotron-1.1 7B model significantly outperforms
AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among
Qwen2.5-7B-based reasoning models on challenging math and code benchmarks,
thereby demonstrating the effectiveness of our post-training recipe. We release
the model and data at: https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B
AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
Authors: name: Jingyuan Qi, name: Zhiyang Xu, name: Qifan Wang, name: Lifu Huang
Published: 2025-06-08 01:33:05

We introduce Autoregressive Retrieval Augmentation (AR-RAG), a novel paradigm
that enhances image generation by autoregressively incorporating knearest
neighbor retrievals at the patch level. Unlike prior methods that perform a
single, static retrieval before generation and condition the entire generation
on fixed reference images, AR-RAG performs context-aware retrievals at each
generation step, using prior-generated patches as queries to retrieve and
incorporate the most relevant patch-level visual references, enabling the model
to respond to evolving generation needs while avoiding limitations (e.g.,
over-copying, stylistic bias, etc.) prevalent in existing methods. To realize
AR-RAG, we propose two parallel frameworks: (1) Distribution-Augmentation in
Decoding (DAiD), a training-free plug-and-use decoding strategy that directly
merges the distribution of model-predicted patches with the distribution of
retrieved patches, and (2) Feature-Augmentation in Decoding (FAiD), a
parameter-efficient fine-tuning method that progressively smooths the features
of retrieved patches via multi-scale convolution operations and leverages them
to augment the image generation process. We validate the effectiveness of
AR-RAG on widely adopted benchmarks, including Midjourney-30K, GenEval and
DPG-Bench, demonstrating significant performance gains over state-of-the-art
image generation models.
Marrying Autoregressive Transformer and Diffusion with Multi-Reference
  Autoregression
Authors: name: Dingcheng Zhen, name: Qian Qiao, name: Tan Yu, name: Kangxi Wu, name: Ziwei Zhang, name: Siyuan Liu, name: Shunshun Yin, name: Ming Tao
Published: 2025-06-11 07:50:31

We introduce TransDiff, the first image generation model that marries
Autoregressive (AR) Transformer with diffusion models. In this joint modeling
framework, TransDiff encodes labels and images into high-level semantic
features and employs a diffusion model to estimate the distribution of image
samples. On the ImageNet 256x256 benchmark, TransDiff significantly outperforms
other image generation models based on standalone AR Transformer or diffusion
models. Specifically, TransDiff achieves a Fr\'echet Inception Distance (FID)
of 1.61 and an Inception Score (IS) of 293.4, and further provides x2 faster
inference latency compared to state-of-the-art methods based on AR Transformer
and x112 faster inference compared to diffusion-only models. Furthermore,
building on the TransDiff model, we introduce a novel image generation paradigm
called Multi-Reference Autoregression (MRAR), which performs autoregressive
generation by predicting the next image. MRAR enables the model to reference
multiple previously generated images, thereby facilitating the learning of more
diverse representations and improving the quality of generated images in
subsequent iterations. By applying MRAR, the performance of TransDiff is
improved, with the FID reduced from 1.61 to 1.42. We expect TransDiff to open
up a new frontier in the field of image generation.
SeqPE: Transformer with Sequential Position Encoding
Authors: name: Huyang Li, name: Yahui Liu, name: Hongyu Sun, name: Deng Cai, name: Leyang Cui, name: Wei Bi, name: Peilin Zhao, name: Taro Watanabe
Published: 2025-06-16 09:16:40

Since self-attention layers in Transformers are permutation invariant by
design, positional encodings must be explicitly incorporated to enable spatial
understanding. However, fixed-size lookup tables used in traditional learnable
position embeddings (PEs) limit extrapolation capabilities beyond pre-trained
sequence lengths. Expert-designed methods such as ALiBi and RoPE, mitigate this
limitation but demand extensive modifications for adapting to new modalities,
underscoring fundamental challenges in adaptability and scalability. In this
work, we present SeqPE, a unified and fully learnable position encoding
framework that represents each n-dimensional position index as a symbolic
sequence and employs a lightweight sequential position encoder to learn their
embeddings in an end-to-end manner. To regularize SeqPE's embedding space, we
introduce two complementary objectives: a contrastive objective that aligns
embedding distances with a predefined position-distance function, and a
knowledge distillation loss that anchors out-of-distribution position
embeddings to in-distribution teacher representations, further enhancing
extrapolation performance. Experiments across language modeling, long-context
question answering, and 2D image classification demonstrate that SeqPE not only
surpasses strong baselines in perplexity, exact match (EM), and
accuracy--particularly under context length extrapolation--but also enables
seamless generalization to multi-dimensional inputs without requiring manual
architectural redesign. We release our code, data, and checkpoints at
https://github.com/ghrua/seqpe.
A Technical Study into Small Reasoning Language Models
Authors: name: Xialie Zhuang, name: Peixian Ma, name: Zhikai Jia, name: Zheng Cao, name: Shiwei Liu
Published: 2025-06-16 12:18:11

The ongoing evolution of language models has led to the development of
large-scale architectures that demonstrate exceptional performance across a
wide range of tasks. However, these models come with significant computational
and energy demands, as well as potential privacy implications. In this context,
Small Reasoning Language Models (SRLMs) with approximately 0.5 billion
parameters present a compelling alternative due to their remarkable
computational efficiency and cost effectiveness, particularly in
resource-constrained environments. Despite these advantages, the limited
capacity of 0.5 billion parameter models poses challenges in handling complex
tasks such as mathematical reasoning and code generation. This research
investigates various training strategies, including supervised fine-tuning
(SFT), knowledge distillation (KD), and reinforcement learning (RL), as well as
their hybrid implementations, to enhance the performance of 0.5B SRLMs. We
analyze effective methodologies to bridge the performance gap between SRLMS and
larger models and present insights into optimal training pipelines tailored for
these smaller architectures. Through extensive experimental validation and
analysis, our work aims to provide actionable recommendations for maximizing
the reasoning capabilities of 0.5B models.