Orionnnn/hub-stats huggingface dataset:
Teaching Metric Distance to Autoregressive Multimodal Foundational
  Models
Authors: name: Jiwan Chung, name: Saejin Kim, name: Yongrae Jo, name: Jaewoo Park, name: Dongjun Min, name: Youngjae Yu
Published: 2025-03-04 08:14:51

As large language models expand beyond natural language to domains such as
mathematics, multimodal understanding, and embodied agents, tokens increasingly
reflect metric relationships rather than purely linguistic meaning. We
introduce DIST2Loss, a distance-aware framework designed to train
autoregressive discrete models by leveraging predefined distance relationships
among output tokens. At its core, DIST2Loss transforms continuous exponential
family distributions derived from inherent distance metrics into discrete,
categorical optimization targets compatible with the models' architectures.
This approach enables the models to learn and preserve meaningful distance
relationships during token generation while maintaining compatibility with
existing architectures. Empirical evaluations show consistent performance gains
in diverse multimodal applications, including visual grounding, robotic
manipulation, generative reward modeling, and image generation using
vector-quantized features. These improvements are pronounced in cases of
limited training data, highlighting DIST2Loss's effectiveness in
resource-constrained settings.
Efficient Test-Time Scaling via Self-Calibration
Authors: name: Chengsong Huang, name: Langlin Huang, name: Jixuan Leng, name: Jiacheng Liu, name: Jiaxin Huang
Published: 2025-02-25 00:21:14

Increasing test-time computation is a straightforward approach to enhancing
the quality of responses in Large Language Models (LLMs). While Best-of-N
sampling and Self-Consistency with majority voting are simple and effective,
they require a fixed number of sampling responses for each query, regardless of
its complexity. This could result in wasted computation for simpler questions
and insufficient exploration for more challenging ones. In this work, we argue
that model confidence of responses can be used for improving the efficiency of
test-time scaling. Unfortunately, LLMs are known to be overconfident and
provide unreliable confidence estimation. To address this limitation, we
introduce Self-Calibration by distilling Self-Consistency-derived confidence
into the model itself. This enables reliable confidence estimation at test time
with one forward pass. We then design confidence-based efficient test-time
scaling methods to handle queries of various difficulty, such as Early-Stopping
for Best-of-N and Self-Consistency with calibrated confidence. Experiments on
three LLMs across six datasets demonstrate the effectiveness of our approach.
Specifically, applying confidence-based Early Stopping to Best-of-N improves
MathQA accuracy from 81.0 to 83.6 with a sample budget of 16 responses,
indicating the efficacy of confidence-based sampling strategy at inference
time.
Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security
  Analysis
Authors: name: Jeffrey Yang Fan Chiang, name: Seungjae Lee, name: Jia-Bin Huang, name: Furong Huang, name: Yizheng Chen
Published: 2025-02-27 18:56:26

Recent advancements in Web AI agents have demonstrated remarkable
capabilities in addressing complex web navigation tasks. However, emerging
research shows that these agents exhibit greater vulnerability compared to
standalone Large Language Models (LLMs), despite both being built upon the same
safety-aligned models. This discrepancy is particularly concerning given the
greater flexibility of Web AI Agent compared to standalone LLMs, which may
expose them to a wider range of adversarial user inputs. To build a scaffold
that addresses these concerns, this study investigates the underlying factors
that contribute to the increased vulnerability of Web AI agents. Notably, this
disparity stems from the multifaceted differences between Web AI agents and
standalone LLMs, as well as the complex signals - nuances that simple
evaluation metrics, such as success rate, often fail to capture. To tackle
these challenges, we propose a component-level analysis and a more granular,
systematic evaluation framework. Through this fine-grained investigation, we
identify three critical factors that amplify the vulnerability of Web AI
agents; (1) embedding user goals into the system prompt, (2) multi-step action
generation, and (3) observational capabilities. Our findings highlights the
pressing need to enhance security and robustness in AI agent design and provide
actionable insights for targeted defense strategies.
General Reasoning Requires Learning to Reason from the Get-go
Authors: name: Seungwook Han, name: Jyothish Pari, name: Samuel J. Gershman, name: Pulkit Agrawal
Published: 2025-02-26 18:51:12

Large Language Models (LLMs) have demonstrated impressive real-world utility,
exemplifying artificial useful intelligence (AUI). However, their ability to
reason adaptively and robustly -- the hallmarks of artificial general
intelligence (AGI) -- remains fragile. While LLMs seemingly succeed in
commonsense reasoning, programming, and mathematics, they struggle to
generalize algorithmic understanding across novel contexts. Our experiments
with algorithmic tasks in esoteric programming languages reveal that LLM's
reasoning overfits to the training data and is limited in its transferability.
We hypothesize that the core issue underlying such limited transferability is
the coupling of reasoning and knowledge in LLMs.
  To transition from AUI to AGI, we propose disentangling knowledge and
reasoning through three key directions: (1) pretaining to reason using RL from
scratch as an alternative to the widely used next-token prediction pretraining,
(2) using a curriculum of synthetic tasks to ease the learning of a
reasoning prior for RL that can then be transferred to natural
language tasks, and (3) learning more generalizable reasoning functions using a
small context window to reduce exploiting spurious correlations between tokens.
Such a reasoning system coupled with a trained retrieval system and a large
external memory bank as a knowledge store can overcome several limitations of
existing architectures at learning to reason in novel scenarios.
PodAgent: A Comprehensive Framework for Podcast Generation
Authors: name: Yujia Xiao, name: Lei He, name: Haohan Guo, name: Fenglong Xie, name: Tan Lee
Published: 2025-03-01 11:35:17

Existing Existing automatic audio generation methods struggle to generate
podcast-like audio programs effectively. The key challenges lie in in-depth
content generation, appropriate and expressive voice production. This paper
proposed PodAgent, a comprehensive framework for creating audio programs.
PodAgent 1) generates informative topic-discussion content by designing a
Host-Guest-Writer multi-agent collaboration system, 2) builds a voice pool for
suitable voice-role matching and 3) utilizes LLM-enhanced speech synthesis
method to generate expressive conversational speech. Given the absence of
standardized evaluation criteria for podcast-like audio generation, we
developed comprehensive assessment guidelines to effectively evaluate the
model's performance. Experimental results demonstrate PodAgent's effectiveness,
significantly surpassing direct GPT-4 generation in topic-discussion dialogue
content, achieving an 87.4% voice-matching accuracy, and producing more
expressive speech through LLM-guided synthesis. Demo page:
https://podcast-agent.github.io/demo/. Source code:
https://github.com/yujxx/PodAgent.
When an LLM is apprehensive about its answers -- and when its
  uncertainty is justified
Authors: name: Petr Sychev, name: Andrey Goncharov, name: Daniil Vyazhev, name: Edvard Khalafyan, name: Alexey Zaytsev
Published: 2025-03-03 16:03:46

Uncertainty estimation is crucial for evaluating Large Language Models
(LLMs), particularly in high-stakes domains where incorrect answers result in
significant consequences. Numerous approaches consider this problem, while
focusing on a specific type of uncertainty, ignoring others. We investigate
what estimates, specifically token-wise entropy and model-as-judge (MASJ),
would work for multiple-choice question-answering tasks for different question
topics. Our experiments consider three LLMs: Phi-4, Mistral, and Qwen of
different sizes from 1.5B to 72B and 14 topics. While MASJ performs similarly
to a random error predictor, the response entropy predicts model error in
knowledge-dependent domains and serves as an effective indicator of question
difficulty: for biology ROC AUC is 0.73. This correlation vanishes for the
reasoning-dependent domain: for math questions ROC-AUC is 0.55. More
principally, we found out that the entropy measure required a reasoning amount.
Thus, data-uncertainty related entropy should be integrated within uncertainty
estimates frameworks, while MASJ requires refinement. Moreover, existing
MMLU-Pro samples are biased, and should balance required amount of reasoning
for different subdomains to provide a more fair assessment of LLMs performance.
SampleMix: A Sample-wise Pre-training Data Mixing Strategey by
  Coordinating Data Quality and Diversity
Authors: name: Xiangyu Xi, name: Deyang Kong, name: Jian Yang, name: Jiawei Yang, name: Zhengyu Chen, name: Wei Wang, name: Jingang Wang, name: Xunliang Cai, name: Shikun Zhang, name: Wei Ye
Published: 2025-03-03 13:22:11

Existing pretraining data mixing methods for large language models (LLMs)
typically follow a domain-wise methodology, a top-down process that first
determines domain weights and then performs uniform data sampling across each
domain. However, these approaches neglect significant inter-domain overlaps and
commonalities, failing to control the global diversity of the constructed
training dataset. Further, uniform sampling within domains ignores fine-grained
sample-specific features, potentially leading to suboptimal data distribution.
To address these shortcomings, we propose a novel sample-wise data mixture
approach based on a bottom-up paradigm. This method performs global
cross-domain sampling by systematically evaluating the quality and diversity of
each sample, thereby dynamically determining the optimal domain distribution.
Comprehensive experiments across multiple downstream tasks and perplexity
assessments demonstrate that SampleMix surpasses existing domain-based methods.
Meanwhile, SampleMix requires 1.4x to 2.1x training steps to achieves the
baselines' performance, highlighting the substantial potential of SampleMix to
optimize pre-training data.
Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemia
Authors: name: Chenxi Wang, name: Tianle Gu, name: Zhongyu Wei, name: Lang Gao, name: Zirui Song, name: Xiuying Chen
Published: 2025-03-03 16:31:45

Human readers can efficiently comprehend scrambled words, a phenomenon known
as Typoglycemia, primarily by relying on word form; if word form alone is
insufficient, they further utilize contextual cues for interpretation. While
advanced large language models (LLMs) exhibit similar abilities, the underlying
mechanisms remain unclear. To investigate this, we conduct controlled
experiments to analyze the roles of word form and contextual information in
semantic reconstruction and examine LLM attention patterns. Specifically, we
first propose SemRecScore, a reliable metric to quantify the degree of semantic
reconstruction, and validate its effectiveness. Using this metric, we study how
word form and contextual information influence LLMs' semantic reconstruction
ability, identifying word form as the core factor in this process. Furthermore,
we analyze how LLMs utilize word form and find that they rely on specialized
attention heads to extract and process word form information, with this
mechanism remaining stable across varying levels of word scrambling. This
distinction between LLMs' fixed attention patterns primarily focused on word
form and human readers' adaptive strategy in balancing word form and contextual
information provides insights into enhancing LLM performance by incorporating
human-like, context-aware mechanisms.
Direct Discriminative Optimization: Your Likelihood-Based Visual
  Generative Model is Secretly a GAN Discriminator
Authors: name: Kaiwen Zheng, name: Yongxin Chen, name: Huayu Chen, name: Guande He, name: Ming-Yu Liu, name: Jun Zhu, name: Qinsheng Zhang
Published: 2025-03-03 02:06:22

While likelihood-based generative models, particularly diffusion and
autoregressive models, have achieved remarkable fidelity in visual generation,
the maximum likelihood estimation (MLE) objective inherently suffers from a
mode-covering tendency that limits the generation quality under limited model
capacity. In this work, we propose Direct Discriminative Optimization (DDO) as
a unified framework that bridges likelihood-based generative training and the
GAN objective to bypass this fundamental constraint. Our key insight is to
parameterize a discriminator implicitly using the likelihood ratio between a
learnable target model and a fixed reference model, drawing parallels with the
philosophy of Direct Preference Optimization (DPO). Unlike GANs, this
parameterization eliminates the need for joint training of generator and
discriminator networks, allowing for direct, efficient, and effective
finetuning of a well-trained model to its full potential beyond the limits of
MLE. DDO can be performed iteratively in a self-play manner for progressive
model refinement, with each round requiring less than 1% of pretraining epochs.
Our experiments demonstrate the effectiveness of DDO by significantly advancing
the previous SOTA diffusion model EDM, reducing FID scores from 1.79/1.58 to
new records of 1.30/0.97 on CIFAR-10/ImageNet-64 datasets, and by consistently
improving both guidance-free and CFG-enhanced FIDs of visual autoregressive
models on ImageNet 256times256.
From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence
  Generation up to 100K Tokens
Authors: name: Tong Wu, name: Junzhe Shen, name: Zixia Jia, name: Yuxuan Wang, name: Zilong Zheng
Published: 2025-02-26 07:10:08

Generating ultra-long sequences with large language models (LLMs) has become
increasingly crucial but remains a highly time-intensive task, particularly for
sequences up to 100K tokens. While traditional speculative decoding methods
exist, simply extending their generation limits fails to accelerate the process
and can be detrimental. Through an in-depth analysis, we identify three major
challenges hindering efficient generation: frequent model reloading, dynamic
key-value (KV) management and repetitive generation. To address these issues,
we introduce TOKENSWIFT, a novel framework designed to substantially accelerate
the generation process of ultra-long sequences while maintaining the target
model's inherent quality. Experimental results demonstrate that TOKENSWIFT
achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B,
14B) and architectures (MHA, GQA). This acceleration translates to hours of
time savings for ultra-long sequence generation, establishing TOKENSWIFT as a
scalable and effective solution at unprecedented lengths. Code can be found at
https://github.com/bigai-nlco/TokenSwift.
DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End
  Full-Length Song Generation with Latent Diffusion
Authors: name: Ziqian Ning, name: Huakang Chen, name: Yuepeng Jiang, name: Chunbo Hao, name: Guobin Ma, name: Shuai Wang, name: Jixun Yao, name: Lei Xie
Published: 2025-03-03 05:15:34

Recent advancements in music generation have garnered significant attention,
yet existing approaches face critical limitations. Some current generative
models can only synthesize either the vocal track or the accompaniment track.
While some models can generate combined vocal and accompaniment, they typically
rely on meticulously designed multi-stage cascading architectures and intricate
data pipelines, hindering scalability. Additionally, most systems are
restricted to generating short musical segments rather than full-length songs.
Furthermore, widely used language model-based methods suffer from slow
inference speeds. To address these challenges, we propose DiffRhythm, the first
latent diffusion-based song generation model capable of synthesizing complete
songs with both vocal and accompaniment for durations of up to 4m45s in only
ten seconds, maintaining high musicality and intelligibility. Despite its
remarkable capabilities, DiffRhythm is designed to be simple and elegant: it
eliminates the need for complex data preparation, employs a straightforward
model structure, and requires only lyrics and a style prompt during inference.
Additionally, its non-autoregressive structure ensures fast inference speeds.
This simplicity guarantees the scalability of DiffRhythm. Moreover, we release
the complete training code along with the pre-trained model on large-scale data
to promote reproducibility and further research.
Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain
  Model
Authors: name: Yaxuan Huang, name: Xili Dai, name: Jianan Wang, name: Xianbiao Qi, name: Yixing Yuan, name: Xiangyu Yue
Published: 2025-02-24 02:14:19

Room layout estimation from multiple-perspective images is poorly
investigated due to the complexities that emerge from multi-view geometry,
which requires muti-step solutions such as camera intrinsic and extrinsic
estimation, image matching, and triangulation. However, in 3D reconstruction,
the advancement of recent 3D foundation models such as DUSt3R has shifted the
paradigm from the traditional multi-step structure-from-motion process to an
end-to-end single-step approach. To this end, we introduce Plane-DUSt3R, a
novel method for multi-view room layout estimation leveraging the 3D foundation
model DUSt3R. Plane-DUSt3R incorporates the DUSt3R framework and fine-tunes on
a room layout dataset (Structure3D) with a modified objective to estimate
structural planes. By generating uniform and parsimonious results, Plane-DUSt3R
enables room layout estimation with only a single post-processing step and 2D
detection results. Unlike previous methods that rely on single-perspective or
panorama image, Plane-DUSt3R extends the setting to handle multiple-perspective
images. Moreover, it offers a streamlined, end-to-end solution that simplifies
the process and reduces error accumulation. Experimental results demonstrate
that Plane-DUSt3R not only outperforms state-of-the-art methods on the
synthetic dataset but also proves robust and effective on in the wild data with
different image styles such as cartoon.Our code is available at:
https://github.com/justacar/Plane-DUSt3R
OneRec: Unifying Retrieve and Rank with Generative Recommender and
  Iterative Preference Alignment
Authors: name: Jiaxin Deng, name: Shiyao Wang, name: Kuo Cai, name: Lejian Ren, name: Qigen Hu, name: Weifeng Ding, name: Qiang Luo, name: Guorui Zhou
Published: 2025-02-26 09:25:10

Recently, generative retrieval-based recommendation systems have emerged as a
promising paradigm. However, most modern recommender systems adopt a
retrieve-and-rank strategy, where the generative model functions only as a
selector during the retrieval stage. In this paper, we propose OneRec, which
replaces the cascaded learning framework with a unified generative model. To
the best of our knowledge, this is the first end-to-end generative model that
significantly surpasses current complex and well-designed recommender systems
in real-world scenarios. Specifically, OneRec includes: 1) an encoder-decoder
structure, which encodes the user's historical behavior sequences and gradually
decodes the videos that the user may be interested in. We adopt sparse
Mixture-of-Experts (MoE) to scale model capacity without proportionally
increasing computational FLOPs. 2) a session-wise generation approach. In
contrast to traditional next-item prediction, we propose a session-wise
generation, which is more elegant and contextually coherent than point-by-point
generation that relies on hand-crafted rules to properly combine the generated
results. 3) an Iterative Preference Alignment module combined with Direct
Preference Optimization (DPO) to enhance the quality of the generated results.
Unlike DPO in NLP, a recommendation system typically has only one opportunity
to display results for each user's browsing request, making it impossible to
obtain positive and negative samples simultaneously. To address this
limitation, We design a reward model to simulate user generation and customize
the sampling strategy. Extensive experiments have demonstrated that a limited
number of DPO samples can align user interest preferences and significantly
improve the quality of generated results. We deployed OneRec in the main scene
of Kuaishou, achieving a 1.6\% increase in watch-time, which is a substantial
improvement.
AI-Invented Tonal Languages: Preventing a Machine Lingua Franca Beyond
  Human Understanding
Authors: name: David Noever
Published: 2025-03-02 23:59:52

This paper investigates the potential for large language models (LLMs) to
develop private tonal languages for machine-to-machine (M2M) communication.
Inspired by cryptophasia in human twins (affecting up to 50% of twin births)
and natural tonal languages like Mandarin and Vietnamese, we implement a
precise character-to-frequency mapping system that encodes the full ASCII
character set (32-126) using musical semitones. Each character is assigned a
unique frequency, creating a logarithmic progression beginning with space (220
Hz) and ending with tilde (50,175.42 Hz). This spans approximately 7.9 octaves,
with higher characters deliberately mapped to ultrasonic frequencies beyond
human perception (>20 kHz). Our implemented software prototype demonstrates
this encoding through visualization, auditory playback, and ABC musical
notation, allowing for analysis of information density and transmission speed.
Testing reveals that tonal encoding can achieve information rates exceeding
human speech while operating partially outside human perceptual boundaries.
This work responds directly to concerns about AI systems catastrophically
developing private languages within the next five years, providing a concrete
prototype software example of how such communication might function and the
technical foundation required for its emergence, detection, and governance.
Liger: Linearizing Large Language Models to Gated Recurrent Structures
Authors: name: Disen Lan, name: Weigao Sun, name: Jiaxi Hu, name: Jusen Du, name: Yu Cheng
Published: 2025-03-03 13:08:00

Transformers with linear recurrent modeling offer linear-time training and
constant-memory inference. Despite their demonstrated efficiency and
performance, pretraining such non-standard architectures from scratch remains
costly and risky. The linearization of large language models (LLMs) transforms
pretrained standard models into linear recurrent structures, enabling more
efficient deployment. However, current linearization methods typically
introduce additional feature map modules that require extensive fine-tuning and
overlook the gating mechanisms used in state-of-the-art linear recurrent
models. To address these issues, this paper presents Liger, short for
Linearizing LLMs to gated recurrent structures. Liger is a novel approach for
converting pretrained LLMs into gated linear recurrent models without adding
extra parameters. It repurposes the pretrained key matrix weights to construct
diverse gating mechanisms, facilitating the formation of various gated
recurrent structures while avoiding the need to train additional components
from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA),
Liger restores the performance of the linearized gated recurrent models to
match that of the original LLMs. Additionally, we introduce Liger Attention, an
intra-layer hybrid attention mechanism, which significantly recovers 93\% of
the Transformer-based LLM at 0.02\% pre-training tokens during the
linearization process, achieving competitive results across multiple
benchmarks, as validated on models ranging from 1B to 8B parameters. Code is
available at https://github.com/OpenSparseLLMs/Linearization.