Orionnnn/hub-stats huggingface dataset:
Solve-Detect-Verify: Inference-Time Scaling with Flexible Generative
  Verifier
Authors: name: Jianyuan Zhong, name: Zeju Li, name: Zhijian Xu, name: Xiangyu Wen, name: Kezhi Li, name: Qiang Xu
Published: 2025-05-17 11:41:44

Large Language Model (LLM) reasoning for complex tasks inherently involves a
trade-off between solution accuracy and computational efficiency. The
subsequent step of verification, while intended to improve performance, further
complicates this landscape by introducing its own challenging trade-off:
sophisticated Generative Reward Models (GenRMs) can be computationally
prohibitive if naively integrated with LLMs at test-time, while simpler, faster
methods may lack reliability. To overcome these challenges, we introduce
FlexiVe, a novel generative verifier that flexibly balances computational
resources between rapid, reliable fast thinking and meticulous slow thinking
using a Flexible Allocation of Verification Budget strategy. We further propose
the Solve-Detect-Verify pipeline, an efficient inference-time scaling framework
that intelligently integrates FlexiVe, proactively identifying solution
completion points to trigger targeted verification and provide focused solver
feedback. Experiments show FlexiVe achieves superior accuracy in pinpointing
errors within reasoning traces on ProcessBench. Furthermore, on challenging
mathematical reasoning benchmarks (AIME 2024, AIME 2025, and CNMO), our full
approach outperforms baselines like self-consistency in reasoning accuracy and
inference efficiency. Our system offers a scalable and effective solution to
enhance LLM reasoning at test time.
SageAttention3: Microscaling FP4 Attention for Inference and An
  Exploration of 8-Bit Training
Authors: name: Jintao Zhang, name: Jia Wei, name: Pengle Zhang, name: Xiaoming Xu, name: Haofeng Huang, name: Haoxu Wang, name: Kai Jiang, name: Jun Zhu, name: Jianfei Chen
Published: 2025-05-16 18:01:54

The efficiency of attention is important due to its quadratic time
complexity. We enhance the efficiency of attention through two key
contributions: First, we leverage the new FP4 Tensor Cores in Blackwell GPUs to
accelerate attention computation. Our implementation achieves 1038 TOPS on
RTX5090, which is a 5x speedup over the fastest FlashAttention on RTX5090.
Experiments show that our FP4 attention can accelerate inference of various
models in a plug-and-play way. Second, we pioneer low-bit attention to training
tasks. Existing low-bit attention works like FlashAttention3 and SageAttention
focus only on inference. However, the efficiency of training large models is
also important. To explore whether low-bit attention can be effectively applied
to training tasks, we design an accurate and efficient 8-bit attention for both
forward and backward propagation. Experiments indicate that 8-bit attention
achieves lossless performance in fine-tuning tasks but exhibits slower
convergence in pretraining tasks. The code will be available at
https://github.com/thu-ml/SageAttention.
Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic
  Reasoning Limits
Authors: name: Xiang Zhang, name: Juntai Cao, name: Jiaqi Wei, name: Yiwei Xu, name: Chenyu You
Published: 2025-05-20 10:32:30

Tokenization is the first - and often underappreciated - layer of computation
in language models. While Chain-of-Thought (CoT) prompting enables transformer
models to approximate recurrent computation by externalizing intermediate
steps, we show that the success of such reasoning is fundamentally bounded by
the structure of tokenized inputs. This work presents a theoretical and
empirical investigation into how tokenization schemes, particularly
subword-based methods like byte-pair encoding (BPE), impede symbolic
computation by merging or obscuring atomic reasoning units. We introduce the
notion of Token Awareness to formalize how poor token granularity disrupts
logical alignment and prevents models from generalizing symbolic procedures.
Through systematic evaluation on arithmetic and symbolic tasks, we demonstrate
that token structure dramatically affect reasoning performance, causing failure
even with CoT, while atomically-aligned formats unlock strong generalization,
allowing small models (e.g., GPT-4o-mini) to outperform larger systems (e.g.,
o1) in structured reasoning. Our findings reveal that symbolic reasoning
ability in LLMs is not purely architectural, but deeply conditioned on
token-level representations.
CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via
  Competition
Authors: name: Nam V. Nguyen, name: Huy Nguyen, name: Quang Pham, name: Van Nguyen, name: Savitha Ramasamy, name: Nhat Ho
Published: 2025-05-19 17:24:26

Sparse mixture of experts (SMoE) offers an appealing solution to scale up the
model complexity beyond the mean of increasing the network's depth or width.
However, we argue that effective SMoE training remains challenging because of
the suboptimal routing process where experts that perform computation do not
directly contribute to the routing process. In this work, we propose
competition, a novel mechanism to route tokens to experts with the highest
neural response. Theoretically, we show that the competition mechanism enjoys a
better sample efficiency than the traditional softmax routing. Furthermore, we
develop CompeteSMoE, a simple yet effective algorithm to train large language
models by deploying a router to learn the competition policy, thus enjoying
strong performances at a low training overhead. Our extensive empirical
evaluations on both the visual instruction tuning and language pre-training
tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE
compared to state-of-the-art SMoE strategies. We have made the implementation
available at: https://github.com/Fsoft-AIC/CompeteSMoE. This work is an
improved version of the previous study at arXiv:2402.02526
Emerging Properties in Unified Multimodal Pretraining
Authors: name: Chaorui Deng, name: Deyao Zhu, name: Kunchang Li, name: Chenhui Gou, name: Feng Li, name: Zeyu Wang, name: Shu Zhong, name: Weihao Yu, name: Xiaonan Nie, name: Ziang Song, name: Guang Shi, name: Haoqi Fan
Published: 2025-05-20 17:59:30

Unifying multimodal understanding and generation has shown impressive
capabilities in cutting-edge proprietary systems. In this work, we introduce
BAGEL, an open0source foundational model that natively supports multimodal
understanding and generation. BAGEL is a unified, decoder0only model pretrained
on trillions of tokens curated from large0scale interleaved text, image, video,
and web data. When scaled with such diverse multimodal interleaved data, BAGEL
exhibits emerging capabilities in complex multimodal reasoning. As a result, it
significantly outperforms open-source unified models in both multimodal
generation and understanding across standard benchmarks, while exhibiting
advanced multimodal reasoning abilities such as free-form image manipulation,
future frame prediction, 3D manipulation, and world navigation. In the hope of
facilitating further opportunities for multimodal research, we share the key
findings, pretraining details, data creation protocal, and release our code and
checkpoints to the community. The project page is at https://bagel-ai.org/
Latent Flow Transformer
Authors: name: Yen-Chen Wu, name: Feng-Ting Liao, name: Meng-Hsi Chen, name: Pei-Chen Ho, name: Farhang Nabiei, name: Da-shan Shiu
Published: 2025-05-20 15:41:05

Transformers, the standard implementation for large language models (LLMs),
typically consist of tens to hundreds of discrete layers. While more layers can
lead to better performance, this approach has been challenged as far from
efficient, especially given the superiority of continuous layers demonstrated
by diffusion and flow-based models for image generation. We propose the Latent
Flow Transformer (LFT), which replaces a block of layers with a single learned
transport operator trained via flow matching, offering significant compression
while maintaining compatibility with the original architecture. Additionally,
we address the limitations of existing flow-based methods in preserving
coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M
model, LFT trained with flow matching compresses 6 of 24 layers and outperforms
directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529),
demonstrating the feasibility of this design. When trained with FW, LFT further
distills 12 layers into one while reducing the KL to 0.736 surpassing that from
skipping 3 layers (0.932), significantly narrowing the gap between
autoregressive and flow-based generation paradigms.
Truth Neurons
Authors: name: Haohang Li, name: Yupeng Cao, name: Yangyang Yu, name: Jordan W. Suchow, name: Zining Zhu
Published: 2025-05-18 00:47:21

Despite their remarkable success and deployment across diverse workflows,
language models sometimes produce untruthful responses. Our limited
understanding of how truthfulness is mechanistically encoded within these
models jeopardizes their reliability and safety. In this paper, we propose a
method for identifying representations of truthfulness at the neuron level. We
show that language models contain truth neurons, which encode truthfulness in a
subject-agnostic manner. Experiments conducted across models of varying scales
validate the existence of truth neurons, confirming that the encoding of
truthfulness at the neuron level is a property shared by many language models.
The distribution patterns of truth neurons over layers align with prior
findings on the geometry of truthfulness. Selectively suppressing the
activations of truth neurons found through the TruthfulQA dataset degrades
performance both on TruthfulQA and on other benchmarks, showing that the
truthfulness mechanisms are not tied to a specific dataset. Our results offer
novel insights into the mechanisms underlying truthfulness in language models
and highlight potential directions toward improving their trustworthiness and
reliability.
NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search
Authors: name: Sunhao Dai, name: Wenjie Wang, name: Liang Pang, name: Jun Xu, name: See-Kiong Ng, name: Ji-Rong Wen, name: Tat-Seng Chua
Published: 2025-05-20 17:59:13

Generative AI search is reshaping information retrieval by offering
end-to-end answers to complex queries, reducing users' reliance on manually
browsing and summarizing multiple web pages. However, while this paradigm
enhances convenience, it disrupts the feedback-driven improvement loop that has
historically powered the evolution of traditional Web search. Web search can
continuously improve their ranking models by collecting large-scale,
fine-grained user feedback (e.g., clicks, dwell time) at the document level. In
contrast, generative AI search operates through a much longer search pipeline,
spanning query decomposition, document retrieval, and answer generation, yet
typically receives only coarse-grained feedback on the final answer. This
introduces a feedback loop disconnect, where user feedback for the final output
cannot be effectively mapped back to specific system components, making it
difficult to improve each intermediate stage and sustain the feedback loop. In
this paper, we envision NExT-Search, a next-generation paradigm designed to
reintroduce fine-grained, process-level feedback into generative AI search.
NExT-Search integrates two complementary modes: User Debug Mode, which allows
engaged users to intervene at key stages; and Shadow User Mode, where a
personalized user agent simulates user preferences and provides AI-assisted
feedback for less interactive users. Furthermore, we envision how these
feedback signals can be leveraged through online adaptation, which refines
current search outputs in real-time, and offline update, which aggregates
interaction logs to periodically fine-tune query decomposition, retrieval, and
generation models. By restoring human control over key stages of the generative
AI search pipeline, we believe NExT-Search offers a promising direction for
building feedback-rich AI search systems that can evolve continuously alongside
human feedback.
Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for
  Real-world Knowledge Injection
Authors: name: Yuwei Zhang, name: Wenhao Yu, name: Shangbin Feng, name: Yifan Zhu, name: Letian Peng, name: Jayanth Srinivasa, name: Gaowen Liu, name: Jingbo Shang
Published: 2025-05-18 08:39:05

Despite significant advances in large language models (LLMs), their knowledge
memorization capabilities remain underexplored, due to the lack of standardized
and high-quality test ground. In this paper, we introduce a novel, real-world
and large-scale knowledge injection benchmark that evolves continuously over
time without requiring human intervention. Specifically, we propose WikiDYK,
which leverages recently-added and human-written facts from Wikipedia's "Did
You Know..." entries. These entries are carefully selected by expert Wikipedia
editors based on criteria such as verifiability and clarity. Each entry is
converted into multiple question-answer pairs spanning diverse task formats
from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290
facts and 77,180 questions, which is also seamlessly extensible with future
updates from Wikipedia editors. Extensive experiments using continued
pre-training reveal a surprising insight: despite their prevalence in modern
LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge
memorization capabilities compared to Bidirectional Language Models (BiLMs),
exhibiting a 23% lower accuracy in terms of reliability. To compensate for the
smaller scales of current BiLMs, we introduce a modular collaborative framework
utilizing ensembles of BiLMs as external knowledge repositories to integrate
with LLMs. Experiment shows that our framework further improves the reliability
accuracy by up to 29.1%.
Understanding Gen Alpha Digital Language: Evaluation of LLM Safety
  Systems for Content Moderation
Authors: name: Manisha Mehta, name: Fausto Giunchiglia
Published: 2025-05-14 16:46:11

This research offers a unique evaluation of how AI systems interpret the
digital language of Generation Alpha (Gen Alpha, born 2010-2024). As the first
cohort raised alongside AI, Gen Alpha faces new forms of online risk due to
immersive digital engagement and a growing mismatch between their evolving
communication and existing safety tools. Their distinct language, shaped by
gaming, memes, and AI-driven trends, often conceals harmful interactions from
both human moderators and automated systems. We assess four leading AI models
(GPT-4, Claude, Gemini, and Llama 3) on their ability to detect masked
harassment and manipulation within Gen Alpha discourse. Using a dataset of 100
recent expressions from gaming platforms, social media, and video content, the
study reveals critical comprehension failures with direct implications for
online safety. This work contributes: (1) a first-of-its-kind dataset capturing
Gen Alpha expressions; (2) a framework to improve AI moderation systems for
youth protection; (3) a multi-perspective evaluation including AI systems,
human moderators, and parents, with direct input from Gen Alpha co-researchers;
and (4) an analysis of how linguistic divergence increases youth vulnerability.
Findings highlight the urgent need to redesign safety systems attuned to youth
communication, especially given Gen Alpha reluctance to seek help when adults
fail to understand their digital world. This study combines the insight of a
Gen Alpha researcher with systematic academic analysis to address critical
digital safety challenges.
Reasoning Path Compression: Compressing Generation Trajectories for
  Efficient LLM Reasoning
Authors: name: Jiwon Song, name: Dongwon Jo, name: Yulhwa Kim, name: Jae-Joon Kim
Published: 2025-05-20 03:21:52

Recent reasoning-focused language models achieve high accuracy by generating
lengthy intermediate reasoning paths before producing final answers. While this
approach is effective in solving problems that require logical thinking, long
reasoning paths significantly increase memory usage and throughput of token
generation, limiting the practical deployment of such models. We propose
Reasoning Path Compression (RPC), a training-free method that accelerates
inference by leveraging the semantic sparsity of reasoning paths. RPC
periodically compresses the KV cache by retaining KV cache that receive high
importance score, which are computed using a selector window composed of
recently generated queries. Experiments show that RPC improves generation
throughput of QwQ-32B by up to 1.60times compared to the inference with full
KV cache, with an accuracy drop of 1.2% on the AIME 2024 benchmark. Our
findings demonstrate that semantic sparsity in reasoning traces can be
effectively exploited for compression, offering a practical path toward
efficient deployment of reasoning LLMs. Our code is available at
https://github.com/jiwonsong-dev/ReasoningPathCompression.
Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task
  Success at Scale
Authors: name: David Noever, name: Forrest McKee
Published: 2025-05-16 22:42:04

This study explores Large Language Models (LLMs) as autonomous agents for
real-world tasks, including freelance software development. This work presents
a new benchmark that evaluates LLMs on freelance programming and data analysis
tasks derived from economic data. We construct the benchmark using synthetic
tasks created from a Kaggle Freelancer dataset of job postings, with all job
prices standardized to USD (median fixed-project price around 250, and an
average of 306). Each task is accompanied by structured input-output test
cases and an estimated price tag, enabling automated correctness checking and a
monetary performance valuation. This approach is inspired by OpenAI's recent
SWE-Lancer benchmark (1,400 real Upwork tasks worth 1M total). Still, our
framework simplifies evaluation using programmatically testable tasks and
predicted price values, making it highly scalable and repeatable. On this
benchmark, we evaluate four modern LLMs - Claude 3.5 Haiku, GPT-4o-mini, Qwen
2.5, and Mistral. We report each model's accuracy (task success rate and
test-case pass rate) and the total "freelance earnings" it achieves (sum of
prices of solved tasks). Our results show that Claude 3.5 Haiku performs best,
earning approximately 1.52 million USD, followed closely by GPT-4o-mini at
1.49 million, then Qwen 2.5 (1.33M) and Mistral ($0.70M). We analyze the
distribution of errors per task and observe that the strongest models solve the
most tasks and rarely fail completely on any project. We discuss the
implications of these results for the feasibility of AI as a freelance
developer, the advantages and limitations of our automated benchmark approach,
and the gap between performance on structured tasks versus the true complexity
of real-world freelance jobs.
EfficientLLM: Efficiency in Large Language Models
Authors: name: Zhengqing Yuan, name: Weixiang Sun, name: Yixin Liu, name: Huichi Zhou, name: Rong Zhou, name: Yiyang Li, name: Zheyuan Zhang, name: Wei Song, name: Yue Huang, name: Haolong Jia, name: Keerthiram Murugesan, name: Yu Wang, name: Lifang He, name: Jianfeng Gao, name: Lichao Sun, name: Yanfang Ye
Published: 2025-05-20 02:27:08

Large Language Models (LLMs) have driven significant progress, yet their
growing parameter counts and context windows incur prohibitive compute, energy,
and monetary costs. We introduce EfficientLLM, a novel benchmark and the first
comprehensive empirical study evaluating efficiency techniques for LLMs at
scale. Conducted on a production-class cluster (48xGH200, 8xH200 GPUs), our
study systematically explores three key axes: (1) architecture pretraining
(efficient attention variants: MQA, GQA, MLA, NSA; sparse Mixture-of-Experts
(MoE)), (2) fine-tuning (parameter-efficient methods: LoRA, RSLoRA, DoRA), and
(3) inference (quantization methods: int4, float16). We define six fine-grained
metrics (Memory Utilization, Compute Utilization, Latency, Throughput, Energy
Consumption, Compression Rate) to capture hardware saturation,
latency-throughput balance, and carbon cost. Evaluating over 100
model-technique pairs (0.5B-72B parameters), we derive three core insights: (i)
Efficiency involves quantifiable trade-offs: no single method is universally
optimal; e.g., MoE reduces FLOPs and improves accuracy but increases VRAM by
40%, while int4 quantization cuts memory/energy by up to 3.9x at a 3-5%
accuracy drop. (ii) Optima are task- and scale-dependent: MQA offers optimal
memory-latency trade-offs for constrained devices, MLA achieves lowest
perplexity for quality-critical tasks, and RSLoRA surpasses LoRA efficiency
only beyond 14B parameters. (iii) Techniques generalize across modalities: we
extend evaluations to Large Vision Models (Stable Diffusion 3.5, Wan 2.1) and
Vision-Language Models (Qwen2.5-VL), confirming effective transferability. By
open-sourcing datasets, evaluation pipelines, and leaderboards, EfficientLLM
provides essential guidance for researchers and engineers navigating the
efficiency-performance landscape of next-generation foundation models.
Efficient Speech Language Modeling via Energy Distance in Continuous
  Latent Space
Authors: name: Zhengrui Ma, name: Yang Feng, name: Chenze Shao, name: Fandong Meng, name: Jie Zhou, name: Min Zhang
Published: 2025-05-19 14:38:59

We introduce SLED, an alternative approach to speech language modeling by
encoding speech waveforms into sequences of continuous latent representations
and modeling them autoregressively using an energy distance objective. The
energy distance offers an analytical measure of the distributional gap by
contrasting simulated and target samples, enabling efficient training to
capture the underlying continuous autoregressive distribution. By bypassing
reliance on residual vector quantization, SLED avoids discretization errors and
eliminates the need for the complicated hierarchical architectures common in
existing speech language models. It simplifies the overall modeling pipeline
while preserving the richness of speech information and maintaining inference
efficiency. Empirical results demonstrate that SLED achieves strong performance
in both zero-shot and streaming speech synthesis, showing its potential for
broader applications in general-purpose speech language models.
MedCaseReasoning: Evaluating and learning diagnostic reasoning from
  clinical case reports
Authors: name: Kevin Wu, name: Eric Wu, name: Rahul Thapa, name: Kevin Wei, name: Angela Zhang, name: Arvind Suresh, name: Jacqueline J. Tao, name: Min Woo Sun, name: Alejandro Lozano, name: James Zou
Published: 2025-05-16 22:34:36

Doctors and patients alike increasingly use Large Language Models (LLMs) to
diagnose clinical cases. However, unlike domains such as math or coding, where
correctness can be objectively defined by the final answer, medical diagnosis
requires both the outcome and the reasoning process to be accurate. Currently,
widely used medical benchmarks like MedQA and MMLU assess only accuracy in the
final answer, overlooking the quality and faithfulness of the clinical
reasoning process. To address this limitation, we introduce MedCaseReasoning,
the first open-access dataset for evaluating LLMs on their ability to align
with clinician-authored diagnostic reasoning. The dataset includes 14,489
diagnostic question-and-answer cases, each paired with detailed reasoning
statements derived from open-access medical case reports. We evaluate
state-of-the-art reasoning LLMs on MedCaseReasoning and find significant
shortcomings in their diagnoses and reasoning: for instance, the top-performing
open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy
and mentions only 64% of the clinician reasoning statements (recall). However,
we demonstrate that fine-tuning LLMs on the reasoning traces derived from
MedCaseReasoning significantly improves diagnostic accuracy and clinical
reasoning recall by an average relative gain of 29% and 41%, respectively. The
open-source dataset, code, and models are available at
https://github.com/kevinwu23/Stanford-MedCaseReasoning.