Orionnnn/hub-stats huggingface dataset:
Scientists' First Exam: Probing Cognitive Abilities of MLLM via
  Perception, Understanding, and Reasoning
Authors: name: Yuhao Zhou, name: Yiheng Wang, name: Xuming He, name: Ruoyao Xiao, name: Zhiwei Li, name: Qiantai Feng, name: Zijie Guo, name: Yuejin Yang, name: Hao Wu, name: Wenxuan Huang, name: Jiaqi Wei, name: Dan Si, name: Xiuqi Yao, name: Jia Bu, name: Haiwen Huang, name: Tianfan Fu, name: Shixiang Tang, name: Ben Fei, name: Dongzhan Zhou, name: Fenghua Ling, name: Yan Lu, name: Siqi Sun, name: Chenhui Li, name: Guanjie Zheng, name: Jiancheng Lv, name: Wenlong Zhang, name: Lei Bai
Published: 2025-06-12 09:29:16

Scientific discoveries increasingly rely on complex multimodal reasoning
based on information-intensive scientific data and domain-specific expertise.
Empowered by expert-level scientific benchmarks, scientific Multimodal Large
Language Models (MLLMs) hold the potential to significantly enhance this
discovery process in realistic workflows. However, current scientific
benchmarks mostly focus on evaluating the knowledge understanding capabilities
of MLLMs, leading to an inadequate assessment of their perception and reasoning
abilities. To address this gap, we present the Scientists' First Exam (SFE)
benchmark, designed to evaluate the scientific cognitive capacities of MLLMs
through three interconnected levels: scientific signal perception, scientific
attribute understanding, scientific comparative reasoning. Specifically, SFE
comprises 830 expert-verified VQA pairs across three question types, spanning
66 multimodal tasks across five high-value disciplines. Extensive experiments
reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08%
and 26.52% on SFE, highlighting significant room for MLLMs to improve in
scientific realms. We hope the insights obtained in SFE will facilitate further
developments in AI-enhanced scientific discoveries.
Steering LLM Thinking with Budget Guidance
Authors: name: Junyan Li, name: Wenshuo Zhao, name: Yang Zhang, name: Chuang Gan
Published: 2025-06-16 17:57:05

Recent deep-thinking large language models often reason extensively to
improve performance, but such lengthy reasoning is not always desirable, as it
incurs excessive inference costs with disproportionate performance gains.
Controlling reasoning length without sacrificing performance is therefore
important, but remains challenging, especially under tight thinking budgets. We
propose budget guidance, a simple yet effective method for steering the
reasoning process of LLMs toward a target budget without requiring any LLM
fine-tuning. Our approach introduces a lightweight predictor that models a
Gamma distribution over the remaining thinking length during next-token
generation. This signal is then used to guide generation in a soft, token-level
manner, ensuring that the overall reasoning trace adheres to the specified
thinking budget. Budget guidance enables natural control of the thinking
length, along with significant token efficiency improvements over baseline
methods on challenging math benchmarks. For instance, it achieves up to a 26%
accuracy gain on the MATH-500 benchmark under tight budgets compared to
baseline methods, while maintaining competitive accuracy with only 63% of the
thinking tokens used by the full-thinking model. Budget guidance also
generalizes to broader task domains and exhibits emergent capabilities, such as
estimating question difficulty. The source code is available at:
https://github.com/UMass-Embodied-AGI/BudgetGuidance.
Language Surgery in Multilingual Large Language Models
Authors: name: Joanito Agili Lopo, name: Muhammad Ravi Shulthan Habibi, name: Tack Hwa Wong, name: Muhammad Ilham Ghozali, name: Fajri Koto, name: Genta Indra Winata, name: Peerat Limkonchotiwat, name: Alham Fikri Aji, name: Samuel Cahyawijaya
Published: 2025-06-14 11:09:50

Large Language Models (LLMs) have demonstrated remarkable generalization
capabilities across tasks and languages, revolutionizing natural language
processing. This paper investigates the naturally emerging representation
alignment in LLMs, particularly in the middle layers, and its implications for
disentangling language-specific and language-agnostic information. We
empirically confirm the existence of this alignment, analyze its behavior in
comparison to explicitly designed alignment models, and demonstrate its
potential for language-specific manipulation without semantic degradation.
Building on these findings, we propose Inference-Time Language Control (ITLC),
a novel method that leverages latent injection to enable precise cross-lingual
language control and mitigate language confusion in LLMs. Our experiments
highlight ITLC's strong cross-lingual control capabilities while preserving
semantic integrity in target languages. Furthermore, we demonstrate its
effectiveness in alleviating the cross-lingual language confusion problem,
which persists even in current large-scale LLMs, leading to inconsistent
language generation. This work advances our understanding of representation
alignment in LLMs and introduces a practical solution for enhancing their
cross-lingual performance.
PersonaFeedback: A Large-scale Human-annotated Benchmark For
  Personalization
Authors: name: Meiling Tao, name: Chenghao Zhu, name: Dongyi Ding, name: Tiannan Wang, name: Yuchen Eleanor Jiang, name: Wangchunshu Zhou
Published: 2025-06-15 17:19:19

With the rapid improvement in the general capabilities of LLMs, LLM
personalization, i.e., how to build LLM systems that can generate personalized
responses or services that are tailored to distinct user personas, has become
an increasingly important research and engineering problem. However, unlike
many new challenging benchmarks being released for evaluating the
general/reasoning capabilities, the lack of high-quality benchmarks for
evaluating LLM personalization greatly hinders progress in this field. To
address this, we introduce PersonaFeedback, a new benchmark that directly
evaluates LLMs' ability to provide personalized responses given pre-defined
user personas and queries. Unlike existing benchmarks that require models to
infer implicit user personas from historical interactions, PersonaFeedback
decouples persona inference from personalization, focusing on evaluating the
model's ability to generate responses tailored to explicit personas.
PersonaFeedback consists of 8298 human-annotated test cases, which are
categorized into easy, medium, and hard tiers based on the contextual
complexity of the user personas and the difficulty in distinguishing subtle
differences between two personalized responses. We conduct comprehensive
evaluations across a wide range of models. The empirical results reveal that
even state-of-the-art LLMs that can solve complex real-world reasoning tasks
could fall short on the hard tier of PersonaFeedback where even human
evaluators may find the distinctions challenging. Furthermore, we conduct an
in-depth analysis of failure modes across various types of systems,
demonstrating that the current retrieval-augmented framework should not be seen
as a de facto solution for personalization tasks. All benchmark data,
annotation protocols, and the evaluation pipeline will be publicly available to
facilitate future research on LLM personalization.
MS4UI: A Dataset for Multi-modal Summarization of User Interface
  Instructional Videos
Authors: name: Yuan Zang, name: Hao Tan, name: Seunghyun Yoon, name: Franck Dernoncourt, name: Jiuxiang Gu, name: Kushal Kafle, name: Chen Sun, name: Trung Bui
Published: 2025-06-14 20:39:32

We study multi-modal summarization for instructional videos, whose goal is to
provide users an efficient way to learn skills in the form of text instructions
and key video frames. We observe that existing benchmarks focus on generic
semantic-level video summarization, and are not suitable for providing
step-by-step executable instructions and illustrations, both of which are
crucial for instructional videos. We propose a novel benchmark for user
interface (UI) instructional video summarization to fill the gap. We collect a
dataset of 2,413 UI instructional videos, which spans over 167 hours. These
videos are manually annotated for video segmentation, text summarization, and
video summarization, which enable the comprehensive evaluations for concise and
executable video summarization. We conduct extensive experiments on our
collected MS4UI dataset, which suggest that state-of-the-art multi-modal
summarization methods struggle on UI video summarization, and highlight the
importance of new methods for UI instructional video summarization.
Forecasting Time Series with LLMs via Patch-Based Prompting and
  Decomposition
Authors: name: Mayank Bumb, name: Anshul Vemulapalli, name: Sri Harsha Vardhan Prasad Jella, name: Anish Gupta, name: An La, name: Ryan A. Rossi, name: Hongjie Chen, name: Franck Dernoncourt, name: Nesreen K. Ahmed, name: Yu Wang
Published: 2025-06-15 19:42:58

Recent advances in Large Language Models (LLMs) have demonstrated new
possibilities for accurate and efficient time series analysis, but prior work
often required heavy fine-tuning and/or ignored inter-series correlations. In
this work, we explore simple and flexible prompt-based strategies that enable
LLMs to perform time series forecasting without extensive retraining or the use
of a complex external architecture. Through the exploration of specialized
prompting methods that leverage time series decomposition, patch-based
tokenization, and similarity-based neighbor augmentation, we find that it is
possible to enhance LLM forecasting quality while maintaining simplicity and
requiring minimal preprocessing of data. To this end, we propose our own
method, PatchInstruct, which enables LLMs to make precise and effective
predictions.
Test3R: Learning to Reconstruct 3D at Test Time
Authors: name: Yuheng Yuan, name: Qiuhong Shen, name: Shizun Wang, name: Xingyi Yang, name: Xinchao Wang
Published: 2025-06-16 17:56:22

Dense matching methods like DUSt3R regress pairwise pointmaps for 3D
reconstruction. However, the reliance on pairwise prediction and the limited
generalization capability inherently restrict the global geometric consistency.
In this work, we introduce Test3R, a surprisingly simple test-time learning
technique that significantly boosts geometric accuracy. Using image triplets
(I_1,I_2,I_3), Test3R generates reconstructions from pairs (I_1,I_2) and
(I_1,I_3). The core idea is to optimize the network at test time via a
self-supervised objective: maximizing the geometric consistency between these
two reconstructions relative to the common image I_1. This ensures the model
produces cross-pair consistent outputs, regardless of the inputs. Extensive
experiments demonstrate that our technique significantly outperforms previous
state-of-the-art methods on the 3D reconstruction and multi-view depth
estimation tasks. Moreover, it is universally applicable and nearly cost-free,
making it easily applied to other models and implemented with minimal test-time
training overhead and parameter footprint. Code is available at
https://github.com/nopQAQ/Test3R.
Discrete Diffusion in Large Language and Multimodal Models: A Survey
Authors: name: Runpeng Yu, name: Qi Li, name: Xinchao Wang
Published: 2025-06-16 17:59:08

In this work, we provide a systematic survey of Discrete Diffusion Language
Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs).
Unlike autoregressive (AR) models, dLLMs and dMLLMs adopt a multi-token,
parallel decoding paradigm using full attention and a denoising-based
generation strategy. This paradigm naturally enables parallel generation,
fine-grained output controllability, and dynamic, response-aware perception.
These capabilities are previously difficult to achieve with AR models.
Recently, a growing number of industrial-scale proprietary d(M)LLMs, as well as
a large number of open-source academic d(M)LLMs, have demonstrated performance
comparable to their autoregressive counterparts, while achieving up to 10x
acceleration in inference speed.
  The advancement of discrete diffusion LLMs and MLLMs has been largely driven
by progress in two domains. The first is the development of autoregressive LLMs
and MLLMs, which has accumulated vast amounts of data, benchmarks, and
foundational infrastructure for training and inference. The second contributing
domain is the evolution of the mathematical models underlying discrete
diffusion. Together, these advancements have catalyzed a surge in dLLMs and
dMLLMs research in early 2025.
  In this work, we present a comprehensive overview of the research in the dLLM
and dMLLM domains. We trace the historical development of dLLMs and dMLLMs,
formalize the underlying mathematical frameworks, and categorize representative
models. We further analyze key techniques for training and inference, and
summarize emerging applications across language, vision-language, and
biological domains. We conclude by discussing future directions for research
and deployment.
  Paper collection: https://github.com/LiQiiiii/DLLM-Survey
BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning
  with Vision-Language Models
Authors: name: Peiyan Li, name: Yixiang Chen, name: Hongtao Wu, name: Xiao Ma, name: Xiangnan Wu, name: Yan Huang, name: Liang Wang, name: Tao Kong, name: Tieniu Tan
Published: 2025-06-09 17:36:34

Recently, leveraging pre-trained vision-language models (VLMs) for building
vision-language-action (VLA) models has emerged as a promising approach to
effective robot manipulation learning. However, only few methods incorporate 3D
signals into VLMs for action prediction, and they do not fully leverage the
spatial structure inherent in 3D data, leading to low sample efficiency. In
this paper, we introduce BridgeVLA, a novel 3D VLA model that (1) projects 3D
inputs to multiple 2D images, ensuring input alignment with the VLM backbone,
and (2) utilizes 2D heatmaps for action prediction, unifying the input and
output spaces within a consistent 2D image space. In addition, we propose a
scalable pre-training method that equips the VLM backbone with the capability
to predict 2D heatmaps before downstream policy learning. Extensive experiments
show the proposed method is able to learn 3D manipulation efficiently and
effectively. BridgeVLA outperforms state-of-the-art baseline methods across
three simulation benchmarks. In RLBench, it improves the average success rate
from 81.4% to 88.2%. In COLOSSEUM, it demonstrates significantly better
performance in challenging generalization settings, boosting the average
success rate from 56.7% to 64.0%. In GemBench, it surpasses all the comparing
baseline methods in terms of average success rate. In real-robot experiments,
BridgeVLA outperforms a state-of-the-art baseline method by 32% on average. It
generalizes robustly in multiple out-of-distribution settings, including visual
disturbances and unseen instructions. Remarkably, it is able to achieve a
success rate of 96.8% on 10+ tasks with only 3 trajectories per task,
highlighting its extraordinary sample efficiency. Project
Website:https://bridgevla.github.io/
Supernova Event Dataset: Interpreting Large Language Model's Personality
  through Critical Event Analysis
Authors: name: Pranav Agarwal, name: Ioana Ciucă
Published: 2025-06-13 19:31:52

Large Language Models (LLMs) are increasingly integrated into everyday
applications. As their influence grows, understanding their decision making and
underlying personality becomes essential. In this work, we interpret model
personality using our proposed Supernova Event Dataset, a novel dataset with
diverse articles spanning biographies, historical events, news, and scientific
discoveries. We use this dataset to benchmark LLMs on extracting and ranking
key events from text, a subjective and complex challenge that requires
reasoning over long-range context and modeling causal chains. We evaluate small
models like Phi-4, Orca 2, and Qwen 2.5, and large, stronger models such as
Claude 3.7, Gemini 2.5, and OpenAI o3, and propose a framework where another
LLM acts as a judge to infer each model's personality based on its selection
and classification of events. Our analysis shows distinct personality traits:
for instance, Orca 2 demonstrates emotional reasoning focusing on interpersonal
dynamics, while Qwen 2.5 displays a more strategic, analytical style. When
analyzing scientific discovery events, Claude Sonnet 3.7 emphasizes conceptual
framing, Gemini 2.5 Pro prioritizes empirical validation, and o3 favors
step-by-step causal reasoning. This analysis improves model interpretability,
making them user-friendly for a wide range of diverse applications.
Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves
  Reasoning Efficiency
Authors: name: Chenlong Wang, name: Yuanning Feng, name: Dongping Chen, name: Zhaoyang Chu, name: Ranjay Krishna, name: Tianyi Zhou
Published: 2025-06-10 01:54:04

Recent advances in large reasoning models have enabled complex, step-by-step
reasoning but often introduce significant overthinking, resulting in verbose
and redundant outputs that hinder efficiency. In this study, we examine whether
explicit self-reflection, signaled by tokens such as "Wait" and "Hmm", is
necessary for advanced reasoning. We propose NoWait, a simple yet effective
approach that disables explicit self-reflection by suppressing these tokens
during inference. Extensive experiments on ten benchmarks across textual,
visual, and video reasoning tasks show that NoWait reduces chain-of-thought
trajectory length by up to 27%-51% in five R1-style model series, without
compromising model utility. NoWait thus offers a plug-and-play solution for
efficient and utility-preserving multimodal reasoning.
From Real to Synthetic: Synthesizing Millions of Diversified and
  Complicated User Instructions with Attributed Grounding
Authors: name: Chiwei Zhu, name: Benfeng Xu, name: Xiaorui Wang, name: Zhendong Mao
Published: 2025-06-04 14:00:47

The pursuit of diverse, complex, and large-scale instruction data is crucial
for automatically aligning large language models (LLMs). While there are
methods capable of generating synthetic instructions at scale, they either
suffer from limited grounding sources, leading to a narrow distribution, or
rely on trivial extensions that fail to produce meaningful trajectories in
terms of complexity. In contrast, instructions that benefit efficient alignment
are typically crafted with cognitive insights and grounded in real-world use
cases. In this paper, we synthesize such instructions using attributed
grounding, which involves 1) a top-down attribution process that grounds a
selective set of real instructions to situated users, and 2) a bottom-up
synthesis process that leverages web documents to first generate a situation,
then a meaningful instruction. This framework allows us to harvest diverse and
complex instructions at scale, utilizing the vast range of web documents.
Specifically, we construct a dataset of 1 million instructions, called
SynthQuestions, and demonstrate that models trained on it achieve leading
performance on several common benchmarks, with improvements that continually
scale with more web corpora. Data, models and codes will be available at
https://github.com/Ignoramus0817/SynthQuestions.
DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
Authors: name: Mingxuan Du, name: Benfeng Xu, name: Chiwei Zhu, name: Xiaorui Wang, name: Zhendong Mao
Published: 2025-06-13 13:17:32

Deep Research Agents are a prominent category of LLM-based agents. By
autonomously orchestrating multistep web exploration, targeted retrieval, and
higher-order synthesis, they transform vast amounts of online information into
analyst-grade, citation-rich reports--compressing hours of manual desk research
into minutes. However, a comprehensive benchmark for systematically evaluating
the capabilities of these agents remains absent. To bridge this gap, we present
DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks,
each meticulously crafted by domain experts across 22 distinct fields.
Evaluating DRAs is inherently complex and labor-intensive. We therefore propose
two novel methodologies that achieve strong alignment with human judgment. The
first is a reference-based method with adaptive criteria to assess the quality
of generated research reports. The other framework is introduced to evaluate
DRA's information retrieval and collection capabilities by assessing its
effective citation count and overall citation accuracy. We have open-sourced
DeepResearch Bench and key components of these frameworks at
https://github.com/Ayanami0730/deep_research_bench to accelerate the
development of practical LLM-based agents.
Don't Pay Attention
Authors: name: Mohammad Hammoud, name: Devang Acharya
Published: 2025-06-12 21:11:06

The Transformer has become the de facto standard for large language models
and a wide range of downstream tasks across various domains. Despite its
numerous advantages like inherent training parallelism, the Transformer still
faces key challenges due to its inability to effectively process sequences
beyond a fixed context window and the quadratic complexity of its attention
mechanism. These challenges have renewed interest in RNN-like architectures,
which offer linear scaling with sequence length and improved handling of
long-range dependencies, albeit with limited parallelism due to their
inherently recurrent nature. In this paper, we propose Avey, a new neural
foundational architecture that breaks away from both attention and recurrence.
Avey comprises a ranker and an autoregressive neural processor, which
collaboratively identify and contextualize only the most relevant tokens for
any given token, regardless of their positions in the sequence. Specifically,
Avey decouples sequence length from context width, thus enabling effective
processing of arbitrarily long sequences. Experimental results show that Avey
compares favorably to the Transformer across a variety of standard short-range
NLP benchmarks, while notably excelling at capturing long-range dependencies.
AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
Authors: name: Polina Kirichenko, name: Mark Ibrahim, name: Kamalika Chaudhuri, name: Samuel J. Bell
Published: 2025-06-10 17:57:30

For Large Language Models (LLMs) to be reliably deployed in both everyday and
high-stakes domains, knowing when not to answer is equally critical as
answering correctly. Real-world user queries, which can be underspecified,
ill-posed, or fundamentally unanswerable, require LLMs to reason about
uncertainty and selectively abstain -- i.e., refuse to answer definitively.
However, abstention remains understudied, without a systematic evaluation
framework for modern LLMs. In this work, we introduce AbstentionBench, a
large-scale benchmark for holistically evaluating abstention across 20 diverse
datasets, including questions with unknown answers, underspecification, false
premises, subjective interpretations, and outdated information. Evaluating 20
frontier LLMs reveals abstention is an unsolved problem, and one where scaling
models is of little use. While recent reasoning LLMs have shown impressive
results in complex problem solving, surprisingly, we find that reasoning
fine-tuning degrades abstention (by 24% on average), even for math and
science domains on which reasoning models are explicitly trained. We find that
while a carefully crafted system prompt can boost abstention in practice, it
does not resolve models' fundamental inability to reason about uncertainty. We
release AbstentionBench to foster research into advancing LLM reliability.