Orionnnn/hub-stats huggingface dataset:
FML-bench: A Benchmark for Automatic ML Research Agents Highlighting the
  Importance of Exploration Breadth
Authors: name: Qiran Zou, name: Hou Hei Lam, name: Wenhao Zhao, name: Yiming Tang, name: Tingting Chen, name: Samson Yu, name: Tianyi Zhang, name: Chang Liu, name: Xiangyang Ji, name: Dianbo Liu
Published: 2025-10-12 06:41:05

Large language models (LLMs) have sparked growing interest in automatic
machine learning research agents. Among them, agents capable of autonomously
proposing ideas and conducting machine learning experiments are particularly
promising, as they maximize research automation and accelerate scientific
progress by iteratively refining ideas based on experimental results. However,
comprehensively evaluating such agents remains challenging. Existing benchmarks
tend to overemphasize engineering aspects while neglecting academic rigor,
creating barriers that obscure a clear assessment of an agent's scientific
capabilities in machine learning research. They also suffer from limited task
diversity, an overemphasis on application-oriented tasks over fundamental
research problems, and limited scalability to realistic research settings. To
address these limitations, we introduce FML-bench, a benchmark designed to
evaluate automatic machine learning research agents on 8 diverse and
fundamental machine learning research problems. It reduces coding burden,
emphasizes fundamental problems rather than specific use cases, offers high
task diversity, and is extensible to real-world machine learning GitHub
repositories. Furthermore, we present a unified evaluation framework with five
complementary metrics, designed to comprehensively assess agent performance on
our benchmark. We evaluate state-of-the-art automatic research agents on
FML-bench, and find that agents employing broad research exploration strategies
outperform those focusing on narrow but deep exploration. These findings
suggest that emphasizing the breadth of exploration may lead to more effective
research outcomes than focusing solely on incremental refinement. Our benchmark
is available at https://github.com/qrzou/FML-bench.
Synthesizing Agentic Data for Web Agents with Progressive Difficulty
  Enhancement Mechanisms
Authors: name: Shrey Pandit, name: Xuan-Phi Nguyen, name: Yifei Ming, name: Austin Xu, name: Jiayu Wang, name: Caiming Xiong, name: Shafiq Joty
Published: 2025-10-15 06:34:46

Web-based 'deep research' agents aim to solve complex question - answering
tasks through long-horizon interactions with online tools. These tasks remain
challenging, as the underlying language models are often not optimized for
long-horizon reasoning and exploration. Prior work has proposed workflows for
constructing instruction-tuning datasets, often leveraging knowledge graphs.
However, such methods typically lack fine-grained control over difficulty and
quality, yielding synthetic data that falls short of capturing the complexity
required for long-horizon reasoning. Furthermore, many studies conflate data
and training effects by comparing models trained under different optimization
recipes, making it difficult to isolate and evaluate the effectiveness of the
data itself. We introduce a two-pronged data synthesis pipeline that generates
question - answer pairs by progressively increasing task complexity until a
frontier baseline web agent fails. The baseline agent plays multiple roles in
this process: attempting the questions, validating factuality, checking for
alternative answers, and enforcing filtering. To evaluate the effectiveness of
our synthesis methods, we adopt a controlled training setup based on
distillation from strong web agents. Experiments across multiple web-based
benchmarks show that our dataset - despite being smaller - enables the training
of more effective web agents than existing datasets. In particular, our data
exhibits twice the diversity in tool-use actions, allowing models trained on it
to achieve stronger performance while avoiding repetitive tool-calling
behaviors.
GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for
  Step-Level Reasoning
Authors: name: Yao Zhang, name: Yu Wu, name: Haowei Zhang, name: Weiguo Li, name: Haokun Chen, name: Jingpei Wu, name: Guohao Li, name: Zhen Han, name: Volker Tresp
Published: 2025-10-16 17:54:07

Process Reward Models (PRMs) aim to improve multi-step reasoning in Large
Language Models (LLMs) by supervising intermediate steps and identifying
errors. However, building effective PRMs remains challenging due to the lack of
scalable, high-quality annotations. Existing approaches rely on costly human
labeling, LLM-based self-evaluation that is prone to hallucination, or Monte
Carlo (MC) estimation, which infers step quality solely from rollout outcomes
and often introduces noisy, misaligned supervision due to credit
misattribution. These issues result in three core limitations: noisy rewards,
low factual fidelity, and misalignment with step-level reasoning objectives. To
address these challenges, we introduce GroundedPRM, a tree-guided and
fidelity-aware framework for automatic process supervision. To reduce reward
noise and enable fine-grained credit assignment, we construct structured
reasoning paths via Monte Carlo Tree Search (MCTS). To eliminate hallucinated
supervision, we validate each intermediate step using an external tool,
providing execution-grounded correctness signals. To combine both step-level
validation and global outcome assessment, we design a hybrid reward aggregation
mechanism that fuses tool-based verification with MCTS-derived feedback.
Finally, we format the reward signal into a rationale-enhanced, generative
structure to promote interpretability and compatibility with instruction-tuned
LLMs. GroundedPRM is trained on only 40K automatically labeled samples,
amounting to just 10% of the data used by the best-performing PRM trained with
auto-labeled supervision. Nevertheless, it achieves up to a 26% relative
improvement in average performance on ProcessBench. When used for reward-guided
greedy search, GroundedPRM outperforms even PRMs trained with human-labeled
supervision, offering a scalable and verifiable path toward high-quality
process-level reasoning.
AnyUp: Universal Feature Upsampling
Authors: name: Thomas Wimmer, name: Prune Truong, name: Marie-Julie Rakotosaona, name: Michael Oechsle, name: Federico Tombari, name: Bernt Schiele, name: Jan Eric Lenssen
Published: 2025-10-14 17:45:17

We introduce AnyUp, a method for feature upsampling that can be applied to
any vision feature at any resolution, without encoder-specific training.
Existing learning-based upsamplers for features like DINO or CLIP need to be
re-trained for every feature extractor and thus do not generalize to different
feature types at inference time. In this work, we propose an inference-time
feature-agnostic upsampling architecture to alleviate this limitation and
improve upsampling quality. In our experiments, AnyUp sets a new state of the
art for upsampled features, generalizes to different feature types, and
preserves feature semantics while being efficient and easy to apply to a wide
range of downstream tasks.
Agentic Design of Compositional Machines
Authors: name: Wenqian Zhang, name: Weiyang Liu, name: Zhen Liu
Published: 2025-10-16 17:59:58

The design of complex machines stands as both a marker of human intelligence
and a foundation of engineering practice. Given recent advances in large
language models (LLMs), we ask whether they, too, can learn to create. We
approach this question through the lens of compositional machine design: a task
in which machines are assembled from standardized components to meet functional
demands like locomotion or manipulation in a simulated physical environment. To
support this investigation, we introduce BesiegeField, a testbed built on the
machine-building game Besiege, which enables part-based construction, physical
simulation and reward-driven evaluation. Using BesiegeField, we benchmark
state-of-the-art LLMs with agentic workflows and identify key capabilities
required for success, including spatial reasoning, strategic assembly, and
instruction-following. As current open-source models fall short, we explore
reinforcement learning (RL) as a path to improvement: we curate a cold-start
dataset, conduct RL finetuning experiments, and highlight open challenges at
the intersection of language, machine design, and physical reasoning.
Mirror Speculative Decoding: Breaking the Serial Barrier in LLM
  Inference
Authors: name: Nikhil Bhendawade, name: Kumari Nishu, name: Arnav Kundu, name: Chris Bartels, name: Minsik Cho, name: Irina Belousova
Published: 2025-10-15 05:22:57

Speculative decoding accelerates LLM inference by using a draft model to look
ahead, but gains are capped by the cost of autoregressive draft generation:
increasing draft size elevates acceptance rates but introduces additional
latency overhead exacerbating the speed-accuracy tradeoff. Prior methods
(Medusa, Hydra, EAGLE) partially reduce draft cost but either degrade
acceptance or introduce overheads that limit scaling. We present Mirror
Speculative Decoding (Mirror-SD), an inference algorithm that breaks the
latency-acceptance tradeoff. Mirror-SD launches branch-complete rollouts from
early-exit signals in parallel with the target model's suffix and explicitly
maps computation across heterogeneous accelerators (GPU and NPU) to exploit
cross-device parallelism. The draft speculates forward continuations for the
target to verify, while the target simultaneously speculates correction paths
for the draft, converting speculation into two complementary execution
pipelines. To further cut draft latency without weakening acceptance semantics,
we add speculative streaming so the draft emits multiple tokens per step. This
dual strategy of parallel heterogeneous execution plus multi-token speculative
streaming pushes speculative decoding toward its ideal regime of high
acceptance with low overhead. On SpecBench with server-scale models from 14B to
66B parameters, Mirror-SD delivers consistent end-to-end gains, achieving
2.8x-5.8x wall-time speedups across diverse tasks and a 30% average relative
improvement over the strongest baseline, EAGLE3.
SimKO: Simple Pass@K Policy Optimization
Authors: name: Ruotian Peng, name: Yi Ren, name: Zhouliang Yu, name: Weiyang Liu, name: Yandong Wen
Published: 2025-10-16 15:40:49

Reinforcement learning with verifiable rewards (RLVR) has advanced the
reasoning capabilities of large language models (LLMs). However, prevailing
RLVR methods exhibit a systematic bias toward exploitation over exploration, as
evidenced by improved pass@1 but reduced pass@K (K>1) performance. To
understand this issue, we analyze training dynamics of RLVR methods by tracking
the token-level probability distributions over vocabulary candidates. Our
analysis reveals a consistent probability concentration effect where the top-1
candidate increasingly accumulates probability mass and suppresses that of
other candidates. More importantly, stronger over-concentration correlates with
worse pass@K performance. Inspired by this finding, we propose Simple Pass@K
Optimization (SimKO), a method designed to mitigate the over-concentration
issue, thereby encouraging exploration. SimKO operates in an asymmetrical
manner. For verified-correct responses, it boosts the probabilities of the
top-K candidates. For verified-incorrect responses, it applies stronger
penalties to the top-1 candidate. We observe that this asymmetric design is
particularly effective at mitigating over-concentration when applied at tokens
with high entropy. Across various math and logical-reasoning benchmarks, SimKO
consistently yields higher pass@K for a wide range of K, providing a simple way
to improve RLVR's exploration.
Efficient Parallel Samplers for Recurrent-Depth Models and Their
  Connection to Diffusion Language Models
Authors: name: Jonas Geiping, name: Xinyu Yang, name: Guinan Su
Published: 2025-10-16 17:59:07

Language models with recurrent depth, also referred to as universal or looped
when considering transformers, are defined by the capacity to increase their
computation through the repetition of layers. Recent efforts in pretraining
have demonstrated that these architectures can scale to modern language
modeling tasks while exhibiting advantages in reasoning tasks. In this work, we
examine the relationship between recurrent-depth models and diffusion language
models. Building on their similarities, we develop a new diffusion forcing
sampler for these models to accelerate generation. The sampler advances by
decoding new tokens at every forward pass of the model, while the latent states
of these tokens can be further refined in parallel through recurrence.
Theoretically, generation with our sampler is strictly more expressive than the
baseline autoregressive generation using the same time budget on modern
hardware. Moreover, this sampler, based on principles from diffusion
literature, can be directly applied to existing 3.5B recurrent-depth
transformers without any tuning, leading to up to a 5x speedup. Consequently,
our findings not only provide an efficient mechanism for parallelizing the
extra computation in recurrent-depth models at inference, but also suggest that
such models can be naturally viewed as strong continuous, though causal,
diffusion language models.
When Models Lie, We Learn: Multilingual Span-Level Hallucination
  Detection with PsiloQA
Authors: name: Elisei Rykov, name: Kseniia Petrushina, name: Maksim Savkin, name: Valerii Olisov, name: Artem Vazhentsev, name: Kseniia Titova, name: Alexander Panchenko, name: Vasily Konovalov, name: Julia Belikova
Published: 2025-10-06 14:36:30

Hallucination detection remains a fundamental challenge for the safe and
reliable deployment of large language models (LLMs), especially in applications
requiring factual accuracy. Existing hallucination benchmarks often operate at
the sequence level and are limited to English, lacking the fine-grained,
multilingual supervision needed for a comprehensive evaluation. In this work,
we introduce PsiloQA, a large-scale, multilingual dataset annotated with
span-level hallucinations across 14 languages. PsiloQA is constructed through
an automated three-stage pipeline: generating question-answer pairs from
Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse
LLMs in a no-context setting, and automatically annotating hallucinated spans
using GPT-4o by comparing against golden answers and retrieved context. We
evaluate a wide range of hallucination detection methods -- including
uncertainty quantification, LLM-based tagging, and fine-tuned encoder models --
and show that encoder-based models achieve the strongest performance across
languages. Furthermore, PsiloQA demonstrates effective cross-lingual
generalization and supports robust knowledge transfer to other benchmarks, all
while being significantly more cost-efficient than human-annotated datasets.
Our dataset and results advance the development of scalable, fine-grained
hallucination detection in multilingual settings.
On Pretraining for Project-Level Code Completion
Authors: name: Maksim Sapronov, name: Evgeniy Glukhov
Published: 2025-10-15 15:55:19

Repository-level pretraining is commonly used to enable large language models
for code to leverage codebase-wide context. This enhances their ability to
generate accurate and context-aware code completions. In this work, we
investigate how different repository-processing strategies affect in-context
learning in OpenCoder, a 1.5B-parameter model. We extend its context window
from 4,096 to 16,384 tokens by training on additional 1B tokens of curated
repository-level data. Despite relying on a smaller dataset than competing
models (which often use hundreds of billions of tokens), our model achieves
comparable performance on the Long Code Arena benchmark. We find that various
repository-processing techniques yield similarly strong results, with the
primary gain coming from adapting to a new rotary positional embedding (RoPE)
scaling parameter. Finally, we show that a simpler file-level training approach
at the original sequence length remains highly effective, opening up
repository-level code completion research to settings with more constrained
data and compute resources.
RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval
  Augmented Generation Systems
Authors: name: Jingru Lin, name: Chen Zhang, name: Stephen Y. Liu, name: Haizhou Li
Published: 2025-10-15 04:13:00

Retrieval-Augmented Generation (RAG) mitigates key limitations of Large
Language Models (LLMs)-such as factual errors, outdated knowledge, and
hallucinations-by dynamically retrieving external information. Recent work
extends this paradigm through agentic RAG systems, where LLMs act as agents to
iteratively plan, retrieve, and reason over complex queries. However, these
systems still struggle with challenging multi-hop questions, and their
intermediate reasoning capabilities remain underexplored. To address this, we
propose RAGCap-Bench, a capability-oriented benchmark for fine-grained
evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs
from state-of-the-art systems to identify common tasks and the core
capabilities required for their execution, then construct a taxonomy of typical
LLM errors to design targeted evaluation questions. Experiments show that
"slow-thinking" models with stronger RAGCap performance achieve better
end-to-end results, underscoring the benchmark's validity and the importance of
enhancing these intermediate capabilities.
ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond
  Semantic Dependency Constraints
Authors: name: Meiqi Wu, name: Jiashu Zhu, name: Xiaokun Feng, name: Chubin Chen, name: Chen Zhu, name: Bingze Song, name: Fangyuan Mao, name: Jiahong Wu, name: Xiangxiang Chu, name: Kaiqi Huang
Published: 2025-10-16 16:19:13

Video generation models have achieved remarkable progress, particularly
excelling in realistic scenarios; however, their performance degrades notably
in imaginative scenarios. These prompts often involve rarely co-occurring
concepts with long-distance semantic relationships, falling outside training
distributions. Existing methods typically apply test-time scaling for improving
video quality, but their fixed search spaces and static reward designs limit
adaptability to imaginative scenarios. To fill this gap, we propose
ImagerySearch, a prompt-guided adaptive test-time search strategy that
dynamically adjusts both the inference search space and reward function
according to semantic relationships in the prompt. This enables more coherent
and visually plausible videos in challenging imaginative settings. To evaluate
progress in this direction, we introduce LDT-Bench, the first dedicated
benchmark for long-distance semantic prompts, consisting of 2,839 diverse
concept pairs and an automated protocol for assessing creative generation
capabilities. Extensive experiments show that ImagerySearch consistently
outperforms strong video generation baselines and existing test-time scaling
approaches on LDT-Bench, and achieves competitive improvements on VBench,
demonstrating its effectiveness across diverse prompt types. We will release
LDT-Bench and code to facilitate future research on imaginative video
generation.
RealDPO: Real or Not Real, that is the Preference
Authors: name: Guo Cheng, name: Danni Yang, name: Ziqi Huang, name: Jianlou Si, name: Chenyang Si, name: Ziwei Liu
Published: 2025-10-16 17:58:25

Video generative models have recently achieved notable advancements in
synthesis quality. However, generating complex motions remains a critical
challenge, as existing models often struggle to produce natural, smooth, and
contextually consistent movements. This gap between generated and real-world
motions limits their practical applicability. To address this issue, we
introduce RealDPO, a novel alignment paradigm that leverages real-world data as
positive samples for preference learning, enabling more accurate motion
synthesis. Unlike traditional supervised fine-tuning (SFT), which offers
limited corrective feedback, RealDPO employs Direct Preference Optimization
(DPO) with a tailored loss function to enhance motion realism. By contrasting
real-world videos with erroneous model outputs, RealDPO enables iterative
self-correction, progressively refining motion quality. To support
post-training in complex motion synthesis, we propose RealAction-5K, a curated
dataset of high-quality videos capturing human daily activities with rich and
precise motion details. Extensive experiments demonstrate that RealDPO
significantly improves video quality, text alignment, and motion realism
compared to state-of-the-art models and existing preference optimization
techniques.
SCas4D: Structural Cascaded Optimization for Boosting Persistent 4D
  Novel View Synthesis
Authors: name: Jipeng Lyu, name: Jiahua Dong, name: Yu-Xiong Wang
Published: 2025-10-08 06:39:33

Persistent dynamic scene modeling for tracking and novel-view synthesis
remains challenging due to the difficulty of capturing accurate deformations
while maintaining computational efficiency. We propose SCas4D, a cascaded
optimization framework that leverages structural patterns in 3D Gaussian
Splatting for dynamic scenes. The key idea is that real-world deformations
often exhibit hierarchical patterns, where groups of Gaussians share similar
transformations. By progressively refining deformations from coarse part-level
to fine point-level, SCas4D achieves convergence within 100 iterations per time
frame and produces results comparable to existing methods with only
one-twentieth of the training iterations. The approach also demonstrates
effectiveness in self-supervised articulated object segmentation, novel view
synthesis, and dense point tracking tasks.
VLA^2: Empowering Vision-Language-Action Models with an Agentic
  Framework for Unseen Concept Manipulation
Authors: name: Han Zhao, name: Jiaxuan Zhang, name: Wenxuan Song, name: Pengxiang Ding, name: Donglin Wang
Published: 2025-10-16 17:18:34

Current vision-language-action (VLA) models, pre-trained on large-scale
robotic data, exhibit strong multi-task capabilities and generalize well to
variations in visual and language instructions for manipulation. However, their
success rate drops significantly when faced with object concepts outside the
training data, such as unseen object descriptions and textures in the dataset.
To address this, we propose a novel agentic framework, VLA^2, which leverages
OpenVLA as the execution backbone and effectively leverages external modules
such as web retrieval and object detection to provide visual and textual
knowledge about target objects to the VLA. This approach mitigates
generalization failure when handling out-of-distribution objects. Based on the
LIBERO simulation environment, we introduced novel objects and object
descriptions to construct a new evaluation benchmark with three difficulty
levels to test the effectiveness of our method. Our framework successfully
outperformed the current state-of-the-art models on our designed hard-level
generalization benchmark. Compared to the standalone OpenVLA baseline, VLA^2
achieves a 44.2% improvement in the success rate in the hard-level benchmark
and an average improvement of 20.2% in all customized environments without any
performance degradation on in-domain tasks. Project website:
https://vla-2.github.io.