Orionnnn/hub-stats huggingface dataset:
Foundational Automatic Evaluators: Scaling Multi-Task Generative
  Evaluator Training for Reasoning-Centric Domains
Authors: name: Austin Xu, name: Xuan-Phi Nguyen, name: Yilun Zhou, name: Chien-Sheng Wu, name: Caiming Xiong, name: Shafiq Joty
Published: 2025-10-20 17:52:06

Finetuning specialized generative evaluators has emerged as a popular
paradigm to meet the increasing demand for scalable evaluation during both
training and test-time. However, recent work has largely focused on applying
new methodology, such as reinforcement learning (RL), to training evaluators,
shying away from large-scale, data-driven development. In this work, we focus
on data scaling, curating a set of 2.5M samples spanning five unique evaluation
tasks (pairwise, step-level, reference-free and reference-based verification,
and single rating) and multiple domains focused on reasoning evaluation. With
our data, we train Foundational Automatic Reasoning Evaluators (FARE), a family
of 8B and 20B (with 3.6B active) parameter evaluators, with a simple iterative
rejection-sampling supervised finetuning (SFT) approach. FARE-8B challenges
larger specialized RL-trained evaluators and FARE-20B sets the new standard for
open-source evaluators, surpassing specialized 70B+ evaluators. Beyond static
benchmarks, we evaluate FARE in real-world tasks: As inference-time rerankers,
FARE-20B achieves near-oracle performance on MATH. As verifiers in RL training,
FARE improves the downstream RL-trained model performance by up to 14.1% vs.
string-matching verifiers. When initialized from FARE, a continually-finetuned
FARE-Code outperforms gpt-oss-20B by 65% on evaluating test-case quality.
Constantly Improving Image Models Need Constantly Improving Benchmarks
Authors: name: Jiaxin Ge, name: Grace Luo, name: Heekyung Lee, name: Nishant Malpani, name: Long Lian, name: XuDong Wang, name: Aleksander Holynski, name: Trevor Darrell, name: Sewon Min, name: David M. Chan
Published: 2025-10-16 17:59:30

Recent advances in image generation, often driven by proprietary systems like
GPT-4o Image Gen, regularly introduce new capabilities that reshape how users
interact with these models. Existing benchmarks often lag behind and fail to
capture these emerging use cases, leaving a gap between community perceptions
of progress and formal evaluation. To address this, we present ECHO, a
framework for constructing benchmarks directly from real-world evidence of
model use: social media posts that showcase novel prompts and qualitative user
judgments. Applying this framework to GPT-4o Image Gen, we construct a dataset
of over 31,000 prompts curated from such posts. Our analysis shows that ECHO
(1) discovers creative and complex tasks absent from existing benchmarks, such
as re-rendering product labels across languages or generating receipts with
specified totals, (2) more clearly distinguishes state-of-the-art models from
alternatives, and (3) surfaces community feedback that we use to inform the
design of metrics for model quality (e.g., measuring observed shifts in color,
identity, and structure). Our website is at https://echo-bench.github.io.
When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM
  Ensembling
Authors: name: Heecheol Yun, name: Kwangmin Ki, name: Junghyun Lee, name: Eunho Yang
Published: 2025-10-17 06:18:29

Ensembling Large Language Models (LLMs) has gained attention as a promising
approach to surpass the performance of individual models by leveraging their
complementary strengths. In particular, aggregating models' next-token
probability distributions to select the next token has been shown to be
effective in various tasks. However, while successful for short-form answers,
its application to long-form generation remains underexplored. In this paper,
we show that using existing ensemble methods in long-form generation requires a
careful choice of ensembling positions, since the standard practice of
ensembling at every token often degrades performance. We identify two key
factors for determining these positions: tokenization mismatch across models
and consensus in their next-token probability distributions. Based on this, we
propose SAFE, (Stable And Fast LLM Ensembling), a framework that selectively
ensembles by jointly considering these factors. To further improve stability,
we introduce a probability sharpening strategy that consolidates probabilities
spread across multiple sub-word tokens representing the same word into a single
representative token. Our experiments on diverse benchmarks, including MATH500
and BBH, demonstrate that SAFE outperforms existing methods in both accuracy
and efficiency, with gains achieved even when ensembling fewer than 1% of
tokens.
Annotation-Efficient Universal Honesty Alignment
Authors: name: Shiyu Ni, name: Keping Bi, name: Jiafeng Guo, name: Minghao Tang, name: Jingtong Wu, name: Zengxin Han, name: Xueqi Cheng
Published: 2025-10-20 13:05:22

Honesty alignment-the ability of large language models (LLMs) to recognize
their knowledge boundaries and express calibrated confidence-is essential for
trustworthy deployment. Existing methods either rely on training-free
confidence estimation (e.g., token probabilities, self-consistency) or
training-based calibration with correctness annotations. While effective,
achieving universal honesty alignment with training-based calibration requires
costly, large-scale labeling. To support annotation-efficient training, we
introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that
first elicits internal confidence using inexpensive self-consistency
supervision, then calibrates this confidence with a small set of correctness
annotations. To support a large-scale study, we release HonestyBench, a
benchmark covering ten free-form QA datasets with 560k training and 70k
evaluation instances annotated with correctness and self-consistency signals.
Experiments show that EliCal achieves near-optimal alignment with only 1k
correctness annotations (0.18% of full supervision) and better alignment
performance on unseen MMLU tasks than the calibration-only baseline, offering a
scalable solution toward universal honesty alignment in LLMs.
Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware
  Finetuning and MLLM Implicit Feedback
Authors: name: Zongjian Li, name: Zheyuan Liu, name: Qihui Zhang, name: Bin Lin, name: Shenghai Yuan, name: Zhiyuan Yan, name: Yang Ye, name: Wangbo Yu, name: Yuwei Niu, name: Li Yuan
Published: 2025-10-19 15:38:06

Instruction-based image editing has achieved remarkable progress; however,
models solely trained via supervised fine-tuning often overfit to annotated
patterns, hindering their ability to explore and generalize beyond training
distributions. To this end, we introduce Edit-R1, a novel post-training
framework for instruction-based image editing based on policy optimization.
Specifically, we utilize Diffusion Negative-aware Finetuning (DiffusionNFT), a
likelihood-free policy optimization method consistent with the flow matching
forward process, thereby enabling the use of higher-order samplers and more
efficient training. Another key challenge here is the absence of a universal
reward model, resulting from the diverse nature of editing instructions and
tasks. To bridge this gap, we employ a Multimodal Large Language Model (MLLM)
as a unified, training-free reward model, leveraging its output logits to
provide fine-grained feedback. Furthermore, we carefully design a low-variance
group filtering mechanism to reduce MLLM scoring noise and stabilize
optimization. UniWorld-V2, trained with this framework, achieves
state-of-the-art results on the ImgEdit and GEdit-Bench benchmarks,
scoring 4.49 and 7.83, respectively. Crucially, our framework is
model-agnostic, delivering substantial performance gains when applied to
diverse base models like Qwen-Image-Edit and FLUX-Kontext, demonstrating its
wide applicability. Code and models are publicly available at
https://github.com/PKU-YuanGroup/UniWorld-V2.
UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
Authors: name: Yuhao Yang, name: Zhen Yang, name: Zi-Yi Dou, name: Anh Nguyen, name: Keen You, name: Omar Attia, name: Andrew Szot, name: Michael Feng, name: Ram Ramrakhya, name: Alexander Toshev, name: Chao Huang, name: Yinfei Yang, name: Zhe Gan
Published: 2025-10-20 17:48:26

Multimodal agents for computer use rely exclusively on primitive actions
(click, type, scroll) that require accurate visual grounding and lengthy
execution chains, leading to cascading failures and performance bottlenecks.
While other agents leverage rich programmatic interfaces (APIs, MCP servers,
tools), computer-use agents (CUAs) remain isolated from these capabilities. We
present UltraCUA, a foundation model that bridges this gap through hybrid
action -- seamlessly integrating GUI primitives with high-level programmatic
tool calls. To achieve this, our approach comprises four key components: (1) an
automated pipeline that scales programmatic tools from software documentation,
open-source repositories, and code generation; (2) a synthetic data engine
producing over 17,000 verifiable tasks spanning real-world computer-use
scenarios; (3) a large-scale high-quality hybrid action trajectory collection
with both low-level GUI actions and high-level programmatic tool calls; and (4)
a two-stage training pipeline combining supervised fine-tuning with online
reinforcement learning, enabling strategic alternation between low-level and
high-level actions. Experiments with our 7B and 32B models demonstrate
substantial improvements over state-of-the-art agents. On OSWorld, UltraCUA
models achieve an average 22% relative improvement over base models, while
being 11% faster in terms of steps. Out-of-domain evaluation on
WindowsAgentArena shows our model reaches 21.7% success rate, outperforming
baselines trained on Windows data. The hybrid action mechanism proves critical,
reducing error propagation while maintaining execution efficiency.
Glyph: Scaling Context Windows via Visual-Text Compression
Authors: name: Jiale Cheng, name: Yusen Liu, name: Xinyu Zhang, name: Yulin Fei, name: Wenyi Hong, name: Ruiliang Lyu, name: Weihan Wang, name: Zhe Su, name: Xiaotao Gu, name: Xiao Liu, name: Yushi Bai, name: Jie Tang, name: Hongning Wang, name: Minlie Huang
Published: 2025-10-20 17:58:56

Large language models (LLMs) increasingly rely on long-context modeling for
tasks such as document understanding, code analysis, and multi-step reasoning.
However, scaling context windows to the million-token level brings prohibitive
computational and memory costs, limiting the practicality of long-context LLMs.
In this work, we take a different perspective-visual context scaling-to tackle
this challenge. Instead of extending token-based sequences, we propose Glyph, a
framework that renders long texts into images and processes them with
vision-language models (VLMs). This approach substantially compresses textual
input while preserving semantic information, and we further design an
LLM-driven genetic search to identify optimal visual rendering configurations
for balancing accuracy and compression. Through extensive experiments, we
demonstrate that our method achieves 3-4x token compression while maintaining
accuracy comparable to leading LLMs such as Qwen3-8B on various long-context
benchmarks. This compression also leads to around 4x faster prefilling and
decoding, and approximately 2x faster SFT training. Furthermore, under extreme
compression, a 128K-context VLM could scale to handle 1M-token-level text
tasks. In addition, the rendered text data benefits real-world multimodal
tasks, such as document understanding. Our code and model are released at
https://github.com/thu-coai/Glyph.
RL makes MLLMs see better than SFT
Authors: name: Junha Song, name: Sangdoo Yun, name: Dongyoon Han, name: Jaegul Choo, name: Byeongho Heo
Published: 2025-10-18 03:37:17

A dominant assumption in Multimodal Language Model (MLLM) research is that
its performance is largely inherited from the LLM backbone, given its immense
parameter scale and remarkable capabilities. This has created a void in the
understanding of the vision encoder, which determines how MLLMs perceive
images. The recent shift in MLLM training paradigms, from Supervised Finetuning
(SFT) to Reinforcement Learning (RL), magnifies this oversight-namely, the
significant lack of analysis on how such training reshapes the vision encoder
as well as the MLLM. To address this, we first investigate the impact of
training strategies on MLLMs, where RL shows a clear advantage over SFT in
strongly vision-related VQA benchmarks. Motivated by this, we conduct a
critical yet under-explored analysis of the vision encoder of MLLMs through
diverse and in-depth experiments, ranging from ImageNet classification and
segmentation to gradient visualization. Our results demonstrate that MLLM's
post-training strategy (i.e., SFT or RL) not only leads to distinct outcomes on
MLLM downstream tasks, but also fundamentally reshapes MLLM's underlying visual
representations. Specifically, the key finding of our study is that RL produces
stronger and precisely localized visual representations compared to SFT,
boosting the ability of the vision encoder for MLLM. We then reframe our
findings into a simple recipe for building strong vision encoders for MLLMs,
Preference-Instructed Vision OpTimization (PIVOT). When integrated into MLLMs,
a PIVOT-trained vision encoder outperforms even larger and more heavily-trained
counterparts, despite requiring less than 1% of the computational cost of
standard vision pretraining. This result opens an effective and efficient path
for advancing the vision backbones of MLLMs. Project page available at
https://june-page.github.io/pivot/
Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset
Authors: name: Claire McLean, name: Makenzie Meendering, name: Tristan Swartz, name: Orri Gabbay, name: Alexandra Olsen, name: Rachel Jacobs, name: Nicholas Rosen, name: Philippe de Bree, name: Tony Garcia, name: Gadsden Merrill, name: Jake Sandakly, name: Julia Buffalini, name: Neham Jain, name: Steven Krenn, name: Moneish Kumar, name: Dejan Markovic, name: Evonne Ng, name: Fabian Prada, name: Andrew Saba, name: Siwei Zhang, name: Vasu Agrawal, name: Tim Godisart, name: Alexander Richard, name: Michael Zollhoefer
Published: 2025-10-17 23:06:36

The Codec Avatars Lab at Meta introduces Embody 3D, a multimodal dataset of
500 individual hours of 3D motion data from 439 participants collected in a
multi-camera collection stage, amounting to over 54 million frames of tracked
3D motion. The dataset features a wide range of single-person motion data,
including prompted motions, hand gestures, and locomotion; as well as
multi-person behavioral and conversational data like discussions, conversations
in different emotional states, collaborative activities, and co-living
scenarios in an apartment-like space. We provide tracked human motion including
hand tracking and body shape, text annotations, and a separate audio track for
each participant.
Distractor Injection Attacks on Large Reasoning Models: Characterization
  and Defense
Authors: name: Zhehao Zhang, name: Weijie Xu, name: Shixian Cui, name: Chandan K. Reddy
Published: 2025-10-17 23:16:34

Recent advances in large reasoning models (LRMs) have enabled remarkable
performance on complex tasks such as mathematics and coding by generating long
Chain-of-Thought (CoT) traces. In this paper, we identify and systematically
analyze a critical vulnerability we term reasoning distraction, where LRMs are
diverted from their primary objective by irrelevant yet complex tasks
maliciously embedded in the prompt. Through a comprehensive study across
diverse models and benchmarks, we show that even state-of-the-art LRMs are
highly susceptible, with injected distractors reducing task accuracy by up to
60%. We further reveal that certain alignment techniques can amplify this
weakness and that models may exhibit covert compliance, following hidden
adversarial instructions in reasoning while concealing them in the final
output. To mitigate these risks, we propose a training-based defense that
combines Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on
synthetic adversarial data, improving robustness by over 50 points on
challenging distractor attacks. Our findings establish reasoning distraction as
a distinct and urgent threat to LRM reliability and provide a practical step
toward safer and more trustworthy reasoning systems.
Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native
  Agentic AI
Authors: name: Jitao Sang, name: Jinlin Xiao, name: Jiarun Han, name: Jilin Chen, name: Xiaoyi Chen, name: Shuyu Wei, name: Yongjie Sun, name: Yuhang Wang
Published: 2025-10-19 05:23:43

The rapid evolution of agentic AI marks a new phase in artificial
intelligence, where Large Language Models (LLMs) no longer merely respond but
act, reason, and adapt. This survey traces the paradigm shift in building
agentic AI: from Pipeline-based systems, where planning, tool use, and memory
are orchestrated by external logic, to the emerging Model-native paradigm,
where these capabilities are internalized within the model's parameters. We
first position Reinforcement Learning (RL) as the algorithmic engine enabling
this paradigm shift. By reframing learning from imitating static data to
outcome-driven exploration, RL underpins a unified solution of LLM + RL + Task
across language, vision and embodied domains. Building on this, the survey
systematically reviews how each capability -- Planning, Tool use, and Memory --
has evolved from externally scripted modules to end-to-end learned behaviors.
Furthermore, it examines how this paradigm shift has reshaped major agent
applications, specifically the Deep Research agent emphasizing long-horizon
reasoning and the GUI agent emphasizing embodied interaction. We conclude by
discussing the continued internalization of agentic capabilities like
Multi-agent collaboration and Reflection, alongside the evolving roles of the
system and model layers in future agentic AI. Together, these developments
outline a coherent trajectory toward model-native agentic AI as an integrated
learning and interaction framework, marking the transition from constructing
systems that apply intelligence to developing models that grow intelligence
through experience.
AsyncVoice Agent: Real-Time Explanation for LLM Planning and Reasoning
Authors: name: Yueqian Lin, name: Zhengmian Hu, name: Jayakumar Subramanian, name: Qinsi Wang, name: Nikos Vlassis, name: Hai "Helen" Li, name: Yiran Chen
Published: 2025-10-17 19:00:08

Effective human-AI collaboration on complex reasoning tasks requires that
users understand and interact with the model's process, not just receive an
output. However, the monolithic text from methods like Chain-of-Thought (CoT)
prevents this, as current interfaces lack real-time verbalization and robust
user barge-in. We present AsyncVoice Agent, a system whose asynchronous
architecture decouples a streaming LLM backend from a conversational voice
frontend. This design allows narration and inference to run in parallel,
empowering users to interrupt, query, and steer the model's reasoning process
at any time. Objective benchmarks show this approach reduces interaction
latency by more than 600x compared to monolithic baselines while ensuring high
fidelity and competitive task accuracy. By enabling a two-way dialogue with a
model's thought process, AsyncVoice Agent offers a new paradigm for building
more effective, steerable, and trustworthy human-AI systems for high-stakes
tasks.
Deep Self-Evolving Reasoning
Authors: name: Zihan Liu, name: Shun Zheng, name: Xumeng Wen, name: Yang Wang, name: Jiang Bian, name: Mao Yang
Published: 2025-10-20 12:51:42

Long-form chain-of-thought reasoning has become a cornerstone of advanced
reasoning in large language models. While recent verification-refinement
frameworks have enabled proprietary models to solve Olympiad-level problems,
their effectiveness hinges on strong, reliable verification and correction
capabilities, which remain fragile in open-weight, smaller-scale models. This
work demonstrates that even with weak verification and refinement capabilities
on hard tasks, the reasoning limits of such models can be substantially
extended through a probabilistic paradigm we call Deep Self-Evolving Reasoning
(DSER). We conceptualize iterative reasoning as a Markov chain, where each step
represents a stochastic transition in the solution space. The key insight is
that convergence to a correct solution is guaranteed as long as the probability
of improvement marginally exceeds that of degradation. By running multiple
long-horizon, self-evolving processes in parallel, DSER amplifies these small
positive tendencies, enabling the model to asymptotically approach correct
answers. Empirically, we apply DSER to the DeepSeek-R1-0528-Qwen3-8B model. On
the challenging AIME 2024-2025 benchmark, DSER solves 5 out of 9 previously
unsolvable problems and boosts overall performance, enabling this compact model
to surpass the single-turn accuracy of its 600B-parameter teacher through
majority voting. Beyond its immediate utility for test-time scaling, the DSER
framework serves to diagnose the fundamental limitations of current open-weight
reasoners. By clearly delineating their shortcomings in self-verification,
refinement, and stability, our findings establish a clear research agenda for
developing next-generation models with powerful, intrinsic self-evolving
capabilities.
FineVision: Open Data Is All You Need
Authors: name: Luis Wiedmann, name: Orr Zohar, name: Amir Mahla, name: Xiaohan Wang, name: Rui Li, name: Thibaud Frere, name: Leandro von Werra, name: Aritra Roy Gosthipaty, name: Andrés Marafioti
Published: 2025-10-20 07:54:46

The advancement of vision-language models (VLMs) is hampered by a fragmented
landscape of inconsistent and contaminated public datasets. We introduce
FineVision, a meticulously collected, curated, and unified corpus of 24 million
samples - the largest open resource of its kind. We unify more than 200 sources
into 185 subsets via a semi-automated, human-in-the-loop pipeline: automation
performs bulk ingestion and schema mapping, while reviewers audit mappings and
spot-check outputs to verify faithful consumption of annotations, appropriate
formatting and diversity, and safety; issues trigger targeted fixes and
re-runs. The workflow further applies rigorous de-duplication within and across
sources and decontamination against 66 public benchmarks. FineVision also
encompasses agentic/GUI tasks with a unified action space; reviewers validate
schemas and inspect a sample of trajectories to confirm executable fidelity.
Models trained on FineVision consistently outperform those trained on existing
open mixtures across a broad evaluation suite, underscoring the benefits of
scale, data hygiene, and balanced automation with human oversight. We release
the corpus and curation tools to accelerate data-centric VLM research.