Orionnnn/hub-stats huggingface dataset:
From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating
  Financial Large Language Models
Authors: name: Ziyan Kuang, name: Feiyu Zhu, name: Maowei Jiang, name: Yanzhao Lai, name: Zelin Wang, name: Zhitong Wang, name: Meikang Qiu, name: Jiajia Huang, name: Min Peng, name: Qianqian Xie, name: Sophia Ananiadou
Published: 2025-08-19 03:52:15

Large Language Models (LLMs) have shown promise for financial applications,
yet their suitability for this high-stakes domain remains largely unproven due
to inadequacies in existing benchmarks. Existing benchmarks solely rely on
score-level evaluation, summarizing performance with a single score that
obscures the nuanced understanding of what models truly know and their precise
limitations. They also rely on datasets that cover only a narrow subset of
financial concepts, while overlooking other essentials for real-world
applications. To address these gaps, we introduce FinCDM, the first cognitive
diagnosis evaluation framework tailored for financial LLMs, enabling the
evaluation of LLMs at the knowledge-skill level, identifying what financial
skills and knowledge they have or lack based on their response patterns across
skill-tagged tasks, rather than a single aggregated number. We construct
CPA-QKA, the first cognitively informed financial evaluation dataset derived
from the Certified Public Accountant (CPA) examination, with comprehensive
coverage of real-world accounting and financial skills. It is rigorously
annotated by domain experts, who author, validate, and annotate questions with
high inter-annotator agreement and fine-grained knowledge labels. Our extensive
experiments on 30 proprietary, open-source, and domain-specific LLMs show that
FinCDM reveals hidden knowledge gaps, identifies under-tested areas such as tax
and regulatory reasoning overlooked by traditional benchmarks, and uncovers
behavioral clusters among models. FinCDM introduces a new paradigm for
financial LLM evaluation by enabling interpretable, skill-aware diagnosis that
supports more trustworthy and targeted model development, and all datasets and
evaluation scripts will be publicly released to support further research.
RynnEC: Bringing MLLMs into Embodied World
Authors: name: Ronghao Dang, name: Yuqian Yuan, name: Yunxuan Mao, name: Kehan Li, name: Jiangpin Liu, name: Zhikai Wang, name: Xin Li, name: Fan Wang, name: Deli Zhao
Published: 2025-08-19 18:00:01

We introduce RynnEC, a video multimodal large language model designed for
embodied cognition. Built upon a general-purpose vision-language foundation
model, RynnEC incorporates a region encoder and a mask decoder, enabling
flexible region-level video interaction. Despite its compact architecture,
RynnEC achieves state-of-the-art performance in object property understanding,
object segmentation, and spatial reasoning. Conceptually, it offers a
region-centric video paradigm for the brain of embodied agents, providing
fine-grained perception of the physical world and enabling more precise
interactions. To mitigate the scarcity of annotated 3D datasets, we propose an
egocentric video based pipeline for generating embodied cognition data.
Furthermore, we introduce RynnEC-Bench, a region-centered benchmark for
evaluating embodied cognitive capabilities. We anticipate that RynnEC will
advance the development of general-purpose cognitive cores for embodied agents
and facilitate generalization across diverse embodied tasks. The code, model
checkpoints, and benchmark are available at:
https://github.com/alibaba-damo-academy/RynnEC
MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds
Authors: name: Bingquan Dai, name: Li Ray Luo, name: Qihong Tang, name: Jie Wang, name: Xinyu Lian, name: Hao Xu, name: Minghan Qin, name: Xudong Xu, name: Bo Dai, name: Haoqian Wang, name: Zhaoyang Lyu, name: Jiangmiao Pang
Published: 2025-08-20 17:50:15

Reconstructing 3D objects into editable programs is pivotal for applications
like reverse engineering and shape editing. However, existing methods often
rely on limited domain-specific languages (DSLs) and small-scale datasets,
restricting their ability to model complex geometries and structures. To
address these challenges, we introduce MeshCoder, a novel framework that
reconstructs complex 3D objects from point clouds into editable Blender Python
scripts. We develop a comprehensive set of expressive Blender Python APIs
capable of synthesizing intricate geometries. Leveraging these APIs, we
construct a large-scale paired object-code dataset, where the code for each
object is decomposed into distinct semantic parts. Subsequently, we train a
multimodal large language model (LLM) that translates 3D point cloud into
executable Blender Python scripts. Our approach not only achieves superior
performance in shape-to-code reconstruction tasks but also facilitates
intuitive geometric and topological editing through convenient code
modifications. Furthermore, our code-based representation enhances the
reasoning capabilities of LLMs in 3D shape understanding tasks. Together, these
contributions establish MeshCoder as a powerful and flexible solution for
programmatic 3D shape reconstruction and understanding.
Tinker: Diffusion's Gift to 3D--Multi-View Consistent Editing From
  Sparse Inputs without Per-Scene Optimization
Authors: name: Canyu Zhao, name: Xiaoman Li, name: Tianjian Feng, name: Zhiyue Zhao, name: Hao Chen, name: Chunhua Shen
Published: 2025-08-20 16:02:59

We introduce Tinker, a versatile framework for high-fidelity 3D editing that
operates in both one-shot and few-shot regimes without any per-scene
finetuning. Unlike prior techniques that demand extensive per-scene
optimization to ensure multi-view consistency or to produce dozens of
consistent edited input views, Tinker delivers robust, multi-view consistent
edits from as few as one or two images. This capability stems from repurposing
pretrained diffusion models, which unlocks their latent 3D awareness. To drive
research in this space, we curate the first large-scale multi-view editing
dataset and data pipeline, spanning diverse scenes and styles. Building on this
dataset, we develop our framework capable of generating multi-view consistent
edited views without per-scene training, which consists of two novel
components: (1) Referring multi-view editor: Enables precise, reference-driven
edits that remain coherent across all viewpoints. (2) Any-view-to-video
synthesizer: Leverages spatial-temporal priors from video diffusion to perform
high-quality scene completion and novel-view generation even from sparse
inputs. Through extensive experiments, Tinker significantly reduces the barrier
to generalizable 3D content creation, achieving state-of-the-art performance on
editing, novel-view synthesis, and rendering enhancement tasks. We believe that
Tinker represents a key step towards truly scalable, zero-shot 3D editing.
Project webpage: https://aim-uofa.github.io/Tinker
FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
Authors: name: Zhiyuan Zeng, name: Jiashuo Liu, name: Siyuan Chen, name: Tianci He, name: Yali Liao, name: Jinpeng Wang, name: Zaiyuan Wang, name: Yang Yang, name: Lingyue Yin, name: Mingren Yin, name: Zhenwei Zhu, name: Tianle Cai, name: Zehui Chen, name: Jiecao Chen, name: Yantao Du, name: Xiang Gao, name: Jiacheng Guo, name: Liang Hu, name: Jianpeng Jiao, name: Xiangsheng Li, name: Jingkai Liu, name: Shuang Ni, name: Zhoufutu Wen, name: Ge Zhang, name: Kaiyuan Zhang, name: Xin Zhou, name: Jose Blanchet, name: Xipeng Qiu, name: Mengdi Wang, name: Wenhao Huang
Published: 2025-08-16 08:54:08

Future prediction is a complex task for LLM agents, requiring a high level of
analytical thinking, information gathering, contextual understanding, and
decision-making under uncertainty. Agents must not only gather and interpret
vast amounts of dynamic information but also integrate diverse data sources,
weigh uncertainties, and adapt predictions based on emerging trends, just as
human experts do in fields like politics, economics, and finance. Despite its
importance, no large-scale benchmark exists for evaluating agents on future
prediction, largely due to challenges in handling real-time updates and
retrieving timely, accurate answers. To address this, we introduce
FutureX, a dynamic and live evaluation benchmark specifically
designed for LLM agents performing future prediction tasks. FutureX is the
largest and most diverse live benchmark for future prediction, supporting
real-time daily updates and eliminating data contamination through an automated
pipeline for question gathering and answer collection. We evaluate 25 LLM/agent
models, including those with reasoning, search capabilities, and integration of
external tools such as the open-source Deep Research Agent and closed-source
Deep Research models. This comprehensive evaluation assesses agents' adaptive
reasoning and performance in dynamic environments. Additionally, we provide
in-depth analyses of agents' failure modes and performance pitfalls in
future-oriented tasks, including the vulnerability to fake web pages and the
temporal validity. Our goal is to establish a dynamic, contamination-free
evaluation standard that drives the development of LLM agents capable of
performing at the level of professional human analysts in complex reasoning and
predictive thinking.
Retrieval-augmented reasoning with lean language models
Authors: name: Ryan Sze-Yin Chan, name: Federico Nanni, name: Tomas Lazauskas, name: Rosie Wood, name: Penelope Yong, name: Lionel Tarassenko, name: Mark Girolami, name: James Geddes, name: Andrew Duncan
Published: 2025-08-15 10:38:15

This technical report details a novel approach to combining reasoning and
retrieval augmented generation (RAG) within a single, lean language model
architecture. While existing RAG systems typically rely on large-scale models
and external APIs, our work addresses the increasing demand for performant and
privacy-preserving solutions deployable in resource-constrained or secure
environments. Building on recent developments in test-time scaling and
small-scale reasoning models, we develop a retrieval augmented conversational
agent capable of interpreting complex, domain-specific queries using a
lightweight backbone model. Our system integrates a dense retriever with
fine-tuned Qwen2.5-Instruct models, using synthetic query generation and
reasoning traces derived from frontier models (e.g., DeepSeek-R1) over a
curated corpus, in this case, the NHS A-to-Z condition pages. We explore the
impact of summarisation-based document compression, synthetic data design, and
reasoning-aware fine-tuning on model performance. Evaluation against both
non-reasoning and general-purpose lean models demonstrates that our
domain-specific fine-tuning approach yields substantial gains in answer
accuracy and consistency, approaching frontier-level performance while
remaining feasible for local deployment. All implementation details and code
are publicly released to support reproducibility and adaptation across domains.
Rapidly Adapting to New Voice Spoofing: Few-Shot Detection of
  Synthesized Speech Under Distribution Shifts
Authors: name: Ashi Garg, name: Zexin Cai, name: Henry Li Xinyuan, name: Leibny Paola García-Perera, name: Kevin Duh, name: Sanjeev Khudanpur, name: Matthew Wiesner, name: Nicholas Andrews
Published: 2025-08-18 19:14:45

We address the challenge of detecting synthesized speech under distribution
shifts -- arising from unseen synthesis methods, speakers, languages, or audio
conditions -- relative to the training data. Few-shot learning methods are a
promising way to tackle distribution shifts by rapidly adapting on the basis of
a few in-distribution samples. We propose a self-attentive prototypical network
to enable more robust few-shot adaptation. To evaluate our approach, we
systematically compare the performance of traditional zero-shot detectors and
the proposed few-shot detectors, carefully controlling training conditions to
introduce distribution shifts at evaluation time. In conditions where
distribution shifts hamper the zero-shot performance, our proposed few-shot
adaptation technique can quickly adapt using as few as 10 in-distribution
samples -- achieving upto 32% relative EER reduction on deepfakes in Japanese
language and 20% relative reduction on ASVspoof 2021 Deepfake dataset.
Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic
  Thought Reward
Authors: name: Yong Deng, name: Guoqing Wang, name: Zhenzhe Ying, name: Xiaofeng Wu, name: Jinzhen Lin, name: Wenwen Xiong, name: Yuqin Dai, name: Shuo Yang, name: Zhanwei Zhang, name: Qiwen Wang, name: Yang Qin, name: Changhua Meng
Published: 2025-08-18 10:23:10

Large language models (LLMs) exhibit remarkable problem-solving abilities,
but struggle with complex tasks due to static internal knowledge.
Retrieval-Augmented Generation (RAG) enhances access to external information,
yet remains limited in multi-hop reasoning and strategic search due to rigid
workflows. Recent advancements in agentic deep research empower LLMs to
autonomously reason, search, and synthesize information. However, current
approaches relying on outcome-based reinforcement learning (RL) face critical
issues such as conflicting gradients and reward sparsity, limiting performance
gains and training efficiency. To address these, we first propose Atomic
Thought, a novel LLM thinking paradigm that decomposes reasoning into
fine-grained functional units. These units are supervised by Reasoning Reward
Models (RRMs), which provide Atomic Thought Rewards (ATR) for fine-grained
guidance. Building on this, we propose Atom-Searcher, a novel RL framework for
agentic deep research that integrates Atomic Thought and ATR. Atom-Searcher
uses a curriculum-inspired reward schedule, prioritizing process-level ATR
early and transitioning to outcome rewards, accelerating convergence on
effective reasoning paths. Experiments on seven benchmarks show consistent
improvements over the state-of-the-art. Key advantages include: (1)
Atom-Searcher scales computation at test-time. (2) Atomic Thought provides
supervision anchors for RRMs, bridging deep research tasks and RRMs. (3)
Atom-Searcher exhibits more interpretable, human-like reasoning patterns.
CAMAR: Continuous Actions Multi-Agent Routing
Authors: name: Artem Pshenitsyn, name: Aleksandr Panov, name: Alexey Skrynnik
Published: 2025-08-18 11:32:26

Multi-agent reinforcement learning (MARL) is a powerful paradigm for solving
cooperative and competitive decision-making problems. While many MARL
benchmarks have been proposed, few combine continuous state and action spaces
with challenging coordination and planning tasks. We introduce CAMAR, a new
MARL benchmark designed explicitly for multi-agent pathfinding in environments
with continuous actions. CAMAR supports cooperative and competitive
interactions between agents and runs efficiently at up to 100,000 environment
steps per second. We also propose a three-tier evaluation protocol to better
track algorithmic progress and enable deeper analysis of performance. In
addition, CAMAR allows the integration of classical planning methods such as
RRT and RRT* into MARL pipelines. We use them as standalone baselines and
combine RRT* with popular MARL algorithms to create hybrid approaches. We
provide a suite of test scenarios and benchmarking tools to ensure
reproducibility and fair comparison. Experiments show that CAMAR presents a
challenging and realistic testbed for the MARL community.
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
Authors: name: Shilong Li, name: Xingyuan Bu, name: Wenjie Wang, name: Jiaheng Liu, name: Jun Dong, name: Haoyang He, name: Hao Lu, name: Haozhe Zhang, name: Chenchen Jing, name: Zhen Li, name: Chuanhao Li, name: Jiayi Tian, name: Chenchen Zhang, name: Tianhao Peng, name: Yancheng He, name: Jihao Gu, name: Yuanxing Zhang, name: Jian Yang, name: Ge Zhang, name: Wenhao Huang, name: Wangchunshu Zhou, name: Zhaoxiang Zhang, name: Ruizhe Ding, name: Shilei Wen
Published: 2025-08-14 13:46:47

AI agents with advanced reasoning and tool use capabilities have demonstrated
impressive performance in web browsing for deep search. While existing
benchmarks such as BrowseComp evaluate these browsing abilities, they primarily
focus on textual information, overlooking the prevalence of multimodal content.
To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising
224 challenging, hand-crafted questions specifically designed to assess agents'
multimodal retrieval and reasoning capabilities. These questions often
incorporate images in prompts, and crucial information encountered during the
search and reasoning process may also be embedded within images or videos on
webpages. Consequently, methods relying solely on text prove insufficient for
our benchmark. Additionally, we provide a verified checklist for each question,
enabling fine-grained analysis of multimodal dependencies and reasoning paths.
Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp
reveals that even top models like OpenAI o3 with tools achieve only 29.02\%
accuracy, highlighting the suboptimal multimodal capabilities and lack of
native multimodal reasoning in current models.
Mind the Generation Process: Fine-Grained Confidence Estimation During
  LLM Generation
Authors: name: Jinyi Han, name: Tingyun Li, name: Shisong Chen, name: Jie Shi, name: Xinyi Wang, name: Guanglei Yue, name: Jiaqing Liang, name: Xin Lin, name: Liqian Wen, name: Zulong Chen, name: Yanghua Xiao
Published: 2025-08-16 13:29:35

While large language models (LLMs) have demonstrated remarkable performance
across diverse tasks, they fundamentally lack self-awareness and frequently
exhibit overconfidence, assigning high confidence scores to incorrect
predictions. Accurate confidence estimation is therefore critical for enhancing
the trustworthiness and reliability of LLM-generated outputs. However, existing
approaches suffer from coarse-grained scoring mechanisms that fail to provide
fine-grained, continuous confidence estimates throughout the generation
process. To address these limitations, we introduce FineCE, a novel confidence
estimation method that delivers accurate, fine-grained confidence scores during
text generation. Specifically, we first develop a comprehensive pipeline for
constructing training data that effectively captures the underlying
probabilistic distribution of LLM responses, and then train a model to predict
confidence scores for arbitrary text sequences in a supervised manner.
Furthermore, we propose a Backward Confidence Integration (BCI) strategy that
leverages information from the subsequent text to enhance confidence estimation
for the current sequence during inference. We also introduce three strategies
for identifying optimal positions to perform confidence estimation within the
generation process. Extensive experiments on multiple benchmark datasets
demonstrate that FineCE consistently outperforms existing classical confidence
estimation methods. Our code and all baselines used in the paper are available
on GitHub.
A Stitch in Time Saves Nine: Proactive Self-Refinement for Language
  Models
Authors: name: Jinyi Han, name: Xinyi Wang, name: Haiquan Zhao, name: Tingyun li, name: Zishang Jiang, name: Sihang Jiang, name: Jiaqing Liang, name: Xin Lin, name: Weikang Zhou, name: Zeye Sun, name: Fei Yu, name: Yanghua Xiao
Published: 2025-08-18 13:07:21

Recent advances in self-refinement have demonstrated significant potential
for improving the outputs of large language models (LLMs) through iterative
refinement. However, most existing self-refinement methods rely on a reactive
process with a fixed number of iterations, making it difficult to determine the
optimal timing and content of refinement based on the evolving generation
context. Inspired by the way humans dynamically refine their thoughts during
execution, we propose ProActive Self-Refinement (PASR), a novel method that
enables LLMs to refine their outputs during the generation process. Unlike
methods that regenerate entire responses, PASR proactively decides whether,
when, and how to refine based on the model's internal state and evolving
context. We conduct extensive experiments on a diverse set of 10 tasks to
evaluate the effectiveness of PASR. Experimental results show that PASR
significantly enhances problem-solving performance. In particular, on Qwen3-8B,
PASR reduces average token consumption by 41.6 percent compared to standard
generation, while also achieving an 8.2 percent improvement in accuracy. Our
code and all baselines used in the paper are available in the GitHub.
Embodied-R1: Reinforced Embodied Reasoning for General Robotic
  Manipulation
Authors: name: Yifu Yuan, name: Haiqin Cui, name: Yaoting Huang, name: Yibin Chen, name: Fei Ni, name: Zibin Dong, name: Pengyi Li, name: Yan Zheng, name: Jianye Hao
Published: 2025-08-19 16:50:01

Generalization in embodied AI is hindered by the "seeing-to-doing gap," which
stems from data scarcity and embodiment heterogeneity. To address this, we
pioneer "pointing" as a unified, embodiment-agnostic intermediate
representation, defining four core embodied pointing abilities that bridge
high-level vision-language comprehension with low-level action primitives. We
introduce Embodied-R1, a 3B Vision-Language Model (VLM) specifically designed
for embodied reasoning and pointing. We use a wide range of embodied and
general visual reasoning datasets as sources to construct a large-scale
dataset, Embodied-Points-200K, which supports key embodied pointing
capabilities. We then train Embodied-R1 using a two-stage Reinforced
Fine-tuning (RFT) curriculum with a specialized multi-task reward design.
Embodied-R1 achieves state-of-the-art performance on 11 embodied spatial and
pointing benchmarks. Critically, it demonstrates robust zero-shot
generalization by achieving a 56.2% success rate in the SIMPLEREnv and 87.5%
across 8 real-world XArm tasks without any task-specific fine-tuning,
representing a 62% improvement over strong baselines. Furthermore, the model
exhibits high robustness against diverse visual disturbances. Our work shows
that a pointing-centric representation, combined with an RFT training paradigm,
offers an effective and generalizable pathway to closing the perception-action
gap in robotics.
Beyond Human Judgment: A Bayesian Evaluation of LLMs' Moral Values
  Understanding
Authors: name: Maciej Skorski, name: Alina Landowska
Published: 2025-08-19 13:05:48

How do large language models understand moral dimensions compared to humans?
  This first large-scale Bayesian evaluation of market-leading language models
provides the answer. In contrast to prior work using deterministic ground truth
(majority or inclusion rules), we model annotator disagreements to capture both
aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty
(model domain sensitivity). We evaluate top language models (Claude Sonnet 4,
DeepSeek-V3, Llama 4 Maverick) across 250K+ annotations from ~700 annotators on
100K+ texts spanning social media, news, and forums.
  Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing
that AI models typically rank among the top 25\% of human annotators, achieving
much better-than-average balanced accuracy. Importantly, we find that AI
produces far fewer false negatives than humans, highlighting their more
sensitive moral detection capabilities.