Orionnnn/hub-stats huggingface dataset:
Trove: A Flexible Toolkit for Dense Retrieval
Authors: name: Reza Esfandiarpoor, name: Max Zuo, name: Stephen H. Bach
Published: 2025-11-03 18:59:57

We introduce Trove, an easy-to-use open-source retrieval toolkit that
simplifies research experiments without sacrificing flexibility or speed. For
the first time, we introduce efficient data management features that load and
process (filter, select, transform, and combine) retrieval datasets on the fly,
with just a few lines of code. This gives users the flexibility to easily
experiment with different dataset configurations without the need to compute
and store multiple copies of large datasets. Trove is highly customizable: in
addition to many built-in options, it allows users to freely modify existing
components or replace them entirely with user-defined objects. It also provides
a low-code and unified pipeline for evaluation and hard negative mining, which
supports multi-node execution without any code changes. Trove's data management
features reduce memory consumption by a factor of 2.6. Moreover, Trove's
easy-to-use inference pipeline incurs no overhead, and inference times decrease
linearly with the number of available nodes. Most importantly, we demonstrate
how Trove simplifies retrieval experiments and allows for arbitrary
customizations, thus facilitating exploratory research.
The Underappreciated Power of Vision Models for Graph Structural
  Understanding
Authors: name: Xinjian Zhao, name: Wei Pang, name: Zhongkai Xue, name: Xiangru Jian, name: Lei Zhang, name: Yaoyao Xu, name: Xiaozhuang Song, name: Shu Wu, name: Tianshu Yu
Published: 2025-10-27 05:11:44

Graph Neural Networks operate through bottom-up message-passing,
fundamentally differing from human visual perception, which intuitively
captures global structures first. We investigate the underappreciated potential
of vision models for graph understanding, finding they achieve performance
comparable to GNNs on established benchmarks while exhibiting distinctly
different learning patterns. These divergent behaviors, combined with
limitations of existing benchmarks that conflate domain features with
topological understanding, motivate our introduction of GraphAbstract. This
benchmark evaluates models' ability to perceive global graph properties as
humans do: recognizing organizational archetypes, detecting symmetry, sensing
connectivity strength, and identifying critical elements. Our results reveal
that vision models significantly outperform GNNs on tasks requiring holistic
structural understanding and maintain generalizability across varying graph
scales, while GNNs struggle with global pattern abstraction and degrade with
increasing graph size. This work demonstrates that vision models possess
remarkable yet underutilized capabilities for graph structural understanding,
particularly for problems requiring global topological awareness and
scale-invariant reasoning. These findings open new avenues to leverage this
underappreciated potential for developing more effective graph foundation
models for tasks dominated by holistic pattern recognition.
How Far Are Surgeons from Surgical World Models? A Pilot Study on
  Zero-shot Surgical Video Generation with Expert Assessment
Authors: name: Zhen Chen, name: Qing Xu, name: Jinlin Wu, name: Biao Yang, name: Yuhao Zhai, name: Geng Guo, name: Jing Zhang, name: Yinlu Ding, name: Nassir Navab, name: Jiebo Luo
Published: 2025-11-03 17:28:54

Foundation models in video generation are demonstrating remarkable
capabilities as potential world models for simulating the physical world.
However, their application in high-stakes domains like surgery, which demand
deep, specialized causal knowledge rather than general physical rules, remains
a critical unexplored gap. To systematically address this challenge, we present
SurgVeo, the first expert-curated benchmark for video generation model
evaluation in surgery, and the Surgical Plausibility Pyramid (SPP), a novel,
four-tiered framework tailored to assess model outputs from basic appearance to
complex surgical strategy. On the basis of the SurgVeo benchmark, we task the
advanced Veo-3 model with a zero-shot prediction task on surgical clips from
laparoscopic and neurosurgical procedures. A panel of four board-certified
surgeons evaluates the generated videos according to the SPP. Our results
reveal a distinct "plausibility gap": while Veo-3 achieves exceptional Visual
Perceptual Plausibility, it fails critically at higher levels of the SPP,
including Instrument Operation Plausibility, Environment Feedback Plausibility,
and Surgical Intent Plausibility. This work provides the first quantitative
evidence of the chasm between visually convincing mimicry and causal
understanding in surgical AI. Our findings from SurgVeo and the SPP establish a
crucial foundation and roadmap for developing future models capable of
navigating the complexities of specialized, real-world healthcare domains.
UniREditBench: A Unified Reasoning-based Image Editing Benchmark
Authors: name: Feng Han, name: Yibin Wang, name: Chenglin Li, name: Zheming Liang, name: Dianyi Wang, name: Yang Jiao, name: Zhipeng Wei, name: Chao Gong, name: Cheng Jin, name: Jingjing Chen, name: Jiaqi Wang
Published: 2025-11-03 07:24:57

Recent advances in multi-modal generative models have driven substantial
improvements in image editing. However, current generative models still
struggle with handling diverse and complex image editing tasks that require
implicit reasoning, underscoring the need for a comprehensive benchmark to
systematically assess their performance across various reasoning scenarios.
Existing benchmarks primarily focus on single-object attribute transformation
in realistic scenarios, which, while effective, encounter two key challenges:
(1) they largely overlook multi-object interactions as well as game-world
scenarios that involve human-defined rules, which are common in real-life
applications; (2) they only rely on textual references to evaluate the
generated images, potentially leading to systematic misjudgments, especially in
complex reasoning scenarios. To this end, this work proposes UniREditBench, a
unified benchmark for reasoning-based image editing evaluation. It comprises
2,700 meticulously curated samples, covering both real- and game-world
scenarios across 8 primary dimensions and 18 sub-dimensions. To improve
evaluation reliability, we introduce multimodal dual-reference evaluation,
providing both textual and ground-truth image references for each sample
assessment. Furthermore, we design an automated multi-scenario data synthesis
pipeline and construct UniREdit-Data-100K, a large-scale synthetic dataset with
high-quality chain-of-thought (CoT) reasoning annotations. We fine-tune Bagel
on this dataset and develop UniREdit-Bagel, demonstrating substantial
improvements in both in-domain and out-of-distribution settings. Through
thorough benchmarking of both open-source and closed-source image editing
models, we reveal their strengths and weaknesses across various aspects.
Towards Robust Mathematical Reasoning
Authors: name: Thang Luong, name: Dawsen Hwang, name: Hoang H. Nguyen, name: Golnaz Ghiasi, name: Yuri Chervonyi, name: Insuk Seo, name: Junsu Kim, name: Garrett Bingham, name: Jonathan Lee, name: Swaroop Mishra, name: Alex Zhai, name: Clara Huiyi Hu, name: Henryk Michalewski, name: Jimin Kim, name: Jeonghyun Ahn, name: Junhwi Bae, name: Xingyou Song, name: Trieu H. Trinh, name: Quoc V. Le, name: Junehyuk Jung
Published: 2025-11-03 18:53:02

Finding the right north-star metrics is highly critical for advancing the
mathematical reasoning capabilities of foundation models, especially given that
existing evaluations are either too easy or only focus on getting correct short
answers. To address these issues, we present IMO-Bench, a suite of advanced
reasoning benchmarks, vetted by a panel of top specialists and that
specifically targets the level of the International Mathematical Olympiad
(IMO), the most prestigious venue for young mathematicians. IMO-AnswerBench
first tests models on 400 diverse Olympiad problems with verifiable short
answers. IMO-Proof Bench is the next-level evaluation for proof-writing
capabilities, which includes both basic and advanced IMO level problems as well
as detailed grading guidelines to facilitate automatic grading. These
benchmarks played a crucial role in our historic achievement of the gold-level
performance at IMO 2025 with Gemini Deep Think (Luong and Lockhart, 2025). Our
model achieved 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-Proof
Bench, surpassing the best non-Gemini models by large margins of 6.9% and 42.4%
respectively. We also showed that autograders built with Gemini reasoning
correlate well with human evaluations and construct IMO-GradingBench, with 1000
human gradings on proofs, to enable further progress in automatic evaluation of
long-form answers. We hope that IMO-Bench will help the community towards
advancing robust mathematical reasoning and release it at
https://imobench.github.io/.
TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images
  Reasoning
Authors: name: Ming Li, name: Jike Zhong, name: Shitian Zhao, name: Haoquan Zhang, name: Shaoheng Lin, name: Yuxiang Lai, name: Wei Chen, name: Konstantinos Psounis, name: Kaipeng Zhang
Published: 2025-11-03 18:40:17

The frontier of visual reasoning is shifting toward models like OpenAI o3,
which can intelligently create and operate tools to transform images for
problem-solving, also known as thinking-with-images in
chain-of-thought. Yet existing benchmarks fail to fully capture this advanced
capability. Even Visual Search, the most common benchmark for current
thinking-with-images methods, tests only basic operations such as
localization and cropping, offering little insight into more complex, dynamic,
and tool-dependent reasoning. We introduce TIR-Bench, a comprehensive
benchmark for evaluating agentic thinking-with-images across 13 diverse tasks,
each requiring novel tool use for image processing and manipulation in
chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from
leading open-sourced and proprietary models to those with explicit tool-use
augmentation. Results show that TIR-Bench is universally challenging, and
strong performance requires genuine thinking-with-images capabilities. Finally,
we present a pilot study comparing direct versus agentic fine-tuning.
Towards Universal Video Retrieval: Generalizing Video Embedding via
  Synthesized Multimodal Pyramid Curriculum
Authors: name: Zhuoning Guo, name: Mingxin Li, name: Yanzhao Zhang, name: Dingkun Long, name: Pengjun Xie, name: Xiaowen Chu
Published: 2025-10-31 15:54:48

The prevailing video retrieval paradigm is structurally misaligned, as narrow
benchmarks incentivize correspondingly limited data and single-task training.
Therefore, universal capability is suppressed due to the absence of a
diagnostic evaluation that defines and demands multi-dimensional
generalization. To break this cycle, we introduce a framework built on the
co-design of evaluation, data, and modeling. First, we establish the Universal
Video Retrieval Benchmark (UVRB), a suite of 16 datasets designed not only to
measure performance but also to diagnose critical capability gaps across tasks
and domains. Second, guided by UVRB's diagnostics, we introduce a scalable
synthesis workflow that generates 1.55 million high-quality pairs to populate
the semantic space required for universality. Finally, we devise the Modality
Pyramid, a curriculum that trains our General Video Embedder (GVE) by
explicitly leveraging the latent interconnections within our diverse data.
Extensive experiments show GVE achieves state-of-the-art zero-shot
generalization on UVRB. In particular, our analysis reveals that popular
benchmarks are poor predictors of general ability and that partially relevant
retrieval is a dominant but overlooked scenario. Overall, our co-designed
framework provides a practical path to escape the limited scope and advance
toward truly universal video retrieval.
Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete
  Denoising Diffusion Process
Authors: name: Jiayi Chen, name: Wenxuan Song, name: Pengxiang Ding, name: Ziyang Zhou, name: Han Zhao, name: Feilong Tang, name: Donglin Wang, name: Haoang Li
Published: 2025-11-03 16:26:54

Vision-language-action (VLA) models aim to understand natural language
instructions and visual observations and to execute corresponding actions as an
embodied agent. Recent work integrates future images into the
understanding-acting loop, yielding unified VLAs that jointly understand,
generate, and act -- reading text and images and producing future images and
actions. However, these models either rely on external experts for modality
unification or treat image generation and action prediction as separate
processes, limiting the benefits of direct synergy between these tasks. Our
core philosophy is to optimize generation and action jointly through a
synchronous denoising process, where the iterative refinement enables actions
to evolve from initialization, under constant and sufficient visual guidance.
We ground this philosophy in our proposed Unified Diffusion VLA and Joint
Discrete Denoising Diffusion Process (JD3P), which is a joint diffusion process
that integrates multiple modalities into a single denoising trajectory to serve
as the key mechanism enabling understanding, generation, and acting to be
intrinsically synergistic. Our model and theory are built on a unified
tokenized space of all modalities and a hybrid attention mechanism. We further
propose a two-stage training pipeline and several inference-time techniques
that optimize performance and efficiency. Our approach achieves
state-of-the-art performance on benchmarks such as CALVIN, LIBERO, and
SimplerEnv with 4times faster inference than autoregressive methods, and we
demonstrate its effectiveness through in-depth analysis and real-world
evaluations. Our project page is available at
https://irpn-eai.github.io/UD-VLA.github.io/.
Generalizing Test-time Compute-optimal Scaling as an Optimizable Graph
Authors: name: Fali Wang, name: Jihai Chen, name: Shuhua Yang, name: Runxue Bao, name: Tianxiang Zhao, name: Zhiwei Zhang, name: Xianfeng Tang, name: Hui Liu, name: Qi He, name: Suhang Wang
Published: 2025-10-29 22:14:25

Test-Time Scaling (TTS) improves large language models (LLMs) by allocating
additional computation during inference, typically through parallel,
sequential, or hybrid scaling. However, prior studies often assume fixed
collaboration architectures (e.g., topologies) and single-model usage,
overlooking that optimal architectures and model combinations can vary across
tasks. Therefore, we study the novel problem of searching for compute-optimal
model combinations and architectures in TTS under a fixed budget. We formalize
it as a multi-LLM collaboration graph, where nodes encode roles and LLM model
assignments, and edges capture information flow. This problem is challenging
because (i) the combinatorial search space is prohibitively large, and (ii)
task-specific requirements demand tailored designs. To address these, we
reformulate the problem as probabilistic graph optimization and, through pilot
experiments, derive three empirical insights into TTS collaboration graphs.
Guided by these insights, we propose Agent-REINFORCE, an LLM-agent-augmented
framework that mirrors the REINFORCE pipeline by mapping
sampling-gradient-update to sampling-feedback-update, where feedback serves as
a textual gradient to update the probabilistic graph and efficiently search for
optimal multi-LLM collaboration graphs. Experiments show that Agent-REINFORCE
outperforms both traditional and LLM-based baselines in sample efficiency and
search performance, and effectively identifies optimal graphs under joint
objectives of accuracy and inference latency.
MisSynth: Improving MISSCI Logical Fallacies Classification with
  Synthetic Data
Authors: name: Mykhailo Poliakov, name: Nadiya Shvai
Published: 2025-10-30 10:52:43

Health-related misinformation is very prevalent and potentially harmful. It
is difficult to identify, especially when claims distort or misinterpret
scientific findings. We investigate the impact of synthetic data generation and
lightweight fine-tuning techniques on the ability of large language models
(LLMs) to recognize fallacious arguments using the MISSCI dataset and
framework. In this work, we propose MisSynth, a pipeline that applies
retrieval-augmented generation (RAG) to produce synthetic fallacy samples,
which are then used to fine-tune an LLM model. Our results show substantial
accuracy gains with fine-tuned models compared to vanilla baselines. For
instance, the LLaMA 3.1 8B fine-tuned model achieved an over 35% F1-score
absolute improvement on the MISSCI test split over its vanilla baseline. We
demonstrate that introducing synthetic fallacy data to augment limited
annotated resources can significantly enhance zero-shot LLM classification
performance on real-world scientific misinformation tasks, even with limited
computational resources. The code and synthetic dataset are available on
https://github.com/mxpoliakov/MisSynth.
Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained
  Classification
Authors: name: William Yang, name: Xindi Wu, name: Zhiwei Deng, name: Esin Tureci, name: Olga Russakovsky
Published: 2025-10-28 05:40:14

Text-to-image (T2I) models are increasingly used for synthetic dataset
generation, but generating effective synthetic training data for classification
remains challenging. Fine-tuning a T2I model with a few real examples can help
improve the quality of synthetic training data; however, it may also cause
overfitting and reduce diversity in the generated samples. We propose a
fine-tuning strategy BOB (BeyondOBjects) to mitigate these concerns for
fine-grained classification. Given a small set of real examples, we first
extract class-agnostic attributes such as scene background and object pose. We
then explicitly condition on these attributes during fine-tuning of the T2I
model and marginalize them out during generation. This design mitigates
overfitting, preserves the T2I model's generative prior, reduces estimation
errors, and further minimizes unintended inter-class associations. Extensive
experiments across multiple T2I models, backbones, and datasets show that our
method achieves state-of-the-art performance in low-shot fine-grained
classification when augmented with synthetic data. Concretely, BOB outperforms
DataDream by 7.4% on the Aircraft dataset (from 50.0% to 57.4% when fine-tuning
a CLIP classifier with five real images augmented with 100 synthetic images).
In three of the four benchmarks, fine-tuning downstream models with 5 real
images augmented with BOB achieves better performance than fine-tuning with 10
real images. Collectively, BOB outperforms prior art in 18 of 24 experimental
settings, with 2+% accuracy improvements in 14 of these settings.
SemCoT: Accelerating Chain-of-Thought Reasoning through
  Semantically-Aligned Implicit Tokens
Authors: name: Yinhan He, name: Wendy Zheng, name: Yaochen Zhu, name: Zaiyi Zheng, name: Lin Su, name: Sriram Vasudevan, name: Qi Guo, name: Liangjie Hong, name: Jundong Li
Published: 2025-10-28 20:11:54

The verbosity of Chain-of-Thought (CoT) reasoning hinders its mass deployment
in efficiency-critical applications. Recently, implicit CoT approaches have
emerged, which encode reasoning steps within LLM's hidden embeddings (termed
``implicit reasoning'') rather than explicit tokens. This approach accelerates
CoT by reducing the reasoning length and bypassing some LLM components.
However, existing implicit CoT methods face two significant challenges: (1)
they fail to preserve the semantic alignment between the implicit reasoning
(when transformed to natural language) and the ground-truth reasoning,
resulting in a significant CoT performance degradation, and (2) they focus on
reducing the length of the implicit reasoning; however, they neglect the
considerable time cost for an LLM to generate one individual implicit reasoning
token. To tackle these challenges, we propose a novel semantically-aligned
implicit CoT framework termed SemCoT. In particular, for the first challenge,
we design a contrastively trained sentence transformer that evaluates semantic
alignment between implicit and explicit reasoning, which is used to enforce
semantic preservation during implicit reasoning optimization. To address the
second challenge, we introduce an efficient implicit reasoning generator by
finetuning a lightweight language model using knowledge distillation. This
generator is guided by our sentence transformer to distill ground-truth
reasoning into semantically aligned implicit reasoning, while also optimizing
for accuracy. SemCoT is the first approach that enhances CoT efficiency by
jointly optimizing token-level generation speed and preserving semantic
alignment with ground-truth reasoning. Extensive experiments demonstrate the
superior performance of SemCoT compared to state-of-the-art methods in both
efficiency and effectiveness. Our code can be found at
https://github.com/YinhanHe123/SemCoT/.
Monopoly Deal: A Benchmark Environment for Bounded One-Sided Response
  Games
Authors: name: Will Wolf
Published: 2025-10-29 01:38:19

Card games are widely used to study sequential decision-making under
uncertainty, with real-world analogues in negotiation, finance, and
cybersecurity. These games typically fall into three categories based on the
flow of control: strictly sequential (players alternate single actions),
deterministic response (some actions trigger a fixed outcome), and unbounded
reciprocal response (alternating counterplays are permitted). A less-explored
but strategically rich structure is the bounded one-sided response, where a
player's action briefly transfers control to the opponent, who must satisfy a
fixed condition through one or more moves before the turn resolves. We term
games featuring this mechanism Bounded One-Sided Response Games (BORGs). We
introduce a modified version of Monopoly Deal as a benchmark environment that
isolates this dynamic, where a Rent action forces the opponent to choose
payment assets. The gold-standard algorithm, Counterfactual Regret Minimization
(CFR), converges on effective strategies without novel algorithmic extensions.
A lightweight full-stack research platform unifies the environment, a
parallelized CFR runtime, and a human-playable web interface. The trained CFR
agent and source code are available at https://monopolydeal.ai.
Rank-GRPO: Training LLM-based Conversational Recommender Systems with
  Reinforcement Learning
Authors: name: Yaochen Zhu, name: Harald Steck, name: Dawen Liang, name: Yinhan He, name: Jundong Li, name: Nathan Kallus
Published: 2025-10-23 02:56:00

Large language models (LLMs) are reshaping the recommender system paradigm by
enabling users to express preferences and receive recommendations through
conversations. Yet, aligning LLMs to the recommendation task remains
challenging: pretrained LLMs often generate out-of-catalog items, violate
required output formats, and their ranking quality degrades sharply toward the
end of the generated list. To this end, we propose ConvRec-R1, a two-stage
framework for end-to-end training of LLM-based conversational recommender
systems. In Stage 1, we construct a behavioral-cloning dataset with a
Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded
demonstrations from powerful blackbox LLMs to warm-start the RL training. In
Stage 2, we propose Rank-GRPO, a principled extension of group relative policy
optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats
each rank in the recommendation list as the unit instead of token (too
fine-grained) or sequence (too coarse), redefining rewards to remove non-causal
credit assignment and introducing a rank-level importance ratio based on the
geometric mean of rank-wise token probabilities to stabilize policy updates.
Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges
faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and
datasets are released at https://github.com/yaochenzhu/Rank-GRPO.