Orionnnn/hub-stats huggingface dataset:
ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible
  Long-term Tracking
Authors: name: Haofeng Liu, name: Mingqi Gao, name: Xuxiao Luo, name: Ziyue Wang, name: Guanyi Qin, name: Junde Wu, name: Yueming Jin
Published: 2025-05-13 13:56:10

Surgical scene segmentation is critical in computer-assisted surgery and is
vital for enhancing surgical quality and patient outcomes. Recently, referring
surgical segmentation is emerging, given its advantage of providing surgeons
with an interactive experience to segment the target object. However, existing
methods are limited by low efficiency and short-term tracking, hindering their
applicability in complex real-world surgical scenarios. In this paper, we
introduce ReSurgSAM2, a two-stage surgical referring segmentation framework
that leverages Segment Anything Model 2 to perform text-referred target
detection, followed by tracking with reliable initial frame identification and
diversity-driven long-term memory. For the detection stage, we propose a
cross-modal spatial-temporal Mamba to generate precise detection and
segmentation results. Based on these results, our credible initial frame
selection strategy identifies the reliable frame for the subsequent tracking.
Upon selecting the initial frame, our method transitions to the tracking stage,
where it incorporates a diversity-driven memory mechanism that maintains a
credible and diverse memory bank, ensuring consistent long-term tracking.
Extensive experiments demonstrate that ReSurgSAM2 achieves substantial
improvements in accuracy and efficiency compared to existing methods, operating
in real-time at 61.2 FPS. Our code and datasets will be available at
https://github.com/jinlab-imvr/ReSurgSAM2.
EnerVerse-AC: Envisioning Embodied Environments with Action Condition
Authors: name: Yuxin Jiang, name: Shengcong Chen, name: Siyuan Huang, name: Liliang Chen, name: Pengfei Zhou, name: Yue Liao, name: Xindong He, name: Chiming Liu, name: Hongsheng Li, name: Maoqing Yao, name: Guanghui Ren
Published: 2025-05-14 18:30:53

Robotic imitation learning has advanced from solving static tasks to
addressing dynamic interaction scenarios, but testing and evaluation remain
costly and challenging due to the need for real-time interaction with dynamic
environments. We propose EnerVerse-AC (EVAC), an action-conditional world model
that generates future visual observations based on an agent's predicted
actions, enabling realistic and controllable robotic inference. Building on
prior architectures, EVAC introduces a multi-level action-conditioning
mechanism and ray map encoding for dynamic multi-view image generation while
expanding training data with diverse failure trajectories to improve
generalization. As both a data engine and evaluator, EVAC augments
human-collected trajectories into diverse datasets and generates realistic,
action-conditioned video observations for policy testing, eliminating the need
for physical robots or complex simulations. This approach significantly reduces
costs while maintaining high fidelity in robotic manipulation evaluation.
Extensive experiments validate the effectiveness of our method. Code,
checkpoints, and datasets can be found at
<https://annaj2178.github.io/EnerverseAC.github.io>.
EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied
  World Models
Authors: name: Hu Yue, name: Siyuan Huang, name: Yue Liao, name: Shengcong Chen, name: Pengfei Zhou, name: Liliang Chen, name: Maoqing Yao, name: Guanghui Ren
Published: 2025-05-14 18:00:19

Recent advances in creative AI have enabled the synthesis of high-fidelity
images and videos conditioned on language instructions. Building on these
developments, text-to-video diffusion models have evolved into embodied world
models (EWMs) capable of generating physically plausible scenes from language
commands, effectively bridging vision and action in embodied AI applications.
This work addresses the critical challenge of evaluating EWMs beyond general
perceptual metrics to ensure the generation of physically grounded and
action-consistent behaviors. We propose the Embodied World Model Benchmark
(EWMBench), a dedicated framework designed to evaluate EWMs based on three key
aspects: visual scene consistency, motion correctness, and semantic alignment.
Our approach leverages a meticulously curated dataset encompassing diverse
scenes and motion patterns, alongside a comprehensive multi-dimensional
evaluation toolkit, to assess and compare candidate models. The proposed
benchmark not only identifies the limitations of existing video generation
models in meeting the unique requirements of embodied tasks but also provides
valuable insights to guide future advancements in the field. The dataset and
evaluation tools are publicly available at
https://github.com/AgibotTech/EWMBench.
Exploring the Deep Fusion of Large Language Models and Diffusion
  Transformers for Text-to-Image Synthesis
Authors: name: Bingda Tang, name: Boyang Zheng, name: Xichen Pan, name: Sayak Paul, name: Saining Xie
Published: 2025-05-15 07:43:23

This paper does not describe a new method; instead, it provides a thorough
exploration of an important yet understudied design space related to recent
advances in text-to-image synthesis -- specifically, the deep fusion of large
language models (LLMs) and diffusion transformers (DiTs) for multi-modal
generation. Previous studies mainly focused on overall system performance
rather than detailed comparisons with alternative methods, and key design
details and training recipes were often left undisclosed. These gaps create
uncertainty about the real potential of this approach. To fill these gaps, we
conduct an empirical study on text-to-image generation, performing controlled
comparisons with established baselines, analyzing important design choices, and
providing a clear, reproducible recipe for training at scale. We hope this work
offers meaningful data points and practical guidelines for future research in
multi-modal generation.
OpenThinkIMG: Learning to Think with Images via Visual Tool
  Reinforcement Learning
Authors: name: Zhaochen Su, name: Linjie Li, name: Mingyang Song, name: Yunzhuo Hao, name: Zhengyuan Yang, name: Jun Zhang, name: Guanjie Chen, name: Jiawei Gu, name: Juntao Li, name: Xiaoye Qu, name: Yu Cheng
Published: 2025-05-13 14:35:51

While humans can flexibly leverage interactive visual cognition for complex
problem-solving, enabling Large Vision-Language Models (LVLMs) to learn
similarly adaptive behaviors with visual tools remains challenging. A
significant hurdle is the current lack of standardized infrastructure, which
hinders integrating diverse tools, generating rich interaction data, and
training robust agents effectively. To address these gaps, we introduce
OpenThinkIMG, the first open-source, comprehensive end-to-end framework for
tool-augmented LVLMs. It features standardized vision tool interfaces, scalable
trajectory generation for policy initialization, and a flexible training
environment. Furthermore, considering supervised fine-tuning (SFT) on static
demonstrations offers limited policy generalization for dynamic tool
invocation, we propose a novel reinforcement learning (RL) framework V-ToolRL
to train LVLMs to learn adaptive policies for invoking external vision tools.
V-ToolRL enables LVLMs to autonomously discover optimal tool-usage strategies
by directly optimizing for task success using feedback from tool interactions.
We empirically validate V-ToolRL on challenging chart reasoning tasks. Our
RL-trained agent, built upon a Qwen2-VL-2B, significantly outperforms its
SFT-initialized counterpart (+28.83 points) and surpasses established
supervised tool-learning baselines like Taco and CogCom by an average of +12.7
points. Notably, it also surpasses prominent closed-source models like GPT-4.1
by +8.68 accuracy points. We hope OpenThinkIMG can serve as a foundational
framework for advancing dynamic, tool-augmented visual reasoning, helping the
community develop AI agents that can genuinely "think with images".
End-to-End Vision Tokenizer Tuning
Authors: name: Wenxuan Wang, name: Fan Zhang, name: Yufeng Cui, name: Haiwen Diao, name: Zhuoyan Luo, name: Huchuan Lu, name: Jing Liu, name: Xinlong Wang
Published: 2025-05-15 17:59:39

Existing vision tokenization isolates the optimization of vision tokenizers
from downstream training, implicitly assuming the visual tokens can generalize
well across various tasks, e.g., image generation and visual question
answering. The vision tokenizer optimized for low-level reconstruction is
agnostic to downstream tasks requiring varied representations and semantics.
This decoupled paradigm introduces a critical misalignment: The loss of the
vision tokenization can be the representation bottleneck for target tasks. For
example, errors in tokenizing text in a given image lead to poor results when
recognizing or generating them. To address this, we propose ETT, an end-to-end
vision tokenizer tuning approach that enables joint optimization between vision
tokenization and target autoregressive tasks. Unlike prior autoregressive
models that use only discrete indices from a frozen vision tokenizer, ETT
leverages the visual embeddings of the tokenizer codebook, and optimizes the
vision tokenizers end-to-end with both reconstruction and caption objectives.
ETT can be seamlessly integrated into existing training pipelines with minimal
architecture modifications. Our ETT is simple to implement and integrate,
without the need to adjust the original codebooks or architectures of the
employed large language models. Extensive experiments demonstrate that our
proposed end-to-end vision tokenizer tuning unlocks significant performance
gains, i.e., 2-6% for multimodal understanding and visual generation tasks
compared to frozen tokenizer baselines, while preserving the original
reconstruction capability. We hope this very simple and strong method can
empower multimodal foundation models besides image generation and
understanding.
Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large
  Reasoning Models
Authors: name: Zhiyuan Hu, name: Yibo Wang, name: Hanze Dong, name: Yuhui Xu, name: Amrita Saha, name: Caiming Xiong, name: Bryan Hooi, name: Junnan Li
Published: 2025-05-15 17:58:33

Large reasoning models (LRMs) already possess a latent capacity for long
chain-of-thought reasoning. Prior work has shown that outcome-based
reinforcement learning (RL) can incidentally elicit advanced reasoning
behaviors such as self-correction, backtracking, and verification phenomena
often referred to as the model's "aha moment". However, the timing and
consistency of these emergent behaviors remain unpredictable and
uncontrollable, limiting the scalability and reliability of LRMs' reasoning
capabilities. To address these limitations, we move beyond reliance on prompts
and coincidental "aha moments". Instead, we explicitly align models with three
meta-abilities: deduction, induction, and abduction, using automatically
generated, self-verifiable tasks. Our three stage-pipeline individual
alignment, parameter-space merging, and domain-specific reinforcement learning,
boosting performance by over 10\% relative to instruction-tuned baselines.
Furthermore, domain-specific RL from the aligned checkpoint yields an
additional 2\% average gain in the performance ceiling across math, coding, and
science benchmarks, demonstrating that explicit meta-ability alignment offers a
scalable and dependable foundation for reasoning. Code is available at:
https://github.com/zhiyuanhubj/Meta-Ability-Alignment
System Prompt Optimization with Meta-Learning
Authors: name: Yumin Choi, name: Jinheon Baek, name: Sung Ju Hwang
Published: 2025-05-14 16:46:15

Large Language Models (LLMs) have shown remarkable capabilities, with
optimizing their input prompts playing a pivotal role in maximizing their
performance. However, while LLM prompts consist of both the task-agnostic
system prompts and task-specific user prompts, existing work on prompt
optimization has focused on user prompts specific to individual queries or
tasks, and largely overlooked the system prompt that is, once optimized,
applicable across different tasks and domains. Motivated by this, we introduce
the novel problem of bilevel system prompt optimization, whose objective is to
design system prompts that are robust to diverse user prompts and transferable
to unseen tasks. To tackle this problem, we then propose a meta-learning
framework, which meta-learns the system prompt by optimizing it over various
user prompts across multiple datasets, while simultaneously updating the user
prompts in an iterative manner to ensure synergy between them. We conduct
experiments on 14 unseen datasets spanning 5 different domains, on which we
show that our approach produces system prompts that generalize effectively to
diverse user prompts. Also, our findings reveal that the optimized system
prompt enables rapid adaptation even to unseen tasks, requiring fewer
optimization steps for test-time user prompts while achieving improved
performance.
J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
Authors: name: Chenxi Whitehouse, name: Tianlu Wang, name: Ping Yu, name: Xian Li, name: Jason Weston, name: Ilia Kulikov, name: Swarnadeep Saha
Published: 2025-05-15 14:05:15

The progress of AI is bottlenecked by the quality of evaluation, and powerful
LLM-as-a-Judge models have proved to be a core solution. Improved judgment
ability is enabled by stronger chain-of-thought reasoning, motivating the need
to find the best recipes for training such models to think. In this work we
introduce J1, a reinforcement learning approach to training such models. Our
method converts both verifiable and non-verifiable prompts to judgment tasks
with verifiable rewards that incentivize thinking and mitigate judgment bias.
In particular, our approach outperforms all other existing 8B or 70B models
when trained at those sizes, including models distilled from DeepSeek-R1. J1
also outperforms o1-mini, and even R1 on some benchmarks, despite training a
smaller model. We provide analysis and ablations comparing Pairwise-J1 vs
Pointwise-J1 models, offline vs online training recipes, reward strategies,
seed prompts, and variations in thought length and content. We find that our
models make better judgments by learning to outline evaluation criteria,
comparing against self-generated reference answers, and re-evaluating the
correctness of model responses.
The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a
  Reasoning Model will Think
Authors: name: Seongyun Lee, name: Seungone Kim, name: Minju Seo, name: Yongrae Jo, name: Dongyoung Go, name: Hyeonbin Hwang, name: Jinho Park, name: Xiang Yue, name: Sean Welleck, name: Graham Neubig, name: Moontae Lee, name: Minjoon Seo
Published: 2025-05-15 11:31:02

Long chain-of-thought (CoT) is an essential ingredient in effective usage of
modern large language models, but our understanding of the reasoning strategies
underlying these capabilities remains limited. While some prior works have
attempted to categorize CoTs using predefined strategy types, such approaches
are constrained by human intuition and fail to capture the full diversity of
model behaviors. In this work, we introduce the CoT Encyclopedia, a bottom-up
framework for analyzing and steering model reasoning. Our method automatically
extracts diverse reasoning criteria from model-generated CoTs, embeds them into
a semantic space, clusters them into representative categories, and derives
contrastive rubrics to interpret reasoning behavior. Human evaluations show
that this framework produces more interpretable and comprehensive analyses than
existing methods. Moreover, we demonstrate that this understanding enables
performance gains: we can predict which strategy a model is likely to use and
guide it toward more effective alternatives. Finally, we provide practical
insights, such as that training data format (e.g., free-form vs.
multiple-choice) has a far greater impact on reasoning behavior than data
domain, underscoring the importance of format-aware model design.
MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine
  Learning Engineering
Authors: name: Rushi Qiang, name: Yuchen Zhuang, name: Yinghao Li, name: Dingu Sagar V K, name: Rongzhi Zhang, name: Changhao Li, name: Ian Shu-Hei Wong, name: Sherry Yang, name: Percy Liang, name: Chao Zhang, name: Bo Dai
Published: 2025-05-12 17:35:43

We introduce MLE-Dojo, a Gym-style framework for systematically reinforcement
learning, evaluating, and improving autonomous large language model (LLM)
agents in iterative machine learning engineering (MLE) workflows. Unlike
existing benchmarks that primarily rely on static datasets or single-attempt
evaluations, MLE-Dojo provides an interactive environment enabling agents to
iteratively experiment, debug, and refine solutions through structured feedback
loops. Built upon 200+ real-world Kaggle challenges, MLE-Dojo covers diverse,
open-ended MLE tasks carefully curated to reflect realistic engineering
scenarios such as data processing, architecture search, hyperparameter tuning,
and code debugging. Its fully executable environment supports comprehensive
agent training via both supervised fine-tuning and reinforcement learning,
facilitating iterative experimentation, realistic data sampling, and real-time
outcome verification. Extensive evaluations of eight frontier LLMs reveal that
while current models achieve meaningful iterative improvements, they still
exhibit significant limitations in autonomously generating long-horizon
solutions and efficiently resolving complex errors. Furthermore, MLE-Dojo's
flexible and extensible architecture seamlessly integrates diverse data
sources, tools, and evaluation protocols, uniquely enabling model-based agent
tuning and promoting interoperability, scalability, and reproducibility. We
open-source our framework and benchmarks to foster community-driven innovation
towards next-generation MLE agents.
WorldPM: Scaling Human Preference Modeling
Authors: name: Binghai Wang, name: Runji Lin, name: Keming Lu, name: Le Yu, name: Zhenru Zhang, name: Fei Huang, name: Chujie Zheng, name: Kai Dang, name: Yang Fan, name: Xingzhang Ren, name: An Yang, name: Binyuan Hui, name: Dayiheng Liu, name: Tao Gui, name: Qi Zhang, name: Xuanjing Huang, name: Yu-Gang Jiang, name: Bowen Yu, name: Jingren Zhou, name: Junyang Lin
Published: 2025-05-15 17:38:37

Motivated by scaling laws in language modeling that demonstrate how test loss
scales as a power law with model and dataset sizes, we find that similar laws
exist in preference modeling. We propose World Preference Modeling$ (WorldPM)
to emphasize this scaling potential, where World Preference embodies a unified
representation of human preferences. In this paper, we collect preference data
from public forums covering diverse user communities, and conduct extensive
training using 15M-scale data across models ranging from 1.5B to 72B
parameters. We observe distinct patterns across different evaluation metrics:
(1) Adversarial metrics (ability to identify deceptive features) consistently
scale up with increased training data and base model size; (2) Objective
metrics (objective knowledge with well-defined answers) show emergent behavior
in larger language models, highlighting WorldPM's scalability potential; (3)
Subjective metrics (subjective preferences from a limited number of humans or
AI) do not demonstrate scaling trends. Further experiments validate the
effectiveness of WorldPM as a foundation for preference fine-tuning. Through
evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly
improves the generalization performance across human preference datasets of
varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5%
on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we
observe significant improvements on both in-house and public evaluation sets,
with notable gains of 4% to 8% in our in-house evaluations.
Few-Shot Anomaly-Driven Generation for Anomaly Classification and
  Segmentation
Authors: name: Guan Gui, name: Bin-Bin Gao, name: Jun Liu, name: Chengjie Wang, name: Yunsheng Wu
Published: 2025-05-14 10:25:06

Anomaly detection is a practical and challenging task due to the scarcity of
anomaly samples in industrial inspection. Some existing anomaly detection
methods address this issue by synthesizing anomalies with noise or external
data. However, there is always a large semantic gap between synthetic and
real-world anomalies, resulting in weak performance in anomaly detection. To
solve the problem, we propose a few-shot Anomaly-driven Generation (AnoGen)
method, which guides the diffusion model to generate realistic and diverse
anomalies with only a few real anomalies, thereby benefiting training anomaly
detection models. Specifically, our work is divided into three stages. In the
first stage, we learn the anomaly distribution based on a few given real
anomalies and inject the learned knowledge into an embedding. In the second
stage, we use the embedding and given bounding boxes to guide the diffusion
model to generate realistic and diverse anomalies on specific objects (or
textures). In the final stage, we propose a weakly-supervised anomaly detection
method to train a more powerful model with generated anomalies. Our method
builds upon DRAEM and DesTSeg as the foundation model and conducts experiments
on the commonly used industrial anomaly detection dataset, MVTec. The
experiments demonstrate that our generated anomalies effectively improve the
model performance of both anomaly classification and segmentation tasks
simultaneously, \eg, DRAEM and DseTSeg achieved a 5.8\% and 1.5\% improvement
in AU-PR metric on segmentation task, respectively. The code and generated
anomalous data are available at https://github.com/gaobb/AnoGen.
Learning to Detect Multi-class Anomalies with Just One Normal Image
  Prompt
Authors: name: Bin-Bin Gao
Published: 2025-05-14 10:25:14

Unsupervised reconstruction networks using self-attention transformers have
achieved state-of-the-art performance for multi-class (unified) anomaly
detection with a single model. However, these self-attention reconstruction
models primarily operate on target features, which may result in perfect
reconstruction for both normal and anomaly features due to high consistency
with context, leading to failure in detecting anomalies. Additionally, these
models often produce inaccurate anomaly segmentation due to performing
reconstruction in a low spatial resolution latent space. To enable
reconstruction models enjoying high efficiency while enhancing their
generalization for unified anomaly detection, we propose a simple yet effective
method that reconstructs normal features and restores anomaly features with
just One Normal Image Prompt (OneNIP). In contrast to previous work, OneNIP
allows for the first time to reconstruct or restore anomalies with just one
normal image prompt, effectively boosting unified anomaly detection
performance. Furthermore, we propose a supervised refiner that regresses
reconstruction errors by using both real normal and synthesized anomalous
images, which significantly improves pixel-level anomaly segmentation. OneNIP
outperforms previous methods on three industry anomaly detection benchmarks:
MVTec, BTAD, and VisA. The code and pre-trained models are available at
https://github.com/gaobb/OneNIP.