Orionnnn/hub-stats huggingface dataset:
UI-R1: Enhancing Action Prediction of GUI Agents by Reinforcement
  Learning
Authors: name: Zhengxi Lu, name: Yuxiang Chai, name: Yaxuan Guo, name: Xi Yin, name: Liang Liu, name: Hao Wang, name: Guanjing Xiong, name: Hongsheng Li
Published: 2025-03-27 15:39:30

The recent DeepSeek-R1 has showcased the emergence of reasoning capabilities
in LLMs through reinforcement learning (RL) with rule-based rewards. Building
on this idea, we are the first to explore how rule-based RL can enhance the
reasoning capabilities of multimodal large language models (MLLMs) for graphic
user interface (GUI) action prediction tasks. To this end, we curate a small
yet high-quality dataset of 136 challenging tasks, encompassing five common
action types on mobile devices. We also introduce a unified rule-based action
reward, enabling model optimization via policy-based algorithms such as Group
Relative Policy Optimization (GRPO). Experimental results demonstrate that our
proposed data-efficient model, UI-R1-3B, achieves substantial improvements on
both in-domain (ID) and out-of-domain (OOD) tasks. Specifically, on the ID
benchmark AndroidControl, the action type accuracy improves by 15%, while
grounding accuracy increases by 10.3%, compared with the base model (i.e.
Qwen2.5-VL-3B). On the OOD GUI grounding benchmark ScreenSpot-Pro, our model
surpasses the base model by 6.0% and achieves competitive performance with
larger models (e.g., OS-Atlas-7B), which are trained via supervised fine-tuning
(SFT) on 76K data. These results underscore the potential of rule-based
reinforcement learning to advance GUI understanding and control, paving the way
for future research in this domain.
ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging
Authors: name: Haoming Xu, name: Shuxun Wang, name: Yanqiu Zhao, name: Yi Zhong, name: Ziyan Jiang, name: Ningyuan Zhao, name: Shumin Deng, name: Huajun Chen, name: Ningyu Zhang
Published: 2025-03-27 02:03:25

This paper presents the ZJUKLAB team's submission for SemEval-2025 Task 4:
Unlearning Sensitive Content from Large Language Models. This task aims to
selectively erase sensitive knowledge from large language models, avoiding both
over-forgetting and under-forgetting issues. We propose an unlearning system
that leverages Model Merging (specifically TIES-Merging), combining two
specialized models into a more balanced unlearned model. Our system achieves
competitive results, ranking second among 26 teams, with an online score of
0.944 for Task Aggregate and 0.487 for overall Aggregate. In this paper, we
also conduct local experiments and perform a comprehensive analysis of the
unlearning process, examining performance trajectories, loss dynamics, and
weight perspectives, along with several supplementary experiments, to
understand the effectiveness of our method. Furthermore, we analyze the
shortcomings of our method and evaluation metrics, emphasizing that MIA scores
and ROUGE-based metrics alone are insufficient to fully evaluate successful
unlearning. Finally, we emphasize the need for more comprehensive evaluation
methodologies and rethinking of unlearning objectives in future research. Code
is available at https://github.com/zjunlp/unlearn/tree/main/semeval25.
Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile
  Gaussian Feature Fields
Authors: name: Shijie Zhou, name: Hui Ren, name: Yijia Weng, name: Shuwang Zhang, name: Zhen Wang, name: Dejia Xu, name: Zhiwen Fan, name: Suya You, name: Zhangyang Wang, name: Leonidas Guibas, name: Achuta Kadambi
Published: 2025-03-26 17:56:16

Recent advancements in 2D and multimodal models have achieved remarkable
success by leveraging large-scale training on extensive datasets. However,
extending these achievements to enable free-form interactions and high-level
semantic operations with complex 3D/4D scenes remains challenging. This
difficulty stems from the limited availability of large-scale, annotated 3D/4D
or multi-view datasets, which are crucial for generalizable vision and language
tasks such as open-vocabulary and prompt-based segmentation, language-guided
editing, and visual question answering (VQA). In this paper, we introduce
Feature4X, a universal framework designed to extend any functionality from 2D
vision foundation model into the 4D realm, using only monocular video input,
which is widely available from user-generated content. The "X" in Feature4X
represents its versatility, enabling any task through adaptable,
model-conditioned 4D feature field distillation. At the core of our framework
is a dynamic optimization strategy that unifies multiple model capabilities
into a single representation. Additionally, to the best of our knowledge,
Feature4X is the first method to distill and lift the features of video
foundation models (e.g. SAM2, InternVideo2) into an explicit 4D feature field
using Gaussian Splatting. Our experiments showcase novel view segment anything,
geometric and appearance scene editing, and free-form VQA across all time
steps, empowered by LLMs in feedback loops. These advancements broaden the
scope of agentic AI applications by providing a foundation for scalable,
contextually and spatiotemporally aware systems capable of immersive dynamic 4D
scene interaction.
Unified Multimodal Discrete Diffusion
Authors: name: Alexander Swerdlow, name: Mihir Prabhudesai, name: Siddharth Gandhi, name: Deepak Pathak, name: Katerina Fragkiadaki
Published: 2025-03-26 17:59:51

Multimodal generative models that can understand and generate across multiple
modalities are dominated by autoregressive (AR) approaches, which process
tokens sequentially from left to right, or top to bottom. These models jointly
handle images, text, video, and audio for various tasks such as image
captioning, question answering, and image generation. In this work, we explore
discrete diffusion models as a unified generative formulation in the joint text
and image domain, building upon their recent success in text generation.
Discrete diffusion models offer several advantages over AR models, including
improved control over quality versus diversity of generated samples, the
ability to perform joint multimodal inpainting (across both text and image
domains), and greater controllability in generation through guidance.
Leveraging these benefits, we present the first Unified Multimodal Discrete
Diffusion (UniDisc) model which is capable of jointly understanding and
generating text and images for a variety of downstream tasks. We compare
UniDisc to multimodal AR models, performing a scaling analysis and
demonstrating that UniDisc outperforms them in terms of both performance and
inference-time compute, enhanced controllability, editability, inpainting, and
flexible trade-off between inference time and generation quality. Code and
additional visualizations are available at https://unidisc.github.io.
LLPut: Investigating Large Language Models for Bug Report-Based Input
  Generation
Authors: name: Alif Al Hasan, name: Subarna Saha, name: Mia Mohammad Imran, name: Tarannum Shaila Zaman
Published: 2025-03-26 14:25:01

Failure-inducing inputs play a crucial role in diagnosing and analyzing
software bugs. Bug reports typically contain these inputs, which developers
extract to facilitate debugging. Since bug reports are written in natural
language, prior research has leveraged various Natural Language Processing
(NLP) techniques for automated input extraction. With the advent of Large
Language Models (LLMs), an important research question arises: how effectively
can generative LLMs extract failure-inducing inputs from bug reports? In this
paper, we propose LLPut, a technique to empirically evaluate the performance of
three open-source generative LLMs -- LLaMA, Qwen, and Qwen-Coder -- in
extracting relevant inputs from bug reports. We conduct an experimental
evaluation on a dataset of 206 bug reports to assess the accuracy and
effectiveness of these models. Our findings provide insights into the
capabilities and limitations of generative LLMs in automated bug diagnosis.
RecTable: Fast Modeling Tabular Data with Rectified Flow
Authors: name: Masane Fuchi, name: Tomohiro Takagi
Published: 2025-03-26 17:12:20

Score-based or diffusion models generate high-quality tabular data,
surpassing GAN-based and VAE-based models. However, these methods require
substantial training time. In this paper, we introduce RecTable, which uses the
rectified flow modeling, applied in such as text-to-image generation and
text-to-video generation. RecTable features a simple architecture consisting of
a few stacked gated linear unit blocks. Additionally, our training strategies
are also simple, incorporating a mixed-type noise distribution and a
logit-normal timestep distribution. Our experiments demonstrate that RecTable
achieves competitive performance compared to the several state-of-the-art
diffusion and score-based models while reducing the required training time. Our
code is available at https://github.com/fmp453/rectable.
Trajectory Balance with Asynchrony: Decoupling Exploration and Learning
  for Fast, Scalable LLM Post-Training
Authors: name: Brian R. Bartoldson, name: Siddarth Venkatraman, name: James Diffenderfer, name: Moksh Jain, name: Tal Ben-Nun, name: Seanie Lee, name: Minsu Kim, name: Johan Obando-Ceron, name: Yoshua Bengio, name: Bhavya Kailkhura
Published: 2025-03-24 17:51:39

Reinforcement learning (RL) is a critical component of large language model
(LLM) post-training. However, existing on-policy algorithms used for
post-training are inherently incompatible with the use of experience replay
buffers, which can be populated scalably by distributed off-policy actors to
enhance exploration as compute increases. We propose efficiently obtaining this
benefit of replay buffers via Trajectory Balance with Asynchrony (TBA), a
massively scalable LLM RL system. In contrast to existing approaches, TBA uses
a larger fraction of compute on search, constantly generating off-policy data
for a central replay buffer. A training node simultaneously samples data from
this buffer based on reward or recency to update the policy using Trajectory
Balance (TB), a diversity-seeking RL objective introduced for GFlowNets. TBA
offers three key advantages: (1) decoupled training and search, speeding up
training wall-clock time by 4x or more; (2) improved diversity through
large-scale off-policy sampling; and (3) scalable search for sparse reward
settings. On mathematical reasoning, preference-tuning, and automated
red-teaming (diverse and representative post-training tasks), TBA produces
speed and performance improvements over strong baselines.
RONA: Pragmatically Diverse Image Captioning with Coherence Relations
Authors: name: Aashish Anantha Ramakrishnan, name: Aadarsh Anantha Ramakrishnan, name: Dongwon Lee
Published: 2025-03-14 01:45:38

Writing Assistants (e.g., Grammarly, Microsoft Copilot) traditionally
generate diverse image captions by employing syntactic and semantic variations
to describe image components. However, human-written captions prioritize
conveying a central message alongside visual descriptions using pragmatic cues.
To enhance pragmatic diversity, it is essential to explore alternative ways of
communicating these messages in conjunction with visual content. To address
this challenge, we propose RONA, a novel prompting strategy for Multi-modal
Large Language Models (MLLM) that leverages Coherence Relations as an axis for
variation. We demonstrate that RONA generates captions with better overall
diversity and ground-truth alignment, compared to MLLM baselines across
multiple domains. Our code is available at: https://github.com/aashish2000/RONA
Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging
Authors: name: Han Wu, name: Yuxuan Yao, name: Shuqi Liu, name: Zehua Liu, name: Xiaojin Fu, name: Xiongwei Han, name: Xing Li, name: Hui-Ling Zhen, name: Tao Zhong, name: Mingxuan Yuan
Published: 2025-03-26 15:34:37

The transition from System 1 to System 2 reasoning in large language models
(LLMs) has marked significant advancements in handling complex tasks through
deliberate, iterative thinking. However, this progress often comes at the cost
of efficiency, as models tend to overthink, generating redundant reasoning
steps without proportional improvements in output quality. Long-to-Short (L2S)
reasoning has emerged as a promising solution to this challenge, aiming to
balance reasoning depth with practical efficiency. While existing approaches,
such as supervised fine-tuning (SFT), reinforcement learning (RL), and prompt
engineering, have shown potential, they are either computationally expensive or
unstable. Model merging, on the other hand, offers a cost-effective and robust
alternative by integrating the quick-thinking capabilities of System 1 models
with the methodical reasoning of System 2 models. In this work, we present a
comprehensive empirical study on model merging for L2S reasoning, exploring
diverse methodologies, including task-vector-based, SVD-based, and
activation-informed merging. Our experiments reveal that model merging can
reduce average response length by up to 55% while preserving or even improving
baseline performance. We also identify a strong correlation between model scale
and merging efficacy with extensive evaluations on 1.5B/7B/14B/32B models.
Furthermore, we investigate the merged model's ability to self-critique and
self-correct, as well as its adaptive response length based on task complexity.
Our findings highlight model merging as a highly efficient and effective
paradigm for L2S reasoning, offering a practical solution to the overthinking
problem while maintaining the robustness of System 2 reasoning. This work can
be found on Github https://github.com/hahahawu/Long-to-Short-via-Model-Merging.
UniHDSA: A Unified Relation Prediction Approach for Hierarchical
  Document Structure Analysis
Authors: name: Jiawei Wang, name: Kai Hu, name: Qiang Huo
Published: 2025-03-20 06:44:47

Document structure analysis, aka document layout analysis, is crucial for
understanding both the physical layout and logical structure of documents,
serving information retrieval, document summarization, knowledge extraction,
etc. Hierarchical Document Structure Analysis (HDSA) specifically aims to
restore the hierarchical structure of documents created using authoring
software with hierarchical schemas. Previous research has primarily followed
two approaches: one focuses on tackling specific subtasks of HDSA in isolation,
such as table detection or reading order prediction, while the other adopts a
unified framework that uses multiple branches or modules, each designed to
address a distinct task. In this work, we propose a unified relation prediction
approach for HDSA, called UniHDSA, which treats various HDSA sub-tasks as
relation prediction problems and consolidates relation prediction labels into a
unified label space. This allows a single relation prediction module to handle
multiple tasks simultaneously, whether at a page-level or document-level
structure analysis. To validate the effectiveness of UniHDSA, we develop a
multimodal end-to-end system based on Transformer architectures. Extensive
experimental results demonstrate that our approach achieves state-of-the-art
performance on a hierarchical document structure analysis benchmark,
Comp-HRDoc, and competitive results on a large-scale document layout analysis
dataset, DocLayNet, effectively illustrating the superiority of our method
across all sub-tasks. The Comp-HRDoc benchmark and UniHDSA's configurations are
publicly available at https://github.com/microsoft/CompHRDoc.
PathoHR: Breast Cancer Survival Prediction on High-Resolution
  Pathological Images
Authors: name: Yang Luo, name: Shiru Wang, name: Jun Liu, name: Jiaxuan Xiao, name: Rundong Xue, name: Zeyu Zhang, name: Hao Zhang, name: Yu Lu, name: Yang Zhao, name: Yutong Xie
Published: 2025-03-23 07:37:24

Breast cancer survival prediction in computational pathology presents a
remarkable challenge due to tumor heterogeneity. For instance, different
regions of the same tumor in the pathology image can show distinct
morphological and molecular characteristics. This makes it difficult to extract
representative features from whole slide images (WSIs) that truly reflect the
tumor's aggressive potential and likely survival outcomes. In this paper, we
present PathoHR, a novel pipeline for accurate breast cancer survival
prediction that enhances any size of pathological images to enable more
effective feature learning. Our approach entails (1) the incorporation of a
plug-and-play high-resolution Vision Transformer (ViT) to enhance patch-wise
WSI representation, enabling more detailed and comprehensive feature
extraction, (2) the systematic evaluation of multiple advanced similarity
metrics for comparing WSI-extracted features, optimizing the representation
learning process to better capture tumor characteristics, (3) the demonstration
that smaller image patches enhanced follow the proposed pipeline can achieve
equivalent or superior prediction accuracy compared to raw larger patches,
while significantly reducing computational overhead. Experimental findings
valid that PathoHR provides the potential way of integrating enhanced image
resolution with optimized feature learning to advance computational pathology,
offering a promising direction for more accurate and efficient breast cancer
survival prediction. Code will be available at
https://github.com/AIGeeksGroup/PathoHR.
Image as an IMU: Estimating Camera Motion from a Single Motion-Blurred
  Image
Authors: name: Jerred Chen, name: Ronald Clark
Published: 2025-03-21 17:58:56

In many robotics and VR/AR applications, fast camera motions cause a high
level of motion blur, causing existing camera pose estimation methods to fail.
In this work, we propose a novel framework that leverages motion blur as a rich
cue for motion estimation rather than treating it as an unwanted artifact. Our
approach works by predicting a dense motion flow field and a monocular depth
map directly from a single motion-blurred image. We then recover the
instantaneous camera velocity by solving a linear least squares problem under
the small motion assumption. In essence, our method produces an IMU-like
measurement that robustly captures fast and aggressive camera movements. To
train our model, we construct a large-scale dataset with realistic synthetic
motion blur derived from ScanNet++v2 and further refine our model by training
end-to-end on real data using our fully differentiable pipeline. Extensive
evaluations on real-world benchmarks demonstrate that our method achieves
state-of-the-art angular and translational velocity estimates, outperforming
current methods like MASt3R and COLMAP.
ViLBench: A Suite for Vision-Language Process Reward Modeling
Authors: name: Haoqin Tu, name: Weitao Feng, name: Hardy Chen, name: Hui Liu, name: Xianfeng Tang, name: Cihang Xie
Published: 2025-03-26 06:38:31

Process-supervised reward models serve as a fine-grained function that
provides detailed step-wise feedback to model responses, facilitating effective
selection of reasoning trajectories for complex tasks. Despite its advantages,
evaluation on PRMs remains less explored, especially in the multimodal domain.
To address this gap, this paper first benchmarks current vision large language
models (VLLMs) as two types of reward models: output reward models (ORMs) and
process reward models (PRMs) on multiple vision-language benchmarks, which
reveal that neither ORM nor PRM consistently outperforms across all tasks, and
superior VLLMs do not necessarily yield better rewarding performance. To
further advance evaluation, we introduce ViLBench, a vision-language benchmark
designed to require intensive process reward signals. Notably, OpenAI's GPT-4o
with Chain-of-Thought (CoT) achieves only 27.3% accuracy, indicating the
benchmark's challenge for current VLLMs. Lastly, we preliminarily showcase a
promising pathway towards bridging the gap between general VLLMs and reward
models -- by collecting 73.6K vision-language process reward data using an
enhanced tree-search algorithm, our 3B model is able to achieve an average
improvement of 3.3% over standard CoT and up to 2.5% compared to its untrained
counterpart on ViLBench by selecting OpenAI o1's generations. We release the
implementations at https://ucsc-vlaa.github.io/ViLBench with our code, model,
and data.
Qwen2.5-Omni Technical Report
Authors: name: Jin Xu, name: Zhifang Guo, name: Jinzheng He, name: Hangrui Hu, name: Ting He, name: Shuai Bai, name: Keqin Chen, name: Jialin Wang, name: Yang Fan, name: Kai Dang, name: Bin Zhang, name: Xiong Wang, name: Yunfei Chu, name: Junyang Lin
Published: 2025-03-26 04:17:55

In this report, we present Qwen2.5-Omni, an end-to-end multimodal model
designed to perceive diverse modalities, including text, images, audio, and
video, while simultaneously generating text and natural speech responses in a
streaming manner. To enable the streaming of multimodal information inputs,
both audio and visual encoders utilize a block-wise processing approach. To
synchronize the timestamps of video inputs with audio, we organize the audio
and video sequentially in an interleaved manner and propose a novel position
embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently
generate text and speech while avoiding interference between the two
modalities, we propose Thinker-Talker architecture. In this framework,
Thinker functions as a large language model tasked with text generation, while
Talker is a dual-track autoregressive model that directly utilizes the hidden
representations from the Thinker to produce audio tokens as output. Both the
Thinker and Talker models are designed to be trained and inferred in an
end-to-end manner. For decoding audio tokens in a streaming manner, we
introduce a sliding-window DiT that restricts the receptive field, aiming to
reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly
sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni
achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench.
Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following
is comparable to its capabilities with text inputs, as evidenced by benchmarks
such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming
Talker outperforms most existing streaming and non-streaming alternatives in
robustness and naturalness.
LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
Authors: name: Kexian Tang, name: Junyao Gao, name: Yanhong Zeng, name: Haodong Duan, name: Yanan Sun, name: Zhening Xing, name: Wenran Liu, name: Kaifeng Lyu, name: Kai Chen
Published: 2025-03-25 18:21:07

Multi-step spatial reasoning entails understanding and reasoning about
spatial relationships across multiple sequential steps, which is crucial for
tackling complex real-world applications, such as robotic manipulation,
autonomous navigation, and automated assembly. To assess how well current
Multimodal Large Language Models (MLLMs) have acquired this fundamental
capability, we introduce LEGO-Puzzles, a scalable benchmark designed
to evaluate both spatial understanding and sequential
reasoning in MLLMs through LEGO-based tasks. LEGO-Puzzles consists of 1,100
carefully curated visual question-answering (VQA) samples spanning 11 distinct
tasks, ranging from basic spatial understanding to complex multi-step
reasoning. Based on LEGO-Puzzles, we conduct a comprehensive evaluation of
state-of-the-art MLLMs and uncover significant limitations in their spatial
reasoning capabilities: even the most powerful MLLMs can answer only about half
of the test cases, whereas human participants achieve over 90\% accuracy. In
addition to VQA tasks, we evaluate MLLMs' abilities to generate LEGO images
following assembly illustrations. Our experiments show that only
Gemini-2.0-Flash and GPT-4o exhibit a limited ability to follow these
instructions, while other MLLMs either replicate the input image or generate
completely irrelevant outputs. Overall, LEGO-Puzzles exposes critical
deficiencies in existing MLLMs' spatial understanding and sequential reasoning
capabilities, and underscores the need for further advancements in multimodal
spatial reasoning.