Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration
Authors: Hao Zhong
Muzhi Zhu
Zongze Du
Zheng Huang
Canyu Zhao
Mingyu Liu
Wen Wang
Hao Chen
Chunhua Shen
Venue: NeurIPS 2025 poster ()
Keywords: RL
Omni
Primary area: deep_learning

Abstract: Long-horizon video-audio reasoning and fine-grained pixel understanding impose conflicting requirements on omnimodal models: dense temporal coverage demands many low-resolution frames, whereas precise grounding calls for high-resolution inputs. We tackle this trade-off with a two-system architecture: a Global Reasoning System selects informative keyframes and rewrites the task at low spatial cost, while a Detail Understanding System performs pixel-level grounding on the selected high-resolution snippets.
Because "optimal" keyframe selection and reformulation are ambiguous and hard to supervise, we formulate them as a reinforcement-learning (RL) problem and present Omni-R1, an end-to-end RL framework built on Group Relative Policy Optimization.
Omni-R1 trains the Global Reasoning System through hierarchical rewards obtained via online collaboration with the Detail Understanding System, requiring only one epoch of RL on small task splits.
Experiments on two challenging benchmarks, Referring Audio-Visual Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show that Omni-R1 not only surpasses strong supervised baselines but also outperforms specialized state-of-the-art models, while substantially improving out-of-domain generalization and mitigating multimodal hallucination.
Our results demonstrate the first successful application of RL to large-scale omnimodal reasoning and highlight a scalable path toward universally foundation models.

Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration
Hao Zhong 1. Muzhi Zhu 1:2g Zongze Du'; Zheng Huang 12, Canyu Zhao',
Mingyu Liu', Wen Wang 2 Hao Chen', Chunhua Shen 122.
Zhejiang University, China 2 Ant Group
Abstract
Long-horizon video-audio reasoning and fine-grained pixel understanding im- pose conflicting requirements omnimodal models: dense temporal coverage demands many low-resolution frames, whereas precise grounding calls for high- resolution inputs. We tackle this trade-off with two-system architecture: Global Reasoning System selects informative keyframes and rewrites the task at low spatial cost, while Detail Understanding System performs pixel-level grounding on the selected high-resolution snippets. Because "optimal" keyframe selection and reformulation are ambiguous and hard to supervise, we formulate them as a ceinforcement-larmiin (RL) problem and present Omni-R1, an end-to-end RL framework built on Group Relative Policy Optimization. Omni-RI trains the Global Reasoning System through hierarchical rewards obtained via online col- laboration with the Detail Understanding System, requiring only one epoch of RL on small task splits. Experiments on two challenging benchmarks, Referring Audio- Visual Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show that Omni-R1 not only surpasses strong supervised baselines but also outperforms specialized state-of-the oh-the-art models, while substantially improving out-of-domain generalization and mitigating multimodal hallucination.
Our results demonstrate the first successful application of RL to large-scale nimodal reasoning and highlight a scalable path toward universally foundation models. Our code is released at: https: htpp:://ee //t::/gttttt ggithuu...... com/ commaimmaai aaimmuofaaffa www.a/oniaaan/aeaaommmmm 00mmi-m1..
Introduction
Enabling models to simultaneously perceive, understand, and reason over omnimodal inputs suuuts-such such as text, video, and audio in complex real-world scenarios remains a longstanding goal artificial
intelligence [1, 2, 3. 4]. Recent advances in omnimodal pretraining and instruction fine-tuning have led to the emergence of omnimodal models, bringing us closer to this objective [5, 6, 7]. Despite recent progress, current omnimodal models exhibit notable limitations in two key areas: (1) long-horizon reasoning over complex temporal sequences in video and audio, and (2) fine-grained spatial understanding at the pixel level. A fundamental challenge underlying these two problems is the inherent trade-off' between temporal coverage and spatial resolution. Long-horizon reasoning [8, 9, 10] over video and audio typically requires high frame rates to capture global temporal context, which significantly increases memory and computational overhead- on low-resolution frames. Conversely, fine- mne-graindd pixel understanding [1 I. 12, 13, 14, 15] demands high-resolution inputs to preserve visual details, which in turn limits the number frames that can
forcing models to operate
"Equal contribution.
tC. Shen is the corresponding author. This work was in part supported by The Pioneer R&D Program of Zhejiang (Grant No. 2025C01011).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Results
Inferring
MVBench
VideMMME-Short
VideoMMME
Detail Understanding System
There are four.
RevOS
OmniBench
How many knives the robot see in the right when can hear water?
User
Omni-Ri
JUA
Global Reasoning System
Mevis
RatAvS-Unsaen
RetAvS-Seen
ThinkoI need to find out the fromes when doing the dishes ..5/ -s/Thinlo
Omni-RI
VideoChat-1
Qwen225-VL-77
Owen2.5-Omns
VISA-138
doob
Arowert Time "0:50" "en__tine" "01:517, "desonitiont doing dchher
high resolution key frames
MiniCPM-o 2.6 2.6-88
Omni R1
Balchuan Omni- 1.5-76
cow shepped reaveenn compan then turned around place?
shiile The nutt sction wsks slettify which cow stiepped with with the monemeni of its componion and thes lurned around in place. The key action here the stupping bock and terning around. The vilco showd heo cond, and one then shaps book and then turds around. The other sae remaiss shotionrry during sctice
presiction
SoZVA pred
Figure 1: Overview of the proposed Omni-RI system for collaborative video understanding. Left: Performance comparison across multiple benchmarks shows Omni-R1 significantly outperforms existing omni-modal and video-reinforced MILMs on both segmentation-centric and reasoning- centric tasks. Top-right: Omni-R1 employs two-stage collaborative framework, integrating detail understanding system (for precise visual QA) and global reasoning system (for temporal grounding and high-resolution key frame identification). Bottom: qualitative example highlights segmentation and reasoning in identifying object-centric actions,
Omni-Rl's 's precise outperforming prior expert models (e.g., Sa2VA) in complex scenarios.
be processed. This trade-off creates a tension between global context modeling and local detail preservation, making it difficult for existing models to excel both simultaneously.
natural way to address this trade-off to decompose the problem into two stages. Accordingly, we frame our solution as two-system architecture:
System 1 (Global Reasoning System) performs coarse-grained global reasoning over long video sequences at low spatial resolution acting as a fast, context-aware selector that identifies critical temporal segments.
System 2 (Detail Understanding System), in contrast, conducts detailed, high nigh-resolution analysis over a small number of keyframes, focusing on precise grounding and fine-grained understanding.
To illustrate how these systems interact, consider task where the goal is to segment the last person to disappear (or make sound) in scene. Systeml first processes the full video (with audio) sequence

---

to determine, through low-resolttion multimodal abstraction, which person is the last to leave visually or to emit sound. then selects a few key segments where this individual appears or speaks. Since System 2 operates only on short segments with high-resolution input and lacks access to long-range temporal or auditory context, System 1 needs to reformulate the original reference task-initially requiring long-horizon multimodal reasoning-ing into a simpler, localized problem. This reformulated task focuses on attributes, identity cues, and object permanence within the selected key segments, making it solvable using only fine-grained visual information. System 2 then takes these key segments and performs fine-grae grained visual grounding directly on the high-resolution input, bypassing the need for global reasoning. This two-system design enables scalable and efficient multimodal reasoning by eliminating the need to process entire videos at high resolution, and effectively addresses the dual challenge of long-horize reasoning and fine- fine-grained visual understanding.
is worth noting that current multimodal models already perform well as Detail Understanding Systems in tasks such as visual grounding [12. 16], OCR (2, 17, 18] and fine-grained image under- standing [19, 20] on high-resolution inputs. Given this progress, the bottleneck in our two-system framework lies primarily in the capabilities of System 1. In this work, we therefore focus improv- ing Global Reasoning System, particularly its ability to select informative key frafra and reformulate the task. However, defining what constitutes an "optimal" keyframe selection or task reformulation is inherently ambiguous and dependent, making it impractical to rely on manually curated SFT data. To address this, we propose Omni-R1, an end-to-end reinforcement learning framework tailored for omnimodal reasoning. Built upon the Group Relative Policy Optimization (GRPO) [21, 22] algorithm, our method simulates online collaboration between System 1 and System 2, applying policy gradient updates guided by a hierarchical reward framework to progressively train System to select keyframes and reformulate tasks in long-horizon, omnimodal settings.
From a reinforcement learning (RL) perspective, although it has proven effective in enhancing reasoning within large language models [21, 22, 23], RL remains underexplored in omnimodal settings. One major challenge lies the lack of effective multimodal reasoning data [24], along with uncertainty about whether language-basee RL techniques can generalize across modalities. While Onni-RI bridges this gap by reformulating long-horizon multimodal understanding as a collaborative process between two systems. In our design, the Global Reasoning System functions that selects keyframes and reformulates tasks for the Detail Understanding System to complete. Such an approach provides a scalable path toward improving temporal reasoning and summarization in omnimodal models, while also opening new opportunities for applying RL. beyond purely linguistic tasks.
RL agent
To validate the effectiveness of Omni-RI, we benchmark it on two especially demanding tasks, namely Referring Audio-Viss Visual Segmentation (RefAVS [25]) and Reasoning Video Object Segmentation (REVOS [26]), both of which require temporal reasoning over video(audio) streams and fine-grained pixel understanding. Training Omni-R1 for just one epoch on the small datasets already lifts performance well beyond our baseline model and even surpasses the strongest, highly specialized ofte--of-the-at models each benchmark. Even more striking, reinforcement learning improves out-of-domain generalization, whereas conventional supervised fine-tuning often weakens it. Omni- R1 achieves higher scores in both pure video- iiileo--ndesstandnng and omnimodal understanding settings, outperforming recent RL methods tailored specifically to video-reasoning tasks. Finally, we conduct a comprehensive suite of diagnostic studies incluudnn ablations over key architectural and training choices and an analysis of RL's impact on multimodal mallucination-whin together highlight the versatility and reliability of our approach. We hope that Omni-RI offers new direction for applying reinforcement learning to future all-modality foundation models.
Our primary contributions are summarized as follows:
We present scalable Global Reasoning, and Detail Understanding two-system archi- tecture that separates long-horizon video-audio reasoning from fine-grained pixel-level grounding, effectively resolving the temporal-spatial trade-off that constrains existing omni- modal models.
We introduce an end-to-end reinforcement-leaming framework Omni-R1, built on Group Relative Policy Optimization that trains System I via hierarchical rewards and simulated collaboration with System 2 to select keyframes and reformulate tasks in long-horizon omnimodal settings.
3

---

With one epoch RL training, Omni-RI surpasses strong supervised baselines and special- ized SOTA methods on RefAVS and REVOS, while markedly improving out-of-domain generalization including video understanding and omnimodal understanding.
2 Related Work
2.1
Omni-modal Large Models
The advent of Large Language Models (LLMs) has revolutionized artificial intelligence, showcasing unprecedented capabiliitess in understanding, generating, and reasoning with textual data [27, 28, 29, 30]. Building upon this foundation, Multimodal Large Language Models (MLLMs) have emerged, integrating multiple data modalities such as vision, language, and audio achieve more holistic understanding of complex tasks [12, 31, 32, 33, 34].
To differentiate from vision-language models (VLMs), multimodal large language models (MLLMs) incorporating the audio modality, such as Qwen2.5-Omni [35], are termed omni--moda models, abbreviated as omni. MiniCPM-o 2.6 [2] extends its vision-language foundation [2] with audio processing capabilities, allowing it to operate across more modalities.
[1]. trained and inferred in a fully end-to-end manner, surpasses GPT-4o-mini on the full-modality leaderboard OmniBench [36]. The recent development of omni-modal models further extends this integration, encompassing visual, linguistic, and auditory modalities to approach comprehensive multimodal understanding [7].
2.2 MLLM with RL
Despite the remarkable progress enabled by supervised learning and instruction tuning, key challenges persist in aligning MLLMs with human preferences, mitigating harmful outputs, and enhancing their performance on complex reasoning tasks. Reinforcement Learning (RL). particularly Reinforcement Learning from Human Feedback (RLHF) [23], has proven effective in addressing these issues within unimodal LLMs, contributing to the success of models like ChatGPT [27]. A notable advancement in this domain is the introduction of DeepSeek-R [21], which employs Group Relative Policy Optimization (GRPO) to enhance reasoning capabilities. GRPO innovatively replaces traditional critic models with a group-based reward normalization approach, reducing computational costs while maintaining performance [22]. This technique has demonstrated that pure RL can effectively develop strong reasoning abilities without reliance on supervised data.
While reinforcement learning techniques have been widely explored in LLMs, their application to MLLMs is still at an early stage. Most recent efforts [37, 38, 39, 40, 41, 42] have primarily focused vision and language modalities, with little attention paid to more comprehensive multimodal integration. Notably, a concurrent work, R1-Omni [43], is the first to include audio in addition to vision and language; however, its focus is limited to single motion recognition task.
In contrast, our work targets more general long- morizoo understanding tasks and conducts a more comprehensive and systematic investigation of omni-modal reinforcement learning. Building on recent advances, we propose Omni-R1, an omni-modal framework that unifies vision, language, and audio processing under end-to-end RL optimization pipeline. Our two-system design, which separates temporal reasoning from spatial perception, enables enhanced long-horizon understanding and fine- grained attention, allowing Omni-RI R1 to better address complex multimodal tasks requiring both structured perception and dynamic maaiii.nmmaing.....
3 Omni-R1
3.1 Task and System Formulation
We consider a long-horizon horizon multimodal understanding task, where the model receives a video sequence V = {21, ......, Fr and a synchronized audio stream A = {a1, az...., )r along with an instruction or query q, The goal to produce task-specific output (e-g., a localized segment, a textual response, or grounding prediction) that reflects both global temporal reasoning and fine- grained visual understanding. To better facilitate global temporal reasoning, we transform the raw instruction g into a high-level instruction Aglobal = T[g) via template- based rewriting function T.

---

Training
Global Reasoning System GRPO Rollouts
Thinko The instruction asks identify the brownish yellww cat. The video shond broenish-yeltee out and enother cut the right. Since the bosk is to segmont the browwiih-ysllzw octs, the relsount durat on should be whin bath broamsh- yellow cots ane visilee evan though they appear of different times with one being right uide the plate some point.
Directly identify the brownish-yellow cat in this video.
O
KAmmerr C ('stort, -imee: unnd im= dearepd "deicriptior"" bbowaish- yollow /Arrwerr
|n|
Omni Data
Key Frumes Selccion
Omni-RI
Detail Understanding System & Reference Model
GRPO Rewards Feedback
Input Frames
Output Grounding
Rx
Rx
X
Omni-RI
RA
R
X
Rg
Rg X
Grounding
Key Frome Quality
Frame Instruction Alignment
Temporal Consistency
Match
Whole Cat
S Wrong
Rx
RA Original Instruction
Ground
Rg
Condiatinn
Enconsishent
Orly
Local Reward
Global Reward
Figure 2: Exclusively trained System video segmentation tasks in an End-to-End RL pipeline, Omni-R1 improved general understanding capabilities.
Stage 1: Global Reasoning System. We reduce the spatial resolution of the video commonly adopted obtain low-resolution stream V {Ø1, ..., ör) suitable for efficient global processing.
Given (V, A, (global). System I produces set of K selected segments(frames) 8 = (81, ....... ... *x)
and corresponding set of local queries { 1 i=1 intended to simplify the reasoning objective for System 2:
8, (q
(6) 1K = 7. (S) A, Aglobat)
Stage 2: Detail Understanding System. System 2 then receives the high-resolution frames Vs = ......... Peg] corresponding to the segments selected by System 1. Given (Vg, ) it performs fine-grained multimodal reasoning and produces the final output:
y = π (S2) (Vs, (Fweel ) (==)
For tasks such as RefAVS and RVOS, one possible instantiation of System 2 is as combination of framm grounding model F grounding and a frozen video segmentation model F (e.g.. SAM2 [44]). Given the selected high-resolution frames Vs = fr....... Us and the corresponding local instructions { 1 the grounding model is applied independently each pair (V,, Hoeal) to predict set of bounding boxes Ba, {6(9) ..... ON.), where each 6,00 IR denotes a box in
(6)
5

---

(21, 12, y2) format. These predicted boxes are then passed to the segmentation model to produce pixel-level instance masks and propagate them temporally across the entire video:
M = F weg (V, Vg. (B,,
where the final output MC = {m mrb is a sequence of temporally aligned masks. The corresponding ground-truth mask sequence is denoted as M .m....., ....... my the binary segmentation mask for frame er. For more details on the segmentation model Feeg, please refer Section B in Appendix.
where each my is
3.2 End-to-End Reinforcement Learning via GRPO
We now turn our focus to optimizing System x (5()), with the goal of improving both the selection of key segments 8 and the formulation of task- task-specific local instructions (P J. in order to better support System 2 in performing fine-grained understanding. However, S, { }. and the System (i.e., π (S2)) are strongly coupled, making it difficult to directly define what constitutes an optimal pair (8, ( })) for downstream performance. As a result, constructing high-quality supervised fine-tuning (SFT) data for (S1)) is infeasible.
Instead, we propose to optimize π (S1) via reinforcement learning by designing a reward function R(8, ((8,(()a))) n (52)) that evaluates the utility of System I's outputs in enabling System 2 to succeed. Under this framework, 7(S)) is trained to explore and generate candidate outputs, and receives feedback from the environment through this reward. Specifically, we adopt GRPO-based policy optimization scheme. At each iteration, we sample N responses from the current policy x(Si) and compute the corresponding rewards In using the reward function R(-). We then normalize the rewards to estimate the advantage of each sample:
A Tn mean (TN.
(I)
std( (rd("I,....TN)
Based on the computed advantages (An}, we perform PPO-style policy gradient updates to improve (S1)
3.3 Hierarchical Reward Design for System
Designing effective reward function that accurately reflects the quality of the action pair (8, {9onnl }) and provides meaningful training signal for System (51) is critical the success of our framework. In this section, we describe our hierarchical reward formulation tailored for the Referring Video Object Segmentation (RVOS) task, which aims guide System 1 to progressively learn to select informative keyframes and generate useful local instructions.
(1)
Due to the strong coupling among 8, (i) }- and System 2 (r((2), relying solely on the final task objective (e.g., segmentation mIoU) as the reward leads unstable and inefficient training. This is because such reward signals are sparse, non non-decomposable and difficult to attribute back to specific decisions made by (S1), To address this, we propose a set of hierarchical reward functions, organized from weakly coupled strongly coupled, and from local to global. These rewards are designed to incrementally shape the learning of System 1, starting from simpler supervision signals and gradually incorporating more task task- specific structure. We define three types of rewards:
Key Frame Quality Reward (Ry)): This reward evaluates the quality of the selected keyframes S, independently of the instructions or the performance of subsequent segmentation.
provides early learning signals to encourage the selection visually salient or semantically diverse frames.
We define the Key Frame Quality Reward a weighted combination of three factors:
Rx X Rdiversiy + Ag + AgResaicncy

---

The first term, Temporal Diversity Reward Raversay. encourages selected frames to spread over the video timeline. rather than being clustered within short segment. This promotes broader temporal coverage and helps the model focus on long-range dynamics.
The second term, Frame Count Regularization Roum- regularizes the number of selected frames K to stay near predefined target Ko, penalizes selections that include either too few or too many frames.
The third term, Object-Centrid Saliency Reward Raaliency rewards keyframes that contain a large visible portion of the target object. This based on the hypothesis that selecting such frames provides stronger visual anchors, which can facilitate more accurate and stable object tracking and segmentation throughout the video. It is calculated as the normalized average GT mask area:
K
area(m).,
Rsaliency
K
max, area(m?)
Together, these components guide System to select keyframes that are temporally diverse, reasonably sparse, and visually informative. Formal definitions of the reward are provided in Appendix Section B.
Inste-Inttuuttio Alignment Reward (Raa) measures how well each local instruction Noeaa aligns with its corresponding keyframe Us.- This reward evaluates whether the instruction provides sufficient grounding cues to locate the correct object in the frame. As it operates independently per frame- instruction pair, it does not depend on the segmentation model Feeg, and thus ignores temporal
(5)
consistency. Concretely, given a frame Us, and its corresponding instruction Oreatt we apply the
grounding model F grrounding predict set of bounding boxes B, (b)") ...... bl.)) We compare these predictions against the ground-truth target boxes B defined for that frame. Since a single
instruction may refer to multiple target objects, both B., and B can contain multiple instances. The reward is computed as the negative Hungarian matching loss commonly used in object detection [45]:
K R
1 K
=
(1 L Hungurian (B B²²))
(2)
This loss is minimized when the predicted boxes exactly match the ground-truth targets.
Global Temporal Consistency Reward (Rg) ) is the most strongly coupled and task-specific reward in framework, directly reflecting the final objective of long-term video object segmentation. Unlike previous rewards, which evaluate the selected keyframes or instructions in isolation, Rg jointly considers how the selected keyframes 8 and local instructions ( influence the performance of System 2 throughout the video. This reward is designed to capture both the spatial accuracy and the temporal consistency of the predicted instance masks. In particular, encourages System to select frames that are critical for robust tracking- such as those appearing after significant object deformations, occlusions, or disappearances-SS so that the segmentation model (c.g.. SAM2) can re-anchor to the target effectively. Formally, given candidate keyframe set 8 and corresponding instructions (1li) } feed them into System (#(()) obtain full sequence of predicted masks MC (m...... }. }.
The reward is computed as the average frame-wiss Intersection over Union (IoU) with the ground-trui masks M (mi..... )
1 Rg T
IoU mi)
Finally, we combine the above three components to form the overall reward used for training System The total reward weighted sum of the three terms:
R == + aaRA + agRg
where ax.4, are the weighting coefficients that control the importance of each reward component.
7

---

Table
Performance comparison across models grouped by Seen and Unseen sets in Ref- AVSBench [25]. Some metrics curated from [25]. J&F represents the average of (3) score and (F) score. f indicates the results are tested on masks predicted by SAM2 according to model's
grounding output.
Seen
Unseen 2
Model
R&F
F
A&F
F
AVSBench [49] + text
37.2
23.2
51.1
43.5
32.4
54.7
AVSegFormer [50] + text
40.2
33.5
47.0
43.1
36.1
50.1
GAVS [51] + text
39.4
28.9
49.8
39.8
29.8
49.7
ReferFormer [52] + audio
40.7
31.3
50.1
39.6
30.4
48.8
R2VOS [53] audio
33.0
25.0
41.0
38.9
27.9
49.8
BEMC [25]
42.8
34.2
51.3
57.2
49.5
64.8
Qwen2.5-0mmi-77
31.6
27.7
35.5
62.3
59.0
65.7
(SFT)
39.1
35.4
42.8
66.2
63.1
69.3
Omni-R1-7B
47.2
43.0
51.4
74.2
71.3
77.0
A
+8.1
+7.6
+8.6
+8.0
+8.2
+7.7
4 Experiments
4.1 Experiments Setting
System 1 and System 2 We adopt Qwen2.5-0mni-77 [7] as our base model, which serves as System responsible for high-level reasoning. To construct lightweight and stable System during training, we use a frozen copy of the same pretrained Qwen2.5-0mmi model, which also functions as reference policy model for guiding optimization. For evaluation, unless otherwise stated, Omni-R1 serving as both System 1 and System for resource efficiency. However, due the modular design and decoupled functionality of the two systems, System 2 can be flexibly replaced with stronger perception module zero-shot manner.
Training Paradigm We train System I on 1,600 samples randomly selected from the RefAVS [25] dataset and 2,600 videos from the ReVOS [26] and ViV [46] datasets for epoch. To further enhance the model's fine-grained understanding capabilities as system 2, we additionally train the model on 2,000 images from refCOCOg [47] for one epoch in the style of SegZero [48]. Unless otherwise specified, all experiments are conducted using policy KIL divergence hyperparameter of 0.04, a group size of 8, and an initial learning rate of x 10-6 under the Adam optimizer with weight decay of 0.01. We adopt sam2-hieras www-hhierrarrree our SAM2 [44] version throughout experiments.
4.2 Referring Video Segmentation
Referring Audio-Visual Segmentation Ref-AVS [25] is specifically designed for audio-visual segmentation tasks, offering diverse and well-annotated collection of samples that require integrated reasoning across both modalities. The dataset comprises 2,908 audio-equipped video clips in the training set, covering 5,366 annotated objects across 39 semantic categories.
We evaluated the performance of our collaborative system on Ref-AVSBench [25] with other Referring AVS methods. Omni-R1 outperforms previous SOTA EMMC [25] by +4.4% J&F in seen and + +17.0 % on unseen set.
Reasoning Video Object Segmentation ReVOS [26] is VOS dataset that emphasizes reasoning about temporal behaviors through implicit object descriptions, comprising 35,074 pairs of instruction- mask sequences derived from 1,042 diverse videos. In contrast traditional referring video segmen- tation datasets, ReVOS includes text instructions that necessitate a sophisticated understanding of both video content and general world knowledge.
For our evaluation, we exclusively employed Sa2VA as System investigate the full reasoning capabilities of Omni-RI as System

---

Table Reasoning Video Object Segmentation performance comparison across different methods, the metric is A&F score(% I means the results are evaluated where Omni-R1-77 serves as System 1 and Sa2VA as System 2(1B and 4B).
ReVOS
Model
Referring
Reasoning
Single
Mulii
Overall
LISA-13B [11]
41.6
TrackGPT-13B [54]
45.0
VISA-13B [26]
50.9
Sa2VA-8B [55]
57.6
Sa2VA-26B [55]
58.4
Qwen2.5-Omni--2
46.3
26.9
38.6
37.4
36.6
Omni-R1-7B
52.5
36.9
45.0
46.6
44.7
Omni-R1-88
61.6
50.7
56.6
47.3
56.2
Omni-R 1-1 1111B
64.1
53.7
59.2
51.0
58.9
Our System exhibits strong performance on video object segmentation tasks under both basic and reasoning-intensive conditions. When deployed as both systems (f), Omni-R1-78 significantly outperforms the base model on ReVOS, achieving a +8.1 % improvement over Qwen2.5-Ommn 5-Ommi-7.. This result underscores enhanced temporal reasoning and fine-grained recaption capabilities.
Furthermore, the collaborative system ($) Omni-R1-11B achieves a score of 58.9% on ReVOS, surpassing much larger segmentation-specillle achieves the best performance across all categories, including the reasoning subset in ReVOS (53.7%), underscoring the effectiveness of our disentangled system architecture and reinforcement learning-bass bbsed training paradigm.
models such as Sa2VA-26B [55]. Notably,
4.3 General Omni-Modal Understanding
In this section, we focus on the impressive progress Omni-RI on to its base model Qwen2.5- S--mni--B and other leading multi-mode models.
tasks, in comparison
Omni-RI shows stable improvements over its base model Qwen2.5-Omni. Omni-RI achieves aver- age improvement of +2.0% % +2.7 % and +3.7% % over baseline on OmniBench [36], VideoMME [59]
Table 3: Performance comparison across models on general understanding QA benchmarks Om- nibench, VideoMME, and MVBench.
Omnibench Avg
VideoMME General
MVBench General
Method
Short
Vision- Vision-Language Models
Qwen2.5-VL-7BBCoT))
56.
71.3
57.4
LLaVA-OOnVission-1 [56]
58.2
56.7
Kangeroo-8B [57]
56.0
61.1
VideoChat-RI [40]
72.2
67.9
Video-RI [39]
59.3
63.9
Sa2VA-26B [55]
52.6
Omni-Modal Language Models
VITA-L.5-7B [58]
33.4
57.3
55.5
MiniCPM-o 2.6-7B [2]
40.5
63.4
58.6
Baichuan- Baichuan-Ommi-l-5-- [1]
42.9
60.1
63.7
Qwen2.5-Omin
47.3
58.3
69.8
66.1
Omni-R Mini-RR-7BB
49.3
60.7
73.0
70.3
A
+2.0
+2.4
+3.2
+4.2

---

and MVBench [60] respectively, surpassing all other open-source source understanding tasks, Omni-RI gains more progress on short videos (less than 2 min) than long videos. This could be attributed to our VOS training videos, where almost all videos are less than 2 minutes, with MeViS even being less than 30 seconds.
Specifically, in video
System I's Strength in General Understanding Tasks Omni-R1 demonstrates significant im- provements, achieving outstanding general performance on omni-modal benchmark OmniBench, where it outperforms all other 7B models in the open-source space. With a score of 73.0 in the short subset of VideoMME, Omni-R1 surpasses VideoChat-RI [40], which was exclusively fine-tuned for Video QA tasks through RL. Additionally, Omni-RI achieves the highest score MVBench, outperforming all other omni-men modil models by a large margin.
The strong performance of Omni-RI across both in-domain and general tasks showcases the effec- tiveness of our reinforcement learning approach. Leveraging System I's multi-modal reasoning, the model excels in task-specifer scenarios and generalizes effectively to unseen tasks, demonstrating its robustness and adaptability real real-world environments.
Conclusion
We present Omni-R1, novel reinforcement learning framework that addresses key limitation in omnimodal models: the trade-off between long-horizon temporal reasoning and fine-grained spatial understanding. By decoupling these objectives into a two-system architecture comprising Global Reasoning System and a Detail Understanding System Omni-RI enables scalable and efficient processing of complex video-audio-tex. inputs.
Through task reformulation and keyframe selection trained via Group Relative Policy Optimization, our approach significantly improves performance on challenging benchmarks like RefAVS and ReVOS, while also enhancing out-of-domain generalization. Our diagnostic studies further confirm the robustness and versatility of the framework. We hope that this work opens new avenues for integrating reinforcement learning into rext-generation omnimodal foundation models.
10

---

References
[1] Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, al. Baichuan-omni- technical report. arXiv preprint arXiv:2501.15366 2025. 1, 4,4,
[2] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level mllm your phone. arXiv preprint arXiv:2408. 0000000110000 01000, 2024. 1,3,4.9
[3] Chiori Hori, Takaaki Hori, Teng-You Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. Attention based multimodal fusion for video description. In Proceedings of the IEEE international conference on computer vision, pages 4193-4202, 2017.
[4] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
[5] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 1
[6] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.05530 2024.
[7] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint 25X2v:2503.20215. 2025. 1, 4, 8
[8] Heqing Zou, Tianze Luo, Guiyang Xie, Fengmao Lv. Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang, et al. From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding. arXiv preprint arXiv:2409. 18938, 2024.
[9] Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, et al. Mavors: Multi lulti-granularity video representation for multimodal large language model. arXiv preprint arXiv:2504. 10068, 2025.
[10] Zheng Huang. Mingyu Liu, Xiaoyi Lin, Muzhi Zhu, Canyu Zhao, Zongze Du, Xiaoman Li, Yiduo Jia, Hao Zhong, Hao Chen, et al. Notvla: Narrowing of dense action trajectories for generalizable robot manipulation. arXiv preprint arXiv:2510.03880 2025.
[11] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li. Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. Proceedings of the IEEE/CVÉ Conference on Computer Vision and Pattern Recognition, pages 9579- 95899, 2024. 1,9
[12] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.14 13923. 2025. 3,
[13] Muzhi Zhu, Hengtao Li, Hao Chen, Chengxiang Fan, Weian Mao, Chenchen Jing, Yifan Liu, and Chunhua Shen. Segprompt: Boosting open-world segmentation via category-level prompt learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 999-1008. 2023.
[14] Muzhi Zhu, Chengxiang Fan, Hao Chen, Yang Liu, Weian Mao, Xiaogang Xu, and Chun- hua Shen. Generative active learning for long-tailed instance segmentation. arXiv preprint arXiv:2406.022 2024. 1

---

[15] Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang. and Chunhua Shen. Unleashing the potential of the diffusion model in few-shot semantic segmentation. Advances in Neural Information Processing Systems, 37:42672-42695, 2024.
[16] Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:23006. /4824, 2023.
[17] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, arXiv preprint Stti::2306.11007 2023. 3
[18) Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473 2024.
[19) Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, AbdeIrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13009-13018, 2024.
[20] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for variety of vision tasks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4818 1818-4829, 2024. 3
[21] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning, arXiv preprint arXiv:2501.12944 2025. 3,4
[22] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint 2xX::2202..33000. 2024. 3,
[23] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. 3,4
[24] Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces. arXiv preprint arXiv:2407.1 1195, 2024. 3
[25] Yaoting Wang, Peiwen Sun, Dongzhan Zhou, Guangyao Li, Honggang Zhang, and Di Hu. Ref- avs: Refer and segment objects in audio-visual scenes. In European Conference on Computer Vision, pages 196-213. Springer, 2024. 3, 8
[26] Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. arXiv preprint arXin:2407.11325 2024. 3, 8,
[27] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08877 2023. 4
[28] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023, 4
[29] Aixin Liu, Bei Feng Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437, 2024. 4
12

---

[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09288. 2023. 4
[31] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint wcXiv:2412.000 2.5271,, 2024. 4
[32] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023. 4
[33] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
[34] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. vl: Towards real-world vision-language understanding, 2024. 4
[35] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He. Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.2022 2025.
[36] Yizhi Li, Ge Zhang. Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang. Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua
Lin. Omnibench: Towards the future of universal mani-language models, 2024. 4,9 9
[37] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement finetuning. arXiv preprint arXiv:2503.01785 2025. 4
[38] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vim-rl: A stable and generalizable r rl-style large vision-language model. arXiv preprint arXiv:2504.07661
2025. 4
[39] Kaituo Feng. Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning milmss arXiv preprint arXiv:2503.21776 2025. 4.9
[40] Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang. Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforce- ment fine-tuning. arXiv preprint ar arXiv:2504.06955 2025. 4, 9, 10
[41] Muzhi Zhu, Yuzhuo Tian, Hao Chen, Chunluan Zhou, Qingpei Guo, Yang Liu, Ming Yang, and Chunhua Shen. Segagent: Exploring pixel understanding capabilities in milms by imitating human annotator trajectories. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3686-3696, 2025. 4
[42] Muzhi Zhu, Hao Zhong, Canyu Zhao, Zongze Du, Zheng Huang, Mingyu Liu, Hao Chen, Cheng Zou, Jingdong Chen, Ming Yang, et al. Active-o3: Empowering multimodal large language models with active perception via grpo. arXiv preprin! arXiv:2505.21457 2025. 4
[43] Jiaxing Zhao, Xihan Wei, and Liefeng Bo. RI-omni: Explainable mnni-nultimodal emotion recognition with reinforcement learning, 2025. 4
[44] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything images and videos. arXiv preprint arXiv:2408.000 007 14, 2024. 5,
[45] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020. 7, 24
13

---

[46] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. In ICCV, 2023.
[47] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. 2016. 8
[48] Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiay Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforeement. arXiv preprint arXiv:2503.06520 2025. 8
[49] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Avsbench: A pixel-level audio- visual segmentation benchmark.
[50] Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visud segmentation with transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12155-12163, 2024. 8
[51] Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. Prompting segmentation with sound is generalizable audio-visual source localizer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5669-5677, 2024. 8
[52] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974 4984, 2022. 8
[53] Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj. and Yan Lu. Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023. 8
[54] Nicholas Stroh. Trackgpt-a generative pre-trained transformer for cross-domain entity trajectory forecasting. arXiv preprint arXiv:2402.0000 2024. 9
[55] Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi. Jiashi Feng, and Ming-Hsuur Yang. Sa2va: Marrying sam2 with Ilava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501. 040001, 2025. 28
[56] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 033.00 2024. 9
[57] Jiajun Liu, Yibing Wang. Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context 2024. 9
video input. arXiv preprint arXiv:2408.
[58] Chaoyou Fu, Haojia Lin, Xiong Wang. Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. V.5a-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01955 2025. 9
[59] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
[60] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-22206, 2024. 10
[61] David F. Crouse. On implementing 2d rectangular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679-1666 2016. 24
[62] Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and Jiaya Jia. Vision- reasoner: Unified visual perception and reasoning via reinforcement learning. arXiv preprint 1200, 2025. 26
14

---

[63] Yunyang Xiong, Chong Zhou, Xiaoyu Xiang, Lemeng Wu, Chenchen Zhu, Zechun Liu, Sak- sham Suri, Balakrishnan Varadarajan, Ramya Akula, Forrest landola, al. Efficient track anything. arXiv preprint arXiv:241 I. 18933, 2024. 27
[64] Kim Sung-Bin, Oh Hyun-Bin, Jung Lee, Arda Senocak, Joon Son Chung, and Tae- Tyu-Hyun Oh. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models. arXiv preprint arXiv:2410.1882 2024. 27
[65] Kaining Ying, Henghui Ding, Guangquan Jie, and Yu-Gang Jiang. Towards omnimodal expres- sions and reasoning in referring audio-visual segmentation. arXiv preprint arXiv:2507.22886. 2025. 30
[66] Henghui Ding, Song Tang, Shuting He, Chang Liu, Zuxuan Wu, and Yu-Gang Jiang. Multimodal referring segmentation: A survey. arXiv preprint arXiv:2508 00265, 2025. 30
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
We don't 't have theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide detailed exposition of the sources for all training data and the specific configurations our training parameters. We believe our work fully reproducible. See Section 4 and Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We plan to release the model and code later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wttps.cc/ppblccc//cccccccpcppppp weaiic/gguie....e CoodeSummii agionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We follow the mainstream evaluation methods, which do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tpr//eerrp/.eeeee cc/ppbbicc/tchicccuiieeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All papers and data are properly credited and cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't 't involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The usage of LLMs is described in detail. See Section 3 and 4.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /ttp:///eeuu.pp/eeeee htt.://c..ccooeeeeeee//2200555 for what should or should not be described.
22

---

A Appendix Overview
This appendix provides additional details on the experimental setup, model architecture along with training pipeline, and suppiementary results that support the findings presented in the main paper.
Implementation Details: this section details additional aspects of our method: our two- system architecture (including user instructions for its MLLM components and prompts for the downstream SAM2 model), our reward design, and the differences between training and inference procedures.
Ablation Studies: this section provides ablation studies of System regarding architectural selection, reward components, and dataset selection.
Visualization Results: this section provides more visualization results, including examples in comparison with other methods and failure case analysis.
More Analysis: this section provides an analysis on the hallucination issue and the influence of resolution on general video understanding tasks.
Limitations and Future Work: this section discusses the limitations of our method and potential future work.
B Implementation Details
User Instructions on Two Systems. To enable MLLMs to perform keyframe selection and referred object captioning, we designed the prompt as shown in the figure 3. We formulate keyframes as time duration segments and assign spatial description text to each duration. Additionally, we observed that during training, the model could be influenced by the timestamp patterns seen in the prompt examples. Therefore, to increase the diversity of keyframe distributions during training and prevent the model from overfitting to specific timestamps, we incorporated randomized timestamps into the prompt, encouraging the model to focus learning keyframe selection and caption rather than simply copying training timestamps. For AVS tasks, we designed a similar prompt (see Figure 4) to guide the model in analyzing the audio content and identifying the corresponding visual grounding description. The prompt emphasizes the need to avoid temporal expressions and instead focus on visual cues.
The intermediate results are then interpreted as frames and paired descriptions before being fed into System 2. The prompt for System 2 follows the official grounding prompt of Qwen2.5-VL, where the output a list of bounding boxes and their corresponding labels in JSÓN format.
Prompt Design for SAM2 Downstream Segmenter. Once the keyframe grounding results obtained from System 2, we assign a unique identifier to each detection result using tuple format: bbox). This ensures that any detection within single GRPO group can be uniquely referenced.
(roll_out_idx, frame_ids
Since all detections within group share the same input context, we optimize inference efficiency by processing all detection results in single forward pass. Specifically, we feed the entire video segment into SAM2, and for each detection tuple, we assign a unique object ID, These object IDs are used conditioning inputs to SAM2 to obtain their respective segmentation masks.
Specifically, we maintain a mapping dictionary between detection tuples and assigned object IDs P tuple + bij_id, which enables us to reverse-mas map SAM2's outputs
e P(tuple)) back to the original detection structure Maupte- The segmentation results then matched to the corresponding grounded predictions and used for reward evaluation.
Reward Design of Rox To enable the model to learn from diverse keyframe selections, design an evaluation reward function, Rx, which assesses both the diversity and quality of the chosen keyframes. This function is formulated as:
Rx A Raiveriity + Xa Raum + RaRaaliency
where the last component, Reatiencys has been clarified the main paper. Here, we detail the first two reward components.
23

---

Firstly, to discourage the model from selecting temporally adjacent keyframes, which can lead redundant System 2 inference, we introduce distribution reward function, Riiversityy This component evaluates the distributional diversity of keyframes by calculating the temporal intervals between them. Specifically, all keyframes are sorted in chronological order. We then compute the temporal interval 4++1 f between each pair of consecutive keyframes. The final Rdiversity value is subsequently determined based on the collection of all such inter-frame temporal intervals.
The diversity reward Rdiversity (S) can be defined as the sum of an overlap punishment term and distribution reward term:
Rdiversity (S) = pherlapppumish |3] + dist reward [D]
where:
8 is the set of selected items. Let 8satted (81, , .M) be the sequence of M [8] items sorted according to the relevant criteria (e.g., timestamps).
is the set of indices for which an "overlap" condition is met between 8; and 81+1- Specifically, assuming idx(s;) gives an identifier for item sy;
J {i fI...., M M-1] idx(s;) idx(s++1))
J] is the number of such identified overlaps (e.g., pairs of consecutive items with identical identifiers)
D is the of indices of items in Swnted that are not considered the start of overlap as defined by J:
D= [j (L....., M} 1i
Therefore, |D] M |3].
overlap_punish is the coefficient for the punishment. For this term to act as a punishment, averlap_punisg should typically be negative value (e.g., -0.2), or if it's positive value, it should be subtracted from the reward.
dist reward is the coefficient for the reward given to items not initiating an overlap.
The formula Rdiversity (8) can also be written
Rdivensity (8) = (overlap_punise - dist_ dist-reward) []] + dist_reward M
Reward Design of RA The specific formulation of RA is follows:
K
RA
1 K
Σ
L Hungarian (B.,: B²,))
where B., denotes the set of predicted bounding boxes at the s--th frame, and B represents the corresponding set of ground truth bounding boxes. The function Hungarian refers to the Hungarian matching loss [45, 61], and K is the total number of selected keyframes.
The Hungarian matching loss Hungarian is computed based on the Intersection-ove--nnii (IoU) between predicted and ground truth bounding boxes. Specifically, a cost matrix M is first constructed using the IoU values between each pair of predicted and ground truth boxes. Then, the Hungarian algorithm applied the negative matrix -M to obtain the optimal one-to-one matching that minimizes the total negative cost, which corresponds to maximizing the overall loU-based matching accuracy.
Reward Design of Rg For Rg, we adopt simple aggregated IoU as the reward function. Specifi- cally, for each detected object, we accumulate the predicted segmentation masks across all objects to construct a per-frame mask set Mc Then, we compute the Intersection-ovee Union (IoU) between the predicted masks and the corresponding ground truth masks Mc on each frame. The final reward obtained by averaging the loU values across all frames.
T
Rg
loU mi)
24

---

Prompt for OMNI-RI as System 1
Given a [frames] seconds video reference instruction: [ref__pe [ref_prompt]] that may involve
temporal behavior, identify the exact object(s) tref_promp- in the video that matches the description.
Select about most relevant moments that contain the referred object(s) with the best view.
Then, simplity identified object into a short and clear visual grounding description that can be used for single-imes image reference at each moment.
Avoid temporal phrases and comparison phrases like "walking". "moving", "approaching". "bigger" or "smaller", but instead describe visible visual cues like clothing, pose, position, or grouping.
Try to select moments that are temporally well-distributed across the video, rather than clustered the same part of the timeline. Avoid selecting multiple timestamps that are adjacent or overlapping: instead, prefer clearly distinct moments that each offer unique visual information. It is better to choose the most relevant and highly representative moments spanning the entire video, rather than picking all from the beginning.
Explain your reasoning in <thinl></think and output the final result in <answer<</answer>
Your final answer should be a JSON object the following format:
<think> your analysis about the video and reference instruction </think> <answer>
{
"start_tai timart_tme;: "00: [start]",
"end_time "00: [end]", " "description": "direct description of referred object(s) at this moment"
</answe>>
Figure 3: Keyframe selection and recaptioning prompt for System 1.
Training and Inference Strategy For video clips, we first feed them into System at a relatively low resolution of per-frame pixel 128 x288x2xx22 x8, which allows to process longer video segments during training and inference. Then System 2 predicts detection results at higher resolution of x0 For VOS tasks, we adopt random uniform sampling strategy during training. selecting between 8 and 24 frames per video to enhance temporal diversity and robustness. All SAM2-based segmentation and reward evaluations are then applied these resampled clips at their original input resolution. For RefAVS tasks, we observed severe cross-modal hallucination issues during preliminary experiments, particularly when reasoning jointly over full-length audio and multi-frame video inputs. To mitigate this, we introduce a simplified variant, RefAID (Referring Audio-Image Detection), which reduces the AVS problem to object detection using only the first video frame and the corresponding full audio query. In this setting, no SAM2 segmentation is used; training is driven solely by detection-base based rewards.
During inference, we resample fixed maximum of 24 frames per video for VOS tasks. Unlike training, segmentation and evaluation are conducted over the full video sequence using SAM2 to align with standard benchmark protocols. For RefAVS, we adopt the same resampling and evaluation procedure as in VOS, ensuring consistency across task settings.
C Ablation Studies
We conduct an ablation study to investigate the effect of progressively designed reward components Rx (keyframe coverage), RA (alignment via Hungarian matching), and Rg (global grounding IoU) on the overall performance. To ensure a fair comparison, all models are trained for one epoch on the ReVOS and MeVIS datasets and are constrained to select exactly four keyframes unless otherwise noted.
Table 4 presents the results across four evaluation subsets of ROVOS: referring, reasoning, single- object, and multi-object. When using the baseline System 2, Rox + Rg achieves the highest overall score (39.9% (39.9%), outperforming the full combination Rxx + R,A + Rg (38.4%). Interestingly, this trend reverses when employing Sa2VA as System 2, where Rx + RA + Rg attains the best overall performance (54.0%). This indicates that while RA may degrade performance with the original
25

---

Prompt for OMNI-RI as System 1
Given a ddration]] audio and a reference instruction: [ref fvef_prompt], which involves temporal and audio-related behavior, first analyze the objects in the image that are producing sound. including both human voices and instrument sounds.
Based on the audio content, identify the exact object [ref_ trem_pr in the image that matches the audio.
Then, simplify the identified object into short and clear visual grounding description that can be unambiguously recognized in single image without relying audio.
Avoid using temporal expressions such as "playing" or "moving"; instead, describe visible visual cues such clothing, pose, position, grouping.
Explain reasoning in <think></think> and output the final result in <answer></answer>
Figure Audio analyzing and recaptioning prompt for System
System 2 due to the locality noises introduced with it, it substantially enhances generalization to alternative System 2 architectures.
We hypothesize that this is because RA is weakly coupled reward, focusing on the alignment be- tween visual features and language prompts the single-frame level. It provides better transferability across different System implementations. In contrast, Rg tends to overfit to the specific structure and behavior of the System 2 used during training, limiting generalization.
These findings validate the effectiveness of Rg as grounding limitations of outcome-basee Rg under two system reinforcement learning settings.
aware aware reward but also highlight the
To assess the impact of architectural choices within our pipeline, perform ablation studies by replacing key components, specifically the grounding and segmentation models, and analyzing their effects on both training and inference performance.
Grounding Model We substitute the original Qwen2.5-Omni-7h System with VisionRea- soner [62], unified visual perception model enhanced through GRPO optimization across grounding, counting, and segmentation tasks. To maintain experimental validity, we keep all training hyperpa- rameters consistent across configurations and test two replacement strategies: (1) replacing the model during training only, and (2) replacing during both training and inference phases.
As seen 5, simply replacing the grounding model during training leads to noticeable gains, suggesting that current approach has further improvements under accurate System A full switch to ResionReasoner during both training and inference yields even greater improvements across most sub-tasks.
Table 4: Ablation study on reward functions Rx. RA and Rg for System We also evaluate Sa2VA System to investigate keyframe performance. The first model is trained with additional 2,000 samples from grounding dataset refcocog.
ReVOS
Method
Referring
Reasoning
Single
Multi
Overall
Omni-R1 refcocog
52.5
36.9
45.0
46.6
44.7
Rx+RA+Rg
44.2
32.5
38.2
41.9
38.4
Rx+
45.5
34.2
39.6
42.6
39.9
Rx+RA
43.1
29.5
36.8
37.5
36.3
Rix
44.1
26.2
36.6
34.1
35.2
Rg
47.3
28.8
38.9
40.4
38.1
Rg Sa2VA
58.1
43.5
50.6
53.5
50.8
Rx + Rg + Sa2VA
60.4
41.8
51.7
51.0
51.1
Rx+RA+Rg Sa2VA
61.8
46.1
54.6
50.7
54.0
26

---

Table 5: Ablation study of different grounding models as System
ReVOS
Method
Referring
Reasoning
Single
Multi
Overall
Original
52.5
36.9
45.0
46.6
44.7
Replace during training
52.2
39.4
45.4
49.2
45.8
Replace also at inference
55.5
41.7
49.1
44.7
48.6
Segmentation Model
In addition, we replace the segmentation model used during inference in 6. Specifically, we switched from SAM2 to TAM [63], a more efficient segmentation tracker, while keeping VesionReasoner as the grounding model (training only).
Table 6: Ablation study of different grounding models as System 2.
ReVOS
Segmentation Model
Referring
Reasoning
Single
Multi
Overall
SAM2 (baseline)
52.2 50.6
39.4
45.4 44.0
49.2
45.8
TAM
39.1
51.3
44.9
D Visualization Results
Mask Quality Since method utilizes the SAM2 model for segmentation, without fine-tuning mask decoder, the final mask output more stable than those methods that rely on additional training on segmentation mask decoder. As be in Figure 5a, in this simple example, method is able to segment the target object with a mask consistent with the ground truth, while Sa2VA predicts the right target but generates mask with holes and noise.
Temporal Reasoning Our System leverages temporal reasoning to improve segmentation aceu- racy. As can be seen in Figure 5b, in this example, one has watch the whole video and analyze the video context to make a correct prediction about the next bottle to be picked up. Our method is able to select the bottle that is about to be picked up, while Sa2VA fails to do and segments the bottle that is already picked up. A similar case is shown in Figure 6a, where one has to leverage world knowledge understanding the target object and our method selects the right object while Sa2VA fails to do so. Both cases show that our method is able to leverage temporal reasoning to improve the segmentation accuracy.
Detail Understanding Our System leverages detail reasoning to improve segmentation accuracy. Figure 5c shows scenario where detail reasoning is needed to figure which wineglass is likely will be finished first. Our System is already able to select the right wineglass but still makes loose description for System 2 to analyze detail information, while Sa2VA fails to understand the instruction and segments all the wineglasses. This shows that our System able to delay detail reasoning for System for detail understanding to improve the segmentation accuracy.
E
More Analysis
During the training of our RefAVS task, we identified significant hallu- cination problem, which we attribute to the complexity of multi-modal video and audio inputs. To systematically evaluate this issue, we conducted targeted assessments on audio-related hallucinations using the JUDGE subset of AVHBench [64], the first comprehensive benchmark designed to evaluate the perception and comprehension abilities of audio-visual large language models (LLMss..
Hallucination Analysis
As shown in Table 8, our base model D0wen2.5-Omni- JUDGE subset. Training on 1600 AVS samples leads to a modest improvement (60.8%). which further enhanced to 61.5% by applying the GRPO KI loss with a reduced coefficient (3 0.001). Notably, increasing the AVS training samples to 10400 does not yield better results, suggesting potential overfitting or task imbalance.
achieves an accuracy of 58.5% on the
27

---

Table Video Object Segmentation performance on MeVIS across different methods, the metric is H&F score(%). I means the results are evaluated where serves as System 1 and Sa2VA-IB as System
MeVIS val_u _
Model
Sa2VA-IB [55]
53.4
Sa2VA-4B [55]
55.4
Wwen2.5-Omni-75
33.6
Ommn-RR-88
55.4
Table Performance on AVHBench (JUDGE subset, total 5302 samples). In the table, AVS tasks are trained on RefAVS dataset and VOS tasks are trained ReVOS and MeViS datasets. The default GRPO KL loss weight 0.04.
AVHBench JUDGE
Method
Correct Answers
Accuracy
Base Model
3100
58.5%
AVS 1600 samples
3222
60.8%
AVS 1600 samples with 3 0.001
3261
61.5%
AVS 10400 samples
3120
58.9%
VOS
3500
66.0%
AVS and VOS
3811
71.9%
On the other hand, training with VOS tasks alone significantly boosts accuracy to 66.0%, and the best performance (71.9%) is obtained by jointly training on both AVS and VOS tasks. This represents a substantial improvement of 13.4% over the base model, demonstrating that multi-task training not only enhances audio-visual grounding but also mitigates hallucination issues more effectively.
These results confirm the effectiveness of leveraging task diversity and balanced reward optimization improving the robustness multimodal reasoning.
Table 9: Performance comparison across different resolutions and the of thinking prompt VideoMME and MVBench. Resolutions are set to either x28x28 28 (default) or 256x28 28 (high). The thinking prompt provides an additional reasoning cue. The reported metric is the average of and F scores (%).
VideoMMM
MVBench Avg
Model
Resolution
Thinking
General
Short
Qwen2.5-Omni
128x28x28
No
58.3
69.8
66.1
Swen2.5-Ommi
256x28x28
No
58.7
69.9
67.0
Qwen2.5-O8 m-mmii
128x28x28
Yes
59.3
70.1
68.1
Qwen2.5-Onm
256x28x28
Yes
59.8
70.9
68.3
Omni-R Ri-A---Av
128x28x28
No
59.0
71.9
68.3
Omni-R O-Ai-RI-AVS
256x28x28
No
59.4
71.9
68.7
Omni-RI-AVS
128x28x28
Yes
59.9
72.1
69.4
Omni-RI-AVS
256x28x28
Yes
60.0
72.1
69.5
Omni-
128x28x28
No
59.7
72.3
68.9
Omni-R1-VOS
256x28x28
No
59.6
72.5
68.9
Omni-R1-VOS
128x28x28
Yes
59.8
72.5
69.8
Omni-RI-VOS
256x28x28
Yes
60.1
72.8
69.9
Rimi-RI-VOS-Ar
128x28x28
No
60.1
72.5
69.1
Amni-RI-VOSSAV?
128x28x28
Yes
60.7
73.0
70.3
28

---

Video Resolution Influence on General Video Understanding Tasks To evaluate the influence of imput resolution and prompting strategy on general video understanding, we compare model performance across different configurations on the VideoMME and MVBench benchmarks, as summarized in Table 9. All models are evaluated under two resolution settings: the default resolution of 128x28x28 x8 and a higher resolution of x (denoted with *), with and without the proposed thinking prompting strategy.
We observe that increasing the input resolution consistently leads to performance gains across all models. For instance, Qwen2.5-Ommi improves from 66.1% % 67.0% on MVBench when evaluated at higher resolution. Similarly, our Omni-R1-AVs model benefits from the resolution increase, achieving a performance gain from 68.3% to 68.7%. These improvements suggest that higher spatial resolution enhances the model's 's ability to capture fine- fine-grained visual details, particularly beneficial for multi-object reasoning and scene comprehension.
In addition to resolution, the thinking prompt designed to guide the model toward structured multi-step reasoning further boosts performance across all tested models. Omni-RI-AVS with thinking achieves 69.4% on MVBench, outperforming its baseline by 1.1%. The combination of both higher resolution and thinking yields the best results overall, with Cimn-RRI-OSS-A VideoMME (general) and 70.3% on MVBench. This indicates that resolution and prompting act as complementary strategies: resolution improves visual precision, while prompting enhances reasoning capability.
+ thinking reaching 60.7% on
However, the performance gain obtained by increasing video resolution is marginal, suggesting that in general understanding benchmarks, resolution plays a limited role, and temporal understanding is more critical than fine-grained spatial details. This observation is consistent with findings in the main paper, where the dual-system design significantly enhances the model's temporal reasoning capabilities and yields notable improvements on reasoning-intensse VOS tasks.
F Limitations and Future Work
Limitations Although our dual-system design significantly enhances the temporal reasoning capa- bility of System 1. the complete functional decoupling between System I and System 2 introduces certain limitations. In particular, System 2 lacks temporal context, which may affect consistency in temporally coherent tasks. This consideration partially motivates our selection of VOS as primary training task: while VOS emphasizes temporal consistency, also provides dense per-frame annotations that allow us design stable training strategies mitigate the context gap-such as frame-wise Hungarian matching loss and aggregated mask-based rewards. During inference, the missing temporal cues in System 2 are partially recovered through SAM2's mask-based processing.
However, when extending to tasks requiring finer temporal sensitivity such as detecting and describing localized anomalous behaviors within specific time span, our current architecture faces new challenges. While System 1 can still progressively narrow down relevant temporal segments, the need for dynamic temporal granularity exposes the limitations of a fully decoupled, one-way reasoning architecture.
Discussion on Future Work Our coarse-to- to-fie reasoning pathway aligns closely with human cognitive intuition, yet we recognize the crucial importance of enabling bidirectional flow of infor- mation between global and local levels. Our current design can be viewed as a context-constraine tree search structure, where only the root node (System 1) has access to full contextual information, while downstream nodes (System 2) operate solely on partial, local input. This diverges from typical hierarchical systems, which often allow child nodes access aggregated information from their parent nodes.
While the VOS task structure inherently compensates for this limitation by providing dense temporal supervision, to further enhance the flexibility of our two-system framework, future work should explore more interactive architectures that facilitate explicit information exchange between System I and System 2. Supporting backtracking within the reasoning tree would empower System to refine its global reasoning using local insights from System 2. Conversely, it would enable System 2 to operate with broader contextual awareness provided by System 1. We firmly believe that transforming this pipeline into bidirectional, cooperative reasoning structure holds immense
29

---

potential for advancing multi-modee modal temporal understanding abilities on more flexible and complex tasks [65. 66].
30

---

Which is white object(s) mode of cellulese that is/ore people s hands?
<think The instruction asks to segment the white object(s) made of cellulese that is/ore hald in people' s hands. The video shows person folding white poper wooden table, The white object question the The person's hands ore visiale they menipulate the paper. The oction of folding the poper the key behovior here, most most relevant duration for this action ("stort_ffe time". "00.00.00 "end time": : "00:00. 50" "description." "white poper being folded")
20
the initial folding
poper c/thinko
Omni-R1 prediction
M
SozVA prediction
(a) In this simple example, both method and Sa2VA able select the right target object, but method able segment target object with mask consistent with the ground truth, while Sa2VA generates a mask with holes and noise.
Which bottle(s) of beveroge will be picked up soor?
ethinlo The instruction asks to identify which bottle(s) of beverage will be picked The video shows of and drinking The beer bottles placed small table front the right picked up
playing suggests of the table the one that will be picked up s/tric/thikke "ordwer": (' L'start_time "00-01.50", 'end_time" 00000..00". "description" "the bottle on the right side of the table")
video,
prodiction
SoZVA
(b) target ohject can only predicted after one watches whole video and makes prediction according to the video context.
Please find the wineglass in which the wine may be finished first?
ethinle The Instruction caks segment winegless which sitting
finished first. The
the gesturing lower right holding winegloss. wine in gloss of the on left oppoors the gloss of the mon the right. Therefore, the winz in the glass of the man on the left likely to be finished first:</tinioo
bottle
wine
"andwar"" ("stant_time time": '00-00.00", timeet "00:00.50" "descniption": "the wineglass in which the wine be finished first")
Omnikki predic
(c) Though our method able to select the right wineglass at System 1, still lets System analyze the detailed information.
Figure 5: Visualization results of our method on three representative VOS cases. Each subfigure illustrates different reasoning pattern.
31

---

Please segment the object(s) that serve az shock absorbers and provide protection?
<thinio The instruction asks to identify the object(s) that shock absorbers and provide protection. In the video, object
pit athlete lands oft long jump. sond is used cushion londing and prevent injury. The nelevont duration from 0.5 1.0 seconds, when the athlete lands in the sand pit.//hinko "onswer" ("start_ffie time". "00:00.50", : "end_time": "00-01.00", "description": "sand pit")
provides
Omni-RI prediction
So2VA prodiction
(a) The case shows the vidco understanding abilities of our method. Our method analyses the function of the sand pit and is able to segment, while Sa2VA fails to do so.
Please segment the object(s) that making a sound of fer the violin. |o|D||||
<thinicIn In the image, there are three musicians: onz playing the violin, one playing the cello, and one playing the guitar. The violin the first instrument be played, the cello the second. The question oaks for the object that positioned left violinist clearly
image, venswer- The cello//onswer>>
Onn k1 prediction
(b) The case shows the vidco temporal understanding abilities of our method. Our method analyses the audio events and finds out the temporal behavior in visual context.
Please segment the object making the fastest rhythm.
|||D|||
<thinleIn the image, there are two main objects producing sound: violinist and pianist. The violinist is the violin, and the pianist playing piana. The violinist likely to making the fastest rhythm due to the likely
of the violin's the
controlled on a stope. violinist wearing long, sparkly dress, and the pianist is weaning block suit. The vollinsst is holding the vialin and bow, and the picnist at the picno. The audio is likely to be focused on the violinist's performance </thinlo vanswer- The molinn/onswes
rhythm very foreground, the pionist is in background,
violinist positioned
Onn-Ri prediction
(c) case shows audio and video understanding abilities our method. Our method analyses the music the video and finds out the object making the fastest rhythm. Figure 6: More visualization results of our method on representative VOS and AVS cases.
32


Reviews:
[{"review_id": "3mATlS3jWE", "reviewer": "Reviewer_vim3", "summary": "This paper introduces Omni-R1, an RL framework built on GRPO training objective for the tasks of Referring Audio-Visual Segmentation and Reasoning Video Object Segmentation. The proposed approach adopts two systems, where the system-1 is responsible for performing global reasoning and output a set of key frames and system-2 is used to perform fine-grained reasoning to generate final answer. Experiments on RefAVS and ReVOS tasks demonstrated the effectiveness of the proposed framework.", "questions": "- For the Key Frame Quality Reward, how to determine the diversity and number of key frames for different videos which are further used to calculate $R_{diversity}$ and $R_{num}$? I think that it is impossible to tell the diversity and key frames number without analyzing the content of videos in varying types and durations.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This paper proposes an effective framework involving GROP to solve the tasks of RefAVS and REVOS.\n- The paper is well-written and clearly organized, making it easy to follow.\n\nWeakness\n- The paper claims that the proposed framework is a omnimodal approach capable of solving video, image and audio tasks. However, the major improvements and contribution lie on the two-stages reasoning on videos.\n- Less training details for the GROP training process, for example the reward curves and rollout number during training.\n- Lack of case studies on the reasoning results of system-1 and system-2, as well as the rollout results in between. Providing these results would help demonstrate the entire process more effectively.\n- Lack of evaluations on image and video understanding tasks.", "was_revised": true, "final_justification": "Most of my concerns, including the training details have been addressed. I will keep my positive score.", "rebuttal": "**W1: Omnimodal Approach**\n\nOne of the central challenges in building an *OmniLLM* lies in handling **long-term temporal reasoning** in combination with **high-resolution fine-grained understanding**. This challenge is especially prominent when integrating modalities like video and audio, which have vastly different token densities: for example, 1 second of audio may require only ~25 tokens, while a single 448×448 video frame typically takes 256 visual tokens. Thus, the core conflict—and bottleneck—resides in *efficiently performing global reasoning over long-range video sequences while preserving detailed understanding at key moments*.\n\nOur dual-system architecture is specifically designed to address this challenge. As mentioned in **L149**, our System 1 can process both **video and audio inputs jointly**. While our method section focuses primarily on video understanding, we also validate the generality of the framework on **RefAVS**, which requires both **audio and visual cues** for temporal localization and grounding. In this task, System 1 learns to reason over the audio and visual sequence and generate global-to-local instructions.\n\nImportantly, our results on RefAVS show that **joint audio-video reinforcement learning improves performance significantly**, demonstrating that our system does not rely solely on vision. Additionally, we show that the same **omnimodal reinforcement learning paradigm** improves performance on **Omnibench**—an image-audio task—further verifying the generality and scalability of our proposed **Omni-R1** framework.\n\nTherefore, while the most substantial improvements are observed on video tasks (which are inherently more complex), our method is **truly omnimodal**, and effective across vision, audio, and multi-modal reasoning scenarios.\n\n**W2: Training Details**\n\nThank you for the suggestion. We will include **reward curves** in the next version of the paper. Due to image restrictions during the rebuttal phase, we provide a brief summary here: \n> During training, the reward component $R_K$ converges quickly and stably, indicating that the model rapidly adapts to the temporal distribution of selected frames. In contrast, $R_A$ exhibits fluctuations around 0.3, which we attribute to inconsistencies in local frame-level grounding. As for $R_G$, the reward shows an oscillating trend throughout training. Notably, high-quality rollout trajectories (where $R_G > 0.5$) ultimately stabilize around 0.8, suggesting effective convergence under high-performing behavior.\n\nRegarding the number of rollouts, we clarify in **L248** that the **group size is 8**, which corresponds to 8 rollouts per update in the GRPO training process.\n\n**W3: Case Studies**\n\nThank you for your suggestion. We will include additional **case studies** in the next version of the paper to illustrate the **outputs of both System-1 and System-2**, as well as the **intermediate results from the rollout process**.\n\nWhile we are currently unable to add new figures during the rebuttal phase, we kindly refer you to **Appendix Figures 3 and 4**, which already show examples of the reasoning behaviors of both systems. These examples provide an initial demonstration of how System-1 and System-2 work together to perform multimodal reasoning.\n\n\n**W4: Evaluations on Image and Video Understanding Tasks**\n\nThank you for the suggestion. In the main paper (Table 3), we evaluate our model on **video understanding benchmarks** such as **VideoMME** and **MVBench**, as well as the **OmniBench** benchmark for image understanding.\n\nTo further assess image-level understanding capabilities, we additionally evaluate on **MME**[1] and **DriveBench**[2]. These results support the generality of our framework across both video and image modalities.\n\n\n**MME Results:**\n\n| Model          | Total Score | Perception Score | Cognition Score |\n|----------------|-------------|------------------|------------------|\n| Qwen2.5-Omini  | 2071.4      | 1556.8           | 514.6            |\n| **Omni-R1**     | **2100.2**  | **1572.0**       | **528.2**        |\n\n**DriveBench Results (Behavior Accuracy):**\n\n| Model           | Behavior Accuracy |\n|-----------------|-------------------|\n| LLaVA 1.5-7B    | 0.100             |\n| LLaVA-NeXT      | 0.230             |\n| Qwen2VL-7B      | 0.300             |\n| Qwen2.5 Omni-7B | 0.330             |\n| **Omni-R1**     | **0.345**         |\n| DriveLM         | **0.440**         |\n\nWe would like to note that **DriveLM is specifically trained on driving-domain data**, while **Omni-R1 is trained in a general-purpose manner** without any driving-specific fine-tuning. Despite this, Omni-R1 still achieves competitive performance across domains. Importantly, compared to the base **Qwen2.5 Omni-7B**, **Omni-R1 consistently improves performance on both tasks** (e.g., MME and DriveBench), demonstrating the effectiveness of our reinforcement learning–based optimization strategy.\n\n\n**Q1: $R_{diversity}$ and $R_{num}$**\n\nThank you for the thoughtful question. We would like to further clarify the design and intent behind the two keyframe quality rewards:\n\n- **$R_{diversity}$** is designed to encourage the selection of **non-overlapping and semantically diverse video segments** as keyframes. It penalizes redundant or closely clustered selections. This formulation is **generalizable across different types and durations of videos**, as it operates purely on the segment selection patterns without requiring prior content-specific assumptions.\n\n- **$R_{num}$** We acknowledges that different videos may require different numbers of keyframes. Therefore, we do **not enforce a fixed number** of selected keyframes. Instead, we define a **soft constraint** via two thresholds: $K_{\\text{min}}$ and $K_{\\text{max}}$. A penalty is applied only when the number of selected keyframes falls outside this range. The model then **learns to select the optimal number** of keyframes based on downstream rewards $R_A$ and $R_G$.\n\nThis design enables the system to **adaptively balance diversity and sufficiency** of information while remaining general-purpose and content-agnostic.\n\n**Reference**\n\n[1] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models\n\n[2] Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives"}, {"review_id": "vv8WFNZNi6", "reviewer": "Reviewer_7n6f", "summary": "The core contributions of this work are embodied in its ​​two-system collaborative architecture​​ based on  Qwen2.5-Omni, which decouples ​​long-horizon reasoning​​ from ​​fine-grained pixel understanding​​ via reinforcement learning. By leveraging hierarchical rewards and simulated collaboration between systems, this study demonstrates that reinforcement learning can effectively optimize keyframe selection and task reformulation in omnimodal settings, even with minimal training data.\nThe contributions of this work include:\n​​(1) Scalable Two-System Architecture​​\nA ​​Global Reasoning System (System 1)​​ for efficient long-range temporal reasoning and keyframe selection.\nA ​​Detail Understanding System (System 2)​​ for high-resolution visual grounding, enabling precise segmentation.\n​​(2) End-to-End Reinforcement Learning Framework​​\nBuilt on ​​Group Relative Policy Optimization (GRPO)​​, requiring only ​​one epoch of RL training​​ on small datasets.\nHierarchical rewards (​​keyframe quality, instruction alignment, global consistency​​) guide policy optimization.\n​​(3) State-of-the-Art Performance & Generalization​​\nOutperforms specialized models (e.g., Sa2VA, EEMC) on ​​RefAVS​​ and ​​REVOS​​ benchmarks.\nImproves ​​out-of-domain generalization​​ in both video and multimodal understanding tasks.", "questions": "According to my understanding (from lines 237 to 250), in the few-shot  rl training, system1 is open and system2 (including sam2) is frozen. However, according to the results in Table 2, the best combination is system1+sa2va, and the paper does not seem to explain how to combine them. I guess that this step uses system1's qwen2.5omni only as a tool for accurately and reasonably extracting frames, and sends these key frames to sa2va, which can achieve such good performance. I am more concerned about the details of this step, because it seems that sa2va does not participate in the training as system2. What caused this improvement in the performance of this combination?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This work proposes a coarse-to-fine reasoning system and design three appropriate reward functions to stimulate the grpo system's understanding and reasoning ability.\n2. The method proposed in this work achieves great performance on these types of benchmarks： referring audio-visual segmentation 、referring video object segmentation、general understanding QA benchmark.\n\nWeaknesses:\n\n1.This work verifies the superiority of the architecture design on many different tasks, but lacks some ablation experiments on the model in the architecture itself, such as replacing the grounding model or segmentation model. These experiments can further verify the reliability of the method.", "was_revised": true, "final_justification": "All my concerns have been fully addressed; the paper is ready for acceptance.", "rebuttal": "**W1: Architecture-Level Ablations**\n\nWe appreciate the reviewer’s suggestion. To evaluate the **impact of architectural choices** within our pipeline, we conduct ablation studies by replacing both the **grounding model** and the **segmentation model**, examining how these changes affect training and inference.\n\n\n**Grounding Model Replacement** \nWe replaced the original **Qwen2.5-Omni-7B** with **VisionReasoner**[1], a unified model for visual perception tasks enhanced via GRPO across grounding, counting, and segmentation. To ensure fair comparison, we aligned all training hyperparameters and tested two settings:\n\n- Replace only during training\n- Replace during both training and inference\n\nResults on **ReVOS** are shown below:\n\n| Method                                  | Referring | Reason | Single Object | Multi Object | Overall |\n|----------------------------------------|-----------|--------|----------------|---------------|---------|\n| Omni-R1 (original)                     | 52.5      | 36.9   | 45.0           | 46.6          | 44.7    |\n| Replace grounding model (training only)| 52.2      | 39.4   | 45.4           | 49.2          | 45.8    |\n| Replace grounding model (full)         | **55.5**  | **41.7** | **49.1**     | 44.7          | **48.6**|\n\nAs seen, simply replacing the grounding model during training leads to noticeable gains. A full switch to VisionReasoner during both training and inference yields even **greater improvements** across most sub-tasks.\n\n**Segmentation Model Replacement** \nWe further replaced the **segmentation model** used during inference. Specifically, we switched from **SAM2** to **TAM** [2], a more efficient segmentation tracker, while keeping VisionReasoner as the grounding model (training only).\n\n| Segmentation Model | Referring | Reason | Single Object | Multi Object | Overall |\n|--------------------|-----------|--------|----------------|---------------|---------|\n| SAM2 (baseline)     | 52.2      | 39.4   | 45.4           | 49.2          | 45.8    |\n| TAM                | 50.6      | 39.1   | 44.0           | **51.3**      | 44.9    |\n\nAlthough the **overall score drops slightly (-1.1%)**, TAM shows **better performance on multi-object tracking**, indicating its advantage in handling complex scenarios.\n\n---\n\n**Conclusion:** These results validate the **robustness and generalizability** of our architecture. Stronger grounding or segmentation models can **further enhance performance**, and our pipeline remains effective under architectural variations.\n\n\n**Q1: Integrating Sa2VA as System 2**\n\nYes, your understanding is correct. The purpose of this experiment is to demonstrate that **System 1 and System 2 are modular and can be decoupled**—replacing System 2 with a stronger module like **Sa2VA** leads to better performance.\n\nAlthough **Sa2VA was not involved in our reinforcement learning training**, it naturally fits the role of System 2. In its original setting, Sa2VA takes as input either the first five frames of a video or five frames uniformly sampled from the whole video, along with the original global instruction.\n\nIn our setup, we leverage **System 1 to select five key frames** and **generate localized, simplified instructions** for each segment. These keyframes and instructions are then fed into Sa2VA for fine-grained segmentation.\n\nThis decoupled design allows System 1 to **fully exploit its strength in global video reasoning**, while Sa2VA focuses on **precise, local-level understanding**. As shown in Table 2, this integration yields improved performance **without any modification to Sa2VA’s parameters or architecture**, further validating the effectiveness of our modular two-system design.\n\n**Reference**\n\n[1] Liu, Y., Qu, T., Zhong, Z., Peng, B., Liu, S., Yu, B., & Jia, J. (2025). VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning. arXiv preprint arXiv:2505.12081.\n\n[2] Xiong, Y., Zhou, C., Xiang, X., Wu, L., Zhu, C., Liu, Z., ... & Chandra, V. (2024). Efficient track anything. arXiv preprint arXiv:2411.18933."}, {"review_id": "MiI37vjr5i", "reviewer": "Reviewer_zCqj", "summary": "This paper addresses the challenge that traditional models struggle to balance global temporal reasoning and local detail understanding simultaneously, proposing a two-system architecture: a Global Reasoning System is trained via GRPO, and then tasks are handed over to an additional downstream task-specific system. The approach demonstrates excellent performance in audio-visual segmentation and video understanding tasks.", "questions": "1. One of the core contributions highlighted in the study is the improvement in out-of-domain generalization. Does System 1 still demonstrate performance gains in scenarios with more significant distribution shifts compared to the training data, such as autonomous driving or medical domains?\n\n2. The design of the Key Frame Quality Reward appears to be influenced by specific data distributions. Will this affect its application in joint training with more diverse data distributions?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The problem addressed in this paper holds significant application value, enabling remarkable performance improvement with minimal data usage.\n2. The paper features thorough and well-documented experiments, ensuring both readability and scientific rigor.\n\nWeaknesses\n1. There is an error in the experimental data, and the delta values in Table 1 are incorrect.\n2. The paper does not discuss the relationship between data scale and RL-driven improvements in target application scenarios. Adding this discussion would help assess the need for labeled data in practical applications.", "was_revised": true, "final_justification": "All of my initial concerns have been addressed. I will keep my rating.", "rebuttal": "**W1: Correction of Δ Values**\n\nThank you for pointing this out. We acknowledge the error in the originally reported delta values and have corrected them accordingly. The revised table is shown below:\n\n| Model                  | Seen (J&F) | Seen (J) | Seen (F) | Unseen (J&F) | Unseen (J) | Unseen (F) |\n|------------------------|:----------:|:--------:|:--------:|:-------------:|:-----------:|:-----------:|\n| Qwen2.5-Omni-7B (SFT)  | 39.1       | 35.4     | 42.8     | 66.2          | 63.1        | 69.3        |\n| Omni-R1-7B             | 47.2       | 43.0     | 51.4     | 74.2          | 71.3        | 77.0        |\n| Δ (corrected)          | **+8.1**   | **+7.6** | **+8.6** | **+8.0**      | **+8.2**    | **+7.7**    |\n\nWe will correct this in the next version of the paper. Thanks again for your careful reading.\n\n\n**W2: Data Scale**\n\nWe believe the relationship between data scale and RL-driven improvements can be analyzed from two perspectives:\n\n**Impact of Data Volume on a Single Task**\n\nWe conducted an experiment on **RefAVS**, comparing models trained with **1600** and **10400** samples. The results are shown below (see Appendix Table 3 for more details):\n\n| Training Samples | RefAVS(F) Seen | RefAVS(F) Unseen | AVHBench Acc |\n|------------------|----------------|------------------|---------------|\n| 1600             | 51.35          | 76.98            | 60.77%        |\n| 10400            | 54.49          | 77.63            | 58.85%        |\n\n\nWe observe that **increasing data size improves task-specific metrics**, particularly in the seen subset. However, it may also introduce slightly **higher hallucination rates**, as reflected in AVHBench accuracy.\n\n \n**Multi-Datasets Joint Training**\n\nWe further investigate the effect of **multi-task data mixing**, especially combining **ReVOS** and **RefAVS** datasets.\n\nAs shown in Appendix Table 4, training on both datasets leads to consistent performance gains even on **ReVOS**—which was not directly targeted during joint training:\n\n- ReVOS overall score: **44.7 → 47.6** (**+2.9**)\n\nThis demonstrates that **RL-based joint training across tasks** and distributions can enhance general video understanding capabilities, without requiring dataset-specific tuning.\n\n**Q1: Out-of-Domain Generalization**\n\nThank you for raising this important point. To evaluate **out-of-domain generalization**, we tested both **Qwen2.5 Omni-7B** and **Omni-R1** on the **DriveBench**[1] dataset, which targets the autonomous driving domain—a distribution notably different from our training data.\n\nThe **behavior prediction task** in DriveBench requires the model to predict future steering angles and vehicle speeds based on multi-view images (from six camera directions). It is worth noting that **DriveLM**, the strongest baseline, is specifically fine-tuned on large-scale driving data.\n\nWe evaluate the **behavior accuracy** metric (as it does not depend on GPT APIs for scoring):\n\n| Model              | Behavior Accuracy |\n|-------------------|-------------------|\n| LLaVA 1.5-7B       | 0.100             |\n| LLaVA-NeXT         | 0.230             |\n| Qwen2VL-7B         | 0.300             |\n| Qwen2.5 Omni-7B    | 0.330             |\n| **Omni-R1**        | **0.345**         |\n| DriveLM (oracle)   | 0.440             |\n\nWhile **DriveLM** benefits from domain-specific fine-tuning, **Omni-R1 is trained in a general-purpose fashion** without any driving-specific data. Despite this, it still **outperforms its baselines**(**Qwen2.5 Omni-7B**), demonstrating clear improvements in **out-of-domain generalization** brought by our reinforcement learning framework.\n\n**Q2: Key Frame Quality Reward**\n\nThank you for the insightful question. We would like to clarify that the underlying principles of the *Key Frame Quality Reward* are generally applicable across video tasks, although the specific hyperparameter settings may vary depending on the dataset and task characteristics.\n\nSpecifically:\n\n1. **$R_{\\text{diversity}}$** encourages the model to select non-overlapping segments as keyframes, promoting diversity. This reward is **task-agnostic** and generally useful across different video scenarios.\n\n2. **$R_{\\text{num}}$** softly constrains the number of selected keyframes. We acknowledge that different videos may require different numbers of keyframes, so we do **not** enforce a fixed value. Instead, we define a reasonable range bounded by $K_{\\min}$ and $K_{\\max}$, and penalize the model only if it selects too few or too many frames. The optimal number of keyframes is **learned implicitly** through the downstream rewards $R_A$ and $R_G$. When training on multi-domain data, we simply need to ensure that the range defined by $K_{\\min}$ and $K_{\\max}$ is sufficiently broad.\n\n3. **$R_{\\text{saliency}}$** is only applied when ground-truth object-level annotations (e.g., masks or boxes) are available, such as in tasks like RefAVS or ReVOS. For high-level video understanding tasks that lack such annotations, we set its weight $\\lambda_3$ to zero.\n\nIn summary, only **$R_{\\text{num}}$** requires mild adjustment when applied to joint training over diverse datasets. The other two reward components remain broadly applicable and robust across various settings.\n\n\n**Reference**\n\n[1] Xie, S., Kong, L., Dong, Y., Sima, C., Zhang, W., Chen, Q. A., ... & Pan, L. (2025). Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives. arXiv preprint arXiv:2501.04003."}, {"review_id": "pG9TNP59Gz", "reviewer": "Reviewer_4Nax", "summary": "The paper presents a novel two-system architecture (Global Reasoning System and Detail Understanding System) that effectively addresses the trade-off between long-horizon temporal reasoning and fine-grained spatial understanding. GRPO is leveraged to select the key frames. The paper demonstrates strong performance on challenging benchmarks like Referring Audio-Visual Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS). The motivation and idea are easy to follow. The writing is good.", "questions": "Please refer to [Weaknesses].", "limitations": "Please refer to [Weaknesses].", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n* Leveraging GRPO to re-formulate the key frames selection is interesting. \n* The performance is good.\n\n[Weaknesses]\n* Leveraging GRPO to re-formulate the key frames selection is interesting. I wonder why the authors conduct this idea in the Referring Video Segmentation Task? For long-video understanding tasks, selecting key frames is important. However, referring video segmentation needs to segment objects per frame. Why is it still necessary to select key frames? Moreover, how to obtain the segmentation mask in the frame that is not selected by system 1.\n* The authors claim that the proposed method addresses the trade-off between long-horizon temporal reasoning and fine-grained spatial understanding. However, this method increases the computational complexity with systems 1 and 2. With a similar computational complexity, we can increase the frame resolution or sample rate to improve the performance. More discussions are needed.\n* The proposed Hierarchical Reward Design for System1 is an important part of this paper. However, it seems that $R_{\\mathcal{A}}$ decreases the performance. If it is useless, why do you still include it in the main paper? Moreover, it is important to show the performance with $R_{\\mathcal{G}}$ alone.", "was_revised": true, "final_justification": "My concerns are solved.", "rebuttal": "**W1: Motivation for Using Referring Video Segmentation (RVS)**\n\n---\n**W1.1 Why Referring Video Segmentation Task?**\n\nThe Referring Video Segmentation (RVS) task can be naturally decomposed into three key components:\n\n1. **Understanding complex user instructions and identifying the referred object across the video**, which requires the model to perform global temporal reasoning over the entire video.\n2. **Once the object is identified**, the model needs to **accurately localize and segment it in high-resolution frames**, requiring strong fine-grained spatial understanding.\n3. **Propagating segmentation masks from sparse keyframes to the entire video** via object tracking, which is more about low-level visual perception and can be handled by off-the-shelf tools like SAM2—thus not the focus of this paper.\n\nThe first two components align well with the challenge we highlight in the Introduction: **the trade-off between long-horizon temporal reasoning and fine-grained spatial understanding**. In contrast, many standard long-video understanding tasks primarily require global reasoning and often **do not demand such precise spatial understanding**. In those cases, System 1 alone may be sufficient to handle the task. Therefore, RVS provides a more rigorous testbed for our proposed two-system architecture.\n\n**W1.3 How Are Masks Generated for Non-Key Frames?**\n\nFor frames not directly selected by System 1, we adopt **SAM2** as a tracking and propagation module. SAM2 is a strong visual foundation model capable of propagating object masks across time based on the sparse predictions on keyframes.\n\nThis approach follows the **standard paradigm** in RVS literature: high-cost grounding and segmentation are performed on sparse frames, and light-weight temporal propagation is used to fill in the rest. Our contribution lies in **learning to select the most informative keyframes**, which significantly boosts both efficiency and segmentation consistency, especially under challenging conditions like motion blur, occlusion, or scene transitions.\n\n**W1.2 Why Is Key Frame Selection Still Necessary?**\n\nGiven the above two premises, the necessity of selecting key frames can be justified from two main perspectives:\n\n1. **Grounding models lack long-range temporal reasoning**: Modern vision-language (grounding-capable) models models (e.g., Sa2VA,Qwen2.5VL,Qwen2.5Omni) excel at precise localization in static images but struggle when provided with long, complex video sequences and ambiguous or complex instructions. If we directly input the full video and a temporally complex prompt, these models struggle to identify the correct object. In contrast, by leveraging the **global reasoning ability of System 1**, we can first select informative key frames and reformulate the instruction into **simpler, localized prompts**.  As a result, the grounding model only needs to perform **fine-grained segmentation on single high-resolution frames** guided by short and unambiguous instructions.\nThese two steps—**key frame selection** and **instruction simplification**—are **strongly coupled**, and together enable the model to more effectively fulfill the **first two components described in W1.1**.\n\n\n2. **More intelligent than rule-based sparsity**: Prior works like Sa2VA also rely on sparse frame segmentation followed by mask propagation. However, their frame sampling is rule-based (e.g., uniformly sampled or first five frames), lacking flexibility and potentially leading to **informational loss**. In contrast, our System 1 can **adaptively select frames with large semantic transitions**, such as those involving **object occlusion, reappearance, or camera scene cuts**. These challenging segments are often the **hard cases** for standard tracking models. By letting System 1 select such frames for precise grounding by System 2, we can significantly **reduce the tracking burden and overall segmentation error**.\n\n---\n\n**W2: Computational Complexity**\n\nWe would like to clarify that the **trade-off between long-horizon temporal reasoning and fine-grained spatial understanding** primarily affects the **training phase**, rather than inference. \n\nAt inference time, increasing frame resolution or sampling rate is indeed a straightforward way to enhance performance. However, if the model **has not been trained with long sequences and high-resolution inputs**, it will lack the necessary capacity for temporal and spatial reasoning—thus limiting the benefit of high-resolution inference. A concrete example can be found in **Appendix Table 3**, where merely increasing input quality does not close the performance gap without structural support.\n\nIn our framework, we decouple training as follows:\n\n- **System 1** operates on inputs with resolution **100×28×28**, with a **maximum of 24 frames**, and is trained end-to-end.\n- **System 2** does **not require training or backpropagation**, and handles only **5 high-resolution frames (1200×28×28)**. Furthermore, because System 2 processes frames independently during inference, we leverage the **vLLM engine** to parallelize its execution efficiently.\n\nIf we were to train the whole pipeline end-to-end using high-resolution frames (e.g., 1200×28×28), the compute budget would limit the number of frames to around **5**, making it impossible to learn long-range video reasoning. This constraint directly limits global temporal understanding. \n\nAs a concrete example, **Sa2VA**, an end-to-end trained model, only supports 5 frames for both training and inference—confirming that **a monolithic approach struggles with long-horizon temporal modeling**.\n\nThus, our proposed two-system architecture offers a **computationally efficient** and **scalable** way to address this trade-off, enabling both global video reasoning and fine-grained visual grounding.\n\n**W3: Reward Design**\n\nThank you for raising this point. We have conducted additional experiments to more comprehensively evaluate the effectiveness of our hierarchical reward design.\n\nAs discussed in the paper, the grounding-based reward $R_G$ is **strongly coupled with System 2**, meaning it depends heavily on the specific outputs of a particular System 2 model. This tight coupling can lead to reduced generalization when switching to a different System 2. Although adding $R_A$ to （ $R_G$ and $R_K$ ） may slightly reduce performance under the same System 2, it **significantly improves generalization** to other System 2 implementations.\n\nTo validate this, we perform ablation studies where we **replace System 2 with Sa2VA** at inference time and evaluate how different reward combinations affect final performance:\n\n| Reward Setting                       | Referring | Reason | Single Object | Multi Object | Overall |\n|---|---|---|---|---------------|---------|\n| $R_{G}$                             | 47.31     | 28.79  | 38.93          | 40.39         | 38.05   |\n| $R_{G} + R_{K}$                     | 45.50     | 34.20  | 39.60          | 42.60         | 39.90   |\n| $R_{G} + R_{K} + R_{A}$             | 44.20     | 32.50  | 38.20          | 41.90         | 38.40   |\n| $R_{G}$ + **Sa2VA**                 | 58.12     | 43.45  | 50.57          | 53.53         | 50.79   |\n| $R_{G} + R_{K}$ + **Sa2VA**         | 60.37     | 41.83  | 51.70          | 51.02         | 51.10   |\n| $R_{G} + R_{K} + R_{A}$ + **Sa2VA** | **61.84** | **46.14** | **54.59**     | **50.68**     | **53.99** |\n\nThe results show that:\n\n- When using Sa2VA as System 2, adding $R_A$ leads to **notable performance improvements**, especially on the **Reasoning** sub-task.\n- This is because $R_A$ is a **weakly coupled reward**, focusing on the alignment between visual features and language prompts at the single-frame level. It provides **better transferability** across different System 2 implementations.\n- In contrast, $R_G$ tends to overfit to the specific structure and behavior of the System 2 used during training, limiting generalization.\n\nIn summary, even if $R_A$ shows limited benefit in fixed setups, it plays an important role in **enabling general-purpose and modular policy learning**, which is essential for a scalable architecture."}]

Author rebuttal:

Meta-review: In this work, the authors present Omni-R1, a two-system RL algorithm for Omnimodal Reasoning. Omni-R1 contains a global reasoning stage to select informative keyframes, and a detailed understanding stage for high-resolution inputs. The two-system architecture is trained via GRPO with specially designed reward functions. The authors provide the experimental results on the RefAVS and REVOS tasks, and the results demonstrate the effectiveness of the proposed method.

All reviewers agree that this work's contribution is strong with the two-system collaborative architecture and GRPO-based RL training.
- Citing Reviewer zCqj (score 5): The problem addressed in this paper holds significant application value, enabling remarkable performance improvement with minimal data usage.
- Citing Reviewer 7n6f (score 4): This work proposes a coarse-to-fine reasoning system and design three appropriate reward functions to stimulate the grpo system's understanding and reasoning ability.

The majority of the reviewers' concerns were the missed implementation details or limited ablation studies. In the author-reviewer discussion phase, the authors address most of the concerns; there are a few additional results that I feel particularly made the work stronger:
1. additional training details for the GROP training process. 
2. more ablation experiments about architectural designs, 
3. additional results on self-driving datasets, which demonstrate the out-of-domain generalization ability.
4. Results and further analysis of the reward design.

Overall, this work is strong and received positive feedback from all reviewers, and I recommend to accept. I clearly see its value to present to the NeurIPS community.
Decision: Accept (poster)