CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos
Authors: Shiu-hong Kao
Yu-Wing Tai
Chi-Keung Tang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Model
Reasoning Video Object Segmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Reasoning Video Object Segmentation is a challenging task, aiming at  generating a mask sequence from an input video given a complex and implicit text query. While existing works finetune Multimodal Large Language Models (MLLM) for the task, they still fail in video inputs given complex temporally-sensitive queries, indicating their lack of temporal and spatial integration in complex scenarios. In this paper, we propose **CoT-RVS**, a novel framework employing the zero-shot Chain-of-Thought (CoT) capability of MLLM to address these complex challenges by **temporal-semantic reasoning**: CoT-RVS analyzes the visible objects within a given frame that possibly match the language query (semantic), and chooses a corresponding keyframe for each object that can be observed effortlessly among all frames (temporal). Notably, the CoT-RVS framework is training-free and compatible with closed-source MLLMs, which can be applied to Reasoning Video Instance Segmentation. Our framework's training-free feature further allows its extension to process online video streams, where the CoT  is used at test time to update the object of interest when  a better target starts to emerge and becomes visible. We conduct extensive experiments on video object segmentation with explicit and implicit queries. The results show that CoT-RVS significantly outperforms previous works in both cases, qualitatively and quantitatively.

Published as a conference paper at ICLR 2026
COT-RVS: ZERO-SHOT CHAIN-OF-THOUGH REASON- ING SEGMENTATION FOR VIDEOS
Shiu-hong Kao
Yu- Yu-Wing Tai Dartmouth College
The Hong Kong University of Science and Technology, National University of Singapore shkao@u. nus .uu.
yu-wing ta retddrrtmutheee edu
Chi-Keung Tang
The Hong Kong University of Science and Technology cktang@cs. uuu.th hk
ABSTRACT
Reasoning Video Object Segmentation is challenging task, aiming at generating a mask sequence from an input video given a complex and implicit text query. While existing works finetune Multimodal Large Language Models (MLLM) for the task, they still fail in video imputs given complex semporally-sensiteee indicating their lack of temporal and spatial integration in complex this paper, we propose COT-RVS, a novel framework employing the zero-shot Chain-of-Thought (CoT) capabilitt of MLLM address these complex challenges by temporal-semant reasoning: CoT-RVS analyzes the visible objects within given frame that possibly match the language query (semantic), and chooses a corresponding keyframe for each object that can be observed effortlessly among all frames (temporal). Notably, the CoT-RVS framework is training- free and compati- ble with closed-source MILMs, which can be applied to Reasoning Video Instance Segmentation. Our framework's 's training-fres feature further allows its extension to process online video streams, where the CoTT is used at test time update the object interest when a better target starts to emerge and becomes visible. We conduct extensive experiments on video object segmentation with explicit and implicit queries. The results show that CoT-RVS significantly outperforms previous works in both cases, qualitatively and quantitatively.
queries, scenarios. In
INTRODUCTION
Reasoning Video Object Segmentation (Reasoning VOS) presents a formidable challenge in the field of computer vision, requiring the generation of mask sequence from input video alongside an implicit and often complex text query (Yan et al. 2024: Bai et al., 2024). Unlike traditional vision-language tasks that directly associate visual data with textual descriptions, Reasoning VOS requires more advanced cognitive capabilities due to the dynamic nature of video data, where factors such temporally sensitive queries, occlusions and disocclusions due rapidly moving object can complicate the segmentation process.
Existing approaches of reasoning segmentation focus on finetuning the multimodal Large Language Models (MLLMs) for segmentation-ban based tokens. While achieving good results in Reasoning VOS, the finetuning process time consuming, and they still suffer from low accuracy when temporal reasoning is required. An implicit and time sensitive query, "Which player makes three-point shot this baskethall game?", for instance, is more challenging than "Which player wearing black?" (ig.. Tp. A query is time or temporally sensitive if the relevant video objects with the specified action context (e.g., three-point shot), do not appear uniformly over time but rather are concentrated at certain intervals, requiring high-level temporal reasoning to correctly find them to the satisfaction of the query requirement. Even though reasoning segmentation has achieved impressive performance on images (Lai et al. 2023 Kao et al. [022 Liu et al. (025) and videos (Bai et al.) (2024). current results are far from perfect in the video domain, showcasing failure integrating temporal information with spatial and textual context in satisfying complex and time-sensitive queries.

---

Published as a conference paper at ICLR 2026
Which player makes three-point shot in this basketball game?
What can seen the frame?
Has the ball been shot yet?
Hlw many and which players are near the three-point line?
Where the ball??
Are there any players taking a shot?
CoT-RVS
Any other players obstructing the view?
Output list: Hobject index: 1, keyframe: object description: "player in black uniform the right side three-point are shooting posture"]]
Figure I: CoT-RVS is a novel framework for Reasoning Video Segmentation utilizing the zero-shot Chain-of-Thoughe (CoT) capability of pretrained multimodal Large Language Models, and correctly segments the fast-moving player right before, during, and after his three-point shot after CoT, given a temporally sensitive query "Which player makes three-point shot in this basketball game?" which requires both spatial and temporal reasoning.
To address this challenge, this paper aims at injecting stronger temporal reasoning ability into segmentation framework through diversely pre-trained MLLMS empowered by zero-shot Chain- of-Thoughts (Wei et al. 2022). We enable new model capabilities through carefully designed tasc-specific prompts and innovative module interaction, a visionary trend also shared by Guo et al.) (2022); Lian et al. (2023); Hu et al. (2023). Thus, we propose CoT-RVS, a novel, modular framework, whose key success lies in its exceptional reasoning capability on the lemporul-semante correlation across keyframe candidates,
example in Fig. 2] At inference time, CoT-RVS analyzes the visible objects within a given frame that possibly matches the language query (semantic), and chooses corresponding keyframe for each instance of interest, where the object can be observed with the least effort among all frames (temporal).
Thus, CoT analysis requires complex, human-level reasoning within and among different video frames, e.g., "Frame 3 and 4 contains the player in shooting posture, which satisfies the user query, while frame 3 is a better keyframe with a clearer view.". We will show in this paper that accurate, in-depth reasoning for this temporal-semanp correlation will significantly improve the model performance processing time sensitive query.
One of the other standout features of the CoT-RVS framework is its training-free nature, mak- ing compatible with open-source and closed-source MLLMs. We leverage the state-of-the-ard Gemma3 (Team et al., 2025) and GPT-4o (Hurst et al. (024)) as the "thinking" agent in our CoT- RVS framework to generate keyframe selectivity. To evaluate the effectiveness of CoT-RVS, we conduct comprehensive experiments on referring VOS datasets (e.g., MeViS (Ding al.) 2023a and Refer-DAVIS-1 (Pont- Tuset et al. (017)) and reasoning VOS datasets (e.g., ReVOS (Yan al. 2024) and Reason VOS (Bai al. (2224). We demonstrate that CoT-RVS significantly outperforms existing referring and reasoning VOS approaches, qualitatively and quantitatively, with its more general formulation based Reasoning Video Instance Segmentation (Reasoning VIS), which can output multiple instance-level mask sequences to satisfy the user query. In addition, we propose a variant version of CoT-RVS for online video streams, where CoT is used to update the target object when a new object starts to appear in the middle of the video that aligns better with the text query.
In summary, contributions include:
2

---

Published as a conference paper at ICLR 2026
Which cat shows a slowdown in movement, possibly
Indicating a loss of interest the cut toy?
VISA:
CoT-RVS:
the white cat seated at the bottom, looking towards the tay with less enthusinsm
Figure 2: Keyframe selection is particularly challenging when processing time sensitive queries. Given a video, where two cats are playing with the cat teaser the beginning. and then the white cat stays static in the second half of the video, existing methods e.g., VISA (Yan et al., 2024) fail to find proper keyframe and directly use the user prompt for reasoning segmentation. In contrast, CoT-RVS extracts temporal-semante correlation from the input video and successfully outputs a more reasonable keyframe with detailed description to the target object.
A modular, zero-shot reasoning VOS framework that coordinates MLLMs, image seg- menters, and video processors in both offline and online settings, without any training or fine-tuning, unlike prior methods.
A keyframe selection pipeline based on CoT prompting, specifically tailored for temporal reasoning, where MLLMs are required to both localize and describe scene-relevant frames, beyond simple object retrieval.
An online reasoning extension that adaptively re-selects keyframes at inference time, feature rarely explored in existing R-VOS systems, enabling streaming video processing.
RELATED WORK
Video segmentation. Video object segmentation (VOS) tracks objects in a given video and output corresponding mask sequences. Early VOS approaches (Maninis et al., 2018 Caelles et al. 2017 17) rely on online first-frame finetuning, thereby constrained to specific applications because the high time
cost. To speed up inference, follow-up works adopt efficient online learning algorithm (Robinson)
al.
2020; Bhat et al., 2020), MRF graph inference (Bao et af.) 2018), temporal CNN (Xu et al.
2019
Hou et al. 2019), capsule routing (Duarte et al., 2019), tracking (Jang & Kim, [017 Wang et al.
2019
Perazzi et al. 2017; Chen et al., 2020). embedding learning (Yang et al. 2020: Voigtlaender
eet al.
2019;
et al. 2018) and space-time time matching (Hu et al. 2018 Oh et al. 2019a] Cheng
cet al.
2021b). Recent VOS methods allow various control modalities, such as categories Cheng
eet al.
2021a; Wang al. [2021] Wu et al. 2022d Zhang al. [023c] 2025 Zhou et al.
2024).
points (Ravi et al. 2024 Rajič et al. 2025 Homayounfar et al. 2021) 1), segmentation masks CCeeg
& Schwing 2022 Cheng al. 2018 Oh et al.
2019b] Perazzi al. 2017 Ravi et al. 2024] text
query (Botach et al. [002aa] Ding et al. 2023a
Wu et | 2023b Seo et al.
2020 Ye et al. 2019
al. 2023b; Wu et al.] 2022b; Miao al. 2023a Cuttano al. 2024). making the segmentation
more flexible and interactive.
Specifically, building on the advanced development of LLM, Reasoning VOS focuses on generating
mask sequences from a highly implicit and complex text query. VISA (Yan et al.) 2024) first selects a keyframe and follows the core idea of LISA (Lai et al., 2023 by generating segmentation-oriee token with finetuned image-based MILMM. Video-LISA (Bai et al., 2024) and HyperSeg (Wei et al. multi-grained manner and concatenate the text query, feeding both outputs. However, none of the prior works incorporate adequate
2024) encode the video input to the LLM for segmentation-baet consideration of CoT reasoning ability for spatial and temporal information in Reasoning VOS. Our CoT-RVS is the first zero-shot-Coll framework compatible with pre-trained MLLM and image segmentation module, making it flexible to a boarder audience. In fact, recent studies, e.g. (Snell et al. 2024), have emphasized that test-time compute and inference-time reasoning are becoming increasingly central to leveraging pre-trained L.LMs. Our method is aligned with this direction.
Chain-of-Thought MLLM. Drawing inspiration from the exceptional reasoning capabilities of Large Language Models (LLMs), Multi-modal Large Language Models (MLLMs) seek to integrate visual components enhance multimodal perception and reasoning. These models are often employed to produce captions or respond to inquiries based on visual and textual inputs, typically achieved
by techniques such as prompt engineering (Liu al. 2023c Shen et al. 2023 Wu et al. 2023a).
3

---

Published as a conference paper at ICLR 2026
Plevee fereeefirddal all the siggodssss two www.whhrlldloommeeooiiiomomoo
Key maskss
Psep Reasoning Image Segmentation
MLLM Fkey K Keyframe Selector
Model
Instance Descctations
Key eyatems
pansrn
weuring while grown
"the parmus in viiille
wanting a white chartoo
Pett Video Processor
Figure 3: CoT-RVS for Reasoning VIS where Reasoning VOS a special case.
cross-attention mechanism Alaaracc et al. 2022), Q-Former (Dai et al. 2023 Li al. 2023a), prompt
tuning (Zhang al. 2023b). linear projection (Koh al. 2023 Liu al. 2023b). and unified model
architectures (Peng et al. 2023) Xiao et al. 2024). These MLLMs have greatly revolutionized the
traditional vision-centric tasks like image segmentation (Lai et al. 2023 Xia et al. [2024 Zhang et al.
2023a: He et al.
2024 Bao et al. 2024), object detection (Zhang et al.
2023a
Zang et al. 2024 You 2024
al. 2023 Chen al. 2023 Ma al. 2024), video segmentation (ann et al.
Bai et al. 2024
Wei et al. 2024), and object tracking [Wang et al. 2024 Munasinghe et al. [023 Zhu et al.
2023
In addition these advancements, CoT-based LLMs have been proposed improve the reasoning capability (Wei et al., [0222] Wang et al. 2022). Existing CoT approaches tend to concentrate on inference through progressive reasoning demonstrations (Zhang al., 2023d] Wei
2022
Lyu al.] 2023) or by utiliziig explicit prompts that clearly outline each step (Kojima et al. 2022).
However, enabling CoT capabilities for MILMMs typically undergo fine-tuning on specifically designed multimodal CoT datasets (Mondal et al.) 2024 Zhang et al. [2023e] Lu et al. 2022) incorporate intricate intermediate representations (Mitra et al. 2024 Suríi al. 2023). which restricts their usability and scalability due data accessibility and time complexity. Closer our work, recent works, such as ThinkFirst (Kao et al. 2025) and Seg-Zero (Liu et al. 2025). have successfully incorporate CoT MILM in reasoning image segmentation, while still falling short of "thinking" in the temporal domain. contrast, we aim to, for the first time, abridge this gap by exploring zero-shot CoT reasoning of pre-trained MLLMs and apply them in the area of video segmentation.
COT-RVS: MULTI-AGENT FRAMEWORK
CoT-RVS is a multi-agent framework that consists of three network components: an MLLM keyframe selector Fkegs a reasoning image segmentation model Fseg. and video processor Fvids see Fig.
Input and Output Settings. Given input video = {4 E = xx1 xa] and text query 9 that can be very complex, CoT--RVS is inherently a Reasoning Video Instance Segmentation model (-) which generates a list of non-overlapping binary mask sequences ÿ = {M, E RTxHxW1**
Ji=1
(xrv g),
where & indicates the number of object instances of interest that satisfy the requirements input query where Reasoning Video Object Segmentation (VOS) is clearly reducible to Reasoning Vidco Instance Segmentation (VIS) by setting i =
Each output mask sequence M, {m,,e F ft-1 corresponds to object instance of interest with the same number of frames and size as Te, having the property of m,, Mmj,t = $ for i + j. Query requirement. CoT-RVS shares similar query setting with the first Reasoning Video Object Segmentation (Yan et al., (024)), where the input query is complex or implicit (e.g.. "Please find the vehicle in the video which can accommodate at least four people.") in contrast to the Referring VOS (Seo et al. 2020) task which takes explicit and direct query (e.g. "Please segment the red car in the video."
3.1 MULTI-AGENTIC FRAMEWORK
MLLM Keyframe selector. The keyframe selector is an MLLM agent, e.g., GPT-40 (Hurst et al.) 2024) Gemma3 TTam al. 2025). that generates keyframe selectivites from a list of keyframe
4

---

Published as a conference paper at ICLR 2026
candidates, denoted {If), with CoT thinking process that analyzes the sempoii correlation among the input video frames. To find all the object instances that meet the query requirement necessitating temporal-semann such that the pertinent instance is most apparent for the MILMM agent to observe from that frame. Thus, the output contains list of selected object instances, each of which corresponds an object index, a respective keyframe, and a keyframe-related object description (e.g., "the player in black jersey at the right side of the frame")
understanding,
select their corresponding keyframes
Refer to Appx. A for the prompting details. In implementation, the agent is prompted to automatically synthesize series of questions for each keyframe candidate and answer them one-by-one. Starting from general semantic information like "What can be seen in the frame?". to temporal-related questions like "Is a better keyfrume?" and eventually ending with detailed questions like "Are there any new objects that meet the query requirement?" this CoT process enables thorough reasoning between the user query and the temporal-semante correlation among the keyframe candidates.
After analyzing all the keyframe candidates, the agent will collect all the per-frame responses and generate an output list composed of all target instances along with the respective keyframes based on its progressive question-anie and-on--edd- pairs. We demonstrate this CoT process in Fig. for visualization.
Reasoning Image Segmentation Model. The task of the reasoning image segmentation model is to predict the key mask of each selected object instance. This agent is prompted by the selected keyframes and the keyframe-relatee object descriptions generated from the MLLM keyframe selector.
Video Processor. The task of the video processor is to track the selected object instances over the entire video. Each key mask generated by the reasoning image segmentation model will be tracked to produce an instance- level mask sequence.
3.2 COT-RVS: REASONING VIDEO INSTANCE SEGMENTATION
CoT can be applied to the entire video when available for reasoning instance segmentation. Given an input video Ty = {4 gHxWx31
1=1 and text query g. we first uniformly sample a number of keyframe candidates {4 E x where 7" [T/E] for some & < T. These candidates, along with the user query (s are then sent to the MILLM keyframe selector Fkey for temporal-semante analysis, as described in [Sec. which eventually outputs a keyframe selectivity S composed of list instance-level text descriptions {5 with respect to their corresponding keyframes { (e.g., "the red car" in 15. "the blue motorcycle" in ..... Specifically, k is the predicted number of target instances. This CoT process can simplified as:
Rrr Sample {)
S1:k, fIA Feey (g, 1irr.)
Following by the extraction of temporal-semaan correlation within video, where time sensitivity has been resolved, now incorporate the image- based segmentation model and video processor to generate cemporal-consittenn mask sequence. In particular, we use the reasoning image segmentation
Keyframe candidates
CoT process
User query
What can be seen in the frame 1?
What be in the frame
frame contains
The frame contains
Any objects satisfying user quary?
Any objects sa satisfying the user query?
No. frome depicts ...
Yes, the object the left side...
MLLM agent
Objeet list [object keyframe: object description: "the white car crossing the intersection, evident with traffic light in view"
Figure 4: Illustration of the CoT Process. The MLLM agent is prompted to synthesize chain- of-thought questions and answers. This CoT process enhances the in-depth temporal and spatial understanding for the keyframe candidates. Based on the CoT result, the MLLM agent outputs an instance-level object list, which contains the object-specific keyframe and description within the frame. Refer to the Appendices. Aland B for the detailed prompt and output examples.
5

---

Published as a conference paper at ICLR 2026
model Fseg generate per-instance key masks {m, "'# based the the instance- level selectivity. These key masks will be sent to the video processor Fesd, i.e. SAM2 (Ravi et al. 2024). to predict the per-instance mask sequences. We formulate this process as:
m: Fsey (st, fi): M, = (un for i 1.2, k. To ensure the mask sequences are not overlapping, we adopt an optional post proocessns operation:
4
m1.1 mi
(5)
mi,1 my mp.e.
(6)
for any t < T and < k, where the final output E RT*Hx is the combination of all Ma,1 E IR NxW
3.3 COT-RVS: ONLINE REASONING VIDEO OBJECT SEGMENTATION
Our approach can be extended to handle online video streams where future frames have yet to be observed. Refer to Fig. 5| The goal of Online Reasoning Video Object Segmentation (Online Reasoning VOS) is generate binary mask sequence
M {m, E R Ji=1 from an online input video stream { R # W x3 x ji=1 and text query q, while the model receives video frames in temporal order and makes predictions only based on frames that have already been seen. Specifically, this process can be modified from Eq. (1) as:
my 1/0 1,,, m1+-1,
The final output can be obtained by combining predictions at all time, i.e., mi, m2..... mr.
The CoT-RVS framework Online Reasoning version shares a similar architecture to the original ReasoningVIS, while "asking" the keyframe selector Fkey for periodic updates of the key mask. Specifically, we introduce a hyper-parameter & for the temporal frequency of keyframe selection. denote key as the selected keyframe at time t, and pt as its corresponding frame index, respectively. When given complex text query and an online video frame 4, where ng + for some E Z. the model sends them to Fkey to justify the the keyframe selectivity 5 E {0, 1}, which indicates whether I should be chosen keyframe. This process be formulated as:
Tkey Null; Po = 0;
(8)
S: = Fkey (g: It);
(9)
TRey St 1 + (1 ((-))) Tkey
(10)
"mxx(t-E,0)"
P: = St -S:) 5:))
Pmmxx(t-.0)=
(11)
for t + + 1, .... Refer Appx. for the prompting details for Eq. (9)
The selected
keyframe T.kry is then used generate mask control and guide the video processor for g consecutive frames. We formulate this process as the following: m. = 0 fkey Null; otherwise,
my Fseg (g. 1 Ikky ; m,
Fvid (my, Iva):
(12)
Which object can move Segment the smallest lest visible
Feey
Key updatee
Reasoning Image
Key mask
MLLM Keyframe Selector
Peey
Segmentation Model
Fais Video Processor
Fvay MLLM Keyfrrme Selector
Key update
Reasoming Image
Key mask
let
Eseg
Segmectation Model
Figure 5: CoT-RVS for Online Reasoning Video Object Segmentation.

---

Published as a conference paper at ICLR 2026
"My friend and each drove our carz to another city. He was driving a white car and leading the way in front of me, he drove too fast and I lon him, He called that he had hoon waiting prfflee light and then crossed Intersection Which
likely to be my friend's car?"
Inpor video
VIdeoL.ISA
Seg-Zero
ThinkFirst
CoT-RVS
Ground Truth
Figure 6: Qualitative results. By thinking with CoT, CoT-RVS demonstrates the temporal
reasoning ability previous works fail to achieve. We compare CoT-RVS with Reasoning VOS method VideoLISA (Bai al. 2024) and CoT Reasoning Image Segmentation methods Seg-Zero (Liu et al.
2025 and TrinkFirst Kao et al., 2025).
Table 1: Referring VOS results on MeVIS bench- mark (Ding et al. 2023a).
Table 2: Referring VOS results Refer-DAVIs 17 benchmark Font-Tused et al. 2017).
Muthods
Online
JKF
J
F
Mothods
Online
JkF
Tradlittonal inctlouds widlour
ability
Traditional metioods wilibou reazoning abulity
12000
URVOS (ee
LBDT (Dann et Purte
29.3
27.8
30.8
YORO TreeiR RUZES
53.3
MTTR Miniere 20122h
X
30.0
24.8
31.2
MESA WirBEi banel
57.9
53.8
62.0
ReferFarmer mn NI22
31.0
29.8
32.2
ReferFarmer Wire 20120
61.1
58.1
64.1
VLT+IC
Ding MINT
35.5
33.6
37.3
SgMg Miao dral 20133
63.3
60.6
66.0
[Digg
Onlinekuter
LLM-hotedd muslonds bith scononing abtiiry
LLM--ased mushhods 1ith zeasoming ability
LISA-7B5 [ai 20233
37.2
35.1
39.4
LISA-7B Lai at M033
64.8
62.2
67.3
LISA-136
2025
37.9
35.8
40.0
LISA- 1315-131-10
puss]
66.0
63.2
68.6
TracGiPT-7 MA
TracGiPT-1388 Zi mill Fute)
41.2
30.3
43.1
TrccGPP-133 33 Vai suzal
66.5
62.7
Vident ISA (O1Sa)) Uimt 20124
X
42.3
39.4
45.2
Videol ISA (OTxx)) Inmiereu 2114
67.7
63.8
71.5
ISA (SooLISAAPP) [Po)* [Bateraa 2000
45.4
41.3
47.6
VideoL (Po)* [B oua 2126)
68.8
64.9
72.7
VISA-7B el 2009
43.5
40.7
46.3
VISA-7B Yan 2000
69.4
66.3
72.5
VISA-13B
70.4
SAMWISE Comoooo sill 2024
49.5
16.6
$2.4
SAMWISE moci 2024
70.5
67.4
74.5
GLUS (Lin ar 2000
X
51.3
48.5
54.2
HyperSeg Werersin 2024
71.2
45.9
42.7
49.1
COTRVV M-T.S---TTTT---
73.9
70.4
77.5
SEL-VV--Gemm3-122
40.3
48.1
74.6
20.9
28.1
COT-RVS.GPT.L
OTSA: One- Token-Seg-All, Po: post-oplimization
OTNA: One- Token-Seg-AAll Po: post-opimizztion.
where L' P1//1-+++1 for any T. Specifically, l' implies the frame index of the most recently selected keyframe. This framework adopts greedy strategy to periodically update the selected keyframe when incoming frame satisfies the query requirement, then using the keyframe to track in the following frames. When none of the previous frames is selected as keyframe, the model outputs nothing. Eventually, we obtain the mask sequence ý with all m.'s.
EXPERIMENTS
In this section, will empirically demonstrate the effectiveness of proposed CoT-RVS. We begin by introducing the experimental settings in Sec. 4.1] followed by showing the qualitative and quantitative results and comparation of CoT-RVS with previous works in Sec. 4.2) Lastly, conduct ablation study to analyze the keyframe candidates for GPT-4o and CoT thinking process in [Sec. 4.3] where more ablation study, e.g. robustness, time and latency analysis, will be provided in the appendix.
4.1 EXPERIMENTAL SETTINGS
Implementation details. We implement our MLLM keyframe selector Fkey with GPT-4o (Hurst
et al. 2024) (version "2024-05-13 13") for the original CoT-RVS and LLaVA1.5-7B (Liu et al. 2023a) for the online CoT-RVS. We also conduct experiments with the open-sourced Gemma3-122 12B
(Team
7

---

Published as a conference paper at ICLR 2026
Table Reasoning VOS results on Reason VOS benchmark (Bai et al. 2024).
Table 4: Reasoning VOS results on ReVOs
benchmark Yan et al.
2024
Methods
Online
JKF
F
Mahooks
Online
5KJ
Traditional mestodd withour reasoning
yoosowing
MTTK [Bosach et al B0221
31.1
29.1
33.1
MITR Betach clai [22222
25.3
25.
25.9
ReforFormer eie surec
32.9
30.2
35.6
LMPM (Ding ot al P07rA
26.4
21.2
31.7
SOC [ et 20r
35.9
33.3
18.5
ReferFamerr Ward 20223
28.1
26.
29.9
33.7
38.7
LLaMA-VID
1-MPMM
26.1
20.9
31.4
Onlineketer Warcemu
and mehhonsen teusiiing
MFM-haeed mustoros wal feammiig abbury
LJSA-138 Laeeral 2023
41.6
59.8
43.5
LISA Lat al. 2025
31.1
29.1
33.1
TracGPT P7 (amrec 2025
43.6
41.8
45.5
Vidoo
2014
TrckGPP-13 (
45.0
43.2
46.8
(Po) (Rareral 2004
49.9
VISA
2022
GLUS (Lin
20033
49.9
47.5
52.4
(TT erar
50.9
CoT STT
52.0
49.5
54.5
GLUS LLia eiinerra me p122
54,9
52.4
57.3
CoTRVV Gemma3-128
50.7
17.5
51.0
HyperSee Werd 31 2024
55.7
53.1
58.4
CVT-RVS-GPT-l
CoT-VV- SEERVS-TTme S-TT
46.2
43.5
48.8
OOSA: One Icken-Srg-AAn All. post-opiimization
NoT-RVS-Gemmal
47.1
43.4
50.9
G-GTToo
(IT): instuction tuning with RC training set.
the vidco
friead and I by busgpoitaton don't Whick obscets
Table 5: Reasoning VOS re-
sults with temporally sensi- tive queries on our sampled T- Reason VOS dataset.
Isput video
CoT-RVS (Orggral)
Experiments
J&J
Please find
Think
Vilaol (CTSA)
35.1
39.6
CoT-RVS- 1718V--Gemm3--12
50.2
46.8
51.6
Iapur video
CoT RVS GPT-4o
55.5
51.2
59.9
Table 6: Ablation study for the GPT-4o input of keyframe candi- dates the Reason VOS bench- mark.
(Onime)
Figure 7: Visual results of the original Reasoning VIS and its online version for Reasoning VOS. The original CoT-RVS be used for instance-level reasoning video segmentation, while the online extension is preferred when the object of interest changes during the tracking process. Zoom in for details and refer to the appendix for more examples.
Experiments [
J&.F
F
66.2
63.1
69.4
T-1
65.5
62.4
68.7
60.0
57.1
62.9
et al. 2025) in CoT-RVS for users with limited access to the OpenAl service. Quantitative studies for different MLLMs are provided in Appendices. FJand to analyze our implementation choices. We adopt Seg-Zero [Liu et al. 2025) the reasoning image segmentation model seg and SAM2 (Ravi al.] 2024) as the video processor Frid, where alternatives with LISA (Lai et al. 2023) and
Cutie Cheng et al. 2024) are also demonstrated in (ppx. H for modularity analysis. We use the pre-trained weights of each module from their official implementation. We set - + for experiments (i.e., the integer value that generates most and closest to 8 keyframe candidates), and { = for online experiments. GPT-4o's 's temperature set to 0.5, and the maximum number of output tokens is 2500. To evaluate the VOS task with the VIS version of CoT-RVS, we take the union of all the predicted instance-level mask(s) as the final output.
Datasets. We compare CoT-RVS with previous works multiple benchmarks, including referring VOS datasets (MeViS (Dig et al. 2023a) and Refer-DAVIS-255 reasoning VOS datasets (ReVOS Yan et al. 2024) and Reason VOS (Bai al. (0244)) To evaluate reasoning VOS with temporally sensitive queries, we manually sample a subset Reason VOS, named VOR. which contains only difficult examples involving time-sensitive queries. Refer to Appx. D for more dataset descriptions.
(Pont- et al. 2017)) and
Evaluation metrics. We follow previous works and evaluate the CoT-RVS performance with region similarity (J), contour accuracy (F), and their average value &J&F). Specifically. J evaluates how well the pixels of the predicted mask match the ground truth overall using intersection over union (IoU), and F combines the precision and recall of boundary pixels, emphasizing the accuracy at the object boundaries.
8

---

Published as a conference paper at ICLR 2026
Input video
Full model
Think wio CoT
w/o Thinking
Input video
Full modal
Tiink wyo CoT
w/o Thinking
(a) Reasoning VIS "Please find all the subjects using mva- whonled transportation
(b) Ressoning VOS "Whon parents see their children potential danger, they oreee concemed ready help.
Who is parent the scene?"
Figure 8: Ablation study. We examine the effectiveness of the Thinking process for keyframe selection. "Think w/o CoT means selecting keyframes without the CoT-guided prompts. "w/o Thinking" means randomly selecting a keyframe for the original method and selecting the first frame as keyframe for its online version. Zoom in for details.
4.2 MAIN RESULTS
Referring VOS. We report the quantitative comparisons of CoT-RVS with previous works the tasks of Referring VOS in Tab. I and Tab. Online methods are highlighted in gray in these tables. Results show that both of the online and offline versions of CoT-RVS significantly outperform the state of the art. CoT-RVS achieves up to 3.5% improvements over the online approaches and 15.6% over the offline approaches. Notably, online CoT-RVS surpasses offline baselines most cases, while given a more challenging task setting.
Reasoning VOS. We demonstrate the quantitative comparison of the Reason benchmark and ReVOS benchmark in Tab. 3 and Tab. 4 respectively, where CoT-RVS achieves up to 18% than Mate-off-hee-rt offline methods and up to 13.3% than state-of-the--e art online methods, showcasing the better reasoning capability of "thinking" first. Implementation based on Gemma3-12B performs slightly results than in GPT-4o, possibly due to the capability of temporal reasoning and image understanding, We further report the qualitative performance of reasoning VOS with temporal- related queries in Fig. 6) When the temporal reasoning capability is needed, CoT-RVS achieves significantly better performance than any existing works. This is also verified in the quantitative results of T-Reason dataset, as shown in Tab. 5]
Reasoning VIS and online Reasoning VOS. We demonstrate the extended features of CoT-RVS in Fig. 7 In specific, the original CoT-RVS able to generate instance-level mask sequences. The online version of CoT-RVS can update the object of interest during the tracking process. These extensions provide CoT-RVS with more flexibility broader audience.
4.3 ABLATION STUDY
Analysis of hyper-paramettr& ç. We sample keyframe candidate for every & frames, then concate- nate the keyframe candidates into grid image for the GPT-4o CoT process. In Tab. the reasoning VOS results on the Reason VOS benchmark (Bai et al. 2024) with different & values. Our empirical results showcase that GPT-4o achieves comparable performance when T (i.e., < 1 + 1), while the image reasoning capability drastically decreases with & = 1 T-1 16 Therefore, to preserve the temporal information in the video without sacrificing GPT-4o reasoning accuracy, we recommend selecting a & value such that < T' < 8.
compare
Analysis of CoT process. We examine the CoT process of the MLLM keyframe selector. Specifically, we show in Fig. 8 that the CoT process significantly improves the reasoning capability. In Reasoning VIS, thinking without CoT may result in missing some instances that require Online Reasoning VOS, thinking without CoT degrades the performance of the keyframe justification, possibly leading to wrong classification of binary selectivity. All of these experiments point the soundness of our proposed CoTT reasoning.
effort to find. In
Limitations. While CoT-RVS demonstrates strong performance in reasoning video segmentation through zero-sho Chain-of-TT Thhou-of-Thought prompting, a key limitation of CoT-RVS is its reliance on pre-trained MLLMs without task-specific fine-tuning. As result, its performance is bounded by the general reasoning capabilities of these models. We believe that fine-tuning MLLMs for temporally-awer multimodal reasoning could further improve segmentation accuracy.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper. we presented CoT-RVS, a novel zero-shot, training-free -free framework that leverages Chain- of of-Thought (CoT)) capabilities of multimodal large language models (MLLMs) to enhance reasoning video segmentation tasks. Our extensive experiments on various benchmark datasets demonstrate that CoT-RVS significantly outperforms ffte-of-thh-ar methods, particularly in challenging scenarios involving complex, implicit and time sensitive queries, occlusions, and rapid object movements. Additionally, extended the framework for online video streams, showcasing its adaptability for broader audience. Overall, CoT-RVS sets a new benchmark in reasoning video segmentation and opens avenues for future research in zero-shot reasoning and its applications in dynamic visual environments.
ACKNOWLEDGMENTS
This work was supported in part by the Research Grant Council of the Hong Kong SAR under Theme-based Research Scheme, grant no. T22-606/23R.
REFERENCES
Jean-Bape SannBaptistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances neural information processing systems, 35:23716-237366 2022.
Ali Athar, Jonathon Luiten, Paul Voigtlaender, Tarasha Khurana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: benchmark for unifying object recognition, segmentation and tracking in video. In WACV, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar a5Xiv:2502. 13223, 2025.
Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language instructed reasoning segmentation videos. arXiv preprint arXiv:2409. 19603, 2024.
Linchao Bao, Baoyuan Wu, and Wei Liu. Can in mrf: Video object segmentation via inference in cnn cnn-based higher-order spatio-temporal mrf, 2018. UKL.nttps://rrav..orr//a////
09453
Xiaoyi Bao, Siyang Sun, Shuailei Ma, Kecheng Zheng, Yuxin Guo, Guosheng Zhao, Yun Zheng, and Xingang Wang. Cores: Orchestrating the dance reasoning and segmentation. In European Conference on Computer Vision, pp. 187-204. Springer, 2024.
Goutam Bhat, Felix Järemo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg, Lue Van Gool, and Radu Timofte. Learning what to learn for video object segmentation, 2020. URL
https://erxiv wttp////////.....1111.
Adam Botach, Evgenii Zheltomozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4985-4995 2022a.
Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022b.
Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset. Laura -aixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation, 2017. URL https: : /ww./xxxvvvo.////.aa//aao org/ass// 1611 05198)
10

---

Published as a conference paper at ICLR 2026
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306.15199, 2023.
Xi Chen, Zuoxin Li, Ye Yuan, Gang Yu, Jianxin Shen, and Donglian Qi. Stare tracker for real-time video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), June 2020.
Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition (CVPR), June 2018
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander Schwing, Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764 2021a.
Ho Kei Cheng and Alexander Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pp. 640-658. Springer, 2022.
Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation, 2021b. URL htttts://arxi...tt//
Ho Kei Cheng. Seoung Wug Oh, Brian Price, Joon- Joon-Young Lee, and Alexander Schwing, Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3151-3161, 2024.
Jingchun Cheng, Yi-Hsuan Tsai, Wei-Chih Hung, Shengjin Wang, and Ming- Hsuan Yang. Fast and accurate online video object segmentation via tracking parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7415-744 1.24,7444, 2018.
Claudia Cuttano, Gabriele Trivigno, Gabriele Rosi, Carlo Masone, and Giuseppe Averta, Samwise: Infusing wisdom in sam2 for text-driven video segmentation, 2024. URL https:
arq/abs/ 2411. 17646
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards purposse vision-language models with instruction tuning. arXiv preprint arXiv:2303.06500 2023.
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16321-16330, 2021.
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. McViS: A large-scale benchmark for video segmentation with motion expressions. In ICCV, 2023a.
Henghui Ding. Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. MOSE: new dataset for video object segmentation in complex scenes. In ICCV, 2023b.
Zihan Ding, Tianrui Hui, Junshi Huang. Xiaoming Wei, Jizhong Han, and Si Liu. Language- bridged spatial-lemporal interaction for referring video object segmentation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 4964 4973, 2022.
Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. Capsulevos: Semi-supervised video object
segmentation using capsule routing, 2019.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. 2022.
arXiv preprint arXiv:221..
Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun- He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned llms with fine graigra visual perception. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 13980- 13990, 2024.
11

---

Published as a conference paper at ICLR 2026
Namdar Homayounfar, Justin Liang, Wei-Chil Ma, and Raquel Urtasun. Videoclick: Video object segmentation with a single click. arXiv preprint arXiv:2101.06445, 2021.
Rui Hou, Chen Chen, Rahul Sukthankar, and Mubarak Shah. An efficient 3d cnn for action/object segmentation in video, 2019. URL U
08895
Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311. 2023.
Yuan-Ting Hu, Jia-Bin Huang, and Alexander G. Schwing. Videomatch: Matching based video object segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Won-Dong Jang and Chang-Su Kim. Online video object segmentation via convolutional trident network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7474-7483, 2017.
doi: 10.1109/CVPR.2017...00
Shiu-hong Kao, Yu-Wing Tai, and Chi-Keung Tang. Think before you segment: High-quality reasoning segmentation with gpt chain of thoughts, arXiv preprint arXiv:2503.07503 2025.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In International Conference Machine Learning, pp. 17283- 17300. PMLR, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. //16, 2022.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308. 00692, 2023.
Dezhuang Li, Ruoqi Li, Lijun Wang, Yifan Wang. Jinqing Qi, Lu Zhang, Ting Liu, Qingquan Xu, and Huchuan Lu. You only infer once: Cross-modal meta- meta-transfer for referring video object segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.
1297-1305, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, and Yan Lu. Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023b.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023c. URL
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.133655 2023.
Lang Lin, Xueyang Yu, Ziqi Pang, and Yu-Xiong Wang. Glus: Global-local reasoning unified into a single large language model for video segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8658-8667, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023b.
12

---

Published as a conference paper at ICLR 2026
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:25033.0550, 2025.
Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, al. Interngp:: Solving fision-centric tasks by interacting with chatgpt beyond language. arXiv preprint wwXiv:235.00666 2023c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yitong Wang, Yansong Tang, Xiu Li, and Yujiu Yang. Soc: Semantic-assisted object cluster for referring video object segmentation, 2023. URL
https://axxiv: wwtp://....../ooottoooooooo 2305. 17011
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong. Marianna Apidianaki, and Chris Callison-Burch Faithful chain-of-thought reasoning. In The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Puciff Chupter of the Association for Computational Linguistics (IJCNLP-AACL 2023), 2023.
Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pp. 417-435. Springer. 2024.
Kevis-Kokitsi Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset Laura Leal-Taixé, Daniel Cremers, and Lue Van Gool. Video object segmentation without temporal information, 2018. URL
https::/arxizs wwtp/orggg///ooooooooooeeeoea. 0009.000013003111
Bo Miao, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Spectrum-guided multi- granularity referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 920-930, 2023a.
Bo Miao, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Spectrum-guided multi- granularity referring video object segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 920-930, October 2023b.
Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14420-14431, 2024.
Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cot: Knowledge augmented multimodal chain-of-thoughts reasoning. arXiv preprint arXiv:2401.12863, 2024.
Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-llava: Pixel grounding large video-language models. arXiv preprint arXiv:2311.13453 2023.
Seoung Wug Oh, Joon- Joon-Young Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019a.
Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Scon Joo Kim. Video object segmentation using space-time memory networks. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9226-9235. 2019b.
13

---

Published as a conference paper at ICLR 2026
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint (rXiv:2306.14824 2023.
Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine- Sorkine-Homung. Learning video object segmentation from static images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2663-2672, 2017.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Hornung,, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.0067. 2017.
Frano Rajič, Lei Ke, Yu- Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. Segment anything meets point tracking. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 9302-9311. IEEE, 2025.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 0000:4448400714 2024.
Andreas Robinson, Felix Järemo Lawin, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg. Learning fast and robust target models for video object segmentation, 2020. URL
00908
Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmen- tation network with large-scale benchmark. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pp. 208-223. Springer, 2020.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 066:38154-8811
2023.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 2rXiv:2408.03314, 2024.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11888-11898, 2023.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma technical report. arXiv preprint arXiv:2503. 19786, 2025.
Paul Voigtlaender, Yuning Chai, Florian Schroff. Hartwig Adam, Bastian Leibe, and Liang-Chich Chen. Feelvos: Fast end-to-end embedding learning for vidco object segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Han Wang. Yongjie Ye, Yanjie Wang, Yuxiang Nie, and Can Huang. Elysium: Exploring object- level perception in videos via mllm. In European Conference on Computer Vision, pp. 166-185. Springer, 2024.
Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip H.S. Torr Fast online object tracking and segmentation: A unifying approach. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 171, 2022.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8741-8750, 2021.
14

---

Published as a conference paper at ICLR 2026
Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: To- wards universal visual segmentation with large language model. arXiv preprint arXiv:2411.7766. 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-though prompting elicits reasoning in large language models. In Advances in neural information processing systems, volume 35, pp. 24824-24831 2022.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303. 04671, 2023a.
Dongming Wu, Xingping Dong, Ling Shao, and Jianbing Shen. Multi-level representation learning with semantic alignment for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1996-5005, 2022a.
Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. Onlinerefer: simple online baseline for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2761-2770, 2023b.
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4974 2022b.
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4974 2022c.
Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. Seqformer: Sequential transformer for video instance segmentation. In European Conference on Computer Vision, pp. 553-569. Springer, 2022d.
Zhuofan Xia, Dongehen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 3858-3869, 2024.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2818-4829, 2024.
Kai Xu, Longyin Wen, Guorong Li, Liefeng Bo, and Qingming Huang. Spatiotemporal enn for video object segmentation, 2019. URL htttps://arxxii org/ abs www.ss//aa 02363
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang. Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In European Conference on Computer Vision, pp. 98-115, Springer, 2024.
Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground- /20032003300 08333
background integration, 2020. URL RRLLttps: htll:tttts///aataaa http://xxxxxvrrgggg/
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 10502-10511, 2019.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang. Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. International Journal of Computer Vision, pp. 1-19, 2024.
Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. Next-chat; An Imm for chat, detection and segmentation. arXiv preprint 2xXiv22311.04490 2023a.
15

---

Published as a conference paper at ICLR 2026
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXh::2303 2023b.
Tao Zhang, Xingye Tian, Yu Wu, Shunping Ji, Xuebo Wang, Yuan Zhang, and Pengfei Wan. Dvis: Decoupled video instance segmentation framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1282-1291, 2023c.
Tao Zhang, Xingye Tian, Yikang Zhou, Shunping Ji, Xuebo Wang, Xin Tao, Yuan Zhang, Pengfei Wan, Zhongyuan Wang, and Yu Wu. Dvis++: Improved decoupled framework for universal video segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In International Conference Learning Representation, 2023d.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923. 2023c.
Yikang Zhou, Tao Zhang, Shunping Ji, Shuicheng Yan, and Xiangtai Li. Improving video segmen- tation via dynamic anchor queries. In European Conference on Computer Vision, pp. 463-463.
Springer, 2024.
Jiawen Zhu, Zhi-Qi Cheng, Jun- Yan He, Chenyang Li, Bin Luo, Huchuan Lu, Yifeng Geng, and Xuansong Xie. Tracking with human-intent reasoning. arXiv preprint arXiv:2312. 17448, 2023.
16

---

Published as a conference paper at ICLR 2026
APPENDIX A COT-RVS COT PROMPT DETAILS
One of the key components of the CoT-RVS framework is the chain-of-thought keyframe selection process with temporal-semantee analysis using GPT-4o and Gemma3 (Eq. (3)). To prompt GPT- 4o Hurst et al. (2024) or Gemma3-12B Team et al. (2025), we first resize and concatenate all the keyframe candidates (r into a merged image I merged E R Tf' x W" x3, where the concatenation is
done on the width dimension if W < HH or on the height dimension if H < W, and max H', W') =
P for some hyperparameter P Z. Then we use the following text prompt:
Chain-of- -houught Prompt:
'You will act a keyframe selection agent for video reasoning task. During each inference, you will be given a grid image that contains multiple keyframes sampled from long video. The keyframes are aligned from top to down or from left right, following their temporal order. You will also be given complex user query that implicitly or explicitly refers to one or more target objects in the video. You need to think in chain of thoughts to analyze each keyframes and find the best keyframes for each target object, where a segmentation model can find the target object in that frume with less effort. Your chain of thoughts should begin with what can seen in each keyfrume, how many objects in total fulfilling the requirements of the user query, etc. Some of the objects may be seriously obscured or blocked by other objects. Some of the objects may camouflaged in their surroundings. Analyze each frame separately get all the visible objects. This chain of thoughts should follow the output format:
" "Chain of Thoughts
Frame 1: <analysis of frame 1>;
Frame canalysis of frame 2>;
...". For the analysis of each frame, you also have to follow the chain-of-thougle format:
" *<question 1>* <answer 1>;
"<question 2> <answer 2>;
...,, where you have to ask question to yourself and answer it. Your answer should be detailed as possible. You should start with broader questions, like "what can be seen in the frame?" to some detailed questions like "rre there any other objects that haven't be listed" and "how many and which objects satisfy the user query?". There will be many questions and answers in the analysis of each frame, helping you to fully understand the frane. The actual questions vary by cases. Generate the in in-depth questions and answers based on your previous analysis. Your thinking process aims find the keyframe for each target object of interest (find all the target objects, each of which corresponds to keyframe) related to the user query. Lastly, you have to output a list of dictionary with a format:
"utpput list: [object index: 1, keyfrume: k_ 1. object description: <description of the object I in keyframe k_1>), bobjecttjidexx: 2, keyframe: 2_2, keyframe 2_2>)....)
<description of the object 2 in
where each element in the list is dictionary with three items, with object index, keyframe index, object description. & is the k-th keyframe in the grid image. object bnjeet_indexx index is numbering integer starting with 1. object description implies the description for that object in a particular frame, helping the model to find the object in that particular frame. For example, valid element in an output list be like 'Output list: [[object index: 1, keyframe: 4, object description: "the man the top left corner of the image"]] IJ', You have to include all objects that fulfill the requirements the user query. Include the objects even if is only partially visible. While choosing the keyframe for any object, you should prioritize those frames where objects are not overlapped. This will help model to better recognize the object. Keep the output list in text format. Don't It use json formatting. The output list begins with the prefix "Output list: ". "followed by square bracket with multiple curly brackets. The square bracket should in the same line, following the format "Ouiput list: [...]". Don't start with new line.
Here is a grid image with (num keyframees keyframes. The user query is "[query]". " Follow the instruction and output the index of the best keyframe.
17

---

Published as a conference paper at ICLR 2026
where ) and {query) respectively refer the number of keyframe candidates T and user query q in Sec. 3.2 Lastly, we parse the output list to retrieve the keyframe selectivity S.
APPENDIX B COT-RVS (ONLINE) COT PROMPT DETAILS
In the online version of CoT-RVS, an incoming frame 4 will be sent to the MLLM keyframe selector (i.e., LLaVA-7B Liu et al. (2023a)) for binary selectivity when & ng + 1. Below is the CoT prompt for our online CoT-RVS:
Online Chain-of-Thought Prompt:
CConsider query "[query)" for an object tracking task. First analyze what can be seen the imput image and then output simple answer with Yes or No justify whether the input image is suitable as a keyframe. good keyframe is an image that contains the target object. Please follow the output format "The image contains .... Therefore, the justification using this image keyframe is <Yes./No.>"
where /query) represents the user query q. The output of the MLLM will be parsed for binary selectivity St.
APPENDIX C HARDWARE DETAILS
CoT-RVS with GPT-40 is implemented with OpenAl API and one RTX-4090 GPU. Other experiments (i.e., the STT-RVS-Geema---- and the online (-T-RVV) implemented with two RTX-4090 GPUs.
APPENDIX D T-REASONVOS DATASET
Reason VOS contains 91 videos (18 from Me Ding et al. (2023a), 26 from MOSE Ding et al. (2023b). 37 from BURST Athar et al. (2023), and 10 from VIPSeg Miao al. (2022) and 458 queries, which was proposed to effectively evaluate the performance of reasoning segmentation in videos Bai al. (024). While being widely used, this benchmark falls short in giving a fair evaluation on temporal reasoning ability. In particular, some considerable number of queries are temporally insensitive and thus less challenging, where the context only refers the appearance or function of the relevant video objects unrelated to the actions object movements. Thus, we believe new dataset is both timely and essential in clearly demonstrating the efficacy CoT-RVS on temporal reasoning. Specifically,
sampled 180 video-query pairs that correspond to 45 videos (8 from MeViS, 18 from MOSE, 16 from BURST, and 3 from VIPSeg) from Reason VOS to produce a new benchmark called RoasonnOS.. We remove the beginning to the end of the video, such as "The mode of transportation capable of transporting the largest group of people". where users can make decisions based independent frames with image segmentation models. Surviving data in T-Reason VOS after removing time-insensitive counterparts focus on videos where the target objects are visible only in a short interval. or queries specifying actions and object movements, e.g., "The thing that has made 1 180-degree turn." This requires models "think" in the temporal dimension find the correct objects interest. The ReeasonVOS dataset and the selection code will be released.
queries with target objects visible from
APPENDIX E COT QUALITATIVE RESULTS
In this appendix, we will demonstrate the CoT outputs of the keyframe selector.
CUT-RVS (Original). Refer to Appx. A for the input format. Below is two qualitative result with T' = 7 and T' respectively.
18

---

Published as a conference paper at ICLR 2026
CoT-RVSS Example 1 (GPT-40):
Input query: Pleaxe segment all the visible subjects that are using some kind of tranportation tool.
Chain of Thoughts: Frame J: *What can be seen in the frame?" There are two individuals visible, both are using bicycles. *Are any subjects using transportation tools?* Yes, both subjects using bicycles. - "HHww many and which objects satisfy the user query? There are two individuals, both satisfy the query as they are using bicycles. *Where is the first subject located?* The first subject. wearing a white shirt, is located at the left center of the frame. "Where is the second subject located?* The second subject, wearing a backpack. is further down the road at the center of the frame.
Frame 2: -Whatt can be seen in the frame?* ?* Three individuals are visible Two in the foreground and one in the background, all using bicycles. *Are any subjects using transportation tools?" Yes, all three subjects are using bicycles. *How many and which objects satisfy the user query? ? Three individuals satisfy the query using bicycles. #Where is the first subject located?* The first subject, wearing a white shirt, is at the center left side of the frame. *Wheree is the second subject located?" The second subject, in light-colored shirt, is to the left of the first subject, occupying the left side of the frame. *Where is the third subject located?* The third subject, with backpack, is in the center back of the frame.
Frame 3: - *What can be seen in the frame?* Two individuals are visible; both are on bicycles. *Are any subjects using transportation tools?* Yes, both are using bicycles. *How many and which objects satisfy the user query?* There are two individuals using bicycles satisfying the query. *Where is the first subject located?* The first subject, wearing a white shirt, is centrally located in the frame. - *Where is the second subject located? The second subject, wearing a backpack, is near the center-back of the frame.
Frame 4: *What can be seen in the frame?* Two individuals are visible, both using bicycles.- *Are any subjects using transportation tools?" Yes, both individuals are using bicycles as transportation. *How many and which objects satisfy the user query?* 2* Two subjects satisfy as they ride bicycles. - *Where is the first subject located?* The first subject, in white shirt, is centrally situated. "Where is the second subject located?" The second subject, with a backpack, at the edge of the frame near the center back.
Frame 5: "What can be seen in the frame?* Two individuals are seen on bicycles. *Are any subjects using transportation tools?* Yes, both on bicycles. *How many and which objects satisfy the user query?* Two subjects, both riding hicycles, satisfy this query. - *Where the first subject located?* The first subject, wearing a white shirt, occupies the central area of the frame.
*Where is the second subject located? The subject with a backpack is in the center background of the image.
-Faame 6: *What can be seen in the frame?* Two individuals on bicycles. " Are any subjects using transportation tools?* Ycs, both are using bicycles. "How many and which objects satisfy the user query?* Two individuals satisfy the query as they use bicycles. *Where is the first subject located?* The first subject, in white shirt, is placed centrally. *Where is the second subject located?* The subject with a backpack appears the central background. Frame 7: - *What can be seen in the frame?" Two individuals on bicycles. " Are any subjects using transportation tools?" Yes, both individuals are using bicycles. "How many and which objects satisfy the user query?" Two individ- uals meet the criteria, both on bicycles. *Where is the first subject located?* The first subject, in a white shirt, is the center. *Where is the second subject located?* The individual with a backpack is further down, towards the center. Output list: [[object index: 1, keyframe: 1. object description: "the individual in a white shirt riding bicycle the left center of the frame"). J. /object index: 2, keyframe: 1. object description: "the individual with a backpack on a bi- cycle at the center further down the road"), (object_ abjjectinndxx: index: 3, keyframe: 2, bbject_ddscriptionn "the individual in light-colored shirt on bicycle at the left side of the frame")]
Figure A1: Input image
Ic
merrged
19

---

Published as a conference paper at ICLR 2026
CoT-RVSS Example 2 (GPT-40):
Input query: Which item the scene serves the purpose of drying hands after they's 've been washed?
Chain of Thoughts: Frame I: *What can be seen in the frame?* A person is holding a green towel in hand, standing
sofa. *Is there any object that serves the purpose of drying hands?* Yes, the green towel being held by the person can serve that purpose. Are there any other objects that haven' been listed?* No, the towel the primary object fulfilling the query in this frame.
2
Frame 2: "What can be seen in the frame?" The person is standing next to a sofa, holding green towel in both hands. "Is the towel clearly visible? Yes, the towel clearly visible and being held prominently. *Does the towel satisfy the user query?* Yes, it serves the purpose drying hands after washing.
Frame 3: *What can he seen in the frame?" The person is no longer visible. the back of the sofa. *Does any object fulfill the purpose of drying hands? ?* Yes, the towel on the sofa still fulfills this purpose.
The towel is now laying
"Is the towel more visible this frame than previous frames?" The towel
distinct but lay still; might not be as instantly recognizable as when held by the
person in previous frames.
-Faaee 4: "What can be seen in the frame?" The frame is similar to Frame
the towel on the back the sofa, and no people are present. "Does fulfill the purpose of drying hands? Yes, the towel on the sofa does. *Is there any difference from Frame 3 regarding the towel's visibility? * No significant differences are noted regarding visibility.
Figure A2: Input image Ic mergedd
Output list: [[ohject_ index: keyframe: abject_descrrptinn "the green towel held prominently by person"))
COT-RVS (Online). Refer to Appx. B for the input format. Below is the qualitative results two sampled frames to and Isc from the same video, where & is set to 4:
CoT-RVS Online Example 1 (LLaVA1.5-BB)
When parents see their children potential danger, they are often concerned and ready to help. Who is the parent in the scene? Chain of Thoughts: The image contains person riding four-wheeler in wooded area. There is clear indication of parent in the scene. Therefore, the justification of using this image as keyframe is No.
Imput query:
Figure A3: Input frame
Jo-
CoT-RVS Online Example 2 (LLaVA1.5-7B):
When parents see their children in potential danger, they often concerned and ready to help. Who the parent in scene?
Input query:
Chain of Thoughts: The image shows the man and the child in a field, with a fence in the background. The man appears to be concerned about the child's the motorcycle. Therefore, the justification of using
safety while they this image as a keyframe Yes.
Figure A4: Input frame 166-
APPENDIX F EFFICIENCY AND COST ANALYSIS
While efficiency is not main focus, CoT's complex reasoning usually requires higher computa- tional complexity, we include the efficiency and cost analysis with each CoT-RVSS component and the comparison with VideoLISA for your reference. All latency and memory measurements are based on 324-sized frames and 8 keyframe candidates for the MLLM selector. The API cost only applies to GPT-4o, which is closed-source model. Results are reported on RTX4090.
In specific, Tab. includes the latency, local memory footprint, and API cost of each module used in CoT-RVS and its online version. The end-to-end analysis of CoT-RVS is further presented in Tab. A2
For the online CoT-RVS, the per-freeeeee wer-frame latency within fframe sampling window is approximated by a+S3+(E-1)7 where is the MLLM inference latency, is the segmentation agent latency, ? is the tracking latency per frame, and S is the binary indicator for frame selection. This yields
20

---

Published as a conference paper at ICLR 2026
upper bound of a+S8 + + for average latency, where y is relatively small (0.04 in Tab. A1 and is hyper-parameetr. Please refer to Tab. A3 for actual latency measurements actoss different MLLMs.
Table Al: Latency of each CoT-RVS component
Module
Latency(s)
Local Memory footprint (GB)
API cost (USS)
L.VVAI,5-78
1.71
15.6
Gemma3-12B
63.95
24.8
GPT-4o
12.01
0.012
Seg Zero
4.57
18.1
SAM2
0.04!
2.6
Average fermframe tracking latency.
Table A2: End-to-end cost comparison with VideoLISA.
Method
Latency(s)
Local Memory footprint (GB) API cost (US$)
VideoLISA (OTSA)
3,46
11.7
HI--RVS-Gemma3-111
69.79
24.8
4PT-RV
17.85
18.1
0.012
Table A3: Ablation study for online keyframe selector on Refer-DAVIS-17. varies depending on the OpenAl server busyness. is implemented on an RT'X4090.
Model
J&F
Avg. Time (sec.)
Qwen-2.5.V2 VL--2 LLL B- Liuel el [2025a]
(2025
72.5
0.18"
73.9
0.24"
GPT-4o Hurst 2024
73.2
0,27*
APPENDIX G ROBUSTNESS ANALYSIS
The quality of keyframe selection is influenced by the choice MILM. evaluate the adherence of MILM to CoT prompt in offline setting, compute the percentage of outputs where the MLLM produces at least one valid object instance per expression (termed "valid outputs"). Results in Tab. A4 A4 show that larger, closed-source MLLMs such as GPT-4o offer significantly more robust and reliable reasoning, while smaller models like -LAVA115-78 and Qwen2.5-VLL VLL-BB fail to follow the output format deseribed in the CoT prompt (refer to Appx. A), thereby resulting in errors of data parsing. Prior works often underutilize this potential of large closed-source models in video segmentation tasks.
Table A4: Robustness analysis of CoT-RVS (original) with different keyframe selector models.
Datasct
MLLM
#Valid outputs/#Expressiinn
Percentage
J&F
J
LLaVA1.5-77
0/785
0%%
Qwen2.5-VL-3B
0/785
0%.
MeViS (val_u)
Gemma3-12B
715/785
91.1%
52.6
48.2
57.0
GPT-4o
761/785
96.8%
60.1
56.0
64.2
LLaVAI.5-7B
0/785
0%.
Qwen2.5- BBB VL-3B
0/785
0%;
Reason
Gemma3-12B
273/458
59.6%
50.7
47.5
$4.0
GPT-4o
444/458
96.9%
65.5
62.4
68.7
Incorrect output format.
APPENDIX H MODULARITY ANALYSIS
We have demonstrated in main paper that GPT-4o and Gemma3 can be used as MLLM keyframe selector. In this appendix, further validate the modularity of CoT-RVS, where adopt LISA Lai cet (003)) and Cutie Cheng et al. (2024) alternatives to Fseg and Frid. respectively. We conduct this ablation under the Refer-DAVIS- 2017 benchmark with Gemma3-122 keyframe selector. Tab. A5
21

---

Published as a conference paper at ICLR 2026
shows that both Seg Zero and SAM2 in default CoT-RVS can be substituted without significantly sacrificing the performance.
Table A5: Ablation study on Ref-DAVIS-17 using Co1-RVS- 12B with different Fseg and Foid-
F Feeg LISA-13B- 11A--33--llama
Fosd
J&FF
F
Cutie
68.8
63.0
72.5
IISA-I3B-Llamaa
SAM2
71.1
67.3
75.0
Seg- Seg-Zero-7B Seg-Zero-7B
Cutie
72.1
68.5
75.7
SAM2
74.6
70.9
78.3
APPENDIX FAILURE CASE
We demonstrate failure case of CoT-RVS in Fig. A5 Specifically, our empirical experience shows that: when multiple object instances are visible in the video, the GPT-4o fails to identify the individual instance. Instead, similar objects are packaged within an element of the output list, egg., "the six geese near the center of the image". This results in the further failure of the reasoning image segmentation model. Furthermore, it is difficult to separate different instances with textual descriptions, which will be sent the image segmentation models for each instance. Eventually, only one instance is presented in the output mask sequence in this case.
"My friend animals
the objects visible
my friend will probubly to pet
Input video
Output list
[object 1, keyframe: 2, object description: geese near the center the image")
CoT-RVS result
Figure A5: A failure of CoT-RVS.
APPENDIX J DISCUSSION
Keyframe selector choice. We adopt GPT-4o as the keyframe selector in the original CoT-RVS. while we also conduct experiments on additional open- open-sourced models for better reproducibility. We notice that smaller models, like LLaVAI.5-1338 Liu et al. (2023a) Qwen2.5-VL-3B Bai et al. (2025), failed to follow our detailed CoT prompts. Gemma3 12B produces reasonable results and performs well in the Referring VOS datasets, while falling short in difficult reasoning tasks. More comprehensive experiments of open-sourced models on larger models, such as Llama4, remain unexplored due the limited computational resources.
Dataset quality. We demonstrate an example from the Reason VOS benchmark Bai et al. (024) of CoT-RVS in Fig. A6 Specifically, notice that CoT-RVS segments the hoodie when asked for an object to alleviate the cold, while the ground-truth truth mask sequence refers to blanket. This showcases that some of the reasoning queries are too ambiguous even for humans. While the benchmark is not our major focus and contribution, we have attempted to avoid such ambiguous queries when sampling the T-Reason VOS dataset.
Future directions. In this work, we are excited to have made significant strides in addressing the challenges of Reasoning Video Object Segmentation (Reasoning VOS) and pushing the boundaries further into Reasoning Video Instance Segmentation (Reasoning VIS) and Online Reasoning VOS. The qualitative and quantitative results of CoT-RVS are indeed impressive, showcasing its potential
22

---

Published as a conference paper at ICLR 2026
"After sitting on the sefa and watching TV for while, started to feel a bit cold. Which object is most likely to help alleviate the cold?"
Input video
Ground-truth
CoT-RVS
Figure A6: Ambiguous case. Zoom in for details.
to enhance segmentation tasks. However, we acknowledge that our evaluation methods for Online Reasoning VIS and Reasoning VOS were not rigorous desired, where only presented the qualitative results instead quantitative results, primarily due to the limitations of available benchmarks in these tasks. This presents opportunity for future research, where aim to develop more comprehensive evaluation frameworks and datasets to thoroughly assess and refine the capabilities of CoT-RVS, ensuring it continues advance the field of video segmentation.
Social impacts. Potential misuse of the CoT-RVS application could include its in unauthorized surveillance or the creation of deceptive media. For example, individuals or organizations could leverage the technology track people without their consent, leading privacy violations and potential harassment. We believe that licensing can be useful tool to prevent such issues, where we will establish specific terms and conditions governing the use of the application by the time we release the code.
APPENDIX K MORE COMPARISON
Fig. A7 contains three examples from T-Reason VOS, We show that proposed CoT-RVS consis- tently outperforms the state-of-the-ar VidcoLISA Bai et al. (2024) on temporally sensitive queries.
23

---

Published as a conference paper at ICLR 2026
driving the road, but have change lanes right becouse white vehicle blocking my way. Which one vehicle causing me change lanes?"
Input video
VIdcoLISA
CoT-RVS
"The vehicle overtokes from left and heads different direction at the intersection.
Input video
VSdcoLISA
CoT-RVS
"The outomobile that merged into my lone abruptly and then left
Input video
VIdeoLISA
CoT-RVS
"If the animal were not on the right track, would probably be intervened and corrected by human. Which one is the wrong track?"
Input video
VIdeoLISA
CoT-RVS
Figure A7: More qualitative comparison. Refer the anonymous website in the supplementary materials for video demo.
24


Reviews:
[{"review_id": "jfJCmCh0X6", "reviewer": "Reviewer_bVYo", "summary": "This paper proposes a training-free referring video object segmentation by chain-of-thought capability of multimodal LLM in multi-agent framework, termed as CoT-RVS. CoT-RVS contains a multimodal LLM keyframe selector, a reasoning image segmentation model, and a video processor that track the masked selected object instances over the entire video. The CoT-RVS can be applied to offline video settings as well as online video settings. Experimental results show CoT-RVS outperforms prior methods on reasoning VOS benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper proposes a framework to use the chain-of-thought capabilitity of pretrained MLLMs with prompting to perform reasoning first and segmentation then. Experiments show this design outperforms prior approaches and maintain training-free, being versatile with proprietary models.\n\n2) The author also proposes a simple adaptation of this framework to online causal video settings, which is relatively underexplored by previous reasoning VOS methods but with great potential values.", "weaknesses": "There are some major concerns as listed follows:\n\n1) For online video settings, could the authors please report the inference FPS and latency of this pipeline? It seems requiring many components' working to finish segmentation and might take too much time when handling online videos.\n\n2) I was wondering whether this is pipeline agentic enough to be called an agent framework? The steps in this pipeline seem highly fixed and rule-based and lacking being autonomous enough. I would like to hear some author's discussion on this.\n\n3) To what extent the performance gain is credited to the SAM2 as a video processor? As one knows, SAM2 excels at video object segementation given prompts. Is this pipeline still effective (and outperforming prior methods) when using SAM1 as a segmentation model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5I8WoXSBNa", "reviewer": "Reviewer_7g6P", "summary": "This paper proposed a novel training-free framework called CoT-RVS. This method leverages the zero-shot CoT capability of MLLM to select the optimal keyframes for segmentation. This framework outperforms existing methods both qualitatively and quantitatively.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed CoT-RVS architecture can be seamlessly extended to online video stream processing (Online Reasoning VOS), showing strong practical potential.\n2.\tThe method primarily relies on the zero-shot capability of MLLMs, requiring no additional training and exhibiting good generalization ability.", "weaknesses": "1.\tFrom a pipeline perspective, the proposed method shows limited distinction from prior training-free works such as AL-Ref-SAM. The explicit introduction of Chain-of-Thought reasoning has also been explored in previous reasoning segmentation studies, indicating limited methodological novelty.\n2.\tAlthough the introduction emphasizes the importance of reasoning over temporal-semantic correlation, this motivation is not clearly reflected in either the method design or the experiments.\n3.\tThe MLLM keyframe selector and reasoning image segmentation model communicate through text, which may lead to inconsistencies between the described instance and the segmented target when multiple similar objects exist.\n4.\tThe method requires long-chain reasoning on each sampled frame, limiting the length and complexity of the input video, and potentially causing issues such as forgetting or hallucination. This restricts scalability to complex or long-duration videos.\n5.\tUsing GPT-4o as the base model makes the comparison with other methods unfair; parameter amount, inference cost, and latency introduced by the CoT mechanism should be reported.\n6.\tThe performance on Reasoning VOS tasks using LLaVA-1.5 and Gemma-3 is not significant, as shown in Table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KXFCxUtvQ4", "reviewer": "Reviewer_bGVR", "summary": "This paper introduces CoT-RVS, a training-free framework for Reasoning Video Object Segmentation. Instead of end-to-end finetuning, the framework is modular. It decomposes the R-VOS task into three distinct stages: an MLLM-based Keyframe Selector that uses zero-shot Chain-of-Thought prompting to perform temporal-semantic reasoning, an off-the-shelf Reasoning Image Segmentation Model to generate a key mask, and a Video Processor (i.e., SAM2) to track the object throughout the video. The paper demonstrates that this framework achieves state-of-the-art performance on several R-VOS benchmarks, with extensions to Reasoning Video Instance Segmentation and online streaming video.", "questions": "1. What is the key difference between this work and ThinkFast?\n2. According to Table A4 in Appendix G, LLaVA-1.5-7B and Qwen2.5-VL-3B are poor at key frames selection. How are the results achieved in the main paper?\n3. Why only CoT-RVS-LLaVA supports online mode, while Gemma and GPT-4o do not support?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea of a training-free, modular framework is compelling. This approach cleverly composes the strengths of large pre-trained models to bypass the need for task-specific finetuning.\n2. The quantitative results are strong, showing the framework outperforms prior SOTA methods on multiple benchmarks.\n3. The framework is shown to be flexible. The authors demonstrate its applicability beyond standard R-VOS, with extensions for Instance Segmentation and Online Reasoning VOS.", "weaknesses": "1. The primary weakness is that the framework is an integration of existing, powerful components rather than a new technical method. It \"stitches together\" a large MLLM, an image segmenter, and a video tracker. The system-level design is a valid engineering contribution, but the paper would be stronger if it more clearly articulated this as a novel contribution beyond \"stitching\".\n2. The paper's central claim, embedded in its title, is the power of the Chain-of-Thought reasoning process. However, this claim is supported almost exclusively by a qualitative ablation in Figure 8. The paper lacks a quantitative ablation study regarding CoT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jr492FP4Hr", "reviewer": "Reviewer_EGe3", "summary": "The paper introduces CoT-RVS, a zero-shot, and training-free framework for Reasoning Video Object Segmentation (Reasoning VOS). The primary goal is to segment objects in a video based on complex, implicit, or temporally-sensitive text queries.The core idea is to decouple high-level temporal-semantic reasoning from low-level segmentation and tracking. The method uses a modular, multi-agent pipeline:MLLM Keyframe Selector, Reasoning Image Segmenter, Video Processor.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A key aspect of the paper's design is its problem decomposition. Instead of using a single model to handle both temporal reasoning and pixel segmentation, it decouples the task into two stages: 1) First, an MLLM ($F_{key}$) performs high-level temporal-semantic reasoning to identify the most suitable keyframe and target description ; 2) Subsequently, specialized vision models ($F_{seg}$, $F_{vid}$) execute segmentation and tracking based on the clear instructions provided by the MLLM . This architecture shifts the challenge of complex temporal understanding from the pixel domain to the semantic reasoning domain.\n2. The paper provides sufficient experiments to validate its method. It was tested not only on four standard benchmarks (MeViS, Refer-DAVIS-17, ReVOS, ReasonVOS) but also on a specially constructed, temporally-sensitive T-ReasonVOS dataset to validate its core hypothesis. The experimental results show that the method achieves performance exceeding the compared SOTA methods on these benchmarks, particularly on T-ReasonVOS \n3. The framework is designed to be training-free and modular. This design allows for the replacement of the MLLM component (e.g., GPT-4o, Gemma3) without retraining. The architecture also supports an extension from single-object (VOS) to multi-instance (VIS) segmentation (via the MLLM outputting a list) and is refactored for an online version (via periodic keyframe updates ).", "weaknesses": "1. the paper's most compelling conclusion—superiority on temporally-sensitive tasks—relies heavily on the newly created T-ReasonVOS dataset . Given that this dataset was manually filtered by the authors and is not yet publicly released, this raises concerns regarding reproducibility and potential selection bias.\n\n2. the ablation studies are incomplete. While they validate the importance of the CoT process and sampling rate, the paper fails to fully explore its claimed modularity by, for instance, swapping the $F_{seg}$ (Seg-Zero) or $F_{vid}$ (SAM2) components to test the framework's generalizability\n\n3. the system is highly dependent on the MLLM's adherence to specific prompt formats.\n\n4. The framework's practical viability is questionable due to high computational latency and cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)