Decomposed Attention Fusion in MLLMs for Training-free Video Reasoning Segmentation
Authors: Su Ho Han
Jeongseok Hyun
Pilhyeon Lee
Minho Shim
Dongyoon Wee
Seon Joo Kim
Venue: ICLR 2026 Poster (2026)
Keywords: MLLMs
Segmentation
Training-free
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multimodal large language models (MLLMs) demonstrate strong video understanding by attending to visual tokens relevant to textual queries. To directly adapt this for localization in a training-free manner, we cast video reasoning segmentation as a video QA task and extract attention maps via rollout mechanism. However, raw attention maps are noisy and poorly aligned with object regions. We propose Decomposed Attention Fusion (DecAF), which refines these maps through two mechanisms: (1) contrastive object-background fusion and (2) complementary video-frame fusion. This method suppresses irrelevant activations and enhances object-focused cues, enabling direct conversion of attention maps into coarse segmentation masks. In addition, we introduce attention-guided SAM2 prompting for obtaining fine-grained masks. Unlike existing methods that jointly train MLLMs with SAM, our method operates entirely without retraining. DecAF outperforms training-free methods and achieves performance comparable to training-based methods on both referring and reasoning VOS benchmarks.

Published as a conference paper at ICLR 2026
DECOMPOSED ATTENTION FUSION IN MLLMS FOR TRAINING-FREE VIDEO REASONING SEGMENTATION
Su Ho Han'
Jeongseok Hyun'
Pilhyeon Lee²
Minho Shim
Dongyoon Wee
Seon Joo Kim
Yonsei University
Inha University
NAVER Cloud
ABSTRACT
Multimodal large language models (MLLMs) demonstrate strong video under- standing by attending to visual tokens relevant to textual queries. To directly adapt this for localization in a training-frr freee manner, we cast video reasoning seg- mentation as video QA task and extract attention maps via rollout mechanism. However, raw attention maps are noisy and poorly aligned with object regions. We propose Decomposed Attention Fusion (DecAF), which refines these maps through two mechanisms: (1) contrastive plementary video-frame framee fusion. This method suppresses irrelevant activations and enhances object-focusee cues, enabling direct conversion attention maps into coarse segmentation masks. In addition, introduce iitention-guider SAM2 prompting for obtaining fine-grained masks. Unlike existing methods that jointly train MI.LMs with SAM, our method operates entirely without retraining. De- cAF outperforms training-free methods and achieves performance comparable to training-baan methods both referring and reasoning VOS benchmarks.
fusion and (2) com-
INTRODUCTION
In recent years, Multimodal Large Language Models (MLLMs) (Lin al. 2023 Chen al. 2024 Zhang et al. [024 Bai et al. 2025 Wang et al., (024a) have rapidly advanced, demonstrating strong performance on challenging video QA benchmarks (Mangalam et al. 2023 Fu al.) (025). These advances reveal their ability to process temporal visual cues and perform complex reasoning over natural language queries. Such capabilities imply that MILMS may also possess inherent local- ization ability in videos, enabling training-free video reasoning segmentation, task that localizes objects corresponding to text-based queries requiring complex reasoning.
2025) have attempted Recent studies Yan et al. 2024 Bai al. 2024 Gong et al. 2025b Lin at.
(Lin al. [024), , SAM2 (Ravi et al. 2024h) for video reasoning segmentation by joint training through efficient fine-tuning meth- ods such as LoRA (Hu et al., 2022). However, these methods require model-specifie training and joint optimization of two foundation models, resulting in significant computation cost and limited generalization capability.
to adapt MLLMs and segmentation foundation models (e.g., SAM
Meanwhile, training-free free approach, Loc-Head (Kang et al.. 2025a). explores the localization ability of MLLMs in the image domain by selecting attention heads responsible for grounding. However, assumes the presence of single object referring object and selects heads based on spatial entropy, making extension to multi-object and temporal video data difficult. Moreover, it relies on heuris- ties to mitigate the visual attention sink phenomenon (Kang et al. (005b)). where certain regions consistently receive dominant attention scores regardless of the instruction, limiting its generaliza- tion across ML..Ms. These observations motivate us to directly examine how attention mechanisms within MILMS contribute to localization, without model-specific modification or training.
To obtain attention maps for object localization without relying on model- or -asciicc design, we start with attention rollout (Abnar & Zuidema 2020). Rollout aggregates attention weights across
"Equal contribution.
Code available at httphttts//// /teetss://gitie http//tttthhh..te com HWW.HHNNSSSSSR /DcAF

---

Published as a conference paper at ICLR 2026
(a) Contrastive Object-Backgronn Fusion
"a person erossing obstacles.
Attention map
Segmentation mask
Object
Background
Fusion
Attention
SAM
to
l
(b) Complementary Video-Frame Fusion
"Who served the ball this round?"
Attention map
Segmentation mask
Video
Frame
Fusion
Attention
SAM
to
Figure 1: Visualization of our method. (a) Noise irrelevant regions is suppressed by contrastive fusion with the background attention map. As shown in the first frame, background activations are removed, and the target object emphasized. (b) Video attention map captures temporal cues, while frame attention map highlights object-centric details. Their fusion resolves conflicts (e.g., identifying the server vs. the hitting player) and produces more consistent localization. The attention mask is obtained directly from the attention map while the SAM mask is generated by SAM2.
layers, revealing visual cues to which the MLLM attends when producing answers. Its applicability attention-based MILMS makes it plausible approach to probing localization ability. How- ever, since the rollout integrates signals from all heads, irrelevant regions and visual attention sinks often dominate, reducing the relative strength of object cues.
To overcome these limitations, we introduce Decomposed Attention Fusion (DecAF), which sup- presses noise and enhances object-focused signals by decomposing and fusing attention maps in two distinct ways. First, contrastive bojject-backrroundd fusion combines the object and background attention maps through simple subtraction. The object attention map obtained with a prompt fo- cusing on the target object, while the background attention map is derived from a contrastive prompt excluding this object. This design effectively suppresses irrelevant activations and highlights the target object signal, illustrated in Fig. (a). Second, complementary video-frame fusion leverages the distinct strengths of video and frame attention a multi-scale manner. Video attention captures temporal context, which is essential when the object is temporarily absent requires temporal rea- soning, but its coarse granularity limits performance on small objects. In contrast, frame attention provides object-centric, fine-gra -rained cues but lacks temporal coherence. By combining these two at- tentions, this fusion maintains clearer object focus while also leveraging temporal context, resulting in more robust attention maps that accurately localize the target object across the video.
2

---

Published as a conference paper at ICLR 2026
With the object localization attention map obtained from our two fusion methods, we first generate video object masks through simple thresholding, which provides reliable localization of the target object but remains coarse due to the low granularity of attention. To obtain denser masks, we extract point prompts from the attention map and apply SAM2 (Ravi et al. 2024). However, these coarse prompts, derived from spurious activations in the attention map, often produce false positives. To address this issue, we propose an attention consistency score that evaluates the alignment between the predicted mask and the underlying attention map, enabling unreliable segmentation masks be filtered out. As shown in Fig. [ this process transforms noisy attention map into a precise and reliable segmentation mask.
We evaluate DecAF across three MLLM families and five datasets. including three referring VOS datasets (Khoreva et al.
2018; Seo et al.) 2020; Ding et al. 2023) and two reasoning VOS Bai et al. 2024). DecAF consistently outperforms prior training- Kang et al. (2025a), both with and without SAM. In addition, the dense
datasets (Yan et al. 2024
approaches (Li et al. 2025
video object masks achieve performance comparable to training- training-based methods (Lai et al. 2024
Yan et al. 2024) Bai et al.)
2024 Lin et al. [025]] Gong et al. 2025bja) a). These results highlight that decomposed attention fusion offers simple and effective framework for training-free free video reasoning segmentation.
2 RELATED WORK
Multimodal Large Language Models. LLMs demonstrate powerful reasoning and cognition
pabilities (Brown et al. 2020 Dubey et al.] 2024: Yang et al. 2024). leading to the development
of MILLMs (Wang al. 2024b Google 2024 Team 2024| Liu al. 2024). These models, built on the transformer architecture (Vaswani et al.
2017). rely on the attention mechanism. Due to the quadratic cost of attention, some MLLMs firstly compress video tokens into fixed number of to- kens via lightweight modules (Jin et al. 2024; Song et al., 20244 Maaz et al. 2024). However, this token compression inevitably sacrifices one-graimed spatial information, unlike LLaVA-style mod- els (Liu et al.] 2023). which use a linear projector to preserve dense spatial features. More recently, Qwen2VL (Wang al., (024b) further advances this line by supporting native-resolution video puts, maintaining both aspect ratio and fine-grained visual details. In this work, we build on such models and focus on exploring the inherent localization ability of MILMS.
Text-conditioned Video Object Segmentation. Early research on referring VOS (RVOS) focuses on localizing the target object from simple textual expressions, typically describing appearance. Datasets such Ref-DAVIS (Khoreva al. 2018) and Ref- YouTueeuobb (Seo et al., 2020) were designed for this setting and only cover single-object cases. More recently, Me (Ding et al,, 2023) introduces motion-centric and more challenging scenarios, including cases where the referred object is absent or where multiple candidates match the expression.
With the advent of powerful MILMs (Liu al. 2023), video reasoning segmentation has emerged, targeting complex expressions that extend beyond appearance or motion cues and require reasoning over world knowledge and temporal context (Yan et al. 2024 Bai al.) 2024). To address this, existing approaches adapt pretrained MLLMs to RVOS via lightweight finetuning strategies such as LoRA (Hu al. 2022). and integrate them with segmentation model such as SAM (Kirillov al. 2023 for precise mask generation, often requiring full finetuning of the mask decoder (Gong et al., 2025b
Lin al. 2025). In contrast, we leverage MLLMs and SAM in a training free manner.
Training-free Text-to-Viste
Grounding with MLLMs. Recently, MLLMS have been studied for training-fres visual grounding tasks (Lin et al. 2024 [i al. 2025) Kang et al.) 2025a). VL.
SAM (Lin et al.) 2024) and TAM (Li et a 20255 leverage the attention rollout mechanism (Abnar & Zuidemal 2020) to localize objects in images, with VL-SAM further refining the masks using SAM.
Both methods identify all objects by enumerating categories during MLLM decoding. In contrast,
Kang et al. (2025a) proposed a method that selects specific attention heads responsible for local- ization, enabling direct grounding of the object referred to by the given expression. However, this method shows poor generalization: attention heads identified on referring datasets transfer poorly and yield low accuracy on reasoning-intensive datasets.
3

---

Published as a conference paper at ICLR 2026
Attention Rollout
Contrastive Object-Background Attention Fusion
Decomposed Attention F Fusion
efjecct referred the gives expression?
MLLM
Ving
Veng
yeer
hyyers
MLLM
Describe the
Attention Rollout V-Max
bar eground of
the vidao.
MLLM
pho
Vesd
(c) Complementary Video-Frame Attention Fusion
Upscaae
<nnwwer>
Rollout Matrix (R4)
MLLM
Vest
1//m
Vis
Reshape
Map
MLLM
Vyve
tokens
Figure 2: Overview of DecAF. (a) Attention rollout with our V-Max normalization produces rollout matrix that accumulates attention across layers, from which visual-token scores for the final query token extracted as attention maps for grounding. (b) Contrastive fusion suppresses attention scores on background regions. (c) Complementary fusion integrates video- and frame-level cues. (d) These fusion methods are combined into the full pipeline to refine noisy attention maps.
3
METHODOLOGY
3.1 OVERVIEW
Given a video and text instruction referring an object, framework produces segmentation masks of the target object(s). The pipeline consists of two stages. First, coarse segmentation masks are obtained from attention score maps computed in an MILM. Second, fine-grained dense segmen- tation masks are generated using SAM conditioned on these attention maps. In the first stage, we propose Decoupled Attention Fusion (DecAF), illustrated in Fig. / which integrates contrastive and complementary fusion strategies with tailored prompting methods. To obtain the attention scores, we adopt attention rollout (Abnar & Zuidemal 2020) with a new normalization technique designed for MLLMs. In the second stage, we introduce a training-free SAM2 prompting pipeline guided by attention maps (Fig. 3. Point queries are first selected by thresholding the attention maps, and SAM2 generates mask tracklets for each query. These tracklets then evaluated with the proposed attention consistency score, which measures whether the predicted masks consistently overlap with high-attention regions across frames. The resulting scores used rank and tracklet candidates.
3.2 ATTENTION ROLLOUT WITH VISION-AWARE NORMALIZATION
We trace the influence of visual tokens on the model's output by propagating attention scores through the transformer layers of MLLMs. To better capture languag--conitiones grounding, we modify the standard attention rollout (Abnar & Zuidema, 2020) with a vision-aware normalization scheme.
Standard rollout. Given the attention tensor A (1) B from the I-th transformer layer, where h is the number of heads and N is the total number of tokens, the hise averaged attention Eq. D and the residual connection is incorporated by adding the identity matrix as Eq. 2 This reflects that a token can either propagate its own representation through the skip connection or attend to other tokens via the attention mechanism. The rollout matrix is then recursively accumulated across layers as Eq. 3 starting from the initialization R()) (1), and producing R(4), which encodes how information flows from each token every other token throughout the network.
matrix is computed
A (I)
A (A + 1)/2. (2)
R (11), (3)
Head-wise weighted aggregation. To reduce the effect of noisy heads, we assign a weight to each head based the strength of its vision attention. For each layer let the original attention tensor
4

---

Published as a conference paper at ICLR 2026
before aggregation be denoted A B*XNN(NN++NN) where N₂ and N, indicate the number of visual and textual tokens, respectively. From A (1) the vision block extracted: A E RANNNN The maximum value over the visual token dimension is then computed as:
m
N. máx
A (1) b, ==1
m (1) E IRANN
(4)
Averaging m (1) over the token dimension finally produces the head-wise weight vector, w( E R The weights are normalized so that max, (axx,u,,)) (w%) 1, and these normalized weights are used to aggregate the heads, resulting in the final attention weights Á (4) -NN((N++N.))
3.3 DECOMPOSED ATTENTION FUSION
The attention rollout mechanism quantifies token-to-token influence. To perform text-conditioned video reasoning segmentation, we cast the task as video question answering, where the goal is to identify category of the object in video referred to by the text instruction. We then exploit the rollout matrix values with the last token as query and visual tokens as keys, using them as attention scores that indicate how visual tokens contribute to answering the video QA, as shown in Fig. (a).
However, the rollout matrix aggregates signals across all heads and layers and is too noisy to serve directly segmentation score map. In addition pervasive noise, we observe strong activations in irrelevant regions, known as the visual attention sink phenomenon. To address this, we introduce Decomposed Attention Fusion (DecAF) to obtain cleaner, object-focused attention maps. As shown in Fig. 2 (d). DecAF applies contrastive fusion within each modality (video and frame) in paral- lel, followed by complementary fusion after upscaling the video-level attention maps match the frame-level size. The resulting attention maps are then converted into coarse segmentation masks via thresholding. Here, we explain with shortened prompts, but the full prompts are in the Appendix.
Contrastive Object-Backgrount Attention Fusion. A key challenge of using attention maps for segmentation is that irrelevant regions often receive very high scores, which cannot be suppressed by simple thresholding. Such visual attention sinks frequently appear regardless of the given instruc- tion. To address this issue, we introduce contrastive fusion, which contrasts attention maps obtained from object-focused and background focused prompts. Subtracting background from object atten- tion effectively highlights the target region while suppressing spurious responses.
The specific process follows Fig. (b). The object attention map is obtained by prompting the model to identify the target object category from the referring expression using an object-focused prompt template, "What is the main object referred to in the given expression?" The rollout attention weights from this response form the positive map. For the background attention map, we first use a cockround-focused prompt such as "Describe the background scene of the video." However, this may cause the target object to be mistakenly attended when it is not the main salient object but still appears in the background. To mitigate this, we additionally insert the identified category Oname into the template, to explicitly exclude the target object from the background attention map. The rollout attention map from this response serves the negative map.
Both object and background attention maps are reshaped into (T, Hp. Wp), where T is the number of frames and (Hp, W;) is the patch grid. Before fusion, Gaussian smoothing applied to both maps to mitigate the sparsity of raw attention weights. The contrastive map, Vetr, is then computed by subtracting the background map from the object map, clamped to remove negative values. Finally, min- max normalization is applied to scale the values into the [0, 1] range.
Complementary Video-Frame Attention Fusion. The softmax operation in attention enforces that all token scores sum to one. With video inputs, this constraint spreads attention across large number of tokens, yielding maps that are relatively sparse and shaped by temporal context. In contrast, with image inputs, attention is concentrated on fewer tokens and tends to emphasize object-centric spatial details. We therefore exploit these complementary properties of video- and frame-level attention maps to achieve more robust localization.
As shown in Fig. (c), we apply the identical attention rollout pipeline individually to the video and frame modalities, where each frame in the image modality is processed along the batch axis. This maxed-modalite design introduces two modifications in the contrastive fusion step. (1) Since background prompting requires an object category, we select single prediction by aggregating out-
5

---

Published as a conference paper at ICLR 2026
High-res frames
Mask tracklets
Atin maps
Point prompts
2 Epg
SAM2
sfrk: 0.98
stri 0.29 X
sahj
Mask Tracklet Scoring
Overview of our SAM prompting pipeline with attention maps. (1) Point queries for SAM2 are obtained from attention maps via thresholding (Tpy). (2) During mask propagation, highly overlapping masks are removed. (3) Spurious mask tracklets are removed using our scoring method.
Figure 3:
puts from both video- and frame-level inputs with object category choice prompt. (2) For min-max normalization, we normalize frame-level maps independently per frame, while video-level maps are normalized globally across all frames. Finally, the two sets of maps are fused by simple averag- ing, combining the global temporal context of video attention with the spatial precision of frame attention.
Our video- fraoo--frmeeaao decoupled prompting enables multi-scale processing, allowing higher-resolution inputs to be used for frame attention. Recent MLLMs, such as InternVL. and LLa LLaVA-NeXT, support dynamic image resolutions via tiling, whereas video inputs remain constrained to lower resolutions (e.g., 448). In contrast, Qwen VL supports native resolutions for both video and image; this case, simply double the width and height for image inputs. To align modalities, low-res video attention maps are upsampled to match the frame-level resolution before fusion.
3.4 SAM2 PROMPTING WITH ATTENTION MAPS
attention maps, V E &T,,H,,,,, where T, is the number of sampled frames and (Hp. Wp) is spatial resolution of visual token grid. Since this resolution is coarse, we introduce a method to prompt SAM2 using the attention maps to produce fine-grained object masks, M E I x Here, we use full frames at high-resolution, rather than the sampled frames used in the MILM. The overall pipeline is illustrated in Fig.
After DecAF process, we obtain the
Point Query Generation. Since SAM requires spatial prompts, we generate point queries directly from attention maps guide object mask prediction. We select visual tokens with attention scores above a threshold Tpy and use their center coordinates as point queries. The set of point queries is defined as Eq. 5 where O, and Og denote half the token width and height, respectively, ensuring that each point corresponds to the token center.
P = (t, - oy, I x oz) VLy. 2 Tpg)
(5)
Frame-wise Prompting and Propagation. Starting from the first frame, the point queries are fed sequentially to SAM2 which produces frame-level masks and propagates them through subsequent frames. This process generates a video mask for each point query (p.). denoted as M; E RTxxIxW together with its confidence score, s SSAM, predicted by SAM2.
Naive thresholding may generate a large number of redundant masks. To reduce computation, we assign object score s obj Vp. +SSA for each predicted mask, where V is attention score of p;- We then apply non-maximum suppression (NMS) using this object score. Two masks are considered overlapping if their loU exceeds threshold (e.g., 0.7), and the one with the lower score is removed. a propagated mask from previous frames highly overlaps with a new mask in the current frame, retain only the one with the higher object score. Through this process, obtain K video mask tracklets, where K << [P), effectively reducing redundancy while keeping high-quality candidates.
Mask Tracklet Scoring and Selection. Since attention maps at low resolution, point queries are not spatially precise and may fall on background regions. Nevertheless. SAM often produces high-confidence masks from such queries (e.g., walls), leading false positives. To suppress these

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of MLLM-based text-conditioned VOS methods that directly compute masks from attention maps (Attn Mask). All methods are training-free free and grouped by MLLM.
Ref-DAVIS
ReasenVOS
ReVOS (Ovant))
ReVOS (eliniig
ReVOS (Ressoting)
Method
MLLM
J&.F
J
F
J&.F
J
F
J&F
J
J&F
J
F
J&FF
F
Loc-Head (CVRe15))
LLaVA-7B
23.2
14.5
14.1
10.2
12.6
10.2
14.1
10.8
11.1
9.5
Loc--Had (cxBeSs)
LLVA--VV7B
15.9
14.1
17.7
12.3
11.2
13.4
13.1
12.0
14.3
14.9
13.8
16.0
11.4
10.3
12.5
DecAF [Ouss
LLaVA-Ov-7B
21.6
24.0
19.1
17.2
19.4
15.0
15.6
17.6
13.7
16.9
19.5
14.4
14.3
15.6
12.9
Loc-Head [CVR)25]
19.0
24.1
14.0
14.1
15.6
12.6
14.6
16.8
12.4
16.5
19.3
13.7
12.7
14.2
11.1
DecAF
IntemVL33 8B
20.7
26.0
15.3
18.4
21.8
14.9
16.7
20.4
13.0
18.2
22.5
13,9
15.2
18.3
12.1
TAM J[CV-15]
Qwen2VL-PP 7B
2.5
1.8
3.3
2.8
2.8
2.9
2.8
2.9
2.6
2.9
3.1
2.7
2.7
2.8
2.6
Loc-Head (eveets)
Owen2V1.-78
11.1
13.2
16.5
10.7
7.2
DecAF [0m)]
Qwen2VL-7BB
20.0
24.8
15.2
13,8
17.5
10.0
15.2
19.8
10.7
17.5
23.2
11.8
13,0
16.4
9.6
TAM (CCC125)
Qwen2.5VL-78
3.5
2.8
4.3
3.7
3.4
3.9
4.0
4.0
4.0
4.1
4.1
4.1
3.8
3.8
3.8
Loc-Head (evиeаs)
Qwen2.5VL
19.1
13.1
16.9
11.0
8.3
DecAF [ous)
Qwen2.5VL VLL-BB
25.3
32.0
18.6
20.6
26.0
15.3
20.2
26.0
14.5
22.1
28.8
15.4
18.3
23,1
13,5
cases, we evaluate each mask tracklet using attention consistency score (s**), which measures whether the mask consistently overlaps with high-attention regions across frames.
For each tracklet i, we then compute combined tracklet score, sir = Avg( (Ve p. SAA. s.a). Tracklets with sfn 2 Tark are retained and propagated across all video frames via SAM2 to generate the final dense segmentation masks. This procedure naturally supports both single-object and multi- object localization by treating each high-confidence query as an independent object hypothesis.
The computation of n as follows. First, we obtain a binary mask for each frame by thresholding the attention map at its mean score f4, (Eq. ( Second, we assign the negative maximum attention score per frame, &e max(Vr...)) Finally, each mask tracklet is downampled to the attention map resolution, M, E RT x H xW, and gac is computed ratio of inner products (Eq.
regions below Pt, (Eq. 7 penalizing low-attention areas.
(8, where (5 denotes the tensor inner product.
Attn [1,
Vt.g.z 1e: pe: otherwise.
(6) Viy.z
ViggZz VLy. 24, (7) 8
(M, V)
(8)
(0,
otherwise.
(M
EXPERIMENTS
4.1 EVALUATION SETTING
Datasets and Evaluation Metrics. We evaluate our method on three referring VOS datasets: Ref- DAVIS (Khoreva al. 2018), Ref-YouTubhe Tob (Seo al. 2020). and MeViS (Ding al. 2023). In addition, we validate it on two reasoning VOS datasets: Reason (Bai et al. (024) and ReVOS (Yan et al. 2024). Note that Reason VOS provides only a test set and used for zero- shot evaluation, whereas the other datasets include training data. For evaluation, we employ the standard VOS metrics: region similarity (T), contour accuracy (F), and their mean (J&F).
Implementation Details. For mask generation directly from attention maps, we apply Otsu's adap- tive thresholding method (Otsu et al.) 1975). By default, attention rollout starts from the middle LLM layer (e.g., 14 for 28 layers of Qwen2.5VL-7B) and SAM prompting threshold values of Terk = 0.8 and Tpg = 0.8. We use publicly released MLLM checkpoints and the SAM2 largee.
4.2 COMPARISON WITH EXISTING METHODS USING MLLMS
Mask without SAM. We evaluate segmentation masks obtained directly from MILM attention maps using simple upscaling and thresholding, and compare them with existing methods in Tab. Uniformly sampled 16 frames are used here. TAM (Li et al., 2025) performs poorly due to its strong dependence on predicted word tokens, making it unable to reliably ground expressions under our object-focused prompt. Further analysis of TAM's failure cases is provided the Appendix. Hoc-Head (Kang et al. 2025a) is also designed for text-conditioned segmentation, but operates in the image domain. Our method consistently outperforms and datasets, with especially large margins datasets require complex reasoning. This suggests
across different MLLMs

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of MLLM-based text-conditioned VOS methods. The upper gray rows corre- spond to boining-based methods, while the lower colored rows correspond to training-free free methods.
Method
MLLM
Ref-DAVIS
ReasonVOS
ReVOS (0waa))
ReVOS (kainning)
ReVOS (keaonny)
J&F
J
F
J&F
J
F
J&F
3
F
J&F
J
F
J&.F
J
F
LISA (CVPR)34)
LLaVA-7B
64,8
62.2
67.3
31.1
29.1
33.1
40,9
39.1
42.7
45.7
44.3
47.1
36.1
33.8
38.4
VISA preves
ChatUniVi-77
69.4
66.3
72.5
46.9
44.9
49.0
50.9
49.2
52.6
43.0
40.6
45.4
VidcoLISA (NoutRS
LLaVa-Phi-3-V
68.8
64.9
72.7
47.5
45.1
49.9
GLUS (CVPR/25)
LLaVA-7B
49.9
47.5
52.4
54,9
52.4
57.3
58.3
56.0
60.7
51.4
48.8
53.9
VRS-HQ (cv1925)
VhatUniVi-7B
76.0
79.4
54,9
59.1
59.8
58.7
Veason-RI (a0i-25000
QVL72.5VL-7B
59.9
56.0
63.8
61.3
58.2
64.4
63.6
60.7
66.5
59.0
55.8
62.2
Loc- LoccHeed (CV21223
LL
55.6
51.5
59.6
37,1
32.9
41.4
35.3
31.1
39.6
39.2
35.0
43.4
31.5
27.2
35.7
Loc-Ileas Ieead (CV2122)
LLaVA OV-7B
24.2
21.3
27.1
32.6
29.7
35.4
31.7
28.3
35.1
32.8
29.2
36.5
30.6
27.4
33.7
DecAF (0ir]
M..VA-OV OV-7B
59.4
54.8
64.0
52.8
49.3
56.3
40.0
35.8
44.1
43.4
39.1
47.6
36.6
32.6
40.7
Loc-Head (CVPR
66.3
62.4
70.2
44.3
41.0
47.5
43.7
39.9
47.5
46.7
42.9
50.6
40.7
36.9
44.5
DecAF [0us]
VL-S
62.8
68.6
58,9
55.1
62.7
47.4
43.7
47,9
55.5
39.5
46.8
Loc-Head (CVPR225
Owen2VL-7B
61.9
65.8
34,0
31.8
44,0
40.8
49.1
56.2
32.6
38.2
DecAF [0zn]
Owen2VL-7B3
64.1
59.4
68.9
52.5
49.0
56.0
45.3
41.6
49.0
52.7
48.9
56.4
37.9
34.3
41.5
Loc-HHea (CV922))
Dwen2.5VL.22
64.6
60.2
68.9
41.1
37.9
44.3
47.0
43.3
50.7
53.1
49.3
56.9
40.8
37.2
44.4
DecAF pooe
Qwen2.5V1
79.5
63.9
67.2
54.2
50.1
58.7
5488
45.4
53.9
Table 3: Comparison on additional datasets.
MeVis
Ref- YTVOS
Method
MLLM
J&F
J
F
J&F
J
VISA [иXC)))
Chat-UniVi-79 7B
44.5
41.8
47.1
61.5
59.8
63.2
VSdcoLIIA INoitP5:4 34)
LLaVA-Phi-3-V
44.4
41.3
47.6
63.7
61.7
65.7
GLUS (CVPR-25)
LLaVA-7B
$1.3
48.5
54.2
67.3
65.5
69.0
VRS-HQ (CV
Chat-UniVi-7B
50.6
47.6
53.7
70.4
68.3
72.5
Veason-RI Jasi
Qwen2.5VL-7B
52.2
48.4
56.0
Loe-Head (c)2
Owen2.5VL-77
39.4
35.2
43.6
51.0
46.8
55,2
DecAF (oar)
Qwen2.5V1.
48.1
44.0
52.1
59.9
56.2
63.5
that our method generalizes more effectively to reasoning-intensiin scenarios, whereas localization heads rely on heuristic head selection and thus exhibit limited robustness.
Despite these relative improvements, attention maps remain very low resolution, and the resulting scores are still below those of conventional segmentation models. In particular, contour accuracy (F) is much lower than region similarity (J), reflecting the inability of low-resolution attention maps to capture fine-grained boundaries - opposite to the trend observed in segmentation-sseciaiize models. These findings suggest that attention masks alone are too coarse for precise segmentation, but they provide a sufficient coarse localization signal to guide SAM prompting (Ravi et al. 2024).
Mask with SAM. We evaluate dense segmentation masks for all video frames using SAM2, and re- port the results in Tabs. [ including both training- based and training-free free methods. Loc- Loc-Head proposes its own SAM prompting method, but it is developed under single-object assumption: the largest bounding box (bbox) is obtained using the convex hull algorithm. Also, prompting with an imprecise bbox may result in segmenting non-target objects. On Ref-DAVIS with LLaVA-7B, Loe-Head's bbox prompting achieves 30.3, whereas our prompting achieves 55.6. This large gap highlights the advantage of our prompting method; thus, we adopt for all subsequent comparisons.
In regards to training-free methods, our method outperforms Loc-Head across different MLLMs and datasets, including the additionally presented MeViS and Ref-YTVOS (Tab. 3.. Although Loc-Head achieves slightly higher scores on Ref-DAVIS with Intern its performance drops substan- tially on ROasonVOS, which requires handling more complex expressions.
Compared with baaning-based methods, our method achieves comparable or even superior perfor- mance. On Ref-DAVIS, our method with Qwen2.5VL-7B outperforms VISA and VIdeoLISA by 5.8 and 6.4 J&F. respectively. On MeViS, our method achieves 48.1, outperforming both VISA (44.5) and VideoLISA (44.4), and reaching performance close to VRS-HQ (50.6). It is worth noting that recent state-of-the-an models (GLUS, VRS-HQ, Veason-RI) leverage trained keyframe selec- tion modules, whereas our method simply employs uniform sampling. Even with this difference, approach surpasses all training- based methods on Reason despite Veason-RI additionally training the same MLLM (Qwen2.5VL) with an RL-based algorithm. This clearly manifests the effectiveness of method.
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study of decomposed attention fusion.
(a) baeker-backgooud contrasting
(b) Video- frame complementing (c) Mclti-scale complementing
MLLM
Obj Bg
Attn Mask
SAM Mask
MELM Vid Frm Ref-D ReasV
MLLM MS
Ref-D
Reas V
Ref-D
ReasV
Ref-D
ReasV
46.0
50.2
54.0
53.7
IVL3
60.0
50.8
IVL3
IVL3
/
12.4
54.7
62.8
58,9
62.8
58.9
V
20.7
18.4
62.8
58.9
65.9
58.6
72.4
60.5
QVL2.5
/
/
14.5
13.8
61.9
58.4
QVL.25
67.4
58.2
QVL2.5
75.2
63.9
25.3
20.6
75.2
63.9
75.2
63.9
Table 5: Ablation study of attention rollout.
Table SAM prompt- ing threshold values.
Table 7: Ablation study of
(a) Rollout method
computing attention consis- tency score (s
Method
Ref-D
Reas'V
Ttrk
7p
Ref-D
ReasV
Rollout Abuus Zuidena 2020
68.4
56.8
Thresh (2) Penalty (8) Ref-D ReasV
0.7
71.0
$9.9
Rollout Max et. 2024
72.9
60.9
0.7
0.8
75.0
62.1
Not Use
60.0
52.9
Rollout
V-Max (Uurs)
75.2
63.9
0.7
74.3
65.7
(b) Starting LLM layer for rollout
0.8
0.7
70.6
61.1
Mcan
Oss
65.1
67.9
61.1
0.8
0.8
75.2
63.9
Mcan
75.2
63.9
Layer index
Qwen2.5VL-78
IntemVL3-SB
0.8
0,9
74.3
65.9
Ref-D
ReasV
Ref-D
ReasV
0.9
0.7
71.6
61.7
0.9
0.8
74.5
64.9
(1/4)
69.2
62.8
62.1
56.8
0.9
0.9
74,2
66.4
(2/4)
75.2
63.9
62.8
58.9
21 (3/4)
73.6
64.1
55.8
60.1
Table 8: Evaluation on other sizes of MLLMs.
Ref-DAWe DAVIS
ReasonVOS
ReVOS
ReVOS
ReVOS
Size
J&&F
J
J&F
J
F
J&F
J
F
J&.F
J
F
J&&F
J
2B
53.5
48.1
58.9
54.1
50,7
57.6
38.1
34,0
42.2
42.3
38.2
46.5
33.9
29.9
38.0
VL3
8B
62.8
56.9
68.6
58.9
55.1
62.7
47.4
43.7
51.2
51.7
47.9
55.5
43.2
39.5
46.8
14B
63.3
58.3
68.4
65.6
62.2
68.9
47.0
43.0
51.0
51.3
47.2
55.4
42.7
38.8
46.6
Owen2VL
7B
64.1
59.4
37.1
46.6
68.9
52.5
47.6
44.4
49.0
56.0
45.3
30.2
41.6
34.0
49.0
52.7
48.9
33.0
56.4
37.9
23.3
34.3
19.8
41.5
QwLn2.5VL
53.6
49.6
39.7
43.8
29.1
7B
75.2
70.9
79.5
63.9
60.5
67.2
54.2
50.1
58.2
58.7
54.8
62.6
49.7
45.4
53.9
4.3 ABLATION STUDY
We Qwen2.5-VI.-7B (QVI.2.5) and Intern (IVI.3) models, and Ref-DAVIS (Ref-D) and ReasonVOS (ReasV) as datasets. By default, results are reported with QVL2.5 and J J&F.
Decoupled Attention Fusion. We evaluate the effectiveness of DecAF. First, we examine object- background contrastive fusion (Tab. 4aj. which substantially improves attention mask accuracy on both referring and reasoning VOS datasets (e.g., 12.4 20.7 and 14.5 - 25.3 on Ref-D) by sup- pressing the irrelevant regions. Similar improvements are also observed for SAM mask accuracy.
Second, video-frame complementary fusion (Tab. 4b further enhances accuracy. For Qwen2.5VL. video-only and frame-only inputs yield 65.9 and 67.4 on Ref-D, respectively, whereas combining both achieves 75.2. Consistent gains are also observed on ReasV and with Int.rn
For video-frame fusion, we adopt multi-scale scheme that leverages higher resolution inputs at the frame level. While Qwen2.5VL supports native resolutions, InternVL and LLaVA-OV models require fixed input size but can handle dynamic high resolution image inputs through tiling. As shown in Tab, 4c, this multi-scale fusion brings additional improvements, particularly for InternVL, whose attention map resolution is very low without tiling.
Attention Rollout. Tab. 5a compares our method with previous attention rollout methods. Our vision-aware weighted normalization further improves accuracy over the method of et al. 2024). We also evaluate different LLM layers for rollout (Tab. 5b). and observe that selecting a middle layer yields the best overall performance.
SAM Prompting. Tab. reports the results with different threshold values to filter point queries (Tpy) and tracklets (Tirk) used in the SAM prompting process. Increasing Tpy helps filter

---

Published as a conference paper at ICLR 2026
Exp: man performing headspin.
Attention
Attention
Mask
SAM Muskk
Exp: The cars stoped because of what?
Attention
map
Attention
Mask
SAM
Mask
Figure 4: Qualitative results of DecAF. Visualization of refined attention maps, coarse attention masks, and dense SAM masks obtained through attention-guided prompting.
target objects or background regions, but too high value may result in missing points. While the optimal threshold combination varies across datasets, our method remains substantially robust to threshold choices, and we use Tpg 0.8 and Terk 0.8 across all datasets and models.
Mask Tracklet Scoring, As shown in Tab. 7 we ablate the attention consistency score (s" which contributes to the mask tracklet score (s Omitting ge and relying only on the object score soby leads to a significant accuracy drop. For Atttn we compare Otsu thresholding and simple averaging to obtain and for V, we evaluate both with and without the penalty term (o). Without Otsu thresholding yields higher accuracy than mean thresholding, as it produces tighter object masks. In contrast, with 8, mean thresholding performs better, it tends to cover the entire object region, while any included background has low attention scores and low s"c.
MLLM Scalability. Tab. 8 shows that larger MLLMs generally yield better performance. In- temvVL3 improves from 53.5 to 63.3 on Ref-D and 54.1 - 65.6 on ReasV while Qwen2.5VL also scales effectively, with its 7B model achieving the best results across all datasets.
Qualitative Results. Fig. illustrates the qualitative results of DecAF. Our method consistently produces refined attention maps that are precisely aligned with the referred target objects. These sults demonstrate that DecAF effectively leverages the inherent localization capabilities of MILMs without any additional training. Furthermore, attention-guided SAM2 prompting successfully produces fine-grainee dense masks. Detailed visualizations for diverse scenarios, including failure cases, are provided in the Appendix.
CONCLUSION
We explore the intrinsic localization ability MILMs by casting video reasoning segmentation as video QA. Based on the attention signals used for answering, we propose Decomposed Attention Fusion (DecAF), which produces robust attention maps convertible into coarse segmentation masks. To obtain fine-grained dense masks, we further introduce an attention-guided SAM2 prompting pipeline. Without training, our method surpasses vriniin-basee methods on ReasonVOS, demon- strating that MLLMs' reasoning capability directly translates into stronger localization for complex expressions. More broadly, since DecAF is agnostic to the underlying modality, it can be extended to localization tasks beyond vision promising direction with emerging Omni-MLL.Ms.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partly supported by Institute of Information & communications Technology Plan- ning & Evaluation (IITP) grant funded by the Korea governmen((MSIT) (No. RS-2024-00457882) the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) RKS-2025-00554790 and Artificial Intelligence Graduate School Program grant funded by Yonsei University (RS-2020-1 1201331).
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the Association for Computational Linguistics (ACL.), 2020.
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint 2rIvv22502.1322. 2025.
Zechen Bai, Tong He, Haiyang Mci, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language instructed reasoning segmentation in videos. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901, 2020.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisto linguistic tasks. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 24185-24198 24198, 2024.
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy, Mevis: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of International Conference Computer Vision (ICCV), pp. 1694-2703 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint urXiv:2407.21788 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal Ilms in video analysis. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 24108-24118, 2025.
Sitong Gong, Lu Zhang, Yunzhi Zhuge. Xu Jia, Pingping Zhang, and Huchuan Lu. Reinforcing video reasoning segmentation think before it segments, 2025a. URL https:///axxi:
orq/abs/ 2aabs///5.... 11538
Sitong Gong, Yunzhi Zhuge, Lu Zhang, Zongxin Yang. Pingping Zhang, and Huchuan Lu. The devil is in temporal token: High quality video reasoning segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025b.
Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint wrXiv:2403.05530. 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proceed-
ings of International Conference on Learning Representations (ICLR), 2022. URL hetps:
//openreviee ew. www.:n///oorum..mmm/eeeeeeerrrr
11

---

Published as a conference paper at ICLR 2026
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceed- ings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 13700-13710, 2024.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Scong Jae Hwang. Your large vision-language model only needs few attention heads for visual grounding. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9339-9350, June 2025a.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.0332/, 2025b.
Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language refer- ring expressions. In Proceedings of Asian Conference on Computer Vision (ACCV), pp. 123-141. Springer, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, et al. Segment anything. In Proceed- ings of International Conference on Computer Vision (ICCV) pp. 4015 4026, 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9579-9589 June 2024.
Yi Li, Hualiang Wang, Xinpeng Ding, Haonan Wang, and Xiaomeng Li. Token activation map to visually explain multimodal llms. In Proceedings of International Conference on Computer Vision (ICCV), 2025.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Lang Lin, Xueyang Yu, Ziqi Pang, and Yu-Xiong Wang. Glus: Global-local reasoning unified into
single large language model for video segmentation. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2025.
Zhiwei Lin, Yongtao Wang, and Zhi Tang. Training- training-frree open-ended object detection and segmen- tation via attention as prompts. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, Visual instruction tuning. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 36, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 26296-26306, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the Association for Computational Linguistics (ACL), 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. Proceedings of Advances in Neural In- formation Processing Systems (NeurIPS), 36:46212-46244 2023.
Nobuyuki Otsu et al. A threshold selection method from gray-level histograms. Automatica, 11 (285-296):23- 27, 1975.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint 2024. URL URI.htttss /arxivvvvvvvv.. org www./ra.////oooooooo/ooooooeaaae abs 2408 00714
12

---

Published as a conference paper at ICLR 2026
Seonguk Seo, Joon- Young Lee, and Bohyung Han. Urvos: Unified referring video object segmenta- tion network with a large-scale benchmark. In Proceedings of European Conference on Computer Vision (ECCV), pp. 208-223. Springer, 2020.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18221-18232 2024.
OpenAl Team. Gpt-4o system card. arXiv preprint arXiv:2410021276, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1219,, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing lsionnlanguage model's s perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024b.
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves, Visa: Reasoning video object segmentation via large language models. In Proceedings of European Conference on Computer Vision (ECCV), 2024.
Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10771, 2024.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024. URL 30- lava next-exeeeee -iideoi
ht.pss: //1ls::///ava -v. g titth..... i
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A USE LARGE LANGUAGE MODELS
We utilized GPT for polishing our manuscript. Our usage is only limited to refining and grammar check of our own written draft.
B PROMPT TEMPLATES
We describe here the prompts used to obtain object and background attention maps in Contrastive Bojket-Background Fusion.
Objuct-focused Prompt. The object attention map is obtained from the attention weights produced when the MILM answers prompt about the target object category referred to in the given expres- sion. The prompt template is shown below:
[Expression]
What is the main object (or objects) referred to in the given expression questien?
Focus on the **primary subject or agent** involved in the described action or behavior.
Respond with a single word (e.g., 'aat', 'person', 'dog') that best describes the target object (s).
Background-focused Prompt. In contrast, the background attention map is derived from the atten- tion weights produced when the MLLM responds to a prompt that asks it to describe the background, excluding the target object category, in single word or short phrase. The prompt template is shown below:
Describe the background scene of the video, excluding any {Object category). Answer the question using single word or phrase.
Object Category Choice Prompt. The quality of this contrastive fusion relies the correctness of the object category. To ensure robust category selection, we first gather category predictions from both video-level and frame-level inputs and then confirm the final target category through an explicit query. The prompt template is shown below:
Here the prompt template:
Given:
Expression: (Expression}
Candidate object class list: {object category list]
Geal: Identify the object class referred to by the expression. Instructions:
1. If the expression is #*clearv, rely it directly (e.g., 'a
person driving car' 'person').
2. the expression is **vague**, use the object class List to support your decision (e.g., check frequency and plausibility).
3. Avoid defaulting te the most frequent class unless the expression
lacks clarity.
Output most likely referred object class just the label.
The final object category used to construct the background-focusee prompts when obtaining video- and frame-level background attention maps. Importantly, the same of prompt templates was applied across all MLLMs and datasets without any dataset-specific or model-specific modifi- cations.
14

---

Published as a conference paper at ICLR 2026
(a)
black bike.
What the main object (or objects) referred the question?
Focus the * subject the involved the described behuvior. Respond with feat?, "person" ) that describes object(s).
(b)
black bicycle. What the object objects) referred
"prrmaay subject involved described single (e-g., "person's "dog")
Respond
target object(s). ecyle
Candidates b, B,
(c)
black bicycle. the object objects) referred **prlmary subject single word (e.g. "cat", 'person', "dog") Involved described target object(s).
expression
Respond
Candidates ieyele, teg, iker.
Figure 5: Analysis of TAM's failure cases
C MORE DETAILS ABOUT PREVIOUS METHODS
2025) exhibits strong sensitivity to the predicted word tokens. As shown in Fig. (a)), when the model predicts the token "bike". the resulting attention map is largely misaligned with the target object. In Fig- (b), when the expression is changed to a black bicycle, the word bicycle split into two tokens, and the first token "b" again shows severe misalignment. Fig.5 displays the attention map for the second token "icycle", which provides relatively better alignment with the target object. These examples demonstrate that TAM's localization is highly unstable and depends heavily how object words generated and tokenized.
TAM. TAM et al.
contrast,
Moreover, decoding object or background categories typically spans multiple tokens, and the origi- nal evaluation protocol reports the best- rest-performing token (i.e., the highest IoU among the predicted tokens) for each class. Such an evaluation overstates performance, underscoring TAM's lack of robustness in practical scenarios.
Loc-Head. Loc-Head (Kang et al., 2025a) was originally proposed in the image domain, where attention maps from MLLMs are used segment the target object referred by given expression in training- free manner. The method consists of two stages; first, identifying localization heads and then generating object masks using the attention weights from these heads.
In reproducing this method, we observed two major limitations. First, the procedure for discov- ering localization heads relies on sampling 1,000 image-text pairs from ReíCOCO. While heads discovered from RefCOCO yielded reasonable performance when evaluated on video datasets, re-discovering heads from samples drawn directly from video datasets led to substantial drop in performance. For example, the Attn-mask (T) score decreased from 24.2 - 19.2 on Ref- DAVIS (Khoreva et al. 2018) and from 18.6 + 4.2 on ReVOS (Yan et al. 2024). Consequently, all experiments in reproduction used the RefCOCO-discovee head-selection process includes heuristic that excludes heads strongly attending to the bottom row to prevent the visual attention sink phenomenon. We found that this heuristic does not generalize across all models. For example, on Qwen2VL, applying the original heuristic resulted in a score of 0.0 because the attention tended to concentrate the right-most column rather than the bottom
heads across datasets. Second, the
15

---

Published as a conference paper at ICLR 2026
row. After adapting the rule to exclude heads that strongly attend to the right- column, the Attn- mask J improved to 23.8. Similarly, for Intern enabling tiliigg during head discovery degraded performance, indicating further sensitivity to preprocessing choices. These results suggest that the Loc-Head procedure does not generalize reliably across either models or datasets.
A second issue arises in producing dense segmentation masks. In Loc-Head, the attention map is first binarized using the mean attention score as a threshold, after which the largest convex hull algorithm is applied to extract bounding box. This bounding box is then used as a prompt SAM for generating a dense mask. However, because the attention map is coarse, the resulting bounding boxes are often inaceurate leading to large degradation in the quality of the SAM masks. When we reproduced this procedure, the performance dropped significantly compared to the paper's 's reported numbers; for instance, on RefCOCO validation the score decreased from 74.2 + 34.4. To ensure a fair comparison, we therefore applied our SAM prompting process consistently across all video datasets.
Overall, these findings highlight that the Loc-Head approach depends heavily on dataset-specific sampling, model-specific heuristics. These issues make it difficult to obtain consistent results across models and datasets. In contrast, our proposed DecAF framework works reliably across different MLLMs and datasets, providing more consistent and generalizable performance compared to the Loc-Head approach.
Loe-Head with LLaVA-OV Details. We attempted the following implementations for adapting Loc-Head LLaVA-OV-7B, but Loc-Head still performs poorly with LLaVA-OV-7B, highlighting its limited robustness across models. 1. Adapting Loc-Head for tiling. LLaVA-OV-7B employs tiling process high-resolution images. As we observed with InterVLL,, extracting localization heads from tiled inputs leads to severe performance degradation. Following this observation, we disable tiling when extracting localization heads and only enable tiling during the computation of attention maps for segmentation. 2. Identifying the appropriate attention-sink region to exclude. Loc-Head removes heads with strong attention the bottom row. Although this details is not de- scribed in the Loc-Head paper, it is implemented in the official code repository [ However, this heuristic does not generalize across different MLLMs. For VL-VV7B..BB we found that addition- ally excluding the left-most column is necessary for the method to produce reasonable results.
D QUALITATIVE RESULTS
We provide qualitative results to demonstrate the effectiveness of our proposed Decomposed Atten- tion Fusion (DecAF) and SAM prompting. Fig. 4 object, multi-object, small-object, temporal reasoning. and world knowledge scenarios. Each exam- ple shows the attention maps obtained through DecAF, the attention masks directly generated from the fused attention maps, and the dense masks obtained via SAM prompting.
6 7
present diverse cases, including single-
Across these scenarios, DecAF consistently produces attention maps that align with instruction- referred target objects, and both the attention masks ans SAM masks accurately capture the object regions. Even in challenging settings involving multiple objects small targets, our approach maintains robust localization and segmentation quality. Moreover, for cases requiring temporal reasoning world knowledge, DecAF effectively leverages the capabilities of MLLMs to generate accurate masks without additional training.
We also report several failure cases (Fig. 9 As shown in Fig. 9 (a), proposed attention con- sistency scoring method may underperform when the target object occupies a large area in certain frames but the attention weights cover only small portion of that region. In such cases, the method assigns a strong penalty, leading to low scores even when the mask tracklet is correctly generated. Similarly (Fig. (c)). when the target object is small and appears only briefly in the video, it occupies only a small fraction of the overall attention weights in the video, which results in low attention con- sistency scores and false filtering. Finally. Fig. (b) shows that when the target object is extremely thin or elongated (e.g., paraglider lines), the attention maps fail to capture its structure, resulting in poor masks.
Link to the official code line for bottom-row exclusion
16

---

Published as a conference paper at ICLR 2026
Exp: airplane(s) Rying side by side in the sky.
Atnentinn
mhip
Attention
Mask
SAM Musk
Figure 6: Qualitative results for the small object case.
Exp: Who served ball this round?
Attention
maa
Attrstin
Mask
SAM
Mask
Figure 7: Qualitative results for the temporal reasoning case.
Table 9: Comparison with naive baselines that directly use the spatial grounding of Qwen2.5-VV VL- 7B together with SAM2. The baselines differ in how Qwen2.5-VL. grounding is applied across the video: All frames performs frame-wise grounding and segmentation, First frame grounds only the first frame and propagates with SAM2, Ref & key frames uses 16 reference frames identify the target and grounds a key frame for propagation, and 16 frames grounds uniformly sampled 16 frames. We report results on Ref-DAVIS and voasonVOS. QVL2.5 denotes Qwen2.5-VL-7B.
Ref-DAVIS
Reason VOS
Method
Sampling
J&F
J
F
J&F
F
QVL.2.5 + SAM2
All frames
53.0
50.1
55.9
44.2
41.9
46.4
QVL2.5 SAM2
Fisrt frame
52.8
50.4
55.2
38.9
36.7
41.2
QVI.2.5 SAM2
Ref & key frames
36.5
33.0
40.0
23.5
21.8
25.2
QVL2.5 SAM2
16 frames
64.8
58.0
71.6
48.0
41.7
54.3
Loc-Head
16 frames
64.6
60.2
68.9
41.1
37.9
44.3
DecAF [Ouss]
16 frames
75.2
70.9
79.5
63.9
60.5
67.2
E NATIVE GROUNDING OF QWEN2.5VL
Tab. presents the results of evaluating video segmentation using the native spatial grounding ca- pability of Qwen2.5-VL-7B (QVL2.5) in a seypoint-prompiinn form together with SAM2. In all settings, QVL2.5 is directly prompted to output the target object's keypoint based on the expression, propagation. Experiments
and SAM2 utilizes this keypoint for segmentation both Ref- Ref-DAVIS and Reason VOS, and the details of the evaluation settings are described below.
conducted
All frames (frame-wise grounding + per-frame SAM2). We apply QVL2.5 independently every frame to obtain the keypoints of target objects for that frame. Each point is then used to prompt SAM2, producing segmentation mask for the corresponding frame without temporal propagation.
17

---

Published as a conference paper at ICLR 2026
(a) Exp: which object product of Apple Inc?
Atentton
mnop
Astentionn
Mask
SAM
Muskk
(b) Exp: The spacious vehicle cupable of accommudating numerous passengers.
Attention
map
Astention
Mask
SAM
Missc
(c) Exp: the most unusual person the video.
Attention
map
Attention
Mask
SAM Mask
Figure 8: Qualitative results for the world knowledge cases.
18

---

Published as a conference paper at ICLR 2026
(a) Exp: motor-bike.
Astention
mhp
Arteriion
Mask
Mask
GT Mack
(b) Exp: paraglider lines.
Amertion
map
Artention
Mask
SAM Miskk
GT Misk
(c) Exp: Among the persons present, have food beverage their possession?
Attention
map
Attention
Mask
SAM Mark
Figure 9: Qualitative examples of failure cases.
19

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study of the object-foeused prompts.
Prompt Design
Ref-DAVIS
Reason VOS
ReVOS (Overall)
Ve (Ratering)
ReVOS
(Reasciing)
J&J
J
J
J&J
J
J
J&J
J
J
J&J
J
J
J&J
J
F
Original
75.2
70.9
79.5
63.9
60.5
67.2
54.2
50.1
58.2
58.7
54.8
62.6
49.7
45.4
53.9
VI (Single Sentence)
69.8
65.0
74.7
63.5
60.3
66.7
56.3
52.3
60.2
60.7
56.8
64.5
51.9
47.8
55.9
V2 (Rephrased)
75.9
71.5
80.4
64.4
61.1
67.8
56.3
52.2
60.3
60.8
56,9
64.7
51.8
47.6
55.9
V3 (Short)
74.2
69.7
78.6
63.8
60.4
67.1
54.9
51.0
58.8
59.8
55.9
63.6
50.1
46.0
54.1
First frame (single-frame grounding SAM2 propagation). We extract the key points only from the first frame using QVL2.5, and then prompt SAM2 to propagate the mask over the full video.
Ref & key frames (video-conditioned grounding SAM2 propagation). Since QVL2.5 does support video-level spatial grounding, we provide 16 uniformly sampled reference frames so that the model can identify the target object using the spatio-temporal context. Using this inferred target information, QVL2.5 localizes the object on the key frame- definee as the first frame among the reference frames-and extracts the corresponding keypoints. SAM2 then propagates this keypoint across the entire video to obtain the final segmentation.
16 frames (QVL2.5 grounding our SAM2 pipeline). In this setting, we uniformly sample 16 frames from each video and use QVL2.5 to extract a keypoint each sampled frame. These key- points are then fed into our SAM2 prompting and propagation process to obtain the final video segmentation masks. For reference, in the same 16-frame setting, Loc-Head derives the keypoints from its attention maps, while DecAF uses its fused attention to obtain them.
Across all settings, native grounding with QVL2.5 shows reasonable performance but consistently remains below that of our method. The All frames' and 'First frame' setups rely on frame-wise grounding and therefore cannot incorporate temporal information, leading to limited accuracy. Al- though the 'Ref & key frames' setting attempts to provide temporal context through reference frames, QVL2.5 does not support video-level visual grounding and fails to reliably extract target keypoints under this prompting scheme.
For fair comparison, we the same uniform 16-frame sampling and apply the identical SAM2 prompting and propagation process as in our method. QVL2.5 grounding performs better than Loc- Head but still falls significantly short of DecAF. These results indicate that, despite not supporting video grounding natively. DecAF effectively leverages the MILM's video understanding capability perform robust target localization. This demonstrates that our approach, while simple, provides an effective solution for video object grounding
F PROMPT ROBUSTNESS ANALYSIS
We evaluate the robustness of DecAF to prompt variations by using multiple formulations of the three prompts in our pipeline: the object-focused prompt, packground-foeusee prompt, and object category choice prompt. For each prompt type, we generate several alternative versions using Chat- GPT and evaluate how these variations affect the model's performance.
Object-focused Prompt. This prompt identifies the target object described in the expression. As shown in Tab. 10 we evaluate three variants: single sentence prompt (VI), a slightly modified version of our original prompt (V2), and prompt that is shorter than the original (V3). Overall, the three versions produce similar performance. Although V1's performance drops slightly on Ref- DAVIS, it slightly improves performance on Re VOS. Despite this small variation, the results indicate that AF robustly handles changes in object-focused prompt. The prompt templates are shown below:
20

---

Published as a conference paper at ICLR 2026
Table 11: Ablation study of the background-focused prompts.
Ref-DAVIS
Reason VOS
Prompt Design
J&F
J
J
J&J
J
J
Original
75.2
70.9
79.5
63.9
60.5
67.2
VI (No Object Category)
72.0
67.4
76.6
63.5
60.3
66.8
V2 (Single Sentence)
75.6
71.2
80.1
62.6
58.9
66.2
V3 (Expression-only)
75.5
71.0
79.9
63.6
60.1
67.0
Table 12: Ablation study of the object category choice prompts.
Ref-DAVIS
ReasonVOS
Prompt Design
J&F
J
F
J&F
J
F
Original
75.2
70.9
79.5
63.9
60.5
67.2
VI (Single Sentence)
73.1
68.5
77.7
62.4
59.0
65.8
V2 (Short)
75.1
70.8
79.4
63.7
60.4
67.0
V3 (Expression-only)
74.5
70.1
79.0
63.5
60.2
66.9
.1 (Single Sentence)
[Expression]
Identify the primary object referred to in the expression and with single word.
v2 (Rephrased)
{Expression)
Identify the primary object referred to in the expression. Focus on the uprimary subject or agent** involved in the described action or behavior. Respond with a single word (e.g., 'cat', 'person', 'dog') that best describes the target object(s)..
v3 (Short)
(Expression)
betermine the primary subject or agent mentioned in the expression question, and provide the object's label within single e word phrase.
Background-focusee Prompt. This prompt identifies the background of the scene. In Tab. we evaluate three variants: (V1) a prompt that queries the background without providing any object- class information, (V2) single sentence prompt, and (V3) a prompt that excludes the object de- scribed in the expression. Since V1 does not supply the object class, the model may occasionally misinterpret the target object part of the background, particularly when the target is not the pri- mary object in the scene. As result, VI tends to perform slightly lower than the original version. Nevertheless, all three variants still show highly similar performance overall, indicating that DecAF remains robust to different formulations of the background prompt. The prompt templates are shown below:
v1 (No Object Category) Describe the 'background scene of the deo. Answer the quest using single word or phrase.
v2 (Single Sentence) Describe the background of the video while excluding any (Object category), using a single word or short phrase.
v3 (Expression-only)
Describe the background scene of the video, excluding the objects referred in the given expression or question (Expression). Answer the question using single word or phrase.
21

---

Published as a conference paper at ICLR 2026
Object Category Choice Prompt. This prompt identifies the final object category using the object classes obtained from both the video- and frame-level predictions. In Tab. 12 we evaluate three variants: single sentence prompt (V1), prompt that is shorter than the original (V2), and prompt that infers the category solely from the expression without providing the class from the object-focused prompt (V3). All variants yield similar performance, showing that DecAF robustly determines the object category across different prompt formulations. The prompt templates are shown below:
V1 (Single Sentence) Given the expression (Expression) and the candidate object classes {Object category list], select the single class label that best matches the object referred in the expression.
V2 (Short)
Using the expression (Expression) and the candidate object classes {object category list}, determine which object class the expression refers to.
If the reference is explicit, rely on the expression; if ambiguous, use the class list as support. Output only the most likely object class.
V3 (Expression-only)
civen:
Expression: [Expression}
Goal: Identify the object class referred to by the expression. (e.g., 'a person driving a car' + 'person').
Output the most likely referred object class just the label.
Overall, these experiments show that DecAF is robust to prompt variations. Importantly, we do not perform any prompt tuning for different MLLMs benchmarks; all experiments in both the main paper and the appendix use the same fixed set of prompts. The consistent results across diverse prompt formulations further demonstrate that DecAF does not rely heavily on the exact choice of prompt and remains stable even when the prompts are varied.
G ADDITIONAL RESULTS OF FIGURE
Fig. prersents additional visualization of Fig. (b). including the final fused attention maps, the at- tention masks obtained directly from the fused attention, the query points used for SAM2 prompting. and the resulting dense SAM masks. The fused attention maps clearly highlight strong activation in the target object. However, because the attention mask produced via frame-wise thresholding, weak attention responses may also be converted into foreground regions. Our SAM prompting pro- cess resolves this issue by deriving query points through thresholding the fused attention map. As shown in Fig. T0 these query points emerge only within the true target region, enabling SAM2 to generate an accurate and dense segmentation mask.
H ANALYSIS OF SIMILAR MULTIPLE OBJECTS SCENARIO
We evaluate our method on the similar multiple objects scenario. Among the 458 samples in the Reason VOS (Bai et al. 2024) dataset, we extract 187 samples corresponding to this challenging case. shown Tab. 13 the performance of the training-based method VRS-HQ (Gong et al., 2025b) decreases from 54.9 to 48.6 (-11.5%), whereas method shows smaller decline from 63.9 to 60.5 (-5.3%). Although performance decreases in this challenging setting, these results indicate that approach maintains relatively stable performance when multiple similar objects present.
and 13 While VRS-HQ RES-HQ often produces masks incorreet objects, method method-even even with complex expressions con.isssettnsssnnnntyyn.... highlights the correct target object in the attention map among multiple similar objects. This accurate localization enables method to generate precise dense segmentation masks for the target objects.
We further provide qualitative comparisons with VRS-HQ in Fig.
22

---

Published as a conference paper at ICLR 2026
Exp: Who served the ball this round??
Vidoo
G1
Attention
Map
Attention
Mask
Query Poinst
SAM
Figure 10: Additional qualitative results for Figure I
Query points are visualized in magenta.
Exp: area around construction entry point, can you identify car positioned next to green truck?
Video
GT Mask
VRS-HQ
SAM Mask
DecAF (ours)
Attemion Map
Attention Mask
Query
SAM
Figure 11: Qualitative results for the similar multiple objects case. Query points are visualized in magenta.
23

---

Published as a conference paper at ICLR 2026
Table 13: Comparison with VRS-HQ on Reason for the similar multiple-object scenario. Fullset and Subset denote the results on all evaluation samples and the similar multiple-object sam- ples, respectively.
Reason VOS
Method
Datasct Type
J&F
J
F
VRS-HQ [CVPR'25]
Fullset Subset
54.9
52.6
57.3
VRS-HQ [CVPR'25]
48.6
45.4
51.9
DecAF [Ours]
Fullset
63.9
60.5
67.2
DecAF [Ours]
Subset
60.5
56.4
64.7
Exp: Which creature seems be keeping distance from the rest its kind?
Video
GT Mask
VRS-HQ
SAM
Maok
DecAF (ours)
Attention
Map
Attentionn Mask
Querr Point
SAM
Figure 12: Qualitative results for the similar multiple objects case. Query points are visualized in magenta.
I PART-LEVEL SEGMENTATION ANALYSIS
Our method is evaluated on object-level video segmentation since there is no video object segmen- tation benchmark focusing on part-level referring expressions. Nonetheless, we also examine its behavior on expressions referring to specific parts of an object, such as "the shirt of the person" or "the glasses of the person", and provide qualitative results for these part-level cases in Figs. 14 15
16 and
and 17
Figs. and 15 show that our DecAF attention maps accurately capture the regions indicated by the expression. When the expression is "the person," the attention map broadly covers the entire person, whereas for "the shirt of the person," it focuses tightly on the shirt region. As a result, both the attention mask and the corresponding query point aligned with the shirt.
A similar trend appears in Figs. 16 and 17 for the smaller regions of "the face" and "the glasses." In both cases, the attention maps effectively highlight the intended part, and the resulting attention
24

---

Published as a conference paper at ICLR 2026
Exp: am driving on the road, but have to change lanes to the right becnuse the white vehicle in front is blocking my way. Which the vehicle causing change
Video
VRS-HQ
SAM Mask
DecAF (ours)
Attention Map
Attenton
Qney
NAM
Figure 13: Qualitative results for the similar multiple objects case. Query points visualized in magenta.
25

---

Published as a conference paper at ICLR 2026
Exp: The person.
Vidoo
Attention
Map
Attantion
Querr PoiR
NAM
Figure 14: Qualitative results for the object-level target case (person). Query points visualized in magenta.
Exp: The shirt the person.
Video
Attention Map
Attention
Query
Point
SAM
Figure 15: Qualitative results for the part-level target case (shirt). Query points are visualized in magenta.
masks and query points reflect this localization. Notably, the attention map for "the glasses" exhibits an even sharper focus due to the expression referring to a more specific and smaller region.
However, despite obtaining well-aligned attention maps and part-level query points, the dense masks produced by SAM2 do not consistently capture the fine-grained target regions. While the face successfully segmented in Fig. [6 prompting SAM2 with only single positive point make ambiguous whether the model should segment the object as a whole or the specific part.
Although part-level segmentation is not the primary focus of our method, these results show that DecAF is still able to perform reliable visual grounding at the part level, producing attention maps that are well aligned with the fine-grained regions referenced by the expression. Incorporating addi- tional cues, such as negative points, could enable SAM2 to produce dense masks that precisely align with the part-level localization provided by our attention maps.
26

---

Published as a conference paper at ICLR 2026
Exp: The person.
Video
Afrention Mop
Attention
Querry Politt
SAM
Figure 16: Qualitative results for the part-level target (face). Query points visualized in magenta.
Attantion
Mdp
Arcusion Mask
Point
Mark
Figure 17: Qualitative results for the part-level target case (glasses). Query points are visualized in magenta.
27

---

Published as a conference paper at ICLR 2026
Exp: Learning an imporiant process for self-immpoeement In the scene, which object is most likely to help enhance knowledge?
Vidoo
G1
Aftection Map
Aftertion
Query Point
BULD
BUL
BUR
SAM
Figure 18: Qualitative results for the non-explicit object expression case. Query points are visualized in magenta.
J ANALYSIS OF NON-EXPLICIT OBJECT EXPRESSION CASE.
Fig. 18 and 20 present qualitative results for expressions in which the target object is not ex- plicitly described or mentioned. As shown in these examples, these expressions provide no direct visual attributes or class-level cues about the target. Instead, identifying the correct object requires reasoning over the scene context (e.g.. "Learning is an important process for self-improvement. In the scene, which object is most likely to help enhance knowledge?"".
Our method naturally handles such challenging cases by formulating video reasoning segmentation Video QA task. By leveraging the MLLM's saasoning-driver attention maps-generated when answering the question formatted with the expression-this design enables the fused attention map to accurately highlight the correct target object even when the expression provides explicit ob- ject description. Furthermore, by utilizing these well-aligned attention maps, our method can also produce accurate dense segmentation masks for the inferred target.
28

---

Published as a conference paper at ICLR 2026
Exp: washing hands, typically to remaining moisture?
Video
G1
Attention Map
Artention
Query
SAM
Figure 19: Qualitative results for the non-explicit object expression case. Query points are visualized magenta.
Exp: construction halledd work shorlage mateerials, Which objectt most likely being awaited??
Vidko
GF
Attemiion
Map
Arcanion
Querry Poiine
SAM
Figure 20: Qualitative results for the non-explicit object expression case. Query points are visualized in magenta.
29


Reviews:
[{"review_id": "dsWEdKi95y", "reviewer": "Reviewer_eWqc", "summary": "This paper investigate the training-free video referring segmentation task, which is relatively underexplored previously.  The main idea of this paper is to cast referring video segementation task as video QA task and extract attention score in video LLM to perform segmentation. This paper presents a systematic pipeline including Decomposed Attention Fusion, DeCAF, and SAM2 refinement. DeCAF mainly includes two stages: 1) object-background attention rollout subtraction, and 2) sparse video and dense frame fusion. The experimental results of this proposed pipeline are on par with training-based methods on several VOS benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) This paper is one of the first cohorts to investigate the scope of the training-free video referring segmentation with LLM and therefore delivers substantial novelty and insights. The paper made several efforts to work this out and the results are on par with the training-based methods, which should be appreciated.\n\n2) This paper proposed several technical methods specifically tailored for video tasks, including video-frame attention fusion, frame-wise propagation, tracklet scoring and selection, showing that the pipeline is not a simply adopted method from image domain to video domain.\n\n3) This paper performed several ablation studies (in Table 5~8) to validate necessities of proposed components in the pipeline, which is of great value.", "weaknesses": "1) In Table 1 and 2, Why DecAF is with LLaVA-OV-7B while Loc-Head is with LLaVA-7B? Could you please show the results of Loc-Head woth LLaVA-OV-7B?\n\n2) Could the authors show the inference time of the whole pipeline of video segmentation, especially compared to the training end-to-end methods? Because it seems that the training free method is bound to include heavy pretrained methods to refine coarse attention scores. Adding the inference time analysis will increase the technical soundness of this paper.\n\n3) In the demo in Figure 1 (b), the author claims the segementation pipeline can resolve conflicts e.g., between the server and hitting player. Could authors elaborate this mechainism more in details? Is it mainly due to frame-wise prompting & progation or mask tracjlet scoring & selection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g7Jg7t7Gm1", "reviewer": "Reviewer_SdCm", "summary": "In this paper, the authors propose a method to use the attention map from the MLLM to extract prompts to get the masks from the SAM2 model. The authors test the performance compared to the performance by directly using the attention map (without SAM2 model). The results show that the masks generated by the SAM2 model have a good performance gain compared to the attention mask.", "questions": "NA", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The experiments demonstrate strong performance on the RefVOS benchmarks.\n\n- The authors evaluate the method's performance across different MLLMs, showing its generalizability.\n\n- The paper includes an ablation study on the proposed modules to validate their effectiveness.", "weaknesses": "- Insufficient and Missing Baselines: The paper's most significant weakness is its baseline setup. While the authors compare against training-free methods like Loc-Head, they omit the most critical and simplest baseline: using the MLLM's native grounding capabilities. Modern MLLMs, particularly Qwen2.5VL (which the authors use), are already capable of generating bounding boxes or points for a given textual expression. A much simpler and more direct training-free baseline would be to use these native grounding outputs (boxes/points) as prompts for SAM2. Given that the paper's best results come from Qwen2.5VL, this \"native grounding + SAM2\" baseline is essential. Without it, it is impossible to determine if the complex, multi-step DecAF process adds any real value or if it is just an overly complicated method to leverage a powerful, pre-existing capability of the MLLM.\n\n- Computational Cost and Practicality: The \"Complementary Video-Frame Fusion\" module raises significant concerns about inference efficiency. As described, the method requires one MLLM pass for the entire video and then additional, separate MLLM passes for each individual frame to obtain the final result. This multi-pass design would be prohibitively slow for videos of any reasonable length, making the method impractical for real-world applications. The authors must provide a detailed analysis of the computational overhead (e.g., latency or FLOPs) compared to a single-pass approach.\n\n- Justification of \"Training-Free\" Claim: The paper's \"training-free\" claim is questionable. In Section 3.2, the method employs \"Head-wise weighted aggregation.\" It is unclear where these \"weights\" originate. If any part of the method relies on optimization or selection using a \"training set,\" the \"training-free\" claim is weakened. In that case, the method should be more thoroughly compared to other training-based methods on an equal footing.\n\n- Limited Novelty and Scope: The paper's novelty appears limited. The framework feels like a straightforward extension of image-domain attention grounding (like Loc-Head) to the video domain. The core technical contributions—the fusion modules—are not particularly novel. \"Contrastive Object-Background Fusion\" appears to be a simple map subtraction, and \"Complementary Video-Frame Fusion\" is a simple averaging of multi-scale maps. Furthermore, the main comparison method, Loc-Head, was designed for images, and the “background” contribution is not video-specific. To better demonstrate the contribution of DecAF, the authors should also evaluate its performance on standard image segmentation benchmarks (e.g., RefCOCO) to show its advantage in both domains.\n\n- Attention Map Ambiguity: The reliance on attention maps is a potential weakness, as these maps may not always align with the intended semantic answer. This ambiguity could be particularly problematic in cases with multiple similar objects or when only part of an object is the correct answer (e.g., segmenting \"the shirt of a person\" from \"the person\"). The authors should justify how the proposed method handles such nuanced cases.\n\n- Ablation Study Concerns: (a question, I am not very sure) A point of clarification is needed regarding the ablation study. In Table 4(a), removing the \"background\" module causes a performance drop far below the Loc-Head baseline. This result is counter-intuitive. Why would removing a single proposed component result in performance significantly worse than the baseline? Intuitively, a model with all proposed modules removed should perform similarly to the baseline. The authors should elaborate on this finding.\n\n\nIn conclusion, I am not convinced of the method's practical utility for the RefVOS task. The computational cost is unclear, and it remains uncertain whether the proposed method offers a great advantage over the much simpler, un-tested baseline of using the MLLM’s native grounding capabilities to prompt SAM2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8KC3S2Ktvc", "reviewer": "Reviewer_M2r8", "summary": "This paper proposes DecAF, a training-free framework for video reasoning segmentation. The core idea is to leverage the inherent localization abilities of Multimodal Large Language Models in the attention maps by treating the task as a video question-answering problem. To address the noise in vanilla attention maps, DecAF introduces Contrastive Object-Background Fusion and Complementary Video-Frame Fusion. The resulting fused attention map provides a strong coarse localization signal, which serves as prompt for the SAM2 model. Experiments show that DecAF significantly outperforms previous training-free methods and achieves performance comparable to SOTA training-based methods.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of exploiting attention map of MLLM as localization signal is interesting. Although using attention map as segmentation has been widely studied in ViT models, it is interesting to see this technique to be applied in the Reasoning VOS task with MLLM as the backbone.\n2. The Contrastive Object-Background Fusion and Complementary Video-Frame Fusion modules are intuitive and show compelling results.\n3. The experiments and ablation studies are comprehensive.", "weaknesses": "1. To achieve the background contrast and video-frame fusion, it seems like the model need to run inference multiple times, which can significantly increase the cost. It is necessary to discuss this and compare with existing methods.\n2. How to correctly get the background attention mask? If I understand correctly, the prompt should specify salient objects. What is the source of salient objects? Is it given by human or?\n3. In reasoning segmentation, the text queries can be complex and sometimes don't explicitly mention a specific object. E.g., it can be a question that simply ask \"Which object in the scene that ...?\" How does this method handle such prompt?\n4. The method is limited to only process object-level segmentation from the design. Is it able to segment part of object?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VxJxTVeMS8", "reviewer": "Reviewer_h74W", "summary": "This paper proposes DecAF (Decomposed Attention Fusion), a training-free framework for video reasoning segmentation that leverages attention maps from Multimodal Large Language Models (MLLMs). The method introduces two key fusion strategies: (1) contrastive object-background fusion to suppress irrelevant activations, and (2) complementary video-frame fusion to combine temporal context with spatial details. The attention maps are converted to coarse masks via thresholding and refined using SAM2 with an attention consistency scoring mechanism. Experiments across five datasets demonstrate performance comparable to training-based methods.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The decomposed attention fusion strategy is intuitive and well-designed. The contrastive object-background fusion effectively addresses the visual attention sink phenomenon, while the complementary video-frame fusion leverages the distinct strengths of different input modalities.\n\n- The approach achieves results comparable to or better than several training-based methods (e.g., outperforming VISA and VideoLISA on Ref-DAVIS).\n\n- Thorough Ablation Studies: The ablation studies (Tables 4-7) systematically validate each component, including fusion strategies, rollout methods, SAM prompting thresholds, and scoring mechanisms.", "weaknesses": "- Limited Novelty in Core Technique: While the fusion strategies are well-designed, the core building block (attention rollout) is not novel. The V-Max normalization (Eq. 4) is a relatively straightforward modification. The main contribution lies in the application and combination of existing techniques rather than fundamental algorithmic innovation.\n\n- Computational Cost Not Discussed: The paper does not analyze computational complexity or runtime. The method requires multiple forward passes through the MLLM (for object, background, video, and frame attention maps), which could be computationally expensive. A comparison of inference time with training-based methods would strengthen the practical claims.\n\n- Dependency on Prompt Engineering: The method heavily relies on carefully designed prompts (Appendix B). The robustness to prompt variations is not studied. Different prompt formulations might significantly affect performance, limiting generalizability.\n\n- Missing Generalization Comparison. This is perhaps the most significant weakness of the paper. The primary advantage of training-free methods over training-based approaches is their ability to generalize to scenarios with limited or no training data. However, the paper completely fails to demonstrate this critical advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)