RAR: Reversing Visual Attention Re-Sinking for  Unlocking Potential in Multimodal Large Language Models
Authors: Zhehan Kan
Xin Li
Yanlin Liu
Xiaochen Yang
Xinghua Jiang
Yinsong Liu
Deqiang Jiang
Xing Sun
Qingmin Liao
Wenming Yang
Venue: ICLR 2026 Poster (2026)
Keywords: MLLMs
visual attention sink
Primary area: foundation or frontier models, including LLMs

Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision-language tasks, yet they frequently exhibit suboptimal output layers, where intermediate decoder layers outperform the final ones, signaling underutilized model capacity. In this work, we delve into the root causes and attribute this issue to the Visual Attention Re-sinking phenomenon, precipitated by attention gradient sparsity driven by textual supervision dominance. This degradation causes attention heads to evolve into sink heads that prioritize low-semantic backgrounds, thereby disrupting modality fusion, neglecting visual information, and biasing outputs toward textual priors, ultimately impairing model performance. To mitigate this, we introduce a parameter-free Sink Attention Dynamic Sparsification (SADS) framework that dynamically identifies and retains all vision heads(concentrating visual attention on semantically relevant regions) while sparsifying sink heads, preserving essential global context through a shared head. Integrated into diverse MLLMs, our framework yields substantial performance gains across 20 benchmarks spanning five task categories (visual grounding, general VQA, OCR-related VQA, vision-centric tasks, and visual hallucination mitigation) surpassing supervised fine-tuning while boosting inference speed by 10.3\%. This approach offers a novel avenue for maximizing MLLMs capabilities.

Published as a conference paper at ICLR 2026
RAR: REVERSING VISUAL ATTENTION RE-SINKING FOR UNLOCKING POTENTIAL IN MULTIMODAL LARGE LANGUAGE MODELS
Zhehan Kan Xin Li
Yanli Liu
Xiaochen Yang²
Xinghua Jiang'
Yinsong Liu²
Deqiang Jiang?
Xing Sun?
Qingmin Liao'
Wenming Yang"
TTsnngua University Tencent Youtu Lab
'The University of Glasgow
ABSTRACT
Multimodal Large Language Models (MLLMs)) have achieved remarkable suc- cess in vision-language tasks, yet they frequently exhibit suboptimal output lay- ers, where intermediate decoder layers outperform the final ones, signaling un- derutilized model capacity. In this work, we delve into the root causes and at- tribute this issue to the Visual Attention Re-sinking phenomenon, precipitated by attention gradient sparsity driven by textual supervision dominance. This degradation causes attention heads to evolve into sink heads that prioritize low- semantic backgrounds, thereby disrupting modality fusion, neglecting visual in- formation, and biasing outputs toward textual priors, ultimately impairing model performance. To mitigate this, we introduce parameter- free Sink Attention Dy- namic Sparsification (SADS) framework that dynamically preserves all vision heads, ensuring focused attention on semantically salient regions, while retain- ing only a minimal subset of sink heads, including a designated shared head to safeguard essential global and contextual information. Integrated into diverse MLLMs, our framework yields substantial performance gains across 20 bench- marks spanning five task categories (visual grounding, general VQA, OCR-related VQA, vision-centric tasks, and visual hallucination mitigation) surpassing super- vised fine-tuning while boosting inference speed by 10.3%. This approach offers a novel avenue for maximizing MILLMs capabilities.
INTRODUCTION
In recent years, Multimodal Large Language Models (MLLMs) have rapidly evolved into a scalable pathway toward Artificial General Intelligence (AGI) (Bai al. 2025 Chen et al.) 2024 Liu al., 2023). These models typically employ a vision encoder and a connector to align image features with text embeddings, which are then processed by an LLM decoder to generate responses (Bai et al., 2025 Chen et al., 2024 Liu et al. (003)). While MLLMs excel in vision languuge tasks like visual question answering, grounding, and captioning, recent findings show mid-to-late vision encoder layers often surpass the output layer, due to CL.IP training fostering rich spatial and semantic features in intermediates (Bolya al. 2025). Analogously, for hallucination mitigation in MILM decoders, mid-layer visual facts are suppressed later, leading to methods that leverage fuse intermediates for enhanced outputs (Wang et al. 2024). However, existing research offers limited insights into the underlying causes and primarily relies post-hoc remedial strategies that fail to fully activate the model's capacity. Thus, addressing "why the output layer in MLLMs is suboptimal" and "how maximize MLLM capabilities by optimizing the output layer" represents critical and urgent challenge.
In this work, we investigate why MILMM output layers suboptimal. Unlike LLMs, MILMS must fuse vision and language, typically processing features in early layers, aligning modalities in mid- layers, and organizing responses in late layers (Zhang et al. 2025). As illustrated Figure I we discover that supervision in existing MLLM training paradigms is entirely textual, devoid of direct
Equaal contribution. Zhehan Kan works done during the internship at Tencent Youtu Lab.
Corresponning author.

---

Published as a conference paper at ICLR 2026
Input
Seperision 9 geal
Iritial State
Our Approach
Imaage & Teat
Laye
Lyyer
Laye
Text
Diogroate
Text-Gelyy Supersisinn
Subeptinna Late-kopar Accursty
Healthy Vision Head
Degenerated Sink
Heads
an Sporsity
Marifaete
Shann Head
Vainn Haac & Presetved Heach
(Lane-apira)
Disect hamds
Layer
Lawer
X
Layw 25
#1 ber viaa taken intropy Sporssty Sink Hoads
Figure I: Overview of visual attention re-sinking in MLLMs caused by text-only supervision, induc- ing gradient sparsity, head degeneration, and suboptimal late-layer accuracy. Our SADS framework sparsifies sink heads while retaining all vision heads and shared head, achieving progressive aceu- racy gains and eliminating re-sinking.
visual oversight. Consequently, gradients for vision tokens rely solely on backpropagation of tex- tual losses through the attention mechanism, constraining its learning capacity vision tokens and rendering the overall gradient distribution increasingly sparse. This sparsity prompts the model, in subsequent forward passes, to concentrate visual attention weights a shrinking subset of vision tokens. As training iterations advance. these weights progressively localize to an even smaller num- ber of tokens, engendering the Visual Attention Re-sinking phenomenon, wherein visual attention in late layers reverts to low-semantic backgrounds. This disrupts the modality fusion balance estab- lished in mid-layers, compelling the model to increasingly rely textual priors rather than deeply integrating visual cues. The degradation escalates with iterations, propagating backward from late mid-layers and ultimately culminating in suboptimal output-layer performance.
Building upon these insights, we introduce a parameter-free Sink Attention Dynamic Sparsifica- tion (SADS) framework. This approach dynamically retains all vision heads during inference while preserving only minimal subset of sink heads, thereby encouraging the model to prioritize visual information without sacrificing critical global and contextual knowledge. Specifically, we observe that the maximum visual attention between vision and sink heads follows bimodal Gaussian dis- tribution. Similarly, within the sink heads, the entropy of non-vision token cross-attention between heads that focus on global knowledge (denoted as sinkg) and those that fully sink attention onto specific non-fixed tokens (denoted as sinks) also adheres to a bimodal Gaussian distribution. By dynamic threshold,
leveraging the valley of this distribution heads along with the sinkG heads, while designating the first head as a shared sink to ensure the preservation of global and contextual information. This framework is architecture-agnostie and readily applicable to diverse MILMMs. In this study, we integrate it into Qwen2.5-Ve VI., Intern VI.2, and LLa -A (Bai et al.) 2025 Chen al., [2024] Liu et al. 2023). Extensive experiments across a variety of tasks demonstrate that fine-tuning with our framework achieves substantial improve- ments over standard supervised fine-tuning (SFT) on numerous visual benchmarks, Furthermore, by streamlining redundant attention weight computations, inference speed is boosted by 10.3%.
framework retains all vision
In summary, our contributions are follows: 1) We conduct an in-depth investigation into the causes of suboptimal MLLM output layers, attributing them to the text-only supervision paradigm in MIL.Ms, which, as training iterates, prompts models to learn modalit--irelevan output short- cuts, sparsifying attention gradients, inducing visual attention re-sinking, leading to suboptimal outputs. 2) We introduce a SADS framework that addresses the visual attention re-sinking issue, optimizes attention gradient sparsity, and achieves output-layer optimality, thereby maximally acti- vating the model's 's capacity. 3) We validate the superior effectiveness and inference efficiency of our method through comprehensive experiments and analyses across five task categories: visual ground- ing, general VQA, rel-tel VQA, vision-centric tasks, and visual hallucination tasks, spanning 20 benchmarks, providing novel framework for advancing ML..Ms.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Best Layer in MLLM Decoders. In MLLMs, the vision encoder's final layer typically extracts visual features, with the decoder's output layer generating responses. However, studies show mid- to-late vision encoder layers often outperform the final one across tasks (Bordes et al. 2022] Chen al. 2020b Ma al. 2024 Shekhar al. 2023 Walmer al. 2022 Zheng et al. 2016). For ex- ample, IGPT's s intermediate layers excelled in image classification (Chen et al.] 2020a). and PE demonstrated CLIP training fosters rich features intermediates across encoders (Bolya et al. 2025). Similar suppression of mid-layer visual facts occurs in MLLM decoders for hallucination mitigation, with fusion methods improving outputs (Wang et al. prior work lacks deep causal insights and relies on post-hoc tixes.
2024 Huang et al. 2024). Yet,
In contrast, we attribute subopti-
mal outputs to visual attention re-sinking from textual supervision and gradient sparsity, proposing SADS for optimal output layers and capacity maximization.
Visual Attention Sink in MLLMs. In LLMs, attention sink involves low-semantic tokens (e.g.. BOS, """) drawing excessive weights (Xiao al. 2023), minimally contributing to inference (Bon darenko et al.] 2023). Recent views frame it as first-token mechanistic mixture suppression to pre- vent collapse (Barbero et al. 2025). In MILLMs, visual attention targets image patches (Aflalo al.). but often misallocates to low-semantic areas, mitigated by registration tokens (Darcet et al., (203) or boosted image weights (Zhu et al. 2025). VAR formalized visual attention sink, linking it to sink token activations like LLMs, and reallocating attention (Kang et al. 2025). Conversely, we pioneer the discovery of visual attention re-sinking MLLMs.
PRELIMINARIES
MLLMs typically feature end-to-end architecture integrating vision encoder, projection mod- ule, and an LLM decoder (Liu et al. [023] Bai et al.] 2025). The vision encoder extracts hierarchical visual features from input images, which projected into modality-aligned latent space bridge visual-textual gaps. These visual embeddings are concatenated with tokenized system prompts and instructions. forming a unified sequence fed into the LLM decoder for autoregressive response gen- eration with causal masking. Each sequence element is discretized token embedding.
Formally, let E N.xd denote N₂ visual tokens and R R.N.xx denote N₂ textual tokens (d: embedding dimension). The concatenated input [v; t] [NN++N))xd is processed by the Transformer-based decoder. Each of I blocks computes:
' = LayerNorm(h e-1 MHA(h--)), h h LayerNormhh - FFN((")),
(1)
where h E R (N N N,, h x), LayerNorm is layer normalization (Ba et al. 2016 and FFN is two-layer feed-forward network with non-linear activation. The melti-head attention (MHA), key to modality fusion, is:
MHA(Q, K, V) Concat(headd,,... ..., head,) W
with Q - f- Wº, K h W V 'WV (W2, W wV, wo E Rddd). Each head i is:
QW (WOKK Vdh
head; = softmax
+ +M| (VW!),
(3)
where de = d/H, H is the number of heads, and M is the causal mask (M,; = 0 if < i, else - Vaswani et al. 2017). We focus cross-modal interactions, specifically textual queries attending to visual keys via the visual attention matrix:
QiK.
Visual Attention = softmax
E NiN.
(4)
Vile
where Q, and K, textual queries and visual keys, analyzed for fusion patterns.
3

---

Published as a conference paper at ICLR 2026
4 WHY Is THE OUTPUT LAYER NOT OPTIMAL?
As illustrated in Figure 2 advanced open-source MLLMS consistently face the challenge of subop- timal output layers, where intermediate decoder layers outperform the final layer. This indicates that the extensive parameter capacity underpinning these models is not fully activated. In this section, investigate the underlying causes of this phenomenon.
Sink Token Hidden States
Vision Token Hidden States
Geber ileen of Thang's chandtriites
1551
(a) Attention Share: Image, System, & Text (green: text, red: system, blue: image).
(b) Hidden State Comparison: Sink and Vision To- kens Lack Major Spikes.
Layer
Layer 15
Layer 23
Layer 30
Layer 34
Layer 35
Sink
Sink Diminish
Re- Re-Sinking
(c) Evolution of visual attention across layers, where the visual attention sink appears in early layers, diminishes middle layers, and re-emerges in later layers. Green boxes indicate semantically relevant visual regions.
Figure 3: Analysis of visual attention on the object "the number of intolerant".
4.1 VISUAL ATTENTION RE-SINKING LEADS TO SUBOPTIMAL OUTPUT LAYERS.
In LLM decoders, intermediate and output lay- ers perform comparably on simple tasks, while deeper output layers excel on more complex ones, without notable suboptimal output layer issues (Fan et al., 2024). Therefore, we focus our investigation on the fusion of visual and lin- guistic modalities. Within Transformer-basee decoder architectures, models rely more heav- ily on attention mechanisms than FFNs to in- ject visual information into the linguistic latent space for modality alignment (Vaswani et al.) 2017). Thus, analysis centers visual at- tention. We decompose the impact of visual at- tention into two aspects: 1) the total atlention allocated images and 2) the distribution of visual attention across vision lokens. As shown in Figure Ba we first compute the attention distribution across image, system, and text components, observing stable pattern across layers without fluctu- ations in image attention allocation in later layers. This suggests that the total attention allocated to images not the primary cause of suboptimal output layers, implying that the issue likely lies in the distribution of visual attention across different vision tokens. According to Equation E we compute the attention weights between output tokens and vision tokens to derive visual attention maps. As illustrated in Figure 3d we observe that visual attention in early layers predominantly concentrates on low- oww-ifoormation background regions; transitions to semantically salient areas in mid-layers, and reverts to www-iformation backgrounds in late layers. We define tokens attracting visual attention to iow-information background regions as sink tokens, those focusing on semantically relevant regions vision tokens, and the resurgence of visual attention toward ow-information backgrounds in late
50
Qwen2. Qwen2.5VL-B8 5VL-3B
Securren (%)) 36.
80
IntwmVV12.21
1.5
Eoscced
Eqpaciad
-0
15
25
Layer
Figure 2: Cross-layer accuracy of the three base models on the OVDEval test benchmark.
4

---

Published as a conference paper at ICLR 2026
layers the visual attention re-sinking phenomenon. We posit that this re-sinking contributes to performance degradation. To validate this hypothesis, we perform a training-free free intervention on the "QA" test set, reallocating attention weights from sink tokens in the last five layers to semantically pertinent vision tokens, which yields a 0.74% accuracy improvement. This confirms that visual attention re-sinking is the primary culprit behind suboptimal output layers. Accordingly, our inves- tigation focuses on late layers to address the question: "Whal causes visual attention re-sinking?"
4.2 ATTENTION GRADIENT SPARSITY LEADS TO VISUAL ATTENTION RE-SINKING
Vision Head
Sink Head
(a) Visual attention of sink head and vision head. Boxes in the images indicate semantically relevant visual regions.
Dendyy
(c) Evolution of visual attention K-value variance in sink heads across layers, exhibit- ing a ceeline-plateau-hiiee sharp increase trend; fluctuations caused by outlier sink tokens, with boxplots showing more sink tokens in early and late layers.
463
06
(b) Distribution of maximum visual attention for vision heads and sink heads, showing a bimodal distribution.
Figure 4: Analysis of vision and sink heads visual attention on the object "the number of intolerant".
Unlike the attention sink in LLMs and VAR, which stems from massive activations in specific hidden state dimen- sions and appears in early layers while vanishing in later ones, as depicted in Figure 3b and Figure 3c| we observe that visual attention sink emerges in early layers, dimin- ishes in mid-layers, and reappears in late layers, without massive activations in specific hidden state dimensions. Therefore, we investigate the characteristies of these sink tokens and the reasons for their resurgence.
Table I: Analysis of the impact of heads OVDEval benchmark.
Methed
Accuracy (%)
Qwen2 V-VL-3B
39.5
ainig
w/ sinkk head
43.0
wio sindee hadd
43.2
wio vision head
42.6
Sink Tokens Concentrate in Sink Heads. For multi-head attention (MHA), we begin by examining the visual attention representations across individual heads. As illustrated in Figure 4aa we identify pronounced disparities among heads in terms of visual attention allocation. We classify heads that direct visual attention toward semantically salient positions as vision heads, whereas those gravi- tating toward low-informatio background regions are termed sink heads. Notably, the maximum visual attention values in vision heads substantially exceed those in sink heads. To substantiate this observation, we perform statistical analysis of the maximum visual attention across 1,600 heads in late layers, revealing a distinct bimodal distribution: sink heads cluster at significantly lower maxi- mum visual attention levels compared to vision heads. This pattern yields two key insights: 1) sink tokens predominantly concentrate within sink heads, and 2) although sink tokens exhibit high rela- rive attention weights, their absolute magnitudes remain low. Consequently, our subsequent analysis focuses on sink heads.
As depicted in Figure 5. compute the cross-attention among all tokens within sink heads and observe distinct patterns in the non-vision token cross- attention: some heads exhibit uniformly dis- persed attention across tokens (preserving global and contextual information), while others sink
5

---

Published as a conference paper at ICLR 2026
attention onto individual low-semantic tokens. Specifically, quantify this pattern via the tropy of the non-vision token attention distribution, defined as follows. Let A E R de- note the attention matrix for a given head, where Lg and LA the query and key sequence lengths, respectively. Let I be the set of indices corresponding to non-vision tokens (keys). Ex- tract the submatrix Asub = A|; E R Renormalize each row of Aub such that ; = _ezz Asubl j, ensuring each sums to 1 over non-image keys. Then, compute the average attention distribution p R [1 where Pj i., E Auu ji for each j I. The entropy is given by H = - ΣEEZ Pj log
We define high-entropy heads, which attend to global and contextual knowledge, as sinke; heads, and low-entropy heads, which sink attention onto individual low-semantic tokens, as sinks heads. This complete sinking of attention onto low-information tokens disrupts effective modality fusion, marginalizes visual cues, and biases outputs toward textual priors, ultimately leading degraded model performance. To validate this, as shown Table [ on the OVDEval benchmark, adding an extra sinks head ablating vision heads sinkg heads leads to significant performance drops. These findings highlight the adverse effects attention sinking to isolated tokens while confirming the importance of visual cues and global context. Moreover, we observe that the entropy values across heads follow a bimodal Gaussian distribution, allowing us to leverage its valley as a dynamic threshold for differentiating sinky and sinkg heads.
Entrapy
Layer 34
Low-Entropy
System
Image
Text
Output
Figure 5: Attention heatmaps of two sink heads. Top: High-entropy sinki head (green box) with distributed attention preserving global context. Bottom: Low-entropy sinkg head (red box) sinking to specific low-semantic tokens.
Re-sinking Tokens Exhibit High Key Variance. On sink heads, we concentrate on the key matrix K in multi-head attention due to its role in modality interactions. As illustrated in Figure 4b) discover that K value variances align with the visual attention re-sinking pattern: decreasing in early layers, stabilizing mid-layers, and surging late layers. From the box plots, mid-layer fluctuations arise from outlier tokens, which are precisely sink tokens. In early and late layers, the increasing number of sink tokens elevates overall K variances. It is this high dimensional variance in K that leads to higher attention weights for sink tokens compared to others.
Attention Gradient Sparsity Causes Sink Token Resurgence. Given that supervision in the final layer of MLLMs during training is entirely textual, devoid of direct oversight for visual signals, the gradients for visual tokens rely exclusively on backpropagation of textual losses through the attention mechanism. This dependency constrains the learning capacity of the attention mechanism on visual tokens, rendering the overall gradient distribution increasingly sparse. Consequently, in subsequent forward passes, the model tends to concentrate visual attention weights diminishing subset of vision tokens. As training goes on, these weights slowly focus on fewer tokens, eventually forming the "visual attention re- re-sinking" phenomenon. To test this cuuse--nd-effeee link with data, as shown in Figure 7 we track the change in attention gradient sparsity and the number of sink heads during the training process. Notably, about 2,000 iterations, gradients in late layers start to thin

---

Published as a conference paper at ICLR 2026
Execution
Layer
Decision
Layr t
Aul
buttect sharet
outpute
Poweed
SADS
Aeros All
Veiin loket
pecbing
Loyar
precution
Firal Active Heads
Rewert
SADS- Imfegetted
Activation Trigger
m
Loyer
Sparaity
SON Nood
Art frem
ceeiding
Merga Shared
Feed
Laysr
Ae hren
(m-
Atantiinn
Shandand
Figure 6: Overview of the SADS framework. Decision (Right): Triggers activation at Layer m upon detecting gradient sparsity and key variance anomalies. Execution (Left): Filters heads using bimodal thresholds on maximum visual attention (separating vision and sink heads) and non-vision token entropy (distinguishing sinkg from sinkg within sinks). Merges all vision heads, retained sinkg heads, and a shared head for computation, preserving global context.
out; by about 3,000 iterations, the number of sink heads in these layers begins to grow, confirming that this sparsity causes the sink tokens to return.
5 SINK ATTENTION DYNAMIC SPARSIFICATION FRAMEWORK
Drawing upon the preceding analysis, we conclude that MLLMS require a sufficient number of vi- sion heads effectively process dense, semantically rich visual information, complemented by minimal subset of sink heads to handle textual and global contextual elements. To this end, we in- troduce the Sink Attention Dynamic Sparsification (SADS) framework. As illustrated Figure 6 this framework first distinguishes vision heads from sink heads via the bimodal distribution of max- imum visual attention. then retains all vision heads while dynamically sparsifying the sink heads: leveraging the bimodal Gaussian distribution of non-vision token cross-attention entropy, it identi- fies and preserves the high-entropy sinkg heads (capture global and contextual knowledge) using the distribution's valley dynamic threshold, and designates the first head as a shared sink to ensure model stability and safeguard essential global information.
5.1 SINK HEAD IDENTIFICATION
Building on the observation from Section 4.2 that the maximum vi- sual attention in vision heads far ex- ceeds that in sink heads, model each layer in SADS using Gaus- sian bimodal distribution, formalized as;p(z) = Ez= (x Mlc o where the valley a arg min, p(x) serves as the maximum visual atten- tion threshold for classifying sink and vision heads. Subsequently, within the identified sink heads, we fur- ther delineate them based the entropy non-vision token cross- attention, which exhibits similar
2000
2000 INFIRITTIOna
400
(a)
(b)
Figure (a) The evolution of attention gradient sparsity across layers over training iterations during training. (b) The evolution of sink heads numbers across layers over training iterations during training.

---

Published as a conference paper at ICLR 2026
bimodal Gaussian distribution. We analogously model this entropy distribution p(e) E (e Ug, o with the valley _ arg min, p(e) acting as the threshold to distinguish high-entropy sinkg heads (focusing on global and contextual knowledge) from low-entropy sinkg heads (sinking attention onto isolated low-semantic tokens). This approach leverages Gaussian Mix- ture Models (GMMs) to capture the inherent separation, ensuring robust identification. We fit the GMM using expectation maximization, which converges efficiently and provides probabilistic as- signments, enhancing reliability in noisy attention distributions.
5.2 SINK ATTENTION DYNAMIC SPARSIFICATION
Leveraging the precise identification of sink heads, as illustrated in Figure 6 we introduce the parameter-free SADS framework. This approach adaptively retains all vision heads per layer while dynamically sparsifying the sink heads: it preserves the high-entropy sinke; heads (capturing global and contextual knowledge) based the valley of the bimodal entropy distribution a threshold, and designates the first head as a shared sink to handle essential textual and global information. Informed by the analysis in Section regardiin visual attention re-sinking in later layers, we acti- vate SADS from layers displaying variance fluctuations in attention keys and gradient sparsity. This selective activation maintains early-layer modality alignment while enhancing efficiency in later layers. During fine-tuning, SADS compels the model to prioritize visual features, thereby averting textual shortcuts and promoting deeper vision-text fusion, as evidenced by reduced hallucinations and improved visual grounding in downstream tasks.
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
MLLMs. In this work, we employ Owen2.5-VL-3BB Qwen2.5-VL-77 Intern Intem L2-2BB and LLaVA-1.5-7B (Bai et al. 2025 Chen al, 2024 Liu et al., 2023) base models. Notably, as parameter-free method, SADS can be readily applied to various MILMss
Qwen2.5-VL-33 VL-32B,
Tasks and Evaluation Benchmarks. We evaluate our method across broad spectrum of tasks, cat- egorized into: visual grounding tasks, general VQA tasks, OCR-related VQA tasks, vision-centric tasks, and visual hallucination tasks. Visual grounding tasks assess MLLMs' visual localiza- tion capabilities, including Referring Expression Comprehension (REC): RefCOCO/g/+ LISA, Re- fGTA, and Object Detection (OD): COCO and OVDEval (Lin et al. 2014 Lai al. 2024 Yao al. 2024). 2) General VQA tasks evaluate MLLMs' comprehension of image-text pairs. encompassing VQA v2, VizWiz, VQA vg, GQA, MME, MMB, MMStar, and A12D (Jia al.
2024 Gurari al. 2018 Krishna et al. 2016 Ainslie et al. 2023, Fu et al.] 2024 Liu et al. 2024, Chen et al.) 2024 Kembhavi et al. (016). 3). 3) OeR-related VQA tasks gauge MI.LMss proficiency
High-granularite
recognition for icon-document pairs, such as InfoVQA, TexIVQA, and Doc (Mathew et al. 2021b Singh et al. 2019). 4) Vision-centric tasks focus on visual-spatial perception in image-text understanding. including MMVP, CVBench, and CLEVER Tong et al. (2024a b); Johnson et al. (2016). 5) Visual hallucination tasks measure the authenticity and reliability of MLLM outputs, featuring POPE and CHAIR (Sun 2025 Li al. 2023).
Training Datasets and Implementation Details. Across the five task categories, we aggregate a total of 670k training samples sourced from RefCOCo, Dcube, VG, GQA, OCR-VQA, Text-VQA, and CLEVER for model fine-tuning, Our hyperparameters remain consistent with those of SFT and the base models across all benchmarks. For layer selection in SADS, we base our choices on the layers exhibiting K variance fluctuations and the onset of gradient sparsity: for Owen-2.5VL-33 we commence from layer 20, for Intern VL2-2B, from layer 15, for LLaVA-1.5-7B, from layer 20.
6.2 QUANTITATIVE RESULTS
Tables 2 and Tables 3 sequentially present the performance of RAR across the five task categories on 20 benchmarks using three base models. is evident that RAR consistently outperforms both SFT and the base models all benchmarks, demonstrating enhanced capabilities in visual local- ization, visual understanding, spatial perception, and hallucination mitigation. This underscores the effectiveness, scalability, and robustness of the RAR method. Notably, on out-of-distributiin
8

---

Published as a conference paper at ICLR 2026
Table Benchmark performance comparison on general VQA and OCR VQA tasks.
General VQA Task
OCR VQA Task
Model
VQA" GQA
VQA
MME MMB MMStar Al2D
InfoVQA
TextVQa DoeVQA
LLAVA-1.5-7B
78.3
61.1
54.6
1808.4
61.1
33.2
55.7
41.2
64.7
69.4
+SFT
79.1
63.1
55.7
1899.6
61.9
34.5
56.2
43.7
65.5
71.2
Ours
80.8
65.2
58.5
2018.8
63.2
36.3
57.1
46.3
67.5
74.4
Qwen2.5VL-3B
76.7
60.4
54.3
2184.1
75.4
53.0
77.9
75.1
78.7
93.0
+SFT
77.9
62.0
55.2
2199.9
75.9
53.7
78.4
75.9
79.0
92.9
Ouus
79.7
64.2
58.1
2276.3
76.9
55.4
79.5
77.3
80.4
93.5
LLter
72.9
55.6
50.1
1864.3
69.1
48.9
73.1
58.8
73.4
86.4
+SFT
74.2
56.9
52.3
1899.1
70.0
49.6
73.9
59.1
73.8
86.6
Ours
75.9
59.0
55.4
2006.5
71.6
50.8
75.7
60.5
75.9
88.2
Qwen2.5VL-7B
81.6
65.8
60.5
2276.3
82.2
64.2
84.1
81.7
80.2
94.8
+SFT
81.9
66.1
61.0
2230.2
82.0
64.5
84.4
82.0
80.7
94.2
Ours
82.6
67.9
62.1
2289.8
83.3
66.0
84.8
82.9
81.3
95.0
Owen2.5V1.-32R
82.9
68.4
63.6
2297.4
83.8
70.3
85.2
83.4
82.8
94.8
+SFT
83.0
68.6
63.9
2255.4
83.8
69.6
85.1
83.0
82.9
94.4
Ours
83.8
69.9
64.5
2326.6
84.5
71.3
85.7
83.8
83.9
95.1
Table 3: Benchmark performance comparison on visual perception tasks.
Visual Grounding Task
Vision Centric Task
Visual Hallucination Task
Model
Re/COCO//g LISA RefGTA OD
OVDEval MMVP CVBench CLEVER CHAIR,
POPE
LLaVA-155-78
76.2
44.2
64.1
19.4
22.7
3.1
57.4
43.6
44.7
85.6
+SFT
77.1
44.7
20.2
23.0
9.7
57.8
44.1
+ Ours
78.9
50.1
66.2
24.8
27.1
15.1
60.4
46.6
41.7
86.4
Qwen2.5-VL-33
84.2
55.3
70.8
32.1
39.5
50.4
67.3
68.7
35.6
86.1
+SFT
84.6
55.3
71.0
32.5
39,9
52.1
68.1
70.0
35.4
86.4
+ Ours
86.8
58.1
72.9
36.7
43.8
54.9
70.1
72.5
32.6
87.4
Intern VL2-2B
77.8
46.1
66.4
21.7
24.9
39.6
56.5
$7.1
37.8
86.2
+SFT
78.1
45.6
66.9
23.2
25.3
40.4
57.2
57.9
37.9
Ours
80.1
48.2
68.9
26.6
29.9
42.7
59.2
59.6
34.3
87.1
Qwen2.5VL SVL-7B
87.1
60.3
74.4
39.3
44.8
55.1
73.6
74.4
32.6
+SFT
87.3
60.1
74.8
39.7
44.4
55.8
73.8
74.8
33.1
89.2
+ Ours
88.2
63.6
76.0
42.1
47.2
$7.0
75.2
75.9
29.7
89.6
Qwen2.5V1.=322
89.8
65.9
77.5
43.1
49.3
60.4
77.2
78.5
28.2
90.3
+SFT
89.7
63.3
77.8
43.3
49.3
60.8
77.1
78.7
30.1
90.1
+ Ours
90.6
67.4
79.3
44.5
51.8
62.5
79.0
80.0
21.2
90.6
benchmarks such as LISA and OVDEval, DocVQA, CVBench, and CLEVER, SFT yields marginal improvements, whereas RAR achieves substantial gains, highlighting its superiority in bolstering visual capabilities. Furthermore, on hallucination tasks, SFT results occasionally fall below those of the base models, which we attribute to iterative training across diverse tasks exacerbating the model's 's bias toward linguistic priors. RAR effectively circumvents this issue, ensuring reliable visual comprchension. Additionally, Table illlustras that SADS significantly enhances inference speed compared to the base models and SFT, attributable to the reduction in computations associated with sink heads' attention.
6.3 ABLATION STUDIES
To directly assess whether the RAR resolves the suboptimal output layer issue, we compare the layer-wise performance of the base model, SFT, and RAR on the OVDEval test set, as shown in Figure Sc Evidently, unlike the performance degra- dation observed in later layers for both the base model and SFT, RAR achieves higher performance and demonstrate pro- gressive capability improvements across all later layers. As depicted in Figure Sal we contrast the visual attention maps in later layers, revealing that RAR effec- tively directs visual attention toward semantically relevant regions, in stark contrast the persistent
Table 4: Comparative analysis of in- ference efficiency and accuracy.
Latency..
Aceuracy!
V-L-325VVL-88
1,332
39,5
Ours
1.195
43.8

---

Published as a conference paper at ICLR 2026
What color is the peach in the photo?
Base
10
(b) Layer-wise Attention Gradient Sparsity: hase model, SFT Model and Our Method.
SFT
28
Ours
20
Performance comparison different lay- ers on OVDEval henchmark: Base model, SFT, and our method.
Layer 34
Layer 35
(a) Qualitative results of attention heatmaps for detect- ing the "peach" across Layer 34 and Layer 35.
Figure 8: Comprehensive ablation studies. (a) Qualitative attention heatmaps. (b) Attention gradient sparsity comparison. (c) Performance comparison across different layers.
illustrate the cross-layer variations in gradient sparsity, demonstrating that RAR mitigates gradient sparsity issues in later layers, whereas the base model and SFT suffer from severe sparsity therein. Furthermore, SFT hibits even more pronounced attention gradient sparsity compared to the base model, substantiating that training iterations exacerbate attention gradient sparsity.
visual attention re-sinking in the base model and SFT. In Figure 8b
We are particularly interested in assessing the scalability of SADS with respect to training data vol- ume. To this end, we perform comprehensive analysis of its performance across varying dataset sizes. As depicted in Figure experiments on the OVDEval benchmark show that as training data increases, SFT yields diminishing returns, with performance gains plateauing markedly. In con- trast, SADS demonstrates a steeper performance ascent, underscoring the visual attention re-sinking phenomenon severely impedes effective data scaling in existing MI.LMs, while SADS effectively mitigates this issue and unlocks superior scaling potential.
7 CONCLUSION
In this work, have elucidated the root causes of sub- optimal output layers in MLLMs, ascribing them to at- tention gradient sparsity precipitated by textual supervi- sion dominance during training. This sparsity engenders the Visual Attention Re-sinking phenomenon, head bi- furcation, and a progressive disregard for visual cues, ultimately culminating in degraded output performance. To mitigate these issues, we introduce the parameter- freee SADS framework, which dynamically retains all vision heads while sparsifying sink heads and ensuring model stability through a designated shared head. Comprehen- sive experiments spanning 20 benchmarks across five di- verse task categories demonstrate that SADS surpasses standard supervised fine-tuning in performance while ac- celerating inference by 10.3%.
40
Aroaet www.
Figure 9: Influence of different head selection strategies on model perfor- mance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was partly supported by the Special Foundations for the Development of Strategic Emerg- ing Industries of NWT..NN..KKN00020020000777700000
ETHICS STATEMENT
Our work is in accordance with the ICLR Code of Ethics. This research did not involve hu- man participants or animal testing. The datasets employed in this work are all publicly available and were utilized accordance with their original licensing and usage terms. These include the training datasets (RefCOCO, Deube, VG, GQA, OCR-VQA, TQA, CLEVER) and the evalu- ation benchmarks (RefCOCO/g/+, LISA, ReGGTA, COCO, OVDEval, VQA v2, VizWiz, VQA vg, GQA, MME, MMB, MMStar, AI2D, Info TextVQA, DOc,QA, MMVP, CVBench, CLEVER, POPE, CHAIR). Our methodology was designed to mitigate potential biases and avoid discrimina- tory results. The data used contains no personally identifiable information, and our experiments do not pose any privacy or security risks. We uphold the principles of research integrity and trans- parency in our work.
REPRODUCIBILITY STATEMENT
Details regarding the setup, such model hyperparameters and training procedures, are outlined in the main paper and appendix to ensure reproducibility. A comprehensive description of our SADS framework, including the specific layer selection criteria for each base model, is also included aid in replication. All evaluation benchmarks used across our five task nategories-vistee visual grounding, general VQA, OCR-related VQA, vision-centrie, and visual hallucination datasets, namely RefCOCO/g/+, LISA, RefGTA, COCO, OVDEval, VQA_v2, VizWiz, VQA.vg, GQA, MME, MMB, MMStar, AI2D, Info VQA, TextVQA, VOc, MMVP, CVBench, CLEVER, POPE, and CHAIR, allowing for consistent re-evaluation.
established public
REFERENCES
Estelle Aflalo, Meng Du, Shao- Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers.
Joshua Ainslie, James Teerphorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points, 2023. URL https: wrw.a.aiivv...oo...a. wwtp://ggg.bbaaaeee 2305 13245.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06-455 2016.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025.
Federico Barbero, Alvaro Arroo,, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why lims attend to the first token? arXiv preprint arXiv:2504.02732. 2025.
Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint vrXiv:2504.33181, 2025.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Jun 2023.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillo- tine regularization: Why removing layers is needed to improve generalization in self-supervised learning. Jun 2022.
11

---

Published as a conference paper at ICLR 2026
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. International Conference on Machine Learning, Intrinninntemaationaal Conference on Machine Learning, Jul 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and GeoffreyE. Hinton, A simple framework for contrastive learning of visual representations. Cornell University arXiv, Cornell University arXiv, Feb 2020b.
The Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of ilms are necessary during inference. arXiv preprint arXiv:2403.02/81, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL htt.tttss///rxii.or///////
2306. 3394
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018.
URL https: /ettps:/aaxxt /ttpsv////avvvvvi http///wwww.....aa././oe/
B02. 08218
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and cetrospection-allocatte on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
In Proceedings of the IEEE/CVE Conference
Ziheng Jia, Zicheng Zhang. Jiaying Qian, Haoning Wu, Wei Sun, Chunyi Li, Xiaohong Liu, Weisi Lin, Guangtao Zhai, and Xiongkuo Min. Vqa²: Visual question answering for video quality assessment, 2024. https:::::
wwwoor//abo.... abss//b////// 037955
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagmostic dataset for compositional language and elementary visual reasoning, 2016. URL http/v/axvvvv.///a org/ www.aawwwwww..aaaaaaaaaaaaaaaaaa
1612. 06890
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv preprint 25x1222503.3332,,2000
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, 2016.
07396
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016. URL /tttts:///axxi.orrrrt wwtp:/iwwwrgg.a../ooooooo 600.32332
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9579-9589, 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating ob- ject hallucination in large lision-language models, 2023. URL https: /wwxivii.v..../////.aawa///awa.v org/ abs/ 2305. 10355
12

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36:34892-34916, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: your multi-modal model an all-around player?, 2024.
abs 210.:022222222
Nanye Ma, Mark Goldstein, MichaelS, Albergo, NicholasM. Boffi, Eric Vanden-Eijndess and Sain- ing Xie. Sit: Exploring flow and diffusion-basse generative models with scalable interpolant transformers. Jan 2024.
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Emest Valveny, and C. V Jawahar. Infographicvqa, 2021a. URL
httpgg/bb//1100/ 12756)
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa on document images, 2021b. URL tttpp:://aa:x
00398
Shashank Shekhar, Florian Bordes, Pascal Vincent, and Ari Morcos. Objectives matter: Understand- ing the impact of self-supervised objectives on vision transformer representations. Apr 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. URL https::/axxiv.oxr
abs/ 020//9904.. 08920
Ao Sun. Chair classifier of hallucination improver, 2025. URL httts:://axiiiii abs/2501..222 02518
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms, 2024a. URL https://aaxxi..oott
abs/ 2406. 6666
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms, 2024b. URL https://arxiv.
org/abs/ 2401. 06209
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidanN. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Information Processing Systems,Neural Information Processing Systems, Jun 2017.
Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava. Teaching matters: Inves- tigating the role of supervision in vision transformers. Dec 2022.
Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.1177,, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. Sep 2023.
Yiyang Yao, Peng Liu, Tiancheng Zhao, Qianqian Zhang, Jiajia Liao, Chunxin Fang, Kyusong Lee, and Qing Wang. How to evaluate the generalization of detection? a benchmark for comprehensive open- vocabulary detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6630-6638, 2024.
Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. Cross-moda. information flow in multimodal large language models, 2025. URL htthhtts:://// //arxivv/// org/ abs 2211. 620
Liang Zheng. Yuchao Zhao, Shengjin Wang. Jingdong Wang, and Qi Tian. Good practice in cnn feature transfer. Cornell University arXiv, Cornell University arXiv, Apr 2016.
13

---

Published as a conference paper at ICLR 2026
Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu, Ibd: Alleviating hallucina- tions in large vision-language models via image-biased biased decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1624-1633, 2025.
A LLM USAGE
A Large Language Model (LLM) employed for language refinement and editing of this manuscript. Its role was to improve clarity, refine phrasing, and check for grammatical consis- lency across the paper. The conceptual and experimental aspects of this research, including the core ideas, methodology. and data analysís, were conducted exclusively by the human authors. The LLM's assistance was confined improving the manuscript's linguistic quality and was used for generating scientific content. The authors retain full responsibility for all content, including any text assisted by the LLM, and have verified its scientific accuracy and originality.
B ADDITIONAL ABLATION STUDIES
Ours
40
Accuracy (96)
43
shared head
Accuracy(%)
heads select
30
Ours
Base
40
10
0
25
40
6.
80
100
25
selection ratio(%)
Number of startup layer
Effect of sink heads selection strategy.
(b) Effect of SADS' start layer.
Figure 10: Comprehensive ablation studies on OVDEval benchmark. (a) Effect of Head Selec- tion Strategy. (b) Effect of SADS' Start Layer.
B. ABLATION STUDIES ON HEAD SELECTION STRATEGY AND START LAYER.
In the preceding experiments, the activation layer SADS is dynamically determined by the onset of variance fluctuations in attention keys (K) and gradient sparsity, while head selection leverages the valley of a bimodal Gaussian distribution fitted maximum visual attention values (for distin- guishing vision and sink heads) and non-vision token cross-attention entropy (for subdividing sink heads into sinkg and sinkg). To evaluate the effectiveness and robustness of these adaptive mech- anisms, we perform ablation studies employing fixed thresholds for both the number of activated layers and sink head retention ratios.
Table 5: Comprehensive ablation studies on
As shown in Figure 10a we first assess the im- pact of varying sink head retention ratios on performance, revealing that our dynamic selec- tion method, based on non-vision token cross- attention entropy, achieves markedly superior re- sults. Among fixed-ratio strategies, retaining ap- proximately 25% of sink heads yields optimal outcomes, consistent with the Pareto principle, whereas higher or lower ratios lead to perfor- mance degradation. Additionally, omitting the designation of a shared head (w/o shared) results in consistent declines across benchmarks, underscoring the critical role of fxee-pooition shared
head selection strategy.
Method
OVDEval
ReCCOCO1
GQAT
Base
29.5
81.2
60.4
SET
39.9
84.6
62.0
0rs heads
heads
40.8
85.]
62.2
100% sink hradss
30.9
84.6
62.0
ficed 25% vink heals)
42.8
86.1
63.5
14

---

Published as a conference paper at ICLR 2026
head in maintaining model stability. To further validate these findings across diverse benchmarks, Tablee|present comprehensive ablation studies on head selection strategies, demonstrating that our method (Ours) outperforms all fixed-ratio variants and the base model. These ablations collectively validate the superiority of SADS's adaptive design.
As depicted in Figure T0b ablating the SADS activation layer further demonstrates that starting from a lower layer (thus activating more layers) induces substantial degradation and garbled outputs, arising from unnecessary sparsification in well-aligned early layers. Conversely, starting from higher layer (activating fewer layers) causes performance drops due to unaddressed visual attention re- si-kinking later layers. These findings highlight the advantages of our dynamic layer selection in balancing modality alignment preservation with targeted late-layer optimization, with the optimal startup layer corresponding to the performance peak observed in the ablation curve.
Table 6: Ablation study across different iterations.
Method
OVDEval
RefCOcO1
GQAt
Base
39.5
SFT (14000 iters)
39.9
Ours (14000 iters)
43.8
64.2
SFT (13000 iters)
39.9
84.5
61.5
Ours (13000 iters)
43.4
86.2
63.8
SFT (12000 iters)
39.6
84.3
61.7
Ours (12000 iters)
43.4
85.7
63.2
SFT (11000 iters)
39.6
84.5
61.0
Ours (11000 iters)
42.8
85.8
63.3
SFT (10000 iters)
39.7
84.4
60.8
Ours (10000 iters)
42.3
85.6
62.5
B.2 ABLATION STUDY ON TRAINING STEPS.
As shown in Table to investigate whether earlier checkpoints during training yield superior per- formance, we evaluate and compare model performance across varying training iterations. Key ob- servations include: (1) Earlier checkpoints exhibit performance degradation, as prematurely halting training mitigates gradient sparsity issues but exacerbates under-training, leading to suboptimal re- sults; (2) Across all checkpoints, method consistently outperforms SFT, underscoring its robust performance and stability.
Table 7: Ablation study across different training objectives.
Method
OVDEvalt
RefCOCOT
GQAT
Base
39.5
84.2
60.4
SFT
39,9
84.6
62.0
regularization
40.8
85.3
62.7
w/ atlention reweight
40.6
85.0
62.3
Ours
43.8
86,8
64.2
B.3 ABLATION STUDY ON DIFFERENT TRAINING OBJECTIVES.
As shown in Table 7 we further examine the performance under different training objec- tives. We explore two alternative strategies: (1) regularization; regularization to elevate the weights of vision heads, implemented via a KL divergence prior that encourages vision heads to align with high-weight distribution (1.2,00.1) while constraining sink heads to low- weight distribution (N(0.33 0.1)), formally expressed as Creg Enevision KL(a,, I[ aZ)) + Ene (L [[ of)): ; (2) attention reweight; attention reweighting on sink tokens within sink heads prioritize semantically relevant visual information, achieved by dynamically scaling
15

---

Published as a conference paper at ICLR 2026
attention weights based visual saliency metries, reweighting d',; رi,j s// Ex ai.b Sk. where 8, denotes the saliency score of token j (upweighting edges toward high-semantic regions while downweighting low-information backgrounds). The results indicate that both approaches yield modest improvements over vanilla SFT. However, our SADS framework substantially out- performs these variants. This superiority stems from the root cause of suboptimal output layers: the Visual Attention Re-sinking phenomenon within sink heads, which impairs model performance by marginalizing visual cues. While regularization and reweighting provide temporary alleviation of sinking, the issue inevitably re-e re-emerges with deeper training iterations. In contrast, our sparsification approach fundamentally eradicates re-sinking, thereby maximizing the model's potential.
B.4 STATISTICAL ANALYSIS IN SINK HEADSS.
randomly sample 2,000 heads and compute their non-image token cross-attention entropy. revealing a pronounced bimodal Gaussian distribution characterized by distinct low-entropy (sinky heads) and high-entropy (sinkg heads) peaks. Consequently, for each layer, we model the entropy distribution across all sink heads using GMM and employ the valley dynamic threshold for differentiation.
As illustrated in the Figure
Original Data (n=2000)
Bimodal Distribution (GMM)
Sink_5 Link_Sti heads
Sink_G G heads
Valley (x=1.7)
2.0
Density
1.5
Valley (1.7, 0.009)
0.5
1.0
2.0
3.0
Non-Vision Token Cross-Attention Entropy
Figure 11: Distribution of non-vision token cross-attention entropy for Sinkg heads and Sinks heads, showing bimodal distribution.
C LIMITATIONS AND FUTURE WORK
Although our method effectively maximizes model capacity at current parameter scales, we ac- knowledge its inherent limitations. While sparsifying redundant gradient spaces has proven effica- cious, infilling techniques may offer a more robust alternative. In future work, we plan to integrate the SADS framework into unified generation-understadding paradigm, leveraging generative pabilities to populate sparse spaces and further enhance multimodal fusion.
D ADDITIONAL QUALITATIVE RESULTS
We provide additional Qualitative results of cross layer visual attention map in the Figure 12 and low-entropy sinks and high-entropy sinkg in Figure 13
16

---

Published as a conference paper at ICLR 2026
Base
Layer 0
Layer
Layer
Layer 16
Layer 21
Layer 26
Layer
Layer 34
Layer 35
SFT
Layer 0
Layer 6
Layer 11
Layer 16
Layer 21
Layer 26
Layer 31
Layer 34
Layer 35
Ours
Layer 0
Layer 6
Layer 11
Layer 16
Layer 21
Layer 26
Layer 31
Layer 34
Layer 35
Figure 12: Qualitative results of visual attention heatmaps for detecting the "mouse" across Layers.
17

---

Published as a conference paper at ICLR 2026
Low- Entrayy
Low Entropy
Lou-Entrogy
High- Entropy
High-Entropy
High Entropy
Figure 13: Qualitative results of low-entropy sinkg and high-entropy sinkg
18


Reviews:
[{"review_id": "OmrC42l7c4", "reviewer": "Reviewer_3kFx", "summary": "This paper studies the root cause of the visual attention resinking phenomenon, and shows that this phenomenon is the reason why output layers of MLLM yield worse performance than middle layers, a frequently observed phenomenon in existing work. The paper finds that the visual attention resinking phenomena is caused by attention gradient sparsity, which makes the gradient distribution over attention heads sparse hence causing sink tokens to concentrate in sink heads. Based on these findings, the paper proposes a \"Sink Attention Dynamic Sparsification\" strategy, which selects vision heads per layer and forces model to focus on vision signals. Experimental results show improvements over baselines.", "questions": "- It seems the gradient sparsity problem gets worse as the training steps increases. If this is the case, I wonder whether the MLLM performance (from the last output layer) would be better at an earlier checkpoint?\n- In Fig 5a, how is the gradient sparsity precisely defined? \n- The framework retains the top 25% of sink heads, is this ratio empirically optimized, and how sensitive is the model performance to this threshold?\n- The tested models used are up to only 7B parameters. Would the same sparsification principle hold in larger models? Due to increased capacity of larger models, it's conceivable that the resinking problem would be alleviated.\n- Would it be possible to address the gradient sparsity problem with better training objectives (e.g. adding regularization terms to upweight vision heads), which address the problem at a more fundamental level?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper developed a systematic approach to diagnose a commonly observed but not well-understood phenomenon. The analysis is logical, insightful and convincing, revealing an overlooked aspect in MLLM training and inference.", "weaknesses": "The proposed approach can be viewed as a patch on existing models rather than a solution to the root cause of the resinking problem. It would strengthen the paper if it also proposed ways to address the issue more fundamentally (for example, through improved training objectives).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7j6W3iJZRT", "reviewer": "Reviewer_7PSW", "summary": "This paper focus on the problem that the intermediate decoder layers outperform the final ones in MLLMs, and attribute this issue to the Visual Attention Re-sinking phenomenon. Then, a parameter-free Sink Attention Dynamic Sparsification framework is proposed. The proposed SADS achieves superior effectiveness and inference efficiency on several benchmarks.", "questions": "Refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It deeply identifies text-only supervision as the cause of suboptimal MLLM output layers.\n2. The SADS framework effectively addresses key issues to optimize output layers.\n3. Comprehensive experiments across 20 benchmarks validate its superiority.", "weaknesses": "1. From Figure 2 and Table 1, it can be observed that the performance degradation caused by the re-sinking phenomenon is limited. It can also be seen from the experiments that the improvement brought by the proposed method is limited.\n2. To demonstrate its effectiveness and  generalizability, more models of different sizes (7B, 13B, ...) should be tested.\n3. An ablation experiment on the proportion of sink heads should be added; additionally, is it reasonable to use a fixed proportion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KcrPBpg9SO", "reviewer": "Reviewer_VNAx", "summary": "The paper presents RAR: a parameter-free method to address the problem of re-sinks in the final layers of a vision-language model, that cause the model to underperform due to undesired attentions. Unhealthy attention heads are removed, which improves the performance.", "questions": "Which subset of the attention heads is unhealthy and why is that, why does that hurt the performance of the last layers? Can you provide insights with some illustrations, and with some statistics, e.g. are they mostly on particular areas in the image, or semantic regions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The observations about the problems in the final layers, the degradation of performance because of that, the illustrations of spurious visual attention, are all interesting. The finding that visual attention is bi-modal and separates the vision heads and sink heads, is interesting too. The performance gains are structural but marginal. The proposed method is effective and simple, which makes the approach feasible for a broader audience.", "weaknesses": "The illustrations are sparse (e.g. Figures 3c, 4a and 8a) and sufficient to convey the main idea, but they are redundant and simplistic, as a consequence a better understanding of the separation into visual and attention heads is not provided. More importantly, the distinction between useful and unhealthy sink heads is not illustrated, and details are lacking to understand this better. \n\nThe whole idea of the paper is to remove the relatively small subset of those unhealthy sink heads. A better understanding of that subset, which ones they are, why they hurt the performance; this is fundamental for a paper that builds on those principles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)