To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
Authors: Jiayun Luo
Wan-Cyuan Fan
Lyuyang Wang
Xiangteng He
Tanzila Rahman
Purang Abolmaesumi
Leonid Sigal
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Transformer
Visual Attention Sink
Attention Sink
Multimodal LLM
Large Vision Langauge Model
Primary area: foundation or frontier models, including LLMs

Abstract: Large Vision Language Models (LVLMs) have recently emerged as powerful architectures capable of understanding and reasoning over both visual and textual information. These models typically rely on two key components: a Vision Transformer (ViT) and a Large Language Model (LLM). ViT encodes visual content into a sequence of image tokens and serves as the perceptual front-end -- the eyes of the model. In contrast, the LLM interprets these tokens to perform high-level reasoning, generates responses, and functions as the cognitive core -- the brain of the model. However, it remains unclear which visual tokens contribute most significantly to understanding and reasoning, and how effectively these signals are propagated from ViT to the LLM. While most existing works have focused on identifying attention sinks, low-semantic tokens receiving disproportionately high attention, within the LLM, we shift the focus to the vision encoder by identifying a class of high-norm visual tokens from ViT, referred to as ViT attention sinks -- a problem that has been rarely studied but is indeed very important for LVLMs. Our findings show that these ViT sinks encapsulate high-level semantic concepts from images, allowing the LLM to perform more effective understanding and reasoning. Despite their importance, these sink tokens are often overlooked in existing LVLM architectures. To explore their contribution, we present both qualitative and quantitative analyses of the information embedded in these sink tokens. We also propose both training-free and training-based approaches to better leverage how this information is interpreted by the LLM, and to what extent. By explicitly utilizing these tokens, we demonstrate substantial improvements across a range of LVLMs and visual reasoning tasks, including but not limited to mathematical problem solving, logical inference, and geometric understanding, highlighting the untapped potential of ViT attention sinks in enhancing visual reasoning.

Published as a conference paper at ICLR 2026
To SINK OR NOT TO SINK: VISUAL INFORMATION PATHWAYS IN LARGE VISION-LANGUAGE MODELS
Jiayun Luo
Wan-Cyuan Fan
Lyuyang Wang'
Xiangteng Hel
Tanzila Rahman
Purang Abolmaesumi! Leonid Sigal
University of British Columbia
Vector Institute for AI
(letitial, wancyuan ) Ccs. ubc. ca
A. Given Image
B. VIT Sinks
C. Propagated VIT Sinks
D. LLM-emerged
Sinks Sinks
All VLM Sinks
Illustration of VIT and LLM attention sinks in LLa In LVLMs, given an image (A), we find that ViT sinks (B) are partially propagated into the LLM as (C), alongside LLM-emerged sinks (D), together outlining all sinks within the VLM (E).
Figure 1:
ABSTRACT
Large Vision Language Models (LVLMs) have recently emerged as powerful ar- chitectures capable of understanding and reasoning over both visual and textual information. These models typically rely on two key components: Vision Trans- former (ViT) and Large Language Model (LLM). ViT encodes visual content into sequence of image tokens and serves as the perceptual front-end - the eyes of the model. In contrast, the LLM interprets these tokens to perform high-level reasoning, generates responses, and functions as the cognitive core the brain of the model. However, remains unclear which visual tokens contribute most significantly to understanding and reasoning, and how effectively these signals are propagated from VIT to the LLM. While most existing works have focused on identifying attention sinks, low-semantic tokens receiving disproportionately high attention, within the LLM, we shift the focus to the vision encoder by identifying a class of high-norm visual tokens from VIT, referred to VIT attention sinks problem that has been rarely studied but is indeed very important for LVLMs. Our findings show that these VIT sinks encapsulate high-level semantic concepts from images, allowing the LLM to perform more effective understanding and reason- ing. Despite their importance, these sink tokens often overlooked in existing LVLM architectures. To explore their contribution, we present both qualitative and quantitative analyses of the information embedded in these sink tokens. We also propose both training-free and training-based approaches to better leverage how this information is interpreted by the L.M,, and what extent. By explicitly utilizing these tokens, we demonstrate substantial improvements across a range LVLMs and visual reasoning tasks, including but not limited to mathematical problem solving, logical inference, and geometric understanding, highlighting the untapped potential of ViT attention sinks in enhancing visual reasoning.
Eqqul contribution. Listed in alphabetical order.
Project page: DTYSink

---

Published as a conference paper at ICLR 2026
Image & Query
VIT
DIYSink
LLM
Output
How menry sheep this image?
EREER (
Baseline
A realistic and complex Image with query on local region.
Two
Dual-ML Projector
Semten Encoder
Four
Query 3
Rownighting
VIT Sinks
Propaggtted VIT Siita
Emarged
What the measure angle LED??
Visull Non
A symbolic and simple image with query focue globel understanding.
Baseline
Use sink tokens
Ours
V
40"
VIT Attn Map
COT-Reweightng
Or
b) MLP-Reweighting
LLM Attn Map
Figure 2: Overview. DIYSink leverages a Dual-MLP Projector to correctly project sink and non-sink tokens, and one of the token selection modules, CoT-RRweighting or MLP-Reeeightinn, to dynamically select the best set of tokens for the L.M based on the specific input.
INTRODUCTION
Large Vision Language Models (LVLMs) Liu ct al. 2023c Zhang et al. 2023 Bai et al. 2023
Awadalla et al. 2023 Chen et al. 2023b Abdin et al. 2024), which integrate the visual capabilities of vision transtormers (e.g.. ViT's) with the generative capabilities of large language models (LLMs), have demonstrated impressive performance across a broad spectrum of multimodal tasks, spanning visual question answering (Antol et al. [201 Goyal et al. 2017), mathematical reasoning (Lindström & Abraham 2022 Lu et al. [2023] Chen et al. (021), ) and many others. As these models continue to improve and find increasing deployment, there has been growing interest in understanding their internal mechanisms, especially their attention dynamics. Attention plays a central role in how LVLMs integrate and align visual and textual inputs. Specifically, attention weights determine how strongly each text (e.g., output) token is influenced by the corresponding (e.g., input) visual token.
One notable emergent behavior in these models is the phenomenon of attention sinks (Xiao al. 2023b) Kang et al. 2025 Sun et al. [024]] Darcet et al. 2024), where the model disproportionately assigns high attention on small subset of tokens; often irrespective of the input. These tokens often corresponding to non-semantically or uninformative regions such as blank image (visual) or punctuation (language) tokens. This pattern has been consistently observed VITs (Darcet et al., 2024; Jiang et al.) 2025) and LLMs (Sun al. 2024 Gu al. 2024). and naturally extends to LVLMs due to their hy brid architecture.
Attention sinks in LVLM models have generally been shown to be detrimental to model performance, with methodologies centered on identifying and de-emphasizing, if not completely removing, them in inference. Specifically, Kang et al. (2025) shows that masking visual sinks leads to little effect and redistributing attention to other tokens actually leads to improvement; Darcet et al. (2024) prepend register tokens absorb high-norm tokens in VIT and achieve better unsupervised object discovery with the register-fir free feature maps. These findings, however, are seemingly in contrast to recent findings in LLM models that find that attention sinks are implicitly useful and encode indispensable biases (Sun et al., 2024), which could be beneficial for long-context handling and to reduce over- mixing (Barbero al.) 2025). This raises an important question Are there any fundamental benefits to sink tokens in LVLM models, and if so, can they be understood and operationalized?
To address this question, we first carefully study the emergence of sink tokens in LVLMs, showing that they result in combination of sink tokens from VIT backbone being propagated to LVLM (Fig. (purple)) and sink tokens from the L.LM itself (Fig. 2 (yellow)) an observation not previously made. We then focus analyzing, the less explored, VIT sinks (Fig. 2 (purlee)) making three core findings: (I) The propagated VIT sinks capture coarse, high-level contextual information, (2) these tokens benefit certain tasks that require high-level image understanding reasoning, and (3) given the seemingly different level of semantics encoded in these sinks vs. other visual tokens (global vs. local), using single learned projection across both diminishes their effectiveness in practice.
Armed with these findings, we propose an effective way to dynamically enhance LVLM performance, by emphasizing and (sometimes) de-emphasizing VIT sinks depending on the task and image content.
2

---

Published as a conference paper at ICLR 2026
Specifically, first, leveraging (1) and taking advantage of the LLM's causal structure, we propose straightforward training-free free approach on enhancing LVLM performance by moving the ViT sink tokens to the front. This shows improvements across variety of LVLM models, and particularly for tasks that require a high-level understanding or reasoning. Second, we introduce DIYSink, illustrated (middle), training-based framework designed help LVLMS make better of ViT sink tokens. The framework relies on dual MLP projection trained separately to most effectively use ViT sink and non-sink tokens by the LLM addressing (3). Given the dual MLP projection LVLM variants, we then leverage two mechanisms to dynamically select which class of tokens should be used in inference (sink, non-sink or both) and to what extent: operationalization observation (2). One mechanism leverages hard selection obtained through Chain-of-Thouull (CoT) routing, while another trains a lightweight soft weighting module with small amount of cross-task data.
in Fig. [
We validate proposed D/YSink using four ViT-LLM combinations, with LigLIP (Zhai et al. 202 and CLIP- VIT (Radford et al. 202T) as vision backbones and Qwen2 YYang et al. 2024). Qwen2.5 (Qwen et al. 2025), Phi-2 (Abdin et al. 2024), and Vicuna (Zheng et al. 2023) language backbones of varying sizes. Our approach consistently improves performance across a wide range of benchmarks.
Contributions:
To summarize, our contributions are two fold. On one hand, they take the form of analysis that sheds light on the role and function of ViT sinks within LVLM models. On the other hand, it leverages those observations, to propose simple yet systematic and effective improvements for both open-source (raining--aased and closed-source (training-free variants of these models.
PRELIMINARIES
Large Vision Language Models (LVLMs) comprise of three key components: (1) Vision Encoder E, (e.g., CLIP, SigLIP): extracts visual features from input images, producing hidden state V = [V1, Ve,..., vvl, V, E R',, where D' is the hidden dimension of ViT and n is a number of image patches. (2) Connector module & (e.g.. Q-Formers (Li et al.. 2023), MLP Projectors (Liu al. 2023ch): maps the visual features to the textual space of the LLM, producing visual tokens Zwis- (3) Language Model & (e.g., LLaMA (Touvron et al., 2023). Qwen (Bai et al. 2023), Phi (Abdin et al. 2024)): an autoregressive transformer that takes in sequence of tokens, namely, system tokens (yys), visual tokens (Ivis), and query tokens (Ix)). and processes them layer by layer to generate outputs (Tou)). Each token has hidden state x E R.D at layer 1, which updated through attention and feedforward networks. During multi- attention (MHA), each token can attend to others. Notably, MHA ViTs non-causal, allowing full token interaction, while MHA in LLMs is causal, restricting attention to past tokens only. Following the definition in KKng et al. (2025). we define the I.b attention weights oi.. to be the strength with which token i attends to j layer and head h. We focus specifically on visual attention, analyzing interactions between Tvis and the text: Iaa and T
Attention sinks and massive activation. (Sun al. (2024] Kang et al. (2025) Darcet al. (0244)
Cancedda (2024); Gu et al. (2024); Yu et al. (2024); Zhang et al. (2024):
:): Ge et al. (023); Xiao et al. (2023a): [Barbero et al. (2025) observed that sink tokens are common transformer-baser models including ViTs and LLMs. These sink tokens, usually receive disproportionately high attention, tend to exhibit abnormally high values (so called massive activation) in particular hidden dimensions, referred to as sink dimensions Daink (Kang et al. (202)), and have high feature norms. As such, sink tokens can be identified by the following definition:
(1) where & is a sink characteristic function, and 7 is the pre-defined threshold. The sink charac- teristic function & can be: (1) feature norms of the token DDacett et al. 2024 & (x =
f' = E I| ø
1x 11 (2) average attention weights on the token : & (x)
FTalll Liez.. oijj" where
1.
denotes the head of interest, and (3) sink dimension value (Sun et al., 2024) of the token
o(x) ') = ma [RMSNorm(x)) "x]] where Dann is the pre-defined dimensions (e.g.. Daink = (1415, 2533) for LLaMA2-7B), and RMSNorm is Root Mean Square Normalization.
In this paper, we focus on VIT sinks in ViT denoted as Propagated ViT sinks from ViT to LLMs as -m and the LLM-emerged sinks Zimm Following Darcet et al. (2024). we identify Z! with the sink characteristic function à as a feature norm in Eq. (1) and 100 (details in Sec. BB.). For the
I.M sinks Z. m we follow Kang al. (0225). defining @ as the sink dimension value with 20.
3

---

Published as a conference paper at ICLR 2026
Attn Weight Analysis
B. Hidden Dim of LLM-emereed Sink
C. Hidden Dim Propogated VIT Sink
Age boom
Men Warr ue a
12-
Aug Mtn for UMM Oulputs
We #o tokens
Migritude
Mapnitude
101
Norom
Token
Hidlen Dimension
Hidden Diminsion
Figure 3: Attention ViT sink tokens and sink dimensions of ViT and LLM sinks in LLaVA- v1.5-7B. (A) compares VIT token norms with the attention assigned during LLM decoding. (B)-(C) show the sink-dimension distributions for emeegeett sinks and ViT-propagated sinks. For enlarged version, please refer to Fig. A10
VIT SINKS IN LVLMS
We now provide our analysis on ViT sink tokens within the LVLMs, examining how they propagate through the model, what information they encode, and how they affect the behavior of the LLM. We show analysis on LLa VA-7B, and also provide the analysis results on other LVLMs in Sec. A The observations are also consistent across vision encoders (i.e., CLIP, SigLIP).
3.1 THE PROPERTY OF PROPAGATED VIT SINKS IN LLMS
Propagation of VIT sink tokens into LLM. ViT sinks, as studied in Darcet et al. (2024). exhibit high vector norms. Given that many related works use attention weight to identify the sink tokens in LLM (Sun et al. [024]] Kang et al. 2025 Barbero et al. 2025 Yu et al., 2024), to quantitatively verify whether these high-norm vectors are being propagated into the LLM as sinks, we visualize the relationship between ViT token norms calculated in ViT and the attention weights they receive from the L.M during output generation, with results averaged across 300 image-question pairs. As shown in Fig. A). the x-axis here represent the categorized token norm. The left y-axis and the yellow line represent the average number of tokens fall in each bin. The right y-axis and the purple bars represent the attention weight assigned from the LLM to the visual tokens in that bin during output generation. We observe a positive correlation between ViT token norm and the attention weight assigned by the LLM during generation. From this, we conclude:
Tokens with higher norms in the VIT are more likely to receive higher atlention weights and become sinks in the L.M.M
Specifically, we find that most tokens have a norm below 60 in the VIT. ViT.Only a small number of tokens (typically 3-5 per image) have norms above 100, yet these high-norm tokens receive substantially higher attention weights, approximately 7 times greater than those assigned to the rest of the tokens. We highlight that this correlation is not architecturally enforced, making it a non-trivial finding. suggests that the LLM implicitly inherits ViT's internal saliency signals, revealing strong inductive link between visual and language components LVLMs.
Hidden dimension breakdown of propagated VIT sink in LLM. We further investigate how visual sink tokens from the VIT propagate to L.M. In Fig. BJ), we plot the hidden dimension magnitude of the LLM-emerged sinks Zim and, in Fig. 3 C), the hidden dimension magnitude of propagated ViT sinks Leit-IImm The values are from the second last layer of the LLM, averaged over 300 examples. Across these examples, we observe that ViT sinks have high activation consistently on certain hidden dimensions (i.e., 982, 2494, and 3263) within the LLM, regardless of the input image prompt. These dimensions are different from the sink dimensions of LLM sinks. In summary:
Visual sink tokens propagate into the LLM as distinct sink tokens, activating different hidden dimen- sions of the sinks emerged in LLM.
Moreover, these high value sink dimensions emerge only after multimodal training. In LLaVA-7B. the LLM's original sink dimensions are (2533, 1415), while the propagated VIT sink tokens activate dimensions (982, 2494, 3263). This distinction is critical: "Prior studies (Kang et al. 2025) identify
4

---

Published as a conference paper at ICLR 2026
A. Interpretation Flow of Sinks
8. Relevance Maps
C. Word Distributions
UM
Srb
Noe.ink
for
NO0 D Duserbuuue
Godks
dog
Sme MIZI
Fidee
litla
4773
ground
Extracted map
Relevance
Madified
0.2.345
Fizcuoncy
ProdMoncy
Figure 4: (A) Flow of obtaining relevancy map and word distribution. (B) Relevance map of sink and non-sink tokens. H12 and H10 denote the respective foreground and background attention head of the penultimate layer of the LLM used to extract the relevance maps. (C) Word Distribution of sink and non-sink tokens from the foreground head (H12). The first and second rows represent the word distributions obtained from 300 images where cat and person are the main objects.
sinks tokens according to LLM original sinks dimensions (Sun et al. 2024 and found redistributing their attention improve overall performance. This may inadvenrtently conflate the effect of two different sinks." We argue that explicitly disentangling them is essential to understanding their distinct behaviors, given the fundamentally different architectures they originate from.
Impact of VIT sink tokens in LLM. With the aforementioned observations, the ViT-propagated sink tokens within the LLM. Lastly, we investigate their importance by analyzing the average attention weights received by ViT sink tokens. By computing the average attention weights from output tokens to target tokens across 1, 000 image-question pairs, we find that, on average, non-sink tokens receive 0.1532% attention per token, LLM--eeeged sink tokens receive 1.27%. and ViT sink tokens receive 1. 1.1%.. This illustrates the importance of ViT sink tokens.
able identify
3.2 WHAT IS IN THE VIT SINKS AND HOW DOES LLM INTERPRET THEM?
As shown the previous section, VIT sinks have a significant impact on the model's output, suggesting they might encode information critical for model learning and inference. To better understand the content and role of these tokens, we examine the attention mechanism in both the ViT and the LLM.
Interpreting sink tokens via Relevance Map in the attention layer. In transformer-based models like ViT, attention maps provide insights into how the model aggregates information (Kovaleva et al., 2019 Reif al. 2019). In these maps, vertical columns indicate how much attention particular token receives from all other tokens, reflecting the relevance or importance of the target token during processing. As illustrated in Fig. (A), visualize the vertical column corresponding the sink in the given attention map. After reshaping and normalizing this attention column, we obtain a 2D map with the same spatial layout as the image patches, referred as Relevance Map. Fig. 4 (B) presents relevance maps for sink and non-sink tokens across three images. Note that the attention maps are extracted from the recond-to-last layer, with head 10 (H10) used for background and head 12 (H12) for foreground in CLIP-ViL We observe that the non-sink token shows high relevance primarily to its local neighbors, whereas the sink token receives attention from tokens broadly distributed across either foreground or background regions, indicating that
VIT sinks capture course mourse-graine, high-level contextual features aligned with the foccs of each attention head.
Decoding VIT sink tokens into word distributions. Using the relevance map, we qualitatively interpret the information encoded in VIT sink tokens. To analyze it quantitatively, we decode visual tokens into word distributions leveraging the LLM. Together with the relevance map, we can analyze multiple images and collect word distribution associated with the target token. Inspired by previous works of ViT concept discovery (Rao et al. 2024] Chen et al., 2023a), we disable attention from all tokens to visual tokens in the LLM,, illustrated by the modified attention mask in Fig. 4 (A), to
5

---

Published as a conference paper at ICLR 2026
A. Task Clustering
B. Performance Analysis
Mixed
Rolatbe purformance change
thflfloooooe 2 $0.0.
Harrwhee to Une dogres of
image deotoz
TinyLLaVA-00500 Siglip-Qan2
logical messening pibera ressoning
Theee bediard
code reasoning
bosted WWen the medise of the rumbbeed?
Global
Local
oiters
0
Image Complexity
Niw
Figure 5: (A) Task Clustering based on GPT-4o annotated Image Complexity and Query Globalness. (B) Performance analysis of two model variants (i.e. Sink-only and Non-sink-only) for evaluating the influence of ViT sink tokens on different tasks.
prevent information exchange. By forwarding these isolated visual tokens through all layers, we are able to map their embeddings the output vocabulary, generating word predictions for each.
We visualize the word distributions for 300 cat images and 300 person images, as shown in Fig. (C). We observe that sink tokens are strongly associated with the main object (i.e., "cat" or "person"), whereas non-sink tokens yield far fewer semantically aligned terms. These quantitative results across multiple images support our finding that ViT sink tokens encode graiee,rained, high-level contextual features and are semantically meaningful.
3.3 TASKS CLASSIFICATION AND PRELIMINARY EXPERIMENT
Interpretation and hypothesis. Above observations that VIT sink tokens are significant and appear to carry gourse-grained high-level contextual information lead to a hypothesis regarding their use. Specifically, (1) we hypothesis that such compact high-level contextual information could be very useful for tasks that require (e.g., scene recognition) and may distract the model when the task is highly local and unlikely to be captured in such high-level context (e.g., localization). Further, (2) given the fixed capacity of these tokens, benefit of them can be reduced for images that are complex and cannot be summarized effectively. To validate this hypothesis, setup simple experiment.
Task taxonomy via query and image properties. To validate our hypothesis, we curated a dataset of 600 muage-query pairs, uniformly sampled from suite of widely used benchmarks, including GQA (Hudson & Manning 2019), TextVQA (Singh et al. 2019b), ScienceQA (Lu et al.) (002). MME (Fu et al.) (0023). Math (Lu et al. 2023). Each instance was annotated using GPT-4o with two ratings: (i) image complexity, measuring the visual density and richness of the scene, and (ii) query globalness, assessing whether the question depends on high-level contextual reasoning or fine-grainee spatial cues. Based on these continuous annotations (normalized to [0. 5]), we categorize each instance into one of three clusters (see Fig. (A)): Global tasks (low image complexity). Local tasks (high complexity and low query globalness), and Mixed tasks (the rest).
Effect of ViT sink tokens on downstream tasks. We investigate the influence of VIT sink tokens during inference using two configurations: (1) Sink-only and (2) Non-sink-only, where we only use ViT sinks or non-sinks as visual input to the LLM for inference. From the results shown Fig. (B), we observe that Sink-only configurations yield strong performance specifically on global tasks, suggesting the compact high-level contextual information in ViT sink can be very useful for tasks that require it. In contrast, for local tasks, removing sinks improves performance, indicating potential interference. This observation confirms that
ViT sink tokens encode useful semantic summaries, but only useful under the right conditions. They are beneficial for tasks with low visual complexity and global semantics but may degrade performance on tasks that demand localized, detail-rich visual processing.
This context- dedendent behavior underscores the importance of adaptive sink usage and motivates further architectural design around task-aware visual token selection. For more analysis results,

---

Published as a conference paper at ICLR 2026
Table Performance impact of reordering VIT sinks to the front. Green and red indicate increase or decrease over the corresponding baseline. Subscripts show the absolute change from baseline.
LLaVA eval
MME
Math Vista
Methods
AVG
GQA
SQA
TeaVQA
MMMU
ALT.
Cogsitin
Conflies
TextTrans
CodleRea
ATJ.
GRR
1.00
1nten.V12..54
70,99
62.31
96.18
73.47
50.20
2332.16
645.35
142.86
200.00
177.50
62.90
65.48
66.53
16.22
Sink to the frot
71.01
02.39
96.06
73.17
50.40
2351.33
645.35
145.00
200.00
175.00
03.30
05.12
68.20
16.22
Mi 3.5 V
66.29
63:48
91.25
05.24
45.20
1887.00
412.14
137.14
$7.50
17.50
43.10
40.21
38.49
10.11
Sinb-oothosfnnt
65.71 4442
63.49
01.31
64.05
1891.27
60.77
433,92
136.42
87.50
112.50
40.57
38.49
61.35
61.23
$1.85
66.20
10
1787:34
709288
129 26
85.00
35,00
37:20
10.96
32.02
16.22
Sinb-wstho-fnet
61.56 v0.1
61.31
$1.99
1766-40 099
301.42
131.42
32.90
37.70 -0.90
31:32
32.02
Motino 7B.D D
61,88
54.53
$6.30
61.49
44.10
1821.98
372.34
122. 14
72.50
87.50
49.40
37.01
35.56
8.11
Sink
4406
56.35
1850.31 63022
127.14
41.20..1.0 -1.0
37.37
Gemmar 128-ii
61.26
45:34
82.55
70.23
47.00
1706-30
$35.80
127.86
112.50
162.50
40.80
48.40
48.95
Sink-ws-th--eett
6025
45:41
$2.50
70.30
48.00
1740.31 4
130.71
130.00
170.00
41.00 -0.00
50.89
Meteor
66.71
62.54
94,29
67.01
39.00
2158.75
508.21
150.00
52.80
40.21
42.68
Sink-oothh-ffott
67.57
94.65
67.07
39.10
2198.47
516.42
141.42 42
100,00
110.00
40.00.0m0
40.21
42.68
13511
including ViT sink tokens behavior across different checkpoints and linear-probing test of the semantics of VIT sink tokens, please refer to the Sec. A..3 and Sec. A.4
4 LARGE VISION LANGUAGE MODEL (LVLM) FRAMEWORK REDESIGN
4.1 TRAINING-FREE APPROACI: FOR WHEN TRAINING IS NOT AVAILABLE.
Motivated by the analysis in Sec. 3.3 we propose a simple yet effective inference-time strategy called sink-to-the-front where VIT sink tokens are repositioned to the beginning of the visual token sequence. This allows subsequent tokens to refer to sink tokens and can benefit mixed and global tasks that require high-level context (see left side of Fig. (5). while maintaining performance on local tasks. This approach requires no additional training and can be applied post hoc to any existing LVLM. Specifically, in the inference phase, given visual token sequence from the vision encoder in a L.LMM, we first identify the ViT sinks by the definition in Eq. (T. where the token characteristic is measured by its feature norm. We jointly reposition them and their corresponding positional embeddings the beginning of the visual token sequence before passing the sequence to the connector and LLM.
4.2 SAANN-RRO-SSCATERR IMPROVING INFORMATION FLOW OF MODELS
The observation in the analysis (Sec. 3.3 suggests that VIT sinks are useful semantic summaries, but only useful under the right conditions. To fully and dynamically leverage the "power" of sink, we propose a main-from-scratel approach, DIYSink, to improve the visual information flow in LVLM.
DIYSink introduces two key design components: (1) Dual-MLPP Projection Layers, which indepen- dently process sink and non-sink ViT tokens, preventing two representations from being conflated; and (2) Dynamic Token Selection Modules, which leverage imputs as a gating mechanism to help the LVLM decide which set of visual tokens (i.e., VIT sinks, non-sinks, or both) to during inference.
Dual-MLP Projection Layers. As discussed in Sec. 3.1 sink tokens exhibit unique characteristics such high activation and large norms that differ significantly from those of non-sink tokens. As result, it is challenging for a shared MLP connector to effectively project both types of tokens into a common semantic space that aligns with the LLM's expectations. To address the issue, in D/YSink, we introduce a dual-ML.P projector. Each MIP is trained exclusively on either sink or non-sink tokens, enabling it to specialize in projecting its token type into an LLM-compatible embedding space. Formally, let Vsint and V sink be two disjoint sets of ViT sink and non-sink visual tokens, respectively. We define two separate MLP connectors: fsink R D' + R for projecting sink, and D for projecting non-sink tokens.
fnon sink: R D'
During pretraining, each connector optimized independently using only its corresponding token:
min LIM (E (Tayss Ssink Vink), ) Ixxt Tour)), min LIM(E Jnon sink (V Voon.sinh sink), Iea, Tomi)) (2)
fook
Facestia
7

---

Published as a conference paper at ICLR 2026
Table 2: Full benchmark comparison. ReW and CoT denote our reweighting module and Chain-of- Thought prompting, respectively. Green indicates increase over the corresponding baseline. i denotes models are fine-tuned on original LLaVA fine-tuning datasets Liu et al. (023c))
Methods
LLaVA eval
MME
VistVista
AVG
GQA
SQA
TeaVQA
MMMU
AII
Cog.
ComRca
TeatIruns CodeRea
All
ALG
GEO
LOG
Baseline"
48.34
$7.98
57.76
47.32
30.50
1381,10
207.14
52.14
50.00
37.50
34.30
22.42
20.92
24.12
DIVSint (D6T)
49.04
58.14
61,08
46
30.50
1456. 78 476
277.50
85 00
80.00
52.50
25. a:
24.91
23,85
24.32
DIVSink (ReW)
49.13 -a29
57.75
60.24
47.42
31.10
1451.6 -2077
229.61
87.14
50.00
37.50
25.20 +2.00
25.27
23.01
24.32
Bascline
48.55
51.09
65.47
41.15
33.50
1455.22
261.79
99:29
50.00
$7.50
25.90
23.49
19.67
18.92
DIVSint (CoT)
52.17 a0
59.99
68.57
46 22
33.90
1523. 18 mero
280.71
110.71
67.50
50.00
26.20 +0.30
29.54
27.20
18.92
DIVSint (ReW)
a3
59.79
36.30
1682.41 -zz
265.71
115.71
27.00 2..0
Baseline
70.22
62.47
74.12
58.35
40.20
1740.00
293.21
115.71
65.00
52.50
30.40
37.01
38.08
18.92
DIVSi (CoT)
63.62
58.54
1758.45 +0606
127,
47.50
.3.0
DIYSiOA (ReW)
41.05
63.58
76.20
60.72
39,00
1761.43 +21.4
299.28
129.28
70.00
47.50
43.00.2330
41,63
43,10
LLaWA-TB-CLIP-WINN-ee
Baseline
56.55
62.65
58.82
35,50
1781.70
782,50
130.00
55.00
42.50
26.00
25.27
DIVSink (C6T)
56.76
62.57
69.56
58.99
35.90
1797.
80
310.36
127.6 867.86
77.50
57.50
26, 40
24.56
21.76
1351
DIYSint: (ReW)
56.59
62.51
70.00
58.16
35,70
1787.00
332,85
127.85
95.00
$7,50
26.60
22.78
18.83
13.51
where LLM the language modeling loss used for pretraining, With the separately-trained MLPs, in the fine-tuning phase, we use them to convert ViT sinks and non-sinks independently. concatenate all sinks with all non-sink tokens, and follow the standard LLM fine-tunine fine-tuning procedure (Liu et al., (023c).
Dynamic token selection. Given the dual MLP projection, we explore two mechanisms to dynam- ically select which class of tokens should be used in inference (sink, non-sink, or both) based on input complexity and task demands. One mechanism leverages hard selection obtained through Chain-of- Thouhhttt (CoT) routing, while the other utilizes a lightweight soft weighting module trained from small amount of cross-task data.
we empirically observed that sink tokens are effective for scene-level, holistic under- standing, whereas non-sink tokens excel at capturing fine-grained details. Based on this observation, we design a two-step chain-of- of-thuught process to pre-classify the given task: (1) determine whether the image is symbolic (with minimal local detail) or real-world photographic scene. and (2) assess whether the question query requires holistic reasoning or local visual understanding. Following these criteria, if the task involves symbolic/simple image and holistic reasoning, tokens for inference. If the task involves a real-world/complee image and local visual understanding, we use only non-sink tokens. For all other mixed ambiguous cases, we leverage both token types to generate the final answer. We note that this is equivalent to hard [0/1] token selection.
In Sec. 3.3
only ViT sink
In addition to the chain-of of of-thought method, we also explore learnable reweighting mechanism to dynamically balance the contribution of sink and non-sink tokens before feeding them into the LLM, as illustrated in the Reweighting MLP section of Fig,
Given an input text question embedding q E Rd obtained from sending input query into frozen sentence encoder, where d represent the hidden dimension of the encoded sentence feature, the reweighting MLP R outputs two scalar weights: [Waink: Wnon-sink = R(q) E R We then use the output weights to reweight the sink and non-sink tokens and input their concatenation into the LLM:
min LLM UR
Husiik fsink( sinnk)] { [wnonsiin- fnon-sink ( wnon-sink)] H. Iout)
(3)
Ia
During training, only the parameters of R are updated, while all other components remain frozen to prevent additional information leakage and maintain the fairness evaluation.
5 EXPERIMENTS
Baselines and training. We evaluate our method on four LVLM (ViT-LLM) configurations. For the vision backbone employ SigI.IP (Zhai et al., 2023) and CLIP-ViT-L. (Radford al. [022))
8

---

Published as a conference paper at ICLR 2026
while for the language backbone use Qwen2 Yang et al. 2024 Qwen2.5 (Qwen al. 2025)
Phi-2 (Abdin et al.] 2024), and Vicuna (Zheng et al. 2023) spanning various scales (0.5B. 3B, 7B) All models are pre-trained and fine-tuned official LLa VA training sets (Liu et al. 2023c For reweighting MLP, training samples are drawn from the training sets of PoxMo (Deitke et al. 2024) and GeoQA (Chen al., 2021). which are distinct from evaluation benchmarks. We 120 examples for 0.5B and 3B models, 240 examples for the 7B model (see Sec. B.4 for details).
Evaluation Benchmarks. We conduct wide range of evaluations on various benchmarks including general benchmarks used in LLaVA (or LLaVA eval) (Liu et al. 2023c), GQA (Hudson & Manning
2019), ScienceQA (Lu al., 2022), TextVQA (Singh et 2019|) MMMU (Yue al. [024). fine- grained benchmark like MME (Fu et al. (2023) and mathematical reasoning benchmark Math Vista (Lu
ct al. 2024b) to examine the influence of D/YSink on LVLMMs following the common protocols of
these benchmarks. We provide benchmark details in Sec. B.5
Training-free method. We evaluate the proposed approach on several recent LVLMs, including
VL2. (Zhu et al.) (2025), Phi-3.5 (Abdin et al.) 2024), DeepSeek-VL (Lu et al. 2024a). Molmo (Deikee et al.J [2024). Gemma3 Teem et al. (2025) ) and Meteor et al. (2024). As shown in Tab. | our lightweight method consistently improves performance on benchmarks require holistic understanding of the image. particularly Math Vista (e.g., 1.8 for Molmo). We also observe modest yet consistent improvements on benchmark with mixed tasks (i.e., MME). Interestingly, we also observe improvements on Meteor Lee et al. (2024), a VLM built on a Mamba-based language model, extend beyond Transformer- based architectures. We
indicating that the benefits of sink-to-the-fie additionally note that DeepSeek- VL's channel-wise concatenation of SAM and SigLIP features creates a blended features, complicating the token behavior; consequently, the gains are smaller.
Training-frone from-scratch method. We evaluate our proposed modules across multiple benchmarks as shown in Tab. 2 Our results show that with correct projection and reweighting of sink tokens, the model consistently improves on global reasoning tasks such as MME code reasoning and Math Vista's algebraic, geometric, and logical reasoning. For instance, applying DIYSink (CoT) or DIYSink(ReW) to the LLaVA-7B (Liu et al. 2023c) both yield 1--p gain on MME code reasoning. On Math Vista, DLYSink (CoT) delivers notable gains of 6.05 and 7.53 points in algebra and geometric reasoning, respectively, when applied to LAALLLa and gains of 4.62 and 5.02 when applied DIYSink (ReW) to MayyLLLVV-Qeee.5555
In addition to domain -omain-specific reasoning improvements, also observe consistent gains commonly evaluated LLaVA benchmarks. For instance, DIYSink (CoT) and DIYSink (ReW) improve the LLaVA- average by 3.62 and 5.79 points, respectively, over LanyLL These results suggest that our approach effectively mitigates representational interference (see Sec. 3.3 and enhances performance across tasks.
Overall, we observe that both explored strategies, CoT and ReW, are effective. The CoT is simpler, but requires additional computation in inference. ReW on the other hand has negligible compute overhead, but does require additional training. Each strategy may be beneficial depending on cireumstances.
Model Ablation. We conduct an ablation study to evaluate the contribution of the dual- MI.P design, which facilitates the conversion of both ViT sink and non-sink information into a format interpretable by the LLM. As shown in Tab. 3 our learned dual-MLP model, even without the token selection module, out- perform baseline across benchmarks. These improvements support our hypothesis that the shared connector design commonly used in LVLMs struggles to effectively convert both ViT sink and non-sink tokens. Due to page limit, we place additional studies in the appendix, includ- ing justification of the design choice of placing sink tokens to the front rather than the end of the sequence (Sec. C.I.. ablation on different connector types (Sec. C.3), computation cost (Sec. (S.c. the human study on the task taxonomy classifier using GPT(Sec. C.II), an analysis of the learned weights in the selection module (Sec. C.4), comparison on compositional scenes (Sec. C.5). and qualitative examples (Sec. D).
Table 3: Ablation of Dual-MLP projector.
Method
SQA
MMMU
MME ALL
MME Cog.
TayyLLaaAA (Qwen)
Baseline
57.76
30.30
138110
207 14
-Dual-MLP (our)
60.63
1439.21
LayyLLV (Phi)
Baseline
60.47
+ (Lall
70.75
34,80
1679.24
262.14
9

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
Large Vision Language Model. LVLMs (Bai et al. 2025 Team et al. 2023 Achiam et al.
2023
Lu et al. 2024a) typically consist of three key components: (i) a Vision Transformer (ViT), (ii)
connector module, and a (iii large language model (LLM), Among open-source LVLMs, widely adopted vision backbones include the CLIP-ViT family (Liu et al. 2023b Abdin et al. 2024).
SigLIP (Chen et al. (223c)). and Intern ViT (Zhu et al. (025). each selected for its strong pretraming and transfer capabilities. To bridge vision and language, most models utilize either a lightweight multi-layer perceptron (MIP) (Chen et al. 2024; Zhou et al. 2024; Deitke et al., 2024) or cross- attention mechanisms (Zhu et al. 2023 Li et al. 2023). In this work, we focus on the visual encoding component and evaluate our proposed method, both in training- free and trining-basee settings, across a range of LVLMs that span all three major VIT families and include seven distinct LLMs.
Attention sinks. Attention sinks have been observed in well-trained LLMs and ViTs (Sun et al. 2024 Darcet et al. 2024), where certain tokens from low semantic meaning regions, such as image background or the language punctuation, are allocated with high attention weights. Early work in
LLMss identified the heginning-offssquenes (BOS) token as a consistent sink (Cancedda 2024 Gu
2024), later extended to show that sink tokens emerge throughout the input (Yu al. 2024). While these tokens can help with long-context modeling and quantization (Zhang et al. 2024 Gc
al.
al.
2023 Xiao et al. 2023a), not all sink are beneficial, some reduce performance (Yu et 2024).
Visual attention also exhibits sinks (Sun et al. 2024) in models such as CLIP, and DINOv2 (Oquab)
2023). Visual sinks, however, are broadly considered not very useful, with majority of works focusing on their mitigation. Kang et al. (2025) identified visual attention sinks and redistribute their
al.
attention to semantically meaningful regions. Huang et al. (2024); Woo al. (2024) propose to
suppress the sink token to improve hallucination. Darcet et al. (2024) add registers in the beginning of the token sequence to absorb the information stored in sinks in ViT and result clearer map for
object discovery. In contrast, to the best of our knowledge, we are the first to empirically study and demonstrate that LVLMs, visual attention sinks, propagated from VIT to LLM, play a crucial role.
7 CONCLUSION
Contrary to prior works, we show that ViT sinks in VLMs encode vital high-level visual context. Based on this insight, introduce two effective methods to leverage them: training ffree repositioning strategy and a training-baas -bsed framework, D/YSink, both validated across diverse benchmarks.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide details of our analysis, methodology. and experiments. All experiments conducted publicly available benchmarks, as detailed in Sec. and Sec. B.5 Our proposed framework, model configurations, and training details, including
hyperparameters, are described Sec. B.4 Furthermore, the specifics of our analysis are documented in the appendix, such as the definitions and thresholds used for identifying VIT sink tokens in Sec. A and Sec. B.I and the prompts used for our GPT task clustering in Sec. [B.2] Last but not least, as stated in Sec. we will release all code and analysis scripts upon acceptance.
ACKNOWLEDGEMENTS
This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chairs, NSERC Canada Research Chair (CRC), AMI-TNN UBC, and NSERC Discovery and Discovery Accelerator Supplement Grants. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govemment of Canada through CIFAR, the Digital Research Alliance of Canada" companies sponsoring the Vector Institute, and Advanced Research Computing at the University of British Columbia. Additional hardware support provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition award.
billiane..cnncc
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404. 2202404..42199 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08775 2023.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 1425-24333 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308. 01300, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 1(2). 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2502 2025.
Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, and Razvan Pascanu. Why do llms attend to the first token? arXiv preprint arXiv:2504.02732, 2025.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. arXiv preprint arXiv:2402.09221, 2024.
Haozhe Chen, Junfeng Yang, Carl Vondrick, and Chengzhi Mao. Interpreting and controlling vision foundation models via text explanations. arXiv preprint arXiv:23 10. 10591, 2023a.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint urXiv:2105: 14517, 2021.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra. Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint 1000943, 2023b.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09) 199. 2023c.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 24Xi::2412..5271,, 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International Conference on Learning Representations (ICLR), 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for sate-of-theeart multimodal models. arXiv preprint arXiv:2409.177 17146 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.

---

Published as a conference paper at ICLR 2026
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 60066690499999 6914, 2017.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprini arXiv:2410. 10781, 2024.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and putropecttionaallcatiee In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
Drew A Hudson and Christopher D) Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. International conference on machine learning, pp. 0651-44664 PMLR, 2021.
Nick Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision transformers don't need trained registers. arXiv preprint arXiv:2506.080/1 2025.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Scong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv preprint Sxkiv:2503.03321,, 2025.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv: 1908.08593 2019.
Byung-Kwan Lee, Chae Won Kim, Beomchan Park, and Yong Man Ro. Meteor: Mamba-based traversal of rationale for large language and vision models. Advances in Neural Information Processing Systems, 37:40278-400 40315, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358. 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05433 2023c.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseck-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.0533
2024a.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
12

---

Published as a conference paper at ICLR 2026
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Sai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024b.
Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157 1947.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025, URL https: http://axvvvvv/om owg/ abs 2
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Disnover- Task-agnostic concept bottlenecks via automated concept discovery. In European Conference Computer Vision, pp. 444 461. Springer, 2024.
Sebastian Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arxiv 2018 arXiv preprint 1rXiv:1811.12880 1811.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. Advances in neural information processing systems, 32, 2019.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019a.
Amanpreet Singh, Vivek Natarjan, Meet Shah, Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pp. 8317-8326, 2019b.
the
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17766 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe
13

---

Published as a conference paper at ICLR 2026
Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dec Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak- Pluciíska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq lqbal, Shashir Reddy. Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu. Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci Vedant Misra, Vincent Roseberry, Vlad Feinberg. Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei. Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL httph:ttp//omm...... /ttppps//arxivv..
-Bptiite
abs 25.3. 19786
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M Kakade, Hao Peng, and Heng Ji. Eliminating position bias of language models: A mechanistic approach. arXiv preprint arXiv:2407.01100, 2024.
Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint 1xXiv:2405.17820 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznee. Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv: 22222309.77747 17453, 2023b.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li. Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin. Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang. Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu. Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://aras www..orgggssooooo 2407 10671
14

---

Published as a conference paper at ICLR 2026
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406. 15765, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024. URL htt.:tttss///eem /ati:///aivvvvv wwtprrxwwwwwrrb////eeee 112111..66222 16502.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings the IEEE/CVF international conference computer vision, pp. 11975-11986, 2023.
Pan Zhang, Xiaoyi Dong Bin Wang. Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, al. Internln-xcomposerr A vision- language large model for advanced text-image comprehension and composition. arXiv preprint
arXiv:2309. 15112, 2023.
Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. Q-hitter: A better token oracle for efficient Ilm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6:381-394, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL/ttts://axxxvvooo abs/2306. 05685
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprini 240.:22002112899 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
15

---

Published as a conference paper at ICLR 2026
ABSTRACT
This document provides supplementary materials for main submission. Section A presents additional analysis results on various LVLMs, including TinyLLa VA-0LLL & 3B, IntemVL-2.5-4B, and LLaVA-7B. Also, we provide additional studies about the impact of VIT sinks in LLM, semantics of ViT sinks, and the emergence of ViT sink tokens during the training this section. Sec- tion describes the implementation details of our analysis, model designs, and benchmarks. Section offers further quantitative analysis, including an ablation study of the Dual-MLP design, as well as inference weights from the reweighting module and the Mfaain-of-Thoueee (CoT) approach. Furthermore, we discuss the training and inference computation cost of our proposed frameworks. Section D provides qualitative results, including reasoning processes, model outputs, and word mapping results. Section discusses the model's limitations and potential social impact.
APPENDIX CONTENTS
VIT Sinks in LVLMS
A.I
Propagation of VIT Sink Tokens into LLM and Hidden Dimension Breakdown
A.2 Impact of ViT sink tokens in LLM
A.3
Semantics of ViT sink tokens
The evolution of propagated VIT sink tokens during LVLM training.
B
Implementation Details
B.
Analysis Details: VIT Sinks LLM
B.2
Analysis Details: Clustering
B.3
Sink Token Position Embedding Details
B 4
Framework Details
B.5
Benchmarks Details
Extra Quantitative Results
Sink Position Ablation
Dual-MIPP Ablation
8
Connector Type Ablation
Reweighting MLP Inference Weights
Performance analysis on dense and compositional scenes
10
C.6
Analysis of the training and inference computational cost
11
Analysis of the sink/non-sing token classification accuracy for DIY Sink
11
C.8
Analysis of Using Chain-of-Though Thought Selection Mechanism as the Soft Selection
12
Parameter Overhead of the DLP Module
12
C.10
Efficiency Analysis of ViT Sink Tokens
13
C.I1
Human Study of Task Taxonomy using GPT
13
Evaluation on Hallucination- Benchmark
14
C.13 Statistical Analysis Model Performance Significancy
14

---

Published as a conference paper at ICLR 2026
D Extra Qualitative Examples
15
D.I DIYSink on LLaVA-7B
15
D.2 Word Mapping Results
15
E Model Limitations and Social Impact
15
Model Limitations and Future Directions
15
E.2 Social Impact
20
LLM usage
20
A VIT SINKS IN LVLMS
A.1 PROPAGATION OF VIT SINK TOKENS INTO LLM AND HIDDEN DIMENSION BREAKDOWN
TinyLLaVA 0.58 (Siglip Qwen-2)
TinyLLaVA 38 (Siglip Phi-2)
InternVL (InternVITe Qwen22.5
LLaVA 78 (CLIP Vicuna)
Hakee Denenion
<abic Dinession
H-SO Diveationn
THE MALI HUMONNN
120
Nf Toben
126
302
Figure Al: Extra analysis results of the sink property. We provide a dimension breakdown of LLM-emerged sinks (first row) and propagated ViT sinks (second row). Additionally, we examine the relationship between VIT token norms, calculated within the ViT, and the attention weights they receive from the LLM during output generation (last row). Results are averaged across 300 image-question pairs.
In Sec. 3 we analyze the properties of ViT sink tokens in large vision-language models (LVLMs), focusing initially on results from LLaVA-7B. To provide a comprehensive comparison, we extend our analysis diverse of LVLMs incorporating various ViT backbones (e.g., CLIP, SigLIP, Intern ViT) and language models (e.g., Qwen-2, Qwen-2.5, Phi-2, Vicuna). Specifically, we exam- ine TinyLLaVA-0.5B. LmyyLLVAA-3B IntemVL--2.544B, and LLaVA-7B, with results presented in Fig. A1
In Sec. we observe a positive correlation between the norm of ViT tokens and the attention weights assigned by the LLM during output generation. We further investigate the generalizability of this observation. As shown in the last row of Fig. AT across all evaluated models, we find that as the norm of ViT tokens increases, the corresponding attention from the LLM also increases during generation. This consistent trend reinforces our earlier observation a positive correlation between ViT token norm and LLM attention. Additionally, by comparing the dimensional breakdown of LLM-emerged sinks (first row) and propagated ViT sinks (second row), we observe that the sets of highly activated hidden dimensions differ between the two sink types, which again support the observation mentioned
2

---

Published as a conference paper at ICLR 2026
in Sec. Note that the InternVL model exhibits notably weaker nigh-activation signals compared to other models. We attribute this to its use of visual features extracted from the final layer of the ViT, whereas the other models rely on features from the second last layer. Our findings suggest that the final ViT layer exhibits weaker sink activation signals relative to the second last layer. Note that all results averaged across 300 image-question pairs.
A.2 IMPACT OF VIT SINK TOKENS IN LLM
We further investigate the importance of ViT proppaagtt sinks by analyzing the average attention weights received by ViT sink tokens, as the results shown in Tab, AT] Specifically, we compute the average attention from output tokens to target tokens across 1000 image-question pairs. Our analysis reveals that, on average. VIT sink tokens receive attention weights that are comparable to or higher than those received by LLM-emerged sink tokens. Moreover, the attention weights assigned ViT sink tokens are significantly higher, ranging from 3 to 10 times greater, than those of non-sink tokens, highlighting the important role of ViT sinks in guiding LLM output.
TinyLLaVA 0,5B
TinyLLaVA 3B
InteraVL 2.5 4B
LLaVA 7B
Propagated VT Sink Atin
0.0072
0,0329
0.0117 0.0054
0,0113
LLM emerged Sink Alin
0.0181
0.0367
0.0127
Non-sink Atta
0.0013
0.0011
0.0032
0.0015
Table A1: Comparison of sink-related metrics across different models.
A.3 SEMANTICS OF VIT SINK TOKENS
In Sec. 3.2, we use relevance map and word distribution to uncover the information captured in the VIT Sink. To further investigate semantics of ViT sink token, we conduct lincar- seoear-poobing analyses at the object level (object classification) and scene level (lighting, composition, and counting).
Table A2: Tasks definition, objectives, and data labeling strategy on the COCO dataset.
Task
Objective
Data & Labeling (on COCO dataset)
Obj class
Tost object closs-scasitivity
Seleet images with only instance from 10 fre- queat categories and label them with the object class.
Lighting
Measure awareness of overall scone brightness
Randomly select 1000 images; each augmented at brightness scales (labels 0-3).
Composition
Determine whether sink tokens encode spatial quadrant
Select images with exactly one object from the top 5 most frequent categories;
quadrant (IL/TR/LL/BR.
Counting
Evaluate soene-level detail via object counting
Seleet images with 1-4 instances from the top most frequent categories; lahel object (1-4).
Analysis setup For each task, we pair COCO images with ground-truth labels (e.g., object category, lighting-level), Given an input image, we extract its sink feature by averaging all sink-token embed- dings from the vision encoder. For comparison, we randomly sample 5 non-sink tokens and average them to form a non-sink feature. We then train a simple linear classifier on these features: high validation accuracy indicates that the feature encodes the target information. We test four different tasks, object classification, lighting, composition, and counting, using the task definitions described in Tab. A2 Note that the number of data for each task: 4000 (training) and 1000 (testing).
Discussion As the results shown in Tab. A3 sink tokens from both CLIP and SigLIP outperform non-sink tokens by 20 30% points on object classification and lighting tasks. They also lead in composition and counting, though the gains which show that sink tokens capture mostly global cues (e.g., object saliency, illumination) with minor local details (e.g., object position, count).
smaller, consistent with the findings in Sec. 3.2
3

---

Published as a conference paper at ICLR 2026
Table A3: Comparison of CLIP and SigLIP across various tasks.
Obj class
Lighting
Composition
Counting
CLIP
SigLIP
CLIP
SigLIP
CLIP
SigLIP
CLIP
SigLIP
Random
0.100
0.100
0.250
0.250
0.250
0.250
0.250
0.250
Non sink
0.512
0.647
0.361
0.515
0.295
0.280
0.447
0.495
0.865
0.722
0.310
0.348
0.555
A.4 THE EVOLUTION OF PROPAGATED VIT SINK TOKENS DURING LVLM TRAINING.
Throughout our training process, the vision encoder (ViT) is kept frozen during both the pretraining and fine-tuning stages. Consequently, the original VIT sink tokens remain static. This raises key question: how the corresponding propagated sink tokens, which are processed by the projector and language model, evolve at different training checkpoints?
To investigate this, we analyzed the hidden states of the propagated sink tokens across various training iterations. For given image, tracked the four hidden dimensions with the highest activation values. The results for our DIYSink 3B model are presented in Tab. A4 The results show that, during the projector pretraining stage, across training iterations, high values. However, in the early stages of fine-tuning, specific hidden dimensions emergeand attain much higher values than the others (e.g., 70.14 in dim 490 in the finetuning of 0.5B model). Thus, appears that ViT sink tokens begin to propagate during the early iterations of fine-tuming neeettnnigg and the set of highly activated dimensions then remains largely consistent throughout the rest of training.
single dimension exhibits disproportionately
Table A4: The behavior of the propogated VIT sink tokens across training progress of DIY Sink- Qwen2-3B.
Pretraining
Fine-tuning
Training Heration 200
800
1400
2000
400
1200
2400
3600
(2311. 9.2)
(1773. 4.2)
(599,5.2)
(1097, 7.7)
(743, 44.0)
(743. 34.3)
(743, 34.2)
(743, 36.2)
(393, 8.6)
(169, 3.8)
(1662, 5.2)
(1195,
(293, 18.2)
(293, 17.3)
(293, 14.1)
(293, 25.3)
(Dim kds: value)
(1254, 8.4)
(1787,5.0) (179
(1625.5.2)
(1948,
10.6)
(573.
(1796, 8.1)
(902, 3.6)
(2521, 4.9)
(599, 4.9)
(1546, 9.0)
(1546, 11.8)
(567, 8.0)
(1948, 9.0)
B IMPLEMENTATION DETAILS
B. ANALYSIS DETAILS: VIT SINKS IN LLM
In Sec. 3.1 we mentioned that we use Eq. I to identify sink tokens. As described in the preliminaries, we use feature norms identify ViT sinks and sink dimension values identify all sinks within the LLM. The values used in the equation for each type of sink are provided in Tab. A6
Justification of For CLIP-based models, we observed drop of token norms from 3+ digit number (e.g., 177.7) to a two-digit norm value (c.g., 47.5). To further verify that the gap between sink and non-sink tokens is large, we train our DIYSink with CLIP+Qwen2-0.5 and SigLIP+Qwen2-0.5BB using different thresholds, including 80, 90, 100, 110, 120. Across all thresholds, the performance variation on MME for both model are small (below 0.2% change). This indicate that the model is unsensitive to the change of threshold, the gap is indeed large and we can probably split sink/non-sink tokens by simply setting a threshold. Same approach applied when finding for SigLIP and other vision backbones.
Additional discussion of the adaptive, data-driven method for sink token selection To further address the potential concern of manual tuning and enhance generalization across various archi- tectures, we have studied an adaptive, data-driven method for sink token selection. Based on the characteristies of sink tokens, know that if the visual tokens are sorted by their norm, significant relative drop in norm will be observed between the last sink token and the first non-sink token. Thus, identify sink tokens by detecting the last significant relative drop (e.g., greater than 10%)
4

---

Published as a conference paper at ICLR 2026
among the top-ranked tokens, allowing the model to adaptively determine the number of sink tokens for each model and dataset. We evaluated this adaptive method across four Large Vision- Jision-Language Model (LVLM) architectures, including
soaniingfrom-scrattee models like TinyLlava-0.5B (Qwen) and Tinyl T'iyLLlave .lava-3B (Phi), and training- -ree- models like Intern V1.2.5-4B and Phi3.5V. The results on the MME benchmark is provided in Tab. [S This table shows that the adaptive method achieves per- formance comparable to the exed-threshold baseline, thereby proving its effectiveness by automating token selection without relying on hard threshold.
Table A5: Comparison of the adaptive method for sink token selection on MME.
Methods
ALL
Perception
Cognition
ComRea
NumCal
TextTrans
CodeRea
Training-from: scratch
Tinyllava 0.5B (Qwen2)
1381,10
1173.96
207.14
82.14
37.50
50.00
37.50
DIYSink (CoT) hard
1456.78
1179.28
277.50
85.00
60.00
80.00
52.50
DIYSink (CoT) adaptive
1455.77
1183.27
272.50
85.00
55.00
80.00
$2.50
Tinyllava 3B (Phi)
1455.22
1193.43
261.79
99.29
55.00
50.00
$7.50
DIYSink (CoT) hard
1823.18
1242.47
280.71
110.71
52.50
67.
50.00
DIYSink (CoT) adaptive
1526.01
1236.01
290.00
115.00
55.00
67.50
52.50
Training-free
Intern VL.2554B
2332.16
1686.81
645.35
142.86
124.99
200.00
177.50
Sink-to-the- (hard)
2351.33
1706.33
645.00
145.00
125.00
200.00
175.00
Sink-to-the-Fii (adaptive)
2357.18
1712.18
147.23
125.00
200.00
172.50
Phi3.5V
1887.94
1475.76
412.14
137.14
90.00
87.50
97.50
Sink-to-the the-front (hard)
1891.27
1457.35
433.92
136.42
97.50
87.50
112.50
monk-to-the-frond (adaptive)
1839.28
1453.17
436.11
136.42
95.00
87.50
112.50
B.2 ANALYSIS DETAILS: CLUSTERING
In Sec. 3.3 we introduce a task taxonomy based on query and image properties and investigate the information encoded in propagated ViT sinks and how the LLM interprets it. Specifically, we leverage GPT-4o to classify image-question pairs according to (1) image complexity. which measures the visual density and richness of the scene, and (2) query globalness, which assesses whether the question requires high-level contextual reasoning in Fig. A3 A3 the detailed prompts used for GaT-based clustering to support reproducibility. We also conduct the human study for verifying the robustness of leveraging GPT as the classifier in the task in Sec.
tne-grained spatial understanding. We provide
B.3 SINK TOKEN POSITION EMBEDDING DETAILS
To preserve the spatial layout of the image in both the training free tink-to-the-front and train-from scratch settings, we apply slightly different adjustments to the position embeddings of tokens. Below, we detail how sink and non-sink tokens are arranged and how their position IDss are assigned. We also include figure help illustrate, please refer to Fig. A2
Training-free: as noted in Sec. 4. the main paper, we preserve the spatial structure by moving both the sink tokens and their original positional embeddings together. This ensures that the spatial layout encoded by the position embeddings remains intact, even though the tokens are reordered in the sequence.
Train-from-see from--rratchh the model is retrained with sink tokens copied the front, while their original positions are filled with constant vectors. This preserves the overall spatial grid and allows the model to learn to interpret the modified sequence without losing spatial consistency.
Importantly, sink tokens typically correspond to only 3-5 background tokens out much larger set of visual tokens (often 576 more). Because they constitute such small fraction of the sequence, repositioning them has minimal impact on the effective spatial layout of the image. This also helps explain why models remain stable despite the reordering.
5

---

Published as a conference paper at ICLR 2026
Training-free
Train-from-scratch
iositionids
Position ids
Original Feature
Original Feature
Position ids
Position ids
S
4
7 8
9
Original Feature
Original Feature
Sink token
Sink token
constant vector
Figure A2: Sink tokens reordering and their respective position ids for training-free free and train-from-
scratch settings
Tiny LLVVA 0.5B
TinyLLaVA
Intern 2.5
LLaVA
(VIT Sink)
120
120
35
100
(Progugated Sink)
(LLM-emerged Sink)
5
25
20
Table A6: Implementation details of sink characteristic function.
B.4 FRAMEWORK DETAILS
The Dual-MLP connector is first separately pretrained and subsequently fine-tuned jointly with the LLM, using the same hyperparameter settings and training examples those employed in the LLaVA framework Liu et al. 2023a).
For the Reweighting MLP module, training samples drawn from the training sets of PIXMO Deitke et al. 2024) and GeoQA Chen et al. (2021), I,, which are distinct from our evaluation benchmarks ks. We stratified sampling strategy based on model scale: 120 examples for 0.5B and 3B models, 240 examples for the 7B model. In each case, the dataset is balanced across three task types: Local tasks (color recognition and counting), Global tasks (geographic reasoning), Mixed tasks (chart and table understanding), with each category comprising one-third of the total samples. We fine-tune only this component while keeping all other modules frozen, in order to prevent additional information leakage and ensure a fair evaluation. The Reweighting MIP is trained for 10 epochs the respective data, and optimized using learning rate le-2 and global batch size of 20. All the training are done using 4 Nvidia A40.
B.5 BENCHMARKS DETAILS
We evaluate method wide range of benchmarks including general benchmarks used in LLaVA (or LLaVA eval) Liu et al. (2023c), GQA Hudson & Manning (2019), ScienceQA [u et al. (2022). TextVQA Singh al. (2019a), MMMU Yue (2024), une-grained benchmark like MME Fu et al. (2023) and mathematical reasoning benchmark Math Vista Lu et al. (024b) to examine the influence of DIYSink on LVLMs following the common protocols of these benchmarks. The GQA dataset large-scale, balanced visual question answering benchmark designed to test and diagnose visual reasoning and scene understanding through compositional, semantically structured questions over real-world images. We evaluated on the test set which contains 12,578 examples. The ScienceQA dataset is a large-scale, multimodal benchmark comprising 21,20 21,208 multimodal multiple-choice science questions, with majority featuring images and text contexts. Each data point is annotated with detailed lectures and explanations to support chain-of chain-of-thought reasoning across diverse subjects like natural, social, and language sciences. We evaluated the test set which contains 4,241 examples. The TextVQ2. dataset requires model to understand the text in the image and reason about it to answer questions. We evaluated the testset which contains 5,00 5,000 examples. The MMMU (Massive

---

Published as a conference paper at ICLR 2026
Image Complesity (0-4)
Glabalness
Mininnll
sujject.
Fiee Level/ Ulto Local.
No baduground ontremaly sample ore (fat cotari.
Focus specchic
the image.
No tadure, depth.
Tasls involv row image data tiny, precke decall (hample: Whar the RGB waue at (100, 1930" this plant part of edge?"
[Fammple a smplerioon or the dlowwing!
Simple:
1- Micro Regiony Fine Fexture
Ocer maan sulject with min suroundingg. Bant bassgroune lite tedture depth. (Bomple: singte apple ona
knokees smrll socures, potter rs, or eccall on port of object. -dil very localeed,
emarpeetall than powels tenure fabvicc")
namateee
(Borppte:
Symbol
busk Light hadiground eimment sed contre but distrct.. Very lnited texture spetial loyering.
Subjectól
Involes denuiiiingg procerties
[toumpler "whe number the door?" "What onis the stire" wearing glasies??
[txample: A persen standing frost wall)
1- local insecctioon / Niaarhy lee bebelonsnipe
3- Moderate Complex by:
-onncern ra rdatioralps actirns bettoren a fro nearby clements. Siinntly broader concet nooied, oftan withir small sub region. [Eaampte: "wchut the penas nolding",
Subjects intractivitucllyy mu (eag ltinn reaching wolking
Bedigm mesringttd denals secondary to man treus.
dog atting the counth")
Mederrte depth sectures
[tammper Asmell scene person drining coffee.]
-
Main Subjest Central Scame Undenstardig: image.
snjert
reltionatios foraground
"What
strong testures, depth, and vius brywiing
[bammtee busting mar kerplase with
hareers
clobed Insepratation / Narnethe:
Requires understanding of endire image: endironmnnt, event moodi mearing.
5- Highly Complex/ Chastic
Ouerapping cutjects, muliqua intercttion happening ae oone
tnoolnes by thais acress diemera interpret the shene stony. [Bxampl: "Whut & happoringin scone7. "Whut & the mood or theme of photo?)
Hoavs toture, reflections, motion blat, dynumis ligtinng Siual regares witnnt fully parce crene (bampter bun street dusing fectual right)
Figure A3: Details GPT-4o prompt for clustering experiments.
Table A7: Additional comparison of Qwen2.5. ReW and CoT denote our reweighting module and Chain-of-Thoon Thought prompting, respectively. Green indicates increase over the corresponding baseline. i denotes models are fine-tuned on original LLaVA fine-tuning datasets.
LLaVA evall
MME
Muth Vista
SQA
MMMU
Cog.
Comfeea
TextTrass
CodeRea
ALG
a
1ig. JP-Qwen2.5..5.
Baseline
58.62
315.00
DIVSiot (ell-CoT)
50,02
58.54
60.99
48.64
32.50
1522.45
241.07
78.57
50,00
55.00
26.00
25/42
24.27
16.22
DVYSint (ReW)
50,3
58.45
60004
48.88
34,00
1526.69
240.71
80.71
50,00
$2.90
26.00 4230
29.54
28,03
16.22
ThyyLLaVA-3.BBSg....P
Beseiin!
70.22
62.47
74:12
58.35
41120
1740.01
203.21
115.71
65.00
52.50
30.40
37.01
38.08
18.92
DIVSint (C6T)
70.98
75.41
39.60
1758.4
41846
.02.64
127.14
47.50
33,40.3u
41.64
42.68
1761,44 .2140
Multi-discipline Multimodal Understanding and Reasoning) benchmark is a comprehensive dataset comprising 11.5K college-level multimodal questions across six disciplines and 30 subjects, designed to evaluate models' expert-level visual perceptual abilities and deliberate reasoning with subject- specific knowledge. We evaluated the testset which contains 900 examples. The MME dataset measures the perception and cognition abilities of LVLMs. Specifically, assesses the models' capability in recognizing the specific object, such its existence, count, position and color, and then compositing the perception information and the Enowledge in LLM to deduce more complex answers. Math Vista comprehensive benchmark designed to evaluate the mathematical reasoning abilities of foundation models within visual contexts. It comprises 6,141 examples drawn from 28 existing multimodal datasets and three newly created oQes-IQTest, FunctionQA, and PuperQA-requiring deep visual understanding and compositional reasoning skills. We evaluated on testmini which contains 1000 examples.
C EXTRA QUANTITATIVE RESULTS
C. SINK POSITION ABLATION
The primary motivation for repositioning sink tokens is to amplify their impact during output generation, which can achieve by leveraging two positional biases in LLMs:
7

---

Published as a conference paper at ICLR 2026
Chain of Thought
Reweighting MLP
Stage 1: the image
hot that the dog 1s weoring?
Symbolte Imag
(A)
(8) Pho Photogr aphic inage?
Sentence Encoder
Stage 2: Is the question
Sink
MLP
Reweighting MLP
(A) Holistic object reasoning task
(B) Local sual understandiing
VIT
LLM
Non- Sink MLP
Decision
Global
Local
Mixed
Figure A4: Illustration of our Chain of Thought inference and Reweighting MLP module.
Causal Attention Bias: In autoregressive Transformers, each token attends only its predecessors, amplifying the influence of early tokens because information flows forward but not backward. By moving sink tokens, together with their original positional embeddings, to the front the sequence. we exploit this bias.
RoPE Recency Bias: Rotary Position Embeddings introduce mild "recency" effect Wang al. (2024): attention weights decay as the relative distance between query and key increases, causing the model to favor tokens nearer the end of the sequence we reposition tokens and reassign their positional embeddings, we can leverage this effect.
Although both biases shape attention, our empirical study, Tab. A8 and Tab. A9 shows that the causal attention bias dominates the RoPE recency bias, enabling front-positioned tokens to exert greater influence. Based on these findings, placing sink tokens at the front ensures they receive stronger, earlier attention, resulting in consistently better performance than appending them at the end.
Table A8: Sink token position ablation with training free models.
Methods
GQA
SQA-image
TosVQQ
MMMU
MME
MathVista
IntermV1.2.5.42
62.1
96.2
73.5
50.2
2332.2
62.9
sink-to-the-end
62.2
96.2
73.3
50.3
2331.5
60.2
So sink to front
62.4
96.1
73.2
50.4
2351.3
63.3
91.3
43.2
1887.9
43.1
sink-to-the-end
63.5
91.0
64,2
43.8
1875.9
43.3
sink-to-the-fron
63.5
91.3
64.1
44.0
1891.3
43.5
Table A9: Sink token position ablation with train-from-seratch models.
Methods
GQA
SQA SQA-image
TexiVQA
MMMU
MMB
MuthVista
Tinyllava 0.5B (Qwen)
$8.0
57.8
47.3
30.3
1381.1
24.3
sink-to-the-end
56.9
Dual MLP with sink-to-the-froot
58.3
60.6
48.4
30.8
1439.2
24.8
Tiylllaaa 3B (Phi)
51.1
68.5
41.2
33.5
1455.2
25.9
Dual MLP with sink the end
49.8
65.9
37.6
34.3
1318.0
26.8
> Dual P with taa--to-the-faee
60.0
70.8
47.7
34.8
1679.2
28.4
C.2 DUAL-MLP ABLATION
From the results Fig. B, we observe a performance gap between the upper-bound achieved using designated tokens for global and local tasks and the baseline model's actual performance. This suggests that the LLM struggles to leverage sink and non-sink tokens simultaneously without interference. To address this, first propose using Dual-MLP Projection Layers to independently process sink and non-sink VIT tokens, allowing for appropriate projections that account for their distinct distributions.
8

---

model archi- Published as a conference paper at ICLR 2026
Table A10: Performance of the Dual-MLP model (default as sink-to-the- (hoott and the sink-only and non-sink only variant, where we only input the respective type of tokens into LLM during inference.
LLaVA eval
MME
MathVista
Methods
-M
SQA-img img
GBO
LLaVA 7B
35.80
68.91
133.33
125.00
42.50
20.92
13.51
25.27
Dual MLP
Ours (defaullt sink-to-the-frome
36.90
69.61
136.67
130.00
55.00
23.43
13.51
26.69
Ours (sink)
33.40
63.29
63.33
130.00
72.50
28.87
21.62
30.96
(non-sink)
21.76
Table A11: Comparison of L.L.Ms with different types connectors. DIYSink improves TinyLLaVA 0.5B (TL) across both Q-Former and Resampler connector variants.
LLaYA eval
MME
MathVista
Method
GQA
SQAsimage
TexVQQ
MMMIl-sl
Cognition
CanRes
TI.- Q-former
47.75
55.78
33.53
31,00
1160.33
221.43
71,43
47.50
50,00
23.30
24.56
23.01
13.11
IL. Q-fermer (COT)
48.01
56.62
33.93
30,00
1173.84
257.86
77.86
75.00
47.50
23,80
25.98
25.10
13.51
TL Resampler
33.65
61.18
45.30
32.00
1400.31
234.29
84.29
50.00
50.00
23.60
21.00
20.08
24.32
TI.- Resampler (COT)
53.75
60.93
45.12
32,00
1427.11
237.50
85.00
50,00
50.00
24.00
22.42
22.18
21
With the connector fixed, we present an additional experiment to further examine whether the performance gap among the three variants, monk-to-the-from, sink-only, and non-sink persist. Similar to Sec. Sec. we modify Dual-MLP model to create the three variants of model by inputting only respective tokens into the LLM during inference. As shown Sec. C.2 the default dual-mip sink-to-the-front consistently outperform the baseline across different tasks, showing the effectiveness of Dual-MLP module. In the mean time, we observe sink and non-sink only variant excels on different subsets of tasks, which indicate that LLM still struggles to jointly leverage sink and non-sink without interfacing each other. Following this, we design the Dynamic Token Selection Modules, as illustrated in Sec. (4.2 and present in Fig. AA4to dynamic reweight the tokens input into LLM in order to better approximate the performance upper bound of the sink and non-sink variants.
C.3 CONNECTOR TYPE ABLATION
To further investigate the impact of connector architectures, studied two cross cross- antss-tttention- based connectors: the Q-former and the Perceiver Resampler Jaegle et al. (202T). The results are provided in Tab. ATT We implemented these connectors within the TinyLLa LawilLLLVA--e-e---00 tecture, replacing the standard MLP connector, and trained the models from scratch to establish baselines. In both architectures, the connector takes 728 visual tokens as input and outputs com- pressed representation of 128 tokens via cross- ross-attention. We then applied our DIYSink method. For a fair comparison with the baseline, we maintained the total output token count 128 by setting the sink token connector (query/latent) size to 2 and the non-sink token size to 126. Following dual-connector training and chain-of-thought inference,
results demonstrate that both model variants with Q-former and the Perceiver Resampler achieve consistent improvements the MME and Math Vista benchmarks. Overall scores across general benchmarks also increased for both models. This indicates that use ViT sink tokens correctly is crucial for enhancing the reasoning capabilities of Large Vision- Language Models (LVLMs) with either cross-attention based connectors or MLP connector.
C.4 REWEIGHTING MLP INFERENCE WEIGHTS
We provided the inference weight for each benchmark we evaluated in the model to further validate finding that sink and non-sink tokens are beneficial to different tasks and reweighting them can help model inference.
As from Fig. A5] the model learned to assign weights global tasks for tasks like geometry reasoning, algebraic reasoning, arithmetic reasoning, code reasoning and commonsense reasoning, which require abstract context-level information and reasoning. For tasks like MME-

---

Published as a conference paper at ICLR 2026
ReW-MLP Learned Weight for LLaVA 78
ReW-MLP Learned Weight for TinyLLaVA 0.58
MathVista
MothVista-ALG
2,44
0.38
1.04
MarhVista-ARI
0.71
MMI-CodeRlee
MME Comflea
125
1.21
1.07
MMM--anddarr
MME-Artwork
9:76
3:26
9:88
PMMe-Posters
0.33
Figure A5: Output weights from the reweighting mlp layer. For each plot, the number the left the weight assigned to sink tokens and the number on the right are the weights assigned to non-sink tokens
Estimated CoT weights for VA
Estimated CoT weights for TinyLLavA 0.58
MaDhVista-GEO
1.81
0..0
1.78
Math
1.69
0.31
3.74
0.26
MathVista ARI
0.55
1.45
0.59
1.41
MME CodeRea
192
0.88
2,00
0,00
MME ComRes
0.67
1.33
0.56
1.M
MME--cone
0.42
3.58
0.24
1.76
MME-Landmark
6.12
0,05
1.66
MME-Artwark
1.31
0.69
MME Posters
0.39
161
Figure A6: Estimated CoT weights. For each plot, the number on the left are the weight assigned to sink tokens and the number on the right are the weights assigned non-sink tokens.
Artwork and MME-Posters, which require more local information to detect the artwork or understand the poster, the model learned assign information. Additionally, we analyze the selection behavior of the CoT approach. To enable quantitative analysis, we define global weights and local weights represent the average CoT selection results. Specifically, in the sink-only case, the global weight is set to 2 and the local weight 0: in the non-sink-only case, the global weight 0 and the local weight is 2; and when both are used, each receives weight of 1. This formulation allows us to compute the average global and local weights for each task. The results are presented in Fig. A6 Notably, the overall trends closely align with the reweighting MI.P results. For reasoning-intensive tasks such as logical and geometric reasoning, the model predominantly leverages ViT sink tokens.
weight non-sink tokens that contains detailed local
C.5 PERFORMANCE ANALYSIS ON DENSE AND COMPOSITIONAL SCENES
As described in Sec. 3.2 we evaluate the impact of VIT sink tokens in LLM across global, local, and mixed tasks. Dense compositional scenes naturally fall under the local and mixed categories. As shown in Fig. b)., using only sink tokens leads to performance drop on these tasks, which supports key finding: "ViT sink tokens excel at global tasks, but fine-grained, localized tasks may require non-sink tokens, alone or with sink tokens, to capture detail.". To address this limitation, we introduce DIYSink, dynamic coken-selection framework that chooses between sink tokens, non-sink tokens, or both, depending on the specific task, thereby improving performance across varied scenarios. To further examine this finding, we evaluated three compositional reasoning benchmarks: (1) Spatial Relation Understanding (BLINK): Probes model's 's ability to infer relative object locations. (2, 3) Position and Counting Tasks (MME): Requires the model locate or enumerate objects in multi- object scenes. The comparison results are shown Tab. tokens underperforms compared to baseline which uses tokens, reaffirming our observation. On the other hand, DiYsink yields consistent gains across all architectures compositional tasks by dynamically selecting between sink/non-sin or both token types based on the visual content and the query, demonstrating the effectiveness of the proposed approach.
A12 We find that the baseline with only sink
10

---

Published as a conference paper at ICLR 2026
Table A12: Model Performance Comparison Across Different Tasks. Tiny (0.5B) refers to TinyLLLaAA..5BB-iiLP-P-vvvee -wee----enn--BB and LLaVA (7B) refers to LLaVA-7B.
BLINK SpaRel
MME position
Method
Tiny (0.5B)
LLaVA (7B)
Tiny (0.5B)
LLaVA (7B)
Tiny (0.5B)
LLaVA (7B)
Buseline
53.85
67.13
101.66
133.33
101.66
160.00
Baseline (Sink Only)
63.33
DIYSink (CoT)
53.15
64.34
123.33
141.67
110.00
165.00
DIYSink (RcW)
53.85
68.53
123.33
131.66
113.33
160.00
C.6 ANALYSIS THE TRAINING AND INFERENCE COMPUTATIONAL COST
We profiled both training and inference cost of three explored DIYSink variants, Dual-peojector only, Dual+CoT routing, and Rual+ReW routing. Note that we use SigLIP Qwen2-0.5B as the base model and measure the cost on Nvidia L40 (48GB). The results are shown in Tab. [A1]] All measurements use batch size of I a single GPU for memory and inference phase profiling, and 4 GPUs for raining-time profiling. Peak memory costs are reported for GPU and CPU cost measurement.
Parameter overhead (+0.2%). This stems from adding an additional MLP projector for sink vs. non-sink token separation, and, the ReW variant, an extra ML. compute per-token reweighting coefficients.
Training overhead. Since we pretrain the projector for sink and nonsink tokens separately, the total training time cost increases (so as FLOPs) roughly by one hour 4xL40s. We note that this can be reduced greatly by parallelly pretraining two MLP projectors.
Inference latency. Our two final models, Dual+CoT and Dual+ReW, exhibit distinct trade-ofit Dual+ReW incurs an additional GPU overhead and increases per-instance latency by roughly 0.01s. In contrast, Dual+CoT leverages the model's own multi-turn conversation routing, adding about 1.6s instance but without any extra parameters. Both methods deliver solid performance gains over the baseline (see Tab, 2 and can be chosen according users' latency and resource requirements. As for the tight-latency scenarios, we can adopt the Dual+ReW routing which captures the majority of DIYSink's benefits while adding only 0.01s of extra inference time.
Table A13: Resource Usage for Tinyllava SigL:P+QWenn-00.5B
Training phase Tinyllava IigL Q...--22-20.
Parameters
Time cost
FLOPs
GPU mem (peak) Ihs
CPU mem (peak)
Baseline w/ Dual
924.09
41981.97 46120.79
4,3784E+15
14.75 GB
32.52 GB
925.9 M
4,8417E=15
17.07 GB
34.93 GB
CoT
46120.79
4.8417E+15
GB
w/ Dual w/ ReW
925.98 M
46437.70
4.8421FF+15
17.07 GB
34.93 GB
Testing phase on GQA)
Parameters
Time cosi (per instance)
FLOPs
GPU mem (peak)
CPU mem (peak)
BS-1
Baseline w/ Dual
1.8617E:+09
7.33 GB
GB
925.93 M
0.0947
0.8620F
7.89 GR
22,78 GB
Dual w/ CoT
925.93
0,2518
6,8232E+09
7.89 GB
22.78 GB
Dual w/ ReW
925.98
0.1057
1.9442F+09
8.42 GB
23.05 GB
C.7 ANALYSIS OF THE SINK/NON-SIN TOKEN CLASSIHCCTION ACCURACY FOR DIYSINK
As shown in the Table A14 sink and non-sink tokens behave very differently on global versus local tasks. When all local questions are mistakenly treated as global, as in the Ours (sink) setting, where only sink tokens used, the performance local tasks drops sharply, often falling well below the baseline. Conversely, in the Ours (nonsink) setting, using only non-sink tokens causes substantial degradation on global tasks, when they misclassified as local.
11

---

Published as a conference paper at ICLR 2026
Table A14: Model performance Local and Global tasks. Local represent tasks that non sink tokens work well and Global represent tasks that sink tokens work well. Sink and non-sink tokens behave very differently on global vs. local tasks. We bold the numbers where DIYSink surpass baseline as we are comparing with baseline. MV. stands for Math Vista.
Local
Global
Methods
GQA
SQA-image
TextVQA
MME Cog.
MV. SciRea
MV. NUM
Tinyllava 0.5B (Qwen)
Baseline
57.98
57.76
47.32
207.14
36.89
11.11
Ours (sink)
39.53
53.94
30.10
270.71
37.70
17.36
Ours (nonsink)
58.07
60.09
48.28
220.00
36.89
11.11
DIYSink (CoT)
58,14
61.08
46.14
277.50
37.70
11.81
DIYSink (ReW)
57.75
60.24
47.42
229.64
38,52
14.58
Tinyllava 3B (Phi)
Baseline
51.09
68.47
41.15
261.79
45.90
13.89
Ours (sink)
39.13
65.79
35,92
315.71
45.90
22.22
Ours (nonsink)
60.02
70.25
47.51
269.64
43.44
19.44
DIYSink (CoT)
59.99
68.57
46.22
280.71
47,54
13.89
DIYSink (ReW)
59,79
70.50
50.77
265.71
42.62
19.44
However, as shown in the DIYSink (CoT) and DITSink (ReW) row of Table A14 we observe consistent improvement in downstream tasks with our dynamic selection module. This demonstrates that DIYSink can adaptively choose the suitable token type for each task and approximate maximum performance, effectively leveraging the strengths of both sink and non-sink tokens. Note that when performing CoT or reweighting in DIYSink, GPT is not involved in the decision-making. GPT classification is only used for the analysis in Section of the main paper.
C.8 ANALYSIS OF USING CHAIN-OF-THOUGHHY SELECTION MECHANISM AS THE SOFT SELECTION
In Sec. we propose both a Chain-of-Thougge (CoT) approach and a learnable reweighting (ReW) mechanism for sink token selection. While the CoT approach performs hard selection by choosing discrete inference modes (sink-only, non-sink-only, or both), the ReW mechanism already serves soft selection mechanism by assigning continuous weights to sink and non-sink tokens. Here, we additionally explore converting the CoT procedure itself into soft selector. Specifically, for given task, we first run the initial 10 VQA samples using the standard CoT to obtain the empirical distribution of sink and non-sink token usage. For instance, in code reasoning task, if the first ten samples yield sink-only inference (6 samples), non-sink-only inference (2 samples), and both (2 samples), the resulting soft weighting values are calculated as Waink (6 + 2)/10 = 0.8 and Wnon-sink = (2 + 2)/10 10 = 0.4. We then rerun the inference for that task using these soft reweighting values. As shown in Table A15 the performance of the Soft-CoT variant is par with the original COT. While the overall score of Soft-CoT slightly higher than the original CoT in some settings, the performance the cognition subtask drop slightly, suggesting trade-off between global and task-specific optimization.
C.9 PARAMETER OVERHEAD OF THE DUAL-MLP MODULE
While the dual-MLPP connector doubles the parameters of the original MLP, the connector constitutes only a very small portion of the overall LVLM. For example, in TinyLLaVA-0.55. the dual-MLPP module introduces only approximately 0.2% additional parameters (1.8M out ~B total), making it unlikely that the observed improvements are driven solely by the parameter increase. To further disentangle this effect, we evaluate a baseline where the MLP depth is doubled (w/ doublemip) while keeping single connector pathway. As shown in Table A16 the doubled-MLP baseline performs comparably to, or only slightly above, the original single-MLP baseline across all three benchmarks, yet it consistently remains below both DIYSink variants (CoT and ReW). This confirms that the
12

---

Published as a conference paper at ICLR 2026
Table A15: Comparison of DIYSink selection mechanisms on TinyLLaVA-005B (Qwen2) and TinyLLaVA-3B (Phi). Bold indicates the best result per model group. ReW denotes the learnable reweighting mechanism, CoT denotes the hard chain-of-thin thain-of-thought selection, and Soft-CoT denotes the proposed soft chain-of-thought variant.
Methods
ALL
Perception
Cognition
ComRea
TextTrans
CodeRea
(inLLLvA-0.58 (Qwen2)
Baseline
1381.10
1173.96
207.1 14
82.14
50.00
37.50
DIY Sink (ReW)
1451.87
1222.23
229.64
87.14
50.00
37.50
DIYSink (CoT)
1456.78
1179.28
277.50
85.00
80.00
52.50
DIYSink (Soft-CoT)
1456.32
1238.46
217.86
87.86
50.00
32.50
TanyLLaVA-3 (Phi)
Baseline
1455.22
1193.43
261.79
99.29
50.00
57.50
DIY (ReW)
1682.41
1416.70
265.71
115.71
57.50
45.00
DIYSink (CoT)
1523.18
1242.47
280.71
110.71
67.50
50.00
DIYSink (Soft-CoT) 1579.85
1310.21
269.64
117.14
50.00
47.50
Table A16: Ablation parameter overhead. "w/ doublemlp" doubles the MLP connector depth while maintaining a single pathway. Our dual-MLP module adds only ~0.2% parameters yet yields consistent improvements, indicating gains arise from the design rather than increased capacity. Bold indicates the best result per column.
Model
LLaVA Eval
MME
Math Vista
LinyLLaVA 0.5B (Qwen2.5)
49.57
1506.68
24.40
TinyLLaVe 0.5B (Qwen2.5) w/ doublemlp
50.18
1500,95
25.10
Ours (CoT)
50.02
1522.45
26.00
Ours (ReW)
50.34
1526.69
26.90
performance gains of DIYSink stem from its architectural design, separately routing sink and non-sink tokens, rather than from simply adding more parameters.
C.10 EFFICIENCY ANALYSIS OF VIT SINK TOKENS
We further explore the efficiency implications of using only the small set of VIT sink tokens for inference. As shown in Table A17 we run baseline models by feeding only the ViT sink tokens into the LLM, which constitute approximately 0.5% of the total visual tokens, and evaluate on reasoning-heese heavy subtasks from MME and Math Vista. Despite using drastically reduced set of visual tokens, sink-only inference yields consistent improvements on high-level global reasoning tasks. On MME, we observe gains numerical calculation, text translation, and code reasoning: on Math Vista, improvements are seen in algebraic, geometry, and scientific reasoning task. These results reinforce our finding that VIT sink tokens encode highly abstract, global visual information. Because these tokens capture distilled semantics most relevant for global reasoning, they enable efficient inference with improved performance on reasoning-centee benchmarks, highlighting their potential for token pruning and lightweight inference strategies.
C.11 HUMAN STUDY OF TASK TAXONOMY USING GPT
To validate the consistency of our GPT-based task taxonomy, we conducted a human verification study involving evaluators who assessed 84 VQA pairs using the same criteria provided to GPT-4o. To first establish reliability ceiling for this inherently subjective task, two independent annotators labeled the same set of 14 VQA pairs, yielding Cohen's 's Kappa scores 0.689 for Image Complexity and 0.583 for Query Globalness (Table These scores indicate that even human experts reach only "substantial" rather than "neferterfect" agreement, reflecting the intrinsic ambiguity these dimensions.
13

---

Published as a conference paper at ICLR 2026
Table A17: Efficiency Analysis Inference using only ViT sink tokens (~0.5% of visual tokens). Sink-only inference consistently improves performance on high-level reasoning subtasks across MME and Math Vista. Bold indicates the best result per model group.
MME
Vista
Methods
NumCal
TextTrans
CodeRea
AL.G
GEO
SCI
GPS
TinLLLVAA-..B-Qwaar
37.50
50.00
37.50
22.42
20.92
36.89
22.12
sink only
50.00
65.00
50.00
25.27
23.85
38.52
25.96
LLaVA 7B
55,00
55.00
42.50
25.27
20.92
39.34
21.15
+ sink only
65.00
95.00
65.00
32.03
30.13
43.44
34.13
Table A18: anter-annotator agreement for the task taxonomy. Cohen's Kappa and correlation coeffi- cients are reported for human-human and human-GPT-40 comparisons across Image Complexity and Query Globalness dimensions.
Image Complexity Query Globalness Correlation Coefficient
Human vs Human Human vs GPT
0.6897
0.5826
0.6667
0.5263
0.4267
0.5120
Against this human baseline, GPT demonstrates robust alignment. When comparing continuous decimal scores with 0.5 tolerance, machine-human agreement achieves Kappa scores of 0.526 (moderate) for Image Complexity and 0.427 (moderate) for Query Globalness, with combined correlation coefficient of 0.512 compared to 0.667 for the human- pair. These results confirm that GPT captures the underlying difficulty distribution with a reliability approaching that of human annotators, supporting its use as a scalable proxy for task classification.
C.12 EVALUATION ON HALLUCINATION--BSSI BENCHMARK
To further demonstrate the effectiveness of our methods, we extend our evaluation to include the POPE benchmark. which targets object ballucination by probing whether specific objects exist in an image. As shown in Table A19 our trained models (CoT and ReW) consistently improve over their respective baselines across all four model scales. We attribute this dynamically reweight toward local non-sink tokens. Since POPE focuses on object existence, a localized task, this reweighting allows the model better attend specific object details, reducing hallucination.
method's ability to
C.13 STATISTICAL ANALYSIS ON MODEL PERFORMANCE SIGNIFICANCY
To assess the statistical significance of both the training- free sink th-te-ttthe-etteee front method and the train- from-scratch DIYSink approach, conduct McNemar's test McNNemar (1947) Raschka (1811) to compare each method against its corresponding baseline.
For the train-from-seee fro-s--ttt--s-cc setting, shown in the right table of Table A20 we compute McNemar's test p-values over all benchmarks reported in Table comparing DIY Sink (ReW) with the baseline while assigning equal weight each benchmark. The results are statistically significant for three out of four models. LLaVA-7B yields p-value of 0.14, which still shows weak significance.
For the training- free methods, as discussed in Section 3, sink tokens benefit the model primarily on tasks requiring high-level, abstract reasoning (the Global and Mixed categories in Figure Accord- ingly, we observe unchanged performance on VAv (mostly Local tasks) and improvements on subtasks from MME and Math Vista (Global and Mixed tasks). To verify the significance of these im- provements, we compute McNemar's 's test p- valuue on MME and Math MathVista, as shown in the left table of Table A20 Three out of four training- free sink-to mokk-to-the--e too-the-frre front variants (Intern- VL2.5-4B, Phi3.5-V, V, and Molmo-7B-DD show statistically significant improvement (p-value < 0.05), with p-values as low 0.005. Finally, although the training-free freee variants yield smaller gains than the rain-from-scratch
14

---

Published as a conference paper at ICLR 2026
Table A19: Results on the POPE hallucination benchmark. Bold indicates the best result per model group. Our method consistently improves over baselines, particularly through dynamic reweighting toward local non-sink tokens.
Model Name
POPE
Model Name
POPE
TivALLaVA 0.5B (Qwen2)
TimyLLaVA-3D (Qwen2.5)
Baseline
86.80
Baseline
85,93
Ours (CoT)
87.15
Ours (CoT)
86.33
Ours (ReW)
86.10
Ours (ReW)
86.10
TinyLLaVA 3B (Phi)
LLaVA 7B
Baseline
84.00
Baseline
87.16
Ours (CoT)
86.13
Ours (CoT)
86.80
Ours (ReW)
85.90
Ours (ReW)
87.30
Table A20: McNemar's P-values for performance significance. The left table reports p-values for training- fareing-freeeee methods and the right table shows p-values for DIYSink.
Model
P-value
Model
P-value
Intern VL2-4B
0.0440
Tiny VA--O.5B (Qwen2)
0.0061
Phi 3.5
0.0490
TinyLLaVA- VA-3B (Phi)
0.0001
Deepseck-vl-7b-chte
0.3900
TinyLLaVA-3B (Qwen2.5)
0.0001
MOLMO-7B-D 0.0050
LLaVA-7B
0.1410
models, we present them as lightweight fallback option, a switch-on mechanism for scenarios where additional data or computation are unavailable.
D EXTRA QUALITATIVE EXAMPLES
D.1 DIYSink ON LLAVA-7B
To better understand how the model behaves when sink tokens are emphasized, compared to the baseline that assigns equal weight to sink and non-sink tokens, present qualitative examples using DIYSink (ReW) on VA-7BB These examples, shown Fig. A7 Fig. A8 and Fig. A9 illustrate the model's reasoning process commonsense reasoning, counting, and arithmetic tasks. We observe that reweighting the sink tokens enables the model to produce more coherent and accurate reasoning, particularly for questions that require global understanding and higher-level inference.
D.2 WORD MAPPING RESULTS
of the word mapping result as illustrated in See. (3.2 Interestingly, we observed accurate mapping of patches with open-form words such the "cup" and "water-other" in the second figure of Fig. All well as the "window-blind" the first figure of Fig. A12 the "cell phone" in the second figure of Fig. A12
We provide some qualiatitve examples in Fig. A1l and Fig. A12
E MODEL LIMITATIONS AND SOCIAL IMPACT
E. MODEL LIMITATIONS AND FUTURE DIRECTIONS
While our training free repositioning strategy generally enhances the VQA capabilities of recent LVLMs, our proposed D/YSink framework-thoud yielding more substantial cerrovementt-requuuie access to training data and certain amount of computational resources, as involves training the model from scratch.
15

---

Published as a conference paper at ICLR 2026
Commonsense Reasoning
?
Does picture usually faster than the car?
LLaVA-7B DIYSink
A: No, the vehicle in the picture, which is bicycle, does not usually run faster Bicycles and rely the rider's physical strength and stamina propel them forward. Cars, on the other hand, are motorized vehicles that use engines to generate power and speed. Bicycles are generally cars,
LLaVA-7B
A: Yes, the bicycle in the picture usually runs faster than the car. Bicycles are human-poweer vehicles that rely on the rider's physical strength and endurance to propel them forward. This than them capable reaching higher speeds
Sink vs. Non-sink tokens weight [1.14, 0.86]
Figure A7: Qualitative comparison of LLaVA 7B DIYSink and LLaVA 7B the Commonsense Reasoning task.
Count
? Are there only three dogs appear this image?
LLaVA-7B DIYSink
A: There are actually four dogs image. The dog running towards the
the other three dogs are running behind The presence four dogs suggests that they might be part of larger group farily of dogs, and they are all enjoying their time together the grassy field.
X
LLavA-7B
A: Yes, there only three dogs visible the image. The reason for this is that the dogs are running field, and the image captures a specific moment in time when they are mation...
Sink vs. Non-sink tokens weight [1.42, 0.58]
Figure A8: Qualitative comparison LLaVA 7B D/YSink and LLaVA 7B on the counting task.
Arithmetic Reasoning
?
Netelle buys 4.5 kilograms of turmeric. What the total cost?
LLaVA-7B DIYSink
A: x. $13.80
pepper
$4 per kg
cumin
$3 per
LLaVA-7B
X
turmeric
$3 per kg
A: The total cost 4.5 kilograms of turmeric $3.20.
cinnamon
$3 per
paprika
$2 per kg
Sink Non-sink tokens weight
[1.08, 0.92]
Figure A9: Qualitative comparison of LLaVA 7B D/YSink and LLaV LA 7B the arithmetic reasoning task.
16

---

Published as a conference paper at ICLR 2026
A. Attn Weight Analysis
Avg # of Takens
Avg Attn Weights for LLM Outputs
102
10
Avg Attn for LLM Dutputs
Aug Tokens
100
10-1
10-2 2
50
100
150
200
250
12 Norm of Taken
B. Hidden Dim of LLM-emerged Sink
C. Hidden Dim of Propagated VIT Sink
Magnitude
Magnitude
60
10
0665
4955
Hidden
Figure A10: Attention to ViT sink tokens and sink dimensions of ViT and LLM sinks in LLaVA- v1.5-7B. Enlarged figure 3.
17

---

Published as a conference paper at ICLR 2026
5ghht
sphht
Eghht
lesses
Maree
Maree
baren
light
Sphht
donds
light
Nuace
kggh
doods
light
deads
toure
laves
leaven
Waree
lovess
gee
Use
Mavres
Goods
light
doodd
lavve
kghht
laave
Marvet
house
light
light
light
kghh
light
docds
Movees
wark
caloe
kghht
Warena
Mavess
Iushs
light
light
laavss
light
laaves
Nouse
kghh
Waves
kghh
Sghtt
1ight
light
kgh
sipht
Right
Signt
MOVER
Mores
Maves
Traa
largee
Sghss
Sagnus
Narwork
lights
treak
bagnc
lights
light
hight
house
light
vouree
Sight
Sight
Sight
Sight
hodre
Sight
lark
ear
Saght
light
light
Sight
sand
Sight
light
light:
Sight
Right
light
dellt
Sight
Sught
lights
Heghtt
Sagltt
Sightt
Sight
light
Sight
bessum
Sight
Sight
houre
Nabile
Nollet
backspack
Sight
Sight
corp
Sight
sight
hight
light
light
Terees
Sight
sight
Pight
Sights
cxft
curo
Sight
Sygtt
Sabie
duale
wood
Nution
phosiee
Sight
take
Sight
thalr
dhalr
dhale
light
Valiee
Saloles
light
Sight
Sight
Right
light
Sight
alhet
sghht
dhalr
dbale
dhals
sight
liiht
Vabidies
light
becks
Sggss
Sight
Sghht
Rgght
Sight
Raghts
Seglis
Sygne
Rights
Sight
Sight
Chale
duafr
dhalr
Sight
light
Sitht
Nabinw
Raghts
laghi
shole
Night
Sight
light
Chall
Sight:
deafr
deale
Sight
Figure All: Qualitative examples of word mapping.
18

---

Published as a conference paper at ICLR 2026
off
oFp
lable
light
Nobla
wiltt
wilet
oef
Oef
light
bosk
lock
light
light
light
monttr
aber
Backkpack
Backsaack
counter
counter
connte
horeel
Ioeel
Eight
Sight
Right
light
light
Ryght
light
sight
Right
tyght
breel
fight
light
light
Sight
light
light
light
Sight
light
Sight
Aight
Sight
Hight
light
pager
markel
light
Wiebew
louse
windew
will
Nowee
Houee
Pouee
babba
alndow
wart gaandl
baiket
wath panee)
doee bilood
Sight
Sightt
Nouse
wnid paaeel
paper
House
light
light
bpht
pepper
Rolket
walee
light
with spannal
glanel
Sight
Sight
Tack
vooft
Pgghs
Sygtt
Sight
Sipht
Sight
Search
houre
Sigh
Sight
Sigh
Sight
Sight
Sight
Sight
hight
Sight
Sight
light
Right
Fight
sight
sight
light
sight
Sight
Right
sight
light
light
light
deelt
Quala
MARA
dall
Coather
Takking
Sagas
Sight
wall- SO0A
dattees
Sapss
Sight
Night
lights
light
lights
Gabie
lightt
Rightt
lights
light
Sight
lights
Sight
light
lights
Sight
Sight
Sight
light
light
light
Hight
lightt
Tight
lights
light
light
light
light
ather
abolt
adwer
booke
borkee
torek
lightt
Vodeet
bonoew
bouree
boooke
light
light
light
light
Bockk
Sight
dealts
lights
light
light
light
light
light
light
MBDI
light
hight
Sight
ligitt
lights
bredl
panel
cleck
light
light
Hight
light
Rgght
light
lights
ganeel
light
Right
bertble
eat
cop
light
Right
light
Right
Sihht
parger
Sight
sabie
syht
hocree
plane
plu
pagert
cep
cranker
col
cal phone
phane
coll
Cell phas
plick
phane
oogger
gaggert
pager
paper
gagger
porgert
pagger
pager
Light
sabie
sabie
Right
phosoe
light
Sagnus
Tagto
1000
salie
Right
Syght
sable
Sgtht
Sabila
Salies
light
sabta
light
Night
Right
Right
light
Vourk
phone
backgaack
Sight
light
light
gager
light
light
light
book
Sight
sight
light
light
Right
Right
Right
Figure A12: Qualitative examples of word mapping.
19

---

Published as a conference paper at ICLR 2026
Additionally, this work primarily focuses on the image and language modalities. Extending DIYSink to other modalities such as video and audio remains an open area for future research and exploration.
E.2 SOCIAL IMPACT
In this work, we demonstrate that VIT sink tokens encode high-level visual context and can be effectively utilized by LVLMs to solve tasks requiring global understanding and reasoning. Building on this insight, we propose two approaches to enhance the use of VIT sinks in existing LVLMs: (1)
a training-free repositioning strategy that prioritizes VIT sink tokens in the input sequence, and (2) DIYSink, training- based framework incorporating dual MLP projectors and task-aware selection modules. Our methods yield significant improvements over the baseline across range of VQA tasks.
While our research focuses improving model performance, it is important to acknowledge potential misuse. In particular, large lssoonlanguuag models (VLMs) could generate harmful visual or textual content if deployed irresponsibly. Although this risk is not specific methods, it highlights the need for further research into safety and ethical considerations in multimodal Al systems.
F LLM USAGE
In this paper, we utilized LLMs as writing assistant. The main usage of LLM was for proofreading, enhancing text fluency, and checking grammar correctness. All the scientific contributions, findings, experimental design, and framework design are the original work the authors.
20


Reviews:
[{"review_id": "2CXnUeRmLF", "reviewer": "Reviewer_qH3u", "summary": "This paper mainly investigates the ViT sink phenomenon in the MLLM domain. The main contributions are: (1) providing many insightful observations about the ViT sink and its underlying mechanisms, and (2) designing applications that leverage these findings to enable MLLMs to route between non-sink and sink states under different scenarios, which is both useful and insightful.", "questions": "* For an image token, the positional embedding indicates its location on the image. However, repositioning these tokens breaks this property. Although the results are promising, I would like to hear the authors’ explanation of this phenomenon.\n\n* The proposed two-step chain-of-thought process could also be redesigned to use a soft rather than hard selection mechanism. For example, one could model the probability that a given question requires holistic reasoning.\n\n* Additionally, are the two introduced MLP parameters summed to equal the original one? If not, the comparison may not be entirely fair.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Valuable observations and insights for the multimodal large language model community.\n\n* The overall logical flow of the paper is well-structured — from motivation, to investigation methods, to applications.\n\n* Comprehensive experiments are conducted across different models and benchmarks.\n\n* Some findings are quite interesting. For example: “Moreover, these high-value sink dimensions emerge only after multimodal training. In LLaVA-7B, the LLM’s original sink dimensions are {2533, 1415}, while the propagated ViT sink tokens activate dimensions {982, 2494, 3263}.”", "weaknesses": "* Some parts are confusing and not well presented or clearly clarified.\n\n  In Section 3.2, some results show that specific head sinks may correspond to either the foreground or the background, while other results indicate that all sink tokens map to the main object of the image. The authors also conclude that ViT sinks capture coarse-grained, high-level contextual features. However, identifying a small main object in a large image seems more like a fine-grained task. I am a bit confused — could the authors provide a clearer and more unified explanation of these observations?\n\n* Some representations could be further improved.\n\n  In the current Figure 3, the leftmost image is not easily readable and should be better presented. A simple improvement would be to split it into two sub-images instead of keeping it as one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "43st5OgwOc", "reviewer": "Reviewer_UaWB", "summary": "This paper shows that ViT “attention sinks” in LVLMs are not noise but compact carriers of global semantics that propagate into the LLM with distinct activation patterns. Leveraging this, the authors propose a training-free token reordering (sink-to-the-front) and a trainable DIYSink with dual MLP projection and dynamic selection; both yield consistent gains, especially on global reasoning tasks, while sinks can hurt highly local tasks.", "questions": "1. Given that only about 1–3% of visual tokens are identified as ViT sinks, have the authors considered using this finding for token pruning or efficient inference? For example, could a “sink-only inference” configuration preserve comparable performance while significantly reducing FLOPs and memory cost?\n\n2. The task taxonomy (global, local, mixed) is annotated by GPT-4o. Was any human verification or inter-annotator check performed?\n\n3. The current evaluation mainly includes structured QA benchmarks (MME, MathVista). Could the authors consider extending experiments to hallucination-oriented benchmarks such as POPE, and high-resolution benchmarks such as V* or HRBench, to further validate the generality of the proposed method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes both a training-free and a training-based approach (Sink-to-the-Front, DIYSink) to utilize sink tokens. These methods are simple yet effective, model-agnostic, and easy to integrate into existing LVLM pipelines without retraining the full model.\n\n2. The discovery that only 1–3% of ViT tokens (the sinks) dominate semantic propagation implies a strong potential for visual token compression and efficient inference. Even though not the main focus, this insight could inspire future research on adaptive token selection and efficient LVLM decoding.", "weaknesses": "1. The efficiency implications of identifying only 1–3% ViT sink tokens are not explored; analyzing their potential for token pruning or lightweight inference would enrich the contribution.\n2. The task taxonomy (global, local, mixed) depends entirely on GPT-4o annotation without human verification, raising concerns about the consistency of the main conclusion.\n3. The experimental coverage is relatively limited, lacking evaluation on hallucination-oriented (e.g., POPE) and high-resolution (e.g., V*, HRBench) benchmarks, which would better demonstrate generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pNIqUX4YjZ", "reviewer": "Reviewer_yLnB", "summary": "The paper studies how large vision-language models (LVLMs) process visual information through looking into the *visual sink tokens.*\n\nThese are vision transformer (ViT) tokens with unusually large activation norms that have large attention score in the language model. The authors show that such tokens carry meaningful global scene information. They analyze how these sink tokens propagate into the language model, identify distinct hidden dimensions they activate, and demonstrate that they encode high-level semantics like object categories.\n\nThe paper also proposes two simple methods to improve LVLM performance with its findings: \n(1) a simple inference-time trick that reorders sink tokens to appear earlier, and \n(2) a training-based design called DIYSink that uses dual-MLP projectors for sink and non-sink tokens. Both improve multimodal reasoning performance on benchmarks like LLaVA eval, MME and MathVista.", "questions": "The connection between this work and previous LVLM token sink study like [1] is not discussed. The paper could better clarify how its findings extend and differ from this work.\n\n[1] Kaduri, O., Bagon, S., & Dekel, T. What’s in the image? A deep-dive into the vision of vision language models. CVPR, 2025.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. It offers the first detailed analysis of how ViT sink tokens propagate into LVLMs and interact with the LLM.  \n\n2. The paper presents clear qualitative and quantitative evidence showing that sink tokens encode global, scene-level semantics, while non-sink tokens focus on local, fine-grained details.   \n\n3. Building on these findings, the authors propose two improvement methods: a training-free Sink-to-the-Front trick and a Dual-MLP design. Both of them are intuitive, simple-yet-effective, compatible with existing models, and lead to consistent gains.  \n\n4. This paper provide a more in-depth understanding to model interpretability and how the vision branch work in current LVLMs.", "weaknesses": "1. The threshold used to distinguish sink and non-sink tokens is manually selected and not adaptive. This choice is heuristic and its robustness is not fully verified. The method lacks an adaptive or learnable way to determine the threshold cutoff, which limits its generalization across models and datasets.\n2. The experiments use CLIP and SigLIP-based LVLMs, all built on ViT encoders. It remains unclear if similar sink behavior and method effectiveness hold in other architectures like Mamba-based vision models that also show high-norm token patterns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QrSpp8Cqqg", "reviewer": "Reviewer_ZDRY", "summary": "This paper investigates the role of attention sink tokens originating from the Vision Transformer (ViT) in Large Vision-Language Models (LVLMs). Unlike prior work that focuses on eliminating or suppressing attention sinks as harmful, this work finds that a subset of ViT sinks propagate into the LLM and encode high-level semantic information. The authors find that ViT sinks help on global reasoning tasks but can hurt performance on fine-grained and local visual tasks. Based on this insight, they propose two methods to leverage sink tokens: a training-free “sink-to-the-front” token reordering strategy, and a training-based DIYSink framework with dynamic token selection. The experiments across multiple LVLM architectures show consistent improvements, especially on MathVista, MME, and reasoning benchmarks.", "questions": "1. Would the same sink behavior appear in models using non-ViT vision encoders? Or when using connector architectures other than an MLP, such as cross-attention or Q-Former? \n2. For DIYSink, how sensitive is the performance to incorrect task classification (e.g. when a “global” task is misclassified as “local”)? \n3. Why does placing sink tokens at the front work better than at the end of the token sequence? Could relative positional encodings influence this outcome?\n4. Have you tried the \"sink-to-the-front\" approach for other VLMs, such as LLaVA models: TinyLLaVA-3B or LLaVA-7B?\n5. Could you also address the points mentioned in the weaknesses list?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- **Novel insights**: The paper questions the common belief that visual attention sinks are harmful. Instead, it shows that ViTs’ sink tokens can carry useful high-level information, which can help LVLMs with global visual reasoning.\n- **Thorough empirical analysis**: The authors study the phenomenon from several angles, such as attention patterns and feature norms. The experiments are thorough and support the main claims well.\n- **Practical proposed methods**: “The proposed “sink-to-the-front” approach is training-free, simple to use, and still leads to improvements.", "weaknesses": "- **Limited theoretical explanation**: The paper offers little insight into the main claim—that ViT sink tokens encode global semantics. The explanation remains largely observational, with no theoretical grounding or analytical model to support why this behavior arises. More intuition or theoretical explanation would strengthen the contributions.\n- **Only marginal improvements**: Although the results show gains, many improvements reported in Tables 1 and 2 are relatively small—often below 1% and sometimes within the range of natural evaluation variance. Statistical tests or error bars would help validate that the gains are meaningful rather than just noise.\n- **Limited evidence on larger models**: The experiments are run on models up to 7B parameters. It is unclear if the improvements would remain for much larger models (> 7B), which may behave differently. Any experiment or discussion on this topic is valuable. \n- **Generalization beyond ViTs**: It is unclear whether the same sink behavior holds —and whether the method would work— for CNN-based or Mamba-based vision encoders, or for architectures that do not use MLP connectors.\n- **Incremental contribution**: Overall, the paper provides an interesting empirical observation but lacks sufficient novelty, depth, or quantitative impact to justify acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WVLqNUrB3D", "reviewer": "Reviewer_4Rra", "summary": "This paper studies the effect of (ViT) attention sink in vision language models. It finds that a few high norm ViT tokens act as visual sinks that receive disproportionate attention in the LLM and carry coarse scene information. Building on this, the paper proposes a training-free token reordering and a trainable MLP connector with dynamic routing to improve various downstream visual reasoning tasks. The findings about that the sink tokens are beneficial for tasks with low visual complexity and global semantics but may degrade performance\non tasks that demand localized, detail-rich visual processing is useful.", "questions": "1. How does RoPE handle the sink token reordering at inference time, as positional embeddings of the sink token are not the same as what the VLMs see during training (i.e. without reordering)? Does moving sinks to the front/back introduce unintended positional biases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1 The paper is well-written and easy to follow.\n\n2 The research topic is timely and interesting. The paper shows a clear empirical pattern linking high norm visual tokens to downstream concept semantics and reasoning capacity.\n\n3 The proposed strategies that can be adopted with low engineering overhead.\n\n4 The use of relevance map and decoding to world distribution to study the propagation of ViT attention sink to VLM is a sound approach.\n\n5 The finding that the ViT sinks in VLMs encode vital high-level visual context and is actually useful for visual reasoning task is a novel finding and good to know for the community.", "weaknesses": "1. While the link between attention sink and visual semantics are evident. Why making the sink token to the front/end would make visual reasoning capacity stronger/weaker in the training-free setting, supposing the connector still struggles to project the re-ordered sink/non-sink tokens to the same embedding?\n2. Currently all experiments are conducted up to 7b models, have the authors consider to test larger models like to see if the observed patterns generalize? \n\n- Figure A9/A10's captions are \"Enter Caption\"\n- Font size in Figure 4/5 too small", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)