See What You Are Told: Visual Attention Sink in Large Multimodal Models
Authors: Seil Kang
Jinyeong Kim
Junhyeok Kim
Seong Jae Hwang
Venue: ICLR 2025 Poster (2025)
Keywords: Large multimodal models
Visual attention sink
Visual attention redistribution
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We find "visual attention sink" in LMMs, where irrelevant visual tokens with high attention weights arise from massive activation in specific dimensions. Also, we introduce Visual Attention Redistribution (VAR) to enhance attention on the image.

Abstract: Large multimodal models (LMMs) "see" images by leveraging the attention mechanism between text and visual tokens in the transformer decoder. Ideally, these models should focus on key visual information relevant to the text token. However, recent findings indicate that LMMs have an extraordinary tendency to consistently allocate high attention weights to specific visual tokens, even when these tokens are irrelevant to the corresponding text. In this study, we investigate the property behind the appearance of these irrelevant visual tokens and examine their characteristics. Our findings show that this behavior arises due to the massive activation of certain hidden state dimensions, which resembles the attention sink found in language models. Hence, we refer to this phenomenon as the visual attention sink. In particular, our analysis reveals that removing the irrelevant visual sink tokens does not impact model performance, despite receiving high attention weights. Consequently, we recycle the attention to these tokens as surplus resources, redistributing the attention budget to enhance focus on the image. To achieve this, we introduce Visual Attention Redistribution (VAR), a method that redistributes attention in image-centric heads, which we identify as innately focusing on visual information. VAR can be seamlessly applied across different LMMs to improve performance on a wide range of tasks, including general vision-language tasks, visual hallucination tasks, and vision-centric tasks, all without the need for additional training, models, or inference steps. Experimental results demonstrate that VAR enables LMMs to process visual information more effectively by adjusting their internal attention mechanisms, offering a new direction to enhancing the multimodal capabilities of LMMs.

Published as a conference paper at ICLR 2025
SEE WHAT YOU ARE TOLD: VISUAL ATTENTION SINK IN LARGE MULTIMODAL MODELS
Seil Kang" Jinyeong Kim Junhyeok Kim
Seong Jae Hwang!
Yonsei University
(seil, Jinyeongl 324, timespt, seongjae)@yonsee...... .ac.kr
ABSTRACT
Large multimodal models (LMMs) "see" images by leveraging the attention mechanism between text and visual tokens the transformer decoder. Ideally, these models should focus key visual information relevant to the text token. However, recent findings indicate that LMMs have an extraordinary tendency to consistently allocate high attention weights to specific visual tokens, even when these tokens are irrelevant to the corresponding text. In this study, we investigate the property behind the appearance of these irrelevant visual tokens and examine their characteristics. Our findings show that this behavior arises due to the mas- sive activation of certain hidden state dimensions, which resembles the attention sink found in language models. Hence, we refer to this phenomenon as the vi- sual attention sink. In particular, our analysis reveals that removing the irrelevant visual sink tokens does not impact model performance, despite receiving high attention weights. Consequently, we recycle the attention to these tokens as sur- plus resources, redistributing the attention budget to enhance focus on the image. To achieve this, we introduce Visual Attention Redistribution (VAR), a method that redistributes attention in image-centric heads, which we identify as innately focusing on visual information. VAR can be scamlessly applied across different LMMs to improve performance wide range of tasks, including general vision- language tasks, visual hallucination tasks, and vision-centric tasks, all without the need for additional training, models, inference steps. Experimental results demonstrate that VAR enables LMMs to process visual information more effec- tively by adjusting their internal attention mechanisms, offering new direction enhancing the multimodal capabilities of LMMs.
INTRODUCTION
Large multimodal models (LMMs) have been actively expanding the capabilities of large language models to multimodal tasks (Liu et al. 2024c b Li et al. 22023b; Bai et al.) 2023 In particu- lar, the LMMs leverage a pre-trained visual encoder (Radford et al. 2021 to process image data and the transformer decoder of large language model to generate text responses (OpenAl 2023 Touvron et al., 2023 Yang et al.) 2024). This straightforward yet powerful architecture has proven highly effective in utilizing visual information from images for vision- language tasks such visual
question answering, image captioning, and visual reasoning (Peng et al. 2023 Alayrac et al. 2022
Tsimpoukelli al. 2021).
To incorporate visual information into text responses, LMMs rely on the attention mecha- nism (Vaswani et al.) (007)) within the transformer decoder. Specifically, when processing multi- modal inputs, the attention weights between visual and text tokens determine how much each text token focuses on the corresponding visual information. For instance, as illustrated the top-left corner of Fig. [ when the text token is 'bird', the model concentrates on visual tokens associated with the bird the image. Intuitively, LMMs should primarily attend the visual tokens that are relevant to each text token.
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2025
Is there a
in the image?
Example
bird
person
Example 2
banana
bowl
Example
knife
cup
Example 4
wine
bottle
Visual attention maps of VLa between specified text tokens and visual to- kens. Attention map visualizes where the model "see" when processing the text token. The model expected to focus only on the visual tokens related to each text token. However, the model also attends to irrelevant visual tokens (red boxes) ) that are unrelated to the corresponding text token. Al- though we visualize the attention maps only for a few specified text tokens, these irrelevant tokens consistently occur in fixed locations across the entire text tokens, including the instructions and the generated responses (see Fig. 13 in Appendix for more examples).
Figure 1:
However, in practice, not all attention is directed toward the relevant visual tokens. As shown in Fig. [ the model also allocates high attention weights to visual tokens which are unrelated to the corresponding text. This phenomenon is widespread LMMs (Woo et al.) [2024] An et al.) 20224) and notable pattern emerges where irrelevant visual tokens consistently appear in fixed locations across different text tokens. For example, in each case illustrated in Fig, irrelevant visual tokens (highlighted in red boxes) consistently occupy the same positions regardless of the text tokens, indicating an underlying pattern. The cause and meaning of this phenomenon remain open questions, motivating this study.
In this work, we explore the underlying property and the characteristics of the irrelevant visual to- kens. We find that these tokens in the visual attention map arise from the massive activation of specific dimensions in the hidden states. This mechanism is analogous to the attention sink observed in language models, where the model consistently assigns large attention weights to tokens with limited semantic meaning (e.g.. 'BOS', '., "in', etc.) (Xiao et al. 2023 Sun et al. 2024a). Irrele- vant visual tokens can be identified by the extreme magnitudes in a few specific dimensions, which refer to visual sink tokens, as they also have limited semantic information from the image. Furthermore, we demonstrate that removing these visual sink tokens does not significantly impact the quality of the model's response, despite the model assigning high attention weights them.
Based on these experiments, we propose that the attention weights assigned to sink tokens can "attention budget". Since recent studies have reported that attention allocated to images is often insufficient compared to that given to text (Chen et al. 2024 Liu et al. (024d), we redistribute the excess attention from sink tokens to image. Also, considering each attention head serves a distinct function (Zheng et al. 2024), we identify the heads that are primarily responsible for focusing on visual information, namely, image- centric heads, based on the presence of visual attention sinks. Finally, we introduce Visual Attention Redistribution (VAR), a two-step method: first, selecting the image-centrie heads, and second, redistributing the attention budget to strengthen image focus within these selected heads.
be recycled
In summary, we uncover the underlying properties of irrelevant visual tokens and demonstrate that, much like sink tokens in language models, they are unnecessary for the model's functioning. To address this, we propose VAR, which reallocates attention from sink tokens to enhance focus on the image. Experimental results show that VAR improves the overall performance of LMMs across a range of tasks, including general vision-languee tasks, visual hallucination tasks, and vision- centric tasks. Notably, VAR can be applied to various models without additional training, models,
2

---

Published as a conference paper at ICLR 2025
or inference steps. This suggests that the existing LMMs can readily benefit from our approach to further enhance their multimodal capabilities by intensifying their attention to images. Our work presents effective method to address the issue of insufficient image attention and offers new perspective on understanding the attention mechanisms within LMMs.
RELATED WORK
Visual attention in large multimodal models. In large multimodal models (LMMs), the attention mechanism between text and images plays a pivotal role in incorporating visual information into the text responses. As such, the model's focus images is typically represented visual attention map (Aflalo et al. 2022 Stan et al. 2024). However, recent findings suggest that LMMs exhibit certain unintuitive behaviors in their visual attention patterns. Specifically, LMMs tend to dispro- portionately focus on a few visual tokens (Woo et al. 2024; Arif et al., 2024h, with some tokens receiving high attention weights regardless of the corresponding text token (An et al. 2024). Addi- tionally, recent works have shown that LMMs often fail to adequately attend to visual information overall (Chen et al., [2024] [iu al. 2024d). To address this issue, visual contrastive decoding (Leng et al.) 2024] Favero et al., 2024) has been proposed, which contrasts the outputs of two models one with and one without visual input to encourage greater reliance on visual cues. Further, other ap- proaches (Zhang et al. 2024b) Zhu et al.) 2024) enhance this by increasing the attention weights assigned to images, ensuring that visual information receives sufficient focus.
Attention sink in language models. Attention sink is intriguing phenomenon in language mod- els, where certain sink tokens with limited semantic meaning (c.g.. 'BOS'.
"'n'. etc.) receive disproportionately high attention weights (Xiao et al. 2023 Ferrando & Voita 2024). Background tokens, which contain little information, in vision transformers also exhibit similar behavior (Darcet al. 2023), suggesting that attention sink is common phenomenon across different modalities. Although sink tokens receive substantial attention weights, they contribute minimally to the model's overall predictions (Kobayashi et al., 2020 Bondarenko al. 2023). Recent research suggests that attention sink arises from the massive activation of specific dimensions within the hidden states of sink tokens, which occurs prior to the high attention allocation (Sun al. [2024]] Cancedda 2024).
Gu et al. (2024) further investigated the factors that contribute to the emergence of attention sink. Additionally, Yu et al. (2024) recalibrated the attention weights assigned to sink tokens in specific attention heads to elicit more accurate responses from language models. We extend the concept of attention sink to the multimodal domain by introducing the idea visual attention sink LMMs.
3 PRELIMINARIES
LMMs typically consist of a visual encoder, a projector, and a large language model, Visual encoder and projector extract visual features from images and project them into text-aligned representations. As shown in the left side of Fig. 2 the large language model receives three types of input: (1) system instructions, (2) visual features from the image, and (3) text including the user's query and preceding context. Then, the model generates responses in autoregressive manner. In this paper, refer to the discrete inputs to the large language model, as well as the embeddings within it, as tokens.
Let the indices of system tokens, visual tokens, and text tokens be denoted as T Ixxt spectively, which are subsets of the indices of all input tokens I. The input is processed through L transformer blocks, each of which consists of multi-head attention (MHA) and werward network (FFN):
8/
Σ MHA (
(1) x
of (FFN((()) + #!
where a E R is the input of the i-th token in the f-th layer and af. a TR D are the output of MHA and FFN, respectively.
We now focus on MHA, which enables interactions between different tokens. Following Elhage al. 2021 the individual input f-1 interact with the previous tokens X X(-1 {0)
2 as
3

---

Published as a conference paper at ICLR 2025
inreleuantt visual tolenn 3 rellevant visual tokers
SBo8e
System
Scale of attention weights
20
<ingg
sim
B
1415
2538
4096
Gamga
<img>
<mp
Reshape
Image
20
Simg?
SimE
0
1415
2533
4096
558.8
Is there a clock this image?
Visual Attention Map
Text
20
2-th Layer of LMM
0
1415
2533
4096
Dimension
Figure 2: Illustration typical architecture of LMMs and investigation of visual attention sink. A large multimodal model receives the image and text as inputs. Each text token interacts with the visual tokens through the attention mechanism in the transformer decoder. We can visualize the interaction in the form of an attention map. We discover that irrelevant visual tokens (marked as red boxes) the attention map have massive activation specific dimensions of hidden states, while relevant visual tokens (marked as blue boxes) do not. Well-known sink tokens (e.g., 'BOS') in language models also exhibit identical patterns in the hidden states.
follows:
(x) (xx--We f- WhITH )
well
MHA (x
(x)-1)
1
a a²h softmax
(2)
DDr
iSi
where w E D the output projection matrix, and w w E Rx are the query and key projection matrices, respectively. at the attention weights from X!- f.h Ejsi 4.h = 1). Fq. indicates that the attention weight Og,j can be interpreted
Q
Si to 2. a
measure of the extent to which the LMM attends to 2; while processing af As we study how text tokens interact with visual tokens to generate responses, focus the attention weights from visual to e, text tokens, i.e., 0i (i I Ivis) and investigate them in the form of visual attention map.
VISUAL ATTENTION SINK
In LMMs, to generate responses that consider visual information, text tokens "see" the image through the attention mechanism in the transformer decoder. Attention from visual tokens (key) to text token (query) is interpreted as the individual text token's 's focus on the visual information. Based on this interpretation, we can investigate attention weights from visual tokens a text token in the form of visual attention map. Visual attention maps express the interaction between text and visual tokens LMMs. Fig. shows the visual attention map between specified text tokens and visual tokens. The model is expected to focus only the visual tokens related the text token.
However, the model also attends to some visual tokens which are irrelevant to the corresponding text token, reported in previous studies (Woo et al.] 2024 An et al. 2024). For example, as shown in the top-right of Fig. T the model assigns high attention weights to the visual tokens (red boxes) unrelated the text token banana. Also, the irrelevant visual tokens exist in fixed locations, regardless of the specific text token. This consistent pattern suggests that the irrelevant visual tokens have their own inherent properties causing their appearance. We interested the property behind the appearance of these irrelevant visual tokens and understanding their meaning in LMMs.
In the following sections, find that the irrelevant visual tokens in the visual attention map stem from the massive activation of specific dimensions of hidden states. This phenomenon is analogous the attention sink in language models (Xiao al. 2023] Sun et al.) 20224), where the model assigns large attention weights to tokens with limited semantic meaning (e.g., BOS). We refer to this phenomenon as visual ottention sink and further analyse its characteristics.
4

---

Published as a conference paper at ICLR 2025
4.1 INVESTIGATING THE PROPERTY OF IRRELEVANT VISUAL TOKENS
We divide the visual tokens with high attention weights in the visual attention map into two cate- gories: irrelevant visual tokens and relevant visual tokens. Irrelevant visual tokens are visual tokens that are not related to the corresponding text token. In contrast, relevant visual tokens are the visual tokens that are related to the corresponding text token. Fig. 2 illustrates the example of irrelevant and relevant visual tokens as red and blue boxes, respectively.
How to distinguish irrelevant visual tokens? We focus on that the irrelevant visual tokens emerge consistently in fixed locations, regardless of the text token. As shown in the bottom-left of Fig. [ whether the text token kni fe or cup, the model consistently attends to the same irrelevant visual tokens. This observation suggests that irrelevant visual tokens appear not due to the text tokens but as a result of their own inherent properties. Therefore, we examine the hidden states of the irrelevant tokens to investigate their unique property. The right side of Fig. 2 shows the hidden states of the irrelevant visual token (red) and the relevant visual token (blue), as well as the 'BOS' token (green).
Irrelevant visual tokens have high activation in specific dimensions. We observe that the hidden states of the irrelevant visual tokens exhibit massive activation in specific dimensions while the relevant visual tokens do not. Also, the dimensions that are highly activated in the irrelevant visual tokens are identical to those of the 'BOS' token, which is known as the representative sink token in language models (Sun et al., 2024a). This observation indicates that the irrelevant visual tokens closely related to the attention sink.
To extend and formalize this observation further, we check the massive activation values of spe- cific dimensions, called sink dimensions Daink. in the hidden states of the tokens. Daink is a set of fixed dimensions that are determined by the base language model of LMMs. For instance, LLaMA- 2 (Touvron et al.) 2023), the base language model for LLaVA-1.55 ------ (Liu et al.] (024a), has Dsink = {1415, 2533 J. We validate that the sink dimensions in LMMS are consistent with those in base language models in Appendix [A_ and utilize the sink dimensions reported in [un et al. 2024a). Given a hidden state æ E IDD of a token, we denote the sink dimension value as follows:
<(x) ma
x/d/V ED =[d]? where x r|] is the d-th dimension of the hidden state.
The hidden states are normalized by the root mean square of the dimensions for stability and we only consider the maximum value among the sink dimensions. As shown in the rightmost side of Fig. 2 the sink dimension value of the irrelevant visual token (red) is significantly higher than that of the relevant visual token (blue).
Sink dimension value can separate irrelevant visual tokens from relevant visual tokens. We incorporate sink dimension value to discriminate the irrelevant visual tokens from the relevant visual tokens. For visual tokens. we plot the pairwise value of sink dimension value and the corresponding attention weights in Fig. Ba). Detailed experimental settings are described in Appendix [33] The sink dimension value distribution of visual tokens with high attention weights is clearly separated into two groups: one with a low sink dimension value and the other with a high sink dimension value. From this analysis, we now define the visual tokens with high sink dimension value visual sink tokens, noting that they are closely related to the attention sink in language models.
Specifically, we set threshold 7 to divide the distribution in Fig. F( and define the indices of the sink tokens as _' [ E I d(arf-)) 2 T). where 25f f is the input hidden state of the j-th token in the b-th layer. In the subsequent analysis, we set 20. We note that the definition of İe also encompasses all indices of sink tokens, including both visual and text tokens. We denote the visual sink tokens as _ = It n Ivis, where Ivis is the set of indices of visual tokens. We refer to other visual tokens as visual non-sink tokens and denote them Ivis I for convenience. While the definition of visual sink tokens I also includes the tokens with low attention weights as shown in Fig. a)), they minimally contribute the model due to their low attention weights. Thus, we can neglect them in the subsequent analysis.
4.2 ANALYSING THE CHARACTERISTICS OF VISUAL SINK TOKENS
As a next step, we analyse the characteristics of the visual sink tokens. Specifically, we conduct the experiments to validate that the visual sink tokens have analogous characteristics to the sink tokens in
5

---

Published as a conference paper at ICLR 2025
(4)
(b)
(c)
(c)
quaryyfmage
sinks
wro sinks
10"2
80-
Atterkon veights
nacklle
10
70-
F2
10-4
60-
chirattoon
50-
10
10
30
Sinkk dimension
5ix
Visual
Sink
Visual
Figure 3: Analysis of visual sink tokens. (a) Scatter plot of sink dimension values and attention weights of visual tokens. (b) Performance comparison between masking visual sink tokens and masking the same number of random visual tokens. Dashed line indicates the performance of the original model. (c) Average attention contributions of visual sink tokens and random visual tokens. (d) Visual attention maps with and without visual sink tokens, where the visual sink tokens are highlighted in red boxes.
language models. The sink token itself does not substantially affect the model's response (Kobayashi jet al., 2020 Bondarenko al., 2023] Yu et al., 2024 Gu et al., 2024). We validate whether the visual sink tokens also do not contribute to the model's output by (1) evaluating the performance of the model with the visual sink tokens masked and (2) measuring the mechanistic contribution of the visual sink tokens to the residual stream
Token Masking Experiment. To evaluate the impact of the visual sink tokens the model's output, we mask the attention from the visual sink tokens to the text tokens. This manipulation makes the model not receive any information from the visual sink tokens. As shown in Fig. b)) masking the visual sink tokens has little impact on the model's performance. In contrast, masking the same number of random visual tokens leads significant performance drop. This result demonstrates that visual sink tokens contribute negligibly to the model's 's responses.
Contribution Analysis. We further investigate the mechanistic contribution of the visual sink tokens to the residual stream. Concretely, we measure attention contributions (Kobayashi et al., 2020 Basu et al. 2024) from the visual sink tokens to the residual stream of the text tokens, which are computed w-
as || aa visual sink tokens exhibit significantly lower attention contributions the residual stream than the other visual tokens. We also qualitatively confirm that the irrelevant visual tokens are clearly filtered by the definition of visual sink tokens, as illustrated in Fig. B().
VoVI. ll. where i E Ina.j E Ivis (see Eq. for derivation). Fig. 3(c) shows that the
Further discussion on visual attention sink. In order to explore visual attention sinks more thor- oughly, we conduct further analysis the visual sink tokens and discuss their characteristics in Appendix A.2 Here, we summarize the key takeaways. (1) Visual sink tokens are mostly located in the background, which is less informative. This observation is similar to the findings in ViT (Darcet et al. 2023). Moreover, considering that attention sink in language models occurs in less semanti- cally meaningful tokens (e.g., tokens also resemble the findings in language models. (2) We discover that visual sink tokens exhibit massive activation at the same dimension Daink as text sink tokens. This evidence suggests that the formation of visual sink tokens and text sink tokens shares the same underlying mechanism inherited from the base language models. In summary, some visual tokens are less semantically meaningful, and LMMs treat them visual sink tokens, similar to the behavior of language models. We leave the investigation of how visual tokens are recognized as sink tokens during training as future work.
"(n") (Ferrando & Voita 2024 Yu et al. 2024). the visual sink
4.3 SURPLUS ATTENTIONS IN VISUAL ATTENTION SINK: CAN WE RECYCLE THEM?
Our experiment reveals that the visual sink tokens do not contribute the model's output, even though they have high attention weights. This motivates us to consider the attention weights to the sink tokens as free resources that be recycled shown that LMMs tend to insufficiently attend to the image compared to text, which possibly leads to suboptimal performance in vasion-language tasks (Chen et al.] 2024 Liu et al. 2024d). This problem can be alleviated by compensating the attention to the image from the attention budget.
"attention budget". Recent studies have
More Details on experimental settings descrihed in Appendix D.3

---

Published as a conference paper at ICLR 2025
Image and Questions
Heads high visual non-sink ratio
Heads w/ low visual non-sink ratio
Example Example
Is there skateboard this image?
Is there red ceat in the image?
Figure 4: Visualization of the attention heads sorted by visual non-sink ratio. We show some at- tention heads with high visual non-sink ratio (left) and low visual non-sink ratio (right). The attention heads with high visual non-sink ratio tend to focus on the visual tokens relevant to the corresponding text token. On the other hand, the attention heads with low visual non-sink ratio have vague attention patterns. The attention heads with high visual non-sink ratio are selected as the image-centric heads.
Furthermore, visual sink tokens can be utilized to calculate true image content. Although visual sink tokens receive high attention weights, they do not have semantic meaning related the correspond- ing text token. Inversely, visual non-sink tokens are closer to the true image content than the visual sink tokens. Hence, we can exploit the attention assigned to the visual non-sink tokens as a measure of how much the attention head focuses on the image. We apply this concept to select the specific attention heads that focus on the image in the subsequent section.
VISUAL ATTENTION REDISTRIBUTION
In this section, we introduce Visual Attention Redistribution (VAR), method to intensify the focus on the image of LMMs based on our discussion in Sec. Our method consists of steps: (1) selecting image-centric heads based on visual attention sink (Sec, 5.1) and (2) redistributing the attention budgets from sink tokens the visual non-sink tokens only in the selected heads (Sec. The overview of VAR is illustrated in Fig. 5
5.1 SELECTING IMAGE-CENTRIC HEADS
In Sec. propose that the insufficient attention weights to the image can be supplemented by redistributing the attention weights from the sink tokens. However, applying redistribution to all attention heads results in significant performance drop (see Table 4.. Given that each attention
head in the transformer possibly has a distinct role (Deiseroth et al. 2023| Zhang et al.) 2024a Ge al. 2024 Zheng et al. 2024), simply redistributing the attention weights all heads may ignore the role of some attention heads, whose function is not related with the interaction with the image. Therefore, selecting image-centric heads, which are responsible for focusing on the image, should proceed before redistributing the attention weights.
Visual attention sink can be utilized select image-centric heads. Since heads with low attention weights to the visual tokens are evidently not focusing on the image, we only consider the heads with high attention weights to the visual tokens. Specifically, for each layer f, we initially discard the heads whose sum of attention weights to the visual tokens is less than 0.2. After that, we incorporate visual attention sink to select image-centric heads. If the model allocates high attention weights to the visual sink tokens, the sum of attention weights to the visual tokens can be high even though the head does not focus on the image. Based on the discussion weights allocated to the visual non-sink tokens can indicate how much each attention head actually focuses on the important visual information. Therefore, following the notations in Sec. we define visual non-sink ratio r, as:
Sec. 4.3 the proportion of attention
f.h
Ejeza ai 2.b
f.h r
(3)
E. Ejera aig
where Tyis and I I denotes the of all visual tokens and the visual non-sink tokens, respec- tively. When visual non-sink ratio p is high, we can expect that the attention head h in layer focuses more on the important visual information.

---

Published as a conference paper at ICLR 2025
Sink tokens
KBOOO
Selected image centric head selected imaga--nnae cantarii
<8O
ERose
atlention weight increased
<ing>>
attertion weight decreased
A
<lmg>
Attention Hoad
simge
<irge
<impp
Zamp>
p x
simgo
<irgl
<1mg>
X
Ainge
<1mg>
<img>
<ingg
Kimg>
Fllmgo
<irgo
thore
saigg
Uhern
Attention Budget
Thvere
(a) Select image centric heads
(b) Redistribute attention weights
Figure 5: Overview of Visual Attention Redistribution (VAR). (a) We select image-centric heads by evaluating visual non-sink ratio; heads with r 2 p are chosen as image-centric heads. (b) VAR redistributes surplus attention weights from sink tokens visual non-sink tokens. The attention budget S2 accumulates a portion p of attention from sink tokens. Finally, visual non-sink tokens receive attention from 52.
Heads with high visual non-sink ratio focus on the important region. To validate the effective- ness of visual non-sink ratio Ti We sort and visualize the attention heads based on the visual non-sink ratio in Fig. 4 We find that the heads with high visual non-sink ratio are more likely to concentrate on the important visual tokens, which are related to a given text token. the other hand, the heads with low visual non-sink ratio exhibit sparse and scattered attention pattern to the various visual tokens. We select attention heads with A.h illustrates the selection process. Here, is a hyperparameter that controls the number of selected heads. We further investigate the characteristics of the image-centric heads Appendix A.3
P the image-centric heads. Fig. 5
5.2 REDISTRIBUTING ATTENTION WEIGHTS
After selecting the image-centric heads, we redistribute the attention weights from the sink tokens to the visual non-sink tokens in the selected heads. Fig. (])) illustrates the redistribution process. We first accumulate the portion p of the attention weights from the sink tokens into the attention budget Ω. The portion < < 1 controls the amount of attention weights to redistribute. For brevity, we will omit the superscript É, h hereafter. The attention weights of the sink tokens decrease to āi = p)
p) ai,j and attention budget is calculated as Ω p Σje ai.j.
Then, allocate the attention budget the visual non-sink tokens (i.c., j Ivis Ivis). Inspired by Yu et al. (2024), we redistribute the attention weights considering the relative importance of the visual tokens. After redistributing the attention weights, the attention weights to the visual non-sink tokens are updated as follows:
Ōi.j = 0 +
(4)
This method ensures that the sum the attention weights remains equal to after redistribution Eyes 0æ.j 1), thereby preserving the overall distribution. Note that the redistribution of attention weights is applied to all text tokens Ipt. including the instructions and the generated responses.
EXPERIMENTS
6.1 EXPERIMENTAL SETTINGS
Model settings. As our method only modifies the attention of the LMM. VAR can be simply applied various LMMs without additional training, models, inference steps. We employ L.LVA--15-7BB
8

---

Published as a conference paper at ICLR 2025
Table I: Benchmark results on general vision-language task.
Model
VQA
GQA
VizWiz
SQA'
VQA
MME
MMB
SEED
LLaVA
MM-Vet
LLaVA-I.5-7B
78.5
62.0
50.0
66.8
58.2
1495.5
64.3
58.6
65,4
31.1
+ Ours
78.6
63.5
53.7
67.3
58.6
1513.8
65.1
60.7
68.1
33.7
LLaVA-1.5-13B
80.0
63.3
53.6
71.6
61.3
1501.2
67.7
61.6
72.5
36.1
+ Ours
81.2
64.9
57.2
72.2
62.1
1534.3
68.1
62.3
74.1
38.4
LLaVA-I.5-DD-13B
81.8
64.7
57.5
71.0
62.5
1500.1
68.8
62.6
72.0
39.4
+ Ours
82.0
65.1
58.8
71.3
63.0
1505.2
69.5
63.3
73.8
40.6
VILA-I3B
80.8
63.3
60.6
73.7
66.6
1507.1
70.3
62.8
73.0
38.8
+ Ours
81.2
63.6
64.2
74.7
67.3
1512.7
71.7
63.0
75.7
39.7
Twen2-VL-7B
82.5
65.4
74.1
84.3
1672.3
83.0
77.9
75.6
63.2
+ Ours
82.8
64.7
67.7
74.2
84.9
1688.5
83.3
78.1
77.3
63.5
ILterVL2--8B
82.0
63.2
63.0
74.2
77.3
1648.
81.7
76.2
73.2
60.0
+ Ours
82.5
63.5
65.1
74.7
78.0
1655.4
82.3
77.1
75.1
61.2
Table 2: Benchmark results on visual hallucination
Table 3:
Benchmark results on vision-
task.
centric task.
Model
CHAIR
POPE (all)
MMHlal
Model
MMVP
CV-Bench CV-Bench"
4s
C₁
Acc.1
Scoret
Hall
LLaVA-L5-7B
3.33
56.8
58.4
LLaVA-1.5-BB
45.0
14.7
85.90
84.76
2.36
51.0
Ours
9.33
57.6
59.0
+ Ours
43.2
13.8
86.53
85.87
4.26
45.1
LLaVA-13--3B
24.7
58.2
58.4
LLAVA-L.--1B
20.6
6.2
85.47
44.3
+ Ours
28.0
59.6
59,9
+ Ours
17.3
5.1
86.12
86.58
4.38
42.7
LLaVA-I5-HD-I3BB
36.0
62.7
65.7
LL-VA-B55-HD-138
42.9
13.2
87.1
85.0
2.35
43.7
+ Ours
39.1
63.8
66.8
+ Ours
40.6
12.8
87.7
87,9
4.15
43.1
VILA-13B
23.1
58.6
57.9
VILA-I3B
31.0
8.8
84.2
83.58
2.40
44.7
29.7
8.0
85.1
85.4
42.9
Ours
28.6
59.8
Qwen2-VI--R
30.5
8.4
87.0
87.5
2.41
44.1
Qwen2-VL.T VL-7B
52.1
63.5
67.6
88.2
Ours
55.6
63.6
67.7
V1.25V
32.4
9.7
86.9
87.8
2.45
43.9
Intern InternVL.2-8 VL.2-8
51.3
61.8
66.8
+ Ours
31.8
9.3
87.5
88.6
4.17
42.7
+ Ours
56.7
62.1
67.1
LLaVA-L.5-13B. LLaVA-I.5-HD-I3B (Liu et al. 2024a),, VILA-13B (Lin et al. 2024). Qwen2-VL. 7B (Wang al., 2024). and Intern VL2-8B (Team 2024 our base models.
Tasks and benchmarks. We evaluate our method on a wide range vision-language benchmarks. The benchmarks are divided into three categories: general vision- lision-language task, visual hallucination task, and vision-centric task. (1) General vision-languag task assesses comprehensive multimodal capabilities of LMMs. We compare method with the hase models across 10 benchmarks. (2) Visual hallucination task evaluates whether the response of the model is consistent with the image content to ensure the trustworthiness and reliability of the model. We use CHAIR (Rohrbach al.) 2018), POPE (Li et al. (023c)), and MMHal-Bench (Sun et al. 2023). (3) Vision-centric task evalu- ates visual understanding capabilities, such as determining the spatial relationship between objects in the image. We use MMVP (Tong et al., (2024b), CV-Bench2D, and CV- CV-Bench3D (Tong et al.) 2024ah. More details on the tasks and benchmarks are provided in the Appendix D.I
Implementation details. We use the same hyperparameters for all the benchmarks in the same task type. We set = 20 and p 0.6 for all experimental settings in our experiments. p is set to 0.8 for general vision-language task in Table [ 0.5 for visual hallucination task in Table 2 and 0.9 for vision-centric task in Table We do not modify the attention heads in the last layer, as the last layer
is considered to have a specialized role (Lad et al. 2024 Sun et al. 2024b).
6.2 MAIN RESULTS
The experimental results on general vision-language task, visual hallucination task, and vision- centric task are presented in Table [ and respectively. VAR reliably improves the performance of the base models across all benchmarks. Despite the diverse characteristies and evaluation set- tings of the benchmarks, VAR demonstrates robust performance without specific hyperparameter tuning for each benchmark. Specifically, Table indicates that VAR effectively mitigates visual hallucination across all benchmarks, and Table demonstrates that complex visual understanding capabilities can be enhanced by editing the attention mechanism the LMMs. is worth noting that LLaVA-1.5-7B with VAR outperforms vanilla LLaVA-1.5-133 POPE, suggesting that there is enough margin to improve the performance by simply enhancing the focus the image without increasing the model size.
GQA, VizWiz, MME, and

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study on selecting image-centric
Table 5: Ablation study on redistributing
heads.
attention weights.
POPE
MME
MM-Vet
Setting
FI
POPE
Acc.
MME
Score
MM-Vet
Score
Setting
F1
Acc.
Score
Score
Baseline
85.9
84.8
1495.5
31.1
Baseline
85.9
84,8
1495.5
Visual Text
86.2
85.3
15039
33.5
w/o Hend selection
0.0
0.0
0.0
0.0
Text
70.3
70.0
1497.3
26.7
w/ Head selection (Ours)
86.5
85,9
1513.8
33.7
Visual (Ours)
86.5
85.9
1513.
33.7
Phil's
Question: the logo "phil's market"?
LLaVA:
LLaVA Ours:
MARKET
No. the word the logo not "phil's market".
Yes,
the logo is "phil's market".
Figure 6: Qualitative analysis of VAR. Visual attention maps before and after applying VAR indi- cate that VAR intensifies the focus the key visual tokens related to the corresponding text token. As a result, the model can generate more accurate responses by seeing the image more effectively.
6.3 ANALYSES AND DISCUSSIONS
Step-wise ablation studies. We conduct ablation studies to investigate the effectiveness of each step of VAR on LLaVA-1.5-7B, We validate the necessity of (1) selecting image-centric heads and (2) redistributing attention weights to visual tokens. First, we compare the model's performance when selecting the image-centric heads versus when not selecting them in Table 4 The result shows that redistributing attention weights across all heads may severely impair the functions of some attention heads, leaving the model unable to generate responses (i.e., 0.0 scores w/o head selection in Table As we discussed in Sec. 5.1 head selection is crucial for performance improvement. Second, we compare the model's 's performance when redistributing the attention budget among both visual and text tokens, only to text tokens, and only to visual tokens (i.e., VAR) in Table Redistributing to text tokens alone yields little improvements, and in some cases, a performance decrease, as the model already sufficiently attends to text tokens. While redistribution to both tokens slightly improves the model's 'ss performance, redistribution to visual tokens is the most effective. The result confirms that our method effectively supplements insufficient visual attention by redistributing the attention budget only to the visual tokens.
Hyperparameters. We explore the impact of the hyperparameters p. and p performance. Be- low, we outline the key findings, with more detailed results provided in Appendix B.2 (1) VAR is robust to the choice of hyperparameters, Performance consistently improves across all tasks within a reasonable range of hyperparameter values. (2) The optimal values of T and are consistent across all tasks. Therefore, single value for and p be applied to all tasks. (3) While the best p value varies across tasks, we demonstrate that the optimal o remains consistent across different bench- marks within the same task type. Based this finding, set single p value for all benchmarks of the same task type.
Discussion.
We qualitatively analyse the visual attention maps of the base model and VAR in Fig. 6
We observe that VAR effectively redistributes the attention budget to the image, enabling the model
to sufficiently focus the key visual tokens. More qualitative results provided in Appendix E Furthermore, our method can be seamlessly incorporated into existing approaches that enhance the performance of LMMs, such as visual contrastive decoding (VCD) (Leng al., (2024). We provide the experimental results with VCD in Appendix B.I Overall results provide the evidence that steer- ing the inner attention mechanism is effective way to improve the multimodal capability of LMMs.
7 CONCLUSION
This paper discovers the properties and characteristics of visual attention sink in LMMs, demon- strating that the model consistently attends to irrelevant parts of the image. Furthermore, we propose Visual Attention Redistribution (VAR) to emphasize the visual information relevant to the corre- sponding text tokens by recycling the surplus attention budget from the visual sink tokens. Exper- imental results imply that LMMS can do better in seeing the image by solely editing the attention map. We hope that our work can contribute understanding the attention mechanism in LMMs and suggest a new direction for improving the multimodal capabilities of LMMs.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We ensure reproducibility by providing detailed explanation of the experimental setup, presenting additional analysis, conducting further experiments on our methods, and offering a comprehensive summary of baseline specifications, all of which are included in Appendix. Our code is included in the supplementary material and all the source codes will be made available to the public.
ACKNOWLEDGEMENTS
This work was supported in part by the IITP RS-2024-00457882 (AI Research Hub Project), NRF RS-2024-003458866 and NRF RS-2023-00219019.
REFERENCES
Estelle Aflalo, Meng Du, Shao-Yer Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. VI-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21406-21411 21415, June 2022.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pix- tral 12b. arXiv preprint arXiv:2410.07072. 2024.
Jean-Baptist Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406.127188 2024.
Kazi Hasan Ibn Arif, Jin Yoon, Dimitrios S. Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource-constrained environments, 2024. URL httttt:/p//axiii
oor/ /bs//bb/// 2408 10945
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision-language model for understanding, local- ization, reading, and beyond. 2023.
Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Mas- siceti. Understanding information storage and transfer in multi-modal large language models. arXiv preprini arXiv:2406.0422 04236, 2024.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Remov- ing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 36:75077-75066. 2023.
Nicola Cancedda. Spectral filters, dark signals, and attention arXiv:2402.092221, 2024.
sinks.
arXiv preprint
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer Plug-and-play inference acceleration for large vision- language models. arXiv preprini arXiv:2403.0611 2024.
Timothée Darcet. Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint 2309 16588, 2023.

---

Published as a conference paper at ICLR 2025
Bjöm Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, and Kristian Kersting. Atman: Understanding transformer predictions through memory efficient attention manipulation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 63437-63460. Curran Associates, Inc., 2023. URL
Confereconfe
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Gan- guli, Zac Hatfell-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination con- trol by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14302, 2024.
Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language mod- els at scale. arXiv preprint arXiv:2403.00824 2024.
Suyu Ge, Yunan Zhang, Liyunn Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024. URLhttps:/ pttptttp:s//ppeeevieiee /pwpreviiewweeeeee/e/ee net forum?
IdduNrFpDPMyo
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6911 2017
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.1077 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
models. In Pro-
Danna Gurari, Qing Li. Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5508-3617, 2018.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? deep-dive into the vision of vision language models. arXiv preprint arXiv:2411. 2024.
Goro Kobayashi, Tatsuki Kuribayashi. Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. arXiv preprint arXiv:2004. 10102, 2020.
12

---

Published as a conference paper at ICLR 2025
Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of Ilms: Stages of infer- ence?, 2024. wttp/ttps///axii..rr////////////
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- lion, pp. 13872-13882, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision vision-language models. arXiv preprint arXiv:2305.1088 2023c.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 26699, 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Eetin-ECCVV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Pro- ceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
attention image: A training-free free method Shi Liu, Kecheng Zheng, and Wei Chen. Paying arXiv:2407.2177), 2024d.
for alleviating hallucination in Ivims. arXiv preprint
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model all-around player? arXiv preprint ar arXi::2307.0688) 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpret- ing visual information processing in vision-language models. arXiv preprint arXiv:2410.07149, 2024.
OpenAl. Gpt-4 technical report.
2023.
URL
Corpus :
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 7748-8763. PMLR, 2021.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint vrXiv:180..02556 2018.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European con- ference computer vision, pp. 166-162. Springer. 2022.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Gabriela Ben Melech Stan, Estelle Aflalo, Raanan Yehezkel Rohekar, Anahita Bhiwandiwalla, Shao- Yen Tseng, Matthew Lyle Olson, Yaniv Gurwicz, Chenfei Wu, Nan Duan, and Vasudev Lal. Lylm-interpret: An interpretability tool for large vision-language models, 2024. URL
03118
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:240..17762 2024a.
Qi Sun, Mare Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers as painters, 2024b. URL https://azxii..oomm www.v//.wgg.b//2....oooo 09298
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yau Yan Gui, Yu-Xiong Wang. Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf. 2023.
OpenGVLabb Team. Internv12, July 2024. URL httpps:/internvll.gg.
2024-07-02 1nt77/vnten
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya lyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal lims. arXiv preprint arXiv:2406. 16860, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092888 2023.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U, Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/prrceeet.nnne inw.riiidddin.n..nnr..e......... www.nneuriicc.mm www.neiiss......errrrrr..e..e files/paper/2017.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing rision-languae model's perception of the world at any resolution, 2024. URL https:////// arxiv org/ abs/2409. 12191
14

---

Published as a conference paper at ICLR 2025
Sangmin Woo, Donguk Kim, Jachyuk Jang, Yubin Choi, and Changick Kim. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405.17820, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 1067,,, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.024444 2023.
Zhongzhi Yu, Zheng Wang. Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Un- veiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406.1177 157655 2024.
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for LLMs. In The Twelfth Interna- fional Conference Learning Representations, 2024a. URL https: / rewieeeeeeweeeeeeemeeeeeeeeeewee net./
Forum X2DW0oeeDD
Yuechen Zhang, Shengju Qian, Bohao Peng, Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for multi-modal llms. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13215-13224, 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 66:46595-46661 2023.
Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Bo Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey, arXiv preprint 2rX1::2409.03752 2024.
Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. hallucinations large vision-language models via image-biased decoding. arXiv:240..18476, 2024.
lbd: Alleviating
arXiv preprint
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL ANALYSIS
A.1
MASSIVE ACTIVATION OF SINK DIMENSIONS
LLaVA-1.5-13B
LLaVA-1.5HHD-13B
Qwen2-VL-7B
InterVVL-888
60
Ka05>
40
40
40
20
20
20
20
2100
47463
2100
4743
458
2070
3584
0
3084
60
4mmg
60
60
60
60
40
40
40
40
simg
20
20
20
20
2100
4745
2100
4745
458
2570
5586
2584
Dimension
Dimension
Dimension
Dimension
Figure 7: The hidden states of BOS, visual sink tokens, other visual tokens in various LMMs. The hidden states of visual sink tokens have massive activation in the same dimensions with the BOS token.
In this section, we demonstrate that sink dimensions D sink visual sink tokens are identical to those of sink tokens in the language models. Based [Snn al. (2024a), sink tokens have the massive activation in very few fixed dimensions. These dimensions are consistent across different layers and heads and agnostic to the input tokens. Also, the dimensions of massive activation are identical before and after the fine-tuning process. The reason is that the massive activation arises as an innate property during the pre-training process (Darcet et al. 2023 Gu et 2024
We find that the statement is also valid for LMMs. For example, the base LLM of LLaVA-1.5- 7B (Liu et al.] (2024a) is Vicuna-1.5-78 1.5-BB (Zheng et al.] 2023), which has been fine-tuned from LLaMA2-7B (Touvron et al. 2023). Sun et al. (2024a) reports that the dimensions of massive activa- tion LLaMA2-7B is Sink = (1415, 2533]. Even though the model fine-tuned with multimodal data, the dimensions of massive activation in LLaVA-1.55
are still Diink {1415, 25233,, as We also check that this observation is consistent across various LMMss in our exper- iments. Therefore, set Dsink based on the pre-trained large language model of L.Ms,, following the previous work Sun et al. (2024a). Specifically, we use Dgink = {1415, 2533) } for LLLVV--155777 1B5-BB, sink {2100, 4 for LLaVA- 1.AA-1.5-133 1.5BBB, LLaVA-1 1.55HDD-11111 1-..V---------.
shown in Fig. 2
and VILA-13B (Lin al. 2024).
Dsink 458, 2570} for Qwen2-VL-7B (Wang et al. 2024). and Dsink {3584] for Intern VL2-
8B (Team 2024
Similar to Fig. we observe that the visual sink tokens also have massive activation in the same dimensions with BOS token, shown in Fig. This observation stands for various LMMs with different architectures, training schemes, and scales. The massive activation in the sink tokens is an innate property of the LMMs, and the visual sink tokens exhibit the same pattern as the sink tokens in the language models. The result suggests that the visual sink tokens share similar characteristics with the sink tokens in the language models.
A.2 FURTHER ANALYSES AND DISCUSSIONS ON VISUAL ATTENTION SINK
We investigate hidden states of visual sink tokens and analyse the characteristics of visual sink tokens in LMMs in Sec. and Appendix(A.1 [ We discover that the visual sink tokens also exhibit a similar pattern as the sink tokens in the language models. This section provides further analyses and discussions on visual attention sink.
Location of visual sink tokens. In Sec. 4 argue that visual sink tokens unrelated to the main subject of the image. We validate this argument using large-scale segmentation datasets, such Pascal-VOC (Everingham al. 2010) and MS-COCO (Lin al. 2014). In the segmentation
16

---

Published as a conference paper at ICLR 2025
Table 6: The proportion of visual sink tokens and random visual tokens located in the back- ground regions. Visual sink tokens are more likely to be located in the background regions com- pared randomly selected visual tokens. The experiment conducted on LLaVA-115-7B.
Token Type
Pascal-VOC
MS-COCO
Visual Sink Tokens
90.5%
93.7%
All Visual Tokens
82.9%
90.5%
60
60
60
Number of visual sink tokens
KBOS>
1415
2533
4546
14155
2533
0046
1412
2532
1146
60
60
3img2
30
20
20
2033
4006
30
35
Layer 0
Layer
Layer 20
Layer
Figure 8: Layer-wise analysis of visual sink tokens. (a) Layer-wise hidden states of visual sink to- kens. The massive activation of hidden states emerges the early layers of LMMs. The phenomenon is consistent with the sink tokens in the language models. (b) Number of visual sink tokens in each layer. The visual sink tokens emerge in the early layers and persist until the last layer. The experi- ment is conducted on LLa 1.5--B and MME Dataset.
data, treat all regions except the main object background. After obtaining the visual attention map between object text tokens and visual tokens, we calculate the proportion of visual sink tokens located within the background regions. We compare the portion of visual sink tokens that not related to the main subject of the image with the portion of all visual tokens that are not related the main subject of the image, as shown in Table 6 The results indicate that the visual sink tokens are more likely to be located in the background regions than other visual tokens, supporting the conclusion that visual sink tokens are unrelated to the main subject of the image. Furthermore, considering that the background regions are semantically less meaningful, the visual sink tokens are similar to the sink tokens ViT (Darcet al. 2023) and language models (Ferrando & Voita 2024] Xiao et al.] 2023; Yu et al. 2024). Specifically. sink tokens in VIT are located in the background regions, and sink tokens in language models have limited semantic meanings (e.g., 'BOS', ".',' etc.)
Layer-wise analysis. To investigate when visual sink tokens emerge in the LMMs we conduct a layer-wise analysis of visual sink tokens. We find that massive activation of hidden states emerges in the early layers of LMMs. For example, as shown in Fig. massive activation emerges in layer 2 of L.LVA-I15-7BB We also count the number of visual sink tokens in each layer. The visual sink tokens emerge in the early layers and persist until the last layer. This result is consistent with the previous work on sink tokens in the language models (Sun al. 2024a). Cancedda (2024) shows that feed- forward network (FFN) in the early layers is responsible for the attention sink of BOS token in the language models. Early FFNs may write massive activation the hidden states of sink tokens, and the massive activation can be preserved in the subsequent layers due to the residual connections. Our finding suggests that the massive activation of the visual sink token shares similar causes with the sink token in the language models.
Further discussions and future works. Considering the results obtained so far, visual sink to- kens are semantically less meaningful and are located in the background regions. These tokens get massive activation from the early layers, resembling the behavior of BOS token. However, the mechanisms behind how LMMs select visual sink tokens and how visual attention sinks emerge during the training process remain open questions. In vision models and language models, sink to- kens emerge result of enough training massive training data (Darcet et al., 2023 Gu et al.) 2024). Since multimodal models are built upon pre-trained language models, the visual attention sink phenomenon in multimodal models more likely inherited from the language models. Sup-
17

---

Published as a conference paper at ICLR 2025
(a)
(b)
80
80
F1 Score
70
F1 Score
70
60
60
ICH
ICH 1
50
Random
Random
0.0
0.2
P
Figure 9: Ablation study on the importance of image-centric heads (ICHs). We evaluate the performance of LLaVA-1155775 with different p on POPE benchmark. (a) The performance drops more significantly when ablating the image-centrie heads than when ablating the random heads. (b) On the other hand, the performance remains relatively stable when ablating the complementary heads.
porting evidence for this is that the sink dimensions in multimodal models are identical to those their base language models, as discussed in Appendix [..] Therefore. investigating how LMMs identify certain visual tokens as visual sink tokens during the multimodal training process could be an intriguing direction for future research.
A.3 ANALYSIS ON IMAGE-CENTRIC HEADS
In Sec. 5.1 select image-centric heads based on r and utilize them to emphasize the important visual information in LMMs. In this section, we explore the characteristics of image-centric heads.
Image-centric heads crucial for visual information processing of LMMs. We validate the importance of image-centric heads (ICHs) by conducting an ablation study. Specifically, we adjust p, which hyperparameter that controls the number of selected heads. is high, the threshold for selecting the image-centric heads is high and the number of selected heads is low. To evaluate the contribution of the image-centric heads, blind the visual information in the image-centric heads. Specifically, we mask the attention from the visual tokens to the text tokens in the centecc head (e, h), using setting similar that described in Sec, 4.2 We compare the performance degradation when ablating the image-centric head versus when ablating random heads. We also evaluate the performance degradation when ablating the complementary heads, which are the heads that are not selected as the image-centric heads.
The results are shown in Fig. 9 In Fig. I(a,, the performance drops more significantly when ablating the image-centric heads than when ablating the random heads for each p. The result indicates that the image-centric heads are crucial for processing visual information in LMMs.. In Fig. b)). when ablating the complementary heads, the performance remains relatively stable throughout different p. This result suggests that the other heads are not responsible for processing visual information. The results demonstrate that the image-centric heads selected by visual non-sink ratio r are indeed essential for integrating visual information into the text responses of LMMs.
Text tokens related to visual information have more image-centric heads. Furthermore, we in- vestigate the relationship between the text tokens and the image-centric heads. We note that the number of selected heads are not fixed for all text tokens. We calculate the portion of image-centric heads for each text token. We find that the text tokens related with visual information have more image-centric heads. For example, shown in Fig. [0] while Sub (the subword token of "Subway') has more image-centric heads than in. The reason is that the text token Sub requires visual infor- mation to generate the text token, while the text token in can be inferred from the question without visual information. This result suggests that the image-centric heads are dynamically selected based on the extent which the text token requires visual information. Therefore, VAR emphasizes im- portant visual information only when is required by the text token, considering the context of the text token.
18

---

Published as a conference paper at ICLR 2025
SERIES
Example
10%
5%
Question: What brand featured this advertisement?
0%
brand
festured
anvert
Figure 10: Relationship between text tokens and image-centric heads. We calculate the portion of image-centric heads for each text token. Text tokens related with visual information have more image-centric heads. The experiment is conducted LLaVA-1.5-7B and LLaVA-Bench (In-the- Wild).
B MORE EXPERIMENTS ON VISUAL ATTENTION REDISTRIBUTION
B. VISUAL CONTRASTIVE DECODING: A CASE STUDY ON INCORPORATING VISUAL ATTENTION REDISTRIBUTION INTO OTHER METHOΟS
Table Evaluation results of VCD with VAR.
Methods
POPE
MME
GQA
Bascline
85.9
1495.5
62.0
+ VCD
87.3
1380.7
66.2
+ VCD VAR
87.6
1597.2
67.1
In Sec. 6.3
we discuss the potential of applying VAR to other methods which enhance the perfor- mance of LMMs in other directions. In this section, provide into the Visual Contrastive Decoding (VCD) (Leng et al.) 2024). VCD mitigates object hallucina- tions in LMMs through contrastive decoding. They require input and a model with distorted visual input. By contrasting the outputs of the two models, VCD effectively emphasizes the visual information and reduces the object hallucinations,
study incorporating VAR
models: model with original visual
Both our method and VCD aim to enhance the focus on the visual information LMMs. However, VCD requires models and additional inference steps, while VAR can be applied model without additional inference steps. Also, we can incorporate VAR into VCD to further en- hance the focus on the visual information. Specifically, we contrast the outputs of the model with VAR and the model with distorted visual input. By contrasting the outputs, we expect that two meth- ods can complement each other and further improve the performance LMMs.
single
We evaluate the performance of VCD with VAR on the MME Yin et al. 2023 POPE LLi et al. 2023c), and GQA (Hudson Manning, 2019) benchmarks. The experiment conducted on LLa 1.5-7B. As shown in Table VCD with VAR improves the performance of VCD on all benchmarks. The results suggest that VAR can be incorporated into various methods to enhance the focus on the visual information in LMMs and further improve the comprehensive performance of LMMss.
B.2 HYPERPARAMETER ANALYSIS
this section, we investigate the impact of hyperparameters on the performance of VAR. The hyper- parameters include p, and p. First, randomly sample 10% of the samples from the benchmark and use the partial samples as "pseudo-validation set" to determine hyperparameters. Second, we apply these hyperparameters to the 100% of the samples in the benchmark and evaluate the im- pact of the hyperparameters on the performance. The results show that VAR works robustly across reasonable ranges of hyperparameters.
19

---

Published as conference paper at ICLR 2025
58.6
58.6
58.6
General Vision- Language Task
251
151
151
58.3
58.3
58.3
58.0
38.0
145
TeXtVOA
57.7
245
MME TeXIVOA
57.7
145
MME TextViA
0.0
0.2
0.6
1.0
0.0
0.6
1.0
Visual Hallucination
86.4-
56
86.4
56
86.4
56
Teak
86.0
55
86.0
55
86.0
55
POPE CHAIRS
POPE CHMiRs
POPE CHAIRS
54
54
34
0
20
40
50
60
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
50.0
59.0
59.0
Vision-centri conn-centiic
57-
57-
57-
Task
58.5
58.5
58.5
56-
CV-Bench-20
56-
CV-Benchi2D
56-
CV-Bench-20
CV-Bench-3D
58.0
CW-Bench3D
58.0
CV-Bench-3D
58.0
20
0.0
0.2
0.6
1.0
0.0
0.6
1.0
Figure 11: Determining hy hyperparameters using 10% of the samples in the benchmark. We evaluate LLAV.-155-7B on the benchmarks with 10% of the samples and determine the hyperpa- rameters. The same optimal hyperparameter values can be obtained from different benchmarks in the same task type.
Determination of hyperparameters. Since most benchmarks lack training validation sets, alternatively determine hyperparameters using a single benchmark per task and apply these hyper- parameters across benchmarks within the same task type. To mitigate the risk of overfitting to spe- cific benchmarks, we allocate a random 10% of the samples as secudo-validation set for hyperpa- rameter tuning. Additionally, we use a different benchmark within the same task type validate the consistency of the hyperparameters across various benchmarks. We also apply another benchmark in the same task type to determine hyperparameters for validating the consistency of the hyperparame- ters across different benchmarks. Specifically, we use MME (Yin et al. 2023) and TextVQA (Singh) et al. 2019) for the general vision-language task, POPE (Li et al. 2023c) and CHAIR (Rohrbach 2018) for the visual hallueination task, and CV-Bench2D and CV-Bench3D (Tong et al.
2024a) for the vision-centric task. Note that for CHAIR, we report CHAIRs, as a lower score indicates better performance in this benchmark.
The results are shown in Fig. 11 The results show that the same optimal hyperparameter values can be obtained from different benchmarks in the same task type. Especially, the optimal values of 7 and are consistent across all tasks, reducing the need for specific hyperparameter settings. Although the optimal P value varies across tasks, the optimal P remains consistent across different benchmarks within the same task type. The results indicate that the hyperparameters are not overfitted to the specific benchmark and VAR is applicable to various tasks with minimal tuning. Based on the results, we set 20 and p 0.6 for all experimental settings in our experiments. is set to 0.8 for the general vasion-language task, 0.5 for the visual hallucination task, and 0.9 for the vision-centrie task. We use the
value for all the benchmarks in the same task type.
Impact of hyperparameters. We evaluate the impact of hyperparameters on the performance of VAR using MME (Yin al. 2023), POPE (Li et al.) 2023c), and CV-Bench (Tong al.) 2024a) as representative benchmarks for the general Dision-language task, visual hallucination task, and vision-centric task, respectively. The results, shown in Fig. 12 consistent with the results ob- tained from the pseudo-valiin ascudo-validation set. Below, we discuss the individual impact of т, P. and p on performance in detail.
20

---

Published as a conference paper at ICLR 2025
General Vision-
Languagee Task
1505
1505
1500
1500
1500
1.0
Visual Hallucination
86.4
86.4
86.4
Tatkk
86.2
POPE
POPE
POPE
58.2-
Vislon-centric
Task
57.8
57.8
57.8
$7.6
57.6
57.5
CV-Bench
CV-Beich
CV-Bench
10
20
30
50
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 12: Impact of hyperparameters the performance of VAR. We evaluate LLAVA-1.5- 7B on MME (Yin et al., 2023). POPE (Li et al., (023c), and averaged CV-Bench 2D+3D Tong al. 2024a) with different hyperparameters: The results show that VAR is robust the choice of nyperparameters, and the performance consistently improves the baseline across all tasks within reasonable range of hyperparameter values.
Impact of т. Hyperparameter is the threshold for selecting sink tokens. We obtain the best performance when т = 20. This result is consistent with Sec. 4.1 that the irrelevant visual tokens and relevant visual tokens clearly separated when 20.
Impact of Hyperparameter is the threshold for selecting image-centric heads. We only find optimal p for one benchmark per task. The suitable p for each task can vary because the optimal degree of head attention required for each task is different. For example, in the visual hallucination task, the model may need to focus more on the image to reduce hal- lucination, so the p is general vision- langual tasks require a balance between text processing and image under- standing, the p the reasonable range of p values, the proposed method robustly improves the performance of various LMMs on various benchmarks. For example, 2 0.5, VAR can consistently improve the performance.
lower value select more image-centric centrec heads. In contrast,
higher value to select fewer image-centric heads. However, in
Impact of p. Hyperparameter is the portion the attention weights from the sink tokens into the attention budget. We find that p 0.6 the best value for all benchmarks, but different values of can improve the performance, indicating that VAR is robust to the choice of p.
Overall, VAR demonstrates robustness to the selection of hyperparameters. Performance shows con- sistent improvement across all tasks within reasonable range of hyperparameter values, indicating that VAR is not highly sensitive to specific hyperparameter choices. This suggests that VAR can enhance the general performance of L.Mss even when hyperparameters vary within an plausible range, providing flexibility in its application across different tasks.
21

---

Published as a conference paper at ICLR 2025
C LIMITATIONS AND FUTURE WORKS
In this study, we utilize raw visual attention map without post-processing in our method. However, the raw attention map may not be perfect for grounding the object because the model may have seen the object incorrectly (Guan et al. 2024) or the visual encoder may misinterpret the image (Tong jet al. 2024b). We can further refine the visual attention map by investigating the localization of the object. Also, although we conduct extensive experiments
various LMMs and benchmarks, it is still worth exploring our methodology on different and larger LMMs (Li et al. 2024 Agrawal et al, 2024). Visual sink tokens can be utilized for various applications. For example, masking visual sink tokens removes noise and produces more reliable and interpretable visual attention map. Moreover, free attention budget from sink tokens be exploited emphasize the specific part of the image, depending on the user's interest. In addition, as discussed in Sec. [4.] understanding how LMMs recognize particular visual tokens as visual sink tokens during multimodal training could be an intriguing direction for future research.
D EXPERIMENTAL DETAILS
D.1 DETAILS OF BENCHMARKS
General Aision-language task. In Table abbreviations of the benchmarks are used for brevity. We use the following benchmarks for general rision-language task:
VQAV2 Goyal et al.
2017
Visual question answering v2.0.
GQA (Hudson & Manning.
2019): Question answering on image scene graphs. We used set splits for the evaluation.
VizWiz (Gurai et al. 2018): We used val set splits for the evaluation.
SQA' (u et al. 2022): y ScienceQA is dataset collected from elementary and high school science curricula, consisting of 21,208 multimodal multiple-choice science questions. Out of these, 10,332 questions include image context, 10,220 include text context, and 6,532 include both. Most questions are annotated with lectures (17,795) and detailed explanations (19,184) to provide general knowledge and specific reasoning for the correct answers. The dataset spans three subjects: natural science, language science, and social science, and organized into 26 topics, 127 categories, and 379 skills.
VQAT (Singh et al.
2019): TextVQA. We used val set splits for the evaluation.
Yin et al. 2023): MME serves thorough evaluation benchmark for large mul- timodal models, assessing both their perception and reasoning capabilities. covers 14 different tasks, including object existence, counting, spatial position, color recognition, poster identification, celebrity recognition, scene understanding, landmark detection, art- work recognition, OCR, commonsense reasoning, numerical calculations, text translation, and code reasoning. We evaluated the model's performance specifically on the perception tasks.
MME
MMB (Liu al. 2023 MMBenchhengllshh MM-Bench consists of around 3,000 multiple-choice questions designed to assess 20 different ability dimensions. Each question has a single correct answer. For evaluation, GPT-4 is used to match the model's prediction with the answer choices, outputting label (A, B, C. D) as the final prediction. We use the circular evaluation strategy to test if vision-language model can successfully solve each single problem. In this paper, we use only the English subset for evaluating our method.
SEED-Bench shows that current multimodal large language models (MLLMs) struggle with understanding spatial relationships between objects, even though they perform well on tasks involving overall image comprehension. This benchmark evaluates models on both images and videos using multiple-choie ques- tions, but focused only the image- section for our evaluation.
SEED' (Li al. 2023a): SLEED-Bench-imardd
WLaVAW (Liu et al. 2024cl: LLaVA-Bench (In-the-Wild) consists of 24 images paired with 60 questions, covering a variety of contexts such as indoor and outdoor environments, memes, paintings, and sketches. The dataset is designed to evaluate how well LMMs handle
22

---

Published as a conference paper at ICLR 2025
more complex tasks and adapt to unfamiliar domains. We perform case studies on this dataset to qualitatively showcase the effectiveness of our proposed VAR method.
MM-Vet (Yu al. 2023 MM-Vet assesses a model's ability engage in visual conversa- tions across various tasks, evaluating both the accuracy and helpfulness of responses using GPT-4. The dataset includes 200 images and 218 questions, each paired with corresponding ground truth answers.
Visual hallucination task. We use the following benchmarks in Table to evaluate the visual hal- lucination performance of the model.
CHAIR (Rohrbach al. 2018): We evaluate algorithm both captioning and visual question answering (VQA) benchmarks. For captioning, we measure object hallucinations using the CHAIR (Captioning Hallucination Assessment with Image Relevance) metries. These metries compare the objects mentioned in the captions with those actually present in the image assess caption quality. CHAIR has two versions: one calculates the percentage of hallucinated objects in the entire caption (CHAIR,), while the other measures the per- centage of captions that contain least one hallucinated object (CHAIRs). CHAIR, and cHAIRs are expressed by the following equations:
# hallucinated objects
# hallucinated captions
CHAIR,
CHAIRs
(5)
# generated objects
# generated captions
POPE (Li et al. 2023c): Polling-based Object Probing Evaluation (POPE) utilizes data
from MSCOCO Lin et al. 2). A A-OKVQA Schwenk et al. (2022). and GQA Hudson & Manning (019). generating 27,000
pairs from 500 images per dataset. This benchmark offers a streamlined approach to assess object hallucination by querying large multimodal models (LMMs) the presence of specific objects in images. Queries are equally divided between existent and non-existent objects (50% each). Negative sam- ples are selected through three strategies: random, popular, and adversarial. In the random setting, missing objects are chosen randomly; in the popular setting, they are picked from a pool of frequently occurring objects; and in the adversarial setting, co-occurring but absent objects are prioritized. Each image is associated with 6 questions, and the evaluation relies on four key metrics: Accuracy, Precision, Recall, and Fl score.
MMHhh-Bennhh (Sun al. 2023): HaMHal--ennh is evaluation benchmark designed to assess hallucination in Large Multimodal Models (LMM). It includes 96 challenging questions based images from OpenImages, along with their corresponding ground-truth answers and image content. Model responses are automatically rated using GPT-4.
Vision-centric task. We use the following benchmarks in Table 3 to evaluate the vision-centric performance of the model.
MMVP (Tong et al. 2024b): MMVP is a dataset of 300 images paired with related ques- tions, curated through a systematic study using the CLIP visual encoder. consists of im- age pairs, called CLIP-blind pairs, where the CLIP vision encoder represents two different images with such similar features that cannot distinguish between them.
CV-Bench (Tong et al., 2024a): CV-Bench addresses the limitations of existing vision- centric benchmarks by providing 2,638 manually inspected examples. It repurposes stan- dard vision datasets such as ADE20K, COCO, and OMNI3D evaluate models on tradi- tional vision tasks in a multimodal context. Using the rich ground truth annotations from these benchmarks, natural language questions are formulated to assess the models' fun- damental 2D and 3D understanding. CV-Bench measures 2D understanding through tasks like spatial relationships and object counting, and 3D understanding through depth order and relative distance. This dataset is structured in multiple-choice question answering (MCQA) format, and models are evaluated based on the accuracy of their responses to each question.
D.2 DETAILS OF EVALUATION SETTING
Experimental Setting, All experiments and evaluations are conducted on a single NVIDIA GeForce RTX A6000 48GB GPU. Only the inference step of L.Ms is used, without any training.
23

---

Published as a conference paper at ICLR 2025
GPT-4 evaluation. LLaVA-Benche (In-the- (n-the-Wild), MM- Vet, and MMHal-bench are benchmarks that use GPT-4 to evaluate model performance. We conducted evaluations using the gpt-4-0613 version, with maximal output length set to 1024.
D.3 DETAILS ANALYSIS
In this section, we provide the detailed experimental settings of Sec.
Investigation of Sink Dimension Values. In Sec. 4.1 we separate the irrelevant visual tokens from relevant visual tokens based on the sink dimensions, as shown in Fig. B]a). First, we selected 10 samples from each category. resulting in total of 100 visual question-answering samples from MME dataset (Yin et al. (023). We then obtain the attention weights from the visual tokens to the text tokens for each sample. Also, we calculate sink dimension value (see Sec. 4.I) for each visual token. Finally, We then plot the pairwise scatter plot of the sink dimension value and the attention weights for all samples. The scatter plot is shown in Fig. B(.)
Token Masking Experiment. In Sec. 4.2 conduct token masking experiments to investigate the impact of visual sink tokens on the performance of LMMs. Specifically, we mask the attention from vişual sink tokens to text tokens blind the model to the visual sink tokens. As discussed in Sec. 4.1 the indices of the visual sink tokens are defined as T {j E Ivis 6(0)) 2 T). where is threshold value. We set 20 for experiments.
We employ the procedure introduced in Geva et al. (023) to mask the attention. Given the query Wet K E RNDx, the attention
matrix Q = x f-1 weeh E NR And the key matrix K
weights A c.h NRN calculated follows:
QK VDr
A softmax
+ M
(6)
where M E RNNN is the causal mask. The causal mask is a upper triangular matrix with negative infinity values (i.e., Mr, -oo for < j), which prevents the model from attending to the future tokens. The procedure, called attention knockout, blocks the attention from æ f-1 to x by setting M.j -o0 We set the attention mask values -0 from the visual sink tokens to the text tokens as follows:
M,j = 00; Vi I I
where Tot is the indices of text tokens and I is the indices of visual sink tokens in the layer l. We then calculate the attention weights 4 and feed them into the subsequent layers.
We also conduct a random-token attention knockout experiment, in which the same number of tokens masked without considering whether they are sink tokens. For instance, if eight sink tokens identified at layer f, we randomly select eight tokens to mask in the corresponding random knockout setting ensure fair comparison. Note that randomness applied differently at each layer. We evaluate the performance of LLa on the POPE dataset (Li et al. 2023c). with results shown in Fig. 3).
While attention knockout is widely used to analyze the influence of specific tokens on model's output (Neo al., 2024; Kaduri al. (2024). this experiment has some limitations. First, setting the attention mask values to -00, which forces the corresponding attention weights to zero, induces diittrddistiitttion behavior, potentially preventing the token knockout effect from being accurately reflected in the model's output. Second, in random knockout, tokens are selected independently at each layer, leading to inconsistencies across layers compared to sink token knockout. Despite these limitations, we believe the token masking experiments offer valuable insights into the influence of visual sink tokens on the model's performance.
Contribution Analysis. In Sec. 4.2 we measure the attribution contributions of visual sink tokens.
ov || for all i Ina and j E I As baseline, similarly calculate the average value for all j E Ivis- Similar to the token masking experiment, we use the LLa VA-1.5-BB model and the POPE dataset (Li et al. 2023c for the analysis, with results shown in Fig. c)..
We compute the average value of ||a e.h e (ea.a--WWW a w
24

---

Published as a conference paper at ICLR 2025
E ADDITIONAL QUALITATIVE RESULTS
Additional qualitative results are provided in Fig.
14 and Fig. 15
Although visualize the attention maps only for few specified text tokens in Fig. | these inrele- vant tokens consistently occur in fixed locations across the entire text tokens, including the instruc- tions and the generated responses. To illustrate this, we visualize the attention maps of various text tokens in Fig. [3 The visual attention maps show that the visual sink tokens consistently receive high attention weights from the text tokens, regardless the content of the text tokens.
Fig. 14 shows the responses generated by the model with and without VAR. We use LLa LLaVA-155-BB
and LLa VA-Bench (n-the-Wild) (Liu et al., (024c) benchmark. We can observe that the model with VAR generates more accurate responses than the model without VAR. In particular, the model with
VAR considers the visual information more effectively and generates more relevant responses the image content.
To evaluate the performance of LMMs in open-ended question-answering tasks, we utilize GPT- 4 (OpenA] 2023) to assess the performance of the model in various aspects. We provide the quali- tative result GPT-4 evaluation in Fig. 15
25

---

Published as a conference paper at ICLR 2025
Q. Is there a person in the image?
Example
A. Yes, there person in the image, who standing rock near the water, <EOS>
Is
there
person
the
image
Yes
who
standing
<EOS>
Q Is there a banana in the image?
Example 2
A. Yes, there is banana the image, placed in bowl on wooden table, <EOS>
Is
there
banana
the
image
Yes
placed
in
bowl
<EOS>
Figure 13: Visual attention maps of various text tokens in LLaVA-115-7B. The visual sink tokens consistently receive high attention weights from the text tokens, regardless of the content of the text tokens. We underline the text tokens that are visualized, and visual sink tokens are marked with red boxes.
26

---

Published as a conference paper at ICLR 2025
The unusual aspect of this image is that a man is standing on clothesling attached to yellow car, which is driving down busy city street. This is not typical sight, as clotheslines are usually found in residential areas backyards, and not common to someone standing
LLaVA
clothesline while car is in motion. The scene captures an unconiventional and unexpected moment, likely drawing and curiosity from passersby.
Example
The unusual aspect this image that
standing on folding table, which is attached yellow car, and ironing clothes. This not typical scene you would expect busy city street, ironing clothes is usually done indoors, such as a home a laundromat. The presence of the folding table and the man's activity in the middle of the street creates interesting and unexpected visual.
LLavA +urs
Question: Describe this photo detail.
To prepare dish, you will
following ingredients:
Shrimp: You
cooked shrimp, depending on your
prefarence.
LLaVA
2. Lemons: You will need few whole lemons, sliced into wedges.In3 In3. Seasonings: You can use salt, pepper, gartic powder, and any other desired seasonings.
Example 2
Instructions:
1. Preheat oven to 400! "F (200 'C).
2. Rinse the shrimp and pat dry with paper towel. ...(omitt
To prepare this diah, you will need fresh lobster tails, lemon wedges,
LLavA + Ours
gartic, and parsley. Here's a step-by-step guide:
the from the lobster tails, and cook them according your preference. You
Clean
tails: Remove claws knuckles
can boil, steam, grill the tails until they are cooked through.
2. Prepare the garlic and parsley: Chop the garlic cloves and parsley leaves finely.
Question: Show the detailed recipe this dish.
Show
Prepare the lemon wedges:
lemon wedges into ...fomit)
scheme, fire-like element the center. The fire appears be burning, giving the logo dynamic and engaging appearance. The design of the logo reminiscent of a fire hydrant, which adds a playful and creative touch to the overall design. The combination the bold red color and the fire-like striking visual impact, making the logo memorable and essity recognizable. The choice of fire hydrant-like shape also suggests e connection to the field of artificial Intelligence. (om(i)
LLaVA logo features
large
LLavA
element
Example 3
The LLaVA logo features red fire-breathing animal, likely dragon or similar creature, with a pair of glasses on its face. The on gray surface, which could be a table or floor. The design the logo quite unique eye-catching, as combines and humorous, anthropomorphic character. The glasses the animal's face add touch of whimsy and personality to the logo, making memorable and appealing. Overail, the LLaVA logo effectively conveys dision-based assistant engaging visual element.
Question: This logo of
The animal standing
LLaVA, Large Language
LLaVA+
Vision Assistant, based the LLaMA architecture.
concept
Please explain this logo in detail, and how do you think its design.
incorporating a
image features man with curly hair, mustache, wearing suit and fie. He smiling and appears be well-dressed individual, The man also wearing glasses, which adds his distinctive appearance. is not possible to determine the exact identity of the person from the image alone, but he could be a businessman, public figure, someane attending
beard,
LLaVA
Example 4
event.
LLaVA + Ours
appearance, smiling and appears who Likely famous scientist intellectual.
image features a man with curly hair
to be enjoying the conversation with the person next to him. The man's
Question: Can you give introducttnti this person?
heir is wild and untamed, and he is wearing suit, which suggests that he well-dressed for the occasion. The man's presence and attire
indicate he prominent figure field science academia.
Figure 14: Qualitative results of the model with and without VAR. We compare the responses generated by LLa with and without VAR. Our method makes the model incorporate visual information more effectively and generate more accurate responses, which are closely related to the image content. While vanilla I.L.VA-1.5-77 incorrectly describes the image ( (red highlights LLaVA-1.5-7B with VAR generates more accurate responses, which are closely related to the image content blue highlights).
27

---

Published as a conference paper at ICLR 2025
Input:".:
[Context]
A top view of highway at night. The highway divided into four sections. From left to right, and four
there three three lancs of traffic moving away from the camera. Most of the cars the four lane saction moving away the camera hava their brake lights There many lene sections. The traffic is light on the two three-lane sections. The four-lane highwey is elevated compared the three-lane highway. lights alongside highway illuminated, On the right side of the highway, there are trees.
trattic approaching the camer,, four
both the four-
[Question]
What the problem this might be facing? What possible solutions?
[Assistant 1]
The city might be facing traffic congestion issues, particularly the four-lane sections of the highway since moet care have their brake lights indicating slow stopped traffic. Possible solutions could include:
Implementing traffic management strategies, such signal timings to improve traffic flo..
control the which enter the highway optimizing traffic
2. Encouraging the use public transportation, carpoeling ride-ste ride-sharing reduce the number of cars the road.
a. Expanding or adding more lanes to case traffic congestion in the four-lanc lanc sections of the highway.
divert traffic from the congested areas. 5. Improving walking and biking infrastructure the city, thus promating more sustainable and non matorizad modes of transport. of Assistant 1]
Exploring alternative
[Assistant 2]
The city is facing significant traffic problem, as evidenced by the heavy traffic on the highway at night. The image shows large number of the highway, with stopped in traffic. This congestion lead to increased travel times, pollution, and higher risk of accidents, IninSome possible solutions to address this prablem include:
1. Implementing public transportation options, such ae buses, trains, or subweys, to encourage people to use theee services instead of personal vahicles.
2. Encouraging cerpooling or ridesharing to reduce the number of cara on the road.
Implementing congestion pricing high-occupancy vahicle (HOV) lanes discourage single-ocoupancy vehicles and promote of more efficient transportation options.
on personel vehicles. 5. Improving traffic management systeme, such se implementing smart treffic signals or real-time traffic updates, to help drivers make decisions avoid unnecessary delays.
Developing and promoting alternetive transportation modes,
eyeling walking, reduce
implementing solutions, city can work towards reducing congestion improving overall transportation efficiency. [End of Assistant 2]
[System]
We
assistants response user question displayed above. asks the question on observing an image. For your reference, the visual content in the image is represented with few sentences describing image.
request your
performance
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives on everall score on scale of where higher score indicates better overall performance. Please first output a single line containing only two volues indicating the scores for Assistant and 2, respectively. The two scores are separated by a space.
the subsequent line, please provide comprehansive explanation of your evaluation, avolding any potential blas and ensuring that order in which the responses were presented does not affect your judgment.
GPT4 Output:
g
Both Assistent and Assistant provided comprehensive, relevant, and sccurate answers question. They both identified the potential problem traffic congestion and suggasted similar solutions such as implementing traftic management strategies, encouraging the use of public transportation, cerpooling or ride-shrring, and improving welking and biking infrastructure, They aiso both sugpested expanding the highway or finding alternative routes. The level of detail in both responses was high, with each assistent providing multiple potential solutions and explaining how these solutions could alleviate the problem. Therefore, they both receive score of -..
Figure 15: Qualitative GPT-4 evaluation results. The example of evaluation of GPT-4 in open- ended generation tasks is illustrated. "Assistant I" and "Assistant 2" refer to "GPT-4" and "LLaVA- 1.5-7B + VAR", respectively.
28


Reviews:
[{"review_id": "0oNAmSTpBx", "reviewer": "Reviewer_1YMG", "summary": "This paper investigates \"visual attention sinks\" in LMMs. The authors pointed out that LMMs often allocate disproportionate attention to certain visual tokens, termed \"visual sink tokens,\" regardless of their relevance to the corresponding text. This paper proposes Visual Attention Redistribution (VAR), a method that identifies and reallocates attention from these sink tokens to more pertinent visual information, enhancing model performance across various vision-language tasks. The authors demonstrate VAR's effectiveness without requiring additional model training.", "questions": "1. Do newer LMMs with anyres and with visual encoder tuned jointly also have the characteristic of visual attention sinks? Do the observation and analyses generalize across LMMs with different language models and sizes?\n\n1. In Fig 8, it looks like LLaVA + VAR has more visual sink tokens qualitatively.\n\n1. Please include the reference of LLaVA-1.5-HD-13B.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The identification of visual attention sinks draws parallels with attention sinks in language models, providing a novel insight into the functioning of LMMs.\n1. Overall well written and easy to follow. Fig 1 and Fig 2 provide a clear and strong motivation. Fig 3-5 also provide a step-by-step motivation and the design choices of the proposed method.\n1. The method improves performance across multiple benchmarks, including general vision-language tasks, visual hallucination tasks, and vision-centric tasks.\n1. The paper evaluates VAR on a wide range of benchmarks and conducts a series of quantitative and qualitative analyses, validating the effectiveness of the proposed VAR method.", "weaknesses": "1. The reviewer's biggest concern is about the determination of hyperparameter $\\rho$. According to the paper and Fig 7(b), it is benchmark-dependent. Furthermore, a poorly chosen $\\rho$ can lead to performance worse than the baseline performance. This parameter greatly limited the applicability and generalizability of the proposed method.\n\n1. The tuning of $\\rho$ on the benchmarks to find the best value is problematic and is against the principle of machine learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wq5jXipKrX", "reviewer": "Reviewer_P74f", "summary": "The paper explores the phenomenon of visual attention sinks in large multimodal models (LMMs). It identifies that these models often allocate high attention weights to irrelevant visual tokens during text-image processing, similar to \"attention sinks\" observed in language models. This issue results in diminished attention to crucial visual information, which can hinder the model’s multimodal understanding. To address this, the authors propose a method called Visual Attention Redistribution (VAR), which reallocates attention from irrelevant \"sink\" tokens to relevant visual tokens. The VAR approach first identifies image-centric attention heads that naturally focus on visual content, then redistributes the excess attention from sink tokens to strengthen attention on the actual image content.", "questions": "1. In Figure 1, does the attention map show the attention scores between visual tokens and a specified text token, or between visual tokens and the output token?\n\n2. In the experiment shown in Figure 4(a) on the POPE benchmark, are the \"random tokens\" randomly selected from only the visual tokens? If not, and they are instead selected from all tokens, wouldn’t this comparison be unfair? If the tokens are indeed chosen only from visual tokens, how is it possible that the model's predictions drop to zero when masking all visual tokens? On the POPE dataset, LLaVA can maintain over 50% accuracy even without image input, so this experimental result is difficult to understand.\n\n3. Regarding the selection of image-centric attention heads, is the VAR strategy applied to all text tokens, or only to specific text tokens, or only to output tokens? If the experiment aligns with the analysis that VAR strategy only applied to specific object token, like \"clock\" in \"Is there a clock in this image?\", how are these particular tokens identified? If applied to all text tokens, is the visual token sink phenomenon consistent across all text tokens, which seems unlikely?\n\n4. During inference, is the attention redistribution performed layer by layer? Specifically, does the forward pass first obtain original results, identify image-centric attention heads, then update attention weights, and proceed with attention weight redistribution in the next layer? \n\nIf the author can address my second and third questions, I will raise my rating.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Insightful Identification of Visual Attention Sink: The authors make an important discovery by identifying the phenomenon of visual attention sinks in LMMs, showing that these models often allocate high attention weights to irrelevant visual tokens. This observation highlights a key inefficiency in multimodal attention mechanisms and contributes to a deeper understanding of model behaviors.\n\n- Effective Attention Redistribution Technique: The proposed Visual Attention Redistribution (VAR) method is both innovative and practical. By reallocating attention from sink tokens to relevant visual tokens, VAR improves focus on key image content, enhancing multimodal model performance without additional training or inference overhead, making it easily applicable across different LMMs.\n\n- Broad Experimental Validation Across Tasks: The paper rigorously validates VAR across a range of multimodal tasks, including visual question answering, visual hallucination reduction, and spatial understanding tasks. This broad evaluation demonstrates the robustness and adaptability of the method, showcasing its effectiveness in various vision-language scenarios.", "weaknesses": "- The results in Figure 4(a) are essential for validating the effectiveness of the sink token filtering strategy, yet they are challenging to interpret. If the randomly selected tokens are drawn from all input tokens, comparing them with visual sink tokens becomes less fair, making it difficult to substantiate the filtering's effectiveness. Conversely, if the random tokens are sampled exclusively from visual tokens, the experimental results appear unreasonable. Even without visual input, the model’s accuracy on the POPE benchmark should remain above 50%, yet the figure shows a final result of 0. Furthermore, when $\\tau < 10$, the tokens being pruned are predominantly standard visual tokens. Despite this, a significant difference in F1-Score persists between the two pruning strategies, contradicting the results seen when $\\tau > 10$. Without a clear explanation of the experimental findings in Figure 4(a), the credibility of the sink token filtering strategy is undermined.\n\n- The scope of the attention redistribution strategy remains ambiguous. Specifically, which tokens are targeted by this strategy? In practice, attention redistribution could apply to both input instruction tokens and output tokens, but the implementation should align with the analytical findings. If the strategy is applied only to specific tokens, it is crucial to specify how these tokens are selected; if applied to all output tokens, the analysis should demonstrate stable consistency in the attention assigned to sink tokens. However, this consistency is not currently evident in the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dM0QQ695Nu", "reviewer": "Reviewer_KaLi", "summary": "This paper studies the phenomenon of visual sink tokens in large multi-modal models. That is, the model allocates high attention scores to visual tokens irrelevant to the corresponding text, and these visual tokens consistently appear in fixed locations. This paper finds that visual sink tokens have high activations in specific dimensions, which provides a way to distinguish them from normal tokens. Furthermore, the paper proposes to recycle the attention to meaningful tokens, helping to improve model performances on a wide range of tasks.", "questions": "1. As mentioned in the weakness, the paper should discuss the relationship between sink tokens in language models and vision models, from the perspective of token properties, emerging mechanisms and proposed methods.\n2. How does the method determine which layer to apply VAR? The supplementary shows that there are less visual sink tokens in the first layer. However, the experiments say that VAR is not applied to the last layer. That does not sound reasonable.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The presentation is really good, first analyzing the property and effects of visual sink tokens, then providing a solution based on these investigations. The figures are also nice to explain relevant concepts and methods.\n2. While the phenomenon of sink tokens has been discussed in language models[1] and vision models[2], this paper further extend it to multi-modal models.\n3. The proposed method is simple yet effective. The model can be modified without further training.\n\n[1] Xiao, Guangxuan, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. \"Efficient Streaming Language Models with Attention Sinks.\" In The Twelfth International Conference on Learning Representations.\n\n[2] Darcet, Timothée, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. \"Vision Transformers Need Registers.\" In The Twelfth International Conference on Learning Representations.", "weaknesses": "1. Insufficient references. There is a work [1] on sink tokens in vision models that is not cited or discussed. \n2. Insufficient discussion on sink tokens. Sink tokens have been observed in different kinds of models, including language models, vision models, and multi-modal models. What's the relationship between sink tokens in different kinds of models? Are they just similar phenomenons, or are there special properties of multimodal models? Moreover, previous works claim that sink tokens exist because of excessive attention scores, so why does it work to recycle these attention scores? From the experiment perspective, the paper is also encouraged to apply previous methods to multi-modal models.\n3. Heavy hyper-parameter tuning for specific tasks. It seems that different tasks are highly sensitive to the selection of $\\rho$. Then is there a train/val/test dataset split? If not, are these hyper-parameters tuned according to the test set? Will these hyper-parameters overfit the test set?\n\n[1] Darcet, Timothée, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. \"Vision Transformers Need Registers.\" In The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gx5WA8boPo", "reviewer": "Reviewer_b7X3", "summary": "In this paper, the authors introduce the phenomenon of visual attention sinks and, through visual analysis, propose that visual attention sink tokens are typically unrelated to the main visual subject information. Furthermore, they demonstrate through experiments how to automatically identify visual sink tokens. Based on this finding, the authors propose a method that utilizes attention budgets to redistribute attention weights. Extensive experiments have shown that this method optimizes both the general visual capabilities of models and the issue of hallucination.\n\nThe paper makes two main contributions:\n1. It introduces the phenomenon of visual attention sinks and analyzes the behavior of LVLMs.\n\n2. The proposed method enhances the performance of LVLMs with low cost.", "questions": "1. Regarding the questions of visual attention sink:\n\n   1.a. I have previously experimented with removing the <bos> token, and the attention sink phenomenon still persisted in the first token. Therefore, I am curious whether new tokens will become visual sink tokens after redistributing the attention weights.\n\n   1.b. The author has visualized tokens with high attention weights in the text, and indeed, for these cases, the visual sink tokens are unrelated to the main subject of the image. However, I believe it is still necessary to prove the trend of visual sink tokens being unrelated to the main subject of the image on a large-scale dataset based on segmentation data.\n\n   1.c. In Table 4.a, the selection of random tokens does not seem to be related to $\\tau$, so I want to confirm whether two masked methods with the same $\\tau$ value mean that the same number of tokens are masked.\n\n2. Regarding the questions of experimental results:\n\n   2.a. The author only chose LLaVA and VILA for evaluation, but the training data for LLaVA is much smaller compared to high-performance open-source models. Therefore, it is necessary to add one or two such models for evaluation, such as Qwen2-VL and InternVL2.\n\n   2.b. LLaVA-HD has several times more image tokens compared to LLaVA, so I speculate that the author's model should perform better on this backbone. However, from Table 1, it appears that the improvement of the proposed method on LLaVA-HD is not as significant as on LLaVA.\n\nIf the author can address these concerns, I will consider raising the score. Additionally, I believe that further analysis on how to utilize this phenomenon during model training could enhance the impact of the article.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing in this paper is very clear, smoothly transitioning from the introduction of the visual attention sink phenomenon to its analysis and the subsequent methodological approach.\n\n2. The concept of visual attention sink proposed in this paper is indeed a direction worthy of deeper exploration within the realm of LVLMs.\n\n3. The experiments in this paper cover a comprehensive range of content and consistently achieve stable improvements across various tasks.\n\n4. The method proposed in this paper is an optimization based on the attention mechanism, making it generalizable and applicable to most LVLMs for experimentation.", "weaknesses": "1. The phenomenon of visual attention sink appears to be similar to the summary tokens proposed in OPERA[1]. Summary tokens imply that higher attention weights are assigned to tokens without semantic information, such as ',' and '\\n'. The visual sink token seems to suggest that within visual tokens, there are also some tokens that play a similar role to summary tokens.\n\n2. The analysis of the visual attention sink phenomenon lacks experimental validation. For more specific explanations, please refer to Question 1.\n\n3. Although the author's evaluation experiments cover a comprehensive set of benchmarks, there is a lack of evaluation across more LVLMs and a detailed analysis of the experimental results. For more specific explanations, please refer to Question 2.\n\n[1] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)