See What You Are Told: Visual Attention Sink in Large Multimodal Models

Authors: Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large multimodal models, Visual attention sink, Visual attention redistribution

Abstract:
Large multimodal models (LMMs) "see" images by leveraging the attention mechanism between text and visual tokens in the transformer decoder. Ideally, these models should focus on key visual information relevant to the text token. However, recent findings indicate that LMMs have an extraordinary tendency to consistently allocate high attention weights to specific visual tokens, even when these tokens are irrelevant to the corresponding text. In this study, we investigate the property behind the appearance of these irrelevant visual tokens and examine their characteristics. Our findings show that this behavior arises due to the massive activation of certain hidden state dimensions, which resembles the attention sink found in language models. Hence, we refer to this phenomenon as the visual attention sink. In particular, our analysis reveals that removing the irrelevant visual sink tokens does not impact model performance, despite receiving high attention weights. Consequently, we recycle the attention to these tokens as surplus resources, redistributing the attention budget to enhance focus on the image. To achieve this, we introduce Visual Attention Redistribution (VAR), a method that redistributes attention in image-centric heads, which we identify as innately focusing on visual information. VAR can be seamlessly applied across different LMMs to improve performance on a wide range of tasks, including general vision-language tasks, visual hallucination tasks, and vision-centric tasks, all without the need for additional training, models, or inference steps. Experimental results demonstrate that VAR enables LMMs to process visual information more effectively by adjusting their internal attention mechanisms, offering a new direction to enhancing the multimodal capabilities of LMMs.

Decision: accept (poster)
Reviewer scores: 8, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1729778758158

thread_type:
official_review

<thread_object>
creator: reviewer_b7x3

<content>
summary:
In this paper, the authors introduce the phenomenon of visual attention sinks and, through visual analysis, propose that visual attention sink tokens are typically unrelated to the main visual subject information. Furthermore, they demonstrate through experiments how to automatically identify visual sink tokens. Based on this finding, the authors propose a method that utilizes attention budgets to redistribute attention weights. Extensive experiments have shown that this method optimizes both the general visual capabilities of models and the issue of hallucination.

The paper makes two main contributions:
1. It introduces the phenomenon of visual attention sinks and analyzes the behavior of LVLMs.

2. The proposed method enhances the performance of LVLMs with low cost.

strengths:
1. The writing in this paper is very clear, smoothly transitioning from the introduction of the visual attention sink phenomenon to its analysis and the subsequent methodological approach.

2. The concept of visual attention sink proposed in this paper is indeed a direction worthy of deeper exploration within the realm of LVLMs.

3. The experiments in this paper cover a comprehensive range of content and consistently achieve stable improvements across various tasks.

4. The method proposed in this paper is an optimization based on the attention mechanism, making it generalizable and applicable to most LVLMs for experimentation.

weaknesses:
1. The phenomenon of visual attention sink appears to be similar to the summary tokens proposed in OPERA[1]. Summary tokens imply that higher attention weights are assigned to tokens without semantic information, such as ',' and '\n'. The visual sink token seems to suggest that within visual tokens, there are also some tokens that play a similar role to summary tokens.

2. The analysis of the visual attention sink phenomenon lacks experimental validation. For more specific explanations, please refer to Question 1.

3. Although the author's evaluation experiments cover a comprehensive set of benchmarks, there is a lack of evaluation across more LVLMs and a detailed analysis of the experimental results. For more specific explanations, please refer to Question 2.

[1] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

questions:
1. Regarding the questions of visual attention sink:

   1.a. I have previously experimented with removing the <bos> token, and the attention sink phenomenon still persisted in the first token. Therefore, I am curious whether new tokens will become visual sink tokens after redistributing the attention weights.

   1.b. The author has visualized tokens with high attention weights in the text, and indeed, for these cases, the visual sink tokens are unrelated to the main subject of the image. However, I believe it is still necessary to prove the trend of visual sink tokens being unrelated to the main subject of the image on a large-scale dataset based on segmentation data.

   1.c. In Table 4.a, the selection of random tokens does not seem to be related to $\tau$, so I want to confirm whether two masked methods with the same $\tau$ value mean that the same number of tokens are masked.

2. Regarding the questions of experimental results:

   2.a. The author only chose LLaVA and VILA for evaluation, but the training data for LLaVA is much smaller compared to high-performance open-source models. Therefore, it is necessary to add one or two such models for evaluation, such as Qwen2-VL and InternVL2.

   2.b. LLaVA-HD has several times more image tokens compared to LLaVA, so I speculate that the author's model should perform better on this backbone. However, from Table 1, it appears that the improvement of the proposed method on LLaVA-HD is not as significant as on LLaVA.

If the author can address these concerns, I will consider raising the score. Additionally, I believe that further analysis on how to utilize this phenomenon during model training could enhance the impact of the article.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729922063790

thread_type:
official_review

<thread_object>
creator: reviewer_kali

<content>
summary:
This paper studies the phenomenon of visual sink tokens in large multi-modal models. That is, the model allocates high attention scores to visual tokens irrelevant to the corresponding text, and these visual tokens consistently appear in fixed locations. This paper finds that visual sink tokens have high activations in specific dimensions, which provides a way to distinguish them from normal tokens. Furthermore, the paper proposes to recycle the attention to meaningful tokens, helping to improve model performances on a wide range of tasks.

strengths:
1. The presentation is really good, first analyzing the property and effects of visual sink tokens, then providing a solution based on these investigations. The figures are also nice to explain relevant concepts and methods.
2. While the phenomenon of sink tokens has been discussed in language models[1] and vision models[2], this paper further extend it to multi-modal models.
3. The proposed method is simple yet effective. The model can be modified without further training.

[1] Xiao, Guangxuan, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. "Efficient Streaming Language Models with Attention Sinks." In The Twelfth International Conference on Learning Representations.

[2] Darcet, Timothée, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. "Vision Transformers Need Registers." In The Twelfth International Conference on Learning Representations.

weaknesses:
1. Insufficient references. There is a work [1] on sink tokens in vision models that is not cited or discussed. 
2. Insufficient discussion on sink tokens. Sink tokens have been observed in different kinds of models, including language models, vision models, and multi-modal models. What's the relationship between sink tokens in different kinds of models? Are they just similar phenomenons, or are there special properties of multimodal models? Moreover, previous works claim that sink tokens exist because of excessive attention scores, so why does it work to recycle these attention scores? From the experiment perspective, the paper is also encouraged to apply previous methods to multi-modal models.
3. Heavy hyper-parameter tuning for specific tasks. It seems that different tasks are highly sensitive to the selection of $\rho$. Then is there a train/val/test dataset split? If not, are these hyper-parameters tuned according to the test set? Will these hyper-parameters overfit the test set?

[1] Darcet, Timothée, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. "Vision Transformers Need Registers." In The Twelfth International Conference on Learning Representations.

questions:
1. As mentioned in the weakness, the paper should discuss the relationship between sink tokens in language models and vision models, from the perspective of token properties, emerging mechanisms and proposed methods.
2. How does the method determine which layer to apply VAR? The supplementary shows that there are less visual sink tokens in the first layer. However, the experiments say that VAR is not applied to the last layer. That does not sound reasonable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730292902814

thread_type:
official_review

<thread_object>
creator: reviewer_p74f

<content>
summary:
The paper explores the phenomenon of visual attention sinks in large multimodal models (LMMs). It identifies that these models often allocate high attention weights to irrelevant visual tokens during text-image processing, similar to "attention sinks" observed in language models. This issue results in diminished attention to crucial visual information, which can hinder the model’s multimodal understanding. To address this, the authors propose a method called Visual Attention Redistribution (VAR), which reallocates attention from irrelevant "sink" tokens to relevant visual tokens. The VAR approach first identifies image-centric attention heads that naturally focus on visual content, then redistributes the excess attention from sink tokens to strengthen attention on the actual image content.

strengths:
- Insightful Identification of Visual Attention Sink: The authors make an important discovery by identifying the phenomenon of visual attention sinks in LMMs, showing that these models often allocate high attention weights to irrelevant visual tokens. This observation highlights a key inefficiency in multimodal attention mechanisms and contributes to a deeper understanding of model behaviors.

- Effective Attention Redistribution Technique: The proposed Visual Attention Redistribution (VAR) method is both innovative and practical. By reallocating attention from sink tokens to relevant visual tokens, VAR improves focus on key image content, enhancing multimodal model performance without additional training or inference overhead, making it easily applicable across different LMMs.

- Broad Experimental Validation Across Tasks: The paper rigorously validates VAR across a range of multimodal tasks, including visual question answering, visual hallucination reduction, and spatial understanding tasks. This broad evaluation demonstrates the robustness and adaptability of the method, showcasing its effectiveness in various vision-language scenarios.

weaknesses:
- The results in Figure 4(a) are essential for validating the effectiveness of the sink token filtering strategy, yet they are challenging to interpret. If the randomly selected tokens are drawn from all input tokens, comparing them with visual sink tokens becomes less fair, making it difficult to substantiate the filtering's effectiveness. Conversely, if the random tokens are sampled exclusively from visual tokens, the experimental results appear unreasonable. Even without visual input, the model’s accuracy on the POPE benchmark should remain above 50%, yet the figure shows a final result of 0. Furthermore, when $\tau < 10$, the tokens being pruned are predominantly standard visual tokens. Despite this, a significant difference in F1-Score persists between the two pruning strategies, contradicting the results seen when $\tau > 10$. Without a clear explanation of the experimental findings in Figure 4(a), the credibility of the sink token filtering strategy is undermined.

- The scope of the attention redistribution strategy remains ambiguous. Specifically, which tokens are targeted by this strategy? In practice, attention redistribution could apply to both input instruction tokens and output tokens, but the implementation should align with the analytical findings. If the strategy is applied only to specific tokens, it is crucial to specify how these tokens are selected; if applied to all output tokens, the analysis should demonstrate stable consistency in the attention assigned to sink tokens. However, this consistency is not currently evident in the analysis.

questions:
1. In Figure 1, does the attention map show the attention scores between visual tokens and a specified text token, or between visual tokens and the output token?

2. In the experiment shown in Figure 4(a) on the POPE benchmark, are the "random tokens" randomly selected from only the visual tokens? If not, and they are instead selected from all tokens, wouldn’t this comparison be unfair? If the tokens are indeed chosen only from visual tokens, how is it possible that the model's predictions drop to zero when masking all visual tokens? On the POPE dataset, LLaVA can maintain over 50% accuracy even without image input, so this experimental result is difficult to understand.

3. Regarding the selection of image-centric attention heads, is the VAR strategy applied to all text tokens, or only to specific text tokens, or only to output tokens? If the experiment aligns with the analysis that VAR strategy only applied to specific object token, like "clock" in "Is there a clock in this image?", how are these particular tokens identified? If applied to all text tokens, is the visual token sink phenomenon consistent across all text tokens, which seems unlikely?

4. During inference, is the attention redistribution performed layer by layer? Specifically, does the forward pass first obtain original results, identify image-centric attention heads, then update attention weights, and proceed with attention weight redistribution in the next layer? 

If the author can address my second and third questions, I will raise my rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730835216687

thread_type:
official_review

<thread_object>
creator: reviewer_1ymg

<content>
summary:
This paper investigates "visual attention sinks" in LMMs. The authors pointed out that LMMs often allocate disproportionate attention to certain visual tokens, termed "visual sink tokens," regardless of their relevance to the corresponding text. This paper proposes Visual Attention Redistribution (VAR), a method that identifies and reallocates attention from these sink tokens to more pertinent visual information, enhancing model performance across various vision-language tasks. The authors demonstrate VAR's effectiveness without requiring additional model training.

strengths:
1. The identification of visual attention sinks draws parallels with attention sinks in language models, providing a novel insight into the functioning of LMMs.
1. Overall well written and easy to follow. Fig 1 and Fig 2 provide a clear and strong motivation. Fig 3-5 also provide a step-by-step motivation and the design choices of the proposed method.
1. The method improves performance across multiple benchmarks, including general vision-language tasks, visual hallucination tasks, and vision-centric tasks.
1. The paper evaluates VAR on a wide range of benchmarks and conducts a series of quantitative and qualitative analyses, validating the effectiveness of the proposed VAR method.

weaknesses:
1. The reviewer's biggest concern is about the determination of hyperparameter $\rho$. According to the paper and Fig 7(b), it is benchmark-dependent. Furthermore, a poorly chosen $\rho$ can lead to performance worse than the baseline performance. This parameter greatly limited the applicability and generalizability of the proposed method.

1. The tuning of $\rho$ on the benchmarks to find the best value is problematic and is against the principle of machine learning.

questions:
1. Do newer LMMs with anyres and with visual encoder tuned jointly also have the characteristic of visual attention sinks? Do the observation and analyses generalize across LMMs with different language models and sizes?

1. In Fig 8, it looks like LLaVA + VAR has more visual sink tokens qualitatively.

1. Please include the reference of LLaVA-1.5-HD-13B.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732355098602

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate all the reviewers for their constructive comments and insightful suggestions. Our research has been further refined and enriched thanks to their feedback. We have uploaded the revised manuscript, incorporating the reviewers' helpful comments. Below, we summarize the major changes made in the revised manuscript:

* More Recent LMMs for Evaluation: We have added two more LMMs, Qwen2-VL and InternVL2, which are more recent. The results show that VAR consistently improves the performance of these models, indicating that the proposed method is applicable to various LMMs. The results are presented in Table 1, 2, and 3. (Reviewer 1YMG, b7X3)
* More Discussions on Visual Attention Sink: We have discussed the characteristics of visual attention sinks, including token properties and emerging mechanisms, in more detail. We have also added the relationship between sink tokens in different models. Research on sink tokens in ViT [1] was incorporated into the related work (Section 2). The discussion is presented in Section 4.2 and Appendix A.2. (Reviewer KaLi)
* Reevaluating Token Masking Experiment in Section 4.2: Recognizing that the token masking experiment setup was misaligned with our intended objectives, we revised the setup and reevaluated the experiment. We also clarified the experimental settings and results. The updated results are presented in Figure 3(b), with further details available in Appendix C.3. (Reviewer P74f, b7X3)
* More Experiments and Discussions on Hyperparameters: We conducted additional experiments to validate the hyperparameter tuning process and demonstrate the robustness of VAR to hyperparameter choices. The results are summarized in Section 6.3 and detailed in Appendix B.2. (Reviewer 1YMG, KaLi)
* Validation of Visual Sink Tokens' Irrelevance to the Main Subject: We added quantitative validation of the trend showing that visual sink tokens are unrelated to the main subject of the image, using segmentation datasets, in Appendix A.2. (Reviewer b7X3)
* Clearer Explanations and Presentations:
  * Figure 3 and Figure 4 were integrated to provide a more coherent presentation of the experimental results.
  * The method for visualizing the attention maps in Figure 1 was clarified, and visual attention maps between all text tokens and visual tokens were included in Figure 13 to demonstrate the consistency of the visual attention sink across all text tokens. (Reviewer P74f)
  * The target tokens of VAR were explicitly detailed in Section 5. (Reviewer P74f)
  * Additional future research directions were included in Section 7. (Reviewer b7X3)

[1] Darcet, Timothée, et al. "Vision Transformers Need Registers." The Twelfth International Conference on Learning Representations. 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733307345327

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final General Response (1/2)

comment:
Dear AC and Reviewers,

We appreciate the time and effort the reviewers have dedicated to providing detailed feedback on our work. During the discussion period, we have received valuable suggestions, such as `(1)` additional experiments on the latest models and Anyres models, `(2)` a more thorough discussion of visual sink tokens with analysis on large-scale datasets, and `(3)` experiments on the robustness of hyperparameters. This feedback has significantly improved our paper, and we are pleased to note that two reviewers have increased their scores from 5 to 6 and 6 to 8, respectively. We are grateful to all the reviewers for giving us the opportunity to strengthen our work and for finding our research interesting and valuable. 

In particular, we had a detailed discussion with Reviewer P74f regarding the interpretation of Figure 3(b). **The primary goal of this experiment is to demonstrate that the visual sink token has a negligible impact on the model's performance**. The key point is that the model remains stable even when the sink token is removed during the inference process, indicating that it does not play a significant role in generating image-related responses. Specifically, this is explained by **the stable performance of the model in the range of $\tau = 20$ to $\tau = 50$, as shown by the red line in Figure 3(b)**. Additionally, through our discussions with the reviewers, we have provided further evidence to support the claim that the visual sink token has a minimal impact on model performance:

1. Removing visual sink tokens does not significantly affect performance, whether *before* or *after* the softmax calculation. (Response to Reviewer P74f (2/3))
2. The final contribution value of the visual sink token $\Vert \alpha ^ {\ell, h} _ {i, j} \boldsymbol{x} _ j ^ {\ell-1} \boldsymbol{W} _ {OV} ^ {\ell, h} \Vert$ is significantly lower than that of other visual tokens. (Further Response to Reviewer P74f (2/3))
3. Visual sink tokens are mainly located in semantically meaningless areas. (Response to Reviewer b7X3 (2/3), Appendix A.2)
4. Visual sink tokens exhibit similar characteristics to text sink tokens, which are known to have a negligible impact on the model (Response to Reviewer KaLi (1/2), Section 4.2, Appendix A.2).

Based on these points, we conclude that the visual sink token has a negligible impact on model performance, which leads us to present Visual Attention Redistribution (VAR) based on this conclusion.

However, in the discussion with Reviewer P74f, a disagreement arose regarding the experiment in Figure 3(b), which **diverged somewhat from the core focus of our experiment**. From our understanding, the reviewer's main concern is why the performance degradation in the "visual sink token" experiment (red line) between $10 < \tau < 20$ is more significant than that in the "random visual token" experiment (blue line) for $\tau > 40$, even though the number of masked tokens is larger in the latter case. Comparing the performance degradation between the two experiments in different ranges of $\tau$ leads to a fuzzy interpretation, as the two experiments are not controlled for the same number of masked tokens, token positions, dynamic vs. fixed masking, and other factors (Further Response to Reviewer P74f (3/3) & Further Response to Reviewer P74f Ⅱ (3/3)). Nevertheless, based on our insights into the visual sink token and related works, we have made our best effort to clarify the interpretation of the experiment.

If we understand the reviewer's response correctly, the main reason for the disagreement is that the reviewer's own experiment results (fixed-position pruning) are not consistent with our experimental results. In the reviewer's experiment, pruning visual tokens did not lead to significant performance degradation, whereas in our experiment, masking resulted in a performance drop. Despite our best efforts, we were unable to replicate the reviewer's results. Furthermore, other studies that conducted masking experiments in similar settings reported significant performance/probability drops with only a few masked tokens [1, 2], which supports our findings.

In this regard, we kindly request that the AC consider the discussion with Reviewer P74f and our responses in the context of the entire paper. The reviewer also mentioned that "*most other aspects of this work appear quite solid*" (Response to Authors (1/2) of Reviewer P74f). We hope that the reviewer's concerns have been addressed in our final response, and we ask the AC to carefully consider the overall aspects of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
### Paper Summary:
This paper introduces and analyzes the "visual attention sink" phenomenon in Large Multimodal Models (LMMs), where irrelevant visual tokens receive disproportionately high attention weights. The authors propose Visual Attention Redistribution (VAR) to reallocate attention from sink tokens to more relevant visual information, improving model performance without additional training.

### Strengths:
1. Novel phenomenon identification [1YMG]: 
> "The identification of visual attention sinks draws parallels with attention sinks in language models, providing a novel insight into the functioning of LMMs."

2. Simple yet effective solution [P74f]:
> "The proposed Visual Attention Redistribution (VAR) method is both innovative and practical. By reallocating attention from sink tokens to relevant visual tokens, VAR improves focus on key image content."

3. Comprehensive validation [b7X3]: 
> "The experiments in this paper cover a comprehensive range of content and consistently achieve stable improvements across various tasks."

### Weaknesses:
1. Hyperparameter sensitivity [1YMG]:
> "The reviewer's biggest concern is about the determination of hyperparameter ρ. According to the paper and Fig 7(b), it is benchmark-dependent."

2. Reproducibility concerns [P74f]:
> "Pruning 25% of visual tokens at fixed positions does not lead to significant performance degradation... Could the authors provide more details about their experimental setup?"

### Justification:
While P74f raised important concerns about experimental reproducibility, these focus primarily on one validation approach rather than the core contribution. The phenomenon is well-documented through multiple approaches, and the solution shows practical benefits. As noted by P74f:
> "Most other aspects of this work appear quite solid"

Two other expert reviewers rated the paper above acceptance threshold, acknowledging its contribution to understanding LMM behavior. However, the authors should add detailed experimental protocols and acknowledge limitations in token masking experiments. 

As a final remark, the identification of the attention sink phenomenon itself, in my opinion, represents a valuable contribution to the field. A paper can merit acceptance even if readers do not fully agree with the authors' proposed explanation, as the explanation why / how modern deep models work is constantly changing. Since no fundamental flaws were found in the technical work, I recommend acceptance.