Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models
Authors: Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, Changick Kim
Accepted at: acl25

Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models

Sangmin Woo* Donguk Kim* Jaehyuk Jang* Yubin Choi Changick Kim
KAIST
{smwoo95, kdu3613, jhyuk, choibinbin, changick}@kaist.ac.kr
Project: https://sangminwoo.github.io/AvisC/

Abstract
Large Vision Language Models (LVLMs) demonstrate strong capabilities in visual understanding and description, yet often suffer from hallucinations—attributing incorrect or misleading features to images. We observe that LVLMs disproportionately focus on a small subset of image tokens—termed blind tokens—which are typically irrelevant to the query (e.g., background or non-object regions). We hypothesize that such attention misalignment plays a key role in generating hallucinated responses. To mitigate this issue, we propose Attentional Vision Calibration (AvisC), a test-time approach that dynamically recalibrates the influence of blind tokens without modifying the underlying attention mechanism. AvisC first identifies blind tokens by analyzing layer-wise attention distributions over image tokens, then employs a contrastive decoding strategy to balance the influence of original and blind-token-biased logits. Experiments on standard benchmarks, including POPE, MME, and AMBER, demonstrate that AvisC effectively reduces hallucinations in LVLMs.

1 Introduction
Large Vision Language Models (LVLMs) (Dai et al., 2024; Zhu et al., 2023; Liu et al., 2023c,b; Bai et al., 2023; Tong et al., 2024a) have recently shown remarkable capabilities in generating coherent and contextually relevant descriptions of visual inputs. Yet, these models are prone to "hallucinations", producing responses that do not accurately reflect the underlying image. Such hallucinations pose a critical challenge for applications, demanding reliability, precision, and trustworthy visual interpretation.

In this work, we hypothesize that a key contributing factor to hallucinations in LVLMs is an attention misalignment during inference. Specifically, we observe that LVLMs often allocate excessive attention to a small subset of image tokens—referred to as blind tokens—which appear to be uninformative or irrelevant to the query (e.g., background or repetitive regions; see Fig. 1). Our preliminary experiments support this observation (Fig. 2): despite receiving high attention weights, these tokens do not typically carry the query-relevant information necessary for precise interpretation.

We posit that this skewed focus may cause LVLMs to rely disproportionately on such blind tokens during text generation, potentially leading to hallucinated or inaccurate responses. Rather than grounding their outputs in the most semantically relevant visual details, the models may be influenced by peripheral or misleading cues, undermining the factual integrity of their descriptions.

To investigate this hypothesis, we propose AvisC, a novel, training-free decoding method that dynamically recalibrates the influence of blind tokens at inference time. AvisC operates in two key stages: first, it analyzes the layer-wise attention distributions to identify image tokens that receive excessive attention and flags them as blind tokens; then, it applies a contrastive decoding strategy (Leng et al., 2023; Favero et al., 2024). By comparing the prediction logits generated using all image tokens with those obtained when blind tokens are selectively zeroed out, AvisC adjusts the final token probabilities to mitigate the undue influence of irrelevant tokens.

Notably, AvisC does not directly manipulate model’s attention mechanism or require any retraining. Instead, it operates solely at the decoding stage by modifying the final token probabilities, reducing the influence of blind tokens while amplifying the impact of non-blind tokens.

We validate our approach on multiple benchmarks—including POPE (Rohrbach et al., 2018), MME (Fu et al., 2024), and AMBER (Wang et al., 2023)—demonstrating that AvisC not
Figure 1: Blind tokens in LVLMs. (Top) Even when the image (\( \mathcal{V} \)) lacks meaningful content for the textual query (\( Q \)), modern LVLMs (Dai et al., 2024; Liu et al., 2023c) still assign disproportionate attention to a few image tokens (i.e., blind tokens). Despite having identical, featureless yellow patches, these tokens dominate the attention distribution. (Bottom) In a real image, overlaying bounding boxes and LLaVA 1.5’s attention map highlights a clear mismatch between blind tokens (red boxes) and genuinely informative regions. Note: attention weights are averaged across all layers for the first generated token. See Appendix A for more examples.

only significantly reduces hallucinations but also enhances both the factual accuracy and descriptive richness of LVLM outputs. Importantly, our method is model-agnostic and requires no additional training data or external modules, making it a practical plug-and-play solution for improving existing LVLM systems.

In summary, our contribution is threefold: (1) We uncover and analyze the phenomenon of blind tokens—image tokens with disproportionately high attention that contribute to hallucinations in LVLMs. (2) We introduce AvisC, a training-free approach that dynamically recalibrates the influence of blind tokens without modifying the underlying model; and (3) We demonstrate, through comprehensive experiments on standard benchmarks, that AvisC effectively reduces hallucinations and enhances the performance of diverse LVLMs.

2 Observations

Modern LVLMs (Dai et al., 2024; Liu et al., 2023c) build upon the transformer architecture (Vaswani et al., 2017), where attention weights are intended to highlight the most relevant tokens for generating the next output token. Intuitively, tokens receiving higher attention should correspond to key elements in the image—an idea that has proven effective in both vision- and text-based transformers (Caron et al., 2021; Ilharco et al., 2021; Vaswani et al., 2017). However, we find that this principle does not always hold in current LVLMs.

Blind tokens in uniform images. A striking illustration of this issue arises when an image has no meaningful content for the query—such as a uniformly colored background. As shown in Fig. 1 (top), even in a plain yellow image with no discernible objects, LVLMs often concentrate most of their attention on a few patches. We refer to these excessively attended yet semantically uninformative patches as blind tokens. This phenomenon echoes findings in vision transformers (Darcel et al., 2023)1, where certain background regions disproportionately attract high attention, possibly serving as global information “reservoirs” at the expense of local, detail-rich areas.

Mismatch between blind tokens and actual objects. Beyond artificially simple images, we also analyze real images from COCO2014 (Lin et al., 2014). We ask LVLMs to describe the given image

1We discuss our findings in relation to (Darcel et al., 2023) in Appendix D.
Figure 2: Blind tokens contribute little to actual predictions. (a) We perform zero-out experiments to measure the impact of blind vs. non-blind tokens. Zeroing out blind tokens (Zero-out > \( \mu + \sigma \)), where attention weights are above mean + standard deviation, leaves the model’s predicted probabilities nearly unchanged, suggesting that these tokens carry minimal object-discriminative information. In contrast, zeroing out non-blind tokens yields near 50:50 probabilities, underscoring their critical role in correct prediction. (b) When non-blind tokens are zeroed out, the models fails to correctly predict previously well-classified instances.

and measure how much attention goes to patches corresponding to object bounding boxes. As shown in Fig. 1 (bottom), many tokens receiving disproportionately high attention (blind tokens) have little overlap with genuine object regions, while actual objects receive comparatively less attention. Specifically, only 3.7% of blind tokens overlap with these regions, and merely 23.2% of total attention weight is goes to them.\footnote{Detailed statistics can be found in Fig. 9.} This mismatch indicates that, despite carrying little or no query-relevant information, blind tokens consume a large share of the model’s attention. Consequently, truly informative tokens that capture critical visual details are under-emphasized, potentially compromising the model’s descriptive quality and reliability.

Zero-out experiments. To better understand the functional role of blind tokens, we conduct a zero-out analysis on LLaVA-1.5 (Liu et al., 2023c), shown in Fig. 2. Specifically, we either zero out the blind tokens or the non-blind tokens, then observe changes in the model’s predicted logits. When we remove blind tokens, the logits remain almost identical to those of the original model—indicating that these tokens contribute little to the final prediction.

By contrast, removing non-blind tokens causes the logits to collapse to near-uniform probabilities, revealing that the essential, object-discriminative information resides in those less-attended tokens.\footnote{For dataset-level zero-out experiment details, see Tab. 7.}

Attention bias and hallucinations. These findings suggest that LVLMs systematically overemphasize certain patches that do not meaningfully aid the prediction process. Consequently, truly informative tokens—often corresponding to the actual objects or key details—receive insufficient attention. We hypothesize that this imbalance predisposes the model to hallucinate, as the generation process leans on blind tokens that fail to encode crucial visual details. In the next sections, we propose a simple yet effective decoding method to mitigate this problem by recalibrating the model’s attention usage at decoding stage, thereby reducing its reliance on blind tokens and improving visual grounding.

Hypothesis. Our hypothesis is that blind tokens arises as a structural byproduct of the deep, layered architecture of these models, similar to the “high-norm outlier tokens” observed in vision trans-
formers (Darcel et al., 2023) (see Appendix D). As information is propagated through layers, global representations from earlier layers are progressively compressed. However, instead of being allocated to semantically meaningful tokens, this global information often becomes concentrated in structurally convenient but semantically irrelevant tokens—frequently those in repetitive or low-information regions. These tokens consequently accumulate disproportionate representational weight and attract excessive attention during decoding. Despite their lack of semantic relevance, they misguide the model’s focus and contribute to the generation of hallucinated content. While establishing a definitive causal link is inherently challenging, our qualitative and quantitative evidence suggest that blind tokens are a recurring and impactful phenomenon in LVLMs. Visualizations and token distribution analyses (see Figs. 6 to 11) demonstrate that blind tokens frequently emerge in spatially uninformative regions and exhibit anomalously high attention scores.

3 Approach: AViSC

We propose AViSC, a test-time approach to enhance visual object understanding in LVLMs during decoding. AViSC dynamically recalibrates the model’s attention at every token generation stepby reducing the over-emphasis on blind tokens—image tokens that receive disproportionate attention despite lacking task-relevant information. An overview of AViSC is shown in Fig. 3. Our approach modifies the decoding process in three key steps: (1) Layer selection: identify layers that exhibit a high proportion of image-related attention; (2) Blind token identification: detect tokens that capture an unusually high share of attention; and (3) Contrastive decoding: adjust output logits to mitigate the influence of these blind tokens.

3.1 LVLM Framework

Uni-modal encoding. LVLMs process visual inputs and textual queries into compact representations. Pre-trained encoders like CLIP (Radford et al., 2021) are commonly used for processing visual data. The text data is tokenized, turning it into a sequence of manageable pieces for further processing.

Cross-modal alignment. Since LLMs natively process only text, a learnable cross-modal alignment module—such as the Q-Former (Li et al., 2023a) or a linear projection layer (Liu et al., 2023c)—transform visual features into tokens compatible with the LLM’s input space. This yields a set of visual tokens, \( V = \{v_0, v_1, \ldots, v_{N-1}\} \), which are then concatenated with text tokens, \( Q = \{\sigma_N, \sigma_{N+1}, \ldots, \sigma_{N+M-1}\} \), forming a unified input sequence of length \( N + M \).

Next token prediction via LLM. The concatenated token sequence is processed by LVLM (parametrized by \( \theta \)) in an auto-regressive manner. The model computes logits \( \ell_t \) for each potential next token:

\[
\ell_t = \log p(\xi_t | V, Q, \xi_{<t}; \theta),
\]

where \( \xi_t \) is the token predicted at timestep \( t \), and \( \xi_{<t} \) denotes the sequence generated up to timestep \( (t-1) \). These logits are converted into a probability distribution via the softmax function:

\[
p(\xi_t) = \text{Softmax}(\ell_t).
\]

from which the next token is sampled.

3.2 Attentional Vision Calibration

Visual hallucinations in LVLMs often arise during decoding when the model’s token selection
is guided by skewed probability distributions that do not faithfully reflect the underlying visual input. Our observations (see Sec. 2) indicate that this problem is linked to an attentional bias toward certain non-relevant tokens, which we term blind tokens. Our approach aims to recalibrate these attention patterns to mitigate hallucinations.

Layer selection. Different layers in LVLMs contribute variably to processing visual information. As illustrated in Fig. 4, models such as InstructBLIP (Dai et al., 2024) and LLaVA-1.5 (Liu et al., 2023c) exhibit different attention distributions across layers. To accommodate these differences, we first select layers that exhibit a high proportion of attention on image tokens. Formally, for the ith layer, we define the attention weight matrix:

\[
A_i = \left[ a_{h,q,k}^{i} \right]_{(h,q,k)=(H,N+M,N+M)}^{(h,q,k)=(1,1,1)},
\]

where \( a_{h,q,k}^{i} \) represents the attention weight from head h, for query q to key l_k in layer i. With image tokens \( \mathcal{V} \in \mathbb{R}^{N \times D} \) and query tokens \( Q \in \mathbb{R}^{M \times D} \), we compute the proportion of attention dedicated to image tokens in layer i:

\[
AP_i^{\text{layer}} = \frac{\sum_h \sum_{k=1}^N a_{h,(N+M),k}^{i}}{\sum_{i,h} \sum_{k=1}^N a_{h,(N+M),k}^{i}},
\]

where H is the total number of attention heads, N is the number of image tokens, and M is the number of query tokens. We then sort the layers by this proportion and select layers using top-P sampling with threshold γ:

\[
\{\text{Selected Layers}\} = \text{top-P}(\{AP_i^{\text{layer}}\}_{i=1}^L, \gamma).
\]

These selected layers provide the basis for further token-level analysis.

Blind token identification. Within the selected layers, we compute the attention proportion for each image token by averaging over all heads:

\[
AP_j^{\text{image}} = \frac{\sum_{i \in \{\text{Selected Layers}\}} \sum_{h=1}^H a_{h,(N+M),[1:N]}^{i}}{|\{\text{Selected Layers}\}| \times H}.
\]

To identify tokens that disproportionately capture attention, i.e., blind tokens, we calculate the mean (\( \mu \)) and standard deviation (\( \sigma \)) of the image attention weights. Tokens with an attention proportion exceeding \( \mu + \lambda \sigma \) (where \( \lambda \) is a hyperparameter) are classified as blind tokens:

\[
\{\text{Blind Token Indices}\} = \{j | AP_j^{\text{image}} > \mu + \lambda \sigma \}.
\]

Contrastive decoding. To mitigate hallucinations, we reduce the influence of blind tokens during decoding via contrastive decoding (Leng et al., 2023; Favero et al., 2024). We construct a biased set of visual tokens by zeroing out non-blind tokens:

\[
\mathcal{V}^* = \bigcup_{j=1}^N \mathbb{I}_{\{j \in \text{Blind Token Indices}\}}(j) v_j.
\]

We then compute the logits for the next token using both the original visual tokens (\( \mathcal{V} \)) and the biased tokens (\( \mathcal{V}^* \)):

\[
\ell_t = \log p(\xi_t|\mathcal{V}, Q, \xi_{<t}; \theta),
\]
\[
\ell_t^* = \log p(\xi_t|\mathcal{V}^*, Q, \xi_{<t}; \theta),
\]

Finally, we adjust the logits using a contrastive scheme and sample the next token from:

\[
\xi_t \sim \text{Softmax}((1+\alpha)\ell_t - \alpha \ell_t^*).
\]

Here, \( \alpha \) controls the strength of the contrast. This adjustment effectively down-weights the contribution of blind tokens and promotes a more balanced attention distribution, thereby reducing hallucinations in the final output.

4 Experiments

Additional experimental results are provided in Appendix C.

4.1 Evaluation Setup

We deliberately avoid constraining LVLMs to provide one-word responses (e.g., “Yes” or “No”) for discriminative tasks, as our analysis (see Tab. 13) shows that such constraints can significantly alter performance. For our experiments, we set P = 0.5 in Eq. (5), \( \lambda = 1 \) in Eq. (7), and \( \alpha = 3 \) for InstructBLIP (Dai et al., 2024) and \( \alpha = 2.5 \) for LLaVA-1.5 (Liu et al., 2023c) in Eq. (10).4

LVLMs. We evaluate AviSC on two state-of-the-art LVLMs: InstructBLIP (Dai et al., 2024) and LLaVA-1.5 (Liu et al., 2023c), both of which use Vicuna 7B (Chiang et al., 2023) as the LLM backbone. InstructBLIP employs the Q-Former (Li et al., 2023a) to efficiently fuse visual and textual features using a fixed number of tokens (e.g., 32 tokens), while LLaVA-1.5 aligns image and text modalities via linear projection layers. Notably, AviSC is model-agnostic and can be integrated into various LVLM architectures.

4Further experimental details are provided in Appendix B.
<table>
  <tr>
    <th rowspan="2">Setup</th>
    <th rowspan="2">Method</th>
    <th colspan="4">InstructBLIP (Dai et al., 2024)</th>
    <th colspan="4">LLaVA-1.5 (Liu et al., 2023c)</th>
  </tr>
  <tr>
    <th>Acc.</th>
    <th>Prec. ↑</th>
    <th>Rec. ↑</th>
    <th>F1 ↑</th>
    <th>Acc.</th>
    <th>Prec. ↑</th>
    <th>Rec. ↑</th>
    <th>F1 ↑</th>
  </tr>
  <tr>
    <td rowspan="4">MS-COCO</td>
    <td><i>base</i></td>
    <td>82.27</td>
    <td>82.84</td>
    <td>81.40</td>
    <td>82.11</td>
    <td>84.47</td>
    <td>83.35</td>
    <td>86.13</td>
    <td>84.72</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>83.37</td>
    <td>83.39</td>
    <td>82.60</td>
    <td>83.24</td>
    <td>84.80</td>
    <td>83.00</td>
    <td><b>87.53</b></td>
    <td>85.20</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>84.37</td>
    <td>84.62</td>
    <td><b>84.00</b></td>
    <td>84.31</td>
    <td>86.00</td>
    <td>85.11</td>
    <td>87.27</td>
    <td>86.18</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>88.73</b></td>
    <td><b>93.88</b></td>
    <td>82.87</td>
    <td><b>88.03</b></td>
    <td><b>87.93</b></td>
    <td><b>88.24</b></td>
    <td><b>87.53</b></td>
    <td><b>87.88</b></td>
  </tr>
  <tr>
    <td rowspan="4">Popular</td>
    <td><i>base</i></td>
    <td>77.77</td>
    <td>74.81</td>
    <td>83.73</td>
    <td>79.02</td>
    <td>82.23</td>
    <td>79.72</td>
    <td>86.47</td>
    <td>82.95</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>78.00</td>
    <td>75.12</td>
    <td>83.73</td>
    <td>79.19</td>
    <td>82.27</td>
    <td>79.19</td>
    <td>87.53</td>
    <td>83.15</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>77.30</td>
    <td>74.10</td>
    <td>83.93</td>
    <td>78.71</td>
    <td>82.83</td>
    <td>79.62</td>
    <td>88.27</td>
    <td>83.72</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>83.90</b></td>
    <td><b>81.33</b></td>
    <td><b>88.00</b></td>
    <td><b>84.53</b></td>
    <td><b>84.33</b></td>
    <td><b>81.71</b></td>
    <td><b>88.47</b></td>
    <td><b>84.96</b></td>
  </tr>
  <tr>
    <td rowspan="4">Adversarial</td>
    <td><i>base</i></td>
    <td>73.13</td>
    <td>69.41</td>
    <td>82.60</td>
    <td>75.46</td>
    <td>77.10</td>
    <td>72.57</td>
    <td>87.13</td>
    <td>79.19</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>75.87</td>
    <td>72.85</td>
    <td>82.47</td>
    <td>77.36</td>
    <td>76.10</td>
    <td>71.50</td>
    <td>86.80</td>
    <td>78.41</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>76.03</td>
    <td>72.47</td>
    <td>83.93</td>
    <td>77.79</td>
    <td>77.70</td>
    <td>73.23</td>
    <td>87.33</td>
    <td><b>79.66</b></td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>81.57</b></td>
    <td><b>80.37</b></td>
    <td>83.53</td>
    <td><b>81.92</b></td>
    <td>77.53</td>
    <td>72.82</td>
    <td>87.87</td>
    <td>79.64</td>
  </tr>
  <tr>
    <td rowspan="4">A-OKVQA</td>
    <td><i>base</i></td>
    <td>81.00</td>
    <td>77.71</td>
    <td>86.93</td>
    <td>82.06</td>
    <td>82.73</td>
    <td>77.43</td>
    <td>92.40</td>
    <td>84.26</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>81.73</td>
    <td>78.67</td>
    <td>87.07</td>
    <td>82.66</td>
    <td>81.30</td>
    <td>75.45</td>
    <td>92.80</td>
    <td>83.23</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>82.33</td>
    <td>77.81</td>
    <td><b>90.47</b></td>
    <td>83.66</td>
    <td>83.57</td>
    <td><b>77.86</b></td>
    <td><b>93.80</b></td>
    <td><b>85.09</b></td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>88.47</b></td>
    <td><b>87.66</b></td>
    <td>89.53</td>
    <td><b>88.59</b></td>
    <td><b>84.60</b></td>
    <td><b>79.29</b></td>
    <td><b>93.67</b></td>
    <td><b>85.88</b></td>
  </tr>
  <tr>
    <td rowspan="4">Popular</td>
    <td><i>base</i></td>
    <td>75.00</td>
    <td>70.14</td>
    <td>87.07</td>
    <td>77.69</td>
    <td>76.10</td>
    <td>69.86</td>
    <td>91.80</td>
    <td>79.34</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>75.33</td>
    <td>70.52</td>
    <td>87.07</td>
    <td>77.92</td>
    <td>75.43</td>
    <td>68.58</td>
    <td>93.87</td>
    <td>79.26</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>75.60</td>
    <td>70.40</td>
    <td>88.33</td>
    <td>78.36</td>
    <td>76.80</td>
    <td>70.20</td>
    <td>93.13</td>
    <td>80.06</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>81.77</b></td>
    <td><b>77.82</b></td>
    <td><b>88.87</b></td>
    <td><b>82.98</b></td>
    <td><b>78.83</b></td>
    <td><b>72.10</b></td>
    <td><b>94.07</b></td>
    <td><b>81.63</b></td>
  </tr>
  <tr>
    <td rowspan="4">Adversarial</td>
    <td><i>base</i></td>
    <td>68.80</td>
    <td>63.57</td>
    <td>88.07</td>
    <td>73.84</td>
    <td>67.90</td>
    <td>62.11</td>
    <td>91.80</td>
    <td>74.09</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>69.70</td>
    <td>64.54</td>
    <td>87.47</td>
    <td>74.27</td>
    <td>67.43</td>
    <td>61.50</td>
    <td>93.20</td>
    <td>74.11</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>69.57</td>
    <td>64.21</td>
    <td>88.40</td>
    <td>74.39</td>
    <td>68.10</td>
    <td>61.99</td>
    <td>93.60</td>
    <td>74.58</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>72.53</b></td>
    <td><b>67.12</b></td>
    <td>88.33</td>
    <td><b>76.28</b></td>
    <td>68.97</td>
    <td><b>62.70</b></td>
    <td><b>93.67</b></td>
    <td><b>75.11</b></td>
  </tr>
  <tr>
    <td rowspan="4">GQA</td>
    <td><i>base</i></td>
    <td>80.00</td>
    <td>77.08</td>
    <td>85.40</td>
    <td>81.02</td>
    <td>82.40</td>
    <td>77.03</td>
    <td>92.33</td>
    <td>83.99</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>81.73</td>
    <td>79.35</td>
    <td>85.80</td>
    <td>82.45</td>
    <td>82.27</td>
    <td>75.85</td>
    <td>94.67</td>
    <td>84.22</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>80.57</td>
    <td>76.77</td>
    <td><b>87.67</b></td>
    <td>81.85</td>
    <td>82.83</td>
    <td>76.64</td>
    <td>94.47</td>
    <td>84.62</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>86.47</b></td>
    <td><b>85.89</b></td>
    <td>87.27</td>
    <td><b>86.57</b></td>
    <td><b>85.00</b></td>
    <td><b>78.81</b></td>
    <td><b>95.73</b></td>
    <td><b>86.45</b></td>
  </tr>
  <tr>
    <td rowspan="4">Popular</td>
    <td><i>base</i></td>
    <td>73.53</td>
    <td>68.80</td>
    <td>86.13</td>
    <td>76.49</td>
    <td>72.03</td>
    <td>65.57</td>
    <td>92.80</td>
    <td>76.84</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>74.10</td>
    <td>69.45</td>
    <td>86.07</td>
    <td>76.87</td>
    <td>71.77</td>
    <td>64.90</td>
    <td>94.80</td>
    <td>77.05</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>74.57</td>
    <td>69.45</td>
    <td><b>87.83</b></td>
    <td><b>77.53</b></td>
    <td>72.83</td>
    <td>66.04</td>
    <td>94.00</td>
    <td>77.58</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>78.00</b></td>
    <td><b>73.68</b></td>
    <td>87.13</td>
    <td><b>79.84</b></td>
    <td><b>74.80</b></td>
    <td><b>67.46</b></td>
    <td><b>95.80</b></td>
    <td><b>79.17</b></td>
  </tr>
  <tr>
    <td rowspan="4">Adversarial</td>
    <td><i>base</i></td>
    <td>68.00</td>
    <td>63.49</td>
    <td>84.73</td>
    <td>72.59</td>
    <td>68.73</td>
    <td>62.54</td>
    <td>93.40</td>
    <td>74.92</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>70.27</td>
    <td>65.43</td>
    <td>85.93</td>
    <td>74.29</td>
    <td>68.27</td>
    <td>62.00</td>
    <td>94.40</td>
    <td>74.84</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>68.90</td>
    <td>64.06</td>
    <td>86.13</td>
    <td>73.47</td>
    <td>68.13</td>
    <td>61.88</td>
    <td>94.47</td>
    <td>74.78</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>73.07</b></td>
    <td><b>67.80</b></td>
    <td><b>87.87</b></td>
    <td><b>76.54</b></td>
    <td><b>69.20</b></td>
    <td><b>62.61</b></td>
    <td><b>95.33</b></td>
    <td><b>75.58</b></td>
  </tr>
</table>

Table 1: POPE benchmark results. AViSC consistently outperforms base decoding and other methods: VCD (Leng et al., 2023) and M3ID (Favero et al., 2024). We reimplemented VCD and M3ID in our evaluation setup.

Benchmarks. (1) POPE (Li et al., 2023b) views hallucination evaluation as a binary classification task (yes/no) with questions on object presence (e.g., "Is there a cat in the image?"). It evaluates both visible and imaginary objects across across three setups: random, popular, and adversarial. (2) MME (Fu et al., 2024) evaluates 14 subtasks—including object existence, count, position, and color—via binary questions. (3) AMBER (Wang et al., 2023) combines generative and discriminative tasks, focusing on hallucinations in object existence, attributes, and relationships. Generative performance is measured by CHAIR, discriminative by F1, with the overall AMBER score computed as ((100 − CHAIR) + F1)/2.

Baselines. We compare AViSC against recent contrastive decoding methods, notably VCD (Leng et al., 2023) and M3ID (Favero et al., 2024), which mitigate hallucinations by enhancing the reference image’s influence relative to the language model’s prior through contrasting output distributions from original and altered visual inputs. Our reimplementations of VCD and M3ID serve as baselines, as they too avoid external models, costly self-feedback, and additional training.

4.2 Benchmark Results

POPE. Table 1 summarizes performance on the POPE benchmark (Li et al., 2023b) across MS-COCO (Lin et al., 2014), A-OKVQA (Schwenk et al., 2022), and GQA (Hudson and Manning, 2019) datasets under the Random, Popular, and Adversarial setups. Overall, AViSC consistently outperforms the baseline (base) and other decoding methods (VCD, M3ID) in most cases, achieving the
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Object-level</th>
    <th colspan="2">Attribute-level</th>
    <th rowspan="2">Total Score</th>
  </tr>
  <tr>
    <th>Existence ↑</th>
    <th>Count ↑</th>
    <th>Position ↑</th>
    <th>Color ↑</th>
  </tr>
  <tr>
    <td rowspan="4">InstructBLIP</td>
    <td>base</td>
    <td>170.19<sub>(±11.12)</sub></td>
    <td>89.52<sub>(±11.04)</sub></td>
    <td>67.62<sub>(±14.04)</sub></td>
    <td>114.76<sub>(±9.60)</sub></td>
    <td>442.09<sub>(±31.51)</sub></td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>172.62<sub>(±8.92)</sub></td>
    <td>98.33<sub>(±15.99)</sub></td>
    <td>71.90<sub>(±13.42)</sub></td>
    <td>117.14<sub>(±10.70)</sub></td>
    <td>459.99<sub>(±16.56)</sub></td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>173.89<sub>(±10.52)</sub></td>
    <td>89.72<sub>(±13.44)</sub></td>
    <td>72.72<sub>(±14.77)</sub></td>
    <td>110.56<sub>(±7.20)</sub></td>
    <td>446.88<sub>(±28.54)</sub></td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>184.76<sub>(±5.56)</sub></b></td>
    <td>82.85<sub>(±12.16)</sub></td>
    <td><b>74.76<sub>(±6.19)</sub></b></td>
    <td><b>131.43<sub>(±4.76)</sub></b></td>
    <td><b>473.80<sub>(±19.67)</sub></b></td>
  </tr>
  <tr>
    <td rowspan="4">LLaVA 1.5</td>
    <td>base</td>
    <td>173.57<sub>(±8.16)</sub></td>
    <td>110.00<sub>(±15.82)</sub></td>
    <td>100.47<sub>(±18.78)</sub></td>
    <td>125.24<sub>(±15.91)</sub></td>
    <td>509.28<sub>(±30.57)</sub></td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>172.14<sub>(±8.09)</sub></td>
    <td>117.14<sub>(±8.76)</sub></td>
    <td>103.33<sub>(±20.56)</sub></td>
    <td>119.52<sub>(±8.58)</sub></td>
    <td>512.14<sub>(±31.82)</sub></td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>178.33<sub>(±6.83)</sub></td>
    <td>107.22<sub>(±14.78)</sub></td>
    <td>96.39<sub>(±5.52)</sub></td>
    <td>127.50<sub>(±8.28)</sub></td>
    <td>509.44<sub>(±22.52)</sub></td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>189.29<sub>(±1.82)</sub></b></td>
    <td>104.76<sub>(±11.66)</sub></td>
    <td><b>106.19<sub>(±13.93)</sub></b></td>
    <td><b>127.86<sub>(±9.13)</sub></b></td>
    <td><b>528.09<sub>(±24.70)</sub></b></td>
  </tr>
</table>

Table 2: MME-Hallucination (Fu et al., 2024) benchmark results. Our method effectively reduces hallucinations at both object and attribute levels, surpassing VCD (Leng et al., 2023) and M3ID (Favero et al., 2024) in Total Score.

![Performance comparison on MME-Fullset. AViSC achieves top performance in 7 of 14 categories with InstructBLIP and 11 in 11 categories with LLaVA-1.5. Beyond minimizing hallucinations, AViSC also boosts the general functionality of LVLMs.](page_101_1012_1242_246.png)

Figure 5: Performance comparison on MME-Fullset. AViSC achieves top performance in 7 of 14 categories with InstructBLIP (Dai et al., 2024) and in 11 categories with LLaVA-1.5 (Liu et al., 2023c). Beyond minimizing hallucinations, AViSC also boosts the general functionality of LVLMs.

highest Accuracy and F1 scores. Additionally, balanced improvements in Precision and Recall suggest reduced errors and better information capture. For InstructBLIP, AViSC yields a significant performance boost—especially in mitigating hallucinations related to object existence—while LLaVA-1.5 shows somewhat lower gains in the more challenging Popular and Adversarial setups. Nonetheless, AViSC proves robust across different datasets and query configurations.

MME-Hallucination. Table 2 presents performance results for InstructBLIP and LLaVA-1.5 on the MME-Hallucination benchmark (Fu et al., 2024). We evaluate both object-level metrics (Existence, Count) and attribute-level metrics (Position, Color). Both models show marked improvements in the Existence category when using AViSC, achieving the highest scores. While VCD slightly outperforms in the Count metric, AViSC excels in Position and Color, leading to superior Total Scores overall. These results affirm that AViSC effectively reduces hallucinations and improves accuracy across multiple dimensions.

MME-Fullset. Figure 5 compares various decoding methods on the MME-Fullset benchmark (Fu et al., 2024) across 14 categories. AViSC achieves top results in 7 categories for InstructBLIP and 11 for LLaVA-1.5. This indicates that AViSC enhances the model’s ability to extract and utilize informative visual features through attention calibration. Although both models experience a slight decline in the Count category with AViSC—and InstructBLIP shows lower performance on OCR tasks—LLaVA-1.5 sees significant OCR improvements, demonstrating that the impact of AViSC can vary across different models. Overall, AViSC delivers superior results across most tasks compared to the baselines.

AMBER. Table 3 shows results on the AMBER benchmark (Wang et al., 2023), which includes both generative and discriminative tasks. AViSC significantly improves discriminative performance (Accuracy and F1) for both InstructBLIP and LLaVA-1.5, outperforming Base, VCD, and M3ID. In generative tasks, it also achieves substantial gains, particularly in the Existence metric, indicating better object detection. Overall, AViSC enables both models to achieve the highest scores across most AMBER metrics.

4.3 Ablation Study

Ablations on \( \alpha \) and \( \lambda \). In our approach, \( \lambda \) is the threshold for identifying blind tokens that receive
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="4">InstructBLIP (Dai et al., 2024)</th>
    <th colspan="4">LLaVA-1.5 (Liu et al., 2023c)</th>
  </tr>
  <tr>
    <th>base</th>
    <th>VCD</th>
    <th>M3ID</th>
    <th>AViSC</th>
    <th>base</th>
    <th>VCD</th>
    <th>M3ID</th>
    <th>AViSC</th>
  </tr>
  <tr>
    <td>Generative</td>
    <td>8.40<sub>(±0.57)</sub></td>
    <td>7.60<sub>(±0.42)</sub></td>
    <td>6.85<sub>(±0.07)</sub></td>
    <td><b>6.70<sub>(±0.28)</sub></b></td>
    <td>7.95<sub>(±0.64)</sub></td>
    <td>6.70<sub>(±0.42)</sub></td>
    <td><b>6.00<sub>(±0.14)</sub></b></td>
    <td>6.25<sub>(±0.07)</sub></td>
  </tr>
  <tr>
    <td>Cover ↑</td>
    <td>46.40<sub>(±1.27)</sub></td>
    <td><b>47.65<sub>(±0.35)</sub></b></td>
    <td>47.20<sub>(±0.71)</sub></td>
    <td>46.65<sub>(±1.48)</sub></td>
    <td>44.45<sub>(±0.21)</sub></td>
    <td>46.50<sub>(±0.28)</sub></td>
    <td><b>48.90<sub>(±0.28)</sub></b></td>
    <td>46.55<sub>(±0.64)</sub></td>
  </tr>
  <tr>
    <td>Hal ↓</td>
    <td>31.10<sub>(±0.64)</sub></td>
    <td>29.90<sub>(±0.99)</sub></td>
    <td><b>27.50<sub>(±0.71)</sub></b></td>
    <td>28.00<sub>(±0.28)</sub></td>
    <td>31.00<sub>(±2.83)</sub></td>
    <td>27.80<sub>(±1.70)</sub></td>
    <td>26.00<sub>(±0.28)</sub></td>
    <td><b>25.60<sub>(±1.70)</sub></b></td>
  </tr>
  <tr>
    <td>Cog ↓</td>
    <td>2.60<sub>(±0.05)</sub></td>
    <td>2.20<sub>(±0.14)</sub></td>
    <td>2.20<sub>(±0.14)</sub></td>
    <td>2.55<sub>(±0.35)</sub></td>
    <td>2.15<sub>(±0.35)</sub></td>
    <td>1.95<sub>(±0.35)</sub></td>
    <td>1.45<sub>(±0.07)</sub></td>
    <td>2.00<sub>(±0.04)</sub></td>
  </tr>
  <tr>
    <td>Discriminative</td>
    <td>Acc. ↑</td>
    <td>68.20<sub>(±0.14)</sub></td>
    <td>69.65<sub>(±0.35)</sub></td>
    <td>69.05<sub>(±0.35)</sub></td>
    <td><b>72.60<sub>(±0.42)</sub></b></td>
    <td>67.00<sub>(±0.71)</sub></td>
    <td>67.30<sub>(±1.41)</sub></td>
    <td>67.25<sub>(±0.21)</sub></td>
    <td><b>70.70<sub>(±0.57)</sub></b></td>
  </tr>
  <tr>
    <td>Prec. ↑</td>
    <td>79.00<sub>(±0.14)</sub></td>
    <td><b>80.70<sub>(±0.42)</sub></b></td>
    <td>79.70<sub>(±0.28)</sub></td>
    <td>72.60<sub>(±0.42)</sub></td>
    <td>85.45<sub>(±0.49)</sub></td>
    <td>86.10<sub>(±1.70)</sub></td>
    <td><b>86.50<sub>(±0.57)</sub></b></td>
    <td>85.45<sub>(±0.21)</sub></td>
  </tr>
  <tr>
    <td>Rec. ↑</td>
    <td>70.70<sub>(±0.42)</sub></td>
    <td>71.60<sub>(±0.42)</sub></td>
    <td>71.25<sub>(±0.35)</sub></td>
    <td><b>76.10<sub>(±0.05)</sub></b></td>
    <td>60.95<sub>(±1.20)</sub></td>
    <td>60.55<sub>(±1.34)</sub></td>
    <td>60.05<sub>(±0.07)</sub></td>
    <td><b>67.55<sub>(±0.92)</sub></b></td>
  </tr>
  <tr>
    <td>F1 ↑</td>
    <td>74.60<sub>(±0.14)</sub></td>
    <td>75.90<sub>(±0.42)</sub></td>
    <td>75.25<sub>(±0.07)</sub></td>
    <td><b>78.60<sub>(±0.28)</sub></b></td>
    <td>71.10<sub>(±0.99)</sub></td>
    <td>71.10<sub>(±1.56)</sub></td>
    <td>70.90<sub>(±0.14)</sub></td>
    <td><b>75.45<sub>(±0.64)</sub></b></td>
  </tr>
  <tr>
    <td>AMBER ↑</td>
    <td>83.10<sub>(±0.35)</sub></td>
    <td>84.15<sub>(±0.05)</sub></td>
    <td>84.20<sub>(±0.07)</sub></td>
    <td><b>85.95<sub>(±0.05)</sub></b></td>
    <td>81.58<sub>(±0.18)</sub></td>
    <td>82.20<sub>(±0.99)</sub></td>
    <td>82.45<sub>(±0.14)</sub></td>
    <td><b>84.60<sub>(±0.35)</sub></b></td>
  </tr>
</table>

Table 3: AMBER (Wang et al., 2023) benchmark results. AViSC outperforms contrastive decoding baselines (Leng et al., 2023; Favero et al., 2024) in both generative and discriminative tasks, achieving the highest AMBER score.

<table>
  <tr>
    <th colspan="5">(a) InstructBLIP (Dai et al., 2024) (\( \lambda = 1 \))</th>
    <th colspan="5">(b) InstructBLIP (Dai et al., 2024) (\( \lambda = 3 \))</th>
    <th colspan="5">(c) LLaVA-1.5 (Liu et al., 2023b) (\( \lambda = 1 \))</th>
  </tr>
  <tr>
    <th>Object</th>
    <th>Attribute</th>
    <th>Total Score</th>
    <th>Object</th>
    <th>Attribute</th>
    <th>Total Score</th>
    <th>Object</th>
    <th>Attribute</th>
    <th>Total Score</th>
    <th>\( \alpha \)</th>
    <th>Exist.</th>
    <th>Count</th>
    <th>Position</th>
    <th>Color</th>
    <th>Total Score</th>
  </tr>
  <tr>
    <td>\( \alpha \)</td>
    <td>Exist.</td>
    <td>Count</td>
    <td>Position</td>
    <td>Color</td>
    <td>\( \lambda \)</td>
    <td>Exist.</td>
    <td>Count</td>
    <td>Position</td>
    <td>Color</td>
    <td>\( \alpha \)</td>
    <td>Exist.</td>
    <td>Count</td>
    <td>Position</td>
    <td>Color</td>
    <td>Total Score</td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>180</td>
    <td>83.33</td>
    <td>80.00</td>
    <td>130</td>
    <td>473.33</td>
    <td>0.0</td>
    <td>180</td>
    <td>75.00</td>
    <td>60.00</td>
    <td>115.00</td>
    <td>430.00</td>
    <td>0.5</td>
    <td>185</td>
    <td>111.66</td>
    <td>103.33</td>
    <td>115.00</td>
    <td>514.99</td>
  </tr>
  <tr>
    <td>2.0</td>
    <td>180</td>
    <td>86.66</td>
    <td>75.00</td>
    <td>135</td>
    <td>476.66</td>
    <td>1.0</td>
    <td>185</td>
    <td>60.00</td>
    <td>65.00</td>
    <td>123.33</td>
    <td>433.33</td>
    <td>2.0</td>
    <td>180</td>
    <td>103.33</td>
    <td>101.66</td>
    <td>120.00</td>
    <td>504.99</td>
  </tr>
  <tr>
    <td>2.5</td>
    <td>180</td>
    <td>85.00</td>
    <td>71.66</td>
    <td>135</td>
    <td>471.66</td>
    <td>1.0</td>
    <td><b>195</b></td>
    <td><b>75.00</b></td>
    <td><b>73.33</b></td>
    <td><b>135.00</b></td>
    <td><b>478.33</b></td>
    <td>2.5</td>
    <td>180</td>
    <td><b>105.00</b></td>
    <td><b>111.66</b></td>
    <td><b>120.00</b></td>
    <td><b>516.66</b></td>
  </tr>
  <tr>
    <td>3.0</td>
    <td><b>195</b></td>
    <td><b>75.00</b></td>
    <td><b>73.33</b></td>
    <td><b>135</b></td>
    <td><b>478.33</b></td>
    <td>1.5</td>
    <td>195</td>
    <td>75.00</td>
    <td>73.33</td>
    <td>135.00</td>
    <td>478.33</td>
    <td>3.0</td>
    <td>180</td>
    <td>105.00</td>
    <td>111.66</td>
    <td>120.00</td>
    <td>516.66</td>
  </tr>
</table>

Table 4: \( \alpha \) and \( \lambda \) ablations on MME-Hallucination (Fu et al., 2024). We set \( \alpha = 3, \lambda = 1 \) for InstructBLIP (Dai et al., 2024) and \( \alpha = 2.5, \lambda = 1 \) for LLaVA-1.5 (Liu et al., 2023b).

excessive attention (see Eq. (7)), and \( \alpha \) controls the strength of contrastive decoding (see Eq. (10)). We conducted ablation experiments on the MME-Hallucination benchmark (Liu et al., 2023d) to study their effects. Tab. 4 (a) and (c) show results using InstructBLIP (Dai et al., 2024) and LLaVA-1.5 (Liu et al., 2023c), respectively, with \( \lambda \) fixed at 1 and \( \alpha \) varied from 0.5 to 3. Overall, performance consistently improves with higher \( \alpha \) values, with InstructBLIP achieving the highest total score at \( \alpha = 3 \) and LLaVA-1.5 at \( \alpha = 2.5 \). These findings suggest that a stronger contrastive signal can better mitigate hallucinations. Additionally, Tab. 4 (b) shows that performance for InstructBLIP improves as \( \lambda \) increases, indicating that restricting the application of our method to a smaller set of highly attended tokens yields better results.

Ablations on \( \gamma \). We further evaluated the sensitivity of our approach to the parameter \( \gamma \), which determines the cumulative threshold for selecting layers based on image attention (see Eq. (5)). Using LLaVA-1.5 with \( \lambda = 1.0 \) and \( \alpha = 2.5 \), our experiments (shown in Tab. 5) reveal that performance remains robust across a range of \( \gamma \) values, except for extreme settings (e.g., \( \gamma = 0.1 \)). Our default value of \( \gamma = 0.5 \) yields high accuracy and balanced metrics on the POPE-COCO-Random benchmark, and achieves the highest total score on the MME-Hallucination benchmark. Overall, these results indicate that our approach is not highly sensitive to \( \gamma \), thereby reducing the need for extensive parameter tuning. Consequently, we fixed \( \lambda = 1.0 \) and \( \gamma = 0.5 \) in our experiments.

Layer Selection. We performed an ablation study to evaluate the effectiveness of our targeted layer selection strategy, which selects layers with the highest proportion of image-related attention for blind-token localization and calibration. To assess the contribution of this mechanism, we compared our method with three variants that manually fixed the selected layers (early, middle, or late five layers in the network). As shown in Tab. 6, our approach consistently outperforms all alternatives across both the POPE-COCO-Random and MME-Hallucination benchmarks. These results demonstrate that our targeted layer selection yields tangible improvements, albeit with modest margins, indicating that selecting layers with stronger image-related attention leads to more accurate blind-token localization and reduces hallucination.

5 Related Work

To mitigate hallucinations in LVLMs, researchers have developed strategies across three levels:
<table>
  <tr>
    <th colspan="6">(a) POPE-COCO-Random</th>
  </tr>
  <tr>
    <th>γ</th>
    <th>Acc.</th>
    <th>Prec.</th>
    <th>Rec.</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>0.5 (Ours)</td>
    <td>87.93</td>
    <td>88.24</td>
    <td>87.53</td>
    <td>87.88</td>
  </tr>
  <tr>
    <td>0.1</td>
    <td>86.77</td>
    <td>83.98</td>
    <td>90.87</td>
    <td>87.29</td>
  </tr>
  <tr>
    <td>0.3</td>
    <td>87.47</td>
    <td>85.35</td>
    <td>90.47</td>
    <td>87.83</td>
  </tr>
  <tr>
    <td>1.0</td>
    <td>88.27</td>
    <td>88.06</td>
    <td>88.53</td>
    <td>88.30</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="6">(b) MME-Hallucination</th>
  </tr>
  <tr>
    <th>γ</th>
    <th>Existence</th>
    <th>Count</th>
    <th>Position</th>
    <th>Color</th>
    <th>Total Score</th>
  </tr>
  <tr>
    <td>0.5 (Ours)</td>
    <td>189.29</td>
    <td>104.76</td>
    <td>106.19</td>
    <td>127.86</td>
    <td>528.10</td>
  </tr>
  <tr>
    <td>0.1</td>
    <td>167.50</td>
    <td>101.80</td>
    <td>103.33</td>
    <td>117.50</td>
    <td>490.13</td>
  </tr>
  <tr>
    <td>0.3</td>
    <td>180.00</td>
    <td>98.33</td>
    <td>114.16</td>
    <td>125.00</td>
    <td>517.49</td>
  </tr>
  <tr>
    <td>1.0</td>
    <td>182.50</td>
    <td>108.33</td>
    <td>109.99</td>
    <td>117.50</td>
    <td>518.32</td>
  </tr>
</table>

Table 5: γ ablations on (a) POPE-COCO-Random and (b) MME-Hallucination benchmarks with LLaVA-1.5 (\( \lambda = 1, \alpha = 2.5 \)).

<table>
  <tr>
    <th colspan="6">(a) POPE-COCO-Random</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>Acc.</th>
    <th>Prec.</th>
    <th>Rec.</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>w/ layer selection (Ours)</td>
    <td>87.93</td>
    <td>88.24</td>
    <td>87.53</td>
    <td>87.88</td>
  </tr>
  <tr>
    <td>Early 5 layers</td>
    <td>87.50</td>
    <td>86.77</td>
    <td>86.27</td>
    <td>87.12</td>
  </tr>
  <tr>
    <td>Mid 5 layers</td>
    <td>87.53</td>
    <td>88.20</td>
    <td>86.07</td>
    <td>87.13</td>
  </tr>
  <tr>
    <td>Last 5 layers</td>
    <td>87.46</td>
    <td>88.56</td>
    <td>85.47</td>
    <td>87.00</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="6">(b) MME-Hallucination</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>Existence</th>
    <th>Count</th>
    <th>Position</th>
    <th>Color</th>
    <th>Total Score</th>
  </tr>
  <tr>
    <td>w/ layer selection (Ours)</td>
    <td>189.29</td>
    <td>104.76</td>
    <td>106.19</td>
    <td>127.86</td>
    <td>528.10</td>
  </tr>
  <tr>
    <td>Early 5 layers</td>
    <td>180.00</td>
    <td>108.33</td>
    <td>106.66</td>
    <td>120.00</td>
    <td>514.99</td>
  </tr>
  <tr>
    <td>Mid 5 layers</td>
    <td>180.00</td>
    <td>108.33</td>
    <td>105.00</td>
    <td>120.00</td>
    <td>513.33</td>
  </tr>
  <tr>
    <td>Last 5 layers</td>
    <td>185.00</td>
    <td>103.33</td>
    <td>105.00</td>
    <td>120.00</td>
    <td>513.33</td>
  </tr>
</table>

Table 6: Ablation of layer selection on (a) POPE-COCO-Random and (b) MME-Hallucination benchmarks. Our targeted layer selection outperforms manual alternatives in all key metrics.

Input-level. These methods improve data quality and diversity by incorporating negative (Liu et al., 2023a) and counterfactual data (Yu et al., 2023a) or through dataset cleansing (Wang et al., 2024; Yue et al., 2024), thereby fostering more robust visual-text alignments during training.

Model-level. Approaches at this level enhance visual representations by increasing image processing resolution (Chen et al., 2023; Liu et al., 2023b; Zhai et al., 2023) or by leveraging advanced vision encoders (He et al., 2024; Jain et al., 2023; Tong et al., 2024b). Typically, these methods involve additional training with auxiliary supervision or reinforcement learning (Zhao et al., 2023; Gunjal et al., 2024; Sun et al., 2023; Yu et al., 2023b).

Output-level. Output-level methods directly refine the generated outputs. Contrastive decoding techniques (Leng et al., 2023; Favero et al., 2024) mitigate hallucinations by contrasting outputs from original and modified visual inputs, while guided decoding leverages external models like CLIP (Radford et al., 2021) or DETR (Carion et al., 2020) to steer generation. Other approaches include training-free methods (Wan et al., 2024; Zhang et al., 2024; Huang et al., 2023) and post-hoc corrections (Lee et al., 2023; Wu et al., 2024).

Our work falls within the output-level category.

Unlike prior contrastive decoding methods that contrast whole-image representations, AVISC analyzes the internal attention patterns of LVLMs to identify blind tokens—tokens that attract excessive attention but contribute little to the final output—and applies a contrastive decoding strategy to recalibrate their influence.

6 Conclusion

We identify and characterize blind tokens in LVLMs—image tokens that receive excessive attention while conveying little task-relevant information. These tokens misdirect the model’s focus, increasing the likelihood of hallucinated responses. To address this, we propose Attentional Vision Calibration (AVISC), a novel, training-free decoding technique that dynamically detects and mitigates the effect of blind tokens using image-wise attention analysis and contrastive decoding. Extensive evaluations on hallucination benchmarks demonstrate that AVISC improves both visual grounding and response accuracy, surpassing existing decoding strategies.

Limitations

While AVISC reduces hallucinations, its effectiveness declines in tasks requiring precise object counting (e.g., the “Count” category in MME and “Number” in AMBER; see Tabs. 14 and 15) This suggests that blind tokens may sometimes carry essential information for quantification. AVISC adds some overhead due to dynamic test-time recalibration but maintains competitive tokens-per-second throughput compared to other contrastive decoding methods. Unlike high-latency beam search approaches (e.g., OPERA), AVISC offers a better efficiency–accuracy trade-off. Though currently sequential, recalibration can be parallelized to reduce wall-clock time at the expense of memory. Performance on MME and AMBER varies with dataset scope and evaluation protocols, which are sensitive to token usage. Still, AVISC consistently lowers hallucination rates and yields statistically significant gains.

Future work. Building on insights from (Darcel et al., 2023), we hypothesize that blind token phenomenon may be intrinsic to large-scale transformer architectures, not limited to LVLMs. Future research will further explore these blind tokens and develop strategies to address them while balancing computational efficiency.
References

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liannmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36.

Timothée Darzet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2023. Vision transformers need registers. arXiv preprint arXiv:2309.16588.

Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. arXiv preprint arXiv:2403.14003.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.

Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135–18143.

Xin He, Longhui Wei, Lingxi Xie, and Qi Tian. 2024. Incorporating visual experts to resolve the information loss in multimodal large language models. arXiv preprint arXiv:2401.03105.

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2023. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. arXiv preprint arXiv:2311.17911.

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. Openclip. If you use this software, please cite it as below.

Jitesh Jain, Jianwei Yang, and Humphrey Shi. 2023. Vocoder: Versatile vision encoders for multimodal large language models. arXiv preprint arXiv:2312.14233.

Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. 2023. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07362.

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.16922.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023a. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c. Visual instruction tuning. Advances in neural information processing systems, 36.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023d. Mm-bench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146–162. Springer.

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525.

Shengbang Tong, Ellis Brown, Penghuo Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. 2024a. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024b. Eyes wide shut? exploring the visual shortcomings of multi-modal llms. arXiv preprint arXiv:2401.06209.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. 2024. Contrastive region guidance: Improving grounding in vision-language models without training. arXiv preprint arXiv:2403.02325.

Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. Amber: An llm-free multi-dimensional benchmark for mlms hallucination evaluation. arXiv preprint arXiv:2311.07397.

Xintong Wang, Jingcheng Pan, Liang Ding, and Chris Biemann. 2024. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715.

Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. 2024. Logical closed loop: Uncovering object hallucinations in large vision-language models. arXiv preprint arXiv:2402.11622.

Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueteng Zhuang. 2023a. Hallucidocor: Mitigating hallucinatory toxicity in visual instruction data. arXiv preprint arXiv:2311.13614.

Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2023b. Rlhf-v: Towards trustworthy mlms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849.

Zihao Yue, Liang Zhang, and Qin Jin. 2024. Less is more: Mitigating multimodal hallucination from an eos decision perspective. arXiv preprint arXiv:2402.14545.

Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. 2023. Halle-switch: Controlling object hallucination in large vision language models. arXiv e-prints, pages arXiv–2310.

Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. 2024. Debiasing large visual language models. arXiv preprint arXiv:2403.05262.

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
Appendix

Contents

A Visualizations & Analysis on Blind Tokens ..................................................................................... 12
B More Experimental Details .......................................................................................................... 13
  B.1 Further Implementation details ............................................................................................ 13
  B.2 Evaluation Benchmarks ...................................................................................................... 15
  B.3 Metrics .................................................................................................................................. 16
C Additional Experiments ............................................................................................................... 16
  C.1 Zero-Out Experiments on POPE-COCO Benchmark ......................................................... 16
  C.2 Inference Time and OPERA ............................................................................................. 17
  C.3 Alternatives to Zero-Out .................................................................................................... 17
  C.4 Results of Larger LVLMs ................................................................................................. 18
  C.5 Additional Evaluation on off-the-shelf LVLMs ................................................................. 18
  C.6 Additional Evaluation on Generative Benchmarks ............................................................. 18
  C.7 POPE (Li et al., 2023b) with Single-Word Constraint ..................................................... 18
  C.8 Detailed Results on MME-Fullset .................................................................................... 19
  C.9 Detailed Results on AMBER Discriminative Tasks ............................................................ 19
D Comparison with "Vision Transformers Need Registers" (Darceit et al., 2023) ................................................. 19
E Qualitative Results ....................................................................................................................... 25

A Visualizations & Analysis on Blind Tokens
In this section, we provide a comprehensive analysis of the attention biases observed in LVLMs through extensive visualizations. Our findings reveal that LVLMs tend to allocate excessive attention to certain image tokens—termed blind tokens—which, despite receiving high attention weights, contribute little to the final prediction logits.

More examples of attention bias. To illustrate the phenomenon, we present several examples using LLaVA-1.5-7B. As shown in Fig. 6, the figure depicts three images—black, Gaussian noise, and white—lacking any semantic or query-related information. Despite the question “Is there a carrot in the image?” and the absence of any meaningful objects or features, the model still concentrates its attention on certain regions. This highlights how blind tokens can dominate the attention mechanism, even when there are no informative cues present. Such behavior demonstrates the tendency of LVLMs to latch onto seemingly random patches in the absence of salient visual details. We define blind tokens as image tokens that draw excessive attention while contributing little to the final prediction logits.

Visualization of blind tokens and target objects. In Fig. 7, we overlay the locations of blind tokens with bounding boxes of target objects on images from the POPE-COCO benchmark. This visualization supports our claim that there is a mismatch between the highly attended blind tokens and the regions containing query-relevant information.

Distributions of bounding boxes and blind tokens. Heatmaps in Fig. 8 illustrate that while object bounding boxes tend to be centered, blind tokens are predominantly located along the image edges, revealing a significant spatial disparity.

Visualization and statistics of blind tokens. We conducted a correlation analysis on 3,000 images from the COCO2014 validation dataset. The results are in Fig. 9. LVLMs were tasked with describing images, and we analyzed the attention distribution over \(24 \times 24\) patches. Our results indicate that, on average, only \(3.7\%\) of blind tokens overlap with actual object regions, with merely \(23.3\%\) of the total attention weight allocated to these regions—highlighting the disconnect between blind tokens and task-relevant information.

Histogram of blind tokens. Fig. 10 presents a histogram of the number of blind tokens and their
Figure 6: Attention distribution for images that lack semantic or query-relevant information. Despite the absence of meaningful content, the model still focuses on certain regions, illustrating how blind tokens can dominate attention even in non-informative scenarios.

Figure 7: Visualization of blind tokens in real images from the POPE-COCO benchmark. Each row displays (left) the original image and (right) the same image with blind tokens highlighted. The red boxes indicate areas where query-related objects are located.

corresponding attention weights. In our evaluation with LLaVA-1.5-7B on the POPE-COCO-Random benchmark, we identified an average of 12.95 blind tokens, which accounted for 33.23% of the total image token attention weight.

Blind tokens and token probability distribution. Fig. 11 visualizes the location of blind tokens for a given image and query, and presents the token logit values of both the baseline model and AVISC.

Figure 8: Comparison of (left) bounding box distribution with (right) the distribution of blind tokens in the COCO dataset. Warmer colors indicate higher density, revealing that bounding boxes cluster near the center while blind tokens are more prevalent around the periphery. This highlights a spatial mismatch between regions containing genuine objects and areas receiving disproportionately high attention.

For example, in the first problem, which asked whether there is a banana in the image, the original probability distribution was: 'No' at 89.62%, 'Yes' at 8.46%, and 'There' at 1.56%. After applying AVISC, the logit distribution shifted to: 'No' at 98.00%, 'There' at 1.35%, and 'Yes' at 0.61%.

B More Experimental Details

B.1 Further Implementation details

Our decoding process employs cut-off sampling following VCD (Leng et al., 2023). Tokens with probability below \( \beta \) times the maximum probability at each generation step are masked. Formally, we consider text tokens \( \xi_t \in \mathcal{H} \) satisfying:

\[
\mathcal{H}(\xi_{<t}) = \{ \xi_t \in \mathcal{H} \mid p (\xi_t \mid \mathcal{V}, Q, \xi_{<t}; \theta) \}
\]

\[
\geq \beta \max_w p (w \mid \mathcal{V}, Q, \xi_{<t}; \theta) .
\]

We set \( \beta = 0.1 \) and limit generation to a maximum of 64 tokens per task. For LLaVA-1.5 (Liu et al., 2023c) experiments, we used the 11ava_v1 conversation template.
<table>
  <tr>
    <th>(a) Avg. # blind tokens in BBox / # BBox tokens</th>
    <td>3.68%</td>
  </tr>
  <tr>
    <th>(b) Avg. BBox token attention propotion</th>
    <td>23.2%</td>
  </tr>
</table>

Figure 9: Visualization and statistics of object bounding boxes and blind tokens in the COCO2014 dataset. Each row displays (from left to right) the original image, bounding box distribution, attention map, and blind tokens (in red boxes). On average, only 3.68% of blind tokens overlap with bounding boxes, while bounding box regions receive just 23.2% of the total attention. This highlights a clear mismatch between regions containing genuine objects and those receiving high attention.

For reproducing VCD (Leng et al., 2023), we followed the official code with \( \alpha = 1.0, \beta = 0.1 \), and a diffusion noise step \( T = 500 \). In our M3ID (Favero et al., 2024) reproduction, we set \( \lambda = 0.2 \). These settings ensure fair comparisons across methods.
Figure 10: Histograms illustrating the distribution of blind tokens in the POPE-COCO-Random benchmark. The left histogram shows the average number of blind tokens per image (about 13), while the right histogram indicates that these tokens account for roughly 33% of the total attention weight. This highlights the disproportionate influence blind tokens exert on the model’s attention.

Q: “Is there a banana in the image?”

Q: “Is there a banana in the image?”

Q: “Is there a grape in the image?”

Q: “What is the color of the leftmost object in the image?”

Figure 11: Visualization of blind tokens and the logit probability distributions before and after AViSC. Each row shows (left) the image with blind tokens highlighted, (center) the model’s original prediction logits, and (right) the logits adjusted by AViSC. By recalibrating attention away from blind tokens, AViSC increases the accuracy and confidence of the model’s responses to queries.

B.2 Evaluation Benchmarks

POPE. We utilize the official POPE benchmark (Li et al., 2023b), which includes 3,000 question-answer pairs (across random, popular, and adversarial setups) with queries of the form “Is there a [object] in the image?”. Performance is
measured by accuracy, precision, recall, and mean F1-score.\footnote{https://github.com/RUCAIBox/POPE}

MME. The MME dataset (Fu et al., 2024) is divided into 10 perceptual categories (existence, count, position, color, posters, celebrity, scene, landmark, artwork, OCR) and 4 cognitive categories (commonsense reasoning, numerical calculation, text translation, code reasoning). We use the official dataset but remove the one-word response constraint to allow natural responses.\footnote{https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation}

AMBER. AMBER (Wang et al., 2023) comprises 1004 images with both generative (e.g., “Describe this image.”) and discriminative (existence, attribute, relation) tasks. We randomly sample 500 questions for generative and 5000 for discriminative tasks, following official protocols.\footnote{https://github.com/junyangwang0410/AMBER.git}

LLaVA-Bench. LLaVA-Bench (Liu et al., 2023c) features 24 images and 60 questions covering diverse contexts (e.g., indoor, outdoor, paintings, sketches) to test LVLM adaptability.\footnote{https://huggingface.co/datasets/lihuoatian/llava-bench-in-the-wild}

B.3 Metrics

Metrics on the MME. For each visual input \( \mathcal{V} \) and its discriminative questions \( \{q_1, q_2\} \), we define accuracy (\( ACC \)) as:

\[
ACC(\mathcal{V}, q_i) = \begin{cases}
1 & \text{if LVLMs}(\mathcal{V}, q_i) \\
& = \text{Answer}(\mathcal{V}, q_i), \\
0 & \text{otherwise}.
\end{cases}
\] (12)

An additional metric, \( ACC+ \) (Fu et al., 2024), is 1 if both answers for an image are correct, and 0 otherwise.

\[
ACC+(\mathcal{V}) = \begin{cases}
1 & \text{if LVLMs}(\mathcal{V}, q_i) \\
& = \text{Answer}(\mathcal{V}, q_i) \text{ for any } i, \\
0 & \text{otherwise}.
\end{cases}
\] (13)

The overall MME score is the sum of \( ACC \) and \( ACC+ \).

Metrics on the generative tasks. Let R denote the response generated for a visual input V. We employ:

(1) \( CHAIR \) (Rohrbach et al., 2018; Wang et al., 2023) evaluates the occurrence of hallucinatory objects in responses to LVLMs. \( CHAIR \) uses an annotated list of objects \( A=\{a_{obj}^1, a_{obj}^2, \ldots, a_{obj}^n\} \) to calculate how often hallucinated objects appear in the responses. Let \( R=\{r_{obj}^1, r_{obj}^2, \ldots, r_{obj}^m\} \) be the list of objects mentioned in the response of LVLMs, the formula for \( CHAIR \) is given as:

\[
CHAIR = 1 - \frac{\text{len}(R \cap A)}{\text{len}(R)}.
\] (14)

(2) Cover (Wang et al., 2023) The Cover metric measures how completely the objects in the response cover the identified objects in the image. Cover calculates the ratio of objects mentioned in the response to the total objects listed. The formula for Cover is:

\[
Cover = \frac{\text{len}(R \cap A)}{\text{len}(A)}.
\] (15)

(3) Hal (Wang et al., 2023) The Hal metric quantifies the presence of hallucinations by checking if the \( CHAIR \) value is not zero, indicating the presence of hallucinations. The Hal is presented by the following formula:

\[
Hal = \begin{cases}
1 & \text{if } CHAIR \neq 0, \\
0 & \text{otherwise}.
\end{cases}
\] (16)

(4) Cog (Wang et al., 2023) The Cog metric evaluates whether the hallucinations in LVLMs responses resemble human cognition. The Cog calculates the ratio of the human hallucinatory object targets, denoted as \( H=\{h_{obj}^1, h_{obj}^2, \ldots, h_{obj}^n\} \) to the objects mentioned in the response. The formula for Cog is:

\[
Cog = \frac{\text{len}(R \cap H)}{\text{len}(R)}.
\] (17)

(5) AMBER Score (Wang et al., 2023) The AMBER Score metric evaluates the comprehensive performance of LVLMs for generative tasks and discriminative tasks. This score combines the CHAIR metric for generative tasks with the F1 metric for discriminative tasks. The formula representing the AMBER Score is as follows:

\[
AMBER\ Score = \frac{1}{2} \times (1 - CHAIR + F1).
\] (18)

C Additional Experiments

C.1 Zero-Out Experiments on POPE-COCO Benchmark

In Tab. 7, we compare how logits change under two strategies on the POPE-COCO benchmark (Rohrbach et al., 2018): (1) zeroing out blind
<table>
  <tr>
    <th>Case</th>
    <th># Case (Base)</th>
    <th># Case (AViSC)</th>
    <th>Logit Yype</th>
    <th>"Yes" Logit</th>
    <th>"No" Logit</th>
    <th>GT Logit - Wrong Logit ↑</th>
  </tr>
  <tr>
    <td rowspan="3">TP (GT = Yes) ↑</td>
    <td rowspan="3">3952</td>
    <td rowspan="3">3958 (+6)</td>
    <td>Baseline</td>
    <td>30.34</td>
    <td>25.68</td>
    <td>4.67</td>
  </tr>
  <tr>
    <td>Zero-out > \( \mu + \sigma \)</td>
    <td>28.78</td>
    <td>25.56</td>
    <td>3.22</td>
  </tr>
  <tr>
    <td>Zero-out < \( \mu + \sigma \)</td>
    <td>20.14</td>
    <td>19.40</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td rowspan="3">TN (GT = No) ↑</td>
    <td rowspan="3">3317</td>
    <td rowspan="3">3536 (+219)</td>
    <td>Baseline</td>
    <td>26.60</td>
    <td>28.88</td>
    <td>2.28</td>
  </tr>
  <tr>
    <td>Zero-out > \( \mu + \sigma \)</td>
    <td>25.82</td>
    <td>28.45</td>
    <td>2.63</td>
  </tr>
  <tr>
    <td>Zero-out < \( \mu + \sigma \)</td>
    <td>18.78</td>
    <td>19.21</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td rowspan="3">FP (GT = No) ↓</td>
    <td rowspan="3">1183</td>
    <td rowspan="3">964 (-219)</td>
    <td>Baseline</td>
    <td>28.01</td>
    <td>27.61</td>
    <td>-0.40</td>
  </tr>
  <tr>
    <td>Zero-out > \( \mu + \sigma \)</td>
    <td>26.75</td>
    <td>27.48</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Zero-out < \( \mu + \sigma \)</td>
    <td>19.33</td>
    <td>19.29</td>
    <td>-0.04</td>
  </tr>
  <tr>
    <td rowspan="3">FN (GT = Yes) ↓</td>
    <td rowspan="3">548</td>
    <td rowspan="3">542 (-6)</td>
    <td>Baseline</td>
    <td>27.42</td>
    <td>28.05</td>
    <td>-0.63</td>
  </tr>
  <tr>
    <td>Zero-out > \( \mu + \sigma \)</td>
    <td>26.41</td>
    <td>27.76</td>
    <td>-1.36</td>
  </tr>
  <tr>
    <td>Zero-out < \( \mu + \sigma \)</td>
    <td>19.08</td>
    <td>19.26</td>
    <td>-0.18</td>
  </tr>
</table>

Table 7: Zero-out experiments on the POPE-COCO benchmark (Rohrbach et al., 2018). We compare how logits change under two strategies: (1) zeroing out blind tokens (Zero-out > \( \mu + \sigma \)) and (2) zeroing out non-blind tokens (Zero-out < \( \mu + \sigma \)). Rows denote true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). For each case, we show the average “Yes” logit, “No” logit, and the difference between the ground-truth (GT) logit and the wrong logit. All results are obtained with LLaVA-1.5-7B (Liu et al., 2023b) on 3,000 MS-COCO (Lin et al., 2014) images.

<table>
  <tr>
    <th>Method</th>
    <th>Acc. ↑</th>
    <th>Prec. ↑</th>
    <th>Rec. ↑</th>
    <th>F1 ↑</th>
    <th>tokens/sec</th>
  </tr>
  <tr>
    <td>base</td>
    <td>84.47</td>
    <td>83.35</td>
    <td>86.13</td>
    <td>84.72</td>
    <td>24.44</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>84.80</td>
    <td>83.00</td>
    <td>87.53</td>
    <td>85.20</td>
    <td>11.53</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>86.00</td>
    <td>85.11</td>
    <td>87.27</td>
    <td>86.18</td>
    <td>13.14</td>
  </tr>
  <tr>
    <td>AviSC</td>
    <td>87.93</td>
    <td>88.24</td>
    <td>87.53</td>
    <td>87.88</td>
    <td>12.28</td>
  </tr>
  <tr>
    <td>OPERA (Beam=2)</td>
    <td><b>89.35</b></td>
    <td><b>90.37</b></td>
    <td><b>88.80</b></td>
    <td><b>89.58</b></td>
    <td>0.17</td>
  </tr>
</table>

Table 8: Comparison of inference time and performance on the POPE-COCO-Random benchmark for LLaVA-1.5. While OPERA achieves the highest performance metrics, it operates at a substantially slower speed compared to the other methods.

<table>
  <tr>
    <th>Case</th>
    <th>Acc. ↑</th>
    <th>Prec. ↑</th>
    <th>Rec. ↑</th>
    <th>F1 ↑</th>
  </tr>
  <tr>
    <td>InstructBLIP</td>
    <td>Zeros 88.50</td>
    <td>93.00</td>
    <td>83.27</td>
    <td>87.86</td>
  </tr>
  <tr>
    <td></td>
    <td>Ones 82.50</td>
    <td>75.48</td>
    <td>96.27</td>
    <td>84.62</td>
  </tr>
  <tr>
    <td></td>
    <td>Noise 86.77</td>
    <td>84.71</td>
    <td>89.73</td>
    <td>87.15</td>
  </tr>
  <tr>
    <td></td>
    <td>Mask <b>88.53</b></td>
    <td>90.14</td>
    <td>86.53</td>
    <td><b>88.30</b></td>
  </tr>
  <tr>
    <td>LLaVA 1.5</td>
    <td>Zeros 87.87</td>
    <td>88.12</td>
    <td>87.53</td>
    <td>87.83</td>
  </tr>
  <tr>
    <td></td>
    <td>Ones 79.97</td>
    <td>72.22</td>
    <td>97.40</td>
    <td>82.94</td>
  </tr>
  <tr>
    <td></td>
    <td>Noise <b>88.47</b></td>
    <td>93.19</td>
    <td>83.00</td>
    <td>87.80</td>
  </tr>
  <tr>
    <td></td>
    <td>Mask 84.77</td>
    <td>86.29</td>
    <td>82.67</td>
    <td>84.44</td>
  </tr>
</table>

Table 9: Design choices for non-blind image token deactivation. Each row presents a different method for handling non-blind tokens (Zeros, Ones, Noise, or Mask), and shows the resulting performance.

tokens (\emph{i.e.}, tokens with attention > \( \mu + \sigma \)) and (2) zeroing out non-blind tokens (\emph{i.e.}, tokens with attention < \( \mu + \sigma \)). Here, removing blind tokens minimally alters the model’s predictions, indicating that they hold little object-discriminative information. In contrast, removing non-blind tokens drastically shifts the logits, underscoring their critical importance. This indicates that blind tokens have a smaller impact on prediction logits than non-blind tokens. Compared to base decoding, AviSC effectively reduces over-emphasis on blind tokens, improving performance, particularly for TN and FP cases.

C.2 Inference Time and OPERA

Tab. 8 presents an efficiency and performance comparison between contrastive decoding methods (AviSC, M3ID, OPERA, and VCD) and AviSC. Inference speed is measured with a TITAN RTX GPU on the POPE-COCO-Random benchmark. OPERA introduces the concept of an “anchor token” and uses this token to guide sentence generation and rollback, thereby mitigating hallucinations. OPERA is implemented on the beam search decod-

ing method of LLMs, so a fair comparison with AviSC is not possible. However, OPERA showed the best performance overall. However, its inference speed was approximately ×72.23 slower than AviSC.

C.3 Alternatives to Zero-Out

Table 9 shows ablation results for various deactivation schemes applied to non-blind image tokens on the POPE-COCO-random benchmark (Li et al., 2023b), using both InstructBLIP (Dai et al., 2024) and LLaVA 1.5 (Liu et al., 2023c). We compare four methods: setting tokens to zero (Zeros), to ones (Ones), replacing tokens with noise (Noise), and masking tokens out in the attention mechanism (Mask). For InstructBLIP, the Mask approach achieves the highest Accuracy and F1 score, while the Zeros method excels in Precision; Ones yields the best Recall, and Noise offers balanced performance across Precision and Recall. For LLaVA 1.5, Noise achieves the highest Accuracy and Precision, whereas Zeros demonstrates consistent, balanced
<table>
  <tr>
    <th rowspan="2">Setup</th>
    <th rowspan="2">Method</th>
    <th colspan="4">LLaVA-1.5 (13B)</th>
  </tr>
  <tr>
    <th>Acc.</th>
    <th>Prec.</th>
    <th>Rec.</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="4">Random</td>
    <td><i>base</i></td>
    <td>83.17</td>
    <td>79.49</td>
    <td>89.40</td>
    <td>84.15</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>82.97</td>
    <td>78.90</td>
    <td>90.00</td>
    <td>84.09</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>83.43</td>
    <td>79.31</td>
    <td>90.47</td>
    <td>84.52</td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td>88.40</td>
    <td><b>86.05</b></td>
    <td><b>91.67</b></td>
    <td><b>88.77</b></td>
  </tr>
  <tr>
    <td rowspan="4">Popular</td>
    <td><i>base</i></td>
    <td>80.93</td>
    <td>76.45</td>
    <td>89.40</td>
    <td>82.42</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>79.67</td>
    <td>74.59</td>
    <td>90.00</td>
    <td>81.57</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>80.90</td>
    <td>75.94</td>
    <td>90.47</td>
    <td>82.57</td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>85.73</b></td>
    <td><b>81.94</b></td>
    <td><b>91.67</b></td>
    <td><b>86.53</b></td>
  </tr>
  <tr>
    <td rowspan="4">Adversarial</td>
    <td><i>base</i></td>
    <td>76.03</td>
    <td>70.74</td>
    <td>88.80</td>
    <td>78.75</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>75.57</td>
    <td>69.86</td>
    <td>89.93</td>
    <td>78.64</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>75.80</td>
    <td>69.97</td>
    <td>90.40</td>
    <td>78.88</td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>79.27</b></td>
    <td><b>73.65</b></td>
    <td><b>91.13</b></td>
    <td><b>81.47</b></td>
  </tr>
</table>

Table 10: Results of LLaVA-1.5-13B on POPE-COCO benchmark.

<table>
  <tr>
    <th rowspan="2">Setup</th>
    <th rowspan="2">Method</th>
    <th colspan="2">LLaVA-OneVision (Qwen2-7B)</th>
    <th colspan="2">Qwen2.5-VL-7B</th>
  </tr>
  <tr>
    <th>Acc.</th>
    <th>F1</th>
    <th>Acc.</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="4">Random</td>
    <td><i>base</i></td>
    <td>88.60</td>
    <td>87.34</td>
    <td>88.84</td>
    <td>87.82</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>90.57</td>
    <td>89.71</td>
    <td>90.47</td>
    <td>89.37</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>89.87</td>
    <td>88.91</td>
    <td>90.77</td>
    <td>88.57</td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>91.46</b></td>
    <td><b>90.84</b></td>
    <td><b>92.36</b></td>
    <td><b>91.50</b></td>
  </tr>
  <tr>
    <td rowspan="4">Popular</td>
    <td><i>base</i></td>
    <td>84.60</td>
    <td>83.79</td>
    <td>85.05</td>
    <td>83.72</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>87.20</td>
    <td>86.50</td>
    <td>87.10</td>
    <td>86.21</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>87.60</td>
    <td>85.65</td>
    <td>88.50</td>
    <td>85.36</td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>89.86</b></td>
    <td><b>89.45</b></td>
    <td><b>90.76</b></td>
    <td><b>90.16</b></td>
  </tr>
  <tr>
    <td rowspan="4">Adversarial</td>
    <td><i>base</i></td>
    <td>83.00</td>
    <td>82.40</td>
    <td>84.07</td>
    <td>82.73</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>86.00</td>
    <td>85.42</td>
    <td>86.50</td>
    <td>85.52</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>86.80</td>
    <td>84.86</td>
    <td>87.30</td>
    <td>84.96</td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>86.12</b></td>
    <td><b>85.83</b></td>
    <td><b>87.62</b></td>
    <td><b>86.93</b></td>
  </tr>
</table>

Table 11: POPE (MS-COCO) results on LLaVA-OneVision-Qwen2-7B and Qwen2.5-VL-7B.

performance across all metrics. Overall, the Ze-ros approach proved most effective in calibrating attention to image tokens and improving model performance.

C.4 Results of Larger LVLM

Tab. 10 presents the performance of each method on the POPE benchmark using the COCO dataset based on the LLaVA-1.5v-13B model. In this experiment setup, compared to the 7B small model shown in Tab. 1, the performance improvement of AViSC is even more pronounced. For other methods (<i>i.e.</i>, VCD, M3ID), the performance increase is slight or, in some cases, decreases depending on the metric. However, AViSC demonstrates robust performance improvement, remaining resilient to changes in the size of LVLMs.

C.5 Additional Evaluation on off-ths-shelf LVLMs

To further validate the model-agnostic robustness and generalizability of AViSC, we extended our evaluation to include recent and diverse LVLMs with varying attention mechanisms. Specifically, we tested AViSC on LLaVA-OneVision-Qwen2-7B, Qwen2.5-VL-7B. As shown in Tab. 11, AViSC consistently outperforms prior methods across all setups on the POPE benchmark, achieving significant reductions in hallucination without compromising the original model capabilities. These results support AViSC’s effectiveness as a test-time, plug-and-play strategy applicable across off-the-shelf LVLMs.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="2">MMHal-Bench</th>
    <th colspan="2">Object-Hallucination</th>
  </tr>
  <tr>
    <th>Score ↑</th>
    <th>HallRate ↓</th>
    <th>CHAIR↑</th>
    <th>CHAIR↓</th>
  </tr>
  <tr>
    <td rowspan="4">InstructBLIP</td>
    <td><i>base</i></td>
    <td>1.84</td>
    <td>0.64</td>
    <td>0.70</td>
    <td>9.1</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>1.75</td>
    <td>0.64</td>
    <td>0.80</td>
    <td>8.9</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>1.70</td>
    <td>0.65</td>
    <td>0.90</td>
    <td>7.6</td>
  </tr>
  <tr>
    <td>OPERA (Beam)</td>
    <td>–</td>
    <td>–</td>
    <td>16.6</td>
    <td><b>6.8</b></td>
  </tr>
  <tr>
    <td rowspan="4">AViSC</td>
    <td><b>2.03</b></td>
    <td><b>0.59</b></td>
    <td><b>0.70</b></td>
    <td>8.3</td>
  </tr>
  <tr>
    <td><i>base</i></td>
    <td>1.59</td>
    <td>0.72</td>
    <td>25.0</td>
    <td>9.2</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>1.96</td>
    <td>0.64</td>
    <td>23.6</td>
    <td>8.4</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td><b>2.14</b></td>
    <td><b>0.61</b></td>
    <td><b>23.2</b></td>
    <td><b>7.3</b></td>
  </tr>
  <tr>
    <td rowspan="4">OPERA (Beam)</td>
    <td>2.15</td>
    <td><b>0.54</b></td>
    <td>45.1</td>
    <td>22.3</td>
  </tr>
  <tr>
    <td>AViSC</td>
    <td><b>2.19</b></td>
    <td><b>0.59</b></td>
    <td><b>22.1</b></td>
    <td>7.8</td>
  </tr>
</table>

Table 12: Results on free-form generative benchmarks.

C.6 Additional Evaluation on Generative Benchmarks

To more comprehensively evaluate the impact of AViSC on generative capabilities, we conducted additional experiments on two free-form generation benchmarks: MMHal-Bench and Object-Hallucination. The results, summarized in Tab. 12, show that AViSC not only preserves the generative quality of the base models (e.g., maintaining or improving MMHal scores), but also consistently reduces hallucination metrics compared to existing approaches. These findings highlight that AViSC serves as a reliable and effective test-time method that retains the strengths of pretrained LVLMs in generative settings.

C.7 POPE (Li et al., 2023b) with Single-Word Constraint

As shown in Tab. 13, we see that imposing a one-word response constraint on LVLMs leads to notable changes in performance compared to Tab. 1. Despite the change in query setup, AViSC shows the best performance on the POPE benchmark. Specifically, precision and recall vary significantly in the COCO random setup comparing scenarios with and without the instruction, "Please answer this question with one word." To mitigate these impacts and better evaluate discriminative capabilities,
<table>
  <tr>
    <th rowspan="2">Setup</th>
    <th rowspan="2">Method</th>
    <th colspan="4">InstructBLIP (Dai et al., 2024)</th>
    <th colspan="4">LLaVA 1.5 (Liu et al., 2023c)</th>
  </tr>
  <tr>
    <th>Acc. ↑</th>
    <th>Prec. ↑</th>
    <th>Rec. ↑</th>
    <th>F1 ↑</th>
    <th>Acc. ↑</th>
    <th>Prec. ↑</th>
    <th>Rec. ↑</th>
    <th>F1 ↑</th>
  </tr>
  <tr>
    <td rowspan="3">MS-COCO</td>
    <td><i>base</i></td>
    <td>81.53</td>
    <td>82.71</td>
    <td>79.73</td>
    <td>81.19</td>
    <td>83.77</td>
    <td>92.31</td>
    <td>73.67</td>
    <td>81.94</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>82.03</td>
    <td>83.77</td>
    <td>79.47</td>
    <td>81.56</td>
    <td>85.43</td>
    <td>93.25</td>
    <td>76.40</td>
    <td>83.99</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>86.03</b></td>
    <td><b>95.53</b></td>
    <td>75.60</td>
    <td><b>84.41</b></td>
    <td>84.67</td>
    <td><b>97.88</b></td>
    <td>70.87</td>
    <td>82.21</td>
  </tr>
  <tr>
    <td rowspan="3">Popular</td>
    <td><i>base</i></td>
    <td>78.47</td>
    <td>77.73</td>
    <td>79.80</td>
    <td>78.75</td>
    <td>82.57</td>
    <td>89.62</td>
    <td>73.67</td>
    <td>80.86</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>79.13</td>
    <td>78.94</td>
    <td>79.47</td>
    <td>79.20</td>
    <td>83.17</td>
    <td>88.36</td>
    <td><b>76.40</b></td>
    <td>81.94</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>84.27</b></td>
    <td><b>91.45</b></td>
    <td>75.60</td>
    <td><b>82.77</b></td>
    <td><b>83.67</b></td>
    <td><b>95.25</b></td>
    <td>70.87</td>
    <td>81.27</td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td><i>base</i></td>
    <td>77.43</td>
    <td>76.09</td>
    <td>80.00</td>
    <td>78.00</td>
    <td>79.77</td>
    <td>83.85</td>
    <td>73.73</td>
    <td>78.47</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>77.23</td>
    <td>76.10</td>
    <td>79.40</td>
    <td>77.72</td>
    <td>80.27</td>
    <td>82.76</td>
    <td><b>76.47</b></td>
    <td>79.49</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>81.83</b></td>
    <td><b>86.20</b></td>
    <td>75.80</td>
    <td><b>80.67</b></td>
    <td><b>81.83</b></td>
    <td><b>90.99</b></td>
    <td>70.67</td>
    <td><b>79.55</b></td>
  </tr>
  <tr>
    <td rowspan="3">A-OKVQA</td>
    <td><i>base</i></td>
    <td>81.33</td>
    <td>78.52</td>
    <td>86.27</td>
    <td>82.21</td>
    <td>84.93</td>
    <td>89.16</td>
    <td>79.53</td>
    <td>84.07</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>81.57</td>
    <td>78.78</td>
    <td>86.40</td>
    <td>82.42</td>
    <td>85.53</td>
    <td>87.64</td>
    <td><b>82.73</b></td>
    <td>85.12</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>87.10</b></td>
    <td><b>89.95</b></td>
    <td>83.53</td>
    <td><b>86.62</b></td>
    <td><b>87.33</b></td>
    <td><b>95.09</b></td>
    <td>78.73</td>
    <td><b>86.14</b></td>
  </tr>
  <tr>
    <td rowspan="3">Popular</td>
    <td><i>base</i></td>
    <td>76.87</td>
    <td>72.69</td>
    <td>86.07</td>
    <td>78.82</td>
    <td>80.90</td>
    <td>81.77</td>
    <td>79.53</td>
    <td>80.64</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>77.30</td>
    <td>73.10</td>
    <td>86.40</td>
    <td>79.19</td>
    <td>81.17</td>
    <td>80.22</td>
    <td><b>82.73</b></td>
    <td>81.46</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>82.47</b></td>
    <td><b>81.79</b></td>
    <td>83.53</td>
    <td><b>82.65</b></td>
    <td><b>85.03</b></td>
    <td><b>90.08</b></td>
    <td>78.73</td>
    <td><b>84.03</b></td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td><i>base</i></td>
    <td>71.40</td>
    <td>66.67</td>
    <td>85.60</td>
    <td>74.96</td>
    <td>74.80</td>
    <td>72.63</td>
    <td>79.60</td>
    <td>75.95</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>72.47</td>
    <td>67.39</td>
    <td>87.07</td>
    <td>75.97</td>
    <td>75.03</td>
    <td>71.87</td>
    <td><b>82.27</b></td>
    <td>76.72</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>76.47</b></td>
    <td><b>73.16</b></td>
    <td>83.60</td>
    <td><b>78.03</b></td>
    <td><b>79.27</b></td>
    <td><b>79.58</b></td>
    <td>78.73</td>
    <td><b>79.16</b></td>
  </tr>
  <tr>
    <td rowspan="3">GQA</td>
    <td><i>base</i></td>
    <td>80.57</td>
    <td>77.47</td>
    <td>86.20</td>
    <td>81.60</td>
    <td>84.80</td>
    <td>87.88</td>
    <td>80.73</td>
    <td>84.16</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>81.73</td>
    <td>79.02</td>
    <td>86.40</td>
    <td>82.55</td>
    <td>85.63</td>
    <td>86.89</td>
    <td><b>83.93</b></td>
    <td>85.38</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>85.30</b></td>
    <td><b>88.57</b></td>
    <td>81.07</td>
    <td><b>84.65</b></td>
    <td><b>87.40</b></td>
    <td><b>95.17</b></td>
    <td>78.80</td>
    <td><b>86.21</b></td>
  </tr>
  <tr>
    <td rowspan="3">Popular</td>
    <td><i>base</i></td>
    <td>74.67</td>
    <td>70.17</td>
    <td>85.80</td>
    <td>77.20</td>
    <td>79.37</td>
    <td>78.59</td>
    <td>80.73</td>
    <td>79.64</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>74.63</td>
    <td>69.94</td>
    <td>86.40</td>
    <td>77.30</td>
    <td>78.73</td>
    <td>76.03</td>
    <td><b>83.93</b></td>
    <td>79.78</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>80.63</b></td>
    <td><b>80.37</b></td>
    <td>81.07</td>
    <td><b>80.72</b></td>
    <td><b>83.33</b></td>
    <td><b>86.66</b></td>
    <td>78.80</td>
    <td><b>82.54</b></td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td><i>base</i></td>
    <td>72.63</td>
    <td>67.78</td>
    <td>86.27</td>
    <td>75.92</td>
    <td>76.00</td>
    <td>74.13</td>
    <td>79.87</td>
    <td>76.89</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>71.93</td>
    <td>67.21</td>
    <td>85.67</td>
    <td>75.32</td>
    <td>76.40</td>
    <td>72.76</td>
    <td><b>84.40</b></td>
    <td>78.15</td>
  </tr>
  <tr>
    <td><b>AViSC</b></td>
    <td><b>77.60</b></td>
    <td><b>75.91</b></td>
    <td>80.87</td>
    <td><b>78.31</b></td>
    <td><b>80.37</b></td>
    <td><b>81.52</b></td>
    <td>78.53</td>
    <td><b>80.00</b></td>
  </tr>
</table>

Table 13: POPE (Li et al., 2023b) results with one-word constraint. We use the instruction "Please answer in one word." at the end of the query text.

we designed experiments that allow the LVLMs to freely make judgments and provide explanations for these judgments rather than restricting them to answers in one word.

C.8 Detailed Results on MME-Fullset

The detailed results on MME-Fullset are provided in Tab. 14. AViSC demonstrates substantial improvements in both LLaVA-1.5 and InstructBLIP across a wide range of perception and recognition tasks. These findings highlight the capability of AViSC to effectively handle diverse tasks, extending beyond hallucination mitigation, and suggest its potential to enhance the ability of LVLMs to accurately interpret and analyze visual information and query text appropriately.

C.9 Detailed Results on AMBER Discriminative Tasks

Tab. 15 presents the performance of the discriminative task on the AMBER benchmark across different categories. The discriminative task in the AMBER benchmark is divided into six categories: 'Existence', 'Attribute', 'State', 'Number', 'Action', and 'Relation', to evaluate the model's performance. For most categories, except for a few, both the LLaVA-1.5 and InstructBLIP models show performance improvements due to the applied AViSC.

D Comparison with "Vision Transformers Need Registers" (Darcel et al., 2023)

Summary of (Darcel et al., 2023). (Darcel et al., 2023) identify artifacts in vision transformer fea-
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Category</th>
    <th colspan="4">LLaVA 1.5 (Liu et al., 2023c)</th>
    <th colspan="4">InstructBLIP (Dai et al., 2024)</th>
  </tr>
  <tr>
    <th>base</th>
    <th>VCD</th>
    <th>M3ID</th>
    <th>AViSC</th>
    <th>base</th>
    <th>VCD</th>
    <th>M3ID</th>
    <th>AViSC</th>
  </tr>
  <tr>
    <td rowspan="8">Perception</td>
    <td>Existence</td>
    <td>173.57 (±8.16)</td>
    <td>172.14 (±8.09)</td>
    <td>178.33 (±6.83)</td>
    <td>189.29 (±1.89)</td>
    <td>170.19 (±11.12)</td>
    <td>172.62 (±8.92)</td>
    <td>173.89 (±10.52)</td>
    <td>184.76 (±5.56)</td>
  </tr>
  <tr>
    <td>Count</td>
    <td>110.00 (±15.82)</td>
    <td>117.14 (±8.76)</td>
    <td>107.22 (±14.78)</td>
    <td>104.76 (±11.66)</td>
    <td>89.52 (±11.04)</td>
    <td>98.33 (±15.99)</td>
    <td>89.72 (±13.44)</td>
    <td>82.85 (±12.16)</td>
  </tr>
  <tr>
    <td>Position</td>
    <td>100.47 (±18.78)</td>
    <td>103.33 (±8.56)</td>
    <td>96.39 (±5.52)</td>
    <td>106.19 (±13.93)</td>
    <td>67.62 (±14.04)</td>
    <td>71.90 (±13.42)</td>
    <td>72.72 (±14.77)</td>
    <td>74.76 (±8.19)</td>
  </tr>
  <tr>
    <td>Color</td>
    <td>125.24 (±15.91)</td>
    <td>119.52 (±8.58)</td>
    <td>127.50 (±8.28)</td>
    <td>127.86 (±9.13)</td>
    <td>114.76 (±9.60)</td>
    <td>117.14 (±10.70)</td>
    <td>110.56 (±7.20)</td>
    <td>131.43 (±4.76)</td>
  </tr>
  <tr>
    <td>Posters</td>
    <td>132.31 (±6.73)</td>
    <td>135.54 (±3.61)</td>
    <td>132.82 (±7.94)</td>
    <td>150.85 (±6.49)</td>
    <td>114.97 (±6.25)</td>
    <td>129.08 (±6.97)</td>
    <td>114.46 (±6.97)</td>
    <td>145.92 (±2.41)</td>
  </tr>
  <tr>
    <td>Celebrity</td>
    <td>114.56 (±6.45)</td>
    <td>118.09 (±7.69)</td>
    <td>113.38 (±0.21)</td>
    <td>125.59 (±2.50)</td>
    <td>113.38 (±3.95)</td>
    <td>123.82 (±4.99)</td>
    <td>114.12 (±2.91)</td>
    <td>120.29 (±9.0)</td>
  </tr>
  <tr>
    <td>Scene</td>
    <td>149.13 (±0.53)</td>
    <td>150.00 (±3.54)</td>
    <td>156.63 (±1.59)</td>
    <td>162.00 (±1.06)</td>
    <td>140.50 (±0.71)</td>
    <td>136.50 (±10.25)</td>
    <td>141.00 (±1.06)</td>
    <td>150.38 (±3.36)</td>
  </tr>
  <tr>
    <td>Landmark</td>
    <td>138.25 (±4.95)</td>
    <td>140.75 (±4.95)</td>
    <td>135.13 (±4.77)</td>
    <td>142.38 (±0.53)</td>
    <td>98.50 (±0.35)</td>
    <td>110.75 (±4.24)</td>
    <td>103.25 (±6.72)</td>
    <td>99.25 (±0.33)</td>
  </tr>
  <tr>
    <td>Artwork</td>
    <td>97.50 (±2.83)</td>
    <td>95.25 (±4.24)</td>
    <td>89.38 (±3.36)</td>
    <td>101.00 (±7.42)</td>
    <td>110.38 (±4.42)</td>
    <td>113.00 (±3.54)</td>
    <td>110.13 (±6.89)</td>
    <td>123.38 (±2.30)</td>
  </tr>
  <tr>
    <td>OCR</td>
    <td>91.25 (±19.45)</td>
    <td>101.25 (±1.77)</td>
    <td>96.25 (±15.91)</td>
    <td>143.75 (±5.3)</td>
    <td>87.50 (±21.21)</td>
    <td>91.25 (±8.84)</td>
    <td>85.00 (±10.61)</td>
    <td>68.75 (±5.3)</td>
  </tr>
  <tr>
    <td rowspan="5">Recognition</td>
    <td>Commonsense Reasoning</td>
    <td>100.36 (±2.53)</td>
    <td>96.79 (±5.56)</td>
    <td>87.14 (±12.12)</td>
    <td>102.86 (±7.07)</td>
    <td>96.43 (±1.01)</td>
    <td>107.14 (±8.08)</td>
    <td>99.64 (±2.53)</td>
    <td>101.79 (±6.57)</td>
  </tr>
  <tr>
    <td>Numerical Calculation</td>
    <td>80.00 (±7.07)</td>
    <td>66.25 (±8.84)</td>
    <td>76.25 (±12.37)</td>
    <td>65.00 (±14.14)</td>
    <td>68.75 (±1.77)</td>
    <td>66.25 (±15.91)</td>
    <td>71.25 (±22.98)</td>
    <td>73.75 (±5.30)</td>
  </tr>
  <tr>
    <td>Text Translation</td>
    <td>75.00 (±3.54)</td>
    <td>86.25 (±22.98)</td>
    <td>65.00 (±14.14)</td>
    <td>77.50 (±17.68)</td>
    <td>63.75 (±5.3)</td>
    <td>91.25 (±1.77)</td>
    <td>53.75 (±5.3)</td>
    <td>86.25 (±1.77)</td>
  </tr>
  <tr>
    <td>Code Reasoning</td>
    <td>62.50 (±10.61)</td>
    <td>61.25 (±1.77)</td>
    <td>71.25 (±15.91)</td>
    <td>71.25 (±5.30)</td>
    <td>73.75 (±5.30)</td>
    <td>57.50 (±0.00)</td>
    <td>81.25 (±1.77)</td>
    <td>76.25 (±5.3)</td>
  </tr>
</table>

Table 14: Results on MME-Fullset (Fu et al., 2024).

<table>
  <tr>
    <th rowspan="2">Category</th>
    <th colspan="4">LLaVA 1.5 (Liu et al., 2023c)</th>
    <th colspan="4">InstructBLIP (Dai et al., 2024)</th>
  </tr>
  <tr>
    <th>base</th>
    <th>VCD</th>
    <th>M3ID</th>
    <th>AViSC</th>
    <th>base</th>
    <th>VCD</th>
    <th>M3ID</th>
    <th>AViSC</th>
  </tr>
  <tr>
    <td>Existence</td>
    <td>68.55 (±0.21)</td>
    <td>67.15 (±1.91)</td>
    <td>68.50 (±0.14)</td>
    <td><b>75.35 (±0.21)</b></td>
    <td>72.05 (±0.49)</td>
    <td>73.20 (±1.27)</td>
    <td>72.95 (±0.21)</td>
    <td><b>81.35 (±0.07)</b></td>
  </tr>
  <tr>
    <td>Attribute</td>
    <td>67.85 (±0.49)</td>
    <td>69.50 (±1.27)</td>
    <td>68.20 (±0.42)</td>
    <td>69.80 (±0.85)</td>
    <td>68.40 (±0.14)</td>
    <td>69.90 (±0.14)</td>
    <td>69.15 (±0.92)</td>
    <td>70.80 (±1.56)</td>
  </tr>
  <tr>
    <td>State</td>
    <td>65.55 (±0.35)</td>
    <td>67.80 (±0.28)</td>
    <td>65.75 (±0.64)</td>
    <td>68.40 (±1.70)</td>
    <td>70.55 (±0.64)</td>
    <td>72.40 (±0.00)</td>
    <td>70.70 (±0.85)</td>
    <td>72.85 (±1.77)</td>
  </tr>
  <tr>
    <td>Number</td>
    <td>69.05 (±0.78)</td>
    <td>68.50 (±2.40)</td>
    <td>68.95 (±0.92)</td>
    <td>67.10 (±1.84)</td>
    <td>60.90 (±0.00)</td>
    <td>60.70 (±0.85)</td>
    <td><b>61.80 (±0.71)</b></td>
    <td>60.85 (±0.49)</td>
  </tr>
  <tr>
    <td>Action</td>
    <td>78.50 (±3.96)</td>
    <td>81.90 (±3.39)</td>
    <td>81.50 (±1.84)</td>
    <td>84.50 (±3.25)</td>
    <td>74.95 (±2.05)</td>
    <td>79.05 (±2.62)</td>
    <td>78.70 (±1.27)</td>
    <td><b>85.20 (±2.40)</b></td>
  </tr>
  <tr>
    <td>Relation</td>
    <td>58.80 (±4.10)</td>
    <td>57.75 (±0.07)</td>
    <td>59.70 (±3.39)</td>
    <td>60.50 (±0.14)</td>
    <td>56.05 (±1.63)</td>
    <td>58.00 (±1.41)</td>
    <td>57.00 (±1.98)</td>
    <td>54.65 (±2.76)</td>
  </tr>
</table>

Table 15: Results on AMBER discriminative tasks (Wang et al., 2023).

ture maps—specifically, “high-norm outlier tokens” that concentrate attention in redundant background areas. These tokens capture significant global information despite lacking local details, leading to poor performance in tasks requiring spatial precision. Notably, when additional memory (register tokens) is introduced, these artifacts vanish.

Differences from blind tokens. While both high-norm outlier tokens and our blind tokens exhibit unusually high attention weights in seemingly irrelevant regions, key differences exist:

• Source of Attention: High-norm tokens are computed within vision transformer layers, whereas our blind tokens are derived from the LLM’s attention in LVLMs (e.g., Vicuna-7B
InstructBLIP

Base: Yes, there is a wooden chair in the image, used primarily by the children.

VCD: There is a chair in the image, with kids sitting on it.

M3ID: Yes, there is a chair in the image.

AvisC: There are several children sitting at tables on mats, with no visible chairs in the image.

LLaVA-1.5

Base: Yes, there is a tv in the image. It sits atop a large modern dresser in the dining and living room area.

VCD: Yes, there is a flat screen television in the living room.

M3ID: Yes, there is a TV in the image, located in the living room area of the house.

AvisC: No, there is no TV in the image. The image features a dining room table with chairs, but it does not show any TV present in the image.

InstructBLIP

Base: No, there isn't a chair in the image.

VCD: There is no chair in the image.

M3ID: No, there is no chair in the image.

AvisC: There is a step stool or a chair in the image, which the little girl is standing on while brushing her teeth.

LLaVA-1.5

Base: No, there is no chair in the image. The image features a boy standing by a wall, surrounded by a bed with a comforter and a pillow.

VCD: No, there is no chair in the image. The scene takes place in a bedroom.

M3ID: No, there is no chair in the image. The image features a young boy standing on a bed with a flowered blanket.

AvisC: Yes, there is a chair in the image, located next to the boy standing on the pillows.

Figure 12: Qualitative examples on POPE (Li et al., 2023b).
InstructBLIP

Base: Yes, there is a black couch in the image.

VCD: Yes, there is a red couch with a black dog sitting on top of it in the image.

AvisC: No, there is a red couch in the image.

LLaVA-1.5

Base: Yes, there is a red and brown scarf wrapped around the teddy bear's neck in the image.

VCD: Yes, there is a brown scarf in the image, wrapped around the teddy bear.

AvisC: No, there is no brown scarf in the image. The teddy bear in the picture is wrapped in a red bandana or a red scarf.

InstructBLIP

Base: No.

VCD: No.

AvisC: Yes, the motorcycle is on the right side of the bus.

LLaVA-1.5

Base: No, there is no blue and yellow fire hydrant in the image.

VCD: Yes, there is a blue and yellow fire hydrant in the image.

AvisC: Yes, there is a blue and yellow fire hydrant in the image, located on the side of the road.

Figure 13: Qualitative examples on MME (Fu et al., 2024).
InstructBLIP

Query: Describe this image.

Base: In the video, a young girl is lying in a large ball pit, surrounded by various sized and colorful balls. She seems to be playing and enjoying herself, smiling and laughing as she plays with her new surroundings.

VCD: In the image, a young child is playing a ball pit, surrounded by colorful balls of different sizes and shapes. The child is fully immersed in the fun and enjoying the experience.

AvisC: The image features a little girl happily laying inside a ball pit filled with multiple colorful balls. She is having a good time as she smiles while being surrounded by the balls. The ball pit is quite full, with numerous balls placed across the area, indicating the child’s enjoyment.

Query: Is there one surfboard in this image?

GT: Yes

Base: No, there are multiple surfboards in the image.

VCD: No, there are multiple surfboards in the image.

AvisC: Yes, there is only one surfboard in the image, being ridden by the man.

InstructBLIP

Query: Describe this image.

Base: The image displays a playful dog in the water, with a tennis ball in its mouth. The dog is floating in the water, seemingly enjoying its time playing in the water. The dog is a medium-sized breed, likely a retriever, and is centered in the water.

VCD: The image showcases a brown dog swimming in a large body of water, holding a toy in its mouth. The dog is covered by water as it moves through the water, possibly retrieving the toy or enjoying the activity of swimming.

AvisC: The image features a golden retriever swimming in a body of water, carrying a piece of something in his mouth. The dog is skillfully navigating the water while holding onto the object, likely a toy or something else it found during its swim.

Query: Are there two lamps in this image?

GT: No

Base: Yes, there are two lamps in this image. Specifically there are two white bedside table lamps on either side of the bed.

VCD: Yes, there are two lamps in the image. One is located on the bedside table and the other is on the nightstand

AvisC: No, there is only one lamp in the image, placed next to the bed on a side table.

Figure 14: Qualitative examples of InstructBLIP (Dai et al., 2024) on AMBER (Wang et al., 2023).

in LLaVA-1.5-7B), with differences in masking strategies.

• Task and Architecture: Vision transformers are optimized for dense prediction tasks, and the emergence of high-norm tokens is sensitive to the training regime (e.g., DINOv1 vs. DINOv2). In contrast, LVLMs integrate a visual encoder with an LLM via an auto-regressive prediction scheme for image-based Q&A tasks.

• Domain: LVLMs project image tokens into an LLM space, altering attention dynamics
LLaVA-1.5

Query: Describe this image.

Base: The image shows a little girl dressed in a red dress, happily riding her yellow bike on a path. Behind her, there are houses likely in a fenced-in area, providing a cozy and quiet atmosphere for her to enjoy her ride.

VCD: The image features a little girl wearing a pink dress and standing next to her bicycle. She appears to be learning how to ride her bike, as she is positioned in front of a fence. The bicycle is situated to the right of the girl.

AvisC: In the image, a young girl wearing a red dress is riding her yellow bicycle near a white picket fence. She appears to be enjoying her ride down the driveways or the street, creating a charming scene.

Query: Does the lamp closed in this image?

Base: Yes, the lamp in the image is closed.

VCD: Yes, the lamp in the image is closed.

AvisC: No, the lamp in this image is open, providing light in the bedroom.

LLaVA-1.5

Query: Describe this image.

Base: A brown dog is skiing through a snow-covered field, creating its own mini-groomed path as it goes. The dog is accompanied by one person in the distance who seems to be providing guidance or looking at the dog as it runs.

VCD: The image captures a golden dog running across a snow-covered field in the middle of a park. The dog is enjoying the open space and appears to be moving at a fast pace. In the background, there are several trees lining the field, adding to the picturesque winter scenery.

Query: Is there a yellow frisbee in this image?

AvisC: In the image, a golden dog is running across a snow-covered field in a park. The dog appears to be enjoying the snow as it moves swiftly through the area. There are several trees surrounding the field, adding to the picturesque winters scene.

Base: Yes, the frisbee in the image is yellow.

VCD: No, the frisbee in this image is not yellow. Instead, it is orange.

AvisC: Yes, the frisbee in this image is yellow.

Figure 15: Qualitative examples of LLaVA-1.5 (Liu et al., 2023b) on AMBER (Wang et al., 2023).

compared to pure vision transformers.

Our experiments show a moderate correlation between high-norm and blind tokens (with \( P(\text{blind token} \mid \text{high-norm token}) = 40.38\% \) and \( P(\text{high-norm token} \mid \text{blind token}) = 31.27\% \)), suggesting shared underlying properties despite their differences. Additionally, blind tokens tend to appear at the beginning and end of the image token sequence—a pattern not clearly observed for
Query: Describe this photo in detail.

Base: Initially, this image portrays three green fruits, possibly ripe figs, with a white spot, harmlessly covering the inside of the fruit. They nestle among leaves, waiting for consumption or simply displayed for viewing. In addition to the three apples, their juxtaposition creates an aesthetic scene. The overall frame is filled with an inviting display of fresh fruits for those who appreciate or consume these delicious treats.

VCD: The image showcases a table with several fresh, ripe fruits cut in half. These fruits are not apples but rather a unique type of fruit that has a slightly purple tint, and their appearance is similar to grapefruits. The table displays four cut pieces of this fruit.

AvisC: The image features four plump purple fruit, possibly plums, placed on a wooden table. Three of the fruits are still whole, while the fourth fruit appears to be split in half, revealing its interior. The table provides an ideal surface for these fruits to be displayed on.

Query: Describe this photo in detail.

Base: In this bustling urban scene, a man in a yellow shirt is perched atop the back of a taxi or a yellow vehicle, possibly performing maintenance work on the car. This unique scene could incorporate elements from promoting an open source project such as a website.

VCD: The image depicts a busy city street with various vehicles and people. A man is prominently riding on the back of a yellow car, holding a blue shirt over the open hood of the vehicle. The car is parked on the side of the road.

AvisC: The image features a busy city street where a man is ironing clothes while sitting on the back of a yellow taxi cab. The taxi is parked, allowing the man to attend to his laundry amidst the urban bustle.

Figure 16: Response comparison on LLaVA-Bench (Liu et al., 2023c). Hallucinations are colored in red. AvisC demonstrates a robust understanding of images and reduces hallucinations in responses.

high-norm tokens in vision transformers.

On reducing dependency on blind tokens. Although high-norm tokens in (Darcel et al., 2023) encode global information, our findings indicate that blind tokens in LVLMs often lack query-relevant details. As shown in Fig. 2, the essential information is typically captured by non-blind tokens. Therefore, reducing the influence of blind tokens via our contrastive decoding scheme—while enhancing the role of non-blind tokens—effectively mitigates hallucinations without sacrificing critical information.

E Qualitative Results

We provide qualitative results on all benchmarks (POPE (Li et al., 2023b), MME (Fu et al., 2024), AMBER (Wang et al., 2023), and LLaVA-Bench (Liu et al., 2023c)) in Figs. 12 to 15. These highlight the differences between sentences generated by standard decoding (Base), VCD (Leng et al., 2023), and those produced by AvisC. The results demonstrate the effectiveness of AvisC in dealing with a variety of challenging visual contexts. Base and VCD often generate descriptions that include errors or hallucinations where elements not present in the image are described. In contrast, AvisC helps counteract these hallucinations, generating sentences that reflect a more accurate comprehension of the image.

Reviewer paper summaries: Large Vision Language Models (LVLMs) excel at understanding and describing visual inputs but often exhibit hallucinations---attributing incorrect or misleading features---due in part to an inherent misalignment in their attention mechanisms. In particular, researcher observe that LVLMs tend to allocate excessive attention to a small subset of image tokens---termed blind tokens---which do not carry the query-relevant information needed for precise image interpretation. To address this, this paper propose Attentional Vision Calibration (AvisC), a test-time approach that dynamically recalibrates the influence of these blind tokens during text generation. AvisC first identifies blind tokens by analyzing the layer-wise attention distribution over image tokens, then applies a contrastive decoding strategy that adjusts the prediction by balancing the original and blind-token-biased logits. Experiments on benchmarks including POPE, MME, and AMBER show that AvisC effectively reduces hallucinations in LVLMs.
Summary of strengths: Problem identification is accurate: the paper accurately identifies a key problem that exists in large-scale visual language models (LVLMs) - the models tend to allocate too much attention to a small fraction of irrelevant image tokens (called blind tokens) in an image when generating text, which can lead to the phenomenon of hallucination. This finding has important implications for understanding the performance bottleneck of LVLMs.

PROPOSING AN INNOVATIVE METHOD: To address the above problem, the paper proposes a novel test-time method, Attentional Visual Calibration (A VIS C). The method dynamically recalibrates the impact of blind tokens during the generation process without modifying the underlying model structure, which is highly practical and generalizable.

Sufficient experimental validation: The paper conducts extensive experiments on several baseline datasets, including POPE, MME, and AMBER, and demonstrates that the A VIS C method is effective in reducing illusory phenomena in LVLMs and improving the factual accuracy and descriptive richness of the generated text.

Significant Performance Improvement: Compared with the baseline method and other comparative methods, A VIS C achieves significant performance improvement in several evaluation metrics, especially in reducing hallucinatory phenomena and improving descriptive accuracy.
Summary of weaknesses: Insufficient exploration of the causes of blind tokens: although the paper identifies the problem of blind tokens, relatively little has been done to explore their causes. A more in-depth analysis of the causes of blind tokens could help to propose more fundamental solutions.

Increased computational overhead: the A VIS C method adds a certain amount of computational overhead by dynamically recalibrating the impact of blind tokens. This can be a limiting factor in certain application scenarios that require high real-time performance.

Limited effectiveness for precise counting tasks: the paper notes that the A VIS C method has relatively limited effectiveness in tasks that require precise counting. This suggests that the method may have limitations when dealing with certain specific types of tasks.
Comments, suggestions and typos: In-depth analysis of the causes of blind tokens: further analyzing the causes of blind tokens and exploring their performance patterns in different model architectures, training data and tasks. This helps to propose more targeted solutions to reduce the generation of blind tokens fundamentally.

Optimize the efficiency of the algorithm: to address the problem of high computational overhead of the AVISC method, optimization of the algorithm can be considered, for example, to improve the efficiency of the algorithm through parallel computation and reduction of unnecessary computational steps. Also, more efficient data structures and algorithm designs can be explored to reduce the computational complexity.

Combining multimodal information: in order to improve the effectiveness of AVISC methods in accurate counting tasks, combining multimodal information, such as text descriptions, voice commands, etc., can be considered. This information can provide more contextual clues for the model, which helps the model to understand the image content and count more accurately.

Introducing external knowledge: Introducing an external knowledge base or common sense knowledge can help the model better understand and interpret image content. For example, pre-trained visual-linguistic models or knowledge graphs can be utilized to enhance the model's ability to recognize and describe objects in an image.

Adaptive tuning of parameters: the parameters (e.g., α and λ) in the AVISC method are adaptively tuned for better performance for different models and tasks. This can be achieved by methods such as meta-learning or hyper-parameter optimization.

Some experiments on mainstream VLMs can be expanded.
Overall assessment scores: 3.5