Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models
Authors: Mingi Jung
Saehyung Lee
Eunji Kim
Sungroh Yoon
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models
Detailed Image Captioning
Attention-based Strategies
Primary area: deep_learning->attention_mechanisms

Abstract: Detailed image captioning is essential for tasks like data generation and aiding visually impaired individuals. High-quality captions require a balance between precision and recall, which remains challenging for current multimodal large language models (MLLMs). In this work, we hypothesize that this limitation stems from weakening and increasingly noisy visual attention as responses lengthen. To address this issue, we propose SPARC (Selective Progressive Attention ReCalibration), a training-free method that enhances the contribution of visual tokens during decoding. SPARC is founded on three key observations: (1) increasing the influence of all visual tokens reduces recall; thus, SPARC selectively amplifies visual tokens; (2) as captions lengthen, visual attention becomes noisier, so SPARC identifies critical visual tokens by leveraging attention differences across time steps; (3) as visual attention gradually weakens, SPARC reinforces it to preserve its influence. Our experiments, incorporating both automated and human evaluations, demonstrate that existing methods improve the precision of MLLMs at the cost of recall. In contrast, our proposed method enhances both precision and recall with minimal computational overhead.

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models
Mingi Jung' Saehyung Lee
Eunji Kim
Sungroh Yoon 123
Abstract
Baseline (0%)
Detailed image captioning is essential for tasks like data generation and aiding visually impaired individuals. High-quality captions require bal- ance between precision and recall, which remains challenging for current multimodal large lan- guage models (MLLMs). In this work, we hy- pothesize that this limitation stems from weak- ening and increasingly noisy visual attention as responses lengthen. To address this issue, we pro- pose SPARC (Selective Progressive Attention Re- Calibration), training-free free method that enhances the contribution of visual tokens during decod- ing. SPARC is founded on three key observations: (1) increasing the influence of all visual tokens re- duces recall; thus, SPARC selectively amplifies vi- sual tokens; (2) captions lengthen, visual atten- tion becomes noisier, so SPARC identifies critical visual tokens by leveraging attention differences across time steps; (3) as visual attention gradu- ally weakens, SPARC reinforces it to preserve its influence. Our experiments, incorporating both automated and human evaluations, demonstrate that existing methods improve the precision of MLLMs at the cost of recall. In contrast, our pro- posed method enhances both precision and re- call with minimal computational overhead. code: http:://ggiabbb.ommmmnnniooossss
Percentage Increase
Precision Recat F1
1. 1. 1
OPÉRA
VCD
VOLCANO
ouRs
Figure 1. Percentage change precision, recall, and Fl-score compared to the results before applying cach method.
ities (Li et al., 2023a; Liu et al., 2024c; Lin et al., 2024). These models leverage the powerful language capabilities of Large Language Models (LLMs) to generate textual descrip- tions from visual inputs (Bai al., 2023; Touvron et al., 2023; Abdin et al., 2024; Peng et al., 2023). This capabil- ity enables MLLMs to effectively perform a wide range of tasks, including Visual Question Answering (VQA), multi- modal reasoning, and image captioning (Chen et al., 2024; Wang et al., 2024; Li et al., 2024a; Liu al., 2024e).
Among these tasks, detailed image captioning aims pro- duce comprehensive yet accurate textual descriptions that capture both key elements and subtle nuances of an im- age. This capability is particularly important in applications such as content creation (Belyaeva et al., 2023; Han et al., 2023) or assistive technology for visually impaired individu- als (Hao al., 2024). However, a major challenge current approaches is hallucination models introduce in- correct irrelevant details compromising the reliability of the generated captions and limiting their practical util- ity (Bai et al., 2024; Liu et al., 2024d; Cui et al., 2023).
1. Introduction
Multimodal Large Language Models (MLLMs) have re- cently gained traction as a transformative approach artifi- cial intelligence by integrating visual and linguistic modal-
Although many methods have been proposed to reduce hal- lucinations (Huang et al., 2024; Leng et al., 2024; Lee et al., 2023; Liu al., 2025), demonstrate that existing meth- ods primarily enhance precision-thh the extent to which a cap- tion accurately reflects image at the expense of recall the extent to which caption comprehensively describes the image-ote often resulting in captions that, while more precise, omit essential details. To highlight this trade-off, we use the CHAIR benchmark (Rohrbach et al., 2018) to assess the effectiveness of existing hallucination mitigation methods in
'Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea 'Interdisciplinary Pro- gram in Artificial Intelligence, Seoul National University AAIIS, ASRI, INMC, and ISRC, Scoul National University. Correspon- dence to: Sungroh Yoon <sryoon @snu.ac.kr'.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
proves caption quality, outperforming existing methods. Un- like conventional methods that struggle balance precision and recall, SPARC effectively enhances both. Furthermore, human evaluations validate the superiority of SPARC in producing more precise and comprehensive captions.
Our contributions are summarized as follows:
Prompt:
Please describe this image distail.
Generated Text
Attention
We empirically show that existing MLLM hallucina- tion mitigation methods overlook recall.
Baseline
The image features bed with two catts lying on it. One cat is located on the left side the bed, while the other cat on the right side. Both
We propose SPARC, a novel attention-based method that improves MLLM image captioning in both pre- cision and recall. We provide empirical evidence sup- porting the design choices of the proposed method.
resting comfortably on pillows, with one cat lying on pillow near the center of the and other cat lying a pillow closer to the right side. In addition to the cots, there one two laptops on the bed. One laptop placed
left side the bed, and other leptop on the right side, A nemote control can also be seen on the left side of the bed.
Through automated and human evaluations, we show that SPARC, while training- free and computationally efficient, effectively enhances both precision and recall.
Ours
The image features bed with two cutt lying top of it. One cat on the left side of the bed, while the other is the right side. Both cats sleeping comfortably, with one of them wearing blue pillow on its head. In addition to the p there are several pillows the bed, including loptop computer placed on the left side of the bed. The bed is covered with green blanket, and the colorful pillows add cozy and inviting ctresphere the bed.
2. Related Work
2.1. Mitigating Hallucinations in MLLMs
MLLMs often generate hallucinations, where text is incon- sistent with visual input, and numerous studies have aimed to address this issue through various methods (Li et al., 2023b; Liu et al., 2024d; Gunjal et al., 2024). Decoding- based methods tackle hallucination by penalizing uncertain token generations through techniques like text aggregation analysis (Huang al., 2024), corrupted visual inputs (Leng et al., 2024; Gong et al., 2024). Self-refinmment strategies are also employed to iteratively align generated captions with visual content (Zhou et al., 2023; Lee et al., 2023). In addi- tion, research indicates that hallucinations often arise from an over-reliance on textual context while neglecting visual information (Zhu et al., 2024; Liu et al., 2025). To address this imbalance, decoding strategies and attention calibration techniques have been developed to enhance the utilization of relevant visual elements based on their importance (Huo et al., 2024; Liu et al., 2025; Li et al., 2025). Hallucination issues intensify with long-form text generation, as models rely more on text and less on image content (Favero et al., 2024; Lee al., 2024; Zhong et al., 2024). Methods such as shortening text length (Yue al., 2024), segmenting refine- ment (Lee al., 2024), and leveraging image-only logits and contrastive decoding (Zhong et al., 2024; Favero et al., 2024) have been explored. Existing solutions have not effec- tively addressed the challenge of enhancing context-relevant visual information, as vision-related signals tend to weaken over longer contexts.
Figure 2. Visualization of image token attention at different context lengths. As the generation context length increases, image atten- tion diminishes, reducing reliance on visual inputs. Our approach mitigates this by preserving image attention, reducing hallucina- tions and cnabling more detailed captions.
terms of recall and precision. Notably, our analysis reveals previously overlooked limitation: these methods signif- icantly reduce model recall (Figure 1). Our work aims to introduce a new approach that balances precision and recall.
In this work, we hypothesize that MLLMs do not fully re- alize their potential in terms of precision and recall. We attribute this limitation to the model's s focus on visual to- kens gradually weakening as it generates longer text and its increasing sensitivity to irrelevant noise, as illustrated in Figure 2. To address these issues, we propose SPARC (Se- lective Progressive Attention ReCalibration), a training-free method designed to enhance the contribution of key visual tokens during the decoding process of MILMs. Specifically, SPARC is built upon the following three principles: (1) naively increasing the influence of all visual tokens reduces recall; therefore, SPARC selectively amplifies the influence of visual tokens; (2) despite noisy visual attention patterns, SPARC identifies key visual tokens by leveraging attention differences across time steps; (3) to compensate for the weakening influence of visual tokens, SPARC accumulates reinforcement effects to sustain their impact
2.2. Visual Attention in MLLMS
MLLMs utilize transformer baased architectures with atten- tion mechanisms to integrate visual and textual modali-
Our experiments demonstrate that SPARC significantly im-
2

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMs
ties (Vaswani, 2017; Basu et al., 2024; Osman al., 2023). During text generation, attention weights do not always fo- cus on the most relevant visual tokens (Zhang et al., 2024;
20
20
Output Tokon Indux
Woo et al., 2024; Jiang et al., 2024). Prior studies have shown that enhancing image attention can help mitigate hal- lucinations by improving the model's ability to better align with relevant visual content (Li et al., 2024b; Xing et al.,
40-
60-
20-
Output
80
80-
2024). Efforts to achieve this include increasing image at- tention, adjusting positional embeddings (Xing al., 2024;
20
40 60
80
20
40
60
10
Li et al., 2025), and correcting biases that direct attention to irrelevant regions. (Jiang et al., 2024; Gong et al., 2024; Kang et al., 2025). Several approaches have been proposed, such as boosting overall image attention (Liu et al., 2025; Jiang et al., 2024) and reweighting key tokens (Xing et al., 2024) better focus on meaningful visual regions. Our findings show that as the model generates longer text, its focus on key visual tokens weakens, while sensitivity ir- relevant noise increases. Existing methods struggle to retain key visual tokens in such cases, making it difficult to pre- serve their relevance. In contrast, our approach reinforces key visual tokens during decoding, ensuring their continued importance. This improves caption detail and accuracy with minimal computational cost.
Output Token Index
Output Token Index
(a)
(b)
Figure 3. Visual attention diversity comparison between (a) the baseline model and (b) the naive attention enhancement approach. The naive approach reduces visual attention diversity, indicating ineffective adaptation to important visual tokens
normalized visual attention weights for the first 100 tokens of each caption. We then compute a pairwise distance ma- trix (100 x 100) of visual attention patterns. The distance is calculated using the Wasserstein distance (Vallender, 1974; Shen et al., 2018).
Figure 3 illustrates visual attention diversity during caption- ing, comparing the baseline model with a naive attention enhancement method (Liu et al., 2025). The results show that naive attention enhancement (right) yields lower dis- tance values compared to the baseline (left), indicating a reduction in visual attention diversity. Notably, naive at- tention enhancement causes the model's visual attention pattern to remain static throughout caption generation. Con- sequently, we consider this a key factor contributing to recall degradation, as it reduces the diversity of objects mentioned in the captions and limits their descriptiveness.
3. Why More Attention Doesn't 't Always Help
Intuitively, enhancing the influence of visual tokens during MLLM decoding could lead to higher-quality captions. A recent study (Liu et al., 2025) has shown that amplifying visual attention can indeed improve MLLMs' precision. However, Figure 1 reveals that this improvement comes at the cost of significant decline in recall. In this section, we explore this issue in depth, analyzing why the naive amplification of visual attention-inereasing all attention weights assigned to visual tokens can lead to captions with lower recall. We identify key factors contributing to these limitations and discuss potential strategies to mitigate them.
These findings highlight the need for a carefully designed approach to visual attention enhancement. Properly identi- fying and emphasizing the most relevant visual tokens is crucial for generating captions that are both contextually rich and informative.
3.1. Does More Attention Reduce Diversity?
When generating a detailed and comprehensive caption for an image, person's gaze naturally moves across differ- regions of the image. Similarly, we hypothesize that MLLMs producing high-recall captions attend diverse locations within the image during decoding. Based on this hypothesis, we analyze the impact of naive attention am- plification on visual attention dynamics. Specifically, we examine the pairwise distances between visual attention patterns obtained during decoding. If an MILLM shifts its at- tention across various regions of an image while generating caption, the distances between its visual attention patterns should be large. To investigate this, we generate 3,000 cap- tions using VA-1.5 (Liu et al., 2024a) and subset of DOCCI (Onoe et al., 2025). During generation, we store the
3.2. Longer Context, More Noise?
We analyzed how the model's attention to visual tokens evolves throughout the caption generation process to inves- tigate why directly amplifying attention based on its mag- nitude can reduce attention diversity. For this analysis, we extracted attention weights from one of the middle-to-late transformer layers, averaged them across attention heads, and normalized the values across all visual tokens. Figure 4 provides a visualization that reveals several notable patterns.
In the early stages of caption generation, attention is primar- ily directed toward image regions corresponding to salient visual elements, thereby ensuring that the visual attention
3

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Image Attention Ratio
2.35
Text Attention Rabio
0.80
2.10-
0.50
9:25-
2:20-
Image 01-Q3
Image
0.85
9:35-
Text
0.00
Text 01 Q3
9:30
0.00
Context length
80
Prompt: Please describe this image in detail.
Figure 5. Average attention weight trends for text and image tokens as function of context length during caption generation. As the context length increases, the proportion of attention allocated to image tokens gradually decreases compared to text tokens. This indicates a significant shift in focus from visual to textual elements during the later stages of caption generation.
Weaken
skiing
skier
skis
skier
Attention
The image features person wearing a green jacket and skiing across snow-covered field, The skier is holding ski poles in their honds, and they appear to enjoying the winter activity. The person the main focus of the scene, with the skis clearly visible beneath them.
The snowy landscape extends beyond the skier, creating a serene and picturesque winter scene. The skier's presence in the middle of the field adds sense of odventure and excitement to the image.
the total attention weight allocated to image tokens and text tokens for each output token during the caption generation process and plot their distribution with respect to the context length. Figure 5 presents the results of this analysis, illus- trating that the context length increases, the proportion of attention allocated to image tokens gradually decreases compared to text tokens. This finding aligns with previous studies (Favero et al., 2024; Lee et al., 2024), which reported that longer captions often lead to increased hallucinations due to decline in the model's focus on relevant visual information.
Figure 4. Visualization of the temporal dynamics of image atten- tion during caption generation. Early in the process, attention is focused on contextually relevant regions, but as the caption grows longer, increasingly shifts toward noise or consistently high- attention tokens.
mechanism itself remains aligned with key parts of the im- age. However, as the caption progresses, this alignment weakens. The intensity of attention on relevant regions di- minishes, while attention increasingly concentrates on noise or on specific visual tokens that consistently attract high attention weights. These noisy attention patterns are often unrelated to the context of generation and tend to recur at the same visual token positions throughout the generation process. These visual tokens, often located in background re- gion, may capture global context (Darcet al., 2024; Kang al., 2025). However their excessive attention dominance can overwhelm local, task-relevant signals. Consequently, commonly used strategies that amplify tokens according to their current attention values may inadvertently exacerbate this problem: as the caption becomes longer, boosting al- ready high-attention tokens can reinforce irrelevant or noisy regions, ultimately impairing the model's 's ability to focus on truly important parts of the image.
To address the diminishing focus on visual information, which can weaken the image's influence and potentially lead to hallucination, it is essential to ensure sustained vi- sual attention throughout the caption generation process. A gradual increase in the emphasis on visual tokens, par- ticularly in the later stages, helps counteract this declining trend. By progressively reinforcing the model's 's visual focus over time, it can better retain and utilize the image con- text, ultimately enhancing the accuracy and relevance of the generated captions.
The detailed implementation details of the above experi- ments can be found in Appendix B.
4. Method
In Section 3, we observed that as the context length gener- ated by the MLLM increases, the attention to visual tokens becomes noisier and its proportion decreases. To address this issue, we propose Selective Progressive Attention Re- Calibration (SPARC), training free approach that incurs minimal additional computational cost. SPARC consists of two key mechanisms: 1) Token selection using the Rela- tive A Activation Score, which robustly selects visual tokens relevant to the actual context and resists the increasing noise as the context length grows, and 2) Selective Progressive Attention Re-Calibration, which progressively reinforces the attention to important visual tokens during each genera-
These findings underscore the need for an adaptive visual attention mechanism that can effectively identify and prior- itize contextually meaningful visual tokens while filtering out noise.
3.3. Does Longer Context Weaken Visual Focus?
We analyzed the model's focus on visual information, con- sidering how it changes with caption length from an atten- tion perspective. Specifically, we generated 3,000 captions using -1.55 with the DOCCI subset and then analyze
4

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
tion step, thereby alleviating the decline in visual attention as the context length increases.
activation score, a dynamic metric that prioritizes tokens with significant relative increases in attention. This score is designed to compare the current attention value with smoothed historical trend, emphasizing relative changes rather than absolute magnitudes. By normalizing these changes with respect to the smoothed values, our method ensures that meaningful variations remain detectable, even as attention scales evolve.
4.1. Preliminaries: Attention Mechanisms in MLLMS
MLLMs are designed to process both image and text inputs, generating text outputs in an autoregressive manner (Li et al., 2024a; Lin et al., 2024). When generating the i-th text token, the models attend to an input sequence of length
Instead of relying solely on instantaneous attention values, our approach focuses on temporal variations in attention. By comparing the current attention value against its historical trend, we minimize the influence of static biases or localized noise, effectively highlighting tokens with sharp increases in relevance even if their raw attention values are not the highest. To stabilize the process, we apply an Exponential Moving Average (EMA), which smooths out transient fluc- tuations and prevents brief spikes or dips in attention from disproportionately influencing the selection process. As the context length grows, attention values may diminish overall, which can obscure important variations. By dynamically adapting to these scale changes through normalization, our method remains sensitive significant shifts in attention, regardless of the overall reduction in attention magnitude over time.
N₂ Ninage + Nust + (),
consisting of Nimage image embeddings, NMmst instruction tokens, and the i - 1 tokens generated so far.
At the (-th layer and h-th attention head, let the query, key, and value vectors be Q.t.A) K((Ih), v((ah) R&,, where i indicates the current token being generated and j ranges over all N₁ tokens in the input (i.e., j E {1.2,... N,)). (1.h). denoting how much the i-th The attention weight t.j
token attends the j-th, is given by
10A = (0 T (C
(L.) oif = softmax, (4
Vd
(1)
Here, d is the scaling factor. Using these attention weights, the output representation of (().)) for the i-th token at the I-th layer and h-th attention head is
Concretely, we employ an Exponential Moving Average (EMA) to track the historical trend of attention weights, allowing us to dynamically compare the current attention weight to its smoothed past values. This enables our method to detect relative increases in attention with high precision, while filtering out transient noise. we first maintain an EMA of the attention weights up to the (i 1)-th step:
N. Σ
of
(1,h)
(1.) ii
V(M
The outputs from all H attention heads at the I-th layer are concatenated and projected back to the original dimension.
ر|_ Bāl_ (1 -8)a(_1.5
(4)
key challenge in MILMs is their limited ability to effec- tively utilize image information during text generation (Zhu et al., 2024; Zhong al., 2024). To address this issue, several methods have been proposed to explicitly enhance attention to image tokens during the text generation pro- cess (Jiang et al., 2024; Zhang et al., 2024). A simple way to boost attention to image tokens is to modify as follows (Liu et al., 2025):
where E [0. 1] isis the smoothing factor that determines the relative weighting of past and current attention values. _ represents the attention weights for the j-th token at the (i 1)-th step, averaged across all attention heads h.
Using this smoothed trend, we define the Relative Activa- tion Score for the j-th image token as:
A (L,) < A
(L.b) + + 1,
ah. dt- d 1
for j (1,2,..., Nimage ).
where a is scaling factor that amplifies the attention given to image tokens.
Here, d/-1,5 represents the EMA-smoothed attention weight for token j at layer 1. This scoring mechanism emphasizes tokens with substantial relative increases in attention, effec- tively prioritizing those that become more salient over time while minimizing the impact of static or irrelevant tokens.
These approaches leverage the premise that attention mech- anisms capture critical visual information, reinforcing at- tention to important visual tokens in proportion to their significance.
Based on these relative activation scores, we apply a thresh- olding mechanism to select the most relevant image tokens. The set of selected tokens S; is defined as:
4.2. Token Selection: Relative Activation Score
To effectively identify contextually relevant visual tokens as the generated context expands, we propose a Relative
S = ( r > 7):
(6)
5

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
where T is a predefined threshold. Tokens exceeding this threshold are treated the significant visual elements for generating the i-th text token. By dynamically adjusting to shifts in attention patterns, this method focuses on visually relevant tokens while mitigating noise and static biases.
Leveraging this cached information, SPARC efficiently up- dates value vectors without additional memory overhead, ensuring seamless recalibration through weighted sums of value vectors (e.g., Equation (2)).
In summary, SPARC addresses attention decay by progres- sively reinforcing the significance of contextually relevant image tokens. This method prevents attention sinks and noise while preserving alignment between visual and textual contexts, resulting in more accurate, diverse, and visually grounded captions.
4.3. Selective Progressive Attention Re-Calibration
As text generation progresses, attention to important image regions often diminishes, leading to a misalignment between visual and textual contexts. To address this issue, we propose Selective Progressive A Attention Re-Calibration (SPARC), a mechanism that dynamically reinforces attention on rele- vant image tokens at each step of text generation. SPARC ensures that contextually significant visual tokens main- tain their importance throughout the captioning process, enabling the model to produce richer and more accurate descriptions.
5. Experiments
We evaluate captioning quality by comparing baseline model performance using existing approaches versus our method. We also assess models with and without the proposed method, conduct human evaluations against conventional techniques, and analyze the methods introduced in Section 4. Qualitative results are provided in Appendix A.
The core idea of SPARC is to compute a cumulative rele- vance measure for each image token and adjust attention weights dynamically based on this measure. To achieve this, we introduce the Selection Count, which quantifies the evolving importance of each image token during text generation. The Selection Count is formally defined as:
5.1. Experimental Setup
Models We conduct experiments on three widely used multi-modal language models (MLLMs); LLaVA-1.5 (Liu et al., 2024a), LLaVA-Next (Liu et al., 2024b), and Qwen2- VI. (Wang et al., 2024), each with 7B parameters. For each model, we generat captions for given images using the prompt: "Please describe this image in detail." The maxi- mum token length for caption generation is set to 512 across all models.
f-1
C.j = -
(7)
1=
where 1(.) is an indicator function that returns 1 if j is selected at step k. and 0 otherwise. This metric accumulates the number of times each token has been deemed relevant in prior steps, reflecting its sustained importance in the evolving context.
Metrics We use two evaluation metrics in our experiments. The first metric, CLAIR (Chan et al., 2023), measures over- all caption quality by assessing alignment with reference captions. It determines whether the generated and reference captions effectively describe the same image, with higher scores indicating better quality. CLAIR leverages GPT- 4o (Hurst et al., 2024) for reliable evaluation of detailed and accurate captions. The second metric, CHAIR (Rohrbach et al., 2018), assesses hallucination by comparing objects mentioned in generated captions with those in reference captions. It provides precision-recall metrics to evaluate the trade-off between correctly identified and erroneously included objects.
At each text generation step i, we first identify the set of contextually relevant tokens S; using our Token Selection method and update the Selection Count C,j for each token j. Based on the updated count, we recalibrate the attention weights by amplifying those of frequently selected tokens:
d/. + d/.3 aii
(8)
where >> 1 is a scaling parameter controlling the ampli- fication of tokens with higher cumulative relevance. This exponential scaling prioritizes consistently relevant tokens while maintaining adaptability to new contexts, mitigating the decline in attention weights observed in longer text gen- cration.
Datasets For CLAIR evaluation, we use the IIW- 400 (Garg et al., 2024) and DOCCI (Onoe et al., 2025) datasets. IIW-400 consists of 400 image-caption pairs, while DOCCI contains 15K pairs. Both datasets provide highly de- tailed, hallucination-free captions, making them well-suited for evaluating caption quality. For CHAIR evaluation, we utilize the MS-COCO 2014 validation dataset (Lin et al., 2014). This dataset includes ground ground-truth object annotations, which enable the calculation of CHAIR metries by compar-
To further enhance computational efficiency, we implement this recalibration by directly adjusting the token's 's value vectors. Specifically, for each j E S,, we update:
v(l - Vi(l)) v.(l.))...
(9)
This is possible because key-v caching in large language models stores value vectors for each token (Wan et al., 2023).
6

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Table Comparison of CLAIR scores on the ITW-400 and DOCCT datasets for the baseline model with existing methods and our pro- posed approach. The results highlight the performance improve- ments achieved by our method.
Table 2. Performance of various methods detailed image cap- tioning using the CHAIR benchmark. The table reports precision. recall, and Fl -core for objects in generated captions. The best scores are bolded, while the second-best scores are underlined.
METHOD
IIW-400
DOCCI
METHODS
PRECISION
RECALL.
F1
BASELINE
56.36
59.26
BASELINE
84.70
79.46
81.99
OPERA
51.02 (65.34)
56.81 (-2.45)
OPERA
84.54
78.82
81.58
VCD
52.16 (-4.20)
55.60 (3366)
VCD
83.22
77.50
80.26
VOLCANO
55.84 (-0.22)
61.09 (+1.83)
VOLCANO
87.64
77.82
82.39
PAI
56.86 (+0.50)
60.09 (40.83)
PAI
90.64
72.44
80.52
OURS
61.49 (45.13)
62.70 (43.44)
OURS
87.72
79.98
83.67
ing the objects mentioned in the generated captions with the reference objects.
Table 3. Comparison CLAII scores on the ITW-400 dataset between the baseline and our method applied to various models. The results demonstrate a consistent performance improvement with approach.
To compare our method with existing approaches, we conduct experiments on the base- line model, 1... The hyperparameters for existing methods are implemented as specified in prior research. For our method, the following parameters are applied across all models: the scaling factor is set to 1.1, the smooth- ing factor 3 is set to 0.1, and the selection threshold t is adjusted for each model. Specifically, + is set to 1.5 for LLaVA-1.5,, 4.0 for LLaVA-Next. and 3.0 for Qwen2-VL. VL. For token selection, we extract visual attention from layer 20 for LLaVA-1.5 and LLaVA-Next, while layer 18 is used for Qwen2-VL. VL. On the other hand, attention recalibration is applied across all layers' attention.
Implementation Details
MODEL
BASELINE
OURS
LLAVA-1.5
56.36
61.49 (45.13)
LLAVA-NEXT
58.86
64.94 (46.08)
QWEN2-VL
78.34
79.70 (+1.3)
in Table 2. To ensure a robust evaluation, we randomly sam- ple 500 instances and repeated the evaluation five times. OPERA (Huang et al., 2024) and VCD (Leng et al., 2024), which rely decoding based strategies fail to improve pre- cision or recall. VOCANO (Lee et al., 2023), which incor- porates feedback to self-revise its initial response, enhances precision but slightly reduces recall. PAI (Liu et al., 2025), which increases attention to the image while incorporating additional decoding techniques, achieves the largest preci- sion gain but suffers the lowest recall, resulting in lower Fl score than method.
5.2. Results
Comparison with Existing Approaches We compar our method with the existing approaches on LLaVA-1.5, using CLAIR scores on the IIW-400 and DOCCI datasets. Table summarizes the results, showing that our method achieves the highest scores across both datasets, significantly outper- forming prior approaches. This improvement demonstrates that our method enhances caption generation by produc- ing captions that are more detailed and better aligned with reference captions.
In contrast, our method successfully improves both precision and recall, achieving the highest Fl score. Specifically, it increases precision by 3.02%p. recall by 0.52%p. and FI score by 1.68%p. Notably, our approach the only one that improves recall compared to the baseline, whereas all other methods sacrifice recall boost precision.
These results demonstrate that our method minimizes incor- rect object inclusions while enhancing the model's s ability to identify relevant objects. This improved balance is crucial for generating captions that are both accurate and compre- hensive, setting new benchmark for high-quality caption generation. The detailed CHAIR metric scores are provided in Appendix F.
Precision-Reed Recall Tradeoff Analysis Generating high- quality captions requires a balanced improvement in both precision and recall. In the CHAIR metric, precision mea- sures the proportion of objects in generated captions that not appear the reference captions, indicating hallucina- tion. Recall quantifies the proportion of reference objects correctly identified in the generated captions. The Fl score combines these metrics to provide holistic assessment of the trade-off' between precision and recall.
Performance Across Different Models To demonstrate the effectiveness of our method across diverse models, we evaluate it on three widely used MLLMs: LLaVA-1.5. V.L.Ve- and Qwen2-Vis VI.. Tables 3 and present the
We compar our method against existing approaches in terms of precision, recall, and F1 score using CHAIR, as shown

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Table Comparison of CLAIR scores the DOCCT dataset be- tween the baseline and our method applied to various models. The results demonstrate a consistent performance improvement with our approach.
Imaae Attontion
0.30-
Baseline Ours
Sink Attention Rato
1.4
0.23
14
0.00
MODEL
BASELINE
OURS
of Context length
LLAVA-1.5
59.26
62.70 (+3.44)
Baseline
Ours
LLAVA-NEXT
62.49
66.99 (+4,50)
(a)
(b)
QWEN2-VL
79.22
80.64 (+1.42)
Figure 7. Visual Attention Analysis. (a) Change in visual attention with increasing context length. SPARC mitigates the decline com- pared to the baseline. (b) Ratio of attention seales between sink and non-sink visual tokens during captioning. SPARC maintains baseline-leve proportions, unlike the naive approach.
Baseline
Ours
Naive
Ours
Recall
Recall
Precision
Precision
5.3. Analyses
60
45
Winning Percentage (%)
Winning Percentage 1%)
Quantitative Evaluation for Visual Attention To ana- lyze the change in visual token attention, we compare against a naive approach and the baseline model. Figure 7(a) illustrates how visual attention changes with increasing context length for each method. It shows that SPARC effec- tively mitigates the decline in visual attention, preserving focus on visual tokens even in longer contexts.
(a)
(b)
Figure Human evaluation results showing winning ratio (%)
of our method compared to (a) the baseline and (b) naive approach terms of precision and recall.
We also quantify how much visual attention is assigned to semantically relevant image regions during the decoding process, using 5,000 randomly sampled images from the MSCOCO 2014 validation set processed by the VA-15 model. Specifically, we calculated the total attention score allocated to image tokens within the region of that object for each generated token corresponding to a ground truth object in the caption. To identify these regions, we employed an open-vocabulary segmentation model al., 2024; Ravi et al., 2024) to generate binary masks for all ground truth objects. During caption generation, proportion of visual attention focused on the corresponding object region relative to the total attention across all image tokens for each object-related token.
CLAIR scores for the IIW-400 and DOCCI datasets, re- spectively. For DOCCI, we randomly select 500 samples for evaluation. Across both datasets. our approach consis- tently improves caption quality, demonstrating its robustness across diverse model architectures. These findings confirm that our approach not only refines precision and recall but also generalizes well across different datasets and model configurations. The consistent performance gains further underscore the adaptability of our method in aligning gener- ated captions more effectively with reference captions, as measured by the CLAIR metric.
measured the
Human Evaluation Results To further assess the precision-recall tradeoff, we conduct a human evaluation. Specifically, we sample 100 captions generated by the LLaVA-1.5 model on images from the IIW-400 dataset. These captions are evaluated for precision and recall by human annotators, who compare different methods and se- lected the better one. The results are then aggregated into a winning ratio, showing how often our method is preferred over the baseline and naive attention enhancement approach.
As shown in Table 5, our method assigns a higher proportion of visual attention image-relevant regions compared to the baseline. In contrast, naive attention scaling allocates less attention to relevant image regions than the baseline. These results indicate that our method achieves better alignment between generated text and semantically relevant visual re- gions, suggesting that the visual attention in our model is less noisy and more accurately focused during the caption- ing process.
As shown in Figure 6, our method achieves a higher recall compared to the baseline while also improving precision. Then compared to naive approach, our approach demon- strates superior recall with a slight decrease in precision. These results indicate that our method effectively balances recall and precision, reducing hallucinations while ensuring comprehensive caption generation.
Additionally, analyze the attention distribution over sink tokens-token identified as unrelated or uninforma- tive-during the caption generation process. Following (Kang et al., 2025), which identifies sink tokens based on hidden state dimensions with exceptionally high values, we compute the ratio of attention scales between sink and non-
8

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Table Comparison of the proportion of visual attention focused on semantically relevant image regions between the baseline, naive attention scaling and our method.
Table 6. Comparison of token generation time (ms/token) between our method and existing approaches, Our method achieves effi- ciency comparable to the baseline, whereas other methods exhibit substantially higher computational overhead.
METHOD
ATTENTION ON RELEVANT REGIONS (%)
MODEL
GENERATION TIME (MS/TOKEN)
A (%)
BASELINE
17.85
NAIVE
15.50 (-2.55)
BASELINE
30.37 = 0.73
OURS
19.17 (-1.3)
OPERA
322.28 = 118.26
+961.3
VCD
59.44 = 0.84
+95.7
VOLCANO
109.98 17.71
+262.2
PAI
57.75 = 0.86
+90.2
sink tokens and plotted the results for the three approaches, as shown in Figure 7(b). The naive approach significantly increases the attention to sink tokens, which can detract from meaningful visual token focus. In contrast, SPARC maintains a sink token attention proportion comparable to the baseline, effectively avoiding unintended amplification of irrelevant tokens.
OURS
31.21 = 0.61
+2.8
pact on model performance. Briefly, we conducted ablation experiments focusing on two key design components: the token selection strategy and the progressive attention calibra- tion mechanism. Specifically, we evaluated the performance when (1) bypassing the token selection process and applying progressive attention calibration to all image tokens, and (2) modifying the token selection strategy to use only the previous step instead of EMA. These variations highlight the individual contributions of both the token selection strat- egy and the progressive attention calibration in enhancing model performance.
These results highlight the robustness of SPARC in pre- serving meaningful visual attention dynamics, even in chal- lenging contexts with extended lengths, while avoiding the pitfalls observed in naive attention reinforcement methods. The detailed implementation details of the above experi- ments can be found in Appendix B.2.
Efficiency Comparisons To demonstrate that method incurs minimal computational overhead, we conducted an efficiency comparison against existing approaches. Specif- ically, we generated image captions using the VL- model on the IIW-400 dataset with an RTXS000 GPU. For each caption, we measured the generation time per output token and then computed the average token generation time across all captions.
Additionally, we conducted ablation studies on four crit- ical parameters: the token selection layer I, token selec- tion threshold т, EMA smoothing factor B, and scaling parameter a, evaluating their impact on performance across LL5 LLaVA-NeXT, and Qwen2-VL. VL. We observe consistent trends across models, with optimal performance typically achieved using mid-to-late layers for token selec- tion and mildly scaled parameter values. While some tuning is required to achieve the best results for each model, our training-fres low computational overhead method makes hyperparameter tuning both efficient and practical, requiring minimal effort to identify optimal settings.
As shown in Table 6, our method achieves a token generation time comparable to the baseline, whereas other methods are significantly slower-by a factor of 2x to 10x. This clearly demonstrates that our approach enables efficient caption generation with minimal computational cost. Although our method incorporates attention amplification mechanisms, these introduce only negligible overhead relative to the orig- inal decoding process. In contrast, many prior methods rely on additional decoding passes, which substantially increase computational burden.
6. Conclusion
In this work, address the challenge of balancing preci- sion and recall in detailed image captioning for multimodal large language models. To mitigate this issue, we propose SPARC, a training-fiet -fre method that enhances the contri- bution of visual tokens during decoding. SPARC identifies critical visual tokens by leveraging attention differences across generation steps and progressively reinforces visual attention counteract its natural decline. Our experimen- tal results, validated through both automated metries and human evaluations, reveal that conventional methods often improve precision at the cost of recall. In contrast, SPARC effectively enhances both precision and recall with minimal computational overhead, offering a simple yet powerful so- lution for improving detailed image captioning in MILMMs.
In terms of memory usage, our method only requires storing the head-wise averaged attention scores for image tokens at each layer from the previous decoding step. For instance, with VL- 1.5, this storage amounts to 32 layers x 576 im- age tokens 2 bytes (float16) 16), totaling less than 40 KB-an overhead that is trivial on modern hardware.
Further ablation studies on the specific param- eters or setting choices used in our approach are provided in Appendix C for a comprehensive analysis of their im-
Ablations

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Acknowledgements
arXiv preprint arXiv:2309. 16609, 2023.
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) TRS-2022-11220959; No.RS-2021-1121115 telligence Graduate School Program (Seoul National Uni- versity)], the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2022R I3BB 1077720; 2022RIA5A70833 ASA7783308717 1), the BK21 FOUR program of the Education and the Research Program for Future ICT Pioneers, Scoul National University in 2025, the Mobile (Xperience(MXX) Business, Samsung Electronics Co., Ltd, and grant from Yang Young Foundation.
Bai, Z.,, Wang, P., Xiao, T., He, T., Han, Z.., Zhang, Z., and
Shou, M. Z. Hallucination of multimodal large language
models: A survey. arXiv preprint arXiv:2404.189800 2024.
Artificial In-
Basu, S., Grayson, M., Morrison, C., Nushi, B., Feizi, S., and Massiceti, D. Understanding information storage and transfer in multi-modal large language models. arXiv preprint arXiv:2406.042255 2024.
Belyaeva, A., Cosentino, J., Hormozdiari, F., Eswaran, K., Shetty, S., Corrado, G., Carroll, A., Mcl.aa, C. Y., and Furlotte, N. A. Multimodal Ilms for health grounded in aadividual-specifee data. In Workshop Machine Learning for Multimodal Healthcare Data, pp. 86-102. Springer, 2023.
Impact Statement
As Al models evolve beyond simple object recognition to generating captions that incorporate contextual and seman- tic understanding, the length and complexity of generated descriptions increasing. This trend makes the challenge of balancing precision and recall in image captioning even more critical. Our research addresses this issue by improv- ing multimodal large language models (MILMss), which can contribute to advancements in AI for accessibility and multi- modal contextual understanding. More accurate and context- aware captions can greatly benefit visually impaired individ- uals by providing richer and more informative descriptions. Additionally, applications in education, automated docu- mentation, and creative content generation can be enhanced through improved captioning capabilities.
Chan, D., Petryk. S., Gonzalez, J. E., Darrell, T., and Canny, J. Clair: Evaluating image captions with large language models. arXiv preprint arXiv:2310.12971 2023.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L.,, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- lisuull-inguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 241998, 2024.
Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L., Zou, J., and Yao, H. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:231 1.03287, 2023.
However, the societal impact of advanced image captioning must be carefully considered. While our method enhances caption quality, Al-generated descriptions can still suffer from hallucinations incorrect details that seem plausible. As Al-generated captions become more refined and detailed, users may develop stronger trust in their accuracy, even when the content is incorrect or misleading. This could lead to unintended consequences, such as misinterpretations of visual content or overreliance on Al-generated descriptions without verification. Ensuring that users remain aware of these limitations is crucial Al-generated content becomes more prevalent in real-world applications.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vi- sion transformers need registers. In The Twelfih In- ternational Conference Learning Representations, 2024. URL https: http::ss///peeriieee...o net//ooruttr forum? id=2dn03ILic1.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2024.
Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A., and Soatto, S. Multi- modal hallucination control by visual information ground- ing. In Proceedings the IEEE/CVF Conference on Com- puter Vision and Patlern Recognition, pp. 14303-14312, 2024.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Garg, R., Burns, A., Ayan, B. K., Bitton, Y., Montgomery, C., Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R. Imageinwords: Unlocking hyper- im- age descriptions. arXiv preprint arXiv:2405.0m 02000000279.... 2024.
Bai, J., Bai, S., Chu, Y. Cui, Z., Dang, K., Deng. X., Fan, Y., Ge, W., Han, Y. Huang, F., et al. Qwen technical report.
10

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MILMS
Gong, X., Ming, T., Wang, X., and Wei, Z. Damro: Dive into the attention mechanism of Ivlm to reduce object hallucination. arXiv preprint arXiv:2410.04514, 2024.
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing. L.. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13872-13882, 2024.
Gunjal, A., Yin, J., and Bas, E. Detecting and prevent- ing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 18135 18143, 2024.
Li, B.. Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H.,, Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a.
Han, Y., Zhang, C., Chen, X., Yang, X., Wang, Z., Yu, G., Fu, B., and Zhang. H. Chartllama: A multimodal Ilm for chart understanding and generation. arXiv preprint arXiv:2311.16488 2023.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip- Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Hao, Y., Yang, F., Huang, H., Yuan, S., Rangan, S., Rizzo, J.-R., Wang, Y., and Fang, Y. A multi-modal foundation model to assist people with blindness and low vision in environmental interaction. Journal of Imaging, 10(5):103, 2024.
Li, J., Zhang, J., Jie, Z., Ma, L.,, and Li, G. Mitigating hallucination for large vision language model by inter- modality correlation calibration decoding, arXiv preprint arXiv:2501.01 0266 2025.
Huang, Q.. Dong, X., Zhang, P., Wang, B., He, C., Wang. J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviating hallu- large language models via over- trust penalty and eetrospection-allccaee of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
Li, K., Patel. O., Viégas, F., Pfister, H., and Wattenberg, M. Inference-ting intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024b.
cination in
In Proceedings
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arX1v:3305.10355, 2023b.
Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., and Zhao, P. Self-introspective decoding: Alleviating hallucina- tions for large dision-lanuuge models. arXiv preprint arXiv:2408.0200: 02032, 2024.
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 26689-26699, 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Rad- ford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Jiang. Z., Chen, J., Zhu, B., Luo, T., Shen, Y., and Yang, X. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucina- tions via attention lens. arXiv preprint 21111:241..16724, 2024.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Kang, S., Kim, J., Kim, J., and Hwang, S. J. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03321, 2025.
2024a.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024b. URL https://llavas www..ggttu..ioo//o 2024-01-30-1lava-ner.
Lee, S., Park, S. H., Jo. Y., and Sco, M. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07366 2023.
Lee, S., Yoon, S., Bui, T., Shi, J., and Yoon, S. Toward robust hyper-detailed image captioning: A multiagent approach and dual evaluation metrics for factuality and coverage. arXiv preprint arXiv:2412. /5444, 2024.
Liu, H., Li, C., Wu, Q., and Lee, Y.J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024c.

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang. K., Hou, L.,, Li, R., and Peng, W. A survey on hallucina- tion in large vision-language models. arXiv preprint arXiv:2402.00255, 2024d.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307009888 2023.
Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free method for alleviating halluci- nation in Ivlms. In European Conference on Computer Vision, pp. 125-140. Springer, 2025.
Vallender, S. Calculation of the wasserstein distance be- tween probability distributions the line. Theory of Probability & Its Applications, 18(4):784-78) 786, 1974.
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang. S., Xi, H., Cao, S., Gu, Y., Li, D., et al. Nvila: Efficient frontier vi- sual language models. arXiv preprint 4rXiv:2412.04468,, 2024e.
Vaswani, A. Attention is all you need. Advances Neural Information Processing Systems, 2017.
Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q. et al. Effi- cient large language models: survey. arXiv preprint arXiv:2312.0365 2023.
Mitra, C., Huang, B., Darrell, T., and Herzig, R. Composi- tional chain of thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2024.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang. J., Ge. W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Onoe, Y., Rane. S., Berger, Z.. Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z., Pont-Tuset, J., Tanzer, G., al. Docci: Descriptions of connected and contrasting images. In European Conference Computer Vision, pp. 291- 309. Springer, 2025.
Woo, S., Kim, D., Jang, J., Choi, Y., and Kim, C. Don't miss the forest for the trees: Attentional vision calibra- tion for large vision language models. arXiv preprint arXiv:2405. 17820, 2024.
Osman, A. A., Shalaby, M. A. W., Soliman, M. M., and Elsayed, K. M. A survey on attention based models for image captioning. International Journal of Advanced Computer Science and Applications, 14(2), 2023.
Xing, Y., Li, Y., Laptev, I., and Lu, S. Mitigating object hal- lucination via concentric causal attention. arXiv preprint arXiv:2410. 15926, 2024.
Yue, Z., Zhang, L., and Jin, Q. Less is more: Mitigating mul- timodal hallucination from an eos decision perspective. arXiv preprint 24Xi / 14545, 2024.
Peng, B., Li, C., He. P., Galley, M., and Gao. J. Instruc- tion tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Zhang, J., Khayatkhoei, M., Chhikara, P., and Ilievski, F. MI.LMs know where to look: Training-free perception of small visual details with multimodal LLMs. In The Thir- teenth International Conference on Learning Represen- tations, 2025. URL ht https: openrevieweeeeeeeeeeeeeeee .ne// forum? daDiaaDDaaaaaa. Y 5mDdmT.
Ravi, N., Gabeur, V., HU, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., and Feichtenhofer, C. Sam 2: Segment anything in images and videos, 2024. URL https: //arxiv. org/ abs/ 2abs/2400 00714.
Zhang, X.,
Quan, Y., Gu, C., Shen, C., Yuan, X., Yan, S., Cheng, H., Wu, K., and Ye, J. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in Ivims. arXiv preprint arXiv:24/ 000::2//1.09900 1,09968, 2024.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q. and Zhang, I.. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Zhong, W., Feng, X., Zhao, I.., Li, Q., Huang, I.., Gu, Y., Ma, W., Xu, Y., and Qin, B. Investigating and mitigating the multimodal hallucination snowballing in large vision- language models. arXiv preprint arXiv:2407.00 00569, 2024.
Rohrbach, A., Hendricks, A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. arXiv preprint 180900080000215666 2018
Shen, J., Qu. Y., Zhang, W., and Yu, Y. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018
Zhou, Y., Cui, C., Yoon, J., Zhang,
L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision- languaee models. arXiv preprint 1rXiv:2310.00754. 2023.
12

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Zhu, L., Ji, D., Chen, T., Xu, P., Ye, J., and Liu, Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476, 2024.
13

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
A. Qualitative Results: Enhanced Caption Quality
A.1. LLaVA-1.5
Baseline
The image features two women standing on snow-covered slope, each wearing skis and holding ski poles. They are both amiling and enjoying their time on the snowy hill. The women alightly the left
positioned close to each other, with right. Their skis are visible, with one pair skis the side of the image and the other pair on the right side. The women are also wearing backpacks, with one backpack the left the right side image. The captures and active winter day spent skiing together.
slightly
side
Ours
The image features women standing snow-covered slope, each wearing skis and holding ski poles. They both smiling enjoying are positioned close to each other, with one woman on the left and the other on the right. In addition to the skis and poles. the women their faces. set against a oloudy sky. creating a picturesque winter landscape.
snowy hill. women
also wearing sunglasses, which women is wearing hai, adding to the wintery atmosphere. scene is
seen
Baseline
The image features bowl filled with delicious and hearty meal, consisting of combination of meat and broccoli. The broccoli pieces are scattered throughout the bow, with some pieces placed closer to the top and others near the bottom. The meat, which appears to be chicken, is also present the bowl, adding the overall flavor texture dish. The bowl placed on dining table, ready be enjoyed by someone.
Ours
The image features bowl filled delicious hearty meal, consisting of combination of meat and broccoli. The bowl is filled with a rich and flavorful soup, containing generrus amount of broccoli pieces. The broccoli pieces scattered throughout the bowl, with some pieces being larger
soup gamished with variety onions, which also distributed throughout the bowl. The dish is served in a bow,, making an appealing and comforting meal.
smaller.
Baseline
The image captures a group of young boys playing soccer on a grassy field. There are at least nine players visible in the scene, with are standing or running around the fleld. soccer ball is located in the center one player attempling to kick it. The players are spread out across the field, with some closer to energy, showcasing
of them actively engaged the game, while others
fleld, with
others further away. The excitement and camaraderie the young players.
lively
Ours
The image captures group
boys playing soccer grassy field. There at least game, while others
players visible the with of actively engaged are standing or running around fleld. soccer ball is located in the center one boy lying the ground. possibly after falling being tackled. The other players across field, same
field, with
others further away
Baseline
The image features close-up colorful fish long. thin tail large, pointy fin. The fish is swimming in body of water, possibly an aquarium, with its head and body prominently visible. The fish appears to be unique and eye-catching species, possibly clownfish, given its distinctive appearance.
Ours
The image features close-up of colorful fish with long. spiky tail, swimming in tank. The fish has vibrant body, The
red color, with distinctive orange and white striped pattern its long and spiky, adding to unique appearance.
The tank filled with various rocks, providing natural and interesting environment for the fish. The rocks are scattered throughout tank, with some located near top, middle, and bottom of image. The fish is positioned in the middle of the tank, surrounded by the rocks.
Figure Comparison captions generated by applying our method to LLaVA-I.5 and the baseline. text highlights incorrect references in the captions, while blue text indicates additional details provided by method compared to the baseline.
14

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
A.2. LLaVA-NeXT
Baseline
The image captures scene from theatrical production, featuring large, intricately designed rack formation the centerpiene. The rock formation, which is the focus of the image, crafled
combination red and brown materials, giving rugged appearance. ll has a large opening the center, inviling the viewer to explore what lies wilhin. The rock formation is silusted on a stage, which is bathed in a warm, orange light that enhances the dramalic effect of the scene. The stage is set against a black curtain, providing stark contrast that further emphasizes the rock formation. the foreground Their presence suggests that they might be depict any actions. The relative
natural
the image, there of the design props
woeden for production. The image does contain any posilions of the abjects are such that the rock formalion is centrally localed on the slage, with the wooden crates positioned in the foreground. The overall composition of the image suggests scene from theatrical performance, with the rock formation serving as significant element the narrative.
The image captures a scene from a theatrical production, featuring large, rock-like prop that dominates the foreground. The prop. which appears to be made of textured material resembling stone wood, has rough, uneven surface with reddish-brown hue. has two prominent, glowing eyes
red and black, giving menacing appearance. mouth open, revealing dark interior, and there small, dark opening on right side. possibly a door or entrance. In the background, there hint of a stage setting with a black curtain on the left side and a white structure that could be part of the set design. On the right side of the image, there is red crate stacked on the floor, and yellow light source is visible, illuminating the dramatic effect
prop's
from right side. The lighting casts shadows on the floor, enhencing the style the image suggests staged performance,
prop. The possibly a play musical, with the prop serving as a significant element the scone.
Baseline
The image captures a scene of a vintage car parked on a street, facing towards the right side of the frame. The painted shade of gray, adomned with flame design its side, adding
vintage charm. It's parked front against the backdrop trees, their green leaves contrasting with the gray of the car. the dislance, power lines stretch across the sky, leading the eye towards the horizon. The image exudes a sense of nostalgia. reminiscent bygone era.
wooden telephone pole, which stands with
clear blue sky. The street on which the car parked
Ours
The image captures a scene of vintage car parked on gravel lot. The car, painted in a striking shade of black, is adomed with flames painted on Its side, adding touch of fiery spint its appearance. The car's design reminiscent of the 1950s, with rounded body and parked on gravel which scattered with small rocks and pebbles. adding texture the scene, the background, telephone pole slands tall, its presence sublly reminding us of the era when such structures were more common. Above the car, a red toy tractor is suspended from power lines, adding an element of whimay to the scene. The tractor, with vibrant red color and playful design, contrasts with the seriousness of the vintage depth the image. The power
root, giving
and timeless appeal.
below. The sky above clear blue, dolled with flulfy while
stretching across the sky create sense openness and freedom. Overall, this image paints picture of moment frozen time, where vinlage meels whimss. and the past meels the present.
clouds
Baseline
The image depicts beige Volkswagen van parked on a sireet at night. The van has visible front grille with the Volkswagen logo, and the headlights are off. The vehicle positioned on the side the road, with the front rear window is clear. There is visible dent
towards the left side of the image. The van's side windows
driver's
door.
is parked next to a curb, and there is a small orange traffic cone placed in front of The background shows a residential area with houses and trees, and the sky is dark, sugpesting it nightlime. There are no visible texts or distinctive markings on the van that provide additional information. The style of the image standard, unfillered photograph with artistic manipulation.
Ours
The image depicts beige Volkswagen Type parked street at night. The van has distinctive rounded front and flat roof, which is typical of the Volkswagen Bus design. It features black grille with the Volkswagen logo at the center, and the headlights are visible. The van has white stripe running allong the side, and side windows are tinted. The vehicle is parked next to a curb, and there is a small orange traffic cone placed in front of the van. The background shows residential houses and trees, and there is street lamp visible in the distance. The lighting suggests is nighttime, visible texts or distinctive markings on the van that provide additional information. The style of the image standard, unembellished photograph with no filters or artistic manipulations.
with
the overall atmosphere is calm and quiet, There no
Figure 9. Comparison of captions generated by applying our method to LLaVA-NeXT and the baseline, Red text highlights incorrect references in the captions, while blue text indicates additional details provided by method compared to the baseline.
15

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Baseline
The image shows a sculpture of fantasy character, which appears be humanoid figure with a stylied,, cartoonish design. The character has large, exaggerated head with prominent nose and a wide, cheerful smile. The figure is wearing a tunic with a belt and a pair pants, and it has a large, round belly. The character's arms raised, and it holding what seems
resembles a log, with a large, curved piece wood serving as the base. The background is out of focus, but it appears to be an outdoor setting with green wall and a framed picture hanging on it. The ground covered with dirt and scatlered debrie. There no visible texts distinguishing selting. style the image is a straighfforward, unembellshee photograph with no filters or artistic effects applied.
small object
right hand.
sculpture standing on
marks provide additional context about the character
Ours
The image shows a sculpture of fantasy character, which appears to be goblin or a similar creature, standing on wooden platform. The sculpture made of a light-colored material. possibly stone concrete, weathered appearance. The character has large, pointed ear, a prominent nose, and wide, grinning mouth. It is wearing a lunic with a bell and a pair of boots, and il holding a small object in its right hand. The character standing slump, with a large The sculpture placed outdoors, as indicated by the natural surroundings and the green wall in the background. There are two framed posters hanging on the wall, but the content of the posters visible suggesting a natural or semi-natural setting. The lighting the image bright, and the photograph is taken from a slightly elevated angle, giving a clear view of the sculpture and its surroundings, There are no visible texts or brands the image. The style of the image straighttorward. unembellished photograph with filters or artistic manipulations.
is detailed with textures and patterns that suggest rugged,
log that has been carved to resemble
visible on the surface.
image. The ground Is covered with dirt and scatterad debris.
Baseline
The image shows model blue and white bost with red stripe the hull, displayed stand. The boat has a prominent white superstructure with a cabin and wheelhouse, There are several windows visible on the cabin and wheelhouse, and the boat is equipped with lifeboat the aft deck. The model detailed with a flag flying from the mast and lifebuoy hanging from the model. There are no visible texts or brands on the model. The style of the image is a straightforward product photograph with a focus on the model boat, intended showcase its design and features.
background
plain, light color, providing a neutral backdrop
highlights
Ours
The image shows a model blue and white boat with a red stripe on the hull, displayed on stand. The bost has a prominent white superstructure with a cabin and wheelhouse, and a blue and white hull with red stripe running along the side. On the bow, there red lifeboat with a white interior. The model is mounted on white stand with two vertical poles supporting The background is a plain, light gray surface, and there a small, circular abject with a flag on top of the model, suggesting it might be decorative element or part of the model's display. The text "RV Coral Reef II visible the side model, indicating The style the image is straightforward. clear photograph with no additional context or embellishmente.
the boat.
Baseline
The image shows a garden scene positioned in the center. The statue is partially abscured by lush greenery, including a variety of flowers and plants. The statue appears to be made of light-colored material, possibly stone concrete,
person wearing straw hat,
brick wall, The wall is constructed of rectangular bricks and has a rough texture. The garden is bordered by a neatly trimmed lawn on the left side, and there are trees and shrubs in the background, adding to the natural setting. The overall style of the image a realistic photograph with focus the garden and the statue. There are visible texts distinntivv brands the image.
set against backdrop
Ours
The image shows a garden soene with focus on a statue of young girl with straw hat, positioned in the center. The statue is made of a light-colared material, possibly stone or concrete, and partiallyy obscured by lush green foliage. The expression, looking towards the viewer. Surrounding the statue are various yellow flowers, including dalsies and possibly other types of flowers, which add a vibrant splash of color to the scene. The flowers are in full bloom, suggesting warm, possibly spring summer season.
depicted with
The garden bed is bordered by low stone wall. which appears to be made of stacked stones, giving it rustic and natural look. The wall is partially covered by the green foliage, blanding the garden's elements together. In the background, there contrasting texture to the softness and well-maintained garden space. There are no visible texts or distinctive branding in the image. The style of the image is realistic photograph with focus on the garden and its decorative elements.
brick wall, which provides garden bed. The overall impression
tranquil
Figure 10. Comparison of captions generated by applying our method to LLaVA-NeXT and the baseline. Red text highlights incorrect references in the captions, while blue text indicates additional details provided by our method compared to the baseline.
16

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
B. Details for Analyses
B.1. Why More Attention Doesn't Always Mean Better Descriptions
Enhanced Attention: A Path to Less Diversity? To analyze the diversity of visual attention during caption generation, we used the baseline model (LLaVA-1.5 7B) to generate captions for 3,000 image samples from the DOCCI dataset. During the caption generation process, measured the attention weights assigned visual tokens when generating output tokens. Specifically,, we extracted visual attention weights from layer 20 and averaged them across attention heads.
For each image sample, we analyzed the visual attention corresponding to the first 100 output tokens. To quantify the variation in visual attention throughout the generation process, we computed the distance between the visual attention distributions of consecutive output tokens within each sample. The distance was calculated using the Wasserstein distance, where each visual attention distribution was first normalized so that the sum of attention weights across all tokens equaled 1.
The results of this analysis are presented in Figure 3, which compares the baseline model with a naive attention enhancement method that proportionally increases visual attention weights. The plot illustrates that the naive enhancement method reduces the variation in visual attention patterns throughout the caption generation process. This finding aligns with our observations that the naive method decreases the diversity of objects mentioned in the generated captions.
20
20
20-
Output Taken Index
Output Taken Index
Output Taken Index
40
60
60
80
80
80
60
40
40
80
Output Token Index
Output Token Index
Output Token Index
Figure 11. Visual attention diversity comparison between (a) the baseline model, (b) the naive attention enhancement approach, and (c) our method. The naive approach reduces visual attention diversity, indicating ineffective adaptation to important visual tokens. Instead, proposed method does not severely degrade visual attention diversity
In contrast, the results shown in Figure 11 demonstrate that our proposed method does not severely degrade visual attention diversity. Instead, our approach dynamically reinforces attention to relevant regions of an image based the evolving context during caption generation. This confirms that our method effectively maintains balance between enhancing visual attention and preserving diversity, leading to more contextually appropriate and varied caption outputs.
In addition to its impact on visual attention, our method also preserves diversity in the content of generated captions. Appendix D presents an analysis of caption diversity, introducing metrics that assess the variation in generated sentences. Using these metries, we evaluate caption diversity across different methods, including those discussed earlier in this appendix.
Longer Contexts Amplify Noisy Attention To analyze how longer contexts influence attention noise, we conducted an experiment using the baseline model (LLaVA-1.5 7B). During the caption generation process for single image, measured the attention weights assigned to visual tokens when generating output tokens. Specifically, visual attention weights were extracted from layer 20 and averaged across attention heads.
Next, we normalized the visual attention weights such that the sum of all attention weights for visual tokens equaled 1. Finally, as shown in Figure 4, we plotted the normalized attention distribution along with the corresponding image to visualize how attention shifts across different tokens in the presence of longer contexts.
17

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Additionally, To address the concern that attention ecores-especiilly in deeper layers-may not directly correspond the original image patches, we conducted additional analyses using saliency maps computed with respect to the attention scores. Specifically, we implemented tradient-weighted attention method inspired by (Zhang et al., 2025), where attention scores are weighted by their gradients with respect to the model's output.
As show in Figure 12, we compare the original attention- results (Figure 4) with those obtained using the saliency maps. The saliency maps reveal similar overall trends: as the context length increases during caption generation, the resulting maps become progressively noisier. This observation supports our original interpretation. Furthermore, recent work (Zhang al., 2025) has demonstrated that the visual attention pattern of MLLMs do indeed align with semantically relevant image regions, particularly in tasks like visual question answering. further validating the use of this method in our analysis.
weightd Attention Score
skiing
skis
Attention Score
skiing
skis
Figure 12. Comparison between visualizations of attention scores and gradient-weighhed attention scores for different context lengths during caption generation. Both methods exhibit similar trends, with increasing noise as the context length grows.
Longer Context, Less Visual Focus To analyze the effect of longer contexts on visual focus, we conducted an experiment using the baseline model (LLaVA-1.5 7B). Caption generation was performed on 3,000 image samples from the DOCCI dataset, and we measured the attention weights assigned to visual tokens during the generation of output tokens.
Specifically, visual attention weights were extracted from layer 20 and averaged across attention heads. For each output token, computed the total attention weights allocated to visual tokens and the total attention weights allocated text tokens (including instruction tokens and generated tokens). These values were averaged across all samples and plotted against different context lengths to examine how visual focus changes as context length increases. The results of this analysis are presented in Figure 5, which illustrates the diminishing visual focus as context length grows.
Figure does not normalize for the total number of tokens, which may obscure the disproportionate decline in attention to visual information as the caption length increases- especially since the number of text tokens naturally grows with longer
18

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
captions. To address this, we additionally analyze the average attention per token by dividing the total attention image and text tokens by their respective token counts. As shown in Figure 13, this normalized analysis reveals that attention image tokens decreases disproportionately faster than to text tokens as the context length increases.
Image Attention Score
0. 0.0006
Image
Text Attention Score
Image Q1-Q3
Text Q1-Q3
0.015
0.0 0.0004
0,010
8.0003
0.0002
0.005
0.0 0.0001
0.000
0
20
40
60
80
Context length
Figure Average attention weight trends for text and image tokens as function context longth during caption generation, computed by dividing attention by the number of respective tokens. This normalized view reveals that attention to image tokens decreases more sharply than text tokens as context length increases.
B.2. Analyses in Section 5.3
Visual Attention Analysis To analyze visual attention during caption generation, we conducted an experiment using the baseline model (LLaVA-1.5 7B) on 3,000 image samples from the DOCCI dataset. During the caption generation process, we measured the attention weights assigned to visual tokens when generating output tokens.
Specifically, visual attention weights were extracted from layer 20 and averaged across attention heads. We computed the total attention weights allocated to visual tokens for each output token and compared the results between the baseline model and the SPARC-enhanced model. These values were plotted to illustrate the differences in visual attention between the two approaches, as shown in Figure 7(a).
Impact on attention sinks To evaluate whether each method strengthens visual attention to contextually relevant regions or amplifies unrelated areas, we analyzed how different approaches affect attention sinks tokens that receive large attention values despite being unrelated to the actual context.
Specifically, we measured the ratio of attention scales by computing the average attention weight of sink tokens divided by the average attention weight of non-sink tokens. This ratio was used to assess the extent to which each method influences attention sinks, as illustrated in Figure 7(b).
For this experiment, we used the baseline model (LLaVA-1,5 7B) to generate captions for 3,000 image samples from the DOCCI dataset. During caption generation, we measured the attention weights assigned to visual tokens at layer 20 and averaged them across attention heads.
To identify visual attention sinks, we followed the approach described in previous work (Kang et al., 2025). Specifically, attention sinks were determined based on high- nigh-dimensional hidden states (self (self-attention layer inputs), where certain dimensions exhibited significantly large values. Tokens with high values in these specific dimensions were classified as sink tokens.
During generation, for each output token, we computed the ratio of average attention weights between sink tokens and non-sink tokens. These values were then averaged across all image samples and plotted for comparison across the baseline model, naive attention enhancement approach, and SPARC.
Results indicate that the naive approach significantly increases the proportion of attention allocated to sink tokens, while SPARC maintains the sink attention ratio at similar levels to the baseline. Additionally, prior results (Figure 7(a)) demon- strated that SPARC counteracts the decrease in attention values caused by longer context lengths. Taken together, these findings suggest that SPARC selectively reinforces attention to important visual tokens as context length increases, rather than indiscriminately amplifying all attention values.
19

---

CLAIR Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
C. Ablation Study on the Effectiveness of Hyperparameter and Setting Varitions
Ablation Study on Setting Choices To demonstrate the effectiveness of the proposed components in our method, we conducted series of ablation experiments. Specifically, we evaluated the impact of our token selection strategy and the progressive attention calibration mechanism, as presented in Table 7. For these experiments, we used L as the baseline model and measured performance using the CLAIR metric on the IIW-400 dataset.
First, we examined the effect of omitting the token selection process and applying the progressive attention calibration to all image tokens. To ensure fair comparison, we set the attention scaling factor to 1.007, following the same scaling strategy as when token selection is employed, as shown in Figure 7(a). While this approach led to performance improvements compared to the baseline, demonstrating the effectiveness of the progressive attention calibration mechanism, the results fell short of the performance achieved by our complete method. This indicates that while progressive attention calibration contributes to performance gains, the token selection strategy further enhances the model's ability to focus on the most informative tokens, leading to superior overall performance.
Next, we analyzed the effect of modifying the token selection approach by comparing the tokens to those from only the immediately preceding step, rather than leveraging the exponential moving average (EMA) across previous steps. This simplified selection mechanism yielded better performance than the baseline but did not match the effectiveness of our proposed method.
These ablation results underscore the contributions of both the token selection strategy and the progressive attention calibration in enhancing model performance. Our full method effectively mitigates attention dilution and ensures the consistent integration of salient visual information throughout the caption generation process.
Table CLAIR scores under different setting choices.
SETTING
BASELINE
$6.35
OURS
61.49
W/O SELECTION
59.10
W/O EMA
60.28
Ablation Study on Hyperparameter Impact To further analyze the effectiveness of the hyperparameters in our method, we conducted series of experiments by systematically varying key parameters and evaluating their impact on performance. Specifically, assessed how changes to individual hyperparameters influenced the model's performance, using LLa as the baseline model and measuring the CLAIR score on the IIW-400 dataset. The parameters examined in this study were those introduced in Section 5.1.
First, we evaluated the impact of the layer at which token selection is applied. Regardless of the layer at which token selection was performed and subsequent attention recalibration was applied, we observed an improvement in CLAIR scores compared to the baseline. Notably, selecting tokens in mid-to-late layers resulted in the most significant performance gains, with the highest CLAIR score observed at layer 20 (Table 8). This finding aligns with prior research, which has shown that the visual token attention patterns in MILMSs tend align more closely with semantically meaningful features in mid-to-late transformer layers (Jiang et al., 2024).
Next, we investigated the impact of the token selection threshold which determines the relative activation score used for token selection (Equation (5)). This score quantifies how much the attention on a given visual token jumps compared to the previous output token during caption generation. The results, presented in Table 8, reveal that excessively low threshold values degrade captioning performance. However, for appropriately chosen values, reinforcing attention on the selected tokens consistently led improvements over the bascline. The best performance was observed at 1.5, where the model achieved the highest CLAIR score.
Then, we evaluated the impact of the EMA (exponential moving average) smoothing factor 3, which controls the degree to which historical attention patterns influence token selection. The results, shown in Table 8, indicate that when 3 = 0, the model relies solely on the difference between the current step's 's attention and that of the immediately preceding step. In contrast, applying EMA smoothing enables a more stable token selection process. We found that using a small value of 0.1 was particularly effective, as allowed token selection to be influenced primarily by the most recent step while still
20

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
incorporating degree of historical information.
Finally, conducted additional ablation experiments the scaling parameter We investigated how increasing affects CLAIR scores and shown in Table 8, found that performance improved up to certain threshold, with the highest CLAIR score observed at = 1.1. However, increasing a beyond this point degraded the model's captioning performance.
Table 8. CLAIR scores according to different hyperparameters for LLaVA.
PARAMETER
LAYER
5 57.34
10 58.18
15 58.30
20 61.49
25 60.85
30
CLAIR
58.29
0.5
1.0
1.5
2.0
2.5 59.50
CLAIR
52.03
58.56
61.49
60.24
B
0.3
0.2
0.15
0.1
0.05
0 60.28
CLAIR
60.00
60.10
61.20
61.49
60.41
Q CLAIR
1.05
1.075
1.1
1.125
1.15
58.60
60.06
61.49
60.90
57.93
Furthermore, we extended the ablation studies additional models and datasets beyond VL..5. Specifically, we applied the same parameter variation framework L.L. VA-NeXT and Qwen2- -i.n using the DOCCI dataset for evaluation. We randomly sampled 500 images and generated captions for each model, and subsequently evaluating the outputs using the CLAIR score.
Following the same protocol as in the LLaVA-1.5 ablation, we systematically varied four key parameters- selection layer 1, token selection threshold т, EMA smoothing factor 3, and scaling parameter a a-and measured their respective impact on performance. Table 9 presents the results for LLaVA-NeXT, while Table 10 shows the results for Qwen2-VL. As reported in Table 4, the configuration used for LLa LLaVA-NeXT was ( 20, 4, a 1.1. 8 = 0.1 with CLAIR score of 66.99, and for Qwen2-VL was I 18, 3, a 1.1, 3 0.1 with CLAIR score of 80.64.
Table 9. CLAIR scores according to different hyperparameters for LLa VA-NeT.
PARAMETER
LAYER
15 65.25
20 66.99
CLAIR SCORE
63.97
64.61
63.58
T
2.5
3.0
3.5
4.0
CLAIR SCORE
68.10
68.60
67.81
66.99
3
0.2
0.15
0.1
0.05
0.0
CLAIR SCORE
65.93
66.34
66.99
66.57
67.80
1.05
1.075
1.1
1.125
CLAIR SCORE
64.20
65.10
66.99
67.26
Across all models (L.5, LLAVA- LLaVA-NeXT, Qwen2-VL.), we observe similar trends regarding the optimal range for the layer, a, 3 and t parameters -typically favoring mid-to-late transformer layers and slightly scaled values. Importantly, our method is training-free and incurs minimal additional computational overhead, making it both practical and efficient for lightweight hyperparameter tuning in real-world scenarios. While some degree of tuning required achieve optimal performance for each model, we highlight that effective settings can be identified with relative ease due to the simplicity and efficiency of our approach.
To further analyze the effect of a, we conducted additional ablation experiments using the CHAIR benchmark, measuring Precision, Recall, and Fl score at the object level based on the caption content as a varied. The results are presented in Figure 14, where we evaluated both LLaVA-1.5 1.5 and LLaVA-NeXX models. Figure 14(a) shows the results for LLaVA-1.5 where performance improved across Precision, Recall, and F1 scores a increased up to 1.1. At a 1.12, Recall slightly
21

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
Table 10. CLAIR scores according different hyperparameters for Qwen2-VT.
PARAMETER
LAYER
10
18 80.64
20
28 79.54
CLAIR SCORE
79.62
79.36
T
2.0
2.5
3.0
3.5
CLAIR SCORE
77.99
78.98
80.64
79.77
8
0.2
0.15
0.1
0.05
0.0
CLAIR SCORE
80.36
79.52
80.64
79.76
79.61
a
1.05
1.075
1.1
1.125
CLAIR SCORE
80.31
80.14
80.64
78.85
decreased compared to the baseline, but Precision improved significantly. Interestingly, at this setting, the Precision level was similar to that of the naive attention enhancement approach (Liu et al., 2025) in Table 2, while maintaining higher Recall, suggesting that our method provides a better balance in the Precision-Recdd for I.I.VA -A-Ne,,, where Precision remained close to the baseline, while Recall increased, indicating that our approach enhances caption quality by improving recall without sacrificing precision.
trade-off. Figure 14(b) presents the results
Abrls
Abha
(a)
(b)
Figure 14. Object-level Precision, Recall, and Fl scores for LLaVA-1.5 (a) and LLaVA-NeXT (b) with scaling parameter &
D. Detailed Caption Similarity Analysis
In the previous sections, we observed that naively enhancing the attention on visual tokens leads to a reduction in the number of objects mentioned the generated captions and decreases the diversity of attention patterns across visual tokens during the captioning process. To further investigate the relationship between the diversity of attention patterns and the variety of objects mentioned in the generated captions, propose metric that quantifies the similarity between sentences within the generated captions. Using this metric, we evaluate the caption diversity of different methods and analyze their effects. Our experimental results demonstrate that naive attention amplification strategies increase the similarity between sentences within captions, thereby reducing overall caption diversity. Furthermore, we show that our proposed token selection method directly enhances the diversity of objects mentioned in captions, supporting the significance of our approach.
Caption Similarity Anlaysis To quantify this effect, we introduce the Caption Similarity Score Csim, which measures the similarity between pairs of sentences within generated caption. It is defined as:
Cesm
Σ (S..8))E
(sim(s 8z)
(10)
Pl
where P denotes the set of all possible sentence pairs within a given caption. Each pair (s1, E P consists of two sentences 8;, 8j, and Ssim (8,, 8,) represents their similarity score. Specifically, Saim is calculated using a large language model (LLM), which evaluates the degree of overlap in visual content described by the paired sentences. higher C.sm score indicates greater similarity between sentences, suggesting lower diversity the generated caption.
22

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
For our experiments, generated captions for images sampled from the ITW-400 dataset and computed the caption similarity scores using LLLMA--32-11B-- (Dubey al., 2024). The prompt used for measuring similarity is follows:
You are trying to tell if two sentences Sentencel: (sentencel) Sentence2: {sentence2) On precise scale from to 100, how likely is it that the two sentences are describing the same visual contents?
describing the same visual contentss
This prompt is similar to the one used in CLAIR to compare the generated caption and the reference caption.
To validate the impact of our approach, we measured Csim for captions generated by the baseline model, the naive attention-enhance method, and our proposed method. As illustrated in Figure 15(a), our results indicate that naive attention amplification strategies, while intended to enhance focus across the entire image, inadvertently lead to higher caption similarity scores. This suggests that the generated captions tend to repeat descriptions of the same visual elements, thereby reducing the diversity of objects and features mentioned, limiting the informativeness and variety of the captions. In contrast, our method increases the diversity of captions by reducing Csim, demonstrating its effectiveness in improving caption variability.
Simiarity
Moon Count
202
Baseline
Ours
Baseline
(a)
(b)
Figure 15. (a) Comparison of caption similarity scores and number of selected tokens. (a) Caption similarity score across different methods. (b) Number dynamically selected tokens during caption generation. Methods that identify more tokens as important tend to yield lower similarity scores, indicating that focusing broader range of tokens inereases diversity in the generated descriptions.
Impact of Token Selection on Caption Similarity We further investigate whether the visual tokens selected by method dynamically align with the most relevant visual content for the generated text. Specifically, examine how frequently each image token is selected during the captioning process to assess its contextual significance. For each caption in the dataset, visual tokens are chosen at each generation step using the Relative Activation Score ( Section 4.2). To quantify the selection dynamics, we used a cumulative Selection Count in Equation (7), which represents the total number of times each token has been selected throughout the caption generation process.
UWith this metric, we assess the extent of dynamic token selection by categorizing tokens according to their total selection count throughout the selection count) and infrequently selected tokens (low selection count). By comparing the number of tokens in the high- selection group across different methods, we evaluate how different approaches influence the degree of dynamic visual token activation during captioning.
process. We classify tokens into groups: frequently selected tokens (high
We analyze the selection dynamics across 3,000 image samples from the DOCCI dataset by measuring the number of tokens in the high-selection group during the captioning process and computing the average. The results are presented in Figure 15(b).
Interestingly, methods that dynamically select larger number of tokens throughout the process tend to produce captions with lower similarity scores. This suggests that a higher degree of dynamic visual token selection contributes to greater variability in generated captions. This aligns with the intuition that focusing on a broader range of tokens increases diversity in the generated descriptions. The naive approach may restrict the model to a narrow set of tokens, thereby limiting caption diversity. In contrast, our proposed selection mechanism remains flexible, emphasizing contextually important tokens without ever-constraining the attention space. This flexibility enables richer and more coherent visual grounding in the final
23

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
captions, overcoming the limitations of earlier approaches.
In summary, findings demonstrate that naive attention-ennhanceeenn strategies inadvertently reduce caption diversity by increasing similarity between sentences. In contrast, our proposed token selection method enhances diversity by dynamically selecting a broader set of visually relevant tokens. These results highlight the importance of effective token selection in generating informative and diverse captions.
E. Evaluation on Other Hallucination Benchmark
While primary focus is on enhancing detailed image captioning, also performed additional experiments the POPE hallucination benchmark (Li et al., 2023b). Inspired by the evaluation setup proposed in (Mitra et al., 2024), where caption-based based reasoning improves MLLM performance on general multimodal tasks, we adopt similar strategy. Specifically, we prompted the model to first generate a caption for the image and then use it as part of the input to answer the question.
We evaluated Qwen2-VL (7B) using three approaches: the baseline model, our proposed method, and naive attention scaling (PAT) (Liu et al., 2025). For the naive method, we identified the optimal hyperparameter (a 0.2) before evaluation.
The table below summarizes the accuracy on the POPE benchmark, considering only those responses that adhered the instruction by including both caption and an answer.
Table 11. Accuracy the POPE hallucination benchmark.
Method
Accuracy (%)
Baseline
82.01
Naive Attn. Scaling
81.45
Ours
83.13
Our method shows an improvement over the baseline, while the naive attention scaling slightly reduces accuracy. Additionally, we evaluated the intructtionfollowwii behavior, measured as the proportion responses in which the model correctly generates an output that includes both a caption and an answer.
Table Instruction following behavior on the POPE benchmark.
Method
Instruction Following (%)
Naive Attn. Scaling Ours
76.84
92.72
These results indicate that naive attention scaling may reduce the model's sensitivity to the input prompt, whereas our method retains alignment with instruction while improving grounding accuracy.
24

---

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in MLLMS
F. CHAIR Metric Results and Precision-Recall Analysis
To provide a more detailed analysis hallucination and precision tradeoff, we present the CHAIR metric results along with recall, and Fl score. The CHAIR metric consists of two components:
Instance-level hallucination rate (CHAIR,)
CHAIR, |hallucinated objects}|
(11)
[all objects mentioned]]
This metric measures the proportion of hallucinated objects among all mentioned objects in generated captions. Since precision in our primary evaluation defined as CHAIR; a lower CHAIR; value directly translates to a higher precision, indicating fewer hallucinated objects in the generated captions.
Caption-level hallucination rate (CHAIR.)
CHAIR, ccaptions with hallucinated object]|
(12)
|all sentences]
This metric captures the proportion of captions containing at least one hallucinated object. A lower CHAIR, value suggests that hallucinations are less frequent across generated captions at the caption level.
Table Table summarizes the evaluation scores, where each value represents the average over five repeated experiments on randomly sampled 500 instances.
Our method adopts a progressive reinforcement mechanism that enhances attention to visual tokens the caption generation progresses. This strategy helps mitigate hallucinations in the latter part of the caption, leading noticeable improvement over baseline methods in later-token precision. However, since this approach primarily addresses hallucinations that occur later in the caption, it is inherently less effective in handling hallucinations that appear early in the sequence. As result, our method may still exhibit limitations when evaluated under metrics such as CHAIR,, which assess hallucination at the overall caption level.
Nevertheless, despite this inherent challenge, our approach demonstrates superior performance in score by achieving better balance between reducing hallucination (increasing precision) and improving recall. These results indicate that our method successfully enhances the model's ability generate captions that both accurate and comprehensive, establishing new standard for high-quality captioning.
Table 13. Comparison of CHAIR metric results on the MS COCO validation dataset for existing methods and our proposed approach. Our method achieves higher score by effectively mitigating hallucination while maintaining recall, demonstrating better balance compared to other approaches.
METHODS
CHAIR,
CHAIR;
RECALL
F1
BASELINE
53.96
15.30
79.46
81.99
OPERA
55.68
15.46
78.82
81.58
VCD
57.92
16.78
77.50
80.26
VOLCANO
46.16
12.46
77.82
82.39
PAI
34.92
9.36
72.44
80.52
OURS
51.52
12.28
79.98
83.67
25


Reviews:
[{"review_id": "pIlDgUrGBQ", "reviewer": "Reviewer_kRcu", "summary": "The authors study the effect of attention variability spatially and temporally and its impact on detailed image captioning with Visual Language Models (VLMs). The authors provide a detailed analysis of methods that tackle attention leaking from the image into the text as the caption grows, and they find that simply increasing the attention on the image makes the captioning focus on only a few objects and noise. Therefore, existing methods that improve the precision of captions do so at the cost of the recall. To mitigate this, they propose a new attention rescaling framework that highlights tokens whose attention scores frequently vary with the captioning process. They show that this approach can improve the alignment with reference captions in relevant datasets with three state-of-the-art VLMs.\n\n## Update after rebuttal\nThe authors have satisfied all my requests, and I hope the new results and discussions are reflected in the final version. Therefore, I update my score to \"Accept\".", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and insightful question. Your comments greatly helped us refine and clarify the paper. Please find our detailed response below—we’re happy to address any further concerns.\n\n**1. Analysis of the “noisy” scores**\n\nWe understand the reviewer’s concern regarding whether the increased attention diversity in our method may be attributed to “noisy” scores. To investigate this, we conducted an analysis of attention “noisiness” by measuring the proportion of attention assigned to **semantically relevant image regions** during decoding, using 5,000 randomly sampled images from the MSCOCO 2014 validation set.\n\nSpecifically, for each generated token corresponding to a ground truth object word in the caption, we computed the **proportion of total attention allocated to image tokens within the region of that object**, relative to the total attention across all image tokens. These object regions were identified using binary masks obtained from an open-vocabulary segmentation model, Grounded SAM2 [1], applied to each object in the image.\n\nThe table below compares the average attention allocated to relevant image regions across three methods: Baseline, Ours, and Naive Attention Scaling. A higher proportion indicates that the model is focusing more accurately on the relevant visual regions, suggesting lower attention noise.\n\n| Method | Attention on Relevant Image Regions (%)|\n| --- | --- |\n| Baseline | 17.85 |\n| Ours | 19.17 |\n| Naive Attn. Scaling | 15.50 |\n\nThese results indicate that our method assigns more focused attention to semantically meaningful image areas compared to the baseline and naive attention scaling, suggesting that our method results in *less noisy* visual attention.\n\nFurthermore, in Fig. 13(a), we report the **caption similarity score**, which measures the semantic similarity between sentences within a generated caption. Captions generated by our method show greater diversity in sentence content, reinforcing that the diversity is not due to randomness or noise in attention, but rather stems from meaningful differentiation in visual grounding and language generation.\n\n[1] Ren et al. Grounded SAM 2: Ground and Track Anything in Videos. IDEA-Research, GitHub, 2024.\n\n---\n**2. More Ablations for Parameters**\n\nWe appreciate the reviewer’s suggestion regarding additional ablation studies on different models and datasets. To address this point, we conducted additional ablation studies on LLaVA-NeXT (7B) and Qwen2-VL (7B) using the DOCCI dataset, analogous to Tables 6–9 in the paper. To avoid redundancy, we kindly refer you to our response to **Reviewer 4gUZ**, where detailed results and observations are provided. In brief, we observed consistent trends across models, with performance generally favoring mid-to-late layers and modest parameter scaling. Since our method is training-free and efficient, such tuning remains practical.\n\n**3. Clarification Regarding “Noisy” Attention Scores**\n\nWe sincerely apologize for any confusion caused by our use of the term “noisy” to describe certain attention scores. As discussed in [2], background tokens can carry important global context and are often beneficial in vision transformers.\n\nOur intent was not to imply that such attention is inherently harmful. Rather, our goal is to **enhance attention to semantically relevant visual regions** during decoding—not to suppress background tokens altogether. In fact, we agree that background tokens can carry useful information, and our method does not explicitly penalize them.\n\nHowever, we found that naive attention scaling may unintentionally amplify attention to globally dominant tokens (including background regions), which can **overwhelm local, task-relevant signals**. This may explain why captions generated with naive attention scaling often exhibit high precision but reduced recall—they tend to overemphasize prominent visual cues while neglecting finer details.\n\nWe sincerely thank the reviewer for pointing this out and will revise the manuscript to replace “noisy” with more accurate wording.\n\n[2] Darcet et al. Vision Transformers Need Registers. ICLR’24.\n\n**4. Attention weight trends for text and image tokens as context length increases**\n\n> Fig. 5 does not account for the share of total tokens.\n> \n\nThank you for the insightful comment. As noted, Fig. 5 did not normalize for the total number of tokens. To address this, we analyzed the average attention per token and found that attention to image tokens decreases disproportionately faster than to text tokens as context length increases.\n\n**5. Clarification on the meaning of “repeating” in evaluation**\n\nThank you for the question. The “repeating” refers to resampling 500 instances from the MSCOCO validation set five times, following prior work [3]. Generation was deterministic with temperature set to 0.\n\nLiu et al. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. ECCV’24. [3]", "claims_and_evidence": "The authors show a detailed analysis of naive attention scaling and a proposed method to numerically improves image captioning, based on the diversity of attention scores throughout the captioning generation. Attention diversity caused by their method is effectively shown in Fig 11c, but it is unclear whether this diversity is allocated to more or less “noisy” scores. The only example is Figure 2, which indeed shows a significant amount of “noisy” scores with the proposed method.\n\n\nThe authors base a significant part of their analysis on the attention scores of a querying token from the caption to the image tokens at a mid-to-late layer of the model. I agree with the implied statement that tokens preserve significant information relating to their initial embedding, but multiple layers of attention should also lead to a significant mixing of information. Therefore, asserting that a particular attention score is “noisy” in this setting needs more substantial evidence. If feasible, a saliency map of the image patches w.r.t. the attention scores of that layer could make this claim more solid. Besides, it is unclear to me that such “noisy” attention scores are inherently harmful as they have been shown to be used for computation by ViTs [1].\n\nFurthermore, the authors claim that this attention diversity correlates with the model not focusing on only a few relevant objects, therefore improving recall. This seems clearly supported by their improved recall scores.\n\n\n[1] Darcet et al. Vision Transformers Need Registers. ICLR’24.", "methods_and_evaluation_criteria": "After a detailed analysis of the limitations of current methods, the authors propose a reasonable method to mitigate the observed shortcomings. The metrics used to evaluate this method, namely object-matched and GPT-4o scored matching between gold captions and generated captions, make sense.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Overall, the authors chose two datasets and two reasonable and comprehensive metrics and showed their approach in different models. I would appreciate an analysis of the “noisy” scores for their approach and naive attention scaling.\nI would also appreciate ablations like those shown in Tables 6-9 for all models, ideally in all datasets. Even if fewer parameters are tried (e.g., only three values per hyperparameter), it would be good to see how sensitive these are across models since, e.g., the final result for Qwen-2-VL uses a different layer.", "relation_to_broader_scientific_literature": "The authors discuss prior work on improving captioning, specifically precision, by mitigating hallucinations. Recent work in this direction [2,3,4] tries to focus the textual generation process more on the visual input. Such past works show the benefits not only for image captioning but for other tasks such as visual question answering. However, the authors of this work show that temporally varying attention is key for higher recall and that simply scaling attention leads to higher precision at the cost of lower recall. In this sense, these findings could also be applied to other image-text tasks.\n\nIt is, however unclear how other techniques [5] that do not rely on increasing attention scores for improving visual attention in VLMs compare to the proposed method. [5] shows they can slightly improve recall while making larger improvement on CHAIR specific metrics. \n\n[2] Huo et al. Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models. ICLR’25.\n[3] Liu et al. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. ECCV’24.\n[4] Li et al. Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding. arXiv’25.\n[5] Xing et al. Mitigating Object Hallucination via Concentric Causal Attention. NeurIPS’24.", "essential_references_not_discussed": "The assumption that high attention scores in mid-to-late layers to tokens related to background patches, is harmful or “noisy” is not empirically supported. No relevant literature is mentioned on that aspect. In contrast, relevant works [1,6] show that large attention scores and/or activations are a key component of ViTs, and can even be employed for better performance.\n\n[6] Sun et al. Massive Activations in Large Language Models. COLM’24.", "other_strengths_and_weaknesses": "Main strengths:\n- Detailed analysis of naive scaling of attention for image captioning, and proof that this correlates with recall in image captioning\n- New training-free method to improve recall and precision in image captioning.\n- Well written paper.\nSummary of weaknesses:\n- No analysis of “noisy” attention scores. If feasible, use saliency map w.r.t. attention scores instead of attention scores directly; in such a deep layer, these might not correspond only to the original patch.\n- Fig 5 does not account for a share of total tokens.\n- No ablations like tables 6-9 for other models & other datasets already shown in the paper. Unclear how difficult it’d be to tune this framework in a new model.\n- The resulting variation of attention scores applying their method is currently in the appendix but seems relevant enough to be in the main paper.", "other_comments_or_suggestions": "Is Figure 5 weighting for the decreasing share of image tokens w.r.t. total tokens? If not, I think that is not very useful. It seems intuitive that as captioning goes on and the image tokens aren’t 100% of the previous tokens, the model has to devote some attention to the text it has already been output. I am not saying they should be weighted the same (clearly from previous work, they are not), but this plot would assume that text tokens need not be attended to. A more reasonable plot would weigh the attention score by the share of total tokens from each type.\nTypos:\nL204: “Specifically, Specifically, “\nL350: “We compar our method”", "questions_for_authors": "L372: “To ensure a robust evaluation, we randomly sample 500 instances and repeated the evaluation five times.” Does the “repeating” here come from 500 sampling subsets or for generating captions with non-determinism (temperature different than 0)? This could be made clearer, and if the reason is the latter, more details on the generation parameters should be provided.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EA5GIv0ibS", "reviewer": "Reviewer_JgJt", "summary": "The paper introduces an ​adaptive attention enhancement mechanism aimed at improving the precision of image captioning while maintaining an acceptable recall rate. Specifically, the ​selective attention enhancement strategy seems powerful according to its significant improvement in the precision of long caption generation, effectively alleviating the ​hallucinations problem in MLLM.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking the time and effort to evaluate our paper. We truly appreciate your insightful comments. Please find our detailed response to your comment below. If you have any further feedback, we would be grateful to hear it.\n\n**1. Efficiency Comparisons**\n\n> Since the paper involves additional attention amplification procedures, efficiency comparisons can be included. e.g., token generation speed.\n> \n\nThank you for your valuable suggestion. While our method involves attention amplification procedures, these introduce **only minimal overhead** compared to the original decoding process. In contrast, many previously proposed approaches require **additional decoding passes**, resulting in significant computational cost.\n\nTo provide a clearer comparison, we measured the **token generation time** (i.e., generation time per output token) across various methods. The following table presents the average generation time per token (in milliseconds) using an **RTX8000 GPU**:\n\n| Method | Token Generation Time (ms/token) |\n| --- | --- |\n| Baseline | 30.37 ± 0.73 |\n| Ours | 31.21 ± 0.61 |\n| Volcano | 109.98 ± 17.71 |\n| PAI | 57.75 ± 0.86 |\n| VCD | 59.44 ± 0.84 |\n| OPERA | 322.28 ± 118.26 |\n\nAs shown, our method performs similarly to the baseline, whereas other methods introduce 2x to 10x slower generation speeds. This demonstrates that our approach achieves **efficient generation with minimal computational overhead**.\n\nRegarding memory usage: our method only requires storing the **head-wise averaged attention scores for image tokens** at each layer from the previous decoding step. For example, in the case of LLaVA-1.5 (7B), this amounts to:\n\n32 (layers) × 576 (image tokens) × 2 bytes (float16) < 40 KB\n\nThis overhead is negligible in modern hardware setups.", "claims_and_evidence": "The paper is well-motivated, with all insights grounded in ​empirical evidence, making the claims convincing. The paper proposes three main insights:\n- ​Naive attention amplification reduces attention diversity, validated in Figure 2.\n​- Noise increases with caption length, demonstrated in Figure 3.\n- ​Visual focus weakens in long contexts, illustrated in Figure 5.\nThrough ​extensive experiments, the paper effectively proves the limitations of existing attention amplification methods and naturally derives the design of ​SPARC. The results also strongly support the proposed approach.", "methods_and_evaluation_criteria": "Yes. With EMA, the formulation of  Relative Activation Score seems plausible. The token selection and attention amplification methods based on the score are easy for implementation.", "theoretical_claims": "The paper does not involve proofs.", "experimental_designs_or_analyses": "Yes. The method is evaluated by CLAIR measuring image-caption alignment and CHAIR measuring hallucination. The comparisons in Table 1 and 2 cover most of SOTA methods for attention strengthening The paper also evaluate their methods plugged into different MLLMs including LLaVA-1.5, LLaVA-Next, and Qwen, demonstrating the consistent effectiveness of SPARC.", "relation_to_broader_scientific_literature": "Precision dropping of MLLM in long caption generation remains a long-standing problem. Most of previous works focus on attention amplification but do not obtain satisfactory results. The paper reveals **the underlying causes supported by experiments** and propose a simple but effective method to address it, which is indeed meaningful.", "essential_references_not_discussed": "To my knowledge, the paper includes most related works.", "other_strengths_and_weaknesses": "## Strengths\n- The observations validated by the paper may motivate many MLLM task, since hallucination in long-context scenarios is a generic problem in LLM. Besides, its attention patterns are also consistently investigated by the community.\n## Weaknesses\n- Since the paper involves additional attention amplification procedures,efficiency comparisons can be included, e.g., token generation speed.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWBni2CgAL", "reviewer": "Reviewer_3wnt", "summary": "This work proposes a training-free method to enhance detailed image captioning with improved balance between precision and recall by re-calibrating the attention values in multimodal large language models (MLLMs). This work first analyzes the attention patterns in MLLMs and finds that 1) trivially enlarging the attention values leads to reduced diversity and lower recall in the captions, and 2) as more tokens are generated and the context becomes longer, the attention becomes more noisy and less focused on visual tokens. Then, this work proposes to select important visual tokens based on the attention dynamics and adjust the attention values accordingly. Experiments show improved overall quality of image captioning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Concerns Regarding Performance Trade-off**\n\n> Compared with baselines, the proposed method significantly improves the recall, but the precision is hurt. For example, as shown in Table 2, the precision is ~3% lower than PAI. Similarly, in Figure 6, the human evaluation suggests a lower precision compared with \"Naive\" (PAI). This may lead to concerns on hallucination and misleading information in the captions. Ideally, the method should be able to adjust its hyperparameters to achieve a higher recall without hurting the precision.\n> \n\nWe appreciate the reviewer’s concern regarding the potential trade-off between recall and precision in our method. However, we would like to clarify that, as shown in **Table 2**, our method achieves **improvements in both precision and recall** compared to the baseline. In contrast, other prior methods, including \"Naive\" (PAI), tend to **improve precision at the cost of recall**.\n\nIn particular, while PAI demonstrates a higher precision than our method, it substantially reduces recall, which may not be acceptable for applications where **completeness of information is critical**. This is especially relevant in **real-world scenarios such as medical image reporting** or **automated content generation**, where omitting important visual elements may lead to misleading or insufficient descriptions. In such cases, **recall can be more crucial than precision**, as overly high precision may inadvertently filter out meaningful or necessary content.\n\n---\n**2. Evaluation on other hallucination benchmark**\n\nThank you for this valuable suggestion. While our primary focus is on **enhancing detailed image captioning**, we also performed additional experiments on the **POPE hallucination benchmark** [2],\n\nInspired by the evaluation setup proposed in [1], where caption-based reasoning improves MLLM performance on general multimodal tasks, we adopt a similar strategy. Specifically, we first generate a caption for the image and then use it as part of the input to answer the question.\n\nWe evaluated Qwen2-VL (7B) using three approaches: the baseline model, our proposed method, and naive attention scaling (PAI). For the naive method, we identified the optimal hyperparameter ($\\alpha$ = 0.2) before evaluation.\n\nThe table below summarizes the accuracy on the POPE benchmark:\n\n| Method | Accuracy (%) |\n| --- | --- |\n| Baseline | 82.01 |\n| Ours | 83.13 |\n| Naive Attn. Scaling | 81.45 |\n\nOur method shows an improvement over the baseline, while the naive attention scaling slightly reduces accuracy. Additionally, we evaluated the instruction-following behavior, measured as the proportion of responses in which the model correctly generates an output that includes both a caption and an answer.\n\n| Method | Instruction Following (%) |\n| --- | --- |\n| Ours | 92.72 |\n| Naive Attn. Scaling | 76.84 |\n\nThese results indicate that naive attention scaling may reduce the model's sensitivity to the input prompt, whereas our method retains alignment with instruction while improving grounding accuracy.\n\n---\n**3. Concerns Regarding the Choice of Hyperparameters**\n\nWe appreciate the reviewer’s thoughtful concern regarding the generalizability of hyperparameter settings across different models. To address this, we conducted additional ablations on LLaVA-NeXT (7B) and Qwen2-VL (7B) using the DOCCI dataset. For detailed results, please refer to our response to **Reviewer 4gUZ**. Briefly, across all models (LLaVA-1.5, LLaVA-NeXT, Qwen2-VL), we observe similar trends regarding the optimal range for the layer, $\\alpha$, $\\beta$, and $\\tau$ parameters—typically favoring mid-to-late transformer layers and slightly scaled values.\nWe also note that our method is **training-free** and imposes **minimal additional computational cost**, making it practical and efficient to perform lightweight hyperparameter searches in real-world applications.\n\n---\n**4. Computational Efficiency Comparison**\n\nThank you for this valuable suggestion. A detailed comparison of runtime and memory overhead—including per-token generation latency and storage requirements—is provided in our response to **Reviewer JgJt**. To avoid redundancy, we kindly refer you to that section for a full breakdown. In brief, our method introduces only minimal overhead compared to the baseline, while remaining significantly more efficient than prior approaches.", "claims_and_evidence": "- Compared with baselines, the proposed method significantly improves the recall, but the precision is hurt. For example, as shown in Table 2, the precision is ~3% lower than PAI. Similarly, in Figure 6, the human evaluation suggests a lower precision compared with \"Naive\" (PAI). This may leads to concerns on hallucination and misleading information in the captions. Ideally, the method should be able to adjust its hyper-parameters to achieve a higher recall without hurting the precision.", "methods_and_evaluation_criteria": "- In addition to dense image captioning, it is suggested to include evaluation on other hallucination benchmarks and general VQA tasks, to ensure a more comprehensive comparison with baselines like PAI. Currently this work only tests CLAIR and CHAIR, which are a bit limited considering the scope of the related baselines. \n\n- In the proposed method, there are a few hyper-parameters that need to be specified. Some are quite different across base MLLMs (see \"Implementation Details\" in Section 5.1). It might be challenging to select the hyper-parameters when applying the method to a new MLLM.", "theoretical_claims": "This work does not include theoretical claims.", "experimental_designs_or_analyses": "No concerns.", "relation_to_broader_scientific_literature": "This work proposes a new approach to dynamically augment the attention to visual tokens in MLLMs to produce better detailed image captioning. However, from the current evaluation, the improvement over existing methods seems not significant, and the usage cases might be limited.", "essential_references_not_discussed": "No concerns.", "other_strengths_and_weaknesses": "No more concerns.", "other_comments_or_suggestions": "- It might be beneficial to include the actual inference cost comparison. As the method needs to monitor and manipulate attention computation in the model, the generation speed may be slowed.\n- There are some typos to be fixed. For example, \"Specifically, Specifically, \" (page 4), \"In Section Section 3\" (page 4).", "questions_for_authors": "No more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0sFHIIxvKG", "reviewer": "Reviewer_4gUZ", "summary": "This paper focuses on improving detailed image captioning quality in VLMs. The authors argue that existing models struggle to maintain strong visual attention when generating longer captions, causing increased noise and reduced recall. To fix this, they propose a method that selectively strengthens visual attention by tracking significant changes in attention values over time. The method also reinforces consistently important visual tokens during decoding.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed and helpful review. We apologize for any confusion caused by the incomplete results in the original submission. Your comments have been very valuable, and we provide our detailed responses below. We would also appreciate any further suggestions or feedback.\n\n**1. Quantitative Evaluation for Attention Quality**\n\nThank you for raising this important point. We agree that intrinsic, quantitative evidence is necessary to directly support our claims regarding attention quality.\n\nTo this end, we analyzed **visual attention** by measuring how much attention is assigned to **semantically relevant image regions** during the decoding process, using 5,000 randomly sampled images from the MSCOCO 2014 validation set. Specifically, for each generated token corresponding to a ground truth object in the caption, we calculated the **total attention score allocated to image tokens** within the region of that object.\n\nTo identify these regions, we used an **open-vocabulary segmentation model (Grounded SAM2 [1])** to generate binary masks for all ground truth objects. During caption generation, for each object word token, we measured the **proportion of visual attention focused on the corresponding object region** out of the total attention score across all image tokens.\n\nThe table below summarizes the results across three methods:\n\n| Method | Attention on Relevant Image Regions (%) |\n| --- | --- |\n| Baseline | 17.85 |\n| Ours | 19.17 |\n| Naive Attention Scaling | 15.50 |\n\nThese results indicate that our method achieves better alignment between generated text and relevant visual regions compared to the baseline and naive attention scaling. This suggests that our model's visual attention is **less noisy** and more focused on semantically meaningful parts of the image during captioning.\n\n[1] Ren, T., Shen, S. Grounded SAM 2: Ground and Track Anything in Videos. IDEA-Research, GitHub, 2024.\n\n---\n**2. Computational Efficiency Comparison**\n\nThank you for your comment. We agree that empirical evidence is important to support our efficiency claims. A detailed comparison of runtime and memory overhead—including per-token generation latency and storage requirements—is provided in our response to **Reviewer JgJt**. To avoid redundancy, we kindly refer you to that section for a full breakdown. In brief, our method introduces only minimal overhead compared to the baseline, while remaining significantly more efficient than prior approaches.\n\n---\n**3. More Ablations for Parameters**\n\nThank you for raising this important concern regarding hyperparameter sensitivity. As mentioned in the main paper, we initially conducted ablation studies on hyperparameters using the **LLaVA-1.5 (7B)** model on the **IIW-400** dataset. The results are presented in Tables 6–9 of the supplementary material.\n\nTo further strengthen our analysis and address the reviewer’s concern, we extended these ablations to additional models and datasets. Specifically, we conducted experiments using **LLaVA-NeXT (7B)** and **Qwen2-VL (7B)** on the **DOCCI** dataset. We randomly sampled 500 images and evaluated the quality of the generated captions using the **CLAIR score**. The results are summarized below.\n\n**LLaVA-NeXT (7B)**\n\nBaseline CLAIR score: **62.49**\n\nOurs (Layer 20, $\\tau$=4, $\\alpha$=1.1, $\\beta$=0.1): **66.99**\n\n| Layer | 10 | 15 | 20 | 25 | 30 |\n| --- | --- | --- | --- | --- | --- |\n| Score | 63.97 | 65.25 | 66.99 | 64.61 | 63.58 |\n\n| $\\tau$ | 2.5 | 3.0 | 3.5 | 4.0 |\n| --- | --- | --- | --- | --- |\n| Score | 68.10 | 68.60 | 67.81 | 66.99 |\n\n| $\\alpha$ | 1.05 | 1.075 | 1.1 | 1.125 |\n| --- | --- | --- | --- | --- |\n| Score | 64.20 | 65.10 | 66.99 | 67.26 |\n\n| $\\beta$ | 0.2 | 0.15 | 0.1 | 0.05 | 0.0 |\n| --- | --- | --- | --- | --- | --- |\n| Score | 65.93 | 66.34 | 66.99 | 66.57 | 67.80 |\n\n**Qwen2-VL (7B)**\n\nBaseline CLAIR score: **79.22**\n\nOurs (Layer 18, $\\tau$=3, $\\alpha$=1.1, $\\beta$=0.1): **80.64**\n\n| Layer | 10 | 18 | 20 | 28 |\n| --- | --- | --- | --- | --- |\n| Score | 79.62 | 80.64 | 79.36 | 79.54 |\n\n| $\\tau$ | 2.0 | 2.5 | 3.0 | 3.5 |\n| --- | --- | --- | --- | --- |\n| Score | 77.99 | 78.98 | 80.64 | 79.77 |\n\n| $\\alpha$ | 1.05 | 1.075 | 1.1 | 1.125 |\n| --- | --- | --- | --- | --- |\n| Score | 80.31 | 80.14 | 80.64 | 78.85 |\n\n| $\\beta$ | 0.2 | 0.15 | 0.1 | 0.05 | 0.0 |\n| --- | --- | --- | --- | --- | --- |\n| Score | 80.36 | 79.52 | 80.64 | 79.76 | 79.61 |\n\nAcross all models (LLaVA-1.5, LLaVA-NeXT, Qwen2-VL), we observe similar trends regarding the optimal range for the layer, $\\alpha$, $\\beta$, and $\\tau$ parameters—typically favoring mid-to-late transformer layers and slightly scaled values.\nWe also note that our method is **training-free** and imposes **minimal additional computational cost**, making it practical and efficient to perform lightweight hyperparameter searches in real-world applications.", "claims_and_evidence": "- The authors claim their method improves visual attention quality, but the paper only shows results on captioning benchmarks. There’s no direct evidence proving attention actually got better. The attention maps provided still look pretty noisy, both before and after applying their method.\n- They mention “minimal computational overhead” but don’t back it up with runtime numbers or memory comparisons, so this efficiency claim isn’t convincing.", "methods_and_evaluation_criteria": "The benchmarks (CLAIR, CHAIR) and datasets (IIW-400, DOCCI, MS-COCO) used to evaluate caption quality are standard and appropriate. However, the evaluation is a bit incomplete—the authors rely solely on extrinsic metrics and provide no convincing intrinsic evaluation (e.g., clearer attention visualizations or quantitative metrics) to directly validate their claim that attention quality improved.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Benchmarks used (CLAIR, CHAIR, human evaluation) are appropriate and sufficient\n- Hyperparameter settings (α, β, τ) are arbitrarily chosen without sensitivity analysis, weakening the reliability of experimental results.\n- Human evaluation uses only 100 examples, limiting generalizability and robustness.\n- Computational overhead is claimed minimal, but no quantitative runtime or memory analysis is presented to support this.", "relation_to_broader_scientific_literature": "This paper studies detailed caption generation in VLMs, connecting closely to previous work on improving attention mechanisms in vision-based models, particularly techniques that aim to enhance visual attention quality. It also aligns with research addressing hallucination in natural language generation, particularly within text summarization, where maintaining factual consistency and reducing irrelevant or incorrect content is a common challenge.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors did a good rebuttal, with the final rating as 2 weak accept and 2 accept. The AC has checked the review, the discussion, the manuscript, and the rebuttal, and agrees with this assessment.
Decision: Accept (poster)