Allegator: Alleviating Attention Bias for Visual-Informed Text Generation

Authors: miso  choi
jinyoung  kim
minseo  yoon
ji soo lee
hyunwoo j kim
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:57:14

Abstract: Large Vision-Language Models (LVLMs) have shown remarkable performances in describing visual information with impressive linguistic ability, powering its diverse application. However, they often generate inaccurate descriptions of visual information, referred to as “hallucination”, therefore resolving this issue remains important for employing LVLMs in real-world scenarios. Although various approaches have been proposed in the literature, mitigating the hallucination in long-form generation remains challenging. We observed the Attention Bias phenomenon in LVLMs, where the model allocates a large amount of attention to a few specific tokens, regardless of inputs. With a thorough analysis of the correlation of Attention Bias with hallucination, we attribute the cause of hallucination to the
internal attention mechanism of Transformers. To ALLEviate hallucination in text GenerATOR (ALLEGATOR), we propose Attention Moderator that refines the attention efficiently in the training stage and Attention Soft-Clipping to guarantee the stable distribution for generating visual-grounded text. We empirically show that our methods enable generating more accurate descriptions by adaptively referring to visuals with sufficient attention. Allegator achieves significant improvements on hallucination benchmarks including POPE, CHAIR and AMBER, especially showing its effectiveness in long-form generation.

Main paper: 
 introduction: 
 Large Vision Language Models (LVLMs) have shown superior performance across various downstream tasks that require visual understanding and reasoning [1, 2, 3] . While LVLMs benefit from the improved capabilities of Large Language Models (LLMs), they also inherit the hallucination problem -generating inaccurate or biased responses, which are often irrelevant to visual input data. Recently, as systems across various applications [4, 5] rely on visual-grounded text generation frameworks, resolving the hallucination has become crucial. Many researchers are delving into this challenge to address the hallucination problem by optimizing LVLMs using labeled datasets with various techniques such as contrastive learning [6] , direct preference optimization [7, 8, 9, 10] , and rectifying [11, 12, 13, 14, 15] . We observed that LVLMs allocate a large amount of attention to a specific token regardless of input images, which we refer to as attention bias. This naturally leads us to the question: "If we alleviate attention bias, will the model experience less hallucination?"Based on the analysis of why a specific token experiences attention bias, we present a simple method that adjusts the attention to more effectively utilize visual information. We propose an AttentionInstruction: Write a detailed description.The image features a large airplane flying through the night sky, with its tail visible in front of it. A full moon is also present above and to one side of this scene, casting an eerie glow on everything around them as they fly by each other at high altitude over Earth's surface. In addition to these two main elements, there are several smaller objects scattered throughout the frame: three birds can be seen perched near trees or buildings along their flight path; another bird flies across partway down from left-to right edge towards center bottom area while yet others appear further away closer toward top middle section.The image features a large airplane flying high in an evening sky with moonlight illuminating its path as it flies past, capturing both celestial bodies together -plane and lunar eclipse.At what time is this photo taken?Baseline:The photo was taken at night.The photo was taken at nighttime, as evidenced by its dimly lit environment and a lack of natural lighting in addition to this dark setting with limited illumination from outside sources there is an open laptop on top of what appears like someone's bed or couch it seems that they are using their computer for entertainment purposes while sitting down Moderator, which is involved in self-attention training, and use refined attention in the generation phase. Furthermore, we scrutinize the relationship between hallucination and the increasing attention to previously generated texts during the generation process. Despite the model's ability to generate fluent and natural text, it fails to properly refer to visual data, resulting in hallucinations. Therefore, we propose to reallocate the attention of text tokens to visual tokens via our Attention Soft-Clipping. This simple remedy ensures LVLMs to generate responses while grounding visual content.We demonstrate Allegator alleviates the hallucination problem through extensive experiments on hallucination benchmarks including POPE [16] , CHAIR [17] , AMBER [18] , as well as various qualitative results. Our method is applicable to any Transformer-based models, and especially in this paper, we apply our method to various LVLMs such as InstructBLIP [1] , MiniGPT-4 [19] , and LLaVA-1.5 [20] . Our contributions are summarized as follows:• Attention Moderator. One key challenge of hallucination lies in the attention disproportion of Transformer. It hinders LVLMs to sufficiently refer to visual information, which leads to incorrect generation. We propose a novel attention refinement method, Attention Moderator, which enables adaptive reference to visual inputs.• Attention Soft-Clipping. Another challenge that auto-regressive Large Language Models experience is the gradual attention decrease in the earlier tokens. In visual-grounded text generation, insufficient attention to visual data often results in hallucinations. We present Attention Soft-Clipping, to guarantee sufficient attention to visual inputs.2 Problem FormulationQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: See Abstract and Sec. 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
attention bias in transformer: 
 We observed a phenomenon where a specific token dominates the attention probabilities in LVLMs, which we term as "Attention Bias". For instance in InstructBLIP [1] , the attention bias mainly appears on the 6 th visual token regardless of the inputs, and the same phenomenon occurs on the specific token in several different LVLMs [19, 20] . Fig. 2a illustrates the attention weight before the softmax operation where each row represents the relevance to the keys given a query. As shown in Fig. 2a , column-wise attention activation pattern occurs in the attention weight matrix due to a large amount of attention values assigned to the specific token. To verify this, we measured the attention weights of visual and instruction tokens before generating text tokens 3 . We observed that high attention scores are assigned to a few specific tokens, where about 48.96% of attention weight is assigned to the 6 th visual token and 17.43% to the 1 st instruction token. We provide additional examples for other LVLMs in the supplement. EQUATION)where A is the attention weight and v is the value state on a corresponding layer.With the residual connection, each layer focuses on learning the residual from the identity mapping instead of full transformation. This often simplifies the training, since the layer output can be obtained by refining the input features. However, due to the softmax operation in F(•), which always has to be summed to one, the non-trivial magnitude of the residual sum will be added, even though there's not much information to update [21, 22] . As a result, to keep the attention output in a small size, the model learns to assign a large amount of attention with a small value to a specific token which often appears as redundant [21] . As in Fig. 2b , we observed that the magnitude of value vectors corresponding to attention bias tokens appears to be small, which is related to the limited updates of residual.While autoregressive text generation is a widely adopted framework by current LLMs and LVLMs, it poses challenges in generating long sequences in that the attention to earlier tokens gradually diminishes. We visualize the attention summation of visual and text tokens in Fig. 3a . As the text generation process continues, an increasing number of tokens are considered at each prediction step, leading to a decrease in attention allocated to each token. As a result, in the case of LVLMs, the attention of visual tokens positioned in the earlier part of the input sequence diminishes over the prediction steps. As in Fig. 3a , attention summation of textual tokens increases while summation corresponding to the visual tokens decreases gradually. Several studies [21, 23, 24, 25] have discussed that high attention scores are assigned to less informative tokens (e.g., dots, quotation marks, and comma). The probability mass of those specific tokens results in a columnwise attention activation pattern in the corresponding token index. We observed that attention of the textual part increases drastically in less informative tokens (referred to as 'attention peak') when such tokens exist during the text generation process. Fig. 3a shows an example of attention peak. As in Fig. 3b , we find that among 375 captions with attention peak, 62.93% captions contain hallucinations, which is about twice as many as captions than those that do not have hallucination. This implies the high correlation of the hallucination phenomenon with attention peak, which inspires us to regulate attention distribution to ensure attention stability for generating visual-informed texts.Based on our insights, we pose the following hypothesis: the attention bias is derived from the model's behavior to update limited residuals. And the consistent pattern in attention arises irrespective of input images. Such constant attention can disturb the LVLMs to fully exploit the various visual information. Furthermore, increasing attention allocation to text, followed by the decline of visual attention, prevents LVLMs from referring to visual information properly. And this disproportion of attention causes the hallucination, especially in cases where the model generates long sequences.We therefore propose simple and efficient approaches to alleviate the hallucination in LVLMs by allowing the model to leverage sufficient visual information. We introduce an Attention Moderator to adjust attention by simply involving additional parameters in the attention mechanism, to alleviate the problem derived from attention bias. Also, in the text generation process, we reset the attention distribution to ensure that the model attends visual information at a sufficient level.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Sec. A Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we introduce a simple method to address the hallucination in LVLMs via alleviating attention bias. Before introducing our method, we briefly review the self-attention mechanism, which is a main module in LVLMs. We then present an Attention Moderator that alleviates attention bias in the training stage (Sec. 3.1). We also propose an Attention Soft-Clipping that is applied in the inference stage for ensuring sufficient attention to refer visual tokens (Sec. 3.2).Self-attention. Given the input sequence x = {x i } N i=1 ∈ R N ×dmodel , self-attention is defined as follows:EQUATIONNote that d h is the hidden dimension of query, key, and value state and each state are derived from linear projection matrices W Q , W K , W V ∈ R dmodel×d h , respectively.We here present Attention Moderator to mitigate the attention bias in LVLMs, which is associated with hallucination, by introducing additional parameters into the attention mechanism. Specifically, we insert a moderator matrix M ∈ R N ×N , which the same dimension as the original attention matrix A, into the attention mechanism as Eq. 3 as where we denote the attention as A ′ to differentiate from original attention in Eq. 2.EQUATIONWe construct a moderator matrix M by introducingW M1 s , W M2 s ∈ R dmodel×r , analogous to W Q , W K ∈ R dmodel×d h ,but with rank r much smaller than d h . Inspired by [26] , we share this Moderator matrices across all layers for efficiency and initialized using a Gaussian distribution, N (0, σ 2 ). In order to adapt the shared moderator M to each layer, we employ scaling vectors, λ b , λ d ∈ R r , arranged in a diagonal matrix form, i.e., Λ b = diag(λ b ) ∈ R r×r . The dimensionality of these vectors is determined by the rank r for each attention head. By multiplying the hidden states x with W M1 s , W M2 s , and scaling with λ b , λ d , the Moderator matrix is computed as follows:M (x) = (M 1,s (x)Λ b ) (M 2,s (x)Λ d ) T √ r ,whereEQUATIONIn our implementation, we analyze two versions of Attention Moderator: The first version involves finetuning both the Moderator and the scaling vectors, denoted as Moderator, and the second version involves the frozen Moderator with randomly initialized W M1 s and W M2 s while finetuning only the scaling vectors λ b , λ d , referred to as Moderator * . By capturing attention bias using the Moderator during training and A ′ has less bias and a higher variability.In the inference stage, we only employ A ′ with a smaller bias to mitigate hallucination asEQUATIONWe demonstrate that finetuning with Moderator * alleviates attention bias in the LVLM (see Fig. 6 ), leading to a reduction in hallucination (see Tab. 3).Attention Soft-Clipping refines a attention score distribution for generating visual-informed texts even especially in the long-form generation. It was motivated by our observation that in the generation step, insufficient (and decreasing) attention to visual inputs in a longer generation may often lead to hallucination. For discussion, we first define the notations of the attention weights corresponding to the visual, instruction, and textual tokens as follows:EQUATIONwhere V, I, T are sets of visual, instruction, and text tokens, respectively, t is the text token generation step, A ij indicates the attention weight of j th key to i th query and | • | denotes the cardinality of token sets. A V , A I , A T determine how much visual, instruction, and textual information to consider, respectively, to generate the output text token. During the text generation process, with more tokens being considered, there is a corresponding attention decrease in visual tokens. To address these issues, we refine the distribution of attention scores by adjusting the portions of attentions to token types.To prevent ∥A T ∥ 1 from taking up a large proportion when generating text token output, we define a threshold α and propose Attention Soft-Clipping with the following condition:EQUATION)where l is the layer index and l s , l e is the start and end layers to which Attention Soft-Clipping is applied, respectively. Further details about layer selection are discussed in the supplement. When the condition (Eq. 7) is satisfied, some of the attention weight of T are assigned to V, leading to a new adjusted output distribution given as:EQUATIONAttention Soft-Clipping enables the model to utilize a sufficient amount of visual information rather than relying too much on previously generated texts, thereby reducing hallucination.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: We have no theoretical result. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We demonstrate the effectiveness of Allegator, which is for alleviating the attention bias problem in LVLMs. We first introduce the detailed setting, comparison of trainable parameter, training data and evaluation metric (Sec. 4.1). We then evaluate the performance on benchmark datasets (Sec. 4.2). Finally, we provide further analysis of our proposed approach and qualitative results (Sec. 4.3). Detailed settings. We refer to the method that combines Moderator and Soft-Clipping as Allegator, and the method that applies Moderator * and Soft-Clipping as Allegator * . We evaluated our proposed method with three baseline models: InstructBLIP (7B) [1] , MiniGPT-4 (7B) [19] , LLaVA-1.5 (7B) [20] . When applying Moderator or Moderator * , we set the Q-Former and linear layer as the trainable in InstructBLIP and MiniGPT-4, while only FC layer as trainable in LLaVA-1.5. We apply Soft-Clipping to layers 4 th through 16 th of the LLM considering the variability of each layers. More implementation details are provided in the supplement.Comparision of trainable parameters. For a fair comarision, we set the rank of Moderator such that our method and LoRA have a comparable number of trainable parameters. Tab. 1 reveals the quantity of trainable parameters in LLM. Here, we use N l , N h to indicate the number of layers and attention heads, respectively and set the rank of Moderator to twice that of LoRA to ensure a comparable number of trainable parameters. LoRA has N l × 2 × 2 × d model × r trainable parameters, where multiplying by 4 refers to the two matrices W Q and W K to which LoRA is applied, as well as the two rank decomposition matrices W A and W B . In contrast, the Moderator has Training data. We use LLaVA-Instruct-150K [2] , which contains 158k unique language-image instruction following samples, including 58k conversations, 23k descriptions, and 77k complex reasoning. Since LLaVA-Instruct-150K is built upon COCO images and leverages GPT-4 for creating the multimodal instruction-following data, the visual and textual diversity of the dataset are constrained. To further enrich the fine-tuning data and train the model effectively, we augment LLaVA-Instruct-150K with other captioning datasets, Flickr30K [27] .N h × 2 × d model × 2r + N l × N h × 2r × 2,Evaluation benchmarks. We adopt two widely used object hallucination benchmarks, POPE [16] and CHAIR [17] . POPE formulates evaluation pipeline as a binary classification by asking a question with "Yes-or-No" answer, for which we report Accuracy, Precision and F1 score on three different settings: Random, Popular, and Adversarial. We report CHAIR in two forms: CHAIR I , which measures the proportion of mentioned objects that are hallucinated, and CHAIR S , which measures the proportion of sentences containing hallucinated objects.Additionally, we evaluate on AMBER [18] for comprehensive evaluation of Allegator * . We specifically report on generative task, where CHAIR, Cover, Hal and Cog scores are measured. CHAIR measures the percentage of hallucinated objects within the response, whereas Cover quantifies how many of the referenced objects are included. Hal identifies the presence of hallucinations, and Cog assesses how closely those hallucinations align with human cognition, focusing on the percentage of predefined hallucinated target objects. For faithful responses, LVLMs should minimize hallucinated objects while ensuring sufficient object coverage. Accordingly, lower scores for CHAIR, Hal, and Cog are desirable, with a higher Cover score being preferred.Performance on POPE benchmark. In Tab. 3, we present the performance of Allegator method on POPE [16] dataset, evaluated across InstructBLIP, MiniGPT-4, and LLaVA-1.5. Our method EQUATIONIncrease in attention variability. We analyze the attention variability, which are proposed in [30] , of the baseline and our method to measure the impact of the Attention Moderator. To verify how dynamically visual information is considered during the text generation process, we apply variability only to the visual part of the attention, which we refer to as Variability vis . Variability vis indicates how visual attention differs over different text inputs, which is analogously defined as follows:EQUATION)where X is the set of generated captions, Āi,j is the mean of A i,j over X, and Z(x) is the denominator. Details about the denominator Z(x) is provided in the supplement. This can be viewed as the mean of normalized absolute deviation and the high variability indicates that the attention is more dynamic. Fig. 5 shows that the model trained with our Attention Moderator exhibits high variability across all layers compared to its baseline. It implies that Attention Moderator promotes the attention to capture visual contents more adaptively. Mitigation of attention bias. We investigated the effect of applying Moderator during finetuning to alleviating Attention Bias compared to the baseline and LoRA finetuning. To evaluate this, we averaged 100 randomly sampled attention maps when generating captions for COCO Caption test split. The attention maps used in this analysis focus on the attention contributing the generation of the first token, denoted as {A i,j (x)} i=|V|+|I|,j=1:|V| . As shown in Fig. 6 , while the sum of attention allocated to visual tokens remains largely unchanged, the magnitude of 6th attention in Moderator is reduced compared to baseline and LoRA finetuning. These results indicate that training with the Moderator allows more tokens to be allocated to diverse visual tokens, thereby enabling a fuller exploitation of varied visual information.Qualitative results. Fig. 1 presents qualitative examples generated from the baseline model and Allegator given the instruction "Write a detailed description." and the question "At what time is this photo taken?". In long-form generation, the baseline struggles to understand the given image generating non-existent objects such as birds or trees, which occurs in the latter part of the sentence.In contrast, Allegator adequately incorporates visual information, successfully capturing complex elements like 'eclipse'. Moreover, in QA-form generation, our model provides answers with evidence considering more of the visual contents, while the baseline lacks comprehensive reasoning. Additional qualitative results that show the effectiveness of Attention Moderator and Attention Soft-Clipping are shown in Fig. 5 . We provide more qualitative results in supplementary material.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: See supplement (Sec. B). Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
related work: 
 Hallucination in LVLMs. LVLMs, despite their outstanding capabilities, suffer from hallucinations, where the texts generated by models are inconsistent with the given image. Several approaches are proposed to address hallucination in LVLMs such as enhancing the training with the proposed datasets [16, 31, 32, 33, 34] , refined instructions [6, 11, 12, 13, 14, 15] and direct preference optimization [7, 8, 9, 10] . There are other lines of work based on contrastive decoding to mitigate hallucination in LVLMs [23, 35, 36, 37, 38] . Our work differs from these approaches by focusing on solving hallucinations in LVLMs in terms of the attention mechanism.Attention bias in transformers. Multiple studies observed outliers or specific patterns in weights and activations, denoted as attention bias, in the text domain. They can be categorized into two groups depending on how the specific patterns are utilized. [39] leverage patterns in the attention map to enable stable and efficient language modeling. On the other hand, [21, 25, 40, 41, 42] aim to eliminate such patterns and activation. In the vision domain, [43] introduces additional learnable tokens to remove undesirable outliers in the attention map of Vision Transformer [44] . Motivated by those works, we first observed specific patterns also exist in LVLMs then proposed a framework that alleviates attention bias in LVLMs, therefore leading to improvements in hallucination.The image features a dining table set with various items, including two cakes. One of the larger and more prominent ones is shaped like an old red truck sitting on top of another smaller piece that resembles half-a car or part cut from it possibly representing some sort of vehicle birthday celebration for someone special in their life! There are also several chairs placed around this scene: one at each end near to where people might sit during meals; three additional seats situated closer together towards center left side of the room area behind them; as well as four other places scattered across different parts within reachable distance throughout the space surrounding these seating arrangements.The scene features a dining table with chairs surrounding it, and on top of this surface is placed an impressive cake in shape like red truck there are two pieces cut from that same birthday treat displayed nearby one piece has been transformed into resembling chevrolet logo another decorated item can be seen to its left.Instruction: Write a detailed description.Baseline:The image features a man riding on the back of his horse, walking along an open beach. He is wearing protective gear and appears to be enjoying himself as he navigates through shallow water near him while being accompanied by another person nearby who may also have joined in for this activity or simply observing from afar. In addition to these two individuals, there are several other people scattered throughout different parts of the scene: one can be seen further away towards left side; others appear closer together at various points around them some even standing close enough that they could potentially interact with each other during their outdoor adventure.The image features a man riding on the back of his horse, walking along an open beach. He is wearing protective gear and appears to be enjoying himself as he navigates through shallow water near some rocks in front him while holding onto reins tightly with both hands firmly grasping them together for control over their mounts' movements during this peaceful moment at sunset time by sea shore side. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See supplement (Sec. G). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 Hallucination is a challenging task that has to be resolved for a broad application of LVLMs. Attention bias that originates from the Transformer mechanism can lead to the occurrence of hallucination, which we demonstrate with a thorough analysis. To resolve the issue, we introduce Allegator, a novel framework that can reduce hallucination by refining attention. Attention Moderator enables dynamic attention to visual information according to given inputs. In addition, we propose Attention Soft-Clipping to ensure stable attention distribution for visual-informed text generation. Our method can reduce the hallucination effectively even in the challenging scenario of long sequence generation. By applying Allegator on several LVLMs, we show the effectiveness for mitigating hallucination in various benchmarks including POPE, CHAIR and AMBER.7 Acknowledgements and Disclosure of Funding NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We will provide code and checkpoint with the safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Sec. 4.1 and supplement (Sec. B). We use publicly available benchmark datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We have not released new asset.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: We have not done crowdsourcing experiments.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We further discuss the limitations of our study and the direction for future work. We finetune our model with the augmented dataset from LLaVA-Instruct-150K dataset. Since the LLaVA-Instruct-150K is a set of GPT-generated multimodal instruction-following data, it can contain some hallucinative captions despite it shows advanced performance. For example, the generated answer from GPT can involve contents that are not aligned with the prompt or instructions. For that reason, even though we mitigate hallucination with the proposed methods, it can be challenging to completely resolve hallucinations from LVLMs. Thus, improving our method for a more robust setting is a possible way for our future work.We apply our Allegator to a widely used LVLMs, InstructBLIP [1] ,MiniGPT-4 [19] and LLaVA-1.5 [20] . InstructBLIP is the vision-language instruction tuned model based on the pretrained BLIP-2 [45] . InstructBLIP architecture adopts a visual encoder, Q-Former, linear layer and LLM, the same as BLIP-2. In this work, we use ViT-g/14 [46] as a visual encoder and Vicuna-7B as LLM.We use default hyperparameters and model configurations provided by InstructBLIP in training.MiniGPT-4 has a similar architecture similar to InstructBLIP, consists of ViT [46] as a visual encoder, Q-Former , linear layer and Vicuna-7B as LLM. Compared to the previous two LVLMs, LLaVA-1.5 leverages only an linear layer to connect the visual encoder and LLM. In LLaVA-1.5, CLIP ViT-L-336px [47] is used as a visual encoder and Vicuna-7B as a LLM. We set rank to r = 32, 64, 32 in the Attention Moderator training for InstructBLIP, MiniGPT-4 and LLaVA-1.5, respectively, and in the Attention Soft-Clipping we set α, l s , l e to 0.2, 4, 16, respectively. All experiments are conducted with 4 NVIDIA RTX A6000 GPUs.In Sec. 2.1, we demonstrated a specific token assigned massive attention values in LVLMs, referred to as "Attention Bias". In this section, we provide additional visualization results of attention bias occurring in other LVLMs [19, 20] . The results are shown in Fig. 9 , 10. Additional results for other attention layers are provided in Fig. 11 , 12, 13. For visibility, we used 150 tokens in MiniGPT-4 [19] for visualization.In this section, we observe the effect of using Attention Moderator in the inference. Note that M is the Attention Moderator and A ′ denotes the attention weight trained with Attention Moderator M . Quantitative Results. In this section, we validate that the attention bias problem is alleviated by Attention Moderator M . In Fig. 6 , we report the POPE result based on how much Attention Moderator M is used in the inference.As the proportion of Attention Moderator included in the inference increases, Precision decreases consistently. Only using the attention weight A ′ in the inference achieves the best performance on all settings in POPE. Since Precision is closely related to hallucination, using Attention Moderator M in the training and not using it in the inference helps reduce hallucination.Qualitative Results. In this section, we provide qualitative examples based on how much Attention Moderator included in the caption generation process for alleviating the hallucination problem. As shown in Fig. 8 , the model generates more accurate captions when only using the attention weight A ′ . Leveraging both attention weight A ′ and Moderator M results in generating more hallucinatory captions compared to only using attention weight A ′ case. The model tends to generate captions containing objects that do not exist in the image or rely on the previously generated texts, therefore leading to hallucinatory captions. We suggest that using the attention weight A ′ only when generating a caption leads to less hallucinatory caption.In this section, we provide more details of the variability metric and denominator. Variability has been proposed in [30] to analyze how the self-attention with a causal mask varies across different inputs. The variability metric is defined as:EQUATION)where x is the sentence from the corpus X, α i,j is the attention weight of j th key to i th query and ᾱi,j is the mean of α i,j over all x ∈ X defined as:ᾱi,j = 1 |X| x∈X α i,j (x).This variability is properly normalized to a range of 0 to 1.In this paper, we modify the variability to examine the change of attention (before softmax) on visual tokens to closely monitor the effect of the proposed moderator. The modified variability of attention on visual tokens is formulated asEQUATIONIn this work, we set X as generated captions based on COCO test set [48] and we limit |x| to 10 the same as [30] although the max length of captions is larger. Z(x) mentioned in Eq. 9 corresponds to the denominator in Eq. 11.We further provide how we set the denominator and its upper bound. We now have: and triangle inequalities used above are as follows:|α i,j (x) -ᾱi,j | = |α i,j (x) + (-ᾱ i,j )| ≤ |α i,j (x)| + | -ᾱi,j | = |α i,j (x)| + |ᾱ i,j |, | x∈X ᾱi,j (x)| ≤ x∈X |α i,j (x)|.The image features a snowy scene with an orange fire hydrant covered in thick layers of frozen white powder. There are several trees visible, some closer to the viewer and others further away from it on either side or behind them. In addition to these elements, there is also another tree located towards one end nearer than halfway down along its length within this winter wonderland setting.Instruction: Write a detailed description.Allegator :The image features a red fire hydrant partially covered in snow, with an orange color that stands out against its surroundings it is located on top of some freshly fallen white powdery flakes and appears to be buried underneath several inches deep.Baseline:The image features a large brown cow with long horns, lying down in the middle of an open field. There are several other cows scattered around her position on both sides and behind them as well some closer to each side while others appear further away from one another or towards different parts of the scene's background area. In total there is at least sixteen cattle visible within this setting: four nearer left-hand corner; three more close by right hand edge (two slightly above); two pairs located next to their respective corners -top pair closest together followed closely afterward by bottom set that appears farther apart but still nearby; five spread out across center region including those resting alongside grassy patches along its length before reaching back into distant areas beyond it where they seem less numerous yet present nonetheless... Instruction: Write a detailed description.The image features a large, long-haired animal with horns laying down in an open field on top of lush green grass there are several cows and sheep scattered throughout this pastoral scene some cattle can be seen grazing or resting near each other while others roam around nearby fields overall, it appears to have been captured during sunny weather conditions.Baseline:The image features a wooden bench with two legs, each made of different colored boxes. These colorful blocks are stacked on top and underneath the leg bases to create an interesting design for this piece furniture in white space setting. There is also another box placed near one end corner that adds more visual interest as well.Instruction: Write a detailed description.Allegator :The image features a wooden bench with two legs, which is placed on top of metal pedestals or posts it has an unusual design as its base and supports are made from old magazines that have been painted in different colors this unique piece appears to be constructed out of recycled materials such as cardboard boxes turned into artwork by using them for decoration.! In this section, we demonstrate how we select the start and end layers of Attention Soft-Clipping. We use the instruction "Write a detailed description." to report CHAIR and variability for layer selection. Variability indicates the degree to which attention at a specific head and layer has various values depending on the input. Since high variability means attention distribution is not consistent and has various values depending on the input, we conjecture that attention with high variability is the input context-dependent attention. On the contrary, when the variability is low, we suppose that the effect of attention bias is large, leading to input context-independent attention. In Tab. 7, we report CHAIR for three layer selection schemes in InsturctBLIP. Note that, applying Attention Soft-Clipping to the model trained with Attention Moderator improves performance regardless of a layer selection scheme. Among three schemes, applying Attention Soft-Clipping from 4 th to 16 th layer achieves the best performance, and from 17 th to 32 th layer results in the least performance gain. As shown in Fig. 5 , variability tends to increase from 4 th to the 16 th layer both in the baseline and ours.We conjecture that the model tries to capture visual information related to the input without relying on bias when variability increases. On the contrary, variability tends to decrease from 17 th to 32 th which means attention bias dependence increases. We suggest that selecting layers where variability increases helps the model to utilize various visual information, leading to hallucination improvement.This work aims to improve hallucination in LVLMs by alleviating attention bias problems. With our proposed method, LVLM can generate accurate and detailed captions for a given input by utilizing diverse and sufficient visual information. To the best of our knowledge, Allegator itself does not involve direct societal impact. However, since this work is based on LVLMs, it may inherit some ethical issues from original LLMs and LVLMs. Further analysis and study of ethical issues would be beneficial and effective.The image features a dining table set with various items, including two cakes. One of the larger and more prominent ones is shaped like an old red truck sitting on top of another smaller piece that resembles half-a car or part cut from it possibly representing some sort of vehicle birthday celebration for someone special in their life! There are also several chairs placed around this scene: one at each end near to where people might sit during meals; three additional seats situated closer together towards center left side of the room area behind them; as well as four other places scattered across different parts within reachable distance throughout the space surrounding these seating arrangements.The image shows a dining room with an elegant wooden table and chairs, set for dinner there are four black leather seats arranged around it in addition to these seating options on one side of the scene is another chair placed closer towards the wall.Instruction: Write a detailed description.The image features a plate filled with sandwiches, cut into small pieces. There are several layers of the same type and size on top each other in an organized manner to create multiple tiers or sections for serving purposes. In total there is at least ten different slices arranged neatly across various parts of this dish's surface area some closer together while others further apart from one another. Each slice has its own unique appearance as they consistently feature cucumbers placed alongside tomatoes along their edges before being covered by bread crusts that form triangular shapes around them.The image shows a plate filled with sandwiches, cut into small triangular pieces the platter is decorated nicely and contains various types of fillings such as ham or turkey along with vegetables like tomatoes. There are multiple layers on top of each other in some areas to create an interesting presentation for guests.Instruction: Write a detailed description.The image features a group of birds standing on the edge or near water, possibly by an ocean. There are several small brown and white bird species present in this scene with some individuals scattered around different parts along one side close to each other while others stand further away from their peers towards either end closer to us as well. In total there is at least ten individual little wading shorebirds visible among them six can be seen closest together next to where they're perched overlooking blue waters behind it all.A group of birds is seen standing on the edge or near water, possibly by an ocean shore there are several small brown and black bird species present in this scene they appear to be gathered together for some reason.Instruction: Write a detailed description. Attention weights before the softmax were used based on the visual tokens and instruction tokens. The sum of results is 1 and obtained from InstructBLIP in 20 th attention layer and averaged over all attention heads. 


Decision: The paper received mixed ratings initially. The main concerns from the reviewers are the experimental results are not solid and the setting of the evaluation steps is not clear. In the rebuttal, the authors provided more details about the experiments and evaluations. Most of the concerns of the reviewers have been addressed. AC made the final recommendation of acceptance.

Reviews: review: paper_summary: paper_summary: This paper addresses the hallucination problem in VLMs through the lens of attention bias. It observes that attention tends to concentrate on a few specific tokens with small value features when the features do not need much update, and that attention is increasingly allocated to text tokens when generating long sequences. The paper then proposes to alleviate these issues by introducing an attention moderator and  performing attention soft-clipping.
strength_weakness: strength_weakness: The evaluation and the comparison with baselines are limited.
* The evaluations are performed only using InstructBLIP. It would be helpful to experiment with at least one or two other VLM varieties to verify the generalizability of the method.
* In Table 1, the performance for the HA-DPO and RLHF baselines are taken directly from their original papers. However, these numbers are not directly comparable, because three of them are based on different VLM varieties (miniGPT4 or LLaVA), and they are finetuned on different data. 
* There is no comparison with baselines (other than regular InstructBLIP) for CHAIR. It would be helpful to at least provide comparison with InstructBLIP+LoRA, as Table 1 shows that just finetuning with LoRA, without applying any specific hallucination mitigation techniques, already gives a large performance gain over regular InstructBLIP on POPE. This can help isolate the contribution of Allegator from the gains from having additional data/training.
questions: questions: 1. As mentioned in weaknesses, it would be helpful to see more comprehensive evaluations, with different VLMs and with more meaningful comparisons with other training-based methods. 
2. A number of qualitative examples from Allegator seem to demonstrate a lack of punctuations/sentence breaks at various places (e.g. the 2nd example in Fig 1 should have sentence breaks before “there is” and “it seems that”; similar for 1st example in Fig 7, and the examples in Fig 9 and 10.). What is the cause of this phenomenon? e.g. are they random unfortunate cases, are they caused by applying attention moderator or soft-clipping, or are there other reasons?
limitations: limitations: The authors have discussed the limitations and societal impact.
paper_summary: paper_summary: The paper tackles the problem of hallucinations in Large Vision-Language Models (LVLMs). The authors identify an Attention Bias phenomenon in LVLMs, where the model allocates a large amount of attention to a few specific tokens, regardless of inputs. They attribute this bias as a cause for hallucinations in LVLMs and propose a solution made of two components: 1. An attention moderator, which refines the attention of a pretrained self-attention layer, enabling adaptive reference to visual inputs. 2. Attention Soft-Clipping. This component aims to solve the challenge whereby auto-regressive Large Language Models experience a gradual attention decrease in the earlier tokens. In visual-grounded text generation, insufficient attention to visual data often results in hallucinations. The method is evaluated against two hallucination benchmarks (POPE, CHAIR), enabling significant improvement.
strength_weakness: strength_weakness: I have concerns with regard to the method and evaluation of the proposed method. 

It is unclear why the method encourages leveraging additional visual information. The method adds additional parameters to the attention mechanism. However, none of the components specifically consider the visual information, not the text. Why would the additional attention parameters also leverage the visual information more than the textural information? 

As for the evaluation, first, while there is a clear advantage in reducing hallucinations, the model may in fact generate less overall information, including factual information. This is in fact visible in Fig. 7. a, where the proposed approach no longer mentions that chairs are "at each end near to where people might sit during meals". This is not evaluated numerically, which makes it difficult to assess the correctness of the approach. 

I would also encourage the evaluation to be performed with respect to corresponding model sizes. As additional parameters are introduced in the additional attention parameters, how do existing baselines of the *same model size* perform on hallucinations? Since no model sizes are provided, this is difficult to assess. 

The authors also mention as motivation that the method  more adaptively refer to visual information then other LVLMs. However this is not demonstrated either numerically of visually. One could consider the attention masks specifically considering visual information as opposed to textual information. 

Minor: Fig. 1 and 7 mark hallucinations in red, and accurate descriptions in blue. However, the non-colored part is not annotated. This part can either be correct or incorrect, so this is unclear to me.
questions: questions: Please refer to the limitations above. I am interested in understanding the performance of the method with respect to the loss of factual information, as well as additional clarifications mentioned above.
limitations: limitations: The authors provide limitations which are adequate in my view.
paper_summary: paper_summary: In this paper, the authors introduce Allegator to reduce hallucination by refining attention. They propose an Attention Moderator to capture visual information according to given inputs and Attention Soft-Clipping to ensure a stable attention distribution for visual-informed text generation. However, some experimental details are missing, and the comparison of the experiments is not appropriate.
strength_weakness: strength_weakness: 1.	In Fig. 2, clarify whether the visualization is obtained from a single example or a collection of images.

2.	Lack of comparison with state-of-the-art methods, such as Opera [1].

[1] Huang Q, Dong X, Zhang P, et al. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13418-13427.

3.	Details of the experiment are missing, such as the specific InstructBLIP used, parameters of the fine-tuning process, the clipping layers, and so on.

4.	No experiments on other families of VLMs. Evaluate the proposed method on LLaVA and MiniGPT4 as HA-DPO did.

5.	In Table 1, performance should be compared on the same network structure with the same parameter amount.

6.	In Table 2, the Mask is not mentioned in the paper.

7.	More datasets should be evaluated, such as COCO, which can be referred to in Opera.

8.	The output text after Allegator is shorter than the baseline. Is the decline in hallucination caused by the shorter output or the proposed method?
questions: questions: 1. Why not using Attention Moderator performs better in the reference stage?
limitations: limitations: 1.	More details of the training process, evaluation method, and other specifics should be included in the paper. 

2.	The evaluation should be compared on the same architecture, and a more adequate assessment should be conducted.

Comments: title: Gentle reminder for Reviewer SG4y
Response to Reviewer
General response and a kind reminder
comment: Dear SG4y, 

Thank you for your valuable insights and the time you've dedicated to reviewing our paper. Your comment helped us significantly to improve the paper. As the rebuttal period is nearing its close, we would like to kindly remind you that there are only a few hours left. We would greatly appreciate your response to confirm that we have adequately addressed your concerns.

Thanks.
We appreciate your response and the improved rating. As you encouraged, we'll incorporate the new experiment results in our manuscripts. Thanks again for your valuable feedback.
Thank the authors for the response and the additional experiments. The rebuttal has addressed my concerns, and the new results have strengthened the work. Based on this, I raised my score from 5 to 6. I encourage the authors to include these new results in the manuscript.
We sincerely thank all the reviewers for their constructive and valuable feedback.

As a follow-up to our rebuttal, we would like to gently remind the reviewers that the discussion period is nearing its end. We hope to take advantage of this open response window to further discuss the paper, address any additional questions, and improve the quality of our paper.

After the main paper submission, we have conducted additional experiments on a broader range of baselines and benchmarks to provide a more comprehensive evaluation, and we were able to address the concerns raised during the rebuttal period.

By the reviewer’s insightful feedback, we could considerably enhance the paper and clarify the ambiguous areas discussed in the review.

Below we summarize our key modifications during the rebuttal phase.

- To address the common concerns raised by **jjA2**, **2KMG**, **SG4y**, we demonstrated the generalizability to other LVLMs (MiniGPT-4, LLaVA-1.5) as well as other benchmarks (POPE-A-OKVQA / GQA, AMBER generative task, MME).
- In response to the feedback from **2KMG** and **SG4y**, we update the main table by verifying our proposed method against the baseline + LoRA in terms of the number of parameters.
- More experiments have been conducted on various aspects, including the comparison with OPERA (Reviewer **SG4y**), the addition of baseline+LoRA to ablation table (Reviewer **jjA2**), evaluation of hallucination regardless of generation length (Reviewer **SG4y**).
- We have addressed the feedback from Reviewer **SG4y** by including more detailed explanations and clarifications about the experimental details.
- Thanks to Reviewer **jjA2**'s thorough feedback regarding the lack of punctuation, we provided detailed explanation for the cause of potential failure in more depth.
- As recommended by Reviewer **2KMG**, we also present the reason for the effect of method and evaluation in terms of factual knowledge.

We will thoroughly consider the reviewer’s comment to make further improvement to our manuscripts (including typo ‘Mask’ and citation ‘OPERA’ by Reviewer **SG4y**, and clear annotation for hallucination by Reviewer **2KMG**).

We thank Reviewer **2KMG** for positive reconsideration after reviewing our rebuttals.

We would greatly appreciate any further feedback or confirmation that our responses have adequately addressed reviewer’s concerns.