Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuning

## Abstract

Abstract Mitigating hallucinations of Large Vision Language Models (LVLMs) is crucial to enhance their reliability for general-purpose assistants.
This paper shows that such hallucinations of LVLMs can be significantly exacerbated by preceding user-system dialogues.
To precisely measure this, we first present an evaluation benchmark by extending popular multi-modal benchmark datasets with prepended hallucinatory dialogues powered by our novel Adversarial Question Generator (AQG), which can automatically generate image-related yet adversarial dialogues by adopting adversarial attacks on LVLMs. On our benchmark, the zero-shot performance of state-of-the-art LVLMs drops significantly for both the VQA and Captioning tasks.
Next, we further reveal this hallucination is mainly due to the prediction bias toward preceding dialogues rather than visual content.
To reduce this bias, we propose Adversarial Instruction Tuning (AIT) that robustly fine-tunes LVLMs against hallucinatory dialogues.
Extensive experiments show our proposed approach successfully reduces dialogue hallucination while maintaining performance.

## 1 Introduction

Developing a general-purpose assistant that interacts with humans through channels such as vision and language is one of the important problems in artificial intelligence.
Inspired by the remarkable success of Large Language Models (LLMs), such as ChatGPT Ouyang et al. (), the community has paid growing interest in developing *multi-modal* assistants, so-called Large Vision Language Models (LVLMs), that align vision foundation models Chen et al. (), Radford et al. () with LLMs to support visual-language instructions.
Many LVLMs including LLaVA Liu et al. (), MiniGPT-4 Zhu et al. (), and InstructBLIP Dai et al. () have shown powerful zero-shot generalization ability in various vision-language tasks such as classification Pham et al. (), Park et al. (), detection Li et al. (), visual question answering (VQA) Song et al. (), and Captioning Xu et al. ().

Despite their great success, several studies have revealed that LVLMs are prone to hallucination issues Ji et al. (), Bang et al. ().
While most studies focus on *object* hallucinations Li et al. (), Liu et al. (), where LVLMs often answer inconsistently with contents of objects in a given image, the effect of user-system *dialogues* on hallucination has received little attention.
Surprisingly, we found that such hallucinations can be significantly exacerbated by preceding user-system dialogues.
For example, as shown in Figure 1(a), certain contents in preceding dialogues (“*eco-friendly*”) conflicting with the current question can distract LVLMs, resulting in incorrect answers (“*Wood*”).
This problem, which we call *dialogue hallucination*, is crucial in practice because a user usually interacts with the system through multi-round chats so that the user can unintentionally attack LVLMs in early chats and get unfaithful answers in later chats.

Figure: Figure 1: (a) shows an example of dialogue hallucination generated by an LVLM (e.g., LLaVA Liu et al. ()) for a test example in ScienceQA dataset; (b) shows the average performance drop of LLaVA and AIT on EvalDial for VQA and Captioning tasks with prepended adversarial dialogues.
Refer to caption: x1.png

In this paper, we first present an evaluation benchmark, EvalDial, to more precisely measure the dialogue hallucination of LVLMs.
Our benchmark is constructed on popular vision-language benchmark datasets for VQA and Captioning tasks Xu et al. ().
Specifically, for each *test* example in each dataset, we create corresponding hallucinatory dialogues that can be prepended to the original test question.
Moreover, to mimic actual user behaviors interacting with the assistant within visual contexts of a given image, we further introduce Adversarial Question Generator (AQG), which automatically generates *image-related* yet *adversarial* dialogues, by steadily incorporating an extra LVLM into the black-box optimization of adversarial attack Ilyas et al. (), Maus et al. ().
With optimization, AQG can generate effective adversarial questions, while GPT-4 or other textual-based red teaming methods struggle to generate such subtle cases.
On EvalDial, the zero-shot performance of state-of-the-art LVLMs drops by up to 37.7$\%$ for the VQA task and 59.6$\%$ for the Captioning task.

To mitigate the dialogue hallucination, we conduct input token attention analysis and embedding distribution analysis. We find that the hallucination is mainly due to the prediction *bias* to preceding dialogues rather than visual contents.
Therefore, we propose Adversarial Instruction Tuning (AIT) that aims to reduce such prediction bias by robustly fine-tuning LVLMs on augmented visual-instruction datasets with hallucinatory dialogues.
Specifically, we introduce masked instruction tuning to focus on the target answers instead of hallucinatory responses from adversarial dialogues.
Extensive experiments on six vision-language datasets in EvalDial demonstrate that AIT successfully reduces the dialogue hallucination while maintaining the performance of LVLM for both VQA and Captioning tasks, as shown in Figure 1(b).

Our main contributions can be summarized as:

- •
We find that LVLMs are prone to hallucination by preceding dialogues.
- •
We present an evaluation benchmark (EvalDial) for dialogue hallucination with a novel adversarial question generator (AQG).
- •
We reveal LVLM’s prediction bias toward hallucinatory dialogues by input token attention analysis.
- •
We propose AIT with masked instruction tuning that successfully reduces the dialogue hallucination on many vision-language datasets.

## 2 Related Work

### 2.1 Instruction-following LVLMs

Instruction-tuning LLMs such as GPT Brown et al. () have significantly enhanced their zero-shot generalization ability in various NLP tasks Wang et al. (), resulting in instruction-following LLMs such as ChatGPT Ouyang et al. ().
Recently, this instruction-tuning idea has been actively extended to *vision-language* domains, and many instruction-following LVLMs have been developed Bai et al. (), Liu et al. (), Zhu et al. (), Dai et al. (), Wang et al. (), Achiam et al. (), Team et al. ().
In general, most LVLMs combine pre-trained vision encoders (e.g., CLIP Radford et al. ()) with LLMs by fine-tuning them on visual-language instruction datasets Zhang et al. (), Koh et al. ().
Notably, LLaVA Liu et al. () projects CLIP to LLaMA Touvron et al. (), and then fine-tunes the models with a projection layer on a visual instruction dataset Peng et al. ().
Similarly, MiniGPT-4 Zhu et al. () uses BLIP-2 Li et al. () as visual encoder and Vicuna as language decoder, and InstructBLIP Dai et al. () uses Q-former as the projection layer.
These models have shown a powerful zero-shot performance in various vision-language tasks including VQA and Image Captioning Xu et al. ().

### 2.2 Hallucinations of LVLMs

LVLMs are prone to hallucination issues as their output descriptions are often inconsistent with the input images and text instructions Ji et al. (), Bang et al. (), Bai et al. ().
Most prior work focuses on *object* hallucination where the output descriptions of objects are non-existent or inaccurate from the given image Rohrbach et al. (), Zhou et al. ().
Many evaluation benchmarks for object hallucination have been proposed Gunjal et al. (), Sun et al. (). POPE Li et al. () converts the hallucination detection as a binary classification, GAVIE Liu et al. () leverages GPT-4 to evaluate the hallucination, and THRONE Kaul et al. () addresses hallucinations in open-ended free-form generations.
To mitigate this, many works tried to enrich the visual-instruction datasets.
LRV-Instruction Liu et al. () reveals existing visual-instruction datasets are biased to positive responses, so they append instructions with negative responses in robust fine-tuning.
HalluciDocter Yu et al. () introduces a hallucination cross-checking paradigm that can recover visual-instruction data. HACL Jiang et al. () proposes a hallucination-augmented contrastive learning framework.
Note that, while some recent literature investigates the effect of deceptive prompts and dialogue hallucination on LVLMs Shi et al. (), Qian et al. (), Chen et al. (), Cao et al. (), they only provide *hand-crafted* or *singular domain* evaluation datasets.

### 2.3 Adversarial Attacks on Language Models

Adversarial attacks aim to ruin output predictions of a model by perturbing the input examples Chakraborty et al. (), Ilyas et al. ().
For attacking LLMs, AdvPrompt Maus et al. () finds adversarial prompts to generate nonsensical text by increasing the perplexity of the output tokens. GCG Zou et al. () obtains adversarial suffix prompts to generate objectionable behavior, such as harmful content. Harmbench Mantas et al. () proposed a framework for large-scale automated red teaming methods and defence of LLMs.
However, since such generated adversarial prompts are *incomprehensive* to humans, e.g., a sequence of random letters, they are *not* directly applicable to generating adversarial dialogues for LVLMs that must be in natural language.

## 3 Dialogue Hallucination and Evaluation Benchmark

We first formulate an LVLM and its dialogue hallucination.
Then, we describe EvalDial, a benchmark we release to evaluate dialogue hallucination, powered by our novel Adversarial Question Generator.

Figure: Figure 2: Overview of dialogue hallucinations on EvalDial. A test example on ScienceQA that LLaVA originally answers correctly becomes hallucinated after three types of prepended dialogues, i.e., General, Random, and Adversarial.
Refer to caption: x2.png

### 3.1 Dialogue Hallucination of LVLMs

Instruction-following LVLM.
For an input image $\text{X}_{\texttt{v}}$ and a user question $\text{X}_{\texttt{q}}$, an *instruction-following LVLM* $f_{\text{LVLM}}$ aims to generate a corresponding output text answer $\text{X}_{\texttt{a}}$. For multi-round conversation, the $t$-th round answer $\text{X}^{t}_{\texttt{a}}$ can be formulated as:

$$ $\text{X}^{t}_{\texttt{a}}=f_{\text{LVLM}}(\text{X}_{\texttt{v}},\text{X}^{<t}_ {\texttt{dialogue}},\text{X}^{t}_{\texttt{q}}),$ (1) $$

where $\text{X}^{<t}_{\texttt{dialogue}}\!\!=\!(\text{X}^{1}_{\texttt{q}},\text{X}^{1
}_{\texttt{a}},\cdots,\text{X}^{t-1}_{\texttt{q}},\text{X}^{t-1}_{\texttt{a}})$ is a sequence of all previous dialogues before asking the $t$-th round question $\text{X}^{t}_{\texttt{q}}$. Here, we denote $i$-th round dialogue $\text{X}^{i}_{\texttt{dialogue}}=(\text{X}^{i}_{\texttt{q}},\text{X}^{i}_{
\texttt{a}})$ as a pair of a user question and the corresponding answer from LVLM at round $i$.

Dialogue Hallucination.
According to hallucination literature Ji et al. (), Bang et al. (), the most inclusive and standard definition of hallucinations is “the generated content that is nonsensical or unfaithful to the given source content”. Based on this, we define the *dialogue hallucination* of LVLMs as in Definition 3.1.

###### Definition 3.1 .

(Dialogue Hallucination)
We call a generated answer $\tilde{\text{X}}_{a}$, that is faithful *without* any dialogue but becomes unfaithful *after* some preceding dialogues, *dialogue hallucination*. That is, the output answer $\tilde{\text{X}}_{a}$ hallucinated by the preceding dialogues is represented as $f_{\text{LVLM}}(\text{X}_{\texttt{v}},\text{X}^{<t}_{\texttt{dialogue}},\text{
X}^{t}_{\texttt{q}})\!=\!\tilde{\text{X}}_{\texttt{a}}$ while $\!f_{\text{LVLM}}(\text{X}_{\texttt{v}},\!\text{X}^{t}_{\texttt{q}})\!=\!{
\text{X}}_{\texttt{a}}$, where ${\text{X}}_{\texttt{a}}$ represents the originally non-hallucinated answer. ∎

Note that, dialogue hallucination can include various types of generated contents, such as wrong answers for VQA Ji et al. (), inaccurate descriptions for Captioning Xu et al. (), and responses of non-existent contents for Object-finding Li et al. ().

###### Definition 3.1 .

### 3.2 EvalDial: An Evaluation Benchmark

We construct EvalDial on top of popular vision-language *test* datasets; ScienceQA Lu et al. (), OKVQA Marino et al. (), GQA Hudson and Manning (), and IconQA Lu et al. () datasets for VQA task, and NoCaps Agrawal et al. (), Flickr-30k Plummer et al. (), and WHOOPS Bitton-Guetta et al. () datasets for Captioning task.
For each test example in each dataset, we create three types of dialogue, i.e., *General*, *Random*, and *Adversarial*, that are prepended into the original test question or instruction. Figure 2 illustrates more details of EvalDial.

- 1.
General Dialogue contains a general question, that can be universally asked to any image, and its corresponding answer is obtained from LVLM. For example, a general dialogue can be *“Q. What is the dominant color in the image? A. It’s blue”*. We extract 10 general questions from GPT by prompting *“Generate 10 general questions for a random image”*. See Appendix A for details.
- 2.
Random Dialogue consists of a pair of random questions, that are completely irrelevant to a given image, and its corresponding answer obtained from LVLM. For example, given a car image, a random dialogue can be *“Q. what kind of animals are these? A. there are no animals”*. To generate such questions, we randomly extract questions from the VQA-v2 dataset Goyal et al. (), which does not have an overlapping set of questions with the aforementioned benchmark test datasets.
- 3.
Adversarial Dialogue contains an *image-related* yet *adversarial* question that causes hallucinations to the original test question. Because real users often have chats related to the context of the given image, it is essential to verify LVLM’s robustness against the image-related but adversarial dialogue. However, generating such subtle questions is very challenging. Thus, we propose AQG, an adversarial question generator based on black-box adversarial attack techniques Andriushchenko et al. (), Maus et al. () as elaborated in Section 3.3. Detailed generated adversarial questions are in Appendix B.

Note that, for all three types of dialogues, EvalDial only contains questions without corresponding answers, since the answers are naturally generated by LVLMs in the test phase.
Evaluation results of state-of-the-art LVLMs on EvalDial can be found in Section 5.

### 3.3 Adversarial Question Generator

To mimic real-world user-system interactions, the adversarial dialogues should be *image-related* and *natural-sounding*, yet *adversarial*.
However, automatically generating such subtle dialogues in any context is very challenging, because LVLMs usually do not know when they hallucinate, which means it is difficult to obtain these adversarial dialogues by simply prompting Gunjal et al. ().
Therefore, as in Figure 3, we propose AQG that can automatically generate natural-sounding adversarial questions by adopting adversarial attack techniques with an extra LVLM. Overall, AQG consists of two common components in adversarial attack; (1) threat model and (2) adversarial target.

Figure: Figure 3: shows the overview of AQG, generating an adversarial dialogue $\text{X}^{adv}_{\texttt{dialogue}}$ (in yellow box) to hallucinate the answer $\text{X}^{adv}_{\texttt{a}}$ (in green box) by incorporating an extra LVLM into the optimization of adversarial attack
Refer to caption: x3.png

Threat Model.
A threat model represents a specific type of attack, *e.g.*, $l_{2}$-bounded noise for image classification Andriushchenko et al. (), or token-restricted prompt for language models Maus et al. ().
Then, the threat model of AQG should be confined to image-related and natural-sounding questions.
To meet this requirement, AQG leverage an extra LVLM $f^{adv}_{\text{LVLM}}$ and force it to generate image-related and natural-sounding dialogues by only updating its prompt token embeddings $\text{Z}_{\texttt{prompt}}={\texttt{tokenize}}(\text{X}_{\texttt{prompt}})$, where $\text{X}_{\texttt{prompt}}$ is an input prompt of $f^{adv}_{\text{LVLM}}$.

The adversarial prompt $\text{X}_{\texttt{prompt}}$ consists of a *fixed* template prompt $\text{X}_{\texttt{template}}$, *e.g.*, “*generate an image-related dialogue about*”, concatenated with an *updatable* context prompt $\text{X}_{\texttt{context}}$ initialized as “*background, colors, history, etc*”, such that $\text{X}^{init}_{\texttt{prompt}}=[\text{X}_{\texttt{template}};\text{X}_{
\texttt{context}}]$.

In optimization, we only perturb the context prompt by injecting a random noise $\epsilon$ into the context token embeddings $\text{Z}_{\texttt{context}}$. The random noise $\epsilon$ is sampled from a Gaussian distribution with the mean of 0 and the standard deviation $\sigma=0.1*\text{AvgDist}$, where AvgDist is the average distance between embeddings of all possible tokens, which is shown to be effective in attacking language models Maus et al. ().

Adversarial Target.
We use the *negative* sentence similarity between the target answer $\text{X}^{tgt}$ and generated answer $\text{X}^{adv}_{\texttt{a}}$ as the adversarial target. Formally, our adversarial target can be denoted as $\mathcal{L}=-{\texttt{Sim}}(\text{X}^{tgt},\text{X}^{adv}_{\texttt{a}})$, where we use CIDEr Vedantam et al. () score as the similarity function.

Figure: Algorithm 1 Adversarial Question Generator (AQG)

Optimization Procedure.
Algorithm 1 details the overall optimization process of AQG, which is self-explanatory.
AQG finds the best adversarial dialogue $\text{X}^{adv}_{\texttt{dialogue}}$ that maximizes target loss $\ell^{tgt}$ by iteratively updating better random noise $\epsilon$. See Appendix C for a more detailed description.

## 4 Adversarial Instruction Tuning

We first provide an input token attention analysis to help understand dialogue hallucination. Based on this, we present a more robust instruction tuning paradigm, *Adversarial Instruction Tuning (AIT)*.

### 4.1 Input Token Attention Analysis

Input feature attention analysis is a popular method to investigate the contribution of input features to model prediction, *e.g.*, GradCAM Selvaraju et al. () for vision models, or token attention map Sundararajan et al. (), Kokhlikyan et al. () for language models.
Here, we introduce a new attention-based metric for LVLM, coined Dialogue Tokens Attention Ratio (DTAR), that helps to analyze the dialogue hallucination in instruction-following LVLMs, by calculating the contribution of preceding dialogues to output answer prediction.

Dialogue Tokens Attention Ratio.
Let $\text{Z}_{\texttt{v}}=W_{\text{Proj}}\cdot f_{\text{VE}}(\text{X}_{\texttt{v}})$ be token embeddings of input image $\text{X}_{\texttt{v}}$, where $W_{\text{Proj}}$ is a linear projection layer that convert the output patches of visual encoder $f_{\text{VE}}$ to input tokens of LLM, and $\text{Z}_{\texttt{dialogue}}=f_{\text{token}}(\text{X}_{\texttt{dialogue}})$ be token embeddings of input preceding dialogue $\text{X}_{\texttt{dialogue}}$. Also, let $P(\text{X}_{\texttt{a}})$ be the probability of output answer tokens.
Then, we define *Dialogue Tokens Attention Ratio (DTAR)* using the *gradient* of input token embeddings $\text{Z}_{\texttt{v}}$ and $\text{Z}_{\texttt{dialogue}}$ with respect to the output token probability $P(\text{X}_{\texttt{a}})$, as in Definition 2.

###### Definition 4.1 .

(Dialogue Tokens Attention Ratio) For each instruction example, DTAR is the ratio of the sum of all absolute attention scores of dialogue tokens over that of all input tokens;

$$ $\vspace{-0.2cm}\sum_{i}\big{[}\sum_{j}|{\partial P(\text{X}_{\texttt{a},i}) \over\partial\text{Z}_{\texttt{dialogue},j}}|/\big{(}\sum_{j}|{\partial P( \text{X}_{\texttt{a},i})\over\partial\text{Z}_{\texttt{dialogue},j}}|+\sum_{k} |{\partial P(\text{X}_{\texttt{a},i})\over\partial\text{Z}_{\texttt{v},k}}| \big{)}\big{]},$ (2) $$

where $X_{\texttt{a},i}$ denotes $i$-th token in the output answer $X_{\texttt{a}}$, $Z_{\texttt{dialogue},j}$ denotes $j$-th token embedding in $Z_{\texttt{dialogue}}$, and $Z_{\texttt{v},k}$ denotes $k$-th token embedding in $Z_{\texttt{v}}$. Intuitively, DTAR means the contribution of preceding dialogues over the input image to output the final answer. ∎

Figure: Table 1: Summary of DTAR scores for correct (non-hallucinated) and hallucinated cases.
[Uncaptioned image]: x4.png

DTAR Analysis for Dialogue Hallucination.
Using LLaVA Liu et al. (), we calculate the DTAR score of hallucinated examples and that of non-hallucinated examples on EvalDial built on ScienceQA dataset.
We select 500 hallucinated examples by preceding adversarial dialogues and 500 non-hallucinated examples, then calculate the DTAR score for each example.
Table 4.1 summarizes the mean and standard deviation of DTAR scores for non-hallucinated and hallucinated cases.
For LLaVA, the DTAR score of hallucinated examples is higher than that of non-hallucinated examples, meaning that LLaVA focuses more on preceding dialogues than image features for the prediction of the hallucinated case.
Similarly, Figure 4 shows the AUC-ROC curves of DTAR score on two cases, hallucinated and non-hallucinated.
The AUC of DTAR score of LLaVA is 0.935, which is high, meaning that LLaVA often relies on hallucinatory dialogue for prediction, thereby causing hallucinations.
Section 4.2 is proposed to suppress this prediction bias toward hallucinatory dialogues.

###### Definition 4.1 .

### 4.2 Adversarial Instruction Tuning (AIT)

To reduce the negative effect of hallucinatory dialogues, we propose AIT to perform instruction tuning on adversarially augmented visual instruction datasets. AIT first generates hallucinatory dialogues and injects them into visual instruction training datasets, and then performs instruction tuning by masking the hallucinatory dialogues in loss calculation.

Hallucinatory Dialogue Generation.
We create hallucinatory dialogues following the protocol of EvalDial in Section 3.2.
Specifically, for each training example of the visual-instruction dataset such as LLaVA-Instruct-665k Liu et al. (), we generate hallucinatory questions $X^{i,adv}_{\texttt{q}}$ in order to hallucinate each round’s question $\text{X}^{i}_{\texttt{q}}$ in the original training example $\text{X}^{<t}_{\texttt{dialogue}}$, and generate its corresponding answers $X^{i,adv}_{\texttt{a}}$ by simply asking the hallucinatory questions to LVLMs.
The hallucinatory question includes all types of dialogues, i.e., General, Random, and Adversarial.
For the training examples with $t$ instruction rounds, we randomly augment $m$ rounds out of $t$ rounds.

Hallucinatory Dialogue Injection.
For each training example, the $i$-th round dialogue $\text{X}^{i}_{\texttt{dialogue}}=(\text{X}^{i}_{\texttt{q}},\text{X}^{i}_{
\texttt{a}})$ can be adversarially augmented by prepending a hallucinatory dialogue $\text{X}^{i,adv}_{\texttt{dialogue}}=(\text{X}^{i,adv}_{\texttt{q}},\text{X}^{
i,adv}_{\texttt{a}})$ as follows:

$$ $\text{X}^{i}_{\texttt{aug}}=(\text{X}^{i,adv}_{\texttt{dialogue}},\text{X}^{i} _{\texttt{dialogue}}).$ (3) $$

That is, if $m=1$ and $i$-th round instruction is chosen to be augmented, then the overall augmented input $\text{X}_{\texttt{aug}}$ for LVLM are formulated as,

$$ $\text{X}_{\texttt{aug}}=(\text{X}_{\texttt{v}},\text{X}^{1}_{\texttt{dialogue} },\cdots,\text{X}^{i}_{\texttt{aug}},\cdots,\text{X}^{t}_{\texttt{dialogue}}).$ (4) $$

Masked Instruction Tuning. As opposed to standard instruction tuning, where LVLM minimizes the cross-entropy loss of answer tokens in all rounds of dialogues, we *mask* answer tokens of hallucinatory dialogues so that they are not factored into calculating the cross-entropy loss.
Therefore, the LVLM is not trained to generate answers in hallucinatory dialogues and can be robustly trained to generate correct answers to subsequent questions despite the presence of preceding hallucinatory dialogues.

## 5 Experiments

Table: Table 2: Zero-shot performance of LVLMs on EvalDial with prepended three types of single-round dialogues, General (Gen), Random (Rand), and Adversarial (Adv). We compare AIT with LLaVA and highlight better performance against dialogue hallucinations in bold. The average relative performance drop ($\%$ Avg Drop) from the None-dialogue case for each LVLM is also presented.

Datasets.
Followed by Section 3.2, we use our proposed benchmark, EvalDial, for evaluating dialogue hallucination. We mainly use EvalDial built on top of OKVQA Marino et al. (), GQA Hudson and Manning (), and IconQA Lu et al. () for VQA task, and NoCaps Agrawal et al. (), Flickr-30K Plummer et al. (), and WHOOPS Bitton-Guetta et al. () for Captioning task.

Algorithms.
We compare the zero-shot performance of AIT with three recently proposed LVLMs: (1) MiniGPT-4 (7B) Zhu et al. (), (2) InstructBLIP (7B) Dai et al. (), and (3) LLaVA-v1.5 (7B) Liu et al. ().

Implementation Details.
AIT uses the same model architecture with LLaVA-v1.5.
For the hyper-parameters of adversarial instruction tuning, we train AIT for 1 epoch with a batch size of 128, and an initial learning rate of 2e-5 with a cosine scheduler.
For hallucinatory dialogue injection, we generate adversarial dialogue examples from LLaVA-Instruct-80k, OKVQA with 9K examples, GQA with 15K examples, IconQA with 29K examples, and 0.5K examples each from NoCaps and Flickr-30K, which are mostly originally included in LLaVA-Instruct-665K.
All methods are implemented with PyTorch 1.8.0 and executed on multiple NVIDIA A100 GPUs. Generating an adversarial dialogue for each image-QA example using AQG takes approximately 50 seconds on a single A100 GPU. By applying quantization, this can be reduced to about 18 seconds. The code is available at [https://github.com/dongmean/LVLM_DialHalu](https://github.com/dongmean/LVLM_DialHalu).

Evaluation.
For VQA task, we use top-1 accuracy that validates whether the ground-truth answer is in the generated sentence. For Image Captioning task, we use CIDEr Vedantam et al. () score, a popular metric to evaluate image captioning quality Xu et al. ().

### 5.1 Main Results on EvalDial

Efficacy of AQG.
Table 2 summarizes the zero-shot performance of LVLMs on EvalDial. Overall, with three types of dialogues prepended, the performance of all existing baselines such as LLaVA, MiniGPT-4, and InstructBLIP are significantly degraded by up to 37.7$\%$ for VQA task and 59.6$\%$ for Captioning task. Among the three types of dialogues, adversarial dialogues generated by AQG show the highest performance drops for every baseline LVLM, which demonstrates the efficacy of AQG.

Efficacy of AIT.
While every baseline LVLM is vulnerable to dialogue hallucinations, AIT maintains the most robust VQA and Captioning performance against dialogue hallucinations.
Numerically, for VQA task with OKVQA, GQA, and IconQA datasets, AIT maintains VQA accuracy within $-$1.0$\%$ to $-$5.4$\%$ drops, while LLaVA drops by $-$2.5$\%$ to $-$13.1$\%$.
Similarly, for Captioning task with Nocaps, Flickr-30K, and WHOOPS datasets, AIT maintains Captioning performance within $-$0.8$\%$ to $-$1.4$\%$ drops, while LLaVA drops by $-$2.4$\%$ to $-$21.0$\%$. Additionally, we evaluated using FAITHSCORE Jing et al. () on the Captioning task, and the results are shown in Appendix D.

### 5.2 In-depth Analysis of AQG

Superiority of AQG over Possible Attacking Methods.
We compare the effectiveness of AQG in attacking subsequent target questions, i.e., triggering dialogue hallucinations. Since there is no comparative attacking approach fit to the dialogue hallucination problem for LVLM, we adopt *one* GPT-prompting based attacking approach, and *four* text-based attacking methods adopted from Harmbench Mantas et al. (), a hallucination generation framework for LLM. For the GPT-prompting attack, we generate adversarial questions by carefully asking GPT to generate hallucinatory questions against the given target questions if prepended (See Appendix E for prompting details). For text-based attacking methods, we use 4 different attacking variants in Harmbench, such as GCG, GCG-Multi ($\text{GCG}^{m}$), PAIR, and TAP.

Table 5.2 shows the performance of AQG compared to other attacking methods. Overall, AQG is the most effective in attacking the original test answer in three VQA datasets, including GQA, OKVQA, and IconQA. In detail, although we carefully ask GPT-4 to generate adversarial questions that can cause hallucination in the subsequent target questions, the generated questions do not degrade the performance of LLaVA; the performance is similar to the None case, where no dialogues are prepended. Also, other text-based attacking methods, from GCG to TAP, are not effective in attacking and sometimes fail to induce the hallucinations. However, AQG is consistently effective in attacking the original questions, showing the necessity of our optimization-based attack by understanding multi-modal semantics.

Figure: Table 3: Comparison of AQG with different attacking methods. The lower, the more effective in attacking.
[Uncaptioned image]: x5.png

Figure: Figure 6: Embedding distribution analysis for three types of dialogues. (a) illustrates the TSNE plot of each dialogue example with its corresponding target image, question, and answer on the joint text-image embedding space of InstructBLIP. (b) shows the density plot of three types of dialogue in terms of the average cosine similarity to the corresponding target image, question, and answer.
Refer to caption: x6.png

Effect of Optimization Steps in AQG.
Figure 5 shows the effect of optimization steps in AQG to attack LLaVA’s captioning performance on Flickr dataset. With more optimization steps in Figure 5, AQG generates more effective adversarial dialogues with higher target loss, thereby successfully attacking the original test captioning performance. This indicates the adversarial objective and optimization process in is well-designed and appropriate to generate better adversarial dialogues in multi-modal semantics.

Distributional Analysis of Avdersarial Examples Generated by AQG. Figure 6 shows the distributions of three types of dialogues, including the adversarial dialogues generated by AQG, with two types of plots: (1) TSNE plot; and (2) density plot.
In the TSNE plot, we visualize all the embeddings of the target image, question, and answer with General, Random, and Adversarial dialogues on a joint embedding space of InstructBLIP. Overall, the General (green) dialogues tend to be located far from the target image, question, and answer (black, grey, and light blue, respectively), while the Random (blue) and Adversarial (red) dialogues appear to be located closer to the target image, question, and answer.
For a more detailed sample-level analysis, in the density plot, we measure the average cosine similarity from each dialogue sample to its corresponding target image, question, and answer, and plot the density. The Adversarial dialogues have the highest average similarity with their corresponding images, questions, and answers, indicating dialogues that are semantically relevant to the Image, Question, and Answer together cause higher levels of hallucination.

**Table 4: Effect of applying masked instruction tuning during AIT on IconQA dataset.**
| Model | GQA | IconQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| None​ | ​Gen​ | ​Rand​ | ​Adv​ | ​None​ | ​Gen​ | ​Rand​ | ​Adv |  |
| ​​​$\text{{AIT}{}}_{m=1}$​​​ | ​​​44.2​​​ | ​​​38.8​​​ | ​​​38.8​​​ | ​​​35.6​​​ | ​​​55.0​​​ | ​​​46.4​​​ | ​​​47.2​​​ | ​​​40.4​ |
| ​​​$\text{{AIT}{}}_{m=2}$​​​ | 44.4​​​ | ​​​39.2​​​ | ​​​36.6​​​ | ​​​35.8​​​ | ​​​61.0​​​ | ​​​44.0​​​ | ​​​47.8​​​ | ​​​41.6​ |
| ​​​$\text{{AIT}{}}_{m=all}$​​​ | ​​​45.6​​​ | ​​​39.6​​​ | ​​​38.9​​​ | ​​​36.8​​​ | ​​​68.6​​​ | ​​​47.6​​​ | ​​​51.4​​​ | ​​​50.0​ |

### 5.3 Ablation Studies of AIT

Effect of Loss Masking on Prepended Adversarial Dialogues.
Table 5 shows the effect of the loss masking on prepended adversarial dialogues in masked instruction tuning. Without loss masking, AIT shows unsatisfactory performance in mitigating dialogue hallucination. This is because the fine-tuned LVLM without loss masking is forced to generate the answer even for injected random or adversarial dialogues, which are out-of-context from the given image or harmful in maintaining the context of the given image, resulting in more hallucinations in later rounds of chats.

Effect of The Number of Injected Hallucinatory Dialogues.
As elaborated in Section 4.2, AIT randomly chooses $m$ rounds for each training example in the visual-instruction dataset to inject hallucinatory dialogues.
Here, we investigate the effect of the number $m$ of injected hallucinatory dialogues. We set $m$ to be 1,2 and all from available dialogues per example since most examples contain 4 rounds of dialogues on average.
To control the effect of data size in the study, we only use LLaVA-Instruct-150K for fine-tuning.
As in Table 5, with more injected adversarial examples used during AIT, the model gets more robust to the adversarial attack.
Therefore, the more hallucinatory dialogues injected into adversarial instruction tuning, the more performance gain we can have.

More studies of the object hallucination baseline Liu et al. () on EvalDial are in Appendix F.

**Table 6: Effect of multi-round prepended dialogues on LVLMs using GQA dataset.**
| Round | LLaVA-v1.5 | AIT |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Gen | Rand | Adv | Gen | Ran | Adv |  |
| 1 | 55.4 | 57.0 | 49.0 | 56.0 | 55.4 | 55.6 |
| 2 | 52.4 | 54.0 | 48.8 | 53.4 | 55.4 | 55.0 |
| 4 | 53.4 | 53.2 | 48.5 | 52.6 | 55.5 | 55.2 |

### 5.4 Results on Multi-round EvalDial

As users usually interact with LVLMs via multi-round dialogues, it is essential to explore the effectiveness of LVLMs in multi-round cases. For General and Random type attacks, we randomly sample one question for each round and prepend them before the target question. For the Adversarial type attack, we generated different adversarial questions by using AQG based on the target question-answer pair for each round. Table 7 shows the results with 1,2, and 4 rounds of prepended dialogues during evaluation on the LLaVA baseline and AIT model.
The results indicate that more prepended questions make the model less likely to answer the target question correctly. However, AIT shows a more robust performance than the baseline on different types of prepended dialogues.
Specifically, as the round of prepended dialogue increases, LLaVA shows a consistent performance drop for all three types of dialogues.
Meanwhile, AIT successfully maintains the performance in General and Random types and shows less performance drop in Adversarial cases.

### 5.5 Sanity Check Experiments: Effect of Prompt Length to Hallucination

LLMs tend to degrade their performance as the prompt length increases Liu et al. (). To check whether this happens in our experiments, we additionally conduct a sanity check experiment. In Table 7, we repeated each test question in EvalDial multiple times (N) to the LVLMs, and measured the accuracy of the last question. Overall, prompt length (# repeats) has minimal impact on LVLM performance, indicating that content is more critical than length.

Figure: (a) VQA example from GQA dataset.
Refer to caption: x7.png

### 5.6 Visualization

Figure 7 visualizes two LVLM prediction examples with adversarial dialogues generated by AQG for VQA and Captioning task, respectively.
Overall, while LLaVA-v1.5 hallucinates answers by preceding adversarial dialogues, AIT can generate correct answers without hallucinations.
For example, as illustrated in Fig 7(a), as the preceding adversarial dialogue contains the word “red” (in yellow box), LLaVA unfaithfully answers “*the color of the person’s hair*” as “*red*” in later chats (colored in red in the green box).
Similarly, in Fig 7(b), the word “*candy*” in the adversarial preceding dialogue (in yellow box) hinders LLaVA from describing the image with the word “*lollipop*” in later chats (marked in red in the green box), which shows the LLaVA’s weakness to the dialogue hallucination.
On the other hand, although the same adversarial dialogue is prepended, AIT generates the correct answer or description of the image without hallucinations, by leveraging the power of robust fine-tuning against augmented adversarial dialogues. More visualization examples can be found in Appendix G.

## 6 Conclusion

In this work, we find that the existing instruction-following LVLMs are prone to be hallucinated by preceding user-system dialogues.
To precisely validate this dialogue hallucination, we construct EvalDial, a large and diverse evaluation benchmark covering popular multi-modal datasets in VQA and captioning tasks, with a novel adversarial dialogue generator AQG.
In addition, to mitigate such hallucination, we provide an in-depth analysis to help understand why such hallucination happens with input token attention analysis, and then propose AIT, a robust instruction-tuning method that maintains or even improves the zero-shot VQA and captioning performance of LVLMs in the presence of hallucinatory dialogues.
We believe that our work can shed light on many applications requiring robust LVLMs such as the Red-teaming of visual-language assistants.

## 7 Reproducibility Statement

For reproducibility, we elaborate on the detailed generation process of our benchmark in Section 3.2, and its examples in Appendix A$\&$B.
The overall process of the proposed algorithms is explained in Section 4, and more detailed algorithm pseudocode is in Section C.
Implementation details and hardware configurations are detailed in Section 5.
We release our code at [https://anonymous.4open.science/r/LMM_hallucination-D52E/](https://anonymous.4open.science/r/LMM_hallucination-D52E/).

## Appendix A General Questions Generated from GPT-3

In Table 8, we show 10 questions generated from GPT-3 using the prompt "Generate 10 general questions for a random image". All these general questions could be asked to any image.

**Table 8: 10 general questions generated from GPT-3.**
| Prompt: "Generate 10 general questions for a random image" |
| --- |
| 1. "What is the geographical location depicted in the image" |
| 2. "Are there any identifiable landmarks or recognizable features in the image?" |
| 3. "What is the dominant color in the image?" |
| 4. "Are there any notable patterns or textures in the image?" |
| 5. "What is the source of light in the image (e.g., natural sunlight, artificial lighting)?" |
| 6. "Does the image evoke a sense of motion or stillness?" |
| 7. "What is the overall mood or atmosphere conveyed by the image?" |
| 8. "How does the image make you feel or what emotions does it elicit?" |
| 9. "What is the primary subject of the image?" |
| 10. What is the main point of focus or point of interest in the image?" |

## Appendix B Adversarial Questions Generated by AQG

In Figure 8 and Figure 9, we showed two examples of AQG-generated human-readable adversarial questions on VQA and Captioning task. Each question-related image and answer is also shown.

Figure: Figure 8: Adversarial dialogue example for VQA task.
Refer to caption: x9.png

Figure: Figure 9: Adversarial dialogue example for Captioning task.
Refer to caption: x10.png

## Appendix C Detailed Explanation of the Optimization Procedure of AQG

**Table 9: Initial context prompt example of AQG.**
| Template: Generate an image-related question regarding |
| --- |
| Context: small objects, background details, expected places, landmarks, related history, painting style, colors, and foods. |

Context Prompt Initialization. Table 9 shows the initialization of the context prompt we used for AQG. During optimization, only the context part is updated with the Gaussian random noise $\epsilon$ at the token level iteratively.

Algorithm Explanation.
Algorithm 1 details the overall optimization process of AQG, which is self-explanatory.
To find the best adversarial dialogue $\text{X}^{adv}_{\texttt{dialogue}}$ with higher target loss $\ell^{tgt}$, AQG starts with an initial prompt $\text{X}^{init}_{\texttt{prompt}}$ and proceeds the black-box optimization steps until round $r$ (in Lines 1–2).
In each optimization step, it samples the gaussian noise $\epsilon$ and the noise injected tokens $\text{Z}_{\texttt{prompt}}=[\text{Z}_{\texttt{template}};\text{Z}_{\texttt{
context}}+\epsilon]$ is fed into the adversarial LVLM, generating the adversarial dialogue $\text{X}^{adv}_{\texttt{dialogue}}$ (in Lines 3–4).
Next, the generated adversarial dialogue $X^{adv}_{\texttt{dialogue}}$ is fed into the original LVLM to hallucinate the answer $X^{adv}_{\texttt{a}}$ (in Line 5).
With the generated answer $\text{X}^{adv}_{\texttt{a}}$, we confirm to update the input tokens $\text{Z}^{prompt}\leftarrow{}\text{Z}^{prompt}+\epsilon$ only if the adversarial target is increased, otherwise we maintain it as $\text{Z}^{prompt}\leftarrow{}\text{Z}^{prompt}$ (in Lines 6–9).
After repeating $r$ rounds of optimization, AQG returns the best adversarial dialogue $\text{X}^{adv}_{\texttt{dialogue}}$. Note that, AQG attacks the input prompt without calculating any gradient in a black-box optimization manner.

## Appendix D Additional evaluation metrics on Captioning

**Table 10: Additional results with FAITHSCORE on Captioning task. The higher, the better.**
| Dialogue Type | LLaVA-v1.5 | AIT |  |  |
| --- | --- | --- | --- | --- |
| None | Adversarial | None | Adversarial |  |
| NoCaps | 0.89 | 0.75 | 0.89 | 0.87 |
| Flickr-30k | 0.91 | 0.80 | 0.91 | 0.90 |
| WHOOPS | 0.86 | 0.72 | 0.87 | 0.86 |

We conducted additional experiments on Captioning tasks with FAITHSCORE, a more complex metric and closer alignment with human semantic understanding Jing et al. (). The results evaluated on LLaVA and AIT are reported in Table 10. AIT is more effective in maintaining the FAITHSCORE than LLaVA, resonating with our main results.

## Appendix E Adversarial Questions Generated by Prompting GPT-4

GPT-4 prompt.
We prompt GPT-4 to generate some adversarial dialogues as a simple baseline to our proposed AQG. Specifically, the prompt we used for VQA task is:
Prompt: Generate an image-related question that a user might ask and answer. This QA pair should be able to hallucinate a large visual-language model when prompting with this question  (a-VQA-question) after prompting with the preceding generated question. Don’t repeat the question. The ability to hallucinate a large visual-language model is very important here. Format the question-answer pair in this way (Que:QUESTION Ans:ANSWER END)

We change the (a-VQA-question) to "*What is described in the image?*" for the Captioning task and keep the rest of the prompt the same. We show two examples of generated questions in Figure 10 for VQA and Captioning tasks. Though GPT-4 can generate more natural-sounding questions, it is hard to effectively hallucinate the large visual language models. We specifically test the GPT-4 generated adversarial questions on all datasets in EvalDial to compare with AQG quantitatively. We evaluate LLaVA-v1.5 and the result in Table 11 shows that the model does not hallucinate much and could achieve higher accuracy when prepended with GPT-4 generated adversarial questions.

Figure: Figure 10: Examples of generated adversarial questions using GPT-4 and AQG on VQA and Captioning tasks.
Refer to caption: x11.png

**Table 11: Effect of GPT4-generated adversarial dialogues to hallucinate LLaVA.**
| Model | OKVQA | GQA | IconQA | NoCaps | Flickr | WHOOPS |
| --- | --- | --- | --- | --- | --- | --- |
| GPT4 | 54.4 | 54.8 | 46.6 | 41.0 | 28.3 | 42.3 |
| AQG | 48.4 | 49.0 | 41.2 | 35.8 | 19.9 | 34.8 |

## Appendix F Result of An Object Hallucination Baseline Liu et al. ( 2023a ) on EvalDial

Because of the severe impact of hallucination on large visual language models, many mitigation methods have been proposed. We use Liu et al. () as a baseline and evaluated on WHOOPs and IconQA datasets, and the result is shown in Table 12. We chose the LRV-Instruction v1 as it uses MiniGPT-4 as its backbone. Even though Liu et al. () used a similar idea in fine-tuning as ours, the performance is not better or sometimes even worse than MiniGPT-4 due to different types of hallucinations. Their focus is on object hallucination, while our finding on dialogue hallucination could still confuse the large visual language model.

**Table 12: Performance of LRV-Instruction-v1 Liu et al. () on IconQA and WHOOPs.**
| Dataset | None | General | Random | Adversarial |
| --- | --- | --- | --- | --- |
| IconQA | 40.6 | 29.6 | 25.6 | 25.2 |
| WHOOPs | 33.1 | 36.4 | 26.8 | 16.2 |

## Appendix G More Visualizations

Figure: (a) VQA example from OKVQA dataset.
Refer to caption: x12.png

Figure 11 shows additional visualization examples of dialogue hallucinations from NoCaps and OKVQA datasets, showing the effectiveness of our AIT compared to LLaVA. We highlighted the hallucinated part in red.

**Table 13: Effect of adding non-adversarial data during AIT. Using a base AIT model, we include additional data from LLaVA-Instruct-665K that does not have adversarial prepended dialogues and evaluated on IconQA.**
| Model | None | General | Random | Adversarial |
| --- | --- | --- | --- | --- |
| AIT | 45.8 | 34.4 | 44.6 | 41.4 |
| +50K | 47.2 | 47.0 | 46.8 | 45.2 |
| +100K | 45.8 | 47.4 | 46.6 | 45.6 |
| +150K | 47.2 | 49.0 | 48.0 | 47.8 |
| +200K | 49.8 | 44.6 | 48.0 | 46.4 |

## Appendix H Effect of Data Size for Non-Adversarial Examples used during AIT

Since most training data do not contain adversarial dialogues, we explore the effect by directly including more non-adversarial prepended dialogue data from LLaVA-Instruct-665K during AIT. We add additional 50K, 100K, 150K, and 200K training data, and the result evaluated on IconQA is shown in Table 13. The performance increases with more non-adversarial examples but too much non-adversarial data also introduces noise, leading to performance fluctuation. With this ablation study, we believe including partial non-adversarial examples during the fine-tuning stage of AIT would help improve the performance.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Does AQG-generated data somehow reflect the real distribution of LVLM's hallucination behaviors? Or is it just a way to create fake adversarial dialogue that serves AIT's finetuning purpose?
soundness: 3
strengths: - The AQG is a reasonable and scalable algorithm for data generation. 
- The experiment design is comprehensive, and the results show the AIT approach effectively mitigates dialogue hallucination.
- There is a potential that both AQG algorithm and AIT strategy can be extended to more tasks and modalities.
summary: This paper addresses the issue of dialogue hallucination in LVLMs, where preceding dialogues can lead to unreliable outputs. 

It introduces a benchmark dataset, EvalDial, using the Adversarial Question Generator from VQA and image captioning datasets. Then, it proposes the Adversarial Instruction Tuning method, which fine-tunes LVLMs against hallucinatory dialogues.

The results suggest most of the well-performing LVLM failed on the EvalDial benchmark, in which AIT successfully reduced dialogue hallucination while maintaining performance.

Overall, this work aims at an important research problem, presents a reasonable and effective approach, and contains insightful discussions. The implementation is reproducible.
weaknesses: An important contribution of the work is to propose AIT as a method to mitigate dialogue hallucination. However, the evaluation only focuses on the EvalDial benchmark, which includes synthetic dialogue augmented from non-dialogue data. This is insufficient to justify AIT's real-world value in countering the visual dialogue hallucination problem.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: (1) The adversarial dialogue sequences used for prepending contain intentionally misleading answers, which likely disrupt baseline models due to their lack of exposure to such cases. Would AIT still demonstrate superior performance if prepend responses were correct, non-hallucinatory answers?

(2) Does the Masked Instruction Tuning approach risk overly discouraging reference to preceding dialogues? The relatively stable performance in Table 6, even as dialogue rounds increase, suggests this possibility. Could this approach prevent the model from effectively using relevant context from previous interactions, which is crucial in real dialogue scenarios where preceding exchanges often contain meaningful information?

(3) It would be valuable to understand how the model’s performance in captioning tasks changes as the length of dialogue rounds increases, particularly for AIT in handling longer, potentially more complex dialogue contexts.
soundness: 3
strengths: The introduction of EvalDial and AQG provides a structured and novel approach to testing model robustness against adversarial dialogue contexts, successfully inducing dialogue hallucinations across multiple models and confirming AQG's effectiveness. Notably, AIT demonstrates superior performance over other models in handling adversarial rounds, highlighting its potential as a reliable technique for mitigating dialogue-induced biases in broader LVLM applications where robustness is essential.
summary: This paper tackles the problem of hallucinations in Large Vision-Language Models (LVLMs), particularly how biases from prior dialogue contexts can lead to incorrect or "hallucinated" responses. To measure and address this, the authors introduce EvalDial, a benchmark created by adding adversarial dialogue sequences to existing multimodal datasets, using a novel Adversarial Question Generator (AQG). Their findings show that these adversarial dialogues significantly impact model accuracy. As a solution, they propose Adversarial Instruction Tuning (AIT), which fine-tunes LVLMs by masking dialogue tokens to reduce the bias from previous conversations. Experimental results demonstrate that AIT successfully minimizes hallucinations across multiple tasks while preserving overall model performance.
weaknesses: (1) The paper’s focus on hallucinated dialogue scenarios limits the ability to gauge the model’s general performance in non-hallucinated contexts. A broader evaluation that includes non-adversarial scenarios would provide a more comprehensive view of the model’s robustness and applicability.

(2) The study relies heavily on the CIDEr metric for evaluating image captioning, limiting insights into overall performance. Results using additional metrics such as BLEU, ROUGE, METEOR, or GPT-aided evaluation (accuracy) would provide a more balanced view of the model’s capabilities.

(3) While the Adversarial Dialogue is designed to be image-related and likely to induce hallucination in response to the original test question, it introduces content that is largely independent of the question itself. This raises the question: if the adversarial content is unrelated to the core test question, would simply resetting the chat between exchanges be a more straightforward solution?

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Regarding the impact of prepended dialogues: Is the hallucination caused by an incorrect prediction made by the LVLM when the user asks a prepended question, which then leads to hallucinations in subsequent questions? Or is the hallucination triggered because the user's prepended question conflicts with the current question (i.e., unrelated to the answer)? Could you provide a detailed analysis of how the information in the preceding dialogues leads to hallucinations in the model?

2. In the benchmark, is a hallucinated prepended dialogue added before each question? If multiple rounds of dialogue are added before the current question, and the hallucinated dialogues are positioned differently within these rounds (e.g., at the beginning or just before the current question), or if the number of hallucinated dialogues varies, how would the model's performance change?

3. It is suggested to add a statistical table to analysis the benchmark.

4. In Table 1, the mean DTAR values for hallucinated cases are 0.37 or 0.25, which are not particularly high. How does this prove that the hallucinations are caused by prepended dialogues? Furthermore, should the impact of the distance of the prepended dialogues from the current answer (whether closer or farther) also be considered in evaluating the influence on the answer?
soundness: 3
strengths: 1. Benchmark on which the performance of existing LVLMs significantly declines. 
2. Adversarial Instruction Tuning method, which has been shown through experiments to reduce the occurrence of hallucinations effectively.
summary: This paper discovers that LVLMs can exhibit hallucination phenomena during user-system dialogues and presents a EvalDial benchmark for validation. Experimental results indicate that the performance of existing LVLMs significantly declines on this hallucination dataset. Furthermore, to mitigate the hallucination issue, the paper proposes an Adversarial Instruction Tuning method, which has been shown through experiments to reduce the occurrence of hallucinations effectively.
weaknesses: Insufficient in-depth analysis of the experiment resulted in the inability to determine whether the dataset was meaningful. The benchmark data analysis is also insufficient. If the following issues in the QUESTIONS are solved, we will consider modifying the score.

1. Regarding the impact of prepended dialogues: Is the hallucination caused by an incorrect prediction made by the LVLM when the user asks a prepended question, which then leads to hallucinations in subsequent questions? Or is the hallucination triggered because the user's prepended question conflicts with the current question (i.e., unrelated to the answer)? Could you provide a detailed analysis of how the information in the preceding dialogues leads to hallucinations in the model?  It is essential to understand the mechanisms behind the mentioned hallucinations in LVLMs. Specifically, there are several following possible assumptions. (1)If the LVLM makes an incorrect prediction when interpreting a prepended question, this could indeed set off a chain reaction of hallucinations. The model might latch onto incorrect information, which it then carries forward into subsequent interactions. This could happen if the model's prediction is misled by the phrasing or context of the prepended question, causing it to deviate from the correct path of reasoning. Hallucinations occur when the model generates responses that are not grounded in the actual experiments. (2)Another hypothesis is that the model can not handle the long text of the prepended question. (3)A prepended question that is unrelated to the current question can also cause hallucinations. This is because the LVLM might struggle to recognize different segments of the dialogue. When the model encounters a question that seems to diverge significantly from the previous context, it may attempt to reconcile the two by creating a fabricated connection. This process can lead to the generation of wrong responses.
You may add specific experiments. For example, you could compare hallucination rates when prepended dialogues contain (1) incorrect information vs (2) unrelated information vs (3) correct, long, and related information.

2. In the benchmark, is a hallucinated prepended dialogue added before each question? If multiple rounds of dialogue are added before the current question, and the hallucinated dialogues are positioned differently within these rounds (e.g., at the beginning or just before the current question), or if the number of hallucinated dialogues varies, how would the model's performance change? You conduct ablation studies with different numbers of hallucinated dialogues and positions.

3. It is suggested to add a statistical table to analyze the benchmark, such as the distribution of dialogue types, and average dialogue length.

4. In Table 1, the mean DTAR values for hallucinated cases are 0.37 or 0.25, which are not particularly high. How does this prove that the hallucinations are caused by prepended dialogues? Furthermore, should the impact of the distance of the prepended dialogues from the current answer (whether closer or farther) also be considered in evaluating the influence on the answer? You may consider significance tests comparing hallucinated vs non-hallucinated cases. At the same time, it is suggested that the author test whether the DTAR value is affected by the relative position of the prepended hallucination dialogue and the current question.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: None
soundness: 2
strengths: 1. The paper presents a novel benchmark that specifically evaluates vision-language models when conversations are corrupted by adversarial questions from previous rounds. This benchmark is powered by the Adversarial Question Generator (AQG), which generates adversarial questions to test model robustness. 

2. The paper proposes a solution, Adversarial Instruction Tuning (AIT), to mitigate the performance drop in VLMs caused by these adversarial dialogues, which is an important contribution to improving the robustness of these models.
summary: The paper introduces a new benchmark designed to evaluate vision-language models (VLM)  when dialogues are corrupted by adversarial questions. While this aims to address dialogue hallucination, such issues appear rare in real-world applications. Additionally, the benchmark is not tested on models known to hallucinate less, which raises concerns about its general applicability. There are also some contradictory statements about hallucinations and examples provided.
weaknesses: 1. The real-world applicability of adversarial questions, as demonstrated in the examples provided, seems limited. The likelihood of such adversarial scenarios occurring in practical use cases appears low, and hence, this might not be a widespread issue. 

2. The hallucinated dialogues presented in Figure 2 may indicate that the underlying model itself is not robust enough. Many recent works, such as ShareGPT4V, have tackled hallucination issues with better training data. The paper does not evaluate its methods on models that are known to hallucinate less, leaving the effectiveness of the proposed method for stronger models unclear. 

3. Section 3.1: Even by the paper’s own definition of hallucination (line 155: “the generated content that is nonsensical or unfaithful to the given source content”), the examples provided don’t clearly qualify as hallucinations. For instance, the responses in Figure 1 and Figure 2 make sense given the incorrect text from the previous dialogue. For example, in Figure 2 (mid-column), the answer "Antarctica" makes sense in relation to the mention of penguins from the prior round.

Meta-review: ### Summary:
This paper proposes a method to mitigate dialogue hallucination in Large Vision Language Models (LVLMs) through:
1. A benchmark called EvalDial that extends existing datasets with adversarial dialogues
2. An Adversarial Instruction Tuning (AIT) approach to make LVLMs more robust against hallucination
3. Experimental results showing AIT can reduce hallucination while maintaining performance

### Strengths:
- Novel benchmark creation: 
> "The paper presents a novel benchmark that specifically evaluates vision-language models when conversations are corrupted by adversarial questions from previous rounds. This benchmark is powered by the Adversarial Question Generator (AQG), which generates adversarial questions to test model robustness."

- Initial empirical validation:
> "The experiment design is comprehensive, and the results show the AIT approach effectively mitigates dialogue hallucination."

### Weaknesses:
- Limited real-world applicability:
> "The real-world applicability of adversarial questions, as demonstrated in the examples provided, seems limited. The likelihood of such adversarial scenarios occurring in practical use cases appears low, and hence, this might not be a widespread issue."

- Insufficient evaluation scope:
> "The paper's focus on hallucinated dialogue scenarios limits the ability to gauge the model's general performance in non-hallucinated contexts. A broader evaluation that includes non-adversarial scenarios would provide a more comprehensive view of the model's robustness and applicability."

- Methodological concerns:
> "Many recent works, such as ShareGPT4V, have tackled hallucination issues with better training data. The paper does not evaluate its methods on models that are known to hallucinate less, leaving the effectiveness of the proposed method for stronger models unclear."


Justification: While the paper presents an interesting direction for addressing dialogue hallucination in LVLMs, the weaknesses in real-world applicability, evaluation methodology, and comparison with state-of-the-art solutions outweigh its contributions. The authors' responses during discussion, while thorough, did not sufficiently address these fundamental concerns. I recommend rejection with encouragement to address these issues for future submission.

Decision: Reject