PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training

Authors: Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, Chunhua Shen
Primary area: foundation or frontier models, including LLMs
Keywords: multi-modal large language models, hallucinations mitigation, hallucinations evaluation, language model priors

Abstract:
This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs)  particularly for dense image captioning tasks. To tackle the challenge, we identify the current lack of a metric that finely measures the caption quality in concept level. We hereby introduce HalFscore, a novel metric built upon the language graph and is designed to evaluate both the  accuracy and completeness of dense captions at a
granular level. Additionally, we identify the root cause of hallucination as the model's over-reliance on its language prior. To address this, we propose PerturboLLaVA, which reduces the model's reliance on the language prior by incorporating adversarially perturbed text during training. This method enhances the model's focus on visual inputs, effectively reducing hallucinations and producing accurate, image-grounded descriptions without incurring additional computational overhead.  PerturboLLaVA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations and achieving improved performance across general multimodal benchmarks.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1729674718154

thread_type:
official_review

<thread_object>
creator: reviewer_3zd9

<content>
summary:
The paper has two main contributions: (1) Introducing an evaluation pipeline for dense image captioning, HalFscore, that measures the quality of dense caption more holistically by constructing a language graph and computing the precision and recall. (2) Introducing an adversarial augmentation method during training that reduces the reliance on language prior, which reduces hallucination.

strengths:
1. This paper addresses a significant gap in the current evaluation of MLLMs
2. The augmentation method is intuitive, and shows nice improvements in the benchmarks checked
3. The authors conduct extensive experiments to demonstrate the effectiveness of the augmentation method

weaknesses:
1. Regarding the human survey, it is unclear to me who the human raters are (the authors, Amazon Mechanical Turk Workers, etc.)
2. Since all the experiments are done on LLaVA-1.5, it is unclear what the effect of the perturbation method on stronger models

questions:
If the annotators are from Amazon Mechanical Turk, how were they recruited? 
If the annotators reported in the paper are the authors, could you also add a comparison with external annotators?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650433758

thread_type:
official_review

<thread_object>
creator: reviewer_bhz8

<content>
summary:
This paper aims to address the issue of hallucination in current multimodal large language models applied to dense image captioning tasks. The authors argue that there is no appropriate metric for evaluating the hallucinations in this context. To address this gap, they introduce HalFscore, a metric based on a language graph that assesses both the accuracy and completeness of dense captions. Additionally, they propose the incorporation of perturbed text to alleviate the over-reliance of language-prior issue, thereby mitigating the hallucination issue. Experimental results on several general multimodal benchmarks demonstrate the effectiveness of their approach.

strengths:
1. The proposed PerturboLLaVA is simple, interesting and effective. By modifying the original training data without employing additional training techniques or supplementary data, this method effectively addresses the hallucination issue when compared to the baseline LLaVA-1.5 (Table 2 & Table 3).

weaknesses:
1. Defination of dense captioning task is unclear: 
As noted in Lines 012 and 038, this paper focuses on addressing the hallucination problem in dense captioning tasks. The authors reference [1] and [2], both of them are related to dense captioning tasks. However, the definitions of dense captioning tasks in [1] and [2] differ from those presented in this paper. Additionally, this paper does not conduct standard evaluations for these tasks as outlined in [1] and [2]. It remains unclear whether this discrepancy constitutes a citation issue, and the authors should provide clarification on this matter.

[1] DenseCap: Fully Convolutional Localization Networks for Dense Captioning, 2016.
[2] Context and attribute grounded dense captioning, 2019.

2. No comparison between HalFscore and existing methods:
The authors solely rely on user studies to justify the effectiveness of the proposed metric. However, they do not provide a comparison with existing metrics. One commonly used baseline is to using gpt-score as evaluation metric (e.g., [3])

[3] LLaVA-RLHF: Aligning Large Multimodal Models with Factually Augmented RLHF, 2023.


3. Additional training cost compare with existing methods.

The methods need to re-train the whole model and cost more training time (the training samples are longer).

4. No evaluation on stronger models.

The paper conducts experiments solely on LLaVA-1.5, which is publicly available as of 2023. More recent and stronger models, such as LLaVA-Next, Intern VL-1.5, and LLaVA-UHD, are not evaluated. It is recommended that the authors include at least one of these recent models to demonstrate the effectiveness of their methods.

questions:
Please refer to weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703588359

thread_type:
official_review

<thread_object>
creator: reviewer_vmbr

<content>
summary:
This paper proposes an evaluation metric called HalFscore to measure the fine-grained image captioning quality in terms of object, attribute, and relation hallucinations. The authors also propose a method to reduce hallucinations by injecting generated perturbation text into training prompts and show that it helps reduce the hallucination compared to several baselines.

strengths:
1. The proposed hallucination metric HalFscore is straightforward but helpful in detecting fine-grained hallucination in terms of objects, attributes, and relations.
2. Based on perturbation text, the proposed training framework PerturboLLaVA forces the model to balance the image content and language prior, avoiding generating hallucinated text in fine-grained captions.

weaknesses:
Both HalFscore and PertuboLLaVA are pretty straightforward. PerturboLLaVA is a simple prompting technique to revise the text prompts for model finetuning. The results are mixed, and there are still performance gaps compared to recent models based on advanced language models like Qwen2 and LLama3.

questions:
Questions:
Can you provide more details on the inconsistent results of RLAI-F in Table 3 between Object HalBench (CHAIRs and CHAIRi) and HalBench?
Comments:
Lines 430, 463: The results in "Table 2"... --> Should be "Table 3"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712097905

thread_type:
official_review

<thread_object>
creator: reviewer_nju1

<content>
summary:
This paper studies hallucination problem in MLLM, especially on dense captioning task. The authors first propose a new metric that can capture both precision recall based on scene graph parsed by a GPT model. The author then propose a training method PerturboLLava which can reduce the model’s reliance on language prior so that the model can focus more on the visual inputs thus reducing hallucination. The authors show the effectiveness of the proposed method on a selected 1k images from DCI dataset.

strengths:
Nice ablation on different version of training templates as well as different relevance levels of perturbation.
The proposed method can be combined with other decoding-based methods to further improve performance.
The authors introduce a new well motivated metric and show that it correlates more strongly with human judgment.
The method is conceptually simple and effective, requiring no additional inference cost. It not only enhances dense captioning but also benefits general multimodal capabilities. Although the authors primarily focus on the dense captioning task, the method is not designed with no specificity on captioning but generally aims to help the model focus on image input over language priors.

weaknesses:
The proposed metric is very similar to SPICE(SPICE: Semantic Propositional Image Caption Evaluation) however the authors do not mention this related work. The main difference is SPICE is using a traditional parser but this paper uses GPT. I hope the authors can add the comparison against SPICE.

I think figure 1 is not super accurate. The authors do use extra data (although augmented from existing data), and also the longer training examples will lead to extra training cost as well.

Having Section 4.2 is good, but I have some concern. I do think equation 5-10 holds, but I think the explanation following the equations is problematic. “During the training of multimodal models, we can assume that the world knowledge embedded in the language model remains unchanged, so p(xk|xp<k) is a perturbation term that cannot be optimized. Therefore, in the training process of multimodal models, our perturbation training method guides the model to optimize towards p(xk|x−p<k,I), transforming it into a fully multimodal model that is unaffected by language priors and relies entirely on image information to answer questions.”
- The statement here holds no matter with or without perturbation.
- My own explanation is, for cross entropy loss, if the model already predicts well, the gradient will be low so the model wouldn’t pay much attention to this token. Without the perturbation, the prior term can already predicts well, so the gradient is low. With the perturbation, the prior term is perturbed to be low, so the gradient is high for the unprior term.
- Essentially originally, for a caption, the tokens that can be inferred from language prior will have less effect on parameter gradients whereas now the gradient is kind of balanced across tokens.

questions:
- In table4,5 the authors show different variants of perturbative visual training. I am wondering if the authors try combining these different versions.
- The training and inference has some discrepancy where training has the perturbation as input but inference does not. Why this kind of discrepancy is not an issue?
- The metric highly rely on the scene graph representation. What are the limitations of this representation?
- Is there analysis on how good the gpt4 model perform perturbation task?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730788353763

thread_type:
official_review

<thread_object>
creator: reviewer_rva9

<content>
summary:
This paper pointed out the hallucination issue in MLLM, where pretrained linguistic knowledge overshadow the visual information. Therefore, this paper proposes a training strategy by augment the text into the one that conflict with the visual content but still aligns with general knowledge. To quantitize the hallucination in fine-grained fashion, it proposes HalFscore (I actually lol at this wordplay), which uses GPT4-o to generate the two scene graph from the two captions and then use GPT4-o to compare them.

strengths:
1.	This paper proposes a new metric to evaluate fine-grained hallucination while also validating the metric by analyzing the correlation with human
2.	This paper proposes a training method that is more fundamental than contrastive decoding.
3.	The paper also provides persuasive experiment results indicating that the hallucination is reduced while preserving the original general VL understanding capability.

weaknesses:
1.	Use GPT-4o as an additional source for perturbation. However, using GPT-o is always expensive on large-scale datasets. I would like to study the impact of the quality of the adversarial language  that is augmented by open-sourced VLMs such as InternVL/QwenVL

questions:
I would like to study the effect of the size of such adversarially augmented data. Would more of such data be more helpful?

I would like to discuss more deeply about such robustness interrupt. Basically you want to construct the data that is unlikely to be generated purely using language bias, thus able to alleviate the overshadowing of the language knowledge. Can we directly construct this training dataset by asking GPT-o to generate these kinds of questions and answers, which directly achieve the paper's goal, instead of purely doing such perturbing over the given QA data? I am concerned actually this perturbing is decreasing the normal VL understanding capability, but because it reduces the hallucination, the general knowledge evaluation (MMbench) is still good
</content>
</thread_object>

</thread>
<thread>
creation date:
1731868805838

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
## General Response: 

We sincerely thank all the reviewers for taking their time to carefully read through our paper and provide valuable comments.

### Contrubution:

We sincerely thank reviewers RVA9, nju1, vMbR, bhZ8, 3ZD9 for well recognizing the contribution of our method:

1. The proposed method is simple and effective.(Reviewer nju1, vMbR, bhZ8,3ZD9), more fundamental than contrastive decoding (Reviewer RVA9), and can be combined with other decoding-based methods to further improve performance.(Reviewer nju1).
2. The authors introduce a new well motivated metric and show that it correlates more strongly with human judgment (Reviewer RVA9,nju1).
3. This paper addresses a significant gap in the current evaluation of MLLMs(3ZD9).

### Promising and Persuasive Results:

- All reviewers agree that our proposed method effectively reduces the hallucination issue in fine-grained image captions (Reviewer RVA9,nju1,vMbR, bhZ8,3ZD9) while preserving the original general VL understanding capability (Reviewer RVA9,nju1,3ZD9).
- Our paper provides extensive and persuasive experiment results(Reviewer RVA9,3ZD9), and nice ablations on different version of training templates as well as different relevance levels of perturbation(Reviewer nju1).

### Summary of changes

We have revised our submission and summarized our updates as follows:

- Problem of table description mentioned by Reviewer nju1 is fixed.
- We addressed the numerical issues of RLAIF-V pointed out by Reviewer vMbR.
- Citation related to dense image captioning  in the introduction mentioned by Reviewer bhZ8 has been updated.
- We have added quantitative results on the additional training overhead and included them in the appendix A.3.
- We have added the interface screenshot of the user study and included them in the appendix A.7.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the hallucination in MLLM. The paper specifically studies the image dense captioning task and points out the root cause of the hallucination is the model biased toward relying on the language prior rather than the image information. To measure the hallucination, the paper proposed a HalFScore which built language graphs on groundtruth caption and the visual LM generated caption. The discrepancy b/w those two graphs are measured as the HalFScore. The paper also proposed a data augmentation technique to improve fighting the hallucination. Concretely, the paper proposed to add a misleading prompt based on language prior to the MLLM training. Although the training time slightly increased, the inference time remained the same and the hallucination reduced.

Strength:
1. The task is quite important and the paper is easy to follow
2. The metrics (HalFScore) is well motivated and has a good alignment with human ratings.
3. The proposed approach achieves good performance.

Weakness:
1. This work might rely on GPT-4o (SoTA MLLM) to generate the misleading prompt for data augmentation. 

Given the overwhelmingly support for this paper, I would recommend accept.