A Unified Hallucination Mitigation Framework for  Large Vision-Language Models
Authors: Yue Chang
Liqiang Jing
Xiaopeng Zhang
Yue Zhang
Venue: Accepted by TMLR ()

Abstract: Hallucination is a common problem for Large Vision-Language Models (LVLMs) with long generations which is difficult to eradicate. The generation with hallucinations is partially inconsistent with the image content. To mitigate hallucination, current studies either focus on the process of model inference or the results of model generation, but the solutions they design sometimes do not deal appropriately with various types of queries and the hallucinations of the generations about these queries. To accurately deal with various hallucinations, we present a unified framework, Dentist, for hallucination mitigation. The core step is to first classify the queries, then perform different processes of hallucination mitigation based on the classification result, just like a dentist first observes the teeth and then makes a plan. In a simple deployment, Dentist can classify queries as perception or reasoning and easily mitigate potential hallucinations in answers which has been demonstrated in our experiments. On MMbench, we achieve a 13.44%/10.2%/15.8% improvement in accuracy on Image Quality, a Coarse Perception visual question answering (VQA) task, over the baseline InstructBLIP/LLaVA/VisualGLM. Our source code will be released on GitHub.

Published in Transactions on Machine Learning Research (10/2024)
A Unified Hallucination Mitigation Framework for Large Vision-Language Models
Yue Chang" Liqiang Jing"
Xiaopeng Zhang" Yue Zhang
The University of Texas at Dallas
hag.:2253333..jggiaaaaggg (hanppacppng0723)))mmaa....mmm wut../oooe.cccccccoouuuttttaa... edu
Reviewed on OpenReview: http:s//ppeeeeevveeeeeeerr/rrreo
Abstract
Hallucination a common problem for Large Vision- Liigun-Languag Models (LVLMs) with long generations which is difficult to eradicate. The generation with hallucinations is partially inconsistent with the image content. To mitigate hallucination, current studies either focus on the process of model inference the results of model generation, but the solutions they design sometimes do not deal appropriately with various types of queries and the hallucinations of the generations about these queries. To accurately deal with various hallucinations, we present a unified framework, Dentist, for hallucination mitigation. The core step is to first classify the queries, then perform different processes of hallucination mitigation based on the classification result, just. like dentist first observes the teeth and then makes plan. In a simple deployment, Dentist can classify queries as perception or reasoning and casily mitigate potential hallucinations in answers which has been demonstrated in our experiments. On MMbench, we achieve a 13.44 13.44%//0.22% 15.8% improvement in accuracy on Image Quality, a Coarse Perception visual question answering (VQA) task, over the baseline InstructBLIP/LLa LLAALL//LLLLAA/ssaaLLLLLLL. AVAisuaaGL.MMM..
1 Introduction
Hallucination in Large Vision- Vision-Langue Language Models (LVLMs) is a critical issue, which manifests as the model's generated content partially deviating from the actual content of the image (Jing et al. 2023a For example, when provided with a image and two questions as input, LVLMs inacourately identify characters' actions and misinterpret relationships between characters, as illustrated Fig. D Such inaccuracies can lead misinformation, potentially degrading the user experience and misleading individuals. This issue underscores the necessity for ongoing improvements to enhance the reliability and accuracy of LVLMs, mitigating the risk of hallucinations and their consequent. misinformation.
To tackle the above challenge, existing work either focuses on optimizing the training data and the parameters of the existing model (Liu et al. 2023b Lu et al. 2023), or correcting the hallucinations during the generation stage without model update (Yin et al. 2023). The former collects high-quality training data, such as adding
negative instances to the training data to avoid overconfidence in the model (Liu et al. 2023a). The latter
mainly utilize the generated object information from the vision foundation model (such as blip2 (Li al.,
2023a to detect hallucinations and eliminate them. For example, Woodpecker (Yin et al. 2023 extracts main objects from the response generated by LVLMs and then verified these objects with object sogmentation
tool [Liu et al. 2023d) and VQA models (Li ct al.) 2023a). Similarly, HalluciDoctor (Yu et al. 2023a) makes the description-oriented answer chunks extract ion and formulates corresponding questions, uses answers for these questions which are gathered from various LVLMs to do the consistency cross-checking and remove hallucinations.
* "Equal contribution.

---

Published in Transactions on Machine Learning Research (10/2024)
Please describe this image.
The image features man and young girl standing close to each other. The girl hugs the man from behind.
What is their relationship?
They have harmonious relationship in the picture, Considering their age, they may a father and daughter.
Figure 1: An example image of hallucination. The generation the model is partially inconsistent with the image, which we call perception hallucination and reasoning hallucination respectively.
There are two main hallucinations in the model response (Ji et al. 2023 perception hallucination and reasoning hallucination, as shown in Fig. | The former is manifested by incorrectly describing image content in the model generation, such as errors when describing object attributes, while the latter refers to the model producing fallacies in logical reasoning answers. Although the previous methods for hallucination mitigation have achieved success, they still have a common problem, that is, when faced with these two types of hallucinations, the fixed verification method may sometimes be inappropriate and ineffective. For example, for reasoning queries, is not effective to object detection pictures to verify whether the object in the answer exists, as shown in Fig. [ addition, sometimes when the corrected answer obtained by the existing method is used as input to perform the same correction once, the answer after the second correction is inconsistent with the first time, which means that one correction did not remove all the hallucinations.
In order to solve this problem, we propose a unified framework for two main kinds hallucination mitigation. descriptive answer or a logical reasoning answer, our framework Dentist will try to correct. the parts of the answer that do not match the content of the picture. Specifically, the framework we steps: (1) Potential hallucination
Whether
proposed is verification loop, and each loop is divided into two classificution divides the query into two categories: perception and reasoning, which also classifies the potential hallucinations in answers when these queries are used as input to LVLMs. (2) cenide-and-coqquee treatment makes the mitigation based on the classification. The generation for the perception query will be verified by the sub-qucstions, while the generation for the reasoning query will be verified with the help of ChoughoffThought (CoT). To ensure that hallucinations are miligated as much as possible, the above verification loop will continue until the revised generation no longer changes semantically significantly or the loop limit is reached.
L. Liu et al. 2023c CHAIR (Rohrbach et al. 2018) and POPE (Li et al. 2023c) using three models: InstructBLIP (Dai et al. 2023), VisualGLM Ding et al. 2021 Du et al. 2022) and LLaVA (Liu et al. 2023c), respectively, to test the effectiveness of our proposed method. In addition, we also conduct a comparative experiment with a current effective LVLM hallucination mitigation method, Woodpecker (Yin al. 2023) Our method demonstrates the effectiveness and superiority in many visual language tasks, and promotes the performance of the baseline models. In particular, in the experiment, we achieve 13.44% 11.1%%115.... 11.8% improvement in accuracy in the visual language task of Image Quality, compared with the baseline models InstructBLIP /LLLLA////////////// VVVVVV//VVilLLM....
We complete quantitative experiments on MMbench [Liu et al. 2023e
Our main contributions are summarized as follows:
We propose a unified framework called Dentist for hallucination classification and mitigation. To the best of our knowledge, we are the first to distinguish treatment based on classification of the potential hallucinations and moreover use validation loop for complete removal of hallucinations.
2

---

Published in Transactions on Machine Learning Research (10/2024)
Our unified framework is easily integrated into various LVLMs. The clear design of the framework also provides convenience for new classifications and treatments to access the framework.
We comprehensively evaluated our method on several hallucination miligation benchmarks (including MMbench, POPE, CHAIR, and LLaVA-QA90) with detailed analysis. As byproduct, we released our code
2 Related Work
2.1 Large Vision-Language Model
Inspired by the success of Large Language Models (LLMs) (Wang et al. 2022 Zhao et al. 2023 Jing et al., 2024a 2023b), the multimodal learning community shifted research attention to LVLMs. LVLMs mainly use the cross-modality aligner to connect the visual encoder (such CLIP (Radford et al., 2021 and
LLMs (such as LLaMA (Touvron et al. 2023) ) to lackle vision-language lasks. For example, LLa VA (Liu
et al. 2023c) connects a vision encoder and a LLM for general-purpose visual and language understanding suggesting practical tips for building a general-purpose visual agent. Mcanwhile, InstructBLII (Dai et al., 2023) introduce an instruction-aware Query Transformer which extracts visual features from the output embeddings of the frozen image encoder, and feeds the visual features as soft prompt input to the frozen LLM. In addition, VisualGLM (Ding et al.) 2021 Du et al.) (022) use Qformer (Li et al.) 2023b) which builds bridge between the visual model and the language model. Though these LVLMs have powerful visual language understanding ability on the generation task, sometimes their outputs still contain hallucinations that need to be corrected. Some studies (Wang et al.
2023b Awal et al. 2023
use LLMs to improve the
performance of LVLMs, which is worth learning from.
2.2 Hallucination
With the progress of rescarch LVLMs, the problem of hallucination has gradually been exposed BBai et al. 2024), and it has attracted more and more attention. Research around hallucination focuses on three
aspects: detecting (Gunjal et al.] 2023 Luo et al.
2024), mitigating (Kang et al., 2023 Lu et al. 2023 Wang
let al. 2023a Yin et al. 2023; Yu et al.) 2023a by
Leng et al. 2023 Huang et al. 2023 Jing & Du 2024).
and evaluating hallucinations (Jing et al.) 2023a
Wu et al. 2024 Jing et al.] 2024b). In this paper, we
mainly focus on hallucination mitigation. Previous works on hallucination mitigation can be divided into two categories: model inference optimization and model generation optimization. The first category focuses on the process of the training and inference of the LVLMs. RLHF-V (Yu ct al. 2023b) collects human feedback at the data level and learns the correctional human feedback at the training level to reduce hallucinations in model generations. Ever (Kng et al.) 2023) points out that mitigating hallucinations in real time during model inference is more appropriate than generating corrections from the model outputs, as the latter is subject to snowballing effects. VIGC (Wang et al., 2023a) uses an iterative method to concatenate the short sentences generated each time, and ensures accuracy by controlling the length of the generation. On the other hand, the sccond category focuses on the aspect of the generation of LVLMs, designing methods obtain hallucination information from the output of the model and do the mitigation. Leng et al. (2023) introduces Visual Contrastive Decoding (VCD) to counteract the statistical biases and mitigate hallucinations by contrasting model outputs generated from original and distorted visual inputs. For example, Woodpecker (Yin et al.) 2023) makes the question formulation and visual knowledge validation base on the keywords which are extracted from the output of the model and uses an LLM to modily the hallucinations in the generated responses.
Despite the success of the existing method, they overlook the diversity of hallucinations which results in a fixed hallucination elimination method that cannot be applied to all hallucination situations well. To solve this problem, propose unified framework for mitigating hallucinations, the core step of which is to classify potential hallucinations caused by different queries.
https://g/ thub. cone//CVaanaauue/eettistt
3

---

Published in Transactions on Machine Learning Research (10/2024)
3 Method
In order to tackle various types of hallucination, our objective is to propose a unified framework for hallucination mitigation. Therefore, we devise & unified hallucination mitigation framework for LVLMs which mainly consists of three major components: potential hallucination classification, divide-and-conquer treatment, and validation loop, as shown Fig.
Qurry
Reasoning Section
Image
Prompt
LYLM
Reerowing
Query Type?
Temp Revised
Verifientien Loop
step
Fiaal
Perception Section
Original
Percopriox
Query
Original
Imsage
Verification Sub-quesiiuns Generation
LVLM
Sub-answers
Aggregation
Potential Hallucination
Clasification
An overview of the proposed method. The components using GPT are indicated in orange. The icons of open and closed eyes indicate whether the component is a pure text task or is related to an image. The black line represents the original part LVLM. The blue line represents the forward path of the verification process, and the orange line represents the feedback path in the verification loop. The core point is customize different methods of mitigating hallucinations by classifying the query. The reasoning section is used to mitigate the hallucinations caused by reasoning queries, while the perception section is used to mitigate the hallucinations caused by perception queries.
Figure 2:
3.1 Task Formulation
Suppose have dataset D ((Q:((200 Y.) where L; is the image, Q; refers to the query of the image and Y, represents the truth answer to the corresponding qucry. Note i represents the index of samples in the dataset D. We omit the index of Qs, I and Y, in the following discussion for the sake of brevity. Thereafter, we feed the image and query sample to the IVLM as follows,
y M(I
(1)
where 8 is the original answer for I) and θ is the parameters of LVLM M. Since the generated response Y suffers from hallucination problem, our aim to devise a hallucination mitigation method to minimize the semantic difference between Y and Y as follows,
min (DY, min R()))),
(2)
where D(-) is the semantic deviation and R(-)) refers to our hallucination mitigation method.
3.2 Potential Hallucination Classification
As we mentioned before, there are two main types of hallucination: perception hallucination and reasoning hallucination. These two types of hallucinations correspond to two types of queries: perception query and reasoning query (Liu al. (023c). The percoption query feature mainly requires the model have the ability perceive visual features, such as attribute recognition, scene description, etc. The hallucinations caused by this type of queries be effectively alleviated through visual level approaches, like object detection. The
4

---

Published in Transactions on Machine Learning Research (10/2024)
reasoning query mainly tests the understanding and reasoning ability of the model and the corresponding potential hallucinations should be mitigated by reasoning level approaches such as CoT which significantly improves the ability of large language models to perform complex reasoning (Wei et al. 2022). Since the type
of potential hallucination in the answer can be judged based on the type of query, we firstly classify queries
into the above two major calegories and then handle the corresponding potential hallucinations. We employ ChatGPT to complete query classification through a prompt as follows,
C = ChalGPT(P.(Q))
(3)
where Pc(-) is a prompt which can instruct ChatGPT to classify the query into perception or reasoning, and C' represents the classification result. Correspouding details can be found Appendix A.3
3.3 londe-and-conqueee Treatment
After query classification, the LVLM responses of perception and reasoning queries need to be processed diffcrently. This because, as we mentioned before, different types of queries examine different capabilities of the IVLM, and the miligation methods required for hallucinations in answers are also different.
To deal with the perception query, need to generate verification sub-questions based on the original query and the original answer with hallucinations generated by the LVLM. The LVLM answers these sub-questions to obtain sub-anse anbwers and finally we aggregate these answers to form the output with fewer hallucinations For reasoning queries, a common phenomenon is that the LVLM only generates the results of logical reasoning and the logical reasoning process we want to see is omitted in the generated content. In response to this situation, the method we propose is to use the CoT.
3.3.1 Visual Verification for Perceptron
LVLMs are prone to hallucinations when generating long descriptive texts (Liu et al., 2023b). This exactly corresponds the situation of the responses of perception queries. We are inspired that when the long answer to perception queries contains hallucinations, we can split. the long answer into short sentences and design verification sub-questions based on the key points in the sentences. We formulate this process follows:
{p) = ChatGPT(P.(Q.X))
(4)
where P.()) is a prompt which can instruct ChatGPT to generate sub-questions & according to the query Q and the original answer Y, and corresponding details can be found in Appendix A.4
After generating verification sub-questions, feed them to the LVLM along with the original image. Thereafter, we can get the verification sub-answers from LVLMs. It is worth mentioning that the LVLM we used for generating the sub-answers is just the original model which has hallucinations that need to be revised. It can probably be replaced with any visual question answering (VQA) model, but would be accompanied by the suspicion of using a better model for better work. To demonstrate the ability of our approach mitigate hallucinations rather than the ability of the rectified models, we chose to use the original LVLM. We formulate this process as follows:
y: M(I
(5)
where yi is the i-th sub-answer.
Then, we aggregate the sub-question answer pairs and feed them ChatGPT with the original answer Ý to refined the hallucinated response follows,
Y = ChatGPT 81 J..... Vn 8)),
(6)
where P.(.) is a prompt that can instruct ChatGPT to aggregate the sub-answers and do the correction, and Y the corrected answer. Details for the prompt P.(.) template which used for aggregating sub-answers can be found in Appendix A.5
5

---

Published in Transactions on Machine Learning Research (10/2024)
3.3.2 Chain of Thought for Reasoning
The answers generated by the LVLM from reasoning queries are not as "clear" as the answers to perception questions, which means the LVLM's answers tend to only contain the results of logical reasoning, but not the process logical reasoning and the basis for that (perception about visual details before reasoning). Therefore, the method of the perception section no longer applicable because of the missing part about perception in the reasoning answer. To solve this problem; that contain more details which are beneficial to our following process. At the same time, the LVLM will also improve the accuracy of reasoning when performing CoT. Add 'Let's think step by step' to the start of the original query to do the CoT and use ChatGPT by prompt to obtain the revised answer as follows:
the CoT prompt method to oblain answers
Y CAaG 8))
Pt(-) refers to 'Let's think step by step' and P.(.) is the prompt that can instruct ChatGPT to correct the original answer Y according to the generation of IVLM M with CoT. Refer to Appendix A.6 for the detailed information about prompt P-(-) template.
3.4 Validation Loop
After the above steps, we obtain the preliminary verified answer which may still contain hallucinations that have not been climinated because of the imperfections of the verification sub-questions generation. order solve this problem, we propose to regard the entire verification framework as a repeated block in the verification loop chain. We show the overall procedure in Algorithm [ The verified answer treated as the original answer and re-verified. The difficulty of loop verification is how to judge when the hallucinations in the answer has been completely removed so the loop can be stopped. We believe that if and only if the verified answer does not change significantly semantically after a new round of verification, it means that all the hallucinations that can be eliminated have been eliminated. On the other hand, if the answer still changes significantly after a specific number of rounds of verification, we believe that there a snowball error phenomenon the verification cycle. We will stop the loop and use only the answer from the first. verification as the final revised answer. We use ChatGPT determine whether the answer has converged and is longer changing semantically. This corresponds to the Similar function in Algorithm | and the prompt template is given in Appendix A.7
Algorithm Dentist
Input: Original question Q original image original answer Y, the large vision-language model M, the maximum iteration T
Output: Corrected answer Y
1: Y -
2: Yi Y
3: for j in 1,2... do
Y, <- Verifu(Q, Q I, M)
then if Y,
end if
8: if Similar (Y, Yt) + Yes then
9: # No improvement in new round of verification
10: return Y
11: else
12: Y
13: end if
14: end for
15: # Arrive the maximum iteration, so return Y;
16: return Y,

---

Published in Transactions on Machine Learning Research (10/2024)
4
Experiment
In this section, we conduct extensive experiments to answer the following research questions:
RQI.
Could our framework improve the current IVLMs?
RQ2. What is the contribution of each component of our Dentist ?
RQ3. What is the intuitive performance of our Dentist ?
4.1
Experiment Settings
Benchmarks. MMBench is novel multi-modality benchmark, which develops fine-grained ability assessment for LVLMs. The MMBench evaluation standard is divided into three levels. The L-1 ability level incorporates Perception and Reasoning, L-2 ability level consists of Coarse Perception, Fine-grained, etc. and L-3 ability level covers Image Style, Image Scene, Image Emotion, elc. Relying on such hierarchical and fine-grained capability assessment, the performance of LVLM can be comprehensively evaluated. The dataset we use is MMBBenh--Tes((NN)
LLaVA-QA90 is also dataset used to evaluate LVLMs. LLVV--A9 contains 90 questions and 30 images taken from COCO Val 2014 (Lin et al. 2014). To evaluate the generated response, we feed the query, image, and model response to GPT-4V (OpenAl] 2023) to get a score of scale of 1 to 10. The prompt template is available in Appendix A.9. We respectively pair baseline LVLMs with Dentist, baseline LVLMs with Woodpecker, and provided their responses to GPT-4V for scoring, which ensures that the scores are mutually referenced, thereby making them more reliable.
Caption Hallucination Assessment with Image Relevance (CHAIR) (Rohrbach et al.) 2018 is a widely-used metric for evaluating object hallucination in image captioning tasks. By comparing the image captions generated by the model with the ground truth objects in the corresponding image, CHAIR evaluates the degree of hallucination of the model and explains the performance of the model. CHAIR has two variants: CHAIR, (C.) and CH AIR, (Ct), both of which reflect the degree of hallucination of the model, the difference being that CHAIR, is at the sentence level and CHAIR; is at the object instance level. The calculation is as follows:
(sentences with hallucinated objec()]
CHAIR.
Mall sentences)]
Khallucinated objects}| Fall mentioned objects]]
CHAIR,
POPE Li et al., 2023ch is also an evaluation method for object hallucination. Three kinds of sampling settings of random, popular, adversarial, are constructed on the dataset according to human annotation or automatic visual segmentation tools. The difference between them lies in the negative sample sampling method. Where, the random setting randomly samples objects that do not exist in the image; the popular setting samples objects that are not present in the current image, but are most common throughout the dataset; the adversarial setting samples objects in the dataset that co-appear most frequently with objects in the current imago, but are not present in the current image. POPE has a high degree of fairness and robustness, which helps to better demonstrate the hallucination mitigation effect of Dentist.
In terms of sampling settings, sample 100 images and construct 6 questions for each type of sampling setting for each image. Each question is "Yes or No" question that transforms the model task into simpler binary classification task. In terms of evaluation metrics, we adopt Accuracy, Precision, Recall and Fl score as the evaluation metries.
Basclincs. We first select currently mainstream LVLMs as our bascline models, including InstructBLIP LLaVA, and VisualGLM. In addition, we also compare against the baseline Woodpecker (Yin et al. 2023 which is training-free hallucination correction method for LVT.Ms.
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Results on MMBench- MMBench-TTss((N)))
(EN). The performance is measured by accuracy, where the improved performance for each partition is highlighted in bold.
InstructBLIP-7B
-LAVA-V1.5-78
VisusIGLM-6B
Ability
Baseline
Dentist
Woodpecker
Baseline
Dentist
Woodpecker
Baseline
Dentist
Woodpecker
Image Topie
60.0
60.0
43.5
97.6
16.44
87.1
52.9
50.2
64.7
Image Quality
0.0
10.6
0.0
0.0
10.2
0.0
15.6
3.5
Course
Perception
Image Emotton
32.7
11.3
12.0
67.5
76.1
41.7
50.6
33.7
Image Scene
58.1
00.3
37.4
86.3
88.3
79.8
68.5
70.3
59.7
Image Style
38.8
37.1
38.8
58.8
55.3
58.8
30.6
35.8
24.7
Fine-graized
OCR
36.4
78.3
53.2
Celebrity Recugnition
40.8
49.2
68.6
60.2
68.6
60322
52.5
55.2
46.6
Perception
[Single-inesaance]
Object Localization
3.9
14.4
8.6
16.3
11.5
14.4
8.6
10.9
8.6
Attribute Recoguition
46.3
51.4
32.5
66.7
70.6
68.7
40.0
13.7
33.3
Fine-grained
Action Recognition
58.5
57.5
20.4
35.2
Perception
Atcribute Comparison
2.6
2.6
0.0
21.2
25.8
6.4
8.8
10.8
6.1
Cross-instanoe
Spatial Relationship
11.1
8.6
11.1
11.1
15.3
11.1
7.3
10.9
8.6
Attribute
Identity Reasoning
68.3
08.3
70.7
86.6
86.6
86.6
81.7
71.2
Reasouing
Function Ressoning
16.2
30.9
71.5
77.8
41.9
37.7
Physical Property
21.0
21.9
26.0
55.0
50.0
53.0
26.0
30.3
17.0
Nature Relation
22.2
27.3
21.7
38.3
38.3
38.3
21.7
30.6
8.0
Relation
Physical Relation
17.3
19.2
28.9
28.9
3.8
3.8
Reasoning
Social Relation
27.6
41.0
36.4
62.8
69.6
62.8
46.2
45.3
17.9
Logle
Image-Text Text
5.9
7.0
15.8
11.9
10.9
12.9
3.9
5.0
8.9
Ressoning
Future Prodietion
46.7
55.0
25.0
43.1
52.1
44.4
21.6
31.0
6.9
Overall
33.9
36.9
32.7
51.0
54.8
51.2
32.0
36.4
28.7
Implementation Details. We utilize GPT-3.5-turbo-06173 to assist in keyword extraction, sub-question gencration, verification loop, and verification answer integration. Experiments have proven that GPT-3.5- turbo can tackle these tasks. On MMBench, we set the experiment rounds to 2: (1) In the first round of evaluation, we have the model generate raw predictions according to MMBench's evaluation rules and submit. them to MMBench's official platform to obtain various accuracy rates; (2) In the second round of evaluation, based on the original prediction of the model, query classification, different verification processes and answer integration are carried out using GPT-3.5-turle (specific details can be found in Section 3). Similarly, we upload the results of the second round of evaluation to the official MMBench platform to obtain various accuracy rates; (3) Finally, we jointly analyze the results two rounds of evaluation to demonstrate the effectiveness and superiority of Dentist.
Previous studies (such as POPE eL al. 2023c ) have found strong correlation between LVLMs hallucination and the length of the generated text, and in our experiments we find this to be true. In the CHAIR evaluation, since the LVLMs we select all have remarkable instruction following ability, we notice that when the LVLM are prompted to "generate as detailed a description as possible", the CHAIR score of the model is much higher than when they are prompted to 'gencrate a short description' (a higher CHAIR score indicates a higher degree of hallucination). This not desirable a common usage scenario. But we can take ad advantage of this feature to better demonstrate the ability of Dentist to mitigate hallucinations, Therefore, in the CHAIR experiment, prompt the model to generate detailed description.
In the following experiments, we limit the maximum number of iterations to 3 (i.e., T in Algorithm is equal to 3) to ensure the effectiveness of the verification and avoid excessive time costs.
4.2 Results (RQ1)
Results MMBench. The results on MMBench summarized in Tab. [ From this table, we have several observations.
(1) The largest accuracy improvement among the three LVLMs exceeds 15.6%, showing that Dentist have excellent correction effects, making obvious improvements in various metrics for the baselines. (2) Dentist performs outstandingly in Image Emotion, Image Quality, Future Prediction, Attribute Recognition, etc., which indicates that Dentist is capable of mitigating hallucination coarse perception, fine-grat gra-gndi perception and logic rcasoning.
(3) Among all metrics, Image Quality shows the highest improvement, which indicates (hat Dentist particularly effective for hallucinations in such problems.
hhttps://platfors
8

---

Published in Transactions on Machine Learning Research (10/2024)
Overall, Dentist has brought significant improvements to the three LVLMs. Comparing Woodpecker's performance, it can be seen that it can also bring many improvements in some perception queries, such bringing a huge 11.8% improvement to VisualGLM on Image Topic. However, in many queries, especially reasoning queries, it even reduces LVLM's performance, such as causing a huge 28.3% decrease in VisualGLM on Social Relations, which is unacceptable. A large part of the reason for the decline performance comes from Woodpecker's 's lack of targeted handling of reasoning problems.
Table 2: Results on LLaVA-QA90. The accuracy, detailedness and logicality metrics are on a scale of 10, and a higher score indicates the better performance. The better performance for each partition is highlighted in bold.
LVLM
Accuracy
Detailedness
Logicality
Precision (%)
Baseline
6.5
4.9
4.3
21.0
InStructBLIP
Woodpecker
6.4
5.5
4.4
20.9
Dentist
7.0
(+0.5)
5.5
(+0.6)
4.8 (+0.5)
22.8
Baseline
6.0
5.3
4.4
27.3
LLaVA
Woodpecker
6.5
5.5
4.5
26.6
Dentist
6.6
(+0.6)
5.8 (+0.5)
5.0 (+0.6)
28.0
Baseline
5.6
5.0
4.0
36.5
VisualGLM
Woodpecker
2.0
1.6
1.3
38.3
Dentist
6.2 (+0.6)
5.8
(+0.8)
4.7 (+0.7)
39.8
Results on LLaVA-QA90. If manual verification required, the evaluation on LLaVA-QA90 is labor- intensive and somewhat subjective. Therefore, it necessary to use a powerful evaluation tool to ensure consistency in evaluation standards while also possessing strong visual language task answering and instruction following abilities. Therefore, consider utilizing the powerful LVLM, GPT-4V. Specifically, we involve GPT- 4V in scoring and evaluating model responses by setting appropriate prompt words. We have designed the following three metries: Accuracy: how accurate is the model response about the image content; Detailedness: level of details of the responses; Logicality: whether the reasoning content of response is reasonable. In addition, we also use GPT--3---ur to calculate the proportion of logical reasoning sentences included in the passage without hallucination, and record it as Precision better check the rationality of the reasoming content. The prompt template is available in Appendix [A.9 We conduct the same evaluation on the current. effective hallucination correction method, Woodpecker, and compare the bascline LVLMs, Dentist, and Woodpecker scores.
Tab. 2 shows the results. Obviously, equipped with our verification method, the models' performance has been comprchensively improved across the three metrics. On average, there is an improvement of over 0.5 points (relative improvement exceeding 13.6%), and Dentist scores better than Woodpecker on all baselines. This indicates that Dentist not only improves the accuracy and detailedness of LVLMs in describing image content, but also enhances the rationality of reasoning content. At the same time, the Precision of Dentist has also improved to some extent, indicating that Dentist increase the proportion of effective reasoning content in IVTMSs answers.
It is worth noting that Woodpecker's score in VisualGLM decreases significantly. We find that the reason for the decrease in score is the failure Woodpecker's object detector. On the one hand, this proves that the Woodpecker over-relies on object detector, and when its detector fails, the correction effect of Woodpecker will become worse; on the other hand, proves that Dentist has stronger robustness.
Results CHAIR.
We compare the effects of Dentist and Woodpecker on mitigating halluuciaatinss. The specific method is as follows: we select 2000 examples in COCO Val 2014, let the bascline model generate corresponding captions, and then apply Dentist and Woodpecker respectively to process the hallucination correction of the captions, and calculate the two metries of the CHAIR to analyze the difference between the two correction methods.
Tab. 3 shows the results. It is easy to draw the following conclusions from the results: (1) The model with remarkable performance may also produce more hallucinations. For example, L.LaVA scored very high
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Comparison of the CHAIR metrics between Dentist and Woodpecker. The best performance for each metric is in bold.
InstructBLIP
LLaVA
VisualGLM
C 4
C
C2 4
C
C
G
Baseline
59.2
22.9
84.0
23.4
44.0
17.8
Woodpecker
56.9
18.5
71.9
19.8
37.0
11.9
Dentist
52.1
16.7
75.2
17.1
36.0
11.3
MMBench, but the CHAIR evaluation shows that LLaVA's hallucinations were more scrious. (2) Dentist demonstrates an ability to reduce hallucinations no less than Woodpecker, and helps the baselines reduce more hallucinations most aspects.
Table 4: Results on POPE.
The best performance for each metric is highlighted in bold.
РOPE
LVLM
Method
Accuracy
Precision
Recall
F1 Score
Yes (%)
Baseline
85.00
88.32
78.67
84.32
45.67
InstructBLIP
Woodpecker
86.21
92.38
78.97
84.63
35.08
Dentist
87.00
94.40
80.67
85.82
41.67
Baseline
79.67
75.47
84.67
75.34
63.00
Random
LLaVA
Woodpecker
84.78
85.64
90.26
83.44
50.23
Dentist
87.33
89.44
92.67
86.98
47.33
Baseline
53.44
51.85
99.12
68.09
95.79
VisualGLM
Woodpecker
69.74
66.83
90.56
76.33
73.82
Dentist
75.88
73.25
87.79
78.95
68.02
Baseline
80.67
81.72
79.00
80.34
48.33
InstructBIIIP
Woodpecker
81.03
88.81
68.97
78.43
37.93
Dentist
83.33
86.76
78.67
82.52
45.33
Baseline
75.67
73.98
90.00
74.79
70.33
Popular
LLaVA
Woodpecker
85.42
84.38
85.39
83.64
50.28
Dentist
87.00
90.90
84.67
86.69
47.67
Baseline
58.31
54.63
99.12
70.44
90.90
VisualGLM
Woodpecker
66.78
62.45
88.34
70.25
88.44
Dentist.
70.53
68.12
87.35
71.20
86.91
Baseline
78.83
76.95
82.33
79.55
53.50
InstructBLIP
Woodpecker
77.59
82.23
68.97
75.47
41.38
Dentist.
80.83
83.33
78.67
80.41
47.83
Baseline
76.67
71.06
92.00
73.40
75.33
Adversarial
LLaVA
Woodpecker
77.62
74.53
88.67
80.73
64.25
Dentist
80.67
78.40
84.67
81.41
54.00
Bascline
54.10
52.21
99.12
68.40
95.12
VisualGLM
Woodpecker
62.67
60.33
85.44
66.78
82.35
Dentist
68.10
63.42
86.90
68.87
80.90
Results on POPE. We evaluate InStructBLIP, LLaVA and VisualGLM for hallucination on POPE respectively, and compare the results of baseline LVLMs, Woodpecker and Dentist. Tab. 4 summarizes the results of POPE under the random, popular, and adversarial sampling settings. It can be seen that, in all sampling settings, VisualGLM is relatively weak, noting that its Recall and Yes Rate are both very high (close to 1), which indicates that VisualGLM produces relatively severe hallucinations of objects not in the image (i.e. negative samples). The reason for this phenomenon is probably that VisualGLM fails to deal with the imbalanced distribution of the datasct during the training process. The other bascline models are above 70% on these metrics. Dentist brings significant improvements to these bascline LVLMs, which validates Dentist's s excellent. performance in mitigating hallucination. Specifically, under the relatively simple
10

---

Published in Transactions on Machine Learning Research (10/2024)
random sampling setting, Dentist obtains gain of 22.44% for VisualGLM in terms of accuracy. In the more challenging popular and adversarial settings, the performance of these baseline LVLMss decline to varying degrees. Dentist continues to show remarkable performance. In particular, Dentist boosts the precision of LLaVA by 16.92% in the popular setting and accuracy of VisualGLM by 14.00% in adversarial sctting. In addition, can be seen from the comparison that Dentist outperforms Woodpecker all sampling settings.
Verification time comparison. While evaluating Dentist and Woodpecker on POPE, we count and analyze the time spent on hallucination correction for both. A total of 100 images and 1,800 questions are provided to Dentist and Woodpecker throughout the evaluation process. For InstructBLIP, Dentist spend total of 7 hours and 49 minutes to complete this task, with an average of 15.6 seconds per query. Woodpecker spend a total of 8 hours and 16 minutes to complete the lask, with an average of 16.5 seconds per query. It can be seen that the the rapidity of Dentist is also better than that of Woodpecker and has a good timeliness.
Comparison with Woodpecker. The similarity between our framework Dentist and Woodpecker is that we both use an LLM to revise the hallucinated response generated by LVLMs. Woodpecker extracts main objects from responses and verifies these objects with object segmentation tool and VQA models, for all hallucinated responses. However, Dentist has a divide-and-conquee treatment. When Dentist is faced with perception query, it tries to verify the main objects in the model response. When faced with reasoning query, Dentist uses CoT to deal with it. This divide-and-conquer treatment accurately mitigates hallucinations and achieves better results.
4.3 Ablation Studies (RQ2)
To explore the effect of the query classification and verification loop, we conduct ablation studies in this section. Query Classification. We study three different variants and evaluate their performance on MMBench. (1) w/ w/o=Classification: we disable the query classification section of Dentist ; (2) w/ 9//oRReaoonnn: we classify all queries into perception for verification; (3) w w/o-Perception: we classify all queries into reasoning for verification. For the sake of brevity, we will refer to these three variants w/o-Cla, w/o-Rea and w/o-Per in the following discussion.
Table 5: Results on MMBench with different variants of InstructBLIP. For more comprehensive evaluation results on LLaVA and VisualGLM, please refer to the Appendix A.
Variants
Perception Accuracy Reasoning Accuracy
Baseline
33.74%
31.15%
Dentist
37.62%
35.92%
w/o- Classification
34.86%
32.73%
w/ o- Reasoning
38.94%
25.48%
w o- Perreppptn
28.34%
38.44%
Tab. 5 shows the results of InstructBLIP. We can that: (1) If query classification is not performed and verification is performed directly (w/o-Cla), the accuracy is not much higher than the baseline, and in some cases there is even a problem of reduced accuracy. Because at this point, the way Dentist corrects the model's answers is completely random, which largely depends on the performance and habits of GPT-3.5-turbo: it can be seen that the perception accuracy may not differ much from the baseline, or slightly higher than the baseline, while the reasoning accuracy may decrease. This because the query classification section tends to treat the problem as perception for processing. (2) If all queries are classified into perception (w/o-Rea) (this is what most current LVLM ballucination mitigation methods do), it can be seen that the perception accuracy is greatly improved, while the reasoning accuracy is greatly attenuated. This is because Dentist also verifies the reasoning problem as perception, so the verification method is not appropriate, resulting in a decrease in accuracy; (3) In the same way, if all problems are classified as reasoning (w/o-Per), the reasoning accuracy is greatly improved, and correspondingly, the perception accuracy reduced; (4) It can also be found that the perception accuracy of w/ w/o-Rca may even be slightly higher than that of Dentist. We speculate that this is due to the misjudgment by GPT-3.5-turf when classifying queries, such as mistakenly categorizing a very small number of perception queries as reasoning, while w/o-Rea precisely corrects this part of the misjudged
11

---

Published in Transactions on Machine Learning Research (10/2024)
perception queries. The same goes for reasoning queries. To verify this problem, we extract 1000 samples from the test results of MMBench and manually count the number of classification errors of Dentist. The results shows that among these 1,000 samples, 33 queries are misclassified, with an error rate of approxi- mately 3.3%. Compared to its improvement LVLMs performance, we believe this quantity is acceptable.
Verification Loop. Verification loop is also a com- ponent that need to study. We conduct additional experiments by varying the number of verification loops in our framework and evaluating it on MM- Bench to demonstrate its effectiveness.
InstructBLIP LLavA VisualGLM
Accuracy
Fig. shows the results, We can see that when the number of verification loop small, there is a slight improvement in accuracy the number of loops increases. However, when the number of cycles is large, the accuracy actually decrease the number of cycles increases. We separately take out one of the cases for observation and found that when the number of cycles is large, the output the LVLM and GPT gradually become chaotic and uncontrollable, which may lead to an avalanche of decrease in the accuracy the model when the number of cycles is large conclude that verification loop is effective, but special attention needs to be paid to limiting its frequency. When the model answer matches the validation answer, it is important to exit the loop validation in timely manner.
25
20
2
10
15
20
Iterations
Figure 3: Results of verification loop
enough. Therefore,
Query: Provide description of the given image.
ChatGPT Classification
Query: What's the profession of the people in this picture?
ChatGPT Classification
Perception
Reasoning
Sub Q&A Verification
Generation
Q: What kind of fruit is in the picture?
Chain-of Shain-of-Thought Prompt
Apple.
The picture are
Q: How many apples are there in the picture?
holding many flowers and plants around him. So the profession of the the gardener.
There three apples the picture.
Answer: There is person halding red apples the picture.
Q: What color are these apples? They red.
Answer: The profession of the man picture is farmer.
Revised Answer:
The picture is holding tools and him. profession man in the picture is gardener.
are many flowers plants
There a person holding three red apples in the picture.
Figure Two testing cases of Dentist The left example shows a perception hallucination produced by LVLM, and the right example shows reasoning hallucination produced by LVLM, and these hallucinations are climinated by Dentist.
4.4 Case Study (RQ3)
We provide two lesting examples in Figure to conduct qualitative analysis. It is obvious from the above example that: In the first example, Dentist classifies the query as 'Perception'. The original response of LVLM that "There a person holding two red apples in the picture', which obviously wrong. After Dentist extracts the keywords "apple', 'two", 'red', etc., three corresponding sub-questions are generated. Dentist then answers the sub-questions one by one. Since the sub-questions more targeted and usually
12

---

wwwp/oorpsst::264444.oo Published in Transactions on Machine Learning Research (10/2024)
have short outputs, the possibility of hallucinations is greatly reduced. After comparing and integrating the answers, Dentist finds out the existing hallucinations (the number of apples), corrects the original answer, and eliminates the perception hallucination.
In the second example, the original response of LVLM infers that the person in the image is a farmer, which is obviously wrong. The model mistakenly draws a conclusion that contrary the facts based on the information in the image. Dentist classifies the query as 'Reasoning' and refines the hallucinated answer according to the content of the CoT and the original output of the model. Dentist matches the original answer and the CoT generation answer, and find that the objects and inference in the content of the two matches, so the final answer is obtained, thus climinating the reasoning hallucination.
In the above cases, Dentist perfectly eliminates the perception and reasoning hallucinations produced by LVLM.
5 Limitations and Future Work
This study acknowledges limitation in the Dentist framework. When performing verification, we take the answers of the verification questions as the ground truth, which may still contain hallucinations, In terms of reasoning hallucination mitigation, the CoT for reasoning we use is relatively simple. In addition, loop verification also increases time cost. In future work, we may refine the CoT for reasoning and add validation of the details obtained from the CoT. In order lo reduce time and money costs, simplifying prompts without. compromising effectiveness is feasible research direction.
6 Conclusion
In this work, we propose a unified framework for hallucination classification and mitigation. We are the first to distinguish treatment based on the classification of hallucinations and use a validation cycle for the removal of hallucinations. Our framework has clear design which is easily integrated into various L.V.Ms, and provides convenience for new classifications and treatments integrate into the framework. To evaluate the effectiveness of our framework, LLaVA- -A990, CHAIR and POPE, which shows that Dentist significantly improve the baseline LVLMs on these benchmarks. At the same time, we compare the results of LLaVA-QA90 and CHAIR with those of Woodpecker, and the results shows that Dentist not only has excellent hallucination correction ability, but also has strong robustness.
conduct experiments the three baseline LVLMs on MMbench,
Acknowledgments
This work was supported by the OpenAl Research Access Program (Award ID: 0000006384), which provided access to advanced GPT models. The authors thank OpenAl for their support and for fostering innovative Al and machine learning research.
References
Rabiul Awal, Le Zhang, and Aishwarya Agrawal. Investigating prompting techniques for zero- and few- shot visual question answering. ArXie, abs/ hts/2306.09999) 2023. URL httpp:://ppi semanticscholar. org/
CorpusID:259187390
Zechen Bai, Pichao Wang, Tianjum Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. ArXiv, abs/2404.18930, 2024. URL
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purposc vision-language models with instruction tuning, ArXiv, hls//2305.05000 2023. URL httph:tt///// //ppi. semaatticccoolaar.oor/roosssl
258615266
13

---

Published in Transactions on Machine Learning Research (10/2024)
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Aduances in Neural Information Processing Systems, 34:19822 19835, 2021.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregrossive blank infilling. In Proceedings of the 60th Annual Meeting of the Associalion for Compulational Linguislics (Volume 1: Long Pupers), pp. 320 335, 2022.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and proventing hallucinations in large vision language models. arxiv preprint or arXiv:2308.003: 06394, 2023. URL https:://bi.orrg/// org/00.....//0...0 4855////r..55//.i..... www.:xiwii........mm 2308
Qidong Huang, Xiao wen Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Neng H. Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospatioon-alloaatio ArXiv, abs/23111.17 2023. URL https::/api..... sem.niicccsmatticchh.o..... Brt/CorpusID::25498881
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55112)1133,, 2023.
Liqiang Jing and Xinya Du. FGAIF: aligning large vision-language models with fine-grained AT feedback. CoRR, abs/2404 /2044000444 2024. doi: 10. 10.48550/ ARXIV 2404.00066. URL https::/doi. org/10 0 Xrv 2404.05046
Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du FAITHSCORE: evaluating hallucinations in large vision-language models. CoRR, abs/2311.01 22111111111777 2023a. doi: 10.48550/
URL
https://dol.org/// 48550 Xiv. 2311.0141 01477
Liqiang Jing, Xuemeng Song, Kun Ouyang, Mengzhao Jia, and Liqiang Nie. Multi-source semantic graph-based multimodal sarcasm explanation generation. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), ACL 2023, Toronlo, Canada, July 9- 9-14, 2023, pp. 11349-11360 Association for Computational Linguistics, 2023b. doi: 1116633/V1/2223AALLLLOGGG.. URL https: org/ 18633//1/2023.c11
635
Liqiang Jing, Xuemeng Song, Xuming Lin, Zhongzhou Zhao, Wei Zhou, and Liqiang Nie. Stylized data-to-text. generation: A case study in the e-commerce domain. ACM Trans. Inf. Syst., 42(1):25:1 25:24, 2024a. doi:
10.1145/3603374. URL hittps://dol org/ 1ttrg/...115/3603311 /14/3663374
Liqiang Jing, Jingxuan Zuo, and Yue Zhang. Fino-grained and explainable factuality evaluation for multimodal summarization. CoRR, abs/2402.11414, 2024b. doi: 10.48550/ AAA.A.222V.22.142222 URL https:://ti.i...
10. 4885558/aaiiv 2402 11414
Haoqiang Kang, Juntong Ni, and Huaxiu Yao. Fver: Mitigating hallucination in large language models
through real-time verification and rectification. arxiv preprint arXiw:2311009114 2023. URL https: :
//doi. 2itoooorr//o.4555//rxi.....2 09114
Sicong Long, Hang Zhang, Guanzhong Chen, Xin Li, Shijian La, Chunyan Miao, and Li Bing. Mitigating objectt hallucinations in large vision- language models through visual contrastive decoding. ArXiv, abs/ abs/2311.16992, 2023. URL httpss//appi: http//ppssssmaicccccocolllla http:///ps..ssooeatttooooooorr/r owtp://oopo..::/555555 2654666833
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023b. URL hhttps://app www.ntacccccoool..oooroooo
256390509
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXie preprint 2r.5: 10355, 2023c.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Laarencce Zitnick. Microsoft coco: Common objects in context. In Computer Eision-ECCH 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjic Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. 2023a. URL https://api sem.anicchmmtt
org/CorpusID: 259251834
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-
modal model with robust instruction tuning. arXiv preprint or Xix:22306 14565, 2023b. URL https:
14565
Haotian Liu, Chunyuan Li, Qingvang Wu, and Yong Jac Lec. Visual instruction tuning. ArXiu, abs/ /30444344 2023c. URL http://app.semmaiiisschooooooooe wwtp:/CrrrsSSS::251111999
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. Xiv preprint arXi:2200.000 05499, 2023d.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 3rXiv:337700628, 2023c. URL httpss://oii. http://ooo.oggg///// 1rg/10. 48550/arXiv. 238.. .62811
Jiaying Lu, Jinmong Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. Evaluation and miligation of agnosia in multimodal large language models. ar preprinl Xr0v 20000044,, 2023. URL https: org/10. (8550/arxiv.2225 04041
Junliang Luo, Tianyu Li, Di Wu, Michael Jenkin, Steve Liu, and Gregory Dudek. Hallucination detection and hallucination mitigation: An investigation. arXiv preprint arXiv:2401.08355 2024. URL https:
7/doi /ttp.//////...5//aaiiiiii 2401. 08358
OPenAL. Gpt-4 tochnical report. ArXiu, abs/ /bb//333308874 2023. URL httpss//arxii.orr/aas///2 08774
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. URL https://api semanticscholar 1ttp://rr.rrrr/rrrus1/22213111 5944
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning, arXiv preprint (tfe::180002155 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaune Lample. Llama: Open and efficient foundation language models. ArXie, hbss/2302.1377/ 2023. URL
http:/ppp...mmmnnicccccoooal hwtptntiiisccoaarror/////ooee wrgg/CoppsII:: 257219404
Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiao wen Dong, Weijia Li, Wei Li, Jiaqi Wang, and Conghui Hc. Vige: Visual instruction generation and correction. ArXiv, abs/2308.1277 2023a. URL httpps://ppi. sew.piicsssmaaicccll.aaa. httpa//aeaaaaaaiaaaaoaoooooonnnr 261 2611000735
Yizhong Wang, Yoganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self- generated instructions. In Annual Meeting of the Association for Computational Linguistics, 2022. URL httt::://api..mmmntiie. ortp/oooolooorrg///puuuss
254877310
Ziyue Wang, Chi Chen, Peng Li, and Yang Liu. Filling the image information gap for vqa: Prompting large language models to proactively ask questions. In Conference on Empirical Methods in Natural Language Processing, 2023b. URL hTR.:https:// https::///pp
15

---

Published in Transactions on Machine Learning Research (10/2024)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Aduances in neural in information processing systems, 35:24824 24837, 2022.
Jinge Wu, Yunsoo Kim, and Honghan Wu. Hallucination benchmark in medical visual question answering. Xiv preprinl (xXx:2401.058800 2024. URL https://doi. org/10. 48550/arXiv. org/10.44550/ariin 2401 05827
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint ar Xix:2310. 16045, 2023. URL https:///oli
18550/arXive 2310. 2390.16045
Qilan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. arXiv preprint arXiv:2311.136: 2023a. URL
48550//r.iv....... 2311 2311.13614
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy milms via behav- ior alignment from fine-grained correctional human feedback. ArXiv, 0bs/2331.000849 2023b. URL
wtapatisccccaaaaoo/roorrrrpuee org/CorpusID 265668723
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, 7. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. survey of large language models. ArXiv, abs/2303. 1330.18223, 2023. URL https: semanticscholar httpa//ccccccccoooor//o/po/////o wrgp/orppssD:2257900555
A Appendix
In this section, we will display all prompt templates used in this framework and other noteworthy figures and tables.
A.1 Multiple Responses Study
In order to comprehensively study the robust ness of Dentist and the consistency of experimental results, we re-conduct POPE evaluation under the random sampling setting, and the other settings are the same as those introduced in Section 4.1 The difference that we let the LVLM test each example 10 times and let our framework process each of these responses. We provide two new basclines: (1) Direct Rejection Baseline (DR Baseline): Tf all the responses of one sample have errors, deem the sample incorrect, else randomly choose one of the responses which has errors to be the answer. (2) Repeated Correction Baseline (RC Baseline): For each example, when our framework detects hallucinations in all ten responses of the model, the hallucinations are corrected normally. We speculate that the performance of DR Baseline will significantly decrease because only the classification function of our framework are enabled to detect errors, while the answer correction function of our framework is disabled, and the performance of RC Baseline should be consistent with Tab.
Table 6: Results on POPE under the random sampling setting.
LVLM
Method
Accuracy
Precision
Recall
F1 Score
Yes Rate
DR Baseline
79.17
81.81
75.00
78.26
38.33
InstructBLIP
RC Bascline
85.56
92.48
82.12
81.37
40.22
DR Baseline
71.70
71.93
74.55
73.21
44.34
LLavA
RC Baseline
87.33
86.97
91.38
82.79
46.41
DR Bascline
50.83
51.79
92.06
66.29
89.16
VisualGLM
RC Baseline
76.88
74.35
86.54
78.18
63.25
16

---

Published in Transactions on Machine Learning Research (10/2024)
From the table, can draw the following conchusions: (1) The results are consistent with our predictions. Using our framework to classify and detect errors without correcting the answers leads to significant drop in the performance of DR Baseline, while the performance of RC Baseline did not show much decline. DR Baseline show a drop of more than 10% in all metrics, espocially the Accuracy of VisualGLM, which drops by 25.05%. (2) This shows that our framework is very effective detecting hallucinations generated by models and correcting the model's answers.
A.2 Discussion on Reproducibility
We provide detailed discussion on the effect of our framework on correcting hallucinations and the repro- ducibility of the above experimental results. It should be emphasized that since our method is training-free, all parameters of the model are fixed. We focus on the following questions: (1) When LVLM repeatedly generates captions for the same image, will it produce the same hallucinations? (2) When using framework to process a series of model responses in (1), can we obtain consistent results? Can we guarantee that the hallucinations can be corrected every time?
We contimue to discuss the results of Appendix A.1 From the DR Baseline, we can see whether the same LVLM will repeatedly hallucinate the same image, and from the RC Bascline, we can see whether our framework's correction of hallucinations is repeatable. We analyze the responses of LVLMs and the results in Tab. 6 and arrive the following conclusions: (1) When the model parameters are fixed, it is very casy to hallucinate the same image repeatedly, long the same image and the prompt with the similar semantic are provided. This is also the reason for the poor performance of DR Baseline, as the same hallucinations repeatedly appears in multiple responses of LVLM. (2) Our framework is still able to find and correct the hallucinations generated by LVLM in the face of repeated tests, and the performance is almost the same as the previous experiment, that is, the performance of RC Baseline not significantly deviated from Tab. which shows that our framework very effective in detecting hallucinations generated by models and correcting hallucinations. Therefore, we believe that experimental results are reproducible and that the iallucination correction capability of our framework is reproducible.
A.3 Query Classification
The prompt template is in Fig.
A.4
Sub-questions Generation
The prompt template is in Fig. 6
A.5
Sub-answers Aggregation
The prompt template is in Fig.
A.6
CoT Verification
The prompt template is in Fig.
A.7
Verification Cycle
The prompt template is in Fig.
9
A.8
Results on MMBench
The results MMBench in Fig.
12
Fig.
13 and Fig.
14
17

---

Published in Transactions on Machine Learning Research (10/2024)
A.9
Prompt for GPT-4V- evaluation and GPT-3.5-aided precision calculation.
The prompl template for GPT-4V-aided evaluation is in Fig. T0 and the prompl template for GPT-3.5-aided
precision calculation is in Fig. II
A.10
Results of ablation study
The results of LL and VisualGLM are in Table 8 and Table g
A.11
Benchmark table
Benchmark details are in Table 10
A.12
Hallucination case
Cases of hallucination mitigation are in Fig.
15
A.13
ChatGPT cost
The cost of calling the GPT-3.5-turbb-000 API is shown in Table 11
We calculated the cost of calling the tp--3.-tturbo-6113 API when evaluating on MMBench. In one round of evaluation, the total cost was about $2.75, with an average cost of $0.0004 per question.
A.14
Model parameter
Details of LVLMs paramcters are shown in Table 12
A.15 GPT-4V-aided evaluation alignment method
When evaluating on LLaVA-QA90, we respectively pair baseline LVLMs with Dentist, bascline IVLMs with Woodpecker, and provided their responses to GPT-4V for scoring. Thus the scores need aligned. The alignment of the scores is followed: Suppose that when the responses of LVLMs and Dentist are provided to GPT-4V for scoring, their scores SBaseline-1 and Dentist respectively, and when LVLMs is paired with Woodpecker, their scores SBaseline-2 and Woodeecker respectivcly. The final aligned scores shown in Table 7
Table 7: The final aligned scores of GPT-4V-aided evaluation.
Score
LVLMS
S SBaeeline-1
Dentist
S Dentist
Woodpecker
Swodppecker X S Baseline-1 S Baseline-
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table 8: Results on MMBench with different variants of LLaVA
Variants
Perception Accuracy Reasoning Accuracy
Baseline
53.52%
50.13%
Dentist 56.83%
51.77%
Dentist/N 54.39%
48.76%
Dentist/P 57.90%
42.63%
Dentist/R 50.43%
52.11%
Table 9: Results on MMBench with different variants of VisualGLM
Variants
Perception Accuracy
Reasoning Accuracy
Bascline
32.31%
31.60%
Dent
36.35%
36.35%
Dentist/A
35.06%
28.73%
Dentist/A
37.83%
22.60%
Dentist/ R
28.26%
37.60%
Prompt
Role:
You are now one of my question classification assistants.
Please help me classify the question into two categories: perception reasoning(binary classification).
Rules:
1.ehe classification result is only "perception" or "reasoning". Choose one of the two to output.
2. the question focuses on perception ability, answer "perception"; the question focuses on logical reasoning ability, answer "reasoning".
3.Don't answer anything else, your answer can only contain 'perception" or "reasoning".
Examples:
1.my input: "How many people are there this picture?"
your answer: "perception'
2.my input: "The person in the picture may do what soon?" your answer: 'reasoning
(add more examples)
Now please classify the following question according the example and then answer "perception" "reasoning":
Figure 5: Prompt template for classification
19

---

Published in Transactions on Machine Learning Research (10/2024)
Prompt
Role:
You are my language assistant for generating sub-questions.
Please generate sub-question: to verify the caption of the picture based on QA- ex-exaeples below.
Rules:
1.ehe number of sub-questions cannot exceed three.
2. Extrract keywords such objects, quantities, and locations to generate sub-questions.
3.Each sub- qubettunt should have different focus.
4.Don't ask repeated questions in different sub-questions.
5.If my input contains multiple choice questions, please generate sub-questions based on the question, options and answers.
Examples:
1.my input:
"Question: Write detailed description for this picture.
Answer: The picture shows man standing on the back of the yellow taxi, with a yellow shirt and black pants, and blue backpack on his back. The taxi is driving on city street with cars and taxis in the background."
sub-questions you generated:
"1.1s there a man standing on the back of taxi in this picture? 2.What color are the T-shirt and pants that man wear?
3. J.Wat'ss the background?"
(add more examples)
Now please generate verification sub- cub-questions based on my input below:
Figure 6: Prompt template for generating sub-questions
Table 10: The number of questions in experiment
Benchmarks Questions
Tasks
LlaVA-QA90
90
generative
POPE
1800
classification
MMBench
6666
classification
CHAIR
2000
generative
Table 11: The cost of calling the GPT-3.5 --T-..5-turbb-063 API
Model (rice(input) Price(output)
$1.5 1M tokens
$2.0
1M tokens
Table 12: Details of the evaluated LVLMs.
LVLM
Overall Parameters
ILIP
8B
VisualGLM
8B
LLaVA
7.2B
20

---

Published in Transactions on Machine Learning Research (10/2024)
Prompt
Role:
You are my language assistant for correcting or remaining my passage.
Below is a passage and some Q&A pairs. You need to modify the passage or just keep unchanged based on the Q&A pairs.
Rules:
1.The information provided by the Q&A pairs is the ground truth, and the information in the passage may contain errors.
2.If the passage conflict with the Q&A pairs, find them and correct the passage based on the Q&AA pairs. Try to make minimal changes to retain the original sentence. Then give me the passage which have been corrected by you.
3.lf the passage has confliction with the Q&A pairs, just keep the original passage and give that.
4.At any time your output should only passage.
Examples:
1.Passage Theee are apples in the picture, they look stale."
Q&A pairs:
Q:How many apples are there in the picture? A:There are three apples in the picture. Q:Do these apples look fresh in the picture? A:No, they look stale. Your output."hert "There are three apples in the picture, they look stale."
(add more examples)
Now give you the passage and some Q&A pairs, please follow the examples and give me the passage you modified:
Figure 7: Prompt template for aggregating sub-answers to form the output after alleviating the hallucinatin tion
21

---

Published in Transactions on Machine Learning Research (10/2024)
Prompt
Role:
You are my language assistant for correcting or remaining my passage.
Below are two passages.
Rules:
T.e second passage the ground truth, the first passage may contain some errors.
2.1f the first passage conflict with the second passage, find them and correct the first passage based on the second passage. Try to make minimal changes to retain the original sentence. Then give me the first passage which has been corrected by you.
3.lf the first passage has no confliction with the second passage, just remain the first passage and give me that.
4.At any time your output should only be passage.
Examples:
1.The first passage: The profession of the man in the picture is a farmer." The second passage: "The man the picture is holding tools and there are many flowers and plants around him. So the profession of the man in the picture is gardener." Your output: "The profession of the man in the picture is gardener."
(add more examples)
Now give you two passages, please follow the rules and examples and give me your output:
Figure Prompt template for CoT verification
Prompt
Role:
You are my language assistant for determining whether there conflict between passages.
Rules:
1.Below are two passages.
2.If there conflicting content between the two passages, you should answer "yes"
3.1f there no conflicting content between the two passages, you should answer "no"
4.At any time You can only answer yes no.
Examples:
1.Passage 1: "There are two apples in the picture, they look stale." Passage 2: "There are three apples in the picture, they look stale."
Your answer: 'yes'
(add more examples)
Now give you two passages, please follow the examples and give me your answer about whether there is conflict between two passages.
Figure 9: Prompt template for judging when the validation cycle can be stopped
22

---

Published in Transactions on Machine Learning Research (10/2024)
Prompt
You are my scoring assistant. You need to score two passages describing the picture based on the content of the picture.
What you need pay special attention to is the hallucination, which refers the conflict between the content of the passages and the content of the picture.
For example, the passage incorrectly describes the shape or color of the object in the picture, or makes wrong inferences based on the content of the picture.
Please rate the two passages on scale of 10, where a higher score indicates better performance, according to the following criteria:
1. Accuracy: Refers to whether the description of the picture by the passage is accurate. Passages with fewer hallucinations should given higher scores.
2. Detailedness: Refers to whether the description of the picture is detailed in the passage. Note that descriptions with hallucinations are not counted. Passages with more details should be given higher scores.
3. Logicality: Refers whether the logical reasoning made by the passage based on the picture content is complex and reasonable. Note that the logical reasoning with hallucinations are not counted. Passages with more logical reasoning should be given higher scores.
Please output single line for each criterion, containing only two values indicating the scores for Passage and 2, respectively.
The two scores separated by space. Following the scores, please provide explanation of your evaluation, avoiding any potential bias and ensuring that the order which the responses were presented does not affect your judgment.
Passage (Original Answer)
Passage (Revised Answer)
Figure 10: Prompt template for GPT-4V-aided evaluation
23

---

Published in Transactions on Machine Learning Research (10/2024)
Prompt
You are my language assistant. You need to calculate the proportion of logical reasoning sentences included in the following two passages respectively.
Specifically, for each passage, you need to follow these steps to calculate the proportion:
Count the number all sentences in the passage.
2. Count the number logical reasoning sentences in the passage.
3. Divide the number of logical reasoning sentences by the number of all sentences to get the proportion.
Please output single line, containing only two values indicating the proportion for Passage and 2, respectively.
This means that you only output two values and not any other text analysis. The two values separated by space.
Passage 1: (Original Answer)
Passage 2: (Revised Answer)
Figure 11: Prompt template for GPT-3.5-aided precision calculation
24

---

Published in Transactions on Machine Learning Research (10/2024)
Future Prediction
Idemtity Reasoning
Function Reasoning
Image Emotion
Celebrity Recognition
Baseline InstructBlip Revised InstructBlip
Image Quality
Attribute Recognition
Image Scene
Action Recognition
$0.00
60.00
70.00
Image Style
10.00
20.00
30.00
40.00
Attribute Comparison
0.80
Structuralized Image-text Understanding
Image Topic
Natural
Sputial Relationship
Relation
Social
OCR
Relation
Reasoning
Physical
Object
Physical
Relation
Localization
Property
Reasoning
Reasoning
Figure 12: Results of InstructBLIP (Bascline and ours) across the 20 ability dimensions defined in MMBench. The blue area is the result of the baseline, and the red area is ours. See the legend. From this figure, intuitively see that our method can enhance the performance of baseline in terms of Image Impression, Image Quality and Future Prodiction, etc. For more comprehensive evaluation results on LLaVA and VisualGLM, please refer to Fig and Fig
25

---

Published in Transactions on Machine Learning Research (10/2024)
Future Prediction
Identity Reasoning
Function Reasoning
Image Emotion
Celebrity Recognition
Baseline -LaVA-V1.5-75
Revised LLaVA-V1.5-77
Image Quality
Attribtee Recognition
Image Scene
Action Recognition
0:00 0. 0 60.00 0.000.0 80.0000 P0.00..000
0.o0
00.00
Image Style
Attribute Comparison
Structuralized Image-text U Understanding
Image Topic
Natural
Spatial Relationship
Relation
Social Relation
OCR
Reasoning
Physical Relation
Object Localization
Physical Property Reasoning
Reasoning
Figure 13: Results of LLaVA on MMBench.
26

---

Published in Transactions on Machine Learning Research (10/2024)
Future Prediction
Identity Reasoning
Function Reasoning
Image Emotion
Celebrity Recognition
Baseline VisualGLM Revised VisualGLM
Image Quality
Auribute Recognition
Image Scene
Action Recognition
0..0 fo. 50.00000 6..00 70.00 80.00 90.00
Image
0:30 t0:00 10.00
Attribute Comparison
Style
Structuralized
Image
Image-text
Topic
Understanding
Natural
Spatial Relationship
Relation
Social
OCR
Relation
Reasoning
Physical Relation
Object Localization
Physical Property
Reasoning
Reasoning
Figure 14: Results of VisualGTMM MMBench.
27

---

Published in Transactions on Machine Learning Research (10/2024)
Please describe this image.
The image features entirely bare tree standing a grassy field. The field is full of weeds and flowers. The sun is setting behind the tree, casting warm glow on the landscape.
image features bare tree standing a grassy field. The field is full of weeds. The sun is setting behind the tree, casting a warm glow on the landscape.
What's the relationship between these two creatures?
The relationship between the two creatures can be described predator- eerattrr-rryyyy relationship.
The relationship between the two creatures, lion and leopard, can be described as competitive relationship.
Figure 15: Example images of our hallucination mitigation. The part of the generation that conflicts with the content of the picture has been corrected.
28


Reviews:
[{"review_id": "RUeqWWRFOs", "reviewer": "Reviewer_e5uA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- I like the idea of applying divide-and-conquer to hallucination mitigation. As the authors mentioned in the paper, there are various types of hallucinations and we cannot solve them with a single fixed method unless the framework internally processes hallucinations differently.\n- The authors provide all the details of the proposed framework, including the prompt templates used for the proposed framework, Dentist.\n- Overall, the paper is well-written and easy-to-read.\n- The superior experimental results show the validity of Dentist.\n\nWeaknesses: The quality of this work is overall good, but I have some concerns about the proposed framework and experiment designs:\n- In Section 3.3, the sub-questions are generated based on the model’s outputs. Thus, if the model does not describe salient features in the image at first, the proposed method cannot guide the model to capture them until the end.\n- The prompt template in Figure 10 for evaluating on LLaVA-QA90 needs two passages as input. How did the authors pair among the baseline LVLMs, Dentist and Woodpecker for getting the scores in Table 2? And why did the authors prompt ChatGPT to compare two passages and score them at once? Have the authors prompt ChatGPT to score a single passage from each of the models?\n- (Figure 10 and Table 2) For evaluating the rationality of inference content, the authors should also have checked the precision, that is the proportion of logical reasoning sentences included in the passage without hallucination, not just measuring the number of logical reasoning s without hallucinations (recall). Also, I think giving higher scores to the passages with more logical reasoning is quite advantageous to the methods that generate the whole reasoning process like CoT. Thus, I am not sure whether comparison with WoodPecker is fair using this metric because it does not use CoT. Lastly, regardless of whether generating reasoning process or not, I believe that the result of reasoning should be evaluated.\n- Please report the performance of WoodPecker on MMbench if possible. It would also be great to report the performance of other previous approaches on the benchmarks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses mitigating hallucinations in Large Visoin-Language Models (LVLMs), a critical issue hampering the performance and reliability of LVLMs. The authors observe that there have been no approaches using different mitigation methods for different types of hallucinations and using a iterative method to fully eliminate the hallucinations in the model's output. Based on this, the paper proposes a new unified framework, Dentist, which first classifies the input query into \"reasoning\" and \"perception\", and mitigates hallucinations using different methods for different types of queries in an iterative manner. The experimental results show the superiority of Dentist on several benchmarks, and the ablation study proves the validity of design choices.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer the above weaknesses. Also I have minor questions/recommendations:\n- The prompt template in Figure 5 does not use model responses, Y hat, while Equation 3 uses them. Is it intended?\n- It seems that the precision boost in LLaVA by Dentist on POPE, random setting is 16.92% points, not 14.83% (Section 4.2)\n- In Section 4.3, it would be great to provide the exact numbers of misclassification by GPT-3.5-turbo, not just mentioning \"We speculate that this is due to the misjudgment by GPT-3.5-turbo ...\".\n- How many iterations did the authors use for the experiments reported in Section 4.1 (T in Algorithm 1).\n- I don't understand the Direct Rejection Baseline (DR Baseline): what do the authors mean by “the sample is directly judged as an error.”? How are they treated when computing precision and recall?\n\nAnd please add a new section addressing the limitations of the proposed method and future work.", "broader_impact_concerns": "Please add the Broader Impact Statement in the final draft.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mrtIReqTrO", "reviewer": "Reviewer_H8tC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Hallucination is an important problem to address\n- The proposed solution is straightforward and does not require additional training or fine-tuning\n- Results show that the proposed method can improve performance over the baseline of querying the LVLM directly\n\nWeaknesses\n- Writing and presentation is poor and is inaccurate and confusing at places (see requested changes for details)\n- Not enough discussion of how Woodpecker is similar to and differs from the proposed framework\n- Some details of the method is missing and evaluation setup are missing\n  - Specifics of the GPT-3.5-turbo model used and cost is not specified. \n  - How is the Similar function in Algorithm 1 implemented? \n  - For MMBench, was the evaluation done with VanillaEval or CircularEval, on MMBench-dev or MMBench-test?\n  - Limited statistics are provided for each dataset that are used for evaluation and scattered and difficult to find.  A table will help. How large was MMBench?  It should be make clear that LLaVA-QA90 has \"90 questions\" not \"90 VQA tasks\".  \n- Lack of analysis on what specific hallucinations are actually reduced by the proposed method.    \n- No discussion of how reproducible the results are \n  - Will the LLM / LVLM give different responses when prompted again?\n  - The datasets used for evaluation are very small and it is unclear whether the experimental results are robust.  For instance, LLaVA-QA90 only has 90 questions over 30 images, and the CHAIR and POPE setting are also very small (50 images for CHAIR, 100 images with 6 questions for each sampling setting).  Can we expect results on such small datasets to be statistically meaningful and robust across runs?  \n- Missing discussion of recent work on halluciation mitigtation in LVLMs (see requested changes)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Paper summary\n- The paper proposes Dentist, a framework for mitigating hallucinations in large vision-language models (LVLMs) by iteratively using a LLM (along with the initial LVLM) to refine the initial LVLM response for VQA or image captioning.  Starting from an initial response (answer) from an LVLM for given query and image, The Dentist framework first takes the query and initial response and classifies whether it it likely to generate perception or reasoning hallucinations.  Based on the likely hallucination type, the framework then selects an appropriate strategy (either a perception-oriented strategy or a chain-of-thought prompting for reasoning) to revise the answer to mitigate the hallucination.  The framework iteratively revise the answer until the answer no longer changes significantly (or a maximum number of iterations is reached).  The components of the framework are all built by prompting ChatGPT.  Experiments on MMBench, LLaVA-QA90, CHAIR, POPE using several LVLMs (InstructBLIP, LLaVA, VisualGLM) show that the proposed framework can improve performance over just using the initial response from the LVLMs.  An ablation study was also conducted on MMBench with InstructBLIP.\n\nMain contributions are:\n- Dentist, a proposed frame for hallucination mitigation in LVLMS by using LLM to help refine the initial LVLM response\n- Set of experiments comparing the proposed framework against the baseline of direct querying of the LVLM, and Woodpecker (prior work that also used LLMs to correct the response).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Include missing details, discussion of similarity and differences to Woodpecker, discussion of reproducibility of results, and analysis of what hallucinations are reduced by the proposed framework (some more qualitative examples should be provided at least). \n- Writing and presentation improvements.  Below are some writing and presentation issues that have been noted.  These are not comprehensive and meant to just be some examples.\n  1. The introduction should be improved to clearly state the tasks being addressed (e.g. VQA and image captioning), and the expected inputs.  It is not clear until section 3 and Figures 2, what the problem setup is is, what are components vs general descriptions, and the inputs and outputs to the framework and its components.      \n\n  2. Some sentences are overly long, confusing, or difficult to parse.  A trimming and re-wording pass is recommended to make the writing more straightforward, clear and easier to understand.  Some examples are given here:\n     a. \"the effect of a method that uses object detection on pictures to verify whether the object in the answer exists will be reduced when the query is the reasoning type\" => the clauses in this sentence should be restructured so it is easier to follow.  it's unclear whether this sentence is trying to say \"for reasoning queries, it is not effective to use object detection on pictures to verify whether the object in the answer exists\" or something else.\n     \n     b. \"Potential hallucination classification divides the query into two categories: perception and reasoning, which also classifies potential hallucinations caused by these queries\"\n        - At this point, it is not clear what is meant by a \"query\" and why that is the appropriate thing to classify (vs using the generated response from the LVLM).  Later in section 3, it becomes more clear what a query is, and from 3.2 (equation 3), it becomes clear that the query and response are used together to determine classification type.\n        - What does \"which also classifies potential hallucinations caused by these queries\" mean?  \n        - It is also good to either introduce the components before hand (\"Potential hallucination classification\") or use typesetting (e.g. italics) to indicate that \"Potential halluciation classification\" is a component in the system (vs just a general concept) \n     \n     c. \"with a detailed superiority analysis\" => superiority analysis does not seem appropriate, please drop this phrase\n     d. It is confusing to have statements that refer to experiments on \"CHAIR\" or \"POPE\" as CHAIR is a metric and POPE is a evaluation strategy\n\n  3. Provide more concrete examples of prompt and response in the main paper\n\n  4. Tone down inaccurate claims \n     - \"we propose a unified framework for all kinds of hallucination mitigation\" => only two types of hallucination is identified and there are just two types of mitigation strategies\n     - \"We conduct massive experiments\" => As noted, the scale of the evaluation datasets are very small.  The claim of \"massive experiments\" are justified.\n\n  5. Other writing issues and clarity improvements:\n     a. Figure 2 improvements \n      - Clearly indicate that the \"potential hallucination classification\" module takes the answer and query as input (currently it is unclear).  \n      - The caption should explain what part uses the LVLM being studied and which part uses ChatGPT.\n      - The figure can either be made more compact or provide example of actual input / output to different components\n     b. Section 3.1: \"Noting i\" => \"Note i\" (i should be typeset in math mode), \"... and we omit\" => it is better to separate into to sentences (\"We omit...\")\n     c. Section 4.1: Use of the word \"dimension\" for the different levels of MMBench.  The MMBench levels are hierarchical.  The word \"dimension\" implies separate axis/dimensions.  That is not the case.  It is also confusing to say that \"we conduct experiments under the setting of the L-3 level abilities\", and for clear to indicate that you \"report evaluation for the most fine-grained L-3 level\".  As the levels are hierarchical, the same answers that are used to obtain the L-3 evaluation results can also be used to obtain the L-2 and L-1 level results.  \n     d. Table 1 improvements \n       -The highest model should be consistently bolded, not just when it is Dentist.  Stylistically, I would recommend removing the % and reducing the number of horizontal lines (only the horizontal line before Overall is needed).  \n       - Should include Woodpecker for comparison\n       - It would be better if the L-3 abiltiies are grouped by the hierarchy (as originally intended by MMBench) vs alphabetically.  It's also unclear if it is necessarily to have the L-3 granularity for the results.  L-2 granularity would be sufficient\n\n     e. Section 4.1: \"we also introduce the baseline Woodpecker\" => \"we compare against the baseline Woodpecker\" (Woodpecker is prior work from Yin et al. 2023, and not introduced by this work)\n     Citation for GPT-3.5-turbo should not be Brown et al. 2020, as it is built with additional work that follows Brown et al. 2020. and the specific version used should be specified.  \n\n     d. Table 4. Please add horizontal lines grouping the LVLMs.  The bolding should be per LVLM.\n     e. Figure 4. Improve caption to provide more details about the two examples.\n     f. Minor: Section 2.2 \"hires an LLM\" => \"uses an LLM\" \n- Include Discussion of recent relevant work. Some examples (not comprehensive) are given\n  1. Hallucination mitigation for LVLMs.    \n  - Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding [Leng et al. CVPR 2024]\n  - Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation [Huang et al. CVPR 2024]\n  - RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback [Yu et al. CVPR 2024]\n  - Hallucination of Multimodal Large Language Models: A Survey [Bai et al. 2024] can potentially be good reference for relevant work\n  2. Using LLMs to improve task performance of VLMs \n  - Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions [Wang et al. 2023]\n  - Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering [Awal et al. 2024]", "broader_impact_concerns": "No Broader Impact Statement is included.  A Broader Impact Statement is recommended to indicate that mitigating hallucination can help the trustworthiness of the LVLM, but hallucinations can still occur.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tdai7Nn3Oa", "reviewer": "Reviewer_3BEp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces a simple but effective framework for mitigating inference on VLLMs.\n2. The paper is well-written and easy to follow.\n3. The experimental setup is sound with a comprehensive range of tasks and baselines. \n\nWeaknesses:\n\n1. The methodological contribution is limited. The contribution is a prompting framework for setting up a verification loop for visual understanding tasks. Though the methodological contribution is limited, the results are compelling.\n2. Time complexity: The approach adds a large time and cost overhead for every VLLM inference as running the classification and the verification loop requires multiple LLM calls to an LLM.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a prompting framework for mitigating hallucinations in VLLMs. They accomplish this by segregating reasoning errors from perception errors and then addressing them by querying either ChatGPT or a VLLM with revised prompts. They validate their approach on various visual understanding tasks (MMbench, POPE, CHAIR, and LLaVA-QA90) and present competitive baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could you present an ensemble baseline where the VLLM is called multiple times, and for each generation, we leverage your classification to detect if there are any errors? For evaluation, if all the generations have errors, deem the sample incorrect, else randomly sample one of the generations where neither a perception nor a reasoning error is detected.\n\nThis baseline aims to understand if Dentist is the best framework for leveraging additional LLM/VLLM calls.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you to the reviewers for their reviews and improvement suggestions, and thank you to the authors for their patience with the delays during the summer break.

Overall, the decision is to accept the paper with minor revisions.

This paper presents a framework for mitigating hallucinations in large vision-language models (LVLMs) through an iterative prompting approach. The key idea is to classify queries as either perception or reasoning related and then apply targeted iterated strategies to refine the model's output. The authors evaluate their approach on several benchmarks and demonstrate improvements over baseline LVLMs and previous hallucination correction methods. The authors have addressed concerns raised by the reviewers in their revision, including clarifying missing details on their experimental setup and expanding their empirical evaluation and literature review, as well as fixing writing issues and improving the presentation.

The following minor revisions are recommended/needed based on comments from the reviewer H8tC's recommendations (which are not visible to the authors):

1. I'm not sure whether the final "Response to Reviewer e5uA" clarifying how the scoring of the baseline LVLM for two methods (Dentist and Woodpecker) are aligned is incorporated into the manuscript. It would be good if that was included.

2. In addition, some parts of the manuscript is unclear (due to wording). There should be a pass over the text to improve the wording. Some suggestions are given below. A more complete proofreading pass to improve wording is recommended.
Writing improvements

3. Please reword to be more accurate:
  - Page 13: "To evaluate the effectiveness of our framework, We conduct massive experiments" => "To evaluate the effectiveness of our framework, we conduct experiments"
4. Please reword to be clearer:
  - Page 6: "Regarding automatically determining whether the answer is no longer changing semantically, we use a prompt to make ChatGPT automatically complete it instead of manual work. In Algorithm 1, the Similar function represents this prompt. Refer to Appendix A.7 for the prompt template." => Please reword to be clearer. For example: "We use ChatGPT to determine whether the answer has converged and is no longer changing semantically. This corresponds to the Similar function in Algorithm 1 and the prompt template is given in Appendix A.7".
  - Page 8: "In Section 3.4, we mention that Dentist adopt validation loops." => cut or reword to be clearer
  - Page 9: "promotes the detail of image description and the rationality of inference content" => please reword
  - Page 9: "can increase effective reasoning content" => please reword
5. Logically, it would be better if "Limitations And Future Work" (Section 6) comes before "Conclusion" (Section 5)
6. Other recommended text rewordings:
  - Page 4: Figure 2 caption: "The orange square components use gpt." => "The components using GPT are indicated in orange."
  - Page 7: "CHAIRi" => "CHAIR_i"
  - Page 8: "Dentristry" => "Dentist" \
    "Limitations And Future Work" => "Limitations and Future Work" \
    "Discussion On Reproductivity" => "Discussion on Reproducibility"

Additionally, the following comments from my own reading of the current revision:
1. " any part of the answer that does not match the content of the picture will be corrected by our framework Dentist." => This is too strong a claim, please tone down.
2. "widely used method dedicated to evaluating" => "evaluation method"
3. "w/o-Per/Rea/Cla" => please use clearer names for the ablations - the table is narrow currently and has sufficient horizontal space to improve readability
4. The authors might want to more explicitly clarify that their verification loop is a fixed point iteration and maybe refer to the theory as future work?
4. Why "dentist"? The paper does not explain the name at all (except for a somewhat strange metaphor). It might be worth choosing a meaningful acronym. 
5. In general, edit phrases to be more concise and plain and thus avoid unclear wording. This will significantly help non-native readers. 

Congratulations on the acceptance, and I'm looking forward to the camera-ready revision.

Best wishes,\
Andreas
Decision: Accept (poster)