Toward a Stable, Fair, and Comprehensive Evaluation of Object Hallucination in Large Vision-Language Models
Authors: Hongliang Wei
Xingtao Wang
Xianqi Zhang
Xiaopeng Fan
Debin Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Vision-Language Models
Multimodal large language models
Multimodal
Object hallucination
Evaluation
Image caption
Primary area: other
TL;DR: The paper reveals  the correlation between the object hallucination degree and the length of image descriptions, and proposes a stable and comprehensive framework for evaluating object hallucination in large vision-language models..

Abstract: Given different instructions, large vision-language models (LVLMs) exhibit different degrees of object hallucinations, posing a significant challenge to the evaluation of object hallucinations. Overcoming this challenge, existing object hallucination evaluation methods average the results obtained from a set of instructions. However, these methods fail to provide consistent evaluation across instruction sets that generate image descriptions of significantly different lengths. In this paper, we present the first systematic investigation of the effect of instructions on object hallucinations in LVLMs, with a specific focus on the role played by image description lengths. A valuable finding is that instructions indirectly affect hallucinations through the length of image descriptions. The longer the image description, the higher the object hallucination degree. Accordingly, we fit an informative length-hallucination curve, upon which a fine-grained evaluation framework named LeHaCE is introduced for evaluating object hallucinations at any given image description length. LeHaCE evaluates the object hallucination degree at a uniform image description length to mitigate the effect of description lengths, promoting stability and fairness. Moreover, LeHaCE incorporates the curve slope as an innovative hallucination evaluation metric, reflecting the extent to which the object hallucination degree is affected by the image description length, achieving a more comprehensive evaluation. Experimental results demonstrate that LeHaCE provides a more stable, fair, and comprehensive evaluation of object hallucinations in LVLMs compared to existing methods.

Toward a Stable, Fair, and Comprehensive Evaluation of Object Hallucination in Large Vision-Language Models
Hongliang Wei, Xingtao Wang', Xianqi Zhang, Xiaopeng Fan, Debin Zhao Harbin Institute of Technology Harbin, China
Abstract
Given different instructions, large vision-language models (LVLMs) exhibit differ- ent degrees of object hallucinations, posing a significant challenge the evaluation of object hallucinations. Overcoming this challenge, existing object hallucina- tion evaluation methods average the results obtained from a set of instructions. However, these methods fail to provide consistent evaluation across instruction that generate image descriptions of significantly different lengths. In this paper, we present the first systematic investigation into the effect of instructions on object hallucinations in LVLMss, with a specific focus on the role played by image description lengths. A valuable finding is that instructions indirectly affect hallucinations through the length image descriptions. The longer the image description, the higher the object hallucination degree. Accordingly, we fit an infor- mative eength-hallucination curve, upon which a tine-grained evaluation framework named LeHaCE is introduced for evaluating object ballucinations at any given image description length. LeHaCE evaluates the object hallucination degree at uniform image description length to mitigate the effect of description lengths, promoting stability and fairness. Moreover, LeHaCE incorporates the curve slope innovative hallucination evaluation metric, reflecting the extent to which the object hallucination degree is affected by the image description length, achieving a more comprehensive evaluation. Experimental results demonstrate that LeHaCE provides a more stable, fair, and comprehensive evaluation of object hallucinations in LVLMs compared to existing methods.
1 Introduction
Drawing inspiration from the remarkable language capabilities exhibited by large language models (LLMs) (1-3), large
4 7 have been well-developed, achieving significant advancements in complex multimodal tasks. However, the practical application of LVLMS is heavily hindered by hallucination phenomena g which refer to situations where objects in image descriptions generated by LVL.Ms are inconsistent with the provided visual content. Consider- able efforts have been dedicated both evaluation 9 and mitigation [2 of hallucination phenomena, leading to notable advancements.
models (LVLMs) 2
A significant challenge in object hallucination evaluation arises from the effect of instructions on object hallucinations [9] Overcoming this challenge, existing object hallucination evaluation methods typically adopt average-based framework, which averages the results obtained from set of instructions. However, as shown in Figure [ this framework fails to provide consistent evaluation across instruction sets that generate image descriptions of significantly varying lengths. Specifically,
"Coresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Gunde-ha- Nodee-
QueaTE
Figure The evaluation results LVLMs on four instruction sets using the CHAIR with the average-based framework. Length refers to the average length of generated image descriptions. Each instruction set consists of six distinct instructions, and there is no overlap between instructions in different sets. All instructions prompt LVLMs to describe the image.
while evaluation results of LVLMs remain consistent across certain instruction sets (e.g., set I and set 2), inconsistencies arise when comparing instruction sets with significantly different average image description lengths (e.g., set 2 and set 3).
In this paper, we present the first systematic investigation into the effect of instructions on object hallucinations in LVLMs. with a specific focus on the role played by the length of image descriptions. Technically, we evaluate lengths and object hallucination degrees (measured by CHAIR scores) of the image descriptions generated by LVLMs under different instructions (see Section 3.1 for more details). The experimental results are shown in Figures 2 & 3 from which we can observe that the degree of object hallucination is primarily influenced by the length of image descriptions, with instructions only indirectly affecting hallucinations through their effect description lengths. The longer the image description, the higher the object hallucination degree, and there a clear linear relation between them. Hence, it is imperative to take into account the length of image descriptions in hallucination evaluation. Unfortunately, the average-base basedd framework can only select instructions, without the ability to directly control the length of image descriptions.
Motivated by the findings, we propose fine- fine-grained evaluation framework called LeHaCE, which fits an informative length-ballucination curve to evaluate object hallucinations at any given image description length within large range. LeHaCE evaluates the object hallucination degree uniform image description length to mitigate the effect of image description length, ensuring stable evaluations for the same LVLM across different instruction sets and fair comparisons among different LVLMs. Moreover, LeHaCE incorporates the curve slope as an innovative hallucination evaluation metric, reflecting the extent to which the object hallucination degree is influenced by the image de- scription length, achieving more comprehensive evaluation. Experiment results 12 representative LVLMs show that LeHaCE can evaluate object hallucinations LVLMs in a more stable, fair, and comprehensive way.
The main contributions of this paper are summarized as follows:
We conduct the first systematic investigation into the effect of instructions object halluci- nations in LVLMSs and find that the degree of object hallucinations is primarily influenced by the length of image descriptions, with instructions only indirectly affecting hallucinations through their effect on image description lengths.
We propose an object hallucination evaluation framework called LeHaCE, which fits an informative hallucaaalliinnooonn curve to evaluate object hallucination at a uniform image description length, realizing more stable and fair evaluation.
We employ the curve slope as an innovative hallucination evaluation metric, reflecting the extent to which the object hallucination degree is affected by the image description length, achieving a more comprchensive evaluation.

---

2 Related work
2.1 Large Vision-Language Models
Inspired by the success of LLMs in NLP 133 researchers have extended LLMs to multimodal
tasks [1528]] proposing numerous LVLMs and achieving new advancements 14 29/35 These
LVLMs align the multi-modal encoders with LLM through multitask fine-tuning and instruction
fine-tuning on multi-modal datasets, enabling LLM to acquire multi-modal perception and instruction- following capabilities. Specifically, to integrate multimodal features, Flamingo (2 proposes cross-attention structure to achieve arbitrary interleaved multi-modal feature fusion. BLIP-2 [35] introduces Q-Former to bridge the visual backbone model and LLM. mPLUG-Ow12 [36] introduces a modality adaptive module to facilitate the fusion between different modules. To enhance generaliza- tion and improve mnstruction-followee fine-tuning and instruction fine-tuning for LVLMs. Among them, LLV-instruction [2), MiniGPT-4 [5] LLaVA 6 and SVIT [7 employ ChatGPT to augment instruction data. To mitigate the risk of catastrophic forgetting of language knowledge during the training process, mPLUG-OW/ (8] and
capabilities, some methods 4 12 37 propose multi-task
VL. 66] perform joint training on pure language and visual-language instructional data. More recently, mPLUG-DocOw (31)
(32), Kosmos-2 (33, Shikra [34]. Cantor further enhance the capabilities of LVLMS in optical character
39.. BuboGPT (30,, and Qwen-VL 7
recognition, document understanding, multi-modal interleaved composition and visual grounding.
2.2
Hallucination in LVLMS
Works on the hallucination in LVLMs focus on two aspects: evaluation and mitigation. For the hallu- cination evaluation, POPE [9] designs a polling-based query method to avoid the influence of instruc- tions on hallucination evaluation. By presenting LVLMs with brief "yes" or "no" questions regarding the target of detection, the evaluation of hallucination is transformed into a simple binary classification task. NOPE [T0 designs a novel benchmark to evaluate the performance of LVLMs in recognizing the non-existence of objects in visual questions. AMBER [1]] designs milti--immensionll LVI.Ms hallu- cination evaluation benchmark without L.LMs, targeting existence, attribute, and relation hallucination. For the hallucination mitigation, L.RV-Instruction [40] creates balanced set of positive and negative instructions to perform robust visual instruction adjustment for LVLMs. VIGC [44 employs an itera- tive approach to generate detailed and accurate answers gradually. Woodpecker I3 proposes post- processing method that utilizes expert models to locate and correct hallucinations from generated text.
While existing methods [9] observe that the hallucination degree of LVLMs is unstable across different instructions, this phenomenon has not been thoroughly investigated to date. This work presents the first comprehensive study investigating the influence of instructions on the hallucination rate of LVI.Ms. Building upon our findings, we propose LeHaCE framework, which can evaluate hallucination of LVLMs in more stable and comprehensive manner. Contrary to polling-bane -ased query methods, LeHaCE can directly evaluate the hallucination rate of image descriptions generated by LVLMs, which is more in line with the practical application scenarios of LVLMs.
3 Hallucination of LVLMs Under Different Instructions
This section provides the investigation into the effect of instructions on hallucinations, with a specific focus on the role played by the length of image descriptions. The experimental settings are presented initially, followed by a comprehensive analysis of the experimental results
3.1 Experimental Settings
In this investigation, twelve popular LVLMs are included, namely Gemini-Pro- Vision pro 2 Qwen-
VL [7], MiniGPT-4 [5) LLaVA (6) InstructBLIP 4. LLaMA-Adapters 41
mPLUG-Ow Ow012
36) mPLUG-OWl 38), MtterLLM-XComponn
[32]. VPGTrans [42]
Otter 43 and Lynx 44 All
LVLMs are prompted by 25 different instructions to generate image descriptions for 256 images
in MSCOCO [5). All descriptions are generated using beam search with beam size of 5. For the instructions, we utilize those from 4 and additionally propose several others, as detailed in the appendix. We use CHAIR S as the evaluation metric for hallucinations, which has two variants:
CHAIR and CHAIRs. Given the ground truth objects in the image, CHAIR, calculates the proportion
3

---

Dive
UNU Ceulloe bared
Ovee
Se
E1-46
llalt Alpphia
PEE
aod
VPCchae
#FLEG-ON22
1226
CHAR
Fo Tioc
231
Canate Uie Stice
VPL Frem
#Piit-dee
1,/m
and 2n
Figure 2: Scatter plots of CHAIR scores and average lengths of the 25 sets of image descriptions generated by 25 instructions. r denotes the Pearson correlation coefficient between the hallucination rates and the average image description lengths, R² and P represent the coefficient of determination and p-value respectively for the linear regression.
of objects that appear in the descriptions but not in the image, while CHAIRs is the proportion of descriptions that include hallucination. Formally, CHAIR, and CHAIRs can expressed as follows:
/hallucinated objects}
CHAIR,
[all mentioned objects)
|
descriitionss with hallucinated objects}|
CHAIR
{ all descriptions)/
For more experimental settings, we use the Pearson correlation coefficient to measure the correlation between the average length and the hallucination rate of image descriptions. Lengths are measured in word count.
3.2 Experimental Analysis
The results are presented in Figure & 3 from which we get two key observations: 1) Fig- ure 2 shows the relationship between the hallucination rate and the average image description length, we can observe that the hallucination rate increases with the average image description length and there is clear linear correlation between them. Specifically, the Pearson correla- tion coefficient between hallucination rates and the average image description lengths exceeds 0.6 for all LVLMs, with 10 LVLMs exceeding 0.8 and 5 LVLMs exceeding 0.9. 2) Figure shows the impact of instructions on the length of image descriptions generated by LVLMs, from which observe that the length of image descriptions generated by the same LVLM with

---

different instructions can vary significantly, e.g., Gemini-Pro Vision with Instruction 11 (101 words in average) v.s. Gemini-Pro-Visse 18 (20 words in average). Furthermore, the length of image descriptions generated by different LVLMs with the same instruction can also differ greatly, e.g.. MiniGPT-4 (11 words in average) v.s. Gemini-Pro- Vision (97 words average) with Instruction 17.
100
with Instruction
100
100
100
180
Based on the aforementioned two observations, we can draw the following conclusions: 1) The degree of object hallucinations is primarily influenced by the length of image descriptions, with instructions only indirectly affecting hallucinations through their effect on image description lengths. Hence, is imperative to take into account the length of image descriptions in hallucination evaluation. However, controlling the length of image descriptions generated by LVLMs is challenging P given that even subtle semantic differ- ences between instructions can significantly impact the output length of LVLMs (shown in Figure B 2) In addition to the hallucination degree. the rate which hallucination degree increase with description length is also a meaningful indicator for char- acterizing the nature of LVLMs hallucinations. Considering both the hallucination degree and the growth rate of hallucination degree can provide a more comprehensive evaluation for hallucinations LVLMs. For example, as shown in the Figure although InstructBLIP has the lowest hallucination degrees in short image descriptions, it exhibits high instability with a rapid increase in hallucination degrees, resulting in high hallucination in long image descriptions.
11
46
11
Mostinta
LZ4is
Cort-
Isted MA
Grobed
eit
Figure 3: The average lengths of image descriptions generated by LVLMs when prompted by different instructions.
Length-Hallucinatonn Curve Based Hallucination Evaluation Framework
In this section, we first introduce the average-baset hallucination evaluation framework and discuss its limitations. Then, we elaborate on the proposed LeHaCE framework and evaluate representative LVLMs with LeHaCE. Finally, the stability of LeHaCE is analysed.
4.1 Average-Based Hallucination Evaluation Framework
The average-based hallucination evaluation framework mitigates the challenge caused by instructions by averaging the hallucination rates over different instructions. Formally, the hallucination rates and average lengths of the image descriptions generated by the LVLM under N instructions are denoted as (e,, )= The average hallucination rate hr and average length & of image descriptions over all instructions can be calculated as follows: hr = 1/2 >yE/n hr, and = * N
4. The werage-based hallucination evaluation framework utilizes for to evaluate the hallucination of LVLMs.
However, due to substantial variations in the average lengths of image descriptions generated by different instruction sets, the average-based framework struggles to mitigate the effect of image description lengths on object hallucinations. resulting in unstable and unfair evaluations. Specifically, (left), as shown in Figure (left), when the average-basse framework evaluates an LVLM under different instruction sets, the inconsistent average image description lengths lead to unstable evaluation. More- over, Figure (right) shows that when the average- based framework evaluates different LVLMS under the same instructions, the inconsistent average image description lengths lead to unfair evaluation.
4.2 Length-Hallucinationn Curve Based Hallucination Evaluation Framework
Section 3.2 reveals the significant effect of image description lengths on the hallucination degree. To mitigate this effect, it is crucial control the description length during the hallucination evaluation.
"This work does not consider controlling length by truncating the generated descriptions, only considering cases where LVLMs generate complete image descriptions, as this better fits practical application scenarios.
5

---

22.5
LLaVA CHMMI Scerason Sot
22,5
ULAV CRUM Scores
LLaVA CHSM Scoreson Ser
IEEECG-O CHMNN Scores
20.0
20.0
LaMtycE Souse
Slope: 0.15
aequal EY DICCRAe
17.5
lengths
17.3
CHAIR:
15.0
CRAIX
15.0
12.5
Stope:
10.0
10.0
7.5
7.5
LeIlaCE
Linequal averraet lengitis milke ABF unfair
LeHaCE campare sarme langht
20
30
40
50
60
70
90
100
Average Description Length (wards)
Average Description Length (words)
Figure 4: Illustrations of the average- sarraee-basedeee evaluation framework (ABF) and our LeHaCE framework. The left figure presents the object hallucination evaluation of LLaVa on two instruction sets. The right figure presents the object hallucination evaluation of I.LaVa and mPL.U-OWl the same set of instructions.
However, controlling the length generated image descriptions is challenging because LVLMs are highly sensitive instructions. To address this challenge, we fit length-hallucination curve to evaluate LVLMs at any desired length. Specifically, based on the clear linear correlation observed in Section 3.2 we assume linear correlation between image description lengths and hallucination rates of LVLMs. Figure intuitivelly illustrates the LeHaCE framework.
Formally, we use {4,, ) INN to represent the average lengths and hallucination rates of image descriptions generated by the LVLM under N instructions. The linear regression curve of { hr, ) which we refer as the Length-Hallucinatinn Curve (LHC), can be formalized follows:
LHC (L) +
(3)
where B and a are:
1= (4, -2) (hr: hr
(4)
Σ (e+ 8)²
hr (3 e)
(5)
mongth-aallucination curve summarizes the trend between the hallucination rate and the description length. The regression coefficient 3 represents the rate at which the hallucination rate increases with the growth of description length. LeHaCE uses the LHC evaluate the hallucination in LVLMs. LeHaCE consists of two metrics:
LeHaCE (f) (H),
LeHaCEGR = 3
(7)
where LeHaCE (c) measures the hallucination rate the specified length é and LeHaCEGR measures the rate at which the hallucination rate increases with the increase in description length. Note that LeHaCE can be built upon any hallucination degree evaluation metric, enhancing their stability, fairness, and comprehensiveness. In this paper, we use CHAIR as the metric for measuring the hallucination degree.
Compared to the weraage-bsedd hallucination evaluation framework, LeHaCE has three advantages. As intuitively shown in Figure 4 LeHaCE can evaluate the hallucination degree of LVLMS uni- form image description length, thereby mitigating the influence of description length on hallucination degree and resulting hallucination degree at multiple lengths and the growth rate of hallucination degree, leading to more comprehensive evaluation.
more stable and fair evaluation. Moreover, LeHaCE can evaluate the
4.3 Evaluation on MSCOCO and NoCaps
We evaluate twelve LVLMS with LeHaCE lengths of 20, 40, 60, and 80 words. This evaluation conducted on subsets of the MSCOCO [45] test set and the NoCaps [46] validation set, each compris-

---

Model
(20)
Lc,(40)
(c,600
(c,(80)
LGaa Lc,(20) Lc,(40)
(60
Lc,(80) Le,
MSCOCO
MiniGPT-4
5.33
6.66
7.98
9.31
0.07
9.27
15.71
22.15
28.59
0.32
InstructBLIP
2.35
5.10
7.86
10.61
0.14
5.61
16.24
26.87
37.50
0.53
Lynx
3.26
6.49
9.72
12.95
0.16
8,00
17.48
26.97
36.46
0.47
LLaVA
7.22
8,30
9.38
10.46
0.05
14.48
20.31
26.14
31.97
0.29
Otter
8.76
12.66
0.19
15.31
88
VPGTTrans
5.77
6.87
7.97
9.08
0.06
9.08
15.01
20.94
26.86
0.30
LLaMA- LLaMA-Adapter-22
6.04
9.29
12.54
15.80
0.16
11.31
22.99
34,66
46.34
0.58
mPLUG-Owl
7.15
10.84
14.52
18.20
0.18
11.18
23.71
36.25
48.79
Gemini-Pro- Vision
4.30
6.15
7.07
0.05
8.00
12.61
17.22
21.83
0.23
InternI M-XComposrr
5.40
7.82
10.25
12.67
0.12
9.48
19.18
28.88
38.58
0.48
Qwen- VL mPLUG-Ow12
3.44
5.36
2.28
9.20
0.10
6.15
15.31
24.47
33.63
0.46
3.92
7.39
10.86
14.33
0.17
8.19
21.66
35.12
48.59
0.67
NoCaps
MiniGPT-4
14.53
16.79
19.05
21.30
0.11
23.75
35.75
47.76
59.77
0.60
InstrucUBLIP
6.52
10.20
13.88
17.56
0.18
13.33
26.39
39.45
52.50
0.65
Lynx
13.79
17.18
20.55
23.96
0.17
36.07
46.11
56.16
66.21
0.50
LLaVA
14.48
0.09
24.15
43.66
53.42
Ofter
15.49
19.03
22.58
26.12
0.18
25.38
38.89
52.40
65.91
0.68
VPGTrans
12.51
14.39
16.26
18.14
0.09
20.39
31.95
43.51
$5.07
0.58
LLaMA-Adapler-22
12.52
16.07
19.62
23.17
0.18
22.44
35.31
48.18
61.04
mPLUG--Wl
12.85
15.84
18.84
21.83
0.15
19.77
30,68
41.60
52.52
0.55
Gemini-PPo Pro- Vision
12.76
15.17
17.57
19.98
0.12
22.63
34.56
46.50
58.44
0.60
InternL IntemLM-XCooposer
10.93
12.74
14.54
16.34
0.09
20.12
31.22
42.33
53.44
0.56
Qwen-VL
8.37
10.69
13.01
15.33
0.12
14.15
25.00
35.85
46.71
0.54
mPLUG-Ow12
6.91
10.82
14.72
18.63
0.20
11.72
25.45
39.17
52.90
0.69
Table I: LeHaCE scores of LVLMs the MSCOCO and NoCaps datasets. LG, and Lcs represent CHAIR, and CHAIRs with the LeHaCE framework. The best result on each metric for each dataset represented in bold, and the second best result is indicated with an underline.
ing randomly selected 256 images. The length hallucination curve in LeHaCE is fitted the CHAIR scores of image descriptions generated by 25 different instructions. To calculate CHAIR scores on No- Caps, we follow the setting proposed in [8 47). All descriptions generated using beam search with a beam size 5. The experiments are conducted with PyTorch on Nvidia GeForce RTX 3090 GPUs.
The results are shown in Table I which demonstrate that LeHaCE can evaluate the object hallucination degree of LVLMs given image description lengths, as well as the growth rate of the hallucination degree, providing fair and comprehensive evaluation. Specifically, 1) For short descriptions, InstructBIIIP achieves best performance on both the MSCOCO and NoCaps datasets. However, its higher growth rate of hallucination degree leads to poor performance on longer descriptions. 2) For medium-length and long descriptions, Gemini-Pro- Proviii-roo-vision and Qwen- VI. exhibit the best performance relatively small growth rate hallucination degree. 3) Gemini- ProiiiPrr--oi Vision and LLaVA exhibit the lowest growth rate in hallucination degree on the MSCOCO and NoCaps datasets, respectively.
the MSCOCO and NoCaps datasets, respectively. This is attributed their
In Table T LVLMs exhibit higher degrees of hallucination on the NoCaps dataset compared the MSCOCO dataset. This is attributed to the fact that LVLMs typically use the MSCOCO for training, making the NoCaps dataset an out-of-distribution dataset. The results show that the distributional differences not only increase the hallucination degree of LVLMMs at various description lengths but also amplify the growth rate of hallucination degree.
4.4 Stability of LeHaCE
As mentioned above, LeHaCE evaluates the hallucination degree of LVLMs in a more stable manner. This subsection verifies the stability of the proposed LeHaCE framework. Specifically, LVLMs are prompted by three sets of different instructions to generate three sets of image descriptions. Each instruction set consists of multiple instructions randomly drawn from pool of 25 instructions, with no overlap between instructions in different sets. The image descriptions generated by different instructions in each set are evaluated using the LeHaCE framework and the average-based framework, respectively. The stability of the LeHaCE and the average-based frameworks on the three sets of image descriptions evaluated using the Relative Standard Deviation (RSD), which defined as the ratio of the standard deviation to the mean RDS o/p. J. The lower the RSD, the more stable
7

---

MSCOCO
Gemini-Pro Vision
Qwen- VL
MiniGPT-4
LLaVA
# of Ins
C₁
LG
Cs
lc.
C₁
1x,
Cs
lc.
C2
Lc
Cs
Lc.
Ci
14,
C2
1.c,
3
0.16
0.25
0.32
0.18
0.29
0.11
0.41
0.09
0.14
0.09
0.22
0.07
0.16
0.23
0.29
0.23
0.14
0.30
0.13
0.38
0.08
0.21
0.13
0.20
0.18
5
0.16
0.15
0.27
0.12
0.22
0.08
0.34
0.05
0.08
0.06
0.15
0.05
0.10
0,08
0.20
0.08
0.13
0.09
0.24
0.10
0.21
0.07
0.32
0.06
0.10
0.04
0.16
0.04
0.10
0.07
0.19
0.06
0.15
0.22
0.11
0.06
0.25
0.06
0.15
0.10
0.06
0.07
8
0.12
0.09
0.20
0.08
0.15
0.06
0.22
0.04
0.06
0.04
0.12
0.03
0.08
0.06
0.15
0.06
InternL
Otter
LLaMA-Adapier--2
mPLUG-Owl
C2
LG
Cs
Lc,
C
C3
Lc,
C1
LG
C3
LG,
C
Le,
Cs
LGs
3
0.19
0.13
0.31
0.13
0.24
0.13
0.31
0.14
0.27
0.75
0.32
0.80
0.19
0.08
0.25
0.06
0.18
0.07
0.27
0.06
0.19
0.07
0.24
0.07
0.20
0.13
0.25
0.11
0.20
0.97
0,26
0.03
0.17
0.08
0.20
0.10
0.27
0.16
0.06
0.02
0,16
0.05
0.27
0.06
0.18
0.09
0.23
0.09
0.17
0.10
0.21
0.09
0.14
0.04
0.18
0.02
7
0.15
0.05
0.22
0.05
0.14
0.05
0.19
0.05
0.15
0.08
0.19
0.07
0.14
0.04
0.17
0.02
0.09
0.05
0.16
0.05
0.10
0.06
0.14
0.06
0.17
0.08
0.21
0.07
0.15
0.04
0.20
0.02
# of Ins
InstrucIBLIP
mPLU'G-W12
Lynx
VPGTrans
Cr
Lc,
Cs
Lc,
C1
LG,
Cs
Lc,
LG,
Cs
Lc,
C1
Lc,
Cs
Lc,
0.43
0.53
0.20
0.34
0.50
0.19
0.14
0.14
0.17
0.14
0.37
0.12
0.44
0.10
0.32
0.05
0.41
0.09
0.13
0.07
0.14
0.06
0.13
0.14
0.30
0.13
5
0.43
0.15
0.50
0.11
0.33
0.04
0.44
0.06
0.12
0.07
0.13
0.05
0.10
0.10
0,26
0.09
0.28
0.34
0.10
0.03
0.37
0.10
0.05
0.10
0.11
0,05
0.07
0.30
0.09
0.35
0.10
0.23
0.03
0.28
0.05
0.12
0.04
0.11
0.02
0.10
0.06
0.21
0.05
8
0.36
0.10
0.41
0.09
0.26
0.04
0.31
0.04
0.11
0.05
0.10
0.03
0.08
0.05
0.17
0.04
NoCaps
Gemini Pro-PProoiieeee Vision
Qwen-VL. VL
MiniGPP:4
LLaVA
# of Ins
C₁
LG
Cs
Les
LG
Cs
Lc,
Lc;
Cs
Lcs
LG,
Cs
L.c,
0.15
0.26
0.30
0.24
0.17
0.28
0.33
0.26
0.08
0.06
0.16
0.14
0.12
0.28
0.14
0.13
0.10
0.28
0.10
0.15
0.10
0.28
0.09
0.07
0.05
0.15
0.06
0.13
0.12
0.23
0.18
5
0.14
0.14
0.27
0.14
0.16
0.15
0.29
0.11
0.05
0.03
0.11
0.04
0.08
0.05
0.18
0.07
0.12
0.11
0.26
0.03
0.11
0.10
0.04
0.18
0.06
7
0.13
0.10
0.22
0.11
0.10
0.06
0.19
0.05
0.06
0.03
0.12
0.04
0.10
0.05
0.17
0.05
8
0.10
0.07
0.18
0.09
0.11
0.05
0.19
0.04
0.04
0.03
0.09
0.04
0.06
0.04
0.12
0.04
Internl. M
Otter
ALaMM--Adppeer
mPLUGGOW
of Ins
C
LG
Cs
Lc,
C
LG,
Cs
Lc,
G
LG,
C2
Lc,
Ci
Lc,
Cs
Lc,
3
0.12
0.58
0.26
0.44
0.16
0.06
0.25
0.09
0.17
0.33
0.25
0.23
0.14
0.15
0,22
0.10
0.09
0.22
0.07
0.05
0.19
0.06
0.20
0.14
0.06
0.08
0.18
0.21
0.05
0.10
0.04
0.15
0.05
0.13
0.04
0.19
0.04
0.11
0.05
0.18
0.04
6
0.09
0.05
0.20
0.03
0.12
0.05
0.18
0.05
0.11
0.04
0.16
0.03
0.09
0.04
0.15
0.03
0.07
0.16
0.03
0.10
0.04
0.14
0.04
0.10
0.05
0.14
0.09
0.04
0.15
0.03
0.05
0.04
0.13
0.03
0.06
0.04
0.10
0.04
0.11
0.05
0.14
0.03
0.10
0.03
0.16
0.03
InstructBLIP
mPLUG-OW12
Lynx
VPGTrans
C
LG
Cs
Lc,
C
LG
C3
LG,
LG
Cs
Lc,
G
LG,
Cs
LG,
0.31
0.12
0.42
0.08
0.34
0.18
0.50
0.20
0.06
0.04
0.07
0.04
0.12
0.17
0.27
0.20
0.26
0.05
0.34
0.04
0.29
0.14
0.43
0.14
0.05
0.03
0.06
0.03
0.10
0.10
0.25
0.08
0.28
0.37
0.05
0.31
0.11
0.45
0.07
0.06
0.03
0.06
0.09
0.08
0.11
0.19
0.04
0.26
0.04
0.27
0.05
0.40
0.03
0.04
0.02
0.04
0.02
0.10
0.07
0.25
0.08
7
0.19
0.04
0.25
0.04
0.19
0.04
0.29
0.03
0.06
0.03
0.06
0.02
0.08
0.07
0.18
0.08
0.25
0.04
0.31
0.03
0.20
0.04
0.29
0.02
0.04
0.02
0.05
0.02
0.07
0.06
0.16
0.06
Table 2: The average RSD CHAIR with the LeHaCE and the average-based frameworks, lower
better. G and Cs respectively represent CHAIR, and CHAIRs with the wverage-based ballucination evaluation framework. Lc, and Lc, respectively represent CHAIR, and CHAIRs with the LeHaCE framework. The best result under each setting is represented in bold.

---

MiniGPT-4
InsirecBBLIP
LLaVA
elns
CHAIISs
CHAIR
CHAIIS
CHAIR
CHAIIS
CHAIR
L2
Lg
L;
Lg
L,
L,
L2
L,
L2
Lg
L
L2
L
L,
Lg
Lg
3
0.07
0.50
0.58
0.09
0.58
0.70
0.20
2.30
2.81
0.28
1.62
2.61
0.23
1.51
1.53
0.23
1.15
1.62
0.18
0.19
0.67
0.10
0.12
1.64
0.20
0.46
2.06
5
0.05
0.04
0.08
0.06
0.06
0.15
0.11
1.00
2.56
0.15
1.31
2.56
0.08
0.14
0.42
0.08
0.14
0.38
0.05
0.06
0.10
0.25
0.06
0.05
0.07
0.06
0.06
0.08
0.10
0.86
1.56
0.09
1.08
0.80
0.07
0.12
0.26
0.06
0.09
0.15
0.03
0.04
0.05
0.04
0.04
0.04
0.09
0.23
0.19
0.10
0.28
0.18
0.06
0.12
0.11
0.06
0.10
0.09
LLaMA-Adapic--22
Lynx
InterLM-XC
#lns
CHAIRs
CHAIR,
CHAIRs
CHAIRi
CHAIR,
CHAIR:
L
L
L,
La
L
L,
Lg
L
3
0.80
2.23
1.76
0.75
1.97
1.69
0.14
0.35
0.36
0.19
0.40
0.40
0.13
1.13
1.91
0.13
2.35
5.83
0.75
1.02
1.06
0.06
12.90
0.07
0.84
0.20
0.07
0.20
1.13
5
0.09
0.33
0.60
0.10
0.39
0.79
0.05
0.14
0.22
0.07
0.15
0.14
0.08
0.23
0.67
0.08
1.08
0.54
0.09
0.19
0.43
0.10
0.27
0.51
0.03
0.04
0.08
0.05
0.07
0.13
0.06
0.12
0.14
0.05
0.10
0.21
0.07
0.09
0.10
0.08
0.12
0.15
0.02
0.05
0.13
0.04
0.08
0.14
0.12
0.12
0.05
0.09
0.08
8
0.07
0.10
0.20
0.08
0.14
0.27
0.03
0.06
0.05
0.05
0.10
0.08
0.05
0.10
0.12
0.05
0.10
0.10
mPLUG-Owl
Otter
VPGTrass
CHAIR;
CHAIR
CHAIRS
CHAIR
CHAIR;
CHAIR;
L
L;
L;
L,
L;
L
L;
L
L
L,
L,
L
0.06
0.93
1.22
0.08
0.67
0.80
0.14
0.49
0.45
0.13
0.59
0.49
0.14
3.27
1.84
0.17
16.25
5.50
0.03
0.12
1.12
0.07
0.10
2.80
0.07
0.13
0.83
0.07
0.12
0.35
0.13
0.87
2.66
0.14
0.85
5.69
0.02
0.10
0.53
0.08
0.77
0.09
0.12
0.17
0.08
0.12
0.16
0.24
0.10
0.16
0.96
0.02
0.05
0.05
0.04
0.07
0.08
0.09
0.10
0.09
0.09
0.11
0.08
0.07
0.14
0.45
0.05
0.10
0.25
0.04
0.06
0.08
0.05
0.09
0.13
0.06
0.05
8
0.02
0.05
0.06
0.04
0.06
0.05
0.06
0.09
0.10
0.06
0.09
0.09
0.04
0.11
0.14
0.05
0.06
0.07
Qwen VL
mPLUG Owl2
Gemini Pro-V
CHAIRS
CHAIR
CHAIR,
CHAIR
CHAIR;
CHAIR;
La
L
L
L
L,
L
L
L;
L
L
L,
La
La
L1
L
L
La
0.99
0.11
2.26
1.67
0.27
0.91
0.91
1.02
0.25
3.92
1.24
4
0.15
0.40
4.91
0.11
0.43
19.84
0..09
0.67
4.33
0.05
0.39
1.76
0.13
0.19
1.67
0.16
0.24
1.87
0.05
2.23
0.06
2.28
1.27
6
0.06
0.09
0.74
0.07
0.12
0.44
0.04
0.08
0.33
0.03
0.10
0.85
0.10
1.00
0.84
0.09
0.38
0.47
0.05
0.08
0.93
0,06
0.08
0.89
0.05
0.09
0.30
0.03
0.15
0.23
0.11
0.09
0.15
0.12
0.13
0.30
0.04
0.06
0.36
0.06
0.08
0.19
0.04
0.08
33.42
0.04
0.11
0.20
0.08
0.08
0.13
0.09
0.11
0.12
Table 3: The average RSD of CHAIR scores with the LeHaCE on MSCOCO with different fitting
methods. L, represents LeHaCE with linear fitting, while L2 and L3 represent LeHaCE with quadratic
and cubic polynomial fitting, respectively.
the results. For the number of instructions in each instruction set, we conduct extensive experiments under five different conditions: 3, 4, 5, 6, 7, and 8. The experiments carried out 10 times using distinct instruction sets, and the final results are determined by averaging the outcomes these 10 experiments.
The results are shown in Table from which we can observe that LeHaCE demonstrates superior stability compared to the average- based framework in nearly all cases. Notably, 1) On the MSCOCO dataset, for the CHAIR, metric, LeHaCE consistently outperforms the average-based framework across all twelve LVLMs when the number of instructions reaches five or more. Similarly, for the CHAIRs metric, LeHaCE exhibits superior performance across all twelve LVLMS when the number of instructions reaches four or more. 2) On the NoCaps dataset, when the number of instructions reaches four or more, LcHaCE consistently outperforms the average-basee framework across all twelve LVLMs on both CHAIR; and CHAIRs metrics. In Table 2 observe that when the number of instructions is very low, such as three, the stability of LeHaCE is compromised due to the difficulty in accurately fitting the length- saighh--llliiiaionn curve. However, with just four five instructions, LeHaCE consistently exhibits superior stability.
To verify the validity of the linear assumption we make in the LeHaCE method, we evaluate the stability of LeHaCE with different fitting methods. Specifically, we assess the RSD of LeHaCE on the MSCOCO dataset when applying linear, quadratic, and cubic fitting. As shown in Table the results indicate that linear fitting significantly outperforms polynomial fitting, particularly when the instruction count is low.

---

Air
wil LafeCE
6 mmb.
Iaass
Iock
Avg K3D er
Aurngo Daorggio (wonk))
Avoy Daseppioo Lagit (word)))
0.30
022
6.00
0.18
000
16
190
1.0
Aurngs Dacripiion (monk)
Avengg Daserprion Longit (words)
Figure 5: Average RSD of CHAIR with the LeHaCE framework at different lengths, lower is better. ABF refers to the average-based evaluation framework.
For the stability of LeHaCE different lengths, the results are shown in Figure 5 from which can see that LeHaCE significantly improves the stability of the CHAIR metrics across a wide range of description lengths. All of these experimental results validate the superior stability of LeHaCE.
5 Conclusion and Limitations
Conclusion: In this paper, we find the degree of object hallucinations is primarily influenced by the length of image descriptions, with instructions only indirectly affecting hallucinations through their effect on image description lengths. The degree of object hallucination and the length of image descriptions exhibit clear positive linear correlation. Based on findings, stable, fair and comprehensive object hallucination evaluation framework named LeHaCE is introduced. Extensive experimental results validate the superiority of LeHaCE over existing frameworks.
Despite exhaustive investigations, this work still has potential limitations. 1) We focus on object hallucination, leaving other types of hallucinations for future work. 2) Due to computational constraints, we evaluate LVLMs on only a subset each dataset. Nevertheless, we conduct thorough experiments across various datasets validate our findings and method. 3) Due to high API fees, we only explore one proprietary business LVLM in our experiments. However, we conduct in-depth analyses eleven open-source LVLMs, validating the broad upplicability of our method. 4) In the typical practice of evaluating hallucination levels in LVL.Ms, multiple instructions are usually used to enhance the stability of the evaluation results. Although LeHaCE cannot be used with just one instruction, this limitation does not affect its ability to provide stable evaluations.
Limitations:
Acknowledgments
This work supported in part by the National Key R&D Program of China (2021 YFF0900500), and the National Natural Science Foundation of China (NSFC) under grants U22B2035 and 62441202.
References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023. doi: 10.48550/ARXIV.23002. 13971. URL https: //ttp:://oo... httpa/////o...o//.../// 48550/ arxii. 2302. 13971
[2] Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov,
10

---

Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Ka- reem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: family of highly capable multimmodal models. CoRR, abs/2312 1805, 2023, doi: 10.48550/ARXXV.23111
(RL.055.RRLL13333333 /tLL/ctttt:///0..rr//110 48550/ /arii//rXvvvvv 2312. 11805
[3] OpenAI. GPT-4 technical report. CoRR,
2023. doi: 10.48500ARXIV22300
08774. URL. httth://pp/:.i..... / 10. 48550/arX3i1 http://.rr/...../////111/11.1 200.:0072.000087777
[4] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson. Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::/ppppp nips cc/ wwwpappr..... lalpppppplesss__lllllllllllpllee files/lle//eeelll wwwpff/aateeeee/eeeeeeeeaaaaaeet (haper/2023/has)) 7a62435e75511a556ffee www.afff477750 www.666677770777762666666666
/311/99363366696 Abstre-bbbbtttt...... Cone--ooofeeeeeee html
[5] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing Vlsionnl understanding with advanced large language models. CoRR, hbs/2304.10592, 2023, doi: 10.48500ARXXV.230440000 10592.. URL htttp:://oi..rs:
48550/ arro:/arXiv... 2304. 10592
[6] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR. abs/2304.084855, 2023. doi: 1044850/ARXIV.204..685.
URL https:///oi.org::
48550/ arviv 2304. 08485
[7] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: frontier large vision-language model with versatile abilities. CoRR, abs/2308. 12966. 2023. doi: 1044550ARXIV.2308.22222 https:://oi. http:::://oo.oo/11.... 48550/ arxiv. 2308. 12966
URL
[8] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 4035 4045. Association for Computational Linguistics, 2018. doi: (0.18653/V1/DI8... 14377 URL https 18653/ v1/
[9] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 292 305. Association for Computational Linguistics, 2023. URL https://aaclathoology org.oooo...ooooooooo...... 2023. emnlp main
20
[10] Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (NOPE) to measure object hallucination vision-language models. 2023. doi: 1048550ARRXIV23100553 URL htt/pst::///do org/ CoRR, wttp:50/.xxxvvv11....om 05338
10. 4445555555Xi10
[11] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi multi-dimensional benchmark for milms hallucination evaluation. CoRR, abs/2311 1.03997, 2023. doi: 10.48550ARXIV.211..7777
URL https
//doi.org/10 485.0/.8855//880 arwiv. 073977
[12] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing ballucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
11

---

[13] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu. Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. CoRR, abs/2310. 16045, 2023. doi: 10448550ARXIV.23100.00 16045.. URL hattps://doi. http///000.rr//0//// 0rg/10.48550. 48550/ arXiv. 2310 16045
[14]
Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, and Conghui He. VIGC: visual instruction generation and correction. CoRR, abs/2308. 127 2023. doi: 10.48550/ARXIV.2388., 12714. httpst/ts://doi org/
10. N0.48550/arx111 2308 12714
[15] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3166 3144, 2015.
[16] Wensheng Pan, Timin Gao, Yan Zhang. Xiawu Zheng, Yunhang Shen, Ke Li, Runze Hu, Yutao Liu, and Pingyang Dai. Semi-supervised blind image quality assessment through knowledge distillation and incremental learning. February 2024.
[17] Xudong Li, Timin Gao, Xiawu Zheng. Runze Hu, Jingyuan Zheng. Yunhang Shen, Ke Li, Yutao Liu. Pingyang Dai, Yan Zhang, and Rongrong Ji. Adaptive feature selection for no-reference image quality assessment using contrastive mitigating semantic noise sensitivity. July 2024.
[18] XuDong Li, Runze Hu, Jingyuan Zheng, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, Ke Li, Yunhang Shen, Yutao Liu, Pingyang Dai, et al. Integrating global context contrast and local sensitivity for blind image quality assessment. July 2024.
[19] Guanyi Qin, Runze Hu, Yutao Liu, Xiawu Zheng, Haotian Liu, Xiu Li, and Yan Zhang. Data- efficient image quality assessment with attention-panel decoder. pages 2091-2100.
[20] Wenrui Li, Xi-Le Zhao, Zhengyu Ma, Xingtao Wang, Xiaopeng Fan, and Yonghong Tian. Motion-decoupled spiking transformer for audio-visual zero-shot learning. In Proceedings of the 31st ACM International Conference Multimedia, page 3994 4002, 2023. ISBN 9798400701085. doi: (0.1145/5511783.3117759 URL https:///ot org/10 1145/ 3581783.361177
[21] Wenrui Li, Penghong Wang, Ruiqin Xiong, and Xiaopeng Fan. Spiking tucker fusion transformer for audio-visual zero-shot learning. IEEE Transactions on Image Processing, 33:4840-485 4852, 2024. doi: 10.
[22] Wenrui Li. Zhengyu Ma, Liang-Jian Jian Deng, Xiaopeng Fan, and Yonghong Tian. Neuron-based spiking transmission and reasoning network for robust image-text retrieval. IEEE Transactions Circuits and Systems for Video Technology, 33(7):3516-355) 2023. doi: 10. 10.11V9/C 2022.323304..
[23] Zhuangzhuang Chen, Zhuonan Lai. Jie Chen, and Jianqiang Li. Mind marginal non-crack regions: Clustering-inspired representation learning for crack segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
12708- June 2024.
[24] Cunhui Dong, Haichuan Ma, Zhuoyuan Li, Li Li, and Dong Liu. Temporal wavelet transform- based low-complexity perceptual quality enhancement of compressed video. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[25]
Zhuoyuan Li, Yao Li, Chuanbo Tang, Li Li, Dong Liu, and Feng Wu. Uniformly accelerated motion model for inter prediction. arXiv preprint 1.XX:224077.1541 2024.
[26] Chuanbo Tang, Xihua Sheng, Zhuoyuan Li, Haotian Zhang, L.i L., and Dong Liu. Offline and online optical flow enhancement for deep video compression. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6):5118-111 Mar. 2024. doi: 10. 1609/aii. v38i6. 28317
28317. https:/// aaai org intp:/inexx.ccc hwtpp/pppppppppaaaaaeaa
[27] Zhuangzhuang Chen, Jin Zhang, Zhuonan Lai, Jie Chen, Zun Liu, and Jiangiang Li. Geometry- aware guided loss for deep crack recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4703-4712 June 2022.
12

---

[28] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[29] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisser- man, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, edi- tors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022. NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http::t/ttppppeeeeeeee mips mips. httppp//eeepppppe.eee//////e///p
[30] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang. Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding multi-modal 1lms. CoRR, abs/2307.0858/ 2023, doi: 10.48550/
ARXIV.2307.0858. URL https::::// :/sttps:/doi..m http///ti..//////////// 4.8450055Xiiy 2307 08581
[31] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-docowl: Modular- ized multimodal large language model for document understanding. CoRR, abs/2307.00299 2023. doi: (048550/ARXIV.2307. 002499.... URI.ttpss: 02499
wttp://0000055./........o http//rXivv.// 2307 orw/ooog///////oo 48550/8500//0
[32] Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internim ditemnl--cooposee A vision-languaag large model for advanced text-image comprehension and composition. CoRR, abs/2300.151112 2023. doi: 1044855//RRIVV2230. org/ 10. 48550//55 2309. 15112
15112. htLpstss::////
[33] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CORR, abs/2306. 14824, 2023. doi: 10.4850//RXIV.2300.. 14824. [RLhhttp:://dat org/ 10. 48550/ aatp/arriiiv.. 2306
14824
[34] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao Shikra: Unleashing multimodal Iim's 's referential dialogue magic. CoRR, abs/2306. 151999 2023. doi: 10.485550/RXXIV22300
15195. URL /ttpp:://ao...ooo org/ 10. 48550/arXivv 2306. 15195
[35] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BIIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, ed- itors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 2023. URL httttt::::ppooeeeeeeeee
mlr prwss http://v221112223/ html
[36] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. CoRR, abs/a311l...aaaaa 1otp/23311.0427,,, 2023. doi: 10.48550/ARXIV2311.. 1..04777
URL /tttps//://doi org/10 10. 0/8500/arXi10 2311.04257
[37] Bo Zhao. Boya Wu, and Tiejun Huang. SVIT: scaling up visual instruction tuning. CoRR, doi: 10.45550/ARXIV.220..0000 abs/2307.04pps 04077, 2023.
URL lhttts:////at httt::://doioor/1100
38550/arXiv 2307 04087
[38] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality. CoRR, abs/2304. 14178, 2023. doi: 10. 00.48550/ARXIV.220 14417744434.
URLhhtts:: /atttps://dof org/ 10. 48550/ arxiv 2304 14178
13

---

[39] Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng. Xing Sun, Liujuan Cao, al. Cantor: Inspiring multi- modal chain-of-thought of milm. October 2024.
[40] Anonymous. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreviess
[41] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter V2: parameter-efficient visual instruction model. CoRR, 0tts/2304.10011 2023. doi: 10.4850/ARXIV2334.5500 URL httpss//://do org/ 488500 2304 15010
[42] Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms. CoRR, abs/2305.01278, 2023, doi: 10-48550/ARXXV22350002228
httpss//////
org/oo/// 10. 48550/ arXiv 2305. 01278
[43] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. CoRR, abs/2305.03726, 2023, doi: URL
10. 0044850/ARXI.223500000
[44] Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training gpt4-style language model with multimodal inputs? CoRR, abs/2307 .22499, 2023. doi: 10.48550/ARXIV22300 02469. URL https //oi.org/// 00..4455//araa
.24669
[45] Ysung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ra- manan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCÓ: common objects context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer
Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014. doi: 10. 10. 10.1100978---31--10666
48. URL https://doo http:///do..og/....11 o0.
978- 178-3-319-10602-11
[46] Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lee. nocaps: novel object captioning at scale. In 2019 IEEE/CVF International Conference
Computer Vision, ICCV 2019, Scoul, Korea (South), October 27 November 2, 2019, pages 8947 8956. IEEE, 2019. doi: 10.1109/ICCV.
2019.00904. URL. htt.:http::://
org/10 10200...1199/ 1109/ICOM ICCV 2019. 00199.09004
[47] Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may be faith- ful: Probing object hallucination in vision-language pre-training. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6. 2023, pages 2128 2140. Association for Computational Linguistics, 2023, doi: 10. 18653V1/ 156. URL httpstt:::/// ///oii.//.////////// org/10 10. 18653/vv/2000 eacl main. 156
14

---

7 Technical Appendices
7.1 Instructions
We primarily referred to the instructions from 4 and additionally designed some instructions, totaling 25 number.
7 Describe the image in sentence.",
I2:" Summarize the image in single sentence.",
Ty: Give one-sentence depiction of the image.',
T4: Provide concise sentence describing the image.",
Ts:" Give brief summary of the image in single sentence.",
1g: Describe this image in short.',
17: Describe this image in few words.,
Ig: Provide brief caption for this image.',
1g: Provide short caption for this image.'
11o: Brieffy describe the content of the image.',
411: Describe this image.'
12 What does the image show?',
113 What can you see in the image?",
14: What is described in the image?",
Js Provide caption for this image.",
76 Describe the objects in this image.'
11: Can you provide a description of the image?",
Iss: What objects or subjects are present in the image?',
119: Describe this image in detail.',
I20: Describe this image in extremely detail.'.
I21:' Provide detailed description of this image.'.
I22: Can you describe the scene in the image in great detail?',
I23: Give a thorough account of what is depicted in this image.',
121 Provide an elaborate and comprehensive analysis of this image.',
125: Give a comprehensive and in-depth description of what is shown in this image.",
7.2 Further Exploration
Why does the hallucination rate of MLLMs increase with the increase in image description length? The underlying reasons behind this phenomenon are difficult to determine, as the output of MLLMs influenced by multiple factors such as visual encoders, language models, and training data. this section, we aim to shed light on this phenomenon by delving into an analysis of common hallucination patterns long image descriptions.
As shown in Figure 7 We found that hallucinations are more likely to occur after some words or phrases that indicate enumeration or introduce additional information, such "in addition". "addition to". "additionally", "include", "including", "such as". "as well" and "also". We refer to these words/phrases as "hallucinogenic words". As shown in Figure left, we performed a statistical analysis on a subset of MSCOCO dataset comprising 256 images to investigate the hallucination rate of image descriptions containing hallucinogenic words. The experimental results demonstrate notable increase in the hallucination rate of image descriptions that incorporate hallucinogenic words, compared to descriptions that lack such words. This phenomenon was consistently observed across all twelve ML..Ms.
15

---

www.phessskhiic
Doctpkeanccerr "aduer "tokdnn"
Dostipann andldingg "sloo" Ozen Dounttooe
Iepornat (operptise)
Ircorrud Dauttplie Corarenn the
Deesitiin Ousnisis HF
Number of Descriptons
Descrigtion Length Incrrsesss from Left Right.
Figure 6: Left:
Comparison of the hallucination rates between image descriptions containing hallucinogenic words and image descriptions without hallucinogenic words. Right: Proportion of hallucinogenic words in image descriptions containing hallucinations under different instructions.
Therefore, we propose a hypothesis that MLLMs are more likely to employ hallucino- genic words in generating lengthy and detailed image descriptions, resulting in higher hallu- cination rate. To validate our hypothesis, we ex- plored the relationship between the occurrence
Morni
MSCODO
NoCape
Scarch
HW
3:855
7.36
15:45
16:73
Le00
5.15
4.77
11:43
11:14
1e:000
17.40
11
14.87
18,09
lens
0.12
0.18
0.34
0.13
21.77
frequency of hallucinogenic words and the av- erage length of image descriptions. The image descriptions with varying average lengths were generated by different instructions. The results are depicted in Figure 6 right. As the length of the description increases, the number of in- correct descriptions containing hallucinogenic words increases and constitutes a significant por- tion of all incorrect descriptions. This validates our hypothesis."
1o
19:36
17.76
33,00
Le (90)
29.07
26.60
44.85
43.75
1,000
38.75
35:43
56.70
$5.37
Team
0.59
0.55
PELCCIIPetiioaa (R011)
52.73
66.39
CLIPSON
0.81
0.81
0.80
0.50
RICCLIPSere
0.81
0.81
0.81
0332
Table 4: Hallucination rate and quality of the gen- erated image descriptions after disabling hallucino- genic words. The values in the table are averaged across 10 MLLMs.
We further explored whether the hallucination rate of MLLMS can be reduced by disabling hallu- cinogenic words. Specifically, we prohibited the use of previously mentioned hallucinogenic words ("in addition," " "addition to," "additionally." "include," "includes," "including," "such as, "as well," "also") during the generation process in the MLLMs, ensuring that the generated image descriptions did not contain these hallucinogenic words. The results, presented in Table clearly demonstrate that disabling hallucinogenic words can significantly decrease the hallucination rate without com- promising the quality of image descriptions. Furthermore, it helps alleviate the tendency for the hallucination rate to increase as the description length grows. These experimental findings not only highlight the substantial impact of hallucinogenic words on the hallucination rate of MLLMs but also offer valuable insights for mitigating hallucination in MLLMSs
7.3 More Experimental Results
The average length and CHAIR scores of image descriptions generated by 12 MILLMs (Gemini-Pro-
Vision pro [2]. VL 77) MiniGPT-4 (5), LLaVA (6), InstructBLIP [4), LdaMe-Adapter-v2
41,, mPLUG-Ow12 (36,, mPLUG-Owl [38] InternLM InternLM-XComposee (32] VPGTrans 42) Otter [43
and Lynx 44 under 25 instructions subsets of MSCOCO 45 and NoCap 46 are shown in
Table 5 and Table 6
respectively.
The hallucination degree of each MLLM with hallucinogenic words disabled are shown in Table and Table Disabling hallucinogenic terms can effectively alleviate hallucinations in MLLMs.
7.4 More Case Studies
Figure 7 showcases examples of image descriptions generated by MILLMs before and after disabling hallucinogenic words.
16

---

MinicPP-4
Tynx
LLVA
Oster
VIR Frans
C2
Lan
C2
C2
Eam
C5
C2
Lum
C2
C2
25
3.26
6.64
43.52
5.63
14.33
34.97
6.20
17.38
18.70
730
14.84
25.45
10.75
20.31
1592
3.75
0.25
60
2,90
5.0%
11.39
1.15
1.56
38
4.70
12:50
19.4
8.00
10.02
$2.30
12.35
26.17
15:30
5.87
7.6
20.0
5.54
9.38
10.35
039
1.56
30.89
6.10
15.21
12.32
5.33
10.16
27.01
10395
20.70
15:30
4.45
0.25
25
5,06
8.59
11.00
0.96
1.56
35.68
4.01
13:38
18.00
6.79
14.06
51.97
13.35
35.04
16.35
5000
7.42
35.17
18.36
14.06
12:05
33.59
15.35
16:11
76:02
7.80
25.52
21.38
6.56
16:43
35.12
18:35
1
18.40
19.55
11.00
0.96
3.50
57.14
6.36
15.65
9.00
21.09
54.59
13.35
33.98
25.71
5.90
8.59
42.79
6.85
16.30
10.88
1:20
1.55
36.79
5:36
14:34
25355
5:06
21:25
53.20
13.45
33.39
10:33
4.20
5,27
1
50
7.95
21.88
10.92
1.21
1.95
64.59
7.10
19:14
27.44
6.92
14.06
57.01
14:34
41.80
66.24
7.78
17.90
03:00
7.00
23.04
90.35
10:02
35.35
05.65
0.85
1932
31.75
3.395
27.34
13.25
21.05
02:01
08.75
8.39
22.27
34.10
14,06
24.25
4.10
8.98
70:31
10.15
29.09
1906
13.26
1.65
1.37
1:39
25.31
0.58
12.50
40.76
4.94
13:28
11:30
31.25
25.35
50.05
16.30
16.76
28.13
16.19
8.53
11:11
6.5%
1.56
4100
6.6
60.71
67.95
15:54
$5.95
9.05
$1.45
5.72
34.77
50.72
8.10
25005
3521
10:30
23:05
50.65
20.29
55.47
55.06
9.34
25.61
47
1.20
22.66
11.00
0396
1.36
35.48
0.32
25.55
-1.4
8.23
20.20
59.64
14.71
35105
46.72
8:25
29 29
6.62
11.33
18.74
3.12
5.86
02.49
13.40
32.81
18.98
3.58
19:14
23.50
15.92
32.81
21.18
6.46
10.94
14.22
00.05
14.99
52.34
041.4
5.28
36.95
34.00
11:48
15.35
51.89
20026
61:34
93:45
833
18.15
100.85
16:05
55.86
61.69
9:3
27.73
08.15
56.79
21.36
63.25
102.59
59
10:16
33.59
64.07
55.25
63.15
23.09
100.25
16.79
58.20
39:45
27.89
70.52
13:41
41.02
10.29
28.13
1.5
73:46
30.05
50:45
21.49
75.47
8.5
20,89
96.74
11.45
43.36
78.71
11:99
35.55
12.01
37.89
90.95
26.89
80:47
101.40
8:54
31:25
9:85
10.47
77.03
6.46
20.80
53.80
5000
21.45
56.50
10
17.50
63.79
21.50
85.65
9.40
30.47
Ins
Len
La5
C5
MPLUG-OW
45
C5
1.00
Gemmni
Pro
Vision
Internl
Len
XX Composer
Cx
Loa
Owen-VL
C5
C5
Len
GPPL0-002
30 50
8.56
18.36
22.71
7.19
12:50
15:35
4.00
7.03
18.10
2:26
4.69
9.96
2337
14.00
18.04
13.67
13.95
14.70
18.89
3.00
1861
15.35
14:54
18.34
300
32.91
9:76
21.09
23.90
6.81
12.89
15.41
2.62
4.30
13.90
7.42
18.00
2.55
10.00
2.10
2.75
19.55
19.07
52.75
87.25
21.27
57.42
19.85
2.02
5.47
22.51
1.0
11:35
40.26
6.05
17.58
64:30
12.32
$3.00
16:56
44.92
72.70
18.32
28.83
10.77
1.50
2.34
22.25
4.29
9:38
27.00
4.22
9.27
41.52
8.50
21.09
11.34
31.25
13.05
12.30
20:700
14:35
4.25
0.50
19.11
6.59
22.7
25.00
4.9
9.77
1.73
2.43
3.15
46.49
10:43
26.17
16.91
6.21
9.38
12:78
5.14
3.50
17:54
4.83
7.03
19:36
5.63
9.38
9.72
2.39
3.13
13.27
38.95
101.23
59.77
75.02
8:47
8.95
18.20
27.00
17:35
19:64
2:30
16:30
3.05
11
29.50
4.96
11.33
70.57
17.19
43.41
33.11
5.48
14.06
38.90
8.28
18.36
28.87
3.74
6.04
30.76
4.45
10.94
18:46
4.75
8.96
10.11
18:30
$2.34
31.80
5.46
12:11
27.32
0.46
11.75
43.25
5.58
1564
$4.97
5.77
13.0
69
11.92
14.77
16.09
11.25
26.17
15:78
4.13
5.08
17.58
5.94
9.77
05.83
7.75
20:31
9.30
2,40
3.13
33
12.15
31.25
64.19
18.56
40.6
45300
6.45
16:0
56.3
10.00
12.03
06.15
7.47
36355
70.1
1204
39.06
59.07
25.76
96.80
21.08
63.35
50.84
7.09
18:36
37.50
8.99
18.36
66.72
8:30
83.73
50.39
13.67
30775
1230
31.01
20.31
17.07
300
60.55
10
23.28
54.84
82.08
66.33
57.10
933
63.67
59.77
10.00
21.65
64.45
85.19
793
35.30
1145
1200
07.94
978
30.45
96.4
1520
07.19
11
85.35
19.39
38.95
10023
21.32
01.72
0333
6:35
18.35
12.29
1124
23.35
86.27
1037
29.06
90.33
15:20
39.77
84
31.25
105.22
22.39
50035
70.30
3.56
31.25
2018
1.59
1933
57.55
775
2813
8037
13.33
46.00
15.00
13.11
38.07
103.27
21.98
02.72
74.96
5.97
18.75
09:35
11:34
35.85
85.0
12.45
44.92
89.39
1559
55.47
92.6
25.14
60.94
13
21.34
06.41
19482
K.88
25.30
60.96
9.95
27.44
105.00
1084
45.70
99.30
1547
60735
13.22
10.25
21.93
6019
$1.99
21009
69:30
25.16
25.02
1095
49.19
1550
33.91
Table 5: The average length and CHAIR scores of descriptions generated by 12 MLLMs prompted
by the 25 instructions on a subset of MSCOCO containing 256 images.
7.5 Broader Impacts
The LeHaCE framework provides a stable, fair, and comprehensive way to evaluate object halluci- nations in large vision-language models. It helps in assessing the usability of large vision- Lasiun-anguuage models, thereby helping to prevent safety incidents.
We acknowledge the potential ethical concerns associated with the use of the MSCOCO dataset, particularly regarding data privacy, copyright, and consent, as the images in this dataset were collected from Flickr without explicit user consent. However, it is important to note that our study's 's methodology and findings are independent of the dataset used. Our research focuses on evaluating the relationship between instruction length and hallucinations in Large Vision-Language Models (LVLMs), and does not rely on or alter the underlying data in the MSCOCO dataset. Nonetheless, we recognize the ethical implications of using such datasets and recommend future research to continue exploring these issues in greater depth.
17

---

Impestinge
Iebaake Carpae Anseleileenn wow gue
Encrbanh
lus Acakk uas bang gaid iauge dlanca pins cabio ceveiies
The
plass lis laaccee opheast This lote: dustooic Aor
The
ME95714
Thar was counto placc moedo ssic vitt talootios clanal hangle Bise pciling
fen vomo tnaut deall dosen piece in surp crtpy
whippl
wit
skrical tas plarni There sevent ofher cieesmic devics ane dirk and
Instuction Canyzu dee souns Imapa SaaaT
Intruction decrbb some Image gaar smme?
magg Meader ploed
wome ventyy regrilents.
ap- Aspica maniaa sii sod toa pilk suauu-- wlick decoadd other phosa
Instldinng
mazt logeche seneeeooo pruting sctriiing beatty
hosnan lagro
part
IntruclL perfii in sene itvhhess
arard
thot vosta efecce aesood thrugh ecmo beathag pirnt birts mose,
denggrandd cisec
randdigg pilcol
Instrutiii
prvee Monder mnd doodge
regumiil fesar timert bas le wrinng be Brmel shil powilcts manada mank
bime lo
ploto lugaato vhiiy decanacd calunta oiiios bemu ubho figheec
them
pesplc
Desarbe
Intraciio Dscr the oluuaaa in image
makerrdk pubee with prop
revenl
edkirap and poupl freits, Berrat
The bagg Runaee pitra with eraas scrroti maddaa obi. The pous sepocd Lide in cobhiid at Eanon pveri to de eveallicionn colicies alpcsseo lee dolk The Mugeriia a datta sisa dight be qudied by alll keail bunc
11200
Yiok amaoghan strdlato Themme
donak ahou pug walling
Tho Siman with
snmaa asar allis pics ce lugend with loped uome dloeo luke by gioid mor thes whe opay buti plas and coua..
rold poffte
ginflo wee thee candly clea cich eito speca fr ablnn dvmmly wil the
Intracter Dhe thepp
Dies
bocodd
(nc- grants wall hhee randle Rilk manicacce tisflace right
sprccal scncle the aioiiis, ths the trnnctt.
Opes
coected vith staaaiii..aaaaoo sdid. chcae,
regetaiis
divnniy
Figure 7: Example of detailed image descriptions generated by beam search and beam search with out hallucinogenic words. The hallucination content is highlighted in red, and the hallucinogenic words are highlighted green.
18

---

Ins
Len
MiniGPT-4
C3
Cs
Len
C5
Len
Lynx
C2
Cs
Len
LLaVA
C2
C5
Len
Otter
C5
Len
VPGTrams
C
C5
21,88
12
17.19
16.21
13.0
36.7
35.50
13.84
42.19
19.09
11.09
18.75
25.20
17.76
33.39
13.54
10.85
15:50
13.38
21.35
39.05
12.25
15.10
26.04
15.97
39.61
33.79
12.12
17.00
16.37
16.05
4.31
35.30
18334
11:05
21.85
17.3
1739
33:45
13.03
28.52
11.05
5.66
7.81
38.95
17.60
45.88
21.50
12.21
24.61
34.21
19.79
39.06
15.51
1541
59.00
15.80
55.13
50.85
17.60
3508
40.11
16.75
47.00
45.34
14.84
35.04
68.05
14.04
56.64
47.30
17.26
45.70
31.91
15.70
36.45
76,89
16.35
53:13
10.16
16.01
02.97
31.57
13.25
29:30
53.95
20.25
15.05
25.17
13.05
26.95
60.38
18.72
51.56
10.95
4.23
5.36
35.79
16.97
43.36
46.28
17.09
2063
33.29
19:00
00.33
24.54
11.25
19.14
15.40
48005
35.71
15.19
14.35
19.00
60.73
21.25
52.73
15:55
55.16
19.25
16.40
72.00
200900
83.72
13.25
66002
67.00
47.92
18.85
43.36
25.92
8.53
17.97
69.55
25:055
60.98
19.81
11.52
20:31
3,64
7.46
5,96
23.54
15.41
25:44
53.16
20012
48005
40.86
1000
$2.09
00:00
20.96
56.25
27.24
13.25
2008
47.86
11.30
45.43
40.37
16.00
56.73
22,89
15.69
16.05
25.81
9:16
18:75
02.31
18.05
53.13
21.47
10.57
21109
17.83
13.02
17.97
27.85
14:37
27.33
55,001
17.41
12.19
11.00
4:40
6.25
38.61
18.35
4800
60.23
14.85
33.88
61.43
22.20
18.44
23.37
10.47
1563
55.47
10.15
21.08
57.00
18.50
12.40
14:33
26.95
41.50
15.45
6:10
63:18
20.35
14:15
11
85.50
21.39
60:16
98.16
25.37
63.07
37.25
21.29
58.59
35.00
18.00
60.55
78.92
26.19
69:14
19.85
19.36
39.77
02.00
21.91
02.11
00.33
2313
06000
38.61
10.14
53.00
03.33
13.00
03.38
80.35
26.35
06.00
59.36
10.20
64.00
57.00
21.90
61.33
92.97
1975
57300
$1.97
20.51
55.61
05.46
18.25
57.42
80.15
26.37
65.30
92.15
17.99
58388
84.68
23.02
65.45
97.89
20.22
60:55
76.56
13.05
02.30
83.05
20.52
63.28
67.57
19.85
09.30
$3.74
19.65
36.10
60.35
21.55
78.73
3500
98.97
61.72
95.43
60.16
76.41
110.76
20.01
98.40
8300S
21.36
63.67
31.12
17.24
67.58
Ins
Len
C,
Ca
Len
mPLUG40l
C4
C4
Len
Gremani-Pr-VViiee
G
Cx
Internl.M-XXomppe
Len
C2
Len
Owen-VL
C,
Len
MPLUG-0W12
C4
31.18
15:16
32.42
20.84
11.99
17.19
15:11
11.75
16:41
13:75
9.83
16.02
16.10
1.26
12.11
10.10
5:18
5.50
23.95
14.04
25.00
17.77
1218
18.36
14:24
8.36
11:35
13.36
3.99
15:03
16.25
8:35
12:30
10500
5.52
13.20
15.63
14,86
10.94
13.39
10.04
17:58
19.55
10:15
6.25
12.11
20:70
13.67
16.16
32.42
0244
7.28
25.76
02.35
79.79
35.69
18.00
1301
19:1
18.95
12:25
21:38
15.45
10.01
32.65
70.22
24.35
57.00
63.05
0153
46.05
10:00
6.30
7.00
13.30
12.00
21.88
17.04
8.31
13.05
11.00
6.05
1.00
38.44
20.65
49:22
29.20
10.10
26.95
14.89
3.82
11:35
16.70
12.09
19.92
19.23
3.95
6.98
9.57
4.79
5.06
46.75
17.35
39000
16.65
12.35
17.38
13.55
9.88
10004
15.34
10.34
17.90
18.05
6:21
9.38
1.45
4.2
4.00
16.77
36.33
42.97
24.44
11.35
13.06
51.38
31.64
67.10
40.54
41.00
$2.50
14.95
52.34
11.07
28.32
30.93
11.05
13.55
10:45
13:15
28.52
36.75
15.64
02:97
29:25
16.15
28:50
34.13
10.85
25.13
25.20
6.76
13:28
19:01
4.70
7.00
18.00
10.85
16:30
73.85
10:15
50000
20.43
15.27
30.47
10.30
7.85
12:50
42.00
11.07
25.05
13:46
5.40
7.45
11.89
20.68
53.91
40.79
15.89
$4.39
20.76
13.57
21.48
16.32
11.20
17.97
22.72
3.11
12.11
9.80
5.48
6.25
16.15
18.08
29331
49.32
22.37
42.07
40.00
18.18
27.6
52.25
15.85
-0.02
59.39
13.72
53.06
39
14.24
20.34
15.85
43.36
57.81
48.17
17.88
46.88
28.23
13.25
68.76
46.00
28.91
12:50
11.79
14:00
6737
07.55
$7.56
14.01
30.50
58.85
26.23
66.41
99.19
57.81
76.83
20.17
65.45
71:39
15.37
50000
76.08
15.10
12.58
91.28
20.55
61.72
18
87.01
25.31
20.70
34.21
2112
30.77
39.00
20.80
57.81
12.00
15.35
50.30
75.26
14.1
45.0
80.65
1001
54.00
82.08
16.74
$1.17
102.22
25.81
61.22
72.84
21.61
62.11
49.79
11.83
32.03
79.31
14.14
4638
69:00
10:34
12.90
84.37
23.55
60.9
99.18
23.99
6133
68.37
20.94
60.16
61.72
15.88
27
75,00
16.15
4835
19.65
1381
52.34
90.38
14.22
63.28
131
59.38
14.82
108.95
15.32
5253
93000
50.08
5037
83.27
80.01
14.97
15.20
52.34
Table 6: The average length and CHAIR scores of descriptions generated by 12 MLLMS prompted
by the 25 instructions on a subset of NoCaps containing 256 images.
19

---

Metric
MiniGiPT-4
InstrucBB
Lynx
BS
BS w/o IIW
BS
BS w/o HIW
BS
BS w/o HW
BS
BS wio HW
BS
BS w/o HW
G
7.39
6.99
6.01
4,95
8.49
8.12
8.77
6.95
15.76
13.32
Le,20)
5.30
3.26
Le,40
6,66
6.36
5.10
4.36
6.49
6.42
8.30
6.81
12.66
11.00
L,,60
7.98
7.53
7,86
6.47
9.72
9.16
9.38
7.96
16.56
14.10
1g,80)
9.31
8.70
10.61
8.59
12.95
11.89
10.46
9.10
20.45
17.21
Lnax
0.07
Cg
19.28
17.95
19.75
17.00
23.34
22.53
22, 84
17.95
41
35.73
Le,(20) 1
9.27
8.98
5.61
5.33
8.00
8.87
14,48
10.32
15,31
14.26
1c,40)
20.31
17.12
29.88
Lr,(60)
22.15
20.65
26.87
23.56
26.97
25.71
26.14
23.92
44.45
38.84
1c,88)
28.59
26.49
37,50
32.67
36.46
34.13
31.97
30.72
$9.02
51.14
Le,ax
0.32
0.29
0.53
0.46
0.47
0.42
0.29
0.34
0.73
0.61
T2I-CLIPRetiival (R@1)1
38.76
38.18
29.66
30.20
39.76
40.18
35.02
33.52
21:22
22.08
CLIPSeare
0.82
0.82
0.80
0.80
0.81
0.81
0.81
0.81
0.78
0.78
RefCLIPScore
0.81
0.81
0.83
0.83
0,80
0.80
0.82
0.82
0.79
0.79
Metric
VPGTiass
LLaMA 2
Interal. MatrrnLMM-X M-Xoopop
Qwen- VL
oiPLUG-QW12
HW
HI
BS
G
7.28
6,84
11.91
11.48
7.54
6.73
6.39
5.63
9.08
7.55
Le,20)
5.77
5.50
6.04
5.78
5.40
5.25
3.44
3.01
3.92
3,48
Le,40)
6.48
9.29
8.97
7,82
1,,60 4
7,97
7.47
12.54
12.15
10.25
3.68
7.28
6.53
10.86
9.23
1g080
9:08
8:45
15,80
15.34
12.67
10.40
9.20
8.29
14.33
12.11
0.06
0.05
0.16
0.16
0.12
0.09
0.10
0.09
0.14
C
17.19
16.64
32.39
31.33
18.03
16.67
20.20
18.15
28.
23.78
Lzy(2)) 1
9.08
8.93
11.31
10.39
9.48
9.49
6.15
5.50
8.19
7.24
1c,(4))
15.01
14.61
22.99
22.09
19.18
17.78
15.31
14.00
21.66
18:95
Lo,(0)
26.07
35.12
1e((8))
26.86
25.96
46.34
45.50
38.58
34,36
33.63
31.00
48.59
42.36
Lo,ox
0.30
0.28
0.58
0.59
0.48
0.41
0.46
0.43
0.67
0.59
Tal-CLIPRerrieall 1R1
CLIPScore
0.80
0.79
0.81
0.81
0.81
0.81
0.82
0.82
0.81
0.81
RefC1 IPScore
0.80
0.30
0.81
0.81
0.82
0.82
0.83
0.83
0.83
0.83
Table 7: Hallucination rate and quality of the generated image descriptions on MSCOCO after
disabling hallucinogenic words. BS stands for Beam Search and HW stands for hallucinogenic Words.
Metric
MisiGPT-4
InstructBLIP
Lyns
LLaVA
Otter
BSS
RS win HW
BS
BS w/n HW
BS
BS HW
BS
BS HW
BS
BS HW
C
19.08
18.27
11.29
10.48
18.99
19.00
15.29
13.72
21.56
20.35
Lo(20)
14.53
14.08
6.52
6.44
13.79
14.53
12.68
11.51
15.49
15.16
Ln,40)
16.79
16.17
10.20
9.73
17.18
17.41
14.48
13.42
19.03
18.27
1c,00
20.57
20.29
22.58
Lc(80)
21.30
20.34
17.56
16.32
23.96
23.17
18:09
17.23
26.12
24.49
1e,ax
0.11
0.10
0,18
0.16
0.17
0.14
0.09
0.10
0.18
0.16
51.47
38.25
48.52
le,(2) 4
23.75
23.53
13.33
13.04
36.07
35.98
24.15
20.90
25, 25.38
25.22
Lo,400)
35.75
35.32
26.39
25.77
46.11
45.99
33.90
32.17
38, 89
37.57
(c,(00)
16
56,00
43.66
43.44
$2.40
49.93
Le,(0)
59.77
58.89
$2,50
51.24
66.21
66.01
53.42
$4.71
65.91
62.28
Leçox
0.60
0.59
0.65
0.64
0.50
0.50
0.49
0.56
0.68
0.62
F21-CciPReerieval ()
47.
CL.PPe
0.82
0.80
0.80
0.81
0.81
0.80
0.76
0.77
RoICLiPSenee
0.82
0.82
0.82
0.83
0.83
0.851
0.81
0.82
0.78
0.79
Metrie
VPCTTian
LLaMA-Adaptee--v2
Interal Compooerr
Qwen- VI.
mPLU//--WW12
BS
BS wio HW
BS
BS wio HW
BS
BS w/o HW
BS
BS w/o HW
BS
BS w/o HW
C
15.07
12.32
11.76
11.01
11.01
10.47
1e(20)
12.51
12.29
12.52
11.69
10.93
10.83
8.37
7.94
6.91
6.89
Lo(40 1
14.39
14.15
16.07
15.41
12.74
12,08
10.69
10.03
10.82
10.56
(0,000)
16.26
19.62
14.54
13.33
13.01
12.12
14.72
14.22
18.14
17.85
23.17
22.85
16.34
14.58
15.33
14.21
18.63
17.89
Lçaa
0.09
0.09
0.18
0.19
0.09
0.06
0.12
0.10
0.20
0.18
16
26.
26,
(2,(20) 4
20.72
20.32
14.15
11.72
Lex40)
31.95
31.60
35.31
33,90
31.22
30,50
25.00
23.50
25.45
24.88
(c,(0)
43.51
43.23
48.18
47.08
42.33
40.68
35, 85
33.45
39.17
3t% (14
Lc,(8)
16.71
52,90
$1.20
14jax 4
0.58
0.58
0.64
0.66
0.56
0.51
0.54
0.50
0.69
0.66
TZI-CLIPRetioall ()
43.94
43,60
42.96
42.62
50.48
50.50
56.14
56.54
44, 14.78
45.48
CLIPSeare
0.79
0.79
0.80
0.80
0.81
ReICLIPSeane
0.50
0.80
0.81
0.81
0.81
0.81
0.83
0.83
0.83
0.83
Table 8: Hallucination rate and quality of the generated image descriptions on NoCaps after dis-
abling hallucinogenic words. BS stands for Beam Search and HW stands for hallucinogenic Words. Disabling hallucinogenic words can alleviate ballucinations in MLLMs.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: Yes
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: Yes
Justification:
We provide detailed discussion of the limitations of our work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: NA
21

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Yes
Justification:
All experimental settings detailed in Sections 3 &
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer:
Yes
Justification:
The code is included in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. httpsscccpppbccccceeceeeeee wwtp://gweeeeeeeeeeeeeeee Baaeeees/oooeemiieeeeeeeee ssinisiiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: Yes
Justification:
All experimental settings are detailed Sections 3 &
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: Yes
Justification: This paper reports the confidence intervals and statistical significance tests in Section 3 The assumptions used our method are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: Yes
Justification:
We detail the information on the computer resources in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: Yes
Justification:
This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: Yes
Justification:
We discuss the broder impacts of our work in Appendix Section 7.5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: NA
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: Yes
Justification: CC-BY 4.0
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: Yes
Justification:
New assets introduced in the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: NA
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: NA
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "zjiJ1J9yYI", "reviewer": "Reviewer_ffbd", "summary": "This work presents comprehensive experiments to study the relationship between description lengths and hallucinations in LVLMs. Based on the observed positive correlation, authors propose an approach of fitting a length-hallucination curve to evaluate object hallucinations. Speciffically, the curve allows for fair comparisons that are not influenced by varying lengths, through providing the hallucination degree corresponding to any given description length. Furthermore, the curve slope reflects the extent to which a LVLM's hallucination degree is affected by description lengths. The evaluation, considering both the value and slope, demonstrates stability and comprehensiveness, as supported by the conducted experiments. The authors' thorough and meticulous research on this issue is highly convincing, and the proposed method effectively showcases its effectiveness.", "questions": "1. Why is the proposed method limited to large vision-language models? Could it be extended to large language models as well? It would be beneficial for the authors to provide a clear explanation or justification for this limitation. \n2. Similarly, are the finding and method presented in this paper applicable to other forms of hallucination beyond object hallucinations, or other tasks, such as VQA?\n3. What could potentially explain the phenomenon observed in Figure 2, where longer output lengths result in higher object hallucination degrees?\n4. How are the 25 instructions used in experiments designed? Are they generated randomly or based on specific rules? Besides, why is it 25, and what difference would there be if there are more or less instructions?", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Hallucinations evaluation is a realistic and crucial task in the field of LVLMs, as hallucinations usually introduce misleading conclusions or even have disastrous outcomes. In this context, the authors perform a detailed experimental analysis on the impact of instructions on hallucinations, providing convincing evidence to support their motivation. Moreover, the proposed curve-based method is a simple yet effective approach, which is well-motivated by the observed linear correlation between description lengths and hallucination rates. The paper is well-written and effectively communicates its main contributions and techniques. Overall, the paper exhibits technical solidity.", "weaknesses": "1. The authors conduct experiments using only the beam search setting. Although I understand that beam search is widely used in hallucination evaluation of LVLMs/LLMs, it remains uncertain whether the observed correlation between the hallucination degree and the description length holds true under different decoding strategies. Thus, I recommend that the authors explore additional commonly used decoding strategies, such as greedy decoding, to provide a more comprehensive analysis. \n2. The paper lacks a study about the influence of the instruction number on the length-hallucination curve. The fitted curve is directly affected by the number of samples, which corresponds to the number of instructions provided. It is therefore essential to thoroughly investigate the minimum number of instructions necessary for the proposed method. \n3. The authors mention in the paper that the proposed method can \"evaluate object hallucinations at any given image description length.\" In reality, when the given length deviates too much from the existing data, the fitting is likely to fail, leading to inaccurate results. The authors should use more cautious wording.\n4. In my opinion, the impact of length might be mitigated by simply controlling the maximum generation lengths.The authors only mention this method in a footnote and believe it does not align with the actual usage scenarios of LVLMs. More in-depth discussions should be provided.\n5. Some minor errors need to be corrected. For example, in line 42, \"Figure 2&3\" should be \"Figures 2&3\".\n6. It appears inappropriate to represent a variable using only two letters. Consider replacing \"hr\" with \"h_r\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1**: The authors conduct experiments using only the beam search setting. Although I understand that beam search is widely used in hallucination evaluation of LVLMs/LLMs, it remains uncertain whether the observed correlation between the hallucination degree and the description length holds true under different decoding strategies. Thus, I recommend that the authors explore additional commonly used decoding strategies, such as greedy decoding, to provide a more comprehensive analysis.\n\n\n**Response to W1**\nThanks for your valuable comment. Following your suggestion, we evaluate the hallucination degree of Qwen-VL and InternLM-XComposer under the greedy decoding strategy. The results are shown in Image 3 of the Author Rebuttal attachment, from which we can observe that the correlation between the hallucination degree and description length remains valid under the greedy decoding setting.\n\nFurthermore, we supplement a comparison of the average-based framework and LeHaCE under the greedy decoding setting. The results are reported in Table 4 of the Author Rebuttal attachment. The results show that LeHaCE's stability still surpasses that of the average-based method under the greedy decoding strategy.\n\n> **W2**: The paper lacks a study about the influence of the instruction number on the length-hallucination curve. The fitted curve is directly affected by the number of samples, which corresponds to the number of instructions provided. It is therefore essential to thoroughly investigate the minimum number of instructions necessary for the proposed method.\n\n**Response to W2:**\nThanks for your comment. We analyze the impact of the number of instructions on the effectiveness of the LeHaCE framework in the second paragraph of Section 4.4: **In Table 2, we observe that when the number of instructions is very low, such as three, the stability of LeHaCE is compromised due to the difficulty in accurately fitting the length-hallucination curve. However, with just four or five instructions, LeHaCE consistently exhibits superior stability.** LeHaCE requires at least two instructions, but too few instructions can affect its consistency. When the number of instructions is greater than or equal to five, LeHaCE demonstrates excellent consistency.\n\n> **W3**: The authors mention in the paper that the proposed method can \"evaluate object hallucinations at any given image description length.\" In reality, when the given length deviates too much from the existing data, the fitting is likely to fail, leading to inaccurate results. The authors should use more cautious wording.\n\n**Response to W3:**\nWe will change the statement to \"evaluate object hallucinations at any given image description length within a large range.\"\n\n>**W4**: In my opinion, the impact of length might be mitigated by simply controlling the maximum generation lengths. The authors only mention this method in a footnote and believe it does not align with the actual usage scenarios of LVLMs. More in-depth discussions should be provided.\n\n**Response to W4:**\nThanks for your comment. Firstly, truncating the output to make the response length uniform does not reflect real usage scenarios, as users typically prefer complete responses. Secondly, truncating the output does not accurately control the description length, as some outputs may not reach the threshold length.\n\n>**W5**: Some minor errors need to be corrected. For example, in line 42, \"Figure 2&3\" should be \"Figures 2&3\".\n\n**Response to W5:**\nThanks for your careful review and we will revise these typos in our final version.\n\n>**W6**: It appears inappropriate to represent a variable using only two letters. Consider replacing \"hr\" with \"h_r\".\n\n**Response to W6:**\nThank you for your suggestion, we will revise it \n\n>**Q1**: Why is the proposed method limited to large vision-language models? Could it be extended to large language models as well? It would be beneficial for the authors to provide a clear explanation or justification for this limitation.\n\n**Response to Q1:**\nThanks for this insightful question. The causes of hallucinations in LVLMs and LLMs share similarities, such as contradictions between the knowledge embedded in the parameters and the information in the context. Therefore, we believe that the proposed method has the potential to be extended to LLMs, which will be our future research direction.\n\n>**Q2**: Similarly, are the finding and method presented in this paper applicable to other forms of hallucination beyond object hallucinations, or other tasks, such as VQA?\n\n**Response to Q2:**\nFollowing your suggestion, we evaluate the relation hallucinations and attribution hallucinations in image descriptions generated by Qwen-VL and InternLM-XComposer under 25 instructions. We conducted a model-based evaluation using Gemini-1.5-flash. The results are shown in Figure 2 of the Author Rebuttal attachment, from which we can observe that the degree of relation and attribution hallucination increases with the length of the descriptions, indicating that our findings and method are applicable to other forms of hallucination. Expanding to other tasks will be the focus of our future work.\n\n>**Q3**: What could potentially explain the phenomenon observed in Figure 2, where longer output lengths result in higher object hallucination degrees?\n\n**Response to Q3:**\nIn Appendix 6.2, we further explore this phenomenon and find that MLLMs are more likely to employ hallucinogenic words in generating lengthy and detailed image descriptions, resulting in a higher hallucination rate. \n\n> **Q4**: How are the 25 instructions used in experiments designed? Are they generated randomly or based on specific rules? Besides, why is it 25, and what difference would there be if there are more or less instructions?\n\n**Response to Q4:**\nTo obtain a diverse and extensive set of instructions, we referred to those from existing works and additionally designed some of our own. The value of 25 was not meticulously chosen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tvGzE2hR1R", "reviewer": "Reviewer_XTeq", "summary": "The paper identifies a pitfall regarding the length of image descriptions in the current average-based LVLM hallucination evaluation framework. To address this, they propose a new Length-Hallucination Curve Based evaluation framework to enhance the fairness of evaluations. The paper observes that the degree of object hallucinations is primarily influenced by the length of image descriptions, with instructions indirectly affecting hallucinations through their impact on description lengths. They suggest using a linear regression curve for evaluation and develop two metrics based on this curve. Extensive experiments on multiple LVLMs with different instruction sets demonstrate the stability of their proposed new evaluation metrics.", "questions": "- The paper grouped 25 different instructions to 5 instruction set. What’s the grouping strategy? How do the author group these instructions? \n\n- The paper claimed that object hallucination is primarily influenced by the length of image descriptions, with instructions only indirectly affecting hallucinations through their effect on description lengths. How is this claim being validated? Specifically, how do the author validate that the length of the image description is the primary cause and is not also affecting the hallucinations indirectly through their effect on some hidden factors ? The observation could be due to the spurious correlation.\n\n- Does the increased length of the image description also capture more real objects, or does it mainly consist of rephrasing and hallucinatory sentences?", "limitations": "Yes, the author adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The observation is intuitive and validate by extensive experiments\n\n- The paper is clearly written and easy to follow\n\n- The evaluation is comprehensive in terms of numerous instructions and LVLMs", "weaknesses": "- Although paper observe the linear relation between the length of the image description and objection hallucination, there are still unanswered questions regarding the justification of the claim. Please see questions below.\n\n- Some minor inconsistent typo, for example, the AEF and ABF in Figure 4.\n\n- The evaluation only use CHAIR scores and scores of other aspects is not evaluated, for example, the detail or the coverage of the real objects in the description as in AMBER.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1:** Although paper observe the linear relation between the length of the image description and objection hallucination, there are still unanswered questions regarding the justification of the claim. Please see questions below.\n\n>**Q2:** The paper claimed that object hallucination is primarily influenced by the length of image descriptions, with instructions only indirectly affecting hallucinations through their effect on description lengths. How is this claim being validated? Specifically, how do the author validate that the length of the image description is the primary cause and is not also affecting the hallucinations indirectly through their effect on some hidden factors ? The observation could be due to the spurious correlation.\n\n**Response to W1 and Q2:**  Thanks for your insightful comment. Intuitively, longer descriptions are more prone to hallucinations because the latter parts of the description can be influenced by the earlier parts, leading to cumulative hallucinations; Experimentally, we conducted extensive experiments across different models, datasets, and decoding strategies, and consistently observed a strong correlation between output length and the hallucination degree in these experiments. Theoretically establishing the causal relationship between output length and the hallucination degree will be a future direction of our work. \n\n***\n\n>**W2**:Some minor inconsistent typo, for example, the AEF and ABF in Figure 4.\n\n**Response to W2:** Thanks for your careful review and we will revise these typos.\n***\n\n>**W3**: The evaluation only use CHAIR scores and scores of other aspects is not evaluated, for example, the detail or the coverage of the real objects in the description as in AMBER.\n\n>**Q3:** Does the increased length of the image description also capture more real objects, or does it mainly consist of rephrasing and hallucinatory sentences?\n\n**Response to W3 and Q3:** Thanks for your valuable comment. The increased length of the image description also captures more real objects.  Following your suggestion, we analyzed the Coverage Ratio of image descriptions generated by 12 LVLMs under 25 instructions, defining the Coverage Ratio as |{real objects in description}|/|{all real objects in figure}|. The results are shown in Figure 1 of the Author Rebuttal attachment, from which we can observe that longer image descriptions result in a higher Coverage Ratio, capturing more real objects. This suggests that the Coverage Ratio is also influenced by length bias.  \n\nFurthermore, we evaluated the stability of the Coverage Ratio when applying LeHaCE and the average-based framework. The results are in Table 2 of the author rebuttal pdf. From the results, we can observe that LeHaCE demonstrates greater stability compared to the average-based framework, and its stability further improves with the addition of more instructions.\n***\n\n>**Q1:** The paper grouped 25 different instructions to 5 instruction set. What’s the grouping strategy? How do the author group these instructions?\n\n**Response to Q1:** Thanks for your comment. As described in Section 4.4, first paragraph: **“Specifically, LVLMs are prompted by three sets of different instructions to generate three sets of image descriptions. Each instruction set consists of multiple instructions randomly drawn from a pool of 25 instructions, with no overlap between instructions in different sets.”** We randomly selected three non-overlapping instruction sets from the 25 instructions to ensure reliable evaluation.\n***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jY0TCs0BR0", "reviewer": "Reviewer_Pvzh", "summary": "This work aims to establish a stable, fair, and comprehensive evaluation method for object hallucinations in large vision-language models. The authors discovered a positive correlation between the length of image descriptions and the degree of object hallucination. Building upon this observation, they developed a hallucination evaluation method named LeHaCE by fitting a length-hallucination curve. LeHaCE enables the evaluation at any given image description length, ensuring stability and fairness in the evaluation process. Additionally, LeHaCE involves the curve slope as a metric to evaluate the influence of image description length on the degree of object hallucination, thereby achieving a comprehensive evaluation. The motivation behind this work is reasonable, and the authors provide many experiments to support their claims. However, it is worth considering that the use of the linear fitting scheme, although straightforward, does somewhat diminish the novelty of the proposed method.", "questions": "1. Does the complexity of the image content, such as the number of objects, influence the extent of hallucination in the model? It would be valuable to investigate additional factors that impact hallucination degrees.\n2. Intuitively, the average-based framework can also be effective as long as there are enough instructions, such as 200 instructions. I'm wondering if this viewpoint is accurate?\n3. Is the relative standard deviation an appropriate approach to evaluate stability, considering that stability in this context essentially refers to the consistency of multiple evaluation results?\n4. Why does this work exclusively focus on object hallucinations? Is this a choice made by the authors or a limitation of the proposed method?\n5. In Figure 5, why does LeHaCE show higher instability on LLaVA and Qwen-VL when the image description length is less than 20 words?", "limitations": "The paper includes a simple discussion of the limitations.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The experimental analysis conducted on instructions and hallucination is compelling and provides strong support for the main argument that the hallucination degree is positively correlated with the length of the description. While previous research (Yifan et al., 2023) has already shown the influence of instructions on hallucinations, this work takes it a step further by proposing that instructions indirectly influence hallucinations through the length of image descriptions. This sheds light on the reason behind the limitations of previous approaches that relied on average-based methods. Overall, this paper offers valuable insights into the evaluation of consistent hallucinations.", "weaknesses": "1. Although the rationale behind the length-hallucination curve is compelling, it is fitted using a relatively simplistic linear approach. Exploring more flexible and intricate fitting approaches is worth considering, as it has the potential to achieve higher fitting accuracy and more effective hallucination evaluation.\n2. Since the proposed method relies on a fitted curve, it needs at least two instructions to evaluate LVLMs and cannot be used with just one instruction.The authors should discuss this limitation.\n3. Lack of indepth discussion on the shortcomings of the proposed method. For instance, as shown in Table 2, why does LeHaCE exhibit poor stability on a few LVLMs when the number of instructions is three?\n4. It seems that the selection of instructions might affect the stability of LeHaCE. It would be helpful to include more discussion on this aspect.\n5. The current paper seems to have lots of results and experiments. As a reader, it is not very easy for me to get the main conclusion for each experiment. It would be good to highlight the conclusions so that the readers can understand the point easier.\n6. Some typos need to be corrected: Line 79: lrv-instruction -> LRV-instruction. Line 92 Nope -> NOPE. Line 81 chatgpt -> ChatGPT. Table 2: Minigpt-4 -> MiniGPT-4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1:** Although the rationale behind the length-hallucination curve is compelling, it is fitted using a relatively simplistic linear approach. Exploring more flexible and intricate fitting approaches is worth considering, as it has the potential to achieve higher fitting accuracy and more effective hallucination evaluation.\n\n**Response to W1:** Following your suggestion, we supplement a comparative experiment for LeHaCE based on linear fitting versus quadratic and cubic polynomial fitting. The results are reported in Table 3 of the author rebuttal pdf.  These results show that linear fitting is significantly superior to polynomial fitting, especially when the number of instructions is small.\n***\n>**W2:** Since the proposed method relies on a fitted curve, it needs at least two instructions to evaluate LVLMs and cannot be used with just one instruction. The authors should discuss this limitation.\n\n**Response to W2:** In the typical practice of evaluating hallucination levels in LVLMs, multiple instructions are usually used to enhance the stability of the evaluation results. Although LeHaCE cannot be used with just one instruction, this limitation does not affect its ability to provide stable evaluations. We will discuss this limitation in the final version.\n***\n\n>**W3:** Lack of indepth discussion on the shortcomings of the proposed method. For instance, as shown in Table 2, why does LeHaCE exhibit poor stability on a few LVLMs when the number of instructions is three?\n\n**Response to W3:** Thank you for your valuable suggestions. LeHaCE exhibits instability with a low number of instructions because such a small sample is insufficient for an adequate fit of the length-hallucination curve. However, when the number of instructions is increased to five or more, LeHaCE consistently demonstrates improved stability.\n***\n\n>**W4:** It seems that the selection of instructions might affect the stability of LeHaCE. It would be helpful to include more discussion on this aspect.\n\n**Response to W4:** For the selection of instructions, a set of instructions that result in significant differences in the model output length will aid in fitting the length illusion curve.\n***\n\n>**W5:** The current paper seems to have lots of results and experiments. As a reader, it is not very easy for me to get the main conclusion for each experiment. It would be good to highlight the conclusions so that the readers can understand the point easier\n\n**Response to W5:** We will revise the paper to highlight the main conclusions of each experiment, making it easier for readers to grasp the key points.\n***\n\n>**W6:** Some typos need to be corrected: Line 79: lrv-instruction -> LRV-instruction. Line 92 Nope -> NOPE. Line 81 chatgpt -> ChatGPT. Table 2: Minigpt-4 -> MiniGPT-4.\n\n**Response to W6:** Thanks for your careful review and we will revise these typos in our final version.\n***\n\n>**Q1:** Does the complexity of the image content, such as the number of objects, influence the extent of hallucination in the model? It would be valuable to investigate additional factors that impact hallucination degrees.\n\n**Response to Q1:** Following your suggestion, We supplement an experiment about the correlation between the hallucination rates and the number of objects in the images for the Gemini, Llava, and MiniGPT-4 models on the COCO dataset, which were 0.08, -0.12, and 0.06, respectively. This indicates that the number of objects has almost no correlation with the hallucination degree of the models. Additionally, we found that LVLMs are more prone to hallucinations on some black-and-white images, suggesting that the style or domain of the images can affect the hallucination degree of LVLMs.\n***\n\n>**Q2:** Intuitively, the average-based framework can also be effective as long as there are enough instructions, such as 200 instructions. I'm wondering if this viewpoint is accurate?\n\n**Response to Q2:** No. Considering that different LVLMs produce outputs of varying lengths for the same instructions, increasing the number of instructions still cannot alleviate the length bias of the average-based framework.\n***\n\n>**Q3:** Is the relative standard deviation an appropriate approach to evaluate stability, considering that stability in this context essentially refers to the consistency of multiple evaluation results?\n\n**Response to Q3:** We believe that Relative Standard Deviation (RSD) is appropriate because the means of different metrics vary, making direct comparisons of standard deviations meaningless.Using RSD as an evaluation metric can eliminate the impact of the mean.\n***\n\n>**Q4:** Why does this work exclusively focus on object hallucinations? Is this a choice made by the authors or a limitation of the proposed method?\n\n**Response to Q4:** Object hallucination is the most common type of hallucination in LVLMs. Following your suggestion, we evaluate the relation hallucinations and attribution hallucinations in image descriptions generated by Qwen-VL and InternLM-XComposer under 25 instructions. We conducted a model-based evaluation using Gemini-1.5-flash. The results are shown in Figure 2 of the Author Rebuttal attachment, from which we can observe that the degree of relation and attribution hallucination increases with the length of the descriptions. This demonstrates that relation hallucination and attribution hallucination are also influenced by length bias, suggesting that our findings and methods are applicable to these types of hallucinations as well. Expanding to other tasks will be the focus of our future work.\n***\n>**Q5:** In Figure 5, why does LeHaCE show higher instability on LLaVA and Qwen-VL when the image description length is less than 20 words?\n\n**Response to Q5:** LLaVa and Qwen-VL have generally large output lengths, with minimums of 19 and 17, respectively (shown in Figure 3). This causes greater fitting deviations in LeHaCE's length-hallucination curve at shorter lengths, resulting in poorer consistency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K8uC3tSPep", "reviewer": "Reviewer_onEV", "summary": "This paper explores the stable evaluation of object hallucinations, which is a crucial challenge in large vision-language models. The authors provide the first systematic analysis of the underlying mechanism through which instructions affect hallucinations, based on comprehensive experiments. They report a linear correlation between the length of descriptions and the levels of object hallucinations. Furthermore, the authors propose a curve-based framework that incorporates description lengths to enable a stable evaluation of hallucinations. What I find particularly novel is that the slope of the curve is incorporated as a metric, which achieves a more comprehensive evaluation.", "questions": "1. Considering that shorter descriptions tend to have fewer hallucinations, have the authors explored the possibility of generating multiple concise descriptions with distinct focuses for the same image, and subsequently merging them into a comprehensive and detailed description?\n2. What factors determine the slope of the length-hallucination curve for the model?\n3. Since the authors introduce the slope of the length-hallucination curve as a valuable evaluation metric, it raises the question of what the intercept of the curve signifies. Is it feasible to incorporate the intercept into the evaluation framework?\n4. Why does the average length of the image description generated by the Otter model, specifically under instruction I12, amount to only 2? Is there any misunderstanding here?", "limitations": "The paper briefly mentioned limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This work might provide valuable insights to the community. Firstly, while the impact of instructions on hallucinations is widely recognized, this work unveils a crucial aspect by demonstrating that instructions exert their influence through the modification of description lengths. This finding illuminates the previously unexplored mechanism underlying instruction-affected hallucinations. Secondly, they employ a curve-based evaluation method instead of relying solely on a single metric, which goes a new way in addressing hallucination evaluation. Thus, this work has the potential to inspire further research and exploration in hallucination evaluation. \n2. The proposed curve-based hallucination evaluation method in this paper is intuitively reasonable, and the author provides substantial experimental evidence to support the motivation behind this method. The experimental results are clearly presented, and the corresponding analyses further enhance the persuasiveness of this work. Overall, the combination of the intuitive approach, extensive experiments, clear presentation of results, and insightful analyses makes this work convincing.", "weaknesses": "1. The proposed method realizes consistent evaluation by calculating the hallucination rate at a uniform length. However, the length distributions of descriptions generated by different LVLMs exhibit variations. In other words, some models tend to produce shorter descriptions while others generate longer ones. In light of this, I have concerns regarding the ability of this method to maintain its effectiveness under such circumstances.\n2. In my view, the hallucination evaluation of a LVLM in practical requires a large instruction set that could simulate real-world applications of the LVLM. If the authors can build such a large instruction set as the benchmark, it would yield a significant contribution to the community.\n3. The authors claim that their proposed evaluation method is fairer compared to other evaluation methods. However, the paper appears to lack experimental results to support this assertion.\n4. The analysis of the stability of the “LeHaCE_GR” is lacking.\n5. The selection of instructions may have a substantial impact on the fitted curve. It would be beneficial for the authors to provide further discussion on this aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1:** The proposed method realizes consistent evaluation by calculating the hallucination rate at a uniform length. However, the length distributions of descriptions generated by different LVLMs exhibit variations. In other words, some models tend to produce shorter descriptions while others generate longer ones. In light of this, I have concerns regarding the ability of this method to maintain its effectiveness under such circumstances.\n\n**Response to W1:** Thanks for your comment. In fact, calculating the hallucination rate at respective average lengths is a practice of the average-based framework, not our method. When the output lengths of different models are inconsistent, the average-based framework indeed suffers from evaluation inconsistency and unfairness. In our LeHaCE framework, by constructing the length-hallucination curve, we can evaluate the hallucination degree of different models at a specified uniform description length, thereby improving the consistency and fairness of the evaluation.\n***\n>**W2:** In my view, the hallucination evaluation of a LVLM in practical requires a large instruction set that could simulate real-world applications of the LVLM. If the authors can build such a large instruction set as the benchmark, it would yield a significant contribution to the community.\n\n**Response to W2:** Thank you for your constructive suggestion. We will give it serious consideration.\n***\n\n>**W3:** The authors claim that their proposed evaluation method is fairer compared to other evaluation methods. However, the paper appears to lack experimental results to support this assertion.\n\n**Response to W3:** Thanks for your comment. In Figure 4 of our paper, we present real data to visually demonstrate the fairness of our LeHaCE method. Our LeHaCE method constructs a length-hallucination curve to evaluate the hallucination levels of LVLMs at a uniform description length. This effectively mitigates the issue of length bias in hallucination level evaluation, providing a fairer evaluation.\n***\n>**W4:** The analysis of the stability of the “LeHaCE_GR” is lacking.\n\n**Response to W4:** Following your suggestion, we supplement an experiment on the stability of LeHaCE_GR.The results are reported in Table 1 of author rebuttal pdf. From the results, we can observe that the consistency of LeHaCE_GR increases with the number of instructions.\n***\n>**W5:** The selection of instructions may have a substantial impact on the fitted curve. It would be beneficial for the authors to provide further discussion on this aspect.\n\n**Response to W5:** Thanks for your valuable question. For the selection of instructions, a set of instructions that result in significant 5.differences in the model output length will aid in fitting the length illusion curve.\n***\n>**Q1:** Considering that shorter descriptions tend to have fewer hallucinations, have the authors explored the possibility of generating multiple concise descriptions with distinct focuses for the same image, and subsequently merging them into a comprehensive and detailed description?\n\n**Response to Q1:** Thank you for your insightful suggestion. We have indeed tried this method. Specifically, we first had the LVLMs list the objects in the image, then prompted the LVLMs to generate descriptions for these objects individually, and finally summarized the information using the LVLM. However, due to the limited ability of LVLMs in listing objects and summarizing, this approach did not yield ideal results. Utilizing multimodal agent technology to achieve this idea with multiple different models is our future research direction.\n***\n>**Q2:** What factors determine the slope of the length-hallucination curve for the model?\n\n**Response to Q2:** We believe that various factors influence the length-hallucination curve of LVLMs, such as the training data, the visual encoder and the language model in LVLMs.\n***\n>**Q3:** Since the authors introduce the slope of the length-hallucination curve as a valuable evaluation metric, it raises the question of what the intercept of the curve signifies. Is it feasible to incorporate the intercept into the evaluation framework?\n\n**Response to Q3:** We believe that the intercept of the length-hallucination curve is not practically meaningful. This is because the output lengths of large vision-language models (LVLMs) are always positive, and for image descriptions, they typically contain at least a dozen words. Given this context, the intercept holds little practical significance.\n***\n**Q4:** Why does the average length of the image description generated by the Otter model, specifically under instruction I12, amount to only 2? Is there any misunderstanding here?\n\n**Response to Q4:** This is because the Otter model always returns the names of the objects in the image rather than a description of the image when given the I12 instruction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers and area chairs for your time and effort during the review process. We are encouraged to hear that our work has **clear and well-written presentations** (by all Reviewers), **good motivation** (by Reviewer Pvzh and Ffbd), **convincing analysis** (by Reviewer Pvzh and onEV), **novel** (by Reviewer onEV) **and effective** (by Reviewer Ffbd) **technical contributions**,  **valuable insights** (by ReviewerPvzh and onEV), **extensive experiments** (by Reviewer Pvzh and onEV), and **comprehensive evaluation** (by reviewer XTeq ). 

During the rebuttal phase, we meticulously give point-by-point responses to your comments, and further add the additional experiments and figures into the one-page supplementary PDF. Especially, 
- we provided extensive evaluations of LeHaCE on more hallucination evaluation metrics, including coverage ratio, relation hallucination, attribute hallucination, and more decoding strategies (greedy decoding), which further validate the effectiveness and versatility of our findings and method. 
- We also provided comprehensive ablation experiments on the fitting methods for LeHaCE, which validated the reasonableness of using linear fitting. 
- Furthermore, we conducted a more comprehensive and in-depth analysis of our findings. 

We hope that our responses adequately address all your concerns and meet the expectations of the conference committee.

Meta-review: This paper finds that the description length strongly correlates with the levels of object hallucinations. Authors further propose to use the slope of the curve as a metric for evaluation. Reviewers consistently gave accept, borderline accept, borderline accept, and accept. Reviewers generally like this paper and think the papeer is novel and easy to understand. The AC agrees on these aspects and recommend accept.
Decision: Accept (poster)