Multi-Object Hallucination in Vision Language Models
Authors: Xuweiyi Chen
Ziqiao Ma
Xuejun Zhang
Sihan Xu
Shengyi Qian
Jianing Yang
David Fouhey
Joyce Chai
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Vision Language Models
Object Hallucination
Visual Prompting
Primary area: machine_vision
TL;DR: We study the problem of multi-object hallucination  and introduce Recognition-based Object Probing Evaluation (ROPE), an automated evaluation protocol.

Abstract: Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images. 
While current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities, this work systematically investigates multi-object hallucination, examining how models misperceive (e.g., invent nonexistent objects or become distracted) when tasked with focusing on multiple objects simultaneously.
We introduce Recognition-based Object Probing Evaluation (ROPE), an automated evaluation protocol that considers the distribution of object classes within a single image during testing and uses visual referring prompts to eliminate ambiguity. 
With comprehensive empirical studies and analysis of potential factors leading to multi-object hallucination, we found that (1) LVLMs suffer more hallucinations when focusing on multiple objects compared to a single object. 
(2) The tested object class distribution affects hallucination behaviors, indicating that LVLMs may follow shortcuts and spurious correlations.
(3) Hallucinatory behaviors are influenced by data-specific factors, salience and frequency, and model intrinsic behaviors.
We hope to enable LVLMs to recognize and reason about multiple objects that often occur in realistic visual scenes, provide insights, and quantify our progress towards mitigating the issues.

Multi-Object Hallucination in Vision Language Models
Xuweiyi Chen 1.2+
Ziqiao Ma
Xuejun Zhang"
Sihan Xu'
Shengyi Qian'
Jianing Yang'
David F. Fouhey Joyce Chai
University of Michigan
Universitty of Virginia
NNww York University
attps://mule object- boott-aallciiaation. github.io.
Abstract
Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images. While current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities, this work systematically investigates multi-object hal- lucination, examining how models misperceive (e.g., invent nonexistent objects or become distracted) when tasked with focusing multiple objects simultancously. Object Probing Evaluation (ROPE), an automated evaluation protocol that considers the distribution of object classes within single image during testing and uses visual referring prompts to eliminate ambiguity. With comprehensive empirical studies and analysis of potential factors leading multi-object hallucination, we found that (1) LVLMs suffer more hallucinations when focusing multiple objects compared to single object. (2) The tested object class distribution affects hallucination behaviors, indicating that LVLMs may follow shorteuts and spurious correlations. (3) Hallucinatory behaviors influenced by data-specific factors, salience and frequency, and model intrinsic behaviors. We hope enable LVLMs to recognize and reason about multiple objects that often occur in realistic visual scenes, provide insights, and quantify our progress towards mitigating the issues.
We introduce Recognition
Introduction
Recent advances in large language models (LLMs) have motivated increasing efforts in adapting them for understanding visual semantics, giving rise surge of large vision language mod- els (LVLMs) 1
37 43 These models, whether explicitly trained with grounding data [70 or without 31 demonstrate an impressive grounded understanding of visual entities. This moti- vates new prompting paradigm based on user-pre bser-provided visual cues, referred to as visual prompt- 58 [88. Despite their promising performances on various downstream applica- tions 36 LVLMs often suffer from object hallucination 44 10 26 where they produce objects not present in given image.
ing [45] 60 27
Although object hallucination was initially observed in image captioning describing multiple ob- jects [44], current benchmarks for object hallucination primarily concentrate the presence of single object class rather than individual entities. These benchmarks either verify if an object class mentioned in the caption can ground to object in the image 44 19]. or probe the model about the existence of an object class, sometimes with additional attributes or relations to other objects [26] 32 There are, however, two key limitations with these setups as shown by case study Figure First, grounding is not simply one-to-one between objects and classes, but many-to-mant -any mapping between objects and phrases (20 34 For instance, "apples" could potentially correspond to multiple referents in Figure [] and the model doesn't 't necessarily need to recognize all of them to provide such a response. Therefore, being able to produce an object that exists an image does not necessarily
Authors contributed equally to this work, alphabetized by last names.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Provide detailed description of the given image. There are also multiple pieces of fresh fruit, such as apples, pears, and lemon. Additionally, there are three jars jam and packet of M&M's Skittles the table. To the side, there's a bottle of water, and utensils including a whisk, a knife, and some spoons placed inside blue mug... (b) Captioning based evaluation.
BAP
GPT-4V
Is there whisk in this image?
User
Yes, there is whisk the image, placed next to knife and bowl on the right side of the table.
GPT-4V
Seloct one and the most appropriate class for each object located within red bounding boxes from the following list: apple, orange, bamana, lemon, pear, plate, bowi,jarr bottle, soda can, fork, whisk, scissors, packaged snack.
(c) Polling- based object probing.
Is there a whisk next to a knife?
User
Yes, there is a whisk next to knife on the table in the image you provided.
GPT-4V
(d) Object attribbue attributeerelation probing.
User
Provide the class names in the format: 'obj1: <class1>, obj2: <class2>, obj3; <class3>, obj4; <class4>, obj5: <class5>', with no additional words punctuations.
Does the caption accurately describe the image: "A whisk placed the right of knife."
User
No, the caption does not accurately describe the image. The whisk is actually placed to the left of the knife on the table.
objı: apple, obj2: knife, objg: fork, obj4: apple, obj5: jar
GPT-4V
(a) Recognition- based object probing.
(e) Counterfactual probing.
Figure 1: A case study that compares our Recognition- broognition-bbsed Object Probing Evaluation (ROPE) benchmark with existing benchmarks for object hallucination in GPT-4V. ROPE offers an automated evaluation protocol with controlled output formatting and uses visual prompts to distinctly ground to objects, thus mitigating referential ambiguity. Unlike binary inquiries relying solely on textual descriptions, ROPE challenges the model to identify multiple objects concurrently. We observe that, while GPT-4V can identify the whisk to the left of a knife when prompted about it, the model hallucinates a "fork" when directly tasked to recognize multiple objects.
indicate that the model is free of hallucinations. Second, explicitly instructing the model to recognize multiple objects poses greater challenges compared to simple yes/no inquiries that contain explicit text descriptions for individual objects. For instance, while the model can correctly identify that a whisk is positioned to the left knife when "a whisk" is deliberately prompted, as shown in Figure db)d), it may hallueinate "fork" when directly prompted to recognize both the whisk and the knife (i.e., Figure [h). This could be due to the common association between knives and forks, which leads potential hallucinations when models are tasked to focus on multiple objects the same time. In real-world applications, multi-object querying is crucial for embodied AI tasks. For example, in cooking scenario, an agent must recognize multiple ingredients and tools simultancously to be effective. We also present case study on autonomous driving (see Figure Siin the appendix), demon- strating how common associations between cars, pedestrians and traffic lights could lead to potential hallucinations. In addition, evaluating multiple objects simultancously. rather than querying each object individually, can significantly save both time and resources. To enable LVLMS to recognize and reason about multiple objects that often occur in realistic visual scenes and to better quantify the complex phenomena we observed, this paper investigates multi-obe obbjcct hallucination, examining how models may misperceive (e.g.. by inventing nonexistent objects or becoming distracted) when tasked to focus on multiple objects concurrently, and which factors cause the hallucinations.
We start by introducing Recognition- Object Probing Evaluation (ROPH) for assessing multi- object hallucination with formatted output control. ROPE features an automated evaluation protocol without black-box neural models or humans as evaluators, and leverages visual prompts to uniquely refer to objects to avoid ambiguity and multiple referents caused by object class names. ROPE considers the distribution of object classes within each image at test time, dividing ROPE into 4 subsets: In-the th- Wild, Homogeneous, Heterogeneous, and Adversarial. For instance, we investigate scenarios where all tested objects belong to the same class or where each tested object represents a different class. We conduct an in-depth analysis of the hallucination behaviors of LVLMS of different scales and training data (e.g., whether grounding data and conversational data are used), and provide comprehensive analysis of potential factors that lead to multi-object hallucination. Our main findings are: (1) LVLMs suffer from more hallucinations when tasked to focus multiple objects, compared
2

---

to focusing on a single object; (2) The tested object class distribution affects the hallucination behaviors, revealing that LVLMs may be following shortcuts and spurious correlations; (3) The hallucinatory behaviors of LVLMs are affected by data-specifie factors, salience and frequency, and model intrinsic behaviors. These findings provide key insights for the development and application of LVLMM, suggesting for more balanced object distributions, diverse annotations, and enhanced multi-object instructions in grounded LVLMs. We hope this work takes a step towards LVLMs that recognize and reason about multiple objects that often occur in realistic visual scenes.
2 Related Work
Large Vision-Language Models. There is growing trend harness and adapt the powerful large language models (LLMs) for multimodal understanding beyond text [ I 65 41]. Especially, visual instruction tuning has gained prominence for its competitive performance with a comparatively moderate amount of data and computational resources, leading to variety of Large Vision- LanguaLa
Models (LVLMs)) 31 30 76
12
56
62
22]. Grounding datasets have been shown to benefit
vision-language pre-training K3
25 34
Researchers have developed family of grounded LVL.Ms
focusing on object grounding to bounding box [40 g 72
3 63
70 39] and segmentation masks [1
69
15 42 71. Of the large space of LVLMs, work is most related to visual prompting [58] 60
and object hallucination [44, 10). The paragraphs below describe the two lines of work in detail.
Visual Prompting. LVLMs demonstrate their grounded understanding of user-provided visual cues,
giving rise to a practical and user-friendly prompting paradigm known as visual prompting [58] 60)
Early work on visual prompting in vision-language models can date back to tuning-
methods 2
61. Recent studies show that LVLMs demonstrate zero-shot understanding of user- aser-provided visual 45| 60]. This observation allows prompting LVLMs by editing images directly in the pixel space, e.g., by adding visual marks or visual text [27]. Starting from Met-of-Marks
cues (e.g., a red circle)
(SoM) prompting [58), several training-free methods have been introduced [24 59 52 Recent
work further enhances visual prompt understanding by additional visual instruction tuning with diverse visual prompts overlaid on the images (.. or explicitly represent visual pointer tokens the
models (21 63 71.. We leverage visual prompting to avoid potential ambiguity in textual descriptions,
especially when evaluating multiple object hallucinations for objects of the same class.
Object Hallucination. Despite their promis- ing performance on benchmarks, these models frequently generate objects that do not exist in the provided images, a problem known as ob-
Design Considerations Multi. Distr. Source Ref. Eval.
Benchmark
#Test
CCEval 68
Scen Text
0.1k
GAVIE
Mixed Text
lk
ject hallucination 44 10]. Several methods
FAITHScore
19
Unseen Text N
2k
have been suggested to mitigate the object hal-
HaELM
64
Unseen Text N
Sk
M-IlalDelect
13
Unseen Text
II
0.8k
lucination issue, such as integrating an external object detector [68], applying visually grounded visual instruction tuning [63] [1] or reinforce- ment learning [46, 13). performing iterative re- finement 775], and adapting the decoding strate- gies [17] To quantify progress on mitigating them, various benchmarks have been developed and have revealed the prevalence of object hallu- cination, even in images that are seen during in- struction tuning [68] (2)]. We contrast our ROPE benchmark against existing benchmarks and se- tups in Table ROPE, which is designed for evaluating multi-object hallucination, is distin- guished in several ways. First, we deliberately consider the distribution of object classes within single image at test time. Object hallucination is observed originally in image captioning ap- plications, where multiple objects are described that the object class distribution in the instruction tuning dataset can influence hallucination pat- terns 26 75 53 the impact of object class distribution within an image at test time remains under-explored. Second, current benchmarks concentrate on the presence of an object class or distinguish instances using textual descriptions like attributes, which can still result in ambiguity and multiple referents. We instead leverage the visual referring prompting setups and use visual cues (i.c..
MMHal-Bench
46
Unseen Text N,H
0.
CHAIR 14
Unseen
46k
AMBER 53
Unseen Text
1k
CIEM 15
Unseen Text
A
5k
NOPF 12
Unseen
A
3k
POPE
Train Unseen
A
0.5k
Seen & Unseen
ROPF (Ours)
Test
Vis. A
Sk
Table 1: An overview of object hallucination benchmarks. For design considerations, we sum- marize the number of tested images, and if multiple classes and object class distribution (at training and test time) are considered. The image sources include those seen or unseen during instruction tun- ing. To refer an object, textual descriptions and visual cues can be adopted. For evaluation, neural models, humans and automatic pipelines are used,
44
While existing research has demonstrated
3

---

marked bounding boxes) to refer to objects. Finally, our evaluation is automated, without black-box -bxx neural models or human evaluators.
3 Recognition-based Object Probing Evaluation
We introduce the Recognition Object Probing Evaluation (ROPE), an automated protocol for assessing LVLMs in multi-object recognition. ROPE specifically measures object hallucination in VLMs within multi-object setting, examining how models may misperceive (e.g.. by inventing nonexistent objects or becoming distracted) when tasked to focus on multiple objects concurrently, and which factors cause the hallucinations.
3.1 Task Setup
Problem Definition. To avoid ambiguity from multiple candidate referents when using text prompts, ROPE leverages visual prompts to uniquely refer to objects. ROPE tasks LVLMs with selecting the best matching class for multiple objects, as referred to by the visual prompt, from a predefined set of object classes. Specifically, each sample in the ROPE protocol consists of quadruple {I, L. (pI.... Pn); (o),... on)): (1) an image I consisting of at least n objects; (2) natural language instruction L that specifies the recognition task, including N candidate object classes C1.... CN; (3) n visual prompts P1.... Pn. each queries an object in the image; and (4) n object classes 01. , On the answers. In this work, we construct dataset with N = 50 and models are tasked with recognizing objects out of 50 candidate object classes. Although we use this dataset as example, ROPE can be applied any dataset containing multiple objects and their bounding boxes.
5. i.c.
Language Instruction Prompts. For a fair comparison that accommodates both gpen-weight and API-based LVLMs, ROPE explicitly instructs models to generate formatted output of object classes, e.g., obj1: <classi>, .... <class>> (Figure 2 This format enables automated evaluation through simple parsing. This format enables automated evaluation through simple parsing, avoiding black-box neural models or human evaluators With different analytical purposes, we designed types of task prompts for Multi- Object queries, as illustrated in Figure 7 and described as follows.
Default: We probe the model to recognize the 5 objects referred to by the visual prompts concur- rently in single turn of prompting. This setting tasks the model with focusing and recognizing all 5 objects simultancously, aiming to capture the complexity involved when the model generates language that includes multiple objects.
Student-Fon Forcing: One potential confounder in the default setting the model's ability to generate data in the specified format. To separate out errors due to following instructions, we force the model to follow the format template and decode only the object tokens for each of the five objects. Ideally, this setting allows the model to focus solely on object recognition.
Seachee-FForcing This setting eliminates cumulative error, allowing the model condition on the correct previous context when generating object classes, leading to upper bound performance in multi-object recognition. We similarly force the model follow the provided template and decode only the object tokens for each of the five objects, but we replace the previously generated object tokens with the ground truth. This essentially follows the few-shot in-context learning setting. Teacher forcing helps especially when LVLMS take shortcuts by repeating the object class list as ordered in the prompt (c.g., LLaVA-7B [1 and Gemini 1.0 Pro [49] in Figure
For comparison, we also designed task prompts for Single-Ohjec query. We probe the model to recognize the object referred to by the visual prompts one at time, repeating this as 5 independent and individual prompts. Unlike Default multi-object query, the model only needs to focus on one object, which can be seen an extension of the POPE (26) setup from yes/no polling to classification. We refer to Appendix[A.I. for the prompt templates for each type of task prompt.
3.2 Dataset Construction
Data Sources and Curation. Since our goal is to evaluate and analyze multi-object hallucina- tion, the image data must contain multiple objects of diverse classes with instance-level semantic annotations. We build our dataset upon existing panoptic segmentation datasets, including MSCOCO-
Panoptic [28] 5 and ADE20K (74), to ensure access to all object instances and their semantic classes.
We note that one can build dataset using the ROPE protocol with any dataset containing multiple objects and their bounding boxes. We describe the data curation pipeline in Appendix A.I
Splits by Query Distributions. As shown in Figure and 3 our initial observations indicate that LVLMs are less likely to hallucinate objects when they are tasked with recognizing the same object

---

Select one and most appropriate class for each object located within red bounding boxes from the following list: apple, orange, banana, Jemon, pear, plate, bowl, bottle, soda con, knife,fork, whisk, scissors, packaged snack.
Provide the class in the format: 'obj1: <classl>, obj2 <class2>, obj3: <class3>, obj4: <class4s, obj5: <class5>>, with
no additional words punctuations.
CPT-4V
Gemini Pro
@-wen-VL-C VL-Chat
LLaVA-7B
obja: apple
obji: apple
objı: apple
obj1: apple
ubj2: knife
obj2: orange
obj2: lemon
obj2: orange
obi3: fork
obis: banana
obj3: bottle
ohi3: hanana
ubj4: apple
obj4: lemon
obj4: packaged snack
obj4: lemon
abj5: jar
obi5: pear
objs: jar
ohj5: pear
Ground Truth
SGPT-40
Gemini 1.5 Pro
Qwen-VL-Max
LLLaVA-348
obja:
obju: packaged snack
obji: fock
obja: packaged snack
obj1: apple
obja: knife
obj2: knifé
obj2: knile
obj2: knife
objj: apple
abj3: whisk
ubja: whisk
objs: whisk
obj3: soda can
objo: apple
obj4: lemon abjs: jar
ubj4: lemon
obj4: lemon
obj4: lemon
obj4: lemon
objs: jar
obi5: jar
obj5:
obj5: pear
Figure 2: A heterogeneous ROPE sample tested with Deafult multi-object query, where each of the
objects belongs to different object classes. We label the output class either correct or hallucinated.
LLLVAA-BB
A LLLVA-7B
objt: applé
ohjt: apple
obj2: apple
obj2: apple
ubj3: apple
obj3: apple
obj4: apple
obj4: apple
objs: apple
obj5: apple
Ground Truth
LLaVA-34B
Ground Truth
LLaVA-34B
obju: apple
obju: apple
obju: apple
objı: apple
obj2: apple
ubj2: apple
obj2: apple
objz: apple
obj3: apple
ubj3: apple
obj3: apple
obj3: apple
abj4: apple
obj4: apple
obj4: apple
obj4: apple
objs: apple
obj5: apple
obi5: orange
objs: apple
Figure 3: A homogencous ROPE sample, where the objects belong to the same object class, and corresponding adversarial ROPE sample, where the last object belongs to a different object class.
class multiple times. However, they tend to make more mistakes when all tasked object classes are different or when a new object class is introduced after multiple repeated tasks. We thus consider the distribution of object classes within each image at test time, dividing ROPE into 4 subsets: Homogeneous, Heterogeneous, and Adversarial, In-the- Wild.
Homogeneous: All the 5 tested objects are of the same class, c.g., c.g., AAAAA.
Heterogeneous: All the 5 tested objects of different classes, e.g., ABCDE.
Adversarial: The first 4 tested objects are of the same class while the last is different, e.g., AAAAB. In-the-Wild Wild: A subset with mixed object class distribution, where the 5 tested objects are randomly chosen and ordered given test image.
Attending to Data Contamination. While data contamination has been explicitly handled in most of the existing benchmarks, object hallucination has been observed even in images that appear in the instruction tuning dataset, such as Visual Genome [68 29]. To evaluate whether multi-object hallucination can be observed both seen and unseen images, and to critically determine if training on these images helps reduce hallucinations, we explicitly split our dataset into Seen and Unseen based on the original split the datasets Depending on the object query distributions (4 splits) and whether the image appears in the training split (2 splits), divide the test into 8 folders.
4 Experiments and Results
4.1
LVLM Baselines
The proposed ROPE framework, in principle, applies to all LVL.Ms that can follow format instructions and understand multiple visual prompts. To cover a variety of LVLMs of different scales and training data (e.g., whether grounding data and conversational data are used), we selected the following LVLMs as baselines.
LVLMs with base LLMs at different scales: LLaVA v1.6 (7B/13B/34B) 31 30 (6B/34B) 644.
and Yi-VL
"We believe this approach is the best practice, but we also acknowledge that the distinction between seen and unseen images may not be strict. Uncurated web images often overlap with public test images, and researchers have no transparent access to the datasets used to train of these LVLMs unfortunately [[]]
5

---

Default Multi-
Guudent-Forciigg
Teacher-Forcing
Single-Objec
Models
Wild
Hom.
Het.
Wild
Hom.
Het.
Wild
Hom.
Het.
Wild
Hom.
Het.
Seen
Yi-VL-6B
2.95
5.65
1.99
3.44
6.80
3.78
5.45
26.25
4.36
0.19
0.30
0.13
Yi-VL-344 -3B
8.50
15.35
3.33
8.97
16.30
4.23
10.09
19.75
4.94
0.22
2.60
0.13
LLaVA-7BB
31.29
67.50
8.00
31.28
67.25
11.22
31.49
92.15
12.37
35.32
62.35
17.37
LLaVA-13B
31.54
67.63
12.64
31.49
73.25
11.54
34.97
94.25
16.03
43.13
80.60
23.91
LLaVA-34B
39.95
85.75
18.85
52.75
85.20
33.91
56.41
95.81
25.31
55.05
86.50
18.97
Qwen VL
2.73
6.60
1.03
6.25
16.00
3.65
18.74
71.50
5.45
8.73
16.05
5.58
Owen VL-C
8.72
16.90
6.67
5.26
8.60
4.10
12.11
47.75
8.08
25.99
43.40
13.21
CogVLM
0.04
0.00
0.00
0,00
0.00
0.00
0.10
0.95
0.00
0.00
0.00
0.00
CLM-
0.00
0.00
0.00
9.86
13.50
6.79
22.64
75.45
11.25
22.65
7.12
CMMVLM-C
12.89
22.75
7.18
25.37
43.63
12.03
28.25
72.80
17.50
30.16
56.00
16.35
LLaVA-7B*
N/A
9.16
16.40
5.51
N/A
11.68
23.55
9.36
GLaMM
N/A
27.11
53.35
13.01
N/A
63.81
81.75
53,40
GroundHOG*
N/A
23.57
30.80
24.23
N/A
44.80
43.10
38.97
IDEFICS
0.00
1.45
0.13
6.25
18.70
0.64
17.37
76.15
10.06
4.62
0.00
0.32
C-2
21.51
37.55
17.31
37.02
70.85
12.69
37.10
73.50
17.44
21.16
38.75
13.65
MiniCPM-V
34.75
59.91
17.37
31.62
62.80
13.65
32.16
68.05
16.79
27.42
55.35
16.92
GPT-4V+
53.80
77.55
40.83
N/A
N/A
55.89
78.25
41.03
GPT-40
71.27
89.25
66.03
N/A
N/A
60.77
73.92
54.31
LLaVA-7B
21.26
52.40
7.69
N/A
N/A
30.59
60.85
12.69
+OPERA
24.07
58.65
7.35
N/A
30.44
60.85
13.27
Unseen
Yi-VL-6B
2.74
3.88
1.14
3.18
4.24
5.20
4.04
10.90
10.57
0.14
0.45
0.08
Yi-VL-34B
7.77
15.63
4.23
10.28
18.04
7.97
11.24
22.49
12.03
0.46
2.37
0,41
LLaVA-7B
30.56
68. 12
10.33
30.55
68.16
10.24
31.89
90.33
13.25
34.88
64.41
16.18
LLaVA-13B
27.56
63.
8.37
27.41
8.37
35.65
91.09
14.80
42.66
71.92
23.41
LLaVA-34B
29.30
79.43
17.72
29.45
91.18
14.39
37.40
95.51
17.92
51.71
77.88
30.81
Owen VL
2.80
1.95
7.06
7.17
16.41
4.15
10.34
58.00
4.07
17.73
31.22
9.51
Owen VT-C
18.86
30.73
8.78
16.16
27.80
7.72
21.81
58.00
11.14
34.20
57.31
15.37
CLgVLM
0.03
0.00
0.00
0.00
0.00
0.00
0.00
0.15
0.00
0.00
0.00
0.00
CLMVLM-G
0.00
0.00
0.00
8.20
1.47
5.77
23.82
81.20
1.81
10.32
10.74
9.11
CLgVLM-C
15.56
26.57
5.53
17.18
41.27
6.02
22.81
56.04
30.56
52.00
13.50
LLaVA-7B*
N/A
7.59
12.12
4.88
N/A
12.71
22.49
8.46
GLaMM
N/A
29.11
$4.53
14.23
N/A
68.65
77.06
$2.28
GroundHOG*
N/A
23.11
24.69
26.26
N/A
40.73
30.37
38.13
IDEFICS
0.39
0.37
0.33
9.03
24.45
2.68
24.80 80
83.02
7 7.64
4.62
3.67
6.50
VL-2
20.99
35.06
15.93
24.64
38.04
23.17
26.74
46.04
26.59
11.13
30.94
5.77
MiniCPM-V
32.96
59.92
16.60
31.77
58.98
14.13
31.87
60.98
16.34
25.56
47.76
14.39
GPT-4V+
45.46
63.12
34.17
N/A
47.34
64.94
35.45
GPT-40
63.27
80.29
34.47
N/A
N/A
63.45
79.84
53.74
LLaVA 7Bt
13,96
31.88
3,98
N/A
26.95
54.41
11.06
+OPERA
13.20
37.14
3.82
N/A
N/A
27.90
56.69
11.22
Mechanistically grounded LVLMs visual prompts by dedicated pointer tokens. We slightly adapt the text prompt and probe the object classes with the highest probabilities. We also apply such probabilistic probing to LLaVA-7B for comparison, as all of three models adopt Vicuna-7B v1.5 [7] as the base LLM. See Appendix[A.] for details.
For GPT models, student/teacher forcing doesn't apply they are API-only. OPERA is implemented hased on LL-B v1.5.
Table 2: Averaged accuracy baselines on the In-the- Wild, Homogeneous, and Heterogeneous splits. The bold marker denotes the best-performing bascline and the underlined marker denotes the scooddbbest-pefoomingg baseline.
LVLMs with conversationallgrronneed instruction tuning: WweassL-BaseCChat (7B) and CogVLMM-Bsse/ChaGGroonnn v1.1 (19B) (55).
Mechanistically grounded LVLMs: GLaMM (7B)
42
and GroundHOG (7B)
LVLMs with RL-based finetuning: MiniCPM-V
66
Other LVLMs: IDEFICS-instruet (9B) GPT-40 [38].
23
MiniCPM-V v2.5 (8B) 16 67 GPT-4V 37 and
For mechanistically grounded LVLMs that take visual prompts through specially designed mecha- nisms, such as pointer tokens in GroundHOG [71], we additionally experiment with their default format and report whichever yields higher performance. For other LVLMs, we overlay the visual prompts on the images using a red bounding box with width of 2 and visual text specifying the object index. presented with white italic font black background with an alpha value of 0.75 for contrast and visibility.

---

100
Accurcy
79.3
100.1
36.7
9733
85.6
100.0
100.0
99.7
100.0
68.2
94,8
95.3
78.8
92.3
AAAAB-SO
50/55.4
52.4
50.6
52.1
54.0
51.
BAAAA-SO
AF
1982
29.0
0.0
13.0
26
0.0
52
BAAAA-IF
obj 1 ob) 2 obj 3 obj 4 obj 5
obj
ob) 2 obj obj 4
0bj 5
obj 1
obj 2 obj 3 obj 4 obj
Object Index
Object Index
Object Index
(a) LLaVA-7B.
(b)
(c) LLaVA-344
Figure 4: The performance of the LLaVA on the adversarial split, organized by the query sequence of AAAAB and BAAAA, reveals significant vulnerabilities the model's 's accuracy dramatically declines for object 5 AAAAB. SO stands for single-object probing and TF stands for teacher-forcing probing.
4.2 Main Results and Findings
We summarize the average results across the splits in Table and present the most important findings below. The full tables appear in Appendix A.2
Our immediate observation is that LVLMs suffer from more hallucinations when tasked with focusing on and recognizing multiple objects compared to single object. Across most of the models and test splits, we find that the average accuracy of single-object queries (i.e.. probing object classes one at time) significantly outperforms that of all three types of multi-object queries. The first exceptions are GPT-4O, MiniCPM-V V, and Cog VLM-2, the latter two leverage LLaMA-3 [35]. Another exception this is when reacher-forcing is applied to homogeneous test splits, which demonstrates an unreasonably high accuracy. We discuss them later this section.
Multi-object tasks introduce more hallucinations.
Heterogeneous queries introduce more hallucinations. We find that for all models and query methods, more heterogeneous queries lead to substantially more hallucinations, with performance decreasing from homogeneous in-the-wild heterogeneous test sets. The impact of heterogeneity applies to even start-of-the-ar LVLMs like GPT-40 (Figure 2). although this performance gap is more significant in ppigh models.
Language bias and shorteuts lead to multi-object hallucinations. In the macher-foreing setting, where there are no cumulative errors, LLa VA models score over 90% accuracy. There are three possible hypotheses for this abnormal observation: (1) LVLMs are smart enough learn object recognition in general through few-shot in-context learning in the teacher-forcing setting: (2) L.V.Ms learn to recognize one specific object through few-shot in-context learning in the teacher-foreing setting; or (3) LVLMs simply exploit language biases and rule-based shorteuts (e.g., repeating previous answers). To reach conclusion on this, we examine an Adversarial split, in which the first four tested objects are of the same class and we probe an object of a different class for the last one (e.g., AAAAB). We compare the single-object query performance with the teacher-foreing performance on the fifth object (object B). We anticipate the following outcomes: If hypothesis (1) is correct, the teacher-foreing performance should outperform the single-object query. hypothesis (2) is correct, the seucher-forcing performance should perform on par with the single-object query. hypothesis (3) is correct, the corcerr-forrigggggo performance should underperform compared to the single-object query. For a controlled comparison in the multi-object setting, we also reverse the order of queries (i.e., BAAAA) and repeat the experiments.
We present the results of LLaVA models on the unseen split in Figure 4. with the full results available in Appendix A.. We find that the model's predictions on class A progressively improve, scoring nearly perfectly starting from the third repetition. However, the model's performance on the last object (with the different class label B) drops to nearly zero, with almost all hallucinations labeling as A. This is in stark contrast to 23. 23.35% if these objects are probed individually or 19.16% when these objects are placed as the first to query in multi-object settings. Our findings suggest that hypothesis (3) is true, indicating that the IVIMs' high performance homogeneous queries could be an illusion resulting from textual shortcuts. We observe that models show lower performance in identifying object class B in single-object analysis, potentially due to the higher salience of object class A in these images.
Multi-object hallucinations occur in both seen and unseen images. We finally investigate whether our observations and findings hold uniformly in both seen and unseen splits. We observe that the gap between multi-object hallucination and single-object hallucination, as well as the reliance on shortcuts, persists. Although most of the models perform slightly better on seen images, the trends remain consistent across both splits. While large-scale training is involved in developing
7

---

these LVLMs, it appears they might not have fully exploited the fine- ane-grained information in the data. Training on these images does not significantly reduce object hallucinations.
4.3 What May Help and What May Not?
Comparing the tested LVL.M, we discuss our observations regarding design considerations that may or may not help reduce multi-object hallucinations.
Scaling the base LLM: data and parameters. We find that using base LLMs with more parameters reduces single-object hallucinations, but may not have the same effect on multi-object hallucinations. We observe a consistent increase in performance with larger LLaVA models in the seen set and in single-object queries, but not in the unseen set with multi-objer queries. One possible explanation for this finding is that LLMs with more parameters are better at memorizing seen images, as the performance gap between seen and unseen images is also more significant in larger models. We also notice that the performance gap between single-object probing and multi-object probing does not apply to MiniCPM-V V and Cog VLM-- which adopt LLaMA-3 (8B) [35] base LLM pre-trained with 15T tokens, as they fail to follow the instruction sometimes. Compared to L.LaVA models developed upon LLaMA-2 (7/13B) (50) and Yi (34B) [64] with 2T and 3T pre-trat pre-training tokens, these models underperform in quantitative measures due to instruction following error but exhibit greater robustness when multiple visual prompts are presented.
Visual instruction fine-tuning: chat and grounding. While it's surprising that conversational tuning reduces multi-object hallucinations, we observe that models without conversational tuning struggle to follow instructions and are prone to shorteuts, such repeating the list of all object class candidates in order or consistently repeating the first candidate. This might also explain why grounded tuning in VLM-L is of little help in reducing multi-object hallucinations thus far. These models typically lack conversational fine-tuning, and there is currently no available grounded dialogue data at scale. While mechanistically grounded LVLMs show strong performances in single-object probing, there remain gap in multi-object probing with student forcing. This could be attributed to a significant portion of the grounded instruction tuning dataset consisting mainly of short captions or questions featuring one single or few Sojjects.RL-basee finetuning approaches, such as MiniCPM-V demonstrate promising results across diverse settings, surpassing single-obje object results in both the Wild and Homogenous settings. Upon inspection, we found that this model demonstrates strong visual in-context learning capability and improves correct recognition when objects of the same classes are probed together.
Decoding and inference time strategy Decoding algorithms like OPERA introduce nuanced improvements specific multi-object settings [8]. In default multi-object tasks, OPERA shows marginal performance enhancements for LLa. but its effectiveness declines in tests with greater heterogeneity, to the point it can even lower performance. This suggests OPERA is beneficial in homogeneous contexts but requires further refinement in handling mixed object scenarios effectively.
5
Analysis of Hallucinatory Behaviors
5.1 Potential Hallucinatory Factors
The task setup described above allows us to evaluate LVLMSs in multi-object hallucinations and identify hallucinatory behaviors. Based on existing literature and our case studies, we further identify potential factors that correlate to and potentially explain these hallucinations.
Data-specific Factors. We consider the following factors that specific to the tested sample (e.g.. object and token positions), and are not relevant to the frequency distribution.
Inpul Order: we consider the order in which the object classes are presented in the input prompt containing all candidates.
Query Homogeneît: We define query homogeneity as the total number of task objects of the same class, normalized by the total number of queried objects (five in this work).
Object Token Position: Zhou et al. [75] has shown that more hallucinations occur in the latter part of captions. In this work, the object indices directly correspond to the object token positions.
Object Homogeneity: We define object homogeneity as the number of object types in the image, calculated upon panoptic annotations.
Object Centrality: Previous research has identified a center bias in datasets and models, indicating that objects are disproportionately located at the center of images in detection models [47 48) We define object centrality as one minus the distance d between the object's 's bounding box center and the image center, normalized by the diagonal distance D from the center to the corner.
8

---

Non Hallucinate
Frequerry
Count
Count
Hallucinate
Hallucinate Mean
Hallucinate
0.20
0.40
0.60
0,80
1,00
5
12.9
16.2
64
(a) Query Homogeneity.
(b) Object token position.
(c) Object Homogeneity.
Frequency
Frequency
Frequenry
0.52 0.56
0.08
0
0.13
0.21 (f) Semantic salience.
(d) Object centrality.
(e) Object salience.
Frequency
Frequency
Frequency
52832
168583
250000
0.6
0,06
0.11 0.12
0.18
(g) Training salience.
(h) Object token entropy.
(i) Visual modality contribution.
Figure 5: A comparison of the distribution of hallucinatory versus non non-hallucinatory object classes in across the unseen split under student forcing.
Salience and Frequency. We consider the following factors that are related to the saliency or frequency of the visual object the object class.
Object Salience: Previous research has shown that smaller objects are harder to detect and ground We define object salience as the ratio of the number of pixels occupied by the object's
[14 34
instance segmentation mask to the total number of pixels in the image.
Semantic Salience: We observe and hypothesize that LVLMs are less likely to hallucinate objects when they co-occur with multiple copies of the same class ("jar" in Figure We define semantic salience as the ratio of the total number of pixels in all instances of the same class, to the total number of pixels in the image.
Training Salience: Previous research has shown that spurious co-occurring patterns in the training data can lead object hallucinations 226 75). We use the log frequency of classes MSCOCO proxy for training salience following previous work, and hypothesize that LVLMs tend to hallucinate more on less frequent objects in the training set.
Model Behaviors. We consider the following factors relevant to the mechanistic behaviors.
Object Token Entropy: Zhou et al. [75] have shown that object hallucinations are more likely when the decoded object tokens have higher log perplexity. In our work, we define object token entropy as the entropy of the logits of the first token in the generated word. Given the softmax logits of the generated word's ' first token, we calculate the entropy using the following formula:
H H(s) - >; 8; log(s;) Simply put, higher entropy indicates greater uncertainty in the model's prediction for the first token, which can lead to more frequent object hallucinations.
Visual Modality Contribution: We hypothesize that LVLMS pay less attention the visual modality during object hallucinations. Motivated by the modality importance score 77]. define Visual Modality Contribution (VMC) as the proportion of attention allocated to visual tokens compared to textual tokens. To quantify this, analyze the attention weights of the last generated token across all heads and layers. The VMC is computed as follows: VMC = =Eieva, /(iev auj +
Exet (kk)), where ai represents the attention weight assigned to visual token head and akj represents the attention weight assigned to textual token à at head j. The sets V and T denote the visual and textual tokens, respectively. By examining the VMC, attention is given to visual inputs in comparison to textual inputs. A lower VMC may indicate higher likelihood object hallucinations due insufficient attention visual cues.
can determine how much
5.2 When Do LVLMs Experience Multi-Object Hallucinations?
In Figure55 we compare the distribution of these factors between hallucinatory and non-hallucinatory objects in the student forcing setting on the unseen split using LLaVA-13B. For continuous values, we use ridgeline plots, and for discrete values with fewer bins, we use bar charts.

---

Artal Class
Frequency
Frequency
Frequency
Prodittd Class
Meen Acteel
Moan Predicod
0.2
0.4
0.6
0.8
1.0
50849 106916
250000
0
10
20
30
40
50
(a) Semantic salience.
(b) Training salience.
(c) Input order.
Figure A comparison of the distribution of actual versus predicted object classes for all hallucinatory objects in the student forcing setting on the unseen split using LLaVA-13B.
sppeesi Factors. We observed that specific data factors, such as query and object homogeneity, significantly influence model performance, with increased hallucination occurring when models process images featuring multiple object classes or a variety of objects. For positional factors, the position of object tokens seems to have minimal impact and the object centrality has only a slight influence as LVLMS tend to hallucinate objects more frequently when they are positioned away from the center. This tendency may stem from reporting bias, as objects mentioned in captions are typically foreground objects that distribute toward the centers of images.
Salience and Frequency. We note that semantic salience significantly affects the model's per- formance, as it is more prone to hallucinate an object class that is less salient within the image. Conversely, the salience of individual objects does not statistically correlate with hallucination inci- dents. This implies that LVLMs may rely more on the presence of co-occurring objects of the same class to predict the labels of queried objects, rather than solely on the presence or salience of the objects themselves. Additionally, training salience plays a crucial role as models are less likely to hallucinate object classes that frequently appear in training.
Intrinsic Behaviors. The intrinsic behaviors of the model provide significant insights into its tendencies to hallucinate. Similar to Zhou et al. [75], we find that models are more prone to hallucination when they experience uncertainty or confusion, especially in scenarios involving multiple objects, as evidenced by higher token entropy. Furthermore, the contribution from the visual modality consistently registers below 20%, suggesting that current LVLMs may rely more heavily on linguistic contexts. There is a marginal increase in the likelihood of hallucination when models pay less attention to the visual context.
5.3 How Do LVLMs Experience Multi-Object Hallucinations?
In Figure 6 we conducted a detailed comparison of the distribution of actual versus predicted object classes within the context of hallucinatory objects, examining factors such as semantic salience, training salience, and input order. Although semantic salience is a key factor in determining whether model hallucinates, it appears to have minimal impact on the prediction of hallucinated objects. Our analysis also shows that models are more likely to hallucinate object classes that are prevalent in the training data, but the reverse is not necessarily true. Additionally, there is a notable preference for models to hallucinate objects that are listed early in the input prompt as candidate classes. Overall, our findings indicate that spurious correlations may lead to hallucinations involving multiple objects.
Discussions and Conclusion
Hallucinations in large vision-language models (LVLMs) can occur at different scales and granu- larities. In this study, we study the problem of multi-objjet hallucination, examining how LVLMs may misperceive when tasked to focus on multiple objects concurrently, and which factors cause the hallucinations. we introduce Recognition based Object Probing Evaluation (ROPE), an automated evaluation protocol designed to account for the distribution of object classes within single image during testing and to use visual referring prompts to reduce ambiguity. Our research provides key insights for the development and application of LVLMs. Since models tend to experience more hallucinations with multiple objects than with single ones, it may be advantageous to probe objects individually in visual prompts to enhance performance. The likelihood of model's hallucinatory output is linked to various data factors and model behaviors. Particularly in situations involving heterogeneous data and low certainty from the model, there is an increased risk of hallucinations, and users should be vigilant. Moreover, our analysis indicates that merely adopting (grounded) instruction tuning and scaling the base language model may not be enough to fully address the issue of object hallucination. There is a need for more balanced object distributions, annotations of objects away from image centers, and an increase in diversity. Introducing instructions that require multiple visual pointers and complex multi-objes reasoning is also crucial.
10

---

Acknowledgement This work was supported in part by NSF IIS- NSF SES-2128623, the DARPA Perceptual Task Guidance (PTG) Program, and the DARPA Machine Common Sense Program. Our experiments have also benefited from the Microsoft Accelerate Foundation Models Research (AFMR) program. We thank the Amazon AGI team for GroundHOG model access. The authors would like to thank Yichi Zhang and anonymous reviewers for their valuable feedback.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. volume 35, pages 23716-23736.
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. 2022. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: vl: A frontier large vision-language model with versatile abilities. arXiv preprint S.di::2308.19666
[4] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11621-11631.
[5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. 2018. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 1209-1218.
[6] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. 2024. Vip-Ilava: Making large multimodal models understand arbitrary visual prompts. In JEEE Conference Computer Vision and Pattern Recognition.
[7] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. 2020. Behind the scene: Revealing the secrets of pre-trained Aiiionaadd-lanuuar models. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 565-580. Springer.
[8] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal lim's referential dialogue magic. arXiv preprint arXiv:2306. 15195.
[9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purper vision-language models with instruction tuning. volume 36.
[10] Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. 2023. Plausible may not be faithful: Probing object hallucination in vision-language pre-training. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2128-2140.
[11] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1786-1305.
[12] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint ar549.2305.04779
[13] Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135-18143.
11

---

[14] Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai. 2012. Diagnosing error in object detectors. In European conference on computer vision, pages 340-353. Springer.
[15] Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. 2023. Ciem: evaluation method for better instruction tuning. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
Contrastive instruction
[16] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, Weilin Zhao, al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395.
[17] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-alloctiio In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition.
[18] Qidong Huang, Xiao wen Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Neng H. Yu. 2023. Opera: Alleviating hallucination in multi-modal large language models via over- trust penalty and crttoospection-allocate
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1--22733.7.
[19] Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. 2023. Faithscore: Evaluating hallucinations in large vision-language models. arXiv preprint arXiv:2311.01477
[20] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1780-1790.
detection for end-to-end multi-modal understanding. In
[21] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2023. Lisa: Reasoning segmentation via large language model. arXiv preprint 206602..08. 00692.
[22] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. 2023. Obelics: An open web-scale filtered dataset of interleaved image-text documents. volume 36.
[23] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, al. 2023. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36.
[24] Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. 2024. Scaffolding coordi- nates to promote vision-language coordination in large multi-moda models. arXiv preprint 102.5:2402.12055.
[25] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975.
[26] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing.
[27] Zongjie Li, Chaozheng Wang, Chaowei Liu, Pingchuan Ma, Daoyuan Wu, Shuai Wang, and Cuiyun Gao. 2023. Vrptest: Evaluating visual referring prompting in large multimodal models. arXiv preprini arXiv:23 /22::23/2.00087.
[28] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vicion-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
12

---

[29] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations.
[30] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved baselines with visual instruction tuning. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. volume 36.
[32] Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2023. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. arXiv preprint arXiv:2310.05333
[33] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Unified-io: A unified model for vision, language, and multi- multi-modal tasks. In The Eleventh International Conference Learning Representations.
[34] Ziqiao Ma, Jiayi Pan, and Joyce Chai. 2023. World-to-worde Grounded open vocabulary acquisition through fast mapping in vision-language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 524-544.
[35] Meta. 2024. Introducing meta Ilama 3: The most capable openly available 1lm to date
[36] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xic, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. 2024. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.0787..
[37] OpenAl. 2023.
Gpt-4v(ision) system card
[38]
OpenAL, 2024.
Hello gpt-4o
[39] Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. 2024. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations.
[40] Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, al. 2023. Detgpt: Detect what you need via reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14172-14189.
[41] Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Li Erran Li. 2024. Affordancellm: Grounding affordance from vision language models. arXiv preprint vrXiv:2401.06344
[42] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. 2024. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition.
[43] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac. Radu Soricut, Angeliki Lazaridou. Orhan Firat, Julian Schrittiiese,, al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300
[44] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
[45] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What does clip know about red circle? visual prompt engineering for vims. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11987-11997.
13

---

[46] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprini arXiv:2309. 145.2.55...5
[47] Gergely Szabó and András Horváth. 2022. Mitigating the bias of centered objects in common datasets. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 4786- 4792. IEEE.
[48] Mohammad Reza Taesiri, Giang Nguyen, Sarra Habchi, Cor-Paul Bezemer, and Anh Nguyen. 2023. Imagenet-hards The hardest images remaining from a study of the power of zoom and spatial biases in image classification. volume 36.
[49] Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean- -aan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1188
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajwwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 092888.
[51] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. volume 34, pages 200-212.
[52] David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. 2024. Contrastive region guidance: Improving grounding in vision-language models without training. arXiv preprint arXiv:2403.02325.
[53] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. An lim-free multi-dimensiinn benchmark for milms hallucina- tion evaluation. arXiv preprint arXhv:231..07397,
[54] Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. 2023. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint 230i::2200.11222
[55] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. 2023. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:231.03009.
[56] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2023. Visionlim: Large language model is also an open-ended decoder for vision-centrie tasks. volume 36.
[57] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. 2024. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[58] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441.
[59] Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. 2024. Fine-grained visual prompting. volume 36.
[60] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv: 17421, 9(1)1.
[61] Yuan Yao, Ao Zhang. Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2024. Cpt: Colorful prompt tuning for pre-trained vasion-language models. Al Open, 5:30-38.
[62] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2144472304.14178
14

---

[63] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations.
[64] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXhv:2403.04653
[65] Shoubin Yu, Jachong Yoon, and Mohit Bansal. 2024. Crema: Multimodal compositional video reasoning via efficient modular adaptation and fusion. arXiv preprint arXiv:2402.05889
[66] Tianyu Yu, Yuan Yao. Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. 2023, Rlhf-v; Towards trustworthy milms via behavior alignment from fine- grained correctional human feedback 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13807-13816.
[67] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. 2024. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220
[68] Bohan Zhai, Shijia Yang. Xiangehen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. 2023. Halle-switch: Rethinking and control- ling object existence hallucinations in large vision language models for detailed caption. arXiv preprint arXiv:2310.0177
[69] Hao Zhang, Hongyang Li. Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. 2023. Llava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.02999.
[70] Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. 2024. Ferret-v2: An im- proved baseline for referring and grounding with large language models. arXiv preprint 244477340..0797.
[71] Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. 2024. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition.
[72] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. 2023. Bubogpt: Enabling visual grounding in multi-modal 1lms. arXiv preprint arXiv:2307.0858)
[73] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena ArXiv, abs/2306.05685.
[74] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641.
[75] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2024. Analyzing and mitigating object ballucination in large vision- language models. In The Twelfth International Conference on Learning Representations.
[76] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing languag understanding with advanced large language models. In The Twelfth International Conference on Learning Representations.
15

---

A Additional Experiments, Results, and Discussions
A.1
Reproducibility
Data Curation Pipeline. Our data curation pipeline involves several essential steps de- signed to prepare and refine dataset for eval- uating multi-object hallucination. The pipeline begins by filtering images and candidate objects to query. We consider valid objects to be those belonging the top 50 "thing" classes and ex- clude objects with a bounding box area less than 1% of the total image area. We discard images containing fewer than 5 valid objects, between
Dataset
Total
COCO
ADE
Wild
1539 1172
732/ /47
807 / 625
Hom.
312 / 490
168 289
144 / 201
Het.
400 / 246
200
200 / 170
Adv.
168 334
54/170
114 164
Table 3: An overview of object hallucination benchmarks. For design considerations, we sum- marize the number of tested images, and if multiple classes and object class distribution (at training and test time) are considered. The image sources include those seen unseen during instruction tun- ing. To refer to an object, textual descriptions and visual cues be adopted. For evaluation, neural models, humans and automatic pipelines are used.
and allow an intersection-over-unee bounding boxes of no more than 0.1, which pre- serves data integrity while ensuring high image quality. We apply this pipeline to MSCOCO- Panoptic 28 and ADE20K 77
Language Instruction Prompt Templates. We illustrate the types of task prompts for Single-Ohject and Multi Multi-Ohject queries in Figure and document the prompts below.
obju: ?. obj2:
obja: ?, obj2: ?,
obj3: ?, obj4: 7, objs:?
objs: ?, obj4: objs:
User
objt:
Predicted
obji:
Ground
apple class
apple
truth
LLaVA
34B
obji: apple, obj2:
Predicted
obji: obj2:
class
Ground
apple
knife
truth
Forced
objt: apple, obja: apple, objs:
objı: fork, objæ: knife, obj3:
apple
fork
obja: ?
can
obja: ?, obja:
objs: ?, obj4: objs:
obja: ?
knifé
obju: apple, obja: apple.
objı: fork, obja: knife,
obju: apple, objo: apple.
obj3: apple, obj4: lemon, objs:
obj3: whisk, obj4; lemon, objs:
objy: apple, obj4: lemon,
objs: soda
obis: pear
pear
soda
(a) Single-object.
(b) Multi-object.
(c) Student forcing,
(d) Teacher forcing.
Figure 7: Different types of instruction settings of ROPE. In a single turn of prompting without format enforcement, we probe the model to recognize the 5 objects referred to by the visual prompts (a) one at time in the single-object setting and (b) concurrently in the multi-object setting. We further enforce the model to follow the format template and decode only the object tokens for each of the five objects (c) without output manipulation in student forcing and (d) replacing previously generated object tokens with the ground truth classes in teacher forcing.
Multi-Object Default Probing, Student Forcing, and Teacher Forcing:
Select one and the most appropriate class for each object located within red bounding boxes from the following list: [CLASS NAMES]. Provide the class names in the format: 'ohj1: <classl>, obj2: <class2>, obj3: <class>>, obj4: <class4>, obj5: <class5>'. with no additional words or punctuations.
Multi-Object Probabilistic Probing:
(GroundHOG) Describe object <PTR> object <PTR> object <PTR> and object <PTR> and object 5 <PTR>, 1: <classl>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <classs>
(GLaMM) What are the classes of [<bbox> LIST] 7 objl: <classl>, obj2: <class2>, obj3: <class>>, obj4: <class4>, ohj5: <class5>
Available suttpprr//uuggggsseo.oo//ttttttt
16

---

Select the single, most appropriate class for obi(obj num) located within the red bounding box from the following list: car, pedestrian, bicycle, motorcycle, bus, truck, troffic light, stop sign, parking meter, bench, fire hydrant, street sign, crossenlk, traffic cone, construction barrier.
Ground Truth
objı: pedestrian
obja: car
obj3: traffic light
Your response should consist solcly of the class name that obj(ineex) belongs to, formatted only the class name, without any extra characters punctuation.
abj4: pedestrian
obi5: traffic light
N VNV
GPT-40
Claude 3:5
LLaVAA-1R
objt: pedestrian
obji: pedestrian
objt: car
obj2:
objz:
AayS
obj3: traffic light
obj3: traffic light
objo: traffic light
obj4: pedestrian
obj4: pedestrian
obj4: traffii light
obj5: traffic light
obj5: traffic light
obj5: traffic light
GPPT-VV
S.Qeen 2.5
LLaVAPhigMini
obji: pedestrian
objt: pedestrian
objt: car
obja: car
obj2: car
obj2: car
obj3:
obi3: traffic light
objo: traffic light
abj4: pedestrian
obj4: pedestrian
obj4: traffic light
obj5: pedestrian
obj5: traffic light
obis: traffie light
Select one and the most appropriate class for each object located within red bounding boxes from the following car, pedestrian, bicycle, motorcgycle bus, truck, traffic light, stop sign, parking meter, fire hydrant,
Ground Truth
obja: pedestrian
sign, crosswalk, traffic cone, construction barrier.
obj2: car
obj3: traffic light
Provide the class names in the format: 'obj1: <classl>, obj2: <class2>, obj3; <class3>, obj4: <class4>, obj5: <class5>', with no additional words or punctuations.
obj4: pedestrian
obi5: traffic light
CANNAA VMV
GPT-40
Claude 3-5
LLLVVA113
obji: pedestrian
obja: pedestrian
obja: car
obj2:
obj2: car
obj2:
AM5
obj3: traffic light
obig: traffic light
obis: car
CBIC
obj4: pedestrian
obj4: pedestrian
obj4:
objs: traffic light
obi5: pedestrian
obis:
GPT-4V
#Qwen
LLaVAPhigMin
obja: pedestrian
obji: pedestrian
obit: eat
obj2: car
obj2: car
obj2: car
obj3:
obj3: traffic light
obig:
obj4: pedestrian
obj4: pedestrian
obj4: car
objs: pedestrian
obis: light
objs:
Figure 8: Single and multi-object hallucination under the default setting nuScenes
(LLaVA) There are five red bounding boxes in this image. each object within the red bounding boxes, identify its class. Provide the class names in the format: 'objl: <class obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>', with additional words or punctuation.
Single Object Default Probing:
Select the single, most appropriate class for obj<obj num> located within the red bounding box from the following list: [CLASS NAMES]. Your response should consist solely of the class name that [bjjobj_mn _umm>> belongs to, formatted as only the class name, without any extra characters or punctuations.
Single Object Probabilistic Probing:
(GroundHOG) Describe object <PTR> a word.
(GLaMM) What is the class of <bbox>?
(LLaVA) Describe the object in the red bounding box labeled obj _bju _nu>> in a word.
Computational Resources. Our experiments conducted on eight A40 and four A100 GPUs slightly over a week. The computational bottleneck was not the numerical accuracy values but the collection of potential hallucinatory factors for analytical purposes, including logits and attention values for each head and layer.
A.2 Additional Experiments and Results
Per-object Performance. We provide the per-object performance in the following Table and T
17

---

Autonomous Driving Case Study. Figure is case study example from the nuScenes dataset for autonomous driving. It illustrates the single and multi-object case, where each object identified independently. The multi-object case exhibits more hallucination error compared to the single-object case. This finding underscores the importance of studying multi-object hallucination, especially in real-world scenarios like autonomous driving, where multiple objects need to be detected accurately at the same time.
B
Limitations, Licenses, and Risks
B.1 Limitations
ROPE represents one of the pioncering efforts to publicly address the issue of multiple object hallucination. However, acknowledge several limitations in work: (1) The lack of transparency in the LVLMs makes it difficult to guarantee that our unseen dataset has not been previously exposed. (2) Our evaluation benchmark uses fixed set of semantic objects, which may introduce bias and impose unnecessary constraints on the LVLMs' ability to follow instructions and reason effectively. (3) The evaluation process be slow, involves performing five inferences per image for both student forcing and teacher forcing.
B.2 Artifacts and licenses
We report list of licenses for all datasets and models used in our experiment in Table We strictly follow all the model licenses and limit the scope of these models to academic research only.
Data Sources
URL
License
MSCOCO 2017
Link
CC BY 4.0
ADE20K
Link
BSD-3-Clause
Software Code
URL
License
LLaVA
Link
Llama Community Licence
Qwen- OWen-VL
Link
Tongyı Qianwen Licence
Cog VLM
Link
Cog VLM Licence
IDEFICS
Link
Llama Community Licence
Yi-VL
Link
YI Community Licence
MiniCPM-V
Link
Apache License 2.0
GLAMM
Link
Apache License 2.0
GPT-4V/40
Link
OpenAl Term of Use
Table 4: License information for the scientific artifacts used.
B.3 Ethical concerns and risks
This study does require human annotators or participants for its interactive experiments. Instead, it utilizes publicly available datasets and content created by models for evaluation purposes. We are aware that these public data might introduce biases and sensitive elements, and is essential for future research to address these concerns, possibly by creating datasets that incorporate fairness-basee filtering and metries.
18

---

Multi- Objee-Object
Student Forcing
Teacher Forcing
Single-Ohjes
Model
Wild Hom.
Het.
Adv.
Wild Hom.
Het.
Adv.
Wild
Hor.
Het.
Adv.
Wild
Hom.
Het.
Adv.
object/ /
66
3.17
3.67
1.22
3.29
3.51
3.88
1.63
3.59
3.51
3.88
1.63
3.59
0.09
1.02
0.00
0..3
V-VL-34B
10.01
16.12
5.69
10.48
10.35
18.16
6.11
11.38
10.35
18.16
6.91
11.38
0.43
2.24
0.00
0.60
LLaVA-7B
14.99
67.14
14
13.11
55.39
34.90
67.15 14
13.41
55.39
34.90
67.
13.41
55.39
34.99
65.71
15.04
$3,89
LLaVA-13B
29.77
62.86
12.60
51.20
29.5
62.86
12.60
50.60
32.34
63.09
12.60
50.60
40.96
22.24
23.17
40.00
LLaVA-34B 34B
26.61
77.55
17.48
62.87
38.5
88.19
14.23
60.97
38.58
88.19
14.23
60.97
48.33
77.55
27.6
66.47
OwenVL
4.01
2.44
5.92
4.19
9.56
20.41
6.91
16.47
10.49
20.82
5.28
17.96
18.39
32.04
9.76
19.76
OwenVL-C
22.53
28.16
13,41
24.25
22.61
33.88
13,41
24.25
23.21
32.86
23,65
34.13
59.80
16.67
47.01
CugVLM
1.20
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVL
0.00
0.00
0.00
0.00
8.94
1.47
9.76
7.02
10.16
16.50
4.17
10.71
11.83
13.97
8.94
7.78
CogVLM-G
17.88
27.76
7.72
27.54
23.18
44.90
9.76
33.83
23.18
44,90
9.76
33.83
30,89
50.82
10.98
42.22
VLM-
24.55
36.33
16.26
31,44
24,89
37.14
18.70
44.61
24,89
37.14
18.70
44,61
10,85
31,43
3.25
27,25
IDEFICS
1.45
1.63
1.22
1.50
11.55
24.2 29
6.10
14.67
11.95
24
6.10
14.67
4.86
1,88
5.28
2.10
MiniCPMLV
33.86
63.21
18.62
55.69
35.59
65.31
15.04
54.79
35.59
65.31
15.04
54.79
25.15
41 16
12.60
37.13
LLBVA-7B*
7.08
11.43
6.50
8.08
13.45
25.75
13.14
24.40
GLaMM
N/A
53.50
50.20
41006
50.30
68.34
77.55
54.07
73.35
GrundHOG*
15.27
20.41
10.98
19.76
40.10
28.98
31.30
32.34
GPT-VV
49.53
67.35
38.21
56.29
A/A
N/A
47.05
64.49
36.59
$7.19
GPT-40)
64.42
80.61
56.10
73.05
N/A
N/A
63.56
81.22
53.66
73.65
obiect 2
6L
2.65
4.08
2.44
3.29
.88
8.37
4.47
5.69
3.68
7.96
5.69
9.58
0.17
0.61
0.00
0.00
VL-BB33
8:55
17.55
3.25
15.5
10.61
19.59
8.54
12.28
11.11
22.24
10.98
14.37
0.51
2.45
0.41
0.60
LLaVA-7B 70
29.77
67 67.96
8.3
56.89
29 52
67.6
8.13
56.89
32.00
84.69
5.28
19.34
14.13
6 02
15.45
50.30
LL
19
63.06
4.07
$2.40
28.2
63.06
4,07
$2.10
37.97
92.37
6.10
85.63
41 64
20.22
21
42.58
34B
30.89
79.39
18.29
68.56
27.56
90.55
15.45
65.81
34,40
91.84
2.85
68.25
52.18
78. 16
30.08
68.86
OwenVL
3.84
2.44
8.98
6.59
10.24
23.06
4.07
20.36
13:05
37.35
4.07
33.53
16.94
27.35
9.35
20.66
OwenVL-C
23.12
35.71
10.57
28.74
17.75
26.12
7.32
19,46
22,61
36.33
11.38
30.24
33.45
53.88
13.82
45.81
CLgVLM
0.00
0.00
0.00
0.30
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
C-
0.00
0.00
0.00
0.00
8.12
1.47
4,88
7.02
25.90
93.50
2.08
94,64
8.39
10.29
6.91
6.89
CLgVLM-G
16.34
28.37
4.07
29.64
15.57
39.80
4,88
34.13
21.81
44,90
2.85
39.52
30.72
$1.43
14.23
43.71
C--VLM-2
25.75
34.08
19.51
28.14
24.72
36.53
22.76
45.21
26.43
40.61
26.83
45.21
11.32
32.45
5.69
25.45
IDEFICS
0.17
0.20
0.41
0.60
11.69
27.7 77.76
4.07
20.06
31,91
91 91.43
2.03
85.93
4.10
3,67
6.50
MiniCPM-V
35.79
57.73
17.00
49.10
31.99
58.37
13,01
53.29
31.48
60.41
13.41
53.29
26.09
49.
14.23
LLaVA-7B*
8.28
12.86
4.47
11.08
10.59
23,00
5.45
GLaMM
54.69
6.50
52.99
69.62
75.92
49.19
GrounHHo
31.31
27.14
32.52
27.25
41.13
29.
40.65
GPT-4V
48.16
67.55
38.18
60.18
47.48
65.51
32.52
57.78
GPT-40)
63,48
80.20
56.50
73.05
N/A
N/A
62.62
80.41
54.07
72.16
object 3
YL-
2.91
3.88
1.22
2.69
3.68
3.88
5.69
5.69
4.02
8.37
17.89
1587
0.17
0.00
0.41
0.30
VL-VL-34B
830
15. 92
6.10
14.67
10.95
17.9 96
9.35
11.98
11 1646
22.86
12.60
15.57
0.43
2.86
0.4
0.30
LLaVA-7B
27.97
68
12.60
57. 19
99
68
12.20
57.1 17.19
29.95
69.80
1.38
99
74
63.2
16:26
$120
LLaVA-13
62.86
8.54
52 40
17
62.86
8.54
52.
12
36.52
100.00
13.01
100.00
43.77
12.65
24.39
45.16
LLaVA-34E 34B
29.97
80.00
21.54
69.16
27.
56
92.91
13.01
67.42
41.20
99.32
8.54
94.79
54.32
75.37
34.15
69.66
OwenVL
2.30
2.03
1.55
7.19
6.48
16.53
4.47
14.37
21.75
91.02
4.47
45.51
18.31
33.47
12.20
23,95
OWe
18.09
33.27
8.54
24.25
14.85
28.16
6.50
19.16
21.67
68.37
6.91
60.78
34.47
57.96
15.85
49.10
CogVLM
0.00
0.00
0.00
0.60
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
Cog VL.MC
0.00
0.60
0.00
0.00
1.47
4.47
7.02
26.02
97.50
6.62
11.79
7.49
CogVLM-G
12.15
25.31
4,88
25.15
13.34
37.55
5.69
31.14
18.39 39
56.73
9.35
50.30
30.12
$2.04
14.23
43.71
C-gVLM-2
19.16
35.31
16.26
27.84
24.98
38.98
24.80
42.81
27.46
43.47
32.93
$1.50
10.50
30.00
6.91
21.56
IDEFICS
0.09
0.00
0.00
0.50
6.25
23.27
1.22
13.77
26.02
99.80
8.94
100.00
5.20
3.67
8.54
2.10
MiniCPM-V
35.12
61.06
13.77
52.10
28.83
55.71
14.63
49.70
29.43
57.14
13.04
47.90
26.52
46.33
14.23
32.93
LLaVA-7B-
8.62
12.65
4.07
10.18
10.53
23.00
9.62
21.43
GLaMM
25.53
56.33
7.32
54.19
77. 96
54.07
73.35
GrundHOG* GPT-4V
25.31
30.08
26.35
40.24
46.96
62.24
34.15
$2.69
V/A
47.73
34.55
GPT-40
61.51
80.41
57.32
74.55
N/A
63.05
79.59
51.22
71.56
obiect 4
VB
2.48
4.29
0.41
3.29
3.17
2.24
1.72
6.29
4.45
13.41
15.27
0.09
0.00
0.30
Yi-VL-34D 34B
6.67
15.31
447
11.98
10.01
18.37
1.72
13.17
11 80
5.45
17.66
0.51
2,04
130
0,60
LLaVA-7B
30.37
68.3
7.32
56.89
30,20
68.5
56.89
29.95
100.00
17.5
100000
17 64
66
15.85
LLVVA-138
26.26
7
$2.40
100.00
100.00
44
26.42
LLaVA
30.58
80.20
18
70.36
31.50
92.15
13.41
67.10
35.60
99.32
6.91
95.26
50.56
77.14
28.05
70.06
OLenVL
2.65
2.03
8.78
7.19
5.20
12.45
4.07
11.08
9.04
97.35
38.32
18.14
30.41
8.94
26.65
OwenVL-C
15.44
29.18
6.50
23.35
12.54
25.51
6.91
18.86
19.97
72.04
13.1
63.77
32.85
40.41
17:48
49.10
CogVLM
0.00
0.00
0.00
0.30
0.00
0.00
0.00
0.00
26.60
47.55
29.67
$7.49
0.00
0.00
0.00
0.00
CogVI.M-C
0.00
0.00
0.00
0.00
7.98
1.47
4.07
7.02
28.00
98.50
2.08
98.81
11.14
10.29
11.38
7.78
CogVLM-G
15.31
25.92
5.69
26.35
17.19
42,86
5.28
36.23
22.75
60,20
0.81
60.48
30,29
55.10
12.60
4341
CLgVLM-2
18.05
34.69
16.26
29.64
25.15
38.57
28.86
41 62
27.63
49.18
20.73
S7.49
10.97
31,84
6.10
26.95
IDEFICS
0.09
0.00
0.00
0.30
8.70
21.47
1.22
13.47
26.88
99.80
8.13
100.00
4.95
3,67
8.13
2,10
MiniCPM-V
30.26
60.86
20.24
$2.40
31.39
58.78
1350
53.89
31.39
61.63
19.92
55.39
25.15
48.37
15.85
38.92
LLaVA-7B*
6.66
11.63
2.44
9.88
13.06
23,00
10.90
21.43
GLaMM
27.27
56.
12
8.54
$4.19
68.
76.33
52.85
74.85
GroundHOG
19:54
25.3T
25.61
25.75
39.42
32.45
36.18
36.83
GPT-VV
45.42
64.08
31.71
50.00
A/A
47.82
66.53
35.77
53.59
GPT-40
63.82
79.80
57.32
74.85
N/A
64.16
79.59
53.66
70.96
object 5
VL
2.48
3.47
0.41
2.10
3.68
2.86
6.50
5.39
4.53
19.18
14
17.66
0.17
0.41
0.00
0.00
VI-VI.338
5.50
13.27
1.63
3.29
9.50
16.12
7.32
9.88
11.38
24.29
14.23
18.26
0.43
2224
0.41
0.30
LL-VA-7B
30.11
67.57
10.16
13.17
30
57
10.16
13.17
32.68
100.00
29
0.00
36.36
65.92
18
18.29
23.35
LLNA-1 13B
27.63
63.27
9.35
10.48
27
9.35
10.48
36.01
100.00
23.88
0.00
42.49
71 02
21
24.54
21
LLaVA- 34B
28.44
80.00
13.01
8.58
25 20
34
15.85
18.71
37.20
99.32
3,66
5.21
53,81
34.1 14.15
34.73
OwenVL
1.19
0.81
4.08
1.50
4.35
9.59
1.25
1.50
7.68
47.35
3.25
0.00
17.11
32.86
7.32
11.98
OwenVL-C
15.10
27.35
4.88
8,38
13.05
25.31
4.47
5.09
21.59
80.41
15.04
6.29
36.09
59,80
13.01
19.16
CugVLM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-C
0.00
0.00
0.00
0.00
8.12
1.47
5.69
7.44
29.00
100.00
0.00
0.60
10.18
12,50
6.50
9.58
CLMVLM-G
16.34
25.51
5.28
6.59
16.60
41.22
4.47
5.69
27.89
73.47
10.57
3,89
30.80
$3.06
15.45
17.66
VLM2L
17.71
34.90
11.38
12.87
23.44
38.98
20.73
12.28
27.29 29
59.80
33.74
12.57
12.02
28,98
6.91
5.09
IDÉFICS
0.17
0.00
0.00
0.00
6.57
23.47
0.81
1.80
27.22
99,80
13.01
0.00
4.01
3.47
4.07
4.79
MiniCPM
29.76
56.75
13.36
19.16
31.05
36.73
14.23
14.07
31.48
60.41
18.29
17.66
25.24
42
15.04
17.9
LLaVA-7B*
7.34
12.04
691
4.79
10.79
23.00
7.69
10.12
GLaMM
2381
55.31
7.72
6.29
69.7
77.35
51 21
53.29
GroundHOG
24.91
25:31
32.11
18.56
42.92
30.00
42.28
3035
GPT-VV
38.41
56.94
31.30 27.54
N/A
46.62
62.65
37.80
31.44
GPT-40
63.74
80.41
54.07 53.59
N/A
64.67
78.37
56.10
$4.49
Table 5: Complete per-object results on the unseen split.
19

---

Multi- Mujei-Object
Student Forcing
Teacher Forcing
Single-Ohjes
Model
Wild Hom.
Het.
Adv.
Wild Hom.
Het.
Adv,
Wild
Hom.
Het.
Adv.
Wild
Hom.
Het.
Adv.
obiect/ /
66
2.92
5.50
1.92
4.76
3.12
5.75
2.56
6.55
3.12
20.75
2.56
6.55
0.06
0.25
0.00
0.60
V-VL-341
8.51
15.75
481
13.10
8.71
16.00
5.13
11.90
8.71
16.00
5.13
11.90
0.19
2.50
0.00
0.60
LLaVA-7B
11.51
65.75
5,00
51.79
31
65.75
10.90
51.79
31 31.45
65.75
10.90
51.79
33.01
62.15
17.95
47.6
LLaVA-138
28.14
65.83
10.91
54.17
28.1 14
72.00
7.69
54.76
28. 14
72.00
7.69
54.76
41.46
81.25
24.04
65.48
LL 34B
34.18
83.50
8.65
67 67.26
48.02
83.75
29.17
72.02
48.02
83.75
29.17
72.02
54.13
85.00
17.20
72.98
OwenVL
2.21
6.00
1.60
4.17
7.41
14.25
1.28
10.71
5,46
15.25
2.24
9.52
8.06
16.50
4.49
17.86
OwenVL-C
5.65
14.25
2,24
10.71
5.59
9.25
2.24
8.33
6.04
8.75
7.74
25.02
43.50
14.10
30.36
CugVLM
0.00
0.00
0.00
0.60
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVL
0.00
0.00
0.00
0.00
11.18
13.50
4.17
16.67
3.44
3,00
1.92
4.76
12.46
24.75
5.13
25.60
CogVLM-G
11.70
18.25
6.09
19.64
24.44
41.67
10.30
29.19
24.61
49.75
10.90
29.87
28.85
55.25
14.74
40.48
VLM-
18.97
37.00
13,78
32.14
35.48
69.50
10,90
33.33
35.48
69.50
10,90
33.33
18.78
39,25
9.62
30,36
IDEFICS
0.00
3.50
0.04
120
7.41
20.25
0.96
14.29
7,41
20.25
0.96
14.29
486
0.00
0.00
0.00
MiniCPM
32.81
66.25
15.38
58.33
3151
72.00
13.14
63.09
31.45
72.25
13.14
59.32
25.87
$4.00
15.38
58.3
LLaVA-7B*
9.62
16.25
6.41
11:31
13.45
55.15
13.14
24.40
GLaMM
45.61
44.50
40.38
42.26
64.33
81.25
55.77
73.81
GrundHOG*
16.11
24.25
16.03
22.62
4336
43.50
45.19
$1.79
GPT-VV
56.79
79.75
41.35
71.43
A/A
N/A
S5.30
76.25
41.35
71.43
GPT-40)
69.98
89.50
66.03
79.76
N/A
N/A
61.35
73.27
54.47
69.46
obiect 2
6L
3.83
5.75
3.21
6.55
3.31
6.25
3.53
7.74
6.17
10.25
481
6.55
0.19
0.25
0.00
1.19
VLBB
9.10
15.50
3.21
13.10
8.90
16.75
5.77
10.12
10.59
21.00
5.77
23.81
0.26
2.50
0.32
199
LLaVA-7B
32.36
67.50
15:00
52.98
32.36
36
67.25
12.50
52.98
35.15
95.00
12.18
91 07
36.32
6
20 19
47.62 47.62
LLAVA-13B
32.42
67.67
13.48
56.55
32.36
36
72.75
14.42
56.55
37.04
99.25
11.34
97.02
43.66
80.50
24
36
72.02
34B
41.26
85.50
25.32
70.24
53.15
85.75
31.41
73.21
57.05
98.10
21.84
94.30
54.52
85.50
18.10
77.38
OwenVL
3.38
6.00
0.96
3.57
5.78
15.75
4.49
12.50
19.36
57.50
5.13
42.86
8.45
14.75
4.81
12:50
OwenVL-C
9.03
16.50
9.62
11.31
5.52
8.75
6.73
6.55
16.89
45.25
9.29
44.05
26.19
40.00
15.71
28.57
CLg
0.06
0.00
0.00
0.60
0.00
0.00
0.00
0.00
0.26
0.00
0,00
0.00
0.00
0.00
0.00
C-
0.00
0.00
0.00
0.00
12.12
13.50
8.65
16.67
29.43
94,00
0.00
94,64
12.38
22.50
7.69
18.45
CogVLM-G
12.15
21.75
6.09
22.02
25. 79
42.83
7.12
30.87
27. 40
55.75
16.67
33.56
31.06
55,00
18.27
42,86
C-gVLM-2
24.95
38.00
21.79
38.10
38.60
72.00
12.50
32.14
36
13
70.25
14.10
31.55
21.90
40.00
14.42
29.76
IDEFICS
0.00
1.00
0.00
0.60
5.78
17.75
0,00
8.33
22 16
77.75
12.18
63.10
4.10
0.00
0.96
0,00
MiniCPM-V
36.19
64.75
19.23
60.71
33.07
62.7
16,67
$7.14
33.79
69.00
17.63
61.31
30.06
60.75
17.63
60.12
LLaVA-7B*
9.10
17.75
6.41
13.10
10.59
23.00
5.45
GLaMM
55.75
6.73
56.55
64.20
80.25
56.41
72,62
GroundHo
31.32
33.50
34.29
30.36
43.60
44.25
41.03
GPT-4V
55.69
78.75
41.35
73.81
V/A
56.27
78,50
37.50
72.62
GPT-40
71.28
89,50
64.10
82.14
N/A
N/A
60.24
74.90
54.88
68.56
object 3
YL-
2.99
6.25
2.56
5.95
1.12
7.00
4.17
8.93
5.26
24.75
6.09
8.33
0.13
0.25
0.32
1.19
Yi-VL-34B
9.10
16.00
160
1488
9.10
16.7
3.24
10.71
11.23
21.75
4.17
36.31
0.19
2.50
0.32
LLaVA-7B
29.26
68.00
5.00
52.
29.
76
67 50
12.18
52
98
40
100.00
86
100000
13.92
6 50
1442
48
LLaVA-13
32.25
68.00
13.18
58.
68
13.50
10.90
57.7 74
38.27
100.00
1163 63
100.00
44.90
$2,00
20.19
67.86
LLVVA-34 34B
44.70
86.50
24.68
70.24
54,00
86.25
36.86
73.81
57.89
98.80
27.63
98.32
53.41
87.50
16.32
77.98
OwenVL
2.86
7.50
0.96
7.14
6.30
18.00
4.49
16.67
21.25
92.50
5.13
88.10
8.58
16.25
5.77
17.26
OWenVL-C
10.66
21.25
7.05
21.43
6.04
9.00
5,13
9.52
16.89
60.75
9.29
61.31
26.19
46.25
10.90
32,74
CogVLM
0.06
0.00
0.00
0.60
0.00
0.00
0.00
0.00
0.13
0.75
0.00
1.19
0.00
0.00
0.00
0.00
Cog VL.
0.00
0.00
0.00
0.00
10.84
13.50
S.77
16.67
21.25
7.05
16.67
CLgVLM-G
12.74
25.75
7.05
24.40
26,89
45.00
14.39
34.23
27.23
81.25
19.23
57.72
29.63
55.50
16.99
42.26
C-gVLM-2
20.27
36.50
15.71
36.31
39.12
72.75
13.14
32.14
36.45
74.75
17.31
49.40
19.36
34.75
12.18
31.55
IDEFICS
0.00
1.00
0.00
0.90
6.30
18.75
0.64
11.90
15.01
92.50
8.65
86.90
5.20
0.00
0.32
0.00
MiniCPM-V
36.19
68.30
20.51
61.31
31 38
60.00
12.50
52.38
30.60
68.25
13.71
54.76
26.15
$3.75
19.87
95.05
LLaVA-7B
8.77
16:50
5.45
11.90
10.53
23.00
9.62
21.43
GLaMM
22 68
55
5.15
54.17
62.70
53.00
GrundHOG*
24.82
32.75
24.68
30.36
35
GPT-4V
$4.45
79.25
40.06
70.24
M/A
$5.62
80.2
39.42
72.02
GPT-40
71.35
90.75
65.71
83,93
N/A
60.07
74.29
53.66
68.26
object 4
VL
3.25
5.50
4.76
4.35
1.75
5.13
10.71
6.30
17.75
8.93
0.26
0.00
Yi-VL-34 34B
8.32
15.75
13.69
9.16
17.00
5.13
11.05
22.75
5.45
14.52
0.26
3,00
0.00
LLaVA-7B
10.00
68
10.90
$4.
30.25
100.00
15.08
100.00
36.84
63.00
17.
49.
LLiVA-138
31.38
68.11
94.05
31 32
11.54
06
100.00
100.00
44.83
80.00
25.
64
LLaVA
40.68
86.75
19.23
73.21
54.06
34.62
73
57.98 96
99.10
97.65
56.14
86.
22.1
80.95
OwenVL
2.47
7.25
0.00
4.76
5.91
14.75
2.88
12.50
22.87
95.50
7.69
95.24
9.75
14.25
6,41
16.07
OwenVL-C
8.58
16.25
7.05
16.07
5 4646
8.00
2,88
7.74
11.96
56.50
10.26
$4.17
24.82
38.50
11.22
28.57
CogVLM
0.00
0.00
0.00
0.60
0.00
0.00
0.00
0.00
0.06
1.00
0.00
0.60
0.00
0.00
0.00
0.00
CogVI.M-C
0.00
0.00
0.00
0.00
10.24
13.50
9.94
16.67
26.25
93.75
0.32
93.45
9.83
21.25
9.29
20.33
CogVLM-G
14.10
24.75
9.62
27.98
26.09
45.33
15.30
34,23
30,91
86.00
23.40
62.08
29.89
57.75
16.99
44.05
CLgVLM-2
22.94
37.75
19.8
36.31
36.58
66.75
14.74
33.93
38,40 40
75.25
20.51
61.31
24.24
37.50
15.71
26.19
IDEFICS
0.00
1.25
0.00
0.60
5.91
18.25
0.64
11.90
21.12
95.00
14.74
93.45
4.95
23,00
0.00
0.00
MiniCPM-V
37.36
64.75
18.59
60.71
32.42
75.75
13
56.55
34,05
67.15
19.55
55.36
29.14
56.00
16.35
62.50
LLaVA-7B*
8.90
15
15.75
5,45
11.31
13.06
23,00
10.90
21,43
GLaMM
20
55.17
75
6.75
55.05
61.91
80.25
55.45
67.86
GroundHOG
20.86
31.50
21.15
30.95
43.99
42.00
35.58
50.00
GPT-VV
$3.09
77.25
43.59
72.62
N/A
56.40
78 75
44.87
74.40
GPT-40
72.25
87.00
68.27
$2.14
N/A
60.67
72.65
$5.69
70.06
object 5
YI- VL
1-75
5.25
0.64
1.19
3.31
7.25
3.53
8.93
6.37
37.75
4.17
0.60
0.32
0.50
0.32
0.60
VI-VI.
7.47
13.75
2.56
3.57
8.97
15.00
138
5.36
8.86
17.28
4.17
4.17
0.19
2,50
0.00
0.60
LLaVA-7B
30.41
60.50
5.00
777
30.41
25
9.62
8.93
32
100.00
11.86
0.00
36.52
62.15
16.99
20.83
LLNA-11 13B
33.01
68.50
11
21
1,43
32
14.55
13.
7.14
35
100.00
18.59
0.00
40.81
79 25
23.81
LLaVA-3A 34B
38.92
86.50
16.35
9.52
54.52
85.00
37.50
32.14
61.15
99.30
22.63
8.05
57.05
75
21.13
32 14
OwenVL
2.73
6.25
1.60
1.79
5.85
17.25
5.13
24.76
96.75
7.05
0.00
8.84
18.50
6.41
4.17
OwenVL-C
9.68
16.25
7.37
4.76
3,70
8.00
3,53
1.79
8.77
67.50
9.29
27.75
48.75
14.10
15.48
CugVLM
0.06
0.00
0.00
0.60
0.00
0.00
0.00
0.00
0.06
3.00
1,00
0.00
0.00
0.00
0.00
0.00
CogVLM-C
0.00
0.00
0.00
0.00
9.98
13,50
S.45
8.93
27.03
92.50
0.00
0.60
10.14
21.15
6.41
9,52
CLMVL
13.78
23.25
7.05
10.12
23.64
43.33
8.94
19.13
31.12
91.25
17.31
8.72
31.38
56.50
14.74
13,10
CLMVLM-2
38.50
15.38
23.21
35.35
73.25
12.18
32.74
39.05
77.75
24.36
19.64
22.09
42.25
16.35
14.88
IDÉFICS
0.00
0.50
0.00
0.60
5.85
18:50
0.96
1.79
21.18
95.25
13.78
0.60
4.01
0.00
0.32
0.00
MiniCPM
31.19
60.00
13.14
17.26
29.69
50
12.82
16.07
30.95
63.00
17.95
16.67
25.87
$2.25
15.38
22.02
LLaVA-7B*
9.42
15.75
385
4.76
10.79
23,00
7.69
10.12
GLaMM
23
50
6.09
4.76
62.90
51.00
$1.2
4643
GroundHOG
24.76
32.00
25.00
26.19
43,86
42.25
37.82
4226
GPT-4V
48.99
72.75
37.82
43.45
A/A
55.88
77.50
41.99
39.8
GPT-40
71.47
89.50
66.35 64.29
N/A
61.52
74.49
52.85
$3,89
Table 6: Complete per-object results on the seen split.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope?
Answer: [Yes]
Justification: Please see Abstract and Section Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section B.1 Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to- to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
21

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: Yes]
Justification: Please see Section and Appendix A.1 Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: We will release both the data and code to the public soon. For code, plan to release both the evaluation template and the code that we have used for data curation. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.cc/pplll
http://ooeSummmsssiooiiio
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines httpp/s///issscce/eeee guidess/Codeeuttmissiioooooo
for more details.
22

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes]
Justification: Please see Section and Section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes]
Justification: Please see Section 4 and Section 5 Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: Please see Appendix A.I Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
23

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppt..ea..e....////////aaa cctpppbllcc/tthicccciieee...eee.
Answer: [Yes]
Justification: Please see Appendix B.3 Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed, either positive negative. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e-g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper does not pose such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
24

---

Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have included licenses for existing assets the Appendix ??.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL..
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpprswipphhoooooooppppp.... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included data curation pipeline in the subsection of Appendix A.1 Data Curation Pipeline and we will release the dataset to the public soon.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
26


Reviews:
[{"review_id": "gDoKUgA7Jh", "reviewer": "Reviewer_4TNS", "summary": "1. The work proposes a **new benchmark evaluation method (through modifying existing datasets)** to measure if VLMs can accurately recognize multiple objects in an image simultaneously. This evaluation pipeline, designed by adding 5 bounding boxes to each image in the dataset and creating a fixed prompt, requires the model to identify which class each object within these boxes belongs to. \n\n2. Using this benchmark dataset, the authors analyze the performance of VLMs in recognizing multiple objects simultaneously and the **factors that may lead to their hallucination**. \n\n3. The contribution of the article lies in its **detailed analysis and insights into understanding VLMs' multi-object recognition capabilities**. Several important takeaways are:\n\n- performance degradation with multiple object recognition: VLMs perform worse when tasked with recognizing multiple objects at once compared to single object recognition. (not convinced by the current experiment results, need more clarification from authors -> see questions below)\n\n- using shortcut to explain hallucination behavior: When multiple objects of the same class are required to predict at the same prompt, VLMs tend to use previously recognized classes to predict subsequent object classes, leveraging shortcuts. It suggests the model simply relies on previously identified classes for subsequent predictions.", "questions": "3. Lines 182/183: **How are seen/unseen datasets divided**? Is it based on the classes from the Visual Genome instruction tuning dataset?\n\n4. Table 2's single-object setup seems confusing. Is it 1) each image with 5 bounding boxes results in **5 independent sessions** with the VLM, or 2) each session asks about one object while retaining history for the next question about another object from another bounding box, or 3) every image only have 1 bounding box? Is the wild/hom/het dataset division of the single-object column different only on the bounding boxes?\n   - Table 2: If it's 1), why is the single-object recognition accuracy so low, especially for LLaVA-34B? How does having 5 bounding boxes in a single-object setup impact results? For LLaVA-34B, performance for single and multiple objects is similar, which contradicts the expectation that performance would drop when asked about multiple objects simultaneously instead of focusing on single-object. Do you have other evidence supporting this claim?\n\n5. Lines 215 and 218: Can you explain the difference between hypothesis 1) and 2)? This section is somewhat confusing.\n\n6. Line 254: The logic here is unclear. Why does grounding tuning have little effect? Are there experimental results supporting this conclusion?\n\n7. Figure 5: Suggest explaining **how hallucinate and non-hallucinate are calculated in the caption**, or provide a reference link to the relevant section. For example, for Figure 5f, is the process as follows: consider every object from every 5k images -> in Toal 25k objects, and classify them into hallucinate and non-hallucinate, calculate everyone's semantic salience? \n\n8. Figures 5f/h/i: **It seems the results of hallucinate and non-hallucinate are reversed?** The descriptions seem contradictory. For instance, Figure 5h suggests that non-hallucinate objects have higher token entropy, **implying that more uncertain tokens are less likely to hallucinate.**\n\n9. Line 298: Why consider the last generated token instead of other positions, like the answer token?\n\n10. Figure 6: Why compare actual class and predicted class? I find it difficult to interpret due to lack information because each image should pair predicted and actual classes. Presenting them as two distributions loses this pairing information. A caption explanation of takeaways for these 3 images might help.\n\n11. Do you think hallucinations generation are entangled with the model's ability to correctly recognize visual text information for the bounding boxes? Have you tested the model's ability to recognize bounding boxes alone? \n\n12. (optional) Have you considered incorporating a user study where users provide bounding boxes and then answer questions? This would measure the model's capability in real-world scenarios where bounding box boundaries might be unclear.", "limitations": "The author has listed their limitation in the appendix B.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The task is well-defined and the evaluation metric is clear (I like the student-forcing and teacher-forcing idea): identify the object from the 5 bounding boxes at the same time.\n\n2. Comprehensive Analysis: It thoroughly examines various factors contributing to hallucination, offering valuable insights into potential causes. It can help the readers to understand hallucination causes in VLMs.", "weaknesses": "1. **Unclear Benchmark Positioning**: The paper's explanation about how this benchmark differs from other datasets is quite confusing (lines 35-48). What specific capability does this benchmark target to measure compared to others? Why is image captioning verification benchmarks insufficient for multiple object consideration? Given a limited evaluation budget, why choose this dataset over others? Is current common benchmarks focus on 1) image captioning, checking if mentioned objects appear in the image and their count, and 2) image grounding, verifying if given descriptions can be located in the image? Could you explain it in more details?\n\n2. **Writing** (minor): Some parts of the writing could be improved for better comprehension. For example, line 53 could explain earlier why the dataset is split into four sets. Readers only understand later through experiment analysis that these sets investigate if LLMs learn shortcuts by using previously predicted classes.\n\n3. **Clarification of Shortcuts and Spurious Correlations**:  The shortcut story is very interesting, could you elaborate this part more? What are the shortcuts and spurious correlations when claiming VLMs use them? What evidence supports this conclusion? IMO Teacher forcing results and Figure 4/5a indirectly support this, but a detailed, systematic analysis in one section would be better for understanding this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 4TNS\n\nWe greatly appreciate your dedicating your valuable time to this detailed review! We address your concerns as follows.\n\n### Weakness\n\nDue to the limited space, we kindly redirect the reviewer to the **General Author Rebuttals**.\n\n### Q3: Seen and Unseen\nThe Seen and Unseen datasets are divided based on whether the images originate from the training or test/validation sets of the original dataset. We use MSCOCO Panoptic and ADE20K datasets to build our datasets. We didn’t use Visual Genome as it does not have an official train/val split and people use the GQA splits conventionally.\n\n### Q4: Single-Object Setup\n\nStatement (1) is correct, the setup has “each image with 5 bounding boxes results in 5 independent sessions with the VLM.” The wild/hom/het dataset division is identical to the Multi-Object setups. The purpose is to keep all irrelevant factors untouched thus the comparisons between setups are fair and controlled.\n\n> Why is the single-object recognition accuracy so low, especially for LLaVA-34B…For LLaVA-34B, performance for single and multiple objects is similar…\n\nWe note that LLaVA-34B performance for single and multiple objects is similar ONLY for the situation of Seen+Heterogeneous data split. For all the other setups, performance for single is better than multiple objects. We are also aware that the performance of LLaVA-34B on Seen+Heterogeneous+Single Object is abnormal. We reran the experiments with 16 bit precision and remove model quantization, the updated number is 19.36%, which is almost the same as the previous score. Upon closer inspection, LLaVA-34B appears to memorize images encountered during pretraining and is prone to focusing on salient objects or objects with multiple occurrences.\n\n> How does having 5 bounding boxes in a single-object setup impact results?\nThere is a very marginal performance increase when keeping only one bounding box compared to five. We report the results with 5 bounding boxes to keep all irrelevant factors untouched thus the comparisons between setups are fair and controlled.\n\n### Q5: Shortcut Hypothesis\nFor hypothesis 1, the model might learn to perform class-agnostic object recognition better due to the ability to perceive context through a few-shot setting. For hypothesis 2, the model might learn to recognize one specific object in a few-shot setting. We expect improved performance on the last object B in the AAAAB setup for hypothesis 1, compared to the single object setting. For hypothesis 2 we expect the performance to be roughly the same.\n\n### Q6: Grounded Tuning\nWe refer specifically to the results for CogVLM-G presented in Table 2. The model performs poorly in the default multi-class setting, but shows relatively better performance under the single-object setting and teacher-forcing setting. We hypothesize that the grounding model fails to improve performance due to either a lack of instruction-following ability. This suggests that grounding alone does not effectively reduce multi-object hallucination in our benchmark, and conversational tuning is critical. We will add more detailed experimental results to support this conclusion in the revised manuscript.\n\n### Q7: Figure Caption Clarity\nWhen calculating hallucinated and non-hallucinated objects, we consider every object from all the unseen images, and classify the model’s prediction into hallucinated and non-hallucinated. We will improve the caption in the revised manuscript.\n\n### Q8: Figure 5 Error\nThank you for catching this bug in the plots, we realize that the legend for Figure 5 was misaligned with its content, which led to confusion. The hallucinated objects are represented in Yellow and the non-hallucinated are Green. We will correct these errors in the revised manuscript and ensure that the descriptions accurately reflect the results. \n\n### Q9: Last Token\nThis is a confusion arising from the way we described the algorithm. In our implementation, which uses a causal transformer, the model generates tokens sequentially. At each step, while generating the next token (which is the most recent token in the context of the transformer), the model attends to all previously generated tokens. This means the model’s attention mechanism considers the entire history of tokens up to that point. We apologize for any misunderstanding caused by our writing and will clarify this in the revised manuscript.\n\n### Q10: Actual Class and Predicted Class\nSorry for the confusion. For a model that hallucinates object A into object B, object A is the actual class, and object B is the predicted class. Indeed, presenting them as two distributions loses this one-to-one pairing information, but Zhou et al. [67] have already studied this setup and we emphasize on the statistical comparison instead.\n\n### Q11: OCR\nYes, we agree that our benchmark setting may entangle hallucinations with the model’s ability to correctly recognize visual text information within the bounding boxes. To address this, we conducted controlled experiments to isolate these factors. In the single-object setting, the model performed very similarly for each of the objects 1/2/3/4/5 within the same image, which indicates that its performance is stable with each visual prompt (See Appendix for the full tables). Also, since the same issue remains for both single and multi-object settings, our findings hold as the factors are controlled. Besides, our visual prompt setups mostly follow that of the set-of-mark prompting [54].\n\n### Q12: User Studies\nThank you for the valuable suggestion. Incorporating a user study where users provide bounding boxes and answer questions would indeed offer a meaningful evaluation of the model’s performance in real-world scenarios with potentially unclear boundaries. Due to time and resource constraints during the rebuttal process, we were unable to implement this approach. However, we consider it an excellent direction for future work and plan to explore it in subsequent research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F4wyyoIEqa", "reviewer": "Reviewer_i299", "summary": "This paper introduce a novel multi-object hallucination evaluation task, which assess model capabilities of classifying  multiple objects given visual prompts or spatial tokens. The benchmark dataset contains both commonly seen images and unseen images regarding the instruction dataset. Evaluation results show that open-source MLLMs and even proprietary GPT-4o struggles on this task.", "questions": "1.The legend for Figure 5 appears to be misaligned with its content, leading to confusion.  Also the discussion about Figure 5 is confusing. The analysis about (c) (f) (h) (i) are not aligned with sub-figures, while discussions about (d) (g) are aligned with the content.  (a) and (b) are not addressed in the discussion at all.\n2. It is unclear which prompting strategy (bounding box or special tokens) is utilized for each model listed in Table 2. This information should be specified for clarity.\n3. Typos: LLaVA-7B (unseen) student-forcing (wild) score should be in bold.", "limitations": "the authors adequately addressed the limitations", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of assessing multi-object hallucination is clear.\n2. The benchmark can facility the research on understanding MLLMs from a new perspective.\n3. The experiments are meticulously designed and cover a broad spectrum of factors.", "weaknesses": "1. The writing in the experiment section, particularly in Section 5, could benefit from significant improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s time and effort reviewing this paper. We thank the reviewer for the positive feedback on the motivation, novelty, and “meticulously designed” experiments. Please see our responses to your questions below.\n\n### W1: Improvement in Writing \n\nSpecifically, we describe our improvements for presentation in the experiment sections, especially Section 5.\n- **Enhanced Definitions and Explanations in Model Behaviors**: We will clarify the definitions and explanations for factors relevant to the mechanistic behaviors of the models. Specifically, we refined the descriptions and formulas for Object Token Entropy and Visual Modality Contribution to ensure they are more comprehensible.\n- **Refined Table and Figure Captions**: We will refine the table and figure captions to provide more context and detail about the settings and the content of each figure. This ensures that readers can better understand the visual data presented and the specific conditions under which the experiments were conducted.\n- **Detailed Analysis in Section 5.2**: We will rewrite and expand the section “When Do LVLMs Experience Multi-Object Hallucinations?” to provide a more detailed and clear analysis of each factor. Each factor is now thoroughly examined, with a deeper discussion of its impact on the model’s performance.\n\nWe kindly redirect the reviewer to the **General Author Rebuttals** for more details.\n\n### Q1: Legend\n\nWe apologize for the confusion caused by a mistake here. The legend for Figure 5 was misaligned with its content. The hallucinated objects are represented in Yellow and the non-hallucinated are Green. We will correct these errors in the revised manuscript and ensure that the descriptions accurately reflect the results.\n\n### Q2: Visual Prompt\n\nWe apologize for any confusion regarding the visual prompting strategies in Table 2 (Lines 194-198). For all LVLMs, we experiment with the bounding box as visual prompts. Specifically for CogVLM-G, we additionally experiment with their special grounding tokens as visual prompt input, as this is their natural visual prompt format and outperforms naive bounding box prompts. We report their performance using special grounding tokens in the Table.\n\n### Q3: Format\n\nThank you for catching this presentation error. We have now corrected this in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99etszBNOO", "reviewer": "Reviewer_i1pJ", "summary": "Previous evaluations of large vision-language model (LVLM) hallucinations have primarily focused on single objects. This paper introduces a novel hallucination evaluation benchmark named ROPE, which simultaneously assesses multiple objects within a single scene during testing. The authors present several empirical findings about multi-object hallucinations, including the statistical correlation in object distribution and its impact on hallucinatory responses.", "questions": "1. Do you have any initial ideas or hypotheses for mitigating multi-object hallucinations?\n2. Adding a baseline methods is not a mandatory rebuttal subject, but will be a good addition to the current version.", "limitations": "Yes, this paper generally addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and clearly articulates the differences between ROPE and previous benchmarks. Especially, observations and motivations are quite reasonable.\n2. Multi-object hallucination is an under-explored aspect in the LVLM hallucination scene. Also, its related findings are valuable and worth further investigation.\n3. The experiments and analyses conducted are extensive and detailed.", "weaknesses": "1. The authors can consider adding a simple baseline method for a prospective solution for mitigating multi-object hallucinations, as a starting point.\n2. Latency analysis in evaluation could be included to provide a more comprehensive analysis. For example, comparing 5 times single-object evaluation vs. multi-object evaluation. Also, especially for student/teacher forcing as they require multiple consecutive inferences. Understanding the time efficiency of the benchmark is important for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We are happy that you found our paper well-written, the problem novel and under-explored, and our experiments/analyses extensive and detailed. We addressed your questions and concerns below. If any residual concerns remain, we would be glad to discuss further. If no concerns remain, we would appreciate it if you could raise your score.\n\n### Q1. Initial ideas or hypotheses to mitigate multi-object hallucinations\n\nWe present an initial ideas on addressing multi-object hallucinations from a training perspective in Section 4.3.\nOur findings indicate that LVLMs tend to hallucinate less in homogeneous cases and more in heterogeneous cases. This suggests that LVLMs may get distracted by other objects and struggle to pay attention to the referred objects. Consequently, we propose a training-free method to mitigate this issue, see details below.\n\n### W1, Q2. Adding a baseline for multi-object hallucinations\n\nOne of the most natural way to make LVLMs \"pay more attention to\" what is inside the bounding box is to enlarge the amount of attention spent on the bounding box region. Since our dataset comes with ground truth bounding box regions and dimensions for each image, we retrieve the corresponding patches in the ViT that contains the bounding box region, and increase the self-attention on those tokens. \n\nAfter some trials and errors, we found:\n- Following ATMAN [1], we keep the selected tokens’ attention the same and scale all other tokens’ attention uniformly down by 0.7. Then after softmax, those bounding box regions will naturally obtain more attention.\n- We have to freeze the attention ratio between vision and text, and manipulate the attention within visual attention. Otherwise, LVLMs output random meaningless tokens.\n\n[1] AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation. Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, Kristian Kersting. NeurIPS, 2023.\n\n|               | Default Multi-Object |                      |                      | Single-Object        |                      |                      |\n|---------------|----------------------|----------------------|----------------------|----------------------|----------------------|----------------------|\n| **Models**    | **Wild**             | **Hom.**             | **Het.**             | **Wild**             | **Hom.**             | **Het.**             |\n| **Seen**                                                                                                                                                                             |\n| LLaVA-1.5                 | 21.26%               | 52.40%               | 7.69%                | 30.59%               | 60.85%               | 12.69%               |\n| LLaVA-1.5 + ATMAN-in-Box | 23.80%         | 55.60%               | 8.50%                | 32.89%               | 63.10%               | 14.85%               |\n| **Unseen**                                                                                                                                                                                          |\n| LLaVA-1.5                 | 13.96%               | 31.88%               | 3.98%                | 26.95%               | 54.41%               | 11.06%               |\n| LLaVA-1.5 + ATMAN-in-Box| 16.00%         | 35.60%               | 4.50%                | 29.00%               | 58.20%               | 13.90%               |\n\n### W2. Latency Analysis\n\nThank you for your valuable feedback. We appreciate your suggestion to include a latency analysis in our evaluation. In response to your comments, we have conducted additional experiments to compare the latency of five single-object evaluations with that of a multi-object evaluation.\n\n| Type                         | Time (per 100 attempts) |\n|------------------------------|-----------------------|\n| 5 times single-object evaluation | 520.34s               |\n| Multi-object evaluation               | 256.59s               |\n| Student forcing                           | 948.32s               |\n| Teacher forcing                           | 928.71s               |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4YjLhmGQJI", "reviewer": "Reviewer_uGDi", "summary": "This paper proposes a benchmark of multi-object hallucinations in Large Vision Language Models (LVLMs). Specifically, this paper explores thoroughly on how LVLMs behaves when multiple objects are prompted in user instructions. They introduce Recognition-based Object Probing Evaluation (ROPE), an automated pipeline used for collection their benchmark. This paper also presents some interesting findings, including that existing LVLMs may follow shortcuts to generate hallucinated responses.", "questions": "## **Q1** \n\nFor Figure 3 (right), ground truth object for obj5 is \"orange\", but both LLaVA-7B and LLaVA-34B hallucinate it as an \"apple\".\nThis paper explains this phenomena in line 165-166 as follows,\n\n> *However, they tend to make more mistakes when all tasked object classes are different or **when a new object class is introduced after multiple repeated tasks***.\n\nIt should be noted that authors use a very specific type of instruction here,\n\n> *Provide the class names in the format: 'obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>', with\nno additional words or punctuations.*\n\nI have a few questions for this example here. \n\n**(1-a)** how LLaVA-7B and LLaVA-34B performs when prompted with \"*Is there an orange in this image?*\" This can be a baseline to this example.\n\n**(1-b)** it should be noted that authors adopt three types of instructions in this paper (line 136-150). What type of instruction is used for this sample?\n\n**(1-c)** noticing that in this paper LVLMs are forced to follow a special format and decode object tokens only, what do authors think about significance of such prompting?\n\n## **Q2**\n\nGiven special instructions (Figure 2) used in this paper, a simple exploration is to train an LVLM (e.g., LLaVA) that follows such instruction well. One reason for why LVLMs performs badly is probably that, these models do not fit these special instructions very well. Authors are suggested to include such models and results accordingly. Specifically, authors can train a LLaVA model with instruction data below. For LLaVA [29] and take ground truth objects in Figure 2 as an example, this could look like,\n\n> *A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions*. \n\n> **USER**: *<image>*. \n\n> **USER**: *Provide the class names in the format: obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>, with no additional words or punctuations*. \n\n> **ASSISTANT**: *obj1: fork, obj2: knife, obj3: whisk, obj4: lemon, obj5: jar*.\n\nThis is also explored in a recent paper [A], which I think will improve this paper and make findings from this paper more convincing.\n\n[A] Object Recognition as Next Token Prediction. CVPR 2024.", "limitations": "Authors include Limitations in Appendix B.1.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**S1: Novelty**. Studying how object tokens affect each other in LVLM inference sounds new. Most papers addressing LVLM object hallucination are focusing on single-object. Multi-object hallucination sounds a new perspective.\n\n**S2: Thorough analysis**. This paper conducts comprehensive analysis and discuss on 9 potential impacting factors that may lead to multi-object hallucination, which are quite a lot experiments. \n \n**S3: Interesting academic findings**. The instruction format that induces LVLM to hallucinate is interesting. This is validated by authors a quite significant factor that leads to multi-object hallucination, which can not be simply remedied by scaling up LLMs.", "weaknesses": "**W1: missing evaluations**. Noticing that this paper mainly focuses on object hallucination, I wonder why authors avoid evaluating published methods that address single object hallucination in LVLMs, including decoding methods like OPERA [15] or RLHF methods [43, A]. It is quite important to check how existing single-object methods perform on the proposed multi-object benchmark. However, this part is missing from this paper, which is encouraged to be presented clearly and thoroughly to support a new benchmark.\n\n**W2: limited significance on real-world use case.** Despite interesting findings and comprehensive analysis on proposed multi-object hallucination, this paper experiments under a very specific type of inference setup. Specifically, their experiments are conducted on a specific type of instruction format (Figure 2) and limits its output tokens to objects only (line 140-150). For proposing a new benchmark, I believe this problem is worth to do some thinking. But this part is missing from this paper. Authors are suggested to discuss more on potential impact from this paper.\n\n[A] RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful review of our paper. We are happy that you appreciate our novelty and found our analyses thorough. We addressed your questions and concerns below. If any residual concerns remain, we would be glad to discuss further. If no concerns remain, we would appreciate it if you could raise your score.\n\n### W1: Missing Experiments.\nWe have added a comparative table in our revised manuscript that clearly presents the performance of OPERA, RLHF (MiniCPM-V), and other relevant methods on our multi-object benchmark. Due to the limit of space, we kindly redirect the reviewer to the **Rebuttal Supplement PDF**.\n- Decoding algorithm (OPERA): In the default multi-object setting, OPERA marginally improves the performance of LLaVA-1.5 in some settings, especially in settings with higher object homogeneity. OPERA decreases the performance on heterogeneous tests.\n- RL-based finetuning (MiniCPM-V): MiniCPM-V shows robust performance across different settings, consistently handling various object types effectively. This approach enhances model performance in multi-object benchmarks that even surpasses single-object settings in Wild and Hom sets. Upon inspection, we found that this model demonstrates strong visual in-context learning capability and improves correct recognition when objects of the same classes are probed together.\n\nIn summary, the heterogeneous test split remains challenging given recent advances in decoding and alignment.\n\n### W2: Limited Significance on Real-world Use Case\n\n**Real-World Use Case 1: Common Scenarios in Embodied AI**\n\nMulti-object querying is particularly relevant in embodied AI applications, such as in cooking scenarios. In a typical kitchen setting, a robot might need to identify and manipulate multiple ingredients and tools simultaneously. For example, preparing a meal might require the robot to locate and retrieve a knife, cutting board, vegetables, and spices all at once. This ability to handle multiple objects at the same time enhances the robot's efficiency and effectiveness, reducing the time taken to complete tasks and improving overall performance.\n\nWe further present a case study in Autonomous Driving (see Figures 1 and 2 in **Rebuttal Supplement PDF**). This case study demonstrates how our approach can be utilized in the automotive industry to enhance the accuracy and efficiency of object recognition systems in vehicles. Figures 1 and 2 are taken from the nuScenes dataset for autonomous driving. Figure 1 illustrates the single-object case, where each object is identified independently. Figure 2 demonstrates the multi-object case, where multiple objects are detected simultaneously. The multi-object case exhibits more hallucination errors compared to the single-object case. This finding underscores the importance of studying multi-object hallucination, especially in real-world scenarios like autonomous driving, where multiple objects need to be detected accurately at the same time.\n\n\n**Real-World Use Case 2: Cost and Time Efficiency**\n\nEvaluating multiple objects simultaneously, rather than querying each object individually, can significantly save both time and resources. \n\n| Type                           | Time (per 100 images) | Cost (per 100 images) |\n|-------------------------------|-----------------------|-----------------------|\n| 5 times single-object evaluation | 512s                  | $0.575                |\n| Multi-object evaluation        | 239s                  | $0.265                |\n\n\n### Q1: Figure 3 Example\n- For Q1-a, prompting with binary inference questions \"Is there an orange in this image?\" has been discussed in prior work, specifically in the POPE [24]. In our paper, we focus on the multi-object setting and our goal is to compare it with a single-object setting and ensure a fair comparison. Our single-object prompts are designed to align with our multi-object prompts for controlled studies, allowing us to directly evaluate and compare the performance under both settings. \n- For Q1-b, the instruction type used for the example in Figure 3 is the default multi-object setting, the same as Figure 2.\n- For Q1-c, we utilize student-forcing and teacher-forcing techniques to mitigate the model's dependence on the output format. These techniques allow the model to focus on predicting the next class rather than conforming to a specific output structure. Furthermore, student-forcing and teacher-forcing help us analyze whether the model learns shortcuts from the text. \n\n### Q2: Instruction Following\n\nIn fact, our student/teacher forcing probing strategies are designed \"to separate out errors due to following instructions, we force the model to follow the format template and decode only the object tokens for each of the five objects. Ideally, this setting allows the model to focus solely on object recognition.\" By decoding the object tokens conditioned on a correct template, student/teacher forcing probing strategies allow us to evaluate the model's performance in both single-object and multi-object settings more fairly, as it factors out the template dependence. See lines 140-150 and Figure 7 in the Appendix for details.\n\nWe appreciate your input and the reference to the recent paper [A]. We acknowledge its relevance to our work and will discuss this paper in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their detailed and thoughtful feedback. We are glad that the reviewers appreciate our motivation, task setup, analysis, and presentation. We hereby respond to the general concerns and update our experimental results.

### General 1: Benchmark Positioning (Reviewer 4TNS)

> What specific capability does this benchmark target to measure compared to others? 

ROPE specifically measures object hallucination in VLMs within a **multi-object setting**, examining how models may misperceive (e.g., by inventing nonexistent objects or becoming distracted) when tasked to focus on multiple objects concurrently, and which factors cause the hallucinations.

> Why is image captioning verification benchmarks insufficient for multiple object consideration?

In comparison to image captioning benchmarks such as CHAIR, ROPE offers two main advantages:
- Captioning benchmarks don’t address grounding and has referential ambiguity. For example, in an image with multiple apples, the ability to generate “there are apples in this image” does not imply that the model can recognize each individual apple correctly. ROPE provides a clear visual prompt that reduces such ambiguity.
- ROPE employs a fixed decoding template, enabling an automatic evaluation protocol, rather than relying on LLMs as evaluators. 

> Given a limited evaluation budget, why choose this dataset over others? 

ROPE is designed to be token-efficient and avoids the need for additional LLMs as evaluators. The benchmark employs a fixed decoding template, saving tokens compared to image captioning benchmarks like CHAIR.

> Is current common benchmarks focus on 1) image captioning, checking if mentioned objects appear in the image and their count, and 2) image grounding, verifying if given descriptions can be located in the image? Could you explain it in more detail?

Current common benchmarks focus on either (1) or use Yes/No questions to probe the models.
- CHAIR is one of the most well-known image captioning benchmarks for hallucination. CHAIR calculates what proportion of words generated are actually in the image according to the ground truth sentences and object segmentations.
- POPE can be seen as the image grounding benchmark for hallucination. POPE formulates the evaluation of object hallucination as a binary classification that prompts LVLMs to output “Yes” or “No”, e.g. “Is there a chair in the image?”

To the best of our knowledge, none of the existing object hallucination benchmarks satisfyingly address grounding and resolve referential ambiguity mentioned above.


### General 2: Shortcut and Spurious Correlation (Review 4TNS)

Yes, the “teacher forcing results and Figure 4” are intended for investigating shortcuts. We elaborate as follows.
- We discuss Shortcuts in Section 4.2 Paragraph 3, which are simple heuristic or rule-based solutions to a problem. In the teacher-forcing setting, we found that LLaVA models score over 90% accuracy. We design an Adversarial split, in which the first four tested objects are of the same class and we probe an object of a different class for the last one (AAAAB). The model's performance on the last object B drops to nearly zero, with almost all hallucinations labeling it as A. This is in stark contrast to 23.35% if these objects are probed individually or 19.16% when these objects are placed as the first to query in multi-object settings.
- We discuss Spurious Correlations in Section 5, which are features that appear to be statistically correlated with predictions. We present a systematic study of data-specific factors, salience/frequency, and model intrinsic behaviors. When the model hallucinates object A to object B, we study how these factors contribute to which object A to be hallucinated and which object B to hallucinate.

### General 3: Writing (Reviewer 4TNS, i299)

- **Enhanced Definitions and Explanations in Model Behaviors**: We will clarify the definitions and explanations for factors relevant to the mechanistic behaviors of the models. Specifically, we refined the descriptions and formulas for Object Token Entropy and Visual Modality Contribution to ensure they are more comprehensible.
- **Refined Table and Figure Captions**: We will refine the table and figure captions to provide more context and detail about the settings and the content of each figure. This ensures that readers can better understand the visual data presented and the specific conditions under which the experiments were conducted.
- **Detailed Analysis in Section 5.2**: We will rewrite and expand the section “When Do LVLMs Experience Multi-Object Hallucinations?” to provide a more detailed and clear analysis of each factor. Each factor is now thoroughly examined, with a deeper discussion of its impact on the model’s performance.
- **Improved Explanation of Dataset Split**: We will enhance the explanation of the dataset split and its purpose earlier in the text, providing a clearer rationale for the division into four sets. This revision ensures that readers understand the motivation behind the dataset split and how it relates to investigating whether LLMs use shortcuts by leveraging previously predicted classes.


### Updated Results and Additional Experiments

**See the PDF attached**

We present a case study in the real-world autonomous driving scenario, and also additional baselines:
- ATMAN-in-Box: A simple, training-free solution that we came up with to improve multi-object hallucination;
- Decoding strategies (OPERA)
- RL-based alignment (MiniCPM-V)
- Mechanistically grounded LVLMs (GLaMM and GroundHog)

Meta-review: The paper introduces a novel evaluation protocol to systematically study multi-object hallucination in LVLMs. The reviewers appreciated the paper's clear motivation, task setup, comprehensive empirical analysis, and effective rebuttal that addressed concerns regarding benchmark positioning, experimental clarity, and evaluation methods. The authors provided additional experiments and clarifications that strengthened the paper, showing the proposed method's advantages over existing benchmarks. Despite some initial concerns about writing clarity and methodological details, the authors' revisions and explanations resolved these issues. The recommendation is to accept.
Decision: Accept (poster)