Multi-Object Hallucination in Vision Language Models

Authors: xuweiyi  chen
ziqiao  ma
xuejun  zhang
sihan  xu
shengyi  qian
jianing  yang
david f fouhey
joyce  chai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 20:36:17

Abstract: Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images. 
While current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities, this work systematically investigates multi-object hallucination, examining how models misperceive (e.g., invent nonexistent objects or become distracted) when tasked with focusing on multiple objects simultaneously.
We introduce Recognition-based Object Probing Evaluation (ROPE), an automated evaluation protocol that considers the distribution of object classes within a single image during testing and uses visual referring prompts to eliminate ambiguity. 
With comprehensive empirical studies and analysis of potential factors leading to multi-object hallucination, we found that (1) LVLMs suffer more hallucinations when focusing on multiple objects compared to a single object. 
(2) The tested object class distribution affects hallucination behaviors, indicating that LVLMs may follow shortcuts and spurious correlations.
(3) Hallucinatory behaviors are influenced by data-specific factors, salience and frequency, and model intrinsic behaviors.
We hope to enable LVLMs to recognize and reason about multiple objects that often occur in realistic visual scenes, provide insights, and quantify our progress towards mitigating the issues.

Main paper: 
 introduction: 
 Recent advances in large language models (LLMs) have motivated increasing efforts in adapting them for understanding visual semantics, giving rise to a surge of large vision language models (LVLMs) [1, 37, 43] . These models, whether explicitly trained with grounding data [71] or without [31] , demonstrate an impressive grounded understanding of visual entities. This motivates a new prompting paradigm based on user-provided visual cues, referred to as visual prompting [45, 60, 27, 58, 58] . Despite their promising performances on various downstream applications [36] , LVLMs often suffer from object hallucination [44, 10, 26] , where they produce objects not present in a given image.Although object hallucination was initially observed in image captioning describing multiple objects [44] , current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities. These benchmarks either verify if an object class mentioned in the caption can ground to an object in the image [44, 19] , or probe the model about the existence of an object class, sometimes with additional attributes or relations to other objects [26, 32] . There are, however, two key limitations with these setups as shown by a case study in Figure 1 . First, grounding is not simply one-to-one between objects and classes, but a many-to-many mapping between objects and phrases [20, 34] . For instance, "apples" could potentially correspond to multiple referents in Figure 1 , and the model doesn't necessarily need to recognize all of them to provide such a response. Therefore, being able to produce an object that exists in an image does not necessarily Provide the class names in the format: 'obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>', with no additional words or punctuations. obj1: apple, obj2: knife, obj3: fork, obj4: apple, obj5: jar Figure 1 : A case study that compares our Recognition-based Object Probing Evaluation (ROPE) benchmark with existing benchmarks for object hallucination in GPT-4V. ROPE offers an automated evaluation protocol with controlled output formatting and uses visual prompts to distinctly ground to objects, thus mitigating referential ambiguity. Unlike binary inquiries relying solely on textual descriptions, ROPE challenges the model to identify multiple objects concurrently. We observe that, while GPT-4V can identify the whisk to the left of a knife when prompted about it, the model hallucinates a "fork" when directly tasked to recognize multiple objects.indicate that the model is free of hallucinations. Second, explicitly instructing the model to recognize multiple objects poses greater challenges compared to simple yes/no inquiries that contain explicit text descriptions for individual objects. For instance, while the model can correctly identify that a whisk is positioned to the left of a knife when "a whisk" is deliberately prompted, as shown in Figure 1(b-d ), it may hallucinate a "fork" when directly prompted to recognize both the whisk and the knife (i.e., Figure 1a ). This could be due to the common association between knives and forks, which leads to potential hallucinations when models are tasked to focus on multiple objects at the same time. In real-world applications, multi-object querying is crucial for embodied AI tasks. For example, in a cooking scenario, an agent must recognize multiple ingredients and tools simultaneously to be effective. We also present a case study on autonomous driving (see Figure ?? and Figure ?? in the appendix), demonstrating how common associations between cars, pedestrians and traffic lights could lead to potential hallucinations. In addition, evaluating multiple objects simultaneously, rather than querying each object individually, can significantly save both time and resources. To enable LVLMs to recognize and reason about multiple objects that often occur in realistic visual scenes and to better quantify the complex phenomena we observed, this paper investigates multi-object hallucination, examining how models may misperceive (e.g., by inventing nonexistent objects or becoming distracted) when tasked to focus on multiple objects concurrently, and which factors cause the hallucinations.We start by introducing Recognition-based Object Probing Evaluation (ROPE) for assessing multiobject hallucination with formatted output control. ROPE features an automated evaluation protocol without black-box neural models or humans as evaluators, and leverages visual prompts to uniquely refer to objects to avoid ambiguity and multiple referents caused by object class names. ROPE considers the distribution of object classes within each image at test time, dividing ROPE into 4 subsets: In-the-Wild, Homogeneous, Heterogeneous, and Adversarial. For instance, we investigate scenarios where all tested objects belong to the same class or where each tested object represents a different class. We conduct an in-depth analysis of the hallucination behaviors of LVLMs of different scales and training data (e.g., whether grounding data and conversational data are used), and provide a comprehensive analysis of potential factors that lead to multi-object hallucination. Our main findings are: (1) LVLMs suffer from more hallucinations when tasked to focus on multiple objects, compared to focusing on a single object; (2) The tested object class distribution affects the hallucination behaviors, revealing that LVLMs may be following shortcuts and spurious correlations; (3) The hallucinatory behaviors of LVLMs are affected by data-specific factors, salience and frequency, and model intrinsic behaviors. These findings provide key insights for the development and application of LVLMs, suggesting for more balanced object distributions, diverse annotations, and enhanced multi-object instructions in grounded LVLMs. We hope this work takes a step towards LVLMs that recognize and reason about multiple objects that often occur in realistic visual scenes. 
related work: 
 Large Vision-Language Models. There is a growing trend to harness and adapt the powerful large language models (LLMs) for multimodal understanding beyond text [51, 1, 65, 41] . Especially, visual instruction tuning has gained prominence for its competitive performance with a comparatively moderate amount of data and computational resources, leading to a variety of Large Vision-Language Models (LVLMs) [31, 30, 9, 76, 12, 56, 62, 22] . Grounding datasets have been shown to benefit vision-language pre-training [33, 25, 34] . Researchers have developed a family of grounded LVLMs focusing on object grounding to bounding box [40, 8, 72, 3, 63, 70, 39] and segmentation masks [21, 69, 57, 42, 71] . Of the large space of LVLMs, our work is most related to visual prompting [58, 60] and object hallucination [44, 10] . The paragraphs below describe the two lines of work in detail. Visual Prompting. LVLMs demonstrate their grounded understanding of user-provided visual cues, giving rise to a practical and user-friendly prompting paradigm known as visual prompting [58, 60] . Early work on visual prompting in vision-language models can date back to tuning-based methods [2, 61] . Recent studies show that LVLMs demonstrate zero-shot understanding of user-provided visual cues (e.g., a red circle) [45, 60] . This observation allows prompting LVLMs by editing images directly in the pixel space, e.g., by adding visual marks or visual text [27] . Starting from Set-of-Marks (SoM) prompting [58] , several training-free methods have been introduced [24, 59, 52] . Recent work further enhances visual prompt understanding by additional visual instruction tuning with diverse visual prompts overlaid on the images [6] , or explicitly represent visual pointer tokens in the models [21, 63, 71] . We leverage visual prompting to avoid potential ambiguity in textual descriptions, especially when evaluating multiple object hallucinations for objects of the same class.Design Considerations #Test Multi. Distr. Source Ref. Eval.CCEval [68] ✔ Seen Text N 0.1k GAVIE [29] ✔ Mixed Text N 1k FAITHScore [19] ✔ Unseen Text N 2k HaELM [54] ✔ Unseen Text N 5k M-HalDetect [13] ✔ Unseen Text H 0.8k MMHal-Bench [46] ✔ Unseen Text N,H 0.1k CHAIR [44] ✔ Unseen A 46k AMBER [53] ✔ Unseen Text A 1k CIEM [15] ✔ Unseen Text A 5k NOPE [32] Unseen A 3k POPE [26] Train Unseen A 0.5kROPE (Ours) ✔ Test Seen & Vis. A 5k UnseenTable 1 : An overview of object hallucination benchmarks. For design considerations, we summarize the number of tested images, and if multiple classes and object class distribution (at training and test time) are considered. The image sources include those seen or unseen during instruction tuning. To refer to an object, textual descriptions and visual cues can be adopted. For evaluation, neural models, humans and automatic pipelines are used.Object Hallucination. Despite their promising performance on benchmarks, these models frequently generate objects that do not exist in the provided images, a problem known as object hallucination [44, 10] . Several methods have been suggested to mitigate the object hallucination issue, such as integrating an external object detector [68] , applying visually grounded visual instruction tuning [63, 71] or reinforcement learning [46, 13] , performing iterative refinement [75] , and adapting the decoding strategies [17] . To quantify progress on mitigating them, various benchmarks have been developed and have revealed the prevalence of object hallucination, even in images that are seen during instruction tuning [68, 29] . We contrast our ROPE benchmark against existing benchmarks and setups in Table 3 . ROPE, which is designed for evaluating multi-object hallucination, is distinguished in several ways. First, we deliberately consider the distribution of object classes within a single image at test time. Object hallucination is observed originally in image captioning applications, where multiple objects are described [44] . While existing research has demonstrated that the object class distribution in the instruction tuning dataset can influence hallucination patterns [26, 75, 53] , the impact of object class distribution within an image at test time remains under-explored. Second, current benchmarks concentrate on the presence of an object class or distinguish instances using textual descriptions like attributes, which can still result in ambiguity and multiple referents. We instead leverage the visual referring prompting setups and use visual cues (i.e., marked bounding boxes) to refer to objects. Finally, our evaluation is automated, without black-box neural models or human evaluators. 
recognition-based object probing evaluation: 
 We introduce the Recognition Object Probing Evaluation (ROPE), an automated protocol for assessing LVLMs in multi-object recognition. ROPE specifically measures object hallucination in VLMs within a multi-object setting, examining how models may misperceive (e.g., by inventing nonexistent objects or becoming distracted) when tasked to focus on multiple objects concurrently, and which factors cause the hallucinations.Problem Definition. To avoid ambiguity from multiple candidate referents when using text prompts, ROPE leverages visual prompts to uniquely refer to objects. ROPE tasks LVLMs with selecting the best matching class for multiple objects, as referred to by the visual prompt, from a predefined set of object classes. Specifically, each sample in the ROPE protocol consists of a quadruple {I, L, ⟨p 2 ). This format enables automated evaluation through simple parsing. This format enables automated evaluation through simple parsing, avoiding black-box neural models or human evaluators With different analytical purposes, we designed 3 types of task prompts for Multi-Object queries, as illustrated in Figure 7 and described as follows.• Default: We probe the model to recognize the 5 objects referred to by the visual prompts concurrently in a single turn of prompting. This setting tasks the model with focusing on and recognizing all 5 objects simultaneously, aiming to capture the complexity involved when the model generates language that includes multiple objects. • Student-Forcing: One potential confounder in the default setting is the model's ability to generate data in the specified format. To separate out errors due to following instructions, we force the model to follow the format template and decode only the object tokens for each of the five objects. Ideally, this setting allows the model to focus solely on object recognition. • Teacher-Forcing: This setting eliminates cumulative error, allowing the model to condition on the correct previous context when generating object classes, leading to upper bound performance in multi-object recognition. We similarly force the model to follow the provided template and decode only the object tokens for each of the five objects, but we replace the previously generated object tokens with the ground truth. This essentially follows the few-shot in-context learning setting. Teacher forcing helps especially when LVLMs take shortcuts by repeating the object class list as ordered in the prompt (e.g., LLaVA-7B [31] and Gemini 1.0 Pro [49] in Figure 2 ).For comparison, we also designed task prompts for Single-Object query. We probe the model to recognize the object referred to by the visual prompts one at a time, repeating this as 5 independent and individual prompts. Unlike Default multi-object query, the model only needs to focus on one object, which can be seen as an extension of the POPE [26] setup from yes/no polling to classification. We refer to Appendix A.1 for the prompt templates for each type of task prompt.Data Sources and Curation. Since our goal is to evaluate and analyze multi-object hallucination, the image data must contain multiple objects of diverse classes with instance-level semantic annotations. We build our dataset upon existing panoptic segmentation datasets, including MSCOCO-Panoptic [28, 5] and ADE20K [74] , to ensure access to all object instances and their semantic classes. We note that one can build a dataset using the ROPE protocol with any dataset containing multiple objects and their bounding boxes. We describe the data curation pipeline in Appendix A.1. Provide the class names in the format: 'obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>', with no additional words or punctuations. Attending to Data Contamination. While data contamination has been explicitly handled in most of the existing benchmarks, object hallucination has been observed even in images that appear in the instruction tuning dataset, such as Visual Genome [68, 29] . To evaluate whether multi-object hallucination can be observed in both seen and unseen images, and to critically determine if training on these images helps reduce hallucinations, we explicitly split our dataset into Seen and Unseen based on the original split of the datasets. 2 Depending on the object query distributions (4 splits) and whether the image appears in the training split (2 splits), we divide the test into 8 folders. 
lvlm baselines: 
 The proposed ROPE framework, in principle, applies to all LVLMs that can follow format instructions and understand multiple visual prompts. To cover a variety of LVLMs of different scales and training data (e.g., whether grounding data and conversational data are used), we selected the following LVLMs as baselines.• LVLMs with base LLMs at different scales: LLaVA v1.6 (7B/13B/34B) [31, 30] and Yi-VL (6B/34B) [64] .Table 2 : Averaged accuracy of baselines on the In-the-Wild, Homogeneous, and Heterogeneous splits. The bold marker denotes the best-performing baseline and the underlined marker denotes the second-best-performing baseline.• LVLMs with conversational/grounded instruction tuning: QwenVL-Base/Chat (7B) [3] and CogVLM-Base/Chat/Grounding v1.1 (19B) [55] . • Mechanistically grounded LVLMs: GLaMM (7B) [42] and GroundHOG (7B) [71] .• LVLMs with RL-based finetuning: MiniCPM-V [66] • Other LVLMs: IDEFICS-instruct (9B) [23] , MiniCPM-V v2.5 (8B) [16, 67] , GPT-4V [37] , and GPT-4O [38] .For mechanistically grounded LVLMs that take visual prompts through specially designed mechanisms, such as pointer tokens in GroundHOG [71] , we additionally experiment with their default format and report whichever yields higher performance. For other LVLMs, we overlay the visual prompts on the images using a red bounding box with a width of 2 and visual text specifying the object index, presented with a white italic font on a black background with an alpha value of 0.75 for contrast and visibility. The performance of the LLaVA on the adversarial split, organized by the query sequence of AAAAB and BAAAA, reveals significant vulnerabilities as the model's accuracy dramatically declines for object 5 in AAAAB. SO stands for single-object probing and TF stands for teacher-forcing probing.We summarize the average results across the splits in Table 2 and present the most important findings below. The full tables appear in Appendix A.2.Multi-object tasks introduce more hallucinations. Our immediate observation is that LVLMs suffer from more hallucinations when tasked with focusing on and recognizing multiple objects compared to a single object. Across most of the models and test splits, we find that the average accuracy of single-object queries (i.e., probing object classes one at a time) significantly outperforms that of all three types of multi-object queries. The first exceptions are GPT-4O, MiniCPM-V, and CogVLM-2, the latter two leverage LLaMA-3 [35] . Another exception to this is when teacher-forcing is applied to homogeneous test splits, which demonstrates an unreasonably high accuracy. We discuss them later in this section.Heterogeneous queries introduce more hallucinations. We find that for all models and query methods, more heterogeneous queries lead to substantially more hallucinations, with performance decreasing from homogeneous to in-the-wild to heterogeneous test sets. The impact of heterogeneity applies to even start-of-the-art LVLMs like GPT-4O (Figure 2 ), although this performance gap is more significant in open-weight models.Language bias and shortcuts can lead to multi-object hallucinations. In the teacher-forcing setting, where there are no cumulative errors, LLaVA models score over 90% accuracy. There are three possible hypotheses for this abnormal observation: (1) LVLMs are smart enough to learn object recognition in general through few-shot in-context learning in the teacher-forcing setting; (2) LVLMs learn to recognize one specific object through few-shot in-context learning in the teacher-forcing setting; or (3) LVLMs simply exploit language biases and rule-based shortcuts (e.g., repeating previous answers). To reach a conclusion on this, we examine an Adversarial split, in which the first four tested objects are of the same class and we probe an object of a different class for the last one (e.g., AAAAB). We compare the single-object query performance with the teacher-forcing performance on the fifth object (object B). We anticipate the following outcomes: If hypothesis (1) is correct, the teacher-forcing performance should outperform the single-object query. If hypothesis (2) is correct, the teacher-forcing performance should perform on par with the single-object query.If hypothesis (3) is correct, the teacher-forcing performance should underperform compared to the single-object query. For a controlled comparison in the multi-object setting, we also reverse the order of queries (i.e., BAAAA) and repeat the experiments.We present the results of LLaVA models on the unseen split in Figure 4 , with the full results available in Appendix A.2. We find that the model's predictions on class A progressively improve, scoring nearly perfectly starting from the third repetition. However, the model's performance on the last object (with the different class label B) drops to nearly zero, with almost all hallucinations labeling it as A. This is in stark contrast to 23.35% if these objects are probed individually or 19.16% when these objects are placed as the first to query in multi-object settings. Our findings suggest that hypothesis (3) is true, indicating that the LVLMs' high performance on homogeneous queries could be an illusion resulting from textual shortcuts. We observe that models show lower performance in identifying object class B in single-object analysis, potentially due to the higher salience of object class A in these images.Multi-object hallucinations occur in both seen and unseen images. We finally investigate whether our observations and findings hold uniformly in both seen and unseen splits. We observe that the gap between multi-object hallucination and single-object hallucination, as well as the reliance on shortcuts, persists. Although most of the models perform slightly better on seen images, the trends remain consistent across both splits. While large-scale training is involved in developing these LVLMs, it appears they might not have fully exploited the fine-grained information in the data.Training on these images does not significantly reduce object hallucinations.Comparing the tested LVLMs, we discuss our observations regarding design considerations that may or may not help reduce multi-object hallucinations.Scaling the base LLM: data and parameters. We find that using base LLMs with more parameters reduces single-object hallucinations, but may not have the same effect on multi-object hallucinations.We observe a consistent increase in performance with larger LLaVA models in the seen set and in single-object queries, but not in the unseen set with multi-object queries. One possible explanation for this finding is that LLMs with more parameters are better at memorizing seen images, as the performance gap between seen and unseen images is also more significant in larger models. We also notice that the performance gap between single-object probing and multi-object probing does not apply to MiniCPM-V and CogVLM-2, which adopt a LLaMA-3 (8B) [35] base LLM pre-trained with 15T tokens, as they fail to follow the instruction sometimes. Compared to LLaVA models developed upon LLaMA-2 (7/13B) [50] and Yi (34B) [64] with 2T and 3T pre-training tokens, these models underperform in quantitative measures due to instruction following error but exhibit greater robustness when multiple visual prompts are presented.Visual instruction fine-tuning: chat and grounding. While it's surprising that conversational tuning reduces multi-object hallucinations, we observe that models without conversational tuning struggle to follow instructions and are prone to shortcuts, such as repeating the list of all object class candidates in order or consistently repeating the first candidate. This might also explain why grounded tuning in CogVLM-G is of little help in reducing multi-object hallucinations thus far. These models typically lack conversational fine-tuning, and there is currently no available grounded dialogue data at scale. While mechanistically grounded LVLMs show strong performances in single-object probing, there remain a gap in multi-object probing with student forcing. This could be attributed to a significant portion of the grounded instruction tuning dataset consisting mainly of short captions or questions featuring one single or few objects.RL-based finetuning approaches, such as MiniCPM-V, demonstrate promising results across diverse settings, surpassing single-object results in both the Wild and Homogenous settings. Upon inspection, we found that this model demonstrates strong visual in-context learning capability and improves correct recognition when objects of the same classes are probed together.Decoding and inference time strategy Decoding algorithms like OPERA introduce nuanced improvements in specific multi-object settings [18] . In default multi-object tasks, OPERA shows marginal performance enhancements for LLaVA-1.5, but its effectiveness declines in tests with greater heterogeneity, to the point it can even lower performance. This suggests OPERA is beneficial in homogeneous contexts but requires further refinement in handling mixed object scenarios effectively.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Please see Section 3 and Appendix A.1 Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
potential hallucinatory factors: 
 The task setup described above allows us to evaluate LVLMs in multi-object hallucinations and identify hallucinatory behaviors. Based on existing literature and our case studies, we further identify potential factors that correlate to and potentially explain these hallucinations.Data-specific Factors. We consider the following factors that are specific to the tested sample (e.g., object and token positions), and are not relevant to the frequency distribution.• Input Order: we consider the order in which the object classes are presented in the input prompt containing all candidates. • Query Homogeneity: We define query homogeneity as the total number of task objects of the same class, normalized by the total number of queried objects (five in this work). • Object Token Position: Zhou et al. [75] has shown that more hallucinations occur in the latter part of captions. In this work, the object indices directly correspond to the object token positions. • Object Homogeneity: We define object homogeneity as the number of object types in the image, calculated upon panoptic annotations. • Object Centrality: Previous research has identified a center bias in datasets and models, indicating that objects are disproportionately located at the center of images in detection models [47, 48] . We define object centrality as one minus the distance d between the object's bounding box center and the image center, normalized by the diagonal distance D from the center to the corner. Salience and Frequency. We consider the following factors that are related to the saliency or frequency of the visual object or the object class.• Object Salience: Previous research has shown that smaller objects are harder to detect and ground to [14, 34] . We define object salience as the ratio of the number of pixels occupied by the object's instance segmentation mask to the total number of pixels in the image. • Semantic Salience: We observe and hypothesize that LVLMs are less likely to hallucinate objects when they co-occur with multiple copies of the same class ("jar" in Figure 2 ). We define semantic salience as the ratio of the total number of pixels in all instances of the same class, to the total number of pixels in the image. • Training Salience: Previous research has shown that spurious co-occurring patterns in the training data can lead to object hallucinations [26, 75] . We use the log frequency of classes in MSCOCO as a proxy for training salience following previous work, and hypothesize that LVLMs tend to hallucinate more on less frequent objects in the training set.Model Behaviors. We consider the following factors relevant to the mechanistic behaviors.• Object Token Entropy: Zhou et al. [75] have shown that object hallucinations are more likely when the decoded object tokens have a higher log perplexity. In our work, we define object token entropy as the entropy of the logits of the first token in the generated word. Given s as the softmax logits of the generated word's first token, we calculate the entropy using the following formula: H(s) =i s i log(s i ). Simply put, higher entropy indicates greater uncertainty in the model's prediction for the first token, which can lead to more frequent object hallucinations. • Visual Modality Contribution: We hypothesize that LVLMs pay less attention to the visual modality during object hallucinations. Motivated by the modality importance score [7] , we define Visual Modality Contribution (VMC) as the proportion of attention allocated to visual tokens compared to textual tokens. To quantify this, we analyze the attention weights of the last generated token across all heads and layers. The VMC is computed as follows: VMC = i∈V αij / i∈V α ij + k∈T α kj , where α ij represents the attention weight assigned to visual token i at head j, and α kj represents the attention weight assigned to textual token k at head j. The sets V and T denote the visual and textual tokens, respectively. By examining the VMC, we can determine how much attention is given to visual inputs in comparison to textual inputs. A lower VMC may indicate a higher likelihood of object hallucinations due to insufficient attention to visual cues.In Figure 5 , we compare the distribution of these factors between hallucinatory and non-hallucinatory objects in the student forcing setting on the unseen split using LLaVA-13B. For continuous values, we use ridgeline plots, and for discrete values with fewer bins, we use bar charts. Data/Task-specific Factors. We observed that specific data factors, such as query and object homogeneity, significantly influence model performance, with increased hallucination occurring when models process images featuring multiple object classes or a variety of objects. For positional factors, the position of object tokens seems to have minimal impact and the object centrality has only a slight influence as LVLMs tend to hallucinate objects more frequently when they are positioned away from the center. This tendency may stem from a reporting bias, as objects mentioned in captions are typically foreground objects that distribute toward the centers of images. Salience and Frequency. We note that semantic salience significantly affects the model's performance, as it is more prone to hallucinate an object class that is less salient within the image. Conversely, the salience of individual objects does not statistically correlate with hallucination incidents. This implies that LVLMs may rely more on the presence of co-occurring objects of the same class to predict the labels of queried objects, rather than solely on the presence or salience of the objects themselves. Additionally, training salience plays a crucial role as models are less likely to hallucinate object classes that frequently appear in training. Intrinsic Behaviors. The intrinsic behaviors of the model provide significant insights into its tendencies to hallucinate. Similar to Zhou et al. [75] , we find that models are more prone to hallucination when they experience uncertainty or confusion, especially in scenarios involving multiple objects, as evidenced by higher token entropy. Furthermore, the contribution from the visual modality consistently registers below 20%, suggesting that current LVLMs may rely more heavily on linguistic contexts. There is a marginal increase in the likelihood of hallucination when models pay less attention to the visual context.In Figure 6 , we conducted a detailed comparison of the distribution of actual versus predicted object classes within the context of hallucinatory objects, examining factors such as semantic salience, training salience, and input order. Although semantic salience is a key factor in determining whether a model hallucinates, it appears to have minimal impact on the prediction of hallucinated objects. Our analysis also shows that models are more likely to hallucinate object classes that are prevalent in the training data, but the reverse is not necessarily true. Additionally, there is a notable preference for models to hallucinate objects that are listed early in the input prompt as candidate classes. Overall, our findings indicate that spurious correlations may lead to hallucinations involving multiple objects.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will release both the data and code to the public soon. For code, we plan to release both the evaluation template and the code that we have used for data curation. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: There is no societal impact of the work performed, either positive or negative. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussions and conclusion: 
 Hallucinations in large vision-language models (LVLMs) can occur at different scales and granularities. In this study, we study the problem of multi-object hallucination, examining how LVLMs may misperceive when tasked to focus on multiple objects concurrently, and which factors cause the hallucinations. we introduce Recognition-based Object Probing Evaluation (ROPE), an automated evaluation protocol designed to account for the distribution of object classes within a single image during testing and to use visual referring prompts to reduce ambiguity. Our research provides key insights for the development and application of LVLMs. Since models tend to experience more hallucinations with multiple objects than with single ones, it may be advantageous to probe objects individually in visual prompts to enhance performance. The likelihood of a model's hallucinatory output is linked to various data factors and model behaviors. Particularly in situations involving heterogeneous data and low certainty from the model, there is an increased risk of hallucinations, and users should be vigilant. Moreover, our analysis indicates that merely adopting (grounded) instruction tuning and scaling the base language model may not be enough to fully address the issue of object hallucination. There is a need for more balanced object distributions, annotations of objects away from image centers, and an increase in diversity. Introducing instructions that require multiple visual pointers and complex multi-object reasoning is also crucial. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does not pose such risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have included licenses for existing assets in the Appendix ??. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have included data curation pipeline in the subsection of Appendix A.1: Data Curation Pipeline and we will release the dataset to the public soon. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A. To refer to an object, textual descriptions and visual cues can be adopted. For evaluation, neural models, humans and automatic pipelines are used.Data Curation Pipeline. Our data curation pipeline involves several essential steps designed to prepare and refine our dataset for evaluating multi-object hallucination. The pipeline begins by filtering images and candidate objects to query. We consider valid objects to be those belonging to the top 50 "thing" classes and exclude objects with a bounding box area less than 1% of the total image area. We discard images containing fewer than 5 valid objects, and allow an intersection-over-union between bounding boxes of no more than 0.1, which preserves data integrity while ensuring high image quality. We apply this pipeline to MSCOCO-Panoptic [28, 5] and ADE20K [74] . 3Language Instruction Prompt Templates.We illustrate the 4 types of task prompts for Single-Object and Multi-Object queries in Figure 7 , and document the prompts below. Provide the class names in the format: 'obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>', with no additional words or punctuations. Computational Resources. Our experiments were conducted on eight A40 and four A100 GPUs slightly over a week. The computational bottleneck was not the numerical accuracy values but the collection of potential hallucinatory factors for analytical purposes, including logits and attention values for each head and layer.Per-object Performance. We provide the per-object performance in the following Table 5 and Table 6 .Autonomous Driving Case Study. Figure 8 is a case study example from the nuScenes dataset [4] for autonomous driving. It illustrates the single and multi-object case, where each object is identified independently. The multi-object case exhibits more hallucination errors compared to the single-object case. This finding underscores the importance of studying multi-object hallucination, especially in real-world scenarios like autonomous driving, where multiple objects need to be detected accurately at the same time.B Limitations, Licenses, and Risks B.1 Limitations ROPE represents one of the pioneering efforts to publicly address the issue of multiple object hallucination. However, we acknowledge several limitations in our work: (1) The lack of transparency in the LVLMs makes it difficult to guarantee that our unseen dataset has not been previously exposed.(2) Our evaluation benchmark uses a fixed set of semantic objects, which may introduce bias and impose unnecessary constraints on the LVLMs' ability to follow instructions and reason effectively.(3) The evaluation process can be slow, as it involves performing five inferences per image for both student forcing and teacher forcing.We report a list of licenses for all datasets and models used in our experiment in Table 4 . We strictly follow all the model licenses and limit the scope of these models to academic research only. 4 : License information for the scientific artifacts used.This study does not require human annotators or participants for its interactive experiments. Instead, it utilizes publicly available datasets and content created by models for evaluation purposes. We are aware that these public data might introduce biases and sensitive elements, and it is essential for future research to address these concerns, possibly by creating datasets that incorporate fairness-based filtering and metrics.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.We believe this approach is the best practice, but we also acknowledge that the distinction between seen and unseen images may not be strict. Uncurated web images often overlap with public test images, and researchers have no transparent access to the datasets used to train some of these LVLMs unfortunately[11].Available at https://huggingface.co/datasets/sled-umich/ROPE 


Decision: The paper introduces a novel evaluation protocol to systematically study multi-object hallucination in LVLMs. The reviewers appreciated the paper's clear motivation, task setup, comprehensive empirical analysis, and effective rebuttal that addressed concerns regarding benchmark positioning, experimental clarity, and evaluation methods. The authors provided additional experiments and clarifications that strengthened the paper, showing the proposed method's advantages over existing benchmarks. Despite some initial concerns about writing clarity and methodological details, the authors' revisions and explanations resolved these issues. The recommendation is to accept.

Reviews: review: paper_summary: paper_summary: 1. The work proposes a **new benchmark evaluation method (through modifying existing datasets)** to measure if VLMs can accurately recognize multiple objects in an image simultaneously. This evaluation pipeline, designed by adding 5 bounding boxes to each image in the dataset and creating a fixed prompt, requires the model to identify which class each object within these boxes belongs to. 

2. Using this benchmark dataset, the authors analyze the performance of VLMs in recognizing multiple objects simultaneously and the **factors that may lead to their hallucination**. 

3. The contribution of the article lies in its **detailed analysis and insights into understanding VLMs' multi-object recognition capabilities**. Several important takeaways are:

- performance degradation with multiple object recognition: VLMs perform worse when tasked with recognizing multiple objects at once compared to single object recognition. (not convinced by the current experiment results, need more clarification from authors -> see questions below)

- using shortcut to explain hallucination behavior: When multiple objects of the same class are required to predict at the same prompt, VLMs tend to use previously recognized classes to predict subsequent object classes, leveraging shortcuts. It suggests the model simply relies on previously identified classes for subsequent predictions.
strength_weakness: strength_weakness: 1. **Unclear Benchmark Positioning**: The paper's explanation about how this benchmark differs from other datasets is quite confusing (lines 35-48). What specific capability does this benchmark target to measure compared to others? Why is image captioning verification benchmarks insufficient for multiple object consideration? Given a limited evaluation budget, why choose this dataset over others? Is current common benchmarks focus on 1) image captioning, checking if mentioned objects appear in the image and their count, and 2) image grounding, verifying if given descriptions can be located in the image? Could you explain it in more details?

2. **Writing** (minor): Some parts of the writing could be improved for better comprehension. For example, line 53 could explain earlier why the dataset is split into four sets. Readers only understand later through experiment analysis that these sets investigate if LLMs learn shortcuts by using previously predicted classes.

3. **Clarification of Shortcuts and Spurious Correlations**:  The shortcut story is very interesting, could you elaborate this part more? What are the shortcuts and spurious correlations when claiming VLMs use them? What evidence supports this conclusion? IMO Teacher forcing results and Figure 4/5a indirectly support this, but a detailed, systematic analysis in one section would be better for understanding this.
questions: questions: 3. Lines 182/183: **How are seen/unseen datasets divided**? Is it based on the classes from the Visual Genome instruction tuning dataset?

4. Table 2's single-object setup seems confusing. Is it 1) each image with 5 bounding boxes results in **5 independent sessions** with the VLM, or 2) each session asks about one object while retaining history for the next question about another object from another bounding box, or 3) every image only have 1 bounding box? Is the wild/hom/het dataset division of the single-object column different only on the bounding boxes?
   - Table 2: If it's 1), why is the single-object recognition accuracy so low, especially for LLaVA-34B? How does having 5 bounding boxes in a single-object setup impact results? For LLaVA-34B, performance for single and multiple objects is similar, which contradicts the expectation that performance would drop when asked about multiple objects simultaneously instead of focusing on single-object. Do you have other evidence supporting this claim?

5. Lines 215 and 218: Can you explain the difference between hypothesis 1) and 2)? This section is somewhat confusing.

6. Line 254: The logic here is unclear. Why does grounding tuning have little effect? Are there experimental results supporting this conclusion?

7. Figure 5: Suggest explaining **how hallucinate and non-hallucinate are calculated in the caption**, or provide a reference link to the relevant section. For example, for Figure 5f, is the process as follows: consider every object from every 5k images -> in Toal 25k objects, and classify them into hallucinate and non-hallucinate, calculate everyone's semantic salience? 

8. Figures 5f/h/i: **It seems the results of hallucinate and non-hallucinate are reversed?** The descriptions seem contradictory. For instance, Figure 5h suggests that non-hallucinate objects have higher token entropy, **implying that more uncertain tokens are less likely to hallucinate.**

9. Line 298: Why consider the last generated token instead of other positions, like the answer token?

10. Figure 6: Why compare actual class and predicted class? I find it difficult to interpret due to lack information because each image should pair predicted and actual classes. Presenting them as two distributions loses this pairing information. A caption explanation of takeaways for these 3 images might help.

11. Do you think hallucinations generation are entangled with the model's ability to correctly recognize visual text information for the bounding boxes? Have you tested the model's ability to recognize bounding boxes alone? 

12. (optional) Have you considered incorporating a user study where users provide bounding boxes and then answer questions? This would measure the model's capability in real-world scenarios where bounding box boundaries might be unclear.
limitations: limitations: The author has listed their limitation in the appendix B.
paper_summary: paper_summary: This paper introduce a novel multi-object hallucination evaluation task, which assess model capabilities of classifying  multiple objects given visual prompts or spatial tokens. The benchmark dataset contains both commonly seen images and unseen images regarding the instruction dataset. Evaluation results show that open-source MLLMs and even proprietary GPT-4o struggles on this task.
strength_weakness: strength_weakness: 1. The writing in the experiment section, particularly in Section 5, could benefit from significant improvement.
questions: questions: 1.The legend for Figure 5 appears to be misaligned with its content, leading to confusion.  Also the discussion about Figure 5 is confusing. The analysis about (c) (f) (h) (i) are not aligned with sub-figures, while discussions about (d) (g) are aligned with the content.  (a) and (b) are not addressed in the discussion at all.
2. It is unclear which prompting strategy (bounding box or special tokens) is utilized for each model listed in Table 2. This information should be specified for clarity.
3. Typos: LLaVA-7B (unseen) student-forcing (wild) score should be in bold.
limitations: limitations: the authors adequately addressed the limitations
paper_summary: paper_summary: Previous evaluations of large vision-language model (LVLM) hallucinations have primarily focused on single objects. This paper introduces a novel hallucination evaluation benchmark named ROPE, which simultaneously assesses multiple objects within a single scene during testing. The authors present several empirical findings about multi-object hallucinations, including the statistical correlation in object distribution and its impact on hallucinatory responses.
strength_weakness: strength_weakness: 1. The authors can consider adding a simple baseline method for a prospective solution for mitigating multi-object hallucinations, as a starting point.
2. Latency analysis in evaluation could be included to provide a more comprehensive analysis. For example, comparing 5 times single-object evaluation vs. multi-object evaluation. Also, especially for student/teacher forcing as they require multiple consecutive inferences. Understanding the time efficiency of the benchmark is important for practical applications.
questions: questions: 1. Do you have any initial ideas or hypotheses for mitigating multi-object hallucinations?
2. Adding a baseline methods is not a mandatory rebuttal subject, but will be a good addition to the current version.
limitations: limitations: Yes, this paper generally addressed the limitations.
paper_summary: paper_summary: This paper proposes a benchmark of multi-object hallucinations in Large Vision Language Models (LVLMs). Specifically, this paper explores thoroughly on how LVLMs behaves when multiple objects are prompted in user instructions. They introduce Recognition-based Object Probing Evaluation (ROPE), an automated pipeline used for collection their benchmark. This paper also presents some interesting findings, including that existing LVLMs may follow shortcuts to generate hallucinated responses.
strength_weakness: strength_weakness: **W1: missing evaluations**. Noticing that this paper mainly focuses on object hallucination, I wonder why authors avoid evaluating published methods that address single object hallucination in LVLMs, including decoding methods like OPERA [15] or RLHF methods [43, A]. It is quite important to check how existing single-object methods perform on the proposed multi-object benchmark. However, this part is missing from this paper, which is encouraged to be presented clearly and thoroughly to support a new benchmark.

**W2: limited significance on real-world use case.** Despite interesting findings and comprehensive analysis on proposed multi-object hallucination, this paper experiments under a very specific type of inference setup. Specifically, their experiments are conducted on a specific type of instruction format (Figure 2) and limits its output tokens to objects only (line 140-150). For proposing a new benchmark, I believe this problem is worth to do some thinking. But this part is missing from this paper. Authors are suggested to discuss more on potential impact from this paper.

[A] RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. CVPR 2024.
questions: questions: ## **Q1** 

For Figure 3 (right), ground truth object for obj5 is "orange", but both LLaVA-7B and LLaVA-34B hallucinate it as an "apple".
This paper explains this phenomena in line 165-166 as follows,

> *However, they tend to make more mistakes when all tasked object classes are different or **when a new object class is introduced after multiple repeated tasks***.

It should be noted that authors use a very specific type of instruction here,

> *Provide the class names in the format: 'obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>', with
no additional words or punctuations.*

I have a few questions for this example here. 

**(1-a)** how LLaVA-7B and LLaVA-34B performs when prompted with "*Is there an orange in this image?*" This can be a baseline to this example.

**(1-b)** it should be noted that authors adopt three types of instructions in this paper (line 136-150). What type of instruction is used for this sample?

**(1-c)** noticing that in this paper LVLMs are forced to follow a special format and decode object tokens only, what do authors think about significance of such prompting?

## **Q2**

Given special instructions (Figure 2) used in this paper, a simple exploration is to train an LVLM (e.g., LLaVA) that follows such instruction well. One reason for why LVLMs performs badly is probably that, these models do not fit these special instructions very well. Authors are suggested to include such models and results accordingly. Specifically, authors can train a LLaVA model with instruction data below. For LLaVA [29] and take ground truth objects in Figure 2 as an example, this could look like,

> *A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions*. 

> **USER**: *<image>*. 

> **USER**: *Provide the class names in the format: obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>, with no additional words or punctuations*. 

> **ASSISTANT**: *obj1: fork, obj2: knife, obj3: whisk, obj4: lemon, obj5: jar*.

This is also explored in a recent paper [A], which I think will improve this paper and make findings from this paper more convincing.

[A] Object Recognition as Next Token Prediction. CVPR 2024.
limitations: limitations: Authors include Limitations in Appendix B.1.

Comments: title: Further Clarifications by Authors
comment: Thank you for your detailed and thoughtful feedback throughout the review process. We’re pleased that our response and additional experiments addressed your concerns.
Thank you for the response and additional experiments provided. After reading, I decide to maintain the current score and evaluation.
It’s great to know that we were able to meet your expectations. Thank you so much for your careful review and the insightful feedback. We genuinely appreciate the time you spent helping us improve our work.
We’re happy to hear that we could cover all your points. Your detailed feedback was really helpful, and we’re grateful for the time you took to review our work and share your thoughts.
We are glad that we addressed all of your concerns. Thank you very much for your thoughtful consideration and feedback. We greatly appreciate your time and effort in reviewing our work.
I appreciate the effort authors put into addressing my feedback, particularly in relation to mitigating multi-object hallucinations and conducting the latency analysis. Also, the addition of a baseline method for multi-object hallucinations is a valuable enhancement.

Overall, I remain inclined to accept and maintain my score.
The authors' responses are clear and my concerns on writing are addressed.
At first, I would like to thank for authors responding to my concerns in details. 

Good to see that authors include single-object hallucination methods, including OPERA and MiniCPM-V in their global response. My major concern of **W1** has been addressed by inclusion of these two approaches. Though, authors are suggested to include more hallucination mitigation methods in future updated manuscript. 

I also recognize potential application value of multi-object hallucination in robotics and embodied AI, which addresses my concern **W2** and improves this paper. 

Further clarifications on **Q1** and **Q2** are clear, I have no further questions.  

In all, I think this paper has its novel contribution and raise my score.
We appreciate your instant feedback! We would like to provide a more detailed clarification regarding Figures 5 (and 6) and the Object Centrality metric.

We first note that the figures and findings themselves are correct, and clarify that the consistency issue arises from typos and presentation ambiguity.
- **(Clarification i) Typo in the legend for Figure 5**: The hallucinated objects are represented in yellow, and the non-hallucinated objects are in green. This color convention is used throughout the paper. In the Figure 5 legend, we mistakenly flipped this for the color blocks.
- **(Clarification ii) Object Centrality**: The object centrality is defined as $1-d/D$ rather than $d/D$. We realize that our wording is confusing and will clarify this in the updated version.

Regarding your specific concerns:

---
> The analysis about (c) (f) (h) (i) are not aligned with sub-figures...

The reason for this misalignment is the flipped legend, as we clarified in **(Clarification i)**.

---
> ...discussions about (d) (g) are aligned with the content...

> ...#313 states 'LVLMs tend to hallucinate objects more frequently when they are positioned away from the center', which means the object centrality (normalized distance d between object and image center, as introduced in #273) positively correlated with the hallucination rate.


- **Figure (d)**: Upon **(Clarification i)** and **(Clarification ii)**, we mean to define the object centrality as $1-d/D$ rather than $d/D$. In this way, the object which is closer to the center of the image has a higher object centrality value. Considering the color for (d) was reversed, it matches our analysis that “the object centrality has only a slight influence as LVLMs tend to hallucinate objects more frequently when they are positioned away from the center.”
- **Figure (g)**: Figure (g) was actually discussed correctly, indicating that non hallucinated objects appear more frequently in training data. This matches our discussion that “models are less likely to hallucinate object classes that frequently appear in the training dataset.” We hope the reviewer can take a second inspection and let us know if there are other confusions.

---
> ...(a) and (b) are not addressed in the discussion at all.

- Figure (a) and (b) are discussed in Section 5.2. Figure (a) is discussed in Lines 309-311, Figure (b) is discussed in Lines 311-314. We analyze that “specific data factors, such as query and object homogeneity, significantly influence model performance, with increased hallucination occurring when models process images featuring multiple object classes or a variety of objects” and “the position of object tokens seems to have minimal impact".

---
> The authors may at least need to clarify the discussion about Figure 5, 6 for the consistency between the proposed analysis and the reported experimental results.

While we clarify the typos and ambiguities in Figure 5, we believe that Figure 6 is consistent with our proposed analysis.
- For (a), there is no significant difference between the model’s predicted class and the actual class, which aligns with our statement that “although semantic salience is a key factor in determining whether a model hallucinates, it appears to have minimal impact on the prediction of hallucinated objects”.
- For (b), the figure shows that the model’s predicted class has a higher training salience than the actual class, matching our analysis that ”models are more likely to hallucinate object classes that are prevalent in the training data”.
- For (c), the figure shows that the position of the predicted class is listed earlier in the input prompt than the actual class, which matches our analysis that “there is a notable preference for models to hallucinate objects that are listed early in the input prompt as candidate classes”. 

We apologize again for the confusion caused by the mistakenly annotated figure, but we believe this might be a minor issue that we can easily fix in the updated manuscript. Our findings and analysis remain correct and valuable to the community. We hope you can take a closer look after we provide these explanations. Again, we appreciate your feedback and believe these clarifications will enhance the clarity and coherence of our findings. 

**Please let is know if you have any concerns that need our clarification!**
The authors may at least need to clarify the discussion about Figure 5, 6 for the consistency between the proposed analysis and the reported experimental results. 

For example, #313 states 'LVLMs tend to hallucinate objects more frequently when they are positioned away from the center', which means the object centrality (normalized distance d between object and image center, as introduced in #273) positively correlated with the hallucination rate. While the Figure 5 (d) shows the yellow distribution (i.e. the hallucination distribution as mentioned in the author rebuttal) center is left to the green distribution.

Overall, the author response does not address my concern about the rigor of in this work from a scientific research view.