Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 04:49:44

Abstract: Large Vision-Language Models (LVLMs) exhibit remarkable capabilities but struggle with "hallucinations"—inconsistencies between images and their descriptions. Previous hallucination evaluation studies on LVLMs have identified hallucinations in terms of objects, attributes, and relations but overlooked complex hallucinations that create an entire narrative around a fictional entity. In this paper, we introduce a refined taxonomy of hallucinations, featuring a new category: Event Hallucination. We then utilize advanced LLMs to generate and filter fine-grained hallucinatory data consisting of various types of hallucinations, with a particular focus on event hallucinations, laying the groundwork for integrating discriminative and generative evaluation methods within our universal evaluation framework. The proposed benchmark distinctively assesses LVLMs' ability to tackle a broad spectrum of hallucinations, making it a reliable and comprehensive tool for gauging LVLMs' efficacy in handling hallucinations. We will release our code and data.

Main paper: 
 introduction: 
 Large Language Models (LLMs) such as GPT-4 [30] , LLaMA [40] , and LLaMA2 [41] have markedly enhanced capabilities in natural language understanding (NLU) and generation (NLG). Building on these advancements, recent Large Vision-Language Models (LVLMs) have shown increased proficiency in handling both textual and visual information, sparking significant interest among researchers. [2, 6, 9, 18, 24, 26, 45, 47] .Despite the promising developments in LVLMs, they broadly face the pivotal obstacle of hallucination, which refers to the discrepancy between factual content in images and the associated generated textual descriptions. As hallucination poses significant concerns for the LVLMs' reliability and robustness [12, 20, 23, 28, 39, 42, 46] , researchers have devised strategies for hallucinations evaluation to bolster the practical deployment of LVLMs, including discriminative and generative methods. The former method directly prompts candidate LVLMs to determine the presence of a particular hallucination, whereas the latter assesses the text produced by these candidate LVLMs.Event Hallucination: "Some zebras are seen in the wild, playfully interacting with each other, while a bird gracefully flies over them. At the same time, a kangaroo is hopping near the zebras, seemingly foraging " Attribute Hallucination: "Some zebras chillin in the wild with a black bird flying over. " Relation Hallucination: "Some zebras chillin in the wild with a bird standing on one's back." Object Hallucination: "Some zebras and elephents chillin in the wild with a bird flying over." Prior research [13, 17, 28] has delineated vision-language mismatches as issues of non-existent objects, incorrect object attributes, or inaccurate object relations, yet it does not encompass the full spectrum of hallucinations observed in LVLMs. For example, as depicted in Figure 1 , LVLM outputs can exhibit more intricate hallucinations, such as "At the same time, a kangaroo is hopping near the zebras, seemingly foraging." This type of hallucination invents a fictional target and weaves an entire narrative around it, including its attributes, relationships, and actions. We categorize these intricate narratives as event hallucinations. Our preliminary experiments indicate that the occurrence of event hallucinations significantly escalates as the output length of LVLMs increases, underscoring its significance as a hallucination phenomenon that warrants attention. However, there is an absence of fine-grained hallucination evaluation benchmarks for LVLMs that comprehensively address the various types of hallucinations-such as artificial objects, relationships, attributes, and events-while also accommodating both discriminative and generative evaluation methods.To this end, we propose a universal, fine-grained hallucination evaluation framework for LVLMs. This framework comprehensively evaluates a broad spectrum of hallucination types, encompassing objects, relationships, attributes, and notably, events with discriminative and generative evaluation methodologies. Specifically, we first develop an automatic annotation pipeline for fine-grained hallucinations, which leverages the sophisticated capabilities of GPT4 to generate and filter hallucinatory data. This pipeline then serves as a solid foundation for unifying discriminative and generative evaluation methodologies in our framework:• For the discriminative evaluation, we construct a dataset that features image captions with hallucinations generated through our pipeline. • For the generative evaluation, our pipeline facilitates the creation of a large-scale hallucinatory dataset. This dataset serves to fine-tune an LVLM into a specialized evaluator, Hal-Evaluator. This evaluator assesses LVLM-generated descriptions and associated images, identifying various hallucination types without needing reference captions.✓ × ✓ × × × × × × × NOPE [29] ✓ × ✓ × × × × × × × CIEM [14] ✓ × ✓ × × × × × × × M-HalDetect [13] × ✓ × × × × ✓ ✓ ✓ × GAVIE [23] × ✓ × × × × ✓ ✓ × × FAITHScore [17] × ✓ × × × × ✓ ✓ ✓ × HaELM [44] × ✓ × × × - - - - × MMHal-Bench [38] × ✓ × × × - - - - × AMBER [43] ✓ ✓ ✓ ✓ ✓ × ✓ × × × Hal-Eval ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓We conduct thorough experiments and analysis with six leading LLMs within our framework, assessing their performance in terms of hallucination under both discriminative and generative paradigms.Our key findings are:• The existing three categories of hallucinations (object, attribute, relation) overlook the existence of event-type hallucinations and are, therefore, insufficient to encompass all types of hallucinations. • Utilizing Chain-of-Thought (COT) significantly helps models minimize hallucinations during discriminative evaluations, particularly those involving relationships and events. • The incidence of hallucinations, especially event hallucinations, increases with the length of the output. Length control becomes a crucial aspect of generative evaluations, affecting comparative performance trends among LVLMs under varied output lengths. • The suitability of evaluation methodology varies according to the type of hallucinations. Using discriminative and generative evaluations together gives a fuller view of tendencies to LVLM hallucination. • The hallucinatory samples used to train our evaluator also serve as effective supervised fine-tuning data for LVLMs, contributing to reducing hallucinations and enhancing their benchmark performance.In summary, we introduce a novel hallucination category (event hallucination) of LVLMs, a universal and fine-grained evaluation framework for LVLMs that spans various hallucination types and unifies discriminative and generative approaches (as shown in Table 1 ), along with some groundbreaking insights to guide future research on vision-language hallucination. covers the spectrum of hallucinations present in LVLMs. For instance, as shown in the left part of Figure 8 , the outputs of LVLMs include more complex hallucinations: " to chase after a squirrel that has invaded its playtime ." We refer to these complex hallucinations as event hallucinations. To further clarify the concept of different hallucinations, we provide strict definitions for four different types of hallucinations in this paper: Object hallucination: The LVLM inaccurately describes an object that does not exist in the image. This could be a misidentification, where the model correctly detects an object's presence but incorrectly labels it, or an additional non-existent object, where the model asserts the presence of an object that does not exist.The LVLM correctly identifies an object that exists in the image, but inaccurately describes the attributes of that object. Attributes could include color, size, shape, position, or any other characteristic that defines the object.The LVLM incorrectly describes the relationship between two or more objects that clearly exists in the image. This could involve misrepresenting spatial relationships (e.g., describing an object as being on top of another when it's actually beside it), functional relationships (e.g., stating that a person is riding a bicycle when they are standing next to it), or other types of interactions or connections between objects.The LVLM not only describes a non-existent target but also constructs complete events around the non-existent target, including its attributes, relations, and actions. This type of hallucination involves a complex interplay of objects, attributes, and relations and often forms a narrative or sequence of actions that does not align with the actual content of the image.Event Hallucination Evaluation: Caption: "a dog carrying a frisbee in its mouth, running excitedly on a grass lawn to chase after a squirrel that has invaded its playtime. " Prompt: Is the content of the image matching the caption description?Attribute Hallucination Evaluation: Caption: "A dog carrying a frisbee in its mouth running on a withered grass lawn. " Prompt: Is the content of the image matching the caption description?Relation Hallucination Evaluation: Caption: "A dog carrying a frisbee in its mouth flying over a grass lawn. " Prompt: Is the content of the image matching the caption description?Object Hallucination Evaluation: Caption: "A cat carrying a Frisbee in its mouth running on a grass lawn. " Prompt: Is the content of the image matching the caption description?Please descript this image in detials.The image features a brown dog flying over a grassy field and a small stream, with a black frisbee in its mouth to chase after a squirrel that has invaded its playtime.There are four kinds of hallucinations in the response of the LVLM:(1) Object Hallucination: small stream, (2) Relation Hallucination : flying over (3) Attribute Hallucination: a black frisbee.(4) Event Hallucination: chase after a squirrel that has invaded its playtime.Here is a revised reponse: " The image features a brown dog running through a grassy field, holding a purple frisbee in its mouth." Building upon these definitions, we further investigate the proportion of different types of hallucinations present within the output of LVLMs. As depicted in Figure 2 , we collected 5,000 image-caption pairs from COCO [22] . We had them described by mPLUG-owl [45] and LLaVA [26] , respectively. Subsequently, we provided both the ground truth image descriptions and the model-generated descriptions to GPT-4 [30] , prompting it to inspect whether these descriptions encompassed hallucinations and to categorize them based on Object, Attribute, Relationship, and Event hallucinations (For more details about the experiment, please refer to our Supplemental Material D.1.). We tabulated the proportions of different types of hallucinations at varying description lengths. As Figure 2 demonstrates, we noted a significant increase in the share of event hallucinations as the length of the description extends. This experimental observation substantiates our finding: The existing three categories of hallucinations (object, attribute, relation) overlook the existence of event-type hallucinations. They are, therefore, insufficient to encompass all types of hallucinations. 
method: 
 We proposed a comprehensive and universal hallucination evaluation benchmark, Hal-Eval. As shown in Figure 3 , Hal-Eval includes both Discriminative Evaluation and Generative Evaluation and can effectively evaluate different types of hallucinations. In the following subsection, we will initially introduce a fine-grained hallucination annotation pipeline. This pipeline is employed to construct both the evaluation dataset for Hal-Eval and a large-scale hallucination detection dataset known as Hal-Data. Hal-Data, in turn, serves as the training data for the Hal-evaluator, an evaluation model designed to perform generative hallucination evaluations. Subsequently, we will delve into an in-depth discussion of the discriminative evaluation process. Finally, we will elucidate the training procedure for the Hal-evaluator and outline how generative evaluations are conducted using this model.The existing multimodal hallucination research lacks large-scale datasets with fine-grained annotations specific to hallucinations. To address this issue, we design an automatic fine-grained hallucination annotation pipeline featuring annotations for four hallucination types and specific hallucination content. Data Annotation. We annotated image-text paired data based on GPT-4. We initially established a rigorous definition for various types of hallucinations as we mentioned in Section 2. Building upon this groundwork, we engaged GPT-4 to rephrase the collated image-text pairs in line with the diverse classifications of hallucinations. This step involved injecting distinctive hallucinatory elements into the original captions. The outcome of this procedure was a collection of image descriptions enriched with specified hallucination categories. Moreover, we delegated to GPT-4 the responsibility of annotating the position of specific hallucinatory content in the image description. Please refer to the supplemental material C.2 for more details. Data Filtering. Following the initial annotation phase, we identified that the quality of the labeled data remained unsatisfactory. Random sampling revealed that approximately 30% of the annotated dataset still harbored noise that failed to meet our stringent labeling criteria. Hence, we proceeded to craft a tailored prompt to commission GPT-4 for the task of purging and refining the noisy annotations, a process thoroughly outlined in the supplemental material C.3. Subsequent to GPT-4's meticulous cleanup operation, a manual verification process ascertained that over 97% of the data accorded with the stipulated annotation standards.Previous benchmarks such as POPE [20] predominantly utilized manually annotated datasets like COCO [22] . However, the COCO dataset is frequently employed to construct general benchmarks such as VQA v2 and visual grounding. These benchmarks are often used for instruction finetuning of LVLM models, which results in evaluation data being in the same domain as the models' finetuning data. This overlap hinders a true assessment of the models' zero-shot hallucination capabilities. To address this issue, we divided our evaluation dataset into two parts: in-domain data from the COCO 2014 validation and COCO 2017 test sets and an out-of-domain dataset randomly sampled from web-based data like CC [5] , SBU [32] , LAION [36] . Data Annotation: We used the automatic annotation process detailed in Section A to annotate both in-domain and out-of-domain evaluation datasets, resulting in 5,000 detailed annotations each. [20] , we also use Accuracy, Precision, Recall, F1 score, and "Yes" ratio as the evaluation metrics. Here, Accuracy represents the number of correctly answered cases, while Precision and Recall, respectively, indicate the proportion of correctly answered questions with responses "Yes" or "No." The F1 score integrates the outcomes of Precision and Recall, which we select as the primary evaluation metric. The "Yes ratio" serves as a reference for analyzing model behaviors.Regarding generative evaluation, current evaluation methods either rely on proprietary models that require subscription fees, such as GPT-4, or depend on fine-tuned large language models (LLMs) that necessitate additional ground truth annotations-a process that is prohibitively expensive. This significantly restricts the scalability of evaluating models. In response, we propose Hal-Evaluator, a reference-free, open-source evaluation model designed specifically to detect hallucinatory content. Hal-Evaluator is fine-tuned on LLaVA 1.5 [24] , which is also an LVLM; as illustrated in Figure 3 , it takes as input the description of an image provided by the LVLMs under evaluation, as well as the corresponding image itself. Following this, it evaluates whether the description contains hallucinations. If hallucinations are detected, they provide the specific content and categorization of the hallucinations. Ultimately, it can even modify the hallucinated information in the description to output an accurate depiction. In this way, our generative evaluation eliminates the need for additional reference annotation, enabling hallucination evaluation based solely on the content of the image.To train the Hal-Evaluator, which is capable of effectively identifying different types of hallucinations, a large-scale, fine-grained hallucinatory image-text dataset is necessary as it facilitates the refinement of training for Hal-Evaluator intended to detect and correct hallucinatory content. However, no dataset of this scale with detailed annotations currently exists. Therefore, we initially constructed Hal-Data, the first large-scale, fine-grained dataset with hallucination annotations, based on the pipeline mentioned in Subsection A.This dataset, referred to as Hal-Data, was generated using an automatic hallucination annotation pipeline and comprises 2 million instances. Hal-Data is split into two parts: Hal-Data 130k, which includes 130,000 GPT-4 annotated and curated image-text pairs, each consisting of an image, a valid image caption, and a hallucination description; and Hal-Data 2M, which includes 2 million image-text pairs created by our caption model trained on the 130,000 high-quality captions from Hal-Data 130k. Below, we detail the creation process for Hal-Data. Data Collection for Hal-Data 130k: To ensure diversity and comprehensiveness, we initially compiled about 200,000 images from various sources, including 80,000 in-domain COCO dataset images [22] , 80,000 web images from sources like CC [5] , SBU [32] , and LAION [36] , and 40,000 image-text datasets from ShareGPT4-V [7] to match the style of LVLM outputs. We then used AFHA to annotate this data, resulting in a final collection of 130,000 meticulously annotated GPT4 instances, named Hal-Data 130k. Generation for Hal-Data 2M: We further selected a subset of 2 million image-caption pairs from current public datasets (see Appendix B.1 for more details) and constructed a large-scale hallucination dataset named Hal-Data 2M. Due to the high cost of using GPT-4, we fine-tuned the open-source large-scale language model LLaMA2 13B [40] on Hal-Data 130k and employed it to modify the image captions of Hal-Data 2M by introducing different types of hallucinations and annotating them.Based on Hal-Data, we fine-tuned LLaVA 1.5 13B [24] , recent SOTA LVLM, with 2M instruction data specifically designed for detecting and correcting hallucinations in image captions, leading to the development of Hal-Evaluator. (For more details, please refer to our supplemental material D.1.)As illustrated in Figure 3 , the input to Hal-Evaluator consists of two parts: an image and the corresponding textual description by the candidate LVLM to be evaluated. We prompt Hal-Evaluator to first determine if the text description contains hallucinations based on the image content. If hallucinations are detected, Hal-Evaluator will identify the type of hallucination and its content. Ultimately, Hal-Evaluator can also correct the hallucinatory content in the image description, providing a revised depiction of the image.To evaluate the generative hallucination of LVLMs, we prompt them to describe images from both an indomain 5K dataset and an out-of-domain 5K dataset mentioned in Subsection 3.2.1 with short length and longer length. These descriptions, coupled with the respective images, are then fed into the pre-trained Hal-Evaluator. Our procedure involves prompting Hal-Evaluator to evaluate the existence and category of any hallucinatory content. Accuracy serves as the principal metric for our evaluation, which measures the proportion of outputs correctly identified as free from hallucinations. Suppose the number of all outputs is 𝑁 , the outputs that contain hallucinations are 𝑁 ℎ , and the accuracy is calculated as 𝐴 = 𝑁 -𝑁 ℎ 𝑁 . Moreover, we track the probability of various types of hallucinations encapsulated in the hallucination ratio. For instance, the number of outputs containing the object hallucination is 𝑁 𝑜 ℎ , and the object ratio 𝑟 𝑜 is calculated as𝑟 𝑂 = 𝑁 𝑜 ℎ 𝑁 ℎ . 
experiments: 
 Hal-Eval is divided into two distinct segments: Discriminative Evaluation and Generative Evaluation. We have opted to assess five widely utilized open-source LVLMs: MiniGPT-4 [47] , InstructBLIP [9] , mPLUG-owl [45] , LLaVA [26] , LLaVA1.5 [24] and one close-source LVLM: GPT4-V [31] .As shown in Table 2 , we evaluate the performance of five models on different types of hallucinations following the method outlined in Subsection 3. First, LLaVA1.5 and LLaVA exhibit a more pronounced predilection for hallucinations when tested against out-of-domain datasets as opposed to in-domain datasets. This trend could possibly be ascribed to the prevalent incorporation of COCO [22] during the instruction tuning phase of the models. Moreover, we noticed that the results derived from the POPE [20] metric indicate a significant tendency among most models to favor "yes" responses. In contrast, within our discriminative evaluations, such a penchant is exclusively noted in the InstructBLIP. This distinction serves to underscore that Hal-eval can effectively avoid the bias of the model towards answering "yes". In the end, our findings indicate that, with the exception of GPT-4, the performance of the currently accessible open-source LVLMs in discriminative evaluations is subpar. These models face challenges in accurately discerning and interpreting hallucinatory content within image descriptions.Our proposed evaluation dataset comprises 5,000 indomain and 5,000 out-of-domain images, which we annotated based on the AFHA framework. To verify the accuracy of the annotations, we randomly sampled 100 cases each from both the in-domain and out-of-domain data for manual validation (Please refer to supplemental material C for more details). We found that after GPT-4's annotation and filtering process, the annotation accuracy rate in the COCO dataset reached 98%. Meanwhile, the annotation accuracy for the out-of-domain dataset stood at 97%. This high level of accuracy in both datasets underscores the effectiveness of our annotation process.For discriminative evaluation, we employed a chain of thought (COT) approach to systematically evaluate whether the LVLM matches the content of images with their respective captions (Refer to Appendix C.4 for more details). As shown in Figure 4 , we observed a significant reduction in discriminative hallucinations on both in-domain and out-of-domain datasets after employing COT to LLaVA 1.5. Based on the above experiments, we have the following finding:Finding 1: Utilizing COT is particularly effective in reducing discriminative hallucinations for LVLMs, especially for those related to relationships and events.We suggest that the increased effectiveness of COT with relationship and event type hallucinations is due to their intrinsic reliance on contextual understanding and inference-making. 3 , our investigation has revealed that contemporary models continue to have a significant inclination toward producing hallucinations. MiniGPT-4 and Instruct-BLIP, displayed robust in-domain accuracy, with the latter achieving 89.1% accuracy when the average output length was approximately 10 tokens. Both mPLUG-owl and LLaVA showed moderate performance across all evaluated metrics, whether tested on in-domain or out-of-domain data. GPT-4V achieved the best result, but we also observed a notable decline in accuracy with increasing output length, accompanied by a significant rise in the proportion of event-type hallucinations. Furthermore, we found that when generating long responses, all models became more prone to producing hallucination content, with the incidence of event hallucinations rising markedly. evaluation and human judgment, we conduct the following experiments. We first select the test dataset from COCO 2014 [22] for human evaluation. This test set comprises 50 images. Each image is supplemented with five reference captions and object labels provided by the COCO dataset. We selected three LVLMs -LLaVA [26] , mPLUG-owl [45] , and instructBLIP [9] -to describe the content of the test set, and we sought the annotation and evaluation from 15 human annotators to evaluate the presence of hallucinations in these data. We made a comparison among five benchmarks: ROUGE-L [21] , BLEU-4 [33] , CHAIR [35] , Hal-EML [44] , GPT4-V along with the module of Hal-Eval -Hal-Eval-Generative. Table 4 delineates the correlation between various evaluation metrics and human judgment regarding LVLM faithfulness, gauged using Pearson's 𝑟 , Spearman's 𝜌, and Kendall's 𝜏. Our generative evaluation distinctly stands out and demonstrates a robust positive correlation, underlining the superior alignment with human perceptions.To further verify the effectiveness of Hal-Evaluator for hallucination detection, we evaluated Hal-Evaluator and GPT-4V (as candidate LVLMs instead of evaluator here) based on the discriminative evaluation of Hal-Eval scripted in subsection 3.2.2, evaluating the detection of different types of hallucinations in 5K coco data. The results disclosed that Hal-Evaluator outperforms GPT-4V in hallucination detection ability, as shown in the Sub-Figure 5 (a).We tasked GPT-4V and Hal-Evaluator with detecting the hallucination content in image descriptions of the evaluation dataset of Hal-Eval. We evaluated the overlap between the hallucination content as identified by GPT-4V and Hal-Evaluator and the annotated hallucination content that exists in the descriptions. The overlap was quantified using the ROUGE-L score, as shown in Sub-figure 5 (b ). The experimental results show that both GPT-4V and Hal-Evaluator can accurately identify the majority of hallucinated content in image descriptions for the first three types of hallucinations (object, attribute, relation). However, when it comes to event hallucinations, GPT-4V struggles to pinpoint the hallucinated content accurately, while Hal-Evaluator demonstrates a reliable identifying capability. Considering that GPT-4V is the current top-performing LVLM yet its difficulty in accurately identifying event hallucinations, this underscores the intrinsic complexity of event hallucinations and Hal-Evaluator's reliability in detecting event-type hallucinations.We investigate the potential correlation between the output length of LVLMs and the occurrence of generative hallucinations. We conducted the following experiments on LLaVA 1.5 and InstructBLIP. Experiments were carried out on LLaVA 1.5 and InstructBLIP wherein both models were prompted to describe the content of images with outputs of varying token lengths, using the Hal-Evaluator to detect and analyze the outputs at each length. The proportions of different types of hallucinations and the accuracy of outputs without hallucinations are visualized in Figure 6 . Combining the results in subsection 4.2.1, the following findings were observed:Finding 2: The incidence of hallucinations, especially event hallucinations, increases with the length of the output. Length control becomes a crucial aspect of generative evaluations, affecting comparative performance trends among LVLMs under varied output lengths.As shown in Table 5 , for event hallucination, most LVLMs perform better in discriminative evaluation compared to other hallucination types. For example, we calculated the average F1 scores for discriminative evaluations across all models for different types of hallucinations, and the average F1 score for event types was 54.3, which is higher than that for other hallucinations. This suggests that models are more resistant to event-type hallucinations. However, when evaluating the models using a generative evaluation approach, we observed a higher frequency of event-type hallucinations in longer output sequences, contradicting the results from discriminative evaluations. We believe this is mainly because event-type hallucinations often contain more complex and rich information inconsistent with the image content, making them easier for models to handle in discriminative evaluations. However, this does not accurately reflect the models' ability to avoid event-type hallucinations effectively. Herefore, generative evaluation is a more effective method for assessing event-type hallucinations. For other types of hallucinations, such as objects, attributes, and relationships, research [20] [29] [14] indicates that discriminative evaluations are sufficiently effective in reflecting whether models tend to such hallucinations. In summary, we have the following findings:Finding 3: The suitability of evaluation methodology varies according to the type of hallucinations. Using both discriminative and generative evaluations together gives a fuller view of tendencies to LVLM hallucination.To validate whether Hal-Data can assist LVLM in eliminating hallucinations and improving performance through instructional fine-tuning, we conducted the following experiment: we constructed instructional data from Hal-Data 130K, and after combining this instructional data with that of LLaVA1. Hal-VL on Hal-Eval and achieve the best performance. In addition to evaluations on Hal-Eval, as shown in Table 7 , we also evaluate Hal-VL with other hallucination benchmark like MMHal-Bench [37] .The experiments demonstrate that Hal-VL significantly outperforms LLaVA 1.5. These results indicate Hal-Data effectively aid models in mitigating hallucinations.We also evaluate Hal-VL on multiple general benchmark such as VQA [11] , GQA [16] and so on. As indicated in the Table 8 , we found that Hal-VL achieved significant advantages on the vast majority of these benchmarks which suggesting Hal-Data can not only mitigating hallucinations but also enhances the overall model performance. These results support that:Finding 4: The hallucinatory samples used to train our evaluator also serve as effective supervised fine-tuning data for LVLMs, contributing to the reduction of hallucinations and enhancement of their benchmark performance.As shown in Figure 7 , we tokenized the annotations of hallucination content in Hal-Data 130K, and visualized them via a word cloud. The results shown in the word cloud align with the definitions of the hallucination content. For instance, the words in the word cloud for object hallucinations are distinguishable nouns, and for attribute hallucinations, they largely consist of features such as colors. This supports the reliability and validity of our labeled dataset. 
related work: 
 Large Vision Language Model: Based on LLMs, there are three principal approaches to constructing LVLMs, all demonstrating potential for robust zero-shot generalization in the vision-language field. For instance, Flamingo [1] utilizes a fixed vision encoder paired with a sizable language model featuring gated cross-attention mechanisms for cross-modality matching. Meanwhile, PaLM-E [10] incorporates visual features via linear layers directly into the pretrained PaLM [8] framework, which delivers strong performance across a spectrum of practical applications. This integration strategy is widely employed by models like LLaVA [27] , Shikra [6] , and others. However, generating long visual sequences remains a notable constraint of this technique. To mitigate this, BLIP-2 [19] , inspired by DETR [4] , conceived a Q-former that effectively condenses the length of visual feature sequences. This concept has since been reflected in Kosmos-1 [15] , mPLUG-Owl [45] , and MiniGPT-4 [48] .Hallucination Evaluation: LVLM hallucination benchmarks specifically aim at non-hallucinatory generation or hallucination discernment. These benchmarks are classified according to the type of evaluation approach they follow: Discriminative (Dis) or Generative (Gen). Discriminative Benchmark: POPE [20] , NOPE [29] , and CIEM [14] are examples of discriminative benchmarks. Each of these benchmarks exclusively directs attention towards object hallucinations and utilizes accuracy as their primary evaluation metric. The metric is calculated by querying the presence of objects within images and comparing the model's responses with the ground truth. Generative Benchmarks: modern research predominantly accentuates generative benchmarks over discriminative ones. Whilst discriminative benchmarks focus mainly on object-level hallucinations, generative benchmarks widen their scope to encompass a more extensive range of hallucinations, such as attribute and relation hallucinations [13, 17, 23, 38, 44] . AMBER [43] emerges as a holistic benchmark that concludes both generative and discriminative tasks. 
conclusion: 
 We introduce a new category, Event Hallucination, into the study of LVLMs and develop a unique evaluation framework leveraging advanced LLMs for data analysis. This approach enhances the understanding and mitigation of hallucinations in LVLMs, marking a significant step forward in assessing and improving model performance.  
 
 Appendix:  


Decision: The paper introduces a type of hallucination, event hallucination, and proposes a novel approach to addressing hallucinations in LVLMs. This paper addresses a highly practical issue, and the authors have devised a comprehensive solution, achieving excellent performance. All reviewers have given positive scores. I believe this paper will provide significant insights to the field. Therefore, I recommend this paper for an oral presentation at MM.

Reviews: review: paper_summary: paper_summary: The paper introduces a type of hallucination, event hallucination, and proposes a novel approach to addressing hallucinations in LVLMs, providing a more reliable toolkit for assessing and addressing the hallucinations of LVLMs.
strength_weakness: strength_weakness: 1. The introduction of event hallucinations and the evaluation benchmark addresses a previously overlooked aspect of LVLM hallucinations.
2. The paper provides a detailed and systematic approach to generating and filtering hallucination data, constructing datasets, and evaluating LVLMs, including both discriminative and generative LVLMs.
3. The paper conducts extensive experiments with leading LVLMs, providing a thorough analysis of their performance in handling different types of hallucinations.
limitations: limitations: 1. In Figure 2, more LVLMs can be tested to illustrate the prevalence of event hallucinations and their characteristic of increasing with token lengths.
2. In Table 2, some models, such as mPLUG-Owl, MiniGPT-4, and InstructBLIP, have similar performance on both in-domain and out-of-domain benchmarks. Out-of-domain data is sampled from web-based datasets like CC, SBU, and LAION. Many LVLMs have already been trained on these datasets, and it is better to use completely unseen data as an out-of-domain benchmark.
3. In Table 4, the three metrics of Hal-Eval-Gen are all less than 0.5, which may suggest that the method does not strongly align with human judgment.
4. What LVLM is used in Figure 1?
5. Some typos:
  1. Figure 4, “Extrinsic” should be “Event”?
  2. Figure 6 caption, “LLaMA1.5” should be “LLaVA1.5”?
paper_summary: paper_summary: The paper introduces a new type of hallucination in Large Vision-Language Models (LVLMs): Event Hallucinations. To better evaluate Event Hallucinations in LVLMs, the authors propose a novel and universal fine-grained hallucination evaluation framework that unifies discriminative and    generative evaluations. The framework details the applicability of both evaluation methods. The authors developed an automatic annotation pipeline to generate fine-grained hallucination data, addressing the gap in large-scale datasets. Leveraging LLaVA 1.5 and the new data, they fine-tuned a high-performance model, Hal-Evaluator, for generative evaluation. Additionally, the authors fine tuned  the Hal-VL model using the new large-scale hallucination data to better mitigate hallucinations and improve model performance.
strength_weakness: strength_weakness: 1. Category Innovation: The paper pioneeringly introduces a new type of hallucination: Event Hallucinations. The experiments demonstrate that Event Hallucinations constitute a significant proportion of all known hallucination types. In Experiment 4.2.2, it was found that Event Hallucinations become more severe as the model's generated output length increases. Additionally, Experiment 4.1.2 shows that COT can reducediscriminative hallucinations in LVLMs, improving the effectiveness in handling relational and event hallucinations.
2. Data Innovation: The paper establishes a comprehensive automatic fine-grained hallucination  annotation process, enriching the data scale for both discriminative and generative evaluations. This fills a gap in the study of LVLM hallucinations. Experiment 4.4 proves that these new large- scale hallucination annotations can be used to fine-tune LVLM models to reduce hallucinations.
3. Evaluation Framework: The Hal-Eval evaluation framework includes both discriminative and generative evaluation results. For discriminative evaluation, the authors created new data using   AFHA and replaced the questions with more natural ones. For generative evaluation, the authors fine-tuned the Hal-Evaluator using the generated large-scale data, which provides more accurate and detailed assessments of LVLM-generated hallucinations, pinpointing the exact locations of     hallucinations in the text.
limitations: limitations: 1. Data Missing: In the out  of  domain_eval_5k and in  of  domain_eval_5k datasets, the new    hallucination descriptions and hallucination type annotations are evident. However, in the Hal- Data 130K SFT dataset, only human inputs and GPT's simple responses about whether the    captions accurately describe the images are visible, without the newly annotated hallucination types and descriptions.

2. Insufficient Experiment Details: The experimental process in Experiment 4.1's discriminative evaluation is not detailed enough. The evaluation metrics are too many and complex, making them difficult to understand, and the authors did not provide a good explanation of these metrics. The experimental process is almost not introduced, leaving questions about how the evaluation   was conducted, whether the descriptions with this type of hallucination also included correct descriptions, and the proportions of these descriptions. These unclear experimental processes make the results confusing.

3. Lack of Confidence in Experimental Data: In Experiment 4.2.2, the authors directly presented the correlation between different benchmark results and human evaluation results without specifying the human evaluation results and the results of LVLM tested based on these benchmarks. The authors did not provide the specific calculation process, raising doubts about the data's authenticity and potentially reducing the experiment's persuasiveness.
paper_summary: paper_summary: The paper introduces a new type of hallucination in Large Vision-Language Models (LVLMs) called Event Hallucination. In particular, the authors propose the Automatic Fine-grained Hallucination Annotation (AFHA) pipeline using GPT-4, constructing the Hal-Data as training data in their Hal-Eval benchmark. These collected hallucinatory samples could be used to fine-tune LVLMs, contributing to reducing hallucinations and better results on discriminative and generative hallucination evaluation methods.
strength_weakness: strength_weakness: * The concept of Event Hallucination involves the intricate integration of vision and language, which is crucial when generating longer text outputs. 
* The idea of Automatic Fine-grained Hallucination Annotation (AFHA) pipeline is interesting. 
* The authors have conducted comprehensive analysis of the need for Event Hallucination, particularly focusing on the length of the output.
limitations: limitations: * In line 448, the authors mention that they fine-tune LLaVA 1.5 13B with their Hal-Data. However, the performance in Table 7 compare to the approaches using LLaVA 7B with other instruction tuning dataset? How can it be a fair comparison? The authors should carefully describe the model and the instruction data difference compare to the approaches in other hallucination benchmark. 
* In Table 7, is there any reason why Hal-VL gets worse performance on Attribute, Comparison, Counting, Relation, and Holistic question type in MMHal-Bench compare to LLaVA-RLHF 7B? 
* In Table 8, what does the symbol ‡ mean? It indicates that the model has been trained on the General VQA dataset, but why is it still under the zero-shot setting?
* The experiments in Table 7 (other benchmark) and Table 8 (other task) could make better discussions.
paper_summary: paper_summary: The paper introduces a novel type of hallucination, termed event hallucination. It argues that prior multimodal hallucination detection methods are capable of recognizing object, attribute, and relation hallucinations but fall short in effectively detecting event hallucinations. To address this gap, an event hallucination dataset was constructed, and several contemporary large models were evaluated and fine-tuned for hallucination detection on this dataset. Furthermore, the study proposes two detection methodologies: generative and discriminative. The discriminative method can uniformly detect various types of hallucinations, while the generative method is also capable of providing the correct output.
strength_weakness: strength_weakness: A novel type of hallucination—event hallucination—is introduced, with a precise definition provided. The existence of this hallucination type in multimodal language models is also validated.
Two hallucination detection methods are proposed: discriminative and generative. The discriminative method employs a unified prompt template, enabling the detection of various types of hallucinations. The generative method involves instruction fine-tuning of LLaVA 1.5 on the proposed Hal-Data, effectively detecting different hallucination types and content while providing correct responses.
limitations: limitations: ● In Section 2, there is an error in the figure citation; Figure 8 does not exist.
● The increased likelihood of hallucinations during out-of-domain testing can be attributed to the inclusion of the COCO dataset during the instruction fine-tuning phase. However, in Section 3.2.1, the description of the out-of-domain dataset does not mention the COCO dataset.
● mPLUG-owl and MiniGPT-4 are relatively older models. For verifying the presence of hallucinations, it would be more appropriate to use the updated models, mPLUG-owl2 and MiniGPT-5.
● The relationships among object, attribute, relation, and event hallucinations have not been validated. For example, in the case of "small stream," where "stream" is identified as an object hallucination and "small" as an attribute hallucination, it is conceivable that "small stream" might also represent an event hallucination, despite being categorized as an object hallucination in Figure 3. Moreover, since event hallucinations encompass object, attribute, and relation hallucinations, it is plausible that the occurrence of any one of these hallucinations could trigger an event hallucination.
● Why the proportion of event hallucinations increases with output length, whereas the proportions of other types of hallucinations decrease In Figure 6.