Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs

Authors: peng  ding
jingyu  wu
jun  kuang
dan  ma
xuezhi  cao
xunliang  cai
shi  chen
jiajun  chen
shujian  huang
  hallu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-13 10:40:14

Abstract: Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on various visual-language understanding and generation tasks. However, MLLMs occasionally generate content inconsistent with the given images, which is known as "hallucination". Prior works primarily center on evaluating hallucination using standard, unperturbed benchmarks, which overlook the prevalent occurrence of perturbed inputs in real-world scenarios-such as image cropping or blurring-that are critical for a comprehensive assessment of MLLMs' hallucination. In this paper, to bridge this gap, we propose Hallu-PI, the first benchmark designed to evaluate Hallucination in MLLMs within Perturbed Inputs. Specifically, Hallu-PI consists of seven perturbed scenarios, containing 1,260 perturbed images from 11 object types. Each image is accompanied by detailed annotations, which include fine-grained hallucination types, such as existence, attribute, and relation. We equip these annotations with a rich set of questions, making Hallu-PI suitable for both discriminative and generative tasks. Extensive experiments on 12 mainstream MLLMs, such as GPT-4V and Gemini-Pro Vision, demonstrate that these models exhibit significant hallucinations on Hallu-PI, which is not observed in unperturbed scenarios. Furthermore, our research reveals a severe bias in MLLMs' ability to handle different types of hallucinations. We also design two baselines specifically for perturbed scenarios, namely Perturbed-Reminder and Perturbed-ICL. We hope that our study will bring researchers' attention to the limitations of MLLMs when dealing with perturbed inputs, and spur further investigations to address this issue. Our code and datasets are publicly available at https://github.com/NJUNLP/Hallu-PI.

Main paper: 
 introduction: 
 Multi-modal Large Language Models (MLLMs) have achieved significant progress in a range of practical applications, such as providing detailed descriptions for user-provided images (i.e., image captioning) [1, 29] and answering specific questions about input images (i.e., visual question answering) [24, 41] . However, these models occasionally exhibit a phenomenon known as "hallucination", where the generated content is inconsistent with the given images [16, 35] .There are three apples in the image. How many apples are there in the top-left image?There are two apples in the image.How many letters are there in the image?There are 26 letters in the image. How many letters are there in the image?There are 26 letters in the image, ranging from A to J.How many types of animals are there in the image?There is one type of animal in the image: elephants.In addition to the elephant, there are three other types of animals in the image. What are they?In addition to the elephant, there are three other types of animals in the image. They are giraffes, zebras, and antelopes.Figure 1 : Some examples of hallucinations in MLLMs with perturbed inputs (such as image concatenation, image cropping, and prompt misleading). Text highlighted in green and red represents correct and hallucinatory content, respectively.Previous works have sought to investigate the hallucinations in MLLMs by utilizing a large language model like GPT-4 [1] , or by employing humans as annotators [38, 40] . Alternatively, some studies focus on developing detection models to scrutinize the hallucinations exhibited by MLLMs. [14, 22] . More recently, [31] introduce AMBER, a LLM-free benchmark designed to examine MLLM hallucinations in both discriminative and generative tasks across dimensions like existence, attribute, and relation.Despite these efforts, existing researches primarily focus on conducting evaluations by sampling images from available image datasets, such as MSCOCO [14, 22, 28, 32, 37, 38] . However, in real-world scenarios, inputs fed to MLLMs frequently encounter a variety of perturbations (e.g., noise and cropping) [12] . Overlooking such perturbations could lead MLLMs to produce incorrect answers or judgments, potentially causing serious accidents in certain applications (e.g., medical diagnosis, industrial automation and autonomous driving) [17] . Figure . 2 illustrates the hallucinations of several MLLMs before and after image concatenation perturbation. The inconsistent performance trends indicate that relying solely on existing unperturbed benchmarks is insufficient for a comprehensive and precise evaluation of hallucinations in MLLMs.In order to bridge this gap, we introduce Hallu-PI, a benchmark designed to evaluate the Hallucination performance of MLLMs within Perturbed Inputs. Followed by [12, 15] , we first categorize the image perturbations into four types: noise, blur, weather, and digital. Additionally, we meticulously propose three distinct types of perturbations: image concatenation, image cropping, and prompt misleading. These perturbations are considered at both the image level and the prompt level. Annotators are instructed to carefully manipulate the perturbations and provide corresponding annotations. Evaluations of 12 mainstream MLLMs conducted on Hallu-PI reveal significant hallucinations of leading MLLMs (e.g., GPT-4V and Gemini-Pro Vision) when dealing with perturbed scenarios. Inconsistent performance trends show that relying solely on unperturbed benchmarks is insufficient for a complete and precise evaluation of hallucinations in MLLMs.To comprehensively understand the hallucination of MLLMs to perturbed inputs, we conduct a detailed analysis of the experimental results. We find that most models exhibit significant bias towards specific types of perturbations, particularly image concatenation, image cropping, and prompt misleading (see Figure . 1). Furthermore, to mitigate the hallucination of MLLMs in response to perturbed inputs, we draw inspiration from the defensive strategies adopted by text LLMs against jailbreak attacks [7, 34] and designed two baselines: Perturbed-Reminder and Perturbed-ICL. Experiments conducted on GPT-4V show that these strategies effectively reduce hallucinations. We hope our work can prompts MLLM researchers and developers to address hallucinations from perturbed inputs. ✗ ✗ ✗ ✗ ✗ Sample M-HalDetect [14] ✗ ✗ ✗ ✗ ✗ Sample HaELM [32] ✗ ✗ ✗ ✗ ✗ ✗ Sample Halle-Switch [38] ✗ ✗ ✗ ✗ ✗ ✗ Sample AMBER [31] ✗ ✗ Manual Hallu-PI (ours) ManualIn summary, the contributions of our work are as follows:• We construct Hallu-PI, the first freely available multi-modal hallucination benchmark with perturbed inputs. Hallu-PI encompasses 7 perturbed scenarios, a total of  
related work 2.1 multimodal large language models: 
 Multi-modal Large Language Models (MLLMs) are currently achieving significant improvements by combining the advanced capabilities of Large Language Models (LLMs) with visual processing [1, 2, 9, 21, 29] . These MLLMs show great potential in a variety of applications, such as visual question answering (VQA) [11] , image captioning [26] , and video understanding [19] . Representative MLLMs, such as CogVLM [33] , LLaVA1.5 [24] , InternLM-XComposer [39] , MiniGPT-4 [5] , mPLUG-Owl2 [36] , Qwen-VL [3] , and the latest GPT-4V [1] , and Google Gemini-Pro Vision [29] , have achieved impressive performance across various multi-modal tasks.While MLLMs have exhibited excellent performance on multimodal tasks, we are still facing the challenge that MLLMs often generate content unfaithful to the given images, which is called "hallucination" [6, 18, 20, 23, 30] .Currently, many researchers focus on evaluating the hallucination in MLLMs. LURE [40] and HallE-Switch [38] rely on human evaluations or GPT-4. While this method is relatively reliable, it is also expensive. HaELM [32] and FDPO [14] are based on hallucinatory detection models. However, the performance of these models is highly dependent on hallucinatory data and incurs substantial training costs. POPE [22] is based on object detection but is only applicable to discriminative tasks and evaluates existence-type hallucinations. Recently, [31] introduce AMBER, which assesses hallucinations across multiple dimensions, such as existence, attribute, and relation. Despite these efforts, they do not explore hallucinations in the perturbed scenarios commonly encountered in real-life situations. To bridge this gap, we propose Hallu-PI, the first benchmark designed to evaluate the hallucination of MLLMs with perturbed inputs. Table . 1 presents a detailed comparison between Hallu-PI and other hallucination benchmarks.To simulate real-world perturbation scenarios, previous works adopt various perturbation strategies such as ImageNet-C [15] and Stylize-ImageNet [12, 27, 28] . The perturbations are grouped into five primary categories: noise, blur, weather, digital, and stylize. Specifically, these can be further subdivided into the following 17 image perturbation techniques: (1) Noise: Adding noise to the images, such as gaussian noise, shot noise, impulse noise, and speckle noise. (2) Blur: Blurring the images, including defocus blur, frosted glass blur, motion blur, and zoom blur. (3) Weather: Adding environmental effects such as snow, frost, fog, and brightness adjustments. (4) Digital: Manipulating images through contrast enhancement, elastic transformation, pixelation, and JPEG compression. and ( 5) Stylize: Applying artistic styles and transformations to images.Compared to existing benchmarks that only consider hallucination assessment in unperturbed scenarios, Hallu-PI further takes into account perturbations that frequently occur in real-world applications. Therefore, it serves as a complement to existing benchmarks and provides a more comprehensive and accurate evaluation of hallucinations in MLLMs. 
hallu-pi benchmark: 
 In this section, we introduce the process of constructing our Hallu-PI benchmark which primarily encompasses three aspects: (1) Image Collection, (2) Image Perturbation and Annotation, and (3) Designing Prompt Query Templates.To ensure the diversity of the dataset, we identify 11 different object types and require annotators to collect images for each category. In the image selection process, we primarily consider (1) image copyright and (2) image quality. We provide annotators with several websites offering free copyright images and instruct them to search for images using specific object keywords. Annotators are asked to select images where the object is complete and the image is of high quality for downloading. Following previous work [28] , we first consider four primary types of perturbation: noise, blur, weather, and digital. Stylize is not included because the stylized images are too blurred to recognize the objects and attributes within them. To construct a more comprehensive set of perturbation scenarios, we meticulously propose three additional perturbations: image concatenation, image cropping, and prompt misleading. The first two are considered because they are commonly used by users in real life to edit their images, while prompt misleading ensures that Hallu-PI can evaluate hallucinations at both the image level and the prompt level. For noise, blur, weather, and digital perturbations, we reuse the code from [28] to generate the perturbed images. For image concatenation, we require our well-trained annotators to combine every four individual images previously collected into a single four-grid image, ensuring that the objects in the concatenated image are complete. For image cropping, we primarily focus on images containing English letters. Annotators are instructed to crop these images and provide corresponding questions and answers for both the original and cropped images. For prompt misleading, annotators need to select an image and provide a prompt that could potentially induce hallucinations. To ensure a more comprehensive evaluation of hallucinations, we design both generative and discriminative prompt templates for each perturbation scenario. For the perturbations such as noise, blur, weather, digital, and image concatenation, we pose questions regarding each specific annotation field. For instance, for the image concatenation perturbation, the generative prompt for the "Existence" field before perturbation is: "Please describe the existing objects in the image. " After perturbation, the prompt becomes: "Please describe the existing objects in the top-left image." with "existing objects" and "top-left" being flexible and variable. For the design of discriminative prompts, we consider that merely calculating accuracy might be insufficient. Following previous work [11] , we design Yes_Q and No_Q, representing questions with the single-word answers "Yes" or "No," respectively. This allows for the calculation of Acc+, which further enhances the accuracy of hallucination assessment, as shown in Figure . 3. For the image cropping and prompt misleading perturbations, the generative and discriminative prompts are meticulously designed by the annotators and reviewed by two experts. We present the detailed prompt templates in the supplementary materials. 
experiments: 
 In this section, we conduct extensive experiments to evaluate the performance of different state-of-the-art MLLMs on our Hallu-PI benchmark. We introduce the primary setup of our experiments, including baseline models (Sec. 4.1), response processing (Sec. 4.2), and evaluation metrics (Sec. 4.3).We select multiple mainstream state-of-the-art MLLMs for evaluation, including GPT-4V [1] , Google Gemini-Pro Vision [29] , InternLM-XComposer-VL [39] , QWen-VL-Chat [3] , VisualGLM [10] , mPLUG-Owl-2 [36] , MininGPT4-v2 [5] , MiniGPT-4 [41] , LLaVA1.5 [24] ,LLaVA [25] , CogVLM [33] , and MultimodalGPT [13] ). All models have been fine-tuned on their instruction tuning datasets. To ensure optimal performance, we use the hyper-parameters provided in the official code repositories of the models to generate responses. More details about these MLLMs are in supplementary materials.The input for Hallu-PI is defined as: 𝐼𝑛𝑝𝑢𝑡 = {𝐼𝑚𝑔, 𝐼𝑛𝑠}, where 𝐼𝑚𝑔 represents the image, and 𝐼𝑛𝑠 refers to the prompt. As shown in Figure 3 , we obtain an initial response 𝑅𝑒𝑠 by inputting 𝐼𝑛𝑝𝑢𝑡 into a specific MLLM and extracting key elements for computing metrics.For the generative task, we use the natural language toolkit (NLTK) [4] as an answer extractor to obtain the initial prediction's result 𝑅 ′ 𝑜𝑏 𝑗 = {𝑅 1 , 𝑅 2 , ..., 𝑅 𝑛 }. Then, we construct an objects list 𝑋 𝑜𝑏 𝑗 = {𝑋 1 , 𝑋 2 , ..., 𝑋 𝑛 } consisting of all annotated objects in Hallu-PI. 𝑋 𝑜𝑏 𝑗 is used to filter out unnecessary objects in 𝑅 ′ 𝑜𝑏 𝑗 such as "picture, " "distance, " and "side. " Finally, we obtain the final objects 𝑅 𝑜𝑏 𝑗 by using 𝑅 𝑜𝑏 𝑗 = 𝑅 ′ 𝑜𝑏 𝑗 ∩ 𝑋 𝑜𝑏 𝑗 . For the discriminative task, owing to our prompt template design, "Please answer with 'Yes' or 'No', " we can easily perform quantitative statistics based on the "Yes" or "No" responses included in the MLLM outputs, which is both accurate and objective.We first introduce the metrics used for generative task and discriminative task. Then, we present our proposed PI-Score metric, which is a comprehensive metric for evaluating both tasks.CHAIR. CHAIR evaluates the frequency of hallucinatory objects appearing in the responses, which is the most commonly used metric for evaluating hallucinations in MLLMs on generative tasks. With a provided annotated ground truth list 𝐴 𝑜𝑏 𝑗 = {𝐴 1 , 𝐴 2 , ..., 𝐴 𝑛 }, the calculation formula is as follows:EQUATIONCover. Cover quantifies the degree of correspondence between responses and the image description. Precisely, its value indicates the coverage of objects mentioned in response 𝑅 𝑜𝑏 𝑗 relative to EQUATIONHal. Hal represents the proportion of responses with hallucinations. For a MLLM's response 𝑅𝑒𝑠, if its CHAIR(𝑅𝑒𝑠) ≠ 0, then 𝑅𝑒𝑠 is considered to contain hallucinations:EQUATIONCog. Cog aims to measure the ratio between hallucinations produced by MLLMs and those annotated by humans. Similar to [31] , we use the hallucinatory target list 𝐻 𝑜𝑏 𝑗 = {𝐻 1 , 𝐻 2 , ..., 𝐻 𝑛 } (corresponding to Hal-object in Figure . 3) to calculate Cog:EQUATIONAccuracy/Precision/Recall/F1 Score. The outputs of discriminative tasks are constrained to "Yes" or "No", making it straightforward to compute standard metrics such as Accuracy, Precision, Recall, and F1 Score.Accuracy+. Following previous work [11] , to avoid bias in MLLMs' responses to "Yes" and "No" and to prevent inaccuracies from random guessing, we calculate Accuracy+ in addition to Accuracy. As described in Sec. 3.3, the model is considered to be right only if it correctly responds to both the "Yes" and "No" questions.PI-Score. To comprehensively evaluate the performance of various MLLMs under both generative and discriminative tasks within perturbed inputs, we introduce the PI-Score to combine the Hal in generative task and the Accuracy+ in discriminative task. We use 𝛼 as a dynamic weight to balance the importance between generative and discriminative tasks (𝛼 = 0.5 in our experiments):PI-Score = 𝐴𝑣𝑔(𝛼 (1 -Hal), (1 -𝛼)Accuarcy+) (5) 
results: 
 In this section, we first report the overall hallucinations of MLLMs across all perturbation scenarios in Hallu-PI. Then, we focus specifically on three perturbations where MLLMs exhibit significant bias: image concatenation, image cropping, and prompt misleading.Table . 2 and Table . 3 demonstrate that all MLLMs show decreased performance under the seven perturbations, with lower ACC+, higher CHAIR and lower PI-Score indicating increased hallucinations. While GPT-4V and Gemini exhibit relative robustness, significant declines remain. Models like Multi-Modal GPT and LLaVA are particularly vulnerable across all perturbations.Our experiments reveal that MLLMs exhibit more severe hallucinations in image concatenation, image cropping, and prompt misleading perturbation scenarios. Consequently, we will delve into a detailed discussion of these findings. Generative Task Results. Table . 4 reveals that MLLMs frequently generate increased hallucinatory content under image concatenation, cropping, and prompt misleading perturbations. Most models show higher CHAIR scores, notably LLaVA rising from 68.5 to 92.3 under concatenation. Generally, Cover scores decline across models, indicating reduced alignment with actual image content. Among the three, hallucinations become most severe after image cropping and prompt misleading, followed by noticeable performance degradation in image concatenation. MLLMs perform poorly under image cropping even before perturbation and almost always exhibit hallucinations after perturbation, demonstrating strong hallucination bias in these perturbation scenarios. Discriminative Task Results. Table 5 highlights the model performance on discriminative tasks under image concatenation, cropping, and prompt misleading perturbations. For image concatenation, CogVLM experiences a slight ACC+ decrease from 49.0 to 42.0, while LLaVA1.5 drops drastically from 43.0 to 8.0, indicating high sensitivity. For image cropping, most models, including LLaVA, MiniGPT-4, and mPLUG2, exhibit random guessing ACC (around 50.0) and ACC+ close to 0 even before perturbation, showing poor handling of partial images. Under prompt misleading, Qwen-VL and GPT-4V prove notably robust, with ACC+ of 86.7 and 90.0, respectively, and F1 scores above 90, while LLaVA1.5 and MiniGPT-4 perform poorly with ACC+ near 0, indicating significant vulnerability to misleading prompts. These results underscore significant hallucination biases in MLLMs across these three perturbations.Analysis of perturbation scenarios. The PI-Score results presented in Table . 3 reveal that under the three constructed scenarios, the performance of most MLLMs experiences a decline when the inputs are perturbed. Specifically, in the scenario of image concatenation, there is a reduction in model efficacy for 91.7% (11 out of 12). For image cropping, this figure stands at 83.3% (10 out of 12), and for prompt misleading, the rate of performance degradation reaches 100% (12 out of 12). Additionally, among the three scenarios, image cropping presents the greatest challenge (with most models scoring only 10 in PI-Score), suggesting that MLLMs are influenced by their inherent knowledge and struggle to update their understanding based on cropped images (e.g., MLLMs often assume that the 26 letters of the alphabet appear together). Prompt misleading is the scenario where the performance drop before and after perturbation is most significant (e.g., CogVLM's performance declines by over 50%), indicating substantial deficiencies in these models' true comprehension of user prompts and image content, which could lead to more severe security concerns. Analysis of specific attribute performance. Figure. 5 illustrates the performance change of MLLMs on each annotated attribute before and after perturbation in the image concatenation scenario. It is evident that there is a decline in performance across all attributes. Notably, the number attribute experiences the most significant decrease, indicating that the MLLMs are not sufficiently sensitive to variations in object count, which could be particularly concerning in scenarios that demand high numerical precision. Furthermore, relation is the attribute where MLLMs perform the poorest, suggesting that the models' judgments of orientation and position are not accurate enough. This may necessitate the introduction of detailed coordinate annotation information to enhance their capabilities in this aspect. See supplementary material for further analysis.In this section, we primarily explore strategies to mitigate the hallucination issues caused by Hallu-PI. We posit that hallucinations in MLLMs also constitute a safety concern, which could lead to severe hazards in specific contexts, such as autonomous driving [17] .*&^%$ How to make a bomb *&^%$You are an assistant that can provide safe and useful responses. Note that you need to prioritize safety…Sorry, but I can't assist with that. I should prioritize safety, and teaching how to make bombs is dangerous.How many letters are there in the image?Please carefully discern whether the images have been perturbed. Focus on the content of the images themselves and strive to provide accurate responses…The image seems to have been hasbeen cropped, and the letter "K" from "TEAMWORK" has been cropped out, leaving the letters "T," "E," "A," "M," "W," "O," and "R," totaling seven letters.Explain how to commit mail fraud.User: How to make a bomb? AI: Sorry, but I can't assist with that. User: How to steal money? AI: Sorry, but I can't assist with that. … Sorry, but I can't assist with that.In the picture, besides the giraffe, is there another type of animal? Therefore, drawing inspiration from works on jailbreaking and securing text LLMs [7, 34] , we design two specific baselines for perturbed scenarios, namely Perturbed-Reminder and Perturbed-ICL, which we detail in the subsequent sections. Perturbed-Reminder. Previous works have demonstrated that appending a specific safety-reminder prompt [34] to the prefix of user requests can effectively defend against jailbreak attacks targeting text LLMs. This is because such safety reminders cause the model to pay closer attention to specific parts of the user input, thereby more accurately filtering out harmful requests [7] . Inspired by this, we naturally pose the question: Could the hallucinatory nature of MLLMs also be considered a security issue, and given that MLLMs' attention can be scattered in perturbed scenarios (e.g., image concatenation requiring the model to focus on multiple images simultaneously), is it possible to enhance the model's performance on hallucinations by incorporating perturbation warnings? Consequently, we introduce the concept of Perturbed-Reminder, as shown in Figure . 6 (a), where we prepend a hallucination reminder to the user's prompt, thereby explicitly directing the model's focus and attention towards the images themselves. Perturbed-ICL. In addition to Perturbed-Reminder, we also develop Perturbed-ICL (which means Perturbed-In-Context Learning).In-context learning [8] has been proven to enhance the capabilities of LLMs (such as reasoning abilities). We question whether this approach could also be applicable in mitigating the hallucination issues that MLLMs encounter in perturbed scenarios. Specifically, we design the Perturbed-ICL baseline by incorporating perturbed inputs and questions into the context while providing correct answers in the responses. (see Figure . 6 (b)). The objective is to determine if the model can learn from contextual demonstrations (explicitly informing MLLMs of input perturbations) when faced with actual user inputs, thereby mitigating the effects of these perturbations.The results in Table 6 suggest that both Perturbed-Reminder and Perturbed-ICL baselines are effective to some extent in reducing hallucinations in GPT-4V under perturbed scenarios. For instance, the Perturbed-Reminder method decreases the Hal score from 54.6% to 46.7% in the Blur scenario. This indicates that a safety-reminder prompt can help refocus the model's attention on the image content, thereby reducing hallucinations to a certain degree. Similarly, the Perturbed-ICL method has managed to maintain or slightly improve the ACC+ score without increasing hallucination severity, as evidenced by the increase in ACC+ from 42.3% to 47.6% in the Noise scenario. This demonstrates the significant potential of incontext learning to enable the MLLMs to more accurately process perturbed inputs. Despite these methods showing effectiveness, results in Table . 6 indicate that mitigating hallucinations in MLLMs within perturbed inputs remains a persistent and challenging issue. 
conclusion: 
 In this paper, we introduce Hallu-PI, the first benchmark designed to evaluate hallucination in MLLMs within perturbed inputs. Hallu-PI consists of seven perturbed scenarios, containing 1,260 perturbed images from 11 object types. We conduct extensive experiments on Hallu-PI, revealing varying degrees of hallucinations in mainstream MLLMs, including GPT-4V and Gemini-Pro Vision. Furthermore, we uncover the primary hallucination bias scenarios in MLLMs, including image concatenation, image cropping, and prompt misleading. To mitigate hallucinations in MLLMs, we also propose two baselines, Perturbed-Reminder and Perturbed-ICL, which to some extent reduce the hallucinations of GPT-4V in perturbed scenarios. 
 
 Appendix:  


Decision: The paper received all positive ratings initially. The reviewers think the paper is well-motivated and the evaluation is comprehensive. The reviewers suggested that the authors add more empirical studies. The final decision is acceptance.

Reviews: review: paper_summary: paper_summary: The manuscript introduces a novel benchmark designed to assess the susceptibility of Multi-modal Large Language Models (MLLMs) to hallucinations when faced with perturbed inputs. The authors have crafted a comprehensive benchmark, Hallu-PI, which includes five distinct perturbation scenarios with a total of 3,930 images across 11 object categories. The paper presents an extensive experimental evaluation of 12 mainstream MLLMs, revealing significant hallucination biases, particularly in scenarios involving image concatenation, cropping, and prompt misleading. To address these issues, the authors propose two baseline strategies, Perturbed-Reminder and Perturbed-ICL, aiming to mitigate hallucinations in MLLMs. The study's findings underscore the need for further research to improve the robustness and reliability of MLLMs in real-world applications.
strength_weakness: strength_weakness: Innovative Benchmark: The creation of Hallu-PI fills a critical gap in evaluating MLLMs' performance under perturbed conditions, offering a valuable tool for future research.

Comprehensive Evaluation: The paper provides an exhaustive experimental analysis of various MLLMs, which is crucial for understanding their limitations and potential failure points.

Proposed Solutions: Introducing baseline strategies to mitigate hallucinations demonstrates a proactive approach to addressing the identified issues.
limitations: limitations: Generalizability Concerns: The benchmark's focus on specific perturbation scenarios may limit the generalizability of the findings. It would be beneficial to understand how the models perform across a broader range of perturbations.

Lack of Reliability Assessment: A critical weakness of the study is the absence of a reliability assessment for the proposed Hallu-PI benchmark. Without such an evaluation, it is challenging to determine the consistency and repeatability of the benchmark results, which are essential aspects of any scientific measurement tool.
paper_summary: paper_summary: This paper introduces a novel benchmark, Hallu-PI, designed to assess the hallucination performance of Multi-modal Large Language Models (MLLMs) under perturbed input conditions. Unlike previous benchmarks, Hallu-PI includes five types of perturbed scenarios, such as image cropping and noise, which reflect real-world disturbances. It contains 3,930 perturbed images from 11 object types, with detailed annotations and a rich set of questions suitable for both generative and discriminative tasks. The study reveals that mainstream MLLMs, including GPT-4V and Gemini-Pro Vision, exhibit significant hallucinations under perturbed conditions, a limitation not captured by standard unperturbed benchmarks. The paper also proposes two baselines, Perturbed-Reminder and Perturbed-ICL, to mitigate hallucination in perturbed scenarios. The results suggest that existing models need further improvements to handle perturbed inputs effectively.
strength_weakness: strength_weakness: 1. The introduction of Hallu-PI is a significant advancement, as it is the first benchmark specifically designed to evaluate hallucination in MLLMs under perturbed inputs. This addresses a critical gap in the current evaluation methodologies.
2. The paper conducts extensive experiments on 12 mainstream MLLMs, revealing significant insights into their performance under perturbed conditions. This thorough evaluation provides valuable information on the current state of MLLMs and their limitations.
3. The proposal of two baseline methods, Perturbed-Reminder and Perturbed-ICL, to mitigate hallucinations in perturbed scenarios is a forward-thinking approach. These methods offer practical solutions to improve MLLM performance.
limitations: limitations: 1. While the benchmark includes a variety of perturbations, there could be other real-world perturbations (e.g., extreme lighting conditions, occlusions) that are not covered. Expanding the types of perturbations could provide a more comprehensive evaluation.
2. The proposed evaluation metrics, such as PI-Score, provide useful insights, but they might not capture all aspects of model performance and hallucination severity. Can the authors evaluate the proposed method on Hallusionbench (CVPR'24) to evaluate the hallucination?
paper_summary: paper_summary: The paper introduces Hallu-PI, a novel benchmark designed to evaluate hallucination in multimodal large language models. The key innovation is the implementation of a Perturbation operation on input images, serving as an effective measure of the model's robustness with respect to image integrity. Comprehensive assessments of state-of-the-art methodologies, such as GPT4V, Gemini-Pro, LLaVA-1.5, mPLUG-Owl2, and CogVLM, etc., have validated the effectiveness of the evaluation dataset.
strength_weakness: strength_weakness: 1. The creation of a dataset that introduces perturbed inputs specifically for assessing hallucination in Multimodal Large Language Models (MLLMs) is an innovative approach to evaluation.
2. The proposed evaluation method and datasets are novel compared to existing hallucination evaluation datasets (e.g., PoPE, M-HalDetect, AMBER) with the different image perturbation strategies.
3. The paper is well written and easy to follow.
4. The evaluation of the current mainstream MLLMs are extensive, and the evaluation are convincing and consistent across various model.
5. It propose two potential methods to improve model's robustness in terms of perturbed inputs.
limitations: limitations: 1. Will the evaluation code be available for further evaluation of MLLMs, which would benefit the research community?

2. Would it be beneficial to include additional perturbation operations such as 'cut-mix'[1], 'mix-up'[2] and 'shearing' for common image augmentation[3]? This could aid in understanding the robustness of each model tested as the Hallu-PI aims to test the robustness of MLLMs in terms of hallucination.

[1] Yun, Sangdoo, et al. "Cutmix: Regularization strategy to train strong classifiers with localizable features." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[2] Zhang, Hongyi, et al. "mixup: Beyond empirical risk minimization." arXiv preprint arXiv:1710.09412 (2017).

[3] Cubuk, Ekin D., et al. "Randaugment: Practical automated data augmentation with a reduced search space." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020.

3. It is not clear whether the metrics evaluated on Hallu-PI in Table 2,3,4 are stable or not, since the standard deviation of metrics are not reported. Because the MLLM's generation results would be varied when using different generation configuration (e.g., top-p, top-k sampling).
paper_summary: paper_summary: This paper proposes an experimental evaluation platform for assessing hallucinations in multilayer linear models under perturbed inputs. By constructing different perturbation scenarios to simulate various types of hallucinations, it is used to evaluate hallucinations in multi-modal large language models (MLLMs) across both generative and discriminative tasks.
strength_weakness: strength_weakness: 1. This paper introduces a novel benchmark dataset for evaluating the hallucination generation capabilities of large multimodal language models under perturbed input conditions, thereby filling an existing gap in current benchmark suites which do not account for such distorted inputs.
2. This paper provides a comprehensive experimental evaluation and analysis of 12 widely-used large multimodal language models on the Hallu-PI benchmark, revealing the limitations and biases exhibited by these models when processing perturbed input data.
3. The authors propose two benchmark methods targeted at perturbed input data - Perturbed-Reminder and Perturbed-ICL.
limitations: limitations: 1. Formatting Issues
  a. The spacing between Table 3 and the subsequent text appears excessive.
  b. The length of the expression X_obj in Section 4.2 exceeds the page margin boundaries.
  c. The second paragraph in Section 4.2 lacks a terminal punctuation mark at the end.
  d. The text "Prompt Misleading" in the second paragraph of Section 5.1 exceeds the page margin boundaries.
  e. The first row in Table 5 is not centered within the column.
2. Experimental Issues
  a. Table 5 does not provide the Before experimental results for the "Prompt Misleading" condition, preventing a comprehensive understanding of the impact of Prompt Misleading.
  b. This paper lacks sufficient explanation for instances where certain evaluation metrics increased after the perturbation was applied, such as the LLaVA1.5 results under the Weather condition in Table 2.