Unified Hallucination Detection for Multimodal Large Language Models (2024)

Abstract: Despite significant strides in multimodal tasks, Multimodal Large Language Models (MLLMs) are plagued by the critical issue of hallucination. The reliable detection of such hallucinations in MLLMs has, therefore, become a vital aspect of model evaluation and the safeguarding of practical application deployment. Prior research in this domain has been constrained by a narrow focus on singular tasks, an inadequate range of hallucination categories addressed, and a lack of detailed granularity. In response to these challenges, our work expands the investigative horizons of hallucination detection. We present a novel meta-evaluation benchmark, MHaluBench, meticulously crafted to facilitate the evaluation of advancements in hallucination detection methods. Additionally, we unveil a novel unified multimodal hallucination detection framework, UNIHD, which leverages a suite of auxiliary tools to validate the occurrence of hallucinations robustly. We demonstrate the effectiveness of UNIHD through meticulous evaluation and comprehensive analysis. We also provide strategic insights on the application of specific tools for addressing various categories of hallucinations1.

heading: 1 Introduction
text: The recent emergence of MLLMs (Ho et al., 2020; OpenAI, 2023; Durante et al., 2024) that more closely mirror human cognition and learning has unleashed unprecedented possibilities for the future of artificial general intelligence (AGI). Despite MLLMs’ impressive abilities, they are susceptible to generating seemingly credible content that contradicts input data or established world knowledge, a phenomenon termed “hallucination”(Liu et al., 2024; Wang et al., 2023a; Huang et al., 2023c;
∗Corresponding author. 1The code can be accessed via https://github.
com/zjunlp/EasyDetect, and the demonstration is available at http://easydetect.openkg.cn.
Tonmoy et al., 2024; Zhang et al., 2023a). These hallucinations hinder the practical deployment of MLLMs and contribute to the dissemination of misinformation. Consequently, detectors that could detect multimodal hallucinations (Yang et al., 2023) within responses from MLLMs are urgently needed to alert users to potential risks and drive the development of more reliable MLLMs.
Although several works have been conducted to detect hallucinations from MLLMs(Zhou et al., 2023; Zhai et al., 2023; Li et al., 2023b; Wang et al., 2023c) or alleviate hallucinations(Xing et al., 2024; Wu et al., 2024), these efforts operate in isolation and have certain limitations when compared with the aspects illustrated in Figure 1: (1) Task Singularity: Current research has primarily concentrated on specific tasks, such as image captioning while neglecting that text-to-image generation, an important component of AGI, also suffers from hal-
3235
lucinations induced by MLLMs. (2) Limited Hallucination Categories: Prior studies have focused on identifying hallucinations at the object level, yet they fail to consider the prevalence of scene-text or factual inconsistencies that also frequently occur in MLLMs. (3) Incomplete Granularity: It would be more valuable to assess hallucinations at a finegrained level, examining individual claims within a response, rather than evaluating the entire response holistically. Considering these constraints hinder rapid progress in practical hallucination detection, it raises the question: Can we develop a unified perspective for detecting hallucinations from MLLMs?
To further investigate this problem, we have broadened the concept of multimodal hallucination within MLLMs to a holistic framework, integrating both image-to-text generation such as Image Captioning (IC) and Visual Question Answering (VQA), as well as text-to-image-synthesis (T2I) – to align with MLLMs’ capabilities of performing varied multimodal tasks. We are committed to exploring a broad spectrum of hallucinatory categories and the intricate nuances of claim-level hallucination through a lens that integrates both modality-conflicting and fact-conflicting hallucinations. Based on the outlined perspectives, We have developed the MultiModal Hallucination Detection Benchmark (MHaluBench) to assess the progress of unified multimodal hallucination detectors for MLLMs and embodied the data framework depicted in Figure 1.
At its core, leveraging MLLMs’ inherent selfdetection mechanisms to pinpoint diverse hallucinations encounters significant hurdles. We further develop a tool-augmented framework for unified hallucination detection, named UNIHD, which integrates evidence from multiple auxiliary tools through the following procedure: (1) Essential Claim Extraction involves extracting the core claims within the generated response for image-totext generation or user queries in text-to-image generation; (2) Autonomous Tool Selection via Query Formulation prompts MLLMs (GPT-4/Gemini) to autonomously generate pertinent questions for each claim. These questions are crafted to determine the specific type of tool required for each claim and to establish the input for the tool’s operation; (3) Parallel Tool Execution deploys a suite of specialized tools to operate concurrently, providing evidence from their outputs to reliably validate potential hallucinations; (4) Hallucination Verification with Rationales aggregates the collected evidence to in-
struct the underlying MLLM to judge whether the claim hallucinatory with rationals for explanation.
We have conducted a thorough evaluation of the UNIHD framework, utilizing the underlying MLLM against the MHaluBench benchmark. Our findings underscore the effectiveness of our approach and confirm that multimodal hallucination detection remains a formidable challenge. In a nutshell, We conclude our contributions as:
• We propose a more unified problem setting for hallucination detection in MLLMs, encompassing a broad spectrum of multimodal tasks and hallucination categories, thus enriching the unified understanding of hallucination in MLLMs.
• We unveil MHaluBench, a meta-evaluation benchmark that encompasses various hallucination categories and multimodal tasks. This benchmark is equipped with fine-grained analytical features, gauging the progress of hallucination detectors.
• We introduce UNIHD, a task-agnostic, toolenhanced framework for the detection of hallucinations in content produced by MLLMs. Our extensive experiments demonstrate the efficacy of this method, underscoring that MHaluBench continues to be a challenging yet vital task.

heading: 2 Preliminaries
text: We explore a unified perspective on hallucination in MLLMs (illustrated in Figure 2) with the aspiration of developing a unified detection framework.
Unified View of Multimodal Hallucination Taxonomy. A prerequisite for unified detection is the coherent categorization of the principal categories of hallucinations within MLLMs. Our paper superficially examines the following Hallucination Taxonomy from a unified perspective: • Modality-Conflicting Hallucination. MLLMs
sometimes generate outputs that conflict with inputs from other modalities, leading to issues
such as incorrect objects, attributes, or scene text. An example in Figure 1 (a) includes an MLLM inaccurately describing an athlete’s uniform color, showcasing an attribute-level conflict due to MLLMs’ limited ability to achieve finegrained text-image alignment.
• Fact-Conflicting Hallucination. Outputs from MLLMs may contradict established factual knowledge. Image-to-text models can generate narratives that stray from the actual content by incorporating irrelevant facts, while text-to-image models may produce visuals that fail to reflect the factual knowledge contained in text prompts. These discrepancies underline the struggle of MLLMs to maintain factual consistency, representing a significant challenge in the domain.
Unified Detection Problem Formulation. Unified detection of multimodal hallucination necessitates the check of each image-text pair a = {v, x}, wherein v denotes either the visual input provided to an MLLM, or the visual output synthetic by it. Correspondingly, x signifies the MLLM’s generated textual response based on the v or the textual user query for synthesizing v. Within this task, each x may contain multiple claims, denoted as {ci}i=1···n. The objective for hallucination detectors is to assess each claim from a to determine whether it is “hallucinatory” or “nonhallucinatory”, providing a rationale for their judgments based on the provided definition of hallucination. Text hallucination detection from LLMs denotes a sub-case in this setting, where v is null.

heading: 3 Construction of MHaluBench
text: To facilitate research in this area, we introduce the meta-evaluation benchmark MHaluBench, which encompasses the content from image-to-text and text-to-image generation, aiming to rigorously assess the advancements in multimodal hallucination detectors. Our benchmark has been metic-
ulously curated to include a balanced distribution of instances across three pivotal tasks, which encompasses 200 exemplars for the task of IC 200 for VQA, and an additional 220 dedicated to Text-to-Image Generation. The comparison of MHaluBench with other benchmarks is detailed in Table 1 and the statistical details are provided in Figure 3 and Figure 4.

heading: 3.1 Hallucinatory Example Collection
text: Image-to-Text Generation. We focus on IC and VQA tasks, drawing samples from the MS-COCO 2014 validation set (Lin et al., 2014) and the TextVQA test set (Singh et al., 2019). We compile generative outputs from mPLUG (Ye et al., 2023), LLaVA (Liu et al., 2023c), and MiniGPT4 (Zhu et al., 2023) to form the core dataset for MHaluBench. These models are representative of current leading MLLMs, characterized by their diverse content generation capabilities and a notable presence of hallucinations, as depicted in Figure 8.
Text-to-Image Generation. We source initial captions from DrawBench (Saharia et al., 2022) and T2I-CompBench (Huang et al., 2023a). These captions are augmented through ChatGPT to include more specific information such as objects, attributes, and factual details, among others. The refined caption guides the DALL-E 2 (Ramesh et al., 2022) and DALL-E 3 model (Betker et al., 2023) in producing visually detailed images.

heading: 3.2 Segment and Claim Extraction
text: Beyond evaluating overall responses, we introduce segmentation at both the segment and claim levels for a multi-granular assessment of hallucinations, enabling more precise feedback to improve model performance (Lightman et al., 2023). We leverage ChatGPT’s advanced instruction-following ability to extract detailed segments and related claims. For image-to-text tasks, we split and extract the model’s textual output into segments and claims;
for text-to-image cases, we break down user queries into fundamental intent concepts, which are subsequently regarded as claims.

heading: 3.3 Human Annotation and Agreement.
text: Our annotation criteria evaluate whether imageto-text output conflicts with the input image or world knowledge and whether text-to-image visuals conflict with claims or world knowledge. Extracted claims are labeled as hallucinatory or nonhallucinatory, with a segment deemed hallucinatory if it contains any such claim; otherwise, it is labeled non-hallucinatory. An entire response is labeled hallucinatory if it includes even one hallucinatory segment. We allocate the dataset uniformly across three annotators with graduate-level qualifications for independent categorization. Decisions in uncertain cases were initially held by individual annotators and later resolved by majority rule. Interannotator reliability, measured by Fleiss’s Kappa (κ), shows significant agreement (κ = 0.822) over the full annotated dataset, indicating a high level of concordance within the range 0.80 ≤ κ ≤ 1.00.

heading: 4 UNIHD: Unified Hallucination Detection Framework for MLLMs
text: We present UNIHD in Figure 5 and follow. The specific prompts are listed in Appendix A

heading: 4.1 Essential Claim Extraction
text: To identify fine-grained hallucinations within the response, claim extraction is a prerequisite. Following the procedure in §3.2, we employ the advanced instruction-following abilities of MLLMs for efficient claim extraction. Specifically, GPT4V/Gemini is adopted as the base LLM to efficiently derive verifiable claims from the outputs of image-to-text models (extracting each response
into individual claims) and text-to-image models (deconstructing user queries into distinct claims) 2.

heading: 4.2 Autonomous Tool Selection Via Query Formulation
text: After extracting essential claims from the input image-text pair a = {v, x}, the challenge of hallucination detection is to aptly match each claim with appropriate aspect-oriented tools. We approach this issue by assessing whether the underlying MLLMs can generate pertinent queries for a given set of claims {ci}i=1···n to provide relevant input to the specific aspect-oriented tool. To facilitate this, we prompt underlying MLLMs like GPT-4V/Gemini to autonomously formulate meaningful queries. Demonstrated in Figure 5, this module yields custom queries for each claim, or “none” when a tool is unnecessary. For example, the framework determines that claim1 calls for the attribute-oriented question “What color is the uniform of the athlete on the right side?” and the object-oriented inquiry “[‘athlete’, ‘uniform’]”, bypassing the need for scene-text and fact-oriented tools.

heading: 4.3 Parallel Tool Execution
text: Leveraging queries autonomously generated from various perspectives, we simultaneously deploy these tools in response to the queries, gathering a comprehensive array of insights to underpin the verification of hallucinations. The specific tools employed in our framework are detailed below, selected for their ability to effectively address a wide range of multimodal hallucination scenarios:
• Object-oriented tool: We employ the open-set object detection model Grounding DINO (Liu et al., 2023d) for capturing visual object information, crucial for detecting object-level hallucinations. For instance, inputting “[‘athlete’, ‘uniform’]” prompts the model to return two
2In subsequent experiments, our framework builds upon the pre-annotated claims available in MHaluBench, and the claim extraction is only necessary in the open-domain setting.
uniform objects and two athlete objects, along with their normalized location coordinates.
• Attribute-Oriented Tool: Dealing with attributes such as positions, colors, and actions, we harness underlying MLLMs (such as GPT-4V and Gemini) to answer the specific attribute-level questions. These responses are leveraged for hallucination verification within the same MLLMs, mirroring a self-reflect akin to (Shinn et al., 2023).
• Scene-Text-Oriented Tool: Should the generated questions for scene text not be exclusively “none”, we then invoke MAERec (Jiang et al., 2023) as our scene-text detection tool, which is capable of identifying scene text within images along with their corresponding normalized fourdimensional coordinates.
• Fact-Oriented Tool: To validate conflicting factual hallucinations, we harness the Serper Google Search API to perform web searches using specific fact-based questions. By extracting and scrutinizing the top results, we obtain a range of snippets from the API’s responses for analysis.
Moreover, UNIHD is tool-agnostic, facilitating the seamless integration of emerging tools and detection strategies to amass tool knowledge, thereby bolstering the process of hallucination verification.

heading: 4.4 Hallucination Verification with Rationales
text: In the concluding phase of our process, we subject each claim, denoted as ci, to a binary prediction to ascertain its hallucinatory status. Claims are categorized as either HALLUCINATORY or NONHALLUCINATORY based on the level of evidence support. To accomplish this, we aggregate the collected evidence from tools with the original image and its corresponding claim list3 into a comprehensive prompt. Subsequently, we instruct our chosen MLLM (GPT-4V or Gemini) to assess each claim’s hallucinatory potential. In doing so, the MLLM also generates insightful explanations to elucidate the rationale behind its judgment.

heading: 5 Experiment

heading: 5.1 Experimental Settings
text: Baselines. We compare UNIHD on MHaluBench4 with two baselines, Self-Check (2-shot)5 and Self-Check (0-shot) based on
3Note that the set a = {v, x}, corresponding to the list of claims, is input into the detectors in a single batch. This operation allows the detectors to capture contextual information while also enhancing efficiency.
4In this paper, we conducted experiments using the evaluation benchmark from our published V0.1 version.
5Self-Check (2-shot) utilize two complete demonstrations based on a = {v, x} rather than only two claims.
CoT (Wei et al., 2022), which assess the capability of the underlying MLLM to identify hallucinations without external knowledge and have shown effectiveness across other various tasks (Chern et al., 2023; Xie et al., 2023). We prompt GPT-4V (gpt-4-vision-preview) and Gemini (Pro Vision) to recognize fine-grained hallucinations and explain the reasoning behind this determination.
Evaluation Perspective. We compute the recall, precision, and Micro-F1 metrics individually for both hallucinatory and non-hallucinatory categories. Additionally, we assess the overall performance by measuring the average Macro-F1 scores at the claim and segment levels. We categorize a segment as non-hallucinatory only if all associated claims are classified as non-hallucinatory; it is deemed hallucinatory if any associated claims do not meet this criterion.

heading: 5.2 Evaluation Results
text: MHaluBench poses a challenging benchmark for multimodal hallucination detection. The segment-level and response-level outcomes are presented in Table 2. Even though all hallucinatory instances in MHaluBench are obtained from opensource MLLMs’ outputs rather than being generated by GPT-4V/Gemini itself, it is noteworthy that the majority of detectors achieve an overall Macro-
F1 score ranging between 70%-80%, exhibiting subpar performance on MHaluBench.
GPT-4V surpasses Gemini as the detector base. GPT-4V-powered detectors consistently outperform Gemini counterparts, achieving higher MacroF1 scores, especially in the text-to-image generation. For instance, Self-Check (0-shot) using GPT4V achieves a claim-level Macro-F1 of 72.82, significantly surpassing Gemini’s Macro-F1 score of 52.98. However, Gemini-powered detectors exhibit better performance in non-hallucinatory categories
UniHD Output: [ ..., {"claim3":"hallucination","reason":"According to the external knowledge, Fanta originated in Germany, not America, during World War II. Therefore, there's a hallucination.."}, ... ]
Self-Check(2-shot) Output: [ ..., {"claim3":"hallucination","reason":"The origin of Fanta being in America during World War II cannot be verified through visual information in the image.", ...} ]
Comparison of Explanation Reasonability 

heading: Here is the claim list:
text: correct prediction 
Image-to-Text User Query: What soda is painted on this building? Here is the claim list: ["claim1": "The soda painted on the building is Fanta", "claim2": "Fanta is a fruitflavored carbonated soft drink.", "claim3": "Fanta originated in America during World War.", "claim4": Fanta's origin was due to limited Coca-Cola availability]
UniHD Output: [{"claim1":"hallucination","reason":"The object detection model has detected five bikes in the image, not three as stated in the claim. Therefore, there's a hallucination."}, ...]

heading: Self-Check(2-shot) Output:
text: [ {"claim1":"hallucination","reason":"There are more than three bikes parked in the image. Therefore, there's a hallucination.", ...} ]
User Query: Three bikes parked outside a local restaurant establishment.
Here is the claim list: ["claim1": "There are three bikes parked.", "claim2": "The bikes are outside a local restaurant."]
Text-to-Image
correct prediction 
for image-to-text tasks, indicating a potential bias towards reduced sensitivity to hallucinations.
UNIHD Empowered by GPT-4V: Superior Detection Across the Board. Table 2 demonstrates that UNIHD, leveraging GPT-4V, consistently outperforms other baseline detectors in image-to-text and text-to-image tasks. Despite the Self-Check (2-shot) showcasing GPT-4V and Gemini’s robust in-context learning, UNIHD markedly exceeds its performance, emphasizing the benefits of integrating external tools for more robust evidence verification and reliable hallucination detection.

heading: 5.3 Analysis

heading: Which Type of Hallucination Can Benefit the
text: Most from Tool Enhancement? Figure 6 shows that UNIHD enhances the detection of scene text and factual hallucinations over Self-Check (2- shot), suggesting that GPT-4V or Gemini’s inherent limitations make the evidence provided by the tool especially valuable. However, UNIHD exhibits minimal improvement in identifying attributelevel hallucinations, potentially attributed to a
lack of specialized tools for direct attribute detection, with self-reflection methods based on GPT4V/Gemini proving to be relatively weak.

heading: Explanation Reasonability of UNIHD. As
text: shown in the upper portion of Figure 7, both the fact-level hallucination “Fanta originated in America during World War.” and the object-level hallucination “There are three bikes parked.” are accurately identified by Self-Check (2-shot) and UNIHD. Comparative analysis reveals that UNIHD excels in synthesizing evidence to provide a more credible and compelling rationale.
Failure Analysis of UNIHD. As shown in the lower part of Figure 7, we present two instances where UNIHD exhibits limitations. The left case demonstrates situations where the tool either generates incorrect evidence or fails to provide useful information, leading to erroneous judgments by the MLLM. On the right, we observe cases where the MLLM maintains its initial bias despite receiving accurate evidence, resulting in incorrect decisions. These scenarios highlight areas for further research
to enhance tool accuracy and to develop MLLMs dedicated to better hallucination detection.

heading: Text-to-Image Hallucination vs. Image-to-Text Hallucination: Which is Easier to Detect?
text: Both baselines and the GPT-4V-enhanced UNIHD show significantly improved performance in identifying hallucinations in text-to-image content over image-to-text content. This can be traced back to the structured nature of manually written user queries for text-to-image tasks, which yield more uniform images. while image-to-text confronts the complexity of natural images with background noise and content generated by MLLMs, characterized by greater diversity and fewer constraints. Consequently, it is intuitively easier to detect discrepancies between text and corresponding images in text-to-image tasks.

heading: Explore UNIHD to Evaluate Hallucination of
text: Modern MLLMs. We designate UNIHD powered by GPT-4V as the golden detector to assess the frequency of hallucinations in MLLMs, including GPT-4V, and Gemini, among others. The findings illustrated in Figure 8 indicate that (1) GPT-4V exhibits the lowest claim-level hallucination ratio across most tested conditions, and (2) the hallucination-based ranking of these MLLMs is generally in agreement with established leaderboards and human evaluation, demonstrating the potential of UNIHD for evaluating hallucinations.

heading: 6 Related Work

heading: 6.1 Hallucinations in MLLM
text: The advent of MLLMs (OpenAI, 2023; Liu et al., 2023c; Ye et al., 2023; Zhu et al., 2023) has highlighted the issue of hallucination (Hu et al., 2024;
Zhang et al., 2023b; Huang et al., 2023b; Rawte et al., 2023; Ji et al., 2023), a crucial concern impacting their dependability. Previous research has primarily focused on three areas: evaluating (Li et al., 2023b; Liu et al., 2023a; Jing et al., 2023), detecting (Wang et al., 2023c; Yang et al., 2023; Yin et al., 2023), and mitigating hallucinations (Wan et al., 2024; Liu et al., 2023b; Huang et al., 2023c; Semnani et al., 2023; Zhao et al., 2024; Leng et al., 2023; Wang et al., 2024; Deng et al., 2024). In a complementary effort, HaELM (Wang et al., 2023c) scrutinizes the challenges associated with POPE (Li et al., 2023b) and suggests training a model based on simulated hallucination samples for detecting multimodal hallucinations. Diverging from prior efforts, this paper addresses a broader problem scope for hallucination detection, introducing a unified multimodal hallucination detection framework, UNIHD, along with meta-evaluation benchmarks, MHaluBench.

heading: 6.2 Harnessing Tool Resources for LLMs
text: Addressing the limitations of LLMs (Chen, 2023; Kang et al., 2024) due to their pre-training confinement, researchers have explored augmenting them with resources like knowledge bases, search engines, and external models, to expand their functionality. Notably, Schick et al. (2023); Hao et al. (2023); Qiao et al. (2023) have developed models that leverage external tools to improve performance in downstream tasks. More recently, Shen et al. (2023); Liang et al. (2023) has unveiled frameworks integrating LLMs with diverse AI models to tackle complex challenges. Building on this, researchers (Peng et al., 2023; Chen et al., 2023) have examined the utilization of external knowledge to mitigate or evaluate hallucinations in LLMs. Adapting these enhancements for MLLMs introduces unique challenges, necessitating the selection of appropriate tools for effective oversight. Our research focuses on automating the selection of functionally diverse tools to enhance multimodal hallucination detection.

heading: 7 Conclusion
text: We introduce a unified problem formulation for multimodal hallucination detection that encompasses a diverse range of multimodal tasks and hallucination types. A fine-grained benchmark dataset, MHaluBench, is also proposed to promote this challenging direction. Alongside this, we
present the unified hallucination detection framework, UNIHD, capable of autonomously selecting external tools with capturing pertinent knowledge to support hallucination verification with rationales. Our experimental results indicate that UNIHD achieves better performance across both image-to-text and text-to-image generation tasks, confirming its universality and efficacy.

heading: Limitations
text: This paper focuses on constructing a unified hallucination detection framework for MLLMs, dubbed UNIHD. Despite the best efforts, our paper still have some limitations.
The Scope of Multimodal Tasks. This paper primarily addresses the detection of multimodal hallucinations from a unified perspective, with a focus on image-to-text tasks (such as Image Captioning and VQA) and text-to-image generation tasks. Nonetheless, it is important to recognize that our framework does not yet encompass other multimodal tasks, such as video captioning, which are also susceptible to hallucinations. Moving forward, we aim to explore the possibilities of incorporating these additional domains into our UNIHD.

heading: Limitations of Closed-Source MLLM Pricing
text: and Inference Speed. Our UNIHD is primarily built upon powerful closed-source models as the foundation. However, closed-source models (Liu et al., 2023c; Zhu et al., 2023; Ye et al., 2023; Bai et al., 2023) often come with a cost, which introduces operational expenses. Additionally, our UNIHD relies on several external tools to provide evidence for enhanced illusion verification, resulting in additional inference time. In the future, we will further explore training open-source dedicated illusion detection models with the tool to further improve effectiveness and reduce costs.
The Scope of Hallucination Categories. In our commitment to developing a comprehensive hallucination detection framework, referred to as UNIHD, for MLLMs, we have made efforts to incorporate various prevalent hallucination categories within MHaluBench and UNIHD, including object, attribute, scene-text, and factual aspects, among others. However, it is important to acknowledge that there are additional categories of hallucinations that have not been covered in our framework, as discussed in the existing literature (Zhang et al., 2023b; Wang et al., 2023a; Mishra et al., 2024;
Huang et al., 2023b; Rawte et al., 2023) . Moving forward, our research will expand its scope to adopt a unified approach towards a wider range of hallucination categories, to strengthen the robustness of our detection mechanisms.

heading: Preliminary Attempts at Tool Utilization. In
text: our early endeavors, we have configured a dedicated tool for detecting a specific type of hallucination, exemplified by the assignment of the Grounded DINO model as the object detection tool of choice. However, it should be acknowledged that the current selection of tools may not represent the optimum choice. It remains imperative to rigorously explore which SOTA object detection models are best suited for the task of multimodal hallucination detection. This necessitates an extensive evaluation of available models to pinpoint the most effective tool that aligns with the nuances and complexities of detection objectives.

heading: Acknowledgement
text: We are grateful for the API services provided by OpenAI and Google, which enabled us to process data and conduct some of our experiments. Part implementation of this work are assisted and inspired by the related hallucination toolkits including FactTool (Chern et al., 2023), Woodpecker (Yin et al., 2023), and others. We follow the same license for open-sourcing and thank them for their contributions to the community. This work also benefits from the public project of mPLUGOwl6, MiniGPT-47, LLaVA8, GroundingDINO9, and MAERec10. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Yongjiang Talent Introduction Programme (2021A-156-G), and Information Technology Center and State Key Lab of CAD&CG, Zhejiang University. This work was supported by Ant Group and Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph.
6https://github.com/X-PLUG/mPLUG-Owl 7https://github.com/Vision-CAIR/
MiniGPT-4 8https://github.com/haotian-liu/LLaVA 9https://github.com/IDEA-Research/ GroundingDINO 10https://github.com/Mountchicken/ Union14M

heading: A Prompt Templates
text: Within this section, we outline the prompt templates designed to guide the foundational MLLM for the autonomous query formulation (illustrated in Table 3-6) and verification of any hallucinated content (shown in Table 7-8).

heading: USER:

heading: SYSTEM:

heading: USER:

heading: SYSTEM:

heading: USER:

heading: SYSTEM:

heading: USER:

heading: SYSTEM:

heading: USER:

heading: SYSTEM:

heading: Future Work
text: In the future, we will further explore training open-source dedicated illusion detection models with the tool to further improve effectiveness and reduce costs. The Scope of Hallucination Categories. In our commitment to developing a comprehensive hallucination detection framework, referred to as UNIHD, for MLLMs, we have made efforts to incorporate various prevalent hallucination categories within MHaluBench and UNIHD, including object, attribute, scene-text, and factual aspects, among others. However, it is important to acknowledge that there are additional categories of hallucinations that have not been covered in our framework, as discussed in the existing literature (Zhang et al. , 2023b; Wang et al. , 2023a; Mishra et al. , 2024;
Huang et al. , 2023b; Rawte et al. , 2023). Moving forward, our research will expand its scope to adopt a unified approach towards a wider range of hallucination categories, to strengthen the robustness of our detection mechanisms