Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

## Abstract

Abstract Large Vision Language Models (LVLMs) are increasingly integral to healthcare applications, including medical visual question answering and imaging report generation. While these models inherit the robust capabilities of foundational Large Language Models (LLMs), they also inherit susceptibility to hallucinations—a significant concern in high-stakes medical contexts where the margin for error is minimal. However, currently, there are no dedicated methods or benchmarks for hallucination detection and evaluation in the medical field.
To bridge this gap, we introduce Med-HallMark, the first benchmark specifically designed for hallucination detection and evaluation within the medical multimodal domain. This benchmark provides multi-tasking hallucination support, multifaceted hallucination data, and hierarchical hallucination categorization.
Furthermore, we propose the MediHall Score, a new medical evaluative metric designed to assess LVLMs’ hallucinations through a hierarchical scoring system that considers the severity and type of hallucination, thereby enabling a granular assessment of potential clinical impacts.
We also present MediHallDetector, a novel Medical LVLM engineered for precise hallucination detection, which employs multitask training for hallucination detection.
Through extensive experimental evaluations, we establish baselines for popular LVLMs using our benchmark. The findings indicate that MediHall Score provides a more nuanced understanding of hallucination impacts compared to traditional metrics and demonstrate the enhanced performance of MediHallDetector. We hope this work can significantly improve the reliability of LVLMs in medical applications. All resources of this work will be released soon.

## 1 Introduction

Large Vision Language Models (LVLMs) [, , , ] inherit the powerful world knowledge of Large Language Models (LLMs) and integrate visual information, enabling them to tackle complex visual-language tasks. However, they also inherit the common problem of hallucinations, where models generate information that is irrelevant or factually incorrect compared to the input. In LVLMs, the hallucination phenomenon is further exacerbated by the lack of visual feature extraction capability, misalignment of multimodal features, incorporation of additional information, and et.al.

Unlike general applications, where hallucinatory contents are somewhat tolerant, hallucinations in the medical domain can disastrously mislead clinical diagnosis or decision-making. Therefore, mitigating medical hallucinations in LVLMs is of paramount importance. However, compared with the significant concentration given to hallucination problems in the general domain, the medical domain so far does not even have a specific method and benchmark for detecting hallucinations, which severely hinders the development of the medical capacity of LVLMs and results in a scarcity of Medical LVLMs (Med-LVLMs) or LVLMs with great medical competence.

In the current rare series of Med-LVLMs [, , ], they still face two major problems: benchmark and evaluation method. From the benchmark dimension, the evaluation of the medical capabilities of existing LVLMs is still performed on outdated benchmarks [, , , , ]. The problem of data leakage during pre-training of large models makes the results obtained from testing on these traditional benchmarks no longer reliable. Furthermore, these traditional medical datasets either have very short answers or unstructured image reports, making a direct evaluation of LVLM outputs extremely difficult, as the outputs are typically well-ordered long texts.

From the evaluation dimension, traditional NLP task metrics such as METEOR, BLEU and et al.  often fail to directly reflect the factual correctness of a language model’s output, typically measuring only shallow similarities to the ground truth. Accuracy, while indicating whether generated content is correct, evaluates at a coarse semantic level and cannot distinguish between degrees of hallucinations in the output. Existing methods like CHAIR [] and POPE [], designed for general LVLM hallucination evaluation, are limited to ’object hallucinations’ in general domains and cannot accommodate the multi-layered complexities of hallucinations in the medical field. Furthermore, they are often constrained to fixed benchmarks or specific types of questions.

To address these issues and enable researchers to evaluate LVLMs’ medical outputs reasonably, we propose solutions from three dimensions: data, evaluation metrics and detection methods.
Firstly, we introduce a hierarchical categorization of hallucinations specific to the medical domain and develop Med-HallMark, the first benchmark for hallucination detection in medical multimodal fields which provides multi-tasking hallucination support, multifaceted hallucination data, and hierarchical hallucination categorization.
Furthermore, we propose the MediHall Score, a new evaluation metric specifically designed for the medical domain, which calculates the hallucination score of the LVLM outputs through hierarchical categorization, providing an intuitive numerical representation of the rationality of the medical texts.
Finally, we present MediHallDetector, the first multimodal medical hallucination detection model designed to detect hallucinations in model output texts with fine granularity. It employs unique design features and customized training methods to enhance its scalability and hallucination detection capabilities.
We not only provide the baseline performance of the most popular LVLMs with medical capacity on the Med-HallMark but also demonstrate the rationality of the medical multimodal hallucination detection method in this paper as well as the superiority of the hallucination detection model MediHallDetector.

In general, the main contributions of this paper are as follows:
(i) We introduce the first benchmark dedicated to hallucination detection in the medical domain, Med-HallMark, and provide baselines for various LVLMs.
(ii) We propose the first hallucination detection model, MediHallDetector, and demonstrate its superiority through extensive experiments.
(iii) We present a new hallucination evaluation metric, MediHall Score, and show its effectiveness relative to traditional metrics through qualitative and quantitative analysis.

## 2 Related work

Large Vision Language Models Hallucinations.
Although LVLMs have shown significant capabilities on a range of multimodal tasks, they still suffer from inevitable performance bottlenecks due to hallucinatory interference [, ]. The hallucination in LVLMs stands for the generation of hallucinatory descriptions that are inconsistent with relevant images and user instructions, containing incorrect objects, attributes, and relationships related to the visual input, thus significantly limiting the usage of LVLMs. Previous studies have shown that even with the current state-of-the-art (SOTA) LVLMs, at least 30% of the hallucinatory text still exists in the form of nonexistent objects, unfaithful descriptions, and inaccurate relationships [].
The hallucination problem is further exacerbated when LVLMs are applied to medical scenarios, not only because most models lack sufficient medical knowledge, but also because the medical issues are more complex and fine-grained. However, there is no current work that systematically investigates hallucinations in LVLMs in medical scenarios.

Hallucinations Detection and Evaluation.
Hallucination detection [] is an important step in addressing potential hallucination disturbances in LVLMs. Current efforts [, , , , , ] can be categorized into two groups: approaches based on off-the-shelf tools and training-based models. In the former, closed-source LLMs or visual tools can be appropriately used for hallucination assessment.
In contrast, training-based approaches aim to detect hallucinations incrementally from feedback.
However, detection methods dedicated to the medical domain remain unproposed. Some work [, ] has used powerful open-source LLMs such as GPT-API to perform detections based on instruction and model responses; however, such models lack both the appropriate medical domain knowledge, as well as being based on textual evaluation only and lacking image inputs.
Meanwhile, traditional NLP metrics cannot intuitively reflect the factual nature of LVLM responses. Several methods [, , , ] have proposed new hallucination detection metrics for generic scenarios; however, these metrics can only be used to assess a certain category of hallucinations in general domains while being limited to the assessment of fixed benchmarks or fixed types of question, and cannot satisfy the needs of assessing complex types of hallucinations in the medical field. Therefore, the dilemma of detecting and assessing medical hallucinations must first be solved if we want to hallucinate LVLM in medical scenarios.

## 3 Med-HallMark

Figure: Figure 1: Illustration of statistical information and construction content of Med-HallMark. We show separately (a) multi-task hallucination support, (b) multifaceted hallucination data, and (c) hierarchical hallucination categorization.
Refer to caption: x1.png

To investigate domain-specific hallucination dilemmas in medical texts, we present Med-HallMark, the first hallucination detection dataset serving the multi-modal healthcare domain. As shown in Figure 1, Med-HallMark provides a comprehensive hallucination awareness benchmark through three significant characteristics, including multi-task hallucination support, multifaceted hallucination data, and hierarchical hallucination categorization.

### 3.1 Multi-task Hallucination Support

Med-HallMark implements support for the following three key dimensions: medical multimodal task types, hallucination detection formats and multidimensional hallucination detection.
Specifically, Med-HallMark covers two primary medical multimodal task types:
Medical Visual Question Answering (Med-VQA) and Imaging Report Generation (IRG) tasks. The former is Question-Answer (QA) pairs that examine the LVLM’s understanding of an image text from a single fine-grained perspective, and the latter is the instruction pairs that require the LVLM to describe a medical image from a global perspective;
Med-HallMark accommodates different hallucination detection needs by including both open-ended and closed-ended questions. This allows users to leverage metrics such as POPE [] and CHAIR [] for closed-ended question hallucination detection, as well as calculate conventional BertScore and ROUGE metrics for open-ended questions from a global perspective.
Our benchmark also supports hallucination detection across four refined dimensions, as illustrated in Figure 1. The questions can be categorized into four types, conventional medical questions, confidence-weakening questions, counterfactual questions, and image depiction questions (More details in Supplementary material) for users to measure the model’s performance from different aspects.

### 3.2 Multifaceted Hallucination Data

Our benchmark includes multifaceted hallucination data, detailed across three primary dimensions: ground truth (GT), LVLM outputs for prompts and annotations of the LVLM-generated content.
The GT provide a reliable standard against which to evaluate LVLM performance. The LVLM outputs for prompts may be correct or may contain hallucinations.
Med-HallMark includes fine-grained annotations for all LVLM responses, detailing both the type of hallucination and its correctness. In fine-grained single-dimension VQA scenarios, each response is labeled with a single hallucination category. In coarse-grained multi-dimension IRG scenarios, the LVLM outputs are segmented into sentences and annotated at the sentence level. This detailed annotation process allows for a thorough evaluation of model performance across different medical tasks.

### 3.3 Hierarchical Hallucination Categorization

In the realm of general LVLMs, hallucinations are commonly categorized into object hallucinations, attribute hallucinations and relational hallucinations[]. However, this categorization fails to address the unique challenges posed by medical text hallucinations adequately. For instance, in the prompt, "What is the function of the organ on the bottom of this image?", an LVLM might respond, "Its function is to store urine produced by the kidneys." when the ground truth is, "Digest food, absorb water, excrete body waste." In this example, it is challenging to discern whether the LVLM’s error stems from a misidentification of the organ at the bottom of the image or a fundamental misunderstanding of stomach function. Therefore, the traditional categorization of hallucinations is neither suitable for complex problem scenarios nor for medical text hallucinations.

To address this gap, we propose a novel hierarchical method specifically designed for medical text hallucinations, which classifies hallucinations based on the severity of their impact on clinical diagnosis or decision-making, as illustrated in Figure 1. We finely categorize the sentence-level hallucination outputs of LVLMs into five distinct levels:

Catastrophic Hallucinations: These involve grossly incorrect judgments, such as misjudging the global health status of the image, misidentifying organs, fabricating organs, fabricating pathologies or lesions on "normal" images, or making incorrect descriptions of the image based on previous errors.

Critical Hallucinations: These generally involve incorrect descriptions of organ functions or pathological categories, fabricating "other types of lesions" on "abnormal" images, resulting in "misanalyses" or "omissions", and incorrect descriptions of the causes of pathologies.

Attribute Hallucinations: These manifest as incorrect judgments or descriptions of the size, shape, location, and number of organs and pathologies and affect diagnostic accuracy to some extent.

Prompt-induced Hallucinations: These hallucinations are induced by prompts containing confused information, often arising from a lack of plausibility or factuality in the prompt and testing the model’s robustness in specific contexts.

Minor Hallucinations: These are often related to judgments about the modality of medical images and how they are collected, which do not seriously affect clinical diagnosis and treatments.

### 3.4 Construction of the Benchmark

Based on the aforementioned characteristics and hierarchical categorization method, we constructed Med-HallMark, as illustrated in Figure 1. To avoid data leakage and privacy issues, the medical images of Med-HallMark are derived from four test datasets, Slake [] and VQA-RAD [] for the Med-VQA task, and MIMIC [] and OpenI [] with clinical reports.

For images from the VQA dataset, we designed a series of questions based on six dimensions: modality, plane, shape, size, organ, location, and pathology, following the methods proposed in []. These original questions are considered conventional medical questions $Q_{Conv}$. We then used the (SOTA) model, LLaVA-Med-pretrained [], to infer answers to $Q_{Conv}$. The generated answers $A_{Conv}$ were manually evaluated. Correct answers $A_{Conv}^{true}$ were directly used as ground truth, while incorrect answers $A_{Conv}^{false}$ were treated as hallucinatory outputs and their ground truths were re-annotated to expedite the annotation process. Incorrect questions $Q_{Conv}^{false}$ were annotated with the hallucination categories introduced in subsection 3.3. To quickly expand the dataset, we utilized the GPT-3.5 API to rewrite $Q_{Conv}$ questions, ensuring the questions’ form did not alter the ground truth, resulting in $Q_{Conv}^{true^{\prime}}$ and $Q_{Conv}^{false^{\prime}}$.

Subsequently, we constructed confidence-weakening questions and counterfactual questions based on $Q_{Conv}$. For confidence-weakening questions, we designed ten prefixes to weaken the model’s confidence and randomly combined them with $Q_{Conv}^{true^{\prime}}$ to create $Q_{Inconfi}$, with the same ground truths as the corresponding $Q_{Conv}^{true^{\prime}}$. For counterfactual questions, we used GPT-4 to generate counterfactual questions $Q_{Counter}$ and their corresponding ground truths based on $Q_{Conv}^{true}$ and its ground truth. The expansion method used for $Q_{Conv}$ was applied to $Q_{Counter}$, resulting in $Q_{Counter^{\prime}}$. Then LLaVA-Med was used to infer answers to the confidence-weakening questions and counterfactual questions and manually annotated by humans.

In the IRG scenario, we sampled 1800 images and their corresponding medical reports from the MIMIC-test and OpenI datasets (sampling methods are detailed in the Supplementary material). To generate concise, de-identified medical reports containing complete key information as the dataset’s ground truth, we processed the medical reports by removing sentences comparing the patient’s previous medical history to ensure the accuracy of independent data. We also de-identified the reports by removing patient-specific, doctor-specific, and visit-specific information. After cleaning the data, we concatenated the Findings and Impressions sections of the reports to form the ground truth for the medical image report generation task.

Following this process, we successfully construct Med-HallMark. The data volume is depicted in Figure 1. Specifically, the data percentages on the Med-VQA and IRG tasks are 57.57% and 42.43%, respectively. Across the data samples, the data partitions for different types of questions are 2871 samples for Conventional Medical; 1350 samples for Confidence-weakening; 1320 samples for Counterfactual, and 1800 samples for Image Depiction.

## 4 MediHall Score

This fine-grained metric MediHall Score is based on the hierarchical categorization of medical text hallucinations introduced in subsection 3.3, which evaluates hallucinations at a fine-grained level and considers two different dimensions of the evaluation scenarios: Med-VQA tasks and IRG tasks.

For Med-VQA tasks, where the LVLM centres on the question and the response is from a fine-grained dimension, the MediHall Score assesses the entire answer to determine the hallucination category and calculates the corresponding score. In IRG tasks, the LVLM’s response typically encompasses descriptions from multiple dimensions, with each sentence potentially containing different types of hallucinations. As illustrated in Figure 1, the MediHall Score evaluates hallucinations at the sentence level and aggregates these to compute the overall score for the entire response.
Hallucination scores are assigned based on the identified category: Catastrophic Hallucinations ($H_{c}=0.0$), Critical Hallucinations ($H_{cr}=0.2$), Attribute Hallucinations ($H_{a}=0.4$), Prompt-induced Hallucinations ($H_{p}=0.6$), Minor Hallucinations ($H_{m}=0.8$), and Correct Statements ($H_{s}=1.0$).

For fine-grained Med-VQA scenarios, the hallucination category and corresponding score $H_{i}$ are determined for each answer. The MediHall Score for a single answer is thus $H_{answer}=H_{i}$.

In coarse-grained IRG scenarios, the score is calculated by averaging the hallucination scores of all sentences within a report. Let $n$ be the number of sentences and $H_{j}$ be the hallucination score for sentence $j$. The score for an individual report $H_{report}$ is given by: $H_{report}=\frac{1}{n}\sum_{j=1}^{n}H_{j}$.
The overall MediHall Score for a set of instructions is derived by averaging the scores across all $k$ reports or answers: $H_{overall}=\frac{1}{k}\sum_{i=1}^{k}H_{i}$.

## 5 MediHallDetector

Figure: Figure 2: Visualization of MediHalldetector related information. (a) Model structure, SFT process and inference objective of MediHalldetector. (b) Examples of questions, LVLM answers and $GT$ for different types of tasks. (c) Comparison of three rounds of evaluation agreement and average inference time for different evaluation models. (d) Comparison of different evaluation models’ agreement with human evaluation preferences in different hallucination texts.
Refer to caption: x2.png

Destination Setup: We require the MediHallDetector $M$ to be able to classify its hallucination levels based on the input image $I$, the original prompt $P$, the LVLM answer $A$ and the ground truth $GT$, which can be denoted as $H_{type}\xleftarrow{}M(I,P,A,GT)$. This destination brings about four requirements for the model: (1). It must accurately differentiate between various levels of hallucinations. (2). It should follow instructions correctly. (3). It must categorize hallucinations in LVLM outputs based on the medical images. (4). The model must maintain flexibility to adapt to different scenarios. These requirements guided the design of the MediHallDetector.

Framework of MediHallDetector: MediHallDetector is built upon the fundamental architecture of the LLaVA model, incorporating a dual-layer fully connected network with GELU activation functions as the connector. The initial weights for MediHallDetector were taken from LLaVA1.5-7B[] to leverage its robust foundational capabilities.

Data Preparation: The training data for MediHallDetector consists of three main categories: traditional medical image-text task data, Med-HallMark data, and specific hallucination evaluation instruction pair data. Traditional medical image-text task data, sourced from SLAKE, VQA-RAD, MIMIC-Test and OpenI datasets shown in Figure 1, helps adapt the general LVLM to the medical domain. Med-HallMark data improves the model’s robustness and prevents the assessment model from being induced to produce hallucinations by the hallucination output itself due to the problem of not having seen the source domain. The hallucination evaluation data, manually annotated medical text described in Section 3, helps the model understand different types of hallucinations and follow the instructions.

Training: MediHallDetector undergoes a single-stage supervised fine-tuning (SFT) process using the combined data described above. During fine-tuning, we froze the visual encoder and connector, and train the full parameters of the MediHallDetector’s language model. The model is trained with a learning rate of 2e-5, utilizing the AdamW optimizer for two epochs. Detailed training parameters and additional settings are provided in the Supplementary material.

## 6 Experiment and Discussion

**Table 1: Comparison results of the different models on the VQA and IRG tasks in Med-HallMark by various evaluation metrics. “R-1/2/L” means the ROUGE-1/2/L. “SF”, “RF”, and “PF” stand for Slake-Finetuned, Rad-Finetuned, and Pathvqa-Finetuned, respectively. “BS” denotes the BertScore. “MR” and “ACC” mean the METEOR and Accuracy, respectively.**
| Model | Med-VQA Task | IRG Task |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BS | MR | R-1 | R-2 | R-L | BLEU | MediHall<br>Score | ACC | BS | MR | R-1 | R-2 | R-L | BLEU | MediHall<br>Score |  |
| BLIP2 | 47.97 | 16.15 | 18.98 | 6.03 | 17.13 | 3.46 | 0.52 | 0.27 | 33.05 | 4.88 | 10.17 | 1.49 | 7.66 | 0.15 | – |
| InstructBLIP-7b | 35.99 | 7.47 | 6.08 | 0.59 | 5.30 | 1.03 | 0.58 | 0.34 | 47.49 | 13.98 | 17.56 | 2.31 | 13.60 | 0.73 | 0.84 |
| InstructBLIP-13b | 36.02 | 7.59 | 6.13 | 0.58 | 5.32 | 1.00 | 0.58 | 0.34 | 47.47 | 13.93 | 17.54 | 2.34 | 13.61 | 0.74 | 0.84 |
| LLaVA1.5-7b | 54.89 | 28.33 | 23.52 | 9.30 | 21.16 | 4.60 | 0.54 | 0.28 | 47.93 | 11.24 | 18.77 | 2.78 | 14.64 | 0.72 | 0.79 |
| LLaVA1.5-13b | 52.82 | 25.98 | 21.52 | 8.20 | 19.38 | 4.18 | 0.51 | 0.23 | 47.96 | 11.80 | 18.35 | 2.40 | 14.49 | 0.67 | 0.81 |
| LLaVA-Med(SF) | 36.67 | 8.80 | 91.17 | 1.40 | 9.10 | 0.03 | 0.61 | 0.39 | 30.49 | 0.61 | 0.27 | 0.00 | 0.27 | 0.14 | – |
| LLaVA-Med(RF) | 35.25 | 6.91 | 6.34 | 1.49 | 6.09 | 0.54 | 0.57 | 0.32 | 33.28 | 2.33 | 2.73 | 0.32 | 2.45 | 0.02 | – |
| LLaVA-Med(PF) | 33.32 | 3.27 | 2.85 | 0.58 | 2.68 | 0.06 | 0.59 | 0.35 | 37.67 | 1.25 | 2.24 | 0.05 | 2.06 | 0.09 | – |
| mPLUG-Owl2 | 55.11 | 29.39 | 22.25 | 8.38 | 19.77 | 3.43 | 0.50 | 0.24 | 64.49 | 40.11 | 32.00 | 13.84 | 28.50 | 6.79 | 0.81 |
| XrayGPT | 44.40 | 14.00 | 10.66 | 1.17 | 9.89 | 0.37 | 0.36 | 0.02 | 62.62 | 25.96 | 27.94 | 6.59 | 22.15 | 3.26 | 0.79 |
| MiniGPT4 | 42.93 | 12.93 | 11.14 | 1.60 | 10.19 | 0.39 | 0.38 | 0.09 | 46.43 | 10.27 | 15.37 | 1.75 | 12.63 | 0.53 | 0.88 |
| RadFM | 43.84 | 11.81 | 11.31 | 2.16 | 10.68 | 1.55 | 0.56 | 0.32 | 41.18 | 3.41 | 5.88 | 0.76 | 4.79 | 0.05 | – |

### 6.1 Baseline Results on the Med-HallMark

To systematically investigate the performance of different models on Med-HallMark across different tasks regarding hallucination detection, we report the proposed MediHall Score and diverse traditional metrics, including BertScore, METEOR, ROUGE-1/2/L, and BLEU.
In Table 1, extensive baselines are provided including BLIP2 [], InstructBLIP-7b/13b [], LLaVA1.5-7b/13b [], mPLUG-Owl2 [], XrayGPT [], MiniGPT4 [], RadFM [].
and LLaVA-Med [] fine-tuned on the Slake (SF), Rad (RF), and Pathvqa (PF) datasets.

Regarding the traditional metrics evaluation, generated responses from most models on the Med-VQA task have relatively low word-level coverage between the responses and the GTs, exhibiting poor content consistency.
For instance, the BLIP family has an average score of only 7.35% on the ROUGE metric. Meanwhile, InstructBLIP-7b and InstructBLIP-13b achieve even worse results on ROUGE-2 with 0.59% and 0.58%, respectively, indicating that the generated content matches poorly with the reference content in terms of vocabulary, word sequences, and overall structure.

In comparison, LLaVA1.5-7b/13b and mPLUG-Owl2 exhibit appreciable precision that is reflected in the METEOR and BLEU metrics.
Specifically, LLaVA1.5-7b/13b obtains scores of 27.16% and 4.39% on the METEOR and BLEU metrics, respectively. mPLUG-Owl2 achieves the best result of 29.39% on the METEOR metrics, demonstrating that the generated content matches the GTs semantically and structurally more than the other baselines.

On the IGR task, mPLUG-Owl2 outperforms the other baseline models on most conventional metrics.
LLaVA-Meds yields the worst result. The intuitive examples are reflected in the ROUGE-1/2/L with scores close to 0%. A plausible explanation is that LLaVA-Med is full-parameters fine-tuned on the Q&A dataset with short answers, and its inability to perform the IRG task, which requires the giving of long contextual responses.
BertScore is considered to be in favourable agreement with human judgment compared to other metrics. In this case, the scores of 64.49% and 62.62% for mPLUG-Owl2 and XrayGPT, respectively. In contrast, BLIP and LLaVA1.5 families achieve comparable performance due to the BertScore of around 47%.

On the MediHall Score metric for assessing hallucinations, we perform corresponding analyses based on different tasks. Specifically, the LLaVA-Med series achieves higher results on the Med-VQA task, which arrive at 0.61, 0.57, and 0.69, respectively, suggesting the production of more correct content and fewer hallucinatory states.

On the IRG task, the LLaVA-Med series, BLIP2, and RadFM cannot produce a computable MediHall Score since the generation format of these models is not suitable for reporting generation scenarios with contextual reasoning properties. Furthermore, MiniGPT4 exhibits favorable performance with the highest score of 0.88.

Figure: Figure 3: Comparison of different Models on hallucination types.
Refer to caption: extracted/5668127/Styles/radar_chart.png

Analysis of six-dimensional hallucination level: Figure 3 provides results on the comparison of the performance of different baselines on distinct hallucination categories. We show the statistics of the presence of hallucinated sentences in the generated responses across baseline models on the Med-VQA task, aiming to fine-grained present the baseline models’ multi-dimension medical competence. The core observations are as follows.

The boundary between catastrophic hallucinations and correct statements clearly differentiates the capabilities of various LVLMs. MiniGPT4 is the worst-performing model, exhibiting extreme tendencies towards both catastrophic hallucinations and correct statements.
All models have insignificant differences in Attribute Hallucinations, and these baselines have similar error boundaries, demonstrating that it’s difficult for LVLMs to correctly judge or describe the size, shape, or number of organs and pathologies.
In the case of prompt-induced hallucinations, primarily caused by counterfactual questions, nearly all models show prompt-induced hallucinations close to or even exceeding the number of catastrophic hallucinations. This indicates that counterfactual questions are effective in challenging the models, revealing that LVLMs are highly vulnerable to such attacks. It also suggests that most LVLMs fail to fully understand medical images from all dimensions, often ignoring information in the questions that is irrelevant to the image facts.

**Table 2: Comparison of different models in Med-HallMark on $Q_{Conv}^{true^{\prime}}$ (‘$\clubsuit$’) and $Q_{Inconfi}$ (‘$\spadesuit$’).**
| Model | Type | BertScore | METEOR | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | MediHall Score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLIP2 | $\clubsuit$ | 50.97 | 19.63 | 24.18 | 8.45 | 21.75 | 2.23 | 0.59 |
| $\spadesuit$ | 52.73 | 21.51 | 26.47 | 10.01 | 24.09 | 3.06 | 0.60 |  |
| LLaVA1.5-13b | $\clubsuit$ | 62.89 | 36.12 | 31.52 | 12.96 | 28.44 | 6.49 | 0.57 |
| $\spadesuit$ | 59.05 | 32.84 | 28.28 | 12.36 | 25.62 | 7.19 | 0.58 |  |
| LLaVA-Med (SF) | $\clubsuit$ | 30.23 | 2.26 | 0.42 | 0.11 | 0.42 | 0 | 0.72 |
| $\spadesuit$ | 27.79 | 2.19 | 0.46 | 0.15 | 0.45 | 0 | 0.74 |  |
| XrayGPT | $\clubsuit$ | 49.16 | 18.87 | 12.80 | 1.44 | 11.85 | 0.37 | 0.34 |
| $\spadesuit$ | 49.29 | 18.8 | 12.89 | 1.51 | 11.97 | 0.38 | 0.35 |  |

Confidence prefix’s impacts on LVLMs: Further, we show in Table 2 the performance comparison between different baselines on traditional correct and confidence-weakening questions by diverse metrics. Four different series of models are selected, including BLIP2, LLaVA1.5-13b, LLaVA-Med (SF), and XrayGPT. On the Confidence-weakening question, the responses generated by BLIP2 and XrayGPT are more discreet and accurate, suggesting that the model robustness across different metrics is incrementally enhanced in the hallucination awareness scenario. In contrast, LLaVA1.5-13b and LLaVA-Med (SF) were consistently degraded in performance across most metrics in the weakened confidence scenario.

### 6.2 Comparison between different Metrics

Table 3 quantitatively illustrates the strengths and weaknesses of different metrics when evaluating LVLM responses. In various question types, LVLM responses may be correct or exhibit hallucinations. As shown in Figure 2(b), regardless of whether the LVLM’s answer $A$ aligns with the ground truth $GT$, the METEOR metric shown in Table 3 fails to directly reflect this alignment.

**Table 3: Comparison between traditional Metrics and MediHall Score. Each metric corresponds to QA pairs with the same ID in Figure 2(b)**
| ID | Accuracy | BertScore | METEOR | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | MediHall Score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 1 | 66.73 | 26.04 | 44.44 | 24.99 | 44.44 | 8.36 | 1.0 |
| 2 | 1 | 46.11 | 49.02 | 0 | 0 | 0 | 0 | 1.0 |
| 3 | 0 | 51.48 | 24.22 | 30.91 | 3.54 | 19.99 | 3.18 | 0.6 |
| 4 | 0 | 66.98 | 40.27 | 33.33 | 10.2 | 30.95 | 5.97 | 0.4 |

ROUGE score evaluates the presence of matching n-grams or subsequences between two texts. ROUGE-1/2/L matches single words, bigrams, and the longest common subsequence, respectively. Although this can detect some content alignment between $A$ and $GT$, it is prone to extreme cases. For example, in the confidence-weakening question shown in Figure 2(b), the model correctly identifies the largest organ in the image as the lung. However, the $A$ "Lung." fails to match "lung" in $GT$ due to the punctuation, resulting in a failed direct match. Additionally, because $A$ contains only one word, ROUGE-2/L cannot be computed. BLEU has similar issues; without any shared n-grams or subsequences between $A$ and GT, BLEU also scores zero. While BLEU does account for significant length differences between A and GT, making it somewhat more versatile than ROUGE, it still weakly measures factual correctness.

BertScore mitigates some of the shortcomings of ROUGE and BLEU but still does not intuitively reflect the factual accuracy or degree of hallucination in medical texts. In ID1/2 where the LVLM answers are entirely correct, the BertScore (%) is 66.73 and 46.11, respectively, indicating a significant and unwarranted disparity.
In contrast, ACC directly shows the correctness of $A$ but is only suitable for evaluating short, fine-grained texts. When $A$ includes complex, multi-dimensional content or belongs to long texts, ACC fails to intuitively measure the LVLM output. For instance, in counterfactual questions, while the LVLM recognizes that the image is a chest X-ray, it does not explicitly state the absence of kidneys, failing to address all aspects of the question. Hence, evaluating the correctness of such a response solely based on ACC is inadequate. In IRG tasks, where the model needs to analyze image content from various dimensions, mere correctness does not capture the model’s judgment of factuality across all dimensions.

In comparison, MediHall Score calculates metrics based on hallucination detection models that classify hallucination levels according to image facts and textual annotations. The calculation method varies according to the requirements of different scenarios. Under conventional and confidence-weakening questions, if $A$ aligns perfectly with the facts, the MediHall Score is $1.0$. For counterfactual questions, the model’s response is affected by the inherent confusion of the question. Thus, even though the answer is not comprehensive, it is categorized as a prompt-induced hallucination, yielding a MediHall Score of 0.6. In the multi-dimensional IRG scenario, MediHall Score evaluates sentence-level hallucinations and aggregates these scores to derive the final score for the response. More examples are provided in the supplementary materials.

### 6.3 Evaluations of the MediHallDetector’s Performance

**Table 4: Ablation studies of different SFT methods.**
| Method | Catas-Hallucinations | Criti-Hallucinations | Attr-Hallucinations | Promp-Hallucinations | Minor Hallucinations | Correct Statements |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC | Recall | ACC | Recall | ACC | Recall | ACC | Recall | ACC | Recall | ACC | Recall |  |
| a | 0.87 | 0.28 | 0.14 | 0.01 | 0.04 | 0.02 | 0.10 | 0.06 | 7.14 | 2.07 | 53.41 | 28.33 |
| b | 34.33 | 15.13 | 42.86 | 1.91 | 17.78 | 5.84 | 12.33 | 6.52 | 0.07 | 0.01 | 0.65 | 0.29 |
| c | 55.22 | 37.37 | 0.00 | 0.00 | 48.89 | 25.00 | 100.00 | 44.24 | 7.14 | 1.28 | 78.41 | 48.59 |
| d | 58.21 | 17.26 | 57.14 | 2.06 | 0.00 | 0.00 | 1.37 | 0.58 | 0.00 | 0.00 | 9.09 | 5.52 |
| e | 76.12 | 45.95 | 71.43 | 7.25 | 53.33 | 31.17 | 98.63 | 48.98 | 42.86 | 8.33 | 68.18 | 55.56 |
| f | 71.64 | 42.48 | 71.43 | 6.49 | 44.44 | 25.64 | 97.26 | 46.10 | 50.00 | 8.43 | 65.91 | 51.33 |
| ours | 83.58 | 54.37 | 85.71 | 11.54 | 68.89 | 43.66 | 98.63 | 55.81 | 64.29 | 15.52 | 70.45 | 65.96 |

To demonstrate the superiority of MediHallDetector as a hallucination detection model, we uniformly sampled 300 inferences from InstructBLIP-13b, LLaVA1.5-13b, and mPLUG-Owl2 across the four tasks in Med-HallMark. These samples were manually annotated for hallucination types following the construction methods outlined in Section 5, forming a test set representative of human preferences for hallucination detection models.

Comparison of different hallucination detection models: We compared our model’s performance with two of the most powerful LLMs: GPT-3.5 and GPT-4 in the context of medical hallucination hierarchy. As depicted in the radar chart in Figure 2(d), the chart illustrates the alignment of detection models with human preferences across six different hallucination levels.
GPT-3.5 predominantly classified responses as Critical and Attribute hallucinations, and only correctly identified approximately 1.14% of these instances, indicating a poor understanding of the nuanced definitions of different hallucination levels and an inability to reasonably assess the correctness of $A$ in more complex hallucination classification scenarios.
While GPT-4 performed better at detecting the correctness of $A$ in complex prompts, they still struggled with the hierarchical categorization of hallucinations. This was particularly evident in distinguishing prompt-induced hallucinations.

Figure 2(c) compares MedHallDetector with GPT-3.5 and GPT-4 across two dimensions: evaluation time and multi-round evaluation consistency. The results show that GPT models tend to produce varying conclusions in each evaluation, whereas MedHallDetector maintains a high consistency rate of 93.88% across three evaluation rounds. Additionally, MedHallDetector achieves the shortest average inference time per evaluation.

Ablation studies of different SFT methods:
We conducted ablation studies to evaluate the performance of MediHallDetectors SFT with different strategies, comparing their ACC (%) and Recall (%) against human preferences across various hallucination categories. The results are summarized in Table 4. In this study, a-f represent the following configurations: baseline model, fine-tuning only the connector, using only traditional task data, using only instruction data, using only MedihallMark data, excluding traditional task data, and models fine-tuned with different data at each of three stages, respectively.
From these experiments, we derived the following insights:
(1). When fine-tuning only the connector with all data, the model fails to adapt effectively to the medical domain.
(2). Performing SFT in a single phase with data from three different tasks allows each type of training data to effectively contribute, leading to incremental improvements in MediHallDetector’s performance.
(3). Sequentially using different task data for SFT across multiple stages is unnecessary. Instead, mixing different task data in a single SFT phase maximizes the performance enhancement of MediHallDetector.

## 7 Conclusion

In this paper, we address the challenges of hallucination detection and evaluation in the application of LVLMs in healthcare. We propose a novel benchmark, evaluation metrics, and detection model specifically designed for the medical domain. In addition to establishing baselines for current mainstream models on the benchmark, we demonstrate the effectiveness of our metrics and model in hallucination evaluation and detection through extensive experimental analysis. We hope this work can significantly improve the reliability of LVLMs in medical applications.

## Appendix A Details of the Proposed Benchmark

### A.1 Details of Med-HallMark

Conventional Medical Questions:
In this paper, conventional medical questions refer to the original questions we constructed based on the original medical image constructs. Conventional questions ask information about the image from a single dimension and are usually clearly expressed without any confrontational or model-confusing vocabulary, such as those illustrated in Figure 2(b) of the main paper as well as in Figure 10.

Confidence-weakening Questions:
Confidence-weakening questions are constructed based on $Q_{true^{\prime}}$, as introduced in Section 3.4 of the main paper. The purpose of these questions is to add prefixes that reduce the model’s confidence, encouraging it to respond more cautiously and thus test its inferential abilities under conditions of uncertainty. We manually created 10 different confidence-weakening prefixes, as illustrated in Figure 4. Each confidence-weakening question is formed by randomly combining $Q_{true^{\prime}}$ with one of these prefixes.

Counterfactual Questions:
Counterfactual questions present prompts that typically include premises contrary to the factual content of the image. These counterfactual premises often involve incorrect descriptions of various dimensions of a medical image, such as shape, size, location, number, or pathology. The questions are then based on these incorrect descriptions. As illustrated in Figure 2(b) of the main paper and Figure 12, counterfactual questions can be understood as adversarial descriptions designed to test whether the model can detect discrepancies between the prompt and the fundamental facts depicted in the image. This type of question evaluates the model’s ability to search for image features based on textual information and determine whether the model truly understands the image rather than merely fitting the training data domain.

In this study, we introduce a method for rapidly constructing counterfactual question-answer pairs. Specifically, we leverage GPT-4’s in-context learning capabilities to generate these pairs without using image information. By incorporating the original questions and their correct answers into instructions, as shown in the prompt context in Figure 5, GPT-4 automatically generates counterfactual question-answer pairs. These pairs, denoted as $Q_{counter}$ and $GT_{counter}$, are subsequently reviewed and verified by human annotators to ensure accuracy.

Image Depiction Questions (IRG):
The medical image report generation (IRG) task can be regarded as a type of image depiction question. In this study, we employ five different manually designed instructions to guide the model in performing IRG tasks, as illustrated in Figure 7. These instructions are crafted to ensure that the model provides comprehensive descriptions of the medical images.

For image sampling, in the original dataset with the medical reports, many of the images were of low quality, resulting in a report that could not accurately describe the corresponding medical images in detail. Therefore, we performed the following image sampling method in this dataset: (1) Removing images with lower resolution; (2) Removing images with smaller image sizes; (3) Selecting 1,800 high-quality data evenly from different patients in different years to avoid the source of the data being concentrated within a particular year.

Figure: Figure 4: Prefix of confidence-weakening questions.
Refer to caption: x3.png

Text Augmentation:
As described in Section 3.4 of the main paper, to rapidly expand the benchmark, we utilized the GPT-3.5 API to augment the original questions of various categories that we constructed. The prompts used for this process are shown in Figure 8. All augmented questions were manually reviewed to ensure that their original meanings were preserved and that the ground truth (GT) still correctly corresponded to the questions.

More Explanations of Hierarchical Hallucination Categorization:
As described in Section 3.3 of the main paper, we categorize hallucinations into five different levels. To facilitate a better understanding of these hallucination levels, Figure 9 illustrates examples of outputs categorized as Catastrophic Hallucination, Critical Hallucination, Attribute Hallucination, and Minor Hallucination. Additionally, Figure 12 presents examples of outputs classified as Prompt-induced Hallucination. These visualizations help readers comprehend the distinctions between different types of hallucinations.

Figure: Figure 5: Prompts for GPT-4 to create counterfactual questions.
Refer to caption: x4.png

Figure: Figure 6: Instructions for MedHallDetector to SFT and inferencing on the VQA and IRG tasks.
Refer to caption: x5.png

Figure: Figure 7: Instructions for baseline model to inference on the IRG task.
Refer to caption: x6.png

Figure: Figure 8: Prompts for GPT-3.5 to expand origin questions.
Refer to caption: x7.png

Figure: Figure 9: Examples of output text at different hallucination levels (Catastrophic Hallucination, Critical Hallucination, Attribute Hallucination, and Minor Hallucination).
Refer to caption: x8.png

Figure: Figure 10: Responses from different models on conventional questions.
Refer to caption: x9.png

Figure: Figure 11: Responses from different models on confidence-weakening questions.
Refer to caption: x10.png

Figure: Figure 12: Responses from different models on counterfactual questions.
Refer to caption: x11.png

Figure: Figure 13: Responses from different models on image depiction questions (IRG).
Refer to caption: x12.png

Figure: Figure 14: Visualization of MedHallDetector and other powerful LVLMS on multimodal hallucination detection.
Refer to caption: x13.png

**Table 5: Detailed Results of the baseline models’ performance on the VQA tasks.**
| model | Bertscore | METEOR<br>score | Rouge-1 | Rouge-2 | Rouge-L |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Recall | Precision | F1 | Recall | Precision | F1 | Recall | Precision | F1 | Recall | Precision | F1 |  |  |
| BLIP2 | 46.86 | 50.94 | 47.97 | 16.15 | 16.21 | 28.52 | 18.98 | 5.14 | 8.83 | 6.03 | 14.61 | 25.95 | 17.13 |
| InstructBLIP-7b | 36.76 | 37.06 | 36.00 | 7.47 | 8.85 | 7.80 | 6.08 | 0.84 | 0.65 | 0.59 | 7.58 | 7.17 | 5.30 |
| InstructBLIP-13b | 36.84 | 37.04 | 36.02 | 7.60 | 9.02 | 7.75 | 6.13 | 0.84 | 0.66 | 0.58 | 7.71 | 7.10 | 5.32 |
| LLaVA1.5-7b | 59.16 | 52.21 | 54.89 | 28.33 | 31.57 | 21.66 | 23.52 | 12.84 | 8.70 | 9.30 | 28.49 | 19.50 | 21.16 |
| LLaVA1.5-13b | 57.15 | 50.26 | 52.82 | 25.98 | 29.74 | 19.79 | 21.52 | 11.79 | 7.64 | 8.20 | 26.87 | 17.83 | 19.38 |
| LLaVA-Med (SF) | 34.34 | 41.48 | 36.67 | 8.80 | 10.60 | 10.16 | 9.17 | 1.75 | 1.60 | 1.40 | 10.53 | 10.06 | 9.10 |
| LLaVA-Med (RF) | 35.07 | 37.80 | 35.25 | 6.91 | 8.07 | 9.85 | 6.34 | 1.82 | 2.29 | 1.49 | 7.82 | 9.52 | 6.09 |
| LLaVA-Med (PF) | 31.59 | 36.55 | 33.32 | 3.27 | 2.24 | 8.64 | 2.85 | 0.45 | 1.83 | 0.58 | 2.08 | 8.32 | 2.68 |
| mPLUG-Owl2 | 60.31 | 51.74 | 55.11 | 29.39 | 33.20 | 19.83 | 22.25 | 12.57 | 7.75 | 8.38 | 29.63 | 17.62 | 19.77 |
| XrayGPT | 50.87 | 39.97 | 44.41 | 14.00 | 20.49 | 8.66 | 10.66 | 2.50 | 0.91 | 1.17 | 19.17 | 8.04 | 9.89 |
| Mini-gpt4 | 48.51 | 39.35 | 42.93 | 12.93 | 22.31 | 9.23 | 11.14 | 4.25 | 1.24 | 1.60 | 20.68 | 8.42 | 10.19 |
| RadFM | 42.58 | 47.09 | 43.84 | 11.81 | 11.40 | 14.07 | 11.31 | 2.19 | 2.74 | 2.16 | 10.78 | 13.33 | 10.68 |

**Table 6: Detailed Results of the baseline models’ performance on Counterfactual Questions.**
| model | Bertscore | METEOR<br>score | Rouge-1 | Rouge-2 | Rouge-L |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Recall | Precision | F1 | Recall | Precision | F1 | Recall | Precision | F1 | Recall | Precision | F1 |  |  |
| BLIP2 | 41.98 | 48.83 | 44.98 | 14.90 | 15.79 | 35.24 | 20.94 | 3.61 | 7.87 | 4.68 | 13.77 | 30.96 | 18.29 |
| InstructBLIP-7b | 52.42 | 46.86 | 49.04 | 19.71 | 28.59 | 15.52 | 18.05 | 3.30 | 2.23 | 2.38 | 23.02 | 12.77 | 14.60 |
| InstructBLIP-13b | 52.63 | 46.77 | 49.14 | 20.10 | 29.10 | 14.96 | 18.06 | 3.29 | 2.14 | 2.31 | 23.28 | 12.13 | 14.46 |
| LLaVA1.5-7b | 57.81 | 54.21 | 55.86 | 28.27 | 29.66 | 21.84 | 24.66 | 8.17 | 5.48 | 6.39 | 26.24 | 19.26 | 21.80 |
| LLaVA1.5-13b | 59.59 | 54.88 | 57.08 | 30.66 | 31.86 | 21.99 | 25.73 | 9.04 | 5.67 | 6.87 | 28.22 | 19.48 | 22.80 |
| LLaVA-Med (SF) | 58.91 | 54.63 | 55.85 | 26.43 | 38.52 | 30.89 | 32.48 | 6.07 | 4.20 | 4.60 | 38.52 | 30.89 | 32.48 |
| LLaVA-Med (RF) | 31.23 | 38.29 | 33.90 | 5.91 | 5.57 | 12.92 | 6.22 | 1.23 | 3.54 | 1.40 | 5.03 | 12.14 | 5.66 |
| LLaVA-Med (PF) | 34.16 | 44.48 | 38.16 | 6.27 | 6.34 | 23.80 | 7.88 | 1.51 | 6.15 | 1.88 | 5.81 | 22.64 | 7.30 |
| mPLUG-Owl2 | 61.52 | 53.07 | 56.88 | 30.78 | 37.13 | 17.11 | 22.69 | 10.33 | 4.19 | 5.66 | 33.17 | 15.24 | 20.23 |
| XrayGPT | 52.49 | 49.77 | 51.02 | 14.81 | 18.24 | 16.93 | 17.02 | 2.02 | 1.69 | 1.77 | 16.91 | 15.71 | 15.79 |
| Mini-gpt4 | 50.24 | 49.36 | 49.63 | 14.43 | 21.10 | 18.01 | 17.94 | 2.80 | 2.04 | 2.12 | 19.37 | 16.47 | 16.43 |
| RadFM | 46.28 | 49.58 | 47.51 | 13.77 | 14.41 | 19.61 | 14.90 | 2.47 | 3.54 | 2.54 | 12.87 | 17.68 | 13.31 |

**Table 7: Detailed Results of the baseline models’ performance on the IRG task.**
| model | Bertscore | METEOR<br>score | Rouge-1 | Rouge-2 | Rouge-L |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Recall | Precision | F1 | Recall | Precision | F1 | Recall | Precision | F1 | Recall | Precision | F1 |  |  |
| InstructBLIP-7b | 49.91 | 45.39 | 47.49 | 13.98 | 18.18 | 18.00 | 17.56 | 2.52 | 2.28 | 2.31 | 14.13 | 13.91 | 13.60 |
| InstructBLIP-13b | 49.87 | 45.37 | 47.47 | 13.93 | 18.16 | 17.99 | 17.54 | 2.56 | 2.31 | 2.34 | 14.14 | 13.93 | 13.61 |
| LLaVA1.5-7b | 52.08 | 44.52 | 47.93 | 11.24 | 16.77 | 23.50 | 18.77 | 2.67 | 3.32 | 2.78 | 13.14 | 18.23 | 14.64 |
| LLaVA1.5-13b | 51.01 | 45.35 | 47.96 | 11.80 | 17.43 | 20.64 | 18.35 | 2.45 | 2.53 | 2.40 | 13.80 | 16.25 | 14.49 |
| mPLUG-Owl2 | 62.21 | 68.19 | 64.49 | 40.11 | 44.70 | 30.08 | 32.00 | 19.54 | 13.45 | 13.84 | 39.80 | 26.83 | 28.50 |
| XrayGPT | 65.32 | 60.37 | 62.62 | 25.96 | 24.67 | 35.01 | 27.94 | 5.95 | 8.23 | 6.59 | 19.61 | 27.68 | 22.15 |
| Mini-gpt4 | 48.57 | 44.80 | 46.43 | 10.27 | 15.16 | 18.94 | 15.37 | 1.88 | 2.05 | 1.75 | 12.58 | 15.41 | 12.63 |

### A.2 Experiments Compute Resources

In this paper, the training of MedHallDetector was performed on 8 A800s and took 2 hours. Inference for all baseline models was performed on a single A800 GPU.

### A.3 Training Details of MediHallDetector

As described in Section 5 of the main paper, our hallucination detection model underwent supervised fine-tuning (SFT) using data from traditional medical visual language tasks, Med-HallMark data, and hallucination detection instruction pair data. The hallucination detection instruction pair data is divided into two parts: instructions for detecting hallucinations in VQA tasks and instructions for detecting hallucinations in IRG tasks. Detailed prompts are shown in Figure 6.
For hallucination detection in IRG tasks, all images are chest X-rays, so minor hallucinations do not occur. Additionally, the prompts used in IRG tasks are clear and detailed enough, eliminating the presence of prompt-induced hallucinations.

### A.4 Comparison between Different Hallucination Detection Methods

To better illustrate the shortcomings of existing general hallucination detection models, we visualized evaluation examples from GPT-4V, GPT-4O, and Gemini. Additionally, to demonstrate the reasoning behind these detection models, we required them to provide explanations for their hallucination-type classifications. The results are shown in Figure 14. The full prompts are presented in Figure 6’s VQA Hallucination-Type Classification Instruction, but here we only show a portion of the instructions, including the origin question, LVLM answer, GT, and medical image.

From the visualization, it is evident that both GPT-4V and GPT-4O followed the instructions well but incorrectly classified the hallucination types in the LVLM outputs. Even when prompted to explain their classifications, they failed to recognize their errors. In contrast, Gemini correctly detected the hallucination types but did not follow the instructions well, providing extensive explanations for its classifications.

### A.5 More Results of Baseline Models in the Med-HallMark

To provide a more comprehensive comparison of different baseline models’ performance on Med-HallMark, we have included additional detailed metrics for various baseline models in both VQA and IRG tasks, as shown in Tables 5 and 7. Additionally, we have provided traditional metrics for each model on counterfactual questions, which are not detailed in the main text, as illustrated in Table 6.

### A.6 Limitations

The limitations of our study are as follows:
Single Language Support: Currently, Med-HallMark contains only English data, and MediHallDetector supports hallucination detection only in English. However, medical contexts often require the use of local languages rather than English.

Need for Further Enhancement: Although our hallucination detection model, MediHallDetector, surpasses strong models like GPT-3.5, GPT-4, and Gemini in detection performance and significantly improves efficiency compared to manual evaluation, there remains a gap compared to human preferences. This is due to the inherent limitations of the base models and the intrinsic challenges of LVLMs.

### A.7 Future Work

Multi-language Support: Presently, Med-HallMark includes only English data, and MediHallDetector supports hallucination detection only in English. In the future, we aim to provide multi-language support, enabling Med-HallMark to include multiple mainstream languages and MediHallDetector to detect hallucinations in texts of different languages.

Provision of More Baselines: We will continue to track contributions from the open-source community and promptly evaluate the latest LVLMs on Med-HallMark across various metrics. This will help demonstrate the hallucination levels of the most advanced models in medical contexts to the community.

## Appendix B Hosting and Maintenance Plan

The authors and corresponding lab will host the dataset and handle maintenance concerns. We have carefully scrutinized the data to ensure that there are no errata for any data points.
We have released partial Med-HallMark to GitHub and will be maintaining and updating it continuously.

## Appendix C Author Statement

The authors declare that they bear all responsibility for violations of rights.

All textual content and annotations constructed in this work are provided under the CC-BY-4.0 license and will be open source. The medical images used in this study are sourced from open datasets. Due to the privacy concerns associated with medical images and the requirements of the source datasets, we will not directly publish all medical images. Instead, we will specify the source and corresponding IDs of all medical images, which must be obtained from the original datasets in compliance with their respective licenses.

Additionally, all textual content in this work is based on the original medical images. By using these texts, you automatically agree to the license terms of the source datasets. Any violations of the source dataset rules by users will result in them bearing responsibility for any infringement issues.

The data released in this study is not intended for any commercial use and may not be modified. Furthermore, these data and models are not recommended for use in real medical scenarios, and we do not assume any responsibility for misuse.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: **Construction of the benchmark:**
1. Could you elaborate on the motivation behind designing confidence-weakening and counterfactual questions beyond the "conventional" questions in established test sets like SLAKE and RAD-VQA? Regarding counterfactual question generation, considering the GPT model is provided with limited information (text-only, single question with its ground-truth answer), how is the quality of GPT-generated counterfactual questions and answers ensured?

2. Robust evaluation of hallucinations in open-ended IRG tasks is inherently challenging, particularly due to synonymous terms (in frameworks like ICD-10, multiple expressions might refer to the same condition). Since the ground truth for open-ended IRG scenarios in this paper derives from medical reports in MIMIC and OpenI test sets, how does the evaluation framework account for potential synonyms in model responses given the limited set of 1800 images and their reports seen by MediHallDetector?

3. In using LLaVA-Med to generate GT responses for Med-VQA, why not utilize the GT already provided in the dataset, given that the questions originate from an established source? For the IRG scenario, are model-generated responses also exclusively from LLaVA-Med? If so, might this narrow distribution affect the generalizability of MediHallDetector, as it is fine-tuned specifically on LLaVA-Med responses?

**Experimental Design:**
1. How much does the image come into play in tuning the MedihallDetector model? It could be necessary to ablate w./w.o. image input, especially when you freeze the image encoder and connector during training.

**Results**
1. For medical VQA tasks (Table 1), why introduce a MediHall Score when accuracy already exists as a metric for Med-VQA? Notably, XrayGPT, with an accuracy of only 0.02, has a MediHall Score of 0.36. Could this indicate potential inflation in the MediHall Score?

2. For medical IRG tasks (Table 2), the results for XrayGPT as measured by MediHallDetector seem counter-intuitive compared to Minigpt-4. In the original XrayGPT study [1], Minigpt-4 was a baseline against which improvements were demonstrated using ROUGE (Table 1 in [1]). Here, XrayGPT does outperform Minigpt-4 in all conventional metrics, but in the MediHall Score, with Minigpt-4 obtaining the highest MediHall Score among evaluated models. Could this raise concerns regarding the reliability of the MediHall Score metric?

3. Could you clarify why models like BLIP2, LLaVA-Med, and RadFM do not receive MediHall Scores for medical IRG tasks? Open-ended IRG tasks are particularly significant since accuracy is already used as a metric for VQA. Presently, only XrayGPT is evaluated in IRG tasks using the proposed model and metric; incorporating additional state-of-the-art models would enhance the proposed metric's robustness. For instance, open-source domain-specific models such as BiomedGPT [2] and Med-Flamingo [3], which perform well on established tasks, or widely used proprietary MLLMs like GPT-4, Claude 3.5, and Gemini Pro could be valuable benchmarks.

**Typo**
1. In Figure 2(a), “pairs” is misspelled.
2. Could you clarify whether XrayGPT is used in constructing the benchmark? It appears in Figure 1(b) but is not referenced in section 3.4.

[1] https://aclanthology.org/2024.bionlp-1.35.pdf
[2] https://arxiv.org/abs/2305.17100
[3] https://arxiv.org/abs/2307.15189
soundness: 2
strengths: The paper is well-motivated in that it addresses a critical and underexplored area in medical LVLMs: detecting and evaluating hallucinations, which is crucial for ensuring safety in clinical applications. The work frames hallucination detection specifically for the medical domain. The MediHall Score introduces a nuanced metric that prioritizes clinical impact by differentiating hallucinations based on their severity and type. The ablation study of different SFT methods for training the detector also adds robustness to the study.
summary: This paper introduces a hallucination detection and evaluation benchmark in the medical multimodal domain. Recognizing that large vision-language models often inherit hallucination tendencies from foundational large-language models, the authors propose a framework for more accurate and detailed hallucination assessment in high-stakes medical applications. 
The paper makes three primary contributions: (i) Med-HallMark, the benchmark dedicated to medical hallucination detection (ii) MediHallDetector, a hallucination detection model tailored for the medical domain; and (iii) MediHall Score, an evaluative metric designed for hallucinations of different severity and types. 
Through extensive experiments, the paper claims that MediHall Score provides more nuanced insights than traditional metrics and that MediHallDetector enhances hallucination detection performance in medical LVLMs.
weaknesses: The paper raises several questions about primarily the soundness of evaluation and the reliability of the proposed metric (detailed in the **Questions** section). For example, the paper limits its IRG evaluation to only one domain-specific model, XrayGPT, which is problematic given that IRG tasks are where hallucination metrics are particularly valuable. Additionally, results for XrayGPT in IRG appear inconsistent with those of Minigpt-4, which was shown to underperform in traditional metrics but achieved the highest MediHall Score among all evaluated models, calling into question the reliability of the MediHall Score. The paper could have strengthened its findings by evaluating other state-of-the-art LVLMs, such as open-source, domain-specific models like BiomedGPT or Med-Flamingo, or proprietary MLLMs like GPT-4 and Gemini Pro, to validate the metric’s robustness. Finally, the experimental design lacks a thorough investigation into the role of image inputs in tuning MediHallDetector. The study could benefit from ablation studies that assess performance with and without image inputs, especially given that the image encoder and connector taken from LLaVA are frozen during training and could have introduced extra errors due to its lack of visual understanding ability. These design choices suggest that the current benchmark version may be limited in scope and could benefit from further refinement and validation.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: •	Did you encounter failures ? Could you elaborate on those cases where MediHallDetector fails and what are the causes.

•	Do you think MediHallDetector could work on other multimodal medical data ? What are the potential challenges and difficulties in extending MediHallDetector to out-of-domain data?

•	Given the rapid evolution of LVLMs, does MediHallDetector need to be updated ?
soundness: 3
strengths: •	Novelty of Benchmark Proposal: the medical benchmark is novel and relevant in assessing LVLMs hallucinations 

•	Robustness: Test time evaluations with several comparisons on annotated sets strengthen the validity of the approach. 

•	Evaluation: Baselines are carefully chosen and evaluated.

•	Clinical Impact: This study includes expert annotators which makes the dataset reliable and applicable.
summary: The paper proposes Med-HallMark, a benchmark to assess hallucination detection in large vision-language models (LVLMs) applied to medicine. This paper suggests MediHallDetector as the first hallucination detection model able to evaluate medical image report generation and visual question answering, and outperforming baselines.
The authors also proposes an evaluation metric, MediHall Score to gauge the level of hallucination in LVLMs’ answers. Both qualitative and quantitative results are provided to show the superiority of MediHallDetector over baseline models.
weaknesses: •	Reproducibility: The authors highlight the benefit of hierarchical categorization but the annotation process is not fully explained for generalization purposes.

•	Evaluation: Lack of discussion for the reasons behind the difference in performance of the models employed.

•	The proposed benchmark includes two modalities X-ray and CT and questions remain on whether it could be applied and work for other imaging modalities.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - The format of the reference is weird. Please check it.
- The figure 2 (d) is vague.
soundness: 3
strengths: - The introduction of Med-HallMark and MediHall Score fills a significant gap by addressing hallucination detection in medical LVLMs.
- The proposed categorization of hallucinations is innovative and highly relevant, allowing for analysis of potential model impacts on medical decision-making.
- The authors conduct extensive experiments comparing popular LVLM models on Med-HallMark.
summary: The paper introduces a new benchmark, Med-HallMark, and proposes evaluation tools specifically tailored to medical contexts. Med-HallMark provides a multi-dimensional framework to identify and assess hallucinations generated by LVLMs. The benchmark includes multi-tasking hallucination support, diverse data for hallucination detection, and a hierarchical categorization of hallucinations based on clinical impact. Additionally, the paper presents MediHall Score, a metric designed to evaluate hallucinations in medical text output, and MediHallDetector, a specialized LVLM aimed at improved hallucination detection through multi-task training.
weaknesses: - Is this very different from the accuracy? After reading the paper, I just feel that this paper just makes a more fine-grained classification of errors, and the classification standard needs to be discussed and re-designed. For example, in the report generation task, the model outputs a nonsense sentence for the chest X-ray, "This is a chest X-ray of a person", which is obviously correct, but it is not what we want. How should this be judged?
So this makes me wonder whether such a benchmark is necessary.
- Some recent related work [1,2,3,4,5,6] is missing. 
- The overall data scale is relatively small, and the medical image modalities involved are limited to radiology.
- How to ensure the accuracy of annotation? 
- Since doctors are hired to do the annotation, have the possible ethical risks been resolved? For example, IRB approval, etc.



[1] Gu Z, Yin C, Liu F, et al. MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context[J]. arXiv preprint arXiv:2407.02730, 2024.

[2] Yan Q, He X, Wang X E. Med-HVL: Automatic Medical Domain Hallucination Evaluation for Large Vision-Language Models[C]//AAAI 2024 Spring Symposium on Clinical Foundation Models. 2024.

[3] Jiang Y, Chen J, Yang D, et al. MedThink: Inducing Medical Large-scale Visual Language Models to Hallucinate Less by Thinking More[J]. arXiv preprint arXiv:2406.11451, 2024.

[4] Xia P, Chen Z, Tian J, et al. CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models[J]. arXiv preprint arXiv:2406.06007, 2024.

[5] Nan Y, Zhou H, Xing X, et al. Beyond the Hype: A dispassionate look at vision-language models in medical scenario[J]. arXiv preprint arXiv:2408.08704, 2024.

[6] Yan Q, He X, Yue X, et al. Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA[J]. arXiv preprint arXiv:2405.20421, 2024.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The authors explicitly defined 5 types of hallucination levels, among which: Catastrophic Hallucination, Critical Hallucinations, Minor Hallucination are severity levels, but the attribute, prompt-induced ones is indeed the `cause` of the hallucinations. There raise doubts of the rationale of such empirical classifications including attribute, prompt-induced to severity levels. 
 
2. As in line 215, the conventional Q_{conv} questions are generated by the GPT3.5, how is the quality guaranteed? How is the initial conv question obtained?  
 
3. LLaVA-Med was used to infer answers, what is the size of the whole dataset? Even though the authors claim that the data has been examined, how is the quality guaranteed?   
 
4. Also, for the dataset part that is relevant to the IRG scenario, the authors use a sampling method to draw 1,800 images and their corresponding medical reports from existing datasets: MIMIC-test and OpenI datasets. This weakens the contribution towards the data as proposed in line 065.  
 
5. In line 255, “This fine-grained metric xxx”, starts abruptly, since the previous paragraph discusses the existing metrics drawbacks. Then it directly refers to “This metric”, which is not transitioning naturally.  
 
6. The MediHALL score is relatively simple, which is built upon the pre-categorized types of hallucination levels and assigns different values in a hierarchical way.  Based on this, the latter human-annotation and fine-tuning are conducted, are they able to maintain the objectiveness?  
 
7. In line 318, which directs the training data details to Figure, it explicitly shows the categories, and the types of training data covered for MEDIHALLDETECTOR, but what about the amount of the data? And how is the instruction pair data derived?  
 
8. While the paper is positioned as the first benchmark for medical LM hallucination evaluation, there is relevant work worth referencing:  

[1] MEDHALU: Hallucinations in Responses to Healthcare Queries by Large Language Models 
[2] MEDIC: TOWARDS A COMPREHENSIVE FRAMEWORK FOR EVALUATING LLMS IN CLINICAL APPLICATIONS 
[3] Faithfulness Hallucination Detection in Healthcare AI
soundness: 2
strengths: The paper proposes a novel benchmark for evaluation the LLMs specially in the Medical domain, and provides a comprehensive framework from dataset, metric, and the fine-tuned evaluation model, which is a completed work through the evaluation pipeline.  
●	The paper is writing in a smooth way, easy to follow and simple to understand.  
●	The work is of good significance, should be a meaningful angle to advance the LLM/LVLMs applications in real-world healthcare domain.
summary: The paper proposes an LLM/LVLM evaluation benchmark, especially for the hallucination in the medical domain tasks. It also proposes a metric to evaluate the severity of hallucination in LVLMs, and a fine-tuned evaluator model is released to perform the universal evaluation for given set information based on (input image I, the original prompt P, the LVLM answer A and the ground truth GT).
weaknesses: ●	The paper did not emphasize the special challenges in Healthcare domain, after the reading, except for the first part of the dataset that involves the `Medical` multi-modality data such as CT, the evaluation process is on the common hallucination challenges from any Vision-language model, it raises the concerns that whether the proposed model stands out from common LVM hallucination evaluation, is other existing work able to solve the same question by simply adapting to the medical data? What is the advantage of the proposed fine-tuned model over other baseline methods?  
●	The hierarchical definition of the MediHALL score seems a bit intuitive, which is a simple evenly scaled value on 5 categories.  
●	The paper lacks discussion on the relevant work that conduct similar hallucination detection in LLMs at Healthcare domain.  
●	It is suggested that the paper could make further clarification on certain questions as in the Question section, if most critical concerns can be addressed, it is plausible to raise the score for the paper’s quality evaluation.

Meta-review: This paper introduces Med-HallMark, a benchmark for detecting and evaluating hallucinations in medical multimodal tasks, along with MediHallDetector, a fine-tuned hallucination detection model, and MediHall Score, a nuanced evaluation metric, collectively advancing the reliability and assessment of large vision-language models in high-stakes medical applications.

Reviewers agreed that the strengths of this paper include its introduction of Med-HallMark and MediHall Score, addressing a critical gap in hallucination detection for medical LVLMs. Meanwhile, Expert-annotated data, nuanced categorization and robust evaluations also contribute to advancing LVLM applications in healthcare.   On the other hand, several common concerns were raised by reviewers, focusing on evaluation soundness, dataset limitations, and reproducibility. Reviewers CLg1, 69eN, and h5iV noted the limited dataset size and restriction to X-ray and CT modalities, which hinder the benchmark's generalizability to other medical domains. The reliability of the MediHall Score was questioned by Reviewers CLg1, 69eN, and Sam2, citing inconsistent results and an overly intuitive hierarchical definition lacking justification. Reviewers h5iV and Sam2 also highlighted the lack of discussion on related work in hallucination detection for healthcare-specific LLMs and insufficient comparisons with existing methods. Additionally, Reviewers 69eN and h5iV criticized the annotation process for being inadequately detailed, raising concerns about reproducibility and ethical considerations such as IRB approval.

No rebuttal was submitted.

Three reviewers leaned towards rejection while one reviewer suggested to be marginally above the acceptance threshold. After carefully considering the reviewers' comments and the absence of a rebuttal from the authors, I have decided to reject the paper. While the paper introduces a novel benchmark and metric for hallucination detection in medical LVLMs, significant concerns remain unresolved. These include the limited dataset size and scope, inconsistent evaluation results, questions about the reliability and design of the MediHall Score, and a lack of detail in the annotation process, raising reproducibility and ethical concerns. Without a rebuttal to address these critical issues, the paper does not meet the standard required for acceptance.

Decision: Reject