Faithfulness Hallucination Detection in Healthcare AI

Authors: prathiksha  rumale
simran  tiwari
tejas g naik
sahil  gupta
dung n thai
wenlong  zhao
sunjae  kwon
victor  ardulov
karim  tarabishy
andrew  mccallum
wael  salloum
Venue: KDD_org_2024_Workshop_AIDSH
Published: 2024-06-05 11:39:47

Abstract: Faithfulness hallucinations, where AI-generated contents diverge from input contexts, pose significant risks in high-stakes environments like healthcare. In clinical settings, the reliability of systems is crucial, as any deviations can lead to misdiagnoses and inappropriate treatments. The burden of summarizing lengthy electronic health records (EHRs) is substantial for clinicians, who face the challenge of extracting key information from extensive notes. Leveraging large language models (LLMs) to summarize these documents can aid clinicians by providing concise and accurate summaries. This study investigates faithfulness hallucinations in medical record summaries generated by LLMs such as GPT-4o and Llama-3. Our detection framework, developed in collaboration with clinicians and supported by a web-based annotation tool, categorizes 5 types of medical event hallucinations. A pilot study involving 100 summaries of medical notes reveals the presence of our categorized hallucinations by recent closed-source and open-source LLMs. The findings highlight the necessity for robust hallucination detection methods to ensure reliability of AI applications in healthcare, ultimately enhancing clinical workflows and improving patient care.

Main paper: 
 introduction: 
 The rapid advances in large language models (LLMs) such as Llama-3 [16] and GPT-4o [1] have captivated the world with their capabilities. Despite being trained as general purpose language models, they can often generate meaningful texts about medical contents, pass medical exams [8, 10, 13] , and perform remarkably well on multiple healthcare and medical tasks [5, 11] . These LLMs can generate summaries, abstract patient information, and answer questions about medical records, demonstrating their potential to assist medical professionals on mundane and time-consuming problems. However, there is an unsolved limitation that these models may hallucinate, producing well-written but nonsensical and unfaithful content. While such hallucinations might be inconsequential in casual conversations, they pose significant risks in healthcare applications where soundness and trustworthiness are crucial. Inaccuracies in high-stakes healthcare settings can lead to severe consequences, including misdiagnoses and inappropriate treatments.A critical challenge in achieving reliable AI in healthcare is addressing model hallucinations. Much existing research focuses on factual hallucination, where LLM generated information is inconsistent with verifiable real-world knowledge. Factually plausible descriptions, however, may still be harmful and misleading, if they are not grounded in the input context. Faithfulness hallucination refers to the phenomenon that models' generated content diverges from the instruction or context provided. This is most typically observed when LLMs produce summaries containing information that contradicts with or is not evident in the original document. Since medical records are highly contextualized and personalized, faithfulness hallucination is of paramount importance to ensuring the safe applications of LLMs in healthcare.In this work, we propose the task of detecting faithfulness hallucinations in medical record summaries. We define 5 medical event hallucination types, incorrect reasoning, and chronological inconsistency, according to which a system should identify unfaithfulness between the original medical record and its summary. To facilitate hallucination annotation, we collaborate with clinicians to carefully craft data annotation guidelines and develop a web-based user interface. We have completed a pilot study using the proposed annotation tool to collect human expert annotations on 50 medical notes and their summaries generated by GPT-4o and Llama-3. Our findings indicate that hallucinations occur within the summaries from both closed-source and open-source LLMs, further confirming the need for hallucination detection in the medical domain.Annotating hallucinations in medical record summaries by human experts is expensive, ushering in the need for an automatic hallucination detection system. In our pilot study, it took 92 minutes on average for a well-trained clinician to label a summary. At the rate of $36 per hour, annotating 100 summaries in our pilot study cost $5508 and 153 working hours in total. To this end, we establish … She took three SL nitro's at home, with some minor improvement in her CP but when it did not resolve she called Dr. ___ referred her to the ER. She denied any associated SOB, nausea/vomiting. She denies any orthopnea, PND, edema or palpitations. … … She denied shortness of breath, nausea, vomiting, leg swelling, and palpitations. An EKG showed an unchanged left bundle branch block (LBBB). ... an extraction-based system and an LLM-based system as baselines for the task. We evaluate the two baselines on the 100 annotated summaries and analyze their strengths and limitations.Our contributions are as follows: (1) We propose the task of faithfulness hallucination detection in medical record summaries and categorize hallucinations types to enable a fine-grained evaluation of facts essential to various medical domains and patient care. 2 Related Work Faithfulness hallucination. [11] explores trust challenges in LLMs by focusing on factual accuracy and faithfulness, emphasizing alignment between model outputs and instructions, context, and logic. The paper delves into the root causes of hallucinations in LLMs, including flawed data, biased training, and limitations during inference. It also introduces the concept of "faithfulness hallucinations, " where model outputs deviate from inputs. To assess the prevalence of hallucination, benchmarks like TruthfulQA [12] , MedHALT [15] and REALTIMEQA [9] have been established and these efforts aim to improve the reliability of LLMs in real-world applications. Our work builds upon this foundation and study the detection of faithfulness hallucinations in healthcare AI.Faithfulness hallucination in medical summaries. A pioneering study [4] investigates faithfulness hallucination in medical text summaries generated by the LED model [7] and argues that model hallucinations have critical impact on patient care. The authors manually extract standard concepts and actions and compound concepts from summaries and let clinical practitioners classify them as "Incorrect", "Missing details", or "Not in Notes". Realizing the specialty of the clinical domain, we categorize a set of 5 hallucination types to be detected, development annotation guidelines and tools, and evaluate hallucinations in summaries from the state-of-the-art LLMs.Evaluating medical tasks. [6] features a systematic review of 519 studies published between Jan. 1st, 2022, and Feb. 19th, 2024, sourced from PubMed and Web of Science, and highlights the lack of standardized task definitions and dimensions of evaluation in the field. These studies were categorized based on evaluation data type, healthcare task, NLP/NLU task, evaluation dimension, and medical specialty. The findings emphasize the necessity of using real patient care data to assess LLMs' effectiveness in administrative tasks. Moreover, the evaluation criteria must extend beyond accuracy to encompass fairness, bias, toxicity, robustness, and deployment considerations across various medical specialties. [14] highlights the limitations of automatic metrics for NLG tasks in the clinical domain. Our work employs human experts to assess hallucinations in LLM-produced summaries. We contribute a guideline for human experts to detect and categorize hallucinations, allowing for more nuanced analysis compared to simple presence-absence judgments.Evaluating medical summaries.[3] establishes a framework for studying "hospital course summarization." The work focuses on the task of automatically generating concise summaries of patient hospital stays using discharge summaries as a reference. The creation of the CLINSUM dataset [2] , a large dataset of patient records with corresponding summaries, helps in the development and evaluation of summarization models. One of their mentioned future works is the development of evaluation methods that assess not only factual accuracy and coherence but also check for clinical relevance. We study the detection of faithfulness hallucinations detect hallucinations in the medical record summaries. 
faithfulness hallucination detection in: 
 Health AI 3.1 Task Definition Faithfulness hallucination in clinical note summaries refers to the phenomena where summarization models generate content that is incorrect or too general according to information in the source clinical notes. These hallucinations can lead to serious consequences in patient care as they undermine the reliability and accuracy of clinical documentation. Incorrectness describes any content in the summary that does not appear in the clinical note or is twisted. This includes generalized information in the medical notes that becomes incorrectly specific in the summary. Specific ⇒ general refers to any content in the summary that are correct but too general compared to the clinical note, such as, medical conditions that are described as hypernyms.Given a medical record 𝑑, its corresponding summary 𝑠 (possibly generated by a summarization model), and a predefined set of hallucination categories H , the objective of the faithfulness hallucination detection task is to detect hallucination occurrences 𝑦, providing a piece of textual evidence and predicting its hallucination category ℎ ∈ H . The evidence is in the form of a text span in the summary when possible. If the text span is unavailable or not comprehensible, the evidence is in the form of free-text explanations.Defining hallucination categories that fit all medical tasks is challenging. The variables of interest can change from one study to another, based on the available information in the medical records and the interests of the summary readers. In this work, we focus on discharge summaries. We collaborate with the clinical team to define a set of hallucination categories based on the structure of the medical notes.Medical event inconsistency. We consider 5 types of hallucinations about different medical events. (1) Patient Information: Hallucinated demographic details and non-medical information about the patient's background including name, age, gender, ethnicity, race, and address. (2) Patient History: Hallucinated information regarding the history of present illness. Many instances of hallucination include incorrectly stated illness or previously unseen health conditions. (3) Symptoms/Diagnosis/Surgical Procedures: Inconsistent symptoms, diagnosis, or procedures found in the patient's current visit details. (4) Medicine Related Instructions: Any disparities or discrepancies noted between the medication instructions documented in the summary and those found in the medical note. (5) Follow-up: Missing information regarding "follow-up" care or instructions provided to the patient. This includes appoint rescheduling and continued monitoring. Examples sections that may have follow-up information include discharge diagnosis, discharge condition, and discharge instructions.Chronological Inconsistency. The order of medical events is not consistent with the sequence documented in the EHR. For example, an event that supposedly happened after another event is described as occurring before it.Incorrect Reasoning. Summary states correct information but the associated reasoning given for it does not make sense or is incorrect. 
data annotation 4.1 guidelines: 
 We proposed our initial guidelines based on the structure of the medical notes and identified potential hallucination types in each part. The guidelines included positive and negative examples for each hallucination type. We recruited a team of 15 clinicians. Each clinician had at least 5 years of experience abstracting medical records and building clinical ontologies. The team provided feedback on the chosen examples and categorized hallucination types. Following trail runs and subsequent revisions, the guidelines grouped the hallucinations into the presented categories based on their impact.We developed a web-based data annotation user interface (UI) that allows annotators to highlight hallucinated spans from summaries and label their categories. The tool also allows annotators to provide free-text explanations for the detected hallucinations. Key features of the data annotation tool are highlighted in Figure 2 . The tool lists extracted spans for annotators to review and amend if necessary. Annotations are automatically uploaded once submitted. The tool also supports loading previously annotated results for consensus and quality control.For our pilot study, we utilized a dataset consisting of 50 medical notes and their corresponding summaries generated by GPT-4o and Llama-3. These medical notes were randomly sampled from the MIMIC-IV (Medical Information Mart for Intensive Care IV) database, specifically focusing on discharge summaries. The selected notes provided a diverse set of patient records, ensuring a representative sample of typical discharge summaries. Each note contained comprehensive details about the patient's medical history, current visit information, symptoms, diagnosis, treatment plans, medication instructions, and follow-up instructions. GPT-4o and Llama-3 were prompted to summarize the medical notes with specific instructions to provide any available information regarding the variables of interest. To promote brevity, the models were additionally prompted to generate the summaries of no more than 𝑛 = 500 words. Explicitly the prompt provided was: Summarization Prompt Summarize the provided clinical note with at most {n} words. Ensure to capture the following essential information, when they exist: the specific cancer type, its morphology, cancer stage, progression, TNM staging, prescribed medications, diagnostic tests conducted, surgical interventions performed, and the patient's response to treatment. {clinical note}Following a training session to familiarize themselves with the annotation guidelines and tool, each clinician was instructed to identify and annotate inconsistencies between the medical notes and their summaries, labeling them according to our hallucination categories. Each medical event inconsistency is also labeled as either incorrect or specific=>general. Clinicians began the annotation process by reading medical records using a search tool for abstraction and extraction of medical events, followed by reading summaries to label hallucinations as per guidelines. They verified suspicious information by cross-referencing the records and summaries. The average annotation time per note was 91.5 minutes. A quality control (QC) member reviewed and corrected annotations, calculating a quality score from a random 5% sample per annotator. A detailed step by step procedure of the annotation process is described in Appendix A. 
hallucination pattern analysis: 
 Table 1 reveals that both GPT-4o and Llama-3 exhibit hallucinations on almost all the examples. Regarding medical event inconsistency, GPT-4o had 21 summaries with incorrect information and 50 summaries with generalized information, while Llama-3 had 19 and 47, respectively. Both models produced many more "specific => general" errors, indicating the tendency of these models to produce less precise words that are potentially more frequently used in the general domain. We conjecture that domain-specific training will alleviate such errors. Among the 5 categories, most errors are from the symptoms/diagnosis and the medicinal instructions. This is unsurprising since these two categories involve the most expert knowledge and terminologies.Considering "Incorrect Reasoning", Llama-3 hallucinates less compared to GPT-4o, with 26 summaries with related errors versus GPT-4o's 44. This is, however, worth noting that Llama-3 generates shorter summaries given the same prompt and does not attempt to infer additional details as much as GPT-4o. GPT-4o had an average of 516.46 words per summary with 27 summaries exceeding 500 words, while Llama-3 had an average of 266.56 words with no summaries exceeding 500 words. This aligns with findings from [3] that longer summaries tend to entail more hallucinations. We add to the finding that these errors often stem from incorrect reasoning in addition to medical event inconsistencies.We qualitatively analyze the hallucination patterns related to "Incorrect Reasoning" by randomly sampling several medical notes. For brevity, we present a few representative examples in Table 2 . Our observations reveal that Llama-3 often omits entirely the details that would have been generated with hallucinations by GPT-4o. When prompted to include these details in the summary and allowed to summarize without a word limit, Llama-3 tends to state the details as-is, with somewhat acceptable interpretations of these details. In contrast, GPT-4o often makes bold, two-step reasoning statements that can lead to hallucinations.Chronological inconsistencies are relatively rare in both models as GPT-4o had 2 summaries with chronological inconsistencies, whereas Llama-3 had only 1. This may be due to the nature of the medical notes studied, which typically span a single patient visit and lack a strong timeline aspect. Further study on a longitudinal dataset could provide more insights into chronological inconsistencies. 
automatic hallucination detection: 
 Human expert annotation of hallucinations is both expensive and time-consuming. To potentially automate this task and scale up the analysis of hallucination patterns across more summarization models, we explore an extraction-based system and an LLM-based system for hallucination detection.In order to scale the hallucination detection, our work employs the use of a commercial solution, Hypercube, to detect medical event hallucinations. Through a combination of medical knowledge bases, symbolic reasoning, and NLP, Hypercube is able to build a comprehensive representation of patient documents. Given a medical document, Hypercube produces a set of events, each having an associated type and its own collection of properties. Since Hypercube identifies explicit spans of text as evidence for properties which are subsequently consolidated into a coherent set of events, the system will overtly seek to address overlapping and potentially conflicting information. Additionally Hypercube leverages a set of medical knowledge bases in order to link the extracted events and properties to concrete definitions with semantic relationships to other concepts. As a consequence, the set of events Hypercube produces is often complete and precise with respect to its medical domain model.We consider the patient journeys extracted by Hypercube, one from the original patient record and another from the LLM summary, denoted 𝐸 𝑑 and 𝐸 𝑠 . Set-membership comparison is employed to compare these sets of extracted events. The hierarchical and semantic nature of the event types and property values allows for partial matching of events with varying specificity. Our systemLlama-3 GPT-4o Llama-3 Incorrect Spec. ⇒ Gen. Incorrect Spec. ⇒ Gen. Incorrect Spec. ⇒ Gen. Incorrect Spec. ⇒ Gen. maps events that are missing or over-specified in 𝐸 𝑠 as compared to 𝐸 𝑑 as "incorrect" hallucinations, since these correspond to events in the summary that do not present in the original medical note. Meanwhile, "specific => general" hallucinations correspond to events that are under-specified with respect to 𝐸 𝑑 . By mapping the events and property types produced by Hypercube onto the aforementioned hallucination categories (Section 3.2), we automatically detect hallucinations in LLM-produced summaries. We analyze 15 randomly sampled GPT-4o summaries and show a few detection examples in Table 3 . We observe that Hypercube tends to over-estimate the number of hallucinations due to limitations of the knowledge base, such as, being case-sensitive and disallowing paraphrases or acronyms. Nonetheless, we observe that Hypercube can detect hallucinations that are otherwise missed by human experts. These findings suggest that Hypercube can be a helpful tool for an initial hallucination detection step, which can then be integrated with human expert review to enhance overall detection accuracy.The LLM-based model leverages the advanced capabilities of large language models to detect hallucinations in medical summaries. As LLMs have demonstrated the ability to self-correct, they might also be capable of detecting hallucinations when appropriately prompted. We employ a straightforward prompt for hallucination detection: "Detect all the inconsistencies between an original document and its summary. The inconsistency could be any details in the summary that could potentially have a different interpretation from the document, or vice versa. "We use this prompt with GPT-4o, providing the original document and its summary as input. This method generates significantly fewer candidate hallucinations compared to Hypercube. Further, most of the detections identified by GPT-4o are false positives, primarily inconsistencies in writing style, such as abbreviations, as shown in Table 4 . Despite this, there are notable cases where the detected hallucinations require complex reasoning, involving the consolidation of multiple pieces of knowledge that would otherwise conflict with each other.We conjecture that LLMs have potential to accelerate hallucination detection due to their ability to perform complicated reasoning and identify inconsistencies. However, the current results indicate that domain-specific training or better prompting are necessary to improve the precision and recall of LLM-based hallucination detection systems. 
conclusion and future work: 
 We propose the task of faithfulness hallucination detection in medical record summaries generated by large language models (LLMs) such as GPT-4o and Llama-3. These hallucinations, where generated content deviates from the input, pose significant risks in leading to misdiagnoses and inappropriate treatments. We develop a framework for annotating faithfulness hallucinations, categorizing their types and implementing a web-based annotation interface. Our pilot annotation study confirms the presence of various hallucination types produced by both closed-source and open-source LLMs. We further establish two automated hallucination detection baselines and show their limitations.Future work should focus on improving automatic detection systems to mitigate the expense of human annotations as well as alleviating faithfulness hallucinations in healthcare AI. Given our initial evaluation of two summarization LLMs, it is imperative to broaden this study to include a wider range of open-source and proprietary models on more diverse medical tasks. By addressing the issue of faithfulness hallucinations, we can move closer to developing trustworthy AI systems that enhance patient care.  
 
 Appendix: Hypothyroidism: TSH was initially found to be 98. She was continued on 150 mcg daily. IV levothyroxine was considered; however, patient was unwilling to undergo IV placement and HCP was not established at that time. TSH was 33 and patient was continued on PO regimen at 137 mcg daily, with endocrinology input.-Hypothyroidism: The patient was non-compliant with levothyroxine. TSH was initially 98 and decreased to 33 with continued PO levothyroxine. Follow-up with endocrinology was arranged. -The patient is an elderly male with a history of HBV/HCV cirrhosis complicated by portal hypertension and hepatocellular carcinoma (HCC) status post resection.-The patient is an elderly male with a history of HBV/HCV cirrhosis complicated by portal hypertension and hepatocellular carcinoma (HCC) status post resection. YES Table 3 : Samples of hallucination detection by Hypercube. 



Reviews: review: title: title: Clear research questions definition and pipeline, but limited amount of data
main_review: main_review: Pros:

1. The paper is clearly written and well-structured. The research question is well-focused and clearly defined.

2. The collaboration with clinicians to craft data annotation guidelines ensures that the framework is clinically relevant and practical.

3. The use of both extraction-based and LLM-based systems for automatic hallucination detection provides a thorough evaluation of different approaches.

Cons:

1. The sample size of pilot study is too small. It seems like 50 clinical notes is not generalisable.

2. The paper lacks a necessary discussion on how this work differentiates itself from or adds to existing research on faithfulness hallucination problems.

3. The paper is a good technical report, but the AC should consider whether it is a suitable fit as a research paper for this workshop.
title: title: A practical and innovative framework to detect faithfulness hallucinations healthcare AI
main_review: main_review: This study focused on the critical issue of faithfulness hallucinations in medical record summaries generated by large language models (LLMs) such as GPT-4o and Llama-3. They developed a framework which was crafted with input from clinicians and implemented using a web-based annotation tool for detecting faithfulness hallucinations, categorizing them into five distinct types. What’s more, a pilot study involving 50 medical notes was conducted to confirm the presence of various hallucination types in summaries produced by both closedsource and open-source LLMs. This study showed excellent quality, clarity, originality and significance. However, the limitations of this study are also evident. Only two LLMs including GPT-4o and Llama-3 were used. Medical record summaries in English were detected but other languages such as Chinese and Arabic were not. Finally, the research on detecting faithfulness hallucinations in large sample medical records should be conducted to validate the robustness of the framework proposed in this study.