Towards Mitigating LLM Hallucination via Self Reflection
Authors: Ziwei Ji
Tiezheng YU
Yan Xu
Nayeon Lee
Etsuko Ishii
Pascale Fung
Venue: EMNLP 2023 Findings (2023)
Keywords: Hallucination
Large Language Model
Medical Question Answering
Generative Question Answering
Question Answering
TL;DR: This paper delves into the phenomenon of hallucination in medical generative QA systems. To tackle this challenge, we present a self-reflection loop methodology that incorporates knowledge acquisition and answer generation.

Abstract: Large language models (LLMs) have shown promise for generative and knowledge-intensive tasks including question-answering (QA) tasks. However, the practical deployment still faces challenges, notably the issue of "hallucination", where models generate plausible-sounding but unfaithful or nonsensical information. This issue becomes particularly critical in the medical domain due to the uncommon professional concepts and potential social risks involved. This paper analyses the phenomenon of hallucination in medical generative QA systems using widely adopted LLMs and datasets. Our investigation centers on the identification and comprehension of common problematic answers, with a specific emphasis on hallucination. To tackle this challenge, we present an interactive self-reflection methodology that incorporates knowledge acquisition and answer generation. Through this feedback process, our approach steadily enhances the factuality, consistency, and entailment of the generated answers. Consequently, we harness the interactivity and multitasking ability of LLMs and produce progressively more precise and accurate answers. Experimental results on both automatic and human evaluation demonstrate the superiority of our approach in hallucination reduction compared to baselines.

Towards Mitigating Hallucination in Large Language Models via Self-Reflection
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Hong Kong University of Science and Technology zjiaddconnect. wwwpcoonnnnttttttt.o..mmmmm hk, pascaleêece ust. hk
Abstract
causes Noonan syndrome?
Large language models (LLMs) have shown promise for generative and knowledge- intensive tasks including grestion-answering (QA) tasks. However, the practical deployment still faces challenges, notably the issue of "hallucination". where models generate plausible-sounding but unfaithful sensical information. This issue becomes particularly critical in the medical domain due the uncommon professional concepts and potential social risks involved. This paper analyses the phenomenon of hallucination in medical generative QA systems using widely adopted LLMs and datasets. Our investigation centers on the identification and comprehension of common problematic answers, with a specific emphasis on hallucina- tion. To tackle this challenge, we present an interactive self-reflection methodology that incorporates knowledge acquisition and answer generation. Through this feedback process, our approach steadily enhances the factuality, consistency, and entailment of the generated harness the
Noonan syndrome is caused by mutation in the PTEN gene. This gene is responsible for regulating cell growth and division, and when mutated, I! can lead development Noonan syndrome
Noonan syndrome caused by mutations one different genes, including PTPN11, SOS1, KRAS, NRAS, RAF1, BRAF, or MEK1.
non-
Reference
Figure 1: An example hallucinated Generative QA? PTEN mutation is not typically associated with Noo- nan syndrome". Thus, despite the plausible sounding answer, is hallucinated one. highlight the hallu- cinated parts in pink and underline them.
ever, despite their potential and recent advance- ments, these models face a concerning issue, "hal- Incination"; ", phenomenon where the model gen- erates plausible-sounding but unfaithful or nonsen- sical information (Ji et al., 2023). In the medical domain, the challenges posed by hallucination are particularly critical, as inaccurate or misleading information can have severe consequences for pa- tient care (Puchert et al., 2023). Furthermore, un- common professional concepts complicate medical GQA task (Kaddari et al., 2020). For the GQA ex- ample in Figure 1, PTEN (Phosphatase and tensin homolog) mutation is not typically associated with Noonan syndrome, so the answer sounds plausible but hallucinated. Therefore, is crucial to under- stand and mitigate these hallucinations to ensure the reliability and safety of Al-enabled medical services (Morley et al., 2020).
answers. Consequently, interactivity and multitasking ability of LLMs and produce progressively more precise and accurate answers. Experimental results both automatic and human evaluation demonstrate the superiority our approach in hallucination reduction compared to baselines.'
1 Introduction
Large language models (LLMs) have shown promise for generative and knowledge-intensin tasks which require world or domain knowl- edge (Petroni al., 2021). One representa- tive task is generative question-answeriig (GQA) which provides relevant information response to queries (Li et al., 202 Su et al., 2023; Nakano et al., 2021) and can also serve to probe the capabil- ities of language models (Wang et al., 2021). How-
Parallel to retrieving external relevant knowl- edge (Lewis et al., 2020; Guu et al., 2020; Izacard et al., 2022), some current works (Yu al., 2023; Wang et al., 2021; Roberts et al., 2020; Xu et al.,
ddirecty generated by Alpaca-Lora. 'According to
https:/www.chhrr chop. edu/ and
hhttps://githhe httpss///githubb.om//ieeeee www/zwwwwwiitcccttotoooo....ctcs jtt.iieeejii/eeeie /tteeiiieeeeefeeeeifiiii Reflection_ Medical
conditions-diin wwwitiiinnddissses//nooooe diseases/nconan- -yndrro https://en. httts::/en.xikipeeee wikipedia. wwwpoeedia..ggg/w//eeee haaokkki/oooonn_syynnre

---

2022; Sun al., 2023) explore leveraging the para- metric knowledge LLMs and tap into their po- tential for knowledge-intensivee tasks. GQA works in other domains (Lin et al., 2021; Su et al., 2022) underscore the importance of addressing halluci- nation and improving faithfulness. However, the current understanding of the extent of hallucination in medical answers generated by LLMs remains unclear, and there is need explore the potential for further improvement in this aspect.
returns to the initial stage, repeating the cycle. Our method fosters a dynamic interaction between the system and its knowledge, enhancing the model's ability to provide accurate, reliable, and factually grounded responses in healthcare settings.
The experimental results of our method show- case its effectiveness across LLMs with varying parameters, including 7B and 175B, all five datasets. This robust performance highlights the generalizabilityy and scalability of our approach, fur- ther validating its efficacy in the context of medical tasks. Our method explores ex- tracting and digging knowledge from LLMs, lever- aging their full potential, and strives to approach their upper performance bound. Our method can also combine other techniques such as utilizing external knowledge and more powerful LLMs in the future to contribute to the development of ro- bust application systems. In summary, the major contributions of this work are threefold:
To fill the gap, this study investigates halluci- nation in the context of medical GQA systems, particularly in general LLMs like Vicuna (Chi- ang et al., 2023), Alpaca-LoRA (Wang, 2023), ChatGPT (OpenAI, 2023a), and medical LLMs like MedAlpaca (Han et al., 2023). and Robin- medical (Diao et al., 2023) on popular med- ical datasets: PubMedQQ (Jin et al., 2019), MedQuAD (Ben Abacha and Demner-Fushman, 2019), MEDIQA2019 (Ben Abacha et al., 2019). LiveMedQA2007 (Ben Abacha al., 2017), MASH-QA (Zhu et al., 2020). We evaluate the incidence of hallucinations, explore the potential cause, and propose strategy to mitigate this issue. Through comprehensive analysis of the recent models, we hope to increase the understanding of hallucination in these systems and provide a road map towards more reliable Al-assisted healthcare.
We provide a comprehensive examination of the hallucination phenomenon in medical GQA systems, particularly those employing five LLMs across five medical GQA datasets.
We propose an innovative self-reflection method to mitigate hallucination in LLMs. The iterative feedback loop process generates, scores, and refines knowledge and answers until they reach satisfactory levels, enhancing the accuracy and reliability of the answers. Experimental results demonstrate the effec- tiveness, generalizability, and scalability of our method. This investigation sheds light on the potential of LLMs as valuable tools for medical GQA tasks, showcasing their abil- ity to provide meaningful insights without ex- plicit training specific datasets.
Current research works (Yin et al., 2023; Burns al., 2022; Rajpurkar al., 2018; Kadavath et al.. 2022: Manakul al., 2023) highlight gap be- tween surface realization and inherent knowledge in NLG tasks. Models can realize they are gener- ating something hallucinated in some way. To re- duce this gap and mitigate hallucinations in medical GQA, we devise an iterative, introspective process that leverages the multi-turn interactivity and multi- task ability of LLMs. Our self-reflective methodol- ogy initiates the generation of pertinent background knowledge for given question, followed by fac- tuality evaluation. Upon detection of discrepancies, the model is urged to self-correct, utilizing its in- herent reflective capacities to refine the knowledge. This cyclical process is repeated until a satisfactory level of factuality is achieved. In the answering stage, employ similar generation gemeration-scorr-reffff score-refine strategy to ensure consistency between the gener- ated answer and the background knowledge. Ad- ditionally, an entailment evaluation conducted between the answer and the question. If the gener- ated answer fails to meet the standard, the process
Related Work
Medical Question Answering Medical QA sys- tems have shown significant potential for enhane- ing information accessibility and understanding in the healthcare field. These systems respond to vari- ous question formats, including Yes/No (Tsatsaro- nis et al., 2015; Jin et al., 2019), multi-choice (Pal et al., 2022; Li al., 2021b; Welbl et al., 2018; Berant et al., 2014; Abacha et al., 2019; Morante et al., 2012), extractive (Tsatsaronis et al., 2015;
Dhingra al., 2018; Šuster and Daelemans, 2018:
Tang et al., 2020; Möller et al., 2020; pampari et al., 2018), and generative (Savery et al., 2020;

---

Category
Question
Generated Answer
Golden Answer
Noonan syndrome is inherited in an autosomal dominant manner... Each child of a person with
Fact Inconsistency
Is Noonan syndrome inherited??
Noonan syndrome is not inherited. It is a ge- netic mutation that occurs sporadically.
Noonan syndrome a 50% chance to inberit condition...
Cardiac reoperations in octogenarions: Do they really benefit? c- c-Kit- -iit-dependeat growth of uveal melanoma cells: a potential therapeutic target?
Yes... sustainable recovery of the survivors of- fering the prospect of a highly independent and satisfying life...
Query Inconsisteacy
Proper use of Vitamin C' to stop smoking: Tak- ing Vitamin by mouth can help smokers quit...
A musele tumer formed in the anterior eyeball of the hody, called uveal melanoma, is nure uumor that often leads to visual impairment the addition GATA3 can reduce scale cancer...
c-kit is vastly expressed in uveal melanoma, suggest that the c-kit molecular patbway may be important uveal melanoma growth...
Yes
Tangentiality
Table 1: The category of problematic answers with corresponding representative examples.
Mollá al., 2016; Jin et al., 2019; Ben Abacha and Demner-Fushmant 2019; Ben Abacha al., 2019, 2017). The introduction of pre-trained lan- guage models has further enhanced the capabilities GQA systems, enabling them to generate fluent and meaningful responses to medical queries (Soni and Roberts, 2020; Liu et al., 2022; Savery et al., 2020; Alsentzer et al., 2019; Kaddari et al., 2020).
struction (Ouyang al., 2022; Wei al., 2021). and chain-of-thir tfotthttttt reasoning (Wei et al.). How- ever, their deployment in practical applications has also surfaced challenges related to the control, bias, and reliability (Tamkin et al., 2021), where halluci- nation has recently become an increasingly visible issue (OpenAI, 2023a; Bang et al., 2023).
Analysis of Hallucination
Hallucination in Generative Question Answer- ing Faithful GQA, which aims generate an- swers strictly grounded in the source text or valid external knowledge, has gained significant research attention (Nakano et al., 2021; Su et al., 2022, 2023). The more faithful the answer is, the less hallucinated content it contains. Other terms like semantic factual correctness can also reflect hallucination level (Li al., 2021a; la; Su et al., 2022). Rationale-Enrichee Answer Generator (REAG) (Li al., 2021a) add an extraction task to obtain an- swer rationale and generate answers with high con- fidence. Sel--befoo--Geeeratt (Su et al., 2022) combines answer generation with machine reading to incorporate fine-grained, answer-related salient information. A benchmark (Lin et al., 2021) mea- sures the truthfulness of answers generated by lan- guage models across various domains. These stud- ies underscore the importance of reducing halluci- nation, a key focus of our work.
In this section, we directly ask LLMs medical ques- tions from five datasets leveraging their zero-shot capability. We then comprehensively evaluate and analyze the generated answers, with a focus on examining the occurrence of hallucination.
3.1 Models
We evaluate the generated answers from five LLMs. including three general LLMs and two LLMs fine- tuned in the medical domain. Vicuna (Chiang et al., 2023) is trained by fine-tuning LLaMA on user-shared conversations from ShareGPT. Alpaca- LoRA (Wang, 2023) employs Low-Rank Adapta- tion (LoRA) to replicate the results of Stanford's Alpaca model. ChatGPT (OpenAI, 2023a) in- terprets prompts and provides comprehensive re- sponses using Reinforcement Learning from Hu- man Feedback (RIHF).. MedAlpaca (Han et al., 2023) is built upon the frameworks of LLaMA and fune-tuned on instruction-tuiinn formatted medical dialogue and QA texts. Robin-medical (Diao et al.. 2023) is fine-tuned LLaMA in the medical domain using LMFlow.
Large Language Models The advent of LLMs, including GPT-3 (Brown et al., 2020), Chat- GPT (OpenAI, 2023a), LLaMA (Touvron et al., 2023), and GPT-4 (OpenAI, 2023b), has revolution- ized natural language processing tasks, showcasing their impressive language capabilities in generat- ing fluent, contextually relevant responses (Brown al., 2020: OpenAI, 2023a; Touvron et al., 2023; Ap 2023b). In addition, emergent abili- ties are revealed from these models, such as in- context learning (Min et al., 2022), zero-shot in-
3.2 Dataset
PubMedQA (Jin al., 2019) is biomedical
QA dataset containing 1k expert-labeled instances which include questions derived from research article titles, abstracts as the context, long an- swers from abstract conclusions, and concise

---

yes/no/maybe answers. MedQuAD (Ben Abacha and Demner-Fushman, 2019) comprises 47,457 QA pairs from National Institutes of Health web- sites and covers various medical topics including diseases, medications, and diagnostic tests. We use the medical QA dataset from MEDIQA2019 (Ben Abacha et al., 2019) challenge and consider an- swers with scores 3 and 4 golden answers. LiveMedQA2017 (Ben Abacha al., 2017) con- tains annotated medical QA pairs for question anal- ysis and answering systems. MASH-QA (Zhu et al., 2020) includes 34k QA pairs from the con- sumer health domain designed for Multiple Answer Spans Healthcare QA. Except for PubMedQA, an- swer annotation in these datasets involves manual extraction and copying from authentic web con- tent. While the answers pertinent and verifiable, there is room for improvement in terms of contex- tual coherence and question linkage. Please see Appendix A for details and an example.
Error Analysis After analyzing 250 directly gen- crated examples from the five models, we classify problematic answers into three categories: Fact Inconsistency, Query Inconsistency, and Tangen- tiality. Please refer to Table and Figure for the representative example and incidence for each cat- egory and model. We consider the first two the hallucination problem.
1. Fact Inconsistency refers to answers that pro- vide information that is inconsistent or in conflict with the fact. It arises when the model fails appropriately recall relevant knowledge when re- sponding to the question. The example answer in Table 1 incorrectly states that Noonan syndrome is not inherited while it is inherited in an autosomal dominant manner.
2. Query Inconsistency refers answers that are unrelated the query or nonsensical. It occurs when the model neither responds to the question nor invokes relevant knowledge appropriately. The example answer in Table discusses the benefits of Vitamin but does not mention cardiac reoperations.
3.3 Evaluation Protocols
3. Tangentiality refers to answers that provide information related to the topic but do not directly address the question. It occurs when the model does not further process mastered knowledge, such as inductive, deductive, and logical reasoning. The example answer Table tangentially discusses uveal membranc but fails to mention the effect of e- C-Kit on uveal membrane.
To evaluate the generation quality, we follow the previous work (Su et al., 2022) utilizing GQA metries: unigram F1 and ROUGE-L (Lin, 2004). However, the widely used n-gram similar- ity metries often fail to discriminate the halluci- nated/incorrect answers and correlate weakly with human judgments (Lee al., 2021; Zhou al., 2021). We introduce Med-NLI (Medical Natural Language Inference) to assess the logical consis- tency/entailment of generated answers with the pro- vided context or the reference answer. We adopt SciFive (Phan et al., 2021), a T5 model pre-trained extensive biomedical corpora. Our evaluation occurs at two levels: Sample-level Med-NLI evalu- ates whether each generated answer entails (1), is neutral (0), or contradicts (-1) the context or the reference answer. Sentence-level Med-NLI deter- mines the same but for each individual sentence within the generated response. We also use CTRL- Eval (Ke et al.. 2022), an unsupervised, reference- free, and task-agnostic evaluation metric that as- sesses generation from various aspects by formulat- ing each aspect into multiple text-infilling tasks. In our work, we specifically employ the consistency aspect of this metric.
Addressing these challenges requires models to recall factual knowledge, contextual understand- ing, and reasoning abilities. Further exploration and development of these capabilities in LLMs are necessary to improve the reliability and trustwor- thiness of generation systems.
70.
Fact Inconsislency
Query Inconsistency Tangentiality
Incidence (%)
50-
1.2
30-
10-
Vicuna
Alpacal
ChaiGPT
MedAlpara
Robin-M
Figure 2: The incidence of each category of problematic answers in each model.
3.4 Results and Discussion
The Effect of Fine-Tuning on Medical Domain LLMs fine-tuned on medical domain texts (Han et al., 2023; Diao et al., 2023) have demonstrated
Table shows the experimental results on auto- matic metrics over the test sets from five datasets.

---

Med
Model
CtrlEval
FlT
R-L
Spl
Sent
4.37e-05
PubMedQA
Google Ngram Frequency
Vicuna-7B
4684
.919
-1.95
12.06
Vicuna-71
.6380
.63126
-1.74
3.13e-05
Alpaca-Lora-7B
0940
1002
-3.25
9.15
11.09
Alpaca-Lora-7B_J
4640
.4475
-1.85
13.69
13.42
CharGPT
5850
.4199
-2.09
18.17
13.48
CharGPT L
6824
.6598
-1.73
23.45
16.54
1.17e.05
MedAlpaca- 7B
2050
.2112
-3.30
9.90
11.20
MedAlpaca--Te 7p L
4720
.4545
-2.38
15.41
14.45
Rohin-medical-- 7B
2900
.2900
-6.73
3.50
3.18
9.38e-08
0
MEDIQA2019
Fine
Fact Inconsistency inconsistency
Query
Tangentialty
Vicuna-7B
8400
.6330
-3.06
23.94
12.81
Vicuna-7B__
8933
.6868
-2.50
24.65
13.80
Figure 3: The Google Ngrams Frequency of each cate-
Alpaca-Lora-7B
72266
6499
-2.48
5.96
4.83
Alpaca-Lora-7B_JJ
8400
.6565
-2.37
10.93
8.35
gory of problematic answers.
ChatGPT
.7667
.741
-2.77
20.02
11.35
ChatGPT_L
8067
.7180
-2.70
21.53
11.85
get eczema?", Robin-medical generates: "(A) All
MedAlpaca-78
6333
5329
-3.08
8.06
6.95
(B) 10% (C) 20% (D) 30%. Output: A. "The dis-
MedAlpaca-78
.7200
.5531
-2.84
11.14
9.04
Rohhin-meheal-777
72000
7414
crepancy between MedAplpaca and Robin-medical indicates that instruction learning is more suitable for LLMs than non-instruction tuning our tasks.
MASH-QA
Vicuna-7B
8103
6403
-2.46
14.75
9.82
Vicuna-7B
8381
.7518
-2.06
20.69
13.47
Due to Robin-medical ' relatively poor generation performance, we exclude it from further experi- ments.
Alpaca-Lora-7B
.2266
.6492
-1.66
15.01
11.71
Alpaca-Lora-7B
8363
.7812
-1.84
15.23
11.95
CharGPT
.6855
.6425
-2.12
23.34
15.28
ChatGPT_L L
.7044
.7476
15.92
MedAlpaaaa-Pe
5629
.4705
-2.28
The Effect of Frequency Considering the im- practicality of measuring frequency according an LLM's pre-train corpus, we use Google N- grams as a proxy of the text distribution in the natural world and pre-training corpora. We ran- domly select 100 samples generated by the general models. We exact the keywords or topics of the questions, which usually are disease names. We take average frequencies between the years 1950- 2019 (McKenna et al., 2023) of these keywords. As in Figure the problematic answers have lower frequencies than fine ones. This suggests that low frequency may be a potential cause of hallucina- tions, which requires more exploration to prove.
MedAAppaca-7B
74455
.6983
-1,96
13.47
11.77
Rebhinmmddicll 7B
.080
.6378
-.13
4.39
5.66
MedQuAD
Vicuna-7B
8411
6564
-2.56
19.64
11.87
Vicuna-7B.
8503
.7355
-2.47
24.04
14.73
Alpaca-Lora-7B
8104
.7580
-2.29
11.86
Alpaca-Loma-BB
8443
.7723
-2.26
14.34
11.25
ChatGPT
8000
.6820
-2.75
25.59
16.01
ChaiGP I.
8317
.7597
-2.57
27.19
16.08
MedAlpaca-77B
6634
.3328
-2.80
12.19
10.61
MedAlpaca-77B 7B_
8343
.7777
-2.60
12.19
10.96
Robin mechii--eiiill-aii 7B
0775
5656
-3.78
4.88
5.96
LiveMedQA2017
Vicuna-7B 7B
.481
.5212
14.63
Vicuna-7B_LL
.6711
.5967
-2.00
26.75
15.21
Alpaca-Lora-7B
3365
.2400
-1.73
12.90
9.68
Alpaca-Lora-BB
.9622
.5090
-1.72
13.22
9.79
CharGPT
6442
.4879
-2.15
25.47
14.70
ChatGPT L
8462
.5627
-2.09
25.93
14.74
MedAlpaaa-778
2019
.1765
-2.18
10.79
8.87
Hallucination Mitigation Method
MedA/ppcca-77 7BB I.
.3750
.2682
-2.17
13.82
10.47
Robin-medical- 7B
.3077
.6277
-5.25
2.40
2.58
4.1 Methodology
Table 2: Automatic evaluation results LLMs with our interactive self-reflection loop (_L) and baselines. "Spl" and "Sent" mean sample and sentence level.
To address the issue of hallucination, propose iterative self-reflection process that leverages the capabilities of LLMs in generating and refining responses. Illustrated in Figure 4, our methodology comprises three loops: Factual Knowledge Acquir- ing Loop, Knowledge-Consisten Answering Loop. and Question-Einaillmen Answering Loop.
improved performance on certain types of ques- tions, such multi-choice QA, benefiting from the availability of abundant training data. How- ever, their performance in GQA tasks suffers from issues like unrelated content, grammar issues, un- warranted templates, non-existent references, and a lack of explanatory reasoning. As in Table 2, bobin-medical obtains the lowest Fl and Rouge-L scores. For example, given the question: "Who can
Factual Knowledge Acquiring Loop First, the model generates background knowledge based
https////ooooooogl//... google. com/ngrams charts the frequencies of any set of search strings using yearly count of n-grams found in printed sources.
"The website

---

Refine Knowledge
Medical
Prompt: The fnchuaiity score for the inciefadgge -1.2 lless than
The consistency score for the so2sdesige fless
Prompt:
(3)
What causes
THBESHOLD FACTUAL)
THRE SHOLD CONS) which means the alignment and consistency between response knowledge are response to improve its consislency.
Noonan syndrome?
which means the knowledge is not strongly
supported by ampirical evidence. Please
the
Unimaled
refine knowledge
factuality
Nan-Factaal
Nan-Comiiner
Acquire Knowledge
Try Answering
Provide background knowledge to answer the given queetion: "Whet causes
Prompt:
Factual
Prompt: Refer the knowledge: "(fnal, knowledge)
Coreestinn
question: "What causes Noonan syndrome? with one paragraph.
Noonen syndrome??
Scorer
LLM
Sconer
Emideed
Generated Knowledge
Generated Anewer
Final Answer
Noonan syndrome genelic
disorder that
mutation.
Noonan syndrome caused mutations changes cartain genes...
Noonan syndrome caused by genetic mutation on chromosome 12...
in specific gene called the "papilion gane.
Figure 4: The overview of interactive self-reflection method, comprising (1) Factual Knowledge Acquiring Loop (yellow), (2) Knowledge-Consise Answering Loop (green), and (3) Question- Entailment Answering Loop (black).
the provided question. This step capitalizes on the inherent ability of LLMs to synthesize context- relevant information, forming the foundation for the subsequent scoring and refining stages.
knowledge to improve its factuality."
This generate-score-refie strategy is repeated interactively until the generated knowledge reaches the satisfactory factuality level. This iterative pro- cedure fosters dynamic and iterative interaction between the system and its generated knowledge. And ensures that the model progressively refines the produced background knowledge, adhering it to established facts.
Then, a factuality evaluation of the generated knowledge is conducted with a customized and reference-free scorer. The factuality scorer is de- signed through in-context instruction learning with the following formula:
Fs(k|D, Q) Σm (og.(ks/ket. T(D,))) (1)
Knowledge-Consieee Answering Loop Once the generated knowledge attains the requisite qual- ity, the model proceeds to generate an answer based on the provided question and the final knowledge with the template: "Refer to the knowledge: "final_ lionallkowwedge""" and answer the question: XXX with one paragraph. " A consistency evaluation of the generated answer is conducted with CTRLEval (introduced in 3.3). If the generated answer's consistency score low- ers the threshold, the model is prompted to intro- spect, self-correct, and revise the answer with "The consistency score for the knowledge is XXX (less than THRESHOLD_COME < which means the alignment and consistency between response and knowledge are low. Please refine the response consistency."
The knowledge to be evaluated [ki, ..., km..
is the few-shot demon- strations that are annotated examples and Q is the given question. T(-) is the prompt template includ- ing the definition of factuality and task description: "Based on Question, please generate the factual knowledge. To do this, please consider these factors: Verifiability, Objectivity, and Reliability of Source. Note that this evaluation should be based on the best available medical Mnowledge.Annnqusstiion AnAnquestion: \nKnowledge: In-context instruction learning is verified in the aspect of relevance, fluency, informativeness, etc. on text generation tasks (Fu et al., 2023).
If the factuality score is lower than the thresh- old set in the evaluation phase, we instruct the model self-reflect and refine the knowledge with the following prompt: "The factuality score for the knowledge is XXX (less than THRESHOLD_FACTUANE
improve its
This scoeaatt-ccor--efinee strategy is repeated until the generated answer reaches the consistency level. This iterative procedure ensures that the model progressively refines the produced answer aligning with the vetted background knowledge, thus maintaining its integrity.
which means the knowledge is not strongly supported by empirical evidence. Please refine the

---

Query- Inconsistent
Fact- Inconsistent.
tradicts cannot be verified by the reference con- texts or websites. "Fact-Consistent"" means that the answer sentence is supported by the given contexts or websites. "Generic" means that the sentence the answer has no statement to judge. Please see Appendix D for details.
Model
Tangentiality.
Vicuna-7B
0.67%
6.04%
8.69%
Vicuna-7B_L L
0.00%
2.00%
7.38% %
ChaiGPT 0.00%
18.00%
8.06%
ChaiGPT_L
0.00%
17.33%
6.33%
Table 3: Human evaluation results on PubMedQA.
1
4.2.2 Results
Mudel
CtrlEval
Fl
R-L
Spl
Sent
Automatic Evaluation Table 2 presents the auto- matic evaluation results for our self-reflection loop approach and the basclines that directly generate an- swers. We observe the superior performance of our method compared to the baselines, as evidenced by both classic overlap metrics and hallucination metrics across all five datasets.
Vicuna-7B (ours)
6380
.3266
-1.74
16.95
13:47
wlo refinement
4520
-5999
-187
16.90
13.13
w/o aspect
4940
6276
-1.75
16.92
13.65
wlo num
6320
.5155
-2.23
16.92
13.35
ChaiGPT I. (ours)
.824
.6598
-1.73
23.45
16.54
w/o refinement
$180
.5422
-1.86
19.60
15.25
wlo aspect
.5520
.6373
-1.87
19.34
15.46
wlo num
21.25
15.97
Our method demonstrates a remarkable increase in MedNLI. For example, Alpaca-Lora-77 with self-reflection loop gains around three times larger than baseline for Sample- and Sentence-level MedNLI on PubMedQA. The improvement in Fi and Rouge-L. scores, which are overlap-based met- rics, relatively modest sometimes. This is pri- marily due to the inherent reliance of these metrics on the accuracy of the golden answers. As such, even though the generated answers exhibit high quality, they may differ from the golden answers, thus impacting the performance of these metrics.
Table 4: Automatic evaluation results for Ablation Study PubMedQA.
Answering Loop After the above two loops, we evaluate the generated an- swer's entailment via sentence-BERR embedding similariy (Reimers and Gurevych, 2019)* to ensure the entailment and answerability. the generated answer does not meet the satisfactory entailment level, the process returns to the initial stage of the framework, and the entire cycle is repeated, iterat- ing through the aforementioned stages.
Notably, our method showcases its effectiveness across language models with varying parameters, including 7B and 175B, across all five datasets.
4.2 Experiments
4.2.1 Evaluation
This robust performance highlights the general- izability and scalability of our approach, further validating its efficacy in the context of medical question-answeriig tasks.
In addition to the automatic metrics described in $ 3.3, we conduct human evaluations using Ama- zon Mechanical Turk6 to further assess the quality of generated answers. The human evaluation for question-consiitency and tangentiality is conducted at the sample level, where we ask annotators to cat- egorize each answer as Query-Inconsisentt, Tan- gential, or Entailed. "Query-Inconsistenn" means that the answer provides information unrelated to the query or is non-sensical and meaningless. "Tan- gential" means that the answer provides informa- tion related the question but doesn't directly address the question. "Entailed" means that the answer directly addresses the question. The hu- man evaluation for fact-consistency conducted at the sentence level, where we ask annotators to categorize each sentence in the answer as Fact- Inconsistent, Fact-Consistent, or Generic. "Fact- Inconsistent" means that the answer sentence con-
Human Evaluation The results in Table 3 demon- strate that our method successfully reduces the per- centage of query inconsistency, tangentiality, and fact inconsistency both Vicuna and ChatGPT, which aligns with the findings from the automatic evaluation. The inter-annotator agreement, mea- sured using Krippendorff's ' alpha (Krippendorff, 2011), 1), indicates high agreement among the anno- tators, with values exceeding 0.8 for question in- consistency and tangentiality, and exceeding 0.7 for fact consistency. Please see Appendix D for detailed results.
4.3 Discussion
4.3.1 Ablation Study
To assess the individual contributions of specific components in our method, we conduct an ablation analysis. The results Table demonstrate the
'ht thttp:://uugiingaacc.. sentence- -tterettataaatteeeemmm-e-------- chttps://wwa wwtp://tukkcoom com/

---

Question
Directly Generated Answer
Generated Answer with Loop
Golden Answer
The evidence from the know]- edge suggests that there is berween
Dose end tidal hon dioside mea-
Measurement tidal caston dioxide (E:CO2) levels in extremely low birth weight infanss in the first of arterial carbon dioside level. This be- cause the relationhip between EIC02 and arte- rial CO2 well eszahished in preterm infants, and there may he significant variations the acen- mmlltion and elimination of CO2 during the first week of life.
Yes. There is good correlation and agreement berween end-tidal CD(2) and arterial (OC2 in EL&W infants in the B/C003 range 30-50 mimig. Bod-
surement correlate
life may
rellable indicator
positive correlation end-tidal carhon dioxide (FTCO2)
with arterial car- bon dioxide in ex- tremely low birth
and antriial carbon dioside (ACO2)
measurements in extremely low birth weight infants
tidal CO12) monitoring can be belpful trending for screening abnormal PsCO(2) valoes in ELBW infants in first week of life.
finst week of life. This means that the two measurements are likely to be reliable indicators of each other...
weight infants in
the first week of life?
Dear Nadine,
The independest protective effect of fe- male gender on multiple organ failure and nesocomial infoction rutes remains significumi in both premenopussal and postnenopassal women when com- pared with similarly aged men. is contrary to previous experimental studies and the known physiologic sex hormone changes that occur afte menopasee in women. These results suggest that factors mones may be responsible for gender- based differences after injury.
Thank you your fetter discussing the potentil role of hormones in the gender dimorphism ob- hemontagic
Characterization
served response injury shock. While early research in this area promis- ing, mach more work remains he done fully understand the mochanisms involved.
The gender dimorphism injury and hemorrhagic shock is com- plex topic. Research males and females may have different responses to injury und hemorrhagic shock serms of blood pressure, organ function, and hormone levels..
of the gender dimorphism after injury and bem- orthagic shock: are bommonal differences re- sponsible?
In your experiments. you observed that PAG bazed GAllAArrgic signating was impaired in responsc to injury and bemorfaggc shock in adult mate rany, which could contribute to the development of depression- and antiery-tiike behavior...
than
Table 5: Example answers generated directly vs. with our self-reflection loop by Vicuna. We underline and bold the parts relevant to the question or aligning with the golden answer; underline and italicize the parts unrelated to the question or conflicting with the golden answer.
performance of different variations of our approach in terms of automatic hallucination metrics.
response to improve its consistency." " As in Table 4, the loop without score number attains lower MedNLI and CtrlEval than the full implemen- tation, indicating that the explicit score numbers contribute to better refinement.
Effect of Refinement To evaluate the impact of refinement, we omit the scoring and refining stages and only conduct the generation stage. We acquire background knowledge based on the question and then answer based on the knowledge. As in Table 4, the answers generated by the loop without re- finement attain lower MedNLI and CtrlEval, which means the refinement is helpful for fewer halluci- nations and higher consistency.
4.3.2 Case Study
The examples in Table 5 demonstrate the effective- ness of our method in addressing fact and query inconsistency. In the first line, the directly gener- ated answer inaccurately states that EtCO2 levels may not be a reliable indicator of arterial CO2 lev- els, while ours accurately indicates the positive correlation between these measures, aligning with the golden answer and fact. In the second line, the directly generated answer is in an email format and mentions irrelevant information: It greets Nadine and mentions PAG-based GABAAergic signaling of adult male rats and depression- and anxicty-like like behavior. In contrast, our responses are more rele- vant and directly address the given question. Please see Appendix for more examples.
Effect of Aspect Description To evaluate the im- pact of providing explicit aspect descriptions for improvement, we omit mentioning the specific as- pect that requires refinement. Instead, we instruct the model to engage in self-reflection by using a more general instruction: "Please refine the wwwlledgerrespoonee
As in Table 4, the an- swers generated by the loop without aspect attain lower McdNLI and CtrlEval,, which means the as- pect description can lead to fewer hallucinations and higher consistency.
Effect of Score Number To examine the influ- ence of exact scores in the evaluation phase, we omit presenting exact values. Instead, we only describe the aspect that requires improvement in the instructions: "The knowledge is not strongly supported by empirical evidence. Please refine the knowledge to improve its factuality." and "The alignment and consistency between response and knowledge are low. Please refine the
5 Conclusion and Future Work
Hallucinations in generation tasks pose significant challenges to AT's accountability and trustworthi- ness. We investigate this problem thoroughly and systematically in the context of medical GQA in general and domain-specific LLMs. To address this challenge, we propose an iterative self-refiection method by adopting a senerate-score-refinn strat- egy on background knowledge and answers. Our

---

method is empirically proven effective, generaliz- able, and scalable in reducing hallucinations. In future work, we will investigate underlying causes hallucination, examine this phenomenon in other generation tasks and extend our method to address challenges associated with these tasks.
Acknowledgement
The authors would like to thank Samuel Cahyawi- Jaya for the discussion and suggestions.
This work has been supported by the China NSFC Project (No. NSFC2IEG14), SAAIR Project (No. Z1286), and HKJCCT2IEG01 (RG192).
Limitations
While our methodology shows promise in miti- gating hallucination, it does not entirely eliminate the possibiliity. The model might still generate un- grounded information, especially in complex or ambiguous scenarios. Currently, our method is still in its early stages and not yet ready for direct real-world deployment. should be viewed as a complementary approach alongside other methods, such as retrieval, with the potential to contribute to more robust application systems in the future.
References
Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushmen 2019. Overview of the mediqa 2019 shared task textual inference, question en- tailment and question answering. Proceedings of the 18th BioNLP Workshop and Shared Task, pages 370-379.
Emily Alsentzer, John Murphy, William Boag, Wei- Hung Weng. Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clini- cal bert embeddings. In Proceedings of the 2nd Clin- ical Natural Language Processing Workshop, pages 72-78.
Our study primarily focuses English medi- cal queries, limiting the generalizability to other languages, domains, and modalities. Further re- search necessary to investigate the potential laneufee challenges, domain adaptation challenges, and multimodality fusion challenges. By addressing these aspects, we can adapt our pro- posed methodology to different contexts, resulting in more comprehensive understanding and ap- plication of approach across diverse linguistic, multi-domain, and multimodal settings.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2332.0444
Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman 2017. Overview of the medical question answering task at trec 2017 liveqa. In TREC 2017.
While this paper has addressed certain issues in this domain, numerous challenges remain, such as empowering LLMs with high-level ability.
Asma Ben Abacha and Dina Demner-Fushman 2019. A question-entailment approach to question answering. BMC Bioinform. 20(1):51:1--11:233
Ethics Statement
We used publicly available or synthetic datasets for our experiments, avoiding any potential harm to individuals or groups. The data used in this study were carefully selected and processed to ensure pri- vacy and confidentiality. No personally identifiable information used, and all data were anonymized prior to analysis.
Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. Overview of the mediqa 2019 shared task on textual inference, question en- tailment and question answering. In ACL-BiONLP 2019.
Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 1499-1510.
In considering the application of our research findings, we acknowledge the potential risks and ethical considerations associated with AT-assisted healthcare. The hallucination issue, in particular, can have significant implications for patient care and clinical decision-making Our work is guided by the principle of "do no harm", aiming to enhance the reliability and safety of medical QA systems rather than substitute for professional medical judg- ment.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 197-1

---

Collin Burns, Haotian Ye. Dan Klein, and Jacob Stein- hardt. 2022. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Representa- lions.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, 7ac Haffield-oodss, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Wei-Linn Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Zakaria Kaddari, Youssef Mellah, Jamal Berrich, Toumi Bouchentouf, and Mohammed G. Belkasmi. 2020. Biomedical question answering: A survey of meth- ods and datasets. In 2020 Fourth International Con- ference On Intelligent Computing in Data Sciences (ICDS), pages 1-8.
Bhuwan Dhingra, Danish Danish, and Dheeraj Ra- jagopal. 2018. Simple and effective semi semi-superr supervisdd question answering. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguisties: Human Lan- guage Technologies, Volume 2 (Short Papers), pages 582-587.
Pei Ke. Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022. Cirleval: An unsupervised reference-free metric for evaluat- ing controlled text generation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages
Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. 2023. Lmflow: An extensible toolkit for finetuning and inference of large foundation models.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Joongbo Shin, and Ky- omin Jung. 2021. KPQA: A metric for generative question answering using keyphrase weights. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2105-2115, Online. Association for Computational Linguisties.
Jinlan Fu, See-Kiong Ng, Zhenghao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire, arXiv prepriat arXiv:2302.04100
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasu- pat. and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages
PMLR.
Tianyu Han, Lisa C Adams, MichhMissllss Papaioan- nou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca-an open-source collection of medical conversational ai models and training data. arXiv preprint 2304.:::3300047..
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmeened generation for knowledge-intensivee nlp tasks. Advances in Neu- ral Information Processing Systems,
Gautier Izacard, Patrick Lewis. Maria Lomeli, Lu- Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03200
Chenliang Li, Bin Bi, Ming Yan. Wei Wang, and Song- fang Huang. 2021a. Addressing semantic drift in gen- crative question answering with auxiliary extraction. In Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 2: Short Papers), pages 942-947.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):
Jing Li, Shangping Zhong, and Kaizhi Chen. 2021b. Mlec-qa: A chinese multi-choice biomedical ques- tion answering dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8862 8874.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 2567-
Chin-Yen Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
guage Processing (EMNLP-IJCNN 2577, Hong Kong, China. Association for Computa- tional Linguistics.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.
Truthfulqa: Measuring how models mimie human falsehoods. arXiv preprint 200.v2210..07958.

---

Lang Liu, Junxiang Ren, Yuejiao Wu, Ruilin Song, Zhen Cheng, and Sibo Wang. 2022. A pre-trained language model for medical question answering based do- main adaption. In Natural Language Processing and Chinese Computing; 11th CCF International Confer- ence, NLPCC 2022, Guilin, China, September 24-25, 2022, Proceedings, Part II, pages 216-227. Springer.
2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Ankit Pal, Logesh Kumar Umapathi, and Malaikan- nan Sankarasubbu. 2022. Medmeqa: A large-scale multi-subject multi-choice dataset for medical do- main question answering. In Conference on Health, Inference, and Learning, pages 248-266 PMLR.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303.0S8: 08896..
Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering electronic medical records. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357-2368.
Nick McKenna, Tianyi Li. Liang Cheng, Moham- mad Javad Hosseini, Mark Johnson, and Mark Steed- man. 2023. Sources hallucination by large lan- guage models on inference tasks. arXiv preprint arXiv:2305. 14552.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis. Majid Yazdani. Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin. Jean Maillard, et al. 2021. Kilt: a benchmark for knowledge in- tensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis. Hannaneh Hajishirzi. and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in io-context learning work? arXiv preprint 2rXiv:2202. 12837.
Long N Phan, James Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Gré- goire Altan-Bonnee 2021. Scifive: text-to-text transformer model for biomedical literature. arXiv preprint 10l.:2306005998
Diego Mollá, María Elena Santiago-Martinez, Abeed Sarker, and Cécile Paris. 2016. A corpus for research in text processing for evidence based medicine. Lan- guage Resources and Evaluation, 50:705-727.
Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. 2020. Covid-qa: A question answer- ing dataset for covid-19. 19. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020.
Patrik Puchert, Poonam Poonam, Christian van Onzenoodt, and Timo Ropinski. 2023. Llmmaps- visual metaphor for stratified evaluation of large language models. arXiv preprint arXiv:2304.00 00400.47.
Roser Morante, Martin Krallinger, Alfonso Valencia, and Walter Daelemans. 2012. Machine reading of biomedical texts about alzheimers disease. In CLEF 2012 Conference and Labs of the Evaluation Forum- Question Answering For Machine Reading Evalua- tion (A4MRE)), Rome/Forner, J.(edit.], J; ea, pages 1-14.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume Short Papers). Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert, Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Jessica Morley, Caio CV Machado, Christopher Burr, Josh Cowls, Indra Joshi, Mariarosaria Taddeo, and Luciano Floridi. 2020. The ethics of ai in health care: a mapping review. Social Science & Medicine, 260:113172.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 5418-54414 1418-5426.
Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim. Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332..
Max Savery, Asma Ben Abacha, Soumya Gayen, and Dina Demner-Fushman 2020. Question-driver sum- marization of answers consumer health questions. Scientific Data, 7(1):222.
OpenAI. 2023a. Chatgpt: Optimizing language models for dialogue.
OpenAl. 2023b. abs/2303.08774.
Gpt-4 technical report. ArXiv,
Sarvesh Soni and Kirk Roberts. 2020. Evaluation of dataset selection for pre gre-training and fine-tumi muenttning transformer language models for clinical question answering. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5532- 5538.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.

---

Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before generate! faithful long form question answering with machine reading. In Findings the Association for Computational Linguistics: ACL 2022, pages 744- 756.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le. Deany Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. Transactions of the Association for Computational Linguistics, 6:287-
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2023. Context generation improves open domain question answering. In Findings of the Association for Computational Linguistics: EACL 2023, pages 193-808, Dubrovnik, Croatia. Association for Com- putational Linguistics.
302.
Yan Xu. Etsuko Ishii Samuel Cahyawijaya, Zihan Liu, Genta Indra Winata, Andrea Madotto, Dan Su, and Pascale Fung. 2022. Retrieval-fres knowledge- grounded dialogue response generation with adapters. In Proceedings of the Second DialDoc Workshop on Document- grrentt-grdddddd Dialogue and Conversational Question Answering, pages 93-107.
Zhiqing Sun, Xuezhi Wang, Tay, Yiming Yang. and Denny Zhou. 2023. decitation-augmentt language models. In The Eleventh International Conference on Learning Representations.
Zhangyue Yin, Qiushi Sun, Qipeng Guo. Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don' ton't know? In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 8653-8666 8665, Toronto, Canada. Association for Computational Linguistics.
Simon Suster and Walter Daelemans. 2018. Clier: dataset of clinical case reports for machine reading comprehension. In Proceedings of NAACL-HLT, pages 1551-1563.
Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. 2021. Understanding the capabilities, limi- tations, and societal impact of large language models. arXiv preprint 2tti:::202222200000
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu. Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh Inter- national Conference on Learning Representations.
Raphael Tang, Rodrigo Nogucira, Edwin Zhang. Nikhil Gupta, Phuong Cam, Kyunghyun Cho, and Jimmy Lin. 2020. Rapidly bootstrapping question swering dataset for covid-19. arXiv preprint arXiv:2004.11339
Hao Zhou, Minlie Huang, Yong Liu, Wei Chen, and Xiaoyan Zhu. 2021. Earl: Informative conversation generation with entity -agnosicc representation learning. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2383-2395
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:230..1397I.
Noowledge-gouunten
Ming 7hu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy. 2020. Question answering with long multiple-span answers. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 3840-3849, Online, Association for Computa- tional Linguistics.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Toannis Partalas, Matthias 7schunke, Michael R Alvers. Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answer- ing competition. BMC bioinformatics, 16(1):1-24
Cunxiang Wang, Pai Liu, and Yue Zhang. 2021. Can generative pre-trained language models serve as knowledge bases for closed-book qa? In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3241-3251.
Eric J. Wang 2023. Alpaca-lora.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:210..0165..

---

A Dataset
B Implementation Details
PubMedQA (Jin et al., 2019) is biomedical QA dataset consisting of 1 expert labeled, 61.2k unla- beled, and 211.3k artificially generated instances. It includes questions derived from research article titles, abstracts as the context, long answers from abstract conclusions, and concise yes/no/maybe answers.
Vicuna is an open-source foundation, equipped with an enriched dataset and a scalable, user- friendly infrastructure. It is trained by fine-tuning LLaMA (Touvron et al., 2023) user-shared con- versations derived from ShareGPT. Vieuna gener- ates more detailed and well-structured responses compared its predecessor, Alpaca, with quality equivalent to ChatGPT. We use the code and check- point from the official library'. We run generation on two GeForce RTX 2080 GPUs with the follow- ing settings: temperature 1.0, max new tokens 512, others are default. The max number of the main loop, knowledge loop, and response loop is 3. The factuality, consistency, and entailment threshold are -1.0, -5.0, and 0.8, respectively. The demo num- ber for the factuality scorer I for PubMedQA; 3 for MedQuAD, MEDIQA2019, MASH-QA, and LiveMedQA2017.
MedQuAD (Ben Abacha and Demner-Fushman, 2019) comprises 47,457 medical QA pairs derived from 12 National Institutes of Health (NIH) web- sites. It spans 37 question categories and covers various medical topics including diseases, medica- tions, and diagnostic tests.
MEDIQA2019 (Ben Abacha et al., 2019) chal- lenge includes three tasks: Natural Language In- ference (NLI), Recognizing Question Entailment (RQE), and QA in the medical domain. For this par- ticular study, we focus on Task 3's dataset, which centers around medical question answering and consider the golden answers with scores 3 and 4 as the correct responses.
Alpaca-LoRA replicates the results of Stanford's Alpaca model, employing a technique known as Low-Rank Adaptation (LoRA). This instruct model, comparable in quality to GPT-3.5, is ca- pable of operating under low resource conditions. Remarkably, even without hyperparameter tuning, Alpaca-LoRA generates outputs par with the original Stanford Alpaca model. We use the code and checkpoint from the official library We run generation on two GeForce RTX 2080 GPUs with the following settings: temperature 1.0, max new tokens 512, others are default. The max number of the main loop, knowledge loop, and response loop is 3. The factuality, consistency, and entail- ment threshold are -1.0, -5.0, and 0.8. respectively. The demo number for the factuality scorer is 1 for PubMedQA, MASH-QA, MEDIQA2019, and MedQuAD; 2 for LiveMedQA22117
LiveMedQA2017 (Ben Abacha et al., 2017) con- tains annotated medical QA pairs, aiding the devel- opment of question analysis and answering systems. The test questions span 26 types within five main categories, each question comprising one or more uub-questions, a focus, and a type. All reference answers, curated from reliable sources and vetted by medical experts, include a URL and relevant comments. A minimum of one reference answer is provided for each test question.
MASH-QA (Zhu al., 2020) is dataset from the consumer health domain designed for Multi- plc Answer Spans Hcalthcare QA. It includes 34k QA pairs where answers may be drawn from non- sequential sections of a lengthy document.
In these datasets, except PubMedQA, answer an- notation undertaken through a manually involved process of extracting and copying from authentic web content. It is imperative to note that although the annotated answers are indeed pertinent and ver- ifiable, it has not undergone refinement, thereby improvements are needed in harmonizing the con- textual coherence and problem linkage. For an example from LiveMedQA20017, given question: "Do Zolmitriptan 5mg tablets contain gluten?", the answer is "Zolmitriptan tablets are available as 2.5 mg (yellow and finctionally-score and mg (pink, not scored) film coated tablets for oral ad- ministration..." "
ChatGPT is designed to interpret prompts and furnish comprehensive responses. It employs Reinforcement Learning from Human Feedback (RLHF), similar InstructGPT (Ouyang al., 2022), albeit with minor differences in data col- lection methodology. The initial model trained through supervised fine-tuning, where human AI trainers conducted dialogues, simulating both the user and AI assistant roles. We use the official API from to generate answers. The max number of
htttps://ggthub..mmm/m/sss///// s 'https://github. http://sshtat.ccmmmmmmm////assss

---

the main loop and response loop 3. The max number of the knowledge loop for MedQuAD and for others. The factuality, consistency, and entailment threshold are -..0, -5.0, and 0.8, respec- tively. The demo number for the factuality scorer is 1 for PubMedQA, MASH-QA, MEDIQA2019, and MedQuAD; 3 for LiveMedQA2017.
airways and small air sacs in your lungs, which can lead to a variety of lung diseases including chronic bronchitis, emphysema, and lung cancer. It also decreases your lung capacity and makes it harder for your lungs to defend against infections and clear out mucus. An
Question: Is it safe to take aspirin every day? Knowledge: For some people, taking aspirin every day can help prevent heart attacks or strokes. However, daily aspirin isn't appropriate for everyone. It cause side effects like gastrointestinal bleeding and isn't t recommended for people with certain health conditions or who take certain medications. Always consult with a heal thcare professional before starting any new medication regimen. An
MedAlpaca is a large language model meticu- lously fine-tuned for medical dialogue and QA applications upon the frameworks of Alpaca and Alpaca-LoRA. These models have been trained using an array of medical texts, including medi- cal flasheards, wikis, and dialogue datasets. We use the code and checkpoint from the official li- brary'". We run generation on two GeForce RTX 2080 GPUs with the following settings: tempera- ture 1.0, max new tokens 512, others are default. The max number of the main loop, knowledge loop, and response loop is 3. The factuality, consistency, and entailment threshold are -1.0, -5.0, and 0.8, respectively. The demo number for the factual- ity scorer is for PubMedQA, MEDIQA2019, and LiveMedQA2017; for MASH-QA, and MedQuAD.
D Human Evaluation
We conduct the human evaluation assess our method's performance GQA for the ability reduce hallucination. In detail, we randomly se- lect 50 question-answe annweer pairs directly generated by Vicuna and 50 question-ansee anwwerr pairs generated with our proposed loop. Each sample is evaluated by three different annotators rule out potential bias. We specify that annotators must meet the fol- lowing qualifications: Their Human Intelligence Task (HIT) approval rates are greater than or equal to 95%, and the numbers of HITs approved are greater than or equal to 5000. The annotators are located in Australia, Canada, the United Kingdom, and the United States. The evaluation cost for fact inconsistency is 0.3 US dollars per sentence, and for question inconsistency and tangentiality, it is 0.15 US dollars per answer. Figure 5 and 6 are the user interfaces (UIs) on Amazon Mechanical Turk for human evaluation of fact-consistency and question-consistency and tangentiality, respectively. The instructions, questions, and examples for anno- tators are shown.
Robin-medical is large language model fine- tuned in the medical domain via LMFlow, exten- sible toolkit for finetuning and inference of large foundation models. We use the code and check- point from the official library!l, We run generation on two GeForce RTX 2080 GPUs with the follow- ing settings: temperature 1.0, max new tokens 512, others are default.
C Factuality Scorer
Here are demonstrations for our factuality scorer: Question: What are the risk factors for heart disease?\n Knowledge: Risk factors for heart disease can be categorized into modifiable and non-modifiable Modi fiabli risk factors include high blood pressure, high cholesterol, smoking, unhealthy diet, physical inactivity, obesity, and excessive alcohol use. Non-modifiable risk factors include age, gender, family history, and race ethnicity. An Question: How does smoking affect lung health?\n Knowledge: Smoking damages the
To identify the Fact Inconsistency generated answers, we ask annotators to refer the context provided in dataset. As a supplement, we re- trieve the related paragraph from WikiMedical- Terms Dataset 12 via Mantence-BERT embedding
/hhtpp::/githubb https////gggthhb....//eress///// com/Kbbresseem/eccllpaaaaa http://gwtbb..oomoopm/mmmm//////
httt::///hgggnggaee..cotttssssst http://ccccc///aeeeeeeeeeee co/ddatasees//gmminn///kkkkkk medical terms
hhttps://huggingte

---

Instructio
Overtic
sapooad ohe wiicted the ausorfedioy Thore tioo opt ane
poot tovesteer weww conumme wan aporterit my the reveroee comeoo wad veriore
Plean clccits Sutton Inatuztiona and REnERovez ExAM Lt caertion
ixpiitted sentencaui:
deddee loads
My derl swo)
Glven rnference contect helow, whather the Elparrannce stji cortinnee conslsient with
Releomnm Comtest
Coaterts
Referance So
amoune peorinem eake
Nperooz
the
Phos talpe Voosa mno regitaa prpettte Paye aua seppect
Spalected sentanca)
The seriacc Cors stant.
Figure 5: The UI for human evaluation fact-consistency.
Quertor:
the tolk, you pood coo whothe The swor reletd quadios ord dieeit oncwas the txeation optione
Tongentall proide information
dechdinncly
Arawembte commitent diaecly
laknng Coass
Question:
crseid the
Shownkord
Answer:
Siyceneec
Wheter conditent quetion directly quastion?
The arover ii queedonniooe seet. Actopuris
ralot Eupater
arawesbl...
Rullagu sorher 200 se
loue Aps
Fastoss: Topereoorr
Figure 6: The UI for human evaluation on and tangentiality.

---

Model
Query-Incossitent)
Tangentiality.
Entailmmnt
Vicuma-7B
0.67%
6.049
93.29%
Vicuna 7
0.00%
2.00%
98.00%
CharGPT
0.00%
18.00%
X2.0055
CharGiT
0
17.33%
82.67%
Model
Fact-Incomistent/
Fact-Consistent
Generic
Vicuna 7B
8.69%
78.07%
13.24%
Vicumi-7B_L
76.96%
15.66%
ChaGPT
8.06%
65.62%
25.32%
CharGPT_
6.33%
77.8%
15.87%
Table 6: Human evaluation results on PubMedQA.
similariy! This dataset contains over 6,000 med- ical terms and their Wikipedia text. In addition, we ask annotators search for references in Wiki- Doc a medical wiki encyclopedia used by the international community of healthcare profession- als.
The results in Table 6 demonstrate that our method successfully reduces the percentage of query inconsistency, tangentially, and fact inconsis- tency in both Vicuna and ChatGPT, Furthermore. we observe that ChatGPT tends to generate more general sentences compared to Vicuna. Our ap- proach can keep generic responses at a relatively low level while improving fact consistency.
E
Case Study
Table presents some example answers gener- ated by ChatGPT, ChatGPT with our self-reflection loop, and GPT4. While ChatGPT provides a nega- tive answer, both ChatGPT with our relf-reflcction loop and GPT4 offer positive answers that align better with the golden answer. ChatGPT suggests that there is evidence in its available literature that shape analysis can differentiate free-floating internal carotid artery thrombus from atheroscle- rotic plaque CTA. In contrast, the golden answer states shape signature highlights the potential to dis- tinguish the two. ChatGPT with our self-reflcction loop acknowledges the role of shape analysis in dif- ferentiation and mentions works on machine learn- ing algorithms and radiomic features that show promising results in classification. GPT4 also sug- gests shape analysis potentially help differenti- ate, but its subsequent content is unrelated to shape analysis itself. These results demonstrate the effec- tiveness of our method and the capabilities of more powerful LLMs.
Phttpss://uuggiigfacc..oo
sentence-transfoommere Sentennttttraattommmeee/al-liiis wwwp://wwwwww..o.co..

---

Can shape analysis differentiate free-floatin internal carotid artery thrombus from atherosclerotic
Question
plaque in patients evaluated with CTA for stroke or transient ischemic attack?
Yes. We identified five quantitative shape descriptors of carotid FFT. This shape "signature" shows potential for supplementing conventional lesion characterization in cases of suspected FFT.
Golden Answer
As an AT language model, don't ' have access to the most recent studies darahase, but based on the so far, couldn't 't find any evidence that shape analysis can differentiate free- floating internal carotid artery thrombus from atherosclerotic plaque in patients evaluated with CTA for stroke or transient ischemic attack. However, other imaging modalities such as MRI and ultrasound may provide more information differentiate hetween the two conditions. is essential consult with a radiologist or a vascular specialist to make a definitive diagnosis and develop the approoprate treatment plan.
literature available
Answer Generated by ChatGPT
studies, shape analysis can play a significant role differentiating free-floating internal carotid artery thrombus from atherosclerotic plaque in patients evaluated with CTA for stroke or transient ischemic attack. Various machine learning algorithms and radiomic features have been used to analyze the shape of the thrombi and plaques in the internal carotid artery, leading to promising results terms accurate classification. This differentiation is crucial the treatment approaches thrombus and plaque are different.
According
Answer Generated by ChatGPT_L.
Shape analysis can potentially help differentiate a free-floating internal carotid artery thrombus from an atheroselerotic plaque in patients evaluated with computed tomography angiography (CTA) for stroke or transient ischemic attack. However, it is essential to note that shape analysis alone might not be sufficient for a definitive diagnosis, and other imaging features and clinical information should be considered.
A free- free-floating thrombus an intraluminal filling defect that not attached to the arterial wall, while an atherosclerotic plaque is a focal thickening of the urterial wall due to the presence of lipid-rich and fibrous tissue. Here are some shape analysis and imaging features that may help differentiate between them:
1. Shape and uppearance: A free-floating thrombus usually appears us a well-defined, clongated, and tubular filling defect within the arterial lumen. In contrast, atherosclerotic plaques tend to be more irregularly shaped and may have caleifications.
2. Location: Thrombi may occur at any location within the internal carotid artery, while atherosclerotic plaques are more commonly found at the carotid bifurcation, the proximal internal carotid artery, and the origin of the external carotid artery.
Answer Generated
by GPT4
3. Attachment: A free- gee-ffoating thrombus not attached to arterial wall may show "bird's beak" or "meniscus" sign at its proximal end, suggesting that it is freely flouting within the lumen. Atherosclerotic plaques are typically atlached to the arterial wall and may cause luminal narrowing or stenosis.
4. Enhancement: In some cases, a free-floating thrombus may show minimal or no enhancement after the administration of intravenous contrast, while atherosclerotic plaques typically enhance to varying degrees.
It important to recognize that these imaging features are not always definitive, and there may be overlap between the appearance of a thrombus and an atherosclerotic plaque. A comprehensive evaluation that includes other imaging techniques (such as magnetic resonance imaging ultrasound) and clinical information (such patient history, symptoms, and risk factors) may be necessary for accurate diagnosis, Additionally, further research and development of advanced imaging techniques and computational methods could improve the differentiation between these two entities in the future.
Table 7: Example answers generated by ChaiGPT, ChatGPT with our self-reflection loop, and GPT4 We underline and bold the concluding parts aligning with the golden answer and underline and italicize the con- cluding parts confliciig with the golden answer.


Reviews:
[{"review_id": "oJxqJzrIKG", "reviewer": "Reviewer_5JJj", "summary": "", "questions": "1. How are the thresholds for factuality score and consistency score chosen?\n\n2. Did you consider retrieving related knowledge from a corpus using information retrieval techniques from a reference corpus instead of generating the knowledge with a model?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback!\n\n## Response to “Reason To Reject about factuality scorer”:\n\nThe motivation for our work (retrieval-free) and retrieval-augmented generation is different:\n1. Retrieval-augmented works emphasize that the inherent knowledge in LMs is deficient, and they add external knowledge to compensate.\n2. Our work aims to harness and maximize the potential of the model's inherent knowledge. Current research works [1-5] highlight a gap between surface realization and inherent knowledge in NLG tasks. Models can realize they are generating something hallucinated/nonsensical in some way. Our method aims to reduce this gap and enhance the performance of the model itself. As base LLMs become more powerful and inherent knowledge becomes more extensive and versatile, it is crucial to leverage knowledge and unleash its capabilities.\n\nGiven our distinct motivation, we do not use external information during generation, which is similar to chain-of-thought (CoT). Thus, our factuality scorer does not refer to the existing knowledge base. If external knowledge is used for scoring, it is equivalent to introducing knowledge retrieval, which is unfair to the baselines.\n\nCorrelation studies between the factuality scorer and human evaluation were conducted by  [6] to verify the effectiveness of in-context instruction learning in diverse NLG tasks. To supplement, we calculate the average factuality score of factual and non-factual cases classified by humans: Factual cases attain -0.89, which is higher than the average score of non-factual (-1.30). This result shows that the factuality score is positively correlated with human evaluation.\n\nOn the other hand, our primary emphasis lies on the interactive self-reflection pipeline itself instead of the factuality scorer. Since the scorer is replaceable, especially with emerging and more reliable alternatives that may become available in the future.\n\n\n> [1] Yin, Zhangyue, et al. \"Do Large Language Models Know What They Don't Know?.\" Findings of the Association for Computational Linguistics: ACL 2023.\n\n> [2] Burns, Collin, et al. \"Discovering latent knowledge in language models without supervision.\" International Conference on Learning Representations (ICLR), 2023.\n\n> [3] Rajpurkar, Pranav, Robin Jia, and Percy Liang. \"Know What You Don’t Know: Unanswerable Questions for SQuAD.\" Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 2018.\n\n> [4] Kadavath, Saurav, et al. \"Language models (mostly) know what they know.\" arXiv preprint arXiv:2207.05221 (2022).\n\n> [5] Manakul, Potsawee, Adian Liusie, and Mark JF Gales. \"Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.\" arXiv preprint arXiv:2303.08896 (2023).\n\n> [6] Fu, Jinlan, et al. \"Gptscore: Evaluate as you desire.\" arXiv preprint arXiv:2302.04166 (2023).\n## Response to “Question 1 about how to choose threshold”:\n\nThe thresholds are hyperparameters, which are empirical values obtained through extensive experiments. First, we sample 250 generated knowledge/answers and calculate their corresponding factuality/consistency scores. Then, we manually assess their quality to classify knowledge into factual or non-factual and classify answers into consistent or inconsistent. This process allows us to find the score gap between the two classes and derive approximate thresholds for the scores. Then, we tune these hyperparameters to optimize the overall performance outcome.\n\n## Response to “Question 2 about retrieval-related method”:\nYes. As outlined in the Introduction and Limitation sections, our method is parallel to the retrieval-related methods and it can be combined with other methods. It should be viewed as a complementary approach alongside other techniques including retrieval, contributing to more robust application systems in the future. \n\nAs mentioned above, our aim is to explore and maximize the inherent potential of the LLMs, which is different from retrieval-augmented generation. We aim to optimize the performance as much as possible within the context of mastering knowledge.  In cases involving knowledge beyond mastery, such as real-time information, we believe retrieval is an effective way.\n\n## Response to “Typos Grammar Style And Presentation Improvements:\nThanks for the correction!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a detailed insight and analysis on the existence of hallucination by several state-of-the-art LLMs in generative question answering in the medical domain. The tendency to hallucinate is a critical weakness in the LLMs and this work takes a step towards addressing this. \n\n2. An innovative method is proposed that does not use any external references to mitigate hallucination. The method can prove to be useful for QA in other domains as well.", "reasons_to_reject": "Original review:\n\nMy main concern is with the method for generating knowledge and its refinement. We know that LLMs themselves have a tendency to generate hallucinated text. Thus, it may not be ideal to rely on LLMs to generate knowledge. I understand that the generated knowledge goes through a refinement to ensure factuality. However, the factuality scorer does not compare the generated knowledge to any reference (e.g., looking at an existing knowledge base/corpus). The score is predicted based on a model that is instruction-tuned in a few-shot setting. In other words, the method relies on the model to predict how factual the knowledge is. At this point, the reliability of the model in predicting the factuality score is unclear. A correlation study between the model’s predictions of the factuality score and human evaluation would be useful to verify this. \n\nIf the factuality scorer is unreliable, this will impact the answer generation as well since the aim is to make the answer grounded in the final knowledge. \n\nWhile both automatic and human evaluation shows a decrease in hallucinations in the answers, it is also essential to evaluate the knowledge generation module more carefully.\n\nAfter rebuttal:\n\nWhile the rebuttal clarifies and reduces my concerns, I would still like to see a correlation study in this paper for the medical domain. I have however, increased my original soundness score from 2 to 3.", "questions_to_authors": "1. How are the thresholds for factuality score and consistency score chosen?\n\n2. Did you consider retrieving related knowledge from a corpus using information retrieval techniques from a reference corpus instead of generating the knowledge with a model?", "paper_topic_and_main_contributions": "This paper studies the existence and mitigation of hallucination in answers generated by LLMs in generative question answering in the medical domain. In particular, the authors first classify the hallucinated answers into 3 types and report their distribution in answers generated by 5 LLMs. Next, an iterative refinement method is proposed that reduces the amount of hallucination by first refining a generated background knowledge and then refining the generated answer based on the background knowledge. Both automatic and human evaluation results are reported and the proposed method seem to reduce the amount of hallucinated answers.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Section 3.3 title -- possible spelling mistake (Protocals)"}, "extra_text": {}}, {"review_id": "ryS74Skez7", "reviewer": "Reviewer_8mL5", "summary": "", "questions": "This is a nice paper overall, but II have the following question.\n1. Choices of LLM are not justified well. PaLM etc are not included - not sure why?\n2. \"After analyzing 250 directly generated examples from the five models\" - I argue this is a very small set to conclude anything!\n3. I am not sure that I understand this - how MedNLI, CtrlEval, F1 etc. reflect there is no hallucination in the follow-up generation - please clarify.\n4. The overall annotation process, inter-annotator agreement, number of annotators etc are completely missing.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to “Question 1 about the choice of LLM”:\n\nOur setting is zero-shot, and most LLMs (Vicuna, Alpaca, MedAlpaca, Robin) we select are based on LLaMa.  Existing papers/blogs [1-2] compare current LLMs and LLaMa achieves the best zero-shot performance at a similar size. LLaMa and its variants are one of the most commonly used open-source LLMs and achieve impressive performance.\nIt's worth noting that, PaLM is more suitable for few-shot learning [3] which does not align with our work. \n\n> [1] Touvron, Hugo, et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).\n\n> [2] https://ai.meta.com/resources/models-and-libraries/llama/\n\n> [3] Chowdhery, Aakanksha, et al. \"Palm: Scaling language modeling with pathways.\" arXiv preprint arXiv:2204.02311 (2022).\n\n## Response to “Question 2 about the number of the case study (human annotation)”:\n\nThe analysis for the case study is extremely time and human source consuming. There are many perspectives to review, and the funding is limited. It takes over one hour to check each example sentence by sentence. The evaluation of 250 QA pairs, comprising roughly 2000 sentences, is conducted by experts and we also did crowd-sourced evaluations as a supplement, costing over 750 person-hour and 1500 USD. Existing works [1-4] also use examples of the same magnitude to analyze (about 100 examples). We are currently asking for more funding to analyze more cases.\n\n> [1] Yang, Shiquan, et al. \"An Interpretable Neuro-Symbolic Reasoning Framework for Task-Oriented Dialogue Generation.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n\n> [2] Yin, Zhangyue, et al. \"Do Large Language Models Know What They Don't Know?.\" Findings of the Association for Computational Linguistics: ACL 2023.\n\n> [3] Dan Su, et al. Read before generate! faithful long form question answering with machine reading. In Findings of the Association for Computational Linguistics: ACL 2022.\n\n> [4] Bang, Yejin, et al. \"A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.\" arXiv preprint arXiv:2302.04023 (2023).\n\n> [5] Liévin, Valentin, Christoffer Egeberg Hother, and Ole Winther. \"Can large language models reason about medical questions?.\" arXiv preprint arXiv:2207.08143 (2022).\n\n## Response to “Question 3 about the metrics”:\n\nCurrently, there is no recognized/common-used metric for hallucination in QA [1]. We try our best to find as many dimensions of metrics as possible to evaluate the hallucination level.\n\nSome works [2-3] in other NLG fields leverage the NLI-based metric to evaluate hallucinations. These metrics define the hallucination score to be the entailment probability between the source/reference and its generated text. These metrics measure the proportion of generated instances that entail, neutral to, and contradict the source/reference. Inspired by them, we introduce MedNLI (Medical Natural Language Inference) [4] to assess the logical entailment of generated answers with the provided context and the reference answer. If the result is “contradiction”, it is identified as “hallucination”.\n\nCTRLEval [5] is an unsupervised, reference-free, and task-agnostic evaluation metric that assesses generation from various aspects (coherence, consistency, and attribute relevance). We focus on the consistency aspect. If the generated answer is factually inconsistent with the context, it is identified as “hallucination”.\n\nUni-gram F1 and RougeL are classical metrics for Generative QA [6] which reflect the overall performance.\n\n> [1] Lee, Hwanhee, et al. \"KPQA: A Metric for Generative Question Answering Using Keyphrase Weights.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n> [2] Laban, Philippe, et al. \"SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization.\" Transactions of the Association for Computational Linguistics 10 (2022): 163-177.\n\n> [3] Dušek, Ondřej, and Zdeněk Kasner. \"Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference.\" Proceedings of the 13th International Conference on Natural Language Generation. 2020.\n\n> [4] Phan, Long N., et al. \"Scifive: a text-to-text transformer model for biomedical literature.\" arXiv preprint arXiv:2106.03598 (2021). https://huggingface.co/razent/SciFive-large-Pubmed_PMC-MedNLI \n\n> [5] Ke, Pei, et al. \"CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n\n> [6] Dan Su, et al. Read before generate! faithful long form question answering with machine reading. In Findings of the Association for Computational Linguistics: ACL 2022.\n\n\n## Response to “Question 4 about the annotation process”:\nWe describe the inter-annotator agreement in Section 4.2.2 (Line 497-502) and the details about human evaluation are in Appendix D (Line 1120-1161). Recognizing that you overlooked this part of our paper, we sincerely hope you improve your score!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "With the proliferation of LLM and their merging abilities, the emerging abilities of lying are also increasing. Hallucination could be detrimental in mission-critical applications like medical question-answering. This paper is quite timely and relevant.", "reasons_to_reject": "I have specific technical queries.", "questions_to_authors": "This is a nice paper overall, but II have the following question.\n1. Choices of LLM are not justified well. PaLM etc are not included - not sure why?\n2. \"After analyzing 250 directly generated examples from the five models\" - I argue this is a very small set to conclude anything!\n3. I am not sure that I understand this - how MedNLI, CtrlEval, F1 etc. reflect there is no hallucination in the follow-up generation - please clarify.\n4. The overall annotation process, inter-annotator agreement, number of annotators etc are completely missing.", "paper_topic_and_main_contributions": "This paper investigates the effect of hallucination in medical question-answering systems. This is indeed an interesting topic, very timely and essential. Mostly this paper is easy to follow and understand. However, I have technical queries - asking relevant questions accordingly.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gA3qcJVkw7", "reviewer": "Reviewer_coTz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback! \n\n## Response to “Reason 1 To Reject about novelty”:\nWe introduce an interactive self-reflection methodology that incorporates knowledge acquisition and answer generation. [1-2] propose “generate-then-answer” pipelines without feedback and interactive loop. They do not explore harnessing the inner self-reflection, interactivity, and multitasking ability of LLM. \n\nSimilarly, Reflexion[3] explores the self-reflection of LLMs. However, this contemporary work was preprinted on June 10, 2023, and our paper was submitted before June 23. Thus, this work should not undermine our novelty. Moreover, [3] only evaluates and refines the final generated text, while our method considers both the intermediate and the final generated text during generation. In more detail, the feedback in [3] is a label while our feedback for refinement includes Aspect Description and Explicit Score Number. We also do ablation studies for these components. Our method fosters a dynamic interaction between the system and its knowledge, progressively enhancing the model's ability to provide factuality, consistency, and entailment in healthcare settings. We hope our work can contribute to a new pattern to utilize the inner knowledge, interactivity, and self-reflection of LLMs and tap into their potential.\n\nIn addition, these arxiv papers are not peer-reviewed.\n\n> [1] Sun, Zhiqing, et al. \"Recitation-augmented language models.\" arXiv preprint arXiv:2210.01296 (2022).\n\n> [2] Yu, Wenhao, et al. \"Generate rather than retrieve: Large language models are strong context generators.\" arXiv preprint arXiv:2209.10063 (2022).\n\n> [3] Shinn, Noah, et al. \"Reflexion: Language agents with verbal reinforcement learning.\" arXiv preprint arXiv:2303.11366 (2023).\n\n## Response to “Reason 2 To Reject about soundness”:\n\nLMs are trained on a wide variety of data from the internet including medicine-related knowledge. Current works [1-2] explore the potential of LM to become a knowledge source. LLMs achieve impressive performance in knowledge-intensive tasks including the medical domain [3-5]. \n\nHowever, currently, a general LM is still not trustworthy enough as a medical knowledge base in practical applications. That’s the reason why we do this work. \nA manually calibrated external medical knowledge base is more reliable. However, the motivation for our work (retrieval-free) and retrieval-augmented generation (using medical knowledge bases) is different:\n1. Retrieval-augmented works emphasize that the inherent knowledge in LMs is deficient, and they add external knowledge to compensate.\n2. Our work aims to harness and maximize the potential of the model's inherent knowledge. Current research works [6-8] highlight a gap between surface realization and inherent knowledge in NLG tasks. Models can realize they are generating something hallucinated/nonsensical in some way. Our method aims to reduce this gap and enhance performance as much as possible within the context of mastering knowledge. As base LLMs become more powerful and inherent knowledge becomes more extensive and versatile, it is crucial to leverage knowledge and unleash its capabilities. \n\nAs we said in the Limitation section, our method is still in its early stages and not yet ready for direct real-world deployment. It should be viewed as a complementary approach alongside other methods, such as retrieval, with the potential to contribute to more robust application systems in the future.\n\n> [1] Wang, Cunxiang, Pai Liu, and Yue Zhang. \"Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n\n> [2] AlKhamissi, Badr, et al. \"A review on language models as knowledge bases.\" arXiv preprint arXiv:2204.06031 (2022).\n\n> [3] Nori, Harsha, et al. \"Capabilities of gpt-4 on medical challenge problems.\" arXiv preprint arXiv:2303.13375 (2023).\n\n> [4] Gilson, Aidan, et al. \"How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment.\" JMIR Medical Education 9.1 (2023): e45312.\n\n> [5]  Liévin, Valentin, Christoffer Egeberg Hother, and Ole Winther. \"Can large language models reason about medical questions?.\" arXiv preprint arXiv:2207.08143 (2022).\n\n> [6] Yin, Zhangyue, et al. \"Do Large Language Models Know What They Don't Know?.\" Findings of the Association for Computational Linguistics: ACL 2023.\n\n> [7] Burns, Collin, et al. \"Discovering latent knowledge in language models without supervision.\" International Conference on Learning Representations (ICLR), 2023.\n\n> [8] Rajpurkar, Pranav, Robin Jia, and Percy Liang. \"Know What You Don’t Know: Unanswerable Questions for SQuAD.\" Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 2018.\n\n## Response to “Typos Grammar Style And Presentation Improvements:\nThanks for your valuable suggestion! We will change the title if it’s allowed in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. For motivations, this paper analyzes the task of large language models in the task of generative medical QA, and summarizes the categories of hallucinations produced by large models in this scenario. This gives us a guiding significance for designing generative medical QA models.\n\n2. For the design of the method, this paper proposes a method of generating answers based on knowledge by using the language model to generate the knowledge corresponding to the answer to the question first, continuously verifying the confidence of the generated knowledge, and finally using the generated knowledge. This method is cleverly designed, and the experimental results also prove the effectiveness of this method.\n\n3. The article has done sufficient experiments to verify that this method is effective on many modern language models, which also shows the versatility of this method.", "reasons_to_reject": "1. Regarding the novelty, the method of this paper has been verified by some existing work. For example, the LLM is allowed to generate knowledge first and then answer. You can refer to these two papers (https://arxiv.org/abs/2210.01296, https://arxiv.org/pdf/2209.10063); about Self-Reflection can refer to this paper (https://arxiv.org/abs/2303.11366). The author needs to elaborate on the uniqueness of this paper compared to these methods. Of course, the reviewer agrees with the necessity of building a credible QA model in the medical field.\n\n2. Regarding the soundness of the method, this paper needs to explain the reliability of using the language model as the knowledge source, especially to explain whether using a general language model can provide better and more reliable knowledge than using a medical knowledge base?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyzes the hallucination problem of large models in the task of generative medical QA, and proposes a method of generating knowledge, verifying knowledge, and generating answers based on knowledge. The experiment proves that the reflection method proposed in this paper can effectively improve the quality of the answers generated by the existing model.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The title is not feasible for this paper. \nThe authors should consider more narrative titles, such as ``Towards Mitigating Hallucination in Medical Question Answering with Large Language Models via Self-Reflection''"}}, {"review_id": "Skqn0OLSF5", "reviewer": "Reviewer_tvQQ", "summary": "", "questions": "See above section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to “Reason 1 To Reject about time efficiency and performance stability”:\nThanks for your valuable feedback! We carefully design guides/instructions for refinement so that reduce the cycle number and improve the time efficiency. We acknowledge that compared to direct generation, our method reduces hallucinations but sacrifices time efficiency. And sampling during generation introduces some randomness. Our future will explore a better guide for refinement or during inference while maintaining the accuracy of QA systems.\n\n## Response to “Reason 2 To Reject about different types of knowledge”:\nThanks for your exciting suggestion! We notice the long-tail problem during the experiment analysis and believe it’s an interesting topic to explore. Due to the limited number of problems addressed by one work, we do not treat it as our main point. Our goal is to enhance generated performance as much as possible within the context of mastering knowledge. In future work, we will improve our current method to treat different types of knowledge/questions differently. We will dig into the deeper causes in knowledge areas. Specifically, we think augmenting with external knowledge or adjusting the distribution of the training dataset can be an effective method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a novel self-enhancement method for knowledge, which enhances its reliability in terms of knowledge; \n2. The paper conducts evaluations on multiple datasets, especially those involving scenarios related to the application of large models (such as in medicine), demonstrating the practical value of the model; \n3. The paper provides a wealth of reproducible details, lending strong credibility to the study.", "reasons_to_reject": "The designed method  does not appear to have inherent advantages in terms of time efficiency and performance stability.\n\nIn fact, the model's ability to generate different types of knowledge is not consistent (for instance, its performance tends to be poorer on long-tail knowledge). For knowledge areas where the model does not excel, the paper may not have conducted a more in-depth analysis of potential error propagation.", "questions_to_authors": "See above section.", "paper_topic_and_main_contributions": "The paper presents a novel knowledge loop design, primarily employed in the prompting process, with a focus on enhancing knowledge consistency during the inference process of large models. The paper has tested the performance across multiple datasets, most of which demonstrate a certain degree of improvement. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the existence and mitigation of hallucination in answers generated by LLMs in generative question answering in the medical domain. In particular, the authors first classify the hallucinated answers into 3 types and report their distribution in answers generated by 5 LLMs. Next, an iterative refinement method is proposed that reduces the amount of hallucination by first refining a generated background knowledge and then refining the generated answer based on the background knowledge. Both automatic and human evaluation results are reported, and the proposed method seems to reduce the amount of hallucinated answers.

The proposed method has been evaluated on top of Vicuna-7B and ChatGPT, showing improved results on 5 biomedical datasets.

During the discussion period, most questions have been answered reasonably well. One of the reviewers complained about novelty, but the paper's focus on the biomedical domain with its unique challenges makes it different from previous work.
Decision: Accept-Findings