Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus
Authors: Tianhang Zhang
Lin Qiu
Qipeng Guo
Cheng Deng
Yue Zhang
Zheng Zhang
Chenghu Zhou
Xinbing Wang
Luoyi Fu
Venue: EMNLP 2023 Main (2023)
Keywords: hallucination detection
large language model
text generation

Abstract: Large Language Models (LLMs) have gained significant popularity for their impressive performance across diverse fields. However, LLMs are prone to hallucinate untruthful or nonsensical outputs that fail to meet user expectations in many real-world applications. Existing works for detecting hallucinations in LLMs either rely on external knowledge for reference retrieval or require sampling multiple responses from the LLM for consistency verification, making these methods costly and inefficient. In this paper, we propose a novel reference-free, uncertainty-based method for detecting hallucinations in LLMs. Our approach imitates human focus in factuality checking from three aspects: 1) focus on the most informative and important keywords in the given text; 2) focus on the unreliable tokens in historical context which may lead to a cascade of hallucinations; and 3) focus on the token properties such as token type and token frequency. Experimental results on relevant datasets demonstrate the effectiveness of our proposed method, which achieves state-of-the-art performance across all the evaluation metrics and eliminates the need for additional information.

Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus
Tianhang Zhang', Lin Qiu², Qipeng Guo?, Cheng Deng', Yue Zhang* Zheng Zhang', Chenghu Zhou', Xinbing Wang' and Luoyi Fu'
SSanghai Jiaotong University, China ²Amazon AWS AI Westlake University, China 'IGSNRR, Chinese Academy of Sciences, China davendw, xwang8, yiluofuj@sjtu.edue edu.cn {quln, gqipeng, zhaz)@amazon. com, yue. www.zzhagwwas.oorr cn
Abstract
2023), which refers to the generated text that is un- truthful or nonsensical (Ji et al., 2023; Bang et al., 2023). Hallucinations are common occurrence in almost all applications (Xu et al., 2023), thus undermining the reliability and trustworthiness of LLMs, especially in scenarios where accuracy and veracity are essential.
Large Language Models (LLMs) have gained significant popularity for their impressive per- formance across diverse fields. However, LLMs are prone hallucinate untruthful or nonsensical outputs that fail to meet user expec- lations in many real-world applications. Exist- ing works for detecting hallucinations in L.MMs either rely on external knowledge for reference retrieval or require sampling multiple responses from the LLM for consistency verification, making these methods costly and inefficient. this paper, we propose novel reference- free, uncertainty-bu -based method for detecting hallucinations in LLMs. Our approach imitates human focus in factuality checking from three aspects: 1) focus on the most informative and important keywords the given text; 2) focus the unreliable tokens in historical context which may lead to a cascade of hallucinations; and 3) focus on the token properties such as token type and token frequency. Experimen- tal results on relevant datasets demonstrate the effectiveness of our proposed method, which achieves state-of-the thetherrrt performance across all the evaluation metrics and eliminates the need for additional information."
Existing studies on hallucination detection for LLMs can be broadly divided into two categories: (i) retrieval-based methods (Min al., 2023;
Liu al., 2023), which evaluate the veracity of the given text against knowledge bases, and (ii) sampling-bused methods (Mündler et al., 2023; Manakul et al., 2023), which assess information consistency between the evaluated text and ad- ditional sampled responses from the same LLM. However, retrieval-based methods depend heavily the external knowledge that may not always be accessible. And sampling- based methods require multiple responses from the LLM for the infor- mation consistency verification or model training. making these methods costly and inefficient.
To address the above issues, we propose a novel reference frree, uncertainty- based method to detect hallucinations in LLMs that are factually incor- rect according to the world knowledge. The pro- posed method relies exclusively on the LLM output text, thereby eliminating the need for additional re- sources like sampled responses from LLM or exter- nal knowledge, well further training based on such data. Our basic idea is to use a proxy language model for calculating the probability of each token in a given text. Based on the calculated probabil- ity, we can compute the hallucination score at both token and sentence level using uncertainty-baase metrics (Guerreiro et al., 2022; Xiao and Wang, 2021), where tokens and sentences with high hallu- cination scores identified as candidate halluci- nated contents. Our assumption is that powerful enough LLM should assign a low probability to to- kens that make up hallucinated information, since
1 Introduction
Large Language Models (LLMs) have garnered substantial attention for their remarkable per- formance across various domains, such as fi- nance (Wu et al., 2023; Lopez-Lira and Tang, 2023), medicine (Javaid et al., 2023; Lee et al., 2023), and education (Tlili et al., 2023; Baidoo- Anu and Owusu Ansah, 2023). These models ex- hibit extraordinary capacity to generate natural language texts with high levels of coherence, flu- ency, and informativeness. Nevertheless, a signifi- cant obstacle confronting LLMS is the risk of pro- ducing hallucinations (Shen et al., 2023b; Sallam,
Cooee can be found at sttps://gitt thub. com/zthang/ focus.

---

(a) Original text
(b) Fecus
Rackenzie Caquatto is an American former artistic gyenast,
Nackenzie Caquatto is an Amorican forner artistic gynnast,
who conpeted at the 2012 Summer olyspics in London.
who corpeted at the 2012 Suiner diympics in London.
Caquatto aas born ip 1992 and began gyanastics at the of three, Sho comsated the uneven bars and balance bean at the 2012 Surmer Olympics..u
Caquatto was boun <OATE> 1992 and began gyanastics at the
age of three She Competed on the uneven bars and balance besn at the 2012 Sumser O1 Smmpics
189
Figure I: (a) Using naive proxy model can hinder the focus on hallucination itself: 1) considering all tokens within the given text may introduce noise; 2) the hallucinated tokens might be assigned high probabilities (green bar) due to the overconfidence problem; 3) factual tokens may receive low probabilities (red bar) due to the underconfidence problem. (b) To strengthen such focus, we imitate how humans perform factuality checking from three aspects: I) focus the informative keywords; 2) focus on the preceding words by propagating the uncertainty through attention weights; 3) focus on the token properties by providing entity type before each named entity.
they deviate from the world knowledge the model has learned during its training stage.
at both sentence-level and passage-level.
Focus on the preceding words: we propagate the uncertainties of previous tokens to the subse- quent ones according to their attention weights to alleviate the overconfidence problem. This up- proach is based on the hypothesis that words that are strongly connected to unreliable tokens may also be influenced by these inaccuracies, which trigger a chain reaction of hallucinations.
The above method serves as a base framework,
which can be limited by the inherent characteristics of the prediction probability from naive proxy model. Such model functions general prob- ability estimator, its predictions reflect syntactic, semantic and other sources of information, which can hinder the focus on hallucination itself as illus- trated in Figure la.
Firstly, the proxy model ignores varying de- grees of informativeness, which may introduce noise. Secondly, the probabilities assigned by LMs are general and can deviate from factuality confi- dence in different contexts. For instance, the proxy model can be overconfident if the historical con- text contains surface tokens that are correlated with a hallucinated token, or the historical context fea- tures exposure bias (Bengio et al., 2015; Iqbal and Qureshi, 2022) due to the auto-regressive nature of generative process. One example is shown in Figure la, where hallucinated tokens "2012 Sum- mer Olympics" are assigned high probabilities. In addition, proxy model can be underconfident if there are many plausible choices of topic directions to continue a context, despite that the hallucination involves different tokens within only one topic di- rection. One example is shown in Figure 1a, where the factual token "1992" received a low probability due to the competitors like "West" and "Coral".
Focus on the token properties: the predicted token probability is conditioned on its entity type (if any) and adjusted by its inverse document frequency (IDF). This results in a probability distribution that aligns more closely with human evaluation in a posterior manner, thus mitigating the underconfidence problem.
In summary, our primary contribution is that we introduce novel reference-free, uncertainty-basee approach for detecting hallucinations LLMs. Our approach does not require additional sampled responses or external knowledge bases, making it simple and cost-effective. Experimental results demonstrate that our proposed method achieves state-of-the performance the WikiBio GPT-3 dataset across various models with different scales, and shows effectiveness in detecting hallucinations within summaries generated by small models.
Related Work
To strengthen the focus on ballucination, we take inspiration from human factuality checking, which can include at least three specific considerations as depicted in Figure lb:
2.1 Hallucinations in Text Generation
Hallucinations are prevalent phenomenon in deep learning-based models employed for various text generation tasks (Xu al., 2023), such as ab- stractive summarization (Huang et al., 2021; Nan et al., 2021), dialogue generation (Dziri et al.,
Focus on the informative keywords: the key- words that express salient information will ex- tracted for the calculation of hallucination scores

---

2022; Rashkin et al., 2021) and question answer- ing (Longpre al., 2021; Su et al., 2022). Hal- lucinations present significant challenges in text generation tasks, as they can lead to inaccurate or misleading results, which is unacceptable in most user-oriented applications (Liu et al., 2022; Xu et al., 2023; Rebuffel et al., 2022).
3 Methods
A proxy model is utilized in our method for uncer- tainty assessment in cases where token-level prob- abilities are inaccessible, such as GPT-3 (Ouyang et al., 2022). Although previous work by Manakul et al. (2023) has demonstrated the ineffective per- formance of using a proxy model, attribute it to the uncertainty metries employed. These metries, such as the average entropy and average loss for all tokens in the sentence, are insufficiently aligned with human evaluation. We believe this issue stems from the inherent disparities in how models and hu- mans perceive and assess information, thus limiting the capability of the uncertainty-bett -based approach for hallucination detection.
2.2 Hallucination Detection
Previous studies on hallucination detection have primarily concentrated on identifying hallucina- tions produced by small models (fewer than lb parameters) that are tailored for specific tasks. For instance, Kasner al. (2021) combined rule- based system and a pretrained language model identify hallucinations in table-to-text genera- tion. Guerreiro et al. (2022) adopted the average log-probability across all the tokens in the output sequence as the model uncertainty metric for de- tecting hallucinations in machine translation. Dale et al. (2022) attempted detect hallucinations by evaluating the percentage of the source contribution the generated text. However, the hallucination patterns exhibited by LLMs tend to be divergent from those in small models (Guerreiro et al., 2023), posing challenges for the generalization of these methods on detecting hallucinations in LLMs. Ac- cordingly, hallucination detection in small models is not within the primary scope of this paper.
To mitigate this problem, we imitate how hu- mans perform factuality checking from three as- pects, which will be discussed in following sections.
3.1 Keywords selection
Prior works (Pagnoni et al., 2021; Kryšcińskii et al., 2020) suggest that entities are the most frequently hallucinated words in text generation. This aligns with the intuition that, when evaluating the verac- ity of generated results, our primary focus lies on keywords that convey the most crucial information. In this regard, only focus on keywords identi- fied by Spacy (Honnibal and Montani, 2017) when calculating the hallucination score at both sentence- level and passage-level. The keywords identified by Spacy can be classified into two groups. The first group comprises 18 distinct types of named entities, including person, location, date, event, organiza- tion, and others. The second group encompasses nouns that do not belong the first group.
The widespread incorporation of LLMs across a diverse range of applications has drawn sub- stantial attention from researchers towards the is- sue of hallucinations within LLMs (Bang et al., 2023; Shen et al., 2023a; Alkaissi and McFarlane, 2023). For instance, Min et al. (2023) introduced FACTSCORE to evaluate the correctness of each atomic fact in the generated text by referencing knowledge source. Mündler al. (2023) aimed detect hallucinations by examining whether two sampled sentences generated at the same position within a text contradict each other. recent work by Manakul et al. (2023) proposed SelfCheckGPT, a black-box approach for detecting hallucinations in L.L.-generated responses. The primary premise of SelfCheckGPT is that when the LLM is uncer- tain about a given concept, the sampled responses may contain inconsistent facts. Nonetheless, these methods either rely on external knowledge bases or multiple responses sampled from LLM, which are resource- -nttensie and inefficient.
Specifically, for a given text r, we will compute a hallucination score h; for the i-th token & in r. To fully utilize both local and global uncertainty infor- mation, h/ is the sum of the negative log probability and entropy when generating f,:
h4 = -log(pp + Hi
(1)
H4 = 2- (()))s(((
(2)
where (v( denotes the probability of generating the token e over all tokens in the vocabulary at position i. The hallucination score h² for the sen- tence s is calculated by a weighted sum, where the

---

The attention heat map after max- max-pooling for all the layers and attention heads when generating the example using llama-30b, where the x-axis only presents first and last sentence, while the y-axis only includes the last sentence due to space constraints. The brightness of each rectangle represents the attention score between the corresponding tokens, with brighter shades indicating higher scores.
Figure 2:
"2012" makes appearances, with the proba- weight is determined by whether t is keyword:
bility of its initial occurrence being significantly lower than the probability of its subsequent appear- ance. The visualized self-attention matrix reveals that considerable attention given to the same phrase in the first sentence (circled with a blue box) when generating "2012 Summer Olympics" in the last sentence. However, the claim "Mackenzie Caquatto competed at the 2012 Summer Olympics in London" is untruthful.
h*
Σ T(4 C (K) * h4 (3)
E.1 1 IIGEERR
where |s] the number of tokens in 8, K. de- notes the set of keywords, I((-) is an indicator func- tion. Moreover, this formulation can be extended to compute the passage-level hallucination score by averaging hallucination scores of keywords in the given passage.
This observation inspired us to introduce a "penalty" for tokens generated with attentions paid to unreliable tokens. In other words, consider the hallucination scores of preceding tokens and apply them penalty to the current token based on their respective attention weights. Here, we only consider propagation between keywords. Specifi- cally, we first check if the current token is a key- word as described in Section 3.1. If not, the penalty is set to zero. If it is a keyword, we normalize the attention weights between the current token and all previous keywords obtain a penalty weight. The penalty for the current token is computed weighted sum of the hallucination scores associ- ated with the previous tokens. Since the penalty can be transmitted to all the subsequent tokens via multi-hop, a coefficient C [o, 1] is introduced to ensure that the penalty diminishes geometrically with the increase in the number of hops.
3.2 Hallucination propagation
Several studies (Guerreiro et al., 2022: Xiao and Wang, 2021) have utilized token probability as a measure for hallucination detection. However, probabilities derived from language model may not accurately reflect the factuality confidence in the generated content. Some hallucinated tokens can be assigned high probabilities when the history context contains hallucinated information, which we term as the overconfidence issue. This issue is exacerbated by the self-attention mechanism that is commonly used in transformer-basee LLMs, since it introduces exposure bias (Bengio et al., 2015; Iqbal and Qureshi, 2022), which refers to the dis- crepancy between training and inference caused by the use of teacher foreing during the training stage. Consequently, the generated text accepted as fac- tual claims, even though it may be non non-factual.
Let h, represent the hallucination score of the i-th token & with an accumulated penalty, the cal- culation of h, can be expressed as follows:
Figure 2 provides an example that illustrates the overconfidence issue. Considering the following text: "Mackenzie Caquatto is American former artistic gymnast, who competed at the 2012 Sum- mer Olympics London. Caquatto was born in 1992, and began gymnastics at the age of three. She competed on the uneven bars and balance beam at the 2012 Summer Olympics." Notably, the term
h h4 + I(ts E K)
(4)
i-1 pi
Wij * h
(5)
i=0

---

I[t K)) * jtt
Wijj Ei T(t E kT) aft
(6)
<PERSON Tockenzie Catiatto is <NpSP (atrican forser artistic gyreast, cospeted at the 2013 Summer Glymsics in
where ps represents the penalty of the i-th token, atty, denotes the attention weight between and I after max-pooling for all the layers and attention heads.
8.62
0.10
0.03
3.3 Probability correction
1992
1993
1994
West
Apart from the overconfidence issue, there are also instances where the model exhibits undercon- fidence, which can also lead to deviations in to- ken probability from factuality confidence. We believe such underconfidence is related to the to- ken properties, including the entity type and token frequency. As shown in Figure 1a, when generat- ing the subsequent words following "Caquatto was born in". The model may have multiple possible choices of topic directions such as "West chester", "Coral Springs", "1992" et al. despite that the hallu- cination involves different tokens within a specific topic direction. Consequently, the probability of generating the date "1992" would be relatively low, given the existence of several other viable options.
Figure 3: An example of providing entity type preceding named entities: Top-3 words that follow the incomplete sentence are all related to dates. Despite having the highest probability in Figure the token "West" is generated with a relatively low probability of 0.03.
stage, given all words are tokenized into sentence pieces. To tackle this problem, we leverage the in- context learning capability of the proxy model by inserting the entity type? preceding every named entity identified by Spacy as shown in Figure 3. The entity type serves constraint in generation, thereby enabling us to approximate the ideal can- didate set c(to:s) in Equation 7 using tokens with a generation probability greater than threshold p. Accordingly, the token probability distribution is corrected to assign higher probability to tokens adhering to the given entity type.
This highlights the stark disparities how mod- els and humans assess information: when eval- uating the plausibility of "1992", the model fo- cuses meticulously on all the feasible choices with different entity types. In contrast, humans tend to intuitively include it within a tailored set of candidate words that predominantly consists of terms related dates. Suppose there are tokens 10n-1 to, (1, ..., - in a model response r. Let c(tox) denote the set of ideal candidate words for 4 given the first i + tokens. According to the Bayes rule, the probability of generating t; given foi-1 and the candidate set can be expressed as:
outlined in previous stud- ies (Raunak al., 2020; van der Poel et al., 2022; Demeter al., 2020), tokens with low frequency are likely to receive lower prediction probabilities, potentially leading to the underconfidence the model. To mitigate this issue, the probability of token is further corrected by its token IDF:
Additionally,
p(t) (t) p(() Evev P(U) # idf(v)
(8)
p(G(loi-1,c((a:))
where p(() denotes the probability of token across all tokens in the vocabulary V with entity type pro- vided. The token IDF is calculated based on 1M documents sampled from RedPajama dataset?.
-p(c(tt:i) tos-1, t4) (ot=1)) plc(tas)ltus-1)
p(t)(tori-1)
3.4 Putting things together
p(o((o:))
To combine all the methods proposed above, we re- place the token probability in Equation and Equa- tion 2 with p(t). Subsequently, we apply hallucina- tion propagation to obtain the token-level hallucina- tion score with penalty accumulated. The sentence-
p(tal (0:i-))
Enec(lon) p(pltos 1)
suggests that when assessing the rationality of given word, the focus should be directed towards similar words rather than encompassing possible choices. However, constructing such a candidate set poses a challenge during the model generation
"The entity type inserted the text do not participate in hallucination propagation or hallucination score calculation.
http::///wuggnngfffe.../t/asssss aadaaaasetsfooghhhrccommp RedPajama-Datte Data-I'T-aam I'T-Saple

---

level and passage-level hallucination scores are cal- culated based on Equation 3.
same metrics employed by SelfCheckGPT. The Area Under the Precision-Recall Curve (AUC-PR) used to measure the performance of sentence- level hallucination detection, while the Pearson and Spearman's 's correlation coefficient are applied to evaluate the agreement between the passage- level hallucination score and human judgement. For space saving, AUC-PR of non-factual class is abbreviated as NonFact or NoFac in the following sections, and similarly for the other classes.
4 Experiments and Results
4.1 Experiment setting
Dataset. We evaluated our proposed method on WikiBio GPT-3 dataset (Manakul et al., 2023), which, the best of our knowledge, is the only publicly accessible dataset for LLM hallucination detection at present. Additionally, to assess the ex- tent to which our proposed method can be applicd for detecting hallucinations produced by different models, and in particular small models, we con- ducted supplementary experiments on the XSum- Faith (Maynez al., 2020) and FRANK (Pagnoni et al., 2021) datasets. Given the primary focus of this paper is hallucination detection in LLM as dis- cussed in Section 2.2, the details and results of the two datasets are provided in Appendix A.
Proxy model. To demonstrate the generalizabil- ity of our proposed method across different scales of LLMs, we conduct experiments on 22 diverse proxy models. The specifie details of these proxy models can be found in Appendix E.
Prompts. In experiments where entity types are not provided, we use the prompt "This is a passage from Wikipedia about {concept]:" Conversely, when entity types are inserted before named entities, the prompt is "Please complete the passage below using appropriate words that follow to the given type with wrapped. This is a passage from Wikipedia about (concept):".
The WikiBio GPT-3 dataset comprises 1908 an- notated sentences from 238 Wikipedia passages generated by text-davinci-003. Each sentence is assigned one of the following labels: 1) major in- accurate, if the sentence is irrelevant to the given topic; 2) minor inaccurate, if the sentence includes non-factual information verifiable via web search; 3) accurate, the sentence does not contain any hallucination. We provided some examples from the dataset in Appendix D. The dataset also in- cluded 20 seechattiaall-ssmpple responses from text-davinci-003 for each passage, but these were not utilized in experiments our method does not necessitate additional sampled responses.
4.2 Main results
The performance comparison between our pro- posed method and the baseline approaches is pre- sented in Table 1. Due space limitations, we only display the results of LLaMA family. Comprehen- sive comparison results for all proxy models can be found in the Appendix H. The hyperparameters and p are set to 0.9 and 0.01, respectively. The baseline results are referenced from Manakul et al. (2023), Other implementation details can be found in Appendix B. Our key findings are as follows: Proxy model surpasses all the baselines. Lever- aging three proposed focus mechanisms, LLaMA- 30b consistently outperforms SelfCheckGPT- Combination" and other baselines across all five metrics. Significantly, this is achieved without re- sorting sampled responses or further training, exhibiting superior efficiency compared Self- CheckGPT. As presented in Table 1, the perfor- mance of LLaMA family improves as the model size increases. However, this improvement not
In accordance with the setting in Manakul et al. (2023), sentences labeled as major inaccurate and minor inaccurate are grouped into the non-factual class, while remaining sentences are grouped into the factual class. For the non-factual* class, we first remove passages where all sentences are labeled major inaccurate. Then, classify remaining major inaccurate sentences as =
Baselines. (i) GPT-3 (Ouyang et al., 2022) Uncer- tainties: GPT-3 mixxtttvvinii-003 API returns top- 5 probabilities for each generated token, which can be used to quantify its uncertainty using negative log probability and entropy. (ii) SelfCheckGPT: SelfCheckGPT (Manakul et al., 2023) is black- box method for detecting hallucinations in LLMs, which demands additional responses sampled from the same LLM for the consistency verification. Metrics. To ensure a fair comparison, we adopt
"We noticed that on June 11th, the authors of SelfCheck- GPT updated their results on GitHub (but not for their arXiv paper). The new approach entails a large number of ChatGPT queries for text inconsistency assessment. We do not include the results in this paper since they are contemporancous with our work, as well as the comparison is not fair.

---

Sentence-levee Metries NonFact
Passage-leva Metries
Method
NonFact
Factual
Pearson
Spearman
GPT-3 Uncertainties
Avg(-loge)
83.21
38.89
53.97
57.04
53.93
Avg(?()
80.73
37.09
52.07
55.52
50.87
Max(-logg
87.51
35.88
50.46
57.83
55.69
Max(H)
85.75
32.43
50.27
52.48
49.55
SelfCheckGPT
BERTSscore
81.96
45.96
44.23
58.18
$5,90
QA
84,26
40.06
48.15
61.07
59.29
Unigram (max)
85.63
41.04
58.47
64.71
64.91
Combination
87.33
44.37
61.83
69.05
67.77
Ours
LLaMA-7B
84.26
40.20
57.04
64.47
54,73
LLaMA- 3B /occs
87.90
43.84
62.46
70.62
63.03
LLaMA- 330 /ocuu
89.79
48.80
65.69
77.15
73.24
LLaMA-65B
89.94
48.69
64.90
76.80
73.01
Tomux
Table I: Performance comparison between proposed method and baseline methods. AUC-PR is adopted as the performance metric for sentence-level hallucination detection. Passage-level performances are measured by Pearson correlation coefficient and Spearman's correlation coefficient with respect to human annotations. Results of GPT-3 and SelfCheckGPT are referenced from the paper (Manakul al., 2023).
linearly correlated to the model size as shown in Figure 9 of Appendix F. LLaMA-653 4-bbbb even exhibits slightly inferior performance compared to LLaMA- 30b in four of the five metrics.
Method
NoFac
NoFac
Fact
Pear.
Spear.
avg(h)
82.07
41.47
47.22
51.03
37.29
+keyword
83.01
41.57
45.82
56.07
44.77
+penalty
86.68
45.27
54.93
59.08
55.84
Moreover, the comprehensive results across 22 proxy models as demonstrated Table affirm that within the same model family, models with more parameters tend to perform better. This can be attributed to their broader and more accurate understanding of world knowledge. In addition, when comparing different model families, models that exhibit superior performance on general NLP tasks often perform well on the WikiBio GPT-3 dataset. These observations provide valuable in- sights for future exploration and enhancement of our hallucination detection method.
+entity type
88.89
46.92
66.12
76.82
71.49
+token idf
89.79
48.80
65.69
77.15
73.24
Table 2: Ablation study of the proposed method using LLaMA-30b (y 0.9, p 0.01).
tricks serves the baseline in the first row, with each trick incorporated incrementally in the ceeding rows. The ablation studies the remain- ing 21 proxy models are detailed in Appendix H. Focus on the informative keywords. By focus- ing on the keywords, improvements are observed across nearly all metrics. Notably, the Pearson and Spearman correlation coefficients are improved by 5.04% and 7.48%, respectively. These results sug- gest stronger correlation between the keyword uncertainties and passage-level human judgments. Focus on the preceding words. When hallucina- tion propagation incorporated on basis of key- word selection, remarkably, substantial improve- ments can be observed across all metrics. Particu- larly, the AUC-PR of the non-factual class exhib- ited significant increase of 3.67% on LLaMA- 30b. This enhancement can be attributed to the successful remediation of the overconfidence issue as discussed in Section 3.2.
Focus allows small-scale models achieve com- parable performance GPT-3. As shown in Table LLaMA-7b achieves comparable or even superior performance when compared with GPT-3 uncertainties. This observation suggests that de- spite being a powerful LLM with 175b parameters, GPT-3 may be similarly plagued by issues of over- confidence and underconfidence. However, neither the attention weights nor the full probability dis- tribution of GPT-3 are accessible, otherwise, the incorporation of focus would enable uncertainties GPT-3 to yield considerably enhanced results.
4.3 Analysis
Table presents the results of our ablation study conducted on LLaMA-30b. The average halluci- nation score in Equation without any proposed
The overall performance LLaMA-30b with y ranging from 0 to (no hallucination propagation when y is set to zero) is illustrated in Figure 10

---

Text
h
h without type
h with type
Paul Taylor is American singer-
12.38
133.96
major-inaceuratt
14.99
4.09
songwritet, multiinnttrumentaiist, and record producer. He is best known the lead singer songwriter the hand Winger,
minor-inee innocrinaccurate
9.70
3.79
accurate*
5.63
2.75
Table 4: The average hallucination scores for each cate- gory with and without entity type information provided.
C. V. Ananda Bose was Indian free- dom fighter. lawyer, and politician. (...) He was memher of the Indian delega- tion United Nations in 1951, He was a member of the Indian delegation
1.36
53.53
"Paul Taylor an American
the second case, the pink sentence duplicates exist- ing text, consequently producing a significantly low value of / owing to the overconfidence problem. With the introduction of the penalty, the halluci- nation score increases by approximately fifty-fold, demonstrating the effectiveness of focusing on the hallucination scores of the preceding words.
the United Nations 1952.
Table 3: Cases detected by hallucination propagation. and h denote the hallucination scores of the highlighted sentences without and with penalty, respectively.
Appendix G. is evident that most metrics improve as increases. However, a performance decline is noticeable when "y exceeds 0.8, indicating that an excessive focus on the preceding words could also lead to deterioration in performance.
The attention heat maps corresponding to the two cases can be found in Appendix C.
4.4.2 Failure cases after entity type provision
To explain the decrease in AUC-PR of the non- factual* class when entity types are specified for each named entity, computed the sentence-level average hallucination score h for each category in Table 4.
Focus on the token properties. Further enhance- ments model performance can be achieved by incorporating entity type information and token IDE, leading to drastic improvements as evidenced in the last two rows. Specifically, the AUC-PR of the factual class increases by 10.76% and both correlation coefficients improve by approximately 18%. This demonstrates the effectiveness of proba- bility correction in mitigating the underconfidence problem as discussed in Section 3.3. Nevertheless, we observe little improvement for the *on-factual* class when considering only the entity type prop- erty on multiple proxy models. The reason behind this observation will be explained in Section 4.4.2.
We notice that the average hallucination score h for all classes decreases when entity type informa- tion is provided, since the probability is corrected to be more confident for the keywords. However, this decrease is especially noticeable in the ma- jor inaccurate category due to the fact that tences labeled major inaccurate contain more hallucinated keywords. As a result, distinguish- ing between major inaccurate and minor inaccurate becomes more challenging. Given that the non- factual* class only includes sentences classified as major inaccurate, this increased difficulty in differ- entiation contributes to the observed decrease in AUC-PR for the non-factual class.
The performance impact of varying P values is depicted in Figure/ 11 of Appendix G. Generally, P = 0.01 delivers optimal results. A large p could lead to the omission of crucial information due to a restricted candidate set, while small might introduce noise by including irrelevant tokens.
5 Conclusion
4.4
Case study
In this paper. we propose a reference-free uncertainty-based method for detecting hallucina- tions in LLMs. The proposed method aims to imi- tate human factuality checking by considering three aspects: focus on informative keywords, focus on preceding words and focus token properties. Our experimental results empirically demonstrate the effectiveness of the proposed method for hal- lucination detection both sentence and passage level, without requiring any external knowledge or training data. We have also analyzed how each
4.4.1
Non-factual cases detected by hallucination propagation
Table 3 showcases two examples of hallucinated content accurately identified by hallucination prop- agation. In the first case, the pink sentence erro- neously assigns the role of singer and songwriter to Paul Taylor, who was actually a keyboardist/gui. tarist of the band Winger. This error originates from the model's preceding hallucination (purple text)

---

the three focus mechanisms impacts the overall per- formance when using different proxy models as the backbone. The results on XSumFaaith and FRANK datasets further showcase the potential capability of the proposed method for detecting hallucinations produced by small models. We hope our work can contribute to the field of LLM research and help improve the reliability and factuality of LLMs.
Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art, performacce....
David Baidoo- Antloo--Auuue and Leticia Owusu Ansah. 2023. Ed- ucation in the era of generative artificial intelligence (ai): Understanding the potential henefits of chatgpt in promoting teaching and learning. Available at SSRN 4337484.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung. et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, ballucination, and interactivity. arXiv preprin: arXiv:2302.044 2402.0003....
Limitations
The keyword identification and named entity recog- nition in our approach is based on Spacy, which may introduce some errors as observed in our prac- tice. For instance, the television drama "The Great Ambition" could erroneously be classified as an organization. Such failures can result in the calcu- lated probability becoming unreliable, leading to a decrease in performance. Additionally, the cat- egories of named entities real-world scenarios are considerably more diverse than those identifi- able by Spacy, such as food, vehicles, and other specialized domains.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace Hc, Connor Leaby. Kyle McDonell, Jason Phang, al. 2022. Gpt-neox-200 sive language model. In Proceedings of BigScience Episode 5-Workshop Challenges & Perspec- lives in Creating Large Language Models, pages 136.
open- autoregres-
A further limitation arises from our assumption that LLM proxies are consistently current with fac- tual knowledge. However, LLMs are not continu- ously updated post-training, hence they may lack recently emerged factual information. This could influence the assigned probabilities and in turn af- fect our hallucination detection's 's effectiveness.
Meng Cao. Yue Dong, and Jackie Chi Kit Cheung. 2022. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 3340-3354.
Ethics Statement
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
We maintained privacy in our approach, as our method does not require user data and we con- ducted experiments publicly available datasets, upholding privacy and anonymity standards. De- spite the intention of this work is to improve LLMs' reliability, potential misuse such as using to en- hance Al-generated misinformation or deepfake content, is condemned. We are dedicated to eth- ical AI research, addressing potential concerns, and maintaining balance between technological progress and ethical responsibility.
Together Computer. 2023. Redpajama: An open source recipe reproduce llama training dataset.
David Dale, Elena Voita, Loic Barrault, and Marta R Costa-jussà. 2022. Detecting and mitigating halluci- nations machine translation: Model internal work- ings alone do well, sentence similarity even better. arXiv preprint arXi::2212.08597
David Demeter, Gregory Kimmel, and Doug Downey, 2020. Stolen probability: A structural weakness of neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2191-2197.
References
Hussam Alkaissi and Samy McFarlane. 2023. Artifi- cial ballucinations in chatgpt: implications scien- tific writing. Cureus, 15(2).
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine
Yue Dong, John Wieting, and Pat Verga. 2022. Faithful to the document to the world? mitigating hallu- cinations via -ntity-linked knowledge in abstractive summarization. arXiv preprint 2204 2.204.13660 13761.

---

Nouha Dziri, Sivan Milton, Mo Yu, Osmar R Zaiane, and Siva Reddy. 2022. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, pages 5271 1-2255.
Ziwei Ji. Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023, Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):
7denčk Kasner, Simon Mille, and Ondiej Dušek. 2021. Text-in-context: Token-level error detection for table- to-text generation. In Proceedings of the 14th Inter- national Conference on Natural Language Genera- tion, pages 259-265.
Alexander R Fabbri, Wojciech Kryšciński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Wojeiech Kryšciñski, Bryan McCann, Caiming Xiong. and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 22 14-2.
Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nli- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics, 10:163-177.
Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. Trueteacher: Lear- ing factual consistency evaluation with large lan- guage models. arXiv preprint arXiv:2305.1 11171.
Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023, Benefits, limits, and risks of gpt-4 as an ai chathot for medieine. New England Journal of Medicine, 1833133:
Nuno M Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. 2023. Hallucinations in large multilingual translation models. arXiv preprint 1.3v:2230316104,
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising gqquence-to-sqquees
Nuno M Guerreiro, Elena Voita, and André FT Martins. 2022. Looking for needle in a haystack: A com- prehensive study of hallucinations in neural machine translation. arXiv preprint arXiv:2208.00 05309.
pre-training for nat- ural language generation. translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics, pages 7871-7880.
Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embed- dings, convolutional neural networks and incremental parsing. To appear, 7(1):411 1200.
Jiongnan Liu, Jiajie Jin, Zihan Wang, Jiehan Cheng. Zhicheng Dou, and Ji-Rong Wen. 2023. Reta-llm: A augmentee large language model toolkit. arXiv preprint 23062230.00221
Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang. Jun Xie, and Yue Zhang. 2020. What have we achieved on text summarization? In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 446-469.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhi- fang Sui, Weizhu Chen, and William B Dolan. 2022. A token-level reference-free hallucination detection benchmark for free-form text generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6723-6737.
Yichong Huang. Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. The factual inconsistency problem in abstractive text summarization: A survey, arXiv preprint arXiv:2 148.9.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Eatity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7052-7063.
Touseef Iqbal and Shaima Qureshi. 2022. The survey: Text generation models in deep learning. Journal of King Saud Sciences, (4(6)225552222
and Information
Mohd Javaid, Abid Haleem, and Ravi Pratap Singh. 2023. Chatgpt for healthcare services: An emerging stage for innovative perspective. BenchCouncil Transactions on Benchmarks, Standards and Evalua- tions, page 100105.
Alejandro Lopez- and Yuehua Tang. 2023. Can chatgpt forecast stock price movements? return pre- dictability and large language models. Return Pre- dictability and Large Language Models (April 6, 2023).

---

Potsawee Manakul, Adian Liusie, and Mark JF Gales, 2023. Selfcheckgpt: Zero-resource black-bor box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303. 08896.
Association for Computational Linguistics: EMNLP 2020, pages
Clément Rebuffel, Marco Roberti, Laure Soulier, Geof- frey Scoutheeten, Rossella Cancelliere, and Patrick Gallinari. 2022. Controlling hallucinations word level in data-to-text generation. Data Mining and Knowledge Discovery, pages 1-37.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
Malik Sallam. 2023. The utility of chalgpt as exam- ple of large language models in healtheare education. research and practice: Systematic review on the fu- ture perspectives and potential limitations. medRxiv, pages 2023-02.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannanch Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.1425. 14251.
Xinyue Shen. Zeyuan Chen, Michael Backes, and Yang Zhang. 2023a. In chatgpt we trust? measuring and characterizing the reliability of chatgpt. arXiv preprint arXiv:2304.0555 08899997
Niels Mündler, Jingxuan He, Slobodan Jenko, and Mar- tin Vechev. 2023. contradcttyy hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint 2.t.:2335./8552..
Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith D Hentel, Beatriu Reig, George Shih, and Linda Moy. 2023b. Chatgpt and other large language models are double-edged swords.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero dos Santos, Henghui Zhu, Dejiao Zhang. Kathleen Mckeown, and Bing Xiang. 2021. Entity-level fac- tual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 2727-2733.
Dan Su, Xiaoguang Li, Jindi Zhang. Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before generate! faithful long form question answering with machine reading. In Findings of the Association for Computational Linguistics: ACL 2022, pages 744- 756.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following 1lama model. https:/ gi thub. cttpb.coo/taatt
Proceedings of the 2018
Ahmed Tlili, Boulus Shehata, Michael Agyemang Adarkwah, Aras Bozkurt, Daniel T Hickey, Ronghuai Huang, and Brighter Agyemang. 2023. What if the devil is my guardian angel: Chatgpt as a case study of using chatbots in education. Smart Learning Envi- ronments, 10(1):15.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedhack. Advances in Neural Information Processing Systems, 35:27730-27744,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Ilama: Open and effi- cient foundation language models. arXiv preprint 2302
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A benchmark for factuality metries. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4812-4829.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09288.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness dialogue with controllable features. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 704-718.
in Knowledge-grouncte
Liam van der Poel, Ryan Cotterell, and Clara Meister, 2022. Mutual information alleviates hallucinations in abstractive summarization. In EMNLP 2022. arXiv.
Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. 2020. On long-tailed phenomena in neural machine translation. In Findings of the
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all

---

you need. Advances in neural information processing systems, 30.
Specifically, for the FRANK dataset, which pro- vides the error type of each sample, we removed the instances that were labeled as OutE (statement con- tains information not present in the source article) for the reason discussed above. For XSumFaith, we excluded the human-writtee summaries since they may differ in style from model-generated sum- maries (Gekhman et al., 2023). The statistics of the two datasets are shown in Table 5.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-
6B: A 6 Billion Parameter Autoregressive Lan- guage Model, https:/ /ugbt/bbbbb......ee/gg
thub. com/kingofllllz/ mesh-transformer--3
Shijie Wu, Ozan Issoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 17564.
Dataset
#Num
&Hallucination
Yijun Xiao and William Yang Wang. 2021. On hal- lucination and predictive uncertainty in conditional language generation. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2734-2744.
XSumFaith 984
90.40
FRANK 1242
57.41
Table 5:
The statistics of XSumFaith and FRANK dataset (test split from SummaC benchmark).
Weijia Xu, Sweta Agrawal, Fleftheria Briakou, Mari- anna J Martindale, and Marine Carpuat. 2023. Un- derstanding and detecting hallucinations in neural machine translation via model introspection. arXiv preprint 0rXiv:2301007770
We report the AUC-PR for the non-factual class and factual class and balanced-accuracy in Ta- ble 6 and Table 7. Our method performs well across all three metrics when applied to the XSum- Faith dataset. However, we observed that, for the FRANK dataset, using only the negative log prob- ability yields better results compared to using the sum of negative log probability and entropy. Fur- thermore, focus on the keywords proves less ef- fective than considering all tokens in the passage. We attribute this discrepancy to the unique char- acteristics of the FRANK dataset, which contains hallucinations such as predicate errors, pronoun errors, and preposition errors. Therefore, we only use the negative log probability of token t as its hallucination score and disregard keyword selec- tion for FRANK dataset. These results highlight the effectiveness of focusing on token property in- formation, but little enhancement is observed when solely relying on hallucination propagation. Fur- ther investigation left for the future work.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diah, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106:
A Results for detecting hallucinations generated by small models
To assess the effectiveness of our proposed method in detecting hallucinations produced by small models, we conducted experiments using two hallucination detection datasets extracted from the test split of the SummaC benchmark (La- ban et al., 2022): XSumFaith and FRANK. These datasets consist of summaries generated by small models such as TransS2S (Vaswani al., 2017), TCONVS2S (Narayan al., 2018), and BART (Lewis et al., 2020).
Although the benchmark includes a total of six datasets, it should be noted that some of them con- tain summarizations not produced by generative models, such Polytope (Huang et al., 2020) and SummEval (Fabbri et al., 2021). Additionally, cer- tain datasets (Kryšcinski al., 2020; Falke al.. 2019) label any content not present in the input as extrinsic hallucinations. However, such extrinsic hallucination might actually be factual (Dong et al., 2022; Cao et al., 2022).
Method
NonFact
Fact
Balanced-
avg(A)
92.79
11:75
$7.65
+keyword
92.65
14.19
56.24
+penalty
92.34
14.97
57.77
+entity type
94.98
18.46
64.77
+token idr
95.13
18.86
64.81
Table 6: Performance of the proposed method using LLaMA-30b-SEF on XSumFaith dataset (7 = 0.9, p 0.01).
*Only about 5% of such cases in XSumFaith as reported in the original paper, hence, we have included this dataset in our analysis.

---

Method
NonFact
Fact
Balanced-Ace
inputting each passage into the proxy model for hal- lucination detection, the entity types are provided before each named entity recognized by Spacy.
avg(-logp)
89.82
79.00
78.79
+penalty
89.87
78.37
79.46
+entity type
90.44
79.78
80.31
E Details of the proxy models
+token idf
90.12 12
80.00
80.70
The 22 proxy models used in our experiments include LLaMA-(7b, 13b, 30b, 65b) (Touvron et al., 2023a), LLaMA-2(7b, 13b, 70b) (Touvron et al., 2023b), OPT-(125m, 1.3b, 13b, 30b) (Zhang et al., 2022), GPT-J-6b (Wang and Komatsuzaki, 2021) GPT-NcoX- (Black et al., 2022), Falcon- [7b, 40b) (Almazrouei et al.. 2023), Vicuna-(7bb 13b, 33b) (Chiang et al., 2023), RedPajama- (3b,
Table 7: Performance of the proposed method using LLaMA-30b-SFT on FRANK dataset (? 0. 0.01).
B Implementation Details
Our experiments were conducted on an AWS 23dn.24xlarge instance, each which equipped with 8 NVIDIA V100 32GiB GPUs, 96 CPU cores, and 768 GiB RAM. In order to prevent the in- fluence of type tags when calculating the token probability, we set the probability of token "<" to zero. When using the SFT version of LLaMA, the prompt as described in Section 1.1 is formatted to follow the Alpaca (Taori et al., 2023) pattern: "### Instruction: (instruction} ##H# Response: [response}".
7b) (Computer, 2023) and instruction tuning ver-
sions of LLaMA- 30b)-SFTS.
F Performance comparison of LLaMA family
Figure 9 presents the performance comparison among the LLaMA family, Models with larger parameter size generally demonstrate superior per- formance on the WikiBio GPT-3 dataset. How- ever, despite being twice the size of LLaMA-30b, LLaMA-65b underperforms across four out of the five evaluated metrics compared to LLaMA-30b.
For the experiments on the two summarization datasets, we excluded instances where the token count exceeded LLaMA's maximum context length of 2048, resulting in the elimination of 16 cases from the XSumFFath dataset. The prompts em- ployed are "[document) TL; and "Summarrze the following text using appropriate words that follow to the given type: (document) TL; without and with the provi- sion of entity types, respectively.
G Hyper parameters analysis
Figure 10 shows the performance of LLaMA-30b with ? ranging from 0 to 1. When 7 no penalty is accumulated to the token hallucina- tion score. Figure 11 depicts the performance im- pact of varying p. Setting either too large or too small leads to decrease in performance.
zero,
Entity types are also provided in the prompts as few-shot examples. For instance, the prompt for the concept "michael savage" is "This passage from <ORG> Wikipedia about <PERSON>
H Additional Results
The main results including all the 22 proxy mod- els are shown in Table 8. As observed Table 9 to Table 29, our method consistently outperforms the performance achieved by solely relying on the uncertainty metric. The optimal setting may vary across models, we attribute this to the different generation patterns exhibited by each model.
michael savage:".
More attention heat map cases
Figure 4 and Figure 5 provide visualizations of the attention heat maps of the two cases mentioned in Section 1.4.1. The attentions that are erroneously directed towards preceding unreliable tokens are marked within a blue box.
D Examples of passages with entity types provided
Figure 6 to Figure 8 illustrate three examples of Wikipedia passages generated by text-davinci-003, along with their corresponding prompts. Before

---

Figure 4: The attention heat map corresponding the first case in Section 4.4.1. Due space limitations, not all sentences are depicted in the figure.
Figure 5: The attention heat map corresponding to the second case in Section 4.4.1. Due to space limitations, not all sentences are depicted in the figure.
Case 1: Michael Savage
Please complete the passage below using appropriate words that follow to the given type with < wrapped.
This is passage from <ORG> Wikipedia about <PERSON> mi nichael savage:
<PERSON> Michael Alan Weiner (born <DATE> March 31, 1942), , better known by his professional name <PERSON> Michael Savage, is <NORP> American radio host, author, activist, nutritionist, and conservative political commentator. He is the host of <ORG> The Savage Nation, nationally syndicated talk show that aired <ORG> Talk Radio Network across <GPE> the United States until <DATE> 2012, and in <DATE> 2009 was the <ORDINAL> second most listened-to radio talk 20 million listeners <CARDINAL> 408 stations across <GPE> the United ted States. Since <DATE> October 2012, <PERSON> Michael <ORG> Cumulus Media Networks. holds master's 's degrees from <ORG> the University of Hawaii medical botany medical anthropology,
show in the country with audience <CARDINAL>
Savage has been syndicated
<WORK_OF OF_ART> Ph.D. from <ORG> the University of California, Berkeley in nutritional ethnomedicine. As
<PERSON> Michael Weiner, has written books on nutrition, herbal medicine, and homeopathy,
Figure 6: text-davinci-003 generated Wikipedia passage about Michael Savage in WikiBio GPT-3 dataset.

---

Case 2: Michael Replogle
Please complete the passage below using appropriate words that follow the given type with wrapped.
This is passage from <ORG> wikipedia about <PERSON> michael replogle:
<PERSON> Michael Replogle (born <DATE> 1951) an <NORP> American environmentalist and + transportation planner. He is the founder and director <ORG> the Institute for Transportation and Development Policy (ITDP), global non-profit organization that works to promote sustainable transport solutions in cities around the world. <PERSON> Replogle has been a leader in the field of sustainable transportation for <DATE> more than four decades, and has been credited with helping to shape the modern urban transport landscape. He has worked with cities in <CARDINAL> more than 40 countries to develop and implement sustainable transport policies and projects, including bus rapid transit, bike-sharing, and cedestrian-friendie streets. He has also been vocal advocate for the use of pricing mechanismy sms to reduce traffic congestion and air pollution.
Figure 7: The text-davinci-003 generated Wikipedia passage about Michael Replogle in WikiBio GPT-3 dataset.
Case 3: Tommy Nutter
Please complete the passage below using appropriate dords that follow to the given type with < wrapped.
This is passage from <ORG> Wikipedia about <PERSON> tommy nutter:
<PERSON> Tommy Nutter (1943-1992) was
British tailor who was major figure in fashion world <DATE> the late 1958s <DATE> early 1970s. was known for his flamboyant style work with <ORG> Rolling Stones, <PERSON> Elton John, and other celebrities. born <GPE> London and began career as an apprentice tailor <DATE> the age of 15, opened his own shop, Nutters of Savile Row, <DATE> 1969. His designs were known their bold colors and patterns, and was one the <ORDINAL> first to introduce the "peacock look" to men's 's fashion. He was also <CARDINAL> one of the <ORDINAL> first use denim in men's suits. He was a major influence on the punk and new wave fashion movements of <DATE> the late 1970s and <DATE> early 1980s. He died AIDS-related complications in <DATE> 1992.
was
Figure 8: The text-davinci-003 generated Wikipedia passage about Tommy Nutter in WikiBio GPT-3 dataset.
puc_pr_no www.onnnnonoooooooo factual fracuul
SUC_PC factual paarson
spearman
auc pl non, faciual auc non factual
auc or, factual
spearman
n00, fastual*
pearson
0.0
0.7
Setersanc
0.7
0.6
0.5
LLMM7-7b
LLaMA-13b
LLbMW-30b
LLaM-65b
0.0
0.2
0.4
0.6
1.0
model
Figure 9: Performance comparison of LLaMA family with varying parameter sizes.
Figure 10: Performance of LLaMA-30b with different
7.

---

Sentence-level Metrics NonFact
Passage-level Metries Spearman
Method
NonFact
Factual
Pearson
GPT-3 Uncertainties
Avg(-logp)
83.21
38.89
53.97
57.04
53.93
Avg(H)
80.73
37.09
$2.07
55.52
50.87
Max(-logp)
87.51
35.88
50.46
57.83
55.69
Max(H)
85.75
32.43
50.27
52.48
49.55
SelfCheckGPT
BERTScore
81.96
45.96
44.23
58.18
55,90
QA
84.26
40.06
48.14
61.07
59.29
Unigram (max)
85.63
41.04
58.47
64.71
64,91
Combination
87.33
44.37
61.83
69.05
67.77
Ours
GPT-J-6B Jocica
77.92
38.56
33.58
15.68
13.95
GPT-NeoX-2000 Joccs
81.40
38.84
39.80
35.03
30.40
OPT-125M facwxxx faeus
73.88
34.74
28.29
-8.04
-5.92
OPT-1.3B foems
73.84
34.00
30.88
1.08
-1.20
OPT-13B facus
79.63
39,97
39.23
23.65
OPT-30B Jocus
79.26
39.49
40.63
31.07
28.67
Falcon- focus
82,25
40.94
41.25
45.19
36.76
Falcon-408, facus
88.11
46.95
58.14
68.63
64.66
Vicuna- 7b, 7brocu
84.14
39.93
53.41
58.78
49.84
Vicuna- 13b 3bfocwa
86.87
41.80
60.25
66.72
58.64
Vicuna-333 -3mma--33y-mm 33bfocua
88,23
44.51
62.10
71.82
65.96
RedPajama-32 focua
82,26
40.49
43.38
46.48
40.56
RedPajama-77 focus
84,68
41.53
50.05
55.55
49.74
LLaMA 7B Focus
84.26
40.20
57.04
64.47
54.73
I.LMA-22 2-7B focus
84,29
41.31
56.64
63.59
48.91
LLaMA- 1.B. Tocus
87.90
43.84
62.46
70.62
63.03
ALaMA-2-13BB focuu
87.28
45.62
63.39
71.57
63.85
LLaMA-13B 11--A---B-FFFFT-.. -STT
youus
88.17
44.62
62.25
71.91
63.81
LLaMA-30B foems
89.79
48.80
65.69
77.15
73.24
AlaMA-30B-SPP focus
90.34
49.17
65.29
77.53
73.10
LLaMA-65B Focuss
89.94
48.69
64.90
76.80
73.01
I. focus
89.95
52.06
65.11
72.36
Table 8: Main results including all proxy models in Section 4.1.
Method
NoFac
NoFac*
Fact
Pear.
Spear.
avg(h)
78.67
37.19
35.81
29.1
16.65
+keyword
79.27
37.28
35.86
36.26
23.61
spearman
+penalty 83.48
43.77
49.22
46.22
38.32
- auc pr. non, fatinall
peanson
+entity type
83.40
39.27
56.36
62.57
52.63
+token
84.26
40.20
57.04
64.47
54.73
0.0
Table 9: Ablation study of the proposed method using LLaMA-7b (7 0.9, p 0.01).
05formance
Method
NoFac
NoFac*
Fact
Spear.
0.6
avg(A)
80.18
38.57
41.21
39.80
25.99
+keyword
80.93
38.85
40.31
45.83
32.37
+penalty
83.26
43.02
43.35
37.92
33.99
+entity type
87.12
43.13
61.72
69.99
60.64
+token idf
87.90
43.84
62.46
70.62
63.03
Figure 11: Performance ILLMM-300 with different
Table 10: Ablation study of the proposed method using LLaMA- ( 0.9, 0.01).
P.

---

Method
NoFac
NoFac*
Fact
Pear.
Spear.
Method
NoFac
NoFac*
Fact
Pear.
Spear:
avg(h)
82.62
40.94
48.74|
52.60
40.85
avg(h)
76.29
27.08
29.65
11.28
3.51
+keyword
83.64
41.00
46.77
58.01
49.44
+keyword
77.50
27.56
32.85
19.50
8.34
+penally
88.06
46.94 96.94
49.49
54.92
56.69
+penalty
74.59
33.83
38.53
23.30
9.87
+entity type|
89.54
47.66
64.27
76.30
72.54
+entity type
83.14
38.45
52.50
56.32
48.41
+token idf
89.94
48.69
64.90
76.80
73.01
#token idf
84.14
39.93
53.41
58.78
49.84
Table 11: Ablation study of the proposed method using LLaMA-65b (y = 0.9, p 0.01).
Table 16: Ablation study of the proposed method using Vicuna-7b (1 0.9, = 0.01).
Method
NoFac
NoFac*
Fact
Pear.
Spear.
Method
NoFac
NoFac
Fact
Pear.
Spear.
avg(h)
80.00
39.28
41.22
39.36
24.80
avg(h)
79.21
36.42
35.44
27.53
17.35
+keyword
81.01
39.29
41.14
46.43
31.94
+keyword
80.58
36.46
37.10
40.37
27.67
+penalty
84.39
45.06
51.36
48.81
40.78
+penalty
84.52
43.10
56.41
51.52
40.13
+entity type|
87.81
44.28
61.81
71.54
62.75
+entity type
86.
76.78
41.35
1..996
67.24
58.90
+token idf
88.17
44.62
62.25
71.91
63.81
+token idf
86.87
41.80
60.25
66.72
58.64
Table 12: Ablation study of the proposed method using LLaMA-13b-SE 13b-SFTT (7 = 0.9. = 0.01).
Table 17: Ablation study of the proposed method using
Vicuna- 13b (7 = 0.9, p = 0.01).
Method
NoFac
NoFac
Fact
Pear.
Spear.
Method
NoFac
NoFac*
Fact
Pear.
Spear.
avg(h)
81,58
42.19
47.56
49.02
35.50
avg(A)
81.96
41.91
42.83
42.09
31.30
+keyword
83.32
42.63
47.13
56.45
45.38
+keyword
82.95
40.87
42.90
49.91
39.35
+penalty 86.95
45.74
59.60
66.56
59.14
+penalty
86.59
47.65
61.022
62.39
55.71
+entity type|
89.92
48.52
65.12
77.48
72.42
+entity type
88. 10
44.40
61.35
71.06
64.53
+token idf
90.34
49.17
65.29
77.53
73.10
+token idf
88.23
44.51
62.10
71.82
65.96
Table 13: Ablation study of the proposed method using LLaMA-30b-SFI (7 = 0.9, p = 0.01).
Table 18: Ablation study of the proposed method using Vicuna-33b (7 = 0.9, p = 0.01).
Method
NoFac
NoFac
Fact
Pear.
Spear.
Method
NoFac
NoFac*
Fact
Pear.
Spear.
avg(h)
77.06
35.47
29.42
9.64
3.88
avg(h)
77.48
32.96
30.24
18.92
5.91
+keyword
77.59
35.61
30.41
19.24
10.98
+keyword
78.32
34.00
32.15
28.78
14.05
+penalty
80.74
42.23
38.03
24.18
20.17
+penalty
82.36
42.41
47.25
42.65
26.51
+entity type|
81.66
41.55
40.03
44.04
36.62
+entity type
82.02
40.46
43.24
46.31
39.44
+tkeen idf
82.25
40.94
41.25
45.19
36.76
+oken idf
82.26
40.49
43.38
46.48
40.56
Table 14: Ablation study of the proposed method using Falcon-7b (y = 1.0. = 0.01).
Table 19: Ablation study of the proposed method using RedPajama-3b (? 1.0, 0.01).
Method
NoFac
NoFac
Fact
Pear.
Spear.
Method
NoFac
NoFac
Fact
Pear.
Spear.
avg(h) 79.72
37.50
32.37
34.00
27.47
avg(h)
79.43
34.37
33.22
36.82
22.56
+keyword
80.55
37.62
35.13
45.45
38.11
+keyword
80.33
35.46
35.92
44.25
30.43
+penalty
87.26
47.22
44.88
47.67
52.01
+penalty
83.57
41.33
44.87
42.34
35.39
+entity type|
87.11
45.74
57.60
68.25
62.46
+entity type
84.34
40.87
50.53
56.38
50.28
+token idf
88.11
46.95
58.14
68.63
64.66
+oken idf
84.68
41.53
50.05
55.55
49.74
Table 15: Ablation study of the proposed method using Falcon-40b ( = 0.9, 0.01).
Table 20: Ablation study of the proposed method using RedPajama-7b (y = 0.9, p = 0.01).

---

Method
NoFac
NoFac*
Fact
Pear.
Spear.
avg(h)
75.64
33.34
28.30
-0.38
-9.30
Method
NoFac
NoFac*
Fact
Pear.
Spear.
+keyword
76.31
33.99
29.61
9.26
-2.55
avg(h)
76.98
33.77
29.65
5.52
-1.62
+penally 77.51
38.05
37.54
25.50
7.06
+keyword
77.61
34.59
31.55
14.25
2.98
+entity type|
77.68
37.73
33.98
17.82
15.29
+penalty
79.67
39.91
43.18
31.42
16.32
+token idf
77.92
38.56
33.58
15.68
13.95
+entity type
79.31
39.32
41.66
33.75
28.98
+token idf
79.26
39.49
40.63
31.07
28.67
Table 21: Ablation study of the proposed method using GPT-J-6b (7 = 1.0, p = 0.01).
Table 26: Ablation study of the proposed method using OPT-30b (v 1.0, p 0.01).
Method
NoFac
NoFac*
Fact
Pear.
Spear.
avg(h)
77.14
33.49
30.71
11.55
3.18
+keyword
77.97
33.70
32.93
23.84
11.53
+penalty
80.77
40.22
43.01
37.46
23.13
Method
NoFac
NoPac*
Fact
Pear.
Spear.
+entity type|
80.12
37.50
38.70
31.24
25.08
avg(h)
77.04
34.22
30.08
28.83
13.62
+token idf 81.40
38.84
39.80
35.03
30.40
+keyword
78.20
35.64
33.23
38.39
23.96
Table 22: Ablation study of the proposed method using GPT-NeoX-2066 (y = 1.0, = 0.01).
+penalty
84.02
42.90
39.17
39.10
41.84
+entity type
83.55
41.12
55.57
62.05
47.94
+token idf
84.29
41.31
56.64
63.59
48.91
Method
NoFac
NoFac
Fact
Pear.
Spear.
avg(h)
71.05
30.96
24.64
-19.84
-23.08
Table 27: Ablation study of the proposed method using LLaMA-2-7b (? = 0.9. 0.01).
+keyworl
71.81
32.65
25.08
-15.16
-19.79
+penalty
72.71
37.25
26.
-9.94
-14.61
+entity type
73.64
34.64
28.12
-9.09
-6.75
+taken idf
73.88
34.74
28.29
-8.04
-5.92
Method
NoFac
NoFac
Fact
Pear.
Spear.
Table 23: Ablation study of the proposed method using OPT-125m (7 = 1.0, p = 0.01).
avg(h)
77.71
34.31
32.24
36.55
22.61
+keyword
79.75
36.24
35.22
47.11
34.73
+penalty
84.36
43.64
51.50
52.88
44.72
Method
NoFac
NoFac
Fact
Pear:
Spear.
+entity type
85.87
43.46
63.20
71.24
59.62
avg(h)
73.73
32.15
26.00
-11.16
-17.54
+token idf
87.28
45.62
63.39
71.57
63.85
+keyword
74.32
33.51
27.05
-3.51
-13.54
+penalty 74.84
37.37
31.13
4.08
-8.62
Table 28: Ablation study of the proposed method using LLaMA-2-13b (1 = 0.9, p= 0.01).
+entity type|
73.50
33.77
30.02
-1.03
-2.59
+token idf 73.84
34.00
30.88
1.08
-1.20
Table 24: Ablation study of the proposed method using OPT-1.3b (7 1.0, = 0.01).
Method
NoFac
NoFac*
Fact
Pear.
Spear:
Method
NoFac
NoFac*
Fact
Pear.
Spear.
avg(h)
79.05
37.35
36.99
49.79
39.58
avg(h)
76.77
33.82
29.75
4.36
-3.96
+keyword
81.50
39.17
40.78
59.18
51.97
#keyword
77.40
34.44
31.67
13.79
1.89
+penalty 86.76
46.39
50.34
55.53
58.53
+penalty
79.72
39.44
40.65
29.13
14.30
+entity type
89.66
51.33
65.14
77.58
72.43
+entity type|
79.06
39.19
38.74
28.42
22.59
+token
89.95
52.06
66.11
76.88
72.36
+token idr
79.63
39.97
39.23
27.88
23.65
Table 29: Ablation study of the proposed method using LLaMA-2-708 (1 0.9, p = 0.01).
Table 25: Ablation study of the proposed method using OPT-13b (y 1.0, p 0.01).


Reviews:
[{"review_id": "qmA5uCinjv", "reviewer": "Reviewer_4M4D", "summary": "", "questions": "(1) Are the results by LLaMA variants (without focus) reported in the paper (in case I did not see the results)?\n\n(2) Is it possible to use some benchmarks as detailed in Section 4.2 in \n\nZha, Yuheng, et al. \"AlignScore: Evaluating Factual Consistency with a Unified Alignment Function.\" arXiv preprint arXiv:2305.16739 (2023).?\n\n(3) Can the proposed approach also work for some other model backbones?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for raising these questions. We will explain each of your concerns below:\n\n*[Reasons to reject 1, Question 1]* \n\nYes, the results by LLaMA variants without focus are reported in the ablation study of our paper. We present the ablation study of LLaMA-30b in Table 2 of Section 4.3 (lines 509\\~512). And for all the 19 proxy models, the ablation study is presented in Table 9 to Table 26 of Appendix H. In the first row of each ablation study table, we presented the result of using loss and entropy of the corresponding model as a token-level hallucination score, **which is the result without focus as you mentioned**. From the results we can observe **incremental effectiveness of our proposed method with each focus trick incorporated, compared to the first row where no focus trick is applied**. We will modify the table captions and relevant content to make it easier to understand our ablation study settings.\n\n*[Reasons to reject 2, Question 2]* \n\nYes, **we also conducted experiments on XSumFaith and FRANK datasets**, which are described in the dataset introduction section (Section 4.1 lines 393\\~399) and **the results are presented in Appendix A**. It's worth noting that **these two datasets are derived from the SummaC benchmark**, which is detailed in the paper you referenced (Zha, Yuheng, et al. \"AlignScore: Evaluating Factual Consistency with a Unified Alignment Function.\" arXiv preprint arXiv:2305.16739 (2023)). Since our paper primarily focuses on hallucination detection in **LLM** (Section 4.1, lines 399\\~402) whose hallucination patterns tend to be divergent from those in small models or generated by humans (Sections 2.2, lines 171~177), we did not include these results in the main text. In the final version, we will consider moving these results to the main text to make them more visible to the readers.\n\n*[Reasons to reject 3, Question 3]* \n\nYes, **our experiments were conducted on 19 diverse backbone models (Section 4.1, lines 445\\~449), including falcon-7b and falcon-40b**. Due to space limitations, we only presented results of the LLaMA family in the main text (Section 4.2, lines 462\\~465). **The full results for all 19 different backbone models, along with corresponding ablation study results, are included in Appendix H**. The ablation study results of all the 19 models demonstrate that **our method significantly enhances the hallucination detection capabilities of these backbone models compared to not using any focus tricks**. In the final version we will add more textual notes in section 4.2 to avoid misleading the readers that we only conducted the experiments on the LLaMA family.\n\n*[Reasons to reject 4]* \n\nGiven that the current state of research on hallucination detection in **LLM** is still exploratory, the availability of viable baselines is limited. In the last two months, we found several related works, and we will consider adding them in the camera-ready version for comparison.\n\n*[Reasons to reject 5]* \n\nTo illustrate how we \"focus on the preceding words\", we used red arrows in Figure 1 to indicate how uncertainties from preceding tokens are propagated to tokens in the succeeding context. To demonstrate \"focus on the token properties\", a green \\<DATE\\> tag is positioned before '1992', showcasing our utilization of in-context learning to guide the model's attention towards the specified token type of the succeeding word. We are sorry for the confusion caused by the mismatch between the caption and figure, we will consider rephrasing the caption to make it easier for readers to follow our ideas in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper studies an important and emerging problem. The problem setting is well motivated.\n\n(2) The proposed solution has good intuitions and easy to follow.\n\n(3) The proposed approach has strong empirical performances.", "reasons_to_reject": "(1) More details can be included in the experiments. For example, in Table 1, LLaMA variants without focus can also be included. LLaMA-30B focus and LLaMA-65B focus clearly outperform the baselines. However, without showing the results by LLaMA variants (without focus), it is unclear whether the improvement is from (i) the difference between LLaMA and GPT or (ii) from the advantages by the proposed focus approach. \n\n(2) I am not very sure about ‘We evaluated our proposed method on WikiBio GPT-3 dataset (Manakul et al., 2023), which, to the best of our knowledge, is the only publicly accessible dataset for LLM hallucination’? There could be multiple benchmarks, for example, as detailed in Section 4.2 in \n\nZha, Yuheng, et al. \"AlignScore: Evaluating Factual Consistency with a Unified Alignment Function.\" arXiv preprint arXiv:2305.16739 (2023).\n\n(3) To show the proposed approach is fundamental and generic, the paper may also consider to show the improvement by another model backbone (e.g., falcon).\n\n(3) (Minor) More baselines can be added. Although it is understandable that the proposed method is not directly comparable to the LLMs, which either relies on external knowledge for reference retrieval or require multiple responses to be sampled from LLM for consistency verification. It is still interesting to show some baselines results, to understand the usability of the developed approach.\n\n(5) (Minor) In Figure 1's caption, it is easy to understand ‘1) focus on the informative keywords’. However, it is difficult to find related illustrations for ‘2) focus on the preceding words; 3) focus on the token properties.’ in the figure.", "questions_to_authors": "(1) Are the results by LLaMA variants (without focus) reported in the paper (in case I did not see the results)?\n\n(2) Is it possible to use some benchmarks as detailed in Section 4.2 in \n\nZha, Yuheng, et al. \"AlignScore: Evaluating Factual Consistency with a Unified Alignment Function.\" arXiv preprint arXiv:2305.16739 (2023).?\n\n(3) Can the proposed approach also work for some other model backbones?", "paper_topic_and_main_contributions": "Existing works for detecting hallucinations in LLMs either relies on external knowledge for reference retrieval or require multiple responses to be sampled from LLM for consistency verification, making these methods costly and inefficient. This paper proposes a novel reference-free, uncertainty-based method for detecting hallucinations in LLMs. Our approach imitates human focus in factuality checking from three aspects: 1) focus on the informative keywords; 2) focus on the unreliable tokens in history context; and 3) focus on the token properties.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mJ9kCWKDGd", "reviewer": "Reviewer_3BQy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper. We appreciate your positive feedback on our approach and findings. Below, we address your comments and indicate how we plan to revise the relevant sections in our paper:\n\n(1) We acknowledge your suggestions for revisions to the abstract. We will make the abstract more concrete and engaging in the final version by reducing the introduction to the hallucination and adding more about the motivation and underlying ideas of our approach.\n\n(2) We agree with your point. We will rephrase the sentences (lines 121\\~124) in the final version to ensure that the message is conveyed concisely and effectively.\n\nThank you again for your suggested changes and we will ensure that our final version will effectively address the concerns you've raised.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Uncertainty based hallucination calculation based on good observation of the hallucination phenomena.\n\n(2) for the overconfident problem, propose penalty approach to mitigate the hallucination\n\n(3) for underconfident problem, propose to adjust the score based on term statistics\n\n(4) extenstive experiments to support the proposed approach ", "reasons_to_reject": "Not necessarily a strong negative point:\n\nThere is improvement space for the writing. Some suggestions are given in other section of the review. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on model based hallucination detection approach. Different from sampling based or retrieval based approach, this paper adopts uncertainty based approach to calculate the hallucination score. Big models are used as proxy language model to calculate the generation probability of each term, which lays the foundation to calculate the uncertainty score. However, a simple setup of the proxy model does not suffice to succeed, which either present \"overconfident\" or \"underconfident\" behavior. \n\nFrom experiment observation, authors of this paper find that (1) there is strong attention relationship among the hallucinated terms, which leads to the \"overconfident\" behavior to generate more hallucination content; (2) while there is difference in next word prediction between real human's mindset and model-based approach. Real person is prone to generate the next word based on semantic info of the context such as entity type to narrow down the next word choice into a small space, but the model would consider and sum up all possibilities to predict next word, which makes the correct next word under-scored. (underconfident). \n\nBased on these observation, authors in this paper propose (1) penalty on the strongly connected hallucinated words; (2) adjust the under scored term based on term statistic properties. To make these penalty and rewards actions happen, the informative keywords in the context should first be identified to support these further adjustment to eventually get the uncertainty score for hallucination detection.\n\nExperiments are conducted extensively on different LLMs mainly on wikiBio dataset. The experiment results well support the proposed approach. Ablation study is also given to present the boost performance of \"penalty\" and term statistics. More extensive experiments are provided in appendix. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "(1) abstract: hallucination is a well known research problem, it might be better to reduce the introduction of hallucination, while introduce more on the proposed approach. For example, what observation inspire authors to propose the approach in the paper? What are the major improvements on the benchmark evaluation? These would make the abstract more concrete and attractive to readers. \n\n(2) Introduction: emphasize more on the findings and proposed approach. Some sentence could be revised in plain words. (e.g. Line 121 to Line 124)"}}, {"review_id": "rIzskgNvf2", "reviewer": "Reviewer_2xZc", "summary": "", "questions": "N.A.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We'd like to address your valuable feedback below:\n\n+ Indeed, \"focus\" is a pivotal concept in our work, we intended to convey the idea of \"focus\" as \"highlighting\" or \"prioritizing\" certain aspects within the context of hallucination detection, similar to how a human might emphasize certain aspects when detecting hallucination. We agree that the term \"focus\" might have been employed in an unnatural manner, leading to confusion. To alleviate this, we will rephrase the sections you pointed out in the final version, ensuring that the meaning and purpose of \"focus\" are explicitly and intuitively explained.\n+ We agree with your comments about exposure bias (lines 265\\~270). In the final version, we will provide an example of this concept and its significance in our work to enhance the clarity of our presentation.\n+ We acknowledge that this is an important point, and we will represent both the 2 probabilities, 0.49 and 0.9, in **Figure 1** in the final version, and carefully revise the wording of lines 272\\~287 to ensure that the conveyed message is aligned with the information presented in the figure.\n\nThank you again for your suggestions and we will ensure that our final version will effectively address the concerns you raised.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1: Explains the class of hallucination methods, why they are not perfect. \n2: Identifies aspects of human behavior used in resolving hallucination. \n3: Explains the concepts through well reasoned text, equations and diagrams.\n4: Performs good experiments, satisfying ablation studies and achieve results higher than baselines. Also appropriate baselines selected.\n5: Fair on limitations - I was impressed.", "reasons_to_reject": "N.A.", "questions_to_authors": "N.A.", "paper_topic_and_main_contributions": "Hallucination plagues the LLM progress and makes them inviable for industry use. Hallucination detection is either based on retrieval of knowledge base or expensive sampling of answers from the LLM. Both these methods have limitations. The paper aims to address these limitations by proposing a novel reference free uncertainty based method for detecting hallucinations. They demonstrate through experiments, human evaluations and ablative analysis that their approach is effective and beats performances by other systems. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N.A.", "Justification_for_Ethical_Concerns": "N.A."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please explain properly what focus means? The use of phrases like line 291 \"...we focus on hallucination scores of preceding tokens by...\" or line 110 \"... to strengthen the focus on hallucination\".  use focus in an unnatural way. To me it feels like the use of the word \"attention\" in transformers paper. But in the latter, they gave a good definition of it. This, otherwise well written paper, was difficult to comprehend every time focus was mentioned unnaturally. \n\nPlease explain exposure bias in lines 265-270 by giving example. People may not be aware of it and it is a key point that must be elaborated. \n\nPlease rephrase lines 272 to 287. They make it seem like the figure 2 will show probabilities 0.49 and 0.90 but it confuses the reader when the figure shows no such thing. I get the points you are trying to make here but it should be rephrased. "}}]

Author rebuttal:

Meta-review: This paper addresses the hallucination problem with a method more efficient than previous attempts that require multiple samples or external knowledge. Their uncertainty based method is said to resemble human focus in focusing on keywords, uncertain tokens in the context, and token properties. The reviewers agree that the paper is sound and exciting and mainly suggest minor revisions to improve the writing. The reviewers note the intuitive approach is valuable and performs well across diverse experimental settings. It appears that the contributions of this paper will be valuable to the community.
Decision: Accept-Main