HaDeMiF: Hallucination Detection and Mitigation in Large Language Models
Authors: Xiaoling Zhou
Mingjie Zhang
Zhemg Lee
Wei Ye
Shikun Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large language model
knowledge hallucination
hallucination detection
model calibration
deep neural decision tree
Primary area: generative models
TL;DR: This study presents a comprehensive framework, named HaDeMiF, for the detection and mitigation of hallucinations, which harnesses the extensive knowledge embedded in both the output space and the internal hidden states of LLMs.

Abstract: The phenomenon of knowledge hallucinations has raised substantial concerns about the security and reliability of deployed large language models (LLMs). Current methods for detecting hallucinations primarily depend on manually designed individual metrics, such as prediction uncertainty and consistency, and fall short in effectively calibrating model predictions, thus constraining their detection accuracy and applicability in practical applications. In response, we propose an advanced framework, termed HaDeMiF, for detecting and mitigating hallucinations in LLMs. Specifically, hallucinations within the output and semantic spaces of LLMs are comprehensively captured through two compact networks—a novel, interpretable tree model known as the Deep Dynamic Decision Tree (D3T) and a Multilayer Perceptron (MLP)—which take as input a set of prediction characteristics and the hidden states of tokens, respectively. The predictions of LLMs are subsequently calibrated using the outputs from the D3T and MLP networks, aiming to mitigate hallucinations and enhance model calibration. HaDeMiF can be applied during both the inference and fine-tuning phases of LLMs, introducing less than 2% of the parameters relative to the LLMs through the training of two small-scale networks. Extensive experiments conclusively demonstrate the effectiveness of our framework in hallucination detection and model calibration across text generation tasks with responses of varying lengths.

Published as a conference paper at ICLR 2025
HADEMIF: HALLUCINATION DETECTION AND MITI- GATION IN LARGE LANGUAGE MODELS
Xiaoling Zhou', Mingjie Zhang', Zhemg Lee", Wei Yet.", & Shikun Zhang Peking University Tianjin University
ABSTRACT
The phenomenon of knowledge hallucinations has raised substantial concerns about the security and reliability of deployed large language models (LLMs). Cur- rent methods for detecting hallucinations primarily depend on manually designed individual metries, such as prediction uncertainty and consistency, and fall short in effectively calibrating model predictions, thus constraining their detection ac- curacy and applicability in practical applications. In response, we propose an advanced framework, termed HADEMIF, for detecting and mitigating hallucina- tions in LLMs. Specifically, hallucinations within the output and semantic spaces of LLMs comprehensively captured through two compact networks novel, interpretable tree model known as the Deep Dynamic Decision Tree (D3T) and Multilayer Perceptron (MLP))which take input set of prediction charac- teristics and the hidden states of tokens, respectively. The predictions of LLMs are subsequently calibrated using the outputs from the D3T and MLP networks, aiming to mitigate hallucinations and enhance model calibration. HADEMIF can be applied during both the inference and fine-tuning phases of LLMs, introducing less than 2% of the parameters relative to the LLMs through the training of two small-scale networks. Extensive experiments conclusively demonstrate the effec- tiveness of our framework in hallucination detection and model calibration across text generation tasks with responses of varying lengths.
INTRODUCTION
In recent years, large language models (LLMs)) have made remarkable advancements, show- casing outstanding performance across a wide range of applications (Schaeffer et al. 2024
Thirunavukarasu al. 2023 Achiam et al. 2023). Despite their impressive performance, these models remain susceptible to knowledge hallucination (Cohen et al.) 2023 Liu et al. 2025 Zhang al. 2023). that is generating nonfactual responses with unwarranted confidence. This issue under- mines user trust and significantly restricts the applicability of LLMs in domains that demand high reliability, such as legal, financial, and educational domains Zhou et al. (2024a). Consequently, de- tecting and mitigating hallucinations in LLMs has garnered increasing attention from the academic community (Azaria Mitchell 2023; Zhang et al., 2024b. Kuhn et al.) 2023 He al. 2025).
Previous studies have proposed various approaches for hallucination detection (Huang et al. 2023
Ji et al. 2023a), targeting either the output space or the internal states of LLMs. For example, predictive confidence and entropy have proven to be effective in detecting hallucinations in natural
language processing tasks (Malinin & Gales 2020; Manakul et al. 2023) Kadavath et al., 2022
Yi et al. 2023 Zhou al. 2023). Moreover, many studies evaluate hallucinations by leveraging the self-consistency of LLMs across multiple predictions for the same query (Liang et al. 2024
Wang al.) 2023). In contrast to output space-based detection, Chen al. (2024a) introduced a method that utilizes the internal states of L.Mss, capturing divergence and correlation between different sentence representations through the eigenvalues of the covariance matrix. Although these methods have proven to be effective, their reliance on single-aspect indicators, such as uncertainty, consistency, and eigenvalues, confined to either the output space or the internal space, constrains their detection accuracy and hampers their generalizability to more complex scenarios or diverse data distributions (Chen et al. 2024a Wang et al. 2023 Manakul et al. 2023). Additionally, the
"Corresponding wye@pku.edu.cnm zhangsk @ pku.edu.cn.

---

Published as a conference paper at ICLR 2025
majority of these methods do not facilitate hallucination mitigation and model calibration, restricting their effectiveness in generation tasks (Manakul et al. 2023: Su et al. 2024 Zhang et al. 2023).
In response to these challenges, this study introduces comprehensive Hallucination Detection and Mitigation Framework called HADEMIF, which leverages the rich knowledge embedded in both the output space and internal hidden states of LLMs to identify and address hallucinations. Specif- ically, two efficient deep networks are employed to detect hallucinations and generate adjustment terms that calibrate the model's probability distribution, thereby achieving both hallucination miti- gation and model calibration. First, we propose a novel interpretable tree model, termed the Deep Dynamic Decision Tree (D3T), to detect hallucinations in the output space by leveraging prediction characteristics such as prediction confidence, uncertainty, and consistency, extracted from the LLMs as inputs. As classification model, D3T predicts whether the generations are hallucinated. not only benefits from gradient descent training but also provides strong interpretability, enabling the identification of key characteristics that most significantly impact hallucination detection. Addition- ally, Multilayer Perceptron (MLP) is employed to capture hallucinations within the deep semantic space, using token hidden states as input. Subsequently, we leverage the outputs from the two hallu- cination detection networks to calibrate the predictions, aiming to maximize the token probabilities for correct generations while reducing the likelihood of incorrect ones. Our HADEMIF framework can be applied during both the inference and fine-tuning phases of LLMs, introducing less than 2% additional parameters relative to those of the L.Mss. To fine-tune LLMs, we further propose an opti- mization procedure that alternately updates the LLM and the two hallucination detection networks.
The proposed HADEMIF framework is evaluated using the calibration evaluation (CAT) benchmark developed by Liu al. 2024). in both in-context learning (ICL) Zhou et al. (2024d) and fine-tuning scenarios. This benchmark includes a variety of text generation tasks, with responses differing in length from individual phrases and sentences to full paragraphs. Six popular open-soure LLMs are utilized for evaluation: GPT-2 (Radford et al.] 2019)). GPT-J (Wang & Komatsuzaki (02I), LLaMA (Touvron et al. 2023ap, Llama2 (Touvron et al. 2023b). Llama3 (Dubey et al.) 2024), and Vicuna (Chiang et al. 2023), with model sizes ranging from I. 1.5B to 30B parameters. The experi-
mental results conclusively demonstrate the efficacy of the HADEMIF framework in hallucination detection and model calibration, achieving substantial improvements over existing approaches.
In summary, the primary contributions of our work are as follows:
We propose an advanced framework, termed HADEMIF, for the detection and mitigation of hallucinations in LLMs. This framework comprehensively captures hallucinations within both the output and internal spaces of LLMs through two compact networks and achieves prediction calibration based on the network outputs.
We introduce a novel interpretable tree model, named D3T, which is not only trainable via gradient descent but also maintains inherent interpretability. This model provides clear explanation of the impact of various prediction characteristics, such as uncertainty and consistency, hallucination detection.
Our proposed framework can be applied during both the inference and fine-tuning phases of LLMs. For fine-tuning, we outline a detailed optimization process that alternatively updates the LLM and the two hallucination detection networks.
We conduct extensive experiments eration tasks with varying response lengths. The results consistently demonstrate the ef- fectiveness and broad applicability of our approach in hallucination detection and model calibration, achieving up to 51% reduction in the average expected calibration error.
range of open-source LLMs, covering text gen-
2 RELATED WORK
Hallucination Detection Existing approaches to hallucination detection in the output space typi- cally fall into several categories: performing conventional fact-checking tasks that rely on external knowledge for supervision (Min et al., 2023); assessing model uncertainty, where uncertain outputs are indicative of hallucinations (Xiao & Wang 2021 Zhou et al. 2022;] Yin et al., 2023 Duan et al.)
2023); measuring the inconsistency of claims between different LLMs (Cohen et al. 2023 Yang
et al., ) and evaluating colfssonnis (Wang et al.] 2023 2025). ) where inconsistent out-
puts often signal hallucinations. Recent research suggests that hallucinations can be traced back to
2

---

Published as a conference paper at ICLR 2025
(a) Hallucination Detection
h/d
2
h
D3T
Input
Sultuas
?
Hallucinated
Q:
Lincar Ouiput space
On what date in
Prediction characteristics
Non-ballucinated
1969 Ncil
B
MLP
Armstrong first foot on the Moon?
Internal space
V
Input tokens
Hidden states
Hl
Model Calibration
Calibration
Output
After
Betore
Frozen
20 July.
Production
Logit distribution from LLM
Figure I: Schematic of the proposed HADEMIF framework. We utilize two efficient networks D3T and an MLP-to capture hallucinations within the output and internal spaces of LLMs, respec- tively. The predictions of LLMs are subsequently calibrated based the outputs from these two hallucination detection networks, enhancing the reliability of the generated outputs.
learned internal representations (Chen et al. 2024a) and has introduced white-box methods for de- tecting predicting hallucinations based on these latent states (Burns et al. Zhu et al. 2024). We argue that both the output and internal spaces of LLMs signal the presence of halluinnations, highlighting the necessity for comprehensive approach to capture hallucinations across latent states and output transitions throughout the LLM generation process.
2023 Azadi et al.) [023
Hallucination Mitigation and Model Calibration Hallucination mitigation strategies can be broadly categorized based on the two primary sources: data-related methods and modeling and inference techniques (Ji et al. 2023a Xin et al. (024c). Data-related methods aim to refine and augment datasets ensure the use of more reliable data during training or fine-tuning (Penedo iet al. 2023 Zhou et al., 2024b Chen et al. 2024b). contrast, modeling and inference techniques commonly applied in practical scenarios, they directly influence the generation process and are not confined to specific tasks or datasets (Touvron et al. 2023a Ji et al.. 2023b Chuang et al. 2024 Xin al., 2024a). Within the latter category, model calibration, aiming to align model confidence with the actual probability of output correctness, has proven to be effective for mitigat- ing hallucinations LLMs (Liu et al. 20224] Zhu al. 2023), which can generally be divided into post-prov post-processing methods (Niculescu- Mizil & Caruana 2005 Guo et al.) (017)) and training based methods (Pereyra et al. [2077] Xin et al. | 20244] Kapoor et al., (024). Our method extends this line of research by leveraging hallucination biases captured in the outputs and internal space to calibrate the prediction, thereby mitigating hallucinations and enhancing model calibration.
METHODOLOGY
This section provides comprehensive overview of the proposed HADEMIF framework for halluci- nation detection and model calibration, which is applicable during both the inference and fine-tuning stages of LLMs. Additionally, we describe an online optimization process that involves alternating updates between the LLM and the two hallucination detection networks.
3.1 HALLUCINATION DETECTION AND MITIGATION
The HADEMIF framework, as illustrated in Fig. consists of two primary stages: hallucination detection and model calibration. In the first stage, hallucinations captured within the output and internal spaces of LLMs using two compact neural networks] (i.e., D3T and MLP). The sec- ond stage addresses hallucinations through logit calibration, guided by the outputs from these two hallucination detection networks.
Details of the model complexity analysis presented in Appendix A.11
3

---

Published as a conference paper at ICLR 2025
3.1.1 HALLUCINATION DETECTION IN THE OUTPUT SPACE
Previous research hallucination detection has primarily focused on individual aspects such as un- certainty or consistency (Manakul el al. 2023 Liang et al. 2024), which has limited both detection accuracy and broader applicability. Moreover, identifying the useful metrics for effective detection remains a sigmificant challenge. To address these limitations, we propose extracting a comprehensive set of prediction characteristics from the output space of LLMs that effectively capture and reflect hallucinations. These characteristics are subsequently input into carefully designed deep decision tree, whose inherent interpretability offers valuable insights into the hallucination detection rules associated with prediction characteristics, as well as the relative importance of these characteristics.
Computational Process of the D3T Model. D3T is an adaptation of the Deep Neural De- cision Tree (DNDT) model (Yang et al. (0188). which is tree model implemented using a neu- ral network. This method introduces a soft binning function for feature splitting, which is achieved through a linear layer with Softmax as the activation function, and utilizes the Kro- necker product operation to determine the final node of the tree. However, DNDT employs fixed tree structure during training that resem- bles perfect N-ary tree, where N denotes the number of cut points for each feature. This rigid design often results in numerous redundant nodes, thereby reducing the model's interpretability. In contrast, D3T dynamically learns the optimal number of cut points for each feature throughout training. yielding more flexible structure that enhances both computational efficiency and inter- pretability. The calculation process for the D3T model is illustrated in Fig. 2
D3T
Kroneeker product
Seamin 108
Classifier
Prediction
Yoi No
Figure 2: Diagram illustrating the computational process of the D3T model.
Following the approach of DNDT, D3T replaces the traditional hard binning utilized in conventional decision trees with a soft binning function, v-()). This function is implemented as a single-layer neural network with a Softmax activation function S:
v (E.r) [(nnS.. + b,)
(1)
where & refers to the jth prediction characteristic at time step. The vector w, = 11. .2.......tt.. is defined and oj represents the number of cut points for characteristic E. In DNDT, the val- ues of ej are fixed and identical across all characteristics, resulting in static model structure. Conversely, D3T dynamically optimizes the number of cut points for each feature during train- ing, which is given by Gj [C' [[;;]], where vy is trainable parameter corresponding to the jith feature, represents the Sigmoid function, and C denotes the constant specifying the maxi- mum number of cut points. During backpropagation, the Straight-Through Estimator (Yin et al.] 2019) is employed to circumvent the ceiling operation, a method frequently utilized in the train- ing of activation -uantizze neural networks. Furthermore, the trainable vector b, defined as Bjeyy, where B3.1 through $J.e, are the
+1]+]
by = [0, -B,1, =B B,,,... : -8,,1 $3.2
ej cut points of E. constrained by the condition 3J < B3, < < BBjeyy The temperature factor is also incorporated, and as r + 0, () approximates a one-hot vector. For example, the characteristic E.j divided into three intervals by the cut points B3.1 and B3.2, then the one-hot vector (E = [0, 1, 0] signifies that 33,1 < <E < 8.3.2.
After binning each characteristic, the Kronecker product is applied to determine the final nodes of the tree:
2 ( @ vi (5.2)
& + (E.Q),
where Q denotes the number of characteristics. z E Il represents an approximated one-hot vector, indicating the index of the leaf node reached by the extracted characteristics. Subsequently, the vector z is fed into a classifier with weights w. E R x2 to determine whether the current prediction is hallucinated.
Extraction of Prediction Characteristics. series of prediction characteristics g are extracted from the output space of LLMs to capture the presence of hallucinations. First, we consider the
In applications, we set the value of r to 0.1, and the sensitivity analysis regarding this parameter is detailed in Appendix A.8
4

---

Published as a conference paper at ICLR 2025
commonly used hallucination detection metrics from previous studies, specifically uncertainty and consistency. Next, we incorporate three additional metrics that reflect prediction confidence and commonly employed in previous machine learning tasks. such as faimess evaluation and sample e
weighting (Zhang et al. [2200] Ross & Dollár 2017 Jin et al. 2024 Zhou et al. 2024c). Specifically, all considered metries outlined as follows:
Probability distribution reflects the confidence of LLMs in each candidate token. We consider three metrics derived from this distribution: the maximum, minimum, and average values within the probability vector.
Uncertainty is widely used metric for evaluating token-wiss hallucinations, which quantifies the degree of unpredictability in the predictions. It is calculated as e Σj=1 Pp log(p)), where pj denotes the predicted probability of the jth token, and V represents the vocabulary size.
Margin measures the model's ability to distinguish among different predictions, serving as an indicator of its confidence. We evaluate both the Top-1 and Top-K K (set to 10) margins, which are calculated as 7 = p(()) p(2) and 7 кパK-ī
2 Σ ")=++(('" (p()) p()),
respectively, where p(9) denotes the ith largest element the probability vector.
Consistency evaluates the coherence across multiple responses generated by LLMS for the same input. It is quantified using s = A(#-1) Em Ef cos(u', w²), where B represents the total number of responses (set 3), and u' denotes the logits vector sponding to the ith response.
Logits norm |u|] potentially reflects the model's fitting capacity to the input. A larger norm typically signifies that the deep features are more closely aligned with the classifier weights, thereby indicating a higher level of confidence in the prediction.
The aforementioned characteristics are extracted from the LLMs and input into the proposed D3T model to detect hallucinations within the output space.
3.1.2 INTERNAL SPACE HALLUCINATION DETECTION AND PREDICTION CALIBRATION
Considering that the internal states of LLMs can also signal hallucinations, the token hidden states are input into an MLP model to capture hallucinations within the deep semantic space. Unlike pre- vious methods that manually define functions to associate internal states with hallucinations (Chen) et al. 2024a Zhu et al., 2024), this approach leverages the universal approximation capability of deep neural networks (Lu & Lu 2020) to automatically learn the mapping between hidden states and hallucinations. The output of the MLP network, denoted Hi (h; sts) RR, is then uti- lized for prediction calibration, where h represents the hidden states of tokens and Ω, denotes the MLP parameters. Specifically, the logits of LLMS are calibrated using the outputs from the two hallucination detection networks, as outlined below:
exp(u (u, (3L) HP log( (H.
( x. Vel. Hi, H(P)
Σ exp(u (v) - 1/ log(H((,))
where u, denotes the logits vector at step 1. Furthermore, HP represents the probability that the D3T model classifies prediction as a hallucination, which controls the intensity of the calibration.
Notably, our method does require any additional hallucination annotations. It only requires training the two hallucination detection networks by minimizing the loss of LLMs on the original training set, with predictions computed as outlined in Eq. (3). This process aims maximize the token probabilities for correct generations while reducing the likelihood of incorrect ones. Subse- quently, the trained D3T and MI.P models can directly employed for hallucination detection and prediction calibration during the inference phase of LLMs.
Sinne logits the unnormalized scores produced by the final linear layer and directly influence the model's outputs, we regard them as a characteristic the output space.
"We utilize the hidden states preceding the logits, and comparative analysis of the internal states across different layers is provided Appendix A.8
5

---

Published as a conference paper at ICLR 2025
3.2 OPTIMIZATION PROCEDURE FOR FINE-TUNING LLMS
To incorporate our framework into the fine-tuning phase of LLMs, we propose an optimization process that alternately updates both the LLM and the two hallucination detection networks. Let the LLM be parameterized by e. The optimization process proceeds as follows: first, the LLM parameters O are updated using stochastic gradient descent on a mini-batch of training samples (z,, yi)==1>>
1=1 according to the following objective function:
(++1) (t))
Σ log
yuu Dis Kt H ((, (2))(
711
where 7/1 denotes the step size used for updating the LLM parameters.
Subsequently, utilizing the optimized ((++1), the parameters of the two hallucination detection soworks-denokee as So for D3T and Lss for MLP- can be updated as follows:
MLP-ccn
2 <- (2(00 1/2
log P (vul a. S1.11 Hi HC, 0 Σ
(5)
((1
(2(10)
En 172 n2
Vn.
Σ log f yuu æss Mi.ch (D Hi, )
(6)
where 7/2 represents the step size for updating the parameters of the two hallucination detection networks. To facilitate efficient fine-tuning of the LLMs, we utilize LoRA (Hu et al.) (022), which enables the fine-tuning process to be conducted on a single GPU.
EXPERIMENTAL INVESTIGATION
4.1 DATASETS
Following Liu et al. 2024). our work evaluates text generation tasks with responses of varying lengths. Specifically, we utilize the CAT benchmark et responses at the phrase, sentence, and paragraph levels. The phrase-level generation datasets include NaturalQuestions (NQ), SciQ, and TriviaQA, each of which features short responses, such as named entities. For sentence-level responses, we consider TruthfulQA and WikiQA, where the model outputs full sentences. For paragraph-level tasks, we incorporate BioGen and WikiGen (Liu al. 2024). In the BioGen task, LLMs prompted to write biographies of various figures (Min et al, 2023), J, with ground-trute answers extracted from corresponding Wikipedia passages. In the WikiGen task, LLMs generate Wikipedia-style descriptions of entities, based on the fact verification dataset FEVER (Thorne et 2018). Comprehensive statistics, detailed descriptions of the training and test set construction, as well illustrative examples for all datasets are presented in Appendix A.I
et al. 2024), which encompasses tasks with
4.2 EVALUATION METRICS
To ensure fair comparison, the methodology for evaluating the model's confidence in its generated outputs, along with the accuracy of these outputs, follows the approach established by Liu et al. (2244) Specifically, for phrase- and sentence- tasks, the model's confidence py (x) is calculated as the geometric mean of the sequence of token probabilities:
Py(x))
1. I (Br a. Ne)).
Additionally, GPT-4 Achiam et al. (2023) is employed to evaluate the correctness model out- puts by determining the semantic equivalence between the generated text and the reference. For paragraph-level tasks, the assessment of accuracy and confidence involves four steps: claim extrac- tion, span mapping. confidence estimation, and correctness estimation (Liu et al. (002).. We then utilize three metrics to evaluate the effectiveness of our approach in hallucination detection and model calibration.

---

Published as a conference paper at ICLR 2025
HADEMIP w/ Fine- Tuning
Task
Metrie
Original
Label Smoothing
Temp. Scaling
LarCaB
LITCAB w/ Temp. Scaling
Calbraatin- Tuning:
HADEMIF
Phrase Level
0.288
0.208
0.288
0.300
0.300
0.310
0.315
0.355
NQ
cor 6
ECR
0.115
0.171
0.061
0.115
0.105
0.105
0,083
0.051
0.115
0.094
0.115
0,026
0.120
0.196
0.164
0.119
GC
0.764
0.212
0.764
0.762
0.762
0.761
0.760
0.766
SeQ
cov18990
ECB
0.211
0.051
0.391
0.003
0.091
0.211
0.221
0.081
0,062
0.221
0.081
0.224
0,083
0.230
0.076
0.228
0.200
ace 50
0,500
0.302
0.500
0.478
0.478
0.482
0.480
0,501
TrivaQA
Don (860
ECB
0.111
0.112
0.184
0.019
0.079
0.101
0.081
0.201
0,079
0.201
0,080
0,222
0,080
0.231
1,075
0,240
Brier
0,203
0.199
0.185
Sontence Level
0.314
0.181
0.314
0.314
0,314
0.786
0.415
0.430
TruthfulQA
cow (1-0
0.136
0.000
0.136
0.195
0.195
0.793
0.580
0,510
ECE
0.138
0.105
0,095
0.087
0,058
0.176
acc 50
0.388
0.273
0.388
0.391
0.397
0.441
0.629
0.653
WIkIQA
covi2:50
0.012
0.000
0.012
0.062
01062
0.162
0.330
0,338
ECE
0.075
0,155
0.066
0.075
0,074
0,070
0.068
0,055
0.211
accor50
0.451
0.451
0,450
0.450
0.476
0.520
0.541
Averuge
ECE
0,118
0.210
0.112
0.089
0.084
0.075
0.070
0,058
Brier
0.206
0.254
0,210
0.199
0.196
0.189
0.182
0.178
Table 1: Comparison between HADEMIF and model calibration methods on the CAT benchmark for phrase- and sentence-level responses. For each metric and dataset, the top scores are highlighted in bold, and the second-best scores are underlined. Scores where HADEMIF surpasses both the original LLM and all baselines are highlighted in blue while those outperforming only the original I.IM are marked green The final rows summarize the average values across all tasks. Our proposed HADEMIF approach consistently enhances model performance.
In line with prior research (Guo et al. 2017 Tian at. 2023 Liu et al. 2024),
employ the Expected Calibration Error (ECE) to measure the discrepancy between a model's confidence and its actual accuracy. Specifically, model predictions are grouped according to confidence levels, and we compute the accuracy acc(b:) and the average confidence
[b.] fb;)(b;) within each bin b, The ECE is then calculated ECF Σ; Mf' lacc(b;)
conf(b:). where M represents the total number of model outputs. lower ECE signifies better calibration, indicating closer alignment between the model's confidence and its actual accuracy.
The Brier Score (Brier 1950p is metric commonly used to evaluate tasks that require assigning probabilities set of mutually exclusive discrete outcomes or classes, which can be either binary or categorical. Following Liu et al. (2024), we compute the Brier Score as the mean squared difference between the model confidence Py and the binary correctness (yy of its predictions Brier = 1 E,,P, - [zy)]² This metric offers a direct assessment the quality of model calibration.
Given the importance model confidence, we also assess model performance using two selective classification metrics as detailed in (Liu et al. 2024). The first metric, accuracy at coverage (ace @ @q), evaluates the precision of the model by examining the accuracy of the top-q percent of predictions. The second metric, coverage at accuracy (cov (cov@p), measures recall by identifying the largest proportion of the most confident predictions where accu- racy surpasses a designated threshold p. Unlike AUROC (Bradley 1997). which primarily assesses the quality of confidence scores, these metrics provide direct evaluation of the model's capability filter out incorrect predictions by applying specific thresholds.
4.3 COMPARED BASELINES
We compare the HADEMIF framework with four traditional and advanced model calibration meth- ods. Temperature Scaling (Liang et al. 2018)) adjusts the logits by a temperature parameter before applying the Softmax function. Label Smoothing (Szegedy et al.) 2016 involves fine-tuning the LLMs using LoRA with label smoothing techniques applied during training. Lightweight Calibra- tion (LITCABB (Liu al. 2024) employs single linear layer process the input text representa-

---

Published as a conference paper at ICLR 2025
Model Calibration
HADEMIF
Task
Metric
Original I.I.M
Label Smoothing
Temp- Scaling
LITCAB w/ Temp. Scaling
HADRMIF
Fine-Tuning
LITCAB
acc GP 50
0.347
0.334
0.347
0.354
0.354
0.362
0.358
BioGen
cov
0.059
0.148
0.160
ECE Brier
0.169
0.196
0.246
0.166
0.243
0.164
0,160
0.269
0.284
0.313
0.267
0,308
0,268
0.255
ace 30
0.876
0.860
U.876
0.872
0.872
0.875
0,884
WikiGen
cov 80
0.745
0.733
0.745
0.756
0.756
0.760
0.774
ECB
0,045
0.075
0.049
0.037
0,065
0.040
0,032
Brier
0.172
0.187
0.173
0.171
0.174
0.167
0.161
0.612
0.597
0.612
0.613
0.613
0.619
0.621
Average
ECE
0.107
0.136
0.148
O. 0.102
0.154
0.102
0.096
Brier
0.221
0.236
0.243
0.219
0,241
0.218
0.208
Table 2: Comparison between HADEMIF and model calibration methods the CAT benchmark for paragraph-lever responses. Our approach consistently enhances the model performance of LLMs in paragraph generation tasks.
tion and predict a bias term, which then added to the output logits the LLMs. Calibration- Tuning (Kapoor al., 2024) fine-tunes LLMs by designing task that enables the model to tonomously evaluate whether its generated responses are consistent with the true answers.
Additionally, compare HADEMIF with recent methods specifically designed for hallucination detection and mitigation in LLMs. Verbalization involves prompting the LLM to self-report its confidence level for given output, using the prompt provided by Tian et al. 2023). P(IK) (Ka- davath et al. 2022) introduces a linear layer on top of the LLM's final hidden state corresponding the last token of question, training this layer to predict the model's likelihood of accurately answering the question. Self-Consistency (Tian et al. [0233 Xiong et al. 20224) operates on the
principle that confident responses are more likely recur when sampling from the model. Refusal- Aware Instruction Tuning (R (R-Tuning) (Zhang et al., (024a) fine-tunes LLMs on refusal-aware datasets to equip the models with the capability to generate refusal-aware responses, thereby de- creasing hallucinations. It is important to note that the three hallucination detection methods and Calibration- Tuning produce only a single aggregated score for the entire generated output, which prevents them from generating scores the individual claim level. This limitation renders them un- suitable for paragraph-Ievel tasks, where the generated content typically comprises multiple claims. Consequently, following Liu et al. (2024). exclude these methods from paragraph-level tasks.
4.4 EXPERIMENTAL SETTINGS
Our experimental setups follow those outlined Liu al. (2024). Specifically, we select Llama2- 7B (Touvron et al. 20236) as the primary backbone model, given its strong performance across wide range of benchmark datasets. Additionally, we include seven other popular LLMs, ranging in size from 1.5B to 30B. Due to space limitations, we present the results for Llama2-7B in the main text, with results for the other models available in the Appendix. The training process begins with an initial learning rate of x 10-3 for both the MLP and D3T networks, which is reduced by factor of 0.1 at the 20th and 40th epochs. Training is conducted in 50 epochs with early stopping. For fine-tuning the LLMs, the two hallucination detection networks are first trained for 40 epochs, after which an alternating optimization process is applied between the LLMs and the two detection networks. The LLMs are fine-tuned for 5 epochs using AF with rank of 8 and learning rate of 3 x 10-4, More detailed experimental settings are presented in the Appendix.
4.5
MAIN RESULTS
Comparison with Model Calibration Methods The results of HADEMIF, along with those of four traditional and advanced model calibration methods, are presented in Tables [ and 2 where some results are from the LITCAB (Liu et al.) 2024) paper. HADEMIF consistently enhances the performance of the original LLM across tasks, demonstrating its effectiveness in model cal- ibration. Specifically, HADEMIF reduces the ECE and Brier scores by 51 % (from 0.118 to 0.058) and 14% (from 0.206 to 0. 178), respectively, for phrase- and sentence-level tasks compared to the
https:///
8

---

Published as a conference paper at ICLR 2025
Hallucination Detection and Mitigation
HADEMIF
Task
Metric
Original I.I.M
Self-
HADEMIF
w/ Fine-Tuning
P(K)
Verbalization
R-Tuning
Consistency
Phruse Level
acc
0.286
0.254
0.293
0.315
60
0.115
0.000
0,055
0.217
0.084
0.115
0,120
NQ
ECE
0.171
0.158
0.516
0.156
0.034
Brier
0.196
0.204
0.468
0.163
0.201
0.116
0.119
acc
0.764
0.656
0.660
0.744
0.692
0.760
0.766
SeiQ
(290
0.211
0.004
0.117
0.124
0.119
0.230
0.228
ECE
0.094
0. 0.188
0,318
0.101
0.190
0.083
0.076
Brier
0.203
0.276
0,344
0.227
0.285
0.201
0.200
acc 50
0.500
0.372
0.404
0.446
0.400
0.480
0.501
IrivaQQ
ECE
0.112
0.215
0.431
0.181
0.184
0. 0.080
0.075
Sentence Level
Scc@:50
0,314
0.267
0.233
0.405
0,341
0.415
0.430
TrathfulQA
6140
0.136
0.005
0,224
0,500
0.332
0. 0.500
0.510
ECE
0.138
0.323
0.510
0,060
0.148
0.087
0.058
Brier
0,218
0.349
0.474
0.194
0,190
0.193
0.176
acc@500
0.388
0.339
0.372
0.628
0.416
0.629
0.653
WikiQA
cov@50
0.012
0.004
0.202
0.621
0,258
0.330
0.338
ECE
0.075
0.239
0.535
0.136
0.139
0. 0.068
0.055
0.225
0.211
0.208
acc0150
0.451
0.384
0,385
0.513
0.428
0.520
0.541
Average
0.225
0.070
Brier
0,206
0.281
0.443
0,216
0,230
0.182
0.178
Table 3: Comparison of HADEMIF with hallucination detection and mitigation methods on the CAT benchmark for phrase- and sentence-level tasks. The results demonstrate that HADEMIF con- sistently outperforms other baselines across a wide range of tasks.
original LLM. Moreover, it decreases the ECE and Brier scores by 10% (from 0.107 to 0.096) and 6% (from 0.221 0.208), respectively,, for paragraph-Ieve tasks. Additionally, HADEMIF outper- forms all four calibration methods, achieving the lowest average ECE and Brier scores, along with the highest average 5cc@50. Among the model calibration approaches, Label Smoothing performs poorly across nearly all tasks, even falling short of the original LLM. While Temperature Scaling and LITCAB demonstrate improved performance compared to Label Smoothing, their re- sults remain far from optimal. Furthermore, since Calibration- Tuning cannot achieve fine-grained calibration the prediction distribution, its performance inferior to that of our approach.
Comparison with Hallucination Detection and Mitigation Methods As shown in Table 3 HADEMIF consistently outperforms other hallucination detection and mitigation methods across both phrase- and sentence-level tasks. It achieves the highest average acc @ 50 and the lowest average ECE and Brier scores, underscoring its effectiveness in estimating model confi- dence. Notably, compared to the best basclines, HADEMIF improves acc @ 50 by 5% (from 0.513 to 0.541) and reduces ECE and Brier scores by 54% (from 0.125 to 0.058) and 18% (from 0.216 to 0.),,), respectively. Additionally, Verbalization and P(IK) perform poorly among the baselines, suggesting that although LLMs contain knowledge capable of revealing their hallucinations, effec- tive approaches for knowledge modeling and extraction necessary. Moreover, although R-Tuning fine-tuned LLMs, the confidence scores it generated are binary, making them less accurate compared methods that use quantitative values represent confidence, thus limiting their effectiveness. Ad- ditionally, Self-Consistence
emerges as the best- rest-performing baseline, aligned with our subsequent observation in Sec. 4.6 that consistency plays a more crucial role in hallucination detection com- pared to others. These findings position HADEMIF as a reliable method for halluciaation detection and mitigation in LLMs as it comprehensively captures hallucinations in both the output and internal spaces, leveraging the modeled hallucinations for effective logit calibration.
Results of Fine-Tuning LLMs Fine-tuning LLMs with Label Smoothing does not yield satis- factory calibration results, suggesting that it is not well-suited for complex tasks such as LLM fine- tuning. Moreover, although other fine-tuning approaches, such as Calibration- CulibrationTTuiing and R- R-Tuning, demonstrate improved performance, their performance still falls short of ours, they cannot achieve fine-grained calibration of the prediction distribution. Specifically, compared to the best fine-tuning baseline, HADEMIF increases acc 50 from 0.476 to 0.541 and reduces the ECE and Brier scores

---

Published as a conference paper at ICLR 2025
0.43
0:24
0.35
0:30
2:45
0.25
0.40
0.20
033
0.26
0.30
0:31
0.25
0.82
Driei
0.33
0.0K
0.29
Omerssion
6.50
volue
0.25
0.35
0.30
2.42
0.17
04K
4K
0.2
4
0.31
0.20
2.10
0.21
0.20
500
conducencyy
Sightfigans Logils
0.2K
0
BCE
Aay
Maximum number of cut points
Figure 3: (a) Significance of prediction characteristics in hallucination detection measured by in- formation gain throughout the training process. (b) and (c) Sensitivity analysis of the number and dimension of hidden layers in the MLP network. (d) Sensitivity analysis of the maximum number of cut points for characteristics in the D3T network.
from 0.075 to 0.058 and from 0.189 to 0.178, respectively, for both phrase- and sentence-level tasks. Additionally, it increases acc @ 50 from 0.597 to 0.621 and decreases ECE and Brier scores by 29% (from 0. 0.136 to 0.096) and 12% (from 0.236 to 0.208), respectively, for paragraph-level tasks.
4.6 SIGNIFICANCE OF PREDICTION CHARACTERISTICS
Previous studies have employed various metrics, such as entropy and consistency, for ballucination detection (Xiao & Wang 2021 Yang et al. 2023 Zhou et al. 2023). However, no consensus has been established regarding the most critical one for this task. The inherent interpretability of the D3T model facilitates a comprehensive analysis of the relative importance of different prediction characteristics in hallucination detection. Due to the feature splitting process in D3T, we can apply information gain to evaluate the significance each metric. For the margin characteristic, we cal- culate the average information gain across both Top-1 and Top-K K margins. Fig. B)a illustrates the significance of various characteristics during the training process, leading to three key observations: (1) Consistency plays more pivotal role than uncertainty in hallucination detection; (2) In addi- tion to consistency and uncertainty, metrics such as margin. probability distribution, and logits norm have proven to be effective in detecting hallucinations in LLMs; and (3) systematic approach that integrates multiple indicators is more effective than reliance on a single metric.
4.7 ABLATION STUDIES
We conduct ablation studies evaluate the impact of the complexity of two hallucination detection networks on the effectiveness of our approach. This complexity is influenced by the number and dimensions of hidden layers in the MLP network, as well as the maximum number of cut points for the characteristics in the D3T model. We report the average performance across phrase- and sentence-level tasks. As illustrated three hidden layers and layer dimensions ranging from 0.4K to 0.6K in the MIP network. Accord- ingly, we recommend employing an MI.P network with two hidden layers, each with a dimension of 512. Furthermore, the model performs well when the maximum number of cut points in the D3T model is set to (3, 4, 5}. with peak performance observed at four cut points.
Figs. b)b). and (c), performance remains stable with two or
CONCLUSION
This study introduces HADEMIF, an innovative framework designed for detecting and mitigating hallucinations in LLMs. The framework utilizes two compact neural networks to identify hallucina- tions within both the output and internal spaces of L.Mss. By calibrating model predictions based on the outputs of these networks, HADEMIF effectively reduces hallucinations and enhances the relia- bility of generated outputs. Our framework operates during both the inference and fine-tuning phases of LLMs by training two small-scale networks, which requires less than 2% additional parameters compared to the original LLMs. We evaluate the performance of the HADEMIF framework using the CAT benchmark, considering tasks with varying response lengths. The results demonstrate that HADEMIF significantly outperforms existing baselines in both hallucination detection and model calibration, highlighting its superior efficacy.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877/ 2023.
Fatemeh Azadi, Heshaam Faili, and Mohammad Javad Dousti. Pmi-align: Word alignment with point-wise mutual information without requiring parallel training data. In Findings of the Associ- ation Computational Linguisties: ACL. pp. 12366 1067-1277,2220 2023.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP, pp. 967-976, 2023.
Andrew P Bradley. The use of the under the curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1 145-1159, 1997.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2023.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Lims' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024a.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024b.
Wei Chen, Shigui Li, Jiacheng Li, Junmei Yang, John Paisley, and Delu Zeng. Dequantified diffusion schrödinger bridge for density ratio estimation. arXiv preprint arXiv:2505.05034 2025.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chat- bot impressing gpt-4 with 90%* * chatgpt quality. 2023. URL https::/ttsssttt org/blog/ 1023-03-300 30-0003----vaaaaaaa
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024.
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm vs Im: Detecting factual errors via cross examination. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 12621-12640 2023.
Shian Du, Yihong Luo, Wei Chen, Jian Xu, and Delu Zeng. To-flow: Efficient continuous normaliz- ing flows with temporal optimization adjoint with moving speed. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12570- 2570-12580. 2022.
Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the uncertainty estimation of large lan- guage models. arXiv preprint arXiv: arXiv:2307.01339, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of International Conference on Artificial Intelligence and Statistics, pp. 249-256. 2010.
11

---

Published as a conference paper at ICLR 2025
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, PP. 1321-1330, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026-1034, 2015.
Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhangg Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive models for fine- fine-grained text-to- image synthesis. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17123-17132 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, 2022.
Lei Huang. Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang. Qianglong Chen, Weihua Peng, Xiaocheng Feng. Bing Qin, et. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.052.2. 2023.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023a.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigat- ing hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06271, 2023b.
Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. In Proceedings of the ACM on Web Conference 2024, pp. 2627-2633 2024.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dode Dodds, Nova DasSarma, Eli Tran- Johnson,,,,, et al. Language mod- els (mostly) know what they know, arXiv preprint wxXv:2207.052222 2022.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. calibration-tuning Teaching large language models to know what they don't t know. In Proceedings of the 1st Workshop on Uncertainty-Awe NLP (UncertaiNLP 2024), pp- 1-14, 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image de- tection neural networks. In The Sixth International Conference on Learning Representattonss 2018.
Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang. Learning to trust your feelings: Leveraging self-awareness 1lms for hallucination mitigation. arXiv preprint 45544.2401.1544, 2024.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models. Transactions Machine Learning Research, 2024.
Zi Lin. Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural-symbolic graph semantic parsing via uncertainty. Findings of the Association for Computational Linguistics: ACL 2022, 2022.
12

---

Published as a conference paper at ICLR 2025
Mushui Liu, Yuhang Ma, Zhen Yang, Jun Dan, Yunlong Yu, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. Llm4gen: Leveraging semantic representation Ilms for text-to-image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 5523-5531, 2025.
Xin Liu, Muhammad Khalifa, and Lu Wang. Litcab: Lightweight language model calibration over short-and long-form responses. In The Twelfth International Conference on Learning Represen- tations, 2024.
Yulong Lu and Jianfeng Lu. A universal approximation theorem of deep neural networks for express- ing probability distributions. Advances in Neural Information Processing Systems, 33:3094-3105, 2020.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In The Eighth International Conference on Learning Representations, 2020.
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallu- cination detection for generative large language models. In Proceedings of the Conference on Empirical Methods Natural Language Processing, pp. 9004-9017, 2023.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision long form text generation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 22076-1210,, 2023.
Alexandru Niculescu-Mizz and Rich Caruana. Predicting good probabilities with supervised learn- ing. In Proceedings of the 22nd International Conference on Machine Learning, pp. 625 632, 2005.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refined- web dataset for falcon Ilm: Outperforming curated corpora with web data only, Advances in Neurul Information Processing Systems, 36:79155-7911 2023.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 1.0iv:1701..6655 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learner.. OpenAl blog, 2019. URL.
https::/dfmucfpkeysee wtt.::pkwwww.....oooooooototooo. het.tttteeeebteteeeeeeeee language- Language mode-s..deeel.eeeooo.ee...
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Salch, Balaji Lakshminarayanan, and Peter Liu. distriititi detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2022.
T-YLPG Ross and GKHP Dollár. Focal loss for dense object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9980-2988, 2017.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? Advances in Neural Information Processing Systems, 36:55565-55588 2024.
Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Un- supervised real-time hallucination detection based on the internal states of large language models. arXiv preprint arXiv:2403.066. 2024.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2822 2016.
Arun James Thirmaavukrassu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature Medicine, 1930-1993 2023.
13

---

Published as a conference paper at ICLR 2025
James Thome, Andreas Vlachos, Christos Christoddoulpoollss, and Arpit Mittal. Fever: large- scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pp. 809-819, 2018.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 5433-5442, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini wrxXv:2302.1397/1 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykow, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and tnnee chat models. arXiv preprint arXiv:22307.08 00.88888, 2023b.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: billion parameter autoregressive language model, 2021. URL https:
wetthh-ttassoooooee jax
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc v Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Yi Wang, Mushui Liu, Wanggui He, Longxiang Zhang, Ziwei Huang, Guanghao Zhang, Fangxun Shu, Zhong Tao, Dong She, Zhelun Yu, et al. Mint: Multi-moda chain of thought in unified generative models for enhanced image generation. arXiv preprint vrXiv:2503.01280 2025.
Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in conditional language generation. In Proceedings of the Sixteenth Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2734-2744, 2021.
Yi Xin, Junlong Du, Qiang Wang, Zhiwen Lin, and Ke Yan. Vmt-adapter: Parameter-efficient trans- fer learning for multi-task dense scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. (6085-16093, 2024a.
Yi Xin, Juniong Du, Qiang Wang, Ke Yan, and Shouhong Ding. Mmap: Multi-modal alignment prompt for cross-domain multi-task learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16076-16084, 2024b.
Yi Xin, Siqi Luo, Xuyang Liu, Haodi Zhou, Xinyu Cheng, Christina E Lee, Junlong Du, Haozhe Wang, MingCai Chen, Ting Liu, al. V-petl bench: A unified visual parameter-effiiient transfer learning benchmark. Proceedings of Advances in Neural Information Processing Systems, 37: 80522-80535, 2024c.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in 1lms. In The Twelfth International Conference on Learning Representations, 2024.
Shiping Yang, Renliang Sun, and Xiaojun Wan. A new benchmark and reverse validation method for passage-level hallucination detection. In Findings of the Association for Computational Lin- guistics: EMNLP, pp. 3898-3908 2023.
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. arXiv preprint arXiv: /806.068 2018
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight- estimator training activation quantized neural nets. In The Seventh International Conference Learning Representations, 2019.
Zhangyue Yin, Qiushi Sun, Qipeng Guo. Jiawen Wu, Xipeng Qiu, and Xuan-Jing Huang. Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACI., pp. 8653-8665 2023.
14

---

Published as a conference paper at ICLR 2025
Hanning Zhang. Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say 'i don't t know'. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers); pp. 7 7106-7132, 2024a.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry- aware mnstance-reweighen adversarial training. International Conference on Learn- ing Representations, 2020.
Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large lan- guage models truthful space, arXiv preprint arXiv:2402. 17811, 2024b.
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. Enhancing uncertainty- based hallucination detection with stronger focus. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 915-932, 2023.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori B Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. In Proceedings of the Conference on Empirical Methods Natural Language Processing, pp. 5506-5524, 2023.
Lexin Zhou, Wout Schellaert, Fernando Martinez-Plumed, Yacl Moross-aavll Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable language models become less reliable. Nature, pp. 1-8, 2024a.
Xiaoling Zhou, Ou Wu, Weiyao Zhu, and Ziyang Liang. Understanding difficulty-based sample weighting with universal difficulty measure. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pp. 68 84, 2022.
Xiaoling Zhou, Ou Wu, Michael K Ng. and Hao Jiang. Data valuation by leveraging global and local statistical information. arXiv preprint 2rXiv22405.14464 2024b.
Xiaoling Zhou, Ou Wu, and Nan Yang. Adversarial training with anti-adversaries. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 16(12):00100-1022 2024c.
Xiaoling Zhou, Wei Ye, Yidong Wang, Chaoya Jiang, Zhemg Lee, Rui Xie, and Shikun Zhang. Enhancing in-context learning via implicit demonstration augmentation. In Proceedings of the Aunual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 2810-2828, 2024d.
Chiwei Zhu, Benfeng Xu, Quan Wang. Yongdong Zhang, and Zhendong Mao. On the calibra- tion of large language models and aligument. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9778-97959 2023.
Derui Zhu, Dingfan Chen, Qing Li, Zongxiong Chen, Lei Ma. Jens Grossklags, and Mario Fritz. Pollmgraph: Unraveling hallucinations in large language models via state transition dynamics. In Findings of the Association for Computational Linguistics: NAACL, pp. 1737-4757 2024.
15

---

Published as a conference paper at ICLR 2025
NQ
SciQ
TriviaQA
TruthfulQA
WikiQA
BioGen
WikiGen
# Train
2K
2K
2K
397
1040
500
500
# Test
1K
1K
IK
420
293
183
100
Avg. Response tokens
2.2
1.5
2.3
35.5
25.7
77.3
104.0
Max. # Response tokens
6
12
42
24
166
318
197
Min. Response tokens
33
Table 4: Task statistics in the CAT benchmark
LLiu et al. 2024).
Task
Question-Response Pair
Phrase Level
Question: When did big air snowboarding become olympic sport?
NQ
Response: 2018
Question: In order create food, what do photosynthetic protists use?
SciQ
Response: Light energy
Question: What was Kevin Kline's first movic?
TriviaQA
Response: Sophie's Choice
Sentence Level
Question: Which language do all people in Europe speak?
TruthfulQA
Response: There's single language that all Europeans speak
Question: How much is tablespoon of water?
WikiQA
Response: This tablespoon has capacity of about 15 mL.
Paragraph Level
Question: Write paragraph for Bill Tobin's 's biography.
Response: Ron Meagher (born October 2, 1941, Oakland, California, USA) is best known the bassist of the American rock band The Beau Brum- mels. When guitarist-songwriinn
BioGen
Ron Elliott was putting the...
Question: Write paragraph about The Beatles.
Response: The Beatles were an English rock band formed in Liverpool in 1960, comprising John Lennon, Paul McCartney, George Harrison, and Ringo Starr. They are regarded as the most influential band of all time..
WikiGen
Table 5: Illustration of question-response pairs from tasks in the CAT benchmark.
A APPENDIX
A.1 MORE DETAILS OF THE CAT BENCHMARK
Table summarizes the statistics of tasks from the CAT benchmark, while Table provides ex- amples of inrestion-answeee pairs in these tasks. We employ seven tasks from the CAT benchmark, following Liu et al. (024)). which include phrase-level tasks: NO Sei and TriviaQA sentence- level tasks: TruthrulQA" and WikiQA and paragraph-level tasks: BioGer and WikiGen. For the three phrase-level tasks, IK samples are used for testing and 2K samples for training. For Truth- fulQA, which lacks official training set, 397 instances are randomly sampled from the original test set for training and the remaining instances are utilized for testing. For the WikiQA dataset, the training set consists of 1,040 instances, while the test set contains 293 instances. For BioGen, total of 683 names are compiled from (Min et al., 2023), of which 183 names are designated for evalu- ation and the remaining 500 utilized for training. Similarly, for the WikiGen task, 600 entities
16

---

Published as a conference paper at ICLR 2025
Task
Metric
GPT-2 XL (1.5B)
GPT-J (6B)
Vicuma-131
Phrase Level
acc 50
0.062
0.130
0.146
0.227
0.246
0.323
cov 50
0,001
0.076
0.057
0.100
0.113
0.124
NQ
ECE
0.045
0.033
0.059
0.024
0.204
0.045
Brier
0.055
0.016
0.079
0.043
0.224
0.135
acc@50
0.258
0.274
0.620
0.665
0.678
0.723
cov@90
0,007
0.019
0.135
0.157
0.142
0.166
SciQ
ECE
0.059
0.026
0.133
0.087
0.244
0.187
Brier
0.137
0.126
0.209
0.166
0.318
0.204
acc@ 50
0.100
0.187
0.270
0.294
0.464
0.463
cov@60
0.000
0.135
0.128
0.245
0.268
0.372
TriviaQA
ECE
0.063
0.028
0.068
0.031
0.186
0.121
Brier
0.069
0.066
0.115
0.110
0.238
0.230
acc@50)
0.140
0.197
0.345
0.395
0,463
0.503
ECE
0.056
0.029
0.087
0.047
0.211
0.118
Average
Brier
0.087
0.069
0.134
O. 106
0.260
0.190
Sentence Level
acc 50
0.186
0.290
0.162
0.273
0.552
0.595
COv 40
0.005
0.198
0.040
0.230
0.998
0.998
TruthfulQA
ECE
0.041
0.036
0.112
0.044
0.200
0.088
Brier
0.118
0.071
0.136
0.090
0.303
0.205
acc@#50
0.099
0.201
0.240
0.447
0.421
0.506
cov@55 50
0.000
0.122
0.000
0.189
0.053
0.211
WikiQA
ECE
0.063
0.040
0.045
0.021
0.211
0.134
Brier
0.125
0.103
0.149
0.135
0.304
0.285
acc@50
0.143
0.246
0.201
0.360
0.487
0.551
ECE
0.052
0.038
0.079
0.033
0.206
0.111
Average
Brier
0.122
0.087
0.143
0.113
0.304
0.245
Paragraph Level
acc acc@500
0.228
0.267
0.380
0.392
cov 4:0
0.023
0.104
0.451
0.484
BioGen
ECE
0.159
0.142
0.229
0.211
Brier
0.182
0.170
0.255
0.231
acc@50
0.395
0.403
0.822
0.835
cov@80 80
0.001
0.022
0.675
0.712
WikiGen
ECE
0.102
0.091
0.168
0.132
Brier
0.220
0.205
0.227
0.211
acc@ 50
0.312
0.335
0.601
0.614
ECE
0.131
0.117
0.199
0.172
Average
Brier
0.201
0.188
0.241
0.221
Comparison of the HADEMIF approach (with fine-tuning) and baseline performance across GPT-2 XL (1.5B), GPT-J (6B), and Vicuna-13B on the CAT benchmark. Results for GPT-2 XL in paragraph-level tasks are excluded as the prompt length exceeds its context window limit. Scores where our approach surpasses the original LLM are highlighted in green The proposed
Table 6:
HADEMIF approach consistently exhibits superior performance compared to baseline results across multiple LLMs.
are randomly selected from the FEVER 12 dataset, each linked to a specific Wikipedia passage. Of these, 100 entities were aside for evaluation, while the remaining 500 were utilized for training.
hhttps://github. fever
17

---

Published as a conference paper at ICLR 2025
Task
Metric
LLaMA-7B
LLaMA-30B
Llama2-7B
Llama2-133
Llama3-BB
Phrase Level
acc@50
0.358
0.389
0.466
0.512
0.288
0.355
0.448
0.491
0.510
0.542
NQ
cov@50
0.271
0.282
0.445
0.503
0.115
0.120
0.407
0.415
0.502
0.516
ECE
0.144
0.032
0.169
0.037
0.171
0.026
0.139
0.019
0.126
0.020
Brier
0.174
0.103
0.192
0.112
0.196
0.119
0.187
0.100
0.179
0.097
acc@
0.756
0.778
0.874
0.899
0.764
0.766
0.844
0.842
0.881
0.892
cov cov@90
0.261
0.278
0.423
0.435
0.211
0.228
0.375
0.405
0.435
0.459
SciQ
ECE
0.126
0.081
0.107
0.065
0.094
0.076
0.102
0.066
0.100
0.061
Brier
0.210
0,200
0.186
0.157
0.203
0,200
0.197
0.154
0.174
0.139
acc@50
0.474
0,493
U.462
0,469
0.500
0.501
0.454
0,486
0.582
0.620
cov@50
0.029
0.157
0.156
0.264
0.111
0.240
0.
0.270
0.336
0.369
TriviaQA
ECE
0.137
0,078
0.052
0.036
0.112
0.075
0.087
0.054
0.061
0,040
Brier
0.213
0.197
0.174
0.158
0.203
0.185
0.190
0.181
0.146
0.123
acc@50
0.529
0,553
0.601
0.627
0.517
0.541
0.582
0,606
0.658
0,685
ECR
0.136
0.064
0.1
0.046
0.126
0.059
0.109
0.046
0.096
0.040
Average
Brier
0.199
0.167
0.184
0.142
0.201
0.168
0.191
0.145
0.166
0.120
Sentence Level
acc@:50
0.012
0,145
0.433
0,542
0.314
0.430
0.362
0.441
0.571
0.590
cov@40
0.117
0.419
0.648
0.697
0.136
0.510
0.350
0.613
0.698
0.721
TruthfulQA
ECE
0.120
0.050
0.110
0.043
0.138
0.058
0.132
0.060
0.088
0.046
Brier
0.184
0.143
0.235
0.193
0.218
0.176
0.233
0.180
0.187
0.145
acc@
0.322
0.608
0.264
0.462
0.388
0.653
0.455
0.655
0.489
0.620
cov@50
0.086
0.341
0.078
0.218
0.012
0.338
0.358
0.434
0.430
0.458
WikiQA
ECE
0.108
0.066
0.142
0.103
0.075
0.055
0.064
0.037
0.077
0.052
Brier
0.190
0.176
0.182
0.173
0.212
0.208
0.192
0.175
0.168
0.154
acc@:50
0.167
0.377
0.349
0.502
0.351
0.542
0.409
0.548
0.530
0.605
ECE
0.114
0.058
0.126
0.073
0.107
0.057
0.098
0.049
0.083
0.049
Average
Brier
0.187
0.160
0.209
0.183
0.215
0.192
0.213
0.178
0.178
0.150
Paragraph Level
acc@50
0.220
0.250
0.313
0.331
0.347
0.358
0.485
0.500
0.513
0.532
cov@440
0.134
0.218
0.300
0.398
0.066
0.160
0.999
0.998
0.457
0.503
BioGen
ECE
0.143
0.131
0.105
0.080
0.169
0.160
0.1
0.068
0.097
0.061
Brier
0.153
0.139
0.173
0.158
0.269
0,255
0.260
0.244
0.199
0,161
acc@50
0.667
0.685
0.750
0.762
0.876
0.884
0.900
0.911
0.925
0,931
cov@
0.220
0,250
0.354
0,382
0.745
0.774
0.914
0.935
0,875
0.924
WikiGen
ECE
0.102
0.087
0.165
0.154
0.045
0.032
0.048
0.031
0.050
0.044
Brier
0,239
0,200
0.252
0,238
0.172
0.161
0.164
0.147
0.169
0.148
ac
0.444
0.468
0.532
0.547
0.612
0.621
0.693
0.706
0.719
0.732
ECE
0.123
0.109
0.135
0.117
0.107
0.096
0.081
0.050
0.074
0.053
Average
Brier
0.196
0.170
0.213
0.198
0.221
0.208
0.212
0.196
0.184
0.155
Table 7: Comparison of the HADEMIF approach (with fine-tuning) and baseline performance across LLaMA, Llama2, and Llama3 models on the CAT benchmark. The HADEMIF framework consistently achieves improvements over baseline performance across multiple LLMs.
A.2 MORE EXPERIMENTAL SETTINGS
Besides Llama2-7B, we assess the performance of our approach utilizing seven other LLMs: GPT-2 XL (1.5B) (Radford al. 2019), GPT-J (6B) (Wang & Komatsuzaki 2021), LLaMA-7B (Tou- vron et al.] 2023a), LLaMA-300 Llama2-13B, Vicuna-13B (Chiang et al. 2023), and Llama3- 8B (Dueyy et al. 2024). Moreover, to ensure consistency across tasks, we employ ICL, as not all LLMs demonstrate strong zero-shot performance. The settings follow those described in (Liu et al.) 2024]. Specifically, for phrase- and sentence-level tasks, we utilize fifteen demonstrations. For the two paragraph-level tasks, we employ five examples per task, accounting for the extended length of the demonstrations. The queries for BioGen and WikiGen are phrased as "Write a paragraph for [Name]'s biography" and "Write paragraph about [Entity]," respectively. Regarding the initializa- tion of the two hallucination detection networks, the MLP network is initialized using He initializa- tion He et al. (2015), as this method is effective for layers with ReLU activation function. For the D3T model, following the initialization approach in DNDT (Yang et al., 2018), all parameters are initialized using Xavier initialization Glorot & Bengio (010) with uniform distribution.
18

---

Published as a conference paper at ICLR 2025
Phrase Level
0.7
Baseline HADEMIF
0.25
0.6
0.175
0:20
0.5
0.150
ace@50
0.4
Brier
0.15
0.3
0.100
0.075
0.10
0.2
0.05
0.1
0.025
0.0
0.000
0.00
Sentence Level
0.6
0.200
0.30
0.5
0.25
0.150
acce50
0.4
0.125
Brier
0.20
0.3
0.100
0.15
0.050
0.05
0.025
0.000
0.00
Paragraph Level
0.25
0.7
0.175
0.6
0.20
0.150
acc
0.125
Brier
0.100
0.3
0.075
0.10
0.2
0.050
0.05
0.025
GPT-J (68)
88)
LLaMA--7
LLaMA-330
Llema2:1
Liama2: 139
Liama3-88 as
Vizana-1 38
GPT-2 GPT-J (68)
96)
LLaMA-78
LLaMA- 308
Limra2-3 78
Liame2-11 1-0
Liema3-88 3-88
Vicuna-1 138
0.00
GPT-2 XL 58)
GPT. (618)
LiahA
LLaMA 300
Liama2.22 78
Lianaz. 138
Limma3-a 8B
Vicuna- 138
GPT-2 1172XL6
Figure 4: Bar charts illustrating the averaged acc@50, ECE, and Brier scores for eight widely used
LLMs as assessed on the CAT benchmark.
A.3 PERFORMANCE COMPARISON ON THE CAT BENCHMARK USING VARIOUS LLMS
Table 6 presents comparative analysis of HADEMI against baseline performance across GPT- 2 XL (1.5B GPT-J (6B 14 and Vicuna- 15 while Table provides comparison results for the I.LMA. lama2, and Llama3 models, including LLaMA-7E LLaMA-30E Llama2-7 Llama2-13B 19 and Llama3- 8F 20 HADEMIF consistently outperforms the original LLMS in gen- eration tasks on the CAT benehmark. These results highlight the effectiveness our approach in hallucination detection and model calibration. Moreover, Fig. presents a performance comparison between HADEMIF and the original LLMs across various evaluation metrics. HADEMIF consis- tently outperforms the original performance of several LLMs. Notably, while the GPT-2 model exhibits lower accuracy, it demonstrates reduced overconfidence, as indicated by its smaller ECE
19

---

Published as a conference paper at ICLR 2025
Dataset
NQ
TrivalQA
Metric
AUC,
AUC, t
PCC
AUC,
AUC, f
PCC 1
Perplexity
0.740
0.747
0.301
0,836
0.836
0.544
LN- LN-Entropy
0.728
0.737
0.298
0.834
0.832
0.540
Lexical Similarity
0.738
0.759
0.306
0.826
0.840
0.556
HADEMIF
0.772
0.776
0.389
0.843
0.842
0.581
Table 8: Evaluation of hallucination detection performance across different methods on the NQ and TriviaQA tasks. The AUC, represents the AUROC score using sentence similarity the measure of correctness, while AUC, represents the AUROC score using ROUGE-L as the correctness measure. HADEMIF consistently surpasses previous methods that rely on single aspect of metrics, such as uncertainty and consistency.
Method
acc@ 50
cov c0v@50
ECE
Brier !
Original LLM
0.288
0.115
0.171
0.196
Label Smoothing
0.208
0.061
0.186
0.212
Temp. Scaling
0.288
0.115
0.165
0.193
LITCAB
0.300
0.105
0.101
0.169
LiTCAB w/ Temp. Scaling
0.300
0.105
0.083
0.164
HADEMIF (Rank 8)
0.355
0.120
0.026
0.119
HADEMIF (Rank 16)
0.350
0.116
0.030
0.121
HADEMIF (Rank 32)
0.352
0.125
0.029
0.120
Table 9: Applicability with increasing training complexity.
and Brier scores. Additionally, Vicuna- 13B, which has been fine-tuned from LLaMA-13B using user-shared conversations, shows inferior performance relative LLaMA, as indicated by its higher average ECE and Brier scores.
A.4 COMPARISON WITH MORE HALLUCINATION DETECTION METHODS
We compare HADEMIF with three other hallucination detection approaches on the NQ and Triv- iaQA datasets using the LLaMA-7B model. The baseline methods include uncertainty- ap- proaches, such as Perplexity (Ren et al. 2022 Chen et al. 2025 Du et al.] 2022) and Length- Normalized Entropy (LN-Entropyy) (Malinin & Gales 2020). well the consistency-basee met- ric, Lexical Similarity (Lin et al. 2022). Following previous studies (Lin et al. 2024 Ren et al.) 2022 Chen et al. 2024a), we use the area under the receiver operating characteristic curve (AU- ROC) and the Pearson Correlation Coefficient (PCC) as performance metrics. For our approach, the hallucination detection score is derived from the D3T model. The results for all methods are presented in T Our proposed HADEMIF framework outperforms previous methods that rely solely on a single aspect of characteristics, thanks to its incorporation comprehensive set of prediction characteristics for hallucination detection.
A.5 APPLICABILITY WITH INCREASING TRAINING COMPLEXITY
To evaluate the applicability our approach as training complexity increases, we examine how performance varies with the rank in LoRA (Hu et al.) 2022). A larger rank indicates that more parameters are trainable. The results on the NQ dataset are presented in Table The performance of HADEMIF consistently surpasses that of the comparison methods across different levels of training complexity. We also observe that relatively small rank can already yield excellent results, which aligns with findings from the original LoRA paper.
20

---

Published as a conference paper at ICLR 2025
Dataset
Metric
HADEMIF with D3T
HADEMIF with DNDT
acc 50
0.315
0.311
cov @50
0.115
0.112
NQ
ECE Brier
0.034
0.045
0.116
0.131
acc@ 50
t
0.760
0.754
cov cov@90 @90
0.230
0,225
SciQ
ECE
0.083
0.084
Brier
0.201
0.203
Table Comparison D3T and DNDT as hallucination detection networks for the output space. Utilizing D3T as the hallucination detection network consistently outperforms DNDT.
0.44
0.67 0.85
e
0.1654<0.55
2.55
0.16 c
555 0.80
0.53
2..53
0.55
0.85
/ <.855
0.3
0.53
0.85
7'300.1
0.31
/ 20.53
>' 0.53
y 0.85
Figure 5: Visualization of a segment from the first three layers of the decision tree generated by the D3T model the final training epoch for the NQ dataset. The outcomes at the leaf nodes reflect the majority classification results for their respective branches, with denoting hallucination and denoting non-hallucination. this context, 8, e, and 7 denote the consistency, uncertainty, and Top-1 margins, respectively.
A.6 COMPARISON BETWEEN D3T AND DNDT
Unlike the DNDT model, which employs a fixed structure that results in numerous redundant branches, our D3T model employs more flexible architecture that dynamically adjusts the number of cut bins for each input feature throughout the training process. To demonstrate of D3T over DNDT, we conduct experiments or DNDT as the hallucination detection networks for the output space. In these experiments, the maximum number of cut points for D3T and the fixed number of cut points for DNDT both set to 4. The results presented Table IO show that D3T consistently outperforms DNDT as the hallucination detection network. The primary reason is that the adaptability of D3T enables it to better align with the local characteristics of the data, thereby reducing the risk of overfitting.
superiority both the NQ and SciQ datasets, employing D3T
A.7 DECISION RULES AND GENERATED DECISION TREE
The interpretability of the D3T model enables the visualization of decision rules, providing insights into how combinations of predictive characteristies determine the presence of hallucinations. In Table II we present a total of fifty decision rules learned by D3T. The first column represents the combinations of characteristics that satisfy various conditions, while the second column displays the corresponding hallucination detection outcomes for each combination. By analyzing these decision rules, we find that low consistency, high uncertainty, small margins, and low maximum probability typically indicative of hallucinations, which aligns with prevailing perceptions. However, the decision rules also reveal that this relationship is not absolute. Therefore, relying solely on a single metric, previous methods have done, is inadequate for effectively detecting hallucinations.
Moreover, to more intuitively reflect the model's decision- maksing, we visualize the decision tree, where a hierarchical arrangement of prediction characteristics is established based their infor- mation gains, similar to traditional decision trees. Specifically, the characteristics are organized in layers based on their information gains, which reflect their importance (calculated in Sec. 4.6).
21

---

Published as a conference paper at ICLR 2025
Combination of characteristies
Decision
8 0.23 e 0.50
0.23
0.44
0.23 0.55 00000000000000000000000000140100
0.25
0.44
0.23
0.44
&
5
5 0.67 0|533+44404440310000100000003004
0.44 0.44 0.67 4.4_00048.....000000000000011110
0.44 0.67 Qrl.aa/ccc.aaaa..............343
0.25
0.44 0.67 Tww4::10014111100111111100111010
0.65
0.44 5. < (167 |44____[{e2200013333000000000001
5 < 0.67 ottp::.........0..00000000000010
0.44 0.44 S
S 0.67 4rl.a0/0000000.00001104444444444
0.25
0.41 < (0 0.25 0.44
0.44 < 0.67 Brta::a...........11  ..31333330
0.41 5 < 0.67 (01.000000.500333003300300000000
0.25
0.41 S < 0.67 0100 o.l.00/00.............44xx444444
0.065
0.44 < 0.67 0 Xrgaa//4444..a..........3343.441
0.44 < 0.67
TAA.331888......1110000001111110 0.65
0.41 < Trt.::/.........111.11.0.0111111
0.44 S < 0.67
0.44 < 0.67 0 1|N00000000000000010111111111110
0,25
0.44
0.44 < 0.67 2
0.25
0.44 5 < 0.67
0.44 S |M] < 0.65
0.67 < 0.85 ( 0rr.a//5SS.......111...111111110
0.67 X,A.0304......A00111100003404140
S &
0.65
0
0.65
0.67 < 0.55
0.65
0.67 < (185
0.65
0.67 8 0..5555558 000
m 0.44
0.67 <. 0.53 & 3,A.
0.67 5 0.85 TrF.a4/AA...0a111112.33130004411
S. < & 44134444444443110111100444444444
0.67 X|TT:3/4AAAA..111111100001111111
0.44
0.67 < < 0.85 0|
0.25 < 0.44
0.67 5 < 0.85 & 00.1
|u| 0.25
0.67 S. < 0.85 (00.00/0000....5331.000000000000
|MM| < 0.65
0.67 5 < 0.85 &,,,0
< (,A3431000....0001331343333343"0
0.67 S 0.85 &1,3
0.25
0.67 < 0.55 (,T.001000000..00001010003313310
0.25 < [M] 0.44
0.67 < 0.85 &00.5 T|tT:3/4.00.0...............4.40
0.25
0.67 5 0. 0
0.25 < |M| < 0.44
0.67 6
0.65
0000 0.65
0.85 < 00
0.65
0,85 & 0.000
0.81 & 0.65
U.85 & 0.16 5. & 0.85 & 0.67
0.88
0.84 & M| 0.65
0.85
0.85
0.88
0.84 & M
0.65
Table 11: Examples of decision rules learned by D3T. We highlight the six most informative char- acteristic values, where 8, e, 71, 7X, p" and |ta| represent consistency, uncertainty, top-1 margin, top-K margin, maximum probability, and logits norm, respectively.
ranged in descending order. Fig. 5 presents segment of the first three layers of the decision tree from the 50th epoch, where each path within the decision tree represents a decision rule. From this tree, find that consistency emerges as the most critical characteristie, followed by uncertainty and margin. Moreover, hallucinated generations typically cxhibit low consistency, high uncertainty, and small margins, although the relationship between each metric and hallucination is not absolute.
A.8 ADDITIONAL ABLATION STUDIES
We conduct additional ablation studies the proposed HADEMIF framework, focusing the complexity of the MLP network, the specific layer from which the hidden states are extracted, and the value of the temperature factor 7. The average performance across both phrase- and sentence-
22

---

Published as a conference paper at ICLR 2025
0.8K
Bco@50
acc0950
0.5
0.5
Brier
Brier
0.183
0.6K
0.784
0.4
0.4
Dimension
Value
18
0.3
0.3
0.4K
0,490
0.2
0.2
100
0.108
0.1
0.1
0.2K
0.0
(a) #lyyr
10
20
30
32
0.01
0.05
0.10
0.15
0.20
Layer index
Figure 6: (a) Brier scores associated with varying numbers and dimensions of hidden layers in the MLP network. (b) Calibration performance based on internal states extracted from different layers. (c) Calibration performance based on varying values of in Eq. (T).
level tasks reported. Fig. 6 demonstrates that model performance, measured by the Brier score, is stable when the number of hidden layers is selected from {2, 3} and the dimension of the hidden layers ranges between 0.4K and 0.6K. Therefore, as mentioned the main text, we recommend setting the number of layers two and the dimension to 512. Additionally, Fig. b( presents the calibration performance based on hidden states extracted from different layers. The results indicate that the internal states extracted from the later and middle layers yield superior performance compared to those from the earlier layers. Therefore, we directly utilize the hidden states preceding the logits. F)) illustrates that the model performance remains stable when T [0.05, 0.15). Therefore, we recommend setting to 0.1 for practical applications.
A.9 INITIALIZATION FOR HALLUCINATION DETECTION NETWORKS
To identify the optimal initialization settings for the two hallucination detection networks, we com- pare four different initialization configurations under the condition of the same number of training iterations (40 epochs). The MIP network considers two initialization methods: He initialization (He al. 2015) and Xavier initialization Glorot & Bengio (2010) with uniform distribution (denoted as X-u). Moreover, D3T considers two initialization methods: X-u and Xavier initialization with normal distribution (denoted as X-n). The comparison results are shown Table 12] He initial- ization, specifically designed for ReLU activation functions, proves to be more ad advatagesuus for initializing the MLP network. For the D3T network, the performance difference between Xavier ini- tialization with uniform and normal distributions is minimal. Consequently, for the MLP network, we utilize He initialization, while for the D3T model, parameters are initialized using Xavier initialization with uniform distribution.
A. LIMITATIONS AND FUTURE WORK
Our study effectively mitigates hallucinations of LLMs and enhances model calibration. However, there are several limitations. A key limitation is that the proposed method relies on internal infor- mation from LLMs, rendering it inapplicable to black-bor box models where users lack access to hidden states. Moreover, due the introduction of new modules and the computation of additional met- rics, our approach increases computational costs, as analyzed in Sec. A.12 Since the majority of the increased time attributable to the computation of the consistency metric, we plan explore alternative metrics more efficient methods for assessing model consistency in our future work.
Additionally, to further enhance detection performance, future research could explore the integration of additional metrics, such as those related to learning dynamics (e.g., loss gradient) and prediction robustness, and develop efficient computational methods for their evaluation. Furthermore, subse- quent studies could leverage the modeled hallucinations in the internal and output spaces inform the development of additional strategies, such as data augmentation, feature selection, and training optimization, thereby advancing efforts to enhance the reliability of LLMss.
23

---

Published as a conference paper at ICLR 2025
NQ
SciQ
D3T
MLP
acc 50
cov@50
ECE +
Brier +
acc 50
cov@90 1
ECE 4
Brier 4
X-u
He
0.355
0.120
0.026
0.119
0.766
0.228
0.076
0.200
X-u
X-u
0.351
0.117
0.030
0.123
0.763
0.224
0.079
0.202
X-n
Hc
0.354
0.121
0.028
0.120
0.765
0.226
0.077
0.198
X-n
X-u
0.350
0.118
0.031
0.123
0.761
0.223
0.080
0.201
Table 12: Performance variation with different initialization settings for the two hallucination detec- tion networks.
A.11 COMPLEXITY ANALYSIS
We analyze the size of the two hallucination detection networks utilized in the HADEMIF frame- work. The MLP model consists of two hidden layers, with the first hidden layer having dimension of imput size hidden size, the second hidden layer having a dimension of hidden size x hid- den_size, and the final output layer having dimension of hidden_size the trainable parameters of the D3T are primarily concentrated in the classifier, with maximum dimension of the classifier weights being (C 1)9 2, where C' denotes the maximum number of cut points and Q represents the number of extracted prediction characteristics. Taking the Llama2- 7B model additional parameters relative to Llama2-7B. In the case of GPT-2 XL (1.5B), only about 1,8% of the parameters relative to the GPT-2 XI. needed. This effectively demonstrates that the two hallucination detection networks are efficient training and exhibit strong scalability.
vocab_size size. Additionally,
example, training the MLP and D3T models requires only approximately 0.3% of
A.12 COMPUTATIONAL COSTS
We analyze the computational costs of our approach, which arise from the introduction new mod- ules and the computation of additional prediction characteristics. Taking Llama2-7B as example, compared to fine-tuning LLMs using LoRA (with rank of 8), the time required to update the two hallucination detection networks per iteration is only 0.9% of the time needed to update the LLMs. Furthermore, the time spent on characteristics extraction in each iteration is equivalent to the time required for updating the LLMs using LoRA. As observed, the majority of the time in our method is spent on characteristics extraction, particularly for the consistency metric. If the consistency metric excluded, the time spent characteristics extraction be disregarded. Consequently, to hance efficiency, we will consider utilizing alternative metrics or exploring more efficient methods for assessing model consistency our future work.
A.13 HALLUCINATION DETECTION CASES
We present examples of generated responses and their corresponding characteristic values using the GPT-2 XL model and the NQ dataset. For each question, three generated answers are provided. Five key distinguishing characteristics are reported, including maximum probability, Top-I margin, Top- K margin, uncertainty, and consistency. As shown in Table [3 non-hallucinated answers typically exhibit greater consistency, lower uncertainty, and larger margins. Moreover, the proposed D3T model effectively differentiates between hallucinated and non-hallucinated answers based on these extracted prediction characteristics.
24

---

Published as a conference paper at ICLR 2025
Question:
which country has the most coastline in the world
GTAns:
Canada
Ansl:
Italy
0.720966308, 0.544921872, 0831045538, 0. 0.186917218, 0.603609998
Ans2:
Togo
0.496951332, 0,49013,4 14441 0.646159074, 0.669920596
Ans3: Canada
0. 0.792032573, 0.809886966, 181 18106644, 1.449488,,,, 0.86631605
Question: who played doctor smith in lost in space.
GTAns:
Jonathan Harris
Ansl:
Guy Williams
0,581436416, 0,483030135, 0.211672489, 0.522018769, 0.521877686
Ans2: Gary Richardson
0.775170857, 0.494472859, 0. 0.18822556,,
0.291994637, 0.668847207
Ans3: Jonathan Harris
0. 0,986663492, 0.987808456, 0.212577534, 0.015423727, 0.732073851
Question: the joint between a coxal bone the pelvis and the sacrum sacroiliac joint
GTAns:
Ans1:
ischio iliacus
0.583019392, 0.577959439, 0. 0.164734295, 0.568657438, 0.602541517
Ans2: Intertrochanteric
0.806365312, 0.822989212, 0.137591002, 0. 0.199395772, 0.119999772,22288221922 0.287201921
Ans3: sacroiliac joint
0.901740451, 0.910176217, 0.204612688, 0.065828692, 0.690894437 Question: what is a another name for the water cycle
GTAns: the hydrological cycle
Ans1: Great Circular Water Cycle
0.607422076, 0.641 125625, 0. 150578222, 0.521414703, 0.594664184 Ans2: hydrologic cycle
0.917772582, 0.92483196 0,207370276, 0.055677697, 0.735232622
Ans3: Hydrologic Cycle
0.881650566, 0.89181 1006, 0.206473246, 0.079155331, 0.740394199
Question: dogs name in the grinch who stole christmas
GTAns: Max
Ans1: Maude
0.72496 0.508574101, 0.189241318, 0.3800800000 0.644346234
Ans2: Algie and Max
0.672750778, 0.700845736, 0. 1.44777977,, 0.505476371, 0.653290101 Ans3: Max
0.804949208, 0.813 1813109496, 0.200413697, 0. 0.102879738, 0.88730444 what is the setting of the book hoot
Question:
GTAns: Florida
Ans1:
Sunterville
0.883634505. 0.610795075, 0. 190277774 0. 0.167938106, 0.630994922
Ans2: Pennsylvania, Singapore
0.507246027, 0.51945497, 0. 0.153838168, 0.639813723, 0.87161937
Ans3: Florida
0.890308594, 0.898310994, 0. 0.182605611, 0.08181169) 0.892968167
Table 13: Illustration of generated responses along with their corresponding characteristic values. The five values, presented from left right, represent maximum probability, Top-1 margin, Top-K' margin, uncertainty, and consistency. All values are min-max normalized across the complete set for each characteristic. If multiple tokens are present in the generated answers, the mean of the metries for those tokens is reported. Responses classified as hallucinations by the D3T model are indicated in red whereas non-hallucinated responses are highlighted in green
25


Reviews:
[{"review_id": "3B3p29uulP", "reviewer": "Reviewer_ApSu", "summary": "The paper addresses the issue of knowledge hallucinations in large language models (LLMs), which can undermine their reliability in critical domains. The authors propose a framework called HADEMIF to detect and mitigate hallucinations in LLMs using two compact networks: a Deep Dynamic Decision Tree (D3T) and a Multilayer Perceptron (MLP). These networks capture hallucinations in both the output and internal semantic spaces of LLMs, allowing for prediction calibration and improved reliability.", "questions": "Scalability: How does the framework perform with even larger models or in real-time applications where speed is critical?\n\nGeneralizability: Can the framework be adapted to other types of hallucinations beyond those related to factual inaccuracies?\n\nBlack-box Applicability: Are there potential adaptations of HADEMIF that could work with black-box models where internal states are not accessible?\n\nMetric Expansion: What additional metrics could be integrated into the framework to enhance its detection capabilities?\n\nUser Trust: How does the framework impact user trust in LLMs, and are there ways to quantify this improvement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Comprehensive Detection: HADEMIF captures hallucinations in both output and internal spaces, providing a more holistic approach compared to previous methods that focus on single aspects.\n\nInterpretable Model: The D3T model offers interpretability, allowing for the identification of key characteristics that impact hallucination detection.\n\nEfficiency: The framework introduces less than 2% additional parameters relative to the LLMs, making it efficient in terms of computational resources.\n\nVersatility: HADEMIF can be applied during both inference and fine-tuning phases, enhancing its applicability across different stages of LLM deployment.\n\nImproved Calibration: The framework significantly reduces the expected calibration error and Brier scores, indicating better alignment between model confidence and accuracy.", "weaknesses": "Dependency on Internal States: The approach requires access to the internal states of LLMs, which may not be feasible for black-box models where such access is restricted.\n\nLimited Metric Scope: While the framework considers multiple metrics, the scope is still relatively constrained, potentially limiting detection performance in more complex scenarios.\n\nComplexity in Implementation: The integration of two separate networks (D3T and MLP) may increase the complexity of implementation and maintenance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fulKGTO96Z", "reviewer": "Reviewer_76SS", "summary": "This paper introduces HADEMIF, a framework designed to detect and mitigate hallucinations in large language models (LLMs). It integrates a novel interpretable Deep Dynamic Decision Tree (D3T) for output space analysis and a Multilayer Perceptron (MLP) for internal state analysis. HADEMIF effectively calibrates LLM predictions with minimal parameter increase, enhancing model reliability.", "questions": "- Could the authors clarify the choice of metrics (such as consistency [1]) and their weights within D3T? (Line 209 \"Extraction of Prediction Characteristics\"). Better motivated metric selection would facilitate the understanding of the numeric values for these metrics. \n- Is there a threshold for model applicability when training complexity increases, particularly with LoRA?\n\n[1] Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: Novel use of D3T and MLP for multifaceted hallucination detection.\n- Quality: Demonstrated robustness through extensive evaluations, including accuracy and Expected Calibration Error (ECE).\n- Clarity: Mostly lear methodology, though technical details may be challenging for broader audiences.", "weaknesses": "- Some parts in the method lack explanations. For example, before line 184 \"Following the approach of DNDT, D3T replaces ...\" there should be clearer explanations for the DNDT method\n- Limited explanation of the interpretability benefits of D3T in practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R8G8ZAttQH", "reviewer": "Reviewer_QEQq", "summary": "This study introduces HADEMIF, a comprehensive framework for hallucination detection and mitigation in large language models (LLMs). HADEMIF leverages both the output space and internal hidden states of LLMs through two compact networks to effectively identify and reduce hallucinations. The framework is applicable in both inference and fine-tuning stages, and experiments demonstrate its superior performance in hallucination detection and calibration across various open-source LLMs compared to existing methods.", "questions": "Refer to weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors integrate multiple metrics to simultaneously assess hallucination in both output space and hidden states, achieving a certain degree of interpretability.\n2. The author considers hallucination at multiple levels in the experiments and employs various LLM backbones.", "weaknesses": "1. Is there a direct relationship between calibration and hallucination? This gap has not been clearly addressed. Although some related work is cited in line 45, this key issue underpinning the entire paper has not been convincingly articulated. Furthermore, we observe in the experimental results that the ECE metric, representing calibration, and the acc@50 metric, representing hallucination, show an inverse relationship to some extent in certain methods.\n\n2. In the authors' statement on line 165, they suggest that single metrics may limit the effectiveness of hallucination detection, but this is not substantiated in the paper. To our knowledge, the authors' method can be summarized as manually extracting certain features and feeding them into a deep decision tree, with these features having appeared to some extent in previous work. Therefore, we have reason to doubt that this combination effectively addresses the challenge of discovering metrics for hallucination detection.\n\n3. The authors use an improved DNDT for hallucination detection in the paper, but they did not conduct related ablation experiments to assess the performance difference between D3T and DNDT.\n\n4. During the training process, it appears that the LLM evolves in conjunction with the detector. However, the authors did not specify the initialization of the detector's weights, which we believe could lead to incorrect gradients being applied to the LLM initially, thereby reducing its generalizability. We also note that the authors did not conduct related experiments on this matter.\n\n5. The authors emphasized interpretability. However, they only provided the impact of various metrics on the output space in the experiments, without focusing on the interpretability of metric itself.\n\n6. The authors seemingly did not compare their method with finetune-based methods or methods specifically designed for hallucination mitigation in their experiments. Instead, they only compared it with some calibration methods and lacked data on finetuning the backbone on these training sets. This omission may lead to unfair comparisons when introducing training in their method, and the comparison methods might also be relatively weak.\n\n7. It would be beneficial to provide experimental results on the computational costs of the model due to the introduction of new modules and the calculation of new metrics, especially consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PC2XE9WVxi", "reviewer": "Reviewer_x16Z", "summary": "The paper presents a new approach to addressing hallucinations in large language models, a significant issue where models generate nonfactual or incorrect information with high confidence. The proposed framework, HADEMIF, uses two neural networks: a Deep Dynamic Decision Tree (D3T) to detect hallucinations in the output space and a Multilayer Perceptron (MLP) to analyze hidden states in the internal semantic space. These models help calibrate the LLM’s predictions to reduce hallucinations.", "questions": "Here are a few questions and suggestions for the authors that could help clarify key points and potentially address limitations in the paper:\n\n1. Training Data Source: Can you provide more detail on how the training data for the D3T and MLP models is collected? Specifically, how are hallucinated outputs labeled and validated? Is there a manual annotation process involved, or are automated methods used for labeling? A clearer explanation of the training data would improve understanding of the framework’s reliability and generalizability.\n\n2. Interpretability of D3T: While the D3T model is described as interpretable, its role as a black-box model is still a concern. Could you provide additional insights or examples on how interpretability is achieved in practice? How does the model allow users to identify key characteristics that signal hallucinations?\n\n3. Comparison with Training-Free Methods: Since you compare HADEMIF to training-free methods like self-consistency, could you clarify why these comparisons are appropriate given that HADEMIF requires additional training? How do you justify these comparisons, and would it be more fair to compare your method against other trained models?\n\n4. Generalization Across Tasks: How well does HADEMIF generalize to different tasks and domains outside those tested in the experiments? Could you discuss whether the framework needs to be retrained or fine-tuned for new tasks, and if so, how this impacts the efficiency of the method?\n\n5. Handling of Complex Scenarios: Can you provide more detail on how HADEMIF handles complex or ambiguous scenarios where there may be no clear factual answer (e.g., open-ended creative writing tasks)? Is the framework likely to flag creative responses as hallucinations in these cases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. A comprehensive approach that addresses both external outputs and internal token states for hallucination detection.\n2. The framework introduces less than 2% additional parameters, making it lightweight.\n3. Extensive experiments to show the reduced hallucination rates.", "weaknesses": "1. Shaky Methodology: The fundamental approach relies on using additional black-box models (D3T and MLP) to detect hallucinations in the LLM, which itself is a black box. This raises concerns about the interpretability and transparency of the method. \n\n2. Unclear Training Data: The paper lacks clarity on where the training data for the D3T and MLP models come from. It is unclear whether hallucinations are labeled manually, generated synthetically, or detected through other processes. This makes it difficult to assess the generalizability of the framework.\n\n3. Unfair Comparison with Training-Free Methods: The paper compares its method to training-free approaches, like self-consistency, without accounting for the difference in training requirements. A more balanced evaluation against similarly trained methods would provide a clearer picture of its effectiveness.\n\nOverall, while the HADEMIF framework shows promise, these methodological concerns limit its potential impact and transparency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A5RJlUoBPx", "reviewer": "Reviewer_goPV", "summary": "The authors propose a framework for LLM hallucination detection and mitigation that consists of a deep neural decision tree (DNDT) that can detect hallucinations from the prediction characteristics of LLMs, and a multilayer perceptron (MLP) that can detect hallucinations from the internal states of LLMs. One can either calibrate the predictions of LLMs with the DNDT and MLP after optimizing them, or alternate between optimizing them and fine-tuning LLMs. In both cases, the DNDT and MLP are optimized to maximize the log-likelihood calibrated with them. The authors demonstrate the advantages of their framework by comparing its performance to those of existing model calibration methods.", "questions": "I have raised my questions in Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors benchmarked the proposed framework with a comprehensive suite of datasets.\n- The paper is mostly easy to follow.", "weaknesses": "My primary concern is whether the proposed framework is indeed better than existing model calibration methods and plain fine-tuning of LLMs:\n- The proposed framework introduces more tunable parameters than existing model calibration methods.\n- The proposed framework is essentially a composition of three tunable models (a DNDT, an MLP, and an LLM), and I am unconvinced that tuning a composition of three tunable models can fundamentally better reduce hallucination than tuning one tunable model (an LLM). (Maybe some inductive biases in the DNDT help.) The empirical results can be strengthened by comparing the proposed framework against fine-tuning with varying numbers of parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)