FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees
Authors: Fan Nie
Xiaotian Hou
Shuhang Lin
James Zou
Huaxiu Yao
Linjun Zhang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Uncertainty Quantification
Hallucination Detection
Neyman-Pearson Classification
Comformal Prediction
Primary area: deep_learning->large_language_models
TL;DR: We introduce FactTest that statistically evaluates whether an LLM can generate correct answers to given questions with provable correctness guarantees.

Abstract: The propensity of large language models (LLMs) to generate hallucinations and non-factual content undermines their reliability in high-stakes domains, where rigorous control over Type I errors (the conditional probability of incorrectly classifying hallucinations as truthful content) is essential. Despite its importance, formal verification of LLM factuality with such guarantees remains largely unexplored.
In this paper, we introduce FactTest, a novel framework that statistically assesses whether an LLM can provide correct answers to given questions with high-probability correctness guarantees. We formulate hallucination detection as a hypothesis testing problem to enforce an upper bound of Type I errors at user-specified significance levels. Notably, we prove that FactTest also ensures strong Type II error control under mild conditions and can be extended to maintain its effectiveness when covariate shifts exist. Our approach is distribution-free and works for any number of human-annotated samples. It is model-agnostic and applies to any black-box or white-box LM. Extensive experiments on question-answering (QA) benchmarks demonstrate that FactTest effectively detects hallucinations and enable LLMs to abstain from answering unknown questions, leading to an over 40% accuracy improvement.

FACTTEST:
Factuality Testing in Large Language Models
with Finite-Sample and Distribution-Free Guarantees
Fan Nie! Xiaotian Hou? Shuhang Lin
James Zou'
Huaxiu Yao? Linjun Zhang? 2
Abstract
high fluency and natural grounding, challenge known as hallucination (Maynez et al., 2020b; Huang et al., 2023; Ji et al., 2023). This tendency undermines the reliability and trustworthiness of the generated content, highlighting the critical need for robust mechanisms to verify the factuality and correctness of LLM outputs.
The propensity of large language models (LLMs) to generate hallucinations and non-factual con- tent undermines their reliability in high-stakes do- mains, where rigorous control over Type I errors (the conditional probability of incorrectly classi- fying hallucinations as truthful content) is essen- tial. Despite its importance, formal verification of LLM factuality with such guarantees remains largely unexplored. In this paper, we introduce FACTTEST, a novel framework that statistically assesses whether an LLM can provide correct an- swers to given questions with high-probabilityy correctness guarantees. We formulate hallucina- tion detection as hypothesis testing problem to enforce an upper bound of Type errors at user-specified significance levels. Notably, we prove that FACTTEST also ensures strong Type II error control under mild conditions and can be extended maintain its effectiveness when covariate shifts exist. FACTTEST is distribution- free and and model-agnostic. It works for any number of human- annotated samples and applies to any -lxck-box or white-box LM. Extensive ex- periments demonstrate that FACTTEST effectively detects hallucinations and enable LLMs to abstain from answering unknown questions, leading to an over 40% accuracy improvement. Code is here.
Existing approaches hallucination detection like retrieval- based methods (Thorne et al., 2018b; Gou al., 2024; Chen et al., 2024) and maining-basede approaches (Zhang et al., 2023) either rely on external databases or resource-intensive fine-tuning processes, which are often impractical or costly. Therefore, uncertainty estimation has emerged as zero- resource alternative for hallucination detection (Varshney et al., 2023; Xiong et al., 2024), operating under the premise that hallucinations are intrinsically tied to the model's un- certainty (Huang et al., 2023). However, none of these methods can provide theoretical guarantees for detection, a critical requirement for deploying LLMs in high-stakes domains (Kumar et al., 2023) where precise control of Type errors (incorrectly flagging hallucination as truthful con- tent) needed for decision-making. For instance, incorrect medical diagnoses the provision of uncertain legal advice could result in detrimental consequences.
To address these limitations, we introduce FACTTEST, a framework that statistically evaluates whether an LLM can reliably generate correct answers to given questions with provable correctness guarantees. We formulate the factuality testing within hypothesis testing framework theoreti- cally control the Type eгror while minimizing the Type II error. Leveraging the fundamental connection between Neyman-Pearge (NP) classification and statistical testing (Tong et al., 2018; Tong, 2013; Scott & Nowak, 2005), we define a score function to quantify correctness and deter- mine an appropriate threshold based on a calibration dataset. This allow L.MSs refuse unknown questions and control the false positive rate for any score function.
1. Introduction
Large Language Models (LLMs) like ChatGPT (Ouyang al., 2022: OpenAI, 2024a) have demonstrated substan- tial advancements across various domains. However, their outputs cannot be fully trusted due their propensity to gen- erate inaccurate and non-facuee non-factual information with seemingly
Our theoretical analysis shows that FACTTEST achieves strong power control when the score function effectively captures output correctness, ensuring not only Type I error control but also low Type II error. We also establish a formal connection between NP classification and PAC-style conformal prediction ( (Vovk, 2012) within our framework,
Stanford University, USA 'Rutgers University, USA 'The University of North Carolina Chapel Hill, USA, Correspondence to: Linjun Zhang <linjun.zhang@ @ rutgers.edu>>.
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
unifying two seemingly distinct methodologies and expand- ing the applicability of conformal prediction. On the other hand, recognizing that the i.idd assumption underlying sta- tistical tests may not always hold in practice, we enhance the robustness of our framework by incorporating an ex- tension to handle covariate shifts through the estimation of density ratios and the use of rejection sampling. Our approach model-agnostic and does not rely on specific data distribution assumptions, making it broadly applicable to any language model. Importantly, it is easy to use and works for any number of numan-annotated samples.
2.1. Problem Formulation
We consider a text generation task in which a language model M will generate its answers (g) based on ques- tion q. Our goal is to statistically evaluate whether M can correctly answer q. We formulate this objective as a hypoth- esis testing problem with the following hypotheses:
Ho : The model M does not answer the question g correctly. H1 The model M answers the question q correctly.
For any question-answer pair (g, a) with to be one of the correct answer for question q, we apply M to generate an an- swer M(). The question-genet -gnnerrttededd answer pair (g. M (9)) is deemed correct the null hypothesis Hg is rejected, i.e., M (g) aligns with a; otherwise, it is deemed incorrect. Let Po represent the distribution of (q, M M(g)) given (() dif- fers from the correct answer, and let P1 denote the distribu- tion of (g, M (4)) when M (q) aligns with the correct answer.
To the best of our knowledge, this study is the first to in- troduce statistical factuality testing for large language mod- els. Evaluation of our proposed framework on question- answering (QA) benchmarks demonstrate several key ad- vantages of our approach: (1) it consistently outperforms base models by substantial margin without requiring ad- ditional training or external data sources; (2) it surpasses fine-tuned baselines by a large margin while utilizing only half of the training data; and (3) it maintains superior perfor- mance on -ntttr--iittitttoon testing data. Notably, the the- oretical guarantees of our method remain valid even when the i.i.d. assumption is violated. We summarize the main contributions below.
Given a dataset {(q), a1)..., (An, an)} C Q x A i..d. ~ Pa.o comprising n question- question-answer answer pairs with Q. A to be the set of all possible questions and answers, respectively, and Pa.a is a distribution of Q A, we apply M to gener- ate answers for all the n questions, resulting in the set Since the distribution Pa(())le of M(4) produced by M given the question g is fully determined by M and independent of
D ((91 ((1)) (1)) a1)..... ...).....((((..) (qn)),
We propose FACTTEST, novel statistical testing frame- work that evaluates the factuality of LLMS while enabling them to decline unknown questions with user-specified Type I error guarantees.
a, we know D i.i.d. P Pa.Mig).a = P (g]ilo Then our goal is to construct predictor fa Q x A + {0, 1} that classifies a pair (g. M(4)) as correct (output 1) or incorrect (output O) while ensuring that the false positive rate, or Type error, does not exceed a pre-specified significance level a. Formally, we seek fa such that the error of predicting incor- rect (g. (q)) as correct is below level a with probability 6, i.e.,
We establish the first formal connection between NP clas- sification and PAC-style conformal prediction within factu- ality testing, providing a unified framework with rigorous guarantees.
at least Ó,
We prove that FACTTEST achieves strong power control under mild conditions, ensuring a low Type error. This power analysis is broadly applicable to standard NP classi- fication problems, not limited to this setting.
PD (F(((.M()))11 (g, M(g)) 1) > S. (I)
where denotes the allowable probability of exceeding a. Note that given any question g. the answer (q) generated by M is randomized. While the distribution of (g) is fully determined by g. the realization M involves additional sampling randomness independent of g. Taking (q. (q)) as input to fa. we enable the predictor to utilize information from the question q, the distribution of M(q) (by asking M the same question q multiple times), and the current realization M (q) of the produced answer.
We extend FACTTEST accommodate covariate shifts by approximating density ratios and employing rejection sam- pling, enhancing its robustness for real-world applications.
We demonstrate that FACTTEST effectively detects halluci- nations while maintaining Type I error below user-specified significance levels, achieving an over 40% improvement in accuracy compared to base models without any fine-tuning. Additionally, it surpasses training-basee based baselines by 30% using only half of the fine-tuning data.
2.2. Finite-sample and Distribution-free Type I Error Control
Calibration Dataset Construction. Following the method- ology of Zhang et al. (2023), we adopt a supervised iden- tification strategy to partition the dataset D into a correct subset D, and an incorrect subset Dg.
2. Statistical Factuality Testing
In this section, we formulate the evaluation of factuality in LLMs as a hypothesis testing problem and introduce our FACTTEST framework to overcome hallucination issues.
Specifically, for each question-generated answer pair
2

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
(g., ((a)) in D. we define an indicator variable y/4 E {0, 1} to indicate the correctness of M (q:) such that
The threshold is then set to: Fa Tay Note that fa is well defined for any no. ensuring Type error control ir- respective of the calibration sample size Specifically, when is small such that v(mu) > 6, the threshold be- comes fa T( +1o+1) +00, causing fa to conservatively classify all pairs (g. (q)) as incorrect, thereby abstaining from answering any question. The derivation is deferred to Appendix. A.
1. if (q:) aligns with the true answer ag:
y4 =
0, otherwise.
Based on these indicators, the dataset is divided into:
D1 [(q, ((.))) E Q A: y i, E [n]},
Do [(a, M(g:)) E e yi = 0, [n]]-
Note that the construction of the indicator vari- able y = (q) aligns with a) for (q)
Theorem 2.1. For any n E N+. with probability at least 6, the constructed classifier fa has type error below a, i.e.
P P..Mii),. defines distribution Pa M()) then the data flg. M(:),, 1i are i..d. samples from P. M())) over all possible combinations of (g, ((), v). and the distri- butions of Do and D are Po = (.) and P1
PD (D 1)
S a 21-8.
Pg. ( respectielly.
With the determined threshold 8.. the predictor f()g.M(4)) = I(f(g, M > fa) is formally de- fined. This classifier ensures that, for a given significance level a, the Type error is controlled below a with high probability 1 - 8. Consequently, when fī(g, (()) fa. we reject the null hypothesis Ho and assert that the model M can answer the question q correctly. Otherwise, the model will output an acknowledgment of uncertainty.
Correctness Predictor based on Score Function. Suppose there is a score function 1) x A R. that measures the correctness of (q, M()). The value is expected to be large if M has the ability to provide factual answer. The predictor (g, M()) can then be defined as:
(a = I(hlg, (g)) > fa)
2.3. Type II Error Control
where Il is the indicator function and f. is a threshold to be determined. The task thus reduces to selecting a threshold t that satisfies the requirement Eq. 1:
The effectiveness of FACTTEST not only hinges on Type I error control but also on ensuring sufficient statistical power to detect true positives. We then analyze the Type II error of the constructed classifier, which the probability of misclassifying correct (g, (q)) from P1 incorrect.
[D (2. (fla, M()) > Fa)= > a) S 8.
(3)
Denote 17(4, (4)) PyrrPote (y 1/4, (M)) to be the conditional probability that (ạ) aligns with the correct answer a given any question q and the generated answer M(g- Note that a question & may have multiple correct answers and a is just one realization from Palg. Therefore, a, and thus y. may still be random given (q. M (q)), implying ngg M (()) may take value in (0, 1). For any classifier f. set Ro(f) P(.M(())~ (f Mg))
Calibration and Threshold Selection To determine the ap-
propriate threshold fa. utilize the calibration subset Do-
Denote the no samples in Do by Do = ((g
(0) Míg ()))
i E [mu]]. For each calibration sample (qi M ())) E
(0)
Do. we compute the score T; = -i (0) M (4; (0) )). We then order these scores in ascending order to obtain the order statisties T11) I ... < Tmn)). and set T T1)
1) (resp. (J) IP ((q.M(g))~>) (f(y, M(2)) = 0)) to be the Type I error (resp. Type II error). It follows from Theorem 1 in (Tong, 2013) that the Bayes optimal classifier fü
t00. Mo- tivated by the seminal works (Vovk, 2012) on the PAC-style conformal prediction and (Tong et al., 2018) on Neyman- Pearson classification, if we set the threshold f, to be the kth smallest score T(,), the probability for fo to have type error greater than a can be controlled in a distribution-fii and finite- saite- manner,
fa E arg Ri(f) s.t. Ro(f) Sa
1oxA-10.11
has the form fā M(2)) I[n(g. M(g)) > Ta) for some To E [0, 1]. Therefore, f the optimal rule for detecting incorrect answers and 77 is an optimal choice of the score function.
Po (P(Pty.M()) M(g)) > T(w)) a)
1- _ (k), k: E [no + 11,
no (
(4)
Suppose there exist an increasing function H and enp >> 0 such that || o f Ti S cn, where H rř M()) H ( M())) is the composition of and f). Let P. = P PymP, (y = 1) denote the marginal probability that M is correct. We define:
j-k when 1: no ukk) is defined to be O. We then determine
k as
k min(k E (no + 1] (kk) < 8),
(5)
3

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
3.2. Type I Error Control under Covariate Shift
To extend the procedure in Section 2 to the covariate shift setting, we take additional step to transform the sam- plcs in Do from Po to Po distributed random variables by rejection sampling.
Ea = [a - Py)
a'=a
a VV a--0-11 V 11g
log
(1 Ta
8'
6r = Ta' Ta + Exp
In the first step, we generate no uniform random vari- ables UI...., Uno iid. Unif B] and select the indexes I {i E [no] U, I w(q,") (0) Mgg
for some constant > 0. If we denote (e(c) F (()) ((nl)) (l7( Tal S E) to be the probabil- ity measure around the classification boundary of fa, then the following theorem states that as long as the score fune- tion f measures the level of correctness of M, the type II error of our algorithm is small.
(0) })}. If we collect
all the samples in Do with indexes in I to form Do [(g M(g, i E I) - {(g. M(:)) : : E [Rol]- Then will be shown in Appendix A that given the selec- tion I by rejection sampling. the selected samples Do follow the target distribution Po i... Do I Lid. Po.
Theorem 2.2. If fg. M (g)) is contimuous random vari- able with (q. (g)) ~ Po. a 2 1/g and Ta tee,te, < 1, an
then with probability at least 1 28, we have
In the second step, we apply the procedure introduced in Section 2 to the incorrect subset Dr. Specifically, given the incorrect subset Do. we calculate the scores T; = i ((:)) and order them in increasing order to get To S ... I T(mo): and set T(ūg+1) +00. Then we set the threshold fa to be T (ky with h satisfies
R1 Rf(fa) R1((fa)
(1- py) + en) Py(1 - Ta --- G ),))
<Ea/ log no
Gals eg)-
3. Extension of FACTTEST to Covariate Shifts
k min(k k E [mo + +1]: +1]:6(K) o(k) S 63,,
The threshold selection procedure developed in Section 2 relies on the assumption that the calibration dataset Do {(1, M(g.) E Q yi = 0} follows the target distribution Po of incorrect question- -eatrion-eeerreed answer pairs. However, labeled samples from the target distribution may not always be available in practice. Instead, people may use the labeled data that they believe to be similar to the target distribution, which necessitates methods handle distribu- tion shifts. In this section, we study the case of covariate shift, where the distribution of the question-generaane swer pairs in the calibration data differs from that in the target distribution, while the conditional distribution of given (q. M (q)) remains the same.
no R ( (1 (1-a)aa no a1- 8(6o + 1)= =0
O1k) =
Since Do I 1.3.d. Po. theoretical results in Section 2 can be directly applied here. Due to limited space, we control the Type error as follows.
Theorem 3.1. With probability at least 1-6, the constructed classifier fa(g, M (4)) = I(î(q, M (g)) > T(ts) (it)) has type I error below a, i.e.
an-
PD PD (a. (fa (g. M()) 1) S a
4. Connection with Conformal Prediction
3.1. Setup
In Sec. 2. we formulate our method within the NP classifica- tion framework by defining an NP classifier for error control. In this section, we explore the inherent relationship between NP classification (Tong et al., 2018) and PAC-style confor- mal prediction (Vovk, 2012), demonstrating that FACTTEST can also be formulated within the lens of conformal pre- diction. To the best of our knowledge, this work presents the first formal articulation of the connection between NP classification and PAC-style conformal prediction.
Suppose we observe n samples D= = {(i, M (4x), : [n]} from the source distribution P (g))) We assume Pala.M(o) Py Pylg.M(g) but Pa.M(n) F Pa.M(g)- i.e., the distribution of questions changes but the oracle rule of de- tecting incorrect answers remains. Following Section 2. we split 'D into a correct subset Dj = ((a, M(1)) 34 = 1 [n]}
()) E [mi]} and an in- correct subset Do [lg,, M (q.)) y = 0, [n]] ((g.)) ()) (q. (0) )) : [nol]. We denote the distribution of Do. Dr to be Po, Pt, respectively. We further denote the density ratio between the target distribution Po of incorrect question- gentrrted answer pair and the source distribution dFle Po to be w(g. (g)) 2P. (g. M (q)). In this section, we assume w is known and satisfies w(q, M (q)) < B for all (g. M()) E Q A.
{(g.") (1)
Challenges of conventional conformal prediction. Recent works applying conformal prediction to LLMs (Kumar et al., 2023; Quach et al., 2024; Li et al., 2024) have primarily focused on answer set coverage guarantee. A naive appli- cation of prediction sets to hallucination detection might involve verifying whether outputs fall within the prediction set. However, this approach is fundamentally flawed for
4

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
the hallucination control problem formulated in Section 2.1, because enforeing (1 -a) coverage guarantee inherently tol- erates incorrect answers within the prediction set, leading to large Type errors. In an extreme scenario, ensuring 100% coverage would require including all possible answers, re- sulting in a 100% Type I error rate. More empirical results can be seen in App. D.I.
Distinctive Features of FACTTEST. To the best of our knowledge, we are the first to introduce statistical factuality testing for LLMs. In addition, while FACTTEST can be interpreted through both NP classification and PAC-style conformal prediction, our key distinction lies in our detailed analysis of Type II error, enabling the control of both Type I and II errors. Further, we identify the optimal score function for constructing the best classifier with minimal Type er- ror, topic that has not been explored in the PAC conformal prediction literature.
PAC-style conformal prediction, Suppose we are given dataset DDP (X,: i [n)) ii.d Px,, where each sample X; E X' follows the distribution Px PAC-style conformal prediction (Vovk, 2012) aims to construct a predictive set r for an independent random element X ~ Px such that
5. Experiments
In this section, we empirically investigate FACTTEST in addressing the hallucination problem of LLMs, focusing on the following questions: Q1: Can FACTTEST improve the accuracy and lead to more factual LLMs? Q2: Can
Por (gx~pg (X $ (1) > a) S 6.
(6)
To this end, (Vovk, 2012) proposes to use conformity score A X' - R. for measuring how well X conforms to the dataset DoP, Given any conformity score A, we define the conformity scores of samples X; in DFP as S: A(X)).
FACTTEST effectively control the Type error? Q3: Can FACTTEST generalize well when covariate shifts exist?
5.1. Experimental Setups
Then a p-value p(X) is defined by
Datasets. Following R-Tuning (Zhang al., 2023), we con- duct experiments on knowledge-extensive QA tasks, catego- rized into two generation tasks. More details are provided in Appendix C.2.
xX)= KS, S. S S(X)++
n+1
Finally, for some >> 0, the predictive set I' is defined by
Given question, the model di- rectly predicts its answer. We include ParaRel (Elazar et al., 2021) and HotpotQA (Yang et al., 2018). For experiments considering distirbution shifts, we utilize ParaRel-OOD as the testing dataset, which comprises questions from different domains compared with ParaRel.
Question-Answerinn
(DPP) = {x E X p(x)= > e).
(7)
It follows from (Vovk, 2012) that as long as
11(++11-1J 1J
Σ
11 0 (1- (1))ma-- 2.,
Multiple-Choice: Given question with several choices, the model chooses one option among A, B and C. We include WiCE (Kamoi et al., 2023) and FEVER (Thorne et al., 2018a).
the coverage guarantee 6 is satisfied.
Duality of PAC-conformal prediction and FACTTEST. If we set the conformity score to be -f, where n is introduced in See. 2.2 and construct the predictive set D defined in 7 using data D = Du. then the predictive set (*(D)) for (g. .Q)) and the classifier f in Sec. 2.2 are closely related as stated in the following lemma:
Evaluating whether M(q) aligns with the answer a depends on the datasets. For question-answeriig datasets, we verify whether the first few tokens contain a. For multiple-choice datasets, check whether M(4) matches exactly
Score Functions. We can fit model to predict correctness or use any off-the-shelf function to serve as f). Particularly, we introduce three entropy-based certainty functions. De- tails about the score functions are deferred to Appendix C.3.
Lemma 1. If we set e such that [c(no + +1) - 1]= = no ke then
ia(g, (())) = ((( M()) & (D.)).
The proof of this lemma is straightforward and thus omitted. This equivalence highlights that NP classification-aused threshold selection aligns with the membership determina- tion in PAC-style conformal prediction, which is akin to the well-known duality between confidence interval and hypoth- esis testing. Under our setting, conformal prediction aims to construct a confidence set for (q, M())ly = O, while Neyman- classification aims to test whether = or not given (g, M (q)).
Vanilla Entropy (VE): and calculate the entropy across k answers.
We query the model M k times
VE(g, M()) Σ P(M
i(q M()) = -V)(
(8)
where ((M(4)))4) the frequency of a predicted answer
5

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Table
The accuracy performance (%) of FACTTEST compared baselines on question question-answeneee answering and multiple- choice datasets using a significance level of a 0.05. For brevity, FACTTEST is abbreviated as FTEST. The notation FTEST-Ve15 denotes the use of a vanilla entropy score function with 15 generated outputs.
Dataset
Model
Base
SeliCheckGPT
Fe
vēgo
ve1s
FTEST seg
FTEST-se1p
FTEST-se:
FTEST kiless
La
36.66
$3.60
60.54
66.75
67.28
60.10
62.50
67.26
78.45
ParaRel
OaeALLaMAA-7 7B
40.38
60.05
74.92
79.87
80.29
65.53
71.40
65,23
76.83
OpenI.IaMA- 138
42.21
59.62
77.37
77.31
79.41
73.49
68.89
73.09
87.84
OpenLLaMA-33
25.72
36.42
50.81
55.19
53.75
45,37
52.55
52.66
55.35
HotpoiQA
28.63
39.16
56.06
59.69
60.67
51.48
53.75
56.56
60.66
LLaMA-13B
20.83
41.78
51.49
54.41
49.74
55.41
57.18
60.69
54.49
OpenLLaMA-33
64.72
66.36
67.65
75,00
68.18
64.71
85.71
66.67
WICE
OpenLLaMAA77
72.96
75.00
50.00
47.37
90.00
100.0
90.00
LLaMA- 13B
56.89
57.39
63.33
45.45
44.44
100.0
82.35
90.00
La
39.74
41.97
60.24
62.50
41.72
82,40
79.23
83,90
FEVER
LLaMA-7B 7B
35.99
40.89
43.92
50.94
51.38
28.69
33.12
33.27
LLaMA- 13B
32.15
41.25
38.74
42.48
46.07
49.92
54.17
52.23
Table 2: The Type I error of FACTTEST on question-answering and multiple-choice datasets when 0.05.
Dataset
Model
FTEST-ve;
FTEST-vejo
FTEST-Ve1s
FTEST-se;
FTEST-Sejo
FTEST-S815
FTEST-kless
LaMALB
0,0508
0.0467
0,0513
0.0479
0,0520
0.0486
0.0342
ParaRel
Op
0.0225
0.0093
0,0145
0.0393
0.0394
0.0435
0.0400
OpenI.LaMA-138
0.0192
0.0087
0,0302
0.0341
0.0477
0.0337
0.0331
OpenLLaMA-33
0.0242
0.0247
0.0272
0,0289
0.0319
0.0297
0.0309
HotpotQA
OpenLLaMMA-7B
0.0273
0.0298
0.0295
0.0344
0.0298
0.0308
0.0266
LLaMA-13B
0.0200
0.0226
0.0367
0.0278
0.0300
0.0286
0.0353
OpenLLaMA-33B
0.0325
0.0089
0.0207
0.0175
0.0029
0.0118
WICE
OpenLLaMA-77
0.0694
0.0579
0.0617
0.0077
0.0
0.0039
LLaMA-13B
0.0266
0.0290
0,0363
0.0
0.0072
0.0024
OpenI LaMA-33
0.0164
0.0005
0.0217
0.0570
0.0471
0.0496
FEVER
LLaMA-7B
0.0598
0.0081
0.0329
0.0392
0.0495
0.0495
LLaMA-13B
0.0172
0.0383
0.0293
0.0459
0.0518
0.0552
M(4) given a question q-
To facilitate comparison with training-basee methods, we randomly split our training dataset, allocating half for instruction-tuning and the remaining half to construct the calibration dataset. We use 15-generation SE as the score function, referring this variant as FACTTEST-L. For com- parative analysis, we include R-Tuning (Zhang al., 2023) as primary baseline, evaluating it on the subset of ques- tions that it is willing to answer. We also consider Finetune- All and Finetune-Hadl which undergo instruction-tuning using the entire and half of the original training dataset, respectively, and are evaluated on the entire test set.
Semantic Entropy (SE): Kuhn et al. (2023) measures uncertainty in natural language generation by accounting for the probability distribution over distinct meanings rather than individual token sequences.
Kernel Language Entropy (KLE): Nikitin et al. (2024) quantifies uncertainty by using semantic similarity kernels between generated answers, allowing for a more nuanced estimation of uncertainty. Notably, this function does not apply to multiple-choice datasets and we only employ it on ParaRel and HotpotQA.
To evaluate the applicability of our framework on black- box APIs, further implement FACTTEST GPT-4o Mini, GPT-4o (OpenAI, 2024b), Gemini-1.5 and Claude- 3.5 (Anthropic, 2024).
Models. In main experiments, we focus on distribution-fre settings, where models do not make specific assumptions about the underlying distribution. We include Base and Self- checkgpt (Manakul al., 2023) as baselines. Base evaluates the original model on the entire test set without any modifica- tions, while Selfcheckgpt and FACTTEST are assessed only on questions for which they can confidently provide answers. We utilize three score functions to implement 9 variants of FACTTEST. Specifically, FACTTEST-ve,, FACTTEST-Se; and FACTTEST-Kle, correspond to using VE, SE and KLE as score functions, respectively, where & denotes the number of sampled outputs for a given question.
Metrics. For models that could only output either the an- swer or an unknown expression, we calculate accuracy on questions that the model is willing to answer:
Ace # of correctly and willingly answered questions # of willingly answered questions
Besides, we also include Type error (False Positive Rate, FPR), and Type II error (False Negative Rate, FNR), as our
6

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Calordted Type lero
Calbruee Type tbr yeu
Gettritedd Type Ennor
Foctlest- VDys
Tactfast -VPt
Factlest-vrs FartFest-Sha
0.7
Factlest- V015
Factlest- V85
Type (Enor
Factffstt
Typo EEror
Factlest
0.1
0.8
0.3
04
0.5
0.0
0.7
0.1
0.3
0.5
Signtficance Level
Signficance Level
Signifisance
(a) ParaRel-3B
(b) WiCE-3B
(c) FEVER-3B
Figure 1: FACTTEST can control the Type error given significance level The caption of each sub- sub-figure consists of the dataset name and the model size.
evaluation metrics.
Openliatoomm
0.35
Implementation. We choose LLaMA-7B, LLaMA- 13B (Touvron et al., 2023), OpenI.LaMA-3B, OpenL.LaMA- 7B, OpenLLaMA-133 (Geng & Liu, 2023) as base models in our main text. Experiments involving Mistral-7B (Jiang (Dubey et al., 2024) and Tulu2-7B (Ivison et al., 2023) are deferred to App. D.6. The temperature set to for evaluation and 0.7 for calcu- lating score functions. We follow Zhang al. (2023) to use LMFlow (Diao al., 2023) to conduct instruction tuning, setting epoch to and learning rate to 2e-5, 5, All experiments are implemented on 4 Nvidia H100-80GB GPUs.
ge0
3:40
200
3.70
-28
-18
-00
-12
-10 -67
et al., 2023). LLaMA-3.2-3BB
Thonked
(a) ParaRel
Openl JOA
1.30
31
125
Theahald
-3 -34
5.2. Main Experimental Results
(b) WICE
We first conduct in-distribution experiments QA datasets ParaRel, HotpotQA, WiCE and FEVER.
Figure 2: The Accuracy-Trreshold Threshold curve. The title of each sub-figure consists of the dataset name, the model size and the certainty function.
Main Performance. The accuracy results are presented in Table 1, where the significance level a for FACTTEST set to 0.05. Additional experimental results for other significance levels (c.g.. a 0.10) are provided in Ap- pendix D.2. Analysis of the results reveals that FACTTEST significantly outperforms baselines by a substantial margin in terms of accuracy on the questions it is willing to answer Notably, FACTTEST can yield an over 40% accuracy im- provement on WiCE and FEVER. These results demonstrate that FACTTEST is more reliable when making predictions and is capable of refusing unknown answers.
error while ensuring that the Type I error remains within the specified upper bound. For > fas the Type er- ror decreases monotonically, whereas the Type II error in- creases monotonically. Figure 2 presents the accuracy-f curve, where ř begins f..1. This curve can be utilized to maximize accuracy, which does not follow a monotonic trend as the threshold f increases, while ensuring that the Type error is controlled below 0.10.
Type I Error. Table 2 represents the Type I error, FPR, of FACTTEST when is set to 0.05. Figure 1 depicts the FPR-a curve. For given significance level we enforce an upper bound on the FPR at a with a high probability guar- antee. Analysis of these figures confirms that our method reliably controls the Type error, thereby validating the the- oretical results presented in Section 2.2. Additional error control analysis including FNR and more FPR results for FACTTEST are available in Appendix D.4.
5.3. Comparing with Finetuned Models
Figure 3 illustrates the accuracy performance of FACTTEST- compared to the baseline methods R- R-Tuning, Finetune-All, and Finetune-Half, We randomly divide into two equal parts: D) for eestruction-tuning and De for constructing the calibration dataset. The pretrained model is finetuned on Dj to obtain Finetune- Half, while Finetune-All is obtained by training the entire dataset D. For R- R-Tuning, we also utilize the entire dataset to finetune the model. It is evident that FACTTEST-t consistently outperforms R- R-Tuning by a large margin, while utilizing only half of the available train- ing data, thereby reducing training costs by 50%. Notably,
Maximizing Accuracy. Given a significance level a, we can determine the threshold fa that minimizes the Type II
7

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
FactTesbt
R-Tuning
Finelune
Pretmined
OpenLLaMA-3b
LLaMA-7b
Accuracy
HolporDA
ParaRel
HotpotQA
Figure 3: The Accuracy performance (%) of FACTTEST trained on half of the data, comparing with training- based baselines. Both R-Tuning and Finetune All utilize all training data for finetuning, while Finetune Half uses the same half of the finetuning data FACTTEST.
FactTest04
FactTestO-Ue
Fineture Pretrained
LpeALLaMA
LLaMA 7b
R-Tuning
FactlestO-se
Caltrattdd Type Error
Calbbated Type
PactteestO-. 5015
FactFestO oh1s
Type Error
0.6
FactTestO
0.6
FactTTsst Kitus
Accuracy
80
FactTesto
FectfiesD--
60
Type
0.2
40
0.0
20
OpenLLaMA-38
LLaMA 7b
Significance Level a
Significance Level
(a)
(b)
(c)
Figure 4: (a) The Accuracy performance (%) of FACTTEST ParaRel-OOD testing dataset. (b)(c) FACTTESTO maintains its ability to control Type error given significance level a when distribution shifts exist.
FACTTEST- yields 34% and 28% accuracy improvement over R- R-Tuning HotpotQA and FEVER, respectively. De- spite the reduced size of the calibration dataset, FACTTEST-t maintains effective control over Type I error, with further details provided in Appendix D.4.
Table 3: The accuracy performance (%) of FACTTEST ap-
plied to GPT-4o-mini. The significance level is chosen as 5%. The number in parentheses Type I error. GPT +
7B/13B means utilizing OpenLLaMA-7B/1133 to calculate certainty scores for GPT-4o mini.
Dataset
Model
FTEST-Se10
FTEST-se15
FTesT-kless
5.4. Extension to Covariate Shifts
GPT-4o mini
52.83
In this subsection, we evaluate the extension of our frame- work, denoted as FACTTESTO (FACTTEST for Out-of- distribution domains), on dataset with distribution shifts.
ParaRel
GPT +
77.78 (0.03)
77.31 (0.03)
83.88 (0.05)
GPT + 13B
76.91 (0.04)
77.67 (0.05)
85.84 (0.04)
GPT-4o mini
75.67
WICE
GPT 7B
81.82 (0.02)
76.67 (0.03)
Setup. We utilize ParaRel for training, consistent with the aforementioned experiments. We randomly split ParaRel- OOD into validation dataset comprising 1,000 samples and testing dataset containing 12k samples. To calculate the density ratio in Section 3 between the target distribution and the source distribution, we employ the training data from the source domain and the validation data from the target domain train a binary classifier and utilize the predicted probability approximate density ratios. Subsequently, select B as the upper quantile of density ratios to filter out anomalous values. We set the default value of y as 90%.
GPT 13B
86.95 (0.01)
81.77 (0.02)
out fine-tuning. Additionally, FACTTESTO effectively en- forces the upper bound on the Type error, thereby main- taining its efficacy
scenarios.
5.5. Extension to Black-box APIS
We further evaluate our framework on black-box models to broaden the applicability of our framework. While score functions like SE and KLE require token probabilities, which are unavailable for black-box APIs, we utilize open- source models to calculate the scores on calibration datasets constructed by black-box models. Table 3 illustrates the performance of FACTTEST on GPT-4o Mini. The results demonstrate that the scores derived from open-source mod- els are effective for black-box APIs, achieving a 33% accu- racy improvement on ParaRel and an 11% improvement on WiCE, while maintaining control over Type error. These
Experimental Results. Figure 4 depicts the performance of FACTTESTO on the ParaRel-OOD testing dataset, along- side the Type error-a curve. The results demonstrate that FACTTESTO-L significantly outperforms baseline methods by large margin. Notably, when utilizing OaenLLaMA-- 3B as the pretrained model, both FACTTESTO-se and FACTTESTO-kle outperform bomiing-based methods with-
8

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
findings illustrate that our framework provides a practical and effective solution for detecting hallucinations in closed- box models. More results involving Claude-3.5, Gemini-1.5 and Go are provided in App. D.6.
independent interest. Additionally, we extend the NP classi- fication framework to account for covariate shifts, enabling it to address more practical, real-world problems.
7. Conclusion: Summary and Limitations
6. Related Work
In this paper, we introduce FACTTEST that leverages the principles of NP classification to provide statistical guar- antees for LLMs. By formulating hallucination detection as hypothesis testing problem, FACTTEST enforces an upper bound on Type errors. It ensures strong power con- trol under mild conditions and can be extended to maintain its effectiveness in the presence of covariate shifts. These theoretical analyses can be seamlessly integrated with the standard NP umbrella algorithm, not limited to our frame- work. Our approach is distribution-free and finite-sample. It applies to any LLM including closed-bo box models. Empirical evaluations have demonstrated that FACTTEST consistently outperforms baselines and maintains superior performance under distribution shifts. Additionally, our framework effec- tively enhances the reliability of black-box APIs.
We review the progress on LLM Factuality and distribution- free and finite-sample inference. More related work is de- ferred to App. B
Factuality of LLMs. The factuality of LLMs is a major problem and of significant research interest (Ji et al., 2023; Maynez et al., 2020a; Li et al., 2023), including halluci- nation detection, mitigation and evaluation (Huang et al., 2023; Wang et al., 2023). Our work relates more to hal- lucination detection, which is imperative for assuring the reliability of the generated content. Kadavath et al. (2022) proposes self-evaluation to verify the prediction. ? trains a classifier based on hidden layer activations. Lee et al. (2023) uses factual and nonfactual prompts and creates a benchmark for measuring the factuality of generations. Man- akul et al. (2023) introduces SelfCheckGPT to fact-check the responses of LLMS in zero-resource fashion. Zhang et al. (2023) instructs LLMs to refuse unknown questions by refusal-aware instruction tuning. However, none of these works have provided theoretical guarantees.
One limitation of our work is the implementation of a lim- ited number of score functions. Exploring additional score functions could further enhance the performance. Further- more, our framework constructs the predictor in an offline manner. Future work could extend FACT TEST to support on- line testing, thus enabling real-time factuality assessments.
Distribution-Free Free and Fimite-Sample Inference. Recent works have extended conformal prediction to provide guar- antees on the outputs of LLMs (Kang et al., 2024; Quach al., 2024; Mohri & Hashimoto, 2024). While these meth- ods focus on improving model outputs, FACTTEST is de- signed to verify correctness and decline unknown questions. Our framework leverages principles from (eyman-PParsoon (NP) classification. The NP classification paradigm dif- fers from standard classification and cost-sensitive learning, which aim at minimizing a weighted combination of Type I and II errors. Instead, it prioritizes controlling the Type I error while minimizing the Type II error, ensuring that the Type error remains below a user-specified threshold a. Rigollet& & Tong (201 l) and Scott & Nowak (2005) pro- posed using empirical risk minimization, and Tong (2013) employed plug-in approaches to construct NP classifiers. more related work by Tong et al. (2018) introduced an umbrella algorithm that achieves Type I error control for any pretrained classifier, while similar techniques were also proposed in the PAC-style conformal prediction literature (Vovk, 2012). However, the methods in Tong et al. (2018) and Vovk (2012) do not provide Type II error guarantees.
Acknowledgements
The research of L.Z is partially supported by NSF CAREER DMS-2340241.
Impact Statement
Our work aims to enhance the reliability of large language models (LLMs) by statistically evaluating the correctness of model outputs. The primary objective is to reduce the risk of misinformation and non-factual content, especially in high-stakes domains where inaccuracies can lead to signifi- cant harm. FACTTEST facilitates responsible deployment of LLMs by enabling them to decline to answer uncertain ques- tions, thereby promoting transparency and trustworthiness in AI systems. While the framework not directly tied to specific ethical risks, it indirectly mitigates the propagation of false misleading information in sensitive domains. Ad- ditionally, its extensibility to black-box APIs supports the development of more robust and fair Al ecosystems. Fur- thermore, we have taken steps to ensure that our research minimizes any privacy or security risks. All datasets used in this study are publicly available and do not contain per- sonally identifiable information, thereby safeguarding user privacy and adhering to data security standards.
Our work takes an initial step to use the NP classification idea to conduct factuality testing for LLMs. Furthermore, the Type II error analysis of our method can be directly applied to the standard NP umbrella algorithm, which is of
9

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
References
Geng, X. and Liu, H. Openllama: An open reproduction of Ilama, May 2023. URL https: httpe////ggh.///utttteeee thub. com/ open openim-reseann wwweereeeaccchc aeteeearchhoopeen1ee1e hatphh/oppeeaaaaaaaaaaaa 1lama.
Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution- free uncertainty quantification, 2022. URL https::/aaaatt //wxxxxxxvvvoog./ abs/2107.07511
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., and Chen, W. Critic: Large language models can self-correct with tool-interactive critiquing, 2024. URL /ettp://arxivv.org/aaaa
Anthropic. Claude 3.5 sonnet.
https: / /www.
anthropic. cwtppppi..ccoooeeeeeeeeeeeeeee heeem//ww/////e clludee 3- 10-5-500 3-5-sonnet, 2024.
Huang, L., Yu, W. Ma, W., Zhong. W., Feng, Z., Wang, H., Chen, Q.. Peng. W., Feng, X., Qin, B., and Liu, T. A sur- vey on hallucination large language models: Principles, taxonomy, challenges, and open questions, 2023.
Azaria, A. and Mitchell, T.
The internal state of an LLM knows when it's lying. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2023, pp. 967-976, Singa- pore, December 2023. Association for Computational Linguistics. doi:
Huang, Y., Liu, Y., Thirukovalluru, R., Cohan, A., and
Dhingra, B. Calibrating long-form generations from large language models, 2024. URL https: /ttp::://axxvvvv//r/ee abs / 22/22402.000 06544.
105533//1/222..nniiigg
68. URL https:/ / aalaccannnn aclannhhlogyy.... org/2023. findings mandings-emni lmmllmmllppppp p. 68/.
Barber, R. F., Candes, E. J., Ramdas, A., and Tibshirani, J. Conformal prediction beyond exchangeability. 2023. URL https: / /arxivv..vv/../////va//aaa/aaaav org/ abs 2302.13155.
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., and Hajishirzi, H. Camels in a changing climate: Enhancing Im adaptation with tulu 2, 2023. https:
13415.
Braverman, M., Chen, X., Kakade, S. M., Narasimhan, K., Zhang, C., and Zhang, Y. Calibration, entropy rates, and memory in language models, 2019. URL https: // //arxiv. org/ aba/ /106. 05664.
//arxiv. org/ abs /21/2 10702.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Chen, J., Kim, G., Sriram, A., Durrett, G., and Choi, E. Complex claim verification with evidence retrieved in the wild, 2024. URL 2305. 1 859.
abs/
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand. F,, Lengyel, G., Lample, G., Saulnier, L.,, Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https:
Detommaso, G., Bertran, M., Fogliato, R., and Roth, A. Multicalibration for confidence scoring in Ilms, 2024. URL https:://aaaatt
org/ aww.aaasaaaa 2404. 04689.
/ //arxiv. org/ abs aat///aa//3300 06825.
Diao, S., Pan, R., Dong, H., Shum, K. S., Zhang, J., Xiong, W., and Zhang, T. Lmflow: An extensible toolkit for fine- tuning and inference of large foundation models, 2023.
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds Z., DasSarma, N., Tran-Johnson, E., Johnston, S., Fl-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language models (mostly) know what they know, 2022.
Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. Shifting attention to rele- vance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 5050- 5063, 2024.
Kamoi, R., Goyal, T. Rodriguez, J. D., and Durrett, G. Wice: Real-world entailment for claims in wikipedia, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models, 2024. URL https: //arxiv. o abs/ / 2407 21783.
Kang, M., Gürel, N. M., Yu, N., Song, D., and Li, B. C- RAG: Certified generation risks for -urreval-augmentte language models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F. (eds.), Proceedings of the 41st International Conference
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., and Goldberg, Y. Measuring and improv- ing consistency in pretrained language models, 2021.
10

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
on Machine Learning, volume 235 of Proceedings of Ma- chine Learning Research, pp. 22963-23000 PMLR, 21- 27 Jul 2024. URL https: /ptpeeediooeeeeseeeeeeeo/.. .ml.. press/ v2////22/////////// ht22//////g2222 html.
1906-1919, Online, July 2020a. Association for Com- putational Linguistics. doi: 10. 10.186553/12000.dmmmaia acl acl-main.174
173. URL
Kuhn, L., Gal, Y., and Farquhar, S. Semantic uncer- tainty: Linguistic invariances for uncertainty estimation in natural language generation, 2023. URL https: //arxiv.oo.. org/ abs/ 2tt//332..000 09664.
Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. On faithfulness and factuality in abstractive summarization. arXiv preprint wrxiv:2005.00661 2020b.
Mohri, C. and Hashimoto, Language models with con- formal factuality guarantees. ArXiv, abs/2402.10978, 2024. URL //tctps://app semanticscholee.. org/ /ttp:Cooou11D:277555555
Kumar, B., Lu, C.. Gupta, G.. Palepu, A., Bellamy, D., Raskar, R., and Beam, A. Conformal prediction with large language models for multi-choice question answer- ing, 2023. URL 18404.
Nikitin, A., Kossen, J., Gal, Y., and Marttinen, P. Kernel language entropy: Fine-grained uncertainty quantification for Ilms from semantic similarities, 2024. URL https:
Lee, N., Ping, W., Xu, P., Patwary, M., Fung, P., Shoeybi, M.. and Catanzaro, B. Factuality enhanced language models for open-ended text generation, 2023.
//arxiv.org/.. /ttpa///..///./ 20003.
OpenAL. Gpt-4 technical report, 2024a. URL https:
Li, S., Han, C., Yu, P., Edwards, C., Li, M., Wang, X., Fung, Y., Yu, C., Tetreault, J., Hovy, E., and Ji, H. Defining new NLP playground. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 11922-11951, Sin- gapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023..mmm hwwdi///222....inn-.aiiaa emnlp. 799. URL https: : /tc//aclaani mocclanthooogy orr.orgg///00 2023. Findings-emnipp 799.
//arxiv.orr/abs//20000 08774
OpenAL Hello gpt-4o, 2024b. URL https://
com/index/ https////hlllogp.....//oo/
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022.
Li, S., Park, S., Lee, I., and Bastani, O. TRAQ: Trustwor- thy retrieval augmented question answering via confor- mal prediction. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies (Volume 1: Long Pupers), pp. 3799-3821, Mexico City, Mexico, June 2024. Association for Computa- tional Linguistics. doi: 10. 10.11665311/2222.aaacccoo 210. URL https:// http://caaattooog..ooooo naacl-long.22 210/.
Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R. Conformal language modeling. In The Twelsth International Conference on Learning Representations, 2024. URL https:// openreview net wow.moiidopraaoeoo Corumm?.-ep2hh.e7744555
Rigollet, and Tong, X. Neyman-pearson classification, convexity and stochastic constraints. Journal of machine learning research, 2011.
Lin, Z., Trivedi, S., and Sun, J. Generating with confi- Uncertainty quantification for black-box large language models, 2024. URL httph::t///// http::///aaxxvv//o// abs 2305 19187.
Scott, C. and Nowak, R. A reyman--earson approach to statistical learning. IEEE Transactions on Information Theory, 51(11):3806-38 3819, 2005.
dence:
Shafer, G. and Vovk, A tutorial on conformal predic- tion, 2007. URL https:t///// http:///axvvvvvi///e wtgpor/g//bb/.....///////// 3 3188.
Manakul, P., Liusie, A., and Gales, J. Selfcheckgpt: Zero- resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08886 2023.
Skorski, M. Bernstiin-type bounds for beta distribution. Modern Stochastics: Theory and Applications, 10(2):211- 228, 2023.
Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. On faithfulness and factuality in abstractive summa- rization. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp.
Thore, J., Vlachos, A., Christodoulopoulls, C., and Mittal, A. Fever: a large-scale dataset for fact extraction and verification, 2018a.

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Thore,, J., Vlachos, A., Cocaraseu, O., Christodoulopoulos, C., and Mittal, A. The fact extraction and verification (fever) shared task, 2018b. URL https: arrii..
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning. C. D. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feed- back. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: / /ppenreview forum? i 3 faCffrm...
Tong, X. A plug-in approach to neyman-pearson classi- fication. Journal of Machine Learning Research, (02):3011-3040, 2013. URL http:/ /tap:://mmrr..//// /tmmmmml.ooorrr// papers/v14/tongl: htmll.
Tong, X., Feng, Y., and Li, J. Neyman-pearson classifica- tion algorithms and np receiver operating characteristics. Science advances, 165) 2018.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A.. Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023.
Varshney, N., Yao, W., Zhang, H., Chen, J., and Yu, D. A stitch in time saves nine: Detecting and mitigating hallu- cinations of Iims by validating low-confidence generation, 2023.
Vovk, V. Conditional validity of inductive conformal pre- dictors. In Asian conference machine learning, pp- 475-490. PMLR, 2012.
Wang, C., Liu, X., Yue, Y., Tang, X., Zhang, T., Jiayang, C., Yao, Y., Gao, W., Hu, X., Qi, Z., Wang. Y., Yang, L., Wang. J., Xie, X., Zhang. Z., and Zhang, Y. Survey factuality in large language models: Knowledge, retrieval and domain specifiiite 2023. URL htLhtpp:://axiv org/ /arg/ab//// 2310 0. 07521.
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. Can Iims express their uncertainty? an empirical evaluation of confidence elicitation in 1ls,, 2024. URL https:/ arxii. org/ abs
Yang, Z., Qi P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering, 2018
Zhang, H., Diao, S., Lin, Y., Fung, R.R. Lian, Q., Wang, X., Chen, Y., Ji, H., and Zhang, T. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.00 1.09677, 2023.
12

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
A. Derivation and Proof
Proof of Equation Assume that when (g, (g)) Po. ilq, M())) has CDF F. denote the a quantile of as F-1 (1 10a) a) inf(x E R F(z) 2 aa. Then we can show that:
(P (a.M (fla, M(1)) T(x) > a) PD(T(M) < <F -
(10)
Considering the property of the order statistics, we have that
PD(T(K) < <F a)) = PD
Σ
ig (0) .M(;; <
(11)
n20
(1 a) a aaaa (k)
<
(12)
j=k
where I(()) is the indicator function, defined as:
{1
1 if )( (0)
A(q::))) Qa
I(f(a 0 M(a (0)
Qa)
=
otherwise
Proof of Theorem 2.1. Theorem 2.1 follows from the definition of k.
Lemma 2 (Theorem 1 in (Skorski, 2023)). Suppose X1,..., Xn are i...d. continuous random variables with CDF F, denote
4(n 2k: +
2 log v0+ +
2k(n k + (n+ 1)2
2 8 log
=
3(n 1) +3
8
then
k
((X
n n S c 21-5.
Proof of Theorem 2.2. At first, simplify the notations in the proof, we argue that the function can be assumed be identity without the loss of generality. To see this, note that k does not depend the choice of i and T(ń T(ky)
I(H o > (H(T)) (n) with (H(T)) to be the k-th smallest order statistic of (H(T) i [no]]. therefore, fa is invariant if we replace f in Section 2 by H i). Consequently, without the loss of generality, we assume H is the identity function. Then the proof of Theorem 2.2 consists of three parts.
1) Firstly, we show that Ro(fa) is not much smaller than a. To see this, since we assume n( (4)) with ( ~ Po a continuous random variable, it follows from the definition of that
((F(T(B-)) 1 - a:) (P ((P(gg (flg, M())) > T(--)) a) >
Here we only consider the case where & 1, as will shown in Equation (13), it holds as long as nio not too small. Since h is deterministic given no. follows from Lemma 2 that
k-1 1o
(((T()))))
'x-1
S 6.
Therefore we have
h-1
Ct-1 <1 a. no +1 1
Denote the event E, as
E F(T()))
- no 1 + c
13

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
it follows from Lemma 2 that P(E,) - 8. Under E1, we know
k
P(T()) - no +
+ +& 1 a
n10 + +hh-1 +
Ek.
which implies
P (G.M(())=F M(g)) > T()) =1 1 1)F(((())
n0
Similarly, since
(( -1-))) 21 P ((F(T()) -
k
+ (x)
21-6,
21 -1-6,
+ 1
know
k
1 S
+ +
140 +
which concludes that
k 2 (1 a)mo.
(13)
Thus we have
a
fh-1 + 6 S V
log 3¹
no
and
(00. > T(k)) >
a cV
log 8"
(14)
n10
2) Secondly, we show T(k) is close to Ta Denote a' = =a a----- cv
a log f.- it follows from Equation (14) that under EJ, we
have
P (( (il,, M(1)) > Twx) > a' = PPG.)())= M()) > Ta)) (oa M())) > 5a + En),
so
((k) < To' + Eg.
Denote Eg
Es (G.. (fg (g)) Taa)) aa).
we know P(E2) 2 8. Under Ez, we have
P P(a.Mal)~ (1(a, M(g)) > ((x) a PI..M(i)) (plg. M(g)) > Ta)
S Pa..Mio)~PP M(g)) > Ta - Gr),
80
T(k) 2 Ta - Ex:
Then we conclude that
IT()) - Tal S Tax
Ta + En = cr.
3) Now we are able to control the excess risk of fa. For any (g. M()),, if we use Y 0 (resp. Y 1) to denote the model
M is uncertain (resp. certain) of q, and denote Py = P(Y 1) to be the marginal distribution for M to be certain, then
dPi
dPb (g, (q)) = n(q. ((9)((1 -pp))
(1 n(g, M()))),,
which is increasing in 1y(g, M (q)). Denote &ar 5. (1-po) if |n[[g M (g)) - Tal S Er + En and To + + en < 1, then (1-Ta)p
dP (g. M()) - fa. -
(1 - pyy ( + +G,)
dP
Py (1 Ta C, C,,) )
14

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Then under Ej Eg, control the excess risk
Ri(fa) - R1(fa)
dP
(g, M())) [[i(g, M(g)) & Twa) - I(n(a, M()) S Fal)
= B()) Po dPo
dPI
) (I M()) - <T()) Iinigg (g)) S ra) T(figg, M(a))) S sTa)) Tinil, M(1)) s TTa)
-M Po
(g. (q)) - Ex
dPb
+ (a.
dPI
(q. (g)) - Ea |LoCe M(2) s Ta,) - I(n(g. M()) S Ta)
B() Po
dPo
t Ro(sa) Ro(ia)
dP
(g. M(g)) - -Ea)
M())~ dPo
ral I c + + cla
log
no
Pyy[1 Ex En )2
(1 - (e) tep)
Galer + te) + Eag
110
a
log
8
Proof of Theorem 3.1. Note that once we show Do I iid.
Po. then similar to Theorem 2.1,
Pp (g) (fa(g, M(g)) 1) s 2
Taking expectation with respect to concludes the result.
remains to prove Do I iid. Po. To this end, suffices show (q. (Q) S w(g, M(4)) ~ Py.M(allu=o Po with (q, (1)) ~ Po. For any measurable set C C Q A, the conditional distribution of (g. (g)) s w(g. M()) can be expressed as
P((, E CIU s wlq, M())) P((g,M(a)) C,U S w(a M()))
IPU S M()))
M(l. ((1, M(a)) EC)
B
E ((g)
M ((g, M())) E EC),
where we have use the facts that P(U S w(g. M M(4)) w(g.
A. w(1, M(g)) and
(a) w(g. M(q)) M(g)) E CC P P(a.M(a))~PP (1g M(g)) EC).
w(g
B. More Related Works
Uncertainty Quantification and Confidence Calibration. Our work relates to a line of work on uncertainty quantification (UQ) for LLMs, as we can employ these functions to assess models' ability to reliably give an answer. Predictive entropy that measures the entropy of the model's predicted token distribution has been used as a simple baseline for UQ in LLMs (Braverman et al., 2019). Kuhn et al. (2023) introduced Semantic Entropy, which incorporates linguistic invariances to measure uncertainty. Most recently, Nikitin al. (2024) introduced Kernel Language Entropy (KLE), which defines positive semidefinite unit trace kernels and quantifies uncertainty using the von Neumann entropy. Lin et al. (2024) identifies that existing methods treat all tokens equally when estimating uncertainty and proposed simple supervised approach for uncertainty estimation in black-box L.Mss using labeled datasets. Duan et al. (2024) proposes jointly shifting attention to more relevant (SAR) components. Besides, recent research on confidence calibration for LLMs has explored several innovative approaches. For example, Tian et al. (2023) elicits verbalized confidences to improve calibration. Huang et al. (2024) proposes confidence elicitation methods for long-fom form generations. Multicalibration (Detommaso et al., 2024) aims to ensure L.MM confidence scores accurately reflect the true likelihood of predictions being correct. These works are
15

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
complementary to ours, as our contribution is a meta-algorithm that works with any uncertainty quantification method to serve score functions. Future developments in this line of work can greatly improve the performance of our framework.
Conformal Prediction. Conformal prediction enables the construction of confidence sets that contain the true outcome with a specified probability (Shafer & Vovk, 2007; Angelopoulos & Bates, 2022; Barber et al., 2023). It has been successfully applied to various black-box machine learning models (Angelopoulos & Bates, 2022) but has limited application in language models (LMs). Specifically, Kumar et al. (2023) provides conformal guarantees on multiple-choice datasets. C-RAG (Kang et al., 2024) provides conformal risk analysis for RAG models and certifies an upper confidence bound. CLM (Quach et al., 2024) extends conformal prediction for LLM generations and provide coverage guarantees. TRAQ (Li et al., 2024) uses conformal prediction to construct prediction sets both for retrieved sets and prediction sets to provide guarantees for RAG. Conformal Factuality (Mohri & Hashimoto, 2024) enables the application of conformal prediction in improving model performance. However, traditional CP methods in LLMs focus primarily on coverage guarantees without differentiating between correct and incorrect samples, thereby lacking explicit error rate controls essential for hallucination detection. In contrast, FACTTEST differs from those works in that aims to evaluate the model's ability to answer correctly, detect hallucinations and explicitly control both Type I and Type II errors.
C. Implementation Details
C.1. Details about Baselines
We provide detailed explanation of the settings used for the baseline.
SAPLMA:
We follow the original settings and implement SAPLMA (Azaria & Mitchell, 2023) to serve as a training-free baseline. SAPLMA trains a classifier to predict statement truthfulness. We evaluate models on questions that the model's ss answer is considered as truthful. The corresponding results are shown in Appendix D.5.
SeI[CheckGPT: We implement SeIPCheckGPT with NLI (Manakul et al., 2023) as recommended by the authors. utilizes Natural Language Inferencce (NLI) model to predict entailment contradiction. We sample five answers and the model will output a probability of contradiction from 0 to 1. We evaluate the base models on questions with the contradiction score less than 0.5.
R- R-Tuning: We follow the settings in the original paper (Zhang et al., 2023): We first construct a refusal-aware dataset by adding prompt 'Are you sure you accurately answered the question based on your internal knowledge?' and the corresponding 'Sure' or 'Unsure' to each question-answer pair, and then finetune the model on this dataset. We evaluate the finetuned model on questions that the model is certain.
C.2. Details about Datasets
We conduct our experiments on four datasets and follow the same train/test split in Zhang et al. (2023), which are described as follows.
ParaRel (Elazar et al., 2021): This dataset comprises factual knowledge with various prompts and relations initially designed for mask prediction. It is utilized to evaluate the model's ability to comprehend paraphrased relational facts. To adapt ParaRel for autoregressive models, Zhang et al. (2023) reformatted it into a prompts with different templates but identical entities were removed for our question-answeriigg task, resulting in 23,133 prompt-answer pairs across 31 domains. Zhang et al. (2023) divided ParaRel into two subsets: the first 15 domains serve as in-domain data, and the remaining 16 domains as out-of-domain data (13974 samples). The in-domain data is further split equally into training and testing sets, consisting of 5575 and 5584 samples.
format. Duplicate
HotpotQA (Yang et al., 2018 2018): It is a question-ansswrrigg dataset that necessitates complex reasoning across multiple documents. We evaluate the model by providing the relevant context documents and questions to assess its ability to generate correct answers. The development set is used as the testing set for our evaluations. The training set contains 10K samples randomly selected from the original dataset while the testing set contains 7405 samples.
WiCE (Kamoi et al., 2023): It is a natural language inference (NI.)) dataset focused on textual entailment. Rach data sample consists of an evidence statement and a claim, and the model must determine whether the evidence supports, partially supports, or does not support the claim. We utilize this dataset as multiple-chor choice questions with three options for each question. The training and testing sets contain 3470 and 958 samples, respectively.
FEVER (Thorne et al., 2018a): FEVER consists of claims paired with supporting evidence from Wikipedia. Each claim is classified as SUPPORTED, REFUTED, or NOT ENOUGH INFO. This dataset is employed to assess the
16

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
models' capability verify the factual accuracy of statements against Wikipedia documents. We utilize FEVER multiple-choice NLJ task with three options for each question: (A) SUPPORTED, (B) REFUTED, (C) NOT ENOUGH INFO. The training set contains 10K samples randomly selected from the original dataset while the testing set contains 9999 samples.
Detailed information about the original datasets and the data preprocessing procedures can be found in Zhang et al. (2023). In Figure 5, we illustrate the distribution of correct and incorrect data within the constructed datasets Do, Dr.
C.3. Details about Score Functions
We implement our framework with three entropy- certainty functions. Details are described as follows.
Vanilla Entropy: The frequeney of predicted answer M (g), in Equation. 8 is calculated by T where m is the number of times M (q), exists in & generations.
Semantic Entropy: Semantic entropy is an entropy which incorporates linguistic invariances created by shared mean- ings (Kuhn et al., 2023), which is computed by the probability distribution over meanings.
SE M()) = - Ex(cla log p(ga())))= = (( a)) log
[x((e))
(15)
Ec where e represents possible meaning-class and p([g) is the probability of the entire answer sequence, that is, the product of the conditional probabilities of new tokens given past tokens. This can be approximated by:
En
SE M()) - Σ logp p(C, 4), flg M(4)) = SEE, M())..
(16)
We follow (Kuhn et al., 2023) to estimate the expectation of 16 given that cannot have access to all possible c. We query M k times and divide the answers into semantic classes C' based on semantic equivalence.
Notably, for multiple-choie datasets including WiCE and FEVER, the outputs are among three choices. this case, we view different tokens having different semantic meanings, and the semantic entropy is thus reduced predictive entropy.
Kernel Language Entropy: Kernel language entropy (KLE) is a generalization of semantic entropy (Nikitin et al., 2024), providing more detailed uncertainty estimates by considering pairwise semantic dependencies between answers or semantic clusters. It quantifies uncertainty by constructing a semantic kernel from the model's & generated answers and computing its von Neumann entropy. Specifically, for given input q, we generate h responses, build positive semidefinite semantic kernel Kgem that captures the semantic relationships among these answers, and then calculate the von Neumann entropy (VNE) of this kernel. The KLE can be defined as:
KLE(g, M()) = VNE(K sem) = [[Ksem log Ksmm), M()) M(E)
(17)
where, K sem the semantic kernel which can implemented from semantic graphs over the LLM outputs.
C.4. Details about FACTTESTO
In order to approximate the density ratio, we randomly split 1000 samples from ParaRel-OOD as validation samples and the remaining 12k samples as testing samples. We then utilize the supervised identification strategy to divide the validation samples into D and D and the training dataset into Do and Dj. We extract the features from the questions in D/, Do by a Tfid/ectorizer,, and label them as 1 (target data) and 0 (source data). We then utilize logistic regression to train binary classifer and use the predicted probability to approximate density ratios.
D. More Experiemnt Results
In this section, we provide more experiment results, including experiments with more significance levels, answer rate analysis, more error control analysis, experiments with more base models and certainty distribution visualizations.
17

---

FACTTEST:
Factuality Testing in Large Language Models with Statistical Guarantees
D.I. Ablation Study
is demonstrated in Sec. FACTTEST can also be formulated with PAC-style conformal prediction (CP) conditioned on y 0. As illustrated in Sec. 4, directly checking whether outputs fall within the conformal prediction set does not work for hallucination detection. In this part, we implement two variants with conformal prediction that could control Type errors to some extent and compare them with FACTTEST to further illustrate the efficacy of our method. Unlike previous works that consider prediction sets for LLM generated outputs, we consider conformal prediction sets around a binary classifier that takes X = (q, .M (q)) as inputs and outputs whether the generated answer is correct (y = 1) or incorrect (y = 0). The model will abstain from answering the question if = is included the prediction set. We then set the non-conformity score [X, y) to be Sigmoid(fi) when y = and (1g when y
Standard Conformal Prediction. The prediction set satisfies marginal coverage:
P( E Cstandard ((ea)) 2 1 -
The standard conformal prediction set is given by:
(_Xiest, 2) Cstandard (Xrest) =
4).
where
[ [N + 1)(1 (al1-a)
-a)
i
Quantile
N
Class-conditonaal Conformal Prediction. The prediction set satisfies class- cans-conniiional coverage:
P(yysst E CClasswise (Xust) Yeea 0) N 1
Suppose there are no samples where y O. The class-conditional conformal prediction set is given by:
CClaswise (wea) = (y 8 [jest: y) S (()))
where
( + 1)(1 a)
g0)) = Quantile
{8; Lhfe
74g
Table 4: The accuracy performance (%) of FACTTEST on ParaRel. The for FACTTEST and two variants are all set 0.05, The percentages inside the parentheses are the Type error. We use 15-generation SE score function.
Dataset
Base Model
Class-Conditional CP
FACTTEST-SC15
OpenLLAMA-3B
65.54 (0.05)
67.26 (0.05)
ParaRel
OpenLLaMA-7B
63.20 (0.05)
65.23 (0.04)
OpenLLaMA-IIB
71.45 (0.05)
73.09 (0.03)
OpenLLAMA-3B
83.17 (0.06)
83.90 (0.05)
FEVER
LLaMA-7B
31.71 (0.06)
33.27 (0.05)
LLaMA-13B 13B
50.41 (0.07)
52.23 (0.05)
Results. Standard CP leverages the full calibration dataset D while Class-conditional CP as well as FACTTEST exclusively utilize Do. We observe that standard CP exhibits excessive conservatism due its requirement to satisfy coverage guarantees for both y 0 and y = -1 simultancously. This often results in rejection of all the questions, leading to 0% predictive accuracy in practice. Table. demonstrates the accuracy and Type error performance of FACTTEST compared with class-conditional CP variant. Class-conditional CP aims at controlling the average type I error, while FACTTEST enforces stricter control by bounding the 5-quantile of all type errors. Therefore, FACTTEST provides statistically more stringent guarantees than conditionl CP, as demonstrated by its lower empirical Type error rates while maintaining competitive accuracy.
D.2. More significance levels.
Table 5 presents the accuracy performance of FACTTEST comparison with base pretrained models at a significance level of a 0.10. Similarly, Table 6 reports the corresponding Type error rates under the same significance level. The results
18

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Table 5: The accuracy performance (%) of FACTTEST compared to Pretrained models on question-ansswering and multiple- choice datasets using a significance level of a = 0.10.
Dataset
Model
Pretrained
FTEST-Ve,
FTEST-VE10
FTEST-Ve1s
FTEST-se,
FTEST-Se10
FTEST-Se1s
FTEST-Klejs
OpenILLaMA-3.
36.66
60.54
67.22
67.10
61.01
62.32
63.05
75.51
ParaRel
OpenLLaMA-77B
40.38
74.92
77.84
76.89
69.00
68.78
64.97
75.36
OpenLLaMA--13B
42.21
77.37
75.17
79.25
68.93
68.45
68.82
79.55
25.72
50.81
49.84
51.68
45,41
45,23
46.88
52.70
HotporQA
OpenLLaMA-7B
28.63
56.06
56.23
55.77
51.10
51,63
52.33
56.73
LLaMA-13B
30.83
51.49
51.45
51.61
53.42
53.12
55.38
53.34
OpenLLaMA-331
64.72
64.40
76.27
61.54
64.71
WICE
OpenLLaMA-71
72.96
63.77
57.32
83.33
85.71
74.19
LLaMA-13B
56.89
63.33
50.00
57.14
75.00
67.44
77.42
OpenLLaMA-3B
39.74
60.24
53.06
52.00
80.71
82.00
82.29
FEVER
LLaMA-7B
35.99
43.92
43.33
47.73
28.69
31.49
32.82
32.15
38.74
42.48
46.79
51.95
53.01
50.92
Table 6: The Type error of FACTTEST question-answering and multiple-choice datasets, with significance level
0.10.
Datasel
Model
FTEST-ve;
FTEST-Ve1o
FTEST-Ve1s
FTEST-SeS
FTEST-se10
FTEST-SE15
FTEST-kless
OpenLLaMA-3B
0.0455
0.0732
0,0783
0.0851
0.0865
0.0905
0.0795
ParaRel
Openl.LaMAA78
0.0225
0.0240
0.0348
0.0799
0.0886
0.0829
0.0781
0pa-11aaMA-1338
0.0192
0.0226
0.0325
0.0849
0.0706
0.0589
0.0709
OpenLLaMA--3B
0.0276
0.0585
0.0521
0.0660
0.0678
0.0651
HotpotQA
Openl.LaMA-7B
0.0295
0.0597
0.0637
0.0631
0.0643
0.0616
0.0590
1.1MM--3B
0.0222
0.0556
0.0675
0.0611
0.0675
0.0503
0.0667
OpenLLaMMA-3B
0.0325
0.0621
0.0414
0.0443
0.0355
0.0325
WICE
OpenLLuMA-77B
0.0694
0.0965
0,1151
0.0347
0.0154
0.0308
L.1MA-13B
0.0266
0.0532
0.0799
0.0169
0.0338
0.0169
OpenLLaMAA3BB
0.0164
0.0418
0,0600
0,1039
0.1053
0.1042
FEVER
LLaMA-7B
0.0598
0.0617
0,0556
0.0928
0.1027
0.1091
13BB
0.0172
0.0828
0.0709
0.0944
0.1059
0.1136
Table 7: The answer rate and accuracy performance (%) of FACTTEST-L. The number in parenthese is Answer Rate, which means the percentage of willingly answered questions.
Dutaset
Model
Finetuned
R- R-Tuning
FTEST-t (a 0.16)
FTEST-t (a 0.10)
FTEST-t (a 0.05)
OpenLLaMA-3B
61.73 (00%) ) 87.42 (37%)
89.91 (46%)
92.73 (31%)
94.26 (17%)
ParaRel
LLaMA-7B
67.73( 100%
89.65 (42%)
92.76 ( 47%)
95.04 (31%)
96.01 (18%)
O
65.56 (10%% ) 67.19 (11%)
92.58 ( (38%)
94.88 ( (36%)
97.82 ( (33%)
FEVER
LLaMA-7B
66.24 (00%) )
66.19 (49%)
95.41 (28%)
95.83 ( (24%)
96.79 (66%
show that Type I error remains effectively controlled with the adjusted While the accuracies 0.10 are slightly lower than those 0.05, FACTTEST continues to significantly outperform base models and maintains lower Type II error rate.
D.3. Answer Rate Analysis
Table 7 presents the answer rate and corresponding accuracy performance (%) of FACTTEST-t in comparison with baseline methods across multiple datasets and models. The findings demonstrate that FACTTEST-t consistently achieves higher accuracy while effectively managing the answer rate through varying significance levels (a). Specifically, FACTTEST-L with a = 0.15 answers 47% questions on ParaRel and acheives 76% accuracy, outperforming R- R-Tuning, which answers 42% of the questions with an accuracy of 99.65%. Similarly, FACTTEST-t maintains superior accuracy performance on FEVER compared to baseline models while managing the answer rate through different significance levels.
19

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Table The accuracy performance (%) of FACTTEST compared baselines on question-answerpnn answering and multiple-choo datasets using significance level a 0.05. For brevity, FACTTEST is abbreviated as FTEST.
Dataset
Model
Base
SelfCheckGPT
SAPLMA
FTEST-Vé15
FTEST-S015
FTEST-Kle1s
FTEST-saplma
OpenLLaMA-3BB
36.66
53.60
67.33
67.28
67.26
78.45
84.77
ParaRel
OpenLLaMA-7B
40.38
60.05
76.18
80.29
65.23
76.83
88.99
OpenLLaMA--138
42.21
59.62
69.62
79.41
73.09
83.84
88.16
OpenLLaMA-3BB 3B
25.72
36.42
25.13
53.75
52.66
55,35
31.96
HolpotQA
Op
28.63
39.16
24.00
60.67
56.56
60.66
31.96
LLaMA-13B
30.83
41.78
38.91
49.74
60.69
54.49
38.69
Op
64.72
66.36
64.40
68.18
66.67
65.67
WICE
OpenLLaMA-77B
72.96
75.00
72.50
47.37
90.00
73.33
LLaMA-13B
56.89
57.39
57.67
44.44
90.00
41.67
OpenL
39.74
41.97
93.21
41.72
83.90
93.21
FEVER
LLMMA-7B
35.99
40.89
74.05
51.38
33.27
63.88
LLMMA-13
32.15
41.25
46.07
52.23
49.81
D.4. Error Control Analysis
Type II Error. Figure 6 shows the FNR-a curve. In this paper, we minimize the Type II error while enforcing the upper bound of Type error at a. The performance of Type error cannot be fully controlled, which mainly depends on how well the score function can quantify model's ability to answer correctly.
Figure illustrates the error control analysis on HotpotQA, highlighting FACTTEST's capability control the Type I error effectively. Figure 8 presents the Type error calibration results of FACTTEST-L across four datasets, complementing the experiments discussed in Section 5.3.
D.5. Additional Results with More Baselines
We include SAPLMA additional baseline and further exploit its classifier logit as classifier-based scoring function, which denote FTEST-sapIma. For FTEST-SapIma, remaining samples for calibration; for the SAPLMA baseline, we train on the full dataset. Table 8 compares FACTTEST 's performance against all the baselines. Note that we omit SAPLMA's accuracy on FEVER with LLaMA- 13B, as it classifies all predictions as incorrect that setting.
train the classifier half of the training set and use the
D.6. Additional Results with More Base Models
To further verify the effectiveness of FACTTEST, we additionally evaluate the performance of FACTTEST on more base models, including more pretrained models, instraction-tuned models and black-box models.
Table 9: The accuracy performance (%) of FACTTEST on four question-answering datasets using Mistral-7B as the base model. The significance level for FACTTEST is set to 0.1. The percentages inside the parentheses are the Type error.
Dataset
Base
SeIICheckGPT
FACTTEST-veis
FACTTEST-se1
FACTTEST-kles
ParaRel
39.79
57.01 (0.25)
65.63 (0.07)
70.20 (0.08)
72.78 (0.08)
HotpotQA
36.48
46.01 (0.46)
61.81 (0.06)
63.06 (0.05)
65.59 (0.05)
FEVER
35.47
41.76 (0.05)
22.99 (0.08)
51.05 (0.08)
WiCE
55.85
56.24 (0.47)
68.81 (0.08)
68.64 (0.08)
Pretrained Model. In Table 9, employ Mistral-7B (Jiang et al., 2023) the pretrained base model to supplement the
results in Table 1.
Instruction-funel Model. In Table 10, we employ LLaMA-3 3.2-3BB -.ntruct (Dubey et al., 2024) and Tulu2-7B (Ivison models.
et al., 2023) as base models to evaluate the performance of FACTTEST on instruction- Black-box Model. In Table 11, we employ Claude-3.5-Somna (Anthropic, 2024) and GPT-4o (OpenAI, 2024b) as base models to evaluate the performance of FACTTEST, with OpenI.LaMA-33. serving as the open-source model to calculate
20

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Table 10: The accuracy performance (%) of FACTTEST using
and Tulu-2-7B as base models. The set to 0.10. The percentages inside the parentheses are the Type error.
Dataset
Model
Base
FACTTEST-se15
FACTTEST-kleis
ParaRel
1amma-3.2-BB-Inttuee
39.34
72.79 (0.08)
80.01 (0.08)
ParaRel
Tulu-2-7B
43.89
75.47 (0.06)
78.49 (0.07)
HotpotQA
1lama-3.2-3B-Instrue
33.40
57.75 (0.06)
60.38 (0.07)
HotpotQA
Tulu-2-7B
32.91
53.54 (0.05)
45.89 (0.10)
WICE
Llama-3.2-3B-Instuu
55.11
75.16 (0.09)
WiCE
Tulu-2-7B
57.20
63.22 (0.08)
FEVER
1lama-3.2-3B-Insttue
33.33
68.48 (0.10)
FEVER
Tulu-2-7B
47.87
69.40 (0.09)
Table 11: The accuracy performance (%) of FACTTEST on ParaRel using Openl.lama778 an open-source model. The significance level is set to 0.1. The percentages inside the parentheses are the Type I error.
Model
Base
SeIFCheckGPT
FACTTEST-SC15
FACTTEST-kless
Claude-3.5-Sonme
58.25
58.96 (0.92)
73.29 (0.08)
79.86 (0.08)
64.23
65.92 (0.86)
76.87 (0.07)
80.01 (0.08)
GPT-4o
66.39
69.71 (0.83)
80.70 (0.07)
82.76 (0.08)
scores.
Our findings reveal that incorporating FACTTEST significantly reduces hallucinations, achieving average accuracy improvement of 24% while maintaining Type errors below the specified a.
D.7. Additional Results with More Score Functions
Table 12: The accuracy and answer rate performance (%) of FACTTEST with significance level 0.1 using 5-generation and 10- 10-generation KLE as score functions.
Dataset
Model
Base
FACTTEST-kles
FACrTEST-kleno
ParaRel
OpenLLAMA-3B
36.66
71.65 (18%)
74.72 (20%)
ParaRel
OpenLLLAMA-7B
40.38
72.99 (20%)
75.90 (20%)
Hotpot
OpenLLAMA-3B
25.72
52.34 (11%)
51.82 (12%)
Hotpot
OpenLLAAMA-7B
28.63
52.45 (11%)
55.92 (13%)
In this section, we present results using additional score functions, including FACTTEST-kle5 and FACTTEST-Kle10, to complement the findings in Table 1.
Besides, we implement FACTTEST-Sogpt to illustrate how our framework can integrate various uncertainty or hallucination quantification methods. For FACTTEST-Sogpt, we utilize the negative value of the SelfCheckGPT-NL. score as the score function. The accuracy and Type error performance for this implementation are summarized in Table 14.
Moreover, to highlight that our framework extends beyond uncertainty -based approaches, we develop classifier-basse variant, FACTTEST-CIS,, and compare it with uncertainty-base ones in Table 15. This variant employs random forest classifier trained on hidden layer activations from both the question and answer, along with probabilistic statistics of the generated answer, to predict the correctness of achieves competitive accuracy, maintains Type I error below the specified threshold, and demonstrates improved Type error rates compared to uncertainty-batt score functions.
answer pairs. The results indicate that FactTest-cls
21

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Table 13: The Type Error of FactTest with a significance level a 0.1 using S-generation and 10-generation KLE as score functions.
Dataset
Model
FACTTEST-kles
FACTTEST-klero
ParaRel
OpenLLAMA-3B
0.0783
0.0778
ParaRel
OpenLLAMA-7B
0.0880
0.0787
Hotpot
OpenLLAMA-3B
0.0656
0.0643
Hotpot
OpenLLAMA-7B
0.0643
0.0654
Table 14: The accuracy and Type Error performance of FACTTEST-scEDD evaluated on ParaRel with OpenLLaMA-3B serving as the base model.
0.05
0.1
0.2
Accuracy
61.82
62.92
59.26
Type error 0.04
0.09
0.17
D.8. Additional Analysis about Unwilling Answered Questions
We perform additional analyses to evaluate the effectiveness of FACTTEST. Table 16 presents the performance of base models on subsets of questions that the model is either unwilling or willing to answer on ParaRel, using FACTTEST-Kle15, Notably, the results for the "Willing" subset correspond directly to the performance of FACTTEST-Kle15, The results show that accuracy on unwilling samples is significantly lower than on the entire dataset and willing samples, highlighting FACTTEST's capability to decline unknown questions effectively.
D.9. Score Distribution
Figure 9 represents the certainty distributions of correct subset and incorrect subset using semantic entropy as the score function.
22

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Table 15: The accuracy, Type I Error and Type II Error performance of FACTTEST-cls evaluated on ParaRel with 0.05.
Base Model
Metric
FACTTRST-ve15
FACTTEST-Se15
FACTTEST-cks
Accuracy (%)
67.28
67.26
85.13
OpenLLAMA-3B
Type error
0.05
0.05
0.04
Type error
0.86
0.85
0.35
Accuracy (%)
80.29
65.23
89.50
OpenLLAMA-7B
Type сгrоr
0.01
0.04
0,03
Type error
0.92
0.87
0.44
Accuracy (%)
79.41
73.09
88.37
Ope
Type error
0.03
0.03
0,04
Type crror
0.91
0.87
0.42
Table 16: The accuracy performance (%) of base models the subset of questions that the model is unwilling willing answer on ParaRel using FACTTEST-Kle15. The is set to 0.1.
Model
Base
Unwilling
Willing
OpenLlama-3B
36.66
27.90
75.51
OpenLlama-7B
40.38
32.93
75.36
OpenLlama-13B
42.21
32.81
79.55
23

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
Correct
Incorrect
ParaRel-7B
ParaRel-13B
Pa
63%
61%
58%
37%
39%
42%
HotPotQA-3B
HotPotQA 7B
HotPotQA 13B
65%
61%
57%
35%
39%
43%
WICE-3B
WiCE-7B
WiCE-13B
32%
23%
39%
68%
61
FEVER-3B
FEVER-7B
FEVER- -1B
60%
62%
67%
40%
38%
Figure 5: The certain and uncertain data distribution the originated datasets obtained from supervised identification strategy. The title of each sub- figure consists of the dataset name and the size of the pre-trainet pre-trained model used evaluate.
24

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
FactTest Ve
1.0
1.0
FactTesst Vi
Type II Error
FaciTest
FactTest vegs
Type Error
Type Error
0.8
Factrest ve
FactFest-
0.6
FactTest Se1s
0.8
0.6
FactTest 5813
0.4
FactFest -
0.4
0.4
-ees
FactTest -eto
0.2
FactTest
0.2
0.4
0.6
0.8
0.2
0.4
0.2
0.8
0.8
Significance Level (a) ParaRel-3B
Significance Level
Significance Level a
(h) WiCE-3B
(c) FEVER-3B
Figure 6: The Type error, or FNR, of FACTTEST given different significance levels. The caption of each sub-figure consists of the dataset name and the model size.
FactTest - Se15
FactTest - SS
Calibrated Typel
OpenLLaMA-36
OpenLL-MMA-36
0.8
LMAAL
Type Error
Type Il Error
LLB
LLMMA-133
0.6
0.4
0.4
0.2
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Significance Level
Significance Level a FNR-a HotpotQA
(a) FPR-a curve HotpotQA
Figure 7: The Type emror and Type emrr of FACTTEST given different significance levels HotpotQA using semantic entropy the score function. The legend represents the base pretrained model.
Typel
Type
HotpolDA
HotootGM
Type Error
Type Error
0.6
ParaRal
FEVER
FEVER
0.4
WICE
0.4
WICE
0.0
0.0
0.2
0.4
0.6
0.2
0.4
0.6
0.8
Significance Level (a) FACTTEST-I-3B
Significance Level
(b) FACTTEST-L-7B
Figure 8: The Type I error calibration results of FACTTEST-t given different significance levels using semantic entropy as the certainty function. The legend represents the dataset name.
25

---

FACTTEST: Factuality Testing in Large Language Models with Statistical Guarantees
ParaRel-3B, seis
ParaRel-78,
ParsRell13B.
Percentage the Data
3.4
Incorrect Correct
Parcentage Data
Incorrect
Percentage the Data
Incorrect
Correct
Certainty intervals
Certainty intervals
Cartainty intervals
Hotpat-3B, 5015
Hotpok-BB, se:s
Hotpot-13B. sess
Percentage the Data
Correct
Incortect
Percentage of the Data
Incorrect Correct
Plrcenttge of the Data
Incomert
Correct
30
Certainty intervals
Certainty intervals
Certainty intervals FEVER-13B, se1s
FEVER-3B, se1s
FEVER-7B, sess
Percentage of the Deta
Incorrect Correct
Percentage of the Dala
Percamaage of the Data
Incorrect
Correct
Correct
Certainly intervals
Certainty intervals
Cartainty Intervals WICE-13B,
WICE-3B, seys
WICE-7B, sets
Percenlage of the Data
Percemage the Deta
Incormect Correct
Parcenaage the Dela
Incorrect
Correct
3.11
Cortainty intervals
Certainty intervals
Cortainty interva
Figure 9: The certainty distribution of the training datasets on certain set and uncertain set. The title of each sub-figure consists of the dataset name, the size of the pre-trained model used to evaluate, the certainty function and the number of generations.
26


Reviews:
[{"review_id": "s7CmP85r5U", "reviewer": "Reviewer_yUoS", "summary": "The paper proposes FactTest, a framework to assess if an LLM can be factual with high probability correction guarantees. FactTest treats hallucination detection as a statistical hypothesis-testing problem. By doing this, it rigorously controls the maximum allowed Type I error rate ensuring hallucinations are not incorrectly classified as factual content at user-specified significance levels.. Additionally, under mild conditions, FactTest provides strong control over Type II errors, preventing truthful responses from being mistakenly rejected. The framework is distribution-free, making no assumptions about underlying data distributions or the number of human annotations. Moreover, FactTest is model-agnostic and applies equally well to black-box and white-box language models. It is also robust against covariate shifts, maintaining its effectiveness despite changes in input distributions. Extensive experiments conducted by the authors on question-answering benchmarks demonstrate that FactTest effectively detects hallucinations, enabling LLMs to abstain from answering uncertain questions, resulting in accuracy improvements of over 40%.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedbacks. We are glad that you acknowledge the validity of our framework and experimental design. Here we provide responses to your concerns one by one.\n\n> ***W1:Threshold of SCGPT***\n\nWe acknowledge that using a threshold of 0.5 for SelfCheckGPT may not be optimal. In our framework, SelfCheckGPT is more suitable as a score function where thresholds are derived with explicit control over Type I error, rather than as a direct baseline. FactTest aims to determine thresholds that guarantee control over Type I error for any score function. In fact, Table 13 in Section D.6 demonstrates the integration of SelfCheckGPT as a score function in our framework.\nFor a more appropriate comparison, we have implemented SAPLMA [1] as a baseline, a classifier specifically designed for hallucination detection that predicts whether an answer is correct. The accuracy (\\%) and Type I error (numbers in parentheses) performance are:\n\n| Dataset  | Base Model | SAPLMA       | FactTest-kle15 |\n| -------- | ---------- | ------------ | -------------- |\n| ParaRel  | 36.66      | 67.33 (0.24) | 78.45 (0.03)   |\n| HotpotQA | 25.72      | 25.13 (0.02) | 55.35 (0.03)   |\n\n[1] Azaria, Amos, and Tom Mitchell. \"The internal state of an LLM knows when it's lying.\"\n\n> ***W2:why accuracy, why not AUC-PR, AUCROC, or AURCC***\n\nThe primary aim of FactTest is rigorous control over Type I error by explicitly setting a rejection threshold for any score function (including UQ) with statistical guarantees. When determining whether an answer is correct using a given score function, a threshold must be selected to distinguish correct from incorrect responses, with each threshold corresponding to a specific Type I error rate. Rather than reducing Type I error, FactTest determines the threshold for any score function that can statistically ensure the Type I error below user-specified $\\alpha$. This operating point is crucial for high-stakes applications where accepting a hallucinated answer even rarely can be unacceptable.\n- Why not AUC-PR, AUCROC, or AURCC: These metrics evaluate the overall effectiveness of uncertainty scores across all possible thresholds. They do not capture performance at the fixed operating point (a specific threshold at a user-specified $\\alpha$) required by our method. These metrics would be more appropriate for evaluating the underlying score functions rather than the thresholding mechanism itself. Users could use such metrics to compare different score functions before applying our FactTest framework.\n- Why accuracy: Since we determine the threshold at a user-specified α and use it to reject answers considered incorrect, we naturally report Type I error (to verify our theoretical guarantees) and accuracy on willingly answered questions (to demonstrate practical utility). This approach directly measures the performance at our chosen operating point rather than averaging across all possible thresholds.\n\n\n> ***W3:type 1 and type 2 don't compare to any baselines***\n\nOur framework aims to control Type I error under a user-specified level α, meaning the threshold determined by FactTest depends on α. UQ methods typically output scores without rejection thresholds, while baselines like R-Tuning use fixed rejection rules, resulting in only one Type I error rate.\nOur Type I error figures (Figure 1) demonstrate performance with different α values, showing that Type I error can almost always be controlled below α using FactTest. Adding a baseline like R-Tuning would simply show a horizontal line, as it doesn't offer the flexibility of varying threshold levels that FactTest provides.\n\n> ***W4:more recent papers on uncertainty***\n\nWe appreciate this feedback and will incorporate discussion of more recent works in our next version to ensure comprehensive coverage of related methods, including but not limited to:\n\n\"[1] proposes a computationally efficient method that leverages semantic diversity to detect hallucinations. [2] revisits standard uncertainty metrics and highlights the limitations of naive entropy-based methods. [3] provides an effective and computationally efficient method to quantify predictive uncertainty in LLMs.\"\n\n[1] Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs\n\n[2] Rethinking Uncertainty Estimation in Natural Language Generation\n\n[3] Improving Uncertainty Estimation through Semantically Diverse Language Generation", "claims_and_evidence": "A lot of the theoretical claims are beyond my expertise. I have focussed some issues regarding the evaluation in the section below.", "methods_and_evaluation_criteria": "I find the evaluation to be less convincing. I've listed my concerns below.\n- In table 1, using selfcheckgpt-NLI to reduce hallucinations using a threshold of 0.5 severely weakens the baseline. The selfcheckgpt paper does not claim that the NLI classifier is calibrated. A threshold of 0.5 could potentially optimize for increased coverage (instead of reduced risk) hence leading to the values not being comparable. \n- If the authors claim FactTest can reduce both type 1 and type 2 errors, why are the only comparisons with the baseline using accuracy? Why not use a metric like AUC-PR [1,2], AUCROC[2], or AU risk coverage curve[3] like several other papers that measure uncertainty estimates for factuality? These metrics measure both type 1 and type 2 errors. \n- The other tables that measure type 1 and type 2 don't compare to any baselines, so it's hard to know if Facttest is actually more reliable at measuring hallucinations than other approaches. \n\n[1] Manakul, Potsawee, Adian Liusie, and Mark JF Gales. \"Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.\" arXiv preprint arXiv:2303.08896 (2023).\n\n[2]Fadeeva, Ekaterina, et al. \"Fact-checking the output of large language models via token-level uncertainty quantification.\" arXiv preprint arXiv:2403.04696 (2024).\n\n[3] Kamath, Amita, Robin Jia, and Percy Liang. \"Selective question answering under domain shift.\" arXiv preprint arXiv:2006.09462 (2020).", "theoretical_claims": "At a very surface level. Many of the theoretical claims made about the connection between NP classification and PAC-style conformal prediction are beyond my expertise.", "experimental_designs_or_analyses": "The experimental design seems valid.", "relation_to_broader_scientific_literature": "The authors have missed more recent papers on estimating uncertainty for factuality.", "essential_references_not_discussed": "Mentioned in evaluation.", "other_strengths_and_weaknesses": "As mentioned, I've only commented on my concerns regarding the empirical evaluations (the improper metrics, and lack of proper comparisons against baselines). \nI am happy to engage in more discussion with the authors about the concerns.", "other_comments_or_suggestions": "-", "questions_for_authors": "Questions listed as concerns regarding evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "78zTXhgCFa", "reviewer": "Reviewer_GerV", "summary": "The paper proposes a framework to provide a statistical guarantee of the correctness of an output generated by LLM. The methodology leverages hypothesis testing and provides guarantees about type I and type II errors. Experiments are conducted on question-answering datasets.\n\n## Update after rebuttal: I have increased my score after authors addressed my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and suggestions. We are glad that you acknowledge our work’s motivation, method and theoretical claims. Here we provide responses and additional experimental results to address your concerns.\n\n> ***W1:guarantees are dependent on training data pairs***\n\nIn Section 3, **we specifically address the covariate shift setting**, which allows the distribution of testing question-answer pairs to differ from the distribution of calibration pairs, provided that the oracle rule (optimal score function) for judging whether an answer is correct remains consistent.\nWe have **already incorporated experiments on OOD datasets** in Section 5.4, where Figure 4 demonstrates that FactTest-O (our extension for out-of-distribution domains) effectively controls Type I error and significantly outperforms baseline methods in terms of accuracy. These results empirically validate our theoretical extension to covariate shifts, showing that our framework maintains its statistical guarantees even when applied to question distributions different from those in the calibration set.\n\n> ***W2:More baselines***\n\nThank you for suggesting additional related works. We believe there may be a misunderstanding about the nature of our contribution. FactTest is **fundamentally a meta-framework that works with any score function** (including uncertainty quantification methods), deriving a statistically guaranteed threshold to determine whether to reject an answer, rather than proposing a new uncertainty scoring mechanism itself.\n\nRegarding the specific papers mentioned:\n\n[1] proposes a metric to evaluate self-consistency that could **serve as a score function within our FactTest framework rather than as a competing baseline**.\n\n[2] is a **benchmarking paper that does not propose a new UQ or hallucination detection method**, but rather evaluates existing approaches with new metrics.\n\n[3] proposes UQ methods for black-box hallucination detection which could **serve as score functions** for FactTest. We have implemented their UDEG approach as a score function within our framework (FactTest-udeg), with results shown below.\n\n[4] is **already implemented** in FactTest as score function, which we denote as FactTest-se in our experiments.\n\n[5] trains a classifier to predict statement truthfulness. We have implemented this as SAPLMA and compared it with FACTTEST. Additionally, we show how it can be used as a score function within our framework (FactTest-saplma).\n\nHere we provide experiments of FactTest compared with [5]. We also implement [5] and udeg in [3] as a score function within FactTest to further demonstrate how these UQ methods could work within our framework.\n\n| Dataset | Base Model | SAPLMA | FactTest-saplma    | FactTest-kle15 |\n| ------- | ---------- | ------ | --- | -------------- |\n| ParaRel | 36.66  |  67.33 (0.24) | 84.77 (0.04) |  78.45 (0.03)     |\n| HotpotQA  | 25.72 |  25.13 (0.02)   | 31.91 (0.04) | 55.35 (0.03)|\n\n\n| Dataset | Base Model | FactTest-udeg5 | FactTest-udeg10  | FactTest-udeg15 |\n| ------- | ---------- | ------ | --- | -------------- |\n| ParaRel | 36.66  |  44.8 (0.04) | 36.53 (0.04) |  36.71 (0.04)     |\n\nThese results further demonstrate how existing uncertainty quantification methods can be integrated into our framework, with FactTest providing statistical guarantees on Type I error control while maintaining or improving accuracy.", "claims_and_evidence": "No issues identified", "methods_and_evaluation_criteria": "No issues identified", "theoretical_claims": "No issues identified", "experimental_designs_or_analyses": "No issues identified", "relation_to_broader_scientific_literature": "The work is related to hallucination detection and statistical guarantees for a prediction.", "essential_references_not_discussed": "Please refer to mentioned baselines in the strength and weakness section.", "other_strengths_and_weaknesses": "Strengths\n\n- The work focuses on providing statistical guarantees while checking the factuality of the generated content.\n\n- The proposed method performs well for covariate shifts as well.\n\nWeaknesses\n\n- The guarantees are dependent on training data pairs. The generalizability of the proposed method could be more convincing with experiments on the OOD dataset.\n\n- Baselines are limited. To make the experiments more comprehensive, the baselines [1,2,3,4,5] could be included.\n\n[1] Chen, Chao, et al. \"INSIDE: LLMs' internal states retain the power of hallucination detection.\" arXiv preprint arXiv:2402.03744 (2024).\n\n[2] Vashurin, Roman, et al. \"Benchmarking uncertainty quantification methods for large language models with lm-polygraph.\" arXiv preprint arXiv:2406.15627 (2024).\n\n[3] Lin, Zhen, Shubhendu Trivedi, and Jimeng Sun. \"Generating with confidence: Uncertainty quantification for black-box large language models.\" arXiv preprint arXiv:2305.19187 (2023).\n\n[4] Farquhar, Sebastian, et al. \"Detecting hallucinations in large language models using semantic entropy.\" Nature 630.8017 (2024): 625-630.\n\n[5] Azaria, Amos, and Tom Mitchell. \"The internal state of an LLM knows when it's lying.\" arXiv preprint arXiv:2304.13734 (2023)", "other_comments_or_suggestions": "Please refer to previous section", "questions_for_authors": "Please refer to previous section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KDiS3jrVWo", "reviewer": "Reviewer_XhUn", "summary": "• Introduces FACTTEST, a framework that statistically evaluates LLM factuality with theoretical guarantees to detect hallucinations\n  \n• Formulates hallucination detection as hypothesis testing, controlling Type I errors (incorrectly classifying hallucinations as truthful) at user-specified significance levels\n\n• Leverages Neyman-Pearson classification techniques to define a score function measuring output correctness and determine appropriate thresholds using calibration datasets\n\n• Provides strong Type II error control (missing actual hallucinations) under mild conditions when the score function effectively captures output correctness\n\n• Extends the framework to handle covariate shifts through density ratio estimation and rejection sampling\n\n• Demonstrates across multiple QA benchmarks that FACTTEST enables models to abstain from answering uncertain questions, improving accuracy by over 40% compared to base models\n\n• Shows FACTTEST outperforms fine-tuned baselines while using less training data", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed reviews and questions you raise to help improve our paper!\n\n> ***W1: statistical significance testing***\n\nWe've conducted bootstrap analysis for 95% confidence intervals. Results available in: https://anonymous.4open.science/r/ICML_rebuttal-8905/icml2025__FactTest-2.pdf. Confidence intervals for all datasets will be included in our revision.\n\n> ***W2: Temperature could affect UQ***\n\nTemperature affects uncertainty quantification but **doesn't limit FactTest**, which controls type I error for any score regardless of temperature. Experiments with different temperatures (see link above) show that while accuracy varies, type I error remains below α.\n\n> ***W3: Score function selection impact***\n\nFor any score function $\\hat\\eta$, the type I error is always controlled below $\\alpha$ (Theorem 2.1). Type II error is nearly optimal with an inflation depending on $\\epsilon_\\eta=\\inf_{H\\text{ increasing}}\\|H\\circ\\hat\\eta-\\eta\\|_\\infty$, which is the deviation between the oracle score $\\eta$ and the used score $\\hat\\eta$ up to increasing transformation $H$ (Theorem 2.2).\n\n> ***W4: Multiple generations and latency***\n\nFactTest works with any score function, including single-generation ones. FactTest-cls in Table 14 (Section D.6) directly predicts answer correctness without multiple generations, with negligible runtime overhead. We expect more efficient score functions developed in the future to further enhance FactTest.\n\n> ***W5: sensitive to threshold choices***\n\nI am afraid there is a misunderstanding of our type I and II error guarantee. Our method selects threshold that guarantees type I error control below α with probability ≥1-δ for any user-specified α,δ. Moreover, our power analysis indicates that the threshold selected always possesses nearly optimal type II error as long as the score function captures the correctness of generated answers. Therefore, **the performance of the selected threshold is always guaranteed**.\n\n> ***W6: False negatives***\n\nFNs occur when score functions poorly separate correct/incorrect samples. This requires better score functions, **orthogonal to our contribution of providing statistical guarantees for any score function**.\n\n> ***W7:binary approach may oversimplify factuality***\n\nWe currently use binary accept/reject decisions at significance level α. A natural extension is to output the answer together with the largest confidence level $1-\\alpha$ for which the answer is rejected, providing a spectrum of factuality..\n\n> ***W8: when/why fails***\n\nWhile FactTest controls type I error for any score function, poor score functions increase type II error. With a constant function, FactTest would reject all answers to maintain type I error bounds.\n\n> ***Q1: long-form***\n\nExtending to long-form is our future direction. We see two approaches:\n1.Document-level analysis: We could apply our framework with score functions designed to measure overall factuality, treating the entire response as a single unit. \n2.Claim-level analysis: Formulate as a multiple testing problem and extends FactTest from controlling false positive rate to false discovery rate. For an answer with $m$ claims $c_1,...,c_m$, we will have:\n$H_{0,j}$: Claim $c_j$ is not correct.\n$H_{1,j}$: Claim $c_j$ is correct.\n\n> ***Q2: runtime overhead***\n\nInference Runtime of FactTest shown in https://anonymous.4open.science/r/ICML_rebuttal-8905/icml2025__FactTest-2.pdf\nSingle-generation function (e.g., FactTest-cls) add negligible overhead.\n\n> ***Q3: Open-domain settings with limited calibration data***\n\nIf the correct answers for in-distribution questions $P_{q,M(q),a}$ are limited, one possiblility is to incoporate OOD questions $\\tilde P_{q,M(q),a}$ for which correct answers are available, and then apply our method in section 3 to address the distribution shift in the calibration samples. Our method in section 3 only requires the oracle rule (optimal score function), of judging whether an answer is correct or not for a question, remains the same, $P(y=1|q,M(q))=\\tilde P(y=1|q,M(q))$. Then it remains to estimate the density ratio of incorrect answers $\\frac{dP_{q,M(q)|y=0}}{d\\tilde P_{q,M(q)|y=0}}$, which equals to\n$$\\begin{align}\n\\frac{dP_{q,M(q)|y=0}}{d\\tilde P_{q,M(q)|y=0}}=\\frac{dP_{q,M(q),y=0}\\tilde P_{y}(0)}{d\\tilde P_{q,M(q),y=0}P_y(0)}=\\frac{dP_{q,M(q)}\\tilde P_y(0)}{d\\tilde P_{q,M(q)}P_y(0)}\\propto\\frac{dP_{q,M(q)}}{d\\tilde P_{q,M(q)}}.\n\\end{align}$$\nSince the multiplicative constant $\\frac{\\tilde P_y(0)}{P_y(0)}$ in the density ratio only affect the efficiency of rejection sampling, provided the range $B$ of uniform random variables is large enough compared to $\\frac{\\tilde P_y(0)}{P_y(0)}$, we can estimate the density ratio $\\frac{dP_{q,M(q)}}{d\\tilde P_{q,M(q)}}$ based on merely unlabeled question-generated answer pairs $(q,M(q))$, which doesn't rely on the correct answers at all. Therefore, we believe our method is still applicable even if correct answers for in-distribution questions are limited.", "claims_and_evidence": "• The Type I error control claim is well-supported by both theoretical analysis (Theorem 2.1) and empirical validation (Table 2, Figure 1) showing error rates consistently below specified significance levels.\n\n• Performance improvement claims are generally substantiated: Table 1 shows significant accuracy gains (e.g., 39.74% to 83.90% on FEVER with OpenLLaMA-3B), though the headline \"40% improvement\" represents best-case scenarios rather than average gains.\n\n• Comparison with fine-tuned models is supported by Figure 3, demonstrating FACTTEST-t outperforms R-Tuning on HotpotQA and  FEVER. On the other hand, the improvement is significantly smaller on other datasets.\n\n• Black-box API applicability is validated in Table 3, though only tested on GPT-4o mini.", "methods_and_evaluation_criteria": "• The evaluation datasets (ParaRel, HotpotQA, WiCE, FEVER) represent a reasonable mix of question-answering and multiple-choice formats, spanning different knowledge domains.\n\n• The metrics (accuracy, Type I error, Type II error) directly align with the paper's goals and theoretical framework, providing clear performance indicators.\n\n• Comparing against both non-training methods (base models, SelfCheckGPT) and training-based approaches (R-Tuning) offers comprehensive benchmarking.\n\n• Testing across model scales (3B to 13B parameters) and architectures helps demonstrate generalizability.\n\n• The evaluation on black-box APIs is particularly valuable for real-world applicability where model internals may be inaccessible.", "theoretical_claims": "I did not look too closely at proofs. A cursory glance indicates they seem correct; the primary limitation is that Type II error control depends on score function quality, which is appropriately acknowledged in the paper.", "experimental_designs_or_analyses": "• **Type I error control experiments** (Table 2, Figure 1): Thoroughly validated across multiple datasets and models. The calibration process correctly maintains error rates below specified α thresholds, with results properly disaggregated by score function and model size.\n\n• **Comparison with fine-tuned models** (Figure 3): The experimental design fairly acknowledges data usage differences - FACTTEST-t uses only half the training data compared to R-Tuning and Finetune-All, strengthening performance claims.\n\n• **Covariate shift experiments** (Figure 4): Limited to a single dataset (ParaRel-OOD), which is adequate for proof-of-concept but insufficient to fully validate generalizability across different distribution shift types.\n\n• **Black-box API validation** (Tables 3): Sound approach of using open-source models to calculate certainty scores for closed models, though limited testing on question-answering tasks only.\n\n• **Limitations**:\n  - No statistical significance testing for accuracy improvements\n  - Temperature settings could affect uncertainty estimation\n  - Limited analysis of score function selection impact on overall performance\n  - No explicit runtime analysis for practical deployment considerations", "relation_to_broader_scientific_literature": "• **Hallucination detection methods**: FACTTEST bridges the gap between three existing approaches:\n  - Retrieval-based methods that require external knowledge bases\n  - Training-based approaches like R-Tuning that need extensive fine-tuning\n  - Uncertainty estimation techniques that lack theoretical guarantees\n\n• **Statistical learning theory**: Builds directly on Neyman-Pearson classification work \n\n• **Selective prediction**: Advances the line of work on LLM \"know when they don't know\" capabilities by providing formal statistical guarantees.\n\n• **LLM factuality benchmarks**: Uses established datasets (ParaRel, HotpotQA, FEVER, WiCE) that have been employed in previous factuality research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n\n• Works with any uncertainty quantification method as the score function, making it adaptable as better estimation techniques emerge.\n\n• Can be applied without fine-tuning, providing immediate benefits to existing models.\n\n• Could be incorporated into existing LLM systems.\n\n**Weaknesses:**\n\n• **Limited to classification/short-form settings**: All experiments focus on question-answering or multiple-choice tasks with short responses. No evaluation on long-form generation where hallucinations often manifest differently and may require different detection approaches.\n\n• Multiple generations required for uncertainty estimation could limit practical deployment in latency-sensitive applications.\n\n• Performance likely sensitive to threshold choices, but limited analysis of this sensitivity.\n\n• Potential for false negatives when correct answers differ syntactically from reference answers.\n\n• The framework's binary approach may oversimplify factuality, which often exists on a spectrum.\n\n• Qualitative analysis of when/why the method fails would strengthen understanding of its limitations.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The framework was evaluated only on QA and multiple-choice tasks. Have you investigated its applicability to long-form generation where hallucinations often manifest differently (e.g., factual inconsistencies within paragraphs)? For example, testing on the dataset from \"Long-form factuality in large language models\". \n\n2. What is the runtime overhead of FACTTEST compared to base models? The requirement for multiple generations (5-15) for uncertainty estimation could create latency issues in practical applications.\n\n3. In many real-world scenarios, comprehensive calibration datasets with known ground truth may not be available. How might FACTTEST be adapted for open-domain settings where correct answers for calibration are limited?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on the fundamental problem of hallucination detection and formulates the problem as hypothesis testing. The proposed method is model-free. The central claim of Type I appears to be well supported and the reviewers appreciated both the importance of the problem as well as the proposed contribution. Therefore, I recommend the acceptance.
Decision: Accept (poster)