Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space
Authors: Xin Qiu
Risto Miikkulainen
Venue: NeurIPS 2024 poster (2024)
Keywords: uncertainty quantification
large language models
trustworthy AI
Primary area: natural_language_processing
TL;DR: A framework is proposed to generate a semantic density metric for measuring confidence of LLM outputs.

Abstract: With the widespread application of Large Language Models (LLMs) to various domains, concerns regarding the trustworthiness of LLMs in safety-critical scenarios have been raised, due to their unpredictable tendency to hallucinate and generate misinformation. Existing LLMs do not have an inherent functionality to provide the users with an uncertainty/confidence metric for each response it generates, making it difficult to evaluate trustworthiness. Although several studies aim to develop uncertainty quantification methods for LLMs, they have fundamental limitations, such as being restricted to classification tasks, requiring additional training and data, considering only lexical instead of semantic information, and being prompt-wise but not response-wise. A new framework is proposed in this paper to address these issues. Semantic density extracts uncertainty/confidence information for each response from a probability distribution perspective in semantic space. It has no restriction on task types and is "off-the-shelf" for new models and tasks. Experiments on seven state-of-the-art LLMs, including the latest Llama 3 and Mixtral-8x22B models, on four free-form question-answering benchmarks demonstrate the superior performance and robustness of semantic density compared to prior approaches.

Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space
Xin Qiu
Risto Miikkulainen
Cognizant AI Labs
Cognizant AI Labs, San Francisco, USA
San Francisco, USA qiuxin.nju@gaate nju@gmail.com
The University of Texas at Austin, Austin, USA ristoBcognizant.com
Abstract
With the widespread application of Large Language Models (LLMs) various domains, con- cems regarding the trustworthiness of LLMs in safety-critical scenarios have been raised, due their unpredictable tendency to hallucinate and generate misinformation. Existing LLMs do not have an inherent functionality to provide the users with uncertainty/confidence metric for each response generates, making it difficult evaluate trustworthiness.. Although several studies aim to develop uncertainty quantification methods for LLMs, they have fundamental limitations, such as being restricted to classification tasks, requiring additional training and data, considering only lexical instead of semantic information, and being prompt-wise but not response-wise new framework is proposed in this paper to address these issues. Semantic density extracts uncertainty/confidenee information for each response from probability distri- bution perspective in semantic space. It has no restriction on task types and is "off-the-shell" for new models and tasks. Experiments on seven state-of the-art LLMs, including the latest Llama 3 and Mixtral-8x22B models, on four free-form question-answenee benchmarks demonstrate the superior performance and robustness of semantic density compared to prior approaches.
Introduction
Large language models (LLMs) have revolutionalizee many domains, such as conversational agents 34]. code generation [41), and mathematical discovery [39]. Given their ability for general reasoning and adaptability new tasks, LLMs are utilized increasingly safety-critical applications, including
healthcare 442 and finance (50) However, existing LLMs have an unpredictable tendency to hallucinate 30 leading to misleading information and risky behaviors. Responses are generated without quantitative indicators for their uncertainty/confidenec making it difficult to evaluate how trustworthy they are. As result, concerns have been raised about their safety [54], hindering a deeper utilization of LLMs in risk-sensitive domains [
Although significant resources have been invested in LLM development, leading to a rapid pace in new model releases, only little progress has been made in building an uncertainty quantification framework for LLMs. An ideal outcome of such a system would be a quantitative metric associated with each response that can be used as an uncertainty/confieees indicator. Users can then build on this metric to evaluate the trustworthiness LLM responses, e.g., establish an automatic system that triggers a warning if the response confidence is below pre-defined threshold.
Following this line of thought, several techniques have been proposed in the literature to extract the uncertainty/confidencc score from LLMs. In addition to the baselines that directly ask the LLM itself to evaluate its own answers 44 cntimation/aalirattin methods into LLMs 6 51 53). However, due to the nature of these traditional methods, they only work on classification problems, not free-form natural language generation (NLG)
21 one further step was to integrate traditional uncertainty
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tasks, which are more general and challenging. Another direction was to fine-tune the original model [25] or train an additional layer or classifier t 28 to output uncertainty/cofiddecee indicators for the responses. The main drawback is that these approaches are not "off-the-shelf" for new tasks and
models: additional task-specific training labels of the ground-truth confidence are needed, and the training needs to be done in model-specific manner, limiting their applicability.
Most importantly, prior work still treats LLM outputs as traditional auto-regressive predictions [29] i.e, the generated responses are simply handled as sequences of lokens/words, considering only their lexical uncertainty/coofideneee. However, due to the unique nature of free-form NLG, tokens that are lexically different may be semantically similar. In most LLM applications, decisions depend on the semantics of responses, and the same semantics can be stated using different words or sentence structures, leading to different lexical tokens. Therefore, umcertainty/confidenn in semantic space is a more essential indicator for trustworthiness of L.M responses than lexical uncertainty/cconidenee.
Semantic entropy [22 is the state-of-the-ar (SOTA) technique in semantic uncertainty 277. However, its current design has two intrinsic limitations. First, the returned uncertainty score is prompt-wise, i.e., the semantic entropy is calculated for each prompt, instead of each response. Considering that LLMs can generate diverse responses for the same prompt, using the same uncertainty score for different responses is problematic [26]. Second, semantic entropy only considers semantic equivalence, which is a binary one-cut measurement, i.e., it only returns whether two responses are considered semantically equivalent or not, without reporting how semantically different the responses are. It does not make use of the more fine-grained semantic differences among the responses, which encode information that can make uncertainty quantification more precise.
To fill these gaps, a framework is developed in this paper for new uncertainty/confiddmce metric, semantic density (SD), that can quantify the confidence of LLM responses in semantic space. Semantic density analyzes the output probability distribution from a semantic perspective and extracts a confidence indicator analogous to probability density. The proposed semantic density metric has the following advantages: (I) The returned confidence metric is response-wise, making it possible to evaluate the trustworthiness of each specific response; (2) it takes the fine- grained semantic differences among responses into account, which makes uncertainty quantification more precise; (3) it does not need any further training or fine-tuning of the original LLM; it is an offfthe-shelf" tool that can directly applied to any pre-trained L.LMs without modifying them; and (4) it does not pose any restrictions on the problem type; in particular, it works for general free-form generation tasks.
The performance of the semantic density metric was compared with six existing uncer- tainty/confidence quantification methods designed for LLMs across four question-ansseringg bench- mark datasets. All the approaches were tested on seven SOTA LLMs, including the latest Llama and Mixtral-8x2215 models. Semantic density performed significantly better than the alternatives across the board, suggesting that it forms a promising foundation for evaluating the trustworthiness of LLM responses. The source codes for reproducing the experimental results reported in this paper are provided at: https://gilubb.oo//oogizzza haw.:/iaaaaaaaalaaasaaaaniieeeee labw/semaatic--enssty--pe
2 Related Work
In the LLM literature, the terms "uncertainty" and "confidence" are used in a mixed manner. A number of studies [51 25
4 [6] treat uncertainty and confidence as two facets of single concept, i.e., lower confidence on one particular response corresponds to higher uncertainty (or lower certainty). Other studies try to further differentiate "uncertainty" from "confidence" [26] and only use
12
"uncertainty" to describe the entire output distribution instead of a specific response [22 I. Both perspectives fall in the same research area of "uncertainty quantifiaation/estiaattor [17] 26 9 16).
and the goals of most existing uncertainty/confidencc metrics are indeed the same: to provide a quantitative indicator of the trustworthiness of LLM responses. For better coverage and clarity, this paper uses "uncertainty quantification" " as general term to describe work related to the assessment of uncertainty confidence of LLMs, and "umcertainty/cooffidnee" to refer to multiple metrics with mixed term definitions. The proposed semantic density is thus an indicator of response-wist "confidence".
Although the main focus of the LLM community is still on developing new models with better performance, number of studies aim at measuring uncertainty/confidencc in LLMs. This section

---

summarizes their basic ideas and potential limitations, which are then targeted in the development of semantic density.
The first direction is to ask the L.M to evaluate the uncerrainty/confidenee of its own responses.
Tian et al. [44) performed an empirical study showing that the inherent conditional probabilities
poorly calibrated in existing LLMs with RLHF (reinforcement learning from human feedback), and that verbal confidence estimates provided by the LLMs are better calibrated. Kadavath et al. [2I] developed an approach where the LLM was asked to evaluate the correctness of its own answer, in the form of the probability "P(True)" that its own answer is correct. An additional "value" head was also trained to predict "P(True)", but it turned out not to generalize well to outt----iitiittionn datasets. In general, the performance of model self- oifllaaalttionn is not as good as other more advanced uncertainty quantification methods (22).
A second direction is to integrate traditional uncertainty quantification methods into LLMs. The effectiveness of temperature scaling [33 for calibrating the output token probabilities of LLMs was verified in Desai and Durrett [6] and Xiao et al. [5]]. Ye al. [53] utilize conformal prediction to quantify the uncertainty of LLMs. However, these approaches are limited to NLP classification tasks; in contrast, the proposed semantic density can be applied the more general and challenging
free-form generation tasks.
A third direction is to perform supervised learning, either by fine-tuning the original LLM or adding an additional layer/classifier, to create uncertaint//confienee indicators. For example Lin et al. [25 fine-tuned the GPT-3 model to verbally express its own confidence level, and Azaria and Mitchell
trained an additional classifier to return the truthful probability of each generated response, based on the hidden layer activations of the LLM. Liu et al. (8 proposed the LitCab framework, in which single linear layer is trained over the LLM's last hidden layer representations to predict bias term, and the model's logits are then altered accordingly to update the response confidence. Since these methods are model-specific and need additional task-specific training labels, they cannot be readily applied to new models and tasks. In comparison, as an unsupervised method, semantic density is oof-thhe-seel" for any new models and tasks, without the need for additional data or modifications to the original LLMs.
A common limitation of all the above methods is that they only consider the lexical information, and do not take into account the semantic relationships between responses. Yet semanties is critical in analyzing LLM outputs. As a fourth direction, semantic entropy 22 is a SOTA technique that quantifies semantic uncertainty for LLMs. It works by grouping the generated samples based on their semantic equivalence, and then generating an entropy-based indicator as an uncertainty metric. Although its performance is promising compared to the other approaches, as discussed in Section M it has two intrinsic limitations: (1) The generated semantic entropy is prompt-wise instead of response-wise, and (2) only one-cut equivalence relationship is considered during semantic analysis. The second issue was considered in recent follow-up [33] that tries to improve semantic entropy. However, did not resolve the first issue, and the proposed uncertainty metric is still prompt-wist limiting its utility when different responses are sampled given the same prompt. The proposed semantic density improves over these two aspects by providing a response-specife confidence indicator and analyzing the semantic relationship in fine-grained manner.
Besides the above four major directions, uncertainty quantification for LLMs has been explored from other angles as well. Lin et al. (26) tested several simple baselines, among which a straightforward measurement of semantic dispersion is robust in evaluating selective response generation. Similarly, Manakul et al. (30] proposed a framework with several variants that use sampling consistency for detecting ballucinations. These two studies assume a very restricted condition in which the original information extracted by these methods limited. Duan et al. 7 proposes a mechanism to shift attention to more relevant components at both token and sentence levels for better uncertainty quantification. Their approach was applied to prompt-wise uncertainty metrics only. The study by Ling et al. [27] focused on a specific in in-context learning setting, aiming to decompose the uncertainty of LLMs into that caused by demonstration quality and that caused by model configuration. These experiments were limited to classification problems. Similarly, Hou et al. [155 decomposed the uncertainty into prompt-wise approach. Xiao and Wang (52) studied
sampling likelihoods of each response are not used; thus, the uncertainty/confidenn
data uncertainty and model uncertainty the connections between hallucination and predictive uncertainty, showing that higher uncertainty is positively correlated with higher chance of hallucination. This study validates the importance
3

---

of a reliable uncertainty measurement in detecting hallucinations of LLMs. Finally, Huang et al. [7 perform an explorative study using simple baselines on uncertainty measurement for LLMs. highlighting the need for more advanced uncertainty quantification methods developed exclusively for L.LMs. This the goal for the current paper as well.
Methodology
This section first defines the LLM uncertainty quantification problem, then describes the design principles and technical details of each algorithmic component, and concludes with a summary of the entire framework. The semantic space defined in Section 3.2 forms theoretical foundation for semantic density. It can be implemented either explicitly through an embedding model or implicitly through an inference model; the latter is the approach used in this paper (details in Section 3.5)
3.1 Problem Statement
Given pre-trained LLM, input prompt and an output sequence [y1, 02. VLL,, where L the number of tokens in y, the target is to produce a confidence metric that positively correlated with the probability of to be true. Note that this metric should be response-wise, i.e., it is calculated a specific y given æ. The metric can be used as quantitative indicator for whether a specific response can be trusted.
3.2 Semantic Space
Theoretically, a semantic space can be any metric space such that a distance function is properly defined to measure the semantic similarity between any output responses, given the input prompt. Note that such a space is prompt-specific, i.e., each prompt results in a specific semantic space in which the distance function measures the contextual semantic similarity between two responses, treating the prompt as a common context.
More concretely, oracle semantic space S is assumed to be Fuclidean space where each point is a D-dimensional vector that represents a contextual embedding of response y given prompt æ:
E x))
where RP,, and B(+}-) an encoder that generates embeddings with the following properties:
1. All the generated embedding vectors normalized to have norm of 1 2
|||| = for E x), Vz, y.
Whereas most existing text embedding models normalize the output vectors have norm of 32], they are rescaled to 2 without changing their direction to make it simpler to integrate them into the kernel function (as explained in Section 3.4).
2. Given prompt æ and two resulting responses and yı, with v; = EEy)(() and by Ez,,kæ), the following constraints exist for three extreme cases:
0. if y: and yj are semantically equivalent given context
||0 wyl
//2 ys and y are semantically irrelevant given context æ
2
1, if yı and yj are semantically contradictory given context: x.
Given the norm requirement in Eq. 2 the above three cases also correspond to Uj. v; 1 Vú and v; = - -ÿ respectively. Note that |! - e,ll is not restricted to the above three values. It can be any value within [0, 1]. depending on the semantic similarity between y: and y given æc
3. Given a prompt æ and three resulting responses y: yj and yv. with v; E E [x), = Eyy) and d2 Exy,
| 1 |0, vall, if y: is semantically closer to yj than to yks given context (4)

---

3.3 Semantic Density Estimator
Given the semantic space S defined in Section 3.2 the underlying probability distribution from which the LLM samples in S provides critical information: If a response is semantically close to many highly probable samples, should be more trustworthy compared to a response that is semantically distant from the major sampling possibilities. A classical technique for estimating probability density is kernel density estimation (KDE) [40] 36]. However, the standard KDE only works for continuous variables, whereas the LLM outputs discrete, i.e., sequences of tokens selected from finite vocabulary. One possible way to extend KDE to accommodate LLM outputs is to build a density estimator
M (ly.(x) = EnK(v. v.)
M Σ
nKK(v. Uz),
(5)
Li=l 74
where x is the input prompt, y. is the target response, i.e., the response that needs confidence estimation, and v, = F(y.lar) In total, Σm=1 n, reference responses are sampled to facilitate the density estimation, where M is the number of unique samples. Each yi represents unique sample; n, is the number of occurrences of y; during sampling, fi ENN n.
ni is the relative frequency of y:
during sampling, and (-)) is a kernel function.
The design of Eq. is similar to an early variant of KDE 37 that was used to handle integer data. However, has the drawback that incorporates no knowledge about the sampling probabilities for each yr. It thus requires large number of samplings, including a sufficient number of duplicated results, to obtain the relative frequency an empirical approximation of the sampling probability. This cost becomes prohibitive for LLMs given how expensive LLM inference generally is.
In contrast with the inherently unknown probability distributions in standard KDE, the output token probabilities can be explicitly calculated in LLM sampling, and with this information, a more sample- efficient estimator can be developed. Given a prompt æ and resulting response y., the semantic density of is defined as
SD y xx)
plyl v.).
(6)
EM p p(y
where v. E(y./z)) |z), and v. = (x) for i= 1, ... M. The M unique responses yi are the reference responses based on which the semantic density of y- estimated. (-)) is a kernel function which will be specified in Section 3.4 and p(ys(z) (for = 1, 2, ... .M is the probability for the orig- inal LLM generate sequence yı given æ. That is, p(y (z) = T= p(ys. 1 V1,, ... 1; æ), where L; is the number of tokens y, and p(y..l.) is the conditional probability to generate token 3t,j- Note that in cases where p(y, (x) is available, y, can also be used as one of the M reference responses.
One advantage of the semantic density estimator of Eq. 6 is that each result ys only needs to be
sampled once; their relative frequency fi can then be estimated as Si Σ p( Given
p(y. le]
sampling budget of M reference responses, is therefore desirable that these M samples are unique (duplications will be removed before calculating Eq. 6) and have high sampling probabilities, that they can cover more sampling regions in the semantic space. In the current implementation, diverse beam search [46]. which tends to generate diverse and highly probable responses, is used to sample the M unique reference responses.
In practice, length-normalized probability 31 29 is usually used to correct the length bias in sequence probability. Moreover, temperature scaling [3 6. [1]] is a simple yet effective method for calibrating the token probabilities during sampling. Both methods can be seamlessly integrated into semantic density: The p(ys (z)) in Eq, c be replaced with ' p/p(yiaa) and the temperature changed during sampling to calibrate each p-))
3.4 Dimension-invarrie Kernel
In standard KDE setup, a commonly used kernel for multi-variate cases is the Epanechnikov kernel g M. which was proved to be the most efficient in terms of asymptotic mean integrated squared
5

---

enror 47 Its original form is
K(v) [(2 + 2) -
(1- 1|/11113/ Pills
where D is the dimension of vector v, I(-) is the gamma function, ||o|| is the 2-norm of v, and 1 condition equals if the condition true, 0 otherwise.
In the semantic density estimator use case, one drawback of the original Epanechnikov kernel is that the normalization coefficient r(2+ 3) changes with the dimension of v. As a result, semantic
densities calculated using embeddings with different dimensionalities are incomparable. This issue may limit the flexibility in selecting embedding methodologies for semantic density calculation. However, the normalization coefficient can be removed to make the kernel function simpler and more flexible, without affecting the performance of confidence measurement. The kernel function of Eq. 6 thus becomes
K(v. - v() (1- 1 11-10 1;,l3)1 ||e lle--e
Although the resulting kernel function does not meet the normalization requirement in standard KDE, fits confidence estimation well. As long as the norm requirements in Eq. 2 B and 4 are fulfilled, any embedding models can be used to generate v. regardless of the embedding dimensionalities. The outcome the kernel function is always within [0. 1]. and kernel value of 1, 2. and correspond to semantically equivalent, irrelevant, and contradictory responses, respectively. As a result, the semantic density also within (0, 1]. with I as the highest semantic density a response can obtain, indicating that all the reference responses are semantically equivalent to it; analogously, it obtains semantic density of 0 when all reference responses are semantically contradictory to it. This consistency the value range makes practical applications of semantic density convenient: Practitioners can set fixed threshold on semantic density to detect unreliable responses.
3.5 Semantic Distance Measurement via the Natural Language Inference (NLI) Model
Although most of the existing text-embedding models work in the semantic space defined in Sec- tion 3.2, they do not perform well in measuring semantic similarities 332 Moreover, they can only consider input texts as whole instead of doing a contextual encoding on part of the input, i.e., they can only obtain ( + y) instead of E(y x)), where æ + y means concatenation of æ and y.
The natural language inference (NLI) classification model 14] has proven to be effective in analyzing the semantic relationship between L.M responses with the prompt as context [22). Given pair of texts, an NLI model performs classification task and outputs the probabilities for them to be seman- tically equivalent ("entailment" class), irrelevant ("neutral" class), or contradictory ("contradiction" class), Given the output class probabilities, the expectation of l/v. will can be obtained as
E([u.. 12) -12. Po (3 y (x) + (x3)² Puly.. y1(x)) 02 Po 1x/x),
= P. v: + Pa Pa. 31(x)
where Pe(y,, y: [x), Pn yi(ær) ar) and (y.. are the probabilities for y. and y, to semanti- cally contradictory ("c" for "contradiction" class), irrelevant ("n" for "neutral" class) and equivalent ("e" for "entailment" class), respectively, given context æ. During implementation, each response y will be concatenated with its prompt æ (with prompt placed before the response) to form one text, ie.. x + y. Each input the NLI model will then be a pair of these texts, analyzing the semantic relationship between two responses given the prompt. The expected value of l|u. e,l1? can then be used in Eq. to obtain the kernel function output.
3.6 Summary the Semantic Density Framework
Algorithm describes how the semantic density metric is deployed on given task and model. The procedure consists of four main steps, i.e., sampling the reference responses, analyzing semantic relationships, calculating the kernel function, and calculating the semantic density.
Computational Cost: In terms of computational cost, only the first two steps involve model inferences. The first step utilizes diverse beam search, in which the group number equals M with

---

Algorithm 1 Procedure for deploying semantic density
Require:
yo: target response that needs confidence measurement
w: original prompt for generating y-
M: number of unique reference responses to be sampled given æ
Ensure:
SD æ): semantic density for y. given
Step I: Reference Response Sampling:
I: sample M unique reference responses yı (for 2. ...... M) with prompt æ on the original LLM using diverse beam search, and record each corresponding length-normalized sampling probability 'vp(y, (x)
Step 2: Semantic Relationship Analysis:
2: for i to M do
3:
obtain (3 y/ (æ) and Pa (y 3/ (z) using NLI classification model
4: calculate expectation B[| v. vill²) = Pe (y., y x+) + 2 (y, y4 /2))
Step 3: Kernel Function Calculation:
5:
to M do 6: calculate kernel function value using the expectation of ||v- - 1y 11112, given by:
K(v.- v (1 1 )): l1llss
Step 4: Semantic Density Calculation:
7: calculate semantic density: SD(y-1æ) EM V (()n 2 VP(y)T))(() vv))
one beam in each group, and thus only M inferences need to be done by the original LLM. The second step requires another M or 2M inferences by the NLI classification model, depending on whether the relationship analysis is performed in a bi-directional manner. Considering the fact that NLI models are usually significantly smaller than LLMs (e.g., the Seberta-large-nn model [14 used in the implementation in this paper only has 1.5 billion parameters), the computational cost is therefore mainly determined by the LLM inferences the first step.
4 Experiments
This section first evaluates the performance of semantic density by comparing it with six existing umcertuinty//cofidenne metrics over various LLMs and benchmarks. After that, two empirical studies performed to investigate the robustness of semantic density when the number of reference responses and sampling strategy for target response are varied.
4.1 Performance Evaluation
Following the usual evaluation approach in the literature [22), the uncertainty/coofidence metric is used quantitative indicator of how likely the response going to be correct. Uncertainty values above a threshold are taken as incorrect while those below are taken as correct (and vice versa for confidence scores). For each threshold, the true positive rate vs. false positive rate then measured. The area under this curve, namely area under receiver operator characteristic curve (AUROC), is calculated for each uncertainty/ccofiiencce metric. The AUROC score equals the probability that randomly chosen incorrect response has a higher uncertainty than a randomly chosen correct response (and vice versa for confidence). perfect uncertainty/confidencc metric would have an AUROC score of while random metric would have 0.5. As an additional performance metric, the area under the Precision-Recee
curve (AUPR) is also calculated. The average AUPR scores over two setups, i.c., using either correct incorrect samples the positive class, reported. Note that higher semantic density corresponds to a higher confidence.
The performance of semantic density (SD) was compared with six existing LLM uncertainty quan-
tification methods: semantic entropy (SE) 22 P(True) [2]. degree (Deg) 26 length nooggt--dd
likelihood (NL) [11 nongghh-oonalized entropy (NE) 29 and predictive entropy (PE) [21]. These
methods were applied seven SOTA open-source LLMs: Llama-2-13B
45 Llama-2-70B (45)
Llama-3-8B 22 Llama-3-70p (2) Mistral-7B [18] Mixtral-8x7B [19 and Mixtral-8x22B [43]
Each LLM was tested on four free-form question-answering datasets commonly used in the literature:
CoQA 38 TriviaQA (20]. SciQ [48] and Natural Questions (NQ) 23] For each question, 10
7

---

Table 1: Performance of different uncertaaintyy/cofideece metrics across various LLMs and datasets
CoQA
AUROC
SD
SE 22
P(Truc) 21
Deg 26
NL 31
NB 29
PE
Llama-2 2-13B
0.783
0.633
0.594
0.734
0.709
0.629
0.647
Llama-2-70B
0.783
0.621
0.576
0.721
0.716
0.617
0.647
Llama-3-8B
0.738
0.599
0.593
0.795
0.676
0.608
0.604
Llama-3-70B
0.789
0.608
0.670
0.729
0.698
0.587
0.641
Mistral-7B
0.788
0.627
0.667
0.737
0.704
0.614
0,632
Mixtral-8x7B
0.786
0.626
0.589
0.728
0.708
0.617
0,651
Mixtral-8x22B
0.791
0.614
0.614
0.726
0.700
0.604
0.649
TriviaQA
AUROC
SD
SE
P(True)
Deg
NL
NE
PE
Llama-2-13B
0.848
0.672
0.589
0.824
0.675
0.574
0.556
Llama-2-70B
0.829
0.677
0.556
0.787
0.714
0.582
0.566
Llama-3-8B
0.866
0.662
0.647
0.796
0.834
0.636
0.622
Llama-3-70B
0.828
0.663
0.654
0.764
0.828
0.611
0.596
Mistral-7B
0.866
0.690
0.589
0.828
0.745
0.615
0.536
Mixtral-8x7BB
0.846
0.685
0.562
0.797
0.795
0.644
0.605
0.829
0.686
0.604
0.762
0.801
0.644
0.607
SciQ
AUROC
SD
SE
P(True)
Deg
NL
NE
PE
Llama-2-13
0.757
0.570
0.572
0.727
0.693
0.513
0.574
Llama-2-70B
0.746
0.643
0.584
0.713
0.637
0.554
0.615
Llama-3-88
0.780
0.611
0,564
0.731
0.686
0.597
0.651
Llama-3-700
0.771
0.613
0.556
0.706
0.724
0.558
0,520
Mistral-7B
0.771
0.618
0.568
0.736
0.669
0.565
0.528
Mixtral-8x7B
0.773
0.612
0.585
0.716
0.726
0.612
0.658
Mixtral-8x22B
0.775
0.620
0.602
0.719
0.715
0.602
0.628
NQ
AUROC
SD
SE
P(True)
Deg
NL
NE
PE
Llama-2 2-13B
0.689
0.581
0.592
0.686
0.588
0.571
0.640
Llama-2-27 2-70B
0.676
0.545
0.531
0.691
0.567
0.573
0.620
Llama-3-8B
0.710
0.583
0,517
0.706
0.601
0.603
0.615
Llama-3-70B
0.723
0.577
0.643
0.714
0.631
0.603
0.615
Mistral-7B
0.680
0.597
0.523
0.676
0.640
0.635
0.631
Mixtral-8x7B
0.729
0.599
0.576
0.720
0.654
0.603
0.608
Mixtral-8x22B
0.709
0.577
0.504
0.704
0.638
0.625
0.680
responses were generated using group beam search and used as reference responses in calculating SD, SE, Deg, NE, and PE (note that P(True) and NL do not need reference responses). Each unique response among these 10 will also be used target response, i.e., the response that needs uncer- tainty/confidence estimation, calculating the AUROC scores uncertainty/confidencc metrics. Detailed experimental configuration and parametric setup is described in Appendix A.
Table and Table A2 lainty/confidence metric across different models and datasets, with the best entry in each configuration highlighted in boldface. SD performs best in out 28 cases for AUROC and 27 out 28 cases for AUPR, demonstrating that it is reliable and robust
Appendix A.4) show the AUROC and AUPR scores of each uncer-
confidence metric for LLM responses. For AUROC, in two cases is outperformed by Deg. After investigation, the inherent sequence likelihood returned by the original LLM was badly calibrated in these two cases. Deg is the only method that ignores the likelihood information during its calculation, making its performance unaffected by this negative factor. However, for the other 26 cases, SD is able to utilize the likelihood information to its advantage and outperform Deg. Appendix AA. includes the results of another variant of SE [10) which is comparable to the original version. SD outperforms it in all the test cases.
To confirm that the observed performance differences in Table are statistically significant, a paired t-test (paired by LLM and dataset) was performed between SD and the other metrics (Table Al

---

CoQA
TriviaQA
SciO
NQ
0.725
AURO scoe
0.775
AUBOC wcre
AUROC ecore
0.750
AUROC
0.025
9.675
0.70
0.725
0.500
0.775
0.700
2
10
10
10
responses
responsse
responses
resporsse
Lioma-2-138
Liama-2-77 708
Liama-3-88
Jama-3-708
Mistral-78
Mixtral-xx78
Mixtrall 8x228
Figure I: AUROC scores of semantic density with one to 10 reference responses. Each subfigure corresponds to the dataset indicated by the subfigure heading. Each curve corresponds to the base LLM identified in the legend below the subfigures. Providing more reference responses generally increases the reliability semantic density, but most cases only four samples are sufficient.
Appendix A.2 The p-values are consistently below 10-6, indicating that the performance gains of SD are strongly statistically significant.
The experiments reported above the setup as Kuhn al. [22] evaluate the correctness of responses: the Rouge-L. [24] between one response and the reference answer is larger than 0.3, then the response deemed correct. In order investigate the influence of the correctness-checknn criterion, the threshold of Rouge-L was varied between 0. and 1.0; the resulting performance of different uncertaintyycconiidecee metrics shown in Figure Al SD consistently outperforms other methods under the different Rouge-L thresholds. Moreover, when the Rouge-L threshold increases, the AUROC scores generally increase for SD whereas they decrease for most other methods. Since higher Rouge-L threshold means stricter correctness checking, the experimental performance gain of SD may be even larger if such checking further improved (e.g. by using more capable LLM).
To further evaluate the generalizability of SD in other types of tasks, an empirical study using summarization task (DUC-20044 355 was performed. Appendix A.6 shows the performance comparisons among different uncertainty/confidenee metrics. SD performs best most test cases, demonstrating good generalizability.
4.2 Robustness Semantic Density
Two additional empirical studies were performed to evaluate the robustness of semantic density when the number of reference responses varies or the sampling strategy for target response changes.
In the first study, the number reference responses was reduced from 10, which is standard setup for existing methods (22), to one, which is the extreme minimum case. Figure I shows the resulting AUROC scores, covering the same four datasets and seven LLMs. Although performance indeed decreases with fewer reference responses, the decrease is minor long the number of references at least four. This result suggests that semantic density can provide reasonable performance even with very limited budget for reference sampling.
In real-world applications, users may have different preferences when generating responses: Some may prefer a greedy sampling strategy while others may need diverse responses. The second study thus investigated how each uncertainty/confidenee metric performs when the target response is sampled using different such strategies. The diverse beam search method inherently utilizes different strategies for each beam group: The first group performs a greedy beam search while later groups encourage more diverse responses. Following Section 4.1 the AUROC scores were calculated for target responses from each group separately, and the results averaged over the four datasets.
As the results in Figure show, semantic density exhibits consistently good AUROC scores across different beam groups. Thus, robust against both more greedy and more diverse sampling strategies, thus covering range of possible user preferences. In contrast, other approaches either perform consistently worse compared to semantic density across different beam groups, or their performance is unstable when the sampling strategy changes.

---

1lama-2-13B
Llama-3-88
Llama-3-708
average AUROC score
AURDE average AUROC score
0.8
0.7
avergge AUROC
AUROC
0.7
0.70
0.65
0.6
0.6
0.55
gro up
group Index
group
group Indiex
averig AUROE score
Mistral-78
average AUIROC score
Mixtral-8x7B
Mixtral-8x228
0.75
0.75
samantic density
P(True)
0.70
entropy
0.45
length-normalizdd likalihood
Deg
0.60
0.60
langth-normalized entropy
predictive entropy
10
10
group
group Index
group
Figure 2: AUROC scores the different beam groups. The plots show the average AUROC scores over the four datasets for the different beam groups in diverse beam search. smaller group index corresponds to the group with more greedy generation strategy, while the group with larger index tends to be more diverse during response generation. Each subfigure corresponds to one of the seven LLMs, as indicated by the subfigure heading. Each curve represents one uncertainty/confidenee metric indicated by the legend at the lower right. Semantic density exhibits consistently better and stable performance across different groups, compared other methods.
Discussion and Future work
In terms of the broader societal impact of this work, the proposed semantic density provides general way to evaluate the trustworthiness of responses generated by LLMs. This ability should have positive impact on real-world applications that are safety-critical, such as healthcare and finance. Practitioners can utilize semantic density as an off-the-shelf indicator to filter out unreliable responses.
One limitation of semantic density is that it needs access to the output probabilities of generated tokens. which may not be available in some proprietary LLMs. In such case, the more expensive variant in Eq. 5 can be considered as alternative. Since semantic density does not require any further access to the internal states or weights of the original I.LMs, Another limitation is that most responses in the current experiments are at the sentence level. However, an extension of semantic density to panaagagg responses should be feasible. Following the existing solutions [28 [0], long response can be decomposed into sentence-level claims (28) or factoids 100. and semantic density then applied to estimate the confidence each claim/factoid.
still widely applicable.
The framework for measuring semantic density is modular, and therefore its main components can be extended in future work. First, new sampling strategies that explicitly encourage a better coverage of semantic space can be developed to generate reference responses. Such extensions should improve the reliability of semantic density further. Second, text embedding methods that can measure contextual semantic similarity between responses more reliably will be helpful well. Third, kernel functions specifically designed for the semantic space should allow the utilization of semantic relationships more efficiently. Fourth, more precise methods for calibrating inherent token probabilities will form a more reliable base for calculating semantic density.
6 Conclusion
This paper proposes semantic density as a practical new metric for measuring the confidence of LLM responses. It overcomes the limitations of existing approaches by utilizing the semantic information an efficient and precise manner. It response-specific, "off-the-shelf". and applicable to free-form generation tasks. Experimental comparisons with six existing uncertainty/confidenee metrics across seven SOTA L.MSs and four benchmark datasets suggest that accurate, robust, and general, and can therefore help deploy LLMs in safety-critical domains.
10

---

References
[1] Lukas Aichberger, Kajetan Schweighofer, Mykyta lelanskyi, and Sepp Hochreiter. 2024. Se- mantically Diverse Language Generation for Uncertainty Estimation in Language Models. arXiv:2406.04306 [cs.LG]
org hatss//446..000 04306
[2] AI@ Meta. 2024. Llama Model Card.
https://gitthub.com
.md
[3] Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When It's Lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguisties, Singapore, 967-976.
https://doi. org/ 10. 18653/ v1/2223...222 windings-emiinn emmlp. 68
[4] Jiuhai Chen and Jonas Mueller. 2024. Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics. Bangkok, Thailand, 5186-5200. https://doi. org/ 10. 11113//1///22// acl- long. 283
[5] Jan Clusmann, Fiona R. Kolbinger, Hannah Sophie Muti. Zunamys I. Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P. Veldhuizen, Sophia J. Wagner, and Jakob Nikolas Kather. 2023. The future landscape of large language models in medicine. Communications Medicine 3, 1 (2023), 141. https: http://ti...ogrrr///eoo
org/10 10. 0000///444665--200037700
[6] Shrey Desai and Greg Durrett. 2020. Calibration of Pre-trained Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 295-302. https://doi. org/10..///. 11.8663/v/222// emnlp main. 21
[7] Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics /Volume 1: Long Papers), Lun- Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 5050-5063. bsttps://dos org/ 10. 11t.11..111111112222222 acl -ong 276
[8] V. A. Epanechnikov. 1969. Parameeti Estimation of Multivariate Probability Density. Theory of Probability & Its Applicaations 14, 1 (1969), 153-158. https://doi.org:
1137/1114019 httpa///aitttt///dboooo//.......
14019
[9] Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. L.MPolygraph: Uncertainty Estimation for Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Yansong Feng and Els Lefever (Eds.). Association for Computational Linguistics, Singapore, 446-461.
https://doi org/ 10. 18653/v1/22223
emmlp-demt demo. 41
[10] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations
in large language models using semantic entropy. Nature 630, 8017 (2024), 625-630. https
7 7/doi.org/10 1038/
00033//41555 s415861666668 00211586-04---10 0222-474221-
[11] Keinosuke Fukunaga and Larry D. Hostetler. 1975. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans. Inf. Theory 21 (1975), 32 40.
https://app. www.seeetaicatioeeeeeoo
org/CorpusIDDDD.....o 15299210
[12] Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024.
A Survey of Confidence Estimation and Calibration in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 6577-6595.
https:/ /ons://do1 /tt.///oiiooooo org/ 10. 118853//1/2000 http://22...a......ee 366
11

---

[13] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70 (Sydney, NSW, Australia) (ICMI. .or, 1321-1330.
[14] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DEBERTA: DECODING- ENHANCED BERT WITH DISENTANGLED ATTENTION. In International Conference on Learning Representations. https://oppeneviiee wttp:ettttoorr/aaae/laaaaaaaaaa
[15] Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2023. Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. 1XXiv23111008711 [cs.CL]
[16] Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. 2023. Un- certainty in Natural Language Processing: Sources, Quantification, and Applications.
arXiv:2306.04459 [cs.CL] attps://arxiv.co http:/orggbbbb////eeee/ee// abs/2306. 22112336604499999
[17] Yuheng Huang. Jiayang Song, Zhijie Wang. Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2023. Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models. 23Xv:23077.1036 [cs.SE]
[18] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv:2310.0682. [cS.CL]
[19] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of Experts. arXiv:2401. 04088 [cs.LG]
[20] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, Vancouver, Canada, 1601-1611.
https: / doi. org/ 11t.:/0111111/111111111
[21] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-oddds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. 2XXv:2207.05222 [cs.CL]
[22] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023, Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In The Eleventh International Conference on Learning Representations. https:///penreevenn wttpo/vwwwweeeeeeeeeeeeeeee net/forum?
[23] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova. Llion Jones, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics (2019).
Ming-Wei
[24] Chin-Yew Lin and Franz Josef Och. 2004. Automatic Evaluation Machine Translation Quality Using Longest Common Subsequence and -iiiaBi Statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACI.-04). Barcelona, Spain, 605-612.
https: /doi. org/10 10. 31////////1111111 12111999555111933
12

---

[25] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. Transactions on Machine Learning Research (2022). https:// openreview.o htt.eeewwwee//rrrrrdddd--s/-///e
[26] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models. arXiv:2305. 19187 [cs.CL]
[27] Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, and Haifeng Chen. 2024. Uncertainty Quantification for In-Context Learning of Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 3357-3370.
https: / //doi. org/ 10. (18633/v1/2000 www.c- 184
[28] Xin Liu, Muhammad Khalifa, and Lu Wang. 2024. LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses. In The Twelfth International Conference on Learning Representations.
[29] Andrey Malinin and Mark Gales. 2021. Uncertainty Estimation in Autoregressive Structured Prediction. In International Conference on Learning Representations. Iuttps://openreviee met/forum?id=jjiee 7bb-2m0
[30]
Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SellCheckGPT: Zero- Black- Box Hallucination Detection for Generative Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9004-9017.
https://doi. org/ 10. 18653 00186633v121 v1/2022...222 emnlp- main.557
[31] Kenton Murray and David Chiang. 2018. Correcting Length Bias in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri. Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, Brussels, Belgium, 212-223. https: org/
10. 10.18653/v1/W13 6322
[32]
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke. Pranav Shyam, Boris Power, Tyna Floundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. 2022. Text and Code Embeddings by Contrastive Pre- Pre-Training, arXiv:2201. 10005 [cs.CL]
[33] Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel Lan- guage Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities. arXiv:2405.20003 [cs.LG] https://arxivi wwt.oog/////////oooooooooooeee// abs///////////// 2405 20003
[34] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Balteseu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael. Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Loper Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei
13

---

Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez. Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo. Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Mokoael Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky. Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever. Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder. Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang. Marvin Zhang, Shengjia Zhao, Tianhao Zheng. Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 Technical Report. arXiv:2303.0877 08774 [Cs.CL]
[35] Paul Over. 2004. DUC 2004 Documents, Tasks, and Measures. duc2004/
https://duce .ist gov/
[36] Emanuel Parzen. 1962. On Estimation of a Probability Density Function and Mode, The Annals of Mathematical Statistics 33, 3 (1962), 1065 - 1076. https://doi. org.o.o..orr// 10. 1214/ /
1177704472
[37] Balaji Rajagopalan and Upmanu Lall. 1995. A kernel estimator for discrete distributions. Journal of Nonparametric Statistics 4, 4 (1995), 409-426. httpp:://dda //ooi. org/10.
10485259508832629 arXiv.htps::/doiiorg/// wttp://///....100000000151111110 10000000000001111553533033
[38] Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A Conversational Question Answering Challenge. Transactions of the Association for Computational Linguistics 7 (2019),
249-266. https://doi. https:////ooooo////// org/10. 1162/tt1ccccccc a aa_2aaallaaaa0666661111111
[39] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2024. Mathematical discoveries from program search with large language models. Nature 625, 7995 (2024), 468 475. https
7/doi.org/10 1038/ 1000///45500 s4158616666661 0005--233-686 06924-6
[40] Murray Rosenblatt. 1956. Remarks on Some Nonparametric Estimates of a Density Function. The Annals of Mathematical Statistics 27,3 (1956), -37. https://doi. org/ 10. 1214/
aoms coms/1177728190
[41] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov,
14

---

Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong. Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. arXiv:2308. 12950 [cs.CI.]
[42] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schackermann, Amy Wang. Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. 2023. Towards Expert-Level Medical Question An- swering with Large Language Models. arXiv:2305.096 [cs.CL]
[43] Mistral AI team. 2024. https://misst
8x22b/
[44]
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 5433-5442,
https: : //doi. org/ 10. 10613/81/22000 emnlp- momlp-main... wa-mmain... 330
[45]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mi- haylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307. .09288 [cs.CL]
[46] Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse Beam Search for Improved Description of Complex Scenes. Proceedings of the AAAI Conference on Artificial Intelligence 32, (Apr. 2018). https://doi. org 10. 1609/ aaai 11.2211.1349
[47] M.P. Wand and M.C. Jones. 1994. Kernel Smoothing. Taylor & Francis. https://books google. wwt.mm/oobbeeddddd00000555/-
[48] Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing Multiple Choice Science Questions. arXiv:1707.06209 [cs.HC]
[49] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Ar Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38-45. https: htaps://wwee //wwww http:/wwwwwwwb..//////////tw aclweb. wwt////toooooggg/2222222 emmlp-demod 6
[50] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. BloombergGPT. A Large
Language Model for Finance. arXiv:2303.17564 [cs.LG]
[51] Yuxin Xiao, Paul Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis- Philippe Morency, 2022. Uncertainty Quantification with Pre-trained Language Models: A
15

---

Large-Scale Empirical Analysis. In Findings of the Association for Computational Linguistics: EMNLP 2022, Yoav Goldberg. Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguisties, Abu Dhabi, United Arab Emirates, 7273-7284 attpss//doo 10..1..1663/v111/
wtt.:///..iinnnnneeeeenee e.t.e.mmmlp...... 538
[52] Yijun Xiao and William Yang Wang. 2021. On Hallucination and Predictive Uncertainty in Conditional Language Generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 2734-2744. https://doi org/ 10. 18653/11/221111111 eacl main. 236
[53] Fanghua Ye, Yang MingMing, Jianhui Pang, Longyue Wang, Derek F Wong, Yilmaz Emine, Shuming Shi, and Zhaopeng Tu. 2024. Benchmarking LLMs via Uncertainty Quantification. arXiv preprint arXiv:2401. 12799 (2024).
[54] Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. 2023. Survey Large Language Models. arXiv:2303. 18223 [cs.CL]
16

---

A Appendix
A.1
Experimental Setup
This section provides the details of the experimental setup for reproducing the results presented in Section The source codes for reproducing the reported experimental results are provided at:
Base LLMs: For all the seven base LLMs, the open-source versions in Huggingface Transformers library [49 were used in the experiments. More specifically, the fol- lowing versions are used: leta-llama/LLama-21111 meta-IlaanaMMeta-taamme
meta- aataa--aaam/eeet---aaa------3 1, mistralaa./iixtralssr/// -atprrv01--x--vv0----- and ning diverse beam search in generating the responses, the generate function is used with Siversity-penaltty O, default temperature of 1.0, and the num_bes num_beam____eeeee www.beeammm_o_eeeeeeee geam_groups equals the number beams so that each group has exactly one beam.
mistralai/Misttaaaa When run-
Correctness Checking: Following the setup in Kuhn et al. 22 for all the reported experimental results except Figure[AT] response considered to be correct the Rouge-L. (24) between it and any of the reference answers is larger than 0.3, after trimming the redundant continuations.
Datasets:
The details of the datasets are as follows:
The coqa-dev-vl. version used with 1596 questions randomly selected for the experiments, using the Huggingface datasets. trainnraaeeeeeee sesti__et___p_____ function with seed value of 10. The prompt format follows the same setup as in Kuhn et al. [22].
CoQA:
The dataset is loaded with the Huggingface datasets API. 1705 questions were randomly selected from the validation split for the experiments, using the Huggingface datasets. taasetssttrii_teee arain_testtsplir function with seed value of 10. A 10-shot prompt format used following the setup in Kuhn et al. [22).
TriviaQA:
SciQ: The teat split from
is used in the experiments. contains 990 questions. 10-shot prompt format is used following the setup in Kuhn et al. [22].
NQ: The test split from https://github.. wttp:/ituuu..ommmmunnnni comm/lanchnll/LLice LitCab is used in the experiments. 1800 questions randomly selected using the Huggingface datasets. train, test_ split function with seed value of 10. A 10-shot prompt format is used following the setup in Kuhn al. [22].
33 used in the experiments. The dataset The following prompt used for all the experiments: "This is a bot that summarizes the following paragraph using one sentence. An Paragraph: [document] An Summary:".
DUC-2004: All the 500 samples in task
is downloaded from
The same exact target response and reference responses are used for all the tested methods. For SD, SE, and Deg, the mi ciooofft///bbb dewerra-large--eaeeeeeeee model from Huggingface Transformers library 44 is used as the NLI classification model. Following Kuhn al. [22]. the probabilities for "contradiction", "neutral" and "entailment" averaged bidirectionally. The parametric setup for the uncertainty metrics tested in Section 4 summarized as follows:
Uncertainty/Confidene Metrics:
semantic density (SD): The same exact steps as described in Algorithm I were implemented, with a fixed temperature of 0.1 applied to rescale the value of each token probability during postprocessing.
semantic entropy (SE): The original implementation and parametric setup from https://github.com/// http://hww....ommmmoo///aa//aaaa cuncrttiicccunetttttyycy was used.
P(True): The original few-shot prompt format from Kadavath et al. (1 is used.
degree (Deg): The "entailment" probability returned by the NLI model (averaged bidirectionally) is used as the similarity between two responses, which then used as the diagonal element the degree matrix.
17

---

length-normalized likelihood (NL): The original form as in Murray and Chiang [3 was imple- mented.
length-normalized entropy (NE): The implementation from https///gthubbbcomm/oeeeiiiiin /semantic _ was used.
predictive entropy (PE): The implementation from http:///gthhbbomm/////o////////e uncertainty was used.
Compute Resources: All the experiments in Section 4 were running on an AWS P4de instance with 96 Intel(R) Xeon(R) Platinum 8275CL CPU 3.00GHz, 1152GB memory and 8 NVIDIA A100 (80GB). The GPU memory required to run the experiments is dependent on the base LLMs, as detailed below:
Llama-2-13:: 33GB GPU memory.
Llama-2-70B: ~-140GB GPU memory.
Llama-3-8B: ~200B GPU memory.
Llama-3-70B: ~110GB GPU memory.
Mistral-7B: GPU memory.
Mixtral-&:7B:~111055 GPU memory.
Mixtral-8x22B: ~300GB GPU memory.
The exact computation time was affected by many factors, e.g., the current workload the machine, the prompt length of the question, the generated response length, whether the cache option is turned on to store the model states of LLMs, etc.
A.2 Results of Statistical Tests
Table A1 shows the p-values of the paired I-tests described in Section 4.1
Table A1: Statistical significance SD's advantage over other methods (p-values of paired f-tests)
SE
P(True)
Deg
NL
NE
PE
SD
4.83E-17
1.71E-15
1.16E-7
4.62E-14
2.35E-15
4.63E-13
A.3 Performances with Different Correctness Thresholds
Figure Al shows the performance changes of the uncertainty/confidence metrics with different Rouge- L thresholds when checking the correctness of responses. SD performs the best overall across these different criteria.
A.4 Performance Evaluation Using AUPR
During this evaluation, each uncertainty/confidencce metric was used as quantitative detector in two cases: 1) detecting incorrect responses, 2) detecting correct responses. The AUPR scores for both cases were calculated. Table A2 shows the average AUPR scores for them. Again, SD performs the best overall.
A.5 Performance of Another Variant of Semantic Entropy
Table A3 shows the performance of a follow-up variant of semantic entropy [10] The performance of this variant comparable with the original semantic entropy, with several cases worse than the original version. The proposed semantic density outperforms this variant in all test cases.
18

---

semanti:
samaatic
lengh--nmmaiied
predicte
Liama-2-13B on CoQA
Llama-2-138 on TriviaQA
Llama-2-138 on SciQ
Liama-2-138 on NQ
0.8
AUROC score
AUROC SCOTE
AUROC SCOTE
AUROC acone
0.7
0.7
0.6
0.5
0.2 0.4 0.6
0.8
1.0
0.2 044 0,5 6-8 1.0
0.2
0.4 0.6
0.8
1.0
0.5
0.2
0.4 0,5 0.8 1.0
correctiesss L
correctness threshold rouge-L
correctiess threshold rouge-L
correctiess threshold rouge-L
Llama-2770 on Llama-2-70B on TriviaQA
Llama-2-708
SciQ
Liama-2-708
AUROC suorescore scoee
SCOT
0.7
AUROC
0.7
AUROC
0.3-
0.5
0.8
1.0
0.8
0.6
1.0
correctrens threshold for rouges
correctrwss threshold for rouge-
correctress threshold for rouge-L
corractness threshold for rouge-L
Llama-3-6e 888 on CoDA
Llama-3-88 on TriviaQA
Llama-3-88 on SeiQ
Llama-3-88 on NO
AUROC scare
AUROC SEOP
0.7
AUROC
0.7
0.7
AUROC
0.6
0.5
0.6
0.2 0.6
1:0
0.2
0.6
0.8
0.8
1.0
rouge-L
shreshold rouge-L
rouge-L
Liama-3-70B on CoQA
Llama-3-70B on TriviaQA
Llama-3-70B Scio
Llama-3-708 NQ
0.8
AUROC soate
0.7
0.8
AUROC
AUROC
0.7
0.5
0.5
0.2 0.4 0.8
0.4 0.6
1.0
0.2
0.4
0.6
0.8
0.2
0.4
0.5
1.0
correctness threshold for rouge
correctiness threshold for rougeiL
correctness thresholl for rouge-L
correctness thresholl for rouge-L
Mistral-7B on CoOA
Mistral-7B on TriviaQA
Mistral-7B on SeiO
Mistral-7B on NO
0.8
AUROC score
AUROC SOOT
AUROC SOOT
0.7
AUROC
0.5
corractrees threshold for
correctnens threshold rougest
corractress threshold for
correctness thresbold for
Mixtral-8x78 on CoOA
Mixtral-8x7B on TriviaOA
Mixtrai-8x78 Selo
Mixtral-8x78 on NQ
AUROC StaR6
AUROO ScOPT
AUROC
0.7
AUROC
0.7
0.6
0.6
0.6
0.6
0.5
0.2 0.4 0.6 0.8
1.0
0.2
0.4
0,5
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.5
0.6 1.0
corractness thrashold for rouge-L
cornectnas threshold rougest
corractrees thrushold for rouge-L
correctnees threshold rouge-L
Mixtral-6x228 CoQA
Mixtral-8x228 TriviaQA
Mixtral-6x226
Mixtral-6x228
AUROC scare
AUROC SCOTT
AUROC 9.7
0.7
AUROC
0.3
correctivess threshold for rougel
correctneds threshold for r
correctiness threshold for rouge-L
correctness threshold for rougeiL
sementic demalty
MTTru)
entrapy
Dep
lenght molleed lieelhood length entropy
prodictivn entropy
Figure A1: Performance Evaluation over Different Correctness Checking Criteria. The Rouge- L threshold for checking response correctness is set 0.1, 0.3, 0.5, 0.7, 0.9 and 1.0, respectively. Each subfigure corresponds to one combination of base LLM and dataset, indicated by the subfigure heading. Each curve corresponds to the performance of one uncertainty/confidencc metric identified in the legend at the very top and bottom. For each subfigure, the horizontal axis indicates the Rouge-L Overall, semantic density exhibits best
threshold, and the vertical axis indicates the AUROC performance under different Rouge-L thresholds. The experimental performance of SD increases when the sereettess-checking criterion becomes stricter (a higher Rouge-L threshold), while most other methods show opposite trends.
19

---

Table A2: Performance evaluation using AUPR-average metric
CoQA
AUPR
SD
P(True)
Deg
NE
Llama-2-133 B-BBB
0.766
0.605
0.589
0.724
0.692
0.617
0.632
Llama-2-70B
0.766
0.587
0.566
0.710
0.686
0.602
0.628
Llama-3-SB
0.718
0.581
0.579
0.779
0.665
0.598
0.595
Alama-3-708
0.765
0.572
0.641
0.714
0.658
0.566
0.606
Mistral-7B
0.773
0.600
0.652
0.730
0.680
0.599
0.618
Mixtral-8x7BB
0.768
0.592
0.580
0.719
0.686
0.599
0.636
Mixtral-8x22B
0.772
0.581
0.605
0.715
0.672
0.585
0.632
TriviaQA
AUPR
P(True)
Deg
NL
NE
Llama-2-11 2-B3B
0.839
0.651
0.589
0.807
0.662
0.570
0.550
Alama-2-700
0.822
0.637
0.543
0.775
0.670
0.572
0.552
Llama-3-88 3-8B
0.855
0.636
0.636
0.781
0.815
0.638
0,616
Llama-3-700
0.817
0.609
0.629
0,750
0.769
0.594
0.573
Mistral-7B
0.857
0.668
0.586
0.813
0.720
0.607
0.529
Mixtral-8x
0.835
0.653
0.546
0.782
0.763
0.628
0.591
Mixtral-8x22B
0.815
0.630
0.573
0.747
0.740
0.616
0.586
SciQ
AUPR
SD
SE
P(True)
Deg
NE
Llama-2-133 2-B
0.744
0.567
0.559
0.695
0.686
0.511
0.556
Llama-2-70B
0.733
0.626
0.574
0.694
0.630
0.549
0.604
Llama-3-88
0.768
0.605
0.560
0.714
0.684
0.590
0.646
Llama-3-70B
0.755
0.597
0.551
0.690
0.722
0.548
0.518
Mistral-7B
0.759
0.612
0.559
0.717
0.664
0.561
0.530
Mixtral-8x7B
0.759
0.612
0.570
0.702
0.726
0.605
0.645
Mixtral-8x22B
0.753
0.604
0.588
0.703
0.700
0.589
0.628
NQ
AUPR
SD
SE
P(True)
Deg
NI.
NE
PF
Llama-2-
0.656
0.561
0.552
0.642
0.589
0.544
0.598
Llama-2-70B
0.655
0.539
0.525
0.648
0.580
0.547
0.591
Llama-3-88
0.663
0.555
0.512
0.644
0.596
0.563
0.572
Llama-3-70B
0.700
0.562
0.608
0.638
0.570
0.580
Mistral-7B
0.654
0.573
0.519
0.637
0.623
0.586
0.586
Mixtral-8x7B
0.700
0.590
0.555
0.692
0.650
0.576
0.583
Mixtral 8x22B
Hixtral-xx22B
0.687
0.571
0.509
0.671
0.645
0.596
0.647
Table A3: Performance of the semantic entropy variant
AUROC
CoQA
TriviaQA
SciQ
NQ
Llama-2-13B
0.587
0.726
0.576
0.576
Llama-2-700
0.570
0.696
0.596
0.555
Mistral-7B
0.606
0.736
0.587
0.582
Meta-Llama-3 L--ma-3-m-388
0.571
0,694
0.582
0.563
Meta-Llamaa337001
0.572
0.734
0.613
0.568
Mixtral-xx7B
0.567
0.699
0.576
0.612
Mixtral-8x22 8x22B
0.562
0.690
0.592
0.565
20

---

Table A4: Performance of different uncertainty metrics for summarization task
DUC 2004
AUROC
SD
SE
Deg
NL
NE
PE
Llama-2-
0.622
0.612
0.606
0.594
0.601
0.521
Llama-2-70B
0.552
0.579
0.549
0.520
0.575
0.534
Llama-3-8B
0.597
0.529
0.590
0,538
0.538
0.521
Llama-3-70B
0.593
0.513
0.592
0.489
0.515
0.433
Mistral-7B
0.679
0.537
0.660
0.673
0.551
0.549
Mixtral-8x7B
0.640
0.532
0.620
0.636
0.576
0.497
Mix22
0.649
0.625
0.639
0.574
0.630
0.532
A.6 Performace Evaluation in Summarization Task
This experiment used Task of DUC-2004 dataset [35], which is single-document summarization task. The LLMs are prompted to summarize the document using one sentence. Table A4 presents the results. performs the best six out of seven test cases, demonstrating its good generalizability to tasks other than question-answering.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The contributions and scope of this work are accurately reflected in the abstract and Section [
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other settings.
It is fine include aspirational goals motivation long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: There are several discussions about the limitations of the proposed semantic density in the current manuscript: (1) The second paragraph of Section 5 discusses the potential limitation regarding the applicability in some restricted cases; (2) the third paragraph in Section 4. discusses a negative factor that may impair the performance of the proposed method; (3) section 4.2 studies other factors that may impact the performance of semantic density, and found it to robust to these factors; (4) the computational efficiency is considered in algorithmic design (the second last paragraph of Section (.3) and discussed Section 336 (5) scalability LLMSs with different sizes are empirically evaluated in Section 4
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "[Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e.g., independence assumptions, noiseless settings, model well-speciffcation, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or a speech-to-tee system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best, judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
22

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
There are no new theorems proposed in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The detailed description about each algorithmic step of the proposed semantic density provided in Section 3.6 The detailed guidelines about the experimental configurations are provided in Appendix A. Source codes are provided in the supplementary material (as an attached zip file).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others replicate the model with the same dataset, provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate the research performed.
also be provided via detailed instructions for how to replicate the
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure source models, it may be that access to the model is limited in some way (e.g.. to registered users), but should be possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Source codes to reproduce all the experimental results in this paper are provided in the supplementary material (as attached zip file). Appendix Dopendix/A.Ippovviiieee [A.]] all the detailed
information needed to reproduce the experimental results, including instructions on data access and preparation, configurations of the proposed method and all other baselines.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines pitttps//niipar ctt.ippiccppbblcc/////cc
guides/CodeSubmise ssionPlicyy yy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines buttps://nijy
www.iimmissiooooooccy... for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 provides clear description of the experimental settings that are sufficient for readers to fully understand the results. Appendix[A.. provides all the configuration details of the experiments. Source codes to reproduce all the experimental results in this paper provided the supplementary material (as an attached zip file).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Statistical tests have been run and reported in Section to verify that the perfor- mance differences between semantic density and other methods are statistically significant, with detailed p-values reported in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: The compute resources used by the experiments in this paper are described in the "Compute Resources" paragraph of Appendix A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. cc/publicc/chiceeuuiieeeo
Answer: [Yes]
Justification:
The research conducted in this paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethies.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Fthics.
The authors should make to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The first paragraph of Section 3 discusses the broader societal impact of the work. Since the proposed method intends to improve the trustworthiness of L.LMs, there is no obvious negative societal impacts.
Guidelines:
The answer NA means that there is societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimesss considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification:
All the models, datasets and source codes used in this work have been properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package weirrsssthhhe.....
should be provided. For popular datasets, paperswi thcode. cww/oat//aaaoo ham/aatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re- re-packagdd, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethies and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "eLF9NnWVSq", "reviewer": "Reviewer_rbw2", "summary": "The paper proposes an approach for estimating uncertainty of LLM outputs. The proposed approach begins by sampling a diverse set of responses, analyses their equivalence using a NLI classification model, and then computes the semantic density using a kernel density estimate.", "questions": "- Sec 3.5: Please describe how the NLI model is used for computing p_c, p_n and p_e. NLI models typically accept pairs of texts, but these probabilities seem to contain three: y_*, y_i and x.\n- Do I understand correctly that embedding models (discussed in Sec 3.2) are never explicitly used in Semantic Density? Instead the NLI model is relied upon for establishing equivalence between texts and for the kernel function? If yes, the current presentation comes across as a bit confusing - please consider rewriting parts of the paper to make this more explicit.\n- As I understand, the proposed approach only estimates uncertainty rather than explicitly calibrating the underlying model - and yet the proposed uncertainty estimation method helps improve predictive performance in Table 1. Am I understanding correctly that the underlying models are somewhat well calibrated to begin with? To put it another way, are there generation tasks where one might expect SD to not be as reliable? For example, a summarization task. If this is a reasonable understanding, please consider discussing where SD is unlikely to work well.\n- The differences highlighted against Semantic Entropy in Lines 105-107 are vague or inaccurate - SE considers responses in computing the SE(x) value so it is unclear how it is \"prompt-wise\", that being said, what are the advantages of obtaining uncertainty for a specific output and prompt pair? What does a \"one-cut equivalence relationship\" mean? Please consider highlighting differences more clearly and discussing the specific advantages/disadvantages that the differences result in.", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach seems sufficiently different from prior work and combines existing ideas meaningfully. The approach is convincingly validated in experiments.\n- The paper is clear for the most part.\n- The problem of uncertainty estimation for generation tasks is meaningful.", "weaknesses": "- The paper is somewhat vague in distinguishing itself from the closest prior work on Semantic Entropy - this makes its specific contribution unclear.\n- Some details of the implementation of semantic density are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response: Semantic Entropy (SE) is indeed the baseline to which we compare, aiming to overcome its  limitations using a different approach, i.e. Semantic Density (SD). Let us first clarify the difference. although the computation of SE involves analysis of multiple reference responses, only one SE score is obtained for each prompt: SE does not quantify the predictive uncertainty of each specific response. As a simple example, assume that given a prompt X, an LLM samples three different responses A, B, and C, among which A is the correct answer while B and C are incorrect. SE returns only one uncertainty score, representing the aleatoric uncertainty introduced by prompt X. In contrast,  SD returns three different uncertainty scores, one for each specific response.\n\nThis difference matters because users can get different responses from the LLM even given the same prompt. In this case, the user who gets a correct response A would be given the same SE score as another user who gets an incorrect response B. Thus, users cannot differentiate the trustworthiness of different responses for the same prompt. In contrast, SD returns different semantic density scores depending on the actual answer returned. A more trustworthy response has a high  score, and thus the users can decide whether a specific response can be trusted. This fundamental difference between SE and the proposed semantic density comes from their inherently different design principles: SE is based on entropy calculation, which can only be prompt-wise, while SD is based on density estimation, which is naturally response-wise. We will add this elaborated discussion in the revision.\n\n\"What does a \"one-cut equivalence relationship\" mean? Please consider highlighting differences more clearly and discussing the specific advantages/disadvantages that the differences result in.”\n\nAs discussed in line 51-55 in the current manuscript, the \"one-cut equivalence relationship\" means that the original SE applies a binary measurement of the semantic relationships between two reference responses. That is, the relationship between two reference responses can only be “equivalent” or “nonequivalent”. In contrast, SD evaluates the semantic relationships among reference responses in a continuous manner: the calculation of SD estimates the degree to which the reference responses are semantically similar. Such a continuous measurement provides a more informative and accurate semantic analysis, making the resulting uncertainty score more precise and reliable. We will further clarify this point in the revision.\n\n---\n\nComment: Some details of the implementation of semantic density are unclear.\n\nResponse: We can see you have posted specific questions following this comment, and will address them one by one below. We will add all these clarifications in the revision to make the implementation details clearer.\n\n---\n\nComment: Sec 3.5: Please describe how the NLI model is used for computing p_c, p_n and p_e. NLI models typically accept pairs of texts, but these probabilities seem to contain three: y_*, y_i and x.\n\nResponse: Each text is a concatenation of the prompt x and response y. The input to the NLI model is one pair of such texts: the first is the concatenation of x and y_i, and the second is the concatenation of x and y_*.  This design follows from the principle that the semantic relationship between a pair of responses (y_i and y_*) needs to be evaluated in the context of the prompt (x). We will add this clarification in the revised version.\n\n---\n\nComment: Do I understand correctly that embedding models (discussed in Sec 3.2) are never explicitly used in Semantic Density? Instead the NLI model is relied upon for establishing equivalence between texts and for the kernel function? If yes, the current presentation comes across as a bit confusing - please consider rewriting parts of the paper to make this more explicit.\n\nResponse: That is right; thanks for pointing out the need to clarify.  The SD approach aims to connect the density estimation to a semantic embedding space, which can be done either explicitly (using an embedding model) or implicitly (using a NLI model). As explained in Line 222-225, NLI models are used in the current implementation due to the lack of contextual embedding models in the existing literature.  Please note that utilization of a NLI model also assumes there exists an implicit contextual embedding space that measures the semantic relationships. We will add this clarification in Section 3 of the revised version.\n\n---\n\nComment: As I understand, the proposed approach only estimates uncertainty rather than explicitly calibrating the underlying model - and yet the proposed uncertainty estimation method helps improve predictive performance in Table 1. Am I understanding correctly that the underlying models are somewhat well calibrated to begin with? To put it another way, are there generation tasks where one might expect SD to not be as reliable? For example, a summarization task. If this is a reasonable understanding, please consider discussing where SD is unlikely to work well.\n\nResponse: The proposed semantic density does not modify the original underlying LLM. Instead, it provides an additional uncertainty measurement for each output response. In our experiments, all the LLMs are not guaranteed to be well calibrated. Indeed, the “NL” column in Table 1 can be seen as a measurement of calibration, and it varies a lot across LLMs.  SD performs robustly across most such variance, but we have also highlighted one case (Line 277-280 in the current manuscript) where its performance (and that of other metrics as well) is negatively affected by a badly calibrated LLM. Good calibration is thus important for uncertainty estimation in general. We will clarify this limitation in the revision.\n\nOn the other hand, we have added a summarization task (DUC2004) to the experiments. SD performs well in it, demonstrating generality across tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eAlwKf69u8", "reviewer": "Reviewer_mViE", "summary": "This paper addresses the problem of LLM’s lack of uncertainty metric for the response it generates. The authors propose to use semantic density to quantify such uncertainty, as it is not restricted to any specific downstream task. In particular, the approach samples reference responses, analyzes semantic relationships, and then calculates the semantic density. Experiments on four QA benchmarks of several Llama and Mistral models are conducted, including the latest Llama 3 13B and Mistral-8x22B, to demonstrate the effectiveness and the robustness of semantic density compared to prior metrics.", "questions": "In Figure 1, different models have inconsistent performance across different tasks. Do you have any explanations?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed metric is not restricted to any specific task but measuring the general LLM ability\n* The experiments show the effectiveness of the proposed approach, outperforming the other previous metrics", "weaknesses": "* Only some of the llama and mistral models are tested\n* The results shown in Table 1 seem to be quite close for all the models (on each task), which are not so indicative for system comparison\n* No further analysis/discussions on different model sizes (7B vs. 70B), nor different architecture (MoE vs. Non-MoE)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment: Only some of the llama and mistral models are tested.\n\nResponse: Thanks for the comment. We want to clarify that at the time of submission (May 22), all the open-sourced LLMs released by MistralAI were included in the experiments, e.g., 'Mistral-7B', 'Mixtral-8x7B', and 'Mixtral-8x22B', and all the open-sourced Llama3 models were included in the experiments, e.g., ‘Llama-3-8B', 'Llama-3-70B’. For Llama2, there were three open-source models, i.e., 'Llama-2-7b', 'Llama-2-13b', and 'Llama-2-70b', and we have included both 'Llama-2-13b' and 'Llama-2-70b' in our experiments. As an older and less capable small-size LLM, 'Llama-2-7b’ is usually replaced by 'Mistral-7B' or ‘Llama-3-8B' in practical usages, so we did not include it in our initial experiments. We will add 'Llama-2-7b’ in the revision for completeness.\n\n---\n\nComment: The results shown in Table 1 seem to be quite close for all the models (on each task), which are not so indicative for system comparison\n\nResponse: Thanks for the comment. Please allow us to further clarify how to interpret the results in Table 1, then we will address your concern. Each column in Table 1 represents one uncertainty quantification method, and each row shows the results for one base LLMs, on top of which we are applying those uncertainty quantification methods. In order to compare the performance among uncertainty quantification methods, we should look at the AUROC scores in each row, the higher the better. We have highlighted the best performing entry in boldface for each row. Regarding your concern here, the performance of other uncertainty quantification methods are not always consistent or close when applying to different base LLMs. The performances of the proposed semantic density (first column) are relatively more consistent across different models (though non-trivial differences can still be observed in some tasks). This actually indicates the better robustness of semantic density compared to other methods: it is able to provide consistently good performance when applying to base LLMs of varying sizes and structures. We will include this discussion in the revision.\n\n---\n\nComment: No further analysis/discussions on different model sizes (7B vs. 70B), nor different architecture (MoE vs. Non-MoE)\n\nResponse: Thanks for the constructive comment. As suggested, we have added an empirical analysis that compares the performance gain of semantic density vs. semantic entropy on different groups of base LLMs. The results indicate that the performance gain of semantic density is not sensitive to the model size or architectures.\n\n---\n\nComment: In Figure 1, different models have inconsistent performance across different tasks. Do you have any explanations?\n\nResponse: Thanks for the question. The reason is that these different LLMs are of different sizes and architectures, and since they are from different LLM families, their training pipelines and training data should also be different. It is therefore natural that their performances vary across different tasks, i.e., they have different capabilities, and they may be knowledgeable when answering one category of questions while ignorant of another topic. This is why we tried to include diverse LLMs: we want to evaluate the robustness of semantic density. We will add these discussions in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4vN1AL5rQY", "reviewer": "Reviewer_kNB4", "summary": "The paper proposes a novel framework for quantifying uncertainty in LLM's responses. The authors introduce the concept of \"semantic density\" (SD), which measures uncertainty from a probability distribution perspective in semantic space, applicable to any pre-trained LLM without additional training. Experiments on various state-of-the-art LLMs and benchmarks demonstrate that SD outperforms existing methods in accuracy and robustness. The paper presentation is overall easy to read.", "questions": "1. The literature review is not comprehensive, there are lots of related works [1, 2, 3] not mentioned or adopted in the experiment.  \n2. The AUROC is not enough the show the strength of the proposed model. It would be more beneficial to show Area under precision-recall curve for the misclassification detection task as well.  \n3. The prompt construction can have more details. In Sec. 3.2, it's worth denoting the final prompt structure, since the in-context learning examples and prompt structures can make the final result different.\n4. The dataset are mostly QA tasks, it seems the model can also handle other NLG tasks. Can you elaborate more on this?\n\n[1] Ling, Chen, et al., \"Uncertainty Quantification for In-Context Learning of Large Language Models.\" (NAACL 2024)   \n[2] Fadeeva, Ekaterina, et al. \"LM-polygraph: Uncertainty estimation for language models.\" (EMNLP 2023)    \n[3] Duan, Jinhao, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. \"Shifting attention to relevance: Towards the uncertainty estimation of large language models.\" ACL (2024).", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel method, semantic density, that quantifies uncertainty in LLM responses using semantic information. \n2. The proposed framework is clear and detailed, including its theoretical foundations and implementation details.\n3. The proposed method is off-the-shelf and can be applied to other scenarios.", "weaknesses": "1. The comparison with existing methods, although extensive, may not cover all possible alternatives. There are recent or lesser-known methods that also warrant consideration.\n2. While the method shows promise for free-form question-answering tasks, its applicability and performance in other types of tasks (e.g., summarization, translation) are not explored, which limits the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment: The comparison with existing methods, although extensive, may not cover all possible alternatives. There are recent or lesser-known methods that also warrant consideration.\n\nResponse: Thanks for the suggestions. We tried to include all the mainstream uncertainty quantification methods for LLMs based on our literature review. Following your suggestion, we have done another literature search for more recent works, even including the papers that are published after the submission deadline of our current manuscript (May 22). We found another variant of semantic entropy from a recent paper published in Nature [4] (published after our submission), and we have added it into our experiments. Based on the results, the performance of the new variant of semantic entropy is similar to the original version of semantic entropy, and the proposed semantic density consistently outperforms it with different base LLMs and datasets. We will include these results in the revision.\n\n---\n\nComment: While the method shows promise for free-form question-answering tasks, its applicability and performance in other types of tasks (e.g., summarization, translation) are not explored, which limits the generalizability of the findings.\n\nResponse: Thanks for your constructive comment. To further demonstrate the generalizability of the proposed method, we have added a summarization task with the commonly used DUC dataset. Based on the experimental results (please see detailed results in general response to all reviewers), the proposed semantic density still shows best performance compared to other counterparts, demonstrating its good generalizability. We will include these results in the revised version.\n\n---\n\nComment: The literature review is not comprehensive, there are lots of related works [1, 2, 3] not mentioned or adopted in the experiment.\n\nResponse: Thanks for the suggestion. We have actually already included an earlier version of [1] in our literature review, we will update it into this newest version. For [2], it is a library that implements some of the existing uncertainty quantification methods for LLMs, without proposing new methods. We will include it for completeness. We will add and discuss [3] in the revised version.\n\n---\n\nComment: The AUROC is not enough to show the strength of the proposed model. It would be more beneficial to show Area under precision-recall curve for the misclassification detection task as well.\n\nResponse: Thanks for this constructive comment. As you suggested, we have added the Area under precision-recall (AUPR) score as an additional performance metric. Please see table 1 in the attached PDF for detailed results. From the experimental results, the proposed semantic density shows consistently better performance compared to other methods, verifying its effectiveness.\n\n---\n\nComment: The prompt construction can have more details. In Sec. 3.2, it's worth denoting the final prompt structure, since the in-context learning examples and prompt structures can make the final result different.\n\nResponse: Thanks for this constructive suggestion. To make the comparisons fair, we used the same prompt template as used in the original semantic entropy paper [5]. We will add the detailed prompt template in the revised version.\n\n---\n\nComment: The dataset are mostly QA tasks, it seems the model can also handle other NLG tasks. Can you elaborate more on this?\n\nResponse: You are right. In principle, the proposed semantic density does not pose any restrictions on the task type, and it should work for a wide range of free-form NLG tasks. Following your previous suggestion, we have added a summarization task to demonstrate this generalizability of semantic density. We will also clarify this point in the revision.\n\n[1] Ling, Chen, et al., \"Uncertainty Quantification for In-Context Learning of Large Language Models.\" (NAACL 2024) \n[2] Fadeeva, Ekaterina, et al. \"LM-polygraph: Uncertainty estimation for language models.\" (EMNLP 2023) \n[3] Duan, Jinhao, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. \"Shifting attention to relevance: Towards the uncertainty estimation of large language models.\" ACL (2024).\n[4] Farquhar, Sebastian and Kossen, Jannik and Kuhn, Lorenz and Gal, Yarin, “Detecting hallucinations in large language models using semantic entropy”, Nature 630, 625–630 (2024)\n[5] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In The Eleventh International Conference on Learning Representations (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d7K3rGHU0Z", "reviewer": "Reviewer_koQE", "summary": "The paper introduces the metric \"Semantic Density\" to quantify the uncertainty of LLMs by measuring the distance of response embeddings in a semantic space.", "questions": "- What do the authors exactly mean by Semantic Density rebuilding the output probability distribution from a semantic perspective? (line 58)\n- In line 277, the authors write \"After investigation, the inherent sequence likelihood returned by the original LLM was badly calibrated in these two cases.\". What is meant by badly calibrated sequence likelihoods? Why didn't the authors account for this by recomputing the sequence likelihood?\n- There exists independent work (Kernel Language Entropy) that is closely related to Semantic Density [2]. The authors should consider discussing this in their paper.\n- It would be insightful to additionally report the performance of existing uncertainty estimation methods when decreasing the number of reference responses instead of only reporting Semantic Density with different models (section 4.2)\n- Semantic Density uses diverse beam search to sample responses, requiring hyperparameters such as diversity penalty and number of groups. How have these hyperparameters been chosen? Is the method sensitive to the specific sampling strategy? An ablation study on the performance of Semantic Density compared to existing uncertainty estimation methods would be insightful.\n- Have the authors considered using sampling strategies other than diverse beam search? For instance, Aichberger et al. (2024) introduced sampling semantically diverse sequences, which could be utilized to sample unique reference responses [1]. \n\n---\n[2] A. Nikitin, J. Kossen, Y. Gal, and P. Marttinen. Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities. arXiv preprint arXiv:2405.20003, 2024.", "limitations": "The authors did not address the limitations of their work, only the limitations of existing uncertainty estimation methods.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Structure**: The problem is well-motivated and the paper is clearly and coherently written.\n- **Theory**: The adaptation of kernel density estimation techniques to semantic space for uncertainty quantification in NLG is novel.\n- **Experiments**: The experiments were performed on multiple datasets and various SOTA LLMs, where Semantic Density shows good empirical performance given the considered correctness metric.", "weaknesses": "- **Theory**: Semantic Density is theoretically not well grounded in uncertainty quantification theory, but proposed as an ad-hoc solution for practical application. So, although it seems intriguing to assign an uncertainty score response-wise rather than prompt-wise, uncertainty theory suggests that the (aleatoric semantic) uncertainty has to be quantified prompt-wise, no matter what ends up being sampled as the response [1]. However, since the true uncertainty score for a given prompt is unknown, the performance of an uncertainty measure is usually evaluated by how well the scores align with the correctness of the most-likely response. \n- **Experiments**: According to the experimental setup in line 897, an answer is considered to be correct if its Rouge-L to any of the reference answers is larger than 0.3 for all datasets. However, since the considered datasets usually result in short responses, a fixed threshold of 0.3 is not sufficient. For instance, in the TriviaQA question *What is the last Grand Slam tennis tournament played in a calendar year?* the reference answer *The US Open* and the sampled response *The Australian Open* achieve a Rouge-L of approximately 0.66. This highlights the fact that considering multiple thresholds is essential to assess the performance of uncertainty metrics.\n\n---\n[1] L. Aichberger, K. Schweighofer, M. Ielanskyi, and S. Hochreiter. Semantically Diverse Language Generation for Uncertainty Estimation in Language Models. arXiv preprint arXiv:2406.04306, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, we will answer your concerns follows the order of your original comments, due to the space limitation.\n\n---\nComment: concern regarding theory\n\nResponse: Thanks for this insightful comment. We have read reference [1] carefully (it was published in arXiv only after the submission deadline, so we were not aware of it before), and will  address your concerns from two perspectives: First, considering the fact that LLMs sample responses stochastically given the same prompt, an uncertainty metric for each specific response has practical value. This is also reflected in reference [1], which  states “Uncertainty estimation in NLG involves assessing the uncertainty of an initially generated text (output sequence) for a given prompt (input sequence)”.The prompt-wise aleatoric uncertainty metric derived in [1] was indeed used to quantify the uncertainty of the most likely response (i.e.\\ the initially generated response). While this estimate provides a good first approximation, assigning the same prompt-wise uncertainty score to different sampled responses is less useful in practical use cases where the trustworthiness of specific responses matters.The main motivation of the proposed semantic density is thus to provide a  solution for such use cases, i.e. to quantify the uncertainty of any specific response (not only the most likely one) that may be sampled by the LLM.. \n\nSecond, as an emerging research direction, uncertainty quantification in LLMs indeed needs more unified definitions of terms related to “uncertainty”. In the literature, different definitions and usages of the term “uncertainty” can be found, and sometimes other terms like “confidence” or “certainty” are used for the same purpose. Definitions of “uncertainty of a response” and “uncertainty of a prompt” should also be defined and differentiated. We will clarify that the “uncertainty” in this paper refers to “uncertainty of a response”, and delineate the different use cases. . We will also discuss reference [1] since we believe that it forms a good theoretical foundation for this emerging direction.\n\n---\n\nComment: concer regarding the experiments\n\nResponse: Thanks for this constructive comment. As suggested, we have added experimental results using Rouge-L with different thresholds ranging from 0.1 to 1.0. According to the results, the performance gain of semantic density is not sensitive to the choice of Rouge-L thresholds.  The proposed semantic density consistently outperforms other methods across this range, further demonstrating that the results are robust.\n\n---\n\nResponse: Sorry for the confusion. We wanted to say that the original output distribution of LLMs is in sequence-of-token space, and we want to consider output distribution in a different semantic space. Theoretically, the output distribution in semantic space can be rebuilt from the output distribution in original sequence-of-token space. We will clarify this point in the revision.\n\n---\n\nResponse: Thanks for the question. Here, “badly calibrated sequence likelihoods” means the original likelihood of the LLM to sample the output token sequence is not well aligned with the correctness of the output sequence. We did not fine-tune or re-calibrate the original LLMs in our current experiments, because we wanted to evaluate the robustness of the proposed semantic density when working with existing LLMs in their original form. Post-hoc fine-tuning and calibration methods are complementary to semantic density, and they can always be applied to further improve the reliability of semantic density.\n\n---\n\n\nResponse: Thanks for the suggestion. We will include and discuss it in “related works”.\n\n---\n\n\nResponse: Thanks for the suggestion. We have added an experiment that investigates the performance change of semantic entropy when decreasing the number of reference responses. The patterns are similar for semantic entropy, but semantic density exhibits consistently better performance when the number of reference responses is reduced. We will add these results in the revision.\n\n---\n\n\nResponse: Thanks for the suggestion! We did not perform an extensive hyperparameter search for diverse beam search, and we used the standard default parameters. As suggested, we have added an ablation study following the setup in reference [1], which tests different choices of diversity penalty. Based on the results, a diversity penalty of 1.0 indeed provides the best performance, but the performance difference between 0.5 and 0.2 is not significant. We will add this empirical study to the revised version.\n\n---\n\n\nResponse: We have also tried standard multinomial sampling, and found that diverse beam search was better in generating unique and diverse generations. The SDLG sampling strategy in reference [1] is indeed a good match with semantic density, i.e., more diverse and likely reference responses would make semantic density more reliable. We will discuss it as a potential replacement for diverse beam search in the revision.\n\n---\n\n\nResponse: We have actually discussed the limitations of the method in several places in the original manuscript. Please see more details and justifications in section 2 of the appended NeurIPS Paper Checklist (page 15, after the references).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for their valuable time and constructive comments. We have considered every comment from each reviewer, and added several experiments as suggested by reviewers. We believe the paper is in a much better form after incorporating constructive suggestions from all the reviewers. Below is a brief summary of the new experimental results we added during rebuttal. We will also use this space to display some of the detailed results:

1. We added a new summarization dataset as suggested by reviewer rbw2 and kNB4. Below shows the detailed results on Mistral-7B and Mixtral-8x7B:
| AUROC        | SD    | SE    | Deg   | NL    | NE    | PE    |
|--------------|-------|-------|-------|-------|-------|-------|
| Mistral-7B   | 0.679 | 0.537 | 0.660 | 0.673 | 0.551 | 0.549 |
| Mixtral-8x7B | 0.640 | 0.532 | 0.620 | 0.636 | 0.576 | 0.497 |

2. We added an experiment that uses area under precision-recall curve (AUPR) score as performance metric, as suggested by reviewer kNB4. Please see Table 1 in the attached PDF for detailed results.
3. We added an experiment that uses different rouge-L threshold for correctness evaluations, as suggested by reviewer koQE. 
4. We added an experiment that uses different hyperparameters for the diverse beam search, as suggested by reviewer koQE.
5. We added an experiment that studies the effect of sampling numbers on semantic entropy, as suggested by reviewer koQE.  
6. We added a comparison to a more recent variant of semantic entropy, as suggested by Reviewer kNB4.

Meta-review: The paper proposes an approach based on kernel density estimation techniques in the semantic space for uncertainty quantification in text generation for large language models. The paper evaluated on question answering data sets, and during the rebuttal process, added results on summarization data sets as well. The reviewers agreed that the approach is principled and the emprical results are strong across benchmarks.

One concern from reviewer koQE is that semantic density lacks a solid theoretical foundation within the context of current uncertainty quantification. However, I think in practical terms, calibration to quantify the confidence of a sampled answer (e.g., at temperature 0) is useful, and the evaluation metric of AUROC also reflects this goal.
Decision: Accept (poster)