MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs (2024)

Abstract: Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. However, their tendency to produce inaccurate or misleading outputs poses a potential risk, particularly in high-stakes environments. Therefore, estimating the correctness of generative LLM outputs is an important task for enhanced reliability. Uncertainty Estimation (UE) in generative LLMs is an evolving domain, where SOTA probability-based methods commonly employ length-normalized scoring. In this work, we propose Meaning-Aware Response Scoring (MARS) as an alternative to length-normalized scoring for UE methods. MARS is a novel scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. We demonstrate that integrating MARS into UE methods results in a universal and significant improvement in UE performance. We conduct experiments using three distinct closed-book question-answering datasets across five popular pre-trained LLMs. Lastly, we validate the efficacy of MARS on a Medical QA dataset. Code can be found here.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7752–7767 August 11-16, 2024 ©2024 Association for Computational Linguistics
Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. However, their tendency to produce inaccurate or misleading outputs poses a potential risk, particularly in high-stakes environments. Therefore, estimating the correctness of generative LLM outputs is an important task for enhanced reliability. Uncertainty Estimation (UE) in generative LLMs is an evolving domain, where SOTA probability-based methods commonly employ length-normalized scoring. In this work, we propose Meaning-Aware Response Scoring (MARS) as an alternative to length-normalized scoring for UE methods. MARS is a novel scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. We demonstrate that integrating MARS into UE methods results in a universal and significant improvement in UE performance. We conduct experiments using three distinct closed-book question-answering datasets across five popular pre-trained LLMs. Lastly, we validate the efficacy of MARS on a Medical QA dataset. Code can be found here.

heading: 1 Introduction
text: Generative Large Language Models (LLMs) have risen in popularity due to their remarkable ability to understand, generate, and process human language at an unprecedented scale and accuracy (Ye et al., 2023; OpenAI, 2023; Touvron et al., 2023). These models have become the state-of-the-art in various fields, including machine translation, content generation, and even scientific research (Huang et al., 2023; OpenAI, 2023) due to their capability to handle diverse tasks such as text summarization, sentiment analysis, and question-answering in a few-shot or zero-shot manner.
*This work does not relate to their position at Amazon.
Despite their growing popularity and success, generative LLMs are not infallible and can sometimes produce erroneous or misleading outputs, especially when dealing with complex reasoning problems or closed-book questions. This limitation becomes particularly critical in question-answering systems used in high-stakes environments. Quantifying the uncertainty of generative LLM responses in such scenarios is not just beneficial but essential for ensuring trustworthy operation. For example, in a medical advice application, accurately assessing the uncertainty of the responses provided by LLMs can prevent the provision of incorrect medical advice. This is crucial because erroneous advice may lead to devastating medical missteps or misunderstandings. Thus, understanding and quantifying uncertainty helps in reliable risk assessment and in maintaining the overall quality of the answers provided, ensuring that users can assess how much reliance they should place on LLM responses.
Uncertainty Estimation (UE) is a well-studied problem in classification scenarios, especially in the computer vision domain (Lakshminarayanan et al., 2017; Gal and Ghahramani, 2016; Shen et al., 2021). The proposed UE methods in classification tasks, which rely on the class probabilities, are not directly applicable to generative LLMs due to the auto-regressive generative structure of LLMs (Malinin and Gales, 2021), which implies that LLMs generate text sequentially by predicting each subsequent word based on the combined context of all preceding words. This process differs significantly from classification tasks, where the output is typically a single label or a set of labels assigned to an entire input, without the sequential and contextaccumulating nature of generative LLMs.
Recent work (Malinin and Gales, 2021), formalizes how to adapt popular UE methods developed for classification tasks to the context of generative LLMs. They propose using length-normalized scoring to estimate the likelihood of a sequence gener-
7752
ated by the model, and the subsequent works (Kuhn et al., 2023; Lin et al., 2023; Chen and Mueller, 2023) utilize that idea of length-normalized scoring.
A downside of these existing UE techniques in the generative LLM literature is treating lengthnormalized scoring like the class probabilities in classification tasks. However, better ways may exist for estimating uncertainty than directly using the length-normalized score of a sequence, as it treats all tokens equally. In reality, each word’s contribution to the sentence’s meaning in the question context might vary. For example, given the question “Which planet is known as the Red Planet?” and with the generated response “Mars is the Red Planet”, the tokens of “Mars” are the most critical ones in the response because those tokens are the ones actually answering the question. Thus, assigning more weight to semantically significant tokens in the response score calculation can improve UE methods, resulting in more accurate predictions.
Based on this word importance intuition, we propose a novel scoring function for generative LLMs called Meaning-Aware Response Scoring (MARS), as outlined in Figure 1. To compute the LLM response score as an input to UE methods, we first assign an importance coefficient to each token in the generation. This importance essentially reflects the impact of masking a token in a sequence on the meaning of the generated response, where tokens with a greater influence on the meaning receive higher importance. By leveraging these meaningaware coefficients (wi in Figure 1), MARS returns the multiplication of the weighted probabilities of the tokens in the generated sequence.
We list our main contributions as follows:
• We propose a novel scoring function for UE
in generative LLMs named Meaning-Aware Response Scoring (MARS).
• We introduce a BERT-like model, efficiently assigning meaning-aware importance weights to the tokens in a single model pass within MARS calculation.
• We explain previous works’ (Malinin and Gales, 2021; Kuhn et al., 2023) design choices from the classification perspective to create a grounded framework for MARS and other scoring functions.
• We evaluate probability-based UE metrics with MARS on question-answer datasets and show that MARS universally improves the UE performance for an extensive list of LLMs.

heading: 2 Background
text: In this section, we will go over probability-based UE methods that our work built on. For a detailed discussion on related works, refer to Appendix A.
In the literature, UE is used as a proxy for the correctness of the model output (Malinin and Gales, 2021; Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017; Band et al., 2021). For generative LLMs in the question-answer context, we consider the most probable sequence as the model output and utilize UE to predict the correctness of the response following Kuhn et al. (2023). The goal of UE is to assign higher scores to incorrect responses, indicating greater uncertainty, and lower scores to correct responses, signifying less uncertainty.

heading: 2.1 Bayesian View to Estimate Uncertainty
text: Bayesian UE is used in machine learning to quantify uncertainty in predictions. It treats model parameters as random variables, assigning a prior
probability distribution to them. Through Bayesian inference, this distribution is updated with training data, yielding a posterior distribution. Prediction uncertainty stems from this posterior distribution. Let {θi}Mi=1 be an ensemble of models sampled from approximate posterior q(θ) ≈ p(θ|D) where D is the training data.
The predictive posterior of input x ∈ X for target y ∈ Y is derived by expectation over the ensemble:
P (y|x,D) = Eq(θ)[P (y|x, θ)]
≈ 1 M
M∑
m=1
P (y|x, θm), (1)
where we have θm ∼ q(θ) ≈ p(θ|D). Using the posterior probability definition, we can define the entropy of predictive posterior as:
H(x,D) = − ∑
y∈Y P (y|x,D) logP (y|x,D). (2)
In classification tasks, commonly used tools for estimating uncertainty are the entropy of the predictive posterior and the negative predictive posterior probability of the most probable answer (Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017; Malinin and Gales, 2021; Xiao et al., 2022; Chen and Mueller, 2023). However, the formulation in (1) is not applicable to generative LLMs because of their auto-regressive generative structure.

heading: 2.2 Uncertainty Estimation (UE) of Auto-Regressive Generative Models
text: Malinin and Gales (2021) formalizes posterior probability definition for auto-regressive generative models where the output s is not a single entity but a sequence of tokens s = {s1, s2, ..., sL}. They simply replace P (y|x, θ) in (1) with sequence probability P (s|x, θ). The probability of a sequence s for a given model parametrized with θ is defined as the multiplication of probabilities of its tokens:
P (s|x, θ) = L∏
l=1
P (sl|s<l,x; θ) (3)
where s<l ≜ s1, s2, .., sl−1 referring to generated tokens before the generation of sl. Kuhn et al. (2023) simplifies the ensemble sampling in (1) by using a single model in the ensemble due to the large size of foundation models. We follow the
simplified version in the rest of the paper:
P (s|x, D)≈P (s|x, θ)= L∏
l=1
P (sl|s<l,x; θ). (4)

heading: 2.3 Length-Normalized Scoring
text: One of the key issues with using sequence probability P (s|x, θ) as a proxy for P (y|x, θ) lies in its tendency to decrease as the sequence length increases. To overcome this issue, Malinin and Gales (2021) uses length-normalized scoring function instead of sequence probability.* Length-normalized scoring P̃ (s|x, θ) is defined as follows:
P̃ (s|x, θ) = L∏
l=1
P (sl|s<l,x; θ) 1 L , (5)
which assigns equal weights to each token in the generation where these weights are inversely proportional to the sequence length L. Although length-normalized scoring P̃ (s|x, θ) does not correspond to an actual probability distribution, Malinin and Gales (2021) and Kuhn et al. (2023) consider P̃ (s|x, θ) as auxiliary probabilities and replace the sequence probability P (s|x, θ) in (4) with the length-normalized scoring given in (5).

heading: 2.4 Entropy-Based UE for Generative LLMs
text: To obtain the entropy of the output for given input x, Malinin and Gales (2021) uses Monte-Carlo approximation over beam-sampled generations of a single model, as going through the entire answer set is infeasible due to its exponential computation complexity. Approximated entropy is defined as:
H(x, θ) ≈ − 1 B
B∑
b=1
ln P̃ (sb|x, θ), (6)
where sb is an output sampled by beam-search and B is the total number of sampled generations.
Kuhn et al. (2023) proposes an alternative entropy definition, named Semantic Entropy (SE), considering the meaning of the generations. They use the same entropy definition in (6), but cluster sampled generations based on their meaning. For example, in response to the question “What is the capital city of France?”, a model might output: “Paris” with score p̃1 and “It’s Paris” with score p̃2. While standard entropy in (6) treats these as distinct
*A scoring function K takes two inputs: the predicted probability p of an event and its actual outcome o, and returns a numerical score (Gneiting and Raftery, 2007).
outputs, SE clusters them together as they convey the same meaning in the question context, forming a single cluster c with summed score p̃1+ p̃2. More formally, cluster scoring is defined as:
P̃ (c|x, θ) = ∑
s,x∈c P̃ (s|x, θ). (7)
SE follows from this cluster scoring P̃ (c|x, θ):
SE(x, θ) = − 1|C|
|C|∑
i=1
log P̃ (ci|x, θ), (8)
where ci refers to each semantic cluster and C is the set of all clusters. In Section 4, we provide an alternative explanation for Semantic Entropy and length-normalized scoring from a classification task perspective.
Negative length-normalized scoring of the most probable answer, standard sequence entropy in (6) and semantic entropy in (8) are the most common probability-based UE methods for generative LLMs (Malinin and Gales, 2021; Kuhn et al., 2023; Chen and Mueller, 2023; Lin et al., 2023) and are visualized in Figure 2. These methods depend on length-normalized scoring and recent work (Duan et al., 2024) replaces that scoring function by considering the meaning of the generation. Similar to Duan et al. (2024), we aim to replace that scoring with MARS. We discussed the differences between our work and Duan et al. (2024) in Appendix A.

heading: 3 Method

heading: 3.1 Key Intuition
text: Existing literature utilizes length-normalized scoring in UE as shown in (5), (6), and (7). Length-
normalized scoring, given in (5), assigns equal importance/weight (1/L) to each token in the generated sentence. The normalization aims to compare the probabilities of short and long sequences more fairly (Malinin and Gales, 2021). Such a normalization method may fall short in considering semantic contribution of tokens, even though it balances length differences across sequences.
To illustrate, consider the following example: Question: “Which planet is known as the Red Planet?” Generated Answer: “Mars is known as the Red Planet". In this answer, the word “Mars” is relatively more important as it directly addresses the question. Other words in the sentence primarily serve syntactic purposes or help achieve humanlike answer. Thus, while designing a scoring function, we should give more importance/weight to the word “Mars”. With this intuition, we want to replace length-normalized scoring and propose an alternative scoring function that assigns importance/weight to each word in the sentence considering both its contribution to the overall meaning in the given context and sequence length.

heading: 3.2 Meaning-Aware Response Scoring
text: Following our word importance intuition, we propose to replace length-normalized scoring P̃ (s|x, θ) in (5), (6), and (7) with Meaning-Aware Response Scoring (MARS). MARS is defined as:
P̄ (s|x, θ) = L∏
l=1
P (sl|s<l,x; θ)w(s,x,L,l), (9)
where w(·) is the weighting function that assigns a weight to each token regarding the generated answer, question context, and sequence length.
We design w(·) as a convex combination of importance coefficient and 1/L, which enables MARS to consider both sequence length and meaning contribution of tokens. Formally, we define
w(s,x, L, l) ≜ 1 2L + u(s,x, l) 2 , (10)
where u(·) is importance function taking three arguments: generated sequence s, contextual information x, and the position l of a token within the sequence. The function u(·) assigns an importance coefficient to each token, where this coefficient ranges between 0 and 1. Additionally, it ensures that the total sum of the importance coefficient for all tokens in a single generation s is 1. Next, we explain how to design the importance function u(·).

heading: 3.3 Importance Function Design
text: We design the token importance function u(·) by measuring the semantic impact of removing a specific token from the generated text. This evaluation of meaning is context-sensitive. In question-answer tasks, which is the focus of this work, the context is defined as the question itself. Thus, u(·) is designed to determine the importance of each token based on its influence on the overall meaning of the response within the context of the question.
To measure the amount of semantic change in the given context, we employ a neural network model originally developed as a question-answer evaluator by Bulian et al. (2022). This model, called BERT matching (BEM), takes three inputs: question, ground truth answer, and predicted answer, returning a probability score indicating answer correctness. For a question x and a generated answer s = {s1, s2, . . . , sL}, we determine the importance of each token as follows: We mask token sl in the generated answer and feed the question x, the original answer s, and masked response sequence s \ {sl} into the BEM model. The output o, ranging from 0 to 1, indicates the impact of the masked token on answer correctness. A token sl with substantial impact yields an output o close to 0, whereas a lesser impact results in an output closer to 1. Hence, we define 1 − o as the preliminary coefficient of sl. Once we compute preliminary coefficients for all tokens, we normalize them using a softmax function with a temperature parameter τ . In our experiments, we set τ = 0.01.
Addressing Token Dependency. Our initial approach for assigning importance coefficients
to tokens assumes their semantic independence even though tokens often exhibit semantic interdependencies. For example, in the sentence “Hamlet is written by William Shakespeare,” tokens “William” and “Shakespeare” are intrinsically linked. Treating such tokens independently ignores linguistic nuances, so we refine our methodology. Instead of masking tokens individually, we mask tokens at the phrase level (details in Appendix B.1). This approach acknowledges and preserves the inherent semantic relationships between closely related tokens, resulting in a more accurate and contextually aware assessment of token importance. In particular, a response s = {s1, s2, . . . , sL} is composed of phrases {h1, h2, . . . , hK}, where each token sl belongs to a phrase hk. We mask phrases one by one and find the importance coefficient of each phrase with BEM model. To translate phraselevel importance coefficients into token-level coefficients, we distribute the importance score to all tokens in the phrase equally. We summarize the enhanced algorithm in Appendix B.2. Further, in Section 5.3, we show that allocating importance score only to the most uncertain token within a phrase also yields comparable results.
Reducing Computation. The necessity of performing a separate neural network pass for each phrase to determine its importance score increases the computational load of the proposed approach. Additionally, detecting phrases themselves requires another neural network pass, further increasing the computational complexity. To address these challenges, we have developed a BERT-like neural network model with 110M parameters (a significantly smaller model compared to LLMs). This model is capable of performing both tasks simultaneously for a given sequence in a single neural network pass: it identifies phrases within the generated text
and their importance scores (see Figure 3). This dual-functionality substantially reduces the computational cost, making the algorithm more efficient and scalable. For detailed model architecture and performance metrics, please refer to Appendix B.

heading: 4 Understanding Generative LLM Probabilities from a Classification Perspective
text: In classification tasks, the class probability reflects the model’s confidence in assigning a specific class to an input. It is inherently tied to the semantics of the class. For instance, if a well-calibrated classifier gives a 75% probability to the label “cat” for a given image, it suggests a 75% likelihood that the given image is indeed a cat. This output probability is not only a numerical value; it conveys a semantic understanding of the image content as a cat. To understand previous works (Kuhn et al., 2023; Malinin and Gales, 2021) from a classification perspective, we propose a new random variable that is directly related to the semantics of the output.
Let Y be a random variable with arbitrary dimension corresponding to the meaning of the sequences generated by an LLM parametrized with θ. The values of Y can be the set of all possible meanings of generated sequences and their contexts. Formally, the set is {g(s, x)}s∈S,x∈X , where g(·) is the meaning function that takes generated sentence s and context x as inputs and returns the meaning as output. A well-calibrated probability distribution of Y ensures that if a model’s generation is more likely to be correct than another, the corresponding probability should be higher. Mathematically, P (Y = g(s1, x1)|θ) > P (Y = g(s2, x2)|θ) if the model’s response s1 to question x1 is more likely to be correct than response s2 to question x2. This is indeed similar to the classification example where well-calibrated class probabilities reflect the output’s actual probability of correctness (Guo et al., 2017). Previous works and MARS have heuristic design choices to make Y more calibrated to get a better uncertainty estimation.
Malinin and Gales (2021) considers g(·) as a one-to-one function which means that each unique sentence in the given context corresponds to different meanings. To achieve better calibration of Y they define its distribution using length-normalized scoring of the generated sequences. This approach ensures a fair comparison between long and short sequences, thereby improving the calibration of Y .
More formally
P (Y = y|θ) = P̃ (s|x, θ)∑ s∈S,x∈X P̃ (s|x, θ) , (11)
where y = g(s, x) and P̃ (s|x, θ) is the length-normalized scoring defined as∏L
l=1 P (sl|s<l, x; θ)1/L. To make the distribution of Y a valid probability distribution, we normalize each P̃ (s|x, θ) by the sum of all possible scores, making their summation 1. By defining Y as above, we essentially create an actual probability distribution of length-normalized scoring.
On the other hand, Kuhn et al. (2023) claims different sequences can have equal meaning. By considering g(·) as a many-to-one function, we can define the distribution of Y with their proposal as follows:
P (Y = y|θ) = ∑
s,x∈cy P̃ (s|x, θ)∑ s∈S,x∈X P̃ (s|x, θ)
(12)
where cy corresponds to the meaning cluster, formally written as cy = {s, x|g(s, x) = y}. By employing this new probability definition within the standard entropy calculation in (6), we obtain the concept of semantic entropy as follows
SE(x, θ) = − 1 B
B∑
b=1
logP (Y = yb|θ) (13)
With the new random variable Y , we essentially write the semantic entropy as the standard MonteCarlo approximated entropy over a total of B distinct meanings. Notice that the normalization term∑ s∈S,x∈X P̃ (s|x, θ) featured in both (11) and (12), acts as a constant across all P (Y = y|θ) calculations, ensuring that Y conforms to a valid probability distribution. Therefore, it only shifts the proposed UE scores which does not affect the performance of accurately predicting the correctness of the model generation. Moreover, by introducing the random variable Y , we not only provide an alternative explanation of the previous works but also create flexibility to define new distributions for Y which may potentially have better calibration and improve the existing UE tools.
Using the definition of Y , we can also rationalize our scoring function MARS. We replace the length-normalized scoring function with MARS
as in (9). MARS considers the semantic contribution of tokens, assigning higher weights to those critical for the correctness of the answer. This emphasis on key tokens potentially makes MARS a more effective scoring function for achieving better calibration of Y .

heading: 5 Experiments

heading: 5.1 Experimental Design
text: In the UE context, we expect that if the model is uncertain about the generated answer, then the answer should be less reliable and tend to be incorrect.
Datasets. We use three closed-book QuestionAnswer (QA) datasets for evaluation: TriviaQA (Joshi et al., 2017), Natural Questions (Kwiatkowski et al., 2019), and WebQA (Chang et al., 2022). We give further details in Appendix C.
Models. Our evaluation consists of 5 popular open-source LLMs. First two models are Llama7B and Llama-7B-chat, where the latter one is finetuned for dialogue use cases (Touvron et al., 2023). We also use Mistral-7B (Jiang et al., 2023) as well as Falcon-7B (Almazrouei et al., 2023) which is fine-tuned on a mixture of chat/instruct datasets. To extend our analysis to larger models, we include Llama-13B (Touvron et al., 2023). We do not perform any further training on these models, rather we use their pre-existing configurations. Following Kuhn et al. (2023), we abstain from assuming any ensemble of the models, considering the significant size and time requirements associated with LLMs.
Baselines. As we focus on the probability-based UE methods, we do not include heuristic-based and black-box methods. We use 3 SOTA probabilitybased UE methods as baselines (see Figure 2 for visualization): 1. Negative length-normalized score (Confidence), which provides the confidence score of the most likely generation only by using its token probabilities as in (5). 2. Entropy as in (6), which requires generating multiple answers to obtain the score for the most likely answer. 3. Semantic Entropy (SE), which considers the meaning of the generated answer while computing entropy, as shown in (8). All 3 baselines depend on lengthnormalized scoring. We replace length-normalized scoring with MARS and arrive at Confidence + MARS, Entropy + MARS, SE + MARS.
Metrics. Following previous works (Malinin and Gales, 2021; Kuhn et al., 2023), we use Area Un-
der the Receiver Operating Characteristic Curve (AUROC) score for our UE performance metric. AUROC quantifies a method’s ability to distinguish between two classes by plotting the true positive rate against the false positive rate for various threshold values. AUROC score is the area under this curve, ranging from 0 to 1. Higher AUROC score indicates a superior performance, while a score of 0.5 implies a random chance. In our case, ground truth is the correctness† of the model response to the question and the prediction is the output of an UE method.

heading: 5.2 Main Results
text: We present our detailed results in Table 1. Upon closer examination of the results, it becomes apparent that the application of MARS consistently improves all baseline methods across various datasets and models. Specifically, MARS yields improvements of up to 5.8 points for Confidence, 6.24 points for Entropy, and 1.51 points for SE.
It is crucial to mention that the choice among the baselines depends on the available computational resources. Confidence score is the least resourceintensive, requiring only a single output generation. Entropy, on the other hand, demands multiple generations (set to 5 in our experiments). SE is the most computationally demanding, needing both multiple generations and O(n2) Natural Language Inference (NLI) model passes for clustering, where n represents the number of generations. We note that the additional computational and memory demands of MARS are relatively minor, approximately 1.5% of the 7b models and 0.8% of the 13b models, because MARS’s importance function is implemented with 110M Bert-like model.

heading: 5.3 Ablation Studies
text: Effect of Phrase Separation. In Section 3.3, we suggest using a phrase-level separation instead of token-level separation in designing the importance function so that tokens having strong relations are evaluated together on their semantic impact on the sequence. To validate this design, we conduct an experiment where we revert to token-level separation. The results in Table 2 demonstrate that while token-level separation outperforms other baselines, phrase-level separation consistently yields superior results, reaffirming the efficacy of our approach.
†We use GPT-3.5-turbo for evaluating the correctness of the model, as in (Lin et al., 2023; Chen and Mueller, 2023).
Importance Coefficient Distribution in Phrases. In Section 3.3, we state that we equally distribute the importance of phrases to each token. Alternative distribution strategies might include prioritization of the least or most uncertain token. Those strategies assign the phrase importance coefficient to the least or most uncertain token of that phrase. In Table 3, we provide AUROC performances when different distribution strategies are adopted. Notably, we find that max-uncertain distribution is nearly as effective as our adopted equally assigning approach. In contrast, the min-uncertain assigning strategy underperforms. This outcome can be
contextualized with a hypothetical scenario: Consider the model’s response is “Shakespeare” to the query “Who wrote Hamlet?”, which is tokenized into “Shake” and “-speare”. Once “Shake” is produced, the subsequent arrival of “-speare” is almost assured. The uncertainty primarily resides in the token “Shake”, making the probability of “-speare” relatively uninformative. Consequently, focusing on the least uncertain (most uninformative) token in a phrase drops the performance of MARS significantly, and focusing on the most uncertain token only is still reasonable.

heading: 5.4 Effect of Sampling Hyperparameters
text: We explore the influence of key hyperparameters on the performance of UE methods that rely on sampling, specifically Entropy and SE. We focus on two critical hyperparameters: Temperature, which adjusts the diversity of the sampling process, and the number of sampling, which dictates how many samples are sampled in entropy calculation.
Temperature. The temperature parameter determines the smoothness of the probabilities while sampling. A higher (lower) temperature value indicates more (less) diverse sampling. Figure 4 presents the AUROC scores for Entropy, SE, and their enhancements via MARS for the Llama2-13b and Mistral-7b models on the TriviaQA dataset. The improvement of MARS is consistent for all temperature values. The choice of temperature is application-dependent: higher temperatures are advisable for tasks demanding creativity, whereas lower temperatures are preferable for applications
where consistency is important.
Number of Sampling. The number of sampled sequences is important for entropy and semantic entropy calculation. More sampling leads to better entropy estimation; however, the cost also increases. Beyond the sampling expense, SE incurs an additional cost from Natural Language Inference (NLI) model passes, a point elaborated in Section 5.2. In Figure 4, we provide the AUROC performance of Llama2-13b and Mistral-7b models on TriviaQA with various sampling numbers. Notably, the efficacy of MARS remains stable across diverse sampling numbers, with its advantages becoming more obvious under lower sampling numbers.

heading: 5.5 UE in Medical QA Dataset
text: Next, we evaluate the UE methods using a medical QA dataset. Publicly available medical QA datasets typically fall into two categories: those with multiple-choice questions (Pal et al., 2022; Kotonya and Toni, 2020; Jin et al., 2021) and those without clear ground truths (Zhu et al., 2019, 2020). To tackle this, we create a subset from the MedMCQA multiple-choice dataset (Pal et al., 2022), selecting questions that can be answered objectively without multiple choices. For this, we collaborate with medical professionals to ensure the accuracy and relevance of the selected questions, yielding a dataset of 415 samples. We use AdaptLLM’s Medicine-Chat (Cheng et al., 2023), a medicaldomain adapted LLaMA-2-Chat-7B model‡. To evaluate the correctness of model-generated responses, we leverage GPT-4 (OpenAI, 2023) and assess response validity in the medical domain.
In Table 4, we provide the AUROC performance of the UE methods. Although MARS still consistently improves the performance of probabilitybased UE methods, AUROC scores are still low compared to Table 1. This might be because of the nature of medical questions. General knowledge questions mostly require a straight, single-
‡https://huggingface.co/AdaptLLM/medicine-chat
sentence answer. On the other hand, although we curated closed-ended questions, medical questions still require a more complex explanation spanning multiple sentences. This difference between domains can affect the prediction performance of the probability-based methods. This observation emphasizes the necessity for further investigation across various specialized fields, including medicine and law. Customized explorations are essential to address domain-specific challenges and optimize UE methods accordingly.

heading: 6 Conclusion
text: We introduce Meaning-Aware Response Scoring (MARS), a novel scoring function designed to replace length-normalized scoring in probabilitybased UE methods when evaluating generative LLMs. MARS consistently and significantly boosts the performance of current probability-based UE methods with minimal additional computational overhead. The efficacy of MARS is shown in three closed-book and closed-ended question-answer datasets and a medical question-answer dataset.

heading: 7 Limitations
text: The importance function model within MARS utilizes an unsupervised methodology, leveraging preexisting models for its formulation. Nonetheless, the performance of MARS can potentially be further enhanced by using human labelers to assign importance coefficients for training the importance function model. Besides, our analysis is limited to the closed-ended question-answering domain in English, where a question has an objective groundtruth answer(s). Extensive analysis of MARS and
other probability-based UE methods on open-ended question-answering tasks and other languages are beyond the scope of the current study and are left as future work.

heading: 8 Ethics Statement
text: Although probability-based UE methods combined with MARS have a remarkable prediction performance on the correctness of generative LLM outputs, it is crucial to acknowledge that these methods do not achieve 100% accuracy. Besides, as LLMs may have biases against gender, ethnicity, age, etc., probability-based methods can carry those biases to UE outputs. Thus, one should be aware of these potential risk factors before employing such probabilistic UE methods in real-world systems. Ensuring fairness, transparency, and accountability in the deployment of these technologies is important in mitigating risks and fostering trust in their application.

heading: A Related Works
text: Uncertainty Estimation (UE) has emerged as a vital concept in various machine learning domains, particularly in Natural Language Processing (NLP). The study of Xiao et al. (2022) concentrates on the UE for tasks like common-sense reasoning and sentiment analysis; Jiang et al. (2021) explores model calibration for UE in the context of multiple-choice question answering; Desai and Durrett (2020) tackles the challenge of UE in specific NLP tasks such as paraphrase detection and natural language inference. These studies represent just a fraction of the UE works in the field of NLP and there is an expanding corpus of research focusing on the investigation of UE in NLP (Hu et al., 2023; Xiao and Wang, 2019; Vazhentsev et al., 2022). The vast majority of these studies only focus on classification and regression tasks, unlike our work where the goal is to study UE for generative LLMs.
Few recent works deal with UE of generative LLMs. Xiao et al. (2020) and Fomicheva et al. (2020) propose heuristic-based uncertainty metrics for generative LLMs considering machine translation. Chen and Mueller (2023), Lin et al. (2023),
Cohen et al. (2023), and Kadavath et al. (2022) propose black-box UE methods for generative LLMs under the assumption that the token probabilities are not accessible. Although these works have experimental validation, they lack a mathematical foundation. Malinin and Gales (2021) is the first study adapting popular uncertainty tools in Bayesian UE literature to the generative LLMs. The main idea of Malinin and Gales (2021) is to utilize length-normalized scoring in computing the entropy of the LLM answers. A more recent approach by Kuhn et al. (2023) further improves this result by introducing the concept of semantic entropy, which considers the meaning of the generated sentences in entropy calculation in uncertainty prediction. Our work is distinct from these works as we no longer utilize length-normalized scoring. Instead, we utilize the proposed MARS in entropy computations, by also taking into consideration token importance to the answer correctness , thereby achieving an improved UE performance.
A.1 Discussion of the Differences with TokenSAR
There is a recent work that also considers the meaning of the words in the generation to estimate uncertainty (Duan et al., 2024). The fundamental difference with our work is that (Duan et al., 2024)’s method is designed as an alternative to the existing probability-based uncertainty methods, whereas, in our work, we propose a scoring function, i.e., MARS, which is compatible with all existing probability-based uncertainty estimation methods. This implies that one can in fact utilize MARS within the framework of (Duan et al., 2024). In particular, in (Duan et al., 2024), authors propose three schemes: TokenSAR (token-level weight assignment), SentSAR (sentence-level weight assignment), and SAR (both token and sentence-level weight assignment). SentSAR and SAR are orthogonal to MARS. SAR is the version of SentSAR where the scoring function in SentSAR is replaced with TokenSAR. In a similar fashion, MARS can be incorporated into the SentSAR approach instead of the TokenSAR.
Thus, we need to discuss our distinction from TokenSAR, which can also be considered as a scoring function. To avoid confusion and clarify our unique approach, below we discuss our distinction from TokenSAR.
• MARS uses BERT-Matching instead of sen-
tence similarity: In our algorithm, we utilize the BERT-Matching (BEM) model which takes the question, ground truth answer, and the generated answer as inputs and returns the probability of the generated answer being correct. To assign importance weights, we remove a set of tokens (a phrase) from the generated sentence and pass the question, generated answer (as ground truth) and token-removed generated answer to the BEM model. We set (1 - output) as the importance weight and normalize weights at the end. We further improve this process by fine-tuning a BERT-like model and increase efficiency (explained in the third bullet). TokenSAR uses sentence similarity model (cross-encoder Roberta-Large) unlike our approach. Sentence similarity model takes two input sentences to measure similarity and they concatenate the question for both inputs. However, we argue that using the BEM model achieves better performance since the goal is to find a token’s importance based on its contribution to the correctness of the generated answer. This difference becomes more visible when the answer is longer and more complex as we demonstrate in the below example. In particular, TokenSAR fails to detect words that actually answer the question so that it (almost) returns uniform importance values. On the other hand, MARS successfully finds the important words and assigns higher weights to them. Let’s consider the following example:
Question: What is the tallest building in the world?
Generated Answer: The Burj Khalifa in Dubai, soaring into the sky, holds the distinction of being the tallest building in the world, a marvel of modern engineering and architecture.
To this question-answer pair, MARS returns the following importance weight assignment:
The Burj Khalifa (0.8428) in (0.0082) Dubai (0.0083) , (0.0082) soaring (0.0084) into (0.0082) the sky (0.0082) , (0.0082) holds (0.0083) the distinction (0.0083) of (0.0082) being (0.0082) the tallest building (0.0082) in (0.0082) the world (0.0082) , (0.0082) a marvel (0.0083) of (0.0083) modern engineering and
architecture (0.0088) . (0.0083)
On the other hand, to the same pair, TokenSAR returns the following importance weight assignment:
The (0.0225) Bur (0.0228) j (0.0318) K (0.0228) hal (0.0228) ifa (0.0319) in (0.0227) Dub (0.0253) ai (0.0232) , (0.0228) so (0.0237) aring (0.0294) into (0.0228) the (0.0228) sky (0.0235) , (0.0229) holds (0.0228) the (0.0227) distinction (0.0234) of (0.0228) being (0.0228) the (0.0229) tall (0.0235) est (0.0227) building (0.0228) in (0.0228) the (0.0228) world (0.0230) , (0.0229) a (0.0228) mar (0.0232) vel (0.0232) of (0.0230) modern (0.0540) engineering (0.0725) and (0.0336) architecture (0.0328) . (0.0232)
In this example, although the phrase “The Burj Khalifa” is the key word answering the question, TokenSAR assigns low weights to its tokens. In fact, according to TokenSAR, tokens of the phrase “The Burj Khalifa” are as important as some of the words/phrases that appear in the question itself such as “the tallest building”. This is not ideal as TokenSAR cannot distinguish between the actual answer and filler words. However, our proposed MARS is able to actually find the important words in the answer thanks to the BEM model we employ during weight assignment.
• MARS addresses token dependencies and process phrases instead of tokens: As we explain in Section 3, we first divide a generated answer into phrases and then assign scores to each of those phrases by using the procedure described in the first bullet point. On the other hand, (Duan et al., 2024) assumes that each generated token is meaningly independent so that they remove tokens from the generation one-by-one and assign importance scores accordingly. However, as we show in Table 2, ignoring token dependencies negatively affects the performance of uncertainty methods. In this sense, our MARS provides a more careful importance score assignment (as we demonstrate in the above example).
• MARS is computationally efficient at inference: As we mention in Section 3, we improve
the computational performance of MARS by fine-tuning a BERT-like model that gives importance scores with phrases in a single forward pass (this is an improvement over the algorithm described in the first bullet). That is, our importance assignment does not depend on the number of tokens in the generated sentence. In stark contrast, (Duan et al., 2024) uses cross-encoder Roberta-Large and their algorithm requires a number of tokens times forward pass for a single generated sentence. Moreover, cross-encoder Roberta-Large has approximately 355M parameters. However, we run our 110M BERT-like model only one time for the generated sentence no matter its length. That is why for a generation comprised of 10 tokens, MARS is 30x computationally more efficient than (Duan et al., 2024).

heading: B Training of BERT-like Model for Importance Function
text: As described in Section 3.3, we optimize the computational efficiency of MARS by training a single Bert-like model with 110M parameters to execute the importance function. This model is an adaptation of the pre-trained Bert-base-uncased§, modified by removing its last layer and incorporating two independent fully-connected (FC) layers. The first FC layer focuses on phrase detection with two output logits: “Begin Phrase” (BP) and “Inside Phrase” (IP), and classifies each token as BP if it marks the start of a phrase or as IP otherwise. This setup enables sentence segmentation into phrases. The second FC layer, tasked with assigning importance coefficients, produces a single output logit for each token’s importance coefficient.
For training data, we take a subset of 69192 question samples from the TriviaQA training set and questions of the whole training set of NaturalQA consisting of 87925. Then, we use these questions as input and feed them to all 7B-sized baseline models (Llama2-7b, Llama2-7b-chat, Mistral-7b, Falcon-7b) to yield the responses. This provides us with question-answer pairs. We use the Flair phrase chunking model to determine phrase labels in the answers, as described in Appendix B.1. For importance coefficient labels per token in the responses, we follow Algorithm 1.
§https://huggingface.co/bert-base-uncased
Sample outputs of our model are provided in Table 5. Here, question and answer are inputs to the model, and the model divides the answer into phrases while assigning importance score to them.
We train the model only for 1 epoch with 5e5 learning rate and 32 batch size. The training process involves a convex combination of two loss functions: cross-entropy for phrase chunking and negative log-likelihood for importance coefficient assignment, with equal weight assigned to both losses. Table 6 displays the training and validation losses at the end of the training, indicating that our training objectives are effectively generalizable to test sets.
B.1 Dividing a Sentence to Phrases To divide a sentence into phrases, we use the Flair phrase chunking model¶ (Akbik et al., 2018), that uses 10 tags which are adjectival, adverbial, conjunction, interjection, list marker, noun phrase, prepositional, particle, subordinate clause and verb phrase. For example, the Flair model divides the sentence “The happy man has been eating at the dinner” as “The happy man”, “has been eating”, “at”, “the diner”.
B.2 Pseudocode of the Importance Function Algorithm
The pseudocode of the importance function algorithm is given in Algorithm 1.
Algorithm 1 Phrase-Level Importance Function Input: Question x, generated answer s = {s1, s2, . . . , sL}, phrases {h1, h2, . . . , hK}, token probabilities {pi = P (si|s<i,x; θ)}si∈s, temperature τ Output: Importance scores I I ← []
1: for k = 1 to K do 2: smasked ← s \ {sl}sl∈hk 3: ok ← BEM(x, s, smasked) 4: for each token sl in phrase hk do 5: I[l]← (1− ok)/|hk| 6: I ← softmax(I, τ) 7: return I

heading: C Experimental Details
text: Datasets. We employ the validation split of the Natural Questions dataset, comprising 3610 sam-
¶https://huggingface.co/flair/chunk-english
ples. Following Kuhn et al. (2023), a subset of 8000 QA pairs is selected from the validation split of the TriviaQA dataset. For WebQA, we combine its training and test splits to form a combined dataset of 6642 samples.
Example Samples from Datasets. We provide data samples from the datasets we used in the evaluation of UE methods in Table 7.
Number of Sampling and Temperature. Following previous work (Kuhn et al., 2023), we sampled 5 samples and used 0.5 as the temperature value for the results presented in Table 1.
Generation Configurations. We use the Huggingface library’s generate function for model generations. We set token “.” as eos_token_id which prevents model to generate long paragraphs to closedbook questions. We set num_beams = 1 which corresponds to greedy decoding.
Computational Cost. We use 40 GB Nvidia A100 GPUs for all the experiments. The total GPUhours for Table 1 is approximately 400. Labeling of the data used for training of BERT-like importance model takes approximately 200 GPU-hours. Fine-tuning of BERT-like model on the importance
dataset takes 7 GPU-hours. Due to expensive computational demands, all presented results are the output of a single run.
Prompts. We use the same 2-shot prompt for all of the models and the datasets for answer generation:
Answer these questions: Question: What is the capital city of Australia? Answer: The capital city of Australia is Canberra. Question: Who painted the famous artwork "Starry Night"? Answer: "Starry Night" was painted by Vincent van Gogh. Question: {sample['question']}? Answer:
To evaluate the correctness of the generated answer, we use gpt-3.5-turbo as the evaluator. The prompt for gpt-3.5-turbo is the following:
You will behave as a question-answer evaluator. I will give you a question, the ground truth of the question and a generated answer by a language model. You will output "correct" if the generated answer is correct regarding question and ground truth. Otherwise, output "false". Question: {question}?, Ground Truth: {answer}, Generated Answer: {generation}

heading: Future Work
text: Extensive analysis of MARS and
other probability-based UE methods on open-ended question-answering tasks and other languages are beyond the scope of the current study and are left as future work