Venue: NAACL 2024

# Uncertainty Quantification for In-Context Learning of Large Language Models

Chen Ling\({}^{1}\), Xujiang Zhao\({}^{2}\), Xuchao Zhang\({}^{3}\), Wei Cheng\({}^{2}\), Yanchi Liu\({}^{2}\),

**Yiyou Sun\({}^{2}\), Mika Oishi\({}^{4}\), Takao Osaki\({}^{4}\), Katsushi Matsuda\({}^{4}\), Jie Ji\({}^{1}\), Guangji Bai\({}^{1}\), Liang Zhao\({}^{1}\), Haifeng Chen\({}^{3}\) \({}^{1}\)Emory University, \({}^{2}\)NEC Labs America, \({}^{3}\)Microsoft, \({}^{4}\)NEC Corporation chen.ling@emory.edu, xuzhao@nec-labs.com, liang.zhao@emory.edu**

###### Abstract

In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) and revolutionized various fields by providing a few task-relevant demonstrations in the prompt. However, trustworthy issues with LLM's response, such as hallucination, have also been actively discussed. Existing works have been devoted to quantifying the uncertainty in LLM's response, but they often overlook the complex nature of LLMs and the uniqueness of in-context learning. In this work, we delve into the predictive uncertainty of LLMs associated with in-context learning, highlighting that such uncertainties may stem from both the provided demonstrations (aleatoric uncertainty) and ambiguities tied to the model's configurations (epistemic uncertainty). We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. Extensive experiments are conducted to demonstrate the effectiveness of the decomposition. The code and data are available at: [https://github.com/lingchen0331/UQ_ICL](https://github.com/lingchen0331/UQ_ICL).

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3357–3370 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Large Language Models (LLMs) have revolutionized diverse domains by serving as general task solvers, which can be largely attributed to the emerging capability: _in-context learning_. By providing demonstrations of the task to LLMs as part of the prompt, LLMs can quickly grasp the intention and make corresponding responses to the particular task Min et al. (2022). In this paradigm, LLMs can quickly adapt to solve new tasks at inference time (without any changes to their weights). Advanced LLMs, e.g., GPT-4 and LLaMA, have achieved state-of-the-art results on LAMBADA (commonsense sentence completion), TriviaQA (question answering) Xie et al. (2021), and many tasks in other domains Ling et al. (2023b, a).

While in-context learning has achieved notable success, LLMs remain vulnerable to well-known reliability issues like hallucination Rawte et al. (2023); Bai et al. (2024). Uncertainty quantification has emerged as a popular strategy to assess the reliability of LLM responses. In the past two years, numerous works Xiao et al. (2022); Lin et al. (2023); Ling et al. (2023c); Amayuelas et al. (2023); Kuhn et al. (2023) have been proposed to quantify the uncertainty of LLM response. These approaches could return a confidence score or directly compute variance/entropy across multiple LLM responses; however, they often overlook the complex nature of LLMs and their reliance on provided demonstrations in in-context learning, so that existing methods may not provide insights into the underlying causes or the interactions among different factors contributing to uncertainty.

A natural question therefore arises: when LLM uses in-context learning to predict a wrong answer with high uncertainty, can we indicate if it is caused by the demonstration examples or by the model itself? Given LLM's responses to a particular task, it's essential to decompose the uncertainty into its primary sources to address the question. Specifically, _Aleatoric Uncertainty (AU)_ refers to variations in the data, often linked to the demonstration examples. As shown in Figure 1 (a), LLM's output can easily be disturbed by inappropriate demonstrations since the provided demonstrations do not cover all possible labels. The noise and potential ambiguity of these demonstrations could bring uncertainty, which, in turn, may hinder the accuracy of the response. In contrast, _Epistemic Uncertainty (EU)_ stems from ambiguities related to the model parameters or different configurations. As depicted in Figure 1 (b), different decoding strategies (e.g., beam search and greedy decoding) and their hyperparameter settings can have different decoding re

sults. Recognizing and quantifying the uncertainty from the model's perspective can also be critical in evaluating the generated responses, which allows us to understand the model's confidence level toward the task and make necessary adjustments (e.g., choosing a more powerful model or conducting an ensemble prediction).

Despite the strides made by existing works in understanding the total uncertainty, the decomposition of uncertainty in the realm of in-context learning remains under-explored. In this work, we propose a novel framework for quantifying the uncertainty of in-context learning to aleatoric and epistemic components from the generated outputs. Our contributions are summarized as follows.

* **Problem.** We formulate the problem of uncertainty decomposition that extracts epistemic and aleatoric uncertainties from the predictive distribution of LLMs with in-context learning.
* **Method.** We propose quantifying both aleatoric and epistemic uncertainty from the mutual information perspective. A novel entropy-based estimation method is also designed to handle the free-form outputs of LLMs.
* **Experiment.** Extensive experiments are conducted to evaluate different aspects of uncertainty, followed by specific applications and case studies to show how two types of uncertainty influence the model's performance.

## 2 Uncertainty Decomposition of In-Context Learning

We first formulate the process of in-context learning as Bayesian Neural Networks with latent variables. Based on the formulation, we propose to decompose the predictive uncertainty into its epistemic and aleatoric components from the mutual information perspective, followed by a novel way to estimate both uncertainties based on the entropy of the prediction's distribution.

### Background

LLMs are typically trained using maximum likelihood estimation on a large corpus of text. The training goal is to maximize the likelihood of the observed data under the model: \({\cal L}(\Theta)=\prod_{i\leq N}p(\omega_{i}|\omega_{1},\omega_{2},\ldots, \omega_{i-1};\Theta)\), where each \(\omega_{i}\in\mathbf{x}\) is a token in a sentence \(\mathbf{x}=[\omega_{1},\dots,\omega_{N}]\), and \(\Theta\) denotes the set of parameters.

Latent Concept.From the Bayesian point of view, LLM's in-context learning ability is obtained by mapping the training token sequence \(\mathbf{x}\) to a latent _concept_\(z\)(Xie et al., 2021). The concept \(z\) is a latent variable sampled from a space of concepts \(\mathcal{Z}\), which defines a distribution over observed tokens \(\omega_{i}\) from a training context \(\mathbf{x}\):

\[p(\omega_{1},\dots,\omega_{N})=\int_{z\in\mathcal{Z}}p(\omega_{1},\dots, \omega_{N}|z)p(z)dz.\]

The concept can be interpreted as various document-level statistics, such as the general subject matter of the text, the structure/complexity of the text, the overall emotional tone of the text, etc.

In-context Learning.Given a list of independent and identically distributed (i.i.d.) in-context demonstrations (contain both question and answer) \([\mathbf{x}_{1},\dots,\mathbf{x}_{T-1}]\) concatenated with a test question (without the task answer) \(\mathbf{x}_{T}\) as prompt. Each demonstration \(\mathbf{x}_{i}\) in the prompt is drawn as a sequence conditioned on the same concept \(z\) and describes the task to be learned. LLMs generate a

Figure 1: Uncertainty in LLM’s prediction can stem from two aspects: a) _Demonstration Quality_: LLMs are likely to make wrong predictions if the demonstrations are inappropriate; b) _Model Configuration_: different decoding strategies (e.g., beam_search and top_k sampling) and their parameter settings may return different predictions.

 

response \({\bf y}_{T}\) to the test question \({\bf x}_{T}\) based on the aggregated prompt \({\bf x}_{1:T}\):

\[p({\bf y}_{T}|{\bf x}_{1:T})=\int_{z\in\mathcal{Z}}p({\bf y}_{T}|{\bf x}_{1:T}, z)p(z|{\bf x}_{1:T})dz.\]

In-context learning can be interpreted as _locating_ a pre-existing concept \(z\) based on the provided demonstrations \({\bf x}_{1:T-1}\), which is then employed to tackle a new task \({\bf x}_{T}\). Including more high-quality demonstrations within the prompt can help refine the focus on the relevant concept, enabling its selection through the marginalization term \(p(z|{\bf x}_{1:T})\). Note that formulating in-context learning as Bayesian inference with latent variables is more of a hypothesis; however, demystifying the in-context learning from the view of Bayesian inference offers a probabilistic interpretation of how LLM learns and adapts to new data in context.

In this work, we focus on quantifying the predictive uncertainty of LLMs in deterministic NLP tasks, such as text classification. Specifically, we address tasks where the training dataset \(\mathcal{D}=\{\mathcal{X},\mathcal{Y}\}\) consists of token sequences \(\mathcal{X}=\{{\bf x}\}\) and their corresponding target outputs \(\mathcal{Y}=\{{\bf y}\}\). For LLMs, the generation process is defined by the function \({\bf y}=f({\bf x},z;\Theta)\), where \(f:\mathcal{X}\times\mathcal{Z}\rightarrow\mathcal{Y}\) is a deterministic function. The output \({\bf y}\) exhibits stochastic behavior, influenced by the latent concept \(z\sim p(z|{\bf x}_{1:T})\) and the model parameters/configurations \(\Theta\) (e.g., temperature, etc.).

### Predictive Uncertainty Formulation of In-context Learning

We formulate the predictive distribution of in-context learning for predicting \({\bf y}_{T}\) given few-shot demonstrations \({\bf x}_{1:T-1}\) and a test case \({\bf x}_{T}\) as:

\[p({\bf y}_{T}|{\bf x}_{1:T}) \approx\int p({\bf y}_{T}|\Theta,{\bf x}_{1:T},z)\] (1) \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} { 

demonstration \({\bf x}_{1:T-1}\)) and epistemic (i.e., model parameter \(\Theta\)) uncertainties. To estimate the EU, we condition Eq. (1) on a specific realization of the model parameter \(\Theta\), yielding \(p({\bf y}_{T}|{\bf x}_{1:T},\Theta)=\int p({\bf y}_{T}|{\bf x}_{1:T},z,\Theta)p( z|{\bf x}_{1:T})dz\) with an associated entropy \(H({\bf y}_{T}|{\bf x}_{1:T},z,\Theta)\). The expected value of this entropy under different demonstration sets can be expressed as \(\mathbb{E}_{z}\left[H({\bf y}_{T}|{\bf x}_{1:T},z,\Theta)\right]\), which serves as a metric to quantify the EU in Eq. (1).

Aleatoric Uncertainty (AU).In terms of AU, the randomness comes from different sets of demonstration \({\bf x}_{1:T-1}\) and their corresponding latent concept \(z\). To estimate AU, we can quantify the mutual information between \({\bf y}_{T}\) and latent concept \(z\), which can often be leveraged as an evaluation metric of AU (Wimmer et al., 2023). As we have already obtained the EU, AU can subsequently be calculated as the discrepancy between the total uncertainty and the epistemic uncertainty:

\[I({\bf y}_{T},z|\Theta)= H\left({\bf y}_{T}|{\bf x}_{1:T},\Theta\right) \tag{2}\] \[-\mathbb{E}_{z}\left[H({\bf y}_{T}|{\bf x}_{1:T},z,\Theta)\right].\]

The entropy \(H\left({\bf y}_{T}|{\bf x}_{1:T},\Theta\right)\) can be approximately calculated as \(-\sum_{t}\left[p(\omega^{{\bf y}_{T}}_{t})\cdot\log p\left(\omega^{{\bf y}_{ T}}_{t}\right)\right]\), where \(p(\omega^{{\bf y}_{T}}_{t})\) represents the probability of each possible next token \(\omega^{{\bf y}_{T}}_{t}\) given the input prompt \({\bf x}_{1:T}\). Therefore, the AU in Eq. (2) can be approximated by sampling many \(z\) (by sampling different sets of demonstrations) to obtain different \({\bf y}_{T}\) conditioning on one set of parameters \(\Theta\). The group of \({\bf y}_{T}\) can then be used to approximate the respective entropies for each group of demonstrations \({\bf x}_{1:T-1}\):

\[I({\bf y}_{T},z|\Theta) \tag{3}\] \[=H\left({\bf y}_{T}|{\bf x}_{1:T},\Theta\right)-\mathbb{E}_{z} \left[H({\bf y}_{T}|{\bf x}_{1:T},z,\Theta)\right]\] \[\approx\sum^{M\times L}H({\bf y}_{T})-\frac{1}{M}\sum_{m=1}^{M} \sum_{l=1}^{L}\left[H({\bf y}^{\Theta_{m},l}_{T})\right],\]

where \([{\bf y}^{\Theta_{m},l}_{T}]\) are obtained corresponding to different demonstrations \([{\bf x}^{1}_{1:T-1},\ldots,{\bf x}^{L}_{1:T-1}]\), and \([\Theta_{1},\ldots,\Theta_{M}]\) are sampled from \(q(\Theta)\). However, in many cases, direct sampling from the posterior is hard since it requires a prohibitive number of samples to approximate it effectively. Beam search is then used as an efficient alternative to find high-quality hypotheses. This approach can be viewed as a form of importance sampling, where hypotheses are drawn from high-probability regions of the space. Each hypothesis \({\bf y}_{T}\) observed during the beam search process is associated with uncertainty, which is importance-weighted in proportion to \(p({\bf y}_{T}|{\bf x}_{1:T},z)\). Beam Search thus serves as a practical and efficient way to sample from the posterior by focusing on the most relevant parts of the hypothesis space. In addition, since calculating the entropy \(H\left({\bf y}_{T}\right)\) entails to obtain the joint probability of the generated tokens \(p({\bf y}_{T})=(\omega^{{\bf y}_{T}}_{1},\ldots,\omega^{{\bf y}_{T}}_{T})\), entropy-based method may only be applicable to white-box LLMs.

Figure 3: Framework of entropy-based uncertainty estimation, which consists of 1) generating \(M\) sequences based on a set of \({\bf x}_{1:T-1}\); 2) selecting token(s) that is relevant to the answer and extract the probabilities; 3) aggregating the token probabilities of \(M\) sequences into a distribution of predicted labels; 4) iterating the process \(L\) times corresponding to \(L\) different demonstration sets and form a probability matrix \(\mathcal{M}\), where the column denotes different demonstration sets and the row denotes labels of the dataset.

 



### Entropy Approximation

The generation of LLMs is generally free-form, which makes the uncertainty estimation for in-context learning still different from well-studied classification models that have specific labels. Specifically, not only may the LLM not always be able to return an expected answer, but the generated sequence may also consist of placeholder tokens. Calculating the entropy of the whole generated sequence would involve redundant tokens. Therefore, in this work, we propose to approximate the entropy of the output \(H({\bf y}_{T})\), and the process is summarized in Figure 3.

Given the probability distributions of the generated tokens \(p({\bf y}_{T})\) for one set of demonstrations, we only select token(s) \(\omega^{{\bf y}_{T}}_{t}\) that directly answer the provided question. Taking the text classification task as an example, LLM is asked to directly output a numerical value standing for a predefined category (e.g., \(0\): Sadness, \(1\): Joy, etc.). The probability of the token \(\omega^{{\bf y}_{T}}_{t}\) that represents the numerical value is then leveraged to denote the overall distribution of \(p({\bf y}_{T})\). We aggregate the answer probabilities from all \(M\) decoded sequences and transform them as an answer distribution (as shown in the top right corner in Figure 3). After repeating the process \(L\) times, where \(L\) corresponds to \(L\) different sets of demonstrations, we have a matrix \(\mathcal{M}\) recording the answer distributions of choosing different demonstrations and model configurations (as shown in the lower right corner in Figure 3). The EU and AU can then be approximated as:

\[EU=\frac{1}{L}\sum H\left(\sigma(\mathcal{M}_{:j})\right),\] \[AU=H\left(\sigma\left(\sum[\mathcal{M}_{:j}]\right)\right)- \frac{1}{L}\sum H\left(\sigma\left(\mathcal{M}_{:j}\right)\right),\]

where \(\sigma(\cdot)\) normalizes the column \(\mathcal{M}_{:,j}\) into a probability distribution, and entropy \(H(\cdot)\) can be calculated as \(-\sum_{k=1}^{K}\left(p(\mathcal{M}_{k,j})*\log(p(\mathcal{M}_{k,j}))\right)\) if the number of labels is \(K\). Note that we have instructed LLMs to not generate tokens with less semantic meaning, such as dashes, spaces, or non-related words. In practice, our adopted LLMs can follow the instruction to only return desired answers so that the whole sentence will be the answer tokens (no need to select tokens).

## 3 Related Works

**Uncertainty Quantification and Decomposition.** Uncertainty quantification aims to measure the confidence of models' predictions, which has drawn attention from various domains Zhao et al. (2020); Ling et al. (2022); Malo et al. (2014). Measuring uncertainty is essential in many real-world NLP applications where making a wrong prediction with high confidence can be disastrous (e.g., assessing the confidence in a translation or a generated piece of information). This is especially important in foundation models since we do not have enough resources to finetune the model Abdar et al. (2021). To better understand the uncertainty, the primary focus is on understanding and categorizing the sources of uncertainty for interpreting the models' outputs more effectively. The output uncertainty can typically be categorized into _Aleatoric Uncertainty_ that arises from the inherent noise in the data, and _Epistemic Uncertainty_ that arises due to inappropriate model architecture or overfitted/underfitted parameters. Existing methods Chowdhary and Dupuis (2013); Depeweg et al. (2017); Malinin and Gales (2020) have come up with various methods (e.g., Bayesian neural network, Deep Ensembles, and Monte Carlo Dropout) to decompose the uncertainty.

**Uncertainty in Language Models.** LLMs have revolutionized the learning and inference paradigm in many domains Chen et al. (2024, 2021), but existing works using LLMs often neglect the importance of uncertainty in their responses. Earlier works Xiao and Wang (2019); Desai and Durrett (2020); Jiang et al. (2021) on uncertainty in language models have focused on the calibration of classifiers (e.g., applying dropout to the model parameters or leveraging ensemble voting) to better assess the confidence of the generated output. When it comes to the era of LLMs, multiple works Xiao and Wang (2021); Xiao et al. (2022); Lin et al. (2022); Yu et al. (2022); Lin et al. (2023); Kuhn et al. (2023); Fadeeva et al. (2023) have been proposed to measure the uncertainty of LLM's prediction in multiple aspects (e.g., lexical uncertainty, text uncertainty, and semantic uncertainty) for multiple NLP tasks. Another line of works Kadavath et al. (2022); Zhou et al. (2023); Amayuelas et al. (2023); Chen et al. (2024) instead tries to analyze how to extract knowledge from a language model correctly and self-evaluate the correctness with a confidence score. However, despite these commendable efforts, existing methods still lack an effective way to directly quantify and decompose the uncertainty inherent in the outputs of LLMs with in-context learning

Experiments

We evaluate the uncertainty decomposition procedure on realistic natural language understanding problems. By comparing state-of-the-art uncertainty quantification methods, we aim to examine what type of uncertainty is the most promising indicator of high confidence for LLMs. In addition, we also provide generalization analysis and two specific out-of-distribution detection applications. Due to the space limit, extra experiments and experiment settings are provided in the Appendix.

### Experiment Setup

We evaluate the decomposed uncertainties on open-source LLMs with different model sizes. We leverage LLAMA-2 (Touvron et al., 2023), which is the most widely applied open LLM, with its \(7\text{B}\), \(13\text{B}\), and \(70\text{B}\) model sizes. The primary experiments are conducted with LLAMA-2 models. In order to further demonstrate the generalization ability of our method, we apply our uncertainty quantification method on OPT-13B (Zhang et al., 2022).

**Data.** We consider different Natural Language Understanding tasks. _1) Sentiment Analysis_: Emotion (Saravia et al., 2018) contains \(2,000\) test cases and six classes; Financial Phrasebank (Financial) (Malo et al., 2014) contains \(850\) financial news and three sentiment classes; Stanford Sentiment Treebank v2 (SST2) (Socher et al., 2013) consists of \(872\) sentences from movie reviews and two classes. _2) Linguistic Acceptability:_ The Corpus of Linguistic Acceptability (COLA) (Warstadt et al., 2019) is about English acceptability judgments, which has \(1,040\) test cases and two classes. _3) Topic Classification:_ AG_News (Zhang et al., 2015) contains \(1,160\) test cases and four classes.

**Demonstration & Model Configuration Sampling.** We evaluate each method on the testing set of each dataset and choose two strategies to randomly sample in-context learning demonstrations. 1) _Random_: we randomly sample demonstrations (training instances with labels) from the training set regardless their labels. 2) _Class_: we randomly sample demonstrations but ensure there is at least one demonstration per label class. To generate various sequences based on one set of demonstrations, we adopt Beam Search with beam width \(=10\) to approximate the sampling process of \(\Theta\sim q(\Theta)\).

**Comparison Methods.** Our study also evaluates the following baseline uncertainty estimation methods: 1) _Likelihood-based Uncertainty_ (Likelihood) (Malinin and Gales, 2020) calculates the sum of log probabilities of all tokens generated from language models and normalizes it by the sequence length. 2) _Entropy-based Uncertainty_ (Entropy) (Xiao and Wang, 2019) calculates the entropy of the probability distributions of the generated tokens. 3) _Semantic Uncertainty_ (Semantic) (Kuhn et al., 2023) is the most advanced entropy-based uncertainty estimation method, which groups generated sequences into clusters according to their semantic embeddings. The average entropy across all groups is viewed as the uncertainty score.

**Evaluation Metrics.** We show the prediction accuracy of each dataset. In addition, we leverage two standard metrics: the Area under Precision-Recall Curve (AUPR) and AUROC (ROC) to evaluate the uncertainty. AUPR calculates the area under the Precision-Recall curve. AP is high when both precision and recall are high, and low when either of them is low across a range of confidence thresholds. ROC represents the likelihood that a correct answer is selected. An ideal ROC rating is \(1\), whereas a random uncertainty estimate would yield \(\text{ROC}=0.5\).

### Quantitative Analysis

We compare the performance of different methods in assessing the misclassification samples based on their perspective uncertainty scores. We follow the procedure: 1) We use LLMs to classify all examples in the dataset with different beam search branches and demonstrations; 2) we use different uncertainty quantification methods to obtain a score associated with each test instance; 3) we assign each example a \(0\) if it was classified correctly or a \(1\) if it was misclassified; and 4) we calculate AUPR and AUROC based on the misclassification rate and uncertainty score. Ideally, misclassified samples should have higher uncertainty scores. The results are shown in Table 1. Note that our proposed method can decompose the uncertainty into epistemic uncertainty (EU) and aleatoric uncertainty (AU), we thus show the performance of EU and AU separately.

As shown in the table, in most cases, our proposed methods (EU and AU) consistently show higher AUPR and ROC scores across all datasets, which indicates a better performance in assessing misclassification samples based on uncertainty scores. Moreover, we also draw some observations from the table. _1. Class Sampling Strategy Proves Superior_: The class sampling strategy 

generally yields higher AUPR and ROC scores across datasets, which proves it is more effective than random demonstration sampling. Class sampling ensures that each class is represented in the sample and reduces sampling bias, which is crucial in scenarios where the dataset might be imbalanced or where certain classes are underrepresented. _2) Increasing Model Size Enhances Performance_: Larger models (moving from 7B to 70B) tend to have better performance in terms of AUPR and ROC. Specifically, there's a general trend of increasing AUPR and ROC scores as model size increases from 7B to 13B to 70B for all comparison methods. Some datasets and metrics do not strictly follow this trend. For instance, in the Emotion dataset, the 70B model sometimes shows a slight decrease in performance compared to the 13B model. The inconsistencies in performance improvement with larger models, especially for EU, hint at the complexity of uncertainty assessment in different contexts and datasets. _3. Treating all tokens equally can be harmful in uncertainty quantification_: both Likelihood and Entropy Uncertainty treat all tokens equally. However, some tokens carry greater relevance and representativeness than others, owing to the phenomenon of "linguistic redundancy". However, most uncertainty estimation methods treat all tokens with equal importance when estimating uncertainty, disregarding these inherent generative inequalities.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & Inference & \multirow{2}{*}{ACC} & \multicolumn{2}{c}{**Likelihood**} & \multicolumn{2}{c}{**Entropy**} & \multicolumn{2}{c}{**Semantic**} & \multicolumn{2}{c}{**Ours (EU)**} & \multicolumn{2}{c}{**Ours (AU)**} \\  & & & AUPR & ROC & AUPR & ROC & AUPR & ROC & AUPR & ROC & AUPR & ROC \\ \cline{2-16}  & LLAMA-7b-random & 0.407 & 0.423 & 0.426 & 0.448 & 0.501 & 0.598 & 0.607 & **0.688** & **0.667** & 0.625 & 0.579 \\  & LLAMA-7b-class & 0.411 & 0.562 & 0.423 & 0.657 & 0.538 & 0.697 & 0.653 & **0.745** & **0.696** & 0.691 & 0.601 \\  & LLAMA-13b-random & 0.501 & 0.597 & 0.613 & 0.584 & 0.503 & 0.612 & 0.625 & **0.645** & **0.681** & 0.559 & 0.585 \\  & LLAMA-13b-class & 0.533 & 0.641 & 0.578 & 0.593 & 0.554 & 0.652 & 0.701 & **0.622** & **0.686** & 0.526 & 0.599 \\  & LLAMA-70b-random & 0.584 & 0.512 & 0.462 & 0.491 & 0.452 & 0.657 & 0.696 & **0.667** & **0.713** & 0.531 & 0.663 \\  & LLAMA-70b-class & 0.592 & 0.537 & 0.484 & 0.469 & 0.442 & 0.622 & 0.689 & **0.659** & **0.721** & 0.612 & 0.693 \\ \cline{2-16}  & LLAMA-7b-random & 0.379 & 0.821 & 0.532 & 0.728 & 0.438 & 0.715 & 0.624 & **0.731** & **0.672** & 0.669 & 0.582 \\  & LLAMA-7b-class & 0.397 & 0.593 & 0.505 & 0.548 & 0.362 & 0.732 & 0.699 & **0.803** & **0.711** & 0.753 & 0.589 \\  & LLAMA-13b-random & 0.476 & 0.894 & 0.571 & 0.652 & 0.463 & 0.705 & 0.545 & 0.718 & 0.512 & **0.729** & **0.573** \\  & LLAMA-13b-class & 0.477 & 0.752 & 0.594 & 0.692 & 0.531 & 0.694 & 0.543 & **0.765** & **0.610** & 0.758 & 0.592 \\  & LLAMA-70b-random & 0.530 & 0.816 & 0.509 & 0.754 & 0.493 & 0.679 & 0.688 & **0.779** & **0.754** & 0.734 & 0.642 \\  & LLAMA-70b-class & 0.537 & 0.668 & 0.469 & 0.623 & 0.439 & 0.774 & 0.649 & **0.893** & **0.804** & 0.739 & 0.659 \\ \hline  & LLAMA-7b-random & 0.856 & 0.149 & 0.636 & 0.135 & 0.587 & 0.244 & 0.593 & **0.286** & 0.683 & 0.205 & **0.702** \\  & LLAMA-7b-class & 0.897 & 0.230 & 0.666 & 0.196 & 0.579 & 0.253 & 0.577 & 0.248 & **0.701** & **0.302** & 0.673 \\  & LLAMA-13b-random & 0.866 & 0.268 & 0.472 & 0.204 & 0.467 & **0.355** & 0.712 & 0.314 & 0.677 & 0.326 & **0.816** \\  & LLAMA-13b-class & 0.928 & 0.178 & 0.425 & 0.113 & 0.439 & 0.343 & 0.631 & **0.397** & **0.836** & 0.367 & 0.639 \\  & LLAMA-70b-random & 0.932 & 0.091 & 0.597 & 0.137 & 0.475 & 0.2esch & 0.565 & **0.318** & **0.764** & 0.298 & 0.571 \\  & LLAMA-70b-class & 0.938 & 0.132 & 0.552 & 0.185 & 0.531 & 0.312 & 0.679 & 0.331 & **0.851** & **0.362** & 0.697 \\ \hline  & LLAMA-7b-random & 0.599 & 0.388 & 0.557 & 0.329 & 0.443 & 0.358 & 0.502 & **0.416** & **0.562** & 0.377 & 0.517 \\  & LLAMA-7b-class & 0.639 & 0.392 & 0.523 & 0.381 & 0.478 & 0.425 & 0.526 & **0.473** & **0.587** & 0.401 & 0.506 \\  & LLAMA-13b-random & 0.652 & 0.389 & 0.498 & 0.287 & 0.512 & 0.433 & 0.562 & 0.469 & **0.572** & **0.488** & 0.565 \\  & LLAMA-13b-class & 0.649 & 0.412 & 0.418 & 0.342 & 0.517 & 0.426 & 0.548 & 0.456 & 0.568 & **0.523** & **0.641** \\  & LLAMA-70b-random & 0.826 & 0.481 & 0.599 & 0.312 & 0.471 & 0.372 & 0.625 & 0.317 & **0.716** & **0.329** & 0.676 \\  & LLAMA-70b-class & 0.852 & 0.357 & 0.612 & 0.397 & 0.588 & 0.397 & 0.613 & 0.389 & **0.727** & **0.425** & 0.682 \\ \hline  & LLAMA-7b-random & 0.646 & 0.238 & 0.472 & 0.265 & 0.463 & 0.312 & 0.612 & **0.448** & **0.634** & 0.361 & 0.537 \\  & LLAMA-7b-class & 0.679 & 0.267 & 0.505 & 0.372 & 0.523 & 0.378 & 0.562 & **0.384** & **0.627** & 0.326 & 0.538 \\  & LLAMA-13b-random & 0.685 & 0.365 & 0.517 & 0.364 & 0.522 & 0.374 & 0.548 & **0.395** & **0.648** & 0.378 & 0.552 \\  & LLAMA-13b-class & 0.685 & 0.378 & 0.528 & 0.359 & 0.413 & 0.411 & 0.566 & **0.429** & **0.654** & 0.401 & 0.569 \\  & LLAMA-70b-random & 0.792 & 0.311 & 0.478 & 0.316 & 0.498 & **0.401** & 0.552 & 0.309 & **0.635** & 0 



### Generalization Capability

In this work, we also show how our method performs when applied to different LLMs. We compare the performance of misclassification rate when using OPT-13B and LLAMA-2-13B. We compute the precision-recall (PR) curve and ROC curve using two backbone LLMs on the Emotion dataset, and the results are shown in Figure 4.

As shown in Figure 4, our method exhibits consistent trends across different LLMs. The precision-recall curves of both uncertainties (Figure 4 (a) and 4 (b)) between the two methods are almost identical, and the model's capability between two LLMs is also reflected in the PR curves of EU. Furthermore, by comparing Figure 4 (c) and 4 (d), the ROC curves of both LLMs show a similar pattern, with the AUC scores not deviating significantly. Specifically, both OPT-13B and LLAMA-2-13B exhibit the same Area Under ROC (AUROC) \({\rm curve}=0.68\) for AU. Since LLAMA-2-13B is a more powerful LLM than OPT-13B, our method can quantify that EU of LLaMA-2-13B (AUROC \(=0.59\)) is better than OPT-13B (AUROC \(=0.55\)). This finding further supports our method maintains its performance irrespective of the underlying model and its robust generalization capability.

### Misclassification Rate with Out of Domain Demonstration

Out-of-domain in-context Demonstration refers to the test instance being coupled with less relevant or out-of-domain demonstrations, which the model may be misled and not handle the test instance reliably. In this work, we analyze the misclassification rate of out-of-domain Demonstration in the Emotion dataset (six-class sentiment analysis task) by providing LLMs with relevant demonstrations (sampled from Finance Phrasebank which is a three-class sentiment analysis task) and complete out-of-domain demonstrations (sampled from COLA which is a binary linguistic acceptability task). We conduct the task with two demonstration selection strategies, and the results are provided in Table 2.

As shown in the table, changes in the performance of the EU are relatively minor under all conditions, suggesting that the model is more stable or less sensitive to the changes in demonstration data within this metric. In contrast, the AU shows more significant fluctuations, which implies that the AU is more sensitive to the quality and relevance of demonstration data. When relevant demonstrations from the Finance Phrasebank sentiment analysis dataset are used, there's a slight improvement or a minor decrease in EU, but a notable decrease in AU. This suggests that even relevant but not identical data can confuse the model, especially for the AU. With out-of-domain demonstrations from COLA, the model's performance drops more significantly, with the AU metric showing a dramatic decrease of up to 17.0%, which indicates that the model struggles significantly when the demonstrations are not relevant to the tas

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{LLaMA-13B-Random} & \multicolumn{2}{c}{LLaMA-13B-Class} \\ \cline{2-5}  & EU & AU & EU & AU \\ \hline Original & \multirow{2}{*}{0.681} & \multirow{2}{*}{0.585} & \multirow{2}{*}{0.686} & \multirow{2}{*}{0.599} \\ Demo & & & & \\ \hline Relevant & \multirow{2}{*}{0.688} & \multirow{2}{*}{0.541} & \multirow{2}{*}{0.671} & \multirow{2}{*}{0.524} \\ Demo & & (\(+1.0\)\%) & (\(-7.5\)\%) & (\(-2.2\)\%) & (\(-12.5\)\%) \\ \hline OOD & \multirow{2}{*}{0.671} & \multirow{2}{*}{0.501} & \multirow{2}{*}{0.673} & \multirow{2}{*}{0.497} \\ Demo & & & & \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of AUROC in misclassified rate on Emotion dataset, where “Original Demo” indicates we sample demonstrations from its original training set, “Relevant Demo” indicates we sample demonstrations from Finance Phrasebank Dataset (a relevant sentiment analysis task, and “OOD Demo” indicates we sample demonstrations from an irrelevant dataset: **COLA**.

Figure 4: The performance of misclassification rate using two backbone LLMs: OPT-13B and LLAMA-2-13B on Emotion dataset. (a) and (b) demonstrate the precision-recall curves (x-axis is the recall and y-axis is the precision) for OPT-13B and LLAMA-2-13B; (c) and (d) demonstrate the ROC curve (x-axis is the false positive rate and y-axis is the true positive rate) for OPT-13B and LLAMA-2-13B, respectively.





### Out-of-domain Demonstration Detection

Out-of-domain (OOD) demonstration refers to coupling a test instance with less relevant or OOD demonstrations, potentially leading the model to be misled and handle the test instance unreliably. In this study, we investigate whether uncertainty scores can effectively distinguish between in-domain and OOD demonstrations. In our labeling scheme, in-domain demonstrations are labeled as \(0\), while OOD demonstrations are labeled as \(1\). AUPR and ROC analyses are performed based on the labels and uncertainty scores, with results summarized in Table 3. Specifically, we conduct experiments on the Emotion dataset, involving two scenarios: in-domain demonstrations (sampled from its training set) and relevant demonstrations (sampled from Finance Phrasebank, a three-class sentiment analysis task). Additionally, we compare in-domain demonstrations with complete OOD demonstrations (sampled from COLA, a binary linguistic acceptability task).

As shown in Table 3, compared to the state-of-the-art Semantic Uncertainty and the AU, the EU demonstrates the best indicator to detect both less relevant and OOD demonstrations. Intuitively, the model's predictions would be impacted by irrelevant and OOD demonstrations and exhibit large variance. AU is less effective than EU in detecting OOD demonstrations since the demonstrations already have large inherent variability. Semantic Uncertainty instead cannot really distinguish what is the root cause of the predictive uncertainty.

### Semantic Out-of-distribution Detection

Semantic out-of-distribution (SOOD) detection refers to distinguishing test samples with semantic shifts from the given demonstrations and the prompt. In this study, we mask out a few classes and ask LLMs to classify test samples into the rest of the classes. The method is expected to return a higher uncertainty score of SOOD test samples. Specifically, we mask two classes _1: sadness_ and _2: anger_ out of six classes from the Emotion dataset



## References

* Abdar et al. (2021) Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochu Cao, Abbas Khosravi, U Rajendra Acharya, et al. 2021. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. _Information fusion_, 76:243-297.
* Amayuelas et al. (2023) Alfonso Amayuelas, Liangming Pan, Wenhu Chen, and William Wang. 2023. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. _arXiv preprint arXiv:2305.13712_.
* Bai et al. (2024) Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. 2024. Beyond efficiency: A systematic survey of resource-efficient large language models. _arXiv preprint arXiv:2401.00625_.
* Chen et al. (2021) Pei Chen, Haibo Ding, Jun Araki, and Ruihong Huang. 2021. Explicitly capturing relations between entity mentions via graph neural networks for domain-specific named entity recognition. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 735-742.
* Chen et al. (2024) Pei Chen, Soumajyoti Sarkar, Leonard Lausen, Balasubramaniam Srinivasan, Sheng Zha, Ruihong Huang, and George Karypis. 2024. Hytrel: Hypergraph-enhanced tabular data representation learning. _Advances in Neural Information Processing Systems_, 36.
* Chen et al. (2022) Pei Chen, Haotian Xu, Cheng Zhang, and Ruihong Huang. 2022. Crossroads, buildings and neighborhoods: A dataset for fine-grained location recognition. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3329-3339.
* Chowdhary and Dupuis (2013) Kamaljit Chowdhary and Paul Dupuis. 2013. Distinguishing and integrating aleatoric and epistemic variation in uncertainty quantification. _ESAIM: Mathematical Modelling and Numerical Analysis-Modelisation Mathematique et Analyse Numerique_, 47(3):635-662.
* Depeweg et al. (2017) Stefan Depeweg, Jose Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. 2017. Uncertainty decomposition in bayesian neural networks with latent variables. _arXiv preprint arXiv:1706.08495_.
* Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. _arXiv preprint arXiv:2003.07892_.
* Fadeeva et al. (2020) Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveeta Goncharova, Alexander Panchenko, Maxim Panov, et al. 2023. Lompolygraph: Uncertainty estimation for language models. _arXiv preprint arXiv:2311.07383_.
* Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. _Transactions of the Association for Computational Linguistics_, 9:962-977.
* Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. _arXiv preprint arXiv:2207.05221_.
* Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. _arXiv preprint arXiv:2302.09664_.
* Lin et al. (2023) Zhen Lin, Shubbhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. _arXiv preprint arXiv:2305.19187_.
* Lin et al. (2022) Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. 2022. Towards collaborative neural-symbolic graph semantic parsing via uncertainty. _Findings of the Association for Computational Linguistics: ACL 2022_.
* Ling et al. (2022) Chen Ling, Junji Jiang, Junxiang Wang, and Zhao Liang. 2022. Source localization of graph diffusion via variational autoencoders for graph inverse problems. In _Proceedings of the 28th ACM SIGKDD_, pages 1010-1020.
* Ling et al. (2023a) Chen Ling, Xuchao Zhang, Xujiang Zhao, Yanchi Liu, Wei Cheng, Mika Oishi, Takao Osaki, Katsushi Matsuda, Haifeng Chen, and Liang Zhao. 2023a. Open-ended commonsense reasoning with unrestricted answer candidates. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 8035-8047.
* Ling et al. (2023b) Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, et al. 2023b. Domain specialization as the key to make large language models disruptive: A comprehensive survey. _arXiv preprint arXiv:2305.18703_.
* Ling et al. (2023c) Chen Ling, Xujiang Zhao, Xuchao Zhang, Yanchi Liu, Wei Cheng, Haoya Wang, Zhengzhang Chen, Takao Osaki, Katsushi Matsuda, Haifeng Chen, et al. 2023c. Improving open information extraction with large language models: A study on demonstration uncertainty. _arXiv preprint arXiv:2309.03433_.
* Malinin and Gales (2020) Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. _arXiv preprint arXiv:2002.07650_

P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala (2014)Good debt or bad debt: detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology65. Cited by: SS3.
* S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer (2022)Rethinking the role of demonstrations: what makes in-context learning work?. arXiv preprint arXiv:2202.12837. Cited by: SS1.
* V. Rawte, A. Sheth, and A. Das (2023)A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922. Cited by: SS1.
* E. Saravia, H. T. Liu, Y. Huang, J. Wu, and Y. Chen (2018)CARER: contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3687-3697. Cited by: SS1.
* R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts (2013)Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642. Cited by: SS1.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, Shruti Bhosale, et al. (2023)LIMA 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: SS1.
* A. Warstadt, A. Singh, and S. R. Bowman (2019)Neural network acceptability judgments. Transactions of the Association for Computational Linguistics7, pp. 625-641. Cited by: SS1.
* L. Wimmer, Y. Sale, P. Hofman, B. Bischl, and E. Hullermeier (2023)Quantifying aleatoric and epistemic uncertainty in machine learning: are conditional entropy and mutual information appropriate measures?. In Uncertainty in Artificial Intelligence, pp. 2282-2292. Cited by: SS1.
* Y. Xiao and W. Yang (2019)Quantifying uncertainties in natural language processing tasks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, pp. 7322-7329. Cited by: SS1.
* Y. Xiao and W. Yang (2021)On hallucination and predictive uncertainty in conditional language generation. arXiv preprint arXiv:2103.15025_. Cited by: SS1.
* Y. Xiao, P. P. Liang, U. Bhatt, W. Neiswanger, R. Salakhutdinov, and L. Morency (2022)Uncertainty quantification with pre-trained language models: a large-scale empirical analysis. arXiv preprint arXiv:2210.04714. Cited by: SS1.
* S. Michael Xie, A. Raghunathan, P. Liang, and T. Ma (2021)An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080. Cited by: SS1.
* J. Yu, A. I. Cristea, A. Harit, Z. Sun, O. Tahir Aduragba, L. Shi, and N. Al Moubayed (2022)Efficient uncertainty quantification for multilabel text classification. In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. Cited by: SS1.
* S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. Tuiona, T. Mihaylov, Q. Ott, S. Shleifer, K. Stuster, D. Simig, P. Soura, A. Sridhar, T. Wang, and L. Zettlemoyer (2022)Opt: open pretrained transformer language models. External Links: 2202.13098 Cited by: SS1.
* X. Zhang, J. J. Zhao, and Y. LeCun (2015)Character-level convolutional networks for text classification. In NIPS, Cited by: SS1.
* Y. Zhang, B. Pan, C. Ling, Y. Hu, and L. Zhao (2024)ElAD: explanation-guided large language models active distillation. arXiv preprint arXiv:2402.13098. Cited by: SS1.
* X. Zhao, F. Chen, S. Hu, and J. Cho (2020)Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems33, pp. 12827-12836. Cited by: SS1.
* K. Zhou, D. Jurafsky, and T. Hashimoto (2023)Navigating the grey area: expressions of overconfidence and uncertainty in language models. arXiv preprint arXiv:2302.13439. Cited by: SS1.

## Appendix A Appendix

### Variance-based Decomposition

Alternatively, we can use the variance as a measure of uncertainty. Let \(\sigma^{2}(\cdot)\) compute the variance of a probability distribution, and the total uncertainty present in Eq. (1) is then \(\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T})\). This quantity can then be decomposed using the law of total variance:

\[\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T})= \sigma^{2}_{q(\Theta)}\left(\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{ 1:T},\Theta]\right) \tag{4}\] \[+\mathbb{E}_{q(\Theta)}\left[\sigma^{2}(\mathbf{y}_{T}|\mathbf{ x}_{1:T},\Theta)\right].\]

where \(\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]\) and \(\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta)\) are mean and variance of \(\mathbf{y}_{T}\) given \(p\left(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta\right)\). \(\sigma^{2}_{q(\Theta)}\left(\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]\right)\) represents the variance of \(\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]\) when \(\Theta\sim q(\Theta)\), which indicates the epistemic uncertainty since it ignores the contribution of \(z\). In contrast, \(\mathbb{E}_{q(\Theta)}\left[\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta) \right]\) in Eq. (4) represents the aleatoric uncertainty since it denotes the average value of \(\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta)\) with 

\(\Theta\sim p(\Theta)\) and ingores the contribution of \(\Theta\) to \({\bf y}_{T}\). Note that variance-based uncertainty decomposition does not involve the probability of the generated tokens, which is applicable to black-box LLMs (e.g., GPT models).

Variance Approximation.In practice, when we are dealing with black-box LLMs (e.g., Chat-GPT), there are multiple hyperparameters (e.g., temperature and top_p) allowing to return different responses. Specifically, \([{\bf y}^{1}_{T},\ldots,{\bf y}^{L}_{T}]\) can be obtained through querying the LLM with different demonstrations \([{\bf x}^{1}_{1:T-1},\ldots,{\bf x}^{L}_{1:T-1}]~{}L\) times. The different set of parameter configurations are denoted as \([\Theta_{1},\ldots,\Theta_{M}]\). The \(\mathbb{E}[{\bf y}_{T}|{\bf x}_{1:T},\Theta]\) can then be calculated as the expected model output given the input data and the model parameters \(\Theta\). Calculate the variance of this expectation with respect to a set of model configurations over all sets of demonstrations gives the epistemic uncertainty. The variance \(\sigma^{2}({\bf y}_{T})\) can also be obtained given a set of few-shot demonstrations over all model parameters. Finally, average this variance over the certain model configuration to obtain the aleatoric uncertainty.

### Dataset Description

**Sentiment Analysis.** 1) Emotion (Saravia et al., 2018) contains \(2,000\) test cases, where LLMs are asked to classify a given sentence with six categories: _sadness_, _joy_, _love_, _anger_, _fear_, _surprise_. 2) Financial Phrasebank (Financiali) (Malo et al., 2014) contains \(850\) test cases, where LLMs are asked to classify a given financial news with three categories: _negative_, _neutral_, _positive_. 3) Stanford Sentiment Treebank v2 (SST2) (Socher et al., 2013) consists of \(872\) sentences from movie reviews and human annotations of their sentiment, where the language model is asked to predict the sentiment from two classes: _positive_ and _negative_.

**Linguistic Acceptability.** 1) The Corpus of Linguistic Acceptability (COLA) (Warstadt et al., 2019) is about English acceptability judgments drawn from books and journal articles on linguistic theory. Each example is a sequence of words annotated with whether it is a grammatical English sentence, and there are \(1,040\) test cases in total.

**Topic Classification.** TC aims at categorizing the given sentence into predefined topics. We adopt AG_News (Zhang et al., 2015) is a dataset that collects more than \(1\) million news articles, where LLMs are asked to classify a given news into four categories: _World_, _Sports_, _Business_, and _SciTech_. There are \(1,160\) test cases in total.

### Experiment Setup

We conduct experiments primarily on llama-2-7b-chat-hf, llama-2-13b-chat-hf, and llama-2-70b-chat-hf, where the model weights are downloaded from the website1. Since we cannot actually "sample" model weights as Bayesian Neural Networks, in order to let LLMs return different outputs, we leverage Beam Search since it considers multiple best options based on beam width using conditional probability, which is better than the sub-optimal Greedy search. The beam search is conducted with the beam size \(10\) and the max number of new tokens is set to be \(16\) uniformly across all datasets. We choose a different number of demonstrations (details are recorded in Table 5) to allow the LLM to achieve the best performance on each dataset, and we sample demonstrations four times uniformly across all datasets.

Footnote 1: [https://ai.meta.com/resources/models-and-libraries/llama-downloads/](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)

### Prompt Template

In this work, we uniformly apply the following prompt template for all datasets. Take the Emotion dataset as an example, we summarize the prompt in Table 6. Note that all datasets use the same template, small modifications are made on the actual label information and different demonstration numbers of different datasets.

### Case Study

Table 7 demonstrates the actual changes in AU and EU when presenting LLMs with different sizes and different demonstrations. Given the test query is: _I had stated to her the reason I feel so fearful is that I feel unsafe_ with the ground truth label is _(4: fear)_, which is a sentence with a negative

\begin{table}
\begin{tabular}{l c c} \hline \hline  & Random & Class \\ \hline Emotion & 6 & 1 per class \\ Financial & 6 & 2 per class \\ SST2 & 4 & 2 per class \\ COLA & 2 & 1 per class \\ AG\_News & 4 & 1 per class \\ \hline \hline \end{tabular}
\end{table}
Table 5: Number of demonstrations selected in each dataset.

 

feeling. For LLaMA-2-7B, by presenting LLMs with more diverse demonstrations (containing both positive and negative sentences), the results would be more diverse between different beam search returned sequences, leading to a relatively higher AU than EU. For LLAMA-2-70B with a larger parameter space and model capability, the EU and AU are significantly reduced, which indicates the model is more confident in the generated output and the variation of data may not influence much to the prediction.

\begin{table}
\begin{tabular}{l l} \hline \hline
**System Prompt** & *** Below is an instruction that describes a task. Clearly follow the instruction and write a short response to answer it. \\ \hline
**Task Description** & *** Instruction: Classify the sentiment in the following text based on the six categories: \\  & [0: Sadness; 1: Joy, 2: Love; 3: Anger; 4: Fear, 5: Surprise]. Provide the information in a structured format WITHOUT additional comments, I just want the numerical label for each text. \\ \hline
**Demonstrations** & *** Here are some examples: \\  & Example 1: Sentence: \textascicentre{!} i didnf feel humiliated} Category: \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentre{!} \textascicentcentre{!} \textascicentre{!} \textascicentcentre{!} \textascicentcentre{!} \textascicentcentre{!} \textascicentcentcentre{!} \textascicent 



\begin{table}
\begin{tabular}{l|p{142.3pt} p{56.9pt} p{78.0pt} p{78.0pt}} \hline
**Testing Query**: I had stated to her the reason I feel so fearful is because I feel unsafe (4: fear) & Extracted Predictions & EU & AU \\ \hline LLaMA-2-7B & 1. i felt anger when at the end of a telephone call (3: anger) & & & \\  & 2. i feel a little mellow today (1: joy) & & & \\  & 3. i don t feel particularly agitated (4: fear) & 0, 0, 0, 1, 3 & 0.171 & 0.372 \\  & 4. i hate it when i feel fearful for absolutely no reason (4: fear) & 4, 3, 4, 4, 4 & & \\  & 5. im updating my blog because i feel shitty (0: sadness) & & & \\ \cline{2-3}  & 1. i am feeling outraged it shows everywhere (4: fear) & & & \\  & 2. i do feel insecure sometimes but who doesnt (4: fear) & & & \\  & 3. i start to feel emotional (0: sadness) & 4, 4, 1, 3, 4 & 0.163 & 0.189 \\  & 4. i feel so cold a href http irish (3: anger) & 4, 4, 4, 5, 4 & & \\  & 5. i feel i have to agree with her even though i can imagine some rather unpleasant possible cases (0: sadness) & & & \\ \hline LLaMA-2-7DB & 1. i felt anger when at the end of a telephone call (3: anger) & & & \\  & 2. i feel a little mellow today (1: joy) & 4, 3, 4, 3, 4 & 0.012 & 0.079 \\  & 3. i don t feel particularly agitated (4: fear) & 4, 4, 2, 4, 4 & & \\  & 4. i hate it when i feel fearful for absolutely no reason (4: fear) & 4, 4, 2, 4, 4 & & \\  & 5. im updating my blog because i feel shitty (0: sadness) & & & \\ \cline{2-3}  & 1. i am feeling outraged it shows everywhere (4: fear) & & & \\  & 2. i do feel insecure sometimes but who doesn (4: fear) & & & \\  & 3. i start to feel emotional (0: sadness) & 4, 4, 4, 4, 4 & 0.004 & 0.009 \\  & 4. i feel so cold a href http irish (3: anger) & 4, 4, 4, 4, 4 & & \\  & 5. i feel i have to agree with her even though i can imagine some rather unpleasant possible cases (0: sadness) & & & \\ \hline \end{tabular}
\end{table}
Table 7: Case study on the actual EU and AU decomposed from the predictive uncertainty