Bayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models (2024)

Abstract: An important requirement for the reliable deployment of pre-trained large language models (LLMs) is the well-calibrated quantification of the uncertainty in their outputs. While the likelihood of predicting the next token is a practical surrogate of the data uncertainty learned during training, model uncertainty is challenging to estimate, i.e., due to lack of knowledge acquired during training. Prior efforts to quantify uncertainty of neural networks require specific architectures or (re-)training strategies, which are impractical to apply to LLMs with several billion parameters, or for black-box models where the architecture and parameters are not available. In this paper, we propose Bayesian Prompts Ensembles (BayesPE), a novel approach to effectively obtain wellcalibrated uncertainty for the output of pretrained LLMs. BayesPE computes output probabilities through a weighted ensemble of different, but semantically equivalent, task instruction prompts. The relative weights of the different prompts in the ensemble are estimated through approximate Bayesian variational inference over a small labeled validation set. We demonstrate that BayesPE approximates a Bayesian input layer for the LLM, providing a lower bound on the expected model error. In our extensive experiments, we show that BayesPE achieves significantly superior uncertainty calibration compared to several baselines over a range of natural language classification tasks, both in zeroand few-shot settings.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 12229–12272 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Generative pre-trained large language models (LLMs) have proven to be very effective at performing a broad variety of natural language processing (NLP) tasks in zero- and few-shot settings (Brown et al., 2020; Touvron et al., 2023; Jiang et al., 2023a). These capabilities make them attractive alternatives to task-specific fine-tuned models, as they do not require large annotated data sets or
gradient updates, but simply take as input relevant text instructions (i.e., prompts) to perform the inference of interest (Petroni et al., 2019; Shin et al., 2020; Sanh et al., 2021; Brown et al., 2020).
However, safely deploying LLMs requires wellcalibrated estimates of the uncertainty in their outputs (Amodei et al., 2016; Hendrycks et al., 2021; Zhou et al., 2024). This is particularly challenging as LLMs are normally not trained to capture their own uncertainty and, due to their size and complexity, it is difficult and expensive to fine-tune or adapt them (Jiang et al., 2021; Kuhn et al., 2023; Osband et al., 2022). In addition, many state-of-the-art LLMs are proprietary and available only via APIs with their weights and architecture hidden from end users, leaving black-box integration as the only option (Katz et al., 2023; Ye et al., 2023a). This precludes the application of common model uncertainty estimation methods for deep learning, i.e., drop-out or deep ensembles (Zhou et al., 2022a; Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017).
To overcome these limitations, we propose Bayesian Prompts Ensembles (BayesPE), a simple yet effective approach for estimating the uncertainty of LLMs in black-box scenarios (i.e. without access to model weights). Similarly to Wightman et al. (2023) and Jiang et al. (2023b), we ensemble the output probabilities obtained by running the LLM classifier with different variations of the prompt task instructions. However, instead of simply combining all prompts with equal contribution, we learn their relative importance through approximate variational inference with a small validation set. Figure 1 schematically illustrates our approach. BayesPE adapts the use of different prompts to the task and LLM of choice, while obtaining well-calibrated estimates of the output uncertainty. We further show that BayesPE approximates a Bayesian input layer to the LLM and provides a lower bound to the reducible error of
12229
the model. In our extensive empirical evaluation, we test BayesPE with several LLMs and a range of text classification tasks, both in zero- and fewshot scenarios. In each case, we measure several uncertainty estimation and calibration metrics and compare to existing competitive baselines.
2 Background & Related Work
Model Uncertainty in Deep Learning Developing deep neural networks (DNNs) that “know when they don’t know” is a long standing challenge in deep learning (Amodei et al., 2016; Zhou et al., 2022a). Unlike other machine learning models, such as Gaussian processes (MacKay et al., 1998; Seeger, 2004; Damianou and Lawrence, 2013) and Markov chain Monte Carlo samplers (Andrieu et al., 2003), DNNs are not capable to self-assess output uncertainty originating from model misspecification, i.e., model or epistemic uncertainty, often resulting in wrong inferences with high confidence for out of distribution data (Gawlikowski et al., 2023). Conversely, Bayesian neural networks (BNNs) provide a principled way to capture this uncertainty by assigning distributions of values to their weights, instead of point estimates (MacKay, 1992; Goan and Fookes, 2020; Jospin et al., 2022). BNNs then estimate epistemic output uncertainty by sampling the weights’ distributions at inference time.
While BNNs are arguably the most principled way to model uncertainty, they are difficult and often prohibitively expensive to train in practice (Graves, 2011; Blundell et al., 2015; Gal and Ghahramani, 2016; Wu et al., 2018). This motivated several approximations to BNNs to efficiently model epistemic uncertainty. Most success-
ful approaches include drop-out, where variation in the weights is approximated by masking random hidden units (Gal and Ghahramani, 2016; Li and Gal, 2017), and deep ensembles, where uncertainty is estimated with multiple models trained on the same data with different random seeds (Lakshminarayanan et al., 2017; Hoffmann and Elster, 2021). In this work, we aim to approximate model uncertainty for black-box pre-trained LLMs. In this setting, uncertainty is intractable and we assume no access to the weights to modify or train the model.
Uncertainty Quantification for LLMs One way to estimate uncertainty for black-box LLMs is to use the probabilities of the generated tokens (Ott et al., 2018; Zhao et al., 2021; Mielke et al., 2022; Yu et al., 2022). Using relative tokens probabilities is proven to provide good calibration for several tasks (Jiang et al., 2021; Stengel-Eskin and Van Durme, 2022). Recent work also models the uncertainty of full sentences taking into account semantic equivalence. Malinin and Gales (2020) use input-output mutual information to capture uncertainty. Kuhn et al. (2023) compute distributions and uncertainty in a representation space specifically trained such that distances capture semantic similarity. Manakul et al. (2023) uses the LLM itself to evaluate similarity among multiple generations.
Using the LLM token probabilities is a reliable estimate of the uncertainty they learn from training data, i.e., data uncertainty. However, it does not necessarily capture model uncertainty, i.e., for inputs where the model is not competent (Osband et al., 2022). Arguably, model uncertainty is more critical to safety, as it assesses when we can trust the predictions of the LLM (Hüllermeier
and Waegeman, 2021). Analogously to approximate Bayesian approaches for deep learning, several studies propose sampling weights with dropout (Fomicheva et al., 2020; Wang et al., 2022). Other approaches build and train specific components instead, designing Bayesian transformer blocks (Sankararaman et al., 2022) or adding an uncertainty component to existing pre-trained LLMs and fine-tuning the whole architecture (Osband et al., 2022). While effective, these approaches need to have access to the model weights and are computationally expensive to apply at scale. A recently proposed approach for black-box scenarios is that of prompting LLMs themselves to generate estimates of their own confidence (Kadavath et al., 2022; Lin et al., 2022; Zhou et al., 2023). However, it is not trivial to construct prompts that result in accurate confidence assessments (Zhou et al., 2024).
Prompt Ensembles Recent works have explored aggregating prompts to improve predictive performance of LLM prompting (Bai et al., 2022; Shi et al., 2023). Similarly to our own work, Hou et al. (2023) assign different weights to different prompts in the ensemble and optimize them with a validation set. However, they train specific verbalizers for each prompt and their objective is based on classification accuracy maximization. In our own work, we perform direct prompting instead and prompts weights are optimized with a Bayesian objective, balancing data likelihood and uniform exploitation of the prompts for improved calibration. Prompts ensembles have also been explored to improve calibration, studying different strategies to generate diverse prompts for this purpose (Wightman et al., 2023; Jiang et al., 2023b). However, these works aggregate all prompts in the ensemble with equal importance. As LLMs are highly sensitive to prompt engineering (Jiang et al., 2020; White et al., 2023), this approach runs the risk of including low performing prompts in the ensemble and degrade overall performance. Conversely, we hypothesize that by learning the importance of different prompts with a Bayesian approach, we can strike a balance between prompt variation and data likelihood maximization, in analogy with the inference of BNN weight distributions given a prior and data.
Automatic Prompt Engineering Given the difficulty in prompt selection, recent work proposed to automatically generate them (Shin et al., 2020; Zhou et al., 2022b; Yang et al., 2023). While re-
lated to the scope of this work, our aim is in an orthogonal direction, as we focus on how to optimally use a series of pre-defined prompts for uncertainty estimation, rather than generating and optimizing a prompt for improving performance in a particular task. In fact, the two approaches are complementary (see Appendix C.3). Some works in the domain of vision-language models also propose Bayesian treatments of automatic prompt generation, balancing out the evidence from a validation set and regularizing priors (Derakhshani et al., 2023; Liu et al., 2023). However, these methods specifically apply to vision-language models, as they formulate the Bayesian combination of prior and posterior in the common text-image embedding space.
3 Bayesian Prompt Ensembling
Problem Description We consider the problem of text classification with a black-box decoder-only LLM via prompting. In a zero-shot setting, the text input (prompt) to the LLM can be separated into two parts: (i) task instructions a, e.g., “is the sentiment of the following review positive or negative?”; and (ii) test input x, e.g., a movie review in a test set. The prompt a is constant to all inputs for a given task, while test inputs x change, e.g., different reviews to be classified as positive or negative. The class probabilities p(y|x, a) for the corresponding label y are inferred by passing the instruction-test data point pair {a, x} as input to the LLM, e.g., “is the sentiment of the following review positive or negative? {review}”, to obtain the probabilities for the next token or sequence that correspond to the particular classes in the vocabulary, e.g., ‘positive’ and ‘negative’. In a few-shot setting, the task instructions can be accompanied with demonstrations (i.e. a small set of labeled data points).
We also assume that we have access to N semantically equivalent task instruction prompts ai ∈ a, and a small validation data set D of M data points xj ∈ x with corresponding groundtruth labels y∗j ∈ y∗. Our main aim is to optimally exploit the LLM through prompting p(y|x, a), the available labeled validation data D and the given batch of prompt instructions a to obtain well-calibrated classification probabilities p(y|x,D). By well-calibrated, we mean that the LLM assigns, on average, the same confidence to its own prediction as the hidden ground-truth, i.e.,
Ep(y|x,D) log p(y|x,D) ≃ log p(y∗|x,D) for new test input data {x, y∗}. This results in a model which produces high confidence when it generate the correct class token and low confidence when it is likely to generate the incorrect class token. Experimentally, we measure this property with a variety of established calibration metrics. In this paper, we do not aim to optimize the definition or automatic generation of the prompts a. We assume these to be fixed and equally relevant a-priori. We focus on the problem of how to optimally employ them for well-calibrated inference. We study performance with different existing prompt generation strategies in Appendix C.3.
Modeling We model the batch of task instruction prompts a as a latent variable, resulting in the LLM having the form p(y|x,D) =∫ p(a|D)p(y|x, a)da. As the LLM is fixed, the conditional p(y|x, a) is independent of the validation data D. The posterior p(a|D) is the desired distribution of task instructions a given D which is intractable to compute directly. We model the batch of prompts as samples from a prior distribution ai ∼ p(a) of semantically equivalent task instruction prompts for the task of interest. We further assume that all prompts in the batch have approximately the same a-priori importance, i.e., p(ai) ≈ C,∀ai. Approximate Variational Inference As the posterior of prompts given the validation data p(a|D) is intractable to compute, we wish to find a distribution q∗(a) to approximate it. We can formulate this task as a KL divergence minimization using variational inference (Graves, 2011; Gal et al., 2016):
q∗(a) = argmin q(a) KL[q(a)|p(a|D)] =
argmax q(a)
Eq(a)[log p(y∗|a,x)]−KL[q(a)|p(a)].
(1)
The reformulation on the second line of Equation 1 is a standard result of variational inference, its derivation is detailed in Appendix A.1. Once q∗(a) has been computed, the classification likelihood for a new input x can be estimated as:
p(y|x,D) ≃ p(y|x) = ∫ q∗(a)p(y|x, a)da. (2)
We argue that the classification likelihood can be interpreted as the addition of a Bayesian input layer, where a ∼ q∗(a) acts as a stochastic variable that
transforms the LLM input x to {a, x}, i.e., the prompt including instructions and text input, by appending the sampled instruction prompt. Figure 2(a) illustrates this analogy.
Fitting a parametric variational density q(a) through the maximization of Equation 1 is challenging, as this distribution would be a generative language model itself with parameters to be trained; sampling a prompt a ∼ q(a) means generating text with defined properties. While this is certainly an interesting research direction for future work, we aim instead to exploit the set of available predefined prompts ai to efficiently build the prompt ensemble. To this end, we estimate the likelihood of Equation 2 using importance sampling from the prior p(a):
p(y|x) ≈ 1 N
N∑
i
q∗(ai) C p(y|ai, x). (3)
A full derivation is presented in Appendix A.2. Note that, with this importance sampling approximation, the likelihood computation only depends on the value of the variational density q∗(a) at the N points a = ai, i.e., the pre-defined prompts. Figure 2(b) illustrates this approximation.
Therefore, we only need to optimize the discrete values q(ai) in the variational inference of Equation 1. We re-parameterize these values as normalized weights wi = q(ai) NC ∈ w, with wi ∈ [0, 1] and
∑ wi = 1 to re-formulate Equation 1 as:
argmax w
N∑
i
wi log p(y ∗|ai,x)
︸ ︷︷ ︸ Validation Data Likelihood
− N∑
i
wi logwi
︸ ︷︷ ︸ Entropy
(4)
A full derivation is presented in Appendix A.3. The BayesPE objective of Equation 4 consists of two terms. The first term encourages high average likelihood of validation data across the weighted prompts. The second term is the entropy of the weights wi ∈ w and pushes them to be uniformly distributed. As log p(y∗|ai,x) =∑M
j log p(y ∗ j |ai, xj), the relative importance of the validation likelihood term increases with M , i.e., the size of the validation data. In the limit of M → ∞, the entropy term vanishes, leading to always sampling the prompt with the highest validation likelihood ai=best while the rest are discarded (w∗i=best = 1, w ∗ i ̸=best = 0). Conversely, with no validation data, i.e., M = 0, only the entropy term remains, leading to uniformly sample task instruction prompts (w∗i = 1/N , ∀i), which is equivalent to the standard ensembling approach proposed by Wightman et al. (2023) and Jiang et al. (2023b).
Once the BayesPE weights w∗i ∈ w∗ have been learned through the optimization of Equation 4, we can infer class probability distributions for new inputs x:
p(y|x) ≈ N∑
i
w∗i p(y|ai, x) (5)
To sum up, we compute the final class probabilities p(y|x,D) given a new test input x by first prompting the LLM N times, with each of the available instruction prompts ai ∈ a to obtain conditional class probabilities p(y|ai, x). Second, we compute the weighted sum of the class probabilities using the weights w∗i ∈ w∗, learned through the optimization of Equation 4. BayesPE can also be interpreted as a Bayesian mixture of experts (Waterhouse et al., 1995), where weights are obtained through approximate MAP inference (details in Appendix A.4 and C.4).
Lower Bound on Model Error BayesPE provides a lower bound on the expected model error for the LLM, even when its weights are not known. Specifically, if prompts a ∼ q∗(a) are semantically
equivalent to each other, the following inequality is valid:
Eq∗(a)Ep(θ|DT )KL[p(y|x, θ, a)||p(y|x, θ∗, a)] ≥ Ep(θ|DT )Eq∗(a)KL[p(y|x, θ, a)||p(y|x, θ)]
(6)
A full derivation is presented in Appendix A.5. In the notation above, we have added the weights θ of the pre-trained LLM which are not available in black-box settings. p(θ|DT ) is the posterior of these weights, given the data set that was used to train the LLM DT . We assume the pre-trained LLM weights to be a sample from this posterior θ ∼ p(θ|DT ). θ∗ are the optimal weights for the LLM, for which its reducible model error is null. Equation 6 states that the expected divergence between the output given any particular prompt p(y|x, θ, a) and the aggregate output p(y|x, θ), i.e., the uncertainty with respect to the input prompts, is a lower bound to the expected divergence between the given LLM p(y|x, θ, a) and the LLM with optimal weights p(y|x, θ∗, a), i.e., the expected reducible model error. This bound indicates that BayesPE approximates the reducible model uncertainty, as opposed to the data uncertainty. Bayesian neural networks provide an analogous bound on the expected model error (Appendix A.6).
4 Experimental Setup
Data: We evaluate BayesPE on a total of ten tasks. Three sentiment analysis tasks: Amazon Reviews (He and McAuley, 2016), Imdb (Pal et al., 2020) and SST-2 (Socher et al., 2013). Three topic modeling tasks: DBPedia 14, Yahoo Answers (Zhang et al., 2015) and TREC (Hovy et al., 2001). Two semantic relation tasks: MRPC (Warstadt et al., 2018) and SNLI (Bowman et al., 2015). Finally, two spam detection tasks: SMS (Almeida et al., 2011) and YouTube (Alberto et al., 2015). These data sets cover a variety of common NLP tasks, number of classes and text input lengths (details in Appendix B.1).
Models: We evaluate the BayesPE framework with five different pre-trained and instruction finetuned LLMs, spanning families and sizes: Falcon7b-Instruct (Penedo et al., 2023), Falcon-40bInstruct (Almazrouei et al., 2023), MPT-7b-Instruct (Team, 2023b), MPT-30b-Instruct (Team, 2023a) and Mistral-7b-Instruct (Jiang et al., 2023a). More details in Appendix B.2.
Baselines: We compare BayesPE to the following baselines:1
• Standard: The LLM is run once with a single prompt, drawn at random ai ∼ a.
• Ensemble: Class probabilities from all instructions prompts ai are summed together (Wightman et al., 2023; Jiang et al., 2023b).
• Best: The prompt ai with the highest validation accuracy is chosen.
Uncertainty calibration metrics: We compute five metrics: negative log-likelihood (NLL), ROCAUC score (AUC), expected calibration error (ECE), maximum calibration error (MCE) and Brier score (Brier) (Gawlikowski et al., 2023; Abdar et al., 2021).
1In early experiments, we also tested confidence selfassessing, following Kadavath et al. (2022), but found the probabilities to be consistently less calibrated than the tokens probabilities of the class words themselves (Appendix B.4).
Prompts and Validation Data: We consider a fixed set of N initially provided prompts ai for each task. We do not aim to investigate prompt definition or engineering and, unless stated otherwise, we simply draw these at random from a pool of prompts generated with GPT-3.5-turbo (Ye et al., 2023b) with no manual editing. The prompts are used to compute Ensemble, Best and BayesPE, while Standard is computed by drawing just one prompt at random. For few-shot experiments, the prompts include five labeled examples from a different pool of validation data. These are drawn at random for each prompt in the set (details in Appendix B.3).
In each task, unless stated otherwise, we use N = 5 initial prompts, 50 validation examples to select the prompt for Best and train the BayesPE weights (details in Appendix B.4), and 200 test examples to compute metrics. The set of prompts and validation examples are drawn at random ten times in each of our experiments to obtain error bars and compute statistical significance of results.
5 Results
Uncertainty Calibration We evaluate performance of BayesPE and baselines on all tasks with all LLMs in zero- and few-shot settings. Table 1 shows detailed results for three of the ten tested tasks using the overall highest performing configuration, i.e., MPT-30b-Instruct with few-shot prompting. While we are primarily interested in uncertainty quantification, we also report macro F1-Score, to verify that predictive performance is not hindered by our approach. Full results are reported in Appendix C.5. Figure 3 shows calibration curves (accuracy vs. coverage) for the same three experiments.
In Table 1, we observe that BayesPE is competitive or better (within statistical significance) across all tasks and all metrics, with the exception of AUC score in Yahoo Answers, for which it is only 0.01 lower than the best baseline. The calibration curves of figure 3 also show BayesPE clearly outperforming competing baselines. For example, on the Imdb data set, BayesPE achieves 97% accuracy at 80% coverage, while competing methods need to reduce coverage to 45 − 55% to achieve the same. We also see that, among competing baselines, the best approach varies with the task (Ensemble for Yahoo Answers and Best for Amazon Reviews and Imdb), meaning no single best baseline can be selected a-priori. Contrarily, BayesPE is always competitive, making it a far more reliable method. We further note that BayesPE often gives a substantial improvement in calibration (NLL and ECE) compared to standard prompting, e.g., NLL of 3.2 vs. 4.0 and ECE of 0.27 vs. 0.46 for the Yahoo Answers data set. Our approach is also competitive or best across the three tasks also in F1-score, meaning that it maintains or improves predictive performance as well.
The trends described above are observed across all experiments. While inspecting all results in detail is difficult (included in Appendix C.5), we present aggregate results, both in terms of average differences in calibration and statistical significance. Table 2 reports the average difference in AUC score, ECE and MCE each method provided across all 100 experiments (ten tasks × and five LLMs in × zero- and few-shot settings), compared to standard prompting2. BayesPE provides substan-
2NLL and Brier score are not comparable across experiments, as their values depend on the number of classes and is not bounded. Therefore, differences are meaningful only
tial improvement in calibration, with appreciably larger reductions in errors than Ensemble and Best, e.g., −32% in ECE vs. −20-26%.
Table 3 reports in how many of our 100 experiments each method was found to be the best, or within statistical significance of the best (twosample p> 0.05 across ten repeats of each experiment). The number in parenthesis reports in how many experiments each method was significantly better than any other (two-sample p< 0.05 with second best). Overall, BayesPE is significantly better or competitive for every metric in the majority of experiments and in more experiments than any competing baseline. In particular, for metrics tied to expected calibration performance (NLL, ECE and Brier) it is substantially more consistent than the baselines, giving best or competitive NLL and Brier score in 96% of the experiments and ECE in 83% of the experiments. In comparison, other baselines were found competitive for these metrics in only 30− 65% of the experiments.
Inference Efficiency The driving factor for the run-time cost of prompting an LLM is the number of forward passes needed to obtain class probabilities (inference). With our approach, after finding the optimal weights w∗ for a set of N ensembles through a validation set, we can fix a forward passes budget at run-time L <= N . We apply a simple greedy approach to select the run-time prompts and their weights by simply taking the L prompts with the highest weights w∗i to construct the weighted ensemble. Using three data sets and Mistral-7bInstruct as the base LLM in few-shot operation, we evaluate NLL with respect to the number of run-time forward passes. We use N = 20 initial prompts to compute the BayesPE weights and the best prompt (Best) baseline with a validation set and then test with a varying budget of L = 1 : 10 prompts at run-time. Experimental details are given in Appendix B.5. Figure 4 shows the results (additional results in supplementary C.2).
Ensemble and BayesPE display a common trend of decreasing NLL as the forward passes budget is increased. This is expected, as in both cases different prompts are increasingly added to the ensemble at increasing LLM forward passes, which results in increasingly better calibration of the final class probabilities. BayesPE is consistently better than or competitive with Ensemble for all experimental settings and all inference budgets. Even for
within the same experiment and not aggregated.
0.17 0.18 0.19 0.20 0.21
0.28
0.32
0.36
0.40
0.44
2.1 2.2 2.3 2.4 2.5 2.6
Amazon Reviews DBPedia 14
Standard
Ensemble
Best
BayesPE
Number of LLM Forward Passes 2 4 6 108
N LL
(↓ )
Number of LLM Forward Passes 2 4 6 108
Number of LLM Forward Passes 2 4 6 108
Yahoo Answers
Figure 4: Negative log likelihood (NLL) vs. number of forward passes through the LLM at run-time. Using any prompt (Standard) and running the best performing prompt on validation (Best) can be performed only with a single prompt and hence a single forward pass. Standard ensembling (Ensemble) and our BayesPE approach exploit more prompts as the forward passes budget is increased.
0.275
0.325
0.375
0.425
2.1
2.3
2.5
2.7 DBPedia 14 Yahoo AnswersAmazon Reviews
0.17
0.19
0.21
0.23
N LL
(↓ )
Number of Validation Examples 0 20 40 8060
Number of Validation Examples 0 20 40 8060
Number of Validation Examples 0 20 40 8060
Standard
Ensemble
Best
BayesPE
Figure 5: Negative log likelihood (NLL) vs. number of labeled validation examples. Using any prompt (Standard) and aggregating all prompts equally (Ensemble) does not require any validation set and are shown as single points at zero validation examples. Choosing the highest accuracy prompt on validation (Best) and BayesPE benefit from increased number of validation examples.
tasks such as Amazon Reviews in few-shot, where performance with a single prompt is comparable across methods (NLL∼ 0.189), as the budget is increased, BayesPE results in a substantially better improvement in calibration (NLL ∼ 0.169 vs. 178 for Ensembles). This means that BayesPE is a more efficient and effective method of obtaining well-calibrated class probabilities with the LLM at any run-time costs constrain. Note that, at a budget of L = 1, BayesPE is equivalent to picking the best performing prompt on validation (Best). As the budget is increased, BayesPE consistently improves performance (lower NLL). This means that, compared to picking the best prompt on validation, BayesPE consistently utilizes additional computational spend to improve calibration performance, e.g. NLL ∼ 0.169 on the Amazon Reviews task for BayesPE with a budget of 5 forward passes, compared to an NLL of 1.89 for picking the best prompt on validation.
Labeled Data Efficiency The second substantial cost factor in prompting LLMs with BayesPE is the number of labeled examples needed in the validation set. We study NLL as a function of the number of validation examples used to estimate LLM uncertainty. As in the experiments above, we use three data sets and Mistral-7b-Instruct as the base LLM in few-shot operation. Experimental de-
tails are given in Appendix B.5. Figure 5 shows the results (additional results in supplementary C.2).
As expected, we observe that with no validation examples, BayesPE is equivalent to Ensemble, as the likelihood term in Equation 4 is null. BayesPE is competitive or better than any baselines for any budget of validation labeled data, e.g., for DBPedia 14 in Zero-Shot we report an NLL of 0.28 for BayesPE with 50 validation examples vs. 0.31 for for both Best and Ensemble. With the Amazon Reviews task, adding five validation examples initially causes to over-fit and degrade performance. However, as the validation data set increases to tens of examples, this over-fitting is mitigated and the BayesPE NLL matches or improves on that of the Ensemble baseline. BayesPE is also consistently better than the Best baseline in all experiments at all values of labeled data budget, e.g., NLL of 2.13 vs. 2.3 with 20 validation examples and 2.1 vs. 2.2 with 90 validation examples for Yahoo Answers. This means that BayesPE provides consistently better uncertainty quantification, independently of the available amount of validation examples.
6 Conclusion
We proposed Bayesian Prompt Ensembles (BayesPE), an effective method to approximately estimate the model uncertainty for classification
with black-box LLMs. We showed that BayesPE constitutes a discrete approximation to a Bayesian input layer and that its uncertainty gives a lower bound on the model error, analogous to that of BNNs. BayesPE is significantly better or comparable to competitive baselines across the majority of experiments. In our analysis, we showed how BayesPE improves uncertainty quantification over baselines as both the inference and labeled data budget increase.
Limitations
Semantic Equivalence Assumption: To develop the theory of BayesPE, we make the assumption that starting prompt are semantically equivalent. This is challenging to ensure in practice, and even to define. For the purpose of prompting, we formulate this property as stated in Equation 7, but as this definition involves a model with no reducible error, it is impossible to ensure in practice with generated prompts. In our experiments, prompts obtained with automatic rephrasing (a-priori or with validation using APE) proved to be a good enough approximation to obtain good calibration from BayesPE. However, better defining semantic equivalence, testing it and ensuring it for input prompts remains to be studied.
Classification Tasks Only: In our extensive experimental benchmark, we evaluate and compare BayesPE for several classification tasks using LLM prompting, but not with text generation. This is because of computational cost of experiments and complexity of comparing with ground-truth. With classification tasks, we can measure several wellestablished calibration and uncertainty metrics that can be readily computed given the output class probability, while in text generation these measures are only recently emerging and are much more computationally expensive to compute and with high variance, as they inevitably involve repeated sampling.
Access to Words Probabilities: In the evaluations presented in this paper we use different LLMs in black-box implementation, i.e, with no access to the weights, but we do use words probabilities to build classification distributions. However, some proprietary LLMs, e.g., GPT-4, do not even make these available and only return text answers to input prompts. BayesPE and other ensemble methods can still be applied, but require sampling multiple
times, extracting classes from the resulting text outputs and aggregating the results to build output probabilities for each prompt. This is more computationally expensive and less robust than using directly class words probabilities when available.
Prompt Ensemble Generation: In this work, we focus on how to optimally exploit a set of prompts assumed to be semantically equivalent, but we do not propose a specific method to generate suitable prompts to build this ensemble. This is an important problem to be investigated in future work, as prompts automatically generated in the different ways we consider in this paper may introduce biases and be overall sub-optimal for the downstream task.
References Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana
Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. 2021. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243– 297.
Túlio C Alberto, Johannes V Lochter, and Tiago A Almeida. 2015. Tubespam: Comment spam filtering on youtube. In 2015 IEEE 14th international conference on machine learning and applications (ICMLA), pages 138–143. IEEE.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance.
Tiago A. Almeida, Jose Maria Gomez Hidalgo, and Akebo Yamakami. 2011. Contributions to the study of sms spam filtering: New collection and results. In Proceedings of the 2011 ACM Symposium on Document Engineering (DOCENG’11).
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565.
Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. 2003. An introduction to mcmc for machine learning. Machine learning, 50:5– 43.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional
ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613–1622.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Andreas Damianou and Neil D Lawrence. 2013. Deep gaussian processes. In Artificial intelligence and statistics, pages 207–215. PMLR.
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. 2023. Bayesian prompt learning for imagelanguage model generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15237–15246.
Giancarlo Ferrari-Trecate, Christopher Williams, and Manfred Opper. 1998. Finite-dimensional approximation of gaussian processes. Advances in neural information processing systems, 11.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR.
Yarin Gal et al. 2016. Uncertainty in deep learning. PhD thesis, University of Cambridge.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. 2023. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513–1589.
Ethan Goan and Clinton Fookes. 2020. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM JeanMorlet Chair, Fall 2018, pages 45–87.
Alex Graves. 2011. Practical variational inference for neural networks. Advances in neural information processing systems, 24.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507–517.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2021. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916.
Lara Hoffmann and Clemens Elster. 2021. Deep ensembles from a bayesian perspective. arXiv preprint arXiv:2105.13283.
Bairu Hou, Joe O’connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2023. Promptboosting: Black-box text classification with ten forward passes. In International Conference on Machine Learning, pages 13309–13324. PMLR.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, ChinYew Lin, and Deepak Ravichandran. 2001. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research.
Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110:457–506.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825.
Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. 2023b. Calibrating language models via augmented prompt ensembles.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know. Transactions of the Association for Computational Linguistics, 8:423–438.
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. 2022. Hands-on bayesian neural networks—a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29–48.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma,
Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023. Gpt-4 passes the bar exam. Available at SSRN 4389233.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
Yingzhen Li and Yarin Gal. 2017. Dropout inference in bayesian neural networks with alpha-divergences. In International conference on machine learning, pages 2052–2061. PMLR.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334.
Xinyang Liu, Dongsheng Wang, Miaoge Li, Zhibin Duan, Yishi Xu, Bo Chen, and Mingyuan Zhou. 2023. Patch-token aligned bayesian prompt learning for vision-language models. arXiv preprint arXiv:2303.09100.
David JC MacKay. 1992. A practical bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472.
David JC MacKay et al. 1998. Introduction to gaussian processes. NATO ASI series F computer and systems sciences, 168:133–166.
Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896.
Sabrina J Mielke, Arthur Szlam, Emily Dinan, and YLan Boureau. 2022. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872.
Toby J Mitchell and John J Beauchamp. 1988. Bayesian variable selection in linear regression. Journal of the american statistical association, 83(404):1023– 1032.
Ian Osband, Seyed Mohammad Asghari, Benjamin Van Roy, Nat McAleese, John Aslanides, and Geoffrey Irving. 2022. Fine-tuning language models via epistemic neural networks. arXiv preprint arXiv:2211.01568.
Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3956–3965. PMLR.
Aditya Pal, Abhilash Barigidad, and Abhijit Mustafi. 2020. Imdb movie reviews dataset.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
Karthik Abinav Sankararaman, Sinong Wang, and Han Fang. 2022. Bayesformer: Transformer with uncertainty estimation. arXiv preprint arXiv:2206.00826.
Matthias Seeger. 2004. Gaussian processes for machine learning. International journal of neural systems, 14(02):69–106.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrievalaugmented black-box language models. arXiv preprint arXiv:2301.12652.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
Elias Stengel-Eskin and Benjamin Van Durme. 2022. Calibrated interpretation: Confidence estimation in semantic parsing. arXiv preprint arXiv:2211.07443.
MosaicML NLP Team. 2023a. Introducing mpt-30b: Raising the bar for open-source foundation models (www.mosaicml.com/blog/mpt-30b).
MosaicML NLP Team. 2023b. Introducing mpt-7b: A new standard for open-source, commercially usable llms (www.mosaicml.com/blog/mpt-7b).
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. 2022. Uncertainty estimation and reduction of pre-trained models for text regression. Transactions of the Association for Computational Linguistics, 10:680–696.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471.
Steve Waterhouse, David MacKay, and Anthony Robinson. 1995. Bayesian methods for mixtures of experts. Advances in neural information processing systems, 8.
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382.
Gwenyth Portillo Wightman, Alexandra DeLucia, and Mark Dredze. 2023. Strength in numbers: Estimating confidence of large language models by prompt agreement. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 326–362.
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, Jose Miguel Hernandez-Lobato, and Alexander L Gaunt. 2018. Deterministic variational inference for robust bayesian neural networks. arXiv preprint arXiv:1810.03958.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023a. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023b. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420.
Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang. 2022. Cold-start data selection for few-shot language model fine-tuning: A prompt-based uncertainty propagation approach. arXiv preprint arXiv:2209.06995.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR.
Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. 2024. Relying on the unreliable: The impact of language models’ reluctance to express uncertainty. arXiv preprint arXiv:2401.06730.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. arXiv preprint arXiv:2302.13439.
Xinlei Zhou, Han Liu, Farhad Pourpanah, Tieyong Zeng, and Xizhao Wang. 2022a. A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications. Neurocomputing, 489:449–465.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022b. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.
A Detailed Derivations
A.1 Derivation of Variational Inference for prompts distribution
We derive here the variational inference maximisation formula of equation 1. Starting from the KL divergence to be minimised:
KL[q(a)|p(a|D)] = KL[q(a)|p(a|x,y∗)] = ∫
q(a) log q(a)
p(a|x,y∗)da =∫ q(a) log
q(a)p(y∗) p(y∗|a,x)p(a)da =∫
q(a) log q(a) p(a) da+ log p(y∗)
− ∫ q(a) log p(y∗|a,x)da.
Because the log probability log p(y∗) does not depend on q(a), we can re-formulate the minimisation as:
argmin q(a)
∫ q(a) log q(a)
p(a) da+ log p(y∗)
− ∫ q(a) log p(y∗|a,x)da =
argmax q(a)
− ∫ q(a) log q(a)
p(a) da
+ ∫ q(a) log p(y∗|a,x)da =
argmax q(a)
Eq(a)[log p(y|a, x)]−KL[q(a)|p(a)]
A.2 Derivation of Importance Sampling Approximation of Classification Likelihood
We derive here the importance sampling approximation of equation 3. After performing variational inference optimisation for the distribution q(a) in equation 1, obtaining the optimised q∗(a), the final classification likelihood is approximated as folows:
p(y|x,D) = ∫ p(a|D)p(y|x, a)da
≃ ∫ q∗(a)p(y|x, a)da.
We can now perform importance sampling with the prior p(a):
∫ q∗(a)p(y|x, a)da = ∫ p(a)
q∗(a) p(a) p(y|x, a)da
≈ 1 N
N∑
i
q∗(ai) p(ai) p(y|x, ai)
≈ 1 N
N∑
i
q∗(ai) C p(y|x, ai)
The last line follows as all propmts are assumed to have the same importance a-priori, i.e., p(ai) ≈ C ∀ai.
A.3 Derivation of Importance Sampling Approximation of Variational Inference Objective
We derive here the importance sampling approximation of equation 4. Starting from the variational inference objective function:
Eq(a)[log p(y∗|a,x)]−KL[q(a)|p(a)]
= ∫ q(a) log p(y∗|a,x)da − ∫ q(a) log q(a)
p(a) da
= ∫ p(a) q(a)
p(a) log p(y∗|a,x)da
− ∫ p(a) q(a)
p(a) log
q(a) p(a) da
≈ N∑
i
q(ai)
p(ai) log p(y∗|ai,x)
− N∑
i
q(ai) p(ai) log q(ai) p(ai) da.
≈ 1 N
N∑
i
q(ai)
C log p(y∗|ai,x)
− 1 N
N∑
i
q(ai)
C log
q(ai)
C .
The last line follows as all propmts are assumed to have the same importance a-priori, i.e., p(ai) ≈ C ∀ai. As the likelihood estimation of equation 3 only depends on q(ai), we only need to optimise these discrete values with variational inference, as we only care about the density of our variational
distribution q(a) at these points:
argmax q(ai),∀ai∈a
1
N
N∑
i
q(ai)
C log p(y∗|ai,x)
− 1 N
N∑
i
q(ai)
C log
q(ai)
C .
We now re-parametrise the density evaluations to be optimised q(ai) with the normalised weights wi = q(ai)/NC ∈ w, with wi ∈ [0, 1] and∑
wi = 1, obtaining the BayesPE optimisation:
argmax w
N∑
i
wi log p(y ∗|ai,x)−
N∑
i
wi logwi
A.4 Relation to Bayesian Mixture of Experts
BayesPE can be viewed as a mixture of experts (MoE) with weights obtained through a lower bound approximation of MAP inference with an entropy maximizing prior over the weights. In the Bayesian MoE formulation, a target function f(x), in our case the target predictive distribution f(x) = p(y|x), is built with a finite sum of basis functions ϕi(x):
f(x) =
N∑
i
wiϕi(x).
In BayesPE, the basis functions correspond to the predictive probabilities obtained by prompting with different instructions prompts ai, i.e., ϕi(x) = p(y|ai, x), recovering:
p(y|x) = N∑
i
wip(y|ai, x),
which is equivalent to equation 5. In Bayesian MoE, the prior over the weights wi is typically assumed to be a Gaussian. There is generally different ways in which to model the posterior distribution of the weights. Arguably the simplest and more closely related to BayesPE is Bayesian linear regression, where a fixed posterior of weights p(w|x,y∗) for the linear combination of the basis function (p(y|ai, x) in our LLM prompting scenario) is learned with the validation data x and y∗. Under the Gaussian prior and likelihood assumption, the posterior is also Gaussian and tractable to compute through matrix inversion (Mitchell and Beauchamp, 1988).
Under this Bayesian MoE perspective, BayesPE models the weights with a maximal entropy prior p(w) = ∏ w−wii s.t. ∑ wi = 1 and wi ∈ [0, 1], instead of a Gaussian. This results in an intractable posterior and the objective of BayesPE is equivalent to performing a lower bound approximation of MAP inference of this posterior on the validation set of observed data:
argmax w
log p(w|x,y∗)
∝ log[p(y∗|w,x)p(w)]
= log[ N∑
i
wip(y ∗|ai,x)] + log[
N∏
i
w−wii ]
≥ N∑
i
wi log p(y ∗|ai,x)−
N∑
i
wi logwi.
The last inequality is obtained through Jensen’s inequality, meaning that the objective of BayesPE (equation 4) is a lower bound on the MAP objective in the Bayesian MoE formulation. We experimentally compare BayesPE to Bayesian linear regression in Appendix C.4.
There exist more advanced methods to model the weights in Bayesian MoEs that could be applied to prompt ensembles. One interesting possibility is modeling the ensemble as a finite dimensional Gaussian Processes (Ferrari-Trecate et al., 1998; Waterhouse et al., 1995). The MoE is modeled as a Gaussian process with Gaussian prior over the weights w and hyper-parameters of the kernel are learned with a validation data set. At inference time, the learned kernel can be used to infer the particular weights combination for the current input. Applying this Gaussian process modeling approach for LLM prompt ensemble is an interesting direction for future research. On one hand, we can expect some advantage from the adaptive nature of the weights at inference time through the kernel, as opposed to the fixed weights learned by BayesPE. On the other, having to compute the covariance matrix at inference time implies running the LLM with all prompts for the new example, precluding the possibility to adjust inference budget at test time (as we do with BayesPE in the experiments of figure 4).
A.5 Derivation of Model Error Lower Bound for Prompts Ensembles
We derive here the inequality of equation 6. First, we expand the definition of our LLM of choice,
used as a classifier p(y|a, x) to include a specification of values for its weights θ; p(y|a, θ, x). We assume no knowledge of the specific values of θ, i.e., black-box scenario, but, as the model was arbitrarily pre-trained with some training data set DT , we assume these weights to be sampled from the posterior p(θ|DT ).
Second, we define optimal weights θ∗ which specify an optimal version of the LLM p(y|a, θ∗, x) which has no reducible error. We can think of this model as a version of the LLM trained with infinite data, leading to vanishing model error. With these definitions, we now formalise the semantic equivalence property of p(a). Specifically, we state:
p(y|x, θ∗, a) = p(y|x, θ∗, a′) ∀a, a′ ∼ p(a) =⇒ p(y|x, θ∗) = p(y|x, θ∗, a) ∀a ∼ p(a).
(7)
This means that we assume the distribution p(a) to capture semantic equivalence, if the optimal model p(y|x, θ∗, a) is invariant to task instructions prompts a drawn from it. This is because the optimal model p(y|x, θ∗, a) is assumed to be trained with infinite data, including all possible semantically equivalent inputs for a given output. In the limit of the BayesPE approximation, this property transfers to the learned variational distribution q∗(a), as we sample it only at prompts ai ∼ p(a). With these definitions in place, we can now derive the inequality of equation 6:
Eq∗(a)Ep(θ|DT )KL[p(y|x, θ, a)||p(y|x, θ∗, a)] =∫ q∗(a) ∫ p(θ|DT ) ∫ p(y|x, θ, a)·
log p(y|x, θ, a) p(y|x, θ∗, a)dθdady =∫ q∗(a) ∫ p(θ|DT )· ∫ p(y|x, θ, a) log p(y|x, θ, a)p(y|x, θ)
p(y|x, θ∗, a)p(y|x, θ)dθdady =∫ p(θ|DT ) ∫ q∗(a) ∫ p(y|x, θ, a)·
log p(y|x, θ, a) p(y|x, θ) dadθdy+∫
p(θ|DT ) ∫ q∗(a) ∫ p(y|x, θ, a)·
log p(y|x, θ)
p(y|x, θ∗, a)dadθdy
Following the condition of equation 7, we can set p(y|x, θ∗, a) = p(y|x, θ∗). Replacing this term in
the second component on the last line above results in: ∫
p(θ|DT ) ∫ q∗(a) ∫ p(y|x, θ, a)·
log p(y|x, θ, a) p(y|x, θ) dadθdy+∫
p(θ|DT ) ∫ ∫ q∗(a)p(y|x, θ, a)da·
log p(y|x, θ) p(y|x, θ∗)dθdy =∫ p(θ|DT ) ∫ q∗(a) ∫ p(y|x, θ, a)·
log p(y|x, θ, a) p(y|x, θ) dadθdy+∫
p(θ|DT ) ∫
p(y|x, θ) log p(y|x, θ) p(y|x, θ∗)dθdy =
Ep(a|a0)Eq(θ)KL[p(y|x, θ, a)||p(y|x, θ∗, a)] = Ep(θ|DT ) [ Eq∗(a)KL[p(y|x, θ, a)||p(y|x, θ)]
+KL[p(y|x, θ)||p(y|x, θ∗)] ]
The inequality of equation 6 follows as the second KL divergence is non-negative. The lower bound of equation 6 means that the uncertainty from aggregating output probabilities from different semantically equivalent prompts, as defined in 7, is a lower bound to the expected model error, defined as the KL divergence between the LLM output probability and the output probability of the optimal LLM with no model error (with weights θ∗). This property actually applies to any method ensembling semantically equivalent prompts, including Wightman et al. (2023) and Jiang et al. (2023b). Therefore, this theoretical finding validates prompt ensemble approaches in general as LLM model error approximators. This lower bound generalizes a property which is intuitive to understand with two prompts; If I know the answer to two prompts must be the same (semantic equivalence condition) and I obtain two different answers, at least one of the two answers is wrong. Equation 6 generalizes this to continuous class probabilities and any number of prompts.
A.6 Derivation of Model Error Lower Bound for Bayesian NNs
The lower bound of equation 6 is analogous to the bound on model error we get with Bayesian neural networks. Consider a Bayesian neural network, with posterior weights distribution p(θ|DT ) and optimal weights θ∗, for which the resulting model
p(y|x, θ∗) has no reducible error.
Ep(θ|DT )KL[p(y|x, θ)||p(y|x, θ∗)] =∫ p(θ|DT ) ∫ p(y|x, θ) log p(y|x, θ)
p(y|x, θ∗)dθ =∫ p(θ|DT ) ∫ p(y|x, θ) log p(y|x, θ)p(y|x)
p(y|x, θ∗)p(y|x)dθ =∫ p(θ|DT ) ∫ p(y|x, θ) log p(y|x, θ)
p(y|x) dθ+∫ p(y|x) log p(y|x)
p(y|x, θ∗)dθ =
Ep(θ|DT )KL[p(y|x, θ)||p(y|x)]+ KL[p(y|x)||p(y|x, θ∗)]
As the second KL divergence in the equation above is non-negative, we can derive the following bound:
Ep(θ|DT )KL[p(y|x, θ)||p(y|x, θ∗)] ≥ Ep(θ|DT )KL[p(y|x, θ)||p(y|x)].
Which is analogous to the bound of equation 6, with the difference that the average divergence is with respect to the change in model’s weights θ instead of prompts a.
B Details of Experiments
B.1 Details of Data Sets For our evaluation, we test on 10 different NLP classification tasks with the respective test sets:
• Amazon Reviews (Appliances): Sentiment analysis on Amazon reviews of appliances. Two classes; positive and negative.
• DBPedia 14: Topic modeling on a data set of encyclopedia descriptions corresponding to 14 categories.
• Glue (mrpc): Binary equivalence judgment on pairs of sentences.
• Imdb: Sentiment analysis on a data set of movie reviews on Imdb. Two classes; positive and negative.
• SNLI: Entailment classification on a data set of pairs of sentences, where the first sentence implies, contradicts or does not imply the second (3 classes).
• SST-2: Sentiment analysis on a Data set of segments from movie reviews. Two classes; positive and negative.
• TREC: Topic modeling on a data set of questions labeled with type of information requested. We are using only the coarse labels (6 classes, reduced to 5 for class balance).
• Yahoo Answers: Topic modeling on a data set of Questions and answers on Yahoo answers. 10 classes.
• YouTube: Spam detection on comments left under YouTube videos. Two classes; Spam and Ham.
• SMS: Spam detection on text messages. Two classes; Spam and Ham.
In all our experiments, we balance the test sets we use to evaluate metrics across classes. This is to ensure that all metrics are meaningful measure of performance in each case, as for heavily unbalanced sets accuracy/likelihood based metrics, such as negative log-likelihood, are not meaningful measures of model performance.
B.2 Details of LLMs We use 5 different open-source LLMs, across different families and sizes. They are listed in Table 4:
B.3 Details of Initial Prompts Generation For each task, we manually write one prompt instruction, which we then automatically rephrase multiple times with gpt-3.5-turbo. The initial manually defined prompt is not validated on any data and the generated prompts are not manually edited. While better prompts can certainly be defined in more principled engineering ways with validation data sets (e.g. Shin et al. (2020)), we are interested in the optimal exploitation of given prompts and avoiding prompt engineering ensures that our findings are not biased to certain manual fine-tuning choices or particular prompt generation methods. The initial prompts for each data set are as follows:
• Amazon Reviews: classify the sentiment of the Amazon review below into one of the following classes:
• DBPedia 14: The following is a description of something on Wikipedia. Which of the following categories does the description fall in?
• Glue (mrpc): Do the two sentences given below have the same meaning?
• Imdb: classify the below movie review into one of the following classes:
• SNLI: Classify the relation between the hypothesis and thesis below into one of the following classes:
• SST-2: classify the sentiment of the movie review extract below into one of the following classes:
• TREC: Classify the question below into one of the following type of information classes:
• Yahoo Answers: classify the question and answer below into one of the following topics:
• YouTube: Is the following Youtube comment spam?
• SMS: Is the following text message spam?
In each case, the above is followed by a numbered list of the class words. The initial prompt for each task is then used to formulate a re-phrasing promt to be given to gpt-3.5-turbo. For the first generated prompt, gpt-3.5-turbo is given the prompt "Rephrase the following task instructions such that the meaning remains unchanged. /n {initial prompt}". For all subsequent prompts, the already generated prompts are provided in a list with the prompt ’Provide one more example of rephrased task instructions, such that the meaning is the same as the original task instructions. /n {list of examples} /n Original instructions: {initial prompt}’. For the experiments of section 5, we generate 10 total prompts, from which we draw 5 in each repetition of the experiments. For few-shot, we also draw at random a different set of 5 labeled examples from a held-out set (different from the set we use as validation in the experiments) and append it to each different re-phrasing of the instructions, in order to obtain the final different prompts ai.
B.4 Details of BayesPE training
the BayesPE weights are trained by performing the maximisation of equation 4, with validation data D = {x,y∗} with the LBFGS optimiser with 100 maximum iterations. The learning rate is set to 10−3, however, if the cost becomes NaN or Inf, the optimisation is restarted with halfed learning rate and the process is repeated until succesful convergence. This occured for a few LLM-task combinations in our experiments. Other than this error recovery strategy, the BayesPE optimisation parameters are constant to all experiments. BayesPE optimisation is inexpensive (1-2 seconds on CPU), as it only needs to optimise a few linear weights for class probabilities with 2-15 dimensions.
B.5 Details of Samples Efficiency Experiments
To test performance at varying budget of LLM forward passes, we use Mistral7bInstruct as the base LLM and evaluate performance on three tasks: Amazon Reviews, DBPedia 14 and Yaho Answers. For these experiments, as in the full benchmark, we use a validation set of 50 examples, approximately balanced across classes. This validation set is used to train the weights for BayesPE and select the prompt with the highest validation accuracy for Best. We test with 400 examples, also balanced across classes. We consider a pool of 50 initial prompts. For every repetition of the experiments, we draw a set of N = 20 at random from them and use them to optimise the BayesPE weights w∗
and determine the highest accuracy prompt for the Best baseline. The forward passes budget L is then varied between 1 and 10. Best and Standard only operate with a single prompt, therefore they only appear as single points at L = 1 in Figure 4. For the baseline Ensemble, L prompts are sampled at random and aggregated to obtain output probabilities. For BayesPE, the prompts corresponding to the top L weights w∗i are taken and aggregated,
weighted by the corresponding weights. The resulting probabilities are re-normalised.
B.6 Details of Labeled Data Efficiency Experiments
To test performance at varying number of labeled validation examples, we use Mistral7bInstruct as the base LLM and evaluate performance on three tasks: Amazon Reviews, DBPedia 14 and Yaho Answers. For each task, we use a test set of 400 examples to evaluate performance and vary the available validation data set size from M = 0 to M = 100. Similarly to the samples efficiency ablations, for each value of validation set size M , we draw N = 20 prompts from a pool of 50 at random and use them on the validation set to choose the prompt with highest accuracy (Best) and to optimise the BayesPE weights w∗. At run time, we use an inference budget of L = 10 LLM forward passes and hence 10 prompts. For each method, these are chosen from the total N = 20 in the same way as in the samples efficiency experiments (Appendix B.5).
C Additional Results
C.1 Early Experiments with LLM Self-Assessing of Confidence
We initially considered the self-assessing of confidence by the LLM itself as a baseline for uncertainty estimation ad calibration of output probabilities. This method consists in feeding the LLM the
original prompt and its original answer and then prompting again to assess if the answer is correct. The probability of the next word confirming correctness is taken as the probability of the class given in the answer, e.g., ’is the above answer correct?’ p(next word = ’yes’) = p(y = chosen class) (Kadavath et al., 2022). In the binary case, the probability distribution is directly derived; if chosen class is 1 p(y = 1) = p(next word = ’yes’) and p(y = 0) = 1 − p(next word = ’yes’). However, in the multi class case, there is not direct mapping. In these cases, we assign the chosen class the confirming word probability (p(next word = ’yes’)) and distribute the remaining probability mass uniformly amongst the remaining classes. We tested this approach in initial ablations with Mistral7bInstruct in zero-shot mode with four tasks. Results are shown in table 5.
We found the standard approach of using the LLM probabilities assigned to classes words (Standard in table 5 and rest of the paper) to consistently and substantially outperform this approach of reevaluating confidence in the given answer (SelfAssess in table 5). As a result, we ruled out this approach as a competitive baseline for BayesPE in all further experiments.
C.2 Additional Efficiency results
Inference Efficiency We show in figure 6 an extended version of the results of figure 4, including zero-shot results. Trends for zero-shot are analogous to those observed in few-shot experiments,
0.17 0.18 0.19 0.20 0.21
0.28
0.32
0.36
0.40
0.44
2.1 2.2 2.3 2.4 2.5 2.6
Amazon Reviews Few-Shot DBPedia 14 Few-Shot
0.175
0.185
0.195 0.205 N LL (↓ )
0.6
0.7
0.8
0.9
2.15
2.25
2.35
2.45
Number of LLM Forward Passes 2 4 6 108
Amazon Reviews Zero-Shot DBPedia 14 Zero-Shot Yahoo Answers Zero-Shot
Standard
Ensemble
Best
BayesPE
Number of LLM Forward Passes 2 4 6 108
Number of LLM Forward Passes 2 4 6 108
Number of LLM Forward Passes 2 4 6 108
N LL
(↓ )
Number of LLM Forward Passes 2 4 6 108
Number of LLM Forward Passes 2 4 6 108
Yahoo Answers Few-Shot
with BayesPE being consistently better than or competitive with Ensemble for all experimental settings and all inference budgets and providing consistent improvement compared to picking the best prompt on validation. For instance, on DBPedia 14 and a budget of 5 forward passes in zero-shot, BayesPE achieves an NLL of 0.65, while standard ensembling can only achieve an average of 0.77.
Labelled Data Efficiency We show in figure 7 an extended version of the results of figure 5, including zero-shot results. Trends are analogous to those observed in the few-shot results, e.g., for DBPedia 14 in Zero-Shot we report an NLL of 0.67 for BayesPE with 50 validation examples vs. 0.75 for Best and 0.80 for Ensemble. In some cases,
most evidently for Yahoo Answers in zero-shot, adding very few validation examples (5-10) initially causes to over-fit and degrade performance (from NLL 2.17 with no validation to 2.25 with five validation examples). However, as the validation data set increases to several tens of examples, this over-fitting is mitigated and the BayesPE NLL matches or improves on that of the Ensemble baseline (e.g. NLL back to 2.17 with 90 validation example for Yahoo Answers in zero-shot).
C.3 Prompt Definition
In all experiments presented above, we have tested the utilization of given prompts independently of any particular engineering, as the set of initial prompts was pre-defined. We investigate here
how different generation strategies for the prompts ai ∈ a affect performance. First, we evaluate performance for different templated prompt augmentation strategies, adopting the different alterations detailed in Jiang et al. (2023b). Second, we test BayesPE in combination with automatic prompt engineering (APE) (Zhou et al., 2022b), which uses the validation set to refine the definition of the initial prompts.
Prompt Augmentation Strategies We adopt the augmentation strategies proposed by Jiang et al. (2023b), adding them one at a time in our experiments to increase prompt variation in each experiment. For these experiments, we use the Yahoo Answers data set, Mistral-7b-Instruct as the base LLM and operate in a few-shot scenarion, providing 5 5 examples in the input prompts. We use 10 prompts to build the ensemble and 50 validation examples. The in-context examples are taken from a separate set of 50 examples. We test with initial prompts defined in the following three ways:
• Rephrase Only: We use GPT-3.5-Turbo to rephrase the task instructions (see Appendix B.3), but in-context examples for few shot and order of multiple choices is fixed.
• Rephrase+Examples: Instructions are rephrased as described above and the 5 in-context examples for few-shot operations are sampled and ordered at random in each of the 10 prompts.
• Rephrase+Examples+MC: Rephrasing and random sampling of the examples are applied as described above and, in addition, multiple choices for the classes presented in the prompts are permuted at random.
The diversity of the prompts icreases with the addition of each included augmentation strategy. We repeat each experiment 10 times, each time drawing at random the 50 validation examples, the incontext examples and the set of 10 re-phrased instructions. Results are shown in table 6. Note that, in these experiments, the Ensemble baseline corresponds to the approach proposed in Jiang et al. (2023b).
In the results of table 6, BayesPE is consistently better or competitive to baselines, providing 48-63% reduction in ECE compared to standard prompting and 4-12% compared to Ensemble, which is the most competitive baseline. We also
note that, unlike all competing strategies, BayesPE is less reliant on the prompt definition strategy. For example the ECE is ∼ 8% for all prompt definitions, while for ensembles it reduces from 9.2% to 8.4% as prompt diversity is increased. This means that BayesPE is more robust to the initial engineering of the set of prompts in the ensemble, alleviating the burden of designing effective and diverse prompts for calibration.
Automatic Prompt Engineering and BayesPE Given a validation set of labeled examples, recent work has proposed different strategies to automatically generate suitable prompts for a given task (Shin et al., 2020; Zhou et al., 2022b; Yang et al., 2023). To test the combination of BayesPE with automatic prompts, we follow Zhou et al. (2022b) and use automatic prompt engineering (APE) to define the initial prompts. We run APE 10 times with the validation set to obtain the initial set of prompts and then apply BayesPE and all baselines, using the same validation set where relevant. For these experiments, we consider the same three tasks as for the efficiency experiments (Amazon Reviews, DBPedia 14 and Yahoo Answers) and use Mistral7BInstruct as the base LLM. We perform experiments in zero-shot format only, as it is not trivial to adapt the APE approach to few-shot scenarios. We run APE (Zhou et al., 2022b) 90 times to obtain an initial pool of prompts. We then select the top 30 based on validation log likelihood. From these, we sample 10 at random and use them for BayesPE, Ensemble and Best. The standard APE baseline is tested by selecting one of the 30 prompts
at random. Prompt sampling is repeated 10 times to obtain error bars and compute statistical significance. In these experiments, the validation set is fixed and is composed of 100 labeled examples for each task. The test set is also fixed and is composed of 400 examples. Table 7 reports results of these experiments.
Using BayesPE in combination with APE (APE+BayesPE), was found to consistently improve calibration performance over APE alone. In particular, for the Amazon Reviews and Yahoo Answers data sets, we observed substantial reduction of about 30% in ECE and 45− 55% in MCE. We also confirm that the deterministic performance, measured by F1-score, is not negatively affected by using BayesPE in combination with APE, remaining competitive or better (difference in F1 from 0 to +0.03) with respect to this metric as well. We also observe that BayesPE remains competitive or better than other baselines methods to exploit multiple prompts and validation data.
C.4 Comparison to Bayesian Mixture of Experts
As described in Appendix A.4, BayesPE can be interpreted as a strategy for Bayesian Mixture of Experts (MoE), using the LLM with different prompts as basis functions. We perform some experiments to compare BayesPE to linear Bayesian regression over the weights of the Bayesian MoE. In particular, we preform Ridge regression, where the prior over the weights is a Gaussian with diagonal covariance. To introduce a regularisation which encourages uniform exploitation of the prompts a-priori, we set the means of the prior over the weights to 1/N and
the diagonal variances to 1. We then infer the posterior using the available validation examples. We compare this approach to BayesPE and the other baselines (Standard, Ensemble and Best). For these experiments, we use the Amazon Reviews, Imdb and Yahoo Answer data sets, an initial ensemble of ten few-shot prompts, designed as described in Appendix B.3, 50 validation examples and Mistral7b-Instruct as the base LLM. Each experiment is repeated 10 times for statistical significance.
As shown in table 8, BayesPE is competitive or better than using linear regression to train the weights w in a Bayesian MoE formulation, with ECE being equal (Imdb) to 20− 56% lower (Amazon Reviews and Yahoo Answers).
C.5 Detailed Results of Benchmark Evaluation
Tables 9-18 report the results of our extensive benchmark evaluations for zero-shot classification in full. Errorbars were obtained by repeating the experiments 10 times, drawing a set of 5 prompts as described in Appendix B.3 each repeat. Tables 19-28 report results for the equivalent few-shot experiments.
C.6 Additional Statistical Significance Results
We report here analogous summary results of our benchmark experiments to those presented in table 3, separated for zero-shot and few shot and then for each individual LLM. Tables 29 and 30 report
summary results for zero-shot and few shot experiments respectively. Tables 31 to 35 reprt summary results for each LLM tested separately. In all tables 29 to 35, numbers in each cell report in how many experiments out of the ones performed each approach was found to be best performing or within statistical significance of the best performing (twosample p-value> 0.05 over the repetitions of each experiment) for each measured metric. The number in parenthesis reports how many times the method was found to be significantly better than any other (p-value< 0.05).

heading: 20(4) 14(1) 19(4) 18(2) 20(4)

heading: Future Work
text: While this is certainly an interesting research direction for future work, we aim instead to exploit the set of available predefined prompts ai to efficiently build the prompt ensemble. To this end, we estimate the likelihood of Equation 2 using importance sampling from the prior p(a):
p(y|x) ≈ 1 N
N∑
i
q∗(ai) C p(y|ai, x). (3)
A full derivation is presented in Appendix A. 2. Note that, with this importance sampling approximation, the likelihood computation only depends on the value of the variational density q∗(a) at the N points a = ai, i. e. , the pre-defined prompts. Figure 2(b) illustrates this approximation. Therefore, we only need to optimize the discrete values q(ai) in the variational inference of Equation 1. We re-parameterize these values as normalized weights wi = q(ai) NC ∈ w, with wi ∈ [0, 1] and
∑ wi = 1 to re-formulate Equation 1 as:
argmax w
N∑
i
wi log p(y ∗|ai,x)
︸ ︷︷ ︸ Validation Data Likelihood
− N∑
i
wi logwi
︸ ︷︷ ︸ Entropy
(4)
A full derivation is presented in Appendix A. 3. The BayesPE objective of Equation 4 consists of two terms. The first term encourages high average likelihood of validation data across the weighted prompts. The second term is the entropy of the weights wi ∈ w and pushes them to be uniformly distributed. As log p(y∗|ai,x) =∑M
j log p(y ∗ j |ai, xj), the relative importance of the validation likelihood term increases with M , i. e. , the size of the validation data. In the limit of M → ∞, the entropy term vanishes, leading to always sampling the prompt with the highest validation likelihood ai=best while the rest are discarded (w∗i=best = 1, w ∗ i ̸=best = 0). Conversely, with no validation data, i. e. , M = 0, only the entropy term remains, leading to uniformly sample task instruction prompts (w∗i = 1/N , ∀i), which is equivalent to the standard ensembling approach proposed by Wightman et al. (2023) and Jiang et al. (2023b). Once the BayesPE weights w∗i ∈ w∗ have been learned through the optimization of Equation 4, we can infer class probability distributions for new inputs x:
p(y|x) ≈ N∑
i
w∗i p(y|ai, x) (5)
To sum up, we compute the final class probabilities p(y|x,D) given a new test input x by first prompting the LLM N times, with each of the available instruction prompts ai ∈ a to obtain conditional class probabilities p(y|ai, x). Second, we compute the weighted sum of the class probabilities using the weights w∗i ∈ w∗, learned through the optimization of Equation 4. BayesPE can also be interpreted as a Bayesian mixture of experts (Waterhouse et al. , 1995), where weights are obtained through approximate MAP inference (details in Appendix A. 4 and C. 4). Lower Bound on Model Error BayesPE provides a lower bound on the expected model error for the LLM, even when its weights are not known. Specifically, if prompts a ∼ q∗(a) are semantically
equivalent to each other, the following inequality is valid:
Eq∗(a)Ep(θ|DT )KL[p(y|x, θ, a)||p(y|x, θ∗, a)] ≥ Ep(θ|DT )Eq∗(a)KL[p(y|x, θ, a)||p(y|x, θ)]
(6)
A full derivation is presented in Appendix A. 5. In the notation above, we have added the weights θ of the pre-trained LLM which are not available in black-box settings. p(θ|DT ) is the posterior of these weights, given the data set that was used to train the LLM DT. We assume the pre-trained LLM weights to be a sample from this posterior θ ∼ p(θ|DT ). θ∗ are the optimal weights for the LLM, for which its reducible model error is null. Equation 6 states that the expected divergence between the output given any particular prompt p(y|x, θ, a) and the aggregate output p(y|x, θ), i. e. , the uncertainty with respect to the input prompts, is a lower bound to the expected divergence between the given LLM p(y|x, θ, a) and the LLM with optimal weights p(y|x, θ∗, a), i. e. , the expected reducible model error. This bound indicates that BayesPE approximates the reducible model uncertainty, as opposed to the data uncertainty. Bayesian neural networks provide an analogous bound on the expected model error (Appendix A. 6). 4 Experimental Setup
Data: We evaluate BayesPE on a total of ten tasks. Three sentiment analysis tasks: Amazon Reviews (He and McAuley, 2016), Imdb (Pal et al. , 2020) and SST-2 (Socher et al. , 2013). Three topic modeling tasks: DBPedia 14, Yahoo Answers (Zhang et al. , 2015) and TREC (Hovy et al. , 2001). Two semantic relation tasks: MRPC (Warstadt et al. , 2018) and SNLI (Bowman et al. , 2015). Finally, two spam detection tasks: SMS (Almeida et al. , 2011) and YouTube (Alberto et al. , 2015). These data sets cover a variety of common NLP tasks, number of classes and text input lengths (details in Appendix B. 1). Models: We evaluate the BayesPE framework with five different pre-trained and instruction finetuned LLMs, spanning families and sizes: Falcon7b-Instruct (Penedo et al. , 2023), Falcon-40bInstruct (Almazrouei et al. , 2023), MPT-7b-Instruct (Team, 2023b), MPT-30b-Instruct (Team, 2023a) and Mistral-7b-Instruct (Jiang et al. , 2023a). More details in Appendix B. 2. Baselines: We compare BayesPE to the following baselines:1
• Standard: The LLM is run once with a single prompt, drawn at random ai ∼ a. • Ensemble: Class probabilities from all instructions prompts ai are summed together (Wightman et al. , 2023; Jiang et al. , 2023b). • Best: The prompt ai with the highest validation accuracy is chosen. Uncertainty calibration metrics: We compute five metrics: negative log-likelihood (NLL), ROCAUC score (AUC), expected calibration error (ECE), maximum calibration error (MCE) and Brier score (Brier) (Gawlikowski et al. , 2023; Abdar et al. , 2021). 1In early experiments, we also tested confidence selfassessing, following Kadavath et al. (2022), but found the probabilities to be consistently less calibrated than the tokens probabilities of the class words themselves (Appendix B. 4). Prompts and Validation Data: We consider a fixed set of N initially provided prompts ai for each task. We do not aim to investigate prompt definition or engineering and, unless stated otherwise, we simply draw these at random from a pool of prompts generated with GPT-3. 5-turbo (Ye et al. , 2023b) with no manual editing. The prompts are used to compute Ensemble, Best and BayesPE, while Standard is computed by drawing just one prompt at random. For few-shot experiments, the prompts include five labeled examples from a different pool of validation data. These are drawn at random for each prompt in the set (details in Appendix B. 3). In each task, unless stated otherwise, we use N = 5 initial prompts, 50 validation examples to select the prompt for Best and train the BayesPE weights (details in Appendix B. 4), and 200 test examples to compute metrics. The set of prompts and validation examples are drawn at random ten times in each of our experiments to obtain error bars and compute statistical significance of results. 5 Results
Uncertainty Calibration We evaluate performance of BayesPE and baselines on all tasks with all LLMs in zero- and few-shot settings. Table 1 shows detailed results for three of the ten tested tasks using the overall highest performing configuration, i. e. , MPT-30b-Instruct with few-shot prompting. While we are primarily interested in uncertainty quantification, we also report macro F1-Score, to verify that predictive performance is not hindered by our approach. Full results are reported in Appendix C. 5. Figure 3 shows calibration curves (accuracy vs. coverage) for the same three experiments. In Table 1, we observe that BayesPE is competitive or better (within statistical significance) across all tasks and all metrics, with the exception of AUC score in Yahoo Answers, for which it is only 0. 01 lower than the best baseline. The calibration curves of figure 3 also show BayesPE clearly outperforming competing baselines. For example, on the Imdb data set, BayesPE achieves 97% accuracy at 80% coverage, while competing methods need to reduce coverage to 45 − 55% to achieve the same. We also see that, among competing baselines, the best approach varies with the task (Ensemble for Yahoo Answers and Best for Amazon Reviews and Imdb), meaning no single best baseline can be selected a-priori. Contrarily, BayesPE is always competitive, making it a far more reliable method. We further note that BayesPE often gives a substantial improvement in calibration (NLL and ECE) compared to standard prompting, e. g. , NLL of 3. 2 vs. 4. 0 and ECE of 0. 27 vs. 0. 46 for the Yahoo Answers data set. Our approach is also competitive or best across the three tasks also in F1-score, meaning that it maintains or improves predictive performance as well. The trends described above are observed across all experiments. While inspecting all results in detail is difficult (included in Appendix C. 5), we present aggregate results, both in terms of average differences in calibration and statistical significance. Table 2 reports the average difference in AUC score, ECE and MCE each method provided across all 100 experiments (ten tasks × and five LLMs in × zero- and few-shot settings), compared to standard prompting2. BayesPE provides substan-
2NLL and Brier score are not comparable across experiments, as their values depend on the number of classes and is not bounded. Therefore, differences are meaningful only
tial improvement in calibration, with appreciably larger reductions in errors than Ensemble and Best, e. g. , −32% in ECE vs. −20-26%. Table 3 reports in how many of our 100 experiments each method was found to be the best, or within statistical significance of the best (twosample p> 0. 05 across ten repeats of each experiment). The number in parenthesis reports in how many experiments each method was significantly better than any other (two-sample p< 0. 05 with second best). Overall, BayesPE is significantly better or competitive for every metric in the majority of experiments and in more experiments than any competing baseline. In particular, for metrics tied to expected calibration performance (NLL, ECE and Brier) it is substantially more consistent than the baselines, giving best or competitive NLL and Brier score in 96% of the experiments and ECE in 83% of the experiments. In comparison, other baselines were found competitive for these metrics in only 30− 65% of the experiments. Inference Efficiency The driving factor for the run-time cost of prompting an LLM is the number of forward passes needed to obtain class probabilities (inference). With our approach, after finding the optimal weights w∗ for a set of N ensembles through a validation set, we can fix a forward passes budget at run-time L <= N. We apply a simple greedy approach to select the run-time prompts and their weights by simply taking the L prompts with the highest weights w∗i to construct the weighted ensemble. Using three data sets and Mistral-7bInstruct as the base LLM in few-shot operation, we evaluate NLL with respect to the number of run-time forward passes. We use N = 20 initial prompts to compute the BayesPE weights and the best prompt (Best) baseline with a validation set and then test with a varying budget of L = 1 : 10 prompts at run-time. Experimental details are given in Appendix B. 5. Figure 4 shows the results (additional results in supplementary C. 2). Ensemble and BayesPE display a common trend of decreasing NLL as the forward passes budget is increased. This is expected, as in both cases different prompts are increasingly added to the ensemble at increasing LLM forward passes, which results in increasingly better calibration of the final class probabilities. BayesPE is consistently better than or competitive with Ensemble for all experimental settings and all inference budgets. Even for
within the same experiment and not aggregated. 0. 17 0. 18 0. 19 0. 20 0. 21
0. 28
0. 32
0. 36
0. 40
0. 44
2. 1 2. 2 2. 3 2. 4 2. 5 2. 6
Amazon Reviews DBPedia 14
Standard
Ensemble
Best
BayesPE
Number of LLM Forward Passes 2 4 6 108
N LL
(↓ )
Number of LLM Forward Passes 2 4 6 108
Number of LLM Forward Passes 2 4 6 108
Yahoo Answers
Figure 4: Negative log likelihood (NLL) vs. number of forward passes through the LLM at run-time. Using any prompt (Standard) and running the best performing prompt on validation (Best) can be performed only with a single prompt and hence a single forward pass. Standard ensembling (Ensemble) and our BayesPE approach exploit more prompts as the forward passes budget is increased. 0. 275
0. 325
0. 375
0. 425
2. 1
2. 3
2. 5
2. 7 DBPedia 14 Yahoo AnswersAmazon Reviews
0. 17
0. 19
0. 21
0. 23
N LL
(↓ )
Number of Validation Examples 0 20 40 8060
Number of Validation Examples 0 20 40 8060
Number of Validation Examples 0 20 40 8060
Standard
Ensemble
Best
BayesPE
Figure 5: Negative log likelihood (NLL) vs. number of labeled validation examples. Using any prompt (Standard) and aggregating all prompts equally (Ensemble) does not require any validation set and are shown as single points at zero validation examples. Choosing the highest accuracy prompt on validation (Best) and BayesPE benefit from increased number of validation examples. tasks such as Amazon Reviews in few-shot, where performance with a single prompt is comparable across methods (NLL∼ 0. 189), as the budget is increased, BayesPE results in a substantially better improvement in calibration (NLL ∼ 0. 169 vs. 178 for Ensembles). This means that BayesPE is a more efficient and effective method of obtaining well-calibrated class probabilities with the LLM at any run-time costs constrain. Note that, at a budget of L = 1, BayesPE is equivalent to picking the best performing prompt on validation (Best). As the budget is increased, BayesPE consistently improves performance (lower NLL). This means that, compared to picking the best prompt on validation, BayesPE consistently utilizes additional computational spend to improve calibration performance, e. g. NLL ∼ 0. 169 on the Amazon Reviews task for BayesPE with a budget of 5 forward passes, compared to an NLL of 1. 89 for picking the best prompt on validation. Labeled Data Efficiency The second substantial cost factor in prompting LLMs with BayesPE is the number of labeled examples needed in the validation set. We study NLL as a function of the number of validation examples used to estimate LLM uncertainty. As in the experiments above, we use three data sets and Mistral-7b-Instruct as the base LLM in few-shot operation. Experimental de-
tails are given in Appendix B. 5. Figure 5 shows the results (additional results in supplementary C. 2). As expected, we observe that with no validation examples, BayesPE is equivalent to Ensemble, as the likelihood term in Equation 4 is null. BayesPE is competitive or better than any baselines for any budget of validation labeled data, e. g. , for DBPedia 14 in Zero-Shot we report an NLL of 0. 28 for BayesPE with 50 validation examples vs. 0. 31 for for both Best and Ensemble. With the Amazon Reviews task, adding five validation examples initially causes to over-fit and degrade performance. However, as the validation data set increases to tens of examples, this over-fitting is mitigated and the BayesPE NLL matches or improves on that of the Ensemble baseline. BayesPE is also consistently better than the Best baseline in all experiments at all values of labeled data budget, e. g. , NLL of 2. 13 vs. 2. 3 with 20 validation examples and 2. 1 vs. 2. 2 with 90 validation examples for Yahoo Answers. This means that BayesPE provides consistently better uncertainty quantification, independently of the available amount of validation examples