Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Authors: Jiefeng Chen
Jinsung Yoon
Sayna Ebrahimi
Sercan O Arik
Tomas Pfister
Somesh Jha
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
selective prediction
adaptation with self-evaluation
TL;DR: Propose a novel framework for adaptation with self-evaluation to improve selective prediction performance in large language models.

Abstract: Large language models (LLMs) have recently shown great advances in a variety of tasks, including natural language understanding and generation. However, their use in high-stakes decision-making scenarios is still limited due to the potential for errors. *Selective prediction* is a technique that can be used to improve the reliability of the LLMs by allowing them to abstain from making predictions when they are unsure of the answer. In this work, we propose a novel framework for adaptation with self-evaluation to improve the selective prediction performance of LLMs. Our framework is based on the idea of using parameter-efficient tuning to adapt the LLM to the specific task at hand while improving its ability to perform self-evaluation. We evaluate our method on a variety of question-answering (QA) datasets and show that it outperforms state-of-the-art selective prediction methods. For example, on the CoQA benchmark, our method improves the AUACC from 91.23\% to 92.63\% and improves the AUROC from 74.61\% to 80.25\%.

Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Jiefeng Chen Jinsung Yoon
Sayna Ebrahimi
Sercan Ö. Arık² Tomas Pfister
Somesh Jha!.2
University of Wisconsin-Maddion (jiefeng, jhajecs wis. saynan,,,y,n,,nn,,ee.eeeeaa saynae, soarik, spfisterr@google.cco
2 Google LLC
(jinsungyoon,
Abstract
Without Selective Prediction
Large language models (LLMs) have recently shown great advances a variety of tasks, including natural language understanding and generation. However, their use in high-stakes decision masiing scenarios is still limited due the potential for errors. Selective prediction is a technique that can be used to improve the reliability of the L.MSs by allowing them to ah- stain from making predictions when they unsure of the answer. In this work, we pro- pose novel framework for adaptation with self-evaluation to improve the selective predic- tion performance of LLMs. Our framework is based the idea of using parfmeter-efffcient tuning to adapt the LLM to the specific task at hand while improving its ability to perform self-evaluation. We evaluate method a variety of question-answering (QA) datasets and show that it outperforms tate-of-theeart selective prediction methods. For example, the CoQA benchmark, our method improves the AUACC from 91.23% 92.63% and im- proves the AUROC from 74.61 % to 80.25%.
Vitanin
LLM
Selective Prediction
Which oham su slodd dottrg?
ArpACt Viamnn Selecion sEGTE
Correct Vitamin
LLM
safety-critical question from TriviaQA dataset: "Which vitamin helps regulate blood clotting?" The OPT-2.7B model incorrectly answers "Vitamin C". when the correct answer "Vitamin K". Without selective prediction. LLMs will directly output the wrong answer which in this case could lead users to take the wrong medicine, and thus causing potential harm, With selective prediction, L.L.s will output low selection score along with the wrong answer and can further output "I don't know!" !" to warn users not to trust it or verify using other sources.
Figure 1:
information be used direct human oversight the queries with the lowest selection score. Se- lective prediction (Geifiman and El- El-vaniv, 2017), broadly refers to the deployment scenario for Al models where humans are involved to maintain overall accuracy by reviewing Al-generated, low- confidence outputs. In this scenario, both human and Al performance are considered together to min- imize human involvement cost. LLMs should be used in the real-worl with enhanced selective pre- diction performance. They should be able to assess the accuracy of their predictions and refrain from making wrong predictions. an LLM detects that an answer might be wrong for a question, should be able to generate an answer with the sentiment of "I don't know!" (as shown in Fig. 1) or defer the answer to human for manual inspection. This will help to ensure that L.MMs are used in reliably, especially for high-stakes applications.
1 Introduction
Large Language Models (LLMs) have recently demonstrated impressive capabilities in many natu- ral language understanding, reasoning and genera- tion tasks, such question answering (Jiang et al., 2021; Singhal et al., 2023), summarization (Tang al., 2023; Zhang et al., 2023b), semantic classifi- cation, and code generation (Poesia al., 2022; Zhang al., 2023a). As LLMs improve their remarkable performance, they are being increas- ingly considered to replace humans to perform high-stakes tasks. For example, LLMS can be used for medical QA to assist patients (Singhal et al., 2022). However, LLMs are not guaranteed to be accurate for all queries, it is important to un- derstand which queries they are reliable for. This
Selective prediction for LLMs is challenging be- cause L.Mss are just trained to predict the next to-
Work done during internship at Google and hefore
joining Amazon.

---

ken given a context but are not guaranteed always predict the correct next token. Also, since LLMs generate an output sequence in an auto-regressive way, they don't it directly produce a confidence score for the output sequence. Thus, obtaining selection scores from LLMs for their oulput sequences is not straightforward. Although there is some research on selective prediction for LLMs, these studies have their own shortcomings. Kadavath et al. pro- pose to use heuristic prompts (e.g., adding prompts like "Is the proposed answer True or False?") to trigger self-evaluation of LLMs. However, those prompts may only work for the LLM used in Kada- vath et al. (2022) and may not generalize to other types of LLMs (e.g., OPT and GPT2 models eval- uated in our work). Some approaches proposed using semantic entropy (Kuhn et al., 2023) or self- consistency (Wang et al., 2022) as measure of uncertainty for selection score. However, they usu- ally require generating multiple output sequences to obtain the uncertainty measure for an input se- quence, which introduces high computational cost and latency at test time. Fine-tuning LLMs on training data from the target question answering task using the standard LLM training loss can im- prove selective prediction performance. This is because fine-tuning can improve the accuracy of the predictions and maximize the likelihood of the ground answer for a given question. However, maximizing the likelihood of the ground an- swer is not the same as minimizing the likelihood of the wrong answers, since LLMs generate output sequences in an auto-regressive way. Even after fine-tuning, some wrong answers may still have high likelihood and be generated by the LLM at test time. Therefore, distinguishing correct and in- correct answers based on likelihood scores alone a challenging task.
self-eval score (see Eq. (11)) to make selective pre- dictions. This makes our method much less com- putationally expensive than solutions that require generating multiple output sequences to obtain the uncertainty measure. Thus, the proposed method is useful for practical applications where high se- lective prediction performance and low inference costs are desired, after deploying the LLM. In such applications, practitioners prefer collecting some training data fine-tune smaller LLMs to achieve high selective prediction performance rather than directly deploying very large pre-trained LLMs with limited selective prediction performance for specific tasks.
We conduct extensive experiments evaluate our proposed framework, ASPIRE. We show that ASPIRE achieves the state-of-the-art selective pre- diction performance on three question answering datasets: CoQA, TriviaQA and SQuAD, using OPT and GPT-2 models. We also provide empirical anal- ysis to delve deeper into our proposed technique.
2 Related Work
Selective Prediction for LLMs. Recently, LLMs (e.g., GPT-4 (OpenAI, 2023) and PaLM (Chowd- hery et al., 2022)) have achieved great success in solving various kinds of Natural Language Gener- ation (NLG) tasks. However, LLMs are still not very reliable and may generate wrong outputs when solving NLG tasks. Due this, selective predic- tion (or sometimes called selective generation (Ren et al., 2022)) is critical for safely deploying LLLMs in the real-world. Diffcrent from selective predic- tion for classification tasks (e.g.. Natural Language Inference (NLJ) tasks) (Xin et al., 2021), selec- tive prediction for LLMS in solving NL. tasks is fundamentally different since the prediction is done regressggyyy over many steps and the possible answer set has an infinite size. Recently, several work propose some uncertainty measures for LLMs, which can be used for selective predic- tion (Si et al., 2022; Kadavath et al., 2022; Varshney et al., 2022; Ren et al., 2022; Kuhn et al., 2023). Some recent work studies selective prediction for solving question answering tasks where questions ambiguous (Cole et al., 2023; Yin et al., 2023). Varshney and Baral (2023) propose selective pre- diction method that inference time leverages an auxiliary model which is trained to distinguish the correct predictions of the QA model from the in- correct ones. Different from previous work, our
To address these challenges of self-evaluation and uncertainty estimation, we propose a novel framework - Adaptation with Self- Seff-Evaluttion to Improve Selective Prediction in L.Ms (ASPIRE)). Unlike previous methods that rely on hand-crafted heuristics or multiple output sequences, our frame- work learns to self-evaluate from target-task data. We do this by training LLMs a subset of the training data from the question-answering tasks. This allows the LLMs to learn distinguish be- tween correct and incorrect answers on their own. We then define a selection score that combines the likelihood of the generated answer with the learned

---

work proposes improve selective prediction per- formance of LLMs in solving question answering tasks by learning self-evaluation during fine-tuning. Parameter Efficient Fine-tuning. Fine-tuning pretrained L.MSs on downstream datasets can bring huge performance gains when compared to us- ing the pretrained LLMs out-of-the-box (e.g.. k- shot inference). However, as LLMs get larger and larger, full fine-tuning becomes very expensive in terms of computational cost and memory require- ments. In addition, massive models might not be data efficient and overfitting issues might be ob- served, yielding suboptimal generalization. To ad- dress these issues, Parameter-Lfffcieen Fine-tuning (PEFT) approaches have been proposed. PEFT ap- proaches only fine-tune small number of (extra) model parameters while freezing most parameters of the pretrained LLMs, thereby greatly decreas- ing the computational and storage costs. has also been shown that PEFT approaches are better than tinent in the low-data regimes and gen- eralize better to out-of-domain scenarios. Existing PEFT approaches include LoRA (Hu et al., 2021), Prefix Tuning (Liu et al., 2021a), Soft Prompt Tun- ing (Lester et al., 2021) and P-Tuning (Liu al., 2021b). In this work, we use Soft Prompt Tun- ing to learn self-evaluation to improve selective prediction performance of LLMs.
We use f generate the output sequence for the given input x by solving the following objective:
y = argmax log f(y x)
(4)
y
It is impossible to solve this objective exactly since the output sequences can be arbitrarily long. How- ever, we can employ some decoding strategy like greedy decoding or beam search to solve it.
To evaluate if the generated output ý is correct or not, we need a set of reference outputs $ and an evaluation metric M : V x V - [0, 1] that can evaluate the similarity of the generated output y compared to the reference output y, E S. With a threshold 7, we can determine the correctness of the generated output if maxy, (M y+) > % then the generated output is correct; otherwise, the generated output wrong. We discuss the specific choices of M and 7 in Section 6.
In selective prediction, we need a rejection op- tion, which is denoted by P. Given training dataset D = {(x', )²)) randomly sampled from target task distribution, we aim to build selective predictor fs V + V U {1} that achieve strong selective prediction performance on Rie the test dataset Dee = ((x', ) where St is the set of reference outputs for the input x². The selective predictor fa is composed of a predictor f V + V and a selection scoring function V* - R. With f and g, the selective predictor [s is proposed as:
3 Problem Setup
Suppose we have a pre-trained LLM f for an ar- bitrary generative modeling task such as question answering. The output can be represented as se- quence of tokens from the vocabulary V. Let V be the space of sequences of tokens. Suppose the logits of f on E V given x E V" is f(u x). The likelihood of the next token following x being is defined as:
fx(x;))= f((x)
if g 2 T, g(x) <T
(5)
where T is a threshold. The accuracy of the se- lective predictor is defined as the fraction of the accepted inputs where the predictions are correct. The coverage of the selective predictor is defined the fraction of the inputs that are accepted. We can tune the threshold and there would be an accuracy-covenge trade-off. (I)
cxp(f(v x))
f(v x) = Evey exp (f(v') x))
achieve certain coverage
whereas the likelihood of generating V* given x is defined as:
We use the area under the accuracy-coverage curve (AUACC) metric to measure selective pre- diction performance and use the area under the receiver operator characteristic curve (AUROC) metric to measure the quality of the selection score estimation. AUACC is the common metrie used for evaluating selective prediction performance (Xin et al., 2021; Yoshikawa and Okazaki, 2023). AU- ROC is equivalent to the probability that a ran- domly chosen correct output sequence has a higher
f( x) = 1T/5 f f(i x. 01 11),
(2)
where y = (I.,.... :) |5| is the length of y. Ú 1] (J1,..., 1) for i > 0 and Y0gj = Ø.
This likelihood can be very small when [y| is very large. To address this issue, we define the normal- ized likelihood
fnormly x) = =f( x)

---

selection score than randomly chosen incorrect oulput sequence. AUROC used in (Kuhn al.. 2023) for evaluating uncertainty estimation meth- ods.
output sequences:
Apf.8p.3
(7)
where & is the number of output sequenees gen- erated. We aim to generate output sequences that have high likelihood (i.e.. ʃ( x; x2) is high). We use the metric M defined in Section 3 to deter- mine the generated output y is correct or not. If My', yy) > ÿ. we label y as a correct output for x; otherwise, we label 5 as wrong output for x. Here, the threshold ý might be different from the threshold 7 used for evaluation. We choose a sufficiently large value of / (e.g., 9 0.9) that the generated wrong outputs wouldn't 't be labeled correct outputs. In Appendix H, we provide more details and analyses on selection of 3.
4 ASPIRE Framework
We propose that LLMs should have the self- evaluation ability such that they should be able distinguish whether their proposed answers for a given question are correct or not. Although some previous work (Kadavath et al., 2022) show that LLMs have good self-evaluation ability with specially designed prompts, those prompts may not transfer to different kinds of LLMs (as shown by our experiments and in Kuhn al. (2023)) and hand-crafting prompts for different kinds of LLMs can be expensive. more effective approach is to collect some training data to employ self- evaluation. Towards this end, we propose novel framework- Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs (ASPIRE).
After sampling high-like high-likelihood outputs for each query, we add adaptable parameters 0, and only tune 0g for learning self-evaluation. Since the out- put sequence generation only depends on & and fp- freezing 0 and the learned Op can avoid changing the prediction behaviors of the LLM when learning self-evaluation. Let be and zn be pair of tokens that represent the words "correct" and "wrong" re- spectively. We can then optimize 0g using the fol- lowing training objective:
Fig. 2 illustrates the proposed framework and the details are explained next.
Given a training dataset for a generative task, we can fine-tune the pre-trained LLM on the train- ing data to improve its prediction performance. Towards this end, parameter efficient tuning tech- niques (e.g., soft prompt tuning (Lester et al., 2021) and LoRA (Hu et al., 2021)) might be employed to adapt the pre-trained LLM the task. given their effectiveness in obtaining strong generaliza- tion with small amount of target task data. Specif- ically, the model parameters 9 of the LLM are frozen and adaptable parameters Op are added for fine-tuning. Only 6g are updated to solve the fol- lowing training objective:
min Exxy) Le + Lrv
0.
f. Rg-SS(xx)) log f f2dl 5; pp. 0s)
Lu = Bg-Su(Xx) log f(zw 5; 0p, 02)
where yc(x,y) is set of correct outputs con- taining the reference output y and k correct out- puts with highest likelihood from A(f, 0p. x). and (x, y) a set of wrong outputs containing by wrong outputs with highest likelihood from A(f, x), ). If A(f, 0y. x) has less than ki correct outputs (or has less than kw wrong outputs), we include all its correct outputs (or all its wrong out- puts) in S. (or Sw). We ensure that Sw contains at least one wrong output. If A(f. 0y. x) doesn' 't con- tain wrong outputs, we add default wrong output (e.g., the empty string) to Sw.
min ((x 8y
(6)
where L is the LLM training loss (e.g. cross- entropy). Such fine-tuning can improve selective prediction performance because it not only im- proves the prediction accuracy, but also enhances the likelihood of correct output sequences.
To further improve selective prediction perfor- mance, propose to fine-tune the LLM to learn self-evaluation. We first use the LLM with the learned O, to generate different answers for each example yx E Dtr, Suppose the decoding algo- rithm used to generate output sequences for each input x is A. would produce a list of generated
After training Op and 0g. we obtain the prediction for the query x via solving the following objective:
y = argmax log f(ý xx; Op))).
We use the beam search decoding method towards this. We define the likelihood of the output y*

---

Vitamin K
Vitamin
Vitamin
Correct
Wrong
Frozen LLM
Frozen LLM
Frozen LLM
Frozen 6p
Adaptable parameters Op
Frozen 8p
Adaptable parameters
Q: Which vitamin assists clotting? A:
a: Which vitamin assists blood clotting? A:
Q: Which vitamin blood
Q: Which vitamin
assists blood
clotting? Vitamin K.
clotting? Vitamin
Task specific tuning
Answer sampling
Self-evaluation learning
Figure 2:
In the proposed framework ASPIRE, we first perform task specific tuning train adaptable parameters 0, while freezing the LLM. Then we use the LLM with the learned e, generate different answers for each training question create dataset for self-evaluation learning. Finally, we train the adaptable parameters 8, to learn self-evaluation using the created dataset while freezing the I.LM and the learned pg.
being correct for the query as:
where / is the length of the prompt. The prompt is then concatenated to the embedded input forming a single matrix [Ø: X] E [[(+++))xde which then flows through the transformer as normal.
P(zel x, x
exp (f(ze x, ; Bp. 02))
(10)
Exe(reit)) exp (f(z x, 8 0p, 0.))
In the proposed framework, we need to train two portions of the prompts O2 E I/ and fg E ! Utilizing soft prompt tuning, the training objective (6) is proposed as:
This score P( x, y*) is referred as the learned self-eval score. Overall, the selection scoring func- tion is proposed as:
y!
g(x) = (1 - a) log fuomn (ý" x; 0p) + a log P(ze x)
(11)
min E (x,) 0p
log (uj](pp X; Yi n]),
y
(12)
where a E [0, 1] is hyper-parametes
where X is the embedding of x and Y is the embedding of Dlj-1]- On the other hand, the train- ing objective (8) is proposed as:
5 Implementation via Soft Prompt Tuning
In the proposed framework, Op and 0, can be trained using parameter efficient tuning approaches. In our work, we focus on Soft Prompt Tuning, as il- lustrated in Fig. 3. The driving force behind this ap- proach lies in the recognition that if can develop prompts that effectively stimulate self-evaluation, it should be possible to discover these prompts through soft prompt tuning in conjunction with tar- geted training objectives.
min Exx,y) 0.
L. + Lrv.
C. = Bg~88(Xx)) log f(zzl X; Y;
Ew Bg - log fz mmll 10p X; 0.])
(13)
where Y is the embedding of y. The inference objective (9) in the framework becomes:
We first briefly introduce the soft prompt tun- ing method proposed by Lester et al. (2021). We consider LLMs based on the Transformer archi- tecture (Vaswani et al., 2017). Given query x ( Transformers first embed the tokens, forming a matrix X xde, where de is the dimension of the embedding space. The soft- prompts are represented as parameters e gixde
yd argmax log f [pp X))
(14)
The learned self-eval score P(z. x, y*) becomes:
P(ae x. x. =
exp (f(zo 10 x;;x;;,0)))
(15)
Ese(ecieu) exp (f(z) 10p: X; 0=)

---

Vitamin K
Correct Likelihood: 0.4
Likelihood: 0.7
Frozen LLM
Frozen LLM
Cache states
Q Qombod
Bp
Q Qembed
Aembed
0g
[soft prompt] Q: Which vitamin assists in blood clotting? A:
[soft prompt] Which vitamin assists blood clotting? A: Vitamin K. [soft prompt]
Figure 3: Implementation of the proposed framework via soft prompt tuning. O,, and 0, are learnable soft prompt embeddings. Q Qembed and Aembed input embeddings for the question and answer respectively. We first generate the answer and the likelihood of the answer, and then compute the learned self-eval score. We can cache the states when generating the answer and reuse those states when computing the learned self-eval score to save computational costs.
where Y is the embedding of y*
across different datasets and LLMs (see Table 1). (Q2) What kinds of decoding algorithms could be used as A for the proposed framework ASPIRE? (A2) Using decoding algorithms that can sample different high-likelihood answers A (e.g.. beam search) is important for ASPIRE to achieve good selective prediction performance (see Table 4). (Q3) What is the effect of the number of training samples for the proposed method ASPIRE?
To generate the output sequence and obtain the selection score a given input sequence, we em- ploy two stages: first, we obtain the generated out- put and the likelihood for the generated output and then, we obtain the learned self-eval score. Since the query of the second stage is constructed by ap- pending some additional tokens to the query of the first stage, the second stage reuse the states in the first stage instead of recomputing them save some computational cost (see Fig. 3).
(A3) More training samples lead to enhanced per- formance and with ~2k samples, ASPIRE can out- perform the baselines without soft prompt tuning significantly on different datasets (see Table 5).
Lastly, we note that the computational complex- ity of the proposed method at test time is (Imea) with lmar being the maximum length of the gen- erated output sequence. In Appendix F, we pro- vide more detailed analysis of the computational complexity of different methods. The predictive entropy and semantic entropy methods have com- plexity of O(m Imaz) where m the number of output sequences sampled for uncertainty estima- tion, which much larger than that of our method.
6.1 Setup
Dataset. We focus on the free-form question answering tasks on the datasets CoQA (Reddy et al., 2019), TriviaQA (Joshi et al., 2017) and SQuAD (Rajpurkar al., 2016). For CoQA and SQuAD, since each question is asked based context paragraph, we evaluate the LLMs in the zero-shot setting. For TriviaQA, since the LLMMs have limited accuracy under the zero-shot setting, we evaluate the LLMs in 5-shot setting. For each dataset, we use a subset of the original training set containing 50K examples for adapting LLMs by default. The details of the datasets are given in Appendix B.
6 Experiments
Our experimental evaluation is focused on the fol- lowing questions:
(Q1) Could a learning-based system using self- evaluation improve selective prediction in LLMs compared to other post-hoc selective prediction alternative??
LLMs. We use OPT (Zhang et al., 2022) and GPT-2 (Radford et al., 2019) models various sizes. For OPT, we consider OPT-350M, OPT-1.3B.
(A1) By learning self-evaluation, we can signifi- cantly improve selective prediction performance

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC 1
AUROC
AUACC
AUROCT
AUACC
AUROC 1
Perplexity
55.93
62.05
22.60
72.88
7.68
$1.90
Predictive Entropy
60.76
67.53
24.83
76,20
10.04
57.21
Pre-trained
Semantic Entropy
63.03
70.50
24.37
75.33
10.38
59.17
GPT2-XL
Self-eval
46.67
50.83
9.30
42.75
7.32
49.56
P(Truc)
46.98
51.17
10.62
44.54
10.69
60.87
Perplexity
83.27
72.79
36.49
79.92
88.73
75.08
Predictive Entropy
83.49
73.44
37.31
82.21
88.25
74.16
Adapted GPT2-XL
Semantic Entropy
84.40
75.16
36.68
81.40
88.62
75.26
Self-eval
69.91
51.90
14.39
43.33
74.26
49.13
with 8,
P(True)
70.63
52.83
13.59
40.59
74.34
49.09
ASPIRE (ours)
85.65
78.32
38.06
83.23
89.86
78.35
Perplexity
75.26
70.16
40.93
78.86
40.82
57.20
Predictive Entropy
75.29
69.16
41.20
78.92
47.18
62.85
Pre-trained OPT-2.7B
Semantic Entropy
76.31
70.96
40.72
78.06
51.53
68.40
Self-eval
62.32
52.26
25.88
59.04
41.78
59.05
P(True)
62.16
51.80
24.88
56.89
34.77
49.42
Pre-trained OPT-30B
Self-eval
71.99
51.10
36.92
48,90
46.24
57.26
P(True)
71.59
51.31
36,20
45.63
43.93
54.26
Perplexity
90.80
74.23
$3.56
81.74
92.86
75.72
Predictive Entropy
90.63
72.87
53.91
82.19
92.96
75.58
Adapted OPT-2.7B with 9p
Semantic Entropy
91.23
74.61
53.58
81.55
93.21
76.53
Self-eval
81.30
50.76
32.98
56.03
86.34
56.99
P(True)
81.14
51.01
33.48
56.27
82.59
49.48
ASPIRE (ours)
92.63
80.25
55.06
84.44
94.73
82.60
Table I: Results of evaluating different methods to compute the selection score when the model's predictions are fixed. All numbers percentages. Bold numbers superior results.
OPT-2.7B and OPT-30B. For GPT-2, we consider GPT2-Medium, GPr2-Large and GPT2-XL. The details of these models are given in Appendix C. Baselines. For selective prediction, we need to get predicted output sequence y* and a selection score g(x) for each input sequence given model f. The model f can be pre-trained LLM or an adapted LLM with Bp trained using the training objective (12). We use the beam-search decoding obtain the predicted output sequence y* and consider the following baselines to compute the selection score gxx): (1) Perplexity; (2) Predictive Entropy; (3) Semantic Entropy (Kuhn et al., 2023); (4) Self-eval; (5) P(True) (Kadavath et al., 2022). More details can be found in Appendix D.
that different from the reference answers (refer to Appendix G for the justifications of the choices of 7). Unless specified, we use 0.7 as default.
Training hyper-parameters. We have two stages of training: the first stage is to train the soft prompt Op using the training objective (12) and the sec- ond stage is to train the soft prompt 0g using the training objective (13). For both stages, we train the soft prompts for 10 epochs using AdamW opti- mizer with a batch size of 8, a learning rate of 0.01 and cosine learning rate scheduling. More training details can be found in Appendix E.
Evaluation metrics. We use the Rouge-L (Lin and Och, 2004) as the evaluation metric M to evalu- ate the similarity of the generated answer to the ref- erence answers following Kuhn et al. (2023). For the threshold that is used determine the correct- ness of the generated answer, we consider relatively larger values of y since we focus on safety-critical applications where accepting a wrong answer is more costly compared to rejecting a correct answer
ASPIRE setup. We use the beam search as the decoding algorithm A. We set the number of beams equal to & and use the & highest scoring beams the answer list A(f, 0p, x). We set / 50, 7 0.9, 10, k 2, k 10 and 0.25 by default. We choose these performance on the validation set from TriviaQA using the OPT-2.7B model. We then use the same hyper-parameters across all datasets and models.
based on the

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC 1
AUROC
AUACC 1
AUROC 1
AUACC
AUROC f
ASPIRE (a= 0.0)
90.80
74.23
53.56
81.74
92.86
75.72
Adapted OPT-2.7B with 0p
ASPIRE (a 0.255)
92.63
80.25
55.06
84.44
94.73
82.60
ASPIRE (a 0.5)
92.56
80.18
54.61
84.33
94.59
82.16
ASPIRE (a= 0.75)
92.05
78.37
52.71
81.52
94.28
80.98
ASPIRE (a 1.0)
91.33
76.08
48.84
76.39
93.77
79.48
Table 2: Results of studying the effect of the hyper-parameter in the proposed selection score (Eq. (II).. All numbers are percentages. Bold numbers are superior results.
CoQA
TriviaQA
SQuAD
OPT-2.7B model. These demonstrate that the self- evaluation approaches are not effective for high ca-
Model
Acc
f
Acc
Pre-trained GPT2-XL
46.27
11.80
7.41
pacity LLMs, and applying the proposed ASPIRE smaller LLMs lead to better selective predic- tion performance compared those self-evaluation approaches with much larger LLMs. Additional re- sults Appendix show that ASPIRE significantly outperforms the baselines across OPT and GPT2 models of different sizes for different values of the Rouge threshold
Adapted GPT2-XL with Op
69.18
17.45
75.44
Pre-trained OPT-2.7B
60.68
21.38
35.95
Pre-trained OPT-30B
71.06
39.36
41.41
Adapted OPT-2.7B with 0p
80.45
29.21
83.27
Table 3: Results of evaluating the accuracy of different LLMs. All numbers are percentages.
6.2 Results
6.3 Empirical Analyses
We first evaluate the accuracy of different LLMs. The results in Table 3 show that after training Op via soft prompt tuming, the accuracy of LLMs is improved significantly. On the CoQA and SQuAD datasets, the adapted OPT-2.7B can even outper- form the pre-trained OPT-30B, which demonstrates that it is possible to adapt smaller LLM to achieve better accuracy than much larger LLM. We then evaluate different methods compute the selection score when the model's predictions are fixed. The results in Table show that the proposed method ASPIRE significantly outperforms the baselines in terms of the AUACC and AUROC metrics across different datasets and LLMs. The results also show that after prompt tuning. the AUACC of different methods is significantly improved the accuracy gets better and the perplexity becomes more mean- ingful in separating correct and wrong answers. Additionally, the results show that the proposed ASPIRE with the adapted OPT-2.7B model can significantly outperform the Self-eval and P(True) baselines with the pre- re-ttrined OPT-30B model in selective prediction performance. Note that on the TriviaQA dataset, although the pre-trained OPT- 30B model has better accuracy than the adapted OPT-2.7B model, the Self-eval and P(True) base- lines with the pre-trained OPT-30B model have much worse selective prediction performance com- pared to the proposed ASPIRE with the adapted
The effect of a. We study the effect of the hyper-parameter a in the proposed selection score (Eq. (11)). The results in Table 2 show that setting a = 0.25 leads to the best performance since it combines the normalized likelihood and the learned self-eval score in a good way. Only using the nor- malized likelihood (i.c., a = 0) only using the learned self-eval score (i.e., worse performance. In practice, the value of be chosen based the performance on the valida- tion data. In Appendix we give results for other models and discuss how we choose a.
1) leads to much
The choices of A. We compare two decoding algorithms beam search and multinomial sam- pling that can be used as A for answer sampling. For beam search, beams as the answer list. For multinomial sam- pling, we consider temperature (denoted as T) in the set {0.1, 1.0, 2.0}. The results Table show that using multinomial sampling with 2.0 or T = 0.1 leads to worse performance compared to other decoding algorithms. If we set a high temperature (T = 2.0) for multinomial sampling, then sample some random answers that might not have high-likelihood. If we set low temper- ature (T 0.1) for multinomial sampling, then we repeatedly sample the same ligelliiel an- swers. Thus, the results suggest that sampling dif- ferent high-likelihood answers is important for our
the k highest scoring

---

CoQA
TriviaQA
SQuAD
Model
Decoding Algorithm
AUACC t
AUROCT
A
AUROC
1
AUROC 1
Multinomial (T=0.1)
83.82
74.22
36.40
80.67
89.75
77.56
Adapted GPT2-XI.
Multinomial (T=1.0)
84.96
76.15
37.03
81.41
90.12
78.71
Multinomial (T=2.0)
83.06
72.96
36.34
80.14
89.41
76.98
with Op
Beam search
85.65
78.32
38.06
83.23
89.86
78.35
Multinomial (T=0.1)
92.04
77.96
55.09
84,28
94.24
80.52
Adapted
Multinomial (T=1.0)
92.60
79.86
55.15
84.29
94.57
82.08
OPT-2.7B with Pp
Multinomial (T=2.0)
92.02
77.91
53.80
82.40
94.15
80.42
Beam search
92.63
80.25
55.06
84.44
94.73
82.60
Table 4: Results of comparing different decoding algorithms for answer sampling in the proposed method. We denote the temperature as T. All numbers are percentages. Bold numbers are superior results.
CoQA
TriviaQA
SQuAD
Model
Method
AUACC1
AUROCT
AUACC1
AUROC +
AUACCT
AUROC T
Pre-trained
Predictive Entropy
75.29
69.16
41.20
78.92
47.18
62.85
OPT-2.7B
Semantic Entropy
76.31
70.96
40.72
78.06
51.53
68.40
ASPIRE (size=1k)
80.87
67.01
45.70
78.98
85.42
71.42
Adapted
ASPIRE (size=2k)
85.71
73.72
46.64
79.24
88.27
75.74
OPT-2.7B with Dp
ASPIRE (size=5k)
87.83
74.58
49.77
82.06
90.09
77.09
ASPIRE (size=10Kg
90.46
78.29
51.88
83.13
92.48
79.46
ASPIRE (size=50k)
92.63
80.25
55.06
84.44
94.73
82.60
Table Results of studying the effect of training set size for the proposed ASPIRE. All numbers are percentages.
method to achieve high accuracy and coverage in selective prediction. The results also show that us- ing beam search leads to similar performance as using multinomial sampling with T 1. So can either in practice.
applying our method to larger LLMs.
Limitations
Higher capacity LLMs are known to often yield superior capabilities. Our work does not include fine-tuning experimental results with the largest and the strongest LLMs in the literature (we have fine-tuning results with LLMs up to 2.7B parame- ters), due ever, the proposed framework can be applied LLMs of any size and similar improvements are expected. We leave the adoption of our methods to larger-scale LLMs to future work.
Training sample efficiency. We perform experi- ments to study the effect of the number of training samples for ASPIRE. We fix the number of training steps to be 50K while varying the size of the train- ing dataset. The results in Table 5 show that more training samples lead to performance improvement and with 2K training samples, ASPIRE can out- perform the baselines without soft prompt tuning by a large margin across different datasets. This underlines that our method, ASPIRE, can signif- icantly improve selective prediction performance even with limited number of training samples.
computational constraints. How-
Ethies Statement
7 Conclusion
LLMs are widely used in various applications nowadays. However, they can generate wrong or misleading answers to questions, which can cause serious consequences in some safety critical ap- plications. The framework proposed in our work can be used improve selective prediction per- formance of LLMs and make their deployments more reliable. However, it noted that the ob- tained selective prediction performances are still not perfect.
In this paper, we proposed a novel framework for adaptation with self-evaluation to improve selective prediction in LLMs. We implemented the frame- work via soft prompt tuning and demonstrated its superior performance over existing methods through extensive experiments. In future work, one could explore implementing our framework via other parameter efficient tuning approaches and

---

Acknowledgements
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. In The Eleventh International Conference on Learn- ing Representations.
We thank all the anonymous reviewers for their careful comments and feedback. The work is partially supported by Air Foree Grant FA9550- 18-1-0166, the National Science Foundation (NSF) Grants CCF-FMiFF-883677, LIS-2005559, Sa'TC-Frontiers 804648, CCF-2046710 10 and CCF- 1652140, and ARO grant number W911NF-17-
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.
The power of scale for parameter-effiiiet prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059.
0405. Jiefeng Chen and Somesh Jha are partially supported by the DARPA-GARD problem under agreement number 885000.
Cew Lin and Franz Josef Och. 2004. Auto- matic evaluation machine translation quality us- ing longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meet- ing of the Association for Computational Linguistics (ACL-04), pages 605-612.
References
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021a. P- tuning v2: Prompt tningg can be comparable to fine- tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. 2021b. Gpt understands, too, arXiv preprint 10Xvv:2103.10385
Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Fisenstein. 2023. Selectively an- swering ambiguous questions. S.ii:2300.14613.
R OpenAI. 2023. Gpt-4 technical report. arXiv.
arXiv preprint
Gabriel Poesia, Oleksandr Polazov, Vu Le, Ashish Ti- wari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code gen- eration from pre-trained language models. arXiv preprint arXiv:2201.11227
Yonatan Geifman and Ran Yaniv. 2017. Selective classification for deep neural networks. Advances in neural information processing systems, 30.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-ehhaneed bert with disentangled attention. arXiv preprint arXiv:200600655.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Lan- guage models unsupervised multitask learners. OpenAl blog, 188):9
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint urXiv:2106.06885
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 1606.05250.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language mod- cls for question answering. Transactions of the Asso- ciation for Computational Linguistics. 9:962-99 -977.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mo- hammad Saleh, Balaji Lakshminarayanan, and Pe- J Liu. 2022. Out-of-distribution detection and lective generation for conditional language models. arXiv preprint arXiv:2209.155:
Mandar Joshi, Funsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: large scale distantly supervised challenge dataset for reading comprehen- sion. arXiv preprint
Chenglei Si, Zhe Gan, Zhengyuan Yang. Shuohang Wang, Jianfeng Wang, Jordan Boyd-rraber, and Li- juan Wang. 2022. Prompting gpt-3 to be reliable. arXiv preprint 22X.9:22100091550
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, 7ac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl,

---

al. 2022. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138.
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding. Joshua B Tenenbaum, and Chuang Gan. 2023a. Planning with large language models for code generation. arXiv preprint arXiv:2333.055//
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl. Heather Cole-Lewis, Darlene Neal, et al. 2023. Towards expert-level medical question an- swering with large language models. arXiv preprint 11X1:2305.006677.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106
Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin Rousseau, al. 2023. Evaluating large language models on medical evidence summarization. medRxiv, pages 2023-04.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023b. Benchmarking large language models for news summarivation. arXiv preprint arXiv:2301.5 13848.
Neeraj Varshney and Chitta Baral. 2023. Post- abstention: Towards reliably re-attempting the ab- stained instances in QA. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics /Volume 1: Long Papers), pages 967-982, Toronto, Canada. Association for Compu- tational Linguistics.
Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Investigating selective prediction approaches across several tasks iid, ood, and adversarial set- tings. In Findings of the Association for Computa- tional Linguistics: ACL 2022, pages 1995-20 2002.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information process- ing systems, 30.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint
Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2017. A groad-coverage challenge corpus for sentence understanding through inference. arXiv preprint X74::1744
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and regularization for natural language processing. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference Natural Lan- guage Processing (Volume Long Papers), pages 1040-1051.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. large language models know what they don't know? arXiv preprint 20x.:3305.1815.
Hiyori Yoshikawa and Naoaki Okazaki. 2023.
Selective-lama: Selective prediction for confidence- aware evaluation of language models. In Findings of the Association for Computational Linguistics:
EACL 2023, pages 1972-1983.

---

A Hardware and Software
where additional question-answer pairs are preced- ing turns of the conversation about the paragraph consisting of questions and reference answers.
We run all experiments using the HuggingFace API on 40GB NVIDIA A100 GPUs in the De- bian GNU/Linux 10 system. We use the OPT and GPT2 models via the HuggingFace transformers li- brary which can be easily adapted for reproducibil- ity. We modify the Trainer class provided by the HuggingFace API for soft prompt tuning. We use the generate() function of the Huggingl'ace API generate answers. Unless specified, we use the default parameters of the generate () function. When generating the answer set A(f, pp. x). we set max_new_ tokkns550 while other cases, we always set max_new_ 2tt__new_tokess=256. The parameters for different decoding strategies are provided be- low:
B.2 TriviaQA
TriviaQA is reading comprchension dataset containing over 650K question-answer--vvdeeee triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and indepen- dently gathered evidence documents, six per ques- tion on average, that provide high quality distant supervision for answering the questions. We focus on TriviaQA as a closed-book QA task (in which the model must answer question without access to supporting paragraph). The training set contains 138,384 question queries while the test contains 17,944 question queries. We split the original test set into new test set containing 8,000 question queries and a validation set containing 9,944 ques- tion queries. We use the new test set for evaluation and use the validation set for hyper-parameter se- lection. We consider the following template with 5-shot prompt construct question queries:
Beam search decoding: we set 1un_beems>1 and do vo_sample==allse If we want get num_beams highest scoring beams, will set num_return_ serruurussseueenessuuu beamm....mmmee.......m We will specify num_beams when using beam search decoding.
Multinomial sampling decoding:
num_beams=1 and do_ do_sample=True. We will specify temperature when using multino- mial sampling decoding.
Q: In which decade did Billboard magazine first publish and American hit chart? A: 30s. Q: What is Bruce Willis' real first name? A: Walter.
Which city does David Soul come from? Chicago. Q: which William wrote the novel Lord Of The Flies? A: Golding. Q: Where in England was Dame Judi Dench born? A: York. Q: [Provided question] A:
Datasets
We use three question answering datasets: CoQA (Reddy et al., 2019), TriviaQA (Joshi et al., 2017) and SQuAD (Rajpurkar et al., 2016) for ex- periments. The details about these datasets given below.
B.1 CoQA
B.3 SQuAD
CoQA is large-scale dataset for Conversational Question Answering systems. The goal of the CoQA challenge is measure the ability of ma- chines to understand text passage and answer a series of interconnected questions that appear in a conversation. CoQA contains 127,0004 questions with answers collected from 8, 8,000+ conversations. The training set contains 108,647 question queries while the test set contains 7,983 question queries. We use the following template to construct question queries:
Stanford Question Answering Dataset (SQuAD) reading comprehension dataset, consisting of questions posed by crowd-worken on set of Wikipedia articles, where the answer to every ques- tion is segment of text, or span, from the cor- responding reading passage. We use the SQUAD 1.1 version, containing 100,000+ question-answer pairs on 500+ articles. The training set contains 86, 66,821 question queries while the test set contains
5,928 question queries. We use the following tem- plate to construct question queries:
[The provided context paragraph] [additional question-answer pairs]
[The provided context paragraph]
Q: [Provided question]
Q: [Provided question]
A:
A:

---

C LLMs
estimation. We use the Deberta-large model (He et al., 2020) that is fine-tuned on the NLI data set, MNLI (Williams et al., 2017) for the bidirectional entailment clustering algorithm.
We perform experiments with OPT (Zhang et al.. 2022) and GPT-2 (Radford et al., 2019) models, which are based on Transformer architecture. For Transformer architecture, there is a limit on the lengths of the sequences we pass the models. The OPT models can handle sequences of up to 2.048 tokens while the GPT-2 models can handle sequences of up to 1,024 tokens. If the sequence length of an input is larger than the maximum se- quence length that allowed, we force the model to output an empty sequence with a -oo selection score.
Self-eval. Self-eval is a simple baseline that ob- tains a selection score from the LLM by asking whether the proposed answer y* is correct or not. Suppose 2, is series of tokens representing the self-evaluation trigger string "The answer ". Sup- pose &, and & are the tokens that represent the words "correct" and "wrong" respectively. Re- call that the logits of the model f on v given x is f(v) x). Then, the self-eval score is computed as:
D Baselines
exp (f(zo ".y",.))
Pize x,
For selective prediction, we need to get predicted output sequence y and selection score g(x) for each input sequence x given a model f. The model f can be a pre-trained LLM or an LLM adapted with prompt tuning using training objective (12). We use the beam-search decoding, with the number of beams being equal to 5, to obtain the predicted output sequence y* We consider the following baselines compute the selection score (xx): Perplexity. Perplexity defined as the exponenti- ated average negative log-likelihood of sequence. The perplexity of the generated output sequence y* is computed as:
Exe(Zn) exp (f(z x, y", 2.))
(18)
P(True). P(True) proposed by Kadavath et al. (2022) a way to estimate the probability that a model's generation is correct by "asking" the model if its answer is correct. It samples m answers and constructs a new natural language question using these possible answers context before asking whether the proposed answer is correct and mea- sures the probability of the completion being True. We set m 4 and use the multinomial sampling with a temperature of 1.0 to sample the answers. The format the prompt is:
perp(ý" |x; = fnorm (y*
x)
(16)
Question: Who was the third president of the United States?
Predictive entropy is a widely used measure of uncertainty. We use the multinomial sampling with temperature of 0.5 to obtain an answer list [f),... ,....y ""] for each input sequence x. The predictive entropy is computed as:
Predictive Entropy.
Here are some brainstormed ideas: James Monroe Thomas Jefferson John Adams
Benjamin Harrison
George Washington
pe(x; f)
log m (5'1x)
(17)
Possible Answer: James Monroe
m
Is the possible answer: (A) True (B) False. The possible answer is:
We set m = 10. This is the same as the length- normalised predictive entropy baseline in Kuhn et al. (2023).
where the "brainstormed answers" are from the set of sampled answers and P(True) (i.e., the likeli- hood of the next token being True) is taken the uncertainty measure.
Semantic Entropy. Semantic entropy is an entropy-based uncertainty measure which uses a bi- directional entailment algorithm for marginalising over semantically-equivaaie
E Training Details
samples (Kuhn et al.,
2023). We follow the settings in Kuhn al. (2023). Specifically, we use the multinomial sampling with a temperature of 0.5 to obtain an answer list of size 10 for each input sequence for uncertainty
We have two stage training: the first stage is train the soft prompt 6, using the training objec- tive (12) and the second stage is to train the soft prompt A, using the training objective (13). For

---

both stages, we train the soft prompt for 10 epochs using AdamW optimizer with a batch size of 8, a learning rate 0.01 and cosine learning rate sched- ule. We remove those data points yx. where x yy| > 700 from the training set D to reduce GPU memory usage during training. Here, [x] is the length of the sequence x. This only removes very small portion of data points from the training set for each dataset (remove 4.02% training data points CoQA, 0% training data points in Trivi- aQA and 0.04% training data points in SQuAD). During training Op or 0s, we always use 20% train- ing data as validation data for selecting the best model among all checkpoints after each training epoch. Training By. we select the best model based on the loss on the validation data. When training 8,, select the best model based on the AUROC on the validation data.
perplexity score, For the predictive entropy method, the computational complexity is O( (maz)) since we need to additionally generate m answers and compute the likelihood of those m answers. For the semantic entropy method, we omit the compu- tational complexity of the bidircctional entailment clustering algorithm since its computational cost is much smaller than that of the generation of the LLM stated in Kuhn al. (2023). Thus, the computational complexity for semantic entropy is O (mas). For the self-eval method, the compu- tational complexity is (lmar) since we only need one additional forward pass to obtain the self-eval score, For the P(True) method, the computational complexity is O(m Imaz) since we need to ad- ditionally generate answers and need one for- ward pass to compute the P(True) score. For the proposed method ASPIRE, the computational com- plexity is ((1maz) since we only need additional one forward pass to obtain the learned self-eval score. Table 6 summarizes the computational com- plexity of different methods at test time.
F Computational Complexity Analysis
The proposed method ASPIRE needs to train two soft prompts Op and 0,, The complexity of training Og using the training objective (12) is the same as the complexity of the standard soft prompt tuning. When training 0g using the training objective (13), the number of training steps is the same as that of training 0p. In each training step of training 0s. we compute gradients for one correct output and two wrong outputs while in each training step of training 8p. we compute gradients for one reference output. Thus, the complexity of training 0g is the same as that of training 8p. Therefore, the complex- ity of the proposed method ASPIRE in the training time is the same as that of the standard soft prompt tuning.
Method
Complexity
Perplexity
(1maz)
Predictive Entropy
O . Imax))
Semantic Entropy
O(m [mex)
Self-eval
(bmar) )
P(True)
O (mox) (bbaaz)
ASPIRE (ours)
Table Computational complexity of different meth- ods in the test time.
G Rouge Threshold for Evaluation
We analyze the computational complexity of dif- ferent methods at test time in terms of the number of forward passes for the LLM. Since the LLM generates the output sequence in an auto-regressive way, the number of forward passes needed depends on the length of the generated output sequence. Suppose the maximum length of the generated out- put sequence is Imaa- To generate an output se- quence given an input sequence, we need one for- ward pass to encode the input sequence and at most Imax forward passes to obtain the output sequence. Thus, for generating the output sequence, the maxi- mum number of forward passes is + mma and the complexity is (. For the perplexity method, the computational complexity is (4maa) since we only need additional one forward pass to obtain the
We use the Rouge-I. (Lin and Och, 2004) metric to evaluate if the predicted answer is correct or not. The Rouge-L metrie produces score in [0, 1]. We need a threshold y to determine whether the predicted answer is correct or not. If the Rouge-L score is larger than the threshold 7, then the pre- dicted answer is correct; otherwise, the predicted answer is wrong. The choice of y depends on the applications. Low values of may lead to label- ing some wrong answers as correct answers while large values of y may lead to labeling some correct answers as wrong answers. If we regard the wrong answer as the positive class, then we can use the precision and recall metrics to evaluate the choice of y. To compute the precision and recall metrics, we need ground-trutt labels for determining the

---

correctness of predicted answers, which requires manual labeling. If the Rouge-L score equal to O (or 1), then answer is wrong (or correct). Thus, we only need to label those samples whose Rouge-L. scores are in (0, 1). To compare different values of 7, we com- pute the precision and recall metrics after manually label 200 samples whose Rouge-L scores are in the range of (0, 1). The results in Table show that larger values of y lead to higher recall but lower precision, while the lower values of y lead higher precision but lower recall. We propose this work for safety-critical applications where accepting a wrong answer is more costly compared to rejecting a correct answer that is different from the refer- ence answers. Thus, we prefer high recall than evaluate
ble 14 (use OPT models and set 7 0.7), Ta- ble 15 (use OPT models and set 7 0.8) and Table 16 (use OPT models and set y = 0.9). The results show that the proposed method ASPIRE significantly outperforms the baselines in terms of AUACC and AUROC across different datasets and LLMs for different values of the Rouge threshold y
mostly sure that the predicted
The Effect of the Hyper-parameter a
We study the effect of the hyper- syper-parameter a in the proposed selection score (Eq. (11)) for our method. The results in Table 17 show that set- ting a = 0.25 leads to the best performance across different datasets and different models. Only us- ing the normalized likelihood (i.c., = 0) or only using the learned self-eval score (i.e., - 1) con- sistently leads to much worse performance. We choose for our method based on the performance on the validation data from the TriviaQA dataset using the OPT-2.7B model. We then use the same value for different datasets and different models. We consider a E {0.0, 0.25, 0.5, 0.75, 1.0} when tuning it. Based on the validation results, we set = 0.25 by default.
high precision. In experiments, different methods under the Rouge-L metric with 7 {0.7,00 0.8, 0, 9} ensure that the recall is at least 90%.
H Rouge Threshold for the Proposed Framework
In the proposed framework ASPIRE, we need the Rouge threshold fy to determine if the generated answer is correct or not. We want to set large enough value of ? that the generated wrong answers won't be labeled as correct answers. To determine the value f,, we manually label the cor- rectness of the 10 generated answers for 50 training examples from each dataset (we have three datasets CoQA, TriviaQA and SQuAD). The answers are generated using the OPT-2.7B 7B model. We find that if we set 0.9, then no wrong answers would be labeled as correct answers. Thus, we set 0.9 for the proposed framework.
K Comparing with Self-Consistency
Self-consistency (Wang et al., 2022) can be used to obtain confidence measures as proposed by Si et al. (2022). We sample 10 times to obtain of different answers for each question using the multinomial sampling with a temperature of 0.5. Among all the generated answers, we take the most frequent answer as the final prediction and its frequency as the selection score. Since self-consistency produces discrete selection scores (in the above setting, the number of possible se- lection scores is 10) and use the composite trapezoidal rule to compute AUACC, it is eas- ier for self-consistency achieve high AUACC compared to those approaches that produce con- tinuous selection scores. Note that the proposed method produce continuous selection scores. Thus, it might not be fair to compare the proposed method with self-consistency. However, even though self- consistency has more advantages in achieving high AUACC, the proposed method ASPIRE still signif- icantly outperforms self-consistency as shown in Table 18. We also observe that Self-Consistency might lead to worse accuracy meaning that the LLM can be consistently wrong.
I Complete Results
In this section, we present the complete results for OPT and GPT2 models of different sizes and dif- ferent Rouge threshold 7. We first evaluate the accuracy of different LLMs. The results are in Ta- ble 8 (set = 0.7), Table 9 (set > = 0.8) and Table 10 (set 7 - 0.9). The results show that after training 9p via soft prompt tuning, the accuracy of LLMs is improved significantly. We then evaluate different approaches compute the selection score when the model's predictions are fixed. The results are in Table 11 (use GPT2 models and set 7 0.7), Table 12 (use GPT2 models and set ? = 0.8), Ta- ble 13 (use GPT2 models and set = 0.9), Ta-

---

CoQA
TriviaQA
SQuAD
7
Precision 1
Recall
Precision f
Recall
Precision
Recall
0.1
100.00
0.00
100.00
0.62
100.00
7.91
0.2
100.00
10.00
100.00
2.50
100.00
34.53
0.3
100.00
22.50
100.00
11.88
98.55
48.92
0.4
100.00
45.62
97.01
40.62
93.58
73.38
0.5
97.98
60.62
97.09
62.50
85.94
79.14
0.6
97.41
70.62
96.19
63.12
84.73
79.86
0.7
93.51
90.00
86.81
98.75
76.16
94.24
0.8
86.59
96.88
81.22
100.00
73.66
98.56
0.9
80.71
99.38
80.00
100.00
69.85
100.00
Table 7: Results of comparing different choices of the Rouge threshold 7. The wrong answer is regarded as the positive class. We the OPT-2.7B model. We manually label 200 samples with Rouge-L. scores in the range of (0, 1) in each dataset and then compute the precision and recall. All numbers are percentages.
CoQA
TriviaQA
SQuAD
Model
Acc f
Acc f
Acc 1
Pre-trained GPT2-Medium
35.12
5,44
4.42
Adapted GPT2-Medium with 8,
57.90
9.04
66.63
Pre-trained GPT2-Large
41.21
8.16
6.09
Adapted GPT2-Large with Op
63.89
12.50
71.34
Pre-trained GPT2-XL
46.27
11.80
7.41
Adapted GPT2-XL. with 8,
69.18
17.45
75.44
Pre-trained OPT-350M
28.76
4.35
13.65
Adapted OPT-350M with 0g
59.46
8.25
64.74
Pre-trained OPT-1.3B
54.13
15.80
30.23
Adapted OPT-1.31 with 0p
76.85
21.73
80.94
Pre-trained OPT-2.7B
60.68
21.38
35.95
Adapted OPT-2.7B with 0y
80.45
29.21
83.27
Table Results of evaluating the accuracy of different LLMs when the Rouge threshold 7 0.7. All numbers percentages.
L Qualitative Evaluation
Question: Who is the most successful UK solo artist in the USA?
We present some concrete examples from the Triv- iaQA dataset show the advantages of the pro- posed method qualitatively. We compare the pro- posed method ASPIRE to the baseline Semantic Entropy. The model for generating answers is the adapted OPT-2.7B 7B with learned fy. The examples below show that some semantic entropy scores for correct predictions are lower than some semantic entropy scores for wrong predictions while the AS- PIRE scores for correct predictions are consistently higher than the ASPIRE scores for wrong predic- tions. The ASPIRE scores are log likelihood scores and can be converted to likelihood scores by taking exponentiation with the base e. Examples where predictions are correct
Answer: Elton John.
Predicted answer: Elton John. Semantic entropy score: -1.1
ASPIRE score: -0.8163
Question: In which decade of the 20th century was Anne Baneroft born?
Answer: 1930s.
Predicted answer: 1930s.
Semantic entropy score: -0.6167
ASPIRE score: -0.9026
Question: The Suez Canal connects the Mediter- ranean Sea to which other Sea?
Answer: Red sea.

---

CoQA
TriviaQA
SQuAD
Model
Acc t
Acc 1
Acc 1
Pre-trained GPT2-Medium
32.12
5.00
2.85
Adapted GPT2-Mcdium with 0p
55.12
8.71
62.92
Pre-trained GPT2-Large
38.16
7.64
3.98
Adapted GPT2-Large with Op
61.04
12.14
67.56
Pre-trained GPT2-XI.
42.67
11.10
5.21
Adapted GPT2-XL with Op
66.49
16.96
71.17
Pre-trained OPT-350M
27.38
4.25
11.15
Adapted OPT-350M with 8,
57.02
8.05
61.29
Pre-trained OPT-1.3B
51.35
15.35
25.73
Adapted OPT-1.3B with Og
74.46
21.26
77.28
Pre-trained OPT-2.7B
57.72
20.71
30.94
Adapted OPT-2.7B with 8p
77.97
28.55
80.04
Table Results evaluating the accuracy of different LLMs when the Rouge threshold y = 0.8. All numbers are percentages.
CoQA
TriviaQA
SQuAD
Model
Acc
Acc f
Acc 1
Pre-trained GPT2-Medium
30.49
4.88
1.99
Adapted GPT2-Medium with Op
53.11
8.53
60.51
Pre-trained GPT2-Large
36.20
7.41
3.00
Adapted GPT2-Large with Op
59.04
11.85
64.98
Pre-trained GPT2-XL
40.32
10.82
4.12
Adapted GPT2-XL with Og
64.59
16.70
68.83
Pre-trained OPT-350M
26.81
4.20
9.62
Adapted OPT-350M with 0p
55.33
8.00
59.35
Pre-trained OPT-1:3B
49.78
15.24
22.79
Adapted OPT-1.3B with 6y
72.78
21.07
74.97
Pre-trained OPT-2.7B
56.06
20.55
27.41 .7.41
Adapted OPT-2.7B with Op
76.45
28.26
78.12
Table Results evaluating the accuracy different L.MSs when the Rouge threshold 7 = 0.9. All numbers percentages.
Predicted answer: Red Sea. Semantic entropy score: 2.2082
Predicted answer: George H W Bush. Semantic entropy score: 0.4547
ASPIRE score: -0.2309
ASPIRE score: -100397
Question: Sun Yat Sen overthrew the emperor in which country establishing a republic after 2000 years of imperial rule?
Question: What Michelle Pfeiffer movie got a boost from the Coolio song Gangsta's Paradise? Answer: Dangerous Minds.
Answer: China.
Predicted answer: Scarface.
Predicted answer: China.
Semantic entropy score: 0.0647
Semantic entropy score: 2.0028
ASPIRE score: -1.0531
ASPIRE score: -0.4205 Examples where predictions are wrong
Question: What was President Gerald Ford's
Question: Who was the director of the CIA from 1976-81?
middle name?
Answer: Rudolph.
Answer: George Bush.
Predicted answer: William.

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC T
AUROC
AUACC
AUROCT
AUACC
AUROC
Perplexity
38.92
55.77
7.67
60.52
4.58
55.35
Pre-trained
Predictive Entropy
45.91
62.89
10.02
67.66
5.99
57.07
GPT2-
Semantic Entropy
48.35
66.30
10.28
68.54
6.18
57.36
Medium
Self-eval
36.16
51.14
6.26
56.74
3.70
43.44
P(Truc)
34.08
48.21
8.24
60.62
5.41
54.33
Perplexity
72.03
67.89
18.02
72.17
81.91
72.38
Adapted GPT2-
Predictive Entropy
72.59
69.42
20.07
77.48
82.00
73.09
Semantic Entropy
73.95
71.54
19.86
77.62
82.35
73.66
Medium
Self-eval
57.94
50.43
9.94
54.68
64.79
46.99
with 0p
P(True)
56.71
48.76
13.55
60.79
65.94
49.13
ASPIRE (ours)
76.32
75.30
20.65
79.41
84.15
77.59
Perplexity
48.57
59.82
13.74
66.51
6.39
53,96
Pre-trained GPT2-
Predictive Entropy
55.04
66.68
16.25
70.46
Semantic Entropy
57.13
69.57
16.02
70.06
8.81
59.24
Large
Self-eval
42.24
51.72
9.78
54.74
46.79
P(True)
36.73
45.69
8.60
48.62
6,83
55.62
Perplexity
77.15
68.15
26.83
77.06
86.26
75.34
Adapted
Predictive Entropy
77.45
69.76
27.83
80.02
86.32
75.65
GPT2-
Semantic Entropy
78.85
71.97
27.61
79.88
86.53
75.90
Large with 0
Self-eval
64.28
50.61
14.26
54.34
70.86
50.81
P(Truc
58.97
45.55
12.38
47.61
70.73
50.09
ASPIRE (ours)
81.30
76.38
29.13
82.14
87.83
79.22
Perplexity
55.93
62.05
22.60
72.88
7.68
$1.90
Predictive Entropy
60.76
67.53
24.83
76.20
10.04
57.21
Pre-trained GPT2-XL
Semantic Entropy
63.03
70.50
24.37
75.33
10.38
59.17
Self-eval
46.67
50.83
9.30
42.75
7.32
49.56
P(Truc)
46.98
51.17
10.62
44.54
10.69
60.87
Perplexity
83.27
72.79
36.49
79.92
88.73
75.08
Predictive Entropy
83,49
73.44
37.31
82.21
88.25
74.16
Adapted GPT2-XL with 8p
Semantic Entrupy
84.40
75.16
36.68
81.40
75.26
Self-eval
69.91
51.90
14.39
43.33
74.26
49.13
P(True)
70.63
52.83
13.59
40.59
74.34
49.09
ASPIRE (ours)
85.65
78.32
38.06
83.23
89.86
78.35
Table 11: Results of evaluating different methods to compute the selection score when the model's predictions are fixed. We the GPT2 models and set the Rouge threshold 7 0.7. All numbers are percentages. Bold numbers are superior results.
Semantic entropy score: -3.9773
ASPIRE score: -2.8203
Question: Kim Camees' nine weeks at No I with Bette Davis Eyes interrupted for week by which song?
Answer: Stars on 45 medley.
Predicted answer: Bette Davis Eyes.
Semantic entropy score: -1.4973
ASPIRE score: -2.2803

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC1
AUROC
AUACC1
AUROC
AUACC
AUROC 1
Perplexity
35.24
54.28
7.03
59.54
2.82
53.29
Pre-trained
Predictive Entropy
42.43
62.42
9.23
66.62
58.15
GPT2-
Semantic Entropy
45.53
66.84
9.52
67.83
4.02
58.53
Medium
Self-eval
32.97
51.13
5.95
57.98
2,06
40.62
P(Truc)
31.05
48.10
7.81
61.51
3.73
55.72
Perplexity
69.36
67.21
17.42
71.77
79.26
72.25
Adapted GPT2-
Predictive Entropy
69.74
68.58
19.38
77.26
79.18
72.70
Semantic Entropy
71.35
71.10
19.22
77.55
79.61
73.53
Medium
Self-eval
55.04
50.31
9.75
55.26
61.43
47.61
with 8p
P(Truc)
53.95
48.66
13.32
61.55
62.07
49.06
ASPIRE (ours)
73.97
75.05
20.12
79.59
82.02
78.02
Perplexity
44.95
58.70
13.06
66.47
4.06
$1.95
Pre-trained GPT2-
Predictive Entropy
51.57
66.32
15.43
70.33
5.61
57.34
Semantic Entropy
54.39
70.24
15.25
70.08
6.25
61.09
Large
Self-eval
39.66
52.36
9.21
54.62
3.15
45.40
P(True)
33.73
45.72
8.20
49.18
4.68
57.51
Perplexity
74.64
67.40
26.20
76.89
83.61
74.57
Adapted GPT2-
Predictive Entropy
74.96
69.07
27.22
80.01
83.57
74.67
Semantic Entropy
76.65
71.82
27.06
79.99
83.81
75.10
Large with 0,
Self-eval
61.88
50.99
13.83
54.15
67.28
51.20
P(True)
56.35
45.90
11.95
47.55
67.13
50.52
ASPIRE (ours)
79.39
76.49
28.43
82.01
85.71
79.27
Perplexity
52.07
61.15
21.54
72.72
5.30
49.81
Predictive Entropy
56.83
66.90
23.65
76.15
7.27
56.53
Pre-trained GPT2-XI.
Semantic Entropy
59.74
70.83
23.23
75.38
7.59
58.85
Self-eval
43.34
51.14
8.81
42.76
5,45
51.47
P(True)
43.24
$1.09
9.81
43.94
8.54
65.61
Perplexity
81.05
71.85
35.61
79.69
86.08
74.71
Predictive Entropy
81.23
72.42
36.42
82.01
85,53
73.62
Adapted GPT2-XL with 8p
Semantic Entropy
82.38
74.62
35.84
81.31
85,93
74.84
Self-eval
67.35
51.89
14.05
43.45
70.30
49.21
P(Truc)
68.02
52.83
13.21
40.48
69.47
48.32
ASPIRE (ours)
83.91
78.09
37.26
83.18
87.76
78.82
Table 12: Results of evaluating different methods to compute the selection score when the model's predictions are fixed. We use the GPT2 models and set the Rouge threshold 7 = 0.8. All numbers are percentages. Bold numbers are superior results.

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC
AUROC
AUACC1
AUROC
AUACC
AUROC 1
Perplexity
33.09
53.03
6.72
58.75
1.74
47.52
Pre-trained
Predictive Entropy
40.52
62.05
8.90
66.11
2.61
56.43
GPT2-
Semantic Entropy
44.11
67.38
9.26
67.57
2.85
57.36
Medium
Self-eval
31.46
51.29
5.88
58.50
1.17
35,24
P(True)
29.24
47.97
7.73
62.30
2.78
58.35
Perplexity
67.42
66.51
17.06
71.50
77.63
72.22
Adapted GPT2-
Predictive Entropy
67.89
68.15
19.06
77.30
77.49
72.55
Semantic Entropy
69.77
71.20
18.94
77.75
78.07
73.77
Medium
Self-eval
53.15
50.51
9.67
55.80
58.95
47.61
with 8p
P(Truc)
51.95
48.59
13.21
62.06
59.55
48.95
ASPIRE (ours)
72.39
74.96
19.81
79.64
80.68
78,49
Perplexity
42.74
57.93
12.56
65.85
2.78
46.87
Pre-trained GPT2-
Predictive Entropy
49.68
66.44
14.89
69.76
3.94
54.80
Semantic Entropy
52.90
71.07
14.76
69.63
4.53
59.75
Large
Self-eval
38.08
52.84
8.97
54.45
2.42
45.97
P(True)
31.71
45.48
8.06
49.66
3.84
60.48
Perplexity
72.97
66.96
25.67
76.60
81.77
74.01
Adapted GPT2-
Predictive Entropy
73.34
68.78
26.69
79.84
81.80
74.31
Semantic Entropy
75.24
71.99
26.59
79.96
75.36
Large with 0
Self-eval
60.35
51.58
13.44
53.82
64.89
51,42
P(True)
54.34
45.68
11.65
47.57
64.77
50.75
ASPIRE (ours)
78.08
76.61
27.97
81.99
84.24
79.37
Perplexity
49.71
60.59
20.96
72.31
46.28
Predictive Entropy
54.74
67.05
23.10
75.93
5.78
55.59
Pre-trained GPT2-XI.
Semantic Entropy
58.07
71.83
22.76
75.33
6.18
59.20
Self-eval
41.19
51.46
8.67
42.97
4.61
53.48
P(True)
40.37
50.77
9.46
43.58
7.30
69.47
Perplexity
79.60
71.40
35.11
79.47
84.69
74.62
Predictive Entropy
79.86
72.25
35.93
81.85
84.23
73.81
Adapted GPT2-XL with 8p
Semantic Entropy
81.25
74.87
35.39
81.19
84.74
75.38
Self-eval
65.61
52.08
13.87
43.51
49.56
P(Truc)
65.90
52.61
12.95
40.31
67.37
48.74
ASPIRE (ours)
82.77
78.11
36.81
83.09
86.57
79.06
Table 13: Results of evaluating different methods to compute the selection score when the model's predictions are fixed. We use the GPT2 models and set the Rouge threshold 7 = 0.9. All numbers are percentages. Bold numbers are superior results.

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC1 f
AUROC f
AUACC1
AUROC
AUACC
AUROC 1
Perplexity
35.37
59.39
6.81
67.09
13.07
50.34
Predictive Entropy
36.55
60.31
7.20
65.04
17.86
59.33
Pre-trained
Semantic Entropy
38.80
64.38
7.31
65.15
19.08
61.66
OPT-350M
Self-eval
30.02
52.69
5.98
61.17
14.00
51.41
P(True)
28.70
50.60
5.29
55.69
17.76
59.55
Perplexity
74.50
70.21
18.13
75,86
80.64
73.76
Predictive Entropy
74.14
68.88
18.73
76.83
80.79
73.46
Adapted
Semantic Entropy
74.94
70.14
18,46
76.91
81.10
73.98
OPT-350M with 8p
Self-eval
60.86
51.67
10.29
57.89
65.48
50.70
P(Truc)
59.20
50.04
8.71
52.05
64.55
50.29
ASPIRE (ours)
75.55
72.37
19.00
78.54
82.59
77.18
Perplexity
69.51
69.32
29.78
74.77
32.43
54.65
Predictive Entropy
69.46
68.48
31.01
75.21
41.06
62.96
Pre-trained OPT-1.3B
Semantic Entropy
70.42
70.46
30.63
74.74
43.33
66.30
Self-eval
56.38
52.86
15.06
49.96
30.74
$1.50
P(True)
57.21
53.19
16.83
51.19
28.88
46.75
Perplexity
88.50
73.64
42.46
79.96
91.45
74.47
Predictive Entropy
88.24
72.38
43.03
80.46
91.46
74.38
Adapted OPT-1.3B with 8,
Semantic Entropy
88.91
74.02
42.70
91.72
75.44
Self-eval
78.52
53.08
20.65
49.24
81.05
51.52
P(True)
79.07
52.76
22.20
50.34
81.58
50.77
ASPIRE (ours)
90.76
79.26
44.03
83.06
93.41
81.17
Perplexity
75.26
70.16
40.93
78.86
40.82
57.20
Predictive Entropy
75.29
69.16
41.20
78.92
47.18
62.85
Pre-trained OPT-2.7B
Semantic Entropy
76.31
70.96
40.72
78.06
51.53
68.40
Self-eval
62.32
52.26
25.88
59.04
41.78
P(True)
62.16
$1.80
24.88
56.89
34.77
49.42
Perplexity
90.80
74.23
53.56
81.74
92.86
75.72
Predictive Entropy
90.63
72.87
$3.91
82.19
92,96
75.58
Adapted OPT-2.7B with 8p
Semantic Entropy
91.23
74.61
53.58
81.55
93.21
76.53
Self-eval
81.30
50.76
32.98
56.03
86.34
P(Truc)
81.14
51.01
33.48
56.27
82.59
49.48
ASPIRE (ours)
92.63
80.25
55.06
84.44
94.73
82.60
Table 14: Results of evaluating different methods to compute the selection score when the model's predictions are fixed. We use the OPT models and set the Rouge threshold = 0.7. All numbers are percentages. Bold numbers are superior results.

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC
AUROC f
AUACC1
AUROC
AUACC
AUROC 1
Perplexity
33.50
58.50
6.64
66.73
10.50
49.05
Predictive Entropy
34.88
59.82
7.03
64.84
14.59
58.98
Pre-trained
Semantic Entropy
37.51
64.67
7.17
65.20
15.74
61.85
OPT-350M
Self-eval
28.73
52.91
5.93
61.83
11.33
50.71
P(True)
27.10
50.36
5.25
56.28
15.38
61.06
Perplexity
72.04
69.26
17.76
75.70
77.72
72.95
Predictive Entropy
71.77
68.12
18.30
76.51
77.91
72.76
Adapted
Semantic Entropy
72.80
69.90
18.06
76.67
78.35
73.57
OPT-350M with 8p
Self-eval
58.65
52.06
10.03
57.87
61.83
50.32
P(Truc)
56.82
50.17
8.58
52.27
61.13
50.18
ASPIRE (ours)
73.56
72.05
18.63
78.42
80,33
77.29
Perplexity
66.09
67.76
29.01
74.46
27.67
$3.61
Predictive Entropy
66.34
67.36
30.21
74.92
35.65
62.44
Pre-trained OPT-1.3B
Semantic Entropy
67.64
70.02
29.91
74.61
38.00
66.50
Self-eval
53.87
53.23
14.73
50.12
26.42
$1.63
P(True)
54.07
52.70
16.44
51.38
23.69
45.44
Perplexity
86.67
72.53
41.59
79.61
89.00
73.48
Predictive Entropy
86.41
71.33
42.18
80.15
89.02
73.35
Adapted OPT-1.3B with 8,
Semantic Entropy
87.27
73.41
41.89
79.75
89.42
Self-eval
76.49
53.45
20.23
49.20
77.85
52.25
P(True)
76.79
52.52
21.65
50.25
77.86
50.71
ASPIRE (ours)
89.48
79.05
43.23
82.84
91.86
81.44
Perplexity
72.00
68.49
39.79
78.43
35.76
56.78
Predictive Entropy
72.23
67.89
40.05
78.49
41.18
61.98
Pre-trained OPT-2.7B
Semantic Entropy
73.64
70.43
39.67
77.81
45.83
68.35
Self-eval
59.51
52.24
25.10
59.02
36.71
59.36
P(True)
58.81
$1.26
24.13
56.80
29.13
48.41
Perplexity
89.10
73.16
52.64
81.56
91.04
74.96
Predictive Entropy
88.95
72.00
$2.97
82.00
91.16
74.86
Adapted OPT-2.7B with 8p
Semantic Entropy
89.80
74.53
52.71
81.47
91.46
75.91
Self-eval
79.12
51.00
32.28
56.03
83.28
56.52
P(Truc)
78.74
50.89
32.95
56.42
79.05
49.26
ASPIRE (ours)
91.49
80.12
54.15
84.28
93.37
82.33
Table 15: Results of evaluating different methods to compute the selection score when the model's predictions are fixed. We use the OPT models and set the Rouge threshold 7 0.8. All numbers are percentages. Bold numbers are superior results.

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACC1
AUROC f
AUACC
AUROC
AUACC
AUROC 1
Perplexity
32.58
57.88
6,50
66.50
8.63
46.42
Predictive Entropy
34.13
59.61
6.91
64.62
12.56
58.33
Pre-trained
Semantic Entropy
36.97
64.81
7.06
65.06
13.82
61.99
OPT-350M
Self-eval
27.98
52.97
5.90
62.10
10.08
51.66
P(True)
26.41
50.23
5.21
56.37
14.01
62.76
Perplexity
70.07
68.20
17.67
75.63
76.12
72.40
Predictive Entropy
69.97
67.41
18.22
76.47
76.44
72.48
Adapted
Semantic Entropy
71.38
69.93
17.98
76.65
77.11
73.81
OPT-350M with 8p
Self-eval
57.05
52.30
9.96
57.81
59.96
50.56
P(Truc)
55.10
50.32
8.53
52.22
59.05
50.03
ASPIRE (ours)
71.94
71.41
18.54
78.39
79.12
77.38
Perplexity
64.03
66.70
28.77
74.31
24.05
$1.41
Predictive Entropy
64.59
66.80
29.98
74.81
31.35
60.95
Pre-trained OPT-1.3B
Semantic Entropy
66.29
70.03
29.72
74.56
34.05
66.05
Self-eval
52.35
53.37
14.64
50.14
24.12
52.63
P(True)
52.51
52.64
16.27
51.38
20.92
45.41
Perplexity
85.21
71.30
41.21
79.43
87.71
73.17
Predictive Entropy
85.05
70.44
41.81
80.00
87.81
73.34
Adapted OPT-1.3B with 8,
Semantic Entropy
86.23
73.38
41.55
79.66
74.81
Self-eval
75.09
53.72
20.07
49.23
75.80
52.60
P(True)
75.16
52.38
21.44
50.22
75.83
51.10
ASPIRE (ours)
88.49
78.52
42.88
82.70
90.79
81.34
Perplexity
70.07
67.37
39.42
78.23
31.18
54.43
Predictive Entropy
70.44
67.03
39.69
78.34
36.14
60.36
Pre-trained OPT-2.7B
Semantic Entropy
72.29
70.35
39.34
77.68
40.96
67.71
Self-eval
57.76
52.07
24.85
58.93
32.56
59.52
P(True)
57.06
50.98
23.96
56.74
25.64
48.02
Perplexity
88.06
72.44
$2.12
81.33
90.01
74.59
Predictive Entropy
87.95
71.48
$2.48
81.81
90.17
74.71
Adapted OPT-2.7B with 8p
Semantic Entropy
88.96
74.50
52.28
81.35
90.47
75.75
Self-eval
77.71
51.04
31,90
55,89
81.27
56.36
P(Truc)
77.16
50.54
32.62
56.33
76.89
48.85
ASPIRE (ours)
90.76
79.94
53.68
84.10
92.52
82.04
Table 16: Results of evaluating different methods to compute the selection score when the model's predictions are fixed. We use the OPT models and set the Rouge threshold 0.9. All numbers are percentages. Bold numbers are superior results.

---

CoQA
TriviaQA
SQuAD
Model
Method
AUACCT 1
AUROCT 1
AUACC 1
AUROCT 1
AUACC
AUROC 1
ASPIRE (a 0.0)
72.03
67.89
18.02
72.17
81.91
72.38
Adapted GPT2-
ASPIRE (a- = 00.25)
76.32
75.30
20.65
79.41
84.15
77.59
ASPIRE (a =0.5)
75.76
75.27
20.24
80.35
83.24
76.50
Medium
ASPIRE (a = 00.55)
73.26
71.99
18.01
77.00
81.76
73.98
with By
ASPIRE 1.0)
67.61
66.72
14.52
72.52
79.60
70.52
ASPIRE (a 0.0)
77.15
68.15
26.83
77.06
86.26
75.34
Adapted GPT2- Large with 0p
ASPIRE (a = 0.25)
81.30
76.38
29.13
82.14
87.83
79.22
ASPIRE (a = 0.5)
80.87
76.39
28.49
82.41
86.97
77.66
ASPIRE (a 0.75)
78.91
73.38
25.32
78.74
85.66
74.95
ASPIRE (a = 1.0)
74.22
68.01
19.77
72.75
83.99
71.74
ASPIRE (a = 0.0)
83.27
72.79
36.49
79.92
88.73
75.08
Adapted GPT2-XL with 8,
ASPIRE (a 0.25)
85.65
78.32
38.06
83.23
89.86
78.35
ASPIRE (a = 0.5)
85.15
78.02
37.22
82,80
88.82
76.12
ASPIRE (a 0.75)
83.03
74.22
33.37
87.47
73.13
ASPIRE (a 1.0)
77.66
66.38
25.49
70.09
85.88
69.89
ASPIRE 0.0)
74.50
70.21
18.13
75.86
80.64
73.76
Adapted OPT-350M with 0p
ASPIRE (a = 0.25)
75.55
72.37
19.00
78.54
82.59
77.18
ASPIRE 0.5)
74.95
72.07
18.81
80.48
81.69
75.90
ASPIRE (a = 0.75)
72.55
68.45
16.21
78.76
80.16
73.27
ASPIRE (a 1.0)
68.02
62.53
12.08
70.31
78.00
70.25
ASPIRE (a = 0.0)
88.50
73.64
42.46
79.96
91.45
74.47
Adapted OPT-1.3B with 0p
ASPIRE (a 0.25)
90.76
79.26
44.03
83.06
93.41
81.17
ASPIRE (a = 0.5)
90.64
79.04
43.70
83.40
93.27
80.91
ASPIRE 0.75)
89.84
76.58
42.05
80.97
92.96
79.90
ASPIRE (a 1.0)
88.66
73.53
38.65
76.34
92.48
78.57
ASPIRE (a = 0.0)
90.80
74.23
53.56
81.74
92.86
75.72
Adapted OPT-2.7B with e,
ASPIRE (a = 0.25)
92.63
80.25
55.06
84.44
94.73
82.60
ASPIRE (a 0.5)
92.56
80.18
54.61
84.33
94.59
82.16
ASPIRE (a = 0.75)
92.05
78.37
52.71
81.52
94.28
80.98
ASPIRE (a 1.0)
91.33
76.08
48.84
76.39
93.77
79.48
Table 17: Results studying the effect of a. numbers are percentages. Bold numbers are superior results.
CoQA
TriviaQA
SQuAD
Model
Method
Acc
AUACC1
Acc 1
AUACCT
Acc 1
AUACCT
Pre-trained OPT-350M
Self- Consistency
24.54
37.83
3.41
7.93
5.75
15,59
Self-Consistency
59.03
74.09
7.41
18.40
63.87
79.99
Adapted OPT-350M with 8
ASPIRE (ours)
59.46
75.55
8.25
19.00
64.74
82.59
Pre-trained OPT-1.3B
Self-Consisiency
50.90
69.14
13.74
30.10
19.89
41.45
Self-Consistency
76.73
88.29
20.34
42.12
80.84
90.68
Adapted O.T-L.3 with 0p
ASPIRE (ours)
76.85
90.76
21.73
44.03
80.94
93.41
Pre-trained OPT-2.7B
Self- Self-Consistency
57.60
75.27
19.57
39.88
23.73
47.78
Self Self-Consistency
80.41
90.61
27.59
52.20
83.11
92.34
Adapted OPT-2.7B with 0p
ASPIRE (ours)
80.45
92.63
29.21
55.06
83.27
94.73
Table 18: Comparing with self-consistency. All numbers are percentages. Bold numbers are superior results.


Reviews:
[{"review_id": "kckkBsiFI4", "reviewer": "Reviewer_1jxs", "summary": "", "questions": "See the reasons to reject the paper", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, which has helped us improve our paper. We appreciate that you think our paper proposes “a simple yet interesting method that outperforms other competing baselines as demonstrated by extensive experiments”.\n\nWe address all your concerns and questions in detail below.\n\n**Question 1**: The description of the metrics is limited. it would be desirable to have an explanation of the metrics used in the paper. Or at least a citation to the metrics would have been good.\n\n**Answer 1**: We use the area under the accuracy-coverage curve (AUACC) metric to measure selective prediction performance and use the area under the receiver operator characteristic curve (AUROC) metric to measure the quality of the selection score estimation. AUACC is the common metric used for evaluating selective prediction performance [1][2]. AUROC is equivalent to the probability that a randomly chosen correct output sequence has a higher selection score than a randomly chosen incorrect output sequence. AUROC is used in Kuhn et al. (2023) for evaluating uncertainty estimation methods. We also use the Rouge-L (Lin and Och, 2004) to evaluate the similarity of the generated answers with the reference answers, following Kuhn et al. (2023). We will clarify these.\n\n[1] Xin, Ji, et al. \"The art of abstention: Selective prediction and error regularization for natural language processing.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n\n[2] Yoshikawa, Hiyori, and Naoaki Okazaki. \"Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models.\" Findings of the Association for Computational Linguistics: EACL 2023. 2023.\n\n**Question 2**: The training objective in Equation 7 would increase the likelihood of negative cases as well resulting in unwanted behavior. Should the objective be: \\mathcal{L}{c} - \\mathcal{L}{w}?\n\n**Answer 2**: The original training objective (Equation 7) is indeed correct. Given the input query $\\textbf{x}$, for those wrong output sequences, we would like to maximize the probability of the “incorrect” token so that the probability of the “correct” token would be minimized. This can result in a low learned self-eval score for the incorrect output sequences, encouraging better selective prediction performance. \n\n**Question 3**: The paper needs a bit of polishing as at times equations are clubbed together. The equations in Sections 4 and 5 can be clubbed together while introducing them.\n\n**Answer 3**: We will polish Sections 4 and 5 to make the equations separate. \n\n**Question 4**: The paper is motivated by the fact that we need to generate multiple sequences during the test time and progress to get rid of them. However, ASPIRE generates multiple answers during the training phase. This should be explicitly mentioned in the paper as it directly conflicts with the claim of not generating multiple sequences.\n\n**Answer 4**: Yes, during the training phase, ASPIRE needs to generate multiple answers. However, it doesn’t conflict with our claim, as our claim is that ASPIRE doesn’t need to generate multiple sequences **at the test time**. Due to this, the proposed ASPIRE has smaller inference costs compared to previous methods.\n\n\n**Question 5**: A bit more analysis on the impact of the number of model parameters is warranted.\n\n**Answer 5**: Larger number of model parameters can lead to better prediction performance. However, it would also lead to higher training and inference costs. Besides, we show that even if the number of model parameters is very large (e.g. using the OPT-30B model. Refer to Answer 2 to the reviewer 7xz9 for details), the model still has limited self-evaluation performance. Thus, the proposed framework is useful for adapting LLMs to learn self-evaluation across various sizes of LLMs. \n\n**Question 6**: Should $k_c$ in line 316 and $k_w$ in line 319 be $z_c$ and $z_w$ respectively? \n\n**Answer 6**: $k_c$ and $k_w$ represent the number of correct / wrong outputs while $z_c$ and $z_w$ are a pair of tokens that represent the words “correct” and “incorrect” respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The methodology is simple yet interesting and performs better than other competing baseline methodologies. \n2. The authors have conducted an appropriate number of experiments on 3 benchmark datasets.", "reasons_to_reject": "1. The description of the metrics is limited. it would be desirable to have an explanation of the metrics used in the paper. Or at least a citation to the metrics would have been good.\n2. The training objective in Equation 7 would increase the likelihood of negative cases as well resulting in unwanted behavior. Should the objective be: \\mathcal{L}_{c} - \\mathcal{L}_{w}?\n3. The paper needs a bit of polishing as at times equations are clubbed together. The equations in Sections 4 and 5 can be clubbed together while introducing them.\n4. The paper motivates by the fact that we need to generate multiple sequences during the test time and progress to get rid of them. However, ASPIRE generates multiple answers during the training phase. This should be explicitly mentioned in the paper as it directly conflicts with the claim of not generating multiple sequences.\n5. A bit more analysis on the impact of the number of model parameters is warranted.\n\n\n", "questions_to_authors": "See the reasons to reject the paper", "paper_topic_and_main_contributions": "The paper proposes the ASPIRE methodology for Question Answering tasks. The ASPIRE methodology prevents an LLM from generating token sequences when they aren't unsure of the answer. First, each example in the training set is used to generate multiple answers given the question. A metric is then used to determine if the generated answers are correct answers or incorrect answers. The model is then trained to increase the likelihood of positive answers and reduce the likelihood of negative answers. The methodology shows strong empirical performance on 3 benchmark dataset on 2 metrics.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Should $k_c$ in line 316 and $k_w$ in line 319 be $z_c$ and $z_w$ respectively?\n"}, "extra_text": {}}, {"review_id": "DpjATfoAMQ", "reviewer": "Reviewer_x6AJ", "summary": "", "questions": "1. The model has been fine-tuned with the training data, and then the fine-tuned model is used to sample answers to the questions from the training set. How you address the problem that the model may have memorize the gold answers? \n\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, which has helped us improve our paper. We appreciate that you think our proposed method “is simple and effective, and the experiments are solid”. \n\nWe address all your concerns and questions in detail below.\n\n**Question 1**: The method may lack novelty. An essential part which is the M(y1, y2) function comes from a previous work (described in 204 ~ 213) (The previous work: Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation). The newly proposed techniques in this paper are only sampling from the model, and then training a classifier.\n\n**Answer 1**: We indeed use the $M(y_1, y_2)$ function to evaluate the similarity of the generated output sequence compared to the reference output sequence. However, using $M(y_1, y_2)$ is not our contribution. Our contributions include proposing a novel training framework for learning self-evaluation to enhance selective prediction performance; and conducting extensive experiments on three question-answering benchmarks using various LLMs to demonstrate the effectiveness of the proposed method in improving selective prediction performance. Although the techniques may look simple, they are novel and effective in improving the selective prediction performance. Also, the proposed method has good training sample efficiency (see Table 5) and introduces low inference costs. \n\n**Question 2**: The model has been fine-tuned with the training data, and then the fine-tuned model is used to sample answers to the questions from the training set. How do you address the problem that the model may have memorized the gold answers?\n\n**Answer 2**: Even if the model memorizes the gold answer for a given question, we can still sample wrong answers from it for the given question (e.g., via beam search). Since LLMs generate the output sequence in an auto-regressive way, even if the likelihood on the gold answer for the given question is high, the likelihood on some wrong answers might also be high. In our experiments, we observe that we could find several high-likelihood wrong answers (e.g., 8 wrong answers out of 10 sampled answers) via beam search for some questions where the LLM could generate a correct answer with greedy-decoding. The proposed method can sample those wrong answers and fine-tune the LLM to distinguish correct and wrong answers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is simple and effective. \n\nThe proposed method is easy to follow and can be implemented easily in practice. The method is mainly about sampling different answers from the model, and these answers may contain noises, then learning a classifier to classify the correct and wrong answers. \n\n2. The experiments are solid. \n\nTo verify the effectiveness of the method, the paper tested three QA datasets. According to the experimental results, the proposed method outperforms the baselines significantly. \n\nThe ablation study further explains how the proposed method works. ", "reasons_to_reject": "1. The method may lack novelty.\n\nAn essential part which is the M(y1, y2) function comes from a previous work (described in 204 ~ 213) (The previous work: Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation). \n\nThe newly proposed technics in this paper are only sampling from the model, and then training a classifier. \n\nThat's why I would think the novelty of the proposed method may be limited. ", "questions_to_authors": "1. The model has been fine-tuned with the training data, and then the fine-tuned model is used to sample answers to the questions from the training set. How you address the problem that the model may have memorize the gold answers? \n\n\n", "paper_topic_and_main_contributions": "This paper studies uncertainty measurement for large language models, specifically the selective prediction for large language models, i.e., the model learns to refuse to answer questions that the model is uncertain about the prediction. \n\nThe paper proposes an effective method for the studied problem, which first tunes the model on the task-specific data, and then uses the tuned model to sample answers(some answers are wrong, some are correct), finally, trains a calibrator to classify the correct and wrong answers. \n\nThe experiments demonstrate the effectiveness of the proposed method. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sqgJxWJsEG", "reviewer": "Reviewer_7xz9", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, which has helped us improve our paper. We appreciate that you think our proposed method is “effective when using medium-size LLMs and achieves the SOTA selective prediction performance on three question answering datasets”. \n\nWe address all your concerns and questions in detail below.\n\n**Question 1**: The authors claim that previous methods such as Self-consistency require multiple sampling, which is not effective. However, the method proposed in this paper requires two staging finetuning, and in addition, the method is task-specific, which means we have to tune the model for each task (and it also requires some training data), which results in a larger cost. In contrast, previous methods are more simple and effective.\n\n**Answer 1**: Although our approach introduces additional training cost, it is more effective compared to previous methods, including self-consistency, at test time. Our approach not only leads to better performance compared to previous methods, but also smaller inference costs, that can be on top of mind for many real-world use cases.\n\n**Question 2**: Experiments in this paper are not sufficient for the claim of \"improve selective prediction in LLMs\", as only GPT2-XL and OPT-2.7B are adopted. I'm not sure about the effectiveness of this method truly LLMs (at least > 7B). Especially, for large models, they show self-evaluation ability without further tuning, so the method proposed in this paper seems unnecessary.\n\n**Answer 2**: We don’t evaluate LLMs with the number of parameters greater than 7B due to computational constraints. However, the proposed approach can be applied to LLMs with any size if there are enough computational resources. Besides, we perform additional experiments to show that even for the pre-trained OPT model with 30B parameters, it still has limited self-evaluation ability (the AUROC of the Self-eval and P(True) methods are below 60%) on the three benchmarks we evaluate (see Table I). Thus, even for large LLMs with more than 7B parameters, further fine-tuning is still needed to boost their selective prediction performance on challenging question-answering datasets. \n\nBesides, the proposed method is useful for some practical applications where high selective prediction performance and low inference costs are desired after deploying the LLM. In those applications, practitioners prefer collecting some training data to fine-tune smaller LLMs to achieve high selective prediction performance than directly deploying very large pre-trained LLMs with limited selective prediction performance on the specific tasks. \n\n\n**Question 3**: Some of the baselines, e.g. Self-eval might be effective on LLMs but weak on small models, so the comparison in table1 is not fair.\n\n**Answer 3**: We perform experiments to compare the proposed ASPIRE with the adapted OPT-2.7B model and the baselines with the pre-trained OPT-30B model. The results in Table I show that  the proposed ASPIRE with the adapted OPT-2.7B model can significantly outperform the Self-eval and P(True) baselines with the pre-trained OPT-30B model in terms of the selective prediction performance (measured by AUACC and AUROC). Note that on the TriviaQA dataset, although the pre-trained OPT-30B model has better accuracy than the adapted OPT-2.7B model, the Self-eval and P(True) baselines with the pre-trained OPT-30B model have much worse selective prediction performance compared to the proposed ASPIRE with the adapted OPT-2.7B model. In summary, our results demonstrate that the self-eval approaches are not effective on LLMs with more than 7B parameters and applying the proposed ASPIRE to smaller LLMs can lead to better selective prediction performance compared to those self-eval approaches with much larger LLMs. \n\n**Table I**: *Comparing the proposed ASPIRE with the adapted OPT-2.7B model and the baselines with the pre-trained OPT-30B model. All numbers are percentages. Bold numbers are superior results.*\n| Dataset  | Model                            | Method        | Accuracy | AUACC   | AUROC  |\n|----------|----------------------------------|---------------|----------|---------|--------|\n| TriviaQA | Pre-trained OPT-30B              | Self-eval     | **39.36**   | 36.92  | 48.90 |\n| TriviaQA | Pre-trained OPT-30B              | P(True)       | **39.36**   | 36.20  | 45.63 |\n| TriviaQA | Adapted OPT-2.7B with $\\theta_p$ | ASPIRE (ours) | 29.21   | **55.06**  | **84.44** |\n| CoQA     | Pre-trained OPT-30B              | Self-eval     | 71.06   | 71.99  | 51.10 |\n| CoQA     | Pre-trained OPT-30B              | P(True)       | 71.06   | 71.59  | 51.31 |\n| CoQA     | Adapted OPT-2.7B with $\\theta_p$ | ASPIRE (ours) | **80.45**   | **92.63**  | **80.25** |\n| SQuAD    | Pre-trained OPT-30B              | Self-eval     | 41.41   | 46.24  | 57.26 |\n| SQuAD    | Adapted OPT-2.7B with $\\theta_p$ | ASPIRE (ours) | **83.27**   | **94.73**  | **82.60** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method proposed in this paper shows effectiveness when using OPT-2.7B and GPT2-XL, achieving the SOTA selective prediction performance on three question-answering datasets.", "reasons_to_reject": "1. The authors claim that previous methods such as Self-consistency require multiple sampling, which is not effective. However, the method proposed in this paper requires two staging finetuning, and in addition, the method is task-specific, which means we have to tune the model for each task (and it also requires some training data), which results in a larger cost. In contrast, previous methods are more simple and effective.\n2. Experiments in this paper are not sufficient for the claim of \"improve selective prediction in LLMs\", as only GPT2-XL and OPT-2.7B are adopted. I'm not sure about the effectiveness of this method truly LLMs (at least > 7B). Especially, for large models, they show self-evaluate ability without further tuning, so the method proposed in this paper seems unnecessary.\n3. Some of the baselines, e.g. Self-eval might be effective on LLMs but weak on small models, so the comparison in table1 is not fair.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the selective prediction of language models, that is force the models to assess the accuracy of their predictions and refrain from making wrong predictions. The main contribution lies in the method of generating self-evaluate data and using it to tune the model for selective prediction.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a selective prediction technique that discourages LLMs from predicting an answer when unsure. Towards this goal, an LLM is fine-tuned on task-specific data and multiple answers are sampled (some correct, some wrong) from the fine-tuned model. A classifier is trained to discriminate between correct and wrong answers, and the model is tuned to refrain from predicting wrong answers. The model incurs low inference costs and is also fairly sample-efficient during training.

All the reviewers have rated this work as good or higher on soundness. Concerns about whether the proposed technique is effective
for larger LLMs was addressed by the authors in their rebuttal. While inference costs are minimal and this is a desired feature in many use-cases, training efficiency does take a bit of a hit using the proposed technique with sampling multiple answers and task-specific finetuning of the model.
Decision: Accept-Findings