Uncertainty Quantification in Retrieval Augmented Question Answering
Authors: Laura Perez-Beltrachini
Mirella Lapata
Venue: Accepted by TMLR ()

Abstract: Retrieval augmented Question Answering (QA) helps QA models overcome knowledge gaps by incorporating retrieved evidence, typically a set of passages, alongside the question at test time.  Previous studies show that this approach improves QA performance and reduces hallucinations, without, however, assessing whether the retrieved passages are indeed useful at answering correctly. In this work, we propose to quantify the uncertainty of a QA model via estimating the utility of the passages it is provided with. We train a lightweight neural model to predict passage utility for a target QA model and show that while simple information theoretic metrics can predict answer correctness up to a certain extent, our approach efficiently approximates or outperforms more expensive sampling-based methods.

Published in Transactions on Machine Learning Research (09/2025)
Uncertainty Quantification in Retrieval Augmented Question Answering
Laura Perez-Beltracchni University of Edinburgh
lperes@ied.co uk
Mirella Lapata University of Edinburgh
mlap@edd.cm ac. uk
Reviewed on
Abstract
Retrieval augmented Question Answering (QA) helps QA models overcome knowledge gaps by incorporating retrieved evidence, typically set of passages, alongside the question at lest time. Previous studies show that this approach improves QA performance and reduces hallucinations, without, however, assessing whether the retrieved passages are indeed useful at answering correctly. In this work, we propose to quantify the uncertainty of a QA model via estimating the utility of the passages is provided with. We train a lightweight neural model to predict passage utility for a larget QA model and show that while simple information theoretic metrics can predict answer correctness up to certain extent, our approach efficiently approximates oulperforms more expensive sampling based methods
1 Introduction
Retrieval augmented Question Answering (QA) allows QA models overcome knowledge gaps at test time through access evidence in the form of retrieved passages (Lewis et. al. 2020 Guu et al. 2020 Tzacard
et al. 2024h. Recent work leverages external retrievers (Chen et al. 2017 Izacard & Grave 2021 and the language understanding and generation capabilities of Large Language Models (LLMs; Brown et al. 2020 Ouyang el al. 2024) to predict answers based on questions and retrieved passages which are provided as input context. In Figure 1 we show an example of a question (Who sings Does He Love Me with Reba?), retrieved passages, and predicted answers.
Retrieval augmented QA architectures have proven beneficial in increasing L.M performance on tail knowledge [zacard al. 2024 Mallen et al. 2023). reducing hallucinations in the generated answers (Shuster al. 2021 and even improving model calibration (Jiang et al. 2021b. However, there are various ways in which retrieval augmented QA can go wrong. The set of retrieved passages is far from perfect (Sciavolino et al.
2021 Yoran et al. 2024 Kasai et al. 2024) containing irrelevant, incomplete, or misleading evidence, the model might be under-trained to read certain passages and reason over those and the qucstion (Izacard et al,
2024 Liu et al., 2024bb Sun et al. 2025 and the question can be ambiguous or unanswerable (Kasai el al.
2024]. Ultimately, QA models may not follow the provided passages (Xie et al.] [2224] Joren et al.) 2025 When faced with uncertainty, QA models should ideally acknowledge it (c.g., by communicating it) rather than risk an incorrect response.
A good deal of previous work has focused on quantifying answer uncertainty in the context of closed-book book QA tasks, where the answer predicted based on a qucstion and the model's encoded knowledge. Sampling- based methods rely on output discrepancies among multiple predictors of the same input (Gal & Ghahramani 2016
Lakshminarayanan et al. (2077). They measure diversity on a set of answers (Kuhn et al. 2023) Chen Mueller 2024) sampled via temperature scaling (Guo et al. 2017), with larger variance indicating higher
Code and data available htap:://giituub.om. http://hwwb..o//aaaaaaaa/a/ggggo /lanhaide/aagg

---

Published in Transactions on Machine Learning Research (09/2025)
Who s sings does he love me wi with Reba?
Linda Davis
Does He Love You. Does He Love You "Does He Love You" is song written Sandy Knox and Billy Stritch, and recorded duet by American country music artists Reba NeEntire and Linda Davis, It was released in August 1993 as the first single from 2.77
Reba's 's albun "Greatest Hits Volune Two". It country music's 's several songs [cont.]
Patti LaBelle
Does He Love You. on Patti LaBelle's albun, Flane; The song features a vocal battle between two female narrotors who are in Both know that the being unfaithful then and wondering who he truly loves. The
love with the big-budget, Jon Small-directed video was filmed over 3 days in mid-1993. It begins with Reba in her dressing roon wearing 1ilac feather gown, where she sees picture
lover, which she glances at. [cont.]
-8.10
Figure 1: Example question from Natural Questions dataset (wiatkowski et al. 2019) with two top-retrieved passages using Contrieve=MSMMACCO (Izacard et al. 2022). On top of each passage, we show the answer generated by GEMMA2-9B when prompted with that passage and the question. The QA model answers correctly (green) only when prompted with the first passage. Numbers at the bottom right of each passage are utility scores predicted by our model (higher values indicate more useful passages).
uncertainty. LLM-based methods rely the QA model's own judgment of uncertainty (adavath et al. 2022 Lin et al. 2022 Tian et al., 2023). Through prompling, the model is encouraged to express its uncertainty (e.g., 0.5 or 'almost certain') either alongside the predicted answer (Lin et al. 2022 Tian et al. 2023) or
after generating it (Kadavath et al.) 2022 Tian et al., 2023). None of these approaches has been applied in the context of retrieval augmented QA (Shorinwa et al. 2025).
In this paper, we focus on answer uncertainty estimation in the context of retrieval augmented QA. We hypothesize that a passage is useful, a model can correctly answer questions based on it. If passages are informative and prime the QA model towards appropriate knowledge Geva et al. (2021), we expect it to produce a correct answer. On the contrary, if passages are irrelevant or misleading and the question falls outside the QA model's knowledge, it is likely to produce an incorrect answer, either factually inaccurate or entirely fabricated. Importantly, this notion of ulility is based on how the target QA model will answer with the provided passages and not on what an external judge (e.g., entailment model) thinks about them. We quantify the utility of retrieved passage with a small neural model trained on utility judgments obtained by observing the target QA model's answering behavior. We borrow ideas from direct uncertainty quantification approaches (Van Amersfoort et al., 2020 Lahlou et al. 2023) but do not decompose uncertainty outline shifts in the input distribution. We make utility predictions for each retrieved passage which then use to estimate the uncertainty of the QA model when prompted with set thereof.
We evaluate our approach on short-form informatiin-ssekknn QA tasks [Rodriguez & Boyd-Graber 2021 (see Figure | for an example). Results on six datasets show that our uncertainty estimator is comparable or outperforms existing sampling-basse methods while being more test-time efficient. Sampling-basee solutions are expensive for in-production QA systems, in terms of latency and cost (e.g., QA engines built on top of proprictary language models would need to process relatively long prompts). Morcover, our experiments reveal that variation is less prominent in model answers in the context of retrieval augmented QA (e.g., the QA model is more confident on incorrect answers supported by retrieved passages in the prompt). Our contributions can be summarized as follows:
We quantify QA model uncertainly via estimating the utility of the passages is provided with. We (contrastively) train a small neural model on utility scores obtained through combining accuracy (is the generated answer correct?) and entailment (is the generated answer supported by the passage?) metrics.
Our approach is lightweight, improves upon more expensive sampling-basel methods, and is not tied to the retriever (and passages) used to prompt the QA model.
2

---

Published in Transactions on Machine Learning Research (09/2025)
2
Related Work
Uncertainty Quantification for Question Answering Several methods have been proposed to predict answer uncertainty the context of closed-book QA; however, none of them has analysed uncertainty in retrieval augmented QA models. Many of these methods rely on the assumption that variation in the model's
output is an expression of its uncertainty (Kuhn et al. 2023: Farquhar et al. 2024: Chen & Mueller] 2024). For example, some approaches (Kuhn et al. 2023 Farquhar et al.) 2024) first cluster answers with similar meaning (in a sample) via Natural Language Inference (NLI) before computing entropy while other work (Cenn& & Mueller, 2024] focuses on black-bor models; they also compute similarities in the set of answers but associate them with the model's self- judgement of confidence. In a recent study, [Soudani et al. (2025) show that, in the context of retrieval augmented QA, these methods tend to exhibit overconfidence and sensitivity to the input context. Indood, our experiments corroborate their analysis. We find that sampling- based methods exhibit less variation in a retrieval set setting, and do not offer significant. advantage over methods such as perplexity LLM prompting for self-assessment (Kadavath et al., 2022). unlike in closed-book settings,
Sequence entropy methods (Kuhn et al., 2023 Farquhar et al. 2024 focus on detecting incorrect answers stemming from arbitrary fluctuations in model outputs (referred to as confabulations). Our approach exlends to additional error sources, including incorrect training data or mislcading evidence. Hou et al. (2024) focus on quantifying aleatoric uncertainty (i.e., uncertainty in the data) caused by ambiguous questions, an approach that could be combined with ours. Sampling- based methods are expensive to run at inference time for a production QA system; they require several inference steps in addition to performing similarity computations, which can become more complex with longer answers (Zhang et al.) 2024b). In contrast, our approach is light-weight at inference time, but requires training data with observations pertaining to the accuracy of the target model. Our approach optionally uses an NLI model at training time, depending on the chosen objective. In Section G we further summarise key differences between our approach and existing uncertainty estimation methods in the context of retrieval augmented QA.
Judging the Quality of Retrieved Passages Previous work has analysed the quality of retrieved passages (Yu et al. 2023 Asai ct al., 2024 Wang et al., 2024 Xu et al.] 2024 Yoran et al. 2024 as they can be irrelevant or misleading. Asai et al. (2024) make use of an external critic model to create training data exemplifying cases where a question requires retrieval (or not) and, in the case that retrieval is needed, whether retrieved passages contain the information (or not) to formulate the answer. Note that the usefulness judgment is made by an external critic. Then, a QA model (i.e., the RAIf-RAG model) is trained on this data to learn to reflect by itself whether passages are supportive and relevant and to predict special tokens indicating this. While it is possible to derive uncertainty from those special tokens' probabilities, they only reffect Self-RAG's uncertainty state. Our proposal is more general and aims at predicting answer uncertainty in zero-shot QA models (e.g., instruction fine-tuned LLMs). Thus, using Self-RAG special tokens would be like using an off-the-shelf sophisticated classifier (i.c., specialized textual entailment) to predict the uncertainty of zero-shot QA model (Yoran et al. 2024). In contrast, predict uncertainty by observing the errors of the target QA model (in a zero-shot setting). Other work creates auxiliary tasks around retrieved passages enforcing the QA model to reason on them; e.g., by taking notes about each passage (Yu et al., 2023) or generating passage summaries (Xu et al., 2024p. These methods also use extreme-scale LLMs to generate training data for fine-tuning a retrieval augmented QA model. Park el al. (2024) select low and high quality in-context examples in order to instruct the LLM to reason on input passages. Concurrent work by [Jren et al. (2025) defines the concept of context sufficiency, i.c., whether the content in the set of retrieved passages is sullicient to answer the question and uses an external judge lo assess this. However, the external judgment might not agree with the answering behavior of the target QA model (i.e., the judge may indicate that the context is sufficient but the model may still incorrectly answer). All these approaches aim at improving QA performance while our primary goal is modeling QA uncertainty.
Using a Separate Model to Predict Confidence Our passage utility predictor is related to methods aiming to estimate error directly (Lahlou et al.) 2023), e.g., by training a secondary model to estimate target model loss; instead, our predictor is trained with sequenco-level metrics, i.c., accuracy and entailment, which measure error indirectly. Some work (Kamath et al. 2020 Zhang et al. 2021 predicts answer correctness
3

---

Published in Transactions on Machine Learning Research (09/2025)
in the context of Reading Comprehension (the task of generating answer based on single supportive passage). However, as there is no retrieval involved, the input passage is by default useful, and the main goal is to detect answer uncertainty due to the QA model being under-trained. In our setting, the number and quality of passages varies leading to additional sources of uncertainty (o.g., mislcading information). Some approaches train specific model to predict answer confidence scores (Dong et al. 2018 Kamath et al. 2020) Zhang ct al. 2021 Mielke et al., 2022) by incorporating various features from the input and model output. Our approach predicts answer uncertainty directly from individual passage utilities and its predictions could also be combined with other features (e.g., output sequence probability).
3 Modeling Answer Uncertainty
We formally define retrieval augmented QA as follows. Given question x and a set of retrieved passages R- = (pI.P2.- } obtained with retricver R, an LLM-based QA model M is prompted to generate answer y to question r noken-by-tokee according to its predictive distribution:
M
Pfyl,, M) II 20, M.. t=1
(1)
We wish to estimate M's uncertainty (i.e., chance of error) of generating y given x and R.
When a retrieved passage is useful to answer given question (such the first passage in Figure for the question Who sings Does He Love Me with Reba?), the QA model is likely to be confident when generating the answer (Linda Davis). When the passage is not useful (such as the second passage in Figure [ the QA model is likely to be uncertain and provide an incorrect answer (Patti LaBelle). Our hypothesis is that the utility of each passage p in R is indicative of the QA model's uncertainty in generating y, when prompted with R. If there are passages in with high utility (c.g., in Figure [ the first passage is useful to answer the question), it is likely that the QA model will be confident when generating answer y. In contrast, if all passages in R have low utility, it is likely that the QA model will be uncertain when generating the answer.
The core of our approach is estimating the utility UM of individual passages for a target QA model M. Specifically, we develop an estimator (s, + (M p}) for each passage E R (Section 3.1. We then leverage the predicted passage utility UM to estimate M's uncertainty when generating answer y to question x based on evidence R, (x, Rt) + uM((F, (fx,,,,)))))) RJ) (Section 3.2
3.1 Passage Utility Prediction
Intuitively, a retrieved passage p is useful for a QA model M if M can correctly answer question when prompted with However, the model's dependence on p may vary. In some cases, M may generate the correct answer even if p does not explicilly contain it, instead it positively primes the model to draw upon its memorized knowledge. In Figure D the first passage has high utility because the QA model generates a correct answer when prompted with it, and explicitly states that "Linda Davis sings alongside Reba. McEntire". Tn contrast, the second passage, while related to the question's 's topic, is not useful. The QA model struggles to answer correctly when prompted with it, suggesting uncertainty. Since this passage does not provide helpful information and leads to incorrect answers, its utility is low.
We estimate the utility of passage p in answering question x for QA model M by combining two key measures, accuracy and entailment:
(a(y) + +e())
UM
2
Accuracy, denoted as a(y), indicates whether the generated answer y is correct, while entailment, denoted as e(y), measures the degree to which p supports y. Accuracy is determined by an evaluator A, and entailment is assessed using an NLI model E. The combined passage utility UM ranges between 0 and 1, given that a takes values in {0. 1} and e falls within the [0, 1] interval. The intuition behind this composite term UM is encourage a ranking of passages that spans broad spectrum: from highly useful ones (producing correct.
4

---

Published in Transactions on Machine Learning Research (09/2025)
answers with strong entailment); through more ambiguous cases (e.g.; correct answers with weak entailment or incorrect answers with strong entailment), to clearly unhelpful passages (producing incorrect answers with weak entailment)
We train a lightweight neural model on dataset DM = {(x,p, uM)) to predict passage utility scores, {a,2] + (Mr(f,,p)); We construct by pairing input questions x and passages p with utility scores um which we obtain after running M on examples (a, p) and computing observed answer accuracy and entailment scores from the QA model M. We retrieve |R| >> 1 passages per question to ensure diverse range of usefulness and create training instances {(z, Pt, 24) p. E R} under model MM. We leverage this data to train the passage utility predictor with contrastive learning scheme. Specifically, if two passages pt and Pj belong to R and pi is more useful than Pj for answering question x, the predicted utility score v should be higher than Vy by margin m, ensuring that pe is ranked above pj. We train the utility predictor with the following ranking objective:
Frank Σ max - =2( w2) + m)),
(3)
where a controls the gold order between p; and Py (e.g., = 1, Ps has higher utility, and conversely v = -1 indicates the opposite ordering) and m is a hyper-parameter.
We train the passage utility predictor with a Siamese neural network consisting of a BERT-based encoder (Devlin et al. 2019) followed by pooling and two MLP layers stacked on top of BERT outputs (Fang et al. 2024). The output layer computes the utility score as v; W,h+ + b, where 4- is the vector representation for pz) from the last hidden layer (the L-th layer) of the network. At inference time, we compute single utility score for each passage. We provide implementation and training details in Section 4
To strengthen the role of accuracy prediction as a training signal and regularize the range of utility values learned by the ranking scheme, we combine the ranking objective in Equation 33 with the following Binary Cross Entropy (BCE; Sculley 2010) objective:
LBCE=
Σ
la x log(p(z,p:)) + (1 a) log(1 p(x.
(4)
+la x log(p(ı, + (1 a) log(1 p(z, p;)]],
where p(x, p.) sigmoid(u;) and a is the target accuracy label a(y) under model M taking values in the set {0, 1]. We train the utility predictor with the following combined objective where À is a hyper-parameter to adjust the LBCE penalty:
C Crank + ABCE
(5)
Both ranking and BCE objectives are compatible with gold annotations that could be provided in active learning interactive settings (Simpson al.] 2020;; Fang et al. 2024). For example, moderators of the QA system would provide judgments the accuracy of the answers it predicts (e.g., correct/ (ineeee/iiorreet))) and the extent to which these are supported by the retrieved passages (c.g., not supported to fully supported).
3.2 Answer Uncertainty Estimation
For our QA task, we want to define an estimator {z, R} + uM((2, R3) which quantifies the uncertainty of model M when generating answer y for question x based on a prompt with passages R. We propose estimating u. directly from the utility scores of individual passages in R. The key intuition is that the higher the utility of one (or more) passages, the less uncertain M will be when generating answer y. Conversely, if all passage utilities in R are low, it is more likely that M will be uncertain about the answer. Specifically, we estimate answer uncertainty uM by taking the maximum utility score among the passages in R:
um((x, R}) = max(u
p E R).
66)
Note that for the pairwise ranking loss, using the sum BM = a(y) e(y) would be equivalent. In Appendix we include ablation experiments with different, implementations the Passage Utility score eM-
5

---

Published in Transactions on Machine Learning Research (09/2025)
Our approach to aggregating passage utilities is intuitive and simple. However, QA models might be sensitive to factors relating to how they are prompted, such as the number or order of passages (Liu et al. 2024b) Xie et al., 2024]. In Appendix C we examine such confounds more closely, comparing question answering accuracy when models are prompted with individual passages in R (our aggregation approach) versus the entire set R. The study shows that there little disagreement bet the two methods and that it is possible to approximate answer uncertainty when prompting with |R] passages while avoiding the combinatorial complexity of estimating uncertainty over all possible combinations of input passages.
4 Experimental Setup
4.1 QA Tasks and Models
We evaluate our approach to predicting answer uncertainty on short-form question answering tasks. Specifically, on the following six datasets: Natural Questions (Kwiiakowski et al. 2019), TriviaQA (Joshi et al. 2017
WebQuestions (Berant et al. 2013), SQuAD (Rajpurkar et al. 2016). and PopQA (Mallen et al. 2023
We also evaluate on RefuNQ (Liu ct al. 2024a), a datasct with unanswerable questions about non-cxisting entities. In Appendix F.1 we describe each dataset, provide example questions, and make available details about the splits used in experiments which follow Lee et al. (2019).
We consider backbone instruction fine-tuned LLMs from different families of similar size. These are Llama- 3.1-8B- Instructt (Al@Meta) 2024). Mistral-7B T-rr-B---nnttt....33 (Jiang et al. 2023). and Gemmma- (Riviere ltt al.) 2024). We also assess answer uncertainty quantification performance on QA models of the same lamily but with different sizes. To this end, we additionally evaluate on Gemma2-27B-it For all QA models, we use a simple prompt including the retrieved passages and the question in the context; the prompt is shown in Table [4 in the Appendix. The QA models' answer is the most likely answer generated with greedy sampling at temperature equal to O. Following previous work on retricval augmented QA, use Contriever-MSMMARCO (Izacard et al. 2022) as our external relriever (Asai et al. 2024) and the larget QA models are prompted with [R] = 5 passages (Yu et al., 2023 Asai et al. 2024 Xu et al. 2024). In
Appendix we provide more details about inference and passage retrieval.
4.2 Evaluation
QA Accuracy A precise metric for measuring accuracy is key when evaluating the quality of uncertainty estimation. Token overlap metrics are imprecise and can over- or reder-estimate accuracy (e.g., 5 will not match five). Thus, we rely on a L.MM-based accuracy evaluator to create training data for the Passage Utility predictor (i.e., A in Section 3.1 and to evaluate retrieval augmented QA performance. We use Owen2-72B-Instruee (Yang et al. 2024 and the prompt proposed by Sun et al. 2024 to obtain accuracy judgments. Details about the LLM evaluator can be found in Appendix
Uncertainty Estimation To assess the quality of answer uncertainty prediction, we follow Farquhar et al. (2244) and report the Area Under the Receiver Operator Curve (AUROC) on detecting incorrect answers (i.c., answers with high uncertainty). In Appendix G.3 we also report the area under the 'rejection accuracy' curve (AURAC) which captures the accuracy a model would have if it refused to answer questions with highest uncertainty. Rejection accuracy is essentially the model's accuracy on the remaining questions. In the main results section, we focus on selective answering performance when models answer 80% of the least uncertain questions versus when always answering. We provide implementation details in Appendix F.2
4.3 Methods
Passage Utility Predictor We train a Passage Utility predictor per QA model and QA task. For cach lask, we curate dataset DM = {x, p, uM)} to train and evaluate a Passage Utility predictor for QA model M. We use the training (and development) questions available for each QA task, considering the top five retrieved passages for each question (i.c., [R] = 5). Note that |R] is a hyper-parameter and other values would be also possible. Larger sizes of [R| would yield more training data, since the Utility predictor takes individual passages her with the question) as input. The target QA model M is first prompted with
6

---

Published in Transactions on Machine Learning Research (09/2025)
passage p R and question to generate answer y. Then, we annotate passages p with utility score computed with the accuracy evaluator A and entailment judge E on generated answer y (Section 3.1p. We use an xxlage model (Lan et al. 2020) optimized on MNLI (Williams et al. 2018 and the VitaminC dataset (Schuster et al.) 2021j. We provide more details about the curated datasets and training of the Passage Utility predictor training in Appendix F.2
Comparison Approaches and Baselines There exist several uncertainty estimation methods which we group in two categories based on whether they require access to logits simply model outputs (see) Fadeeva et al. 2023 for additional methods). We choose the highest scoring ones compare with here and include additional results in Appendix G.3 for completeness.
Information- Theoretic Measures. We compare against uncertainty estimation methods that are based on the predictive probabilitios of the target QA model. Let denote an answer generated with probability ply|æ, R; M) which is computed as:
lol
plyl,, R: M)
II p(w.le1..-1, R;
The Perplexity (PPL) model M boils down to calculating token-level entropy as based on the average negative log-likol loglliiel of the generated tokens:
PPL(I, R, M) exx {- 1 log plu 21,
(8)
Regular entropy, on the other hand, is computed over sequences, quantifying the entropy of the answers. It defined as E[-log (x R; M)] where the expected value, E, is computed on sequences y sampled from the conditional distribution P(Y a R; M), where random variable denotes the answer sequences, and and R are the input question and retrieved passages, respectively. In practice, regular entropy is approximated via Monte-Carlo integration, i.e., sampling N random answers from P(Y|æ, R;M):
RE(x, R, M)
N Σ log P(y x. (x,
1
where P(y" R. is the length normalised version of P(y" M).
Kuhn et al. 2023 propose Semantic Entropy, a variant of regular entropy that disregards uncertainty related to the surface form of the generated answers. The method works by sampling several possible answers to cach question and grouping the set of N samples into M clusters (with M s N) that have similar meanings (which are determined the basis of whether answers in the same cluster entail each other bidirectionally). The average answer probability within each cluster is:
SE(r. R,
E m=1
Pm (2, log Pm (x,
(10)
where (x, is estimated as follows:
(m) Eyec., P(u/e. R;M) IM Eyec... Plylo, R;M)
(11)
LLM-based Measures. We compare with p(true) which uses the same LLM-based target QA model to assess whether the answers it produces are correct (Kadavath et al. 2022). We follow the p(true) variant used in previous work (Kuhn et al. 2023). The QA model is prompted with the question and a set of candidate answers (consisting the most likely answer and a sample of N answers) and is instructed to respond whether the most likely answer is true false (i.e., correct/ correct/incorenne /incorrctt. The score produced by this approach is the
7

---

Published in Transactions on Machine Learning Research (09/2025)
probability of model M generating the token True. This method needs several in-context examples to work well; following Kuhn et al. (2023), we use 20 in-context examples. Note that since our backbone LLMs are recent models with a large context (unlike [uhnn et al. 2023). all 20 examples are fed in the context making p(true) an expensive but very strong approach. In the context of retricval augmented QA, we modify p(true) to include in the prompt the set of retrieved passages for the quest ion of interest. We provide the prompt used by p(true) in Appendix F.3 Note that p(true) can be considered as a specialized powerful entailment. model and thus we do not include entailment based methods relying on off-the-shelf NLI models which have been shown to perform poorly (Yoaan et al., 2024b.
For approaches that require sampling, we follow previous work (Farquhar et al. 2024 and take N = 10 samples, which we generate with multinomial sampling. We set the sampling temperature to 1, with nucleus sampling (P = 0.9; Holtzman et al. 2020) and top-K sampling (K = 50; Fan et al. 2018), and use a different random seed to draw each sample. We provide further details about inference in Appendix F.2 and report inference costs for cach approach in Appendix
5 Results and Analysis
Passage Utility is effective across model families, sizes, and QA tasks. Table ! summarizes our uncortainty estimation results (test set) with four QA models (CEMMA2-9B/22) LLAMA-3.1-81) and MISTRAI--BB-OO.3) across six QA datasets (results on the development set are included in Appendix G.3 We boldface the highest AUROC value for each QA model and datasot pair and mark with the next best value that is significantly different from it at < 0.05. We use the paired De Long test (DeLong et al. 1988 to compute whether two AUROC values are significantly different
In general, answer perplexity (PPL) performs rather poorly, especially with GEMMA2-9B/27B. Perplexity is likely to underperform with less calibrated models, such as those which have undergone instruction tuning (Tian al. ).23). Regular Entropy shows little improvement upon PPL but by ignoring surface form choices and focusing on meaning, Semantic Entropy improves AUROC scores. p(true) performs well at detecting answer uncertainty matching or surpassing Semantic Entropy. Overall, we observe that the gap among these methods' performance is lower than in the context of closed-book QA studied in previous work (Farquhar let al. 2024). We hypothesize that, on one hand, our recent QA models admitting more in-context examples benefiting p(truc) and, on the other hand, that retrieved passages in the prompt make QA models' outputs less varied. Our Passage Utility approach performs on par or outperforms all other methods with a single small-model inference step on each input passage.
Passage Utility performs particularly well on challenging question answering tasks represented by datascts like PopQA and RefuNQ. In these cases, our light-weight uncertainty estimation model works better than p(true) which requires the same QA model (i.e., the same backbone LLM) to judge the correctness of its own generated answers. We speculate that for questions with high uncertainty, i.e. where the model does not have the knowledge to answer (e.g., questions about non-existing concepts in RefuNQ), it confidently generates a response and also fails at assessing it. We attribute the Passage Utility's success the fact that it has been specifically trained to detect situations where the target QA model is prone to answer incorrectly (i.c., when provided with retrieved passages of lower relevance). The six QA tasks pose different retrieval challenges. On TQA, retrieval results are often of good quality: for 73% of the questions, the top-5 retrieved passages contain the gold answer string. In contrast, on PopQA the percentage reduces to 63% and on RefuNQ the quality of retricval is deliberately low (as it consists of unanswerable questions). Across models (Table I. our approach is comparable to p(true) when retrieved passages contain the answer and excels in cases of low quality retrieval.
Passage Utility also performs well with different QA model sizes (within the same family), i.e., GEMMA2-99 and 27B (Table D.. We observe a noticeable decrease in performance for most information- thoommttin--thooetiiic models when using the biggest GEMMA model (27B). We attribute this to the fact that the 27B model more confidently makes less errors and its calibration may be affected more by the fine-tuning step TTan et al. 2023). ). p(truc),
on the other hand, benefits from the largest model's context understanding and memorized knowledge.
WWe use the library in https:://gitii /ttggs/////thub....// thub. com/ ccwp//rrritannyMMsstakk.i to compute significance scores.
8

---

Published in Transactions on Machine Learning Research (09/2025)
GEMMA2-9B WebQ
GEMMA2-27B WebQ
NQ
TQA
SQuAD PopQA RefuNQ AVG
NQ
TQA
SQuAD PopQA RefuNQ
AVG
PPL
0.64
0.68
0.52
0.53
0.59
0.51
0.58
0.64
0.50
0.53
0.59
0.58
0.51
0.56
p(true)
0.73
0.75*
0.67
0.63
0.81*
0.62
0.70
0.77
0.83
0.67
0.68
0.78
0.60
0.72
Regular Entropy
0.66
0.69
0.54
0.56
0.61
0.51
0.60
0.67
0.54
0.55
0.59
0.62
0.51
0.58
Semantic Entropy
0.70
0.73
0.57
0.64"
0.73
0.59*
0.66
0.69
0.62*
0.59*
0.63
0.66
0.58
0.63
Passage Utility
0.76
0.85
0.69
0.78
0.86
0.79
0.79
0.73*
0.82
0.69
0.80
0.85
0.78
0.78
LLAMA-3.1-8P
MISTRAL-7B-V0.3
NQ
TQA
WebQ SQuAD PopQA RefuNQ AVG
NQ
TQA
WebQ SQuAD PopQA RefuNQ AVG
PPL p(true)
0.75
0.80
0.68
0.74
0.83
0.60
0.73
0.63
0.71
0.57
0.65
0.64
0.62
0.64
0.79
0.88
0.74
0.77
0.85
0.67
0.78
0.73
0.82
0.68
0.74
0.75
0.68
0.73
Regular Entropy
0.76*
0.81
0.71
0.78
0.83*
0.65
0.76
0.64
0.75
0.62*
0.65
0.66
0.60
0.65
Semantic Entropy
0.72
0.82*
0.66
0.78*
0.81
0.59
0.73
0.66
0.78*
0.66
0.73
0.74
0.61
0.70
Passage Utility
0.77
0.82
0.72
0.83
0.87
0.81
0.80
0.74
0.83
0.68
0.82
0.85
0.80
0.79
Table 1: AUROC values for QA models based on GEMMA2-9B/ / 27B, LLAMA-3.1-8B, and MISTRAL-7B-VO.3 on Natural Questions (NQ), TriviaQA (TQA), WebQuestions (WebQ), SQuAD, PopQA, and RefuNQ test sets. The best values (per model and dataset) are highlighted in bold; we also mark with next best values which are significantly different using the paired De Long test (p < 0.05). For example, on TQA with GEMMA2-9B, p(true), the second best performing is significantly different from the Passage Utility which performs best and by extension models with lesser values than p(truc) are also significantly different.
ChaSmAa-227 -7BB
pilces)
MCROC
Kirn)
PPL
Regular Entropy
Passage Utillty
0.7
p(true)
Semantic Entropy
SIE
Average QA Accuracy
0.70
0.7
0.5 700
L5k
0.65
Number of Inferenns Tiiees
Figure 2: Average AUROC across our six development sets (y-axis) with respect to number of input tokens inference time (x-axis) and number of parameters (size of the circles). We present results for perplexity (PPL), Semantic Entropy (SE), p(true), and Passage Utility (PU). We exclude Entropy (which is close to SE) for readability. We compare the smaller GEMMA2-9B and its bigger version 27B. Thinner circles positioned in the left corner are better.
0.6
G9B
G27B
L8B
M7B
Model variants
Figure 3: Average accuracy (across our six test
sets) with GEMMA2 sizes 9B and 27B (G9B, G27B), LLAMA-3. 1-8B (LSB), and MISTRAL-7B- v0.3 (M7). Black dots: QA models always answer; colour dots: QA models answer 80% of the cases they are most confident about.
It important to note that our approach lite-weight at inference time. In Figure 2 we report average AUROC per method with respect to the number of input tokens at inference time and the number of parameters involved in uncertainty estimation. We report scores for GEMMA2-9B and its bigger version GEMMA2-27B. As can be scen, our approach which is based on BERT encoder (Devlin et al. 2019) with 110M parameters and a small number of input tokens achieves on aggregate better performance than more expensive approaches. For the 27B QA model, p(true) edges closer to passage utility, however, at the expense of thousands of input tokens and ~2 billions more parameters. This indicates that p(truc) will be less efficient in QA settings where latency and cost are critical. In Appendix B we provide a cost analysis in terms of model calls.
9

---

Published in Transactions on Machine Learning Research (09/2025)
Passage Utility leads to selective answering. Model uncertainty can be used to decide whether to provide an answer to question or not. Figure g shows average accuracy when the target QA models choose to answer 80% of the cases they are most confident about. For comparison, we also show QA accuracy when always answering, i.c., black bold dots. All uncertainty quantification approaches improve model accuracy. Across different LLM QA backbones, Passage Uuility performs on par with or better than more expensive uncertainty cstimation approaches such as p(true). For instance, when looking at selective performance according to Passage Utility, the biggest GEMMA2-27B model improves by +9 points (0.74 vs 0.65). In Appendix
report the full set of selective accuracies at different thresholds.
Passage Utility shows potential for retricval reranking. To further assess the quality of the Passage Utility scores and to highlight their potential for retrieval reranking we carry out the following ablation study. Since most previous work (Asai et al. 2024 Xie et al.) 2024 Yoran et al.) 2024) on retrieval augmented QA prompts the QA model with the top-5 (or less) input passages, we hypothesize that our passage utility score could be an effective reranking method after retrieval (Nogueira et al. 2019 Ma el al., 2024 Yao al., 2024).. We test this hypothesis by computing the accuracy of the CEMMA2-9B QA model when prompted with the top-k passages (with k in the range of (5, 3, 1}) out of a sample of [ = 10 passages provided by an external retriever.
In Figure we compare performance under two passage-ranking strategies: one based on relevance scores from an external retriever (gray), and the other based on the QA model's self-assessed utility of individual passages, following Yao al. (2024). We report two self- celf-assessment variants, one using the predictions of the Passage Utility model (red) and another one based on the perplexity of the QA model when answering with individual retrieved passages (bluc). Passages that yield answers with lower perplexity should be ranked first. Figure 4 shows average accuracy values across five development sets (NQ, TQA, WebQ, SQuAD, and PopQA), at different cutoff values (k). As can be seen, the QA model achieves higher accuracy when passages are ranked according to their utility. This finding suggests that Passage Utility scores indeed reflect which passages are useful for the target QA model.
Passage Utility struggles with multi-hop questions. Our approach estimates uncertainty by predicting individual passage utility and selecting the maximum utility score from set of retrieved passages. A potential limitation of this method emerges in multi-hop questions that require evidence from multiple passages (Yang) et al. 2018 Pal et al. 2022). Specifically, the QA model is unable to answer correctly when prompted with any individual passage (resulting in uniformly low utility scores), yet succeeds when given the complete set of passages (in which case the maximum utility score over individual passages fails to reflect this outcome). To quantify this limitation, we evaluate our approach on the widely used Pot dataset (Vang et al., 2018). using the splits as provided by Trivedi et al. (2023), and the same retrieval scttings as defined above (Section Table 2 shows AUROC values for all uncertainty estimation methods. In Appendix E we provide analysis on small set of multi-hop questions making use of synthetic HotPotQA data.
Across models, our approach is on par with p(true) and sequence entropy, and better than perplexity (Table 2 Manual inspection of 100 examples from the development. set with GEMMA2-9B as the target QA model, reveals two major trends. Firstly, the QA model frequently (48 cases) manages to correctly answer multi-hop questions using only one of the required 'hop' passages, often not needing the entire set. This observation aligns with recent findings in .Jren et al. (2025). Secondly, in numerous instances (29 cases), the retrieved passages did not contain any useful evidence, leading the model to answer incorrectly even when prompted with the full set. This underscores the inherent difficulty of effective retrieval for complex questions. Many studies (Jeong et al. 2024 Trivedi et al. 2023 Lin et al. 2025) tackle this challenge through sophisticated QA pipelines. Notably, certain approaches decompose complex questions into sub-questtan that can be answered independently. In such multi-hop pipelines, our approach could be naturally applied at the sub-question level.
6 Discussion
Key Properties and Usability Scenarios Each uncertainty quantification approach comes with its own advantages and limitations. This entails that the choice of a specific method depends on criteria like available resources, desired latency, and the necessary level of control and trustworthiness for the QA system. For
10

---

Published in Transactions on Machine Learning Research (09/2025)
RAG RAG+PU RAG+PPL
Average Accuracy
G9B
G27B
LSB
0.6
PPL
0.61
0.59
0.73
0.70
p(true)
0.75*
0.78
0.81
0.78
0.5
Regular Entropy
0.64
0.60
0.77
0.71
Semantic Entropy
0.70
0.65*
0.74
0.73
Passage Utility
0.78
0.77
0.77
0.74"
Number of Retrieved Passages
Table 2: AUROC values for QA models based on GEMMA2-9B 922BBB 8B, and MISTRAI-7BB-V.33 on Pot test sct. Best values per model are high- lighted in bold; we also mark with" next best values which significantly different using the paired De Long test (p < 0.05).
Figure 4: Average RAG accuracy with GEMMA2-9B across the five QA development sets. Points on the x-axis correspond to different context sizes, when laking the top-k, passages according to query rele- vance (gray) and self-assessment via perplexity (PPL; blue), and Passage Utility (PU; red).
/2BB, LLAMA-3.1
Low latency
No Training Dala
No Fine-tuning
Recognises
Mitigates
Erroneous Evi- confidence
dence
PPL p(true)
Semantic Entropy
Passage Utility
Table 3: Categorization of the uncertainty estimation approaches studied in Section according to different properties (table headers).
example, in high-stakes applications, method that favours higher rates of false positives (thereby allowing human intervention) and reduces the chance of overconfident false negatives would be better, even if it requires training data and regular updates.
In Table we summarise existing features (and limitations) of various uncertainty estimation approaches as we compare them to our work. The first column reports latency, as previously discussed in Figure 2 Retrieval augmented QA models can err as result of being exposed erroncous sources such as mislcading passages (Xie et al.) 2024) or inaccurate training data (Vu al. 2024). Information-theoreti methods are not equipped with explicit mechanism to deal with these cases (Farquhar et al. 2024 Soudani et al. 2025
Moreover, these methods are also known to suffer from over-confidence (Simhi et al. 2025 Soudani al.
2025 Sung et al. 2025). While p(truc) may be able to detect these challenging cases to certain extent, Passage Utility can be specifically trained to recognise them. In terms of supervision, both p(true) and Passage Utility require task-specific training examples, and the performance both approaches deteriorates on out-of-distribution examples (Table 8).
Training Data Requirements Our approach requires question-answer pairs curate a dataset with retrieved passages and utility scores for training. However, general task specific training datascts could be generated semi-automatically Li & Zhang (0024); Wei et al. (2024) Moreover, in experiments that in some QA tasks such as WebQ or PopQA this training data can be relatively small, i.e., in the region of 2.5k or 10k respectively.
show
Fine- TunnnT Requirements Our approach, by design, requires fine-tuning to adapt to new QA tasks or models, as core aim model the accuracy behavior of the target QA model. To enhance versatility across QA tasks, unified training set encompassing diverse QA tasks could be compiled to train a single passage utility predictor. More practically, advanced training schemes promoting generalization, such
11

---

Published in Transactions on Machine Learning Research (09/2025)
meta-learning with varied set of QA tasks and QA model examples, could be employed to develop single passage utility predictor.
In addition, while the utility predictor may necessitate recalibration for distinct QA models or tasks, its performance on aat-of-distribution scenarios (Appendix D establishes it as a robust warm-up base model. This allows for subsequent fine-tuning with a reduced number of target examples (Kamath et al. 2020) Zhang et al., 2021). Finally, as the utility predictor relies on small model, the cost of fine-tuning in terms of resources and time is low.
7 Conclusions
In this work we address uncertainly estimation in the context of retrieval augmeented QA with a method that relies on individual passage utilities. Key in our approach is the definition of utility in terms of the behaviour of the QA model and whether it is able to provide correct answer given retrieved passage. We train small neural model passage utility judgements elicited from the QA model's responses and use utility predictions to estimate answer uncertainly. Experimental results show that our uncertainty estimator is competitive or better than existing strong methods while being heght- Future work could extend
this approach to long-form gencration tasks (Stelmakh et al., 2022 Gao ct al. 2023 Min ct al. 2023 Zhang
et al. 2024a where evaluating whether answer correctness is more challenging (Zhang et al. 2024b and to multi-modal QA scenarios (Borszuuovszki et al. 2025).
Acknowledgments
We thank the action editor and anonymous reviewers for their constructive feedback. We want to thank Ivan Titov and Gustavo Giméncz for useful discussions; and Alex Yu for sharing the synthotic multi-hop QA scripts. We are also grateful to Harry Sfyrakis and the UK Research and Innovation via the Innovate UK award (10092998) to Algomo Limited. This work was supported by the UK Engineering and Physical Sciences Rescarch Council (grant EP EP/WON (PP/002876/1) We also gratefully acknowledge the support of the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh.
References
Al@Meta. Llama 3 model card. 2024. URL http://///tthhhcoooo////////////
CARD_md
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024. URL htt.://tss..eeeeee/ee/////////es
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question- answer pairs. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533 1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://accaatthology -rtp://-11o///11111660010
Mirko Borszukovszki, Ivo Pascal De Jong, and Matias Valdenegro- Toren. Know what you do not know: Verbalized uncertainty estimation robustness on corrupted images in vision-language models. In Trista Cao, Anubrata Das, Tharindu Kumarage, Yixin Wan, Satyapriya Krishna, Ninareh Mehrabi, Jwala Dhamala, Anil Ramakrishna, Aram Galystan, Anoop Kumar, Rahul Gupta, and Kai-Wei Chang (eds.), Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pp. 247-265, Albuquerque, New Mcxico, May 2025. Association for Computational Linguistics. ISBN 979-8-89176-2388. doi: 00.18633/v1/2222 trustnlp-main. 16. URL httppss//ccllattooooo..o///22//. htt.://rw........tuuutmaaaaaaaa.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):321-38)) 345, 1952. ISSN 00063444, 14643510. URL http:
hww.jjsoooorggtaaaie 2334029
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Danicl Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish,, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https::/ heurips.cc/papees 1ttp:pppr....aass//p///////////a htt.://1111111706666664644711110
pdf.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia answer open-domain questions. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18633/11P117-1177 URL https: //aclanthologgy.o wrwpey/ogg////1111/7
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lúnguistics (Volume 1: Long Papers), pp. 5186-5200, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https:
//aclanthology.uo wwwggg..org.ooo wtrg/224..eeom (e2224..alllaee long. 283
Elizabeth R. DeLong, David M. DeLong, and Daniel L. Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3): 837-845, 1988. ISSN 0006341X, 15410420. URL http://www..jtott wtw.://ssoor.gg/ttallee/255s
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapler of the Association for Compulational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171- Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653 /v 1/N19-14223 URL https://aclanthology.oorg
Li Dong, Chris Quirk, and Mirella Lapata. Confidence modeling for neural semantic parsing. In Iryna Gurevych and Yusuke Miyao (ods.), Proceedings of the 56th Annual Meeting of the Association for Computational Länguisties (Volumee 1: Long Papers), pp. 743-753, 753, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 18.11863/v1// /11/////1//11////// 1069. CRL https:///claanhooogg http://ccaantttllooo.oorr///---0 org/P18-1069
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. LM-polygraph: Uncertainty estimation for language models. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 446-461, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/vl v1/2023 10023.emllp-dem..1.. URL httpps:/aacaanthooogy.rrr/ org/2222 http://////0222...eeeemmmoo .11,
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Tryna Gurevych and Yusuke Miyao (ods.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889 898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.11653//11/11 1082. URL httpss//aclaanthology hwwpo/oooyyo..ygg1111 org//188-
Haishuo Fang, Jeet. Gor, and Edwin Simpson. Efficiently acquiring human leedback with Bayesian deep learning. In Raúl Vázquez, Hande Celikkanat, Dennis Ulmer, Jörg Tiedemann, Swabha Swayamdipta, Wilker Aziz, Barbara Plank, Joris Baan, and Maric- Mario-Catherine de Marnoffe (ods.), Proceedings of the 1st Workshop on Uncertainty-Auare NLP (UncertaiNLP 2024), pp. 70-80, St Julians, Malta, March 2024. Association for Computational Linguistics. URL https::/acclanthoooo.ooo//////// uncertainlp-1.7
13

---

Published in Transactions on Machine Learning Research (09/2025)
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language 0038//441586-024-7774
models using semantic entropy. Nature, 2024. URL https://doi.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (cds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 22 Jun 2016. PMLR. URL https: https:://proceedinggs. .llr press/v48/gal16..u html
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 6488-6488, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/w1/2023.mmm// mma.:eeennnpm-aan3.e. URL https: //aclanthology. org/ wwwgg/222..ccm wwmlp-mminn 3988
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tan Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484 5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1018653vv1222/ http://v//222..emlmmaaaaaaa memnlp-main.4446 446. URL https://aclaatthoooss/ hrwpetthooooooooooooe.oooo /020. /tt.:///........mm......... 446/
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL https://proceedinng. mlr, wwtt.pppeesssseeueoeo/ 7a html,
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3929 3938.. PMLR, 13-18 Jul 2020.
mlr. wwwppprsssivv... /tt//v119//// w1111119990 1uu22a.. html
Lovisa Hagström, Youna Kim, Hacun Yu, Sang goo Lee, Richard Johansson, Hyunsoo Cho, and Isabelle Augenstein. Cub: Benchmarking context utilisation techniques for language models, 2025. URL https //arxiv.org/abs/220 16518
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.
In International Conference on Learning Representations, 2020. URL https://oppmeeeieee.nee/ffoooo id=rygG0yrFvHl
Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decomposing uncertainty for large language models through input clarification ensembling. In Forty-first International Conference on Machine Learning, 2024. URL https://oppenreiieet wttp://wwwww.....oooooooooooooae
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapler of the Association for Computational Linguistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguistics. doi: 1/6653/v1/22// main.74. URL http:/ss//acclattoooogooooo/// wtw.://nttooooooooo//2221/1 eew.222a.clmmmi.....
Gautier Izacard, Mathilde Caron, Lucas Hosscini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transuctions on Machine Leornóng Research, 2022. ISSN 2835-8856. URL https:///penreviiew.c jtt.ee/iiiiiettffooou/ddddiiii pX1760
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosscini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24(1), mar 2024. ISSN 1532-4435.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Soyeong Jeong, Jinheon Back, Sukmin Cho, Sung Ju Hwang, and Jong Park. Adaptive-rag: Learning to adapt retrieval-augmmented large language models through question complexity. In NAACI, 2024. URL
https://arxivv.rrr/bs//// 14403 14403
Albert Q, Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavand, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and 06825
William El Sayed. Mistral 7b, 2023. URL
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Compulational Linguisttees, 9:962 977, 09 2021. ISSN 2307-387X. doi: (0.1162/lacl a 00407. URL httpp:///oi..rr /ttp::///ooo.o//.../.o www.1t11caacccaa 00407
Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da- Chenh Juan, Ankur Taly, and Cyrus Rashtchian. Sufficient context: A new lens on retrieval augmented generation systems. In The Thirteenth International Conference on Learning Representations, 2025. URL
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina. Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Anreua Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601 1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/vl 11/88/11/11111// URL.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac (latfield-Dodde Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022. URL abs/ abtaaabb//222222 2207 85221
https://arxiv. http://arxiiv.ooraa////////
Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684 5696, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1 20l/22200.allmaaiii
URL https::/aallatthoolgg
org/ wwwgg/222.000 acl-main. 503
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. Realtime qa: what's the answer right now? In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncer- tainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. URL
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wi Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering rescarch. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/Lac)
00276. URL https://aclanthoooog.ooo 182
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, pp. 611-626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702297. doi: 10. 00.114//600000636133666 URL https: 14t 36116661000066663661
15

---

Published in Transactions on Machine Learning Research (09/2025)
Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL http:://ppppeeeivvvveeeeerrorres Certification.
Expert
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:
Paper...
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL http://perreeiieeeeeee/roorrr/ae
Kenton Lee, Ming-Wos Wei Chang, and Kristina Toutanova. Tatent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086-6096, Florence, Ttaly, July 2019. Association for Computational Linguistics. doi: 10. N0.18653/v11P199 grglaanthooogy.or///11---
URL. https:
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Scbastian Riedel, and Douwe Kicla. Retrieval- intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Tnc.
augmented generation for knowledge-intensivee
ISBN 9781713829546.
Kunze Li and Yu Zhang. Planning first, question second: An LLM-guided method for controllable question generation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Bangkok, Thailand, August 2024. Association for Compulational Linguislics: ACL 2024, pp. 211./22.2.nnning..... findings--ccl... acI.280. URL https::/acclatthllooe / Computational Linguistics. doi: 10.18653/v1. 20242222..22f. 2024. few.ffinding-aae acl. 2807
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume 6, pp. 87-100, 2024. URL httpps://ppocceeeinggs mlsys. wirg/appr__fiir hape://Do.htaaaa22eeeeeeeeeeeaar
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https::/openrreviewweee www.:eww.eeffrrmmm
id=8s8K2UZGTZ
Xin Lin, Zhenya Huang, Zhiqiang Zhang, Jun Zhou, and Enhong Chen. Explore what 1lm does not know in complex question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 39(23): Apr. 2025. doi: 10.1609/ 111699/aai.vva wtw..3399933333.33 URL https://ojs. aaai wwwaiiorg/indx.ppppeeee phtpeeexxppppp/AA///pp article/view/346BB
Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. Examiniig Ilms' uncertainly expression towards questions outside parametric knowledge, 2024a.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157 173, 2024b. doi: 10.1162/tacl 00638. URL https:///cclanttollooy
org/2024. 2tg/2224.tacl11.9 tac1-1.9
Shayne Longpre, Kartik Perisctla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity- based knowledge conflicts in question answering. Tn Maric-Franeine Moens, Xuanjing Huang, Lucia
16

---

Published in Transactions on Machine Learning Research (09/2025)
Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 2021. Association for Computational Linguistics. doi: 10.18653/v1 /220//1//222////// www.p-main.n..... URL https: //aclanthology.org// org/2021. eenlp- main. 565/
Online and Punta Cana, Dominican Republic, November
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retricvol, SIGIR '24, pp. 2421-2425, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314.. doi: 10.1145/3600 /32/6666222667775511111 URL 3657951
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and con-parametric memories. Tn Anna Rogers, Jordan Grader, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 10.18653/v (0.18653/v1/20223 ac02..ac-oo URL https://aclanttooloy/ wwt.:/gg///////// acl-long.
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-8722 2022. doi: 10.1162/tacl a 00494. URL httpss://aclanthooogy.oooo org/2022. tacl-1.50
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-lau Yih, Pang Koh, Mohit Iyyer, Luke Leemorrer and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text. generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076- 12100, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 11/5633/112022..mm/// http//222.....mmmaaaae 741. URT. https::/acclanthlloogo/oo org/ 2023. wem.-m-maimnnnnn. 7417]
Rodrigo Nogucira, Wei Yang, Kyunghyun Cho, and Jimmy J. Lin. Multi-stage document ranking with bert. ArXiv, abs/ abs/1910.14424, 2019. URL
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS 222, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871088.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.), Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 248 260. PMLR, 07-08 Apr 2022. URL https::/prooeeddnngs mlr, http:/iinw.......esssisiie//eeee wwtplrrrsssss//ssa22222/2. html
Seongll Park, Seungwoo Choi, Nahyun Kim, and Jay- Yoon Lee. Enhancing robustness of retrieval-augmeentr
language models with in-context learning. In Wenhao Yu, Weijia Shi, Michihiro Yasunaga, Meng Jiang, Chenguang Zhu, Hannaneh Hajishirzi, Luke Zettlemoyer, and Zhihan Zhang (eds.), Proceedings of the 3rd Workshop on Knowledge Augmented Methods for NLP, pp. 93-102, Bangkok, Thailand, August. 2024. Association for Computational Linguisties. doi: 2222://////2224..ooeeeeeeeeeee //aclanthology.org// wrw.//24.. httg://222...ooeeegeeeel.e. knowledgenlp-1.77
URL https:
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16- 1264. URL httpss//aadaantooooo..or/////
17

---

Published in Transactions on Machine Learning Research (09/2025)
Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L'eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram'e, Johan Ferret, Peter Liu, Pouya Dehghani Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vicillard, Piotr Stańczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behmam Neyshabur, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Boxi Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Christoper A. Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozi' nska, D. Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Pluce nska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Zhou, Joana Carrasqucira, Joana Iijazi, Jocelyn Becker, Joe Fernandez, Joost R. van Amersfoort, Josh Gordon, Josh Lipschultz, Joshua Newlan, Junsong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, L. Sifre, L. Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Gorner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, thew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengehong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona. Merhej, Reena. Jana, Reza Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, S. Mc Carthy, Sarah Perrin, S'ebastien Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomás Kociský, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wonming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotrula, Phoebe Kirk, Auand Rao, Minh Giang, Ludovic Peran, Tris Brian Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Auca Dragan, Slav Petrov, Oriol Vinyals, Jeffrey Dean, Demis Hassabis, Koray Kavukcuoglu, Cl'ement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size. ArXiu, hbs/2448.0001 2024. URL
https://apl semaaniccaant htt.://aa.r.rr/rruuss/////////// 270843336
Pedro Rodriguez and Jordan Boyd-Graber. Evaluation paradigms in question answering. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9630-9642, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. v2222211112221//2 10121eemnll-mmm main. 758. URL hnttps://aclanthol.gg org/ otgg/221..mmmpmmmin.. 7587
Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C!' robust fact verification with contrastive evidence. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Humon Language Technologies, pp. 024-643, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1 100///112/221// http//22.nnacclleee maacl-main.5 52. URL httt:::://cclanhoooooyooor wtw.:ooo///......1lloaaaa. www.caaaaaiiia.... main.mainn.. 52
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric questions challenge dense retrievers. In Francine Mocns, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (cds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6138 6148, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2000 /ttp://112221.cmppmmmo/ 496. URL htttps://aclanthoolge
/etrg//2211..2 emnlp-main maall.-mai.444 496
D. Sculley. Combined regression and ranking. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '10, pp. 979-988, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781450300551, doi: 111144/1135804418835525 URL /ttp::///ooi.orrr//..11///1//331 1835928
18

---

Published in Transactions on Machine Learning Research (09/2025)
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar. A survey uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Comput. Suro., June 2025. ISSN 0360-0300. doi: 10.1145/3744238, URL https:://bi..rgg//// org/ 1145/ 3744238
Just Accepted.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3784-3803, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/ hatp://vv1222.iiii/ wii.ei.iiiing-eeee findings--mmll.32222 URL https: /ttps://ac.aanthology.orrt
Findings-emnlp... 320/
Adi Simhi, Ita Itzhak, Faul Barez, Gabriel Stanovsky, and Yonatan Belinkov. Trust me, i'm wrong: High-certainty hallucinations in 1ims, 2025. URL https:///rxii. http///aiii..or/b//////22.. 12964
Edwin Simpson, Yang Gao, and Iryna Gurovych. Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization. Transaclions of the Associalion for Compulational Linguistics, 8:759 775, 12 2020. ISSN 2307-387X. doi: 10.1162/taci 00344. URL https: / /tt.///ooiioooo /ttp::///do..or//188
1162/tacl_a000320
Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. Why uncertainty estimation methods fall short in RAG: An axiomatic analysis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 16596-16616, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89176--56-5. doi: (1.1863//v1/200 httd:///////....iinnng//ga/ ac1.852. URL httts://accanntooooyy.orr//// http:///22...iiinnninnn.. findings-acl. 852/
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wae Wei Chang. ASQA: Factoid questions meet long-form answers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 8273 8288, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18633/v1 wwtp:/22..mmmma...mmb... URL https://aclaanthology/ org 20022. emnlp-main. 566/
Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Hcad-to-tail: How knowledgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? In Kevin Duh, Helena Gomez, and Steven Bethard r, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingwistics: Human Language Technologies (Volume 1: Long Papers), pp. 311-325, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.naacl-long.15 URL https:://aclantholoog wwwpollogyoogggggooooooooo org/ wwgg2224..nacl-1oom. naacl-long. 18
ZhongXiang Sun, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li. ReDeEP: Detecting hallucination in relrieval-augmentte generation via mechanistic interpretability. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttpss://openrevien
het/forrmmidd=zzzZZzze
Yoo Yeon Sung, Eve Fleisig, Yu Hou, Ishan Upadhyay, and Jordan Lee Boyd-Graber. GRACE: A granular benchmark for evaluating model calibration against human calibration. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (ods.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19586-19587, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 8916-----0-- doi: (0.18653/v1 /2025. acl-long.962. URL https://aclaantoolog.. orge 2etgg/2225.20 acn- 962/
.rg
Junya Takayama and Yuki Arase. Relevant and informative response generation using pointwise mutual information. In Yun-Nung Chen, Tania Bedrax- Weiss, Dilek Hakkani- -urr, Anuj Kumar, Mike Lewis, Thang-Minh Luong, Pci-Hao Su, and Tsung-Hsien Wen (ods.), Proceedings of the First Workshop on NLP for Conversational AI, pp. 133-138, Florence, Italy, August 2019. Association for Computational URL http:://aaclanttoooo...oo/////
Linguistics. doi: 10.18653/v1/W199-40
19

---

Published in Transactions on Machine Learning Research (09/2025)
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), ) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. -443, Singapore, December 2023. Association for Computational Linguistics. doi: (10553/v1/222.en/// main. 330. URL https://aclanttollonn wrw. httg:/223.eeeee 101123..mmllppmem mamnp-mmii.330
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensinn multi-step questions. In Anna Rogers, Jor- dan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61sl. Annual Meeting of the Asso- ciation for Computational Linguistics ( (Volume 1: Long Papers), pp. 10014-10033 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v 2ww.22222....2nngpe.. https://aclanthologyy hwwpanthoooooooooo////eee org/2023. acl-long. 5577/
URL
Joost Van Amersfoort, Lewis Smith, Yee Whye Tch, and Yarin Gal. Uneertainty estimation using single deep deterministic neural network. In Hal Dauné IIT and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9690 9700. PMLR, 13 18 Jul 2020. URL htttp:://proceeeiinnggg www.mmlrrrrrr... www...ppeesvvilii http:///////aaaaaararrrrrrro
mlr
Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13697-13720, Bangkok, Thailand, August 2024. issociation for Computational Linguistics. doi: 10.18653 v w /ttp///2222.iin-aa/aaa/a acc.nggs-all.1..... URL https: //aclanthology.c wwt.:rgg////00 2024. findings-acl.c. 8137
Yuhao Wang, Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, and Ji-Rong Wen. Rear: A relevance-aware retrieval-augmented framework for open-domain question answering. arXiv preprint arXiv:2402.17497, 2024. URL
17497
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. URL
fttps://arxiw.rrg/abs////2 24112
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Tong Papers), pp- 2018. Association for Computational Linguisties. doi: 10. 10.18653/w1/N18- 1101 URL hntps:://cclaathoolooy Nr8/N18-1101
New Orleans, Louisiana, June
1101.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations, 2024. URL
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-auggmenter LMs with context com- pression and selective augmentation. In The T Twelfth International Conference on Learning Representations, 2024. URL /ttpps:/oppenreeeee wttp://wwnnn//oooommmmmmm..ii//e
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng Hc, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingleng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. Xrviv preprint 210:24077.1061,, 2024.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. HotpoiQQ: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tuujü (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2869-2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.1105....... 1 1259. URL http:///c..nnttoooooorrrr///////
Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, and Juanzi Li. Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation, 2024. URL
19215
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmmentd language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations, 2024.
URL
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-ave anggeettd language models, 2023. URL http::///axxi..or/bbbsss 2311 09210
Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. LUQ; Long-text uncertainty quantification for LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (ods.), Proceedings of the 2024 Conference on Empirioal Methods in Natural Language Processing, pp. 5244 5262, Miami, Florida, USA, November 2024a. Association for Computational Linguisties. doi: 10.18653/ v1 1002l...nlpmmaae 299. URL https://aallanhoology org. 2204g/2224.cmm emnlp-main. 299/
Huajian Zhang, Yumo Xu, and Laura Perez- Beltrachini. Fine-gret sane-grained natural language inference based faithfulness evaluation for diverse summarisation tasks. In Yvette Graham and Matthew Purver (ods.), Proceedings of the 1 18th Conference of the European Chapler of the Associalion for Computational Lénguislics (Volume 1: Long Papers), pp. 1721-1722, St. Julian's, 's Malta, March 2024b. Association for Computational Linguistics. URL https: http:s://aaaaatthool...
org/ 2224./2244... http//2222..222210 eacl-long. 102
Shujian Zhang, Chengyue Gong, and Eunsol Choi Knowing more about questions can help: Improving calibration in question answering. In Chengqing Zong, Fei Xia, Wenjic Li, and Roberto Navigli (eds.), Findings of the Association for Compulational Linguislics: ACL-IJCNLP 2021, pp. 1958 1970, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v www.20....iiing........... URL http:///claatholooyoorgg////// findings-acl.17
A
Ablation of the Passage Utility Predictor Training Objective
Table | shows AUROC results on answer uncertainty prediction with Passage Utility estimators trained with different variants of the objective in Equation (5) The first row shows the full objective (sce training details in Section F.2), the second row shows a variant where the ranking objective uses only entailment utility annotations (e), and in the third row the objective is solely based on accuracy prediction (LBCe). As can be seen, there is a drop in performance when the pairwise ranking loss is not used (i.e., last line of Table 4h; this component of the objective provides a smoother signal on passage utility, which is empirically beneficial. However, when the pairwise ranking loss only based on entailment, performance drops by several points, highlighting the importance of combining both to model QA answering behaviour.
Table 5 reports various ablation studies (with GEMMA2-9B on the WebQ development set) with different instantiations of the Passage Utility score tM and our training objective. We present different ways to combine accuracy (a) and entailment (e) scores to induce utility rankings with the pairwise ranking loss Lrank (-) (Equation 3). These include average (a + e)/2 as in Equation 2 addition (a + e), addition but inverting the entailment score ((e|1 - e) + a) when the passage yields an inaccurate answer (a = 0), or when one of the two is given a zero weight, i.c., (a) or (e) alone. We asses these ranking variants when using the ranking objective alone (rightmost block) as well as when combined with the binary cross-entropy objective LACE (Equation 5 (middle block); we also report performance when only the LaCE objective is
21

---

Published in Transactions on Machine Learning Research (09/2025)
Objective Terms
G9B
L8B
M7B
Cranks (e + a)/2 + XLacE
0.80
0.81
0.80
Lrank, (e) + X LBCE
0.73
0.73
0.71
LBCE
0.78
0.78
0.80
Table 4: Answer uncertainty estimation with Passage Utility predictors trained with different variants of the objective in Equation 5). AUROC values for GEMMA2-9B (G9B), LLAMA3.1-8B (L8B); and MISVRAI-7B--O.3 (M7B) are averaged over development sets.
LACK
Drank +
XLBEE
Lrank
(e + a)/2
(e+
([e]1 - +
(e)
(e +
a)/2
(e +
((el1 - e)
+a)
(e)
(a)
0.71
0.75
0.74
0.72
0.63
0.65
0.63
0.62
0.61
0.53
Table 5: Answer uncertainty estimation with Passage Utility prodictors trained with different variants of the AUROC values are for GEMMA2-9B and the WebQ development set.
objective in Equalion (2)
used (leftmost block). All model variants are trained with rank-basod model selection and X = 0.25 following
training details in Section F.2
When looking at the pairwise ranking objective alone (rightmost block), entailment dominates the ranking and the utility scores learnt; (e) pairwise ranking as well as (a (a+00 e) variants yicld utility scores that have similar discriminative power. The (a) variant exhibits the worst performance, given that many pairs are discarded due to having the same utility. AUROC values improve when the pairwise ranking is combined with the cross-entropy objective (middle block). In this case, in addition to enforcing the pairwise ranking with Crank, (.), the utility scores regularised. In other words, the utility scores of a pair of passages with the same accuracy (i.e., only ordered by entailment) will end up closer (differ less) than the utility scores of a pair of passages with different accuracies (differ more). This is reminiscent of the Bradley-Tery model BBaddley & Terry) 1952): in the first case, one passage is better than the other with a small probability, while in the second case the probability of one being better than the other higher. The exception here is the Lrank, (e) + LBCE variant where the pairwise ranking by entailment may often contradict the binary cross-entropy signal.
B Test Time Cost of Uncertainty Estimation Methods
Table 0 shows the cost of estimantng uncertainty for question æ, measured by the number of inference calls required. Simple information theoretic methods (e.g., PPL) require a single call to the target QA model with the retrieval augmented QA prompt (i.e., [/[] retrieved passages and question x). However, approaches that estimate uncertainty based on diversity (e.g., Regular Entropy, Semantic Entropy, and p(true)) require generating N answers, i.c., N inference calls with the retrieval augmented QA prompt. In addition, Semantic Entropy requires the computation of answer clusters (i.e., grouping answers with the same meaning), additional calls to entailment model are required to compare the set of sampled answers. p(true) requires additional LLM call to elicit Frue/False answer but with very long prompt including in-context examples and the assessment question with the |R] retrieved passages, sampled and most likely answers, and question x (see Table 15 In contrast, our approach requires []] utility predictions with a BERT-sizer model. For instance, in our experimental setup with = 10 samples and retrieval augmented QA with |R] = 5, the Semantic Entropy approach would require 11 forward passes with the QA model prompted with passages (one the greedy candidate and 10 random samples) plus 45 calls to the entailment model. Our approach requires 5 forward passes with the BERT-based model.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Methods
Inference Calls at Test Time
PPL
1G
p(true)
(N + +1
Regular Entropy
(N + +1JG
Semantic Entropy
(N + I)G+ (*)
Passage Uuility
[R] BERT-F
Table 6: Number and type of inference calls required to estimate answer uncertainty for question and set of retrieved passages R. G means inference is performed with a retrieval augmeetedd QA model, i.e,, an LLM forward pass with the prompt including the set of [R] retrieved passages and question æ generate candidate answer y. E is inference with an evaluation model, e.g,, a forward pass to ask an LLM for correctness in p(true) or a forward pass with an entailment model in Semantic Entropy. Bert-F is an inference call to predict passage utility for passages in R and question x.
G9B G27B
L8B
M7B
a 0.12
0.13
0.15
0.15
RAce LINDACO
0.04
0.04
0.05
0.06
RAcot INDACe
0.01
0.01
0.01
0.01
Table 7: Average proportion of cases (five development sets) where at least one individual passage in R leads correct answer (InDAcct) but the QA model prompted with R yields an incorrect one (RAcc.) (and vice versa RAcc INDACC
C Generality of the Uncertainty Aggregation Strategy
To validate the intuition behind our uncertainty estimation step (Section 3.2 we compare the behaviour of the QA model when prompted with individual passages in R versus when prompted with R (i.e., the top-|R| passages). In particular, we want to inspect the proportion of cases where our stratege of taking the maximum uitility score among the passages in does not agree with the entire set accuracy. In other words, we are interested in cases where the QA model is accurate with at least one individual passage (INDAcc t) but answers incorrectly when promtped with R and vice versa. In this study, we consider two individual passage accuracy variants (related to the combined definition of Passage Utility Section 3.1 One is based accuracy (see Section 4 and is either 0 or 1. The other one is also based on accuracy but smoothed by entailment e (computed by an off-the-shelf entailment model; see Section 4 and downgrades cases where a = into a = e < 0.5. The latter occurs in cases where the QA model produces an answer that is accurate but not entailed by the passage.
We analyse the behaviour of four QA models (LLMs) across five datasets in our experimental setup. Table shows the proportion of model disagroements across development sets. We see that such disagreements amount to relatively small number in both settings, i.e., when at least one individual passage in R yields correct answer (INDACC but the QA model prompted with R yields an incorrect answer (RAcc.) and vice versa (RAcc1 1 INDACC 4). The results in Table 7 confirm that our aggregation approach based on individual passages is fairly general. It approximates answer uncertainty when prompting with [R] passages while avoiding the complexity of estimating uncertainty over all possible combinations input passages terms of number and order.
Our study is related to the issue of understanding LLM sensitivity to external evidence Xie et al. 2024 Liu et al. 2024b), i.e., how the type of evidence (supportive, contradictory, irrelevant, misleading). the amount, and order of presentation affect LLM predictions and interact with parametric knowledge. The Passage Cuility predictor is trained to predict the error of a target QA model (LLM) on answering questions, independent of the type of passage or any memorized knowledge. Given a question passage pair, if the LLM relies on its memorized knowledge rather than adapting to the passage and still produces the correct answer, or conversely, adapts to the passage but produces an incorrect answer, then the Passage Utility predictor should reflect this outcome by predicting correct incorrect answer accordingly Xie et al. (2024).
23

---

Published in Transactions on Machine Learning Research (09/2025)
Note that the Passage Utility predictor is mcant to be synchronized with the target QA model and judgments of what is (or not) correct answer. If an answer to question changes, the target QA model answer correctness on this question may also change, and the Passage Utility predictor should also reflect this (i.e., it should be adapted with now examples). Multiple passage interactions studied in datascts with synthetic evidence (Longpre et al. 2021; Xie el al. 2024) are observed to a lesser extent. our experiments with six datasots and external retrievers. This has also been recently pointed out Hagström et al. (2025). Our approach could be combined with additional features to capture more complex interactions (Dong al. 2018). Investigating and understanding the relation between QA model uncertainty and (improving) context utilization is an interesting topic on its own right (Xie et al. 2024 Longpre et al. 2021 Hagström et al.
2025 but out of scope for this paper.
D Out-of-distribution Gemeralization of Uncertainty Estimation
We assess the generalization ability of our Passage Utility estimator both in terms of new QA tasks and QA models. We train a unique Passage Utility predictor for the GEMMA2-9B model. Following previous work on question answering and out-of-distribution (o.o.d) scenarios (Kamath et al. 2020 Zhang et al., 2021), we train it on the SQuAD dataset and then use to predict zero-shot (i.e., without further line-tuning) passage utilities on all other datasets (test set). As p(true) relies on 20 in context training examples, also evaluate its ability to generalise in out-of-distribution settings.
Table S (QA Task block) shows AUROC for answer uncertainty estimation in o.o.d scenarios. As an upper bound, the i.i.d block of the table shows AUROC values in the in-distribution scenario for Passage Utility and p(true). We compare o.o.d performance w.r.t. PPL and Semantic Entropy which do not rely on training examples. Although Passage Utility performance decreascs in o.o.d scttings, it remains competitive in four out of five datasets. In these four cases, it is always statistically significantly different. from the PPL method and comparable to p(true) and Semantic Entropy. Interestingly, p(true)'s performance also drops in all o.o.d fixed number of in-context learning examples is neither a robust nor test sets showing that relying scalable adaptation method.
To understand the observed performance drop, we conducted a comparative analysis of passage utility predictions for 50 question-passen data (i.i.d) versus one trained on SQuAD (o.o.d). We sampled 10 WebQ test questions: 5 for which the predicted utility decreased from high under i.i.d. conditions to low under o.o.d. conditions (changing from true negative to false positive), and 5 for which it incrcased in the opposite direction (changing from true positive to false negative). In both scenarios, the o.o.d predictions were predominantly influenced by token overlap and similarity. We hypothesize that when faced with o.o.d questions (e.g., in terms of type, length, or topic), the prediction mechanism defaults to predictions based on a general notion of question-passage similarity, disregarding whether the QA model can answer the question with the given passages. For passages with low (high) similarity to the question, it will predict low (high) utility scores. This behaviour may be a consequence of under-training, suggesting that the Passage Utility predictor cannot predict the accuracy of the target QA model on these o.o.d questions and their corresponding retrieved passages.
pairs. We examined predictions from a utility predictor trained on WebQ
To empirically validate these observations, we calculate Spearman's p correlation coefficients between the predicted Passage Utility scores and the corresponding retriever relevance scores (as computed by Contriever- MSMARCO; Izacard et al. 2022) for each passage, examining correlations separately for both i.dd and o.o.d predictions. We use the Retriever Score because it measures the semantic relation between questions and retrieved passages. In agrcement with our manual inspection, there positive correlation between Passage Utility and Retriever Score in the o.o.d setting (0.536 with p-value < 0.01), suggesting that Passage Utility falls back to textual similarity. In contrast, the i.i.d setting exhibits a negative and weaker correlation (-0.318 with p-value < 0.05), indicating that training successfully aligns Passage Utility predictions with the target QA model's actual performance rather than superficial retriever similarity scores.
We also evaluate generalisation to a new QA model, by training the Passage Utility predictor on utility labels obscrved for GEMMA2-9B and use its predictions to estimate uncortainty for the bigger GEMMA2-27B model. We also evaluate p(true) in this o.o.d setting, i..,, we create p(true) prompts as usual with GEMMA2-27B generated answers (greedy and sampling), but ask GEMMA2-9B to judge the probability of the most likely
24

---

Published in Transactions on Machine Learning Research (09/2025)
GEMMA2-9B
NQ
TQA
WebQ
PopQA
RefuNQ
p(true)
0.73
0.75
0.67
0.81
QA Task
Passage Utility
0.76
0.85
0.69
0.86
PPL
0.64
0.68
0.52
0.59
0.51
0.o.d
Semantic Entropy
0.70
0.73
0,58*
0.73
0.59*
p(true) (SQuAD)
0.67
0.63
0.63
0.72
0.62
Passage Utility (SQuAD)
0.65
0.79
0.60
0.72
0.79
GEMMA2-27B
NQ
TQA
WebQ
PopQA
RefuNQ
p(true)
0.77
0.83
0.67
0.79
0.60
QA Model
Passage Utility
0.73
0.82
0.69
0.87
0.80
PPL
0.64"
0.50
0.53
0.53
0.51
v.o.d
Semantic Entropy
0.68
0.62
0.59
0.66
0.58
p(true) (GEMMA2-9BB
0.73
0.71
0,64
0.75*
0.59*
Passage Utility (GEMMA2-9B)
0.75
0.80
0.68
0.87
0.79
Table 8: Out-of-domain performance of Passage Utility predictor for GEMMA2-9B both in terms of QA task and QA model. i.i.d blocks report AUROC values from our main in-distribution experiments (Table D o.o.d blocks contain the o.o.d comparison. Best values are highlighted in bold; we also mark with next best values which are siguificantly different using the paired De Long lest (p < 0.05). For the QA task, Passage Utility and p(true) are supervised with SQuAD data and evaluated on NQ, TQA, WebQ, PopQA, and RefuNQ test data. For the QA model, Passage Utility predictors are trained on GEMMA2-9B and usod to estimale uncertainty for GEMMA2-27B; p(true) assessment provided by GEMMA2-9B.
answer being truc. Table S (QA Model block) shows AUROC results; for reference, we include in-distribution values (i.i.d block) and compare the o.o.d results with comparison methods. Passage Utility outperforms all other methods across the board, and while still competitive, p(true) exhibits a higher decrease in performance.
These preliminary results suggest that in the context of retrieval augmented QA, models behave alike (also suggested by the distribution of observed correct porrec//nnoon /incorrectt individual passage utilities in Table 12) This highlights practical benefits of our approach, such as training a base Passage Utility predictor using data generated by less expensive model or developing more general predictor applicable across multiple QA models.
Finally, we train a unique Passage Utility predictor for all QA tasks and assess its generalisation capabilities. To this end, we train a predictor for the GEMMA2-9B model on a random sample drawn from the five training sets of size equivalent to the training sets of the individual QA task predictors. Specifically, we took 10k from NQ, TQA, and SQuAD, 3k from PopQA, all WebQ, totalling 35,474 instances for training and 500 instances from each for validation; the number of pairwise instances in the final curated training set is 354,379. For p(true), we mixed 4 randomly sampled examples from each dataset as in-context training examples. We follow the same training procedure as described in Section F.2 (with combined model selection and = We show AUROC results in Table g Across the six test sets, the unique Passage Utility predictor trained on a mix of QA tasks (bottom block) achieves similar performance to the individual predictors trained on per-task datasets (top block). The unique predictor keeps comparable or better performance than p(true) and outperforms PPL and Semantic Entropy. This preliminary study suggests that is fcasible train more general predictor for various QA lasks.
E Synthetic Qualitative Analysis of Passage Utility in Multi-hop QA
In multi-hop QA Passage Utility may fail at cases where the QA model cannot answer when prompted with any individual 'hop' passage, but answers correctly when prompted with the full set. To analyse this limitation, we carry out a qualitative analysis. As discussod Scction 5 there are two scenarios in retrieval augmented multi-hop QA (given existing datasets and current LLMs). Either models can answer with one passage (thus multi-hop) retrieval completely fails. Thus, be able to pinpoint actual multi-hop QA
25

---

Published in Transactions on Machine Learning Research (09/2025)
NQ
TQA
WebQ
SQuAD
PopQA
RefuNQ
Task
((true)
0.73
0.75
0.67
0.68
0.81
0.62
Passage Utility
0.76
0.85
0.69
0.80
0.86
0.79
PPL
0.64"
0.68
0.52
0.59
0.59
0.51
Unique
Semantic Entropy
0.70
0.73*
0.58
0.63"
0.73
0.59
p(true)
0.71
0.70
0.68
0.63*
0.75"
0.66"
Passage Utility
0.69
0.83
0.68
0.79
0.85
0.79
Table 9: Generalisation of a Unique Passage Utility predictor for GEMMA2-9B trained on a mix of QA tasks and evaluated on the six test sets (lower block). Best values are highlighted in bold; we also mark with next best values which are significantly different using the paired De Long lest (p 0.05). The upper block reports AUROC values for the Passage Utility from predictors trained for GEMMA2-9B on indiviual QA tasks' training datasets. We also compare p(true) when in-context lcarning examples are all from the same QA task (upper block) versus from mix of tasks (lower block).
cases and expose the limitations of our approach, we make use of synthetic HotPotQA data Yang et al. 2018
First, we use the sets of gold passages provided in the HotPotQA dataset to categorise question types as follows. We prompt the QA model with individual gold passages and with combinations thereof, measuring accuracy. Out of the 500 test questions, 475 are answerable with only one gold 'hop' passage, 13 require multiple passages, and 12 unanswerable. Secondly, insert gold 'hop' passages within the retrieved set for those where retrieval completely fails. Then, run QA and uncertainty estimation. In 6 of 13 cases, our approach fails to predict that the QA model's answer will be correct. It is important to note, however, that these 13 cases represent only 2.6% of the total (13 out of 500).
focus the 13 requiring multi-hop questions and
F Experimental Details
F.1 Datasets and Splits
In our experiments, we six QA tasks which we describe below. Table 10 shows dataset statistics and example question-answers pairs.
Natural Questions (NQ; Kwiatkowski et al. 2019) is a QA dataset compiled from real user questions submitted to the Google search engine. As part of the dataset curation process, annolators judge the quality of questions and associate them with a short answer that can be extracted from a related Wikipedia page.
TriviaQA (TQA; Joshi al. 2017) question answering datasel designed for training and evaluating machine learning models on open-domain question answering tasks. The dataset was created by gathering questions from trivia websites, along with their corresponding answers, to provide broad range of factual questions.
WebQuestions (WebQ; Berant al. 2013) was mined off questions generated with the Google Suggest API. The answers to the questions are defined as Freebase entities (i.e., their string label) and were elicited by Amazon Mechanical Turk (AMT) annotators.
SQuAD Rajjurkar et al.) 2016 contains questions formulated by AMT annotators based on given Wikipedia paragraph, with the answer being a short span in that paragraph. Annotators were encouraged to use paraphrasing when writing the question. The answer types not only cover named entities but also other categories such as noun- and verb-phrases.
PopQA Mallen et al. (2023) an open-domain QA dataset, focusing on popular culture topics, such movies, TV shows, music, and sports. It contains question-answer pairs derived from (subject, relation,
26

---

Published in Transactions on Machine Learning Research (09/2025)
Dataset
Train
Dev
Test
Example Question
Example Answer
NQ
79,168
8,757
3,610
Who plays Letty in Bring on all or nothing?
Francia Raisa
TQA
78,785
8,837
11,313
Who was the first artistic director of the National Theatre in London?
Lord Laurence Olivier
WebQ
2,474
361
2,032
What party was Andrew Jackson?
Democratic Retrralicnn Party
SQuAD
78,713
8,886
10,570
What the Grotto at Notre Dame?
A Marian place of prayer and reflection
PopQA RefuNQ
10,000
1,267
3,000
Who was the director Champion? Who does the voice over in the Re- quirtion?
Rabi Kinagi
2,173
Table 10: Dataset statistics, number of instances per Train/ (Dev)) /Test sets, and example question-answer pairs (all taken from the Dev set except for RefuNQ).
object) triples in Wikidata. Triples were translated into natural language and the object entity was taken to be the gold answer. The collection process focused on gathering questions about subject entities of varying popularity.
RefuNQ Liu et al. (2024a) is derived from NQ and consists of answerable and unanswerable questions. Unanswerable questions are created by replacing entities in the original NQ question by non-existing concepts.
We follow previous work [ee et al. 2019) and use only the question and gold answers, i.e., the open versions of NQ, TQA, and SQuAD. We use the unfiltered TQA dataset. We follow the train/de//tes splits as used previous work Lee et al.] (2019) and randomly split PopQA. RefuNQ only provides a test set so our experiments on this dataset are zero-shot from a Passage Utility predictor trained on SQuAD. We follow Farquhar et al. (2024) and use 400 test examples randomly sampled from the original larger test datasets for evaluation of uncertainty quantification.
F.2
Implementation Details
QA Models
For all question answering tasks, we use the off-the-shelf Contriever-MSMARCO tool (Izaaard 2022) to retrieve sets of passages R for question x from Wikipedia and the official Wikipedia embeddings
ct al.
based
(2018 snapshot) as our document knowledge-base. For PopQA, we follow the work of Mallen et al. who also use the full 2018 English Wikipedia dump.
2023
The QA prompt used for all models (embedded in the corresponding chat templates) is shown in Table 14 For inference, we set the maximum number of generated tokens 50 for both the greedy (most likely answer well temperature scaled (sampled candidates) decoding. We use vLLM for inference (Kwon et al. 2023 For all models, inference was run on single A100-80GB GPU.
Curated Passage Utility Dataset We train our passage utility predictor on a dataset D curated from benchmark D, e.g., WebQ, consisting of question and gold answer pairs (x, (x,y). For each question we retrieve the top-k passages. Then, we pair question x and retrieved passages p with utility scores UM which we obtain after running the QA model M on inputs (a, and computing the generated answer accuracy and entailment scores (Section 3.1). i.e., we create tuples (I,p, UM). From the set of tuples for question x, we derive (b) instances for our pairwise ranking loss.
In experiments, we use & = 5 retrieved passages per question. Table 11 reports the size (number of training instances) of the curated datasets for each QA task and model. From each question and set of lop-5 retrieved passages, we derive 10 pairwise ranking instances, discarding those that have equal utilities (e.g., from the WebQ training split with 2,474 question-answer pairs, we curate 24,720 instances with MISTRAL-7B-VO.3) As our top-5 passages are obtained via a real retrieval module, i.c., not synthetically assembled, there are questions for which all passages in the top-5 set lead to correct (incorrect) answer. In these cases, the
27

---

Published in Transactions on Machine Learning Research (09/2025)
Models
NQ
TQA
WebQ
SQuAD
PopQA
GEMMA2-9B
395,138
393,175
21,721
393,285
99,770
GEMMA2-2T7
395,426
393,477
24,723
393,293
99,777 778
1LAMA-3.1-8PP
790,862
393,465
24,713
393,288
99,787
MISTRAL-7B--O.3
395,397
393,474
24,720
393,271
99,772
Table 11: Number of pairwise training instances in the curated datasets to train the Passage Utility predictor with the combined pairwise ranking and binary cross-entropy losses (Section
Incorrect
Mixed
Correct
GEMMA2-9B
21%
52%
27%
GEMMA2-27B
20%
53%
27%
LLAMA-8.1-8B
22%
55%
23%
MISTRAL-7B-VO. 21%
56%
23%
Table 12: Number of training instances curated datascts to train the passage utility prodictor with the combined pairwise ranking and binary cross-entropy losses (Section 5
pairwise ranking is dominated by the entailment score (i.e., accuracy is the same). Table 12 shows the distribution of questions with all retrieved passages leading to the same accuracy (Correct/Incorroct) mixed (Mixed) accuracies in the curated dataset for each QA task and model.
Passage Utility Predictor Training Details We train a different predictor for each target QA model and QA task. Given the large number of predictors required in our experiments, we initially tested the hyper-parameters used in Fang et al. (2024) on the NQ dataset and choose a set thereof for all predictor instances. We train each predictor for 3 epochs, with a batch size of 32 examples, learning rate equal 2e-5, and weight decay 0.001 (with the exception of LLAMA-3.1-81) and WebQ where we used 0.01). For each predictor performed scarch values for X, i.c., the contribution of the LEC loss (Equation 5 and different criteria for model selection, i.e., the best at pairwise ranking or at both pairwise ranking and accuracy prediction (combined).
Table 13 shows the configuration for each predictor. Table cells show selection criteria (R for ranking and C for combined) and the value for X. At inference time we predict a single Passage Utility score given by the selected best checkpoint. For all predictor instances (except for all WebQ and PopQA predictors and the predictor for LLAMA-3.1-8B and NQ), we half of the available training data to speed up experiments. Training and inference was run on single A100-40GB GPU; training ranges from 2 to 12 hours depending the dataset.
Comparison Approaches In this section, we describe additional answer uncertainty estimation methods (for which we present supplementary results in Section G Maximum Sequence Probability (MSP) is based on the probability of the most likely answer and is computed as:
MSP(2, M) = P(ylæ, ((ylaRR;M) R)
(12)
Note that, in contrast to PPL(x, reported in the main section of the paper, this metric is biased by answer length, i.c., identifying an answer to have low probability (low confidence) because of its length. Despite the fact that QA models are instructed to produce short answers, they do not always follow instructions. For this reason, we consider perplexity a more accurate metric. Indeed, answer length could indicate that the model is uncertain about the answer. Thus, we also estimate answer uncertainty from the Average Answer Length (AvgAnsLen) as the average number of words in the sampled answers. As seen in Section G.3 Table 20 MSP and AvgAnsLen perform similarly across the board.
28

---

Published in Transactions on Machine Learning Research (09/2025)
Models
NQ
TQA
WebQ
SQuAD
PopQA
HotPotQA
GEMMA2-9B
C, 0.25
C, 0.25
0.25
C,
C,
C, 0.25
GEMMA2-211 -7B
C, 0.25
C, 1
C, 1
C, 0.25
R, 0.25
C, 1
LLAMA-3.1-33
C, 0.25
C, 0.25
0.25
C,
C,
C, 0.25
MISTRAL-7B-VO.3
R, 0.25
C,
C, 0.25
C, 0.25
C, 0.25
C, 1
Table 13: This table shows the À value and selection criteria (R for pairwise ranking or C for combined pairwise ranking and accuracy prediction) for each Passage Utility predictor in our experiments.
We also report Cluster Assignment (CA) which is a variant of SE without answer probabilities where the probability of each generated meaning (i.e., a cluster) is approximated from the number of answers in the cluster. We found that in general CA estimations are very close to Semantic Entropy ones.
Another uncertainty estimation approach is the negative mean Point- Mutual Information (PMI: Takayama & Arase 2019) over lokens; i.e., compares the probability of answer sequence y given prompt with question æ and passages R w.r.t the probability given by M to y without any context. Intuitively, the higher the point-wisc mutual information, the more certain the QA model is on generating y (i.c., the answer is related to or depends on æ and R). PMI is computed as:
plyclyi 1-1, M)
PMI(x, M,
log
(13)
We also report Retriever Score as baseline for Passage Utility, Instead of using the predicted Passage Utility use the original relevance score assigned by the external retriever (i..,, Contriever MS-MARCO).
We use the implementation provided in Farquhar et al. (024g to compute Regular Entropy, Semantic Entropy, Cluster Assignment, and p(true). Note that we do not include the supervised baseline reported in Farquhar et al. (2024) as the authors show that underperforms simple information-theoretee metrics and in addition only works for white-box models. Note that while AvgAnsLen and Retriever Score do not strictly provide scores the [0, 1] interval, the package that computes AUROC finds ranking decision thresholds 4
Metrics We use the implementation provided in Farquhar al, (024 to compute AUROC, Accuracy X% of rejection, and AURAC metrics.
We use Qwen2-72B-Insirru specifically, we use the Activation-aware Weight Quantization (Lin et al.) 2024). version Qwen2-72B-Inn Instruct- AWQ. We prompt the accuracy evaluator with the prompt proposed in Sun et al. (2024) as we found it to perform well on our datasets. The accuracy evaluation (AccLM) prompt is shown in Table 16 In a sample of 840 generated answers human and L.I.-based judgment. of correctness agreed 98% of the time (Sun et al., 2024.
(Yang et al. 2024) to oblain accuracy judgments (i.e., A judge, Section
F.3 Prompts
The prompt we use for our QA models shown in Table 14 Table 15 illustrates the prompt used for the p(true) baseline. Table 16 shows the prompt used for the LLM-based accuracy metric.
G Additional Results
G.1 Reference Retrieval Augmented QA Accuracy
Table 18 shows retrieval augmented QA performance for the five QA models on the development and test. sets of our six tasks. We report accuracy based on token overlap (Acc) as computed in previous work, ie.,
https::/scikittllesee
det_curre.hhtl
29

---

Published in Transactions on Machine Learning Research (09/2025)
Retrieval augmented QA prompt
Knowledge: passage
passage
[[]]]
passage
Answer the following question with a very short phrase.
Question: question
Table 14: Prompt designed as user turn for QA models.
p(true) prompt
Question: question Brainstormed Answers: likely sampled
sampled answer
Possible answer: most likely answer
Is the possible answer:
A) True
B) False
The possible answer is: correct choice
Knowledge:
passage
passage
[[]]]] passage
Question: question
Brainstormed Answers: likely answer sampled
sampled answer N
Poesible answer: most likely answer
Is the possible answer:
A) True
B) False
The possible answer is:
Table 15: Prompt used for p(true) approach. Items in blue are filled with in-context examples from the
training set and the current example being evaluated. N represents the number of sampled answers. The "sequence of in-context examples" prefix is sequence of examples taken from the training split with the same question format but with the answer to The possible answer is: resolved.
whether the gold answer is contained in the generated answer MMalenn et al. 2023 Asai et al. 2024 Xie et al., 2024) and accuracy using an LLM as judge (AccLM). Note that AccLM much higher than Acc across the board, which highlights the importance of using a better accuracy metric, especially when the target QA models not fine-tuned.
30

---

Published in Transactions on Machine Learning Research (09/2025)
Accuracy evaluation prompt.
You check whether the prediction of question-answering system to a question correct. You should make the judgment based on a ground truth answers provided to you. Your response should 'correct' the prediction correct or 'incorrect"
prediction is wrong.
Question:
Who authored The Taming of the Shrew (published in 2002)?
Ground truth: ("William Shakespeare', "Roma Gill']
Prediction: W Shakespeare
Currectness: correct
Question: Who authored The Taming of the Shrew (published in 2002)?
Ground truth: [ ['William Shakespeare', 'Roma Gill"]
Prediction: Roma Gill and W Shakespeare Correctness: correct
Question: Who authored The Taming of the Shrew (published in 2002)?
Ground truth: ['William Shakespeare', 'Roma Gill')'
Prediction: Roma Shakespeare
Correctness: incorrect
Question: What country Maharashtra Metro Rail Corporation Limited located in?
Ground truth: ["India"]
Prediction: Maharashira
Correctness: incorrect
Question: What's the job of Song Kang-ho in Parasite (2019)?
Ground truth: ["actor']
Prediction: He plays the role of Kim Ki-tack, the patriarch of the Kim family.
Correctness: correct
Question: Which did Michael Oakeshott belong to?
Ground truth: ("20th-century philosophy"]
Prediction: 20th century."
Correctness: correct
Question: Edward Tise (known for Full Metal Jacket (1987)) is in what department?
Ground truth: ('suund department". Prediction: Infantry Division, United States Army Correctness: incorrect
Question: What wine region is Finger Lakes AVA a part of?
Ground truth: ["New York wine")
Prediction: Finger Lakes AVA
Correctness: incorrect
Question: qucstion
Ground truth: gold answer
Prediction: generated answer Correctness:
Table 16: Prompt used for LLM-based accuracy evaluation.
G.2 Results on Individual Passage Utility Prediction
Beyond using Passage Utility to estimate uncertainty in retrieval augmented QA, evaluate how it performs on its own. Table 17 shows AUROC scores when using Passage Utility to predict accuracy for individual passages. We evaluate on the same samples from the test. sets in Section 5 and Table [ but per passage. We also include a perplexity baseline (PPL). Overall, results in Table 17 follow a similar pattern as those the QA setting with top-5 passages (Table D.. Across the board, Passage Utility demonstrates strong performance in predicting the usefulness of individual passages. Morcover, these results highlight that the quality of uncertainty estimation strongly depends the quality of individual Passage Utility predictions.
31

---

Published in Transactions on Machine Learning Research (09/2025)
GEMMA2-9B
GEMMA2-27B
NQ
TQA WebQ SQuAD PopQA RefuNQ AVG
NQ
TQA WebQ SQuAD PopQA RefuNQ AVG
PPL
0.64 0.62
0.53
0.56
0.45
0.50
0.55
0.61
0.61
0.53
0.57
0.52
0.54
0.56
Passage Utility
0.77 0.80
0.71
0.74
0.90
0.71
0.77
0.74
0.8
0.69
0.76
0.90
0.72
0.77
LLAMA-3.1-SI
MISTRAL-7B-VuL
NQ TQA WebQ SQuAD PopQA RefuNQ AVG
NQ
TQA WebQ SQuAD PopQA RefuNQ AVG
PPL
0.69 0.78
0.67
0.67
0.65
0.68
0.69
0.64
0.71
0.59
0.64
0.62
0.68
0.65
Passage Utility
0.74 0.80
0.71
0.78
0.91
0.75
0.78
0.75
0.77
0.70
0.78
0.88
0.78
0.78
Table 17:
AUROC values for the Passage Utility and perplexity baseline on individual passages on the six test sets (NQ, TQA, WebQ, SQuAD, PopQA, and RefuNQ).
NQ
TQA
WebQ
SQuAD
PopQA
RefuNQ
Development
Aoc
AccLM
Acc
AccLM
Acc
AceLM
Acc
AccLM
Ace
AccLM
Ace
AccLM
GEMMA2-9B
0.48
0.66
0.74
0.80
0.46
0.66
0.38
0.60
0.51
0.52
GEMMA2-27BB
0.48
0.66
0.75
0.81
0.49
0.67
0.38
0.60
0.52
0.52
LLAMA-3.1-8B
0.48
0.62
0.71
0.77
0.53
0.64
0.39
0.57
0.51
0.49
MISTRAL--B--O..3
0.48
0.62
0.72
0.76
0.52
0.69
0.37
0.58
0.53
0.51
NQ
TQA
WebQ
SQuAD
PopQA
RefuNQ
Test
Acc
AccLM
Acc
AccLM
Acc
AccLM
Arc:
AccLM
Ace
AceLM
Ace
AccLM
GEMMA2-9B
0.49
0.65
0.74
0.80
0.40
0.66
0.43
0.60
0.50
0.52
0.26
0.40
GEMMA2-27B
0.48
0.65
0.76
0.81
0.41
0.66
0.42
0.61
0.51
0.53
0.26
0.39
LLAMA-3.1-8B
0.49
0.61
0.71
0.77
0.44
0.63
0.43
0.58
0.50
0.49
0.27
0.36
MISTRAL-7B-VO.3
0.62
0.72
0.77
0.47
0.66
0.41
0.58
0.51
0.50
0.26
0.35
Table 18: QA model performance (with [R] 5) on the development and test sets. model-based accuracy (Acc and AccLM). AccLM is computed by Qwen2-77BB-Instruc.
We report token- and
G.3
Detailed Uncertainty Estimation Results
Table 19 shows AURAC scores on the test sets. Table 20 shows the performance of uncertainty quantification approaches on the development set. We report AUROC and AURAC.
Examples of False Positives and Negatives
Tables 21 24 illustrate the working of Passage Utility for answer uncertainty estimation. As we report /incorrect decision threshold; for the purpose of this discussion, we assume a decision point at 0.5 and analyze clear success and failure cases. For each example, we show the question, gold, and generated answers in the top block. Then, we show three retrieved passages with their estimated Passage Utility and a final block with ten sampled answers, their grouping into clusters, and the Cluster Assignment entropy.
AUROC scores, we do not set any
Table 21 shows example for SQuAD question and the LLAMA-3.1-8B QA model. In this case, the QA model correctly answers and the Passage Utility estimate is high (i.c., indicating the answer is correct). Table 22 illustrates a case where LLAMA-3. 3-8BBss answer is incorrect and all Passage Utilities are very low (i.e., indicating the answer is incorrect). The example from NQ in Table 23 shows case where all Passage Utilitics are low but the QA model (GEMMA2-9B) answers correctly. The first passage is useful, the second does not explicitly mention the answer but still primes the QA model to answer correctly, while the third passage mentions the answer.
32

---

Published in Transactions on Machine Learning Research (09/2025)
GEMMA2-9B WebQ SQuAD PopQA RefaNQ
GEMMA2-27B WebQ SQuAD PopQA RefuNQ
NQ
TQA
NQ
TQA
PPL
0.69 0.84
0.63
0.57
0.56
0.45
0.67
0.78
0.63
0.62
0.56
0.45
p(true)
0.75 0.85
0.71
0.63
0.71
0.53
0.76
0.89
0.73
0.67
0.70
0.54
Regular Entropy
0.70 0.84
0.63
0.59
0.57
0.46
0.69
0.79
0.64
0.61
0.58
0.45
Semantic Entropy
0.71 0.85
0.64
0.65
0.64
0.51
0.69
0.81
0.67
0.63
0.50
Passage Utility
0.76 0.90
0.72
0.74
0.73
0.64
0.72
0.88
0.73
0.74
0.73
0.64
LLAMA-3.1-4
MISTRAL-7B-V0.3
NQ
TQA WebQ SQuAD PopQA RefuNQ
NQ
TQA
WebQ SQuAD PopQA RefuNQ
PPL
0.73 0.87
0.71
0.68
0.69
0.54
0.67
0.83
0.66
0.66
0.61
0.54
p(true)
0.76 0.89
0.75
0.70
0.71
0.59
0.71
0.86
0.70
0.69
0.65
0.56
Regular Entropy
0.73 0.87
0.72
0.70
0.69
0.56
0.67
0.84
0.69
0.65
0.62
0.51
Semantic Entropy
0.71 0.87
0.71
0.70
0.67
0.54
0.68
0.85
0.69
0.51
Passage Utility
0.74 0.87
0.73
0.73
0.71
0.65
0.72
0.87
0.75
0.71
0.66
Table 19: AURAC values for QA models based on DEEMMA2-9B/77B LLAMA-3.1-8B, and MISTRAL-7B-VO.33 on Natural Questions (NQ), TriviaQA (TQA), WebQuestions (WebQ), SQuAD, PopQA, and RefuNQ test sets.
In Table 24 Passage Utility scores are high estimating a correct answer for the TQA test question: however, GEMMA2-9B answers with the incorrect magazine name. Note that none of the passages corresponds to the National Geographic magazine but have high token overlap with the question (in particular the first and second passages).
33

---

Published in Transactions on Machine Learning Research (09/2025)
AUROC WebQ SQuAD PopQA
AURAC
GEMMA2-9B
NQ
TQA
NQ
TQA
WebQ SQuAD PopQA
PPL
0.61
0.52
0.58
0.66
0.56
0.67
0.78
0.67
0.65
0.52
MSP
0.64
0.60
0.64
0.71
0.61
0.69
0.80
0.69
0.67
0.56
PMT
0.53
0.46
0.32
0.50
0.48
0.64
0.75
0.64
0.57
0.50
p(true)
0.70
0.71
0.66
0.73
0.83
0.72
0.84
0.70
0.69
0.71
Regular Entropy
0.64
0.54
0.60
0.70
0.58
0.69
0.78
0.68
0.67
0.54
Cluster Assignment
0.68
0.65
0.65
0.70
0.68
0.71
0.82
0.70
0.67
0.60
Semantic Entropy
0.67
0.69
0.64
0.72
0.69
0.71
0.84
0.69
0.68
0.61
Avg Angtneee
0.61
0.64
0.65
0.63
0.68
0.68
0.83
0.71
0.65
0.61
Retriever Score
0.53
0.62
0.67
0.64
0.65
0.82
0.63
0.67
0.61
Passage Utility
0.72
0.84
0.75
0.85
0.85
0.75
0.89
0.77
0.77
0.71
AUROC WebQ SQuAD
AURAC WebQ SQuAD
GEMMA2-27B
NQ
TQA
PopQA
NQ
TQA
PopQA
PPL
0.61
0.63
0.68
0.79
0.65
0.67
0.52
MSP
0.66
0.67
0.70
0.82
0.67
0.69
0.56
PMI
0.51
0.52
0.56
0.54
0.56
0.64
0.78
0.67
0.62
0.56
p(true)
0.76
0.73
0.69
0.69
0.79
0.78
0.84
0.74
0.71
0.70
Regular Entropy
0.65
0.53
0.56
0.64
0.53
0.71
0.78
0.66
0.67
0.52
Cluster Assignment
0.66
0.67
0.39
0.66
0.66
0.71
0.82
0.67
0.68
0.60
Semantic Eutropy
0.64
0.67
0.39
0.68
0.66
0.69
0.82
0.68
0.69
0.60
AvgAnsLen
0.63
0.68
0.65
0.60
0.69
0.69
0.83
0.72
0.66
0.61
Retriever Score
0.56
0.60
0.31
0.69
0.65
0.67
0.81
0.64
0.71
0.62
Passage Utility
0.73
0.75
0.72
0.84
0.87
0.75
0.86
0.75
0.78
0.73
AUROC
AURAC WebQ
ULAMA-3.1-8B
NQ
TQA
WebQ SQuAD
PopQA
NQ
TQA
SQuAD
PopQA
PPL
0.75
0.78
0.68
0.75
0.81
0.76
0.85
0.71
0.71
0.68
MSP
0.77
0.80
0.71
0.76
0.85
0.76
0.85
0.72
0.72
0.70
PMI
0.55
0.52
0.54
0.58
0.64
0.73
0.60
0.61
0.53
p(true)
0.80
0.86
0.82
0.85
0.78
0.87
0.74
0.75
0.71
Regular Entropy
0.77
0.76
0.76
0.85
0.71
0.72
0.69
Cluster Assignment
0.75
0.75
0.85
0.71
0.67
Semantic Eutropy
0.70
0.74
0.75
0.86
0.72
0.68
AvgAnsLen
0.73
0.69
0.69
0.73
0.82
0.71
0.67
0.69
Retriever Score
0.58
0.63
0.34
0.68
0.66
0.65
0.79
0.62
0.66
0.60
Passage Utility
0.78
0.85
0.74
0.82
0.86
0.75
0.87
0.75
0.75
0.71
AUROC
AURAC WebQ
MISTRAI-7B-vO
NQ
TQA
WebQ SQuAD
PopQA
NQ
TQA
SQuAD
PopQA
PPL
0.66
0.70
0.60
0.63
0.66
0.69
0.84
0.72
0.63
0.63
MSP
0.70
0.75
0.65
0.71
0.77
0.70
0.85
0.73
0.68
0.67
PMI
0.38
0.33
0.42
0.42
0.30
0.53
0.68
0.62
0.52
0.39
p(true)
0.72
0.82
0.71
0.75
0.74
0.71
0.87
0.76
0.71
0.64
Regular Entropy
0.67
0.71
0.63
0.66
0.68
0.69
0.85
0.73
0.66
0.63
Cluster Assignment.
0.72
0.81
0.68
0.73
0.76
0.71
0.87
0.75
0.68
0.66
Semantic Entropy
0.72
0.80
0.68
0.73
0.76
0.71
0.87
0.76
0.69
0.66
AvgAusLen
0.66
0.75
0.65
0.68
0.81
0.69
0.85
0.73
0.67
0.70
Retriever Score
0.55
0.65
0.63
0.81
0.67
0.63
Passage Utility
0.76
0.81
0.85
0.74
0.87
0.78
0.75
0.71
Table 20: Answer uncertainty estimation for QA models CRMMAA-9/27BB, 1LAMA-3.1-8B, and MISTRAL- 7B-v0.3 on NQ, TQA, WebQ, SQuAD, and PopQA development sets. We report AUROC and AURAC.
34

---

Published in Transactions on Machine Learning Research (09/202))
Question:
How many platforms does Manchester Oxford Road have?
Generated Answer: Four through platforms and terminating bay platform.
Gold Answer:
Originally one, later expanded.
Manchester Oxfard Road railway station. Gloucester Street (now Whitworth Street Mest) to reach their north front. There was single platform the north side of the line through Manchester London Road (now "Manchester Pic adilly") and second platforn west-facing siding. To allow for extra trains in connection with the Nanchester Art Treasures Exhibition in 1857, extra platforns and sidings
of complaints the want of punctuality "especially as at Oxford Road station there is only one platforn used for both passengers and milk". -8.42
neeting
holders
included
terminating, through.
Transport in Manchester. platforns are planned as part of the Northern Hub praject however 2017 the construction was not confirmed. through platforms. Longestion
present there twelve terminating platforms but often incur delays for following trains down the line at Uxford Road. The new proposals will provide up to 709 more trains per day upgrades. The station
at peak times and trains
nore people was rated the worst in the UK following a 2009 study and plans far
space
by
year.
has undergone
-8.26
Five platforns.
Manchester Oxford Road railway station. Manchester Oxford Road railway station is a rai lway station in Manchestar, England, at the rebuilt in It the secord busiest of four
junction of whitworth Street and exford Street. opened 1849 stations in Manchester city centre. The station serves the southern part of Manchester city centre, the University of Nanchester Manchester Metropolitan University, on Eastbound trains beyond Piccadilly to and The station consists of four through platforns and one terminating bay platform. 1.34
restwards
Marrington, Chester, Llandudno, Liverpool, Blackpaol.
('Four through platforns and one terninating bay platforn.', "Four through platforms one terninating bay.", "Four through platforns and terninating bay platforn. '], ['Four through and terminating. through platforms and terminating bay.', 'Four through and one bay. 'J, ['Five platforns.', 'Five platforns.', "Five platforns.', 'Five platforns.")
1.09
Table 21: True negative example (from NQ development set): Passage Utility predicts the right answer well as the QA model (GEMMA2-9B).
35

---

Published in Transactions on Machine Learning Research (09/2025)
Question: Which company largeted by the NAACP for not having fair practices?
Generated Answer: Target Corporation.
Gold Answer: Lockheed Aircraft Corporation.
Target. Corporation.
Target Corporation. of Colored People has repeatecly given Target failing grades on its annual Econonic Reciprocity Initiative report card, a measure of the company's "commitrent to the African-Anerican citizenry". In 2883 and 2885, the NAAC? has rated Target "p" this report; in 2804, Target rated "D-". again, representative explained, "Target views diversity as being inclusive of all people from all different backgrounds, not just group."" February 2806, National Federation of the Blind filed class [cont.]
2606, when Target asked it didn't participate in the survey
-2.64
None, the NAAP was involved in the Duke lacrosse case.
Reactions to the Dake lacrosse case. formed an opinion on the case. North Caralina NAACP Legal Redress Chair, Al McSurely, explained that "The NACP stands for fair play for all parties, zealous investigation and deep concern for the survivers racist/sexist attacks." " the evidence to the contrary, using the case to pronote the group's cause, and implying guilt. McSurely stated that "[w]ithin five nisogynist verbal assaults, [cont.]
time, have criticized the NAAPP for making statements that portrayed the players racist despite
the threatened
with
-2.77
Philadelphia Transit Company (PTC).
Phi Philadelphi transit strike of 1944, (PATHU), Frank Carney, proved to be equally reticent and clained that he was not authorized by the union nembers to consider request to allow promotions black employees. The black PTC employees enlisted the help NAACP and started lobbying the federal authorities, particularly the Fair Employment Practices Comnission (FEPC), intervane. The Fair Employment Practices commission, created by an executive order of the President in 1941, was charged witth ensuring non-discrimination employment practices by governnent contractors. [cont.]
-2.60
['Target, '], D'Target Corporation.'s "Target Corporation.', 'Target Corporation. "Target Corporation.', 'Target Corporation 'Target corporation." 'Target Corporation.'. "Target Corporittion ('Target Corporation the National Federation the Blind filed lawsuit egainst Michigan. ']
0.33
Table 22: True positive example (from SQuAD development set): Passage Utility predicts the answer provided by the QA model (TLAMA-3. 1-8B) which is incorrect.
36

---

Published in Transactions on Machine Learning Research (09/202))
Question: What do Alfred Newman, John Williams, Johnny Green, Andr Previn, John Barry and Alan Menken have common?
Generated Answer:
They are all Oscar-winning film composers. Gold Answer: Winning the Academy award for Best Original Score.
They all mul tiple Acadeny Awards for film
Alan Menken. his other film musicals also nominated: Alan Alan Irwin
(born July 949) Anerican musical theatre and film score composer and planist. Menken is best known for his scores for films produced by Walt Disney Animation Studios. scores for Little hin two Acadeny Awards. He also composed the scores for "Little Shop of Horrors" (1986), "Nowsies" (1992), "The Hunzhback of Notre Dame (1995), "Hercules" (1997), "Home
(1989 ), "Beauty
(1991), "Aladdin" (1992), and "Pocahentas" (1995) each
Range" (2084), "Enchanted" (2807), "Tangled" (2010), and "Sausage Party" (2016) -2.22
They all prolific Oscar winners in music categories.
Alan Henken.
musical theatre works for Broadway elsewhere. stage hits include "Little Shop of Horrors" (1982). "A Christna Carol" (1994) and "5 ister Act" (2889). Menken has collaborated Ashman, Rice, Glenn Slater, Stephen Schwartz David Zippel. eight Acadeny wins (four each for Best Score and Best Song), Nenken is the second most prolific Oscar winner the music categories after Alfred Newrnn, who -2.30
based his Disney films, other
with such lyricists
has nine Oscars. He has also won eleven Gramny Awards, Tony Award and other honors.
all Academy Award-winning compasers.
ngs from writing new songs with Benj Pasek and Justin Poul for a live-action film version of "Aladdin" and is also working with Lin-Hanuel Miranda new
Menken. Beast",
naterial
and Menken is collaborating
live-action film adaptation Schwartz write new sorgs for Disenchanted; the equel to "Enchanted". With eight Acadeny Awards (four each for best score and best song), only composer Alfred Newnan (nine wins) and Malt Disney (22 wins) have
Hermaid". Henken
once again working Stephen
-2.31
['They are all Academy Award-wining composers.', 'They are all Acadany Award-winning composers.', "They are all Oscar-winning film composers." ', 'They are all Oscar-winning film composers.") 'J, ['They are multiple-tine Oscar winners music.", 'They are prolific Aradeny kward-winning corposers. 'J, ['They all multi-Oscar winning film composers.', 'They all multiple Ostar-winning composers.', ;, 'They are prolific oscar-winning film composers." '], ('They are all prolific film music conposers." "]
1.28
Table 23: False positive example (from TQA development set): Passage Utility predicts wrong answer whereas the prediction of the QA model (GEMMA2-9B) is correct.
37

---

Published in Transactions on Machine Learning Research (09/202))
Question: One Step Behind' is a tribute act to which band? Generated Answer: Furniture.
Gold Answer: North London Invaders.
Def eppard.
Steps Behind. Two Behind Steps Behind" is 1993 song by English Active" and "Last Action Here" Soundtrack. It reached V5 on the "Billboard" loot Mainstrean Rock Tracks chart, and al2 on the Billboardiot Though the band's last major hit in the us. In the 1993 "Metal Edge" Readers" Choice Awards, the song was voted "Song of the Year" and "Best Song Fron Novie Soundtrack." Def Leppard have
band Def Leppard from their albun "Retro
nanage
songs following years, generally considered
3.29
Def Leppard.
Steps Behind. and footage. song for her 2887 albun "Obsession".. Two Steps Behind "Two Steps Behind" is 1993 song by English hard rock band Def Leppard from their
was aired August Filipino-Chinesd singer Rachelle covered the
"Billboard" Hainstrean Rock Tracks chart, and #12 the "Billboard" Hot 100. Though the band did manage chart few nore songs in the following years, this generally considered to be the band's 1ast najor hit the US. In the
Active"
Action
Soundtrack.
reached
2.42
Split Enz.
Step Ahead (split song). unavailable Australasian narkets until when became available iunes)) The video clip to "One Step Ahead" has keyboardist Eddie Rayner performing "Marche sur place". the pantomine illusion walk created by Decroux and Paradise)
Barrault (seen
technique Micheel 1983. One Step Ahead (Split Enz song) "One Step Ahead" is 1988 sang New Zealand art rack group Split Enz from the ir studio albun "Maiata". The song continued group's success their move towards their own version new wave
French film
base
on
-2.93
('Furniture', 'Furniture', "Furni ture', Turniture', FFunnturr',
'Furn ture', Furnituree, Furniture',
'Furni ture',
Furnituree]
Table 24: False negative (from TQA development sct): Passage Utility predicts correct answer, and the answer by the QA model (GEMMA2-9B) is wrong.
38


Reviews:
[{"review_id": "di4xedaUmb", "reviewer": "Reviewer_r8hv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed framework is firmly rooted in problem structure: by emphasizing passage utility instead of response variability, the method capitalizes on the inherent retrieval‑QA pipeline. Moreover, avoiding repeated LLM sampling yields substantial advantages in settings constrained by latency or cost. \n- For smaller LLMs, the authors provide compelling empirical evidence. The results are clearly presented, and the accompanying discussion effectively reinforces the core claims.\n- The exploration of reranking via Passage Utility (Figure 4) is particularly noteworthy. These experiments reveal that utility‑based reranking not only serves as an uncertainty proxy but also enhances QA accuracy relative to standard retriever relevance scores, thereby underscoring the dual applicability of the approach.\n\nWeaknesses:\n- The necessity of training a neural predictor introduces a dependency on sufficient data, potentially constraining applicability. As illustrated in Appendix D, out‑of‑distribution generalization remains limited; thus, in low‑resource contexts, the method’s viability may be compromised.\n- Section 4’s organization could benefit from revision. Although comprehensive, the presentation occasionally shifts abruptly between topics, which may impede reader comprehension.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce an intuitive approach for quantifying the uncertainty of LLM-generated responses within a retrieval-augmented QA framework. Rather than relying on multiple sample generations, their method predicts the utility of individual retrieved passages via a lightweight neural network and employs this estimate to derive an uncertainty measure for the LLM’s response. Empirical evaluations across six QA benchmarks and three smaller-scale LLMs demonstrate that this strategy outperforms conventional information‑theoretic uncertainty metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following are not critical but would strengthen the work:\n- Reorganize Section 4 to establish a more coherent narrative flow.\n- In Equation 2, it appears that a passage yielding an accurate yet non‑entailing response is assigned identical utility to one that is inaccurate but entails the target. Please clarify if this interpretation is correct, and if so whether this equivalence is intentional. If intentional, provide an intuitive rationale for its desirability.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bBN4O2lDN6", "reviewer": "Reviewer_f1fE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is clearly written and easy to understand.\n- The proposed method is innovative, and sufficient technical details are introduced. \n- The approach is thoroughly evaluated with extensive performance analysis.\n\nWeaknesses\n- My primary concern relates to the dataset used for training the passage utility predictor. The paper currently does not provide sufficient details about this dataset. Including statistics and a more comprehensive description of the curated dataset would be valuable and enhance the paper’s clarity. \n- A separate passage utility predictor needs to be trained for each base model and task, which is a limitation of the current method and experimental setup. It raises the question of whether it is possible to develop a more general passage utility predictor that can be shared across different models and tasks. Minor comment: In this context, the passage utility predictor is analogous to a reward model, which is often trained using the Bradley-Terry (BT) loss. It would also be interesting to explore alternative loss functions for training of the passage utility predictor.\n- Since the overall performance of the method depends on the effectiveness of the passage utility predictor, it is important to include a separate evaluation of the predictor’s performance. This would provide valuable insight into how well the predictor functions and its impact on the final results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new approach for assessing uncertainty in retrieval-augmented question answering. The authors hypothesize that the usefulness of each retrieved passage reflects the model’s uncertainty in producing the correct answer. To leverage this, they propose training a compact model to estimate passage utility, which is then used to predict answer uncertainty. The method is lightweight, efficient, and effective, and is evaluated across multiple QA datasets with comprehensive baseline comparisons.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some of the requested changes are directly related to the previously mentioned weaknesses. In addition, some aspects of the presentation could be improved to enhance clarity. \n  - In Equation 4, there are two instances of $ p $ in $ p(x, p) $, which is somewhat unusual. It might be clearer to use a different notation for the inner variable, such as $ p_1 $ or $ p_a $.\n  - In Table 1, the term \"Utility Ranker\" is used in the experimental results without being formally defined in the text.\n  - In Figure 2, it is unclear which data points correspond to which methods, as the legend lacks sufficient detail.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0y3WTHt9Mh", "reviewer": "Reviewer_FhU7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strenghts\n- The paper addresses the important problem of uncertainty quantification in retrieval-augmented QA systems.\n- The proposed method shows performance improvements on several QA models and datasets. \n- The proposed contrastive learning framework with a pairwise ranking loss is a well-designed choice for passage utility prediction. By directly optimizing for relative utility differences between passages conditioned on the same question, the objective function of Eq.(3) enables consistent learning of passage ranking. \n\n## Weaknesses\n\n- The proposed Utility Ranker is a passage utility predictor trained according to the output behavior of the QA model M. When switching to a new QA model, the existing utility predictor becomes non-reusable because it requires retraining for each model change, which creates a weakness of frequent fine-tuning requirements.\n\n- It may appear to combine existing research areas, such as passage quality assessment and uncertainty quantification.\n\n- The explanation for the passage utility definition in Equation 2 is unclear.\n\n- The analysis and explanation of the HotPotQA experiment are sparse compared to other experiments.\n\n- Unlike entropy-based or p(true)-style estimators directly based on model outputs or self-judgment, the utility ranker relies on supervised learning based on QA results and entailment judgments, making it seem dependent on models and tasks.\n\n- Although the contrastive ranking objective is well designed to model relative phrase utility, the pairwise comparison approach appears unlikely to guarantee global ranking consistency.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a passage-level utility prediction approach for uncertainty quantification in retrieval-augmented QA systems. This method provides a novel uncertainty estimation that can identify which specific passages contribute to unreliable QA performance and offers an alternative to existing answer-level uncertainty methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- While the experimental settings and implementation cite `Farquhar et al. (2024)`, the related work section does not thoroughly discuss how this research relates to the authors' study. I suggest the authors describe the relationship and differences with this research in more detail.\n\n- Can you explain why a simple average of accuracy and entailment is used in Eq.(2)?\n\n- Analysis seems needed on how it works when $υ_{\\mathcal{M}}  = a(y)$ and when $υ_{\\mathcal{M}}  = e(y)$ in  Eq.(2). Or it would be good if you could report changes according to α values in $υ_{\\mathcal{M}} = α·a(y) + (1-α)·e(y)$.\n\nThe OOD performance drops in Table 6, and it would be good to analyze this reason further, even if you mentioned leaving it as future work.\n\n- The authors state about HotPotQA results that \"This evaluation shows that the idealized multi-hop behavior studied with synthetic passages occurs to a lesser extent in current LLMs.\" However, doesn't this stem from limitations in the authors' experimental design rather than dataset issues? Particularly, isn't the authors' aggregation method (Eq. 6) inappropriate for multi-hop scenarios?\n\n- As I understand it, since retraining is required whenever the model or task changes, frequent fine-tuning makes it difficult to apply to new QA environments without retraining. Can you discuss more about these limitations?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)