Axiomatic Analysis of Uncertainty Estimation For Retrieval Augmented Generation
Authors: Heydar Soudani, Evangelos Kanoulas, Faegheh Hasibi
Accepted at: acl25

Why Uncertainty Estimation Methods Fall Short in RAG:
An Axiomatic Analysis

Heydar Soudani
Radboud University
The Netherlands
heydar.soudani@ru.nl

Evangelos Kanoulas
University of Amsterdam
The Netherlands
e.kanoulas@uva.nl

Faegheh Hasibi
Radboud University
The Netherlands
faegheh.hasibi@ru.nl

Abstract

Large Language Models (LLMs) are valued for their strong performance across various tasks, but they also produce inaccurate or misleading outputs. Uncertainty Estimation (UE) quantifies the model’s confidence and helps users assess response reliability. However, existing UE methods have not been thoroughly examined in scenarios like Retrieval-Augmented Generation (RAG), where the input prompt includes non-parametric knowledge. This paper shows that current UE methods cannot reliably estimate the correctness of LLM responses in the RAG setting. We propose an axiomatic framework to identify deficiencies in existing UE methods. Our framework introduces five constraints that an effective UE method should meet after incorporating retrieved documents into the LLM’s prompt. Experimental results reveal that no existing UE method fully satisfies all the axioms, explaining their suboptimal performance in RAG. We further introduce a simple yet effective calibration function based on our framework, which not only satisfies more axioms than baseline methods but also improves the correlation between uncertainty estimates and correctness.

1 Introduction

Large Language Models (LLMs) have recently demonstrated promising capabilities in various tasks, including question-answering, and various classification and clustering tasks (Jin et al., 2025; Lin et al., 2024a; Soudani et al., 2024b; Trivedi et al., 2023). However, LLMs are prone to generating incorrect information for multiple reasons, such as lack of parametric knowledge (Mallen et al., 2023), temporal knowledge shifts (Zhao et al., 2024a; Kordjamshidi et al., 2024), or noisy information introduced through retrieved documents in Retrieval-Augmented Generation (RAG) (Soudani et al., 2024a; Min et al., 2023). As a result, the trustworthiness of LLM-generated responses has become a critical concern, directly impacting user satisfaction (Hou et al., 2024; Mahaut et al., 2024).

Uncertainty Estimation (UE) is a widely studied approach for assessing the reliability of LLM outputs. A UE method assigns an uncertainty score to each (input, output) pair, reflecting its truthfulness. Ideally, a perfect UE method would assign lower uncertainty to correct samples and higher uncertainty to incorrect ones (Duan et al., 2024). While existing UE methods mainly focus on scenarios where the input is just a query, real-world applications like RAG involve non-parametric knowledge in more complex prompts (Huang et al., 2024). Research shows that non-parametric knowledge significantly influences LLM responses, often aligning them with the provided context (Cuconasu et al., 2024; Mallen et al., 2023). Despite this, it is unclear how current UE methods account for non-parametric knowledge.

In this paper, we investigate a critical question: (RQ1) How do UE methods perform when the input prompt includes non-parametric knowledge,

![Desired behavior of uncertainty estimation methods with and without RAG. For instance, the first row indicates that when an LLM generates a correct response both without and with RAG (i.e., the retrieved document supports the internal belief of the LLM), the uncertainty in the RAG setup should decrease compared to the no-RAG setup. These principles form an axiomatic framework for evaluating and understanding uncertainty behavior in RAG.](page_1012_370_388_246.png)

Figure 1: Desired behavior of uncertainty estimation methods with and without RAG. For instance, the first row indicates that when an LLM generates a correct response both without and with RAG (i.e., the retrieved document supports the internal belief of the LLM), the uncertainty in the RAG setup should decrease compared to the no-RAG setup. These principles form an axiomatic framework for evaluating and understanding uncertainty behavior in RAG.
such as in RAG? We study UE in the context of RAG with retrievers of varying effectiveness: (i) a deliberately weak synthetic retriever that returns irrelevant documents, (ii) an idealized retriever that consistently ranks the gold document at the top, and (iii) several widely used retrievers with varying performance levels. Our findings unveil that the performance of existing UE methods is inconsistent and mainly deteriorates when non-parametric knowledge is included in the input prompt. Most notably, improvements on the proposed UE methods in the literature do not add up when considering RAG setup.

Against this background, it is clear that UE requires a methodological departure; existing methods are developed without paying attention to the specific properties that UE methods must satisfy in the RAG setup. The question that arises here is: (RQ2) What properties can guarantee optimal performance of UE considering LLMs’ both parametric and non-parametric knowledge? We approach this question theoretically using axiomatic thinking, which is proven effective in various fields and tasks, including information retrieval (Fang and Zhai, 2005; Bondarenko et al., 2022), interpretability (Chen et al., 2024; Parry et al., 2025), and preference modeling (Rosset et al., 2023). In axiomatic thinking, a set of formal constraints is defined based on desired properties, which are then used as a guide to search for an optimal solution. In this work, we define an axiomatic framework for UE and establish five axioms considering the desired behavior of a UE method with and without external knowledge. Our axiomatic analysis reveals that current UE methods can satisfy only two axioms, violating the remaining three axioms in the majority of cases.

The axiomatic framework helps explaining deficiencies of existing UE methods for the RAG setup. The next question is: (RQ3) Can the axiomatic framework guide us in deriving an optimal UE method? We use the constraints of the axiomatic framework to define a calibration function based on three components. We implement three instantiations of this function and apply it to different UE methods on a number of representative datasets. The results show that the derived functions are not only more stable than the existing UE methods but also improve overall performance with respect to AUROC. This highlights two key insights: first, satisfying the axioms leads to performance improvements, and second, existing UE methods can still be used for RAG by incorporating an axiomatically informed coefficient.

The main contributions of this paper include:
(1) Analyzing existing UE methods and showing their deficiencies in RAG setup.
(2) Proposing an axiomatic framework for UE with five formalized constraints and demonstrating deficiencies of existing methods in satisfying them.
(3) Introducing a calibration function guided by axioms and showing consistent improvements of the UE methods as a result of alignment with axioms.

2 Background

UE methods are typically divided into white-box approaches, which utilize token probabilities and entropy (Kadavath et al., 2022; Kuhn et al., 2023), and black-box approaches, which rely solely on final outputs (Lin et al., 2024b; Band et al., 2024). This section reviews methods of both categories that are explored in this paper. For further details on related work, see Appendix A.

2.1 White-box Methods

Predictive Entropy (PE) for generative models quantifies uncertainty as the entropy of responses for an LLM input. The entropy is maximized when all outcomes are equally likely, indicating low informativeness (Kadavath et al., 2022; Kuhn et al., 2023). Given an LLM parametrized by \( \theta \) and an input \( x \), the LLM uncertainty is estimated by computing entropy using Monte-Carlo approximation:

\[
PE(x, \theta) = -\frac{1}{B} \sum_{b=1}^{B} \ln P\left(r_b \mid x, \theta\right),
\]

where \( r_b \) is a beam-sampled response and \( B \) is the number of samples. The probability of generating a response \( r = \{r^1, r^2, ..., r^N\} \), comprising \( N \) tokens, given the input \( x \) is computed as the product of the conditional probabilities of each token, given its preceding tokens and the input \( x \). For a model with parameters \( \theta \), the sequence probability is defined as:

\[
P(r \mid x, \theta) = \prod_{n=1}^{N} P\left(r^n \mid r^{<n}, x; \theta\right),
\]

where \( r^{<n} \) denotes the tokens generated before \( r^n \).

Semantic Entropy (SE) (Kuhn et al., 2023) extends PE by incorporating the semantic meaning of sampled responses. In this approach, generated samples are clustered into semantic clusters \( c_i \in C \),
and SE is defined as:
\[
SE(x, \theta) = -\frac{1}{|C|} \sum_{i=1}^{|C|} \log \tilde{P}(c_i \mid x, \theta),
\]
where \( c_i \) represents a semantic cluster, containing semantically similar responses. The cluster score \( \tilde{P}(c_i | .) \) is computed as:
\[
\tilde{P}(c_i \mid x, \theta) = \sum_{r \in c_i} P(r \mid x, \theta).
\]

**Length Normalization and Semantic Awareness** are two important components in UE. It has been observed that the sequence probability in Equation (2) is biased against longer generations (Malinin and Gales, 2021). To address this, a length-normalized probability is introduced to generate equal weighting of tokens and reduce bias toward shorter sequences:
\[
P_{ln}(r \mid x, \theta) = \prod_{n=1}^N P\left(r^n \mid r^{<n}, x; \theta\right)^{\frac{1}{N}}.
\]

MARS (Bakman et al., 2024) and Token-SAR (Duan et al., 2024) further refined this approach by incorporating semantic importance. These approaches assign weights based on each token’s contribution, resulting in the meaning-aware probability:
\[
P_{me}(r \mid x, \theta) = \prod_{n=1}^N P\left(r^n \mid r^{<n}, x; \theta\right)^{w(r,x,N,n)}
\]
where \( w(r, x, N, n) \) is the importance weight for the \( n \)-th token. Both the length-normalized and meaning-aware probabilities can be used in the PE (1) and SE (3) equations.

**2.2 Black-box Methods**

We examine state-of-the-art semantic similarity-based methods (Lin et al., 2024b), following these steps: (i) generate \( B \) sampled responses \( \{r_1, \ldots, r_B\} \) for a given input \( x \); (ii) compute pairwise similarity scores \( a_{i,j} = a(r_i, r_j) \) between the responses; and (iii) derive uncertainty from these scores. Three approaches are proposed for computing uncertainty scores, described below.

*Sum of Eigenvalues (EigV)* (Lin et al., 2024b). SE groups responses into semantic equivalence subsets and uses their count (*NumSet*) as an uncertainty metric; greater diversity implies higher uncertainty. To compute a more nuanced and continuous value for uncertainty than *NumSet*, Lin et al. (2024b) define uncertainty as:
\[
U_{EigV}(x) = \sum_{k=1}^B \max\left(0, 1 - \lambda_k\right),
\]
where \( \lambda_1, \ldots, \lambda_B \) are the eigenvalues of symmetric normalized Graph Laplacian (von Luxburg, 2007), defined as:
\[
L := I - D^{-\frac{1}{2}} W D^{-\frac{1}{2}}.
\]
Here, \( W \) represents a symmetric weighted adjacency matrix for a graph, where each node represents a response \( r_i \) for input \( x \) and weights are \( w_{i,j} = (a_{i,j} + a_{j,i})/2 \). The degree matrix \( D \) is defined as:
\[
D_{i,j} = \begin{cases}
\sum_{j' \in [B]} w_{i,j'} & \text{if } i = j, \\
0 & \text{if } i \neq j.
\end{cases}
\]
(5)

*Degree Matrix (Deg)* relies on the degree matrix in Eq. (5) to computer uncertainty. Here, the intuition is that \( D \) reflects node connectivity, and nodes with higher degrees indicate confident regions in the LLM (Lin et al., 2024b). Building on this, the uncertainty score is computed by:
\[
U_{Deg}(x) = \operatorname{trace}(B I - D)/B^2.
\]

*Eccentricity (ECC)* is defined as the average distance of response embeddings from their centroid, which can serve as an uncertainty measure. Since access to the embeddings is not possible in black-box LLMs, the embeddings are driven from graph Laplacian. Let \( \mathbf{u}_1, \ldots, \mathbf{u}_k \in \mathbb{R}^B \) be the \( k \) smallest eigenvectors of \( L \). For each response \( r_j \), define the embedding as \( \mathbf{v}_j = [u_{1,j}, \ldots, u_{k,j}] \) (Ng et al., 2001), and its centroid as \( \mathbf{v}_j' = \mathbf{v}_j - \frac{1}{B} \sum_{j'=1}^B \mathbf{v}_{j'} \). Uncertainty is computed as:
\[
U_{ECC}(x) = \left\| \left[ \mathbf{v}_1^\top, \ldots, \mathbf{v}_B^\top \right] \right\|_2.
\]

**3 Axiomatic Framework**

The assumption of an axiomatic framework for UE is that by satisfying a set of formal constraints, a UE method would likely have an optimal correlation with correctness for both RAG and no-RAG setups. To define the framework, we introduce five *axioms* based on a set of *functions* that form our search space for an optimal UE. These axioms, while necessary, do not represent an exhaustive list, as increasing the number of axioms can, in reality, introduce stringent, contradictory, or biased constraints. In the following, we introduce the functions and constraints of our axiomatic framework.

**3.1 Functions**

We define UE as the task of learning a function \( \mathcal{U} \) that predicts a score \( s \), quantifying the LLM’s uncertainty for its output (Liu et al., 2024). Formally, let \( x \) be the input given to a generative LLM \( \mathcal{M}_\theta \), parameterized by \( \theta \). The uncertainty estimator
function is formulated as follows:
\[
\mathcal{U} : \mathcal{M}_\theta(x), r \mapsto s
\]
where the input consists of an LLM with the given input \( x \) and a generated response \( r \). In a no-RAG setting, the input \( x \) is only the query \( q \), while for the RAG setup, the input \( x \) consists of a query \( q \) and a context \( c \), denoted as \( \mathcal{M}_\theta(q, c) = r \). We define context \( c \) broadly, including an individual document or a set of documents.

Before defining the axioms, we introduce functions that formalize the relation between a context, a query, and an LLM-generated response. These functions, defined based on Natural Language Inference (NLI) (Pavlick and Callison-Burch, 2016; Williams et al., 2018), are as follows:

Entailment (\( c \models (q, r) \)): Given the context \( c \), a human can infer that \( r \) is the correct response to the query \( q \); i.e., the premise \( c \) entails the hypothesis \( (q, r) \) (asymmetric relation).

Contradiction (\( c \bot (q, r) \)): Given the context \( c \), a human can infer that \( r \) is an incorrect response to \( q \); i.e., the premise \( c \) contradicts the hypothesis \( (q, r) \) and vice versa (symmetric relation).

Independence (\( c \# (q, r) \)): Given the context \( c \), a human cannot infer any information about the correctness of response \( r \) to query \( q \); i.e., the premise \( c \) does not guarantee the truth or falsity of hypothesis \( (q, r) \) and vice versa (symmetric relation).

Equivalence (\( r_1 \equiv r_2 \)): Two LLM responses, \( r_1 \) and \( r_2 \), convey the same meaning; i.e., the premise \( r_1 \) entails the hypothesis \( r_2 \) and vice versa (symmetric relation).

3.2 Axioms

The axioms are defined based on two key assumptions to ensure the validity of axioms and the four aforementioned functions:

Assumption 1. The context \( c \) is trustworthy and contains factually correct information.

Assumption 2. The context \( c \), given to the LLM for the query \( q \), does not contain contradictory information about the query \( q \).

We now define five constraints that any reasonable UE method should satisfy, considering LLM’s both parametric and non-parametric knowledge. Our working hypothesis is that UE is a proxy for the correctness of the model (Bakman et al., 2024). Two of these constraints are proven based on this hypothesis, and three of them are intuitively driven.

Theorem 1 (Positively Consistent). \( \forall q, c \) if \( \mathcal{M}_\theta(q) = r_1, \mathcal{M}_\theta(q, c) = r_2, r_1 \equiv r_2, c \models (q, r_2) \), then \( \mathcal{U}(\mathcal{M}_\theta(q), r_1) > \mathcal{U}(\mathcal{M}_\theta(q, c), r_2) \).

This constraint states that if applying RAG does not alter the LLM’s response and the RAG context supports LLM’s generated response \( r_2 \), then LLM’s internal belief aligns with the context. In such a scenario, the uncertainty after applying RAG should be lower than before, as the retrieved context reinforces the LLM’s prior knowledge. For instance, consider the example in Figure 1. Given the query, "What is the name of Manchester United’s stadium?" if the LLM initially generates the correct response, "Old Trafford," and the input context mentions "Old Trafford" as the name of the stadium, then the uncertainty value after applying RAG should be lower than before.

Theorem 2 (Negatively Consistent). \( \forall q, c \) if \( \mathcal{M}_\theta(q) = r_1, \mathcal{M}_\theta(q, c) = r_2, r_1 \equiv r_2, c \bot (q, r_2) \), then \( \mathcal{U}(\mathcal{M}_\theta(q), r_1) < \mathcal{U}(\mathcal{M}_\theta(q, c), r_2) \).

This constraint states that if the LLM’s response remains unchanged after applying RAG, but the retrieved context \( c \) contradicts the generated response \( r_2 \), then the LLM’s internal belief does not align with the context. In such a case, the uncertainty after applying RAG should be higher than before, as the retrieved information challenges the LLM’s internal belief. For example, in Figure 1, if LLM’s response before and after RAG is "Wembley Stadium," and RAG context contradicts the LLM’s response, then the uncertainty of the RAG response should increase. This means that although the LLM persists with its incorrect response, it does so with a lower confidence.

Theorem 3 (Positively Changed). \( \forall q, c \) if \( \mathcal{M}_\theta(q) = r_1, \mathcal{M}_\theta(q, c) = r_2, \neg(r_1 \equiv r_2), c \bot (q, r_1), c \models (q, r_2) \), then
\[
\mathcal{U}(\mathcal{M}_\theta(q), r_1) > \mathcal{U}(\mathcal{M}_\theta(q, c), r_2).
\]

Theorem 3 directly follows from the statement in the following lemma:

Lemma 1. If \( \mathcal{M}_\theta(x_1) = r_1, \mathcal{M}_\theta(x_2) = r_2, r_1 \) is False, \( r_2 \) is True, then
\[
\mathcal{U}(\mathcal{M}_\theta(x_1), r_1) > \mathcal{U}(\mathcal{M}_\theta(x_2), r_2).
\]

Proof. Given Assumptions 1 and 2 and \( c \bot (q, r_1) \), then response \( r_1 \) is False. Similarly, given that \( c \models (q, r_2) \), then response \( r_2 \) is True. Given these events and Lemma 1, then \( \mathcal{U}(\mathcal{M}_\theta(q), r_1) > \mathcal{U}(\mathcal{M}_\theta(q, c), r_2) \). \( \square \)

This constraint states that if the LLM’s response changes from \( r_1 \) to \( r_2 \) after applying RAG, and the RAG context \( c \) supports \( r_2 \) while contradicting
\( r_1 \), then the estimated uncertainty for \( r_2 \) should be lower than one for \( r_1 \). For example, consider the case illustrated in Figure 1. If the LLM initially generates "Wembley Stadium" but then, after seeing a context containing the correct response, changes its output to "Old Trafford," the uncertainty of "Old Trafford" with RAG should be lower than the uncertainty of "Wembley Stadium" without RAG.

**Theorem 4 (Negatively Changed).** \( \forall q, c \) if \( \mathcal{M}_\theta(q) = r_1, \mathcal{M}_\theta(q, c) = r_2, \neg(r_1 \equiv r_2), c \models (q, r_1), c \bot (q, r_2) \), then
\[
\mathcal{U}(\mathcal{M}_\theta(q), r_1) < \mathcal{U}(\mathcal{M}_\theta(q, c), r_2).
\]
This theorem follows from the statement in the Lemma 1 with the following proof.

*Proof.* The proof is similar to that of Theorem 3. Given Assumptions 1 and 2 and \( c \models (q, r_1) \), then response \( r_1 \) is correct. Similarly, response \( r_1 \) is incorrect because \( c \bot (q, r_1) \). Based on Lemma 1 and these events, then \( \mathcal{U}(\mathcal{M}_\theta(q), r_1) < \mathcal{U}(\mathcal{M}_\theta(q, c), r_2) \). \( \square \)

This constraint states that if the LLM’s response changes from \( r_1 \) to \( r_2 \) after applying RAG, where \( r_1 \) is correct, and \( r_2 \) is incorrect, then the estimated uncertainty of \( r_2 \) should be higher than the one for \( r_1 \). In the example of Figure 1, the LLM generates the correct response "Old Trafford" and changes its response to "Wembley Stadium" in the RAG setup, which is incorrect. In this scenario, the uncertainty of the RAG response should be higher than that of the original response without RAG.

**Theorem 5 (Neutrally Consistent).** \( \forall q, c \) if \( \mathcal{M}_\theta(q) = r_1, \mathcal{M}_\theta(q, c) = r_2, r_1 \equiv r_2, c \#(q, r_1) \), then \( \mathcal{U}(\mathcal{M}_\theta(q), r_1) \approx \mathcal{U}(\mathcal{M}_\theta(q, c), r_2) \).

This constraint states that if the LLM’s response remains unchanged after applying RAG, and the retrieved context \( c \) is unrelated to the query and responses \( r_1 \) and \( r_2 \), then the context neither supports nor contradicts the LLM’s belief. In this case, the estimated salary should remain similar. For example, consider the query "Who wrote the book The Origin of Species?". If, in the RAG setup, the LLM is provided with the context shown in Figure 1, which is unrelated to the query, then as long as the response remains unchanged, the uncertainty value should remain unaffected.

3.3 Instantiation

To empirically examine UE methods against these axioms, we need to define a specific instantiation of functions in our framework (cf. Sec. 3.1).

We introduce two instantiations of these functions: *reference-based* and *reference-free*. The reference-based instantiation assumes the existence of a benchmark containing ground truth responses to queries. Such a benchmark is not available for reference-free instantiation.

**Reference-based.** In this setup, we rely on ground truth labels to check the condition of each axiom. We assume that for every \( q \), the correct response \( \hat{r} \) is available in our ground truth. The implementation of *Entailment* and *Contradiction* functions then boils down to comparing the generated response \( r \) against the ground truth response \( \hat{r} \). The comparison is performed using a matching function \( \mathcal{E}(r_1, r_2) \), which assesses whether the two responses are equivalent. This function is also used to implement the *Equivalence* function (cf. Sec 3.1). For datasets containing factual queries with short responses, \( \mathcal{E}(.) \) is an Exact Match (EM) function, which returns *True* if and only if the two responses are identical on a token-by-token basis (Mallen et al., 2023). Using this setup, the following conditions can be inferred for our axioms:

*Axiom 1.* \( \mathcal{E}(r_1, r_2) = True, \mathcal{E}(r_2, \hat{r}) = True \).
*Axiom 2.* \( \mathcal{E}(r_1, r_2) = True, \mathcal{E}(r_2, \hat{r}) = False \).
*Axiom 3.* \( \mathcal{E}(r_1, r_2) = False, \mathcal{E}(r_1, \hat{r}) = False, \mathcal{E}(r_2, \hat{r}) = True \).
*Axiom 4.* \( \mathcal{E}(r_1, r_2) = False, \mathcal{E}(r_1, \hat{r}) = True, \mathcal{E}(r_2, \hat{r}) = False \).
*Axiom 5.* \( \mathcal{E}(r_1, r_2) = True, c \) is not relevant to \( q \).

**Reference-free.** Since access to the correctness labels of LLM’s responses limits the applicability of axioms to unseen queries, we propose a reference-free implementation of axioms. Specifically, we leverage an NLI classifier to assess the relationship between the generated response and the context, denoted as \( \mathcal{R}(.) \). Following (Kuhn et al., 2023; Lin et al., 2024b), we implement *Entailment* by merging entailment and neutral classes into a single class. The contradiction class of the NLI classifier is considered for the *Contradiction* function. Similar to the reference-based instantiation, function \( \mathcal{E}(.) \) is used for *Equivalence*. Using these definitions, the axioms are defined as follows:

*Axiom 1.* \( \mathcal{E}(r_1, r_2) = True, \mathcal{R}(c, q, r_2) = Entailment \).
*Axiom 2.* \( \mathcal{E}(r_1, r_2) = True, \mathcal{R}(c, q, r_2) = Contradiction \).
*Axiom 3.* \( \mathcal{E}(r_1, r_2) = False, \mathcal{R}(c, q, r_1) = Contradiction, \mathcal{R}(c, q, r_2) = Entailment \).
Axiom 4. \( \mathcal{E}(r_1, r_2) = False,\ \mathcal{R}(c, q, r_2) = Entailment,\ \mathcal{R}(c, q, r_2) = Contradiction \).
Axiom 5 mirrors the reference-based setup, due to the limitations of existing NLI methods in predicting the neutral relation.

4 Derivation of a Calibration Function

In this section, we derive a calibration function that improves existing UE methods using our axiomatic framework. To recap, our formal constraints are built around four functions that are examined for LLM responses without RAG (\( r_1 \)) and with RAG (\( r_2 \)). In the reference-free instantiation of our framework (cf. Sec. 3.3), we showed that these functions are of two types: (i) Equivalence that examines the relation between two LLM-generated responses, represented as \( \mathcal{E}(r_1, r_2) \), and (ii) other functions that examine entailment, contradiction, and independence relations between context, query, and an LLM generated response, represented as \( \mathcal{R}(c, q, r) \). We define a calibration coefficient by searching the space of our axiomatic constraints using these two types of functions:
\[
\alpha_{ax} = k_1 \cdot \mathcal{E}(r_1, r_2) + k_2 \cdot \mathcal{R}(c, q, r_1) + \\
k_3 \cdot \mathcal{R}(c, q, r_2),
\]
where \( k_1, k_2, k_3 \) are hyper parameters, and \( r_1, r_2 \) represent LLM generated responses without and with RAG, respectively. The calibrated UE function for RAG is then defined as:
\[
\mathcal{U}(\mathcal{M}_\theta(c, q), r_2)^{cal} = (k_4 - \alpha_{ax}) \cdot \mathcal{U}(\mathcal{M}_\theta(c, q), r_2).
\]
The hyper parameters \( k_1–k_4 \) are set to satisfy the axioms using a validation set. This calibration enables increasing the uncertainty score of RAG for samples associated with axioms 2 and 4 while decreasing it for samples related to axioms 1 and 3.

4.1 Instantiation

We propose three instantiations of the calibration function, where three different models are used to implement \( \mathcal{R} \).

CTI. The first model is based on the Context-sensitive Token Identification (CTI) task, which has been applied in self-citation and groundedness evaluation (Sarti et al., 2024; Qi et al., 2024). In this approach, each token in \( r = \{r^1, r^2, \ldots, r^N\} \) is evaluated using a contrastive metric \( m \) (e.g., KL divergence, comparing the LLM’s response distributions with and without the context. The resulting scores are \( \{m_1, m_2, \ldots, m_N\} \), where \( m_n = \mathrm{KL}(P(r^n | r^{<n}, (q, c); \theta) \parallel P(r^n | r^{<n}, q; \theta)) \). These scores are converted into binary values via the selector function \( S_{CTI} \). The overall relation score is then computed as:
\[
\mathcal{R}(c, q, r) = \frac{1}{N} \sum_{n=1}^N S_{CTI}(m_n).
\]

NLI. The second model employs an NLI-based approach that quantifies the relationship using entailment probability:
\[
\mathcal{R}(c, q, r) = \mathcal{N}_e(c, (q, r)).
\]

MiniCheck. Finally, the third model employs MiniCheck (Tang et al., 2024), which performs sentence-level fact-checking using a fine-tuned model. It produces a score between 0 and 1 indicating how well the \( r \) is grounded in the \( c \):
\[
\mathcal{R}(c, q, r) = \text{MiniCheck}(c, (q, r)).
\]

In all three instantiations, the equivalence function \( \mathcal{E}(r_1, r_2) \) is an NLI classifier, wherein the entailment probability serves as a continuous measure of similarity between \( r_1 \) and \( r_2 \) (Kuhn et al., 2023); formally \( \mathcal{E}(r_1, r_2) = \mathcal{N}_e(r_1, r_2) \).

5 Experimental Setup

Datasets. We evaluate our approach on three open-book QA datasets, Natural Questions (NQ-open) (Lee et al., 2019), TriviaQA (Joshi et al., 2017), and POPQA (Mallen et al., 2023). For each dataset, we randomly sample 3,000 examples as the test set. We create a validation set for each dataset, comprising 300 samples, which is used to compute calibration coefficients as described in Section 4. For NQ-open and TriviaQA, the validation set is sampled from the training set, whereas for POPQA, it is derived from the test set.

Methods. We evaluate three white-box UE methods: PE, SE, and MARS applied to PE and SE (denoted as PE+M and SE+M), as well as three black-box methods: Deg, ECC, and EigV (cf. Sec. 2).

Experimental setup. Our experiments involve the reproduction of existing UE methods for the RAG setup. To ensure a fair comparison, we employ LLMs that are used in the original papers: Llama2-chat 7B and Mistral-7B. For uncertainty computation, 10 responses per query are generated with a temperature setting of \( T = 1 \); for correctness evaluation, the most likely response is considered.

Following Kuhn et al. (2023), we use Deberta-large model fine-tuned on MNLI as NLI classifier. BM25, Contriever (Izard et al., 2022), and BM25+Reranker are used as retrievers. Manually chosen relevant and irrelevant documents are denoted with Doc\(^+\) and Doc\(^-\), respectively.
<table>
  <tr>
    <th rowspan="2">LLM Unc.</th>
    <th colspan="2">PopQA</th>
    <th colspan="4">PopQA</th>
  </tr>
  <tr>
    <th>No Doc</th>
    <th>Doc</th>
    <th>BM25</th>
    <th>Cont.</th>
    <th>ReRa.</th>
    <th>Doc<sup>+</sup></th>
  </tr>
  <tr>
    <td>PE</td>
    <td>1.29</td>
    <td>1.11<sup>*</sup></td>
    <td>0.54<sup>*</sup></td>
    <td>0.46<sup>*</sup></td>
    <td>0.35<sup>*</sup></td>
    <td>0.34<sup>*</sup></td>
  </tr>
  <tr>
    <td>SE</td>
    <td>4.86</td>
    <td>4.37</td>
    <td>3.45</td>
    <td>3.30</td>
    <td>3.13</td>
    <td>3.19</td>
  </tr>
  <tr>
    <td>PE+M</td>
    <td>1.59</td>
    <td>1.34</td>
    <td>0.65</td>
    <td>0.55</td>
    <td>0.44</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>SE+M</td>
    <td>5.38</td>
    <td>4.71</td>
    <td>3.62</td>
    <td>3.43</td>
    <td>3.23</td>
    <td>3.27</td>
  </tr>
  <tr>
    <td>Deg</td>
    <td>0.52</td>
    <td>0.32</td>
    <td>0.12</td>
    <td>0.09</td>
    <td>0.06</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>0.71</td>
    <td>0.54</td>
    <td>0.22</td>
    <td>0.17</td>
    <td>0.12</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>4.25</td>
    <td>2.28</td>
    <td>1.42</td>
    <td>1.31</td>
    <td>1.18</td>
    <td>1.17</td>
  </tr>
  <tr>
    <td>Mistral-v0.3</td>
    <td>PE</td>
    <td>1.51</td>
    <td>0.94</td>
    <td>0.84</td>
    <td>0.69</td>
    <td>0.62</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td></td>
    <td>SE</td>
    <td>5.66</td>
    <td>3.73</td>
    <td>3.68</td>
    <td>3.53</td>
    <td>3.41</td>
    <td>3.26</td>
  </tr>
  <tr>
    <td></td>
    <td>PE+M</td>
    <td>2.35</td>
    <td>1.42</td>
    <td>1.26</td>
    <td>1.05</td>
    <td>0.92</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td></td>
    <td>SE+M</td>
    <td>6.47</td>
    <td>4.05</td>
    <td>3.98</td>
    <td>3.77</td>
    <td>3.60</td>
    <td>3.45</td>
  </tr>
  <tr>
    <td></td>
    <td>Deg</td>
    <td>0.48</td>
    <td>0.05</td>
    <td>0.07</td>
    <td>0.06</td>
    <td>0.05</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td></td>
    <td>ECC</td>
    <td>0.68</td>
    <td>0.03</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.05</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td></td>
    <td>EigV</td>
    <td>4.18</td>
    <td>1.08</td>
    <td>1.16</td>
    <td>1.17</td>
    <td>1.11</td>
    <td>1.08</td>
  </tr>
</table>

Table 1: Average uncertainty values for various settings. Lighter colors indicate lower uncertainty. Statistically significant differences are compared to No Doc are marked with *.

Metrics. We report the Exact Match for correctness and AUROC (Bakman et al., 2024). We report on statistical significance using Wilcoxon test with \( p \)-value < 0.01; see Appendix B for further details.

Calibration Function. We perform a grid search on the validation set of each dataset to determine the axiomatic coefficients (\( k_1, k_2, k_3, k_4 \)) as described in Section 4. This grid search simultaneously pursues two objectives: satisfying the axioms and maximizing the overall AUROC. For the CTI method, the optimal coefficients are (0.05, 0.20, 0.75, 1.30); for the NLI and MiniCheck methods, the optimal coefficients are (0.05, 0.05, 0.90, 1.20) consistently across all datasets. We observed that the calibration coefficient values are consistent across different datasets and LLMs, which is expected given that the range of uncertainty scores does not vary significantly across datasets and LLMs. Specifically, \( k_3 \) consistently takes higher values than other values. The lower value of \( k_3 \) for CTI compared to other NLI and MiniCheck is due to its higher error rate in capturing the relationship between the retrieved document and the generated output. We found that decreasing \( k_3 \) (or increasing \( k_1 \) and \( k_2 \)) consistently leads to lower AUROC scores, while reducing \( k_1 \) or \( k_2 \) results in fewer satisfied axioms. \( k_4 \) remains relatively stable across configurations.

6 Results

6.1 Uncertainty Changes with RAG
(RQ1) examines how the performance of UE methods and their associated uncertainty values vary with and without context in the input prompt. Figures 2 and 4 present accuracy and AUROC for different RAG settings. We observe inconsistent behavior of UE methods with and without RAG across different datasets, often displaying drop AUROC for RAG cases, except for Doc<sup>+</sup>. While AUROC should be independent accuracy, the results suggest a correlation between the performance of the RAG method and AUROC; especially when considering irrelevant and relevant documents. This indicates a bias of current UE methods towards RAG generations.

To assess this bias further, we report on average uncertainty values of these methods in Tables 1 and 5. The results reveal that incorporating any context results in lower uncertainty values. Even the inclusion of irrelevant contexts, which do not enhance accuracy, leads to a significant reduction in uncertainty scores. This suggests that current UE methods produce lower uncertainty values in the RAG setup without adequately accounting for the relevance of the context.

![Bar chart comparing AUROC and Accuracy for PopQA across different RAG settings](page_1012_180_410_312.png)

Figure 2: Comparison of AUROC between no-RAG and RAG settings for Llama2-chat.

6.2 Axiomatic Evaluation

The second research question (RQ2) investigates properties (i.e., axioms) of UE methods that guarantee optimal performance, and assesses how these axioms are satisfied by current UE methods. Tables 2 and 6 present the change in the average uncertainty value of Llama2-chat, without and with RAG, for Axioms 1–4 using the Reference-based implementation. The results indicate that Axioms 2 and 4 are largely unmet. Furthermore, MARS, although being a state-of-the-art white-box method, does not demonstrate improved axiom compliance. Similar trends are observed for Mistral and other datasets (see Table 7), underscoring the generalizability of these findings. Additionally, the Reference-free implementation of axioms (Table 9) strongly correlate with the Reference-based findings, confirming that UE methods completely fail to satisfy Ax-
<table>
  <tr>
    <th rowspan="2">UE</th>
    <th colspan="3">PopQA</th>
  </tr>
  <tr>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>c</sup></th>
  </tr>
  <tr>
    <th colspan="4">Axiom 1: Positively Consistent ↓</th>
  </tr>
  <tr>
    <td>PE</td>
    <td>0.735 → 0.419 *</td>
    <td>0.735 → 0.408 *</td>
    <td>1.242 → 0.340 *</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>3.781 → 3.205 *</td>
    <td>3.791 → 3.158 *</td>
    <td>4.682 → 3.113 *</td>
  </tr>
  <tr>
    <td>PE+M</td>
    <td>0.896 → 0.483 *</td>
    <td>0.881 → 0.458 *</td>
    <td>1.530 → 0.406 *</td>
  </tr>
  <tr>
    <td>SE+M</td>
    <td>4.102 → 3.286 *</td>
    <td>4.091 → 3.248 *</td>
    <td>5.146 → 3.173 *</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>1.951 → 1.166 *</td>
    <td>2.025 → 1.143 *</td>
    <td>4.074 → 1.078 *</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>0.417 → 0.110 *</td>
    <td>0.426 → 0.094 *</td>
    <td>0.710 → 0.055 *</td>
  </tr>
  <tr>
    <td>Deg</td>
    <td>0.220 → 0.048 *</td>
    <td>0.230 → 0.043 *</td>
    <td>0.496 → 0.022 *</td>
  </tr>
  <tr>
    <th colspan="4">Axiom 2: Negatively Consistent ↑</th>
  </tr>
  <tr>
    <td>PE</td>
    <td>1.068 → 0.746</td>
    <td>0.820 → 0.593</td>
    <td>1.083 → 0.597</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>4.163 → 3.548 *</td>
    <td>4.104 → 3.381 *</td>
    <td>4.388 → 4.107</td>
  </tr>
  <tr>
    <td>PE+M</td>
    <td>1.309 → 0.844</td>
    <td>1.016 → 0.782</td>
    <td>1.328 → 0.684</td>
  </tr>
  <tr>
    <td>SE+M</td>
    <td>4.599 → 3.700 *</td>
    <td>4.481 → 3.610 *</td>
    <td>4.764 → 4.221</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>2.453 → 1.338 *</td>
    <td>2.088 → 1.274 *</td>
    <td>2.758 → 1.910</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>0.541 → 0.197 *</td>
    <td>0.477 → 0.152 *</td>
    <td>0.503 → 0.443</td>
  </tr>
  <tr>
    <td>Deg</td>
    <td>0.286 → 0.101 *</td>
    <td>0.228 → 0.073 *</td>
    <td>0.343 → 0.254</td>
  </tr>
  <tr>
    <th colspan="4">Axiom 3: Positively Changed ↓</th>
  </tr>
  <tr>
    <td>PE</td>
    <td>1.375 → 0.347 *</td>
    <td>1.416 → 0.298 *</td>
    <td>1.342 → 0.268 *</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>4.889 → 3.015 *</td>
    <td>5.091 → 3.013 *</td>
    <td>4.884 → 3.051</td>
  </tr>
  <tr>
    <td>PE+M</td>
    <td>1.708 → 0.398 *</td>
    <td>1.735 → 0.374 *</td>
    <td>1.604 → 0.340 *</td>
  </tr>
  <tr>
    <td>SE+M</td>
    <td>5.514 → 3.072 *</td>
    <td>5.681 → 3.082 *</td>
    <td>5.379 → 3.099 *</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>4.131 → 1.139 *</td>
    <td>4.733 → 1.114 *</td>
    <td>4.449 → 1.102 *</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>0.790 → 0.085 *</td>
    <td>0.823 → 0.081 *</td>
    <td>0.780 → 0.072 *</td>
  </tr>
  <tr>
    <td>Deg</td>
    <td>0.547 → 0.044 *</td>
    <td>0.588 → 0.035 *</td>
    <td>0.544 → 0.032 *</td>
  </tr>
  <tr>
    <th colspan="4">Axiom 4: Negatively Changed ↑</th>
  </tr>
  <tr>
    <td>PE</td>
    <td>0.933 → 0.636</td>
    <td>1.006 → 0.558</td>
    <td>1.252 → 0.463</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>4.152 → 3.552 *</td>
    <td>4.192 → 3.409 *</td>
    <td>4.830 → 3.690 *</td>
  </tr>
  <tr>
    <td>PE+M</td>
    <td>1.164 → 0.714 *</td>
    <td>1.298 → 0.748 *</td>
    <td>1.689 → 0.747</td>
  </tr>
  <tr>
    <td>SE+M</td>
    <td>4.553 → 3.690 *</td>
    <td>4.653 → 3.608 *</td>
    <td>5.381 → 4.007 *</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>2.593 → 1.449 *</td>
    <td>2.557 → 1.412 *</td>
    <td>3.567 → 1.449 *</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>0.540 → 0.262 *</td>
    <td>0.548 → 0.220 *</td>
    <td>0.707 → 0.237 *</td>
  </tr>
  <tr>
    <td>Deg</td>
    <td>0.320 → 0.128 *</td>
    <td>0.320 → 0.115 *</td>
    <td>0.463 → 0.140 *</td>
  </tr>
</table>

Table 2: Comparison of changes in average uncertainty values for Axioms 1–4 before (left) and after (right) applying RAG with Llama2-chat. Colors green and deep red indicate significant changes aligning or conflicting with axioms, respectively. Color shallow red represents non-significant changes conflicting with axioms. Significance is marked by *.

ioms 2 and 4. This further shows the reliability of reference-free implementation for axiomatic evaluation of UE methods.

To evaluate Axiom 5, we add irrelevant context (\( Doc^c \)) for each query. Table 3 shows that only PE+M and SE+M partially satisfy Axiom 5 for Llama2. For Mistral (Table 8), all methods pass Axiom 5 for POPQA but not for the other datasets. These findings suggest that none of the existing UE methods fully satisfy Axiom 2, 4, and 5.

6.3 Axiomatic Calibration

Our third research question (RQ3) examines how our axiomatic framework can lead to designing an optimal UE method. Tables 4 and 10 present AUROC and percentage of samples passing the axioms 1–4 before and after applying our calibration method. Axiom 5 is not assessed, as retrievers tend to retrieve relevant documents. We perform the experiments on four representative (and not cherry-picked) UE methods, as the results generalize to other methods as well. The calibration function is implemented using the three models described in Section 4.1, and Contriever is employed for RAG.

The results show that calibration MiniCheck outperforms all implementations, improving percentages of all axioms for EigV and ECC and for most axioms in open-box methods. Most importantly, the results show as the percentage of samples satisfying the axioms increases, the AUROC improves, showing the empirical validity of our axioms in improving UE methods. Moreover, Figures 3 and 5 show that after calibration, the RAG AUROC becomes comparable to or even better than the No Doc baseline, suggesting that our calibration method successfully compensates for the inefficiencies of existing UE methods in RAG.

<table>
  <tr>
    <th>Unc.</th>
    <th>NQ-open</th>
    <th>TriviaQA</th>
    <th>PopQA</th>
  </tr>
  <tr>
    <td>PE</td>
    <td>2.072 → 2.248 *</td>
    <td>0.872 → 1.155 *</td>
    <td>0.897 → 0.909 *</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>5.253 → 5.471 *</td>
    <td>3.863 → 4.158 *</td>
    <td>3.897 → 4.319 *</td>
  </tr>
  <tr>
    <td>PE+M</td>
    <td>4.791 → 4.805</td>
    <td>1.415 → 1.699 *</td>
    <td>1.031 → 1.130 *</td>
  </tr>
  <tr>
    <td>SE+M</td>
    <td>7.993 → 7.933</td>
    <td>4.540 → 4.817 *</td>
    <td>4.297 → 4.591</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>2.211 → 2.446 *</td>
    <td>1.757 → 1.870 *</td>
    <td>2.270 → 2.218</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>0.512 → 0.625 *</td>
    <td>0.382 → 0.448 *</td>
    <td>0.490 → 0.507</td>
  </tr>
  <tr>
    <td>Deg</td>
    <td>0.265 → 0.333 *</td>
    <td>0.171 → 0.211 *</td>
    <td>0.256 → 0.309</td>
  </tr>
</table>

Table 3: Comparison of changes in average uncertainty values for Axiom 5 before (left) and after (right) applying RAG with Llama2-chat. Color coding and significance markers follow those in Table 2.

In this paper, we examined existing uncertainty estimation (UE) for the RAG setup and showed they systematically generated low uncertainty values in the RAG setup without considering the relevance of the given context to the query. We further proposed an axiomatic evaluation framework for UE in the RAG setup and defined five formal constraints that a UE method should satisfy when processing both parametric and non-parametric knowledge. These axioms were empirically validated across multiple representative datasets, UE methods, and LLMs. Our results showed that none of the existing UE methods pass all the axiom, pinpointing the problem in these methods. We further derived a calibration function for adjusting UE methods in the RAG setup and improvements in both axiomatic evaluation and correlation with correctness. Future work includes developing a UE method designed to naturally conform to the estab-
<table>
  <tr>
    <th rowspan="2">UE</th>
    <th colspan="4">NQ-open</th>
    <th colspan="4">TriviaQA</th>
    <th colspan="4">PopQA</th>
  </tr>
  <tr>
    <th>A1 (%)</th>
    <th>A2 (%)</th>
    <th>A3 (%)</th>
    <th>A4 (%)</th>
    <th>AUROC</th>
    <th>A1 (%)</th>
    <th>A2 (%)</th>
    <th>A3 (%)</th>
    <th>A4 (%)</th>
    <th>AUROC</th>
    <th>A1 (%)</th>
    <th>A2 (%)</th>
    <th>A3 (%)</th>
    <th>A4 (%)</th>
    <th>AUROC</th>
  </tr>
  <tr>
    <td>PE</td>
    <td>60.19</td>
    <td>47.85</td>
    <td>77.35</td>
    <td>51.16</td>
    <td>64.87</td>
    <td>45.53</td>
    <td>43.78</td>
    <td>70.26</td>
    <td>66.88</td>
    <td>68.18</td>
    <td>66.19</td>
    <td>42.46</td>
    <td>87.57</td>
    <td>38.17</td>
    <td>61.59</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>61.49</td>
    <td>44.17</td>
    <td>76.43</td>
    <td>53.88</td>
    <td>65.38</td>
    <td>46.00</td>
    <td>43.78</td>
    <td>69.23</td>
    <td>68.47</td>
    <td>69.29</td>
    <td>69.63</td>
    <td>39.73</td>
    <td>87.95</td>
    <td>38.17</td>
    <td>63.04</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>66.02</td>
    <td>47.24</td>
    <td>77.57</td>
    <td>55.43</td>
    <td>67.21</td>
    <td>48.45</td>
    <td>45.77</td>
    <td>71.28</td>
    <td>68.47</td>
    <td>69.40</td>
    <td>68.77</td>
    <td>41.10</td>
    <td>88.15</td>
    <td>41.22</td>
    <td>63.09</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>76.05</td>
    <td>37.42</td>
    <td>83.75</td>
    <td>51.93</td>
    <td>69.85</td>
    <td>51.36</td>
    <td>49.23</td>
    <td>74.10</td>
    <td>69.75</td>
    <td>71.92</td>
    <td>69.34</td>
    <td>39.73</td>
    <td>89.48</td>
    <td>39.70</td>
    <td>64.31</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>77.35</td>
    <td>33.75</td>
    <td>91.53</td>
    <td>36.05</td>
    <td>67.49</td>
    <td>50.14</td>
    <td>35.82</td>
    <td>84.62</td>
    <td>54.78</td>
    <td>73.44</td>
    <td>71.92</td>
    <td>31.51</td>
    <td>94.07</td>
    <td>29.01</td>
    <td>63.79</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>77.02</td>
    <td>25.76</td>
    <td>89.47</td>
    <td>40.31</td>
    <td>67.09</td>
    <td>56.54</td>
    <td>39.30</td>
    <td>79.74</td>
    <td>56.69</td>
    <td>72.63</td>
    <td>78.51</td>
    <td>26.03</td>
    <td>91.21</td>
    <td>26.72</td>
    <td>62.58</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>79.61</td>
    <td>40.49</td>
    <td>86.72</td>
    <td>50.00</td>
    <td>69.77</td>
    <td>68.96</td>
    <td>46.77</td>
    <td>80.77</td>
    <td>62.74</td>
    <td>74.72</td>
    <td>71.63</td>
    <td>38.36</td>
    <td>92.73</td>
    <td>41.22</td>
    <td>67.86</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>88.02</td>
    <td>32.52</td>
    <td>91.53</td>
    <td>46.90</td>
    <td>75.88</td>
    <td>73.28</td>
    <td>49.75</td>
    <td>82.82</td>
    <td>67.20</td>
    <td>79.79</td>
    <td>77.94</td>
    <td>31.51</td>
    <td>94.07</td>
    <td>41.22</td>
    <td>72.49</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>65.37</td>
    <td>12.88</td>
    <td>88.56</td>
    <td>24.42</td>
    <td>63.94</td>
    <td>37.16</td>
    <td>24.38</td>
    <td>86.15</td>
    <td>39.17</td>
    <td>70.00</td>
    <td>55.30</td>
    <td>6.85</td>
    <td>92.93</td>
    <td>20.61</td>
    <td>62.42</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>77.35</td>
    <td>20.25</td>
    <td>90.16</td>
    <td>34.50</td>
    <td>66.82</td>
    <td>66.89</td>
    <td>30.85</td>
    <td>86.15</td>
    <td>48.41</td>
    <td>72.54</td>
    <td>80.80</td>
    <td>19.18</td>
    <td>93.50</td>
    <td>29.77</td>
    <td>61.51</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>80.91</td>
    <td>27.61</td>
    <td>91.76</td>
    <td>35.27</td>
    <td>69.44</td>
    <td>60.21</td>
    <td>41.79</td>
    <td>87.69</td>
    <td>51.59</td>
    <td>75.58</td>
    <td>73.35</td>
    <td>35.62</td>
    <td>95.60</td>
    <td>38.17</td>
    <td>67.60</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>88.67</td>
    <td>23.93</td>
    <td>93.82</td>
    <td>34.88</td>
    <td>73.60</td>
    <td>74.88</td>
    <td>40.30</td>
    <td>90.00</td>
    <td>55.41</td>
    <td>78.34</td>
    <td>83.09</td>
    <td>24.66</td>
    <td>96.75</td>
    <td>32.82</td>
    <td>72.18</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>61.49</td>
    <td>9.82</td>
    <td>83.06</td>
    <td>18.99</td>
    <td>63.57</td>
    <td>34.24</td>
    <td>14.43</td>
    <td>73.59</td>
    <td>30.89</td>
    <td>68.23</td>
    <td>52.44</td>
    <td>6.84</td>
    <td>87.38</td>
    <td>18.32</td>
    <td>62.06</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>75.73</td>
    <td>23.31</td>
    <td>87.18</td>
    <td>37.98</td>
    <td>67.37</td>
    <td>65.47</td>
    <td>31.84</td>
    <td>77.69</td>
    <td>53.19</td>
    <td>69.92</td>
    <td>78.80</td>
    <td>23.29</td>
    <td>90.82</td>
    <td>34.35</td>
    <td>61.75</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>78.64</td>
    <td>32.52</td>
    <td>87.18</td>
    <td>42.64</td>
    <td>68.96</td>
    <td>58.04</td>
    <td>42.79</td>
    <td>77.44</td>
    <td>59.87</td>
    <td>71.31</td>
    <td>71.35</td>
    <td>32.88</td>
    <td>92.16</td>
    <td>42.75</td>
    <td>66.44</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>86.08</td>
    <td>26.99</td>
    <td>89.93</td>
    <td>39.54</td>
    <td>71.81</td>
    <td>72.44</td>
    <td>41.29</td>
    <td>82.31</td>
    <td>58.92</td>
    <td>74.94</td>
    <td>79.37</td>
    <td>21.92</td>
    <td>94.84</td>
    <td>35.87</td>
    <td>71.39</td>
  </tr>
</table>

Table 4: Percentage of samples passing the axioms before and after calibration for Contriver with Llama2-chat. The results show that as the number of samples passing the axioms increases, the AUROC also improves. Bold values indicate the best performance for each dataset, while underlined values represent the best performance achieved by a UE method and its calibrated variants.

![Bar chart comparing AUROC between no-RAG and calibrated RAG settings for TriviaQA across different UE methods](page_1012_1012_420_246.png)

Figure 3: Comparison of AUROC between the no-RAG and calibrated RAG settings for Llama2-chat for TriviaQA. AUROC improves significantly, either surpassing the no-RAG setting or reducing the gap between them.

lished axioms. Another direction is assessing these axioms in long-form responses and uncertainty-based applications, such as Active RAG.

Limitations

Axiomatic Uncertainty Estimator. In this study, we evaluate existing uncertainty estimation (UE) methods within the RAG setup and delineate the optimal behaviors that these methods should exhibit. Although we introduce a calibration function in Section 4, it may be more effective to develop an axiomatic UE model that inherently adheres to the prescribed axioms. Future research should leverage these principles in the construction of UE methods.

Comprehensiveness of the Axioms. As discussed in Section 3, while our current axioms address most cases, additional axioms may be needed to cover all sample types. For example, consider when an LLM produces a different output after incorporating a context, and both the initial and augmented responses contradict the context. In this scenario, our framework does not specify a change in uncertainty, though supplementary axioms might address this gap. Future research should develop axioms for such cases.

Scalability and Applications. We investigated the impact of incorporating context into the input prompt on uncertainty measures. However, we did not explore other input modalities, such as multi-modal RAG, or alternative response formats, such as long-form responses, each of which presents unique challenges. Furthermore, applications of uncertainty estimation, such as Adaptive RAG (Cheng et al., 2024; Tao et al., 2024), hallucination detection (Geng et al., 2024), reasoning monitoring (Yin et al., 2024), and LLM-as-Judgment (Lee et al., 2024; Dietz et al., 2025), fall outside the scope of this study. Future research should extend these findings to encompass diverse input types, response formats, and UE applications.

Acknowledgments

This publication is part of the project LESSEN with project number NWA.1389.20.183 of the research program NWA ORC 2020/21 which is (partly) financed by the Dutch Research Council (NWO).
References

Enrique Amigó, Hui Fang, Stefano Mizzaro, and Chengxiang Zhai. 2020. Axiomatic thinking for information retrieval: introduction to special issue. Inf. Retr. J., 23(3):187–190.

Yavuz Faruk Bakman, Duygu Nur Yaldız, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. MARS: meaning-aware response scoring for uncertainty estimation in generative LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ACL, pages 7752–7767.

Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. 2024. Linguistic calibration of long-form generations. In Forty-first International Conference on Machine Learning, ICML.

Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völks, and Matthias Hagen. 2022. Axiomatic retrieval experimentation with ir_axioms. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3131–3140.

Catherine Chen, Jack Merullo, and Carsten Eickhoff. 2024. Axiomatic causal interventions for reverse engineering relevance computation in neural retrieval models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR, pages 1401–1410.

Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangye Yin, Yunfan Shao, Linyang Li, Tianxiang Sun, Hang Yan, and Xipeng Qiu. 2024. Unified active retrieval for retrieval augmented generation. In Findings of the Association for Computational Linguistics: EMNLP, pages 17153–17166.

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for RAG systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR, pages 719–729.

Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Elise Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, and Nick Craswell. 2025. LLM-evaluation tropes: Perspectives on the validity of LLM-evaluations. arXiv preprint arXiv:2504.19076.

Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of freeform large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pages 5050–5063.

Hui Fang, Tao Tao, and ChengXiang Zhai. 2004. A formal study of information retrieval heuristics. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 49–56.

Hui Fang and ChengXiang Zhai. 2005. An exploration of axiomatic approaches to information retrieval. In SIGIR 2005: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 480–487.

Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koepll, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL, pages 6577–6595.

Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2024. Decomposing uncertainty for large language models through input clarification ensembling. In Forty-first International Conference on Machine Learning, ICML.

Xinneng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, and Edgar Dobriban. 2024. Uncertainty in language models: Assessment through rank-calibration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 284–312.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res.

Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050.

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516.

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL, pages 1601–1611.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas
Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. abs/2207.05221.

Parisa Kordjamshidi, Qiang Ning, James Pustejovsky, and Marie-Francine Moens. 2024. Spatial and temporal language understanding: Representation, reasoning, and grounding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 5: Tutorial Abstracts), pages 39–46.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations ICLR.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452–466.

Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2024. Are LLM-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on LLM-based evaluation.

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL, pages 6086–6096.

I-Fan Lin, Faegheh Hasibi, and Suzan Verberne. 2024a. Generate then refine: Data augmentation for zero-shot intent detection. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13138–13146.

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024b. Generating with confidence: Uncertainty quantification for black-box large language models. Trans. Mach. Learn. Res., 2024.

Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. 2024. Uncertainty estimation and quantification for LLMs: A simple supervised approach. CoRR.

Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, and Lluís Márquez. 2024. Factual confidence of LLMs: on reliability and robustness of current estimators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pages 4554–4570.

Andrey Malinin and Mark J. F. Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In 9th International Conference on Learning Representations, ICLR.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pages 9802–9822.

Sewon Min, Kalpesh Krishna, Xinxin Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 12076–12100.

Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. 2001. On spectral clustering: Analysis and an algorithm. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS], pages 849–856.

Andrew Parry, Catherine Chen, Carsten Eickhoff, and Sean MacAvaney. 2025. Mechir: A mechanistic interpretability framework for information retrieval. In Advances in Information Retrieval - 47th European Conference on Information Retrieval, ECIR, pages 89–95.

Ellie Pavlick and Chris Callison-Burch. 2016. Most "babies" are "little" and most "problems" are "huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL.

Jirui Qi, Gabriele Sarti, Raquel Fernández, and Arianna Bisazza. 2024. Model internals-based answer attribution for trustworthy retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 6037–6053.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Corby Rosset, Guoqing Zheng, Victor Dibia, Ahmed Awadallah, and Paul N. Bennett. 2023. Axiomatic preference modeling for longform question answering. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 11445–11475.
Gabriele Sarti, Grzegorz Chrupala, Malvina Nissim, and Arianna Bisazza. 2024. Quantifying the plausibility of context reliance in neural machine translation. In The Twelfth International Conference on Learning Representations, ICLR.

Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. 2024a. Fine tuning vs. retrieval augmented generation for less popular knowledge. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2024, pages 12–22.

Heydar Soudani, Roxana Petcu, Evangelos Kanoulas, and Faegheh Hasibi. 2024b. A survey on recent advances in conversational data generation. CoRR, abs/2405.13003.

Liyan Tang, Philippe Laban, and Greg Durrett. 2024. Minicheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 8818–8847.

Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. 2024. When to trust LLMs: Aligning confidence with response quality. In Findings of the Association for Computational Linguistics, ACL, pages 5984–5996.

Harsh Trivedi, Niranjian Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014–10037.

Michael Völske, Alexander Bondarenko, Maik Fröbe, Benno Stein, Jaspreet Singh, Matthias Hagen, and Avishek Anand. 2021. Towards axiomatic explanations for neural ranking models. In ICTIR ’21: The 2021 ACM SIGIR, pages 13–22.

Ulrike von Luxburg. 2007. A tutorial on spectral clustering. Stat. Comput., 17(4):395–416.

Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pages 1112–1122.

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations ICLR.

Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. Do not design, learn: A trainable scoring function for uncertainty estimation in generative LLMs. abs/2406.11278.

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Junqi Dai, Qinyuan Cheng, Xuanjing Huang, and Xipeng Qiu. 2024. Reasoning in flux: Enhancing large language models reasoning through uncertainty-aware adaptive guidance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pages 2401–2416.

Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah A. Smith. 2024a. Set the clock: Temporal alignment of pretrained language models. In Findings of the Association for Computational Linguistics, ACL, pages 15015–15040.

Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2024b. Knowing what LLMs DO NOT know: A simple yet effective self-detection method. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL, pages 7051–7063.

Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. 2024c. Enhancing contextual understanding in large language models through contrastive decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL, pages 4225–4237.

Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. 2024. Relying on the unreliable: The impact of language models’ reluctance to express uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pages 3623–3643.
Appendix

A Related Work

Uncertainty Estimation (UE) seeks to quantify the confidence of LLMs in their predictions (Hou et al., 2024; Zhao et al., 2024b). UE methods are commonly divided into two groups: black-box and white-box approaches. Black-box methods rely solely on the LLM’s outputs without accessing internal layers or generation logits. In addition to the semantic similarity-based methods discussed in Section 2, other black-box techniques exist. For example, verbalization methods prompt the model to explicitly report its confidence (e.g., “How confident are you that the answer is correct?”). Xiong et al. (2024) highlight that two key factors influence the quality of verbalized confidence: (i) the prompting strategy, which includes techniques such as vanilla, Chain-of-Thought (CoT), self-probing, multi-step, and Top-K prompting, and (ii) the sampling strategy, employing methods like self-random sampling, prompting-based elicitation, and misleading prompts to generate multiple responses. Additionally, Epi-M (Zhou et al., 2024) incorporates epistemic markers into the input prompt to facilitate well-calibrated confidence scores.

White-box approaches, by contrast, leverage access to next-token prediction probabilities for uncertainty calculation. Beyond the methods covered in Section 2, several techniques have been proposed. For instance, P(True) (Kadavath et al., 2022) measures the probability that a model assigns to the correctness of a given response by appending a sentence such as Is the possible answer: (A) True (B) False. The possible answer is: so that the probability of generating “True” or “False” serves as the measure. Similarly, P(IK) (Kadavath et al., 2022) estimates the likelihood that the model “knows” the correct answer, that is, the probability of generating the correct response when sampling at unit temperature. Furthermore, LARS (Yaldiz et al., 2024) introduces a learning-based approach by training a scoring model on token probabilities to enhance uncertainty prediction.

Axiomatic Evaluation. Axiomatic thinking refers to a problem-solving approach guided by a set of axioms closely aligned with conventional scientific methodologies (Amigó et al., 2020). More generally, this approach seeks solutions that satisfy all predefined axioms, that is, the desirable properties a solution should possess.

Axiomatic thinking has been successfully applied to the study of Information Retrieval (IR), thereby contributing both to the theoretical understanding and the practical enhancement of existing retrieval models. The objective of Axiomatic IR is to establish formal constraints, or axioms, that delineate the essential properties an effective ranking model must satisfy (Völkske et al., 2021). In this context, Fang et al. (2004) formally defined six fundamental constraints derived from empirical observations of common characteristics in traditional retrieval functions. These constraints correspond to intuitive retrieval heuristics, such as term frequency weighting, term discrimination weighting, and document length normalization. Building on this foundation, Fang and Zhai (2005) proposed an axiomatic framework for the development of retrieval models. Their framework comprises an inductive scheme for function definitions, which provides a common basis for the analytical comparison of different retrieval functions, as well as a set of formalized retrieval constraints adapted from (Fang et al., 2004). These axioms have been further examined in subsequent studies. For example, Chen et al. (2024) employed causal interventions to identify specific attention heads that encode a robust term frequency signal, thereby aligning with one of the original axioms.

Beyond IR, axiomatic approaches have been extended to other domains. For instance, Rosset et al. (2023) defined axioms representing the qualities that humans value in long-form answers, including usefulness, relevance, groundedness, truthfulness, and thoroughness. They generated training data corresponding to these principles and subsequently used it to train a preference model.

B Experimental Setup

Datasets. We conduct our experiments on three open-book Question Answering (QA) datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023). The NQ dataset comprises a large-scale collection of real-world queries derived from Google search data. Each entry includes a user query and the corresponding Wikipedia page that contains the answer. The NQ-open dataset (Lee et al., 2019), a subset of NQ, differs by removing the restriction of linking answers to specific Wikipedia passages, thereby emulating a more gen-
Figure 4: Comparison of AUROC between no-RAG and RAG settings.

<table>
  <tr>
    <th rowspan="2">LM Unc.</th>
    <th colspan="6">NQ-open</th>
    <th colspan="6">TriviaQA</th>
    <th colspan="6">PopQA</th>
  </tr>
  <tr>
    <th>No Doc</th><th>Doc<sup>+</sup></th><th>BM25</th><th>Cont.</th><th>ReRa.</th><th>Doc<sup>+</sup></th>
    <th>No Doc</th><th>Doc<sup>+</sup></th><th>BM25</th><th>Cont.</th><th>ReRa.</th><th>Doc<sup>+</sup></th>
    <th>No Doc</th><th>Doc<sup>+</sup></th><th>BM25</th><th>Cont.</th><th>ReRa.</th><th>Doc<sup>+</sup></th>
  </tr>
  <tr>
    <td>PE</td><td>1.98</td><td>1.92</td><td>1.53 *</td><td>1.41 *</td><td>1.31 *</td><td>1.19 *</td><td>1.14</td><td>1.42 *</td><td>1.05</td><td>1.03</td><td>0.90 *</td><td>0.96 *</td><td>1.29</td><td>1.11 *</td><td>0.54 *</td><td>0.46 *</td><td>0.35 *</td><td>0.34 *</td>
  </tr>
  <tr>
    <td>SE</td><td>5.40</td><td>5.09 *</td><td>4.29 *</td><td>4.20 *</td><td>3.99 *</td><td>3.88 *</td><td>4.39</td><td>4.48 *</td><td>3.89 *</td><td>3.85 *</td><td>3.66 *</td><td>3.73 *</td><td>4.86</td><td>4.37 *</td><td>3.45 *</td><td>3.30 *</td><td>3.13 *</td><td>3.19 *</td>
  </tr>
  <tr>
    <td>PEM</td><td>3.90</td><td>3.89</td><td>3.33 *</td><td>3.26 *</td><td>3.12 *</td><td>2.97 *</td><td>1.74</td><td>2.06 *</td><td>1.64</td><td>1.64 *</td><td>1.46 *</td><td>1.51 *</td><td>1.59</td><td>1.34 *</td><td>0.65 *</td><td>0.55 *</td><td>0.44 *</td><td>0.45 *</td>
  </tr>
  <tr>
    <td>SEM</td><td>7.41</td><td>6.93 *</td><td>5.97 *</td><td>5.88 *</td><td>5.62 *</td><td>5.49 *</td><td>5.16</td><td>5.21 *</td><td>4.51 *</td><td>4.50 *</td><td>4.22 *</td><td>4.30 *</td><td>5.38</td><td>4.71 *</td><td>3.62 *</td><td>3.43 *</td><td>3.23 *</td><td>3.27 *</td>
  </tr>
  <tr>
    <td>Llama2-chat</td><td>0.52</td><td>0.36 *</td><td>0.16 *</td><td>0.13 *</td><td>0.09 *</td><td>0.07 *</td><td>0.31</td><td>0.29 *</td><td>0.17 *</td><td>0.15 *</td><td>0.11 *</td><td>0.16 *</td><td>0.52</td><td>0.32 *</td><td>0.12 *</td><td>0.09 *</td><td>0.06 *</td><td>0.05 *</td>
  </tr>
  <tr>
    <td>Deg</td><td>0.64</td><td>0.60 *</td><td>0.29 *</td><td>0.23 *</td><td>0.17 *</td><td>0.14 *</td><td>0.56</td><td>0.53 *</td><td>0.33 *</td><td>0.29 *</td><td>0.23 *</td><td>0.31 *</td><td>0.71</td><td>0.54 *</td><td>0.22 *</td><td>0.17 *</td><td>0.12 *</td><td>0.10 *</td>
  </tr>
  <tr>
    <td>ECC</td><td>3.06</td><td>2.48 *</td><td>1.57 *</td><td>1.42 *</td><td>1.28 *</td><td>1.21 *</td><td>2.52</td><td>2.21 *</td><td>1.65</td><td>1.57 *</td><td>1.41 *</td><td>1.48 *</td><td>1.68</td><td>2.45 *</td><td>2.28 *</td><td>1.42 *</td><td>1.31 *</td><td>1.18 *</td><td>1.17 *</td>
  </tr>
  <tr>
    <td>EigV</td><td>5.61</td><td>4.37 *</td><td>4.32 *</td><td>4.33 *</td><td>4.19 *</td><td>4.05 *</td><td>4.29</td><td>4.27 *</td><td>3.76 *</td><td>3.74 *</td><td>3.57 *</td><td>3.67 *</td><td>5.66</td><td>3.73 *</td><td>3.68 *</td><td>3.53 *</td><td>3.41 *</td><td>3.26 *</td>
  </tr>
  <tr>
    <td>PEM-v1.3</td><td>4.25</td><td>2.51 *</td><td>3.29 *</td><td>3.61 *</td><td>3.48 *</td><td>3.36 *</td><td>1.73</td><td>1.88 *</td><td>1.51 *</td><td>1.54 *</td><td>1.36 *</td><td>1.41 *</td><td>2.35</td><td>1.42 *</td><td>1.26 *</td><td>1.05 *</td><td>0.92 *</td><td>0.80 *</td>
  </tr>
  <tr>
    <td>SEM</td><td>7.65</td><td>5.42 *</td><td>5.94 *</td><td>6.19 *</td><td>6.01 *</td><td>5.85 *</td><td>4.99</td><td>4.98 *</td><td>4.35 *</td><td>4.37 *</td><td>4.12 *</td><td>4.27 *</td><td>6.47</td><td>4.05 *</td><td>3.98 *</td><td>3.77 *</td><td>3.60 *</td><td>3.45 *</td>
  </tr>
  <tr>
    <td>Deg</td><td>0.37</td><td>0.16 *</td><td>0.13 *</td><td>0.10 *</td><td>0.07 *</td><td>0.05 *</td><td>0.20</td><td>0.18 *</td><td>0.10 *</td><td>0.10 *</td><td>0.07 *</td><td>0.19 *</td><td>0.48</td><td>0.05 *</td><td>0.07 *</td><td>0.06 *</td><td>0.05 *</td><td>0.03 *</td>
  </tr>
  <tr>
    <td>ECC</td><td>0.54</td><td>0.20 *</td><td>0.18 *</td><td>0.15 *</td><td>0.11 *</td><td>0.08 *</td><td>0.37</td><td>0.32 *</td><td>0.17 *</td><td>0.19 *</td><td>0.13 *</td><td>0.17 *</td><td>0.68</td><td>0.03 *</td><td>0.08 *</td><td>0.08 *</td><td>0.05 *</td><td>0.04 *</td>
  </tr>
  <tr>
    <td>EigV</td><td>2.83</td><td>1.49 *</td><td>1.40 *</td><td>1.32 *</td><td>1.23 *</td><td>1.37 *</td><td>2.04</td><td>1.65 *</td><td>1.36 *</td><td>1.39 *</td><td>1.25 *</td><td>1.41 *</td><td>4.18</td><td>1.08 *</td><td>1.16 *</td><td>1.17 *</td><td>1.11 *</td><td>1.08 *</td>
  </tr>
</table>

Table 5: Average uncertainty values for various settings. Lighter colors indicate lower uncertainty. Statistically significant differences are compared to No Doc are marked with *.

eral real-world scenario. We obtain the gold documents for each query from the corpus and dataset annotated by (Cuconasu et al., 2024) 1, in which the gold documents are integrated with the original corpus. For evaluation, we use the test set containing 2,889 queries. TriviaQA consists of trivia questions sourced from the web (Jeong et al., 2024). To ensure a dataset size comparable to NQ-open, we randomly sample 3,000 queries from its development set. POPQA is an open-domain QA dataset designed to evaluate factual knowledge, particularly regarding long-tail entities. Constructed from 16 diverse relationship types in Wikidata, POPQA is originally a closed-book dataset comprising 14,000 QA pairs without gold document annotations. Consequently, following (Soudani et al., 2024a), we consider the summary section of the corresponding Wikipedia page as the gold document. Since POPQA is entirely based on Wikipedia, we employ the same corpus for retrieval. To maintain consistency with the other datasets, we randomly select 3,000 samples from the test set.

Language Models. In accordance with established baselines, we select two generative LLMs: Llama2-chat-7B and Mistral-7B. For inputs that are not augmented with retrieved documents, we employ the following template: "Answer the question. Question: <question> Answer:" For inputs augmented with retrieved documents, we utilize this template: "You are given a question, and you MUST respond with an answer (max 10 tokens) using either the provided document or your memorized knowledge. Document: <context> Question:<question> Answer:". Although more sophisticated prompts were examined in preliminary experiments, the marginal

1Dataset: florin-hf/nq_open_gold
<table>
  <tr>
    <th rowspan="2">UE</th>
    <th colspan="3">BM25</th>
    <th colspan="3">Contriever</th>
    <th colspan="3">Doc<sup>+</sup></th>
    <th colspan="3">BM25</th>
    <th colspan="3">Contriever</th>
    <th colspan="3">Doc<sup>+</sup></th>
    <th colspan="3">BM25</th>
    <th colspan="3">Contriever</th>
    <th colspan="3">Doc<sup>+</sup></th>
    <th colspan="3">PopQA</th>
  </tr>
  <tr>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>NQ-open</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
  </tr>
  <tr>
    <td colspan="24"><b>Axiom 1: Positively Consistent ↓</b></td>
  </tr>
  <tr>
    <td>PE</td><td>1.445 → 1.194 *</td><td>1.535 → 1.216 *</td><td>1.549 → 1.159 *</td><td>0.700 → 0.753 *</td><td>0.718 → 0.743 *</td><td>0.731 → 0.724 *</td><td>0.735 → 0.419 *</td><td>0.735 → 0.408 *</td><td>1.242 → 0.340 *</td><td>SE</td><td>4.656 → 3.933 *</td><td>4.756 → 3.907 *</td><td>4.800 → 3.823 *</td><td>3.644 → 3.412 *</td><td>3.664 → 3.424 *</td><td>3.738 → 3.388 *</td><td>3.781 → 3.205 *</td><td>3.791 → 3.158 *</td><td>4.682 → 3.113 *</td><td>PE+M</td><td>3.389 → 3.124 *</td><td>3.412 → 3.052 *</td><td>3.437 → 3.069 *</td><td>1.051 → 1.110 *</td><td>1.131 → 1.178 *</td><td>1.141 → 1.120 *</td><td>0.896 → 0.483 *</td><td>0.881 → 0.458 *</td><td>1.530 → 0.406 *</td><td>SE+M</td><td>6.640 → 5.778 *</td><td>6.705 → 5.632 *</td><td>6.740 → 5.667 *</td><td>4.142 → 3.832 *</td><td>4.212 → 3.898 *</td><td>4.293 → 3.824 *</td><td>4.102 → 3.286 *</td><td>4.091 → 3.248 *</td><td>5.146 → 3.173 *</td><td>EigV</td><td>2.030 → 1.270 *</td><td>2.129 → 1.189 *</td><td>2.166 → 1.112 *</td><td>1.622 → 1.318 *</td><td>1.617 → 1.234 *</td><td>1.679 → 1.254 *</td><td>1.951 → 1.166 *</td><td>2.025 → 1.143 *</td><td>4.074 → 1.078 *</td><td>ECC</td><td>0.479 → 0.149 *</td><td>0.538 → 0.120 *</td><td>0.557 → 0.071 *</td><td>0.346 → 0.228 *</td><td>0.338 → 0.169 *</td><td>0.367 → 0.180 *</td><td>0.417 → 0.110 *</td><td>0.426 → 0.094 *</td><td>0.710 → 0.055 *</td><td>Deg</td><td>0.227 → 0.084 *</td><td>0.262 → 0.061 *</td><td>0.270 → 0.035 *</td><td>0.144 → 0.087 *</td><td>0.142 → 0.066 *</td><td>0.155 → 0.067 *</td><td>0.220 → 0.048 *</td><td>0.230 → 0.043 *</td><td>0.496 → 0.022 *</td>
  </tr>
  <tr>
    <td colspan="24"><b>Axiom 2: Negatively Consistent ↑</b></td>
  </tr>
  <tr>
    <td>PE</td><td>2.317 → 2.261 *</td><td>2.230 → 2.153 *</td><td>2.232 → 2.194 *</td><td>1.543 → 1.478 *</td><td>1.534 → 1.438 *</td><td>1.495 → 1.528 *</td><td>1.068 → 0.746 *</td><td>0.820 → 0.593 *</td><td>1.083 → 0.597 *</td><td>SE</td><td>5.626 → 4.998 *</td><td>5.515 → 4.848 *</td><td>5.572 → 4.841 *</td><td>4.715 → 4.460 *</td><td>4.672 → 4.291 *</td><td>4.897 → 4.638 *</td><td>4.163 → 3.548 *</td><td>4.104 → 3.381 *</td><td>4.388 → 4.107 *</td><td>PE+M</td><td>5.284 → 4.891 *</td><td>5.052 → 4.904 *</td><td>5.665 → 5.652 *</td><td>2.716 → 2.633 *</td><td>2.381 → 2.249 *</td><td>2.594 → 2.597 *</td><td>1.309 → 0.844 *</td><td>1.016 → 0.782 *</td><td>1.328 → 0.684 *</td><td>SE+M</td><td>8.566 → 7.579 *</td><td>8.377 → 7.471 *</td><td>8.914 → 7.962 *</td><td>5.978 → 5.521 *</td><td>5.737 → 5.170 *</td><td>6.109 → 5.733 *</td><td>4.599 → 3.700 *</td><td>4.481 → 3.610 *</td><td>4.764 → 4.221 *</td><td>EigV</td><td>2.410 → 1.694 *</td><td>2.454 → 1.375 *</td><td>2.340 → 1.216 *</td><td>2.147 → 1.802 *</td><td>2.271 → 1.700 *</td><td>2.654 → 2.508 *</td><td>2.453 → 1.338 *</td><td>2.088 → 1.274 *</td><td>2.758 → 1.910 *</td><td>ECC</td><td>0.564 → 0.302 *</td><td>0.600 → 0.240 *</td><td>0.542 → 0.166 *</td><td>0.554 → 0.382 *</td><td>0.561 → 0.331 *</td><td>0.617 → 0.600 *</td><td>0.541 → 0.197 *</td><td>0.477 → 0.152 *</td><td>0.503 → 0.043 *</td><td>Deg</td><td>0.304 → 0.172 *</td><td>0.314 → 0.113 *</td><td>0.299 → 0.069 *</td><td>0.274 → 0.194 *</td><td>0.294 → 0.186 *</td><td>0.353 → 0.325 *</td><td>0.286 → 0.101 *</td><td>0.228 → 0.073 *</td><td>0.343 → 0.254 *</td>
  </tr>
  <tr>
    <td colspan="24"><b>Axiom 3: Positively Changed ↓</b></td>
  </tr>
  <tr>
    <td>PE</td><td>2.113 → 0.909 *</td><td>1.989 → 0.939 *</td><td>2.006 → 0.847 *</td><td>1.481 → 0.665 *</td><td>1.413 → 0.702 *</td><td>1.403 → 0.653 *</td><td>1.375 → 0.347 *</td><td>1.416 → 0.298 *</td><td>1.342 → 0.268 *</td><td>SE</td><td>5.606 → 3.589 *</td><td>5.459 → 3.589 *</td><td>5.500 → 3.544 *</td><td>4.970 → 3.347 *</td><td>4.966 → 3.469 *</td><td>4.972 → 3.287 *</td><td>4.889 → 3.015 *</td><td>5.091 → 3.013 *</td><td>4.884 → 3.051 *</td><td>PE+M</td><td>3.479 → 2.056 *</td><td>3.420 → 1.991 *</td><td>3.416 → 2.012 *</td><td>2.001 → 0.917 *</td><td>2.026 → 1.020 *</td><td>1.930 → 0.938 *</td><td>1.708 → 0.398 *</td><td>1.735 → 0.374 *</td><td>1.604 → 0.340 *</td><td>SE+M</td><td>7.268 → 4.703 *</td><td>7.069 → 4.616 *</td><td>7.101 → 4.637 *</td><td>5.790 → 3.648 *</td><td>5.804 → 3.825 *</td><td>5.760 → 3.579 *</td><td>5.514 → 3.072 *</td><td>5.681 → 3.082 *</td><td>5.379 → 3.099 *</td><td>EigV</td><td>3.692 → 1.420 *</td><td>3.561 → 1.182 *</td><td>3.551 → 1.159 *</td><td>3.588 → 1.245 *</td><td>3.625 → 1.346 *</td><td>3.650 → 1.277 *</td><td>4.131 → 1.139 *</td><td>4.733 → 1.114 *</td><td>4.449 → 1.102 *</td><td>ECC</td><td>0.756 → 0.144 *</td><td>0.701 → 0.111 *</td><td>0.714 → 0.115 *</td><td>0.801 → 0.163 *</td><td>0.807 → 0.218 *</td><td>0.810 → 0.179 *</td><td>0.790 → 0.085 *</td><td>0.823 → 0.081 *</td><td>0.780 → 0.072 *</td><td>Deg</td><td>0.507 → 0.065 *</td><td>0.484 → 0.057 *</td><td>0.488 → 0.051 *</td><td>0.497 → 0.076 *</td><td>0.502 → 0.093 *</td><td>0.504 → 0.079 *</td><td>0.547 → 0.044 *</td><td>0.588 → 0.035 *</td><td>0.544 → 0.032 *</td>
  </tr>
  <tr>
    <td colspan="24"><b>Axiom 4: Negatively Changed ↑</b></td>
  </tr>
  <tr>
    <td>PE</td><td>1.609 → 1.695 *</td><td>1.621 → 1.635 *</td><td>1.598 → 1.688 *</td><td>0.945 → 1.325 *</td><td>0.889 → 1.364 *</td><td>1.034 → 1.396 *</td><td>0.933 → 0.636 *</td><td>1.006 → 0.558 *</td><td>1.252 → 0.463 *</td><td>SE</td><td>4.899 → 4.457 *</td><td>4.899 → 4.437 *</td><td>4.915 → 4.497 *</td><td>4.160 → 4.312 *</td><td>4.157 → 4.273 *</td><td>4.297 → 4.339 *</td><td>4.152 → 3.552 *</td><td>4.192 → 3.409 *</td><td>4.830 → 3.690 *</td><td>PE+M</td><td>3.446 → 3.653 *</td><td>3.522 → 3.692 *</td><td>3.465 → 4.158 *</td><td>1.566 → 2.123 *</td><td>1.306 → 1.946 *</td><td>1.486 → 2.178 *</td><td>1.164 → 0.714 *</td><td>1.298 → 0.748 *</td><td>1.689 → 0.747 *</td><td>SE+M</td><td>6.764 → 6.286 *</td><td>6.803 → 6.377 *</td><td>6.643 → 6.442 *</td><td>4.953 → 5.121 *</td><td>4.769 → 4.933 *</td><td>4.983 → 5.088 *</td><td>4.553 → 3.690 *</td><td>4.653 → 3.608 *</td><td>5.381 → 4.007 *</td><td>EigV</td><td>2.062 → 1.582 *</td><td>2.244 → 1.503 *</td><td>2.233 → 1.367 *</td><td>2.089 → 1.908 *</td><td>2.141 → 1.908 *</td><td>2.399 → 2.131 *</td><td>2.593 → 1.449 *</td><td>2.557 → 1.412 *</td><td>3.567 → 1.449 *</td><td>ECC</td><td>0.594 → 0.332 *</td><td>0.565 → 0.295 *</td><td>0.490 → 0.270 *</td><td>0.501 → 0.453 *</td><td>0.542 → 0.456 *</td><td>0.614 → 0.555 *</td><td>0.540 → 0.262 *</td><td>0.548 → 0.220 *</td><td>0.707 → 0.237 *</td><td>Deg</td><td>0.301 → 0.163 *</td><td>0.294 → 0.148 *</td><td>0.308 → 0.123 *</td><td>0.239 → 0.237 *</td><td>0.253 → 0.251 *</td><td>0.313 → 0.299 *</td><td>0.320 → 0.128 *</td><td>0.320 → 0.115 *</td><td>0.463 → 0.140 *</td>
  </tr>
</table>

Table 6: Comparison of changes in average uncertainty values for Axioms 1–4 before (left) and after (right) applying RAG with Llama2-chat. Axioms are implemented using the Reference-based method. Colors green and deep red indicate significant changes aligning or conflicting with axioms, respectively. Color shallow red represents non-significant changes conflicting with axioms. Significance is marked by *.

improvement they offered relative to the simple template did not justify their use, particularly given the increased risk of model overfitting. Furthermore, following MARS (Bakman et al., 2024), we utilize the Huggingface library’s “generate” function for model output generation. We designate the token ". " as the "eos_token_id" to prevent the model from generating overly lengthy paragraphs in response to closed-book questions. We also set “num_beams” to 1, corresponding to greedy decoding.

Retrieval Models. We employ a suite of retrieval models to acquire relevant contexts for the RAG approach. The models utilized include BM25 (Robertson and Zaragoza, 2009), Contriever (Izacard et al., 2022), and a two-stage re-ranking system. In the two-stage configuration, BM25 is applied for initial retrieval, followed by re-ranking using a pre-trained cross-encoder model, specifically, ms-marco-MiniLM-L-6-v2 from the sentence-transformers library. Additionally, we report results for two variations: Doc+, in which the gold context is incorporated into the input prompt, and Doc−, in which an irrelevant context is substituted. Although several methods exist to obtain irrelevant contexts (Zhao et al., 2024c), in our experiments, these are generated by randomly sampling a context from the corpus.

NLI Models. A NLI classifier takes a sequence pair \( (x_1, x_2) \) and outputs a label \( y \in \{Contradiction, Neutral, Entailment\} \) with corresponding probabilities. The two sequences are concatenated with a separator token [SEP] before input. To study ordering effects, we consider both \( x_1[SEP]x_2 \) and \( x_2[SEP]x_1 \). In the reference-free setting (Section 3.3), if either order yields a contradiction, the input is labeled as such; otherwise, it is labeled as entailment. In Section 4.1, we use
<table>
  <tr>
    <th>UE</th>
    <th colspan="3">BM25</th>
    <th colspan="3">NQ-open</th>
    <th colspan="3">TriviaQA</th>
    <th colspan="3">PopQA</th>
  </tr>
  <tr>
    <th></th>
    <th>Contriever</th>
    <th>Doc*</th>
    <th>Doc†</th>
    <th>Contriever</th>
    <th>Doc*</th>
    <th>Doc†</th>
    <th>Contriever</th>
    <th>Doc*</th>
    <th>Doc†</th>
    <th>Contriever</th>
    <th>Doc*</th>
    <th>Doc†</th>
  </tr>
  <tr>
    <th colspan="13">Axiom 1: Positively Consistent ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>1.620 → 1.332 *</td><td>1.538 → 1.288 *</td><td>1.544 → 1.232 *</td><td>0.531 → 0.483 *</td><td>0.549 → 0.494 *</td><td>0.570 → 0.456 *</td><td>0.893 → 0.673 *</td><td>0.886 → 0.638 *</td><td>1.368 → 0.419 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE</td><td>4.874 → 4.164 *</td><td>4.876 → 4.060 *</td><td>4.941 → 3.922 *</td><td>3.460 → 3.265 *</td><td>3.508 → 3.299 *</td><td>3.565 → 3.241 *</td><td>4.063 → 3.361 *</td><td>4.162 → 3.354 *</td><td>5.379 → 3.112 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>PE+M</td><td>3.682 → 3.283 *</td><td>3.380 → 3.188 *</td><td>3.395 → 3.080 *</td><td>0.943 → 0.903 *</td><td>0.987 → 0.948 *</td><td>1.010 → 0.869 *</td><td>1.220 → 0.942 *</td><td>1.195 → 0.836 *</td><td>2.115 → 0.615 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE+M</td><td>6.710 → 5.863 *</td><td>6.531 → 5.746 *</td><td>6.594 → 5.602 *</td><td>3.839 → 3.638 *</td><td>3.913 → 3.715 *</td><td>3.971 → 3.615 *</td><td>4.315 → 3.539 *</td><td>4.424 → 3.459 *</td><td>6.087 → 3.224 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>EigV</td><td>1.724 → 1.285 *</td><td>1.788 → 1.172 *</td><td>1.901 → 1.069 *</td><td>1.277 → 1.129 *</td><td>1.294 → 1.162 *</td><td>1.344 → 1.114 *</td><td>1.614 → 1.119 *</td><td>1.837 → 1.095 *</td><td>3.781 → 1.041 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ECC</td><td>0.356 → 0.169 *</td><td>0.381 → 0.104 *</td><td>0.405 → 0.043 *</td><td>0.155 → 0.082 *</td><td>0.164 → 0.094 *</td><td>0.187 → 0.076 *</td><td>0.260 → 0.050 *</td><td>0.288 → 0.052 *</td><td>0.621 → 0.021 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Deg</td><td>0.175 → 0.088 *</td><td>0.185 → 0.053 *</td><td>0.208 → 0.023 *</td><td>0.063 → 0.038 *</td><td>0.067 → 0.042 *</td><td>0.076 → 0.030 *</td><td>0.129 → 0.045 *</td><td>0.157 → 0.033 *</td><td>0.426 → 0.016 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <th colspan="13">Axiom 2: Negatively Consistent ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>2.460 → 2.303 *</td><td>2.353 → 2.321 *</td><td>2.377 → 2.374 *</td><td>1.512 → 1.397 *</td><td>1.226 → 1.228 *</td><td>1.477 → 1.421 *</td><td>0.933 → 0.589 *</td><td>0.804 → 0.450 *</td><td>1.196 → 0.570 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE</td><td>5.846 → 5.233 *</td><td>5.614 → 5.074 *</td><td>5.619 → 4.966 *</td><td>4.697 → 4.384 *</td><td>4.449 → 4.133 *</td><td>4.936 → 4.699 *</td><td>4.407 → 3.314 *</td><td>4.290 → 3.215 *</td><td>4.620 → 3.442 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>PE+M</td><td>6.014 → 5.908 *</td><td>5.523 → 5.664 *</td><td>5.783 → 5.920 *</td><td>2.917 → 2.797 *</td><td>2.660 → 2.513 *</td><td>2.777 → 2.833 *</td><td>1.376 → 0.901 *</td><td>1.230 → 0.762 *</td><td>1.631 → 0.686 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE+M</td><td>9.067 → 8.557 *</td><td>8.493 → 8.092 *</td><td>8.728 → 8.147 *</td><td>6.033 → 5.699 *</td><td>5.462 → 5.100 *</td><td>6.121 → 6.009 *</td><td>4.819 → 3.551 *</td><td>4.702 → 3.456 *</td><td>4.875 → 3.504 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>EigV</td><td>2.177 → 1.529 *</td><td>2.047 → 1.303 *</td><td>1.869 → 1.071 *</td><td>1.648 → 1.472 *</td><td>1.655 → 1.489 *</td><td>2.284 → 2.025 *</td><td>2.041 → 1.098 *</td><td>2.055 → 1.181 *</td><td>2.188 → 1.143 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ECC</td><td>0.507 → 0.256 *</td><td>0.437 → 0.166 *</td><td>0.453 → 0.040 *</td><td>0.367 → 0.223 *</td><td>0.394 → 0.243 *</td><td>0.476 → 0.394 *</td><td>0.338 → 0.065 *</td><td>0.411 → 0.069 *</td><td>0.514 → 0.041 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Deg</td><td>0.260 → 0.134 *</td><td>0.227 → 0.080 *</td><td>0.210 → 0.022 *</td><td>0.153 → 0.127 *</td><td>0.152 → 0.120 *</td><td>0.254 → 0.205 *</td><td>0.200 → 0.030 *</td><td>0.194 → 0.044 *</td><td>0.260 → 0.055 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <th colspan="13">Axiom 3: Positively Changed ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>1.972 → 1.038 *</td><td>1.972 → 1.120 *</td><td>2.020 → 1.097 *</td><td>1.492 → 0.531 *</td><td>1.446 → 0.515 *</td><td>1.452 → 0.510 *</td><td>1.837 → 0.734 *</td><td>1.727 → 0.566 *</td><td>1.458 → 0.403 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE</td><td>5.861 → 3.808 *</td><td>5.813 → 3.855 *</td><td>5.898 → 3.838 *</td><td>5.527 → 3.337 *</td><td>5.569 → 3.326 *</td><td>5.497 → 3.364 *</td><td>6.309 → 3.349 *</td><td>6.227 → 3.276 *</td><td>5.662 → 3.104 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>PE+M</td><td>3.917 → 2.599 *</td><td>4.061 → 2.810 *</td><td>4.063 → 2.762 *</td><td>2.544 → 0.959 *</td><td>2.545 → 1.033 *</td><td>2.480 → 0.927 *</td><td>2.935 → 0.970 *</td><td>2.666 → 0.867 *</td><td>2.244 → 0.594 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE+M</td><td>7.587 → 5.162 *</td><td>7.662 → 5.299 *</td><td>7.746 → 5.303 *</td><td>6.542 → 3.690 *</td><td>6.606 → 3.798 *</td><td>6.465 → 3.716 *</td><td>7.365 → 3.467 *</td><td>7.156 → 3.436 *</td><td>6.439 → 3.211 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>EigV</td><td>3.745 → 1.168 *</td><td>3.449 → 1.131 *</td><td>3.547 → 1.119 *</td><td>3.575 → 1.191 *</td><td>3.611 → 1.179 *</td><td>3.470 → 1.210 *</td><td>5.124 → 1.054 *</td><td>5.217 → 1.055 *</td><td>4.323 → 1.040 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ECC</td><td>0.653 → 0.089 *</td><td>0.633 → 0.072 *</td><td>0.661 → 0.069 *</td><td>0.756 → 0.110 *</td><td>0.752 → 0.104 *</td><td>0.747 → 0.131 *</td><td>0.854 → 0.024 *</td><td>0.841 → 0.024 *</td><td>0.700 → 0.025 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Deg</td><td>0.471 → 0.053 *</td><td>0.450 → 0.048 *</td><td>0.466 → 0.037 *</td><td>0.462 → 0.053 *</td><td>0.471 → 0.047 *</td><td>0.454 → 0.063 *</td><td>0.614 → 0.022 *</td><td>0.615 → 0.021 *</td><td>0.492 → 0.016 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <th colspan="13">Axiom 4: Negatively Changed ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>1.480 → 1.284 *</td><td>1.570 → 1.490 *</td><td>1.518 → 1.256 *</td><td>0.791 → 1.173 *</td><td>0.833 → 1.144 *</td><td>0.881 → 1.021 *</td><td>0.941 → 0.881 *</td><td>1.014 → 0.807 *</td><td>1.660 → 0.913 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE</td><td>4.957 → 4.252 *</td><td>5.039 → 4.543 *</td><td>4.775 → 4.116 *</td><td>4.173 → 4.356 *</td><td>4.212 → 4.319 *</td><td>4.392 → 4.174 *</td><td>4.569 → 3.875 *</td><td>4.739 → 3.709 *</td><td>5.853 → 3.783 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>PE+M</td><td>3.045 → 2.901 *</td><td>3.421 → 3.597 *</td><td>3.159 → 2.924 *</td><td>1.383 → 1.989 *</td><td>1.361 → 2.107 *</td><td>1.424 → 1.849 *</td><td>1.323 → 1.354 *</td><td>1.447 → 1.303 *</td><td>2.705 → 1.735 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SE+M</td><td>6.368 → 5.630 *</td><td>6.674 → 6.349 *</td><td>6.181 → 5.549 *</td><td>4.743 → 5.076 *</td><td>4.720 → 5.081 *</td><td>4.954 → 4.796 *</td><td>4.958 → 4.241 *</td><td>5.184 → 4.062 *</td><td>6.835 → 4.446 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>EigV</td><td>2.087 → 1.415 *</td><td>2.115 → 1.497 *</td><td>1.906 → 1.375 *</td><td>1.850 → 1.593 *</td><td>1.944 → 1.710 *</td><td>2.103 → 1.594 *</td><td>2.522 → 1.200 *</td><td>2.565 → 1.222 *</td><td>4.209 → 1.159 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ECC</td><td>0.440 → 0.208 *</td><td>0.438 → 0.238 *</td><td>0.351 → 0.151 *</td><td>0.362 → 0.293 *</td><td>0.378 → 0.323 *</td><td>0.420 → 0.298 *</td><td>0.437 → 0.091 *</td><td>0.492 → 0.119 *</td><td>0.700 → 0.068 *</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Deg</td><td>0.243 → 0.138 *</td><td>0.252 → 0.149 *</td><td>0.222 → 0.109 *</td><td>0.175 → 0.183 *</td><td>0.190 → 0.203 *</td><td>0.233 → 0.180 *</td><td>0.259 → 0.091 *</td><td>0.280 → 0.092 *</td><td>0.479 → 0.065 *</td><td></td><td></td><td></td>
  </tr>
</table>

Table 7: Comparison of changes in average uncertainty values for Axioms 1–4 before (left) and after (right) applying RAG with Mistral-v0.3. Axioms are implemented using the Reference-based method. Colors green and deep red indicate significant changes aligning or conflicting with axioms, respectively. Color shallow red represents non-significant changes conflicting with axioms. Significance is marked by *.

<table>
  <tr>
    <th>Unc.</th>
    <th>NQ-open</th>
    <th>TriviaQA</th>
    <th>PopQA</th>
  </tr>
  <tr>
    <td>PE</td><td>2.227 → 1.778 *</td><td>0.657 → 0.780 *</td><td>1.014 → 1.087 *</td>
  </tr>
  <tr>
    <td>SE</td><td>5.453 → 4.964 *</td><td>3.570 → 3.892 *</td><td>3.976 → 4.021 *</td>
  </tr>
  <tr>
    <td>PE+M</td><td>5.634 → 4.293 *</td><td>1.223 → 1.374 *</td><td>1.686 → 1.759 *</td>
  </tr>
  <tr>
    <td>SE+M</td><td>8.543 → 7.216 *</td><td>4.089 → 4.463 *</td><td>4.310 → 4.521 *</td>
  </tr>
  <tr>
    <td>EigV</td><td>1.696 → 1.637 *</td><td>1.256 → 1.496 *</td><td>1.215 → 1.452 *</td>
  </tr>
  <tr>
    <td>ECC</td><td>0.357 → 0.335 *</td><td>0.154 → 0.300 *</td><td>0.059 → 0.362 *</td>
  </tr>
  <tr>
    <td>Deg</td><td>0.160 → 0.206 *</td><td>0.056 → 0.128 *</td><td>0.093 → 0.140 *</td>
  </tr>
</table>

Table 8: Comparison of changes in average uncertainty values for Axiom 5 before (left) and after(right) applying RAG with Mistral-v0.3. Color coding and significance markers follow those in Table 7.

the maximum entailment probability from the two orders.

**Computational Cost.** We conducted all experiments using one Nvidia A100 GPUs with 40 GB of memory, accumulating approximately 250 GPU hours. Due to the substantial computational demands, all results presented are based on a single run.
Table 9: Comparison of changes in average uncertainty values for Axioms 1–4 before (left) and after (right) applying RAG with Llama2-chat. Axioms are implemented using the Reference-free method. Colors green and deep red indicate significant changes aligning or conflicting with axioms, respectively. Color shallow red represents non-significant changes conflicting with axioms. Significance is marked by *.

<table>
  <tr>
    <th rowspan="2">UE</th>
    <th colspan="6">NQ-open</th>
    <th colspan="6">TriviaQA</th>
    <th colspan="6">PopQA</th>
  </tr>
  <tr>
    <th>A1 (%)</th>
    <th>A2 (%)</th>
    <th>A3 (%)</th>
    <th>A4 (%)</th>
    <th>AUROC</th>
    <th>A1 (%)</th>
    <th>A2 (%)</th>
    <th>A3 (%)</th>
    <th>A4 (%)</th>
    <th>AUROC</th>
    <th>A1 (%)</th>
    <th>A2 (%)</th>
    <th>A3 (%)</th>
    <th>A4 (%)</th>
    <th>AUROC</th>
  </tr>
  <tr>
    <td>PE</td>
    <td>69.77</td>
    <td>42.95</td>
    <td>80.54</td>
    <td>45.21</td>
    <td>64.40</td>
    <td>63.88</td>
    <td>39.05</td>
    <td>87.24</td>
    <td>61.30</td>
    <td>79.14</td>
    <td>70.57</td>
    <td>22.22</td>
    <td>89.17</td>
    <td>44.23</td>
    <td>65.72</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>67.91</td>
    <td>40.39</td>
    <td>78.60</td>
    <td>48.85</td>
    <td>64.82</td>
    <td>65.67</td>
    <td>37.87</td>
    <td>86.01</td>
    <td>64.04</td>
    <td>79.70</td>
    <td>72.91</td>
    <td>14.82</td>
    <td>89.63</td>
    <td>46.80</td>
    <td>67.14</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>70.81</td>
    <td>42.31</td>
    <td>80.54</td>
    <td>52.81</td>
    <td>66.50</td>
    <td>65.88</td>
    <td>42.01</td>
    <td>86.42</td>
    <td>64.04</td>
    <td>79.78</td>
    <td>74.23</td>
    <td>24.69</td>
    <td>93.09</td>
    <td>46.80</td>
    <td>68.65</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>78.05</td>
    <td>32.05</td>
    <td>85.41</td>
    <td>49.18</td>
    <td>67.15</td>
    <td>67.53</td>
    <td>44.38</td>
    <td>87.24</td>
    <td>64.38</td>
    <td>80.25</td>
    <td>74.22</td>
    <td>17.88</td>
    <td>91.01</td>
    <td>48.72</td>
    <td>68.63</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>76.40</td>
    <td>32.69</td>
    <td>90.54</td>
    <td>37.62</td>
    <td>65.66</td>
    <td>63.88</td>
    <td>31.36</td>
    <td>91.36</td>
    <td>51.71</td>
    <td>78.83</td>
    <td>74.22</td>
    <td>16.05</td>
    <td>95.39</td>
    <td>33.97</td>
    <td>68.53</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>74.12</td>
    <td>31.41</td>
    <td>87.57</td>
    <td>42.57</td>
    <td>65.13</td>
    <td>53.22</td>
    <td>40.21</td>
    <td>85.60</td>
    <td>59.93</td>
    <td>76.28</td>
    <td>70.57</td>
    <td>12.35</td>
    <td>93.55</td>
    <td>37.18</td>
    <td>67.10</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>70.39</td>
    <td>42.95</td>
    <td>87.03</td>
    <td>49.51</td>
    <td>66.92</td>
    <td>51.57</td>
    <td>48.52</td>
    <td>86.83</td>
    <td>62.67</td>
    <td>77.01</td>
    <td>69.01</td>
    <td>25.93</td>
    <td>94.24</td>
    <td>39.74</td>
    <td>70.53</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>78.47</td>
    <td>30.13</td>
    <td>89.73</td>
    <td>42.57</td>
    <td>69.66</td>
    <td>68.60</td>
    <td>45.86</td>
    <td>89.30</td>
    <td>56.16</td>
    <td>77.65</td>
    <td>80.21</td>
    <td>12.35</td>
    <td>94.01</td>
    <td>42.31</td>
    <td>70.10</td>
  </tr>
  <tr>
    <td>EigV</td>
    <td>54.66</td>
    <td>14.10</td>
    <td>85.40</td>
    <td>27.72</td>
    <td>63.03</td>
    <td>19.24</td>
    <td>23.08</td>
    <td>85.19</td>
    <td>41.44</td>
    <td>72.25</td>
    <td>41.15</td>
    <td>6.17</td>
    <td>93.08</td>
    <td>26.92</td>
    <td>66.35</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>71.22</td>
    <td>24.36</td>
    <td>87.30</td>
    <td>37.95</td>
    <td>65.06</td>
    <td>47.93</td>
    <td>47.34</td>
    <td>88.89</td>
    <td>60.27</td>
    <td>74.79</td>
    <td>68.23</td>
    <td>16.05</td>
    <td>93.55</td>
    <td>39.74</td>
    <td>65.23</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>69.98</td>
    <td>38.46</td>
    <td>88.65</td>
    <td>41.91</td>
    <td>67.29</td>
    <td>48.64</td>
    <td>50.89</td>
    <td>87.24</td>
    <td>60.27</td>
    <td>74.48</td>
    <td>65.88</td>
    <td>32.10</td>
    <td>95.16</td>
    <td>39.10</td>
    <td>68.40</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>77.85</td>
    <td>28.85</td>
    <td>92.16</td>
    <td>36.63</td>
    <td>68.45</td>
    <td>68.81</td>
    <td>45.56</td>
    <td>90.95</td>
    <td>53.43</td>
    <td>78.51</td>
    <td>83.07</td>
    <td>12.35</td>
    <td>96.31</td>
    <td>37.18</td>
    <td>70.39</td>
  </tr>
  <tr>
    <td>ECC</td>
    <td>53.00</td>
    <td>13.46</td>
    <td>81.62</td>
    <td>26.07</td>
    <td>62.87</td>
    <td>18.31</td>
    <td>14.79</td>
    <td>78.60</td>
    <td>35.62</td>
    <td>71.72</td>
    <td>40.62</td>
    <td>4.93</td>
    <td>92.16</td>
    <td>23.08</td>
    <td>66.28</td>
  </tr>
  <tr>
    <td>+CTI</td>
    <td>72.05</td>
    <td>29.48</td>
    <td>86.48</td>
    <td>39.60</td>
    <td>66.76</td>
    <td>47.13</td>
    <td>50.29</td>
    <td>82.71</td>
    <td>60.95</td>
    <td>76.22</td>
    <td>67.45</td>
    <td>18.52</td>
    <td>94.24</td>
    <td>37.82</td>
    <td>68.36</td>
  </tr>
  <tr>
    <td>+NLI</td>
    <td>70.18</td>
    <td>39.74</td>
    <td>87.29</td>
    <td>43.23</td>
    <td>67.59</td>
    <td>48.35</td>
    <td>50.29</td>
    <td>84.36</td>
    <td>62.32</td>
    <td>73.77</td>
    <td>65.88</td>
    <td>29.63</td>
    <td>95.62</td>
    <td>36.53</td>
    <td>69.91</td>
  </tr>
  <tr>
    <td>+MCH</td>
    <td>79.08</td>
    <td>32.05</td>
    <td>90.81</td>
    <td>37.29</td>
    <td>68.80</td>
    <td>68.74</td>
    <td>43.19</td>
    <td>90.12</td>
    <td>53.08</td>
    <td>77.67</td>
    <td>81.77</td>
    <td>11.11</td>
    <td>96.08</td>
    <td>36.53</td>
    <td>72.71</td>
  </tr>
</table>

Table 10: Percentage of samples passing the axioms before and after calibration for Contriver with Mistral-v0.3. The results show that as the number of samples passing the axioms increases, the AUROC also improves. bold values indicate the best performance for each dataset, while underlined values represent the best performance achieved by a UE method and its calibrated variants in terms of axiomatic satisfaction.
Figure 5: Comparison of AUROC between the no-RAG and calibrated RAG settings for Llama2-chat for NQ-open and POPQA datasets. AUROC improves significantly, either surpassing the no-RAG setting or reducing the gap between them.

Figure 6: Comparison of AUROC between the no-RAG and calibrated RAG settings for Mistral-v0.3. AUROC improves significantly, either surpassing the no-RAG setting or reducing the gap between them.
<table>
  <tr>
    <th rowspan="2">UE</th>
    <th colspan="3">NQ-open</th>
    <th colspan="3">TriviaQA</th>
    <th colspan="3">PopQA</th>
  </tr>
  <tr>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
  </tr>
  <tr>
    <th colspan="10">Axiom 1: Positively Consistent ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>55.357 → 55.357</td><td>60.194 → 61.489</td><td>61.793 → 61.598</td><td>41.454 → 43.124</td><td>45.532 → 46.002</td><td>45.853 → 46.401</td><td>62.369 → 68.293</td><td>66.189 → 69.628</td><td>79.511 → 81.040</td>
  </tr>
  <tr>
    <td>SE</td><td>66.964 → 70.982</td><td>77.346 → 77.023</td><td>79.337 → 83.041</td><td>47.446 → 55.894</td><td>50.141 → 56.538</td><td>52.191 → 58.059</td><td>69.338 → 79.443</td><td>71.920 → 78.510</td><td>88.073 → 86.544</td>
  </tr>
  <tr>
    <td>PE+M</td><td>57.589 → 57.589</td><td>61.165 → 63.430</td><td>62.278 → 62.378</td><td>44.008 → 44.499</td><td>43.744 → 45.720</td><td>46.870 → 48.044</td><td>62.021 → 68.293</td><td>69.628 → 73.639</td><td>81.040 → 82.875</td>
  </tr>
  <tr>
    <td>SE+M</td><td>64.286 → 68.304</td><td>76.375 → 76.375</td><td>72.904 → 73.099</td><td>47.348 → 55.599</td><td>48.354 → 57.008</td><td>52.504 → 58.059</td><td>69.686 → 80.488</td><td>73.639 → 79.656</td><td>89.602 → 88.073</td>
  </tr>
  <tr>
    <td>EigV</td><td>58.036 → 71.875</td><td>65.372 → 77.346</td><td>69.981 → 84.795</td><td>35.265 → 64.047</td><td>37.159 → 66.886</td><td>38.498 → 57.199</td><td>53.659 → 83.275</td><td>55.301 → 80.802</td><td>81.346 → 91.743</td>
  </tr>
  <tr>
    <td>ECC</td><td>54.464 → 72.321</td><td>61.489 → 75.728</td><td>68.226 → 84.016</td><td>32.122 → 62.279</td><td>34.243 → 65.475</td><td>34.977 → 55.634</td><td>50.523 → 80.836</td><td>52.436 → 78.797</td><td>77.064 → 88.379</td>
  </tr>
  <tr>
    <td>Deg</td><td>55.804 → 57.589</td><td>64.401 → 65.049</td><td>70.565 → 70.565</td><td>34.283 → 35.069</td><td>36.595 → 37.065</td><td>37.950 → 38.419</td><td>54.007 → 55.401</td><td>55.014 → 55.874</td><td>81.346 → 81.346</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 2: Negatively Consistent ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>46.237 → 44.086</td><td>47.853 → 44.172</td><td>47.059 → 44.118</td><td>52.299 → 50.575</td><td>43.781 → 43.781</td><td>56.477 → 56.218</td><td>49.275 → 44.928</td><td>42.466 → 39.726</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <td>SE</td><td>34.409 → 31.183</td><td>33.742 → 26.380</td><td>31.618 → 28.676</td><td>42.529 → 41.379</td><td>35.821 → 39.303</td><td>45.078 → 50.777</td><td>34.783 → 21.739</td><td>31.507 → 26.027</td><td>42.857 → 28.571</td>
  </tr>
  <tr>
    <td>PE+M</td><td>39.247 → 39.247</td><td>42.945 → 38.037</td><td>47.794 → 46.324</td><td>49.425 → 47.701</td><td>41.791 → 41.791</td><td>52.332 → 53.886</td><td>44.928 → 43.478</td><td>43.836 → 42.466</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <td>SE+M</td><td>31.720 → 30.108</td><td>31.288 → 26.380</td><td>35.294 → 36.029</td><td>41.379 → 37.356</td><td>35.323 → 38.308</td><td>44.301 → 51.295</td><td>33.333 → 17.391</td><td>30.137 → 27.397</td><td>42.857 → 28.571</td>
  </tr>
  <tr>
    <td>EigV</td><td>19.355 → 31.720</td><td>12.883 → 20.245</td><td>5.147 → 26.471</td><td>29.885 → 34.483</td><td>24.378 → 30.846</td><td>37.047 → 50.259</td><td>15.942 → 13.043</td><td>6.849 → 19.178</td><td>42.857 → 28.571</td>
  </tr>
  <tr>
    <td>ECC</td><td>14.516 → 37.097</td><td>9.816 → 23.313</td><td>5.882 → 30.147</td><td>19.540 → 35.057</td><td>14.428 → 31.841</td><td>21.503 → 58.031</td><td>10.145 → 20.290</td><td>6.849 → 23.288</td><td>28.571 → 28.571</td>
  </tr>
  <tr>
    <td>Deg</td><td>20.968 → 20.968</td><td>17.178 → 15.951</td><td>5.147 → 6.618</td><td>29.885 → 31.034</td><td>24.378 → 24.876</td><td>36.788 → 42.487</td><td>13.043 → 12.329</td><td>12.329 → 12.329</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 3: Positively Changed ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>82.215 → 81.544</td><td>77.346 → 76.430</td><td>82.557 → 81.541</td><td>73.402 → 72.634</td><td>70.256 → 69.231</td><td>74.870 → 73.830</td><td>82.331 → 83.083</td><td>87.572 → 87.954</td><td>84.314 → 84.540</td>
  </tr>
  <tr>
    <td>SE</td><td>93.289 → 93.289</td><td>91.533 → 89.703</td><td>93.057 → 91.194</td><td>86.445 → 83.632</td><td>84.615 → 79.744</td><td>88.042 → 83.882</td><td>93.233 → 90.226</td><td>94.073 → 91.205</td><td>92.543 → 88.235</td>
  </tr>
  <tr>
    <td>PE+M</td><td>81.544 → 79.886</td><td>77.574 → 76.888</td><td>80.271 → 78.493</td><td>76.982 → 77.749</td><td>73.590 → 72.308</td><td>80.069 → 77.296</td><td>88.346 → 87.594</td><td>90.822 → 90.440</td><td>84.389 → 84.691</td>
  </tr>
  <tr>
    <td>SE+M</td><td>90.604 → 88.591</td><td>88.787 → 85.812</td><td>88.654 → 86.198</td><td>86.957 → 84.143</td><td>84.359 → 80.000</td><td>88.562 → 84.922</td><td>93.609 → 92.857</td><td>94.455 → 92.543</td><td>93.439 → 89.668</td>
  </tr>
  <tr>
    <td>EigV</td><td>90.604 → 91.611</td><td>88.558 → 90.389</td><td>89.077 → 91.025</td><td>86.189 → 85.166</td><td>86.154 → 86.154</td><td>83.709 → 86.308</td><td>91.353 → 90.977</td><td>92.925 → 93.499</td><td>86.652 → 89.367</td>
  </tr>
  <tr>
    <td>ECC</td><td>82.886 → 87.919</td><td>83.066 → 87.185</td><td>82.557 → 86.622</td><td>79.028 → 80.563</td><td>73.590 → 77.692</td><td>75.390 → 80.243</td><td>86.466 → 89.850</td><td>87.380 → 90.822</td><td>82.730 → 86.652</td>
  </tr>
  <tr>
    <td>Deg</td><td>90.604 → 90.940</td><td>87.414 → 87.643</td><td>89.331 → 89.670</td><td>85.934 → 86.189</td><td>86.410 → 85.128</td><td>85.442 → 85.789</td><td>91.353 → 90.977</td><td>92.543 → 92.543</td><td>86.576 → 86.501</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 4: Negatively Changed ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>51.136 → 52.273</td><td>51.163 → 53.876</td><td>49.231 → 50.769</td><td>66.944 → 66.389</td><td>66.879 → 68.471</td><td>66.372 → 63.717</td><td>42.045 → 39.773</td><td>38.168 → 38.168</td><td>27.586 → 27.586</td>
  </tr>
  <tr>
    <td>SE</td><td>36.080 → 40.625</td><td>36.047 → 40.310</td><td>44.615 → 40.000</td><td>55.556 → 58.889</td><td>54.777 → 56.688</td><td>52.212 → 57.522</td><td>31.818 → 36.364</td><td>29.008 → 26.718</td><td>25.287 → 22.989</td>
  </tr>
  <tr>
    <td>PE+M</td><td>47.727 → 49.716</td><td>50.388 → 53.876</td><td>50.769 → 56.923</td><td>63.333 → 64.722</td><td>66.242 → 65.287</td><td>64.602 → 65.487</td><td>38.636 → 38.636</td><td>32.061 → 31.298</td><td>26.437 → 27.886</td>
  </tr>
  <tr>
    <td>SE+M</td><td>38.636 → 41.193</td><td>40.699 → 42.636</td><td>41.538 → 49.231</td><td>55.278 → 57.500</td><td>53.503 → 56.369</td><td>53.097 → 55.752</td><td>31.250 → 33.523</td><td>28.244 → 24.427</td><td>24.138 → 20.690</td>
  </tr>
  <tr>
    <td>EigV</td><td>24.432 → 34.091</td><td>24.419 → 35.271</td><td>16.923 → 18.462</td><td>38.333 → 51.944</td><td>39.172 → 48.408</td><td>38.938 → 51.327</td><td>21.591 → 35.795</td><td>20.611 → 29.771</td><td>8.046 → 12.644</td>
  </tr>
  <tr>
    <td>ECC</td><td>19.602 → 39.205</td><td>18.992 → 37.984</td><td>16.923 → 30.769</td><td>30.556 → 57.500</td><td>30.892 → 53.185</td><td>26.549 → 60.177</td><td>18.182 → 44.318</td><td>18.321 → 34.351</td><td>8.046 → 19.540</td>
  </tr>
  <tr>
    <td>Deg</td><td>25.284 → 26.989</td><td>24.806 → 27.132</td><td>20.000 → 23.077</td><td>42.500 → 45.278</td><td>42.357 → 44.904</td><td>42.478 → 44.248</td><td>22.727 → 23.864</td><td>19.084 → 19.084</td><td>11.494 → 11.494</td>
  </tr>
</table>

Table 11: Changes in the percentage of samples that satisfy the axioms before and after calibration for Llama2-chat. The relation function \( \mathcal{R} \) is implemented using CTI.
<table>
  <tr>
    <th rowspan="2">UE</th>
    <th colspan="3">NQ-open</th>
    <th colspan="3">TriviaQA</th>
    <th colspan="3">PopQA</th>
  </tr>
  <tr>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>†</sup></th>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>†</sup></th>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>†</sup></th>
  </tr>
  <tr>
    <th colspan="10">Axiom 1: Positively Consistent ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>55.357 → 60.714</td><td>60.194 → 66.019</td><td>61.793 → 66.667</td><td>41.454 → 44.695</td><td>45.532 → 48.730</td><td>45.853 → 47.966</td><td>62.369 → 63.415</td><td>66.189 → 68.481</td><td>79.511 → 80.122</td>
  </tr>
  <tr>
    <td>SE</td><td>66.964 → 73.661</td><td>77.346 → 80.259</td><td>79.337 → 79.922</td><td>47.446 → 58.743</td><td>50.141 → 58.231</td><td>52.191 → 57.433</td><td>69.338 → 73.868</td><td>71.920 → 74.785</td><td>88.073 → 81.040</td>
  </tr>
  <tr>
    <td>PE+M</td><td>57.589 → 62.054</td><td>61.165 → 67.961</td><td>62.278 → 67.057</td><td>44.008 → 46.660</td><td>43.744 → 48.260</td><td>46.870 → 49.687</td><td>62.021 → 63.763</td><td>69.628 → 72.206</td><td>81.040 → 82.263</td>
  </tr>
  <tr>
    <td>SE+M</td><td>64.286 → 70.089</td><td>76.375 → 77.023</td><td>72.904 → 74.269</td><td>47.348 → 58.350</td><td>48.354 → 58.325</td><td>52.504 → 57.825</td><td>69.686 → 76.655</td><td>73.639 → 75.072</td><td>89.602 → 84.709</td>
  </tr>
  <tr>
    <td>EigV</td><td>58.036 → 71.429</td><td>65.372 → 82.201</td><td>69.981 → 86.160</td><td>35.265 → 59.136</td><td>37.159 → 60.960</td><td>38.498 → 59.077</td><td>53.659 → 73.868</td><td>55.301 → 74.785</td><td>81.346 → 85.933</td>
  </tr>
  <tr>
    <td>ECC</td><td>54.464 → 70.982</td><td>61.489 → 78.641</td><td>68.226 → 84.795</td><td>32.122 → 55.403</td><td>34.243 → 58.043</td><td>34.977 → 55.399</td><td>50.523 → 72.474</td><td>52.436 → 71.347</td><td>77.064 → 84.404</td>
  </tr>
  <tr>
    <td>Deg</td><td>55.804 → 56.250</td><td>64.401 → 64.401</td><td>70.565 → 70.955</td><td>34.283 → 35.069</td><td>36.595 → 36.877</td><td>37.950 → 38.185</td><td>54.007 → 55.052</td><td>55.014 → 57.307</td><td>81.346 → 81.040</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 2: Negatively Consistent ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>46.237 → 47.312</td><td>47.853 → 47.239</td><td>47.059 → 46.324</td><td>52.299 → 54.598</td><td>43.781 → 45.274</td><td>56.477 → 58.549</td><td>49.275 → 49.275</td><td>42.466 → 41.096</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <td>SE</td><td>34.409 → 38.172</td><td>33.742 → 38.037</td><td>31.618 → 31.618</td><td>42.529 → 48.276</td><td>35.821 → 43.284</td><td>45.078 → 56.218</td><td>34.783 → 33.333</td><td>31.507 → 32.877</td><td>42.857 → 57.143</td>
  </tr>
  <tr>
    <td>PE+M</td><td>39.247 → 43.011</td><td>42.945 → 41.718</td><td>47.794 → 50.735</td><td>49.425 → 54.023</td><td>41.791 → 41.294</td><td>52.332 → 56.218</td><td>44.928 → 46.377</td><td>43.836 → 45.205</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <td>SE+M</td><td>31.720 → 38.710</td><td>31.288 → 36.196</td><td>35.294 → 33.824</td><td>41.379 → 45.977</td><td>35.323 → 39.801</td><td>44.301 → 55.440</td><td>33.333 → 30.435</td><td>30.137 → 32.877</td><td>42.857 → 42.857</td>
  </tr>
  <tr>
    <td>EigV</td><td>19.355 → 35.484</td><td>12.883 → 26.380</td><td>5.147 → 20.588</td><td>29.885 → 46.552</td><td>24.378 → 38.806</td><td>37.047 → 58.290</td><td>15.942 → 26.087</td><td>6.849 → 32.877</td><td>42.857 → 42.857</td>
  </tr>
  <tr>
    <td>ECC</td><td>14.516 → 43.011</td><td>9.816 → 32.515</td><td>5.882 → 25.000</td><td>19.540 → 55.172</td><td>14.428 → 42.786</td><td>21.503 → 76.425</td><td>10.145 → 34.783</td><td>6.849 → 32.877</td><td>28.571 → 57.143</td>
  </tr>
  <tr>
    <td>Deg</td><td>20.968 → 23.656</td><td>17.178 → 17.791</td><td>5.147 → 8.824</td><td>29.885 → 34.483</td><td>24.378 → 26.866</td><td>36.788 → 45.596</td><td>13.043 → 13.043</td><td>12.329 → 13.699</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 3: Positively Changed ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>82.215 → 84.228</td><td>77.346 → 77.574</td><td>82.557 → 81.964</td><td>73.402 → 73.913</td><td>70.256 → 70.513</td><td>74.870 → 74.003</td><td>82.331 → 84.586</td><td>87.572 → 88.145</td><td>84.314 → 84.615</td>
  </tr>
  <tr>
    <td>SE</td><td>93.289 → 88.591</td><td>91.533 → 86.270</td><td>93.057 → 86.113</td><td>86.445 → 84.910</td><td>84.615 → 80.513</td><td>88.042 → 84.749</td><td>93.233 → 91.729</td><td>94.073 → 92.161</td><td>92.534 → 87.029</td>
  </tr>
  <tr>
    <td>PE+M</td><td>81.544 → 85.235</td><td>77.574 → 79.863</td><td>80.271 → 80.610</td><td>76.982 → 77.238</td><td>73.590 → 72.821</td><td>80.069 → 78.683</td><td>88.346 → 88.346</td><td>90.822 → 90.057</td><td>84.389 → 85.143</td>
  </tr>
  <tr>
    <td>SE+M</td><td>90.604 → 87.248</td><td>88.787 → 84.211</td><td>88.654 → 82.557</td><td>86.957 → 84.655</td><td>84.359 → 81.026</td><td>88.562 → 86.482</td><td>93.609 → 92.105</td><td>94.455 → 93.690</td><td>93.439 → 88.235</td>
  </tr>
  <tr>
    <td>EigV</td><td>90.604 → 92.617</td><td>88.558 → 91.533</td><td>89.077 → 90.517</td><td>86.189 → 87.724</td><td>86.154 → 87.436</td><td>83.709 → 86.655</td><td>91.353 → 93.609</td><td>92.925 → 95.602</td><td>86.652 → 90.875</td>
  </tr>
  <tr>
    <td>ECC</td><td>82.886 → 88.255</td><td>83.066 → 87.185</td><td>82.557 → 86.791</td><td>79.028 → 84.655</td><td>73.590 → 77.436</td><td>75.390 → 78.163</td><td>86.466 → 91.353</td><td>87.380 → 92.161</td><td>82.730 → 88.537</td>
  </tr>
  <tr>
    <td>Deg</td><td>90.604 → 89.933</td><td>87.414 → 86.270</td><td>89.331 → 89.162</td><td>85.934 → 86.189</td><td>86.410 → 86.154</td><td>85.442 → 84.749</td><td>91.353 → 91.353</td><td>92.543 → 92.352</td><td>86.576 → 86.652</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 4: Negatively Changed ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>51.136 → 56.250</td><td>51.163 → 55.426</td><td>49.231 → 58.462</td><td>66.944 → 68.611</td><td>66.879 → 68.790</td><td>66.372 → 69.027</td><td>42.045 → 42.614</td><td>38.168 → 41.221</td><td>27.586 → 31.034</td>
  </tr>
  <tr>
    <td>SE</td><td>36.080 → 49.432</td><td>36.047 → 50.000</td><td>44.615 → 52.308</td><td>55.556 → 65.000</td><td>54.777 → 64.013</td><td>52.212 → 64.602</td><td>31.818 → 42.045</td><td>29.008 → 41.985</td><td>25.287 → 31.034</td>
  </tr>
  <tr>
    <td>PE+M</td><td>47.727 → 52.273</td><td>50.388 → 56.977</td><td>50.769 → 55.385</td><td>63.333 → 66.667</td><td>66.242 → 67.834</td><td>64.602 → 67.257</td><td>38.636 → 38.636</td><td>32.061 → 35.115</td><td>26.437 → 29.885</td>
  </tr>
  <tr>
    <td>SE+M</td><td>38.636 → 51.136</td><td>40.699 → 53.488</td><td>41.538 → 56.923</td><td>55.278 → 62.778</td><td>53.503 → 64.013</td><td>53.097 → 61.947</td><td>31.250 → 38.068</td><td>28.244 → 39.695</td><td>24.138 → 29.885</td>
  </tr>
  <tr>
    <td>EigV</td><td>24.432 → 35.795</td><td>24.419 → 36.047</td><td>16.923 → 33.846</td><td>38.333 → 57.500</td><td>39.172 → 53.185</td><td>38.938 → 53.982</td><td>21.591 → 36.932</td><td>20.611 → 38.931</td><td>8.046 → 18.391</td>
  </tr>
  <tr>
    <td>ECC</td><td>19.602 → 43.466</td><td>18.992 → 42.636</td><td>16.923 → 36.923</td><td>30.556 → 65.556</td><td>30.892 → 59.873</td><td>26.549 → 65.487</td><td>18.182 → 46.591</td><td>18.321 → 42.748</td><td>8.046 → 25.287</td>
  </tr>
  <tr>
    <td>Deg</td><td>25.284 → 29.545</td><td>24.806 → 27.907</td><td>20.000 → 24.615</td><td>42.500 → 47.222</td><td>42.357 → 48.089</td><td>42.478 → 48.673</td><td>22.727 → 24.432</td><td>19.084 → 21.374</td><td>11.494 → 16.092</td>
  </tr>
</table>

Table 12: Changes in the percentage of samples that satisfy the axioms before and after calibration for Llama2-chat. The relation function \( \mathcal{R} \) is implemented using NLI.
<table>
  <tr>
    <th rowspan="2">UE</th>
    <th colspan="3">NQ-open</th>
    <th colspan="3">TriviaQA</th>
    <th colspan="3">PopQA</th>
  </tr>
  <tr>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
    <th>BM25</th>
    <th>Contriever</th>
    <th>Doc<sup>+</sup></th>
  </tr>
  <tr>
    <th colspan="10">Axiom 1: Positively Consistent ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>55.357 → 70.089</td><td>60.194 → 75.081</td><td>61.793 → 75.634</td><td>41.454 → 48.134</td><td>45.532 → 51.364</td><td>45.853 → 51.174</td><td>62.369 → 68.293</td><td>66.189 → 69.341</td><td>79.511 → 81.040</td>
  </tr>
  <tr>
    <td>SE</td><td>66.964 → 78.571</td><td>77.346 → 86.408</td><td>79.337 → 90.058</td><td>47.446 → 72.299</td><td>50.141 → 73.283</td><td>52.191 → 73.083</td><td>69.338 → 81.533</td><td>71.920 → 77.937</td><td>88.073 → 86.544</td>
  </tr>
  <tr>
    <td>PE+M</td><td>57.589 → 68.750</td><td>61.165 → 78.317</td><td>62.278 → 76.023</td><td>44.008 → 48.723</td><td>43.744 → 51.646</td><td>46.870 → 52.034</td><td>62.021 → 70.035</td><td>69.628 → 71.347</td><td>81.040 → 81.957</td>
  </tr>
  <tr>
    <td>SE+M</td><td>64.286 → 75.446</td><td>76.375 → 86.731</td><td>72.904 → 87.329</td><td>47.348 → 71.709</td><td>48.354 → 72.907</td><td>52.504 → 72.457</td><td>69.686 → 82.230</td><td>73.639 → 78.223</td><td>89.602 → 87.156</td>
  </tr>
  <tr>
    <td>EigV</td><td>58.036 → 77.769</td><td>65.372 → 87.702</td><td>69.981 → 93.177</td><td>35.265 → 70.432</td><td>37.159 → 74.882</td><td>38.498 → 74.257</td><td>53.659 → 82.927</td><td>55.301 → 83.095</td><td>81.346 → 95.413</td>
  </tr>
  <tr>
    <td>ECC</td><td>54.464 → 76.786</td><td>61.489 → 86.084</td><td>68.226 → 92.398</td><td>32.122 → 66.306</td><td>34.243 → 72.437</td><td>34.977 → 70.736</td><td>50.523 → 80.488</td><td>52.436 → 79.370</td><td>77.064 → 93.272</td>
  </tr>
  <tr>
    <td>Deg</td><td>55.804 → 57.143</td><td>64.401 → 65.372</td><td>70.565 → 70.175</td><td>34.283 → 35.560</td><td>36.595 → 37.535</td><td>37.950 → 39.202</td><td>54.007 → 54.704</td><td>55.014 → 55.587</td><td>81.346 → 81.346</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 2: Negatively Consistent ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>46.237 → 52.151</td><td>47.853 → 39.877</td><td>47.059 → 38.971</td><td>52.299 → 57.471</td><td>43.781 → 49.254</td><td>56.477 → 60.622</td><td>49.275 → 46.377</td><td>42.466 → 39.726</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <td>SE</td><td>34.409 → 40.323</td><td>33.742 → 34.969</td><td>31.618 → 27.941</td><td>42.529 → 54.023</td><td>35.821 → 49.751</td><td>45.078 → 56.995</td><td>34.783 → 37.681</td><td>31.507 → 31.507</td><td>42.857 → 71.429</td>
  </tr>
  <tr>
    <td>PE+M</td><td>39.247 → 46.774</td><td>42.945 → 34.969</td><td>47.794 → 40.441</td><td>49.425 → 58.621</td><td>41.791 → 48.756</td><td>52.332 → 59.585</td><td>44.928 → 43.478</td><td>43.836 → 39.726</td><td>57.143 → 71.429</td>
  </tr>
  <tr>
    <td>SE+M</td><td>31.720 → 44.086</td><td>31.288 → 34.969</td><td>35.294 → 30.882</td><td>41.379 → 51.149</td><td>35.323 → 48.756</td><td>44.301 → 57.254</td><td>33.333 → 34.783</td><td>30.137 → 31.507</td><td>42.857 → 71.429</td>
  </tr>
  <tr>
    <td>EigV</td><td>19.355 → 31.183</td><td>12.883 → 24.540</td><td>5.147 → 18.382</td><td>29.885 → 44.253</td><td>24.378 → 40.299</td><td>37.047 → 52.073</td><td>15.942 → 21.739</td><td>6.849 → 24.658</td><td>42.857 → 42.857</td>
  </tr>
  <tr>
    <td>ECC</td><td>14.516 → 36.022</td><td>9.816 → 26.994</td><td>5.882 → 21.324</td><td>19.540 → 49.425</td><td>14.428 → 41.294</td><td>21.503 → 65.026</td><td>10.145 → 31.884</td><td>6.849 → 21.918</td><td>28.571 → 57.143</td>
  </tr>
  <tr>
    <td>Deg</td><td>20.968 → 26.882</td><td>17.178 → 18.405</td><td>5.147 → 9.559</td><td>29.885 → 37.931</td><td>24.378 → 30.846</td><td>36.788 → 50.000</td><td>13.043 → 15.942</td><td>12.329 → 13.699</td><td>57.143 → 57.143</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 3: Positively Changed ↓</th>
  </tr>
  <tr>
    <td>PE</td><td>82.215 → 91.946</td><td>77.346 → 83.982</td><td>82.557 → 84.589</td><td>73.402 → 76.726</td><td>70.256 → 74.103</td><td>74.870 → 74.697</td><td>82.331 → 86.842</td><td>87.572 → 89.484</td><td>84.314 → 84.691</td>
  </tr>
  <tr>
    <td>SE</td><td>93.289 → 93.060</td><td>91.533 → 90.847</td><td>93.057 → 89.331</td><td>86.445 → 88.491</td><td>84.615 → 82.821</td><td>88.042 → 84.575</td><td>93.233 → 94.361</td><td>94.073 → 94.073</td><td>92.514 → 89.216</td>
  </tr>
  <tr>
    <td>PE+M</td><td>81.544 → 91.275</td><td>77.574 → 84.439</td><td>80.271 → 83.065</td><td>76.982 → 79.028</td><td>73.590 → 75.385</td><td>80.069 → 79.029</td><td>88.346 → 89.850</td><td>90.822 → 91.396</td><td>84.389 → 85.143</td>
  </tr>
  <tr>
    <td>SE+M</td><td>90.604 → 93.289</td><td>88.787 → 90.847</td><td>88.654 → 87.214</td><td>86.957 → 89.258</td><td>84.359 → 83.846</td><td>88.562 → 85.442</td><td>93.609 → 95.113</td><td>94.455 → 94.264</td><td>93.439 → 90.121</td>
  </tr>
  <tr>
    <td>EigV</td><td>90.604 → 94.295</td><td>88.558 → 93.822</td><td>89.077 → 91.871</td><td>86.189 → 90.026</td><td>86.154 → 90.000</td><td>83.709 → 89.081</td><td>91.353 → 96.241</td><td>92.925 → 96.750</td><td>86.652 → 94.646</td>
  </tr>
  <tr>
    <td>ECC</td><td>82.886 → 89.933</td><td>83.066 → 89.931</td><td>82.557 → 88.400</td><td>79.028 → 87.724</td><td>73.590 → 82.308</td><td>75.390 → 84.749</td><td>86.466 → 93.985</td><td>87.380 → 94.837</td><td>82.730 → 92.911</td>
  </tr>
  <tr>
    <td>Deg</td><td>90.604 → 89.933</td><td>87.414 → 86.499</td><td>89.331 → 89.331</td><td>85.934 → 86.701</td><td>86.410 → 85.128</td><td>85.442 → 83.882</td><td>91.353 → 91.729</td><td>92.543 → 92.352</td><td>86.576 → 86.275</td>
  </tr>
  <tr>
    <th colspan="10">Axiom 4: Negatively Changed ↑</th>
  </tr>
  <tr>
    <td>PE</td><td>51.136 → 55.682</td><td>51.163 → 51.550</td><td>49.231 → 58.462</td><td>66.944 → 69.745</td><td>66.372 → 63.717</td><td>42.045 → 40.909</td><td>38.168 → 39.695</td><td>27.586 → 32.184</td><td></td>
  </tr>
  <tr>
    <td>SE</td><td>36.080 → 48.011</td><td>36.047 → 46.512</td><td>44.615 → 53.846</td><td>55.556 → 65.833</td><td>54.777 → 67.197</td><td>52.212 → 63.717</td><td>31.818 → 46.023</td><td>29.008 → 41.221</td><td>25.287 → 36.782</td>
  </tr>
  <tr>
    <td>PE+M</td><td>47.727 → 51.420</td><td>50.388 → 50.775</td><td>50.769 → 61.538</td><td>63.333 → 69.167</td><td>66.242 → 67.834</td><td>64.602 → 65.487</td><td>38.636 → 38.068</td><td>32.061 → 36.641</td><td>26.437 → 31.034</td>
  </tr>
  <tr>
    <td>SE+M</td><td>38.636 → 50.568</td><td>40.698 → 48.450</td><td>41.538 → 56.923</td><td>55.278 → 62.778</td><td>53.503 → 66.879</td><td>53.097 → 64.602</td><td>31.250 → 43.182</td><td>28.244 → 39.695</td><td>24.138 → 34.483</td>
  </tr>
  <tr>
    <td>EigV</td><td>24.432 → 35.227</td><td>24.419 → 34.496</td><td>16.923 → 32.308</td><td>38.333 → 55.278</td><td>39.172 → 55.414</td><td>38.938 → 53.982</td><td>21.591 → 34.091</td><td>20.611 → 32.824</td><td>8.046 → 17.241</td>
  </tr>
  <tr>
    <td>ECC</td><td>19.602 → 42.330</td><td>18.992 → 39.535</td><td>16.923 → 33.846</td><td>30.556 → 61.389</td><td>30.892 → 58.917</td><td>26.549 → 61.062</td><td>18.182 → 41.477</td><td>18.321 → 35.878</td><td>8.046 → 21.839</td>
  </tr>
  <tr>
    <td>Deg</td><td>25.284 → 29.830</td><td>24.806 → 28.295</td><td>20.000 → 26.154</td><td>42.500 → 49.167</td><td>42.357 → 49.363</td><td>42.478 → 50.442</td><td>22.727 → 26.136</td><td>19.084 → 22.137</td><td>11.494 → 19.540</td>
  </tr>
</table>

Table 13: Changes in the percentage of samples that satisfy the axioms before and after calibration for Llama2-chat. The relation function \( \mathcal{R} \) is implemented using MiniCheck.

Reviewer paper summaries: This paper systematically explores the reliability of uncertainty estimation (UE) methods when large language models perform generation based on the Retrieval-Augmented Generation (RAG) approach. First, the authors observe that existing uncertainty estimation methods exhibit bias regardless of whether the RAG retriever provides relevant or irrelevant documents. Second, the authors propose an axiomatic evaluation framework to verify whether UE under the RAG setup satisfies five proposed axioms. Results show that none of the existing UE methods fully satisfy all these axioms. To address this issue, the authors further propose a calibration method for existing UE approaches based on the axiomatic framework, which improves the correlation between UE and correctness.
Summary of strengths: 1. Uncertainty is an important metric for evaluating model generation behavior, especially for assessing the trustworthiness of RAG-based generation. This paper is the first to critically examine and challenge existing uncertainty estimation methods under the RAG setup, offering valuable insights for the RAG research community to think critically about the use of UE methods.
2. The paper selects representative white-box and black-box UE methods and conducts experiments across multiple RAG task datasets, making the results representative and convincing.
3. The proposed calibration method contributes to a potential approach for evaluating model trustworthiness during generation in the RAG setting, promoting the development of trustworthy AI.
Summary of weaknesses: 1. The details of the calibration method's setup are not fully disclosed. In Section 4, the calibration method relies on hyperparameters $k_1$–$k_4$, which, according to the paper, are derived from a `validation set`. However, the paper neither specifies the details of this validation set nor conducts ablation studies on these hyperparameters. This lack of detail may raise concerns about the robustness of the proposed calibration method.
2. Table 4 and Figure 3 require additional contextual information. For Table 4, it would be helpful to clarify in the caption what the bold and underlined values represent. Regarding Figure 3, it is not referenced in the main text. While it may relate to Section 6, this omission makes it inconvenient for readers.
Comments, suggestions and typos: - Line 520: Following (Kuhn et al., 2023) ->Following Kuhn et al. (2023)
- Line 545: esp. when -> especially when
- Line 530: p-value -> $p$-value
Overall assessment scores: 3