When to Speak, When to Abstain: Contrastive Decoding with Abstention
Authors: Hyuhng Joon Kim, Youna Kim, Sang-goo Lee, Taeuk Kim
Accepted at: acl25

When to Speak, When to Abstain: Contrastive Decoding with Abstention

Hyuhng Joon Kim1, Youna Kim1, Sang-goo Lee1,2, Taeuk Kim3 *
1Seoul National University, 2IntelliSys, Korea, 3Hanyang University
{heyjoonkim, anna9812, sglee}@europa.snu.ac.kr
kimtaeuk@hanyang.ac.kr

Abstract

Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks by leveraging pre-trained (i.e., parametric) and external (i.e., contextual) knowledge. While substantial efforts have been made to enhance the utilization of both forms of knowledge, situations in which models lack relevant information remain underexplored. To investigate this challenge, we first present a controlled testbed featuring four distinct knowledge access scenarios, including the aforementioned edge case, revealing that conventional LLM usage exhibits insufficient robustness in handling all instances. Addressing this limitation, we propose Contrastive Decoding with Abstention (CDA), a novel training-free decoding method that allows LLMs to generate responses when relevant knowledge is available and to abstain otherwise. CDA estimates the relevance of both knowledge sources for a given input, adaptively deciding which type of information to prioritize and which to exclude. Through extensive experiments, we demonstrate that CDA can effectively perform accurate generation and abstention simultaneously, enhancing reliability and preserving user trust.

1 Introduction

Large Language Models (LLMs) (Team et al., 2023; Achiam et al., 2023; Dubey et al., 2024) acquire extensive parametric knowledge during pre-training, enabling them to attain remarkable performance across a wide range of tasks. Although parametric knowledge can be comprehensive and highly informative, it is inherently bounded by the scope of the pre-training corpus. Consequently, LLMs become less reliable when processing inputs from underrepresented domains, such as those including domain-specific (Kandpal et al., 2023; Raja et al., 2024; Feng et al., 2024a) or outdated data (Lazaridou et al., 2024; Kasai et al., 2023; Zhao et al., 2024a).

To overcome this challenge, approaches that integrate previously unseen information during inference have emerged (Buttcher et al., 2016; Yin et al., 2016; Karpukhin et al., 2020). They provide external information to LLMs as contextual knowledge, expanding the knowledge boundary beyond what is learned from training.

Since LLMs are generally exposed to two distinct sources of information—parametric and contextual knowledge—they are expected to adaptively leverage both to maximize performance. Despite efforts to enhance such desired behavior, scenarios where neither parametric nor contextual knowledge is available—often encountered in real-world settings—remain largely underexplored. Compelling models to respond imprudently in such cases heightens the risk of hallucination, diminishes reliability, and introduces potential dangers in high-stakes applications.

Therefore, it is crucial for LLMs to abstain from responding when necessary information is inaccessible (Varshney et al., 2024; Zhang et al., 2024a; Wen et al., 2024) while preserving performance

Figure 1: This study considers four possible scenarios based on the existence of the model’s parametric and contextual knowledge. The model is expected to respond reliably by either (1) generating correct responses leveraging any form of relevant knowledge available or (2) abstaining from producing potentially inaccurate or misleading outputs when no relevant knowledge exists.
when relevant knowledge is available. However, such behavior requires a precise assessment of the knowledge and the integration of this assessment into the generation process, both of which are inherently challenging.

In this work, we first present a controlled testbed, where the accessibility of both types of knowledge for a query is explicitly determined. In contrast to typical scenarios where definitively confirming the availability of parametric or contextual knowledge is imprecise, our experimental setup facilitates controlled investigations, covering all scenarios depicted in Figure 1. Experimental results on this testbed indicate that existing methods for LLMs lack sufficient robustness in effectively handling all the considered scenarios.

To this end, we propose Contrastive Decoding with Abstention (CDA), a novel, training-free decoding method that enables LLMs to not only leverage relevant parametric or contextual knowledge during generation but also abstain when no appropriate knowledge is available. During the decoding process, CDA assesses the relevance of both forms of knowledge, adaptively determining the knowledge to attend to during generation. Moreover, CDA steers the models towards abstention if no relevant knowledge is available. The relevancy is estimated as the uncertainty associated with the knowledge in response to a specific query.

Extensive experiments with four LLMs on three question-answering (QA) datasets (Zhang et al., 2023; Etezadi and Shamsfard, 2023) demonstrate that CDA effectively enables LLMs to abstain in the absence of relevant knowledge while maintaining existing capabilities without additional training. Further validation against training-based methods demonstrated CDA’s robust generalization capabilities, while evaluations in retrieval-augmented generation (RAG) setting highlight its effectiveness across practical scenarios.

2 Related Work

2.1 Contrastive Decoding

Contrastive decoding (CD) controls text generation by contrasting different output distributions and steers the model in the desired direction. DExperts (Liu et al., 2021) employs an ensemble of an “expert” and an “anti-expert” for tasks such as detoxification. Li et al. (2023) contrasts the output distributions of a large LM and a small LM for open-ended text generation. CD is also proven effective in domains such as reasoning (O’Brien and Lewis, 2023) and machine translation (Waldendorf et al., 2024). Recently, there has been growing interest in context-aware contrastive decoding (CCD) (Zhao et al., 2024b; Kim et al., 2024b; Qiu et al., 2024; Shi et al., 2024b), which enables the model to leverage both parametric and contextual knowledge during decoding, tackling tasks such as knowledge conflicts (Longpre et al., 2021; Chen et al., 2022; Zhou et al., 2023). Despite the promising results, existing approaches assume that at least one knowledge source is always available. In practice, LLMs frequently encounter situations with no relevant knowledge, a gap that these methods fail to bridge. To address this limitation, we expand the scope to include such edge cases and propose a novel approach of integrating abstention to CCD.

2.2 Abstention in LLMs

LLMs often generate unintended or undesirable responses, such as hallucinations (Maynez et al., 2020; Ji et al., 2023; Jiang et al., 2024), biases (Sap et al., 2020; Feng et al., 2023), and harmful or unsafe outputs (Anwar et al., 2024; Ye et al., 2024; Zhang et al., 2024b). In such instances, it is appropriate for the model to abstain (Kamath et al., 2020; Feng et al., 2024b; Srinivasan et al., 2024) from generating unintended content. Abstention can be employed for unanswerable (Sulem et al., 2022; Amayuelas et al., 2024) or ambiguous (Min et al., 2020; Kim et al., 2024a) queries. Furthermore, models may abstain when relevant parametric knowledge is absent (Ahdrizt et al., 2024; Kim and Thorne, 2024). Abstention can be facilitated by utilizing confidence scores of generations (Sun et al., 2022; Kuhn et al., 2023; Duan et al., 2024) or training the model for abstention capabilities (Zhang et al., 2024a; Sun et al., 2024; Cohen et al., 2024). Unlike previous approaches, this work proposes a training-free decoding method, enabling off-the-shelf models to abstain when necessary.

3 Testbed Design for Controlled Analysis

The primary objective of this work is to enable the model to dynamically adjust its behavior based on the presence and absence of its knowledge. Specifically, the model must effectively address all four scenarios depicted in Figure 1. However, as we lack prior information regarding the model’s possessed knowledge, it is challenging to determine and evaluate whether the model should provide an answer
or abstain from a given query. Thus, we construct a testbed by explicitly controlling the accessibility of the knowledge to simulate all the scenarios. This section first formulates the problem and describes the setup process as illustrated in Figure 2. Further details are in Appendix A.

3.1 Problem Formulation

This paper focuses on QA tasks, which facilitate a clear assessment of the knowledge usage of the model. Parametric knowledge (\( \mathcal{P} \)) is defined as the knowledge the model acquires during pre-training, and contextual knowledge (\( \mathcal{C} \)) refers to the external knowledge provided within the input at inference time. The knowledge is deemed relevant if it contains information capable of generating an accurate response to the query. For a given query \( x \) and a context \( c \), the objective is to produce the ground-truth answer \( y \) when relevant knowledge is available or to abstain otherwise.

Figure 1 illustrates the scenarios addressed in this work. Inputs are defined answerable if one or more relevant knowledge are present (\( \mathcal{P}=1 \) or \( \mathcal{C}=1 \)). With relevant parametric knowledge (\( \mathcal{P}=1 \)), the model is expected to generate the correct answer regardless of \( c \). On the other hand, the model should generate grounded on \( c \) given relevant contextual knowledge (\( \mathcal{C}=1 \)). When no relevant knowledge is available (\( \mathcal{P}=0 \) and \( \mathcal{C}=0 \)), the query is considered unanswerable, and the model should refuse to generate incorrect responses. Thus, a reliable model should properly generate an accurate answer or abstain grounded on the possessed knowledge.

3.2 Initial Dataset Construction

The testbed utilizes three extractive QA datasets from the MRQA benchmark (Fisch et al., 2019); Natural Questions (NQ) (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), and TriviaQA (Joshi et al., 2017). Each dataset consists of a query \( x_i \), an answer \( y_i \), and a pre-defined context \( c_i \) containing one or more answer spans. We split \( c_i \) into 100-word spans containing \( y_i \) to avoid excessively long contexts. Through preprocessing, we construct \( \mathcal{D}_{\text{init}} = \{ (x_i, c_i, y_i) \}_{i=1}^{N_{\text{init}}} \).

3.3 Parametric Knowledge Estimation

To estimate the model’s parametric knowledge, we assess the generation consistency (Wang et al., 2023)\footnote{We assume the context is always factual and only focus on the relevance to the query. While incorporating the context’s factuality is practical, we consider it orthogonal to this study.} for a query \( x_i \in \mathcal{D}_{\text{init}} \). We prompt the model with the parametric template \( \mathcal{T}_p(\cdot) \) from Figure 3a, which relies solely on the model’s parametric knowledge for the prediction. By sampling \( n \) responses, we compute the consistency rate \( r = \frac{m}{n} \), where \( m \) is the number of correct responses. If \( r = 0 \), we assume the model lacks relevant parametric knowledge for \( x_i \). These samples are collected as \( \mathcal{D}_{\mathcal{P}=0} = \{ (x_i, c_i, y_i, p_i = 0) \}_{i=1}^{N_{\mathcal{P}=0}} \). On the other hand, the model is con-

![The overall process of dataset construction for the testbed.](page_120_120_1247_246.png)

Figure 2: The overall process of dataset construction for the testbed.

<table>
  <tr>
    <th>Answer the following question.</th>
    <th><i>&lt;few-shot demonstrations&gt;</i></th>
    <th>Question: <i>&lt;question&gt;</i></th>
    <th>Answer:</th>
  </tr>
</table>

(a) Parametric template \( \mathcal{T}_p(\cdot) \).

<table>
  <tr>
    <th>Answer the following question based on the given context.</th>
    <th><i>&lt;few-shot demonstrations&gt;</i></th>
    <th>Context: <i>&lt;context&gt;</i></th>
    <th>Question: <i>&lt;question&gt;</i></th>
    <th>Answer:</th>
  </tr>
</table>

(b) Contextual template \( \mathcal{T}_c(\cdot) \).

<table>
  <tr>
    <th>Answer the following question based on the given context. If you are unsure or don’t know the correct answer, simply respond with “Unknown”.</th>
    <th><i>&lt;few-shot demonstrations&gt;</i></th>
    <th>Context: <i>&lt;context&gt;</i></th>
    <th>Question: <i>&lt;question&gt;</i></th>
    <th>Answer:</th>
  </tr>
</table>

(c) Explicit abstention template \( \mathcal{T}_a(\cdot) \).

Figure 3: List of inference templates.
sidered to pose relevant parametric knowledge if \( r > \eta \) for a pre-defined threshold \( \eta \), given its consistent accuracy. These samples are grouped into \( \mathcal{D}_{\mathcal{P}=1} = \{ (x_i, c_i, y_i, p_i = 1) \}_{i=1}^{N_{\mathcal{P}=1}} \). The resulting dataset is defined as \( \mathcal{D}_{\mathcal{P}} = \mathcal{D}_{\mathcal{P}=0} + \mathcal{D}_{\mathcal{P}=1} \).

3.4 Contextual Knowledge Estimation

In this stage, we select relevant and irrelevant contexts for a given query.

Relevant Context Selection  We provide the model with a contextual template \( \mathcal{T}_c(c_i, x_i) \) from Figure 3b, where \( (x_i, c_i) \in \mathcal{D}_{\mathcal{P}} \). The model can leverage contextual knowledge by providing \( c_i \) as the input. We further verify the relevance of \( c_i \) by computing the consistency rate \( r \). Samples with \( r > \eta \) are defined as the relevant context \( c_i^+ \) and are grouped into \( \mathcal{D}_{C=1} = \{ (x_i, c_i^+, y_i, p_i) \}_{i=1}^{N_{C=1}} \).

Irrelevant Context Selection  For each sample \( (x_i, c_i^+) \in \mathcal{D}_{C=1} \), we select an irrelevant context candidate \( c_j^{\text{train}} \) from the training set. The candidate is selected with the highest SBERT (Reimers and Gurevych, 2019) embedding similarity to \( c_i^+ \) to avoid overly unrelated contexts. We then prompt the model with \( \mathcal{T}_c(c_j^{\text{train}}, x_i) \) and measure the consistency rate \( r \). Only candidates with \( r = 0 \) are considered irrelevant context \( c_i^- \), ensuring that \( c_i^- \) does not provide any unintended relevant information. The resulting dataset is defined as \( \mathcal{D}_{C=0} = \{ (x_i, c_i^+, c_i^-, y_i, p_i) \}_{i=1}^{N_{C=0}} \).

3.5 Final Dataset Construction

Finally, we randomly select an equal number of samples with \( p_i = 0 \) and \( p_i = 1 \), constructing \( \mathcal{D} = \{ (x_i, c_i^+, c_i^-, y_i, p_i) \}_{i=1}^{N} \). Note that the number of selected data varies across models due to their distinct knowledge boundaries.

4 Contrastive Decoding with Abstention

Contrastive Decoding with Abstention (CDA) is a novel decoding method integrating abstention within the CCD process. This section provides a detailed description of the overall process.

4.1 Preliminary

Given a model \( \theta \) at decoding step \( t \), the parametric knowledge distribution \( d_t^p \) and the contextual knowledge distribution \( d_t^c \) is defined as follows:

\[
d_t^p = \operatorname{logit}_\theta(y_t \mid \mathcal{T}_p(x, y_{<t})),
\]
\[
d_t^c = \operatorname{logit}_\theta(y_t \mid \mathcal{T}_c(c, x, y_{<t}))
\] (1)

where \( y_{<t} \) are previously generated tokens. CCD measures the final output distribution \( d_t^o \) as an ensemble of \( d_t^p \) and \( d_t^c \) as:

\[
d_t^o = d_t^p + w_t^c (d_t^c - d_t^p)
\] (2)

The weight \( w_t^c \) should precisely quantify the relevance of \( c \), ensuring a higher weight when \( c \) is deemed relevant.

4.2 Incorporating Abstention

To enable CDA to properly abstain, we incorporate the abstention distribution \( d_t^a \) computed from an explicit abstention instruction \( \mathcal{T}_a(\cdot) \) in Figure 3c.

\[
d_t^a = \operatorname{logit}_\theta(y_t \mid \mathcal{T}_a(c, x, y_{<t}))
\] (3)

CDA expands Eq. 2 by applying \( d_t^a \) for the final output distribution, where the weight for \( d_t^a \) is defined as \( w_t^a = 1 - w_t^p - w_t^c \).

\[
\begin{align*}
d_t^o &= d_t^p + w_t^c (d_t^c - d_t^p) + w_t^a (d_t^a - d_t^p) \\
&= (1 - w_t^c - w_t^a) d_t^p + w_t^c d_t^c + w_t^a d_t^a \\
&= w_t^p d_t^p + w_t^c d_t^c + (1 - w_t^p - w_t^c) d_t^a
\end{align*}
\] (4)

Intuitively, \( w_t^a \) decreases when the model is confident of a possessed knowledge, whereas it increases when both knowledge are uncertain.

4.3 Knowledge Relevance Assessment via Uncertainty Calibration

A key requirement for CDA is that the weights \( w_t^p \) and \( w_t^c \) should effectively quantify the relevance of the corresponding knowledge. We assess the relevance as the uncertainty of the corresponding knowledge regarding \( x \). Specifically, we utilize the entropy (Malinin and Gales, 2021; Abdar et al., 2021), a widely used measure to assess the uncertainty of the knowledge for a query (Kuhn et al., 2023; Duan et al., 2024), particularly prevalent in CCD (Kim et al., 2024b; Qiu et al., 2024). For an output distribution \( d \), the entropy \( \mathcal{H} \) is defined as:

\[
\mathcal{H} = - \sum_{i=1}^{|\mathcal{V}|} d_i \log d_i,
\] (5)

where \( d_i \) is the \( i \)th token of the vocabulary \( \mathcal{V} \). The parametric uncertainty \( \mathcal{H}_t^p \) and contextual uncertainty \( \mathcal{H}_t^c \) are derived from their respective distributions \( d_t^p \) and \( d_t^c \).

Nonetheless, directly comparing \( \mathcal{H}_t^p \) and \( \mathcal{H}_t^c \) are imprecise since they are conditioned on distinct inputs and possibly miscalibrated. To address this,
we “calibrate” (Zhao et al., 2021; Holtzman et al., 2021; He et al., 2024) the uncertainty measures by accounting for the model’s intrinsic bias. Specifically, we estimate the bias with a “content-free” null prompt, replacing specific inputs \( x_i \) and \( c_i \) with placeholder tokens \( \bar{x} \) and \( \bar{c} \) to remove any specific semantic information. Applying the templates \( \mathcal{T}_p(\bar{x}) \) and \( \mathcal{T}_c(\bar{c}, \bar{x}) \) yields the parametric null distribution \( d^p_t \) and the contextual null distribution \( d^c_t \), along with their corresponding entropy values \( \hat{H}^p_t \) and \( \hat{H}^c_t \). The confidence for the knowledge is quantified as the additional information provided by the input relative to the null prompt.

\[
r^p_t = \frac{\max(\hat{H}^p_t - \hat{H}^p_0, 0)}{\hat{H}^p_t}, \quad r^c_t = \frac{\max(\hat{H}^c_t - \hat{H}^c_0, 0)}{\hat{H}^c_t}.
\] (6)

We obtain the final weights \( w^p_t \) and \( w^c_t \) from Eq. 4 by normalizing \( r^p_t \) and \( r^c_t \), respectively.

\[
w^p_t = \frac{r^p_t}{r^p_t + r^c_t} \cdot r^p_t, \quad w^c_t = \frac{r^c_t}{r^p_t + r^c_t} \cdot r^c_t
\] (7)

When a particular knowledge source provides substantial additional information relative to the null prompt, it is assigned a higher weight, whereas the absence of knowledge results in a reduced weight.

4.4 CDA with Momentum (CDA-m)

At each decoding step, previous content may unintentionally steer the model toward irrelevant knowledge. To mitigate this, we apply momentum, updating the current weight \( w_t \) as a convex combination with the previous weight \( w_{t-1} \).

\[
w_t \leftarrow \alpha \, w_{t-1} + (1 - \alpha) \, w_t
\] (8)

Here, the hyperparameter \( \alpha \) controls the influence of the previous step on the current step. Applying momentum helps stabilize the decoding process by smoothing abrupt weight changes.

Figure 4: Illustration of all possible results. The model should generate correct answers for answerable queries (N₁) and abstain for unanswerable queries (N₅). Any other responses (N₂, N₃, N₄) are classified as incorrect.

5 Experiments

5.1 Experimental Setting

The experiments utilize the testbed from Section 3 and four instruction-tuned models including LLAMA3 8B INSTRUCT (Dubey et al., 2024), LLAMA2 7B & 13B CHAT (Touvron et al., 2023), and MISTRAL 7B INSTRUCT (Jiang et al., 2023). Results are averaged over three different random seeds. Further details are stated in Appendix B.

5.2 Evaluation Metric

To measure the overall performance across two distinct tasks, we adopt three metrics, each reflecting unique aspects of performance, based on the five possible results in Figure 4.

Answerable Prediction F1 (F1_ans) For answerable queries, we compute F1_ans (Kim et al., 2024a) as the harmonic mean of precision (\( \frac{N_1}{N_1+N_2+N_3} \)) and recall (\( \frac{N_1}{N_1+N_2+N_5} \)). The prediction is considered correct if it contains the ground-truth answer (Mallen et al., 2023; Schick et al., 2023).

Abstention F1 (F1_abs) The model should abstain from incorrect responses for unanswerable queries while minimizing over-abstention. F1_abs (Kim et al., 2024a) measures such behaviors by incorporating both precision (\( \frac{N_5}{N_5+N_2+N_3} \)) and recall (\( \frac{N_5}{N_5+N_2+N_5} \)). A prediction is deemed as an abstention if it contains any pre-defined abstention phrases (Amayuelas et al., 2024; Kim et al., 2024a).

Reliability Score (RS) RS (Xu et al., 2024) is the weighted sum of accuracy (Acc., \( \frac{N_1}{N} \)) and coverage (Cov., \( \frac{N_1+N_2+N_5}{N} \)), where N is the total number of samples and \( \alpha \) is set as the answer rate (\( 1 - \frac{N_3+N_4}{N} \)).

\[
\text{RS}(\alpha) = \alpha \times \text{Cov.} + (1 - \alpha) \times \text{Acc.}
\] (9)

RS prioritizes accuracy at a lower answer rate while avoiding errors with a higher coverage at a high answer rate. We also report the accuracy and coverage for a more thorough analysis.

5.3 Baselines

To evaluate the effectiveness of our approach, we compare different inference methods as baselines.

Direct Prompting includes contextual prompting (CONTEXT) employing \( \mathcal{T}_c(\cdot) \) and abstention prompting (ABSTAIN) utilizing \( \mathcal{T}_a(\cdot) \), with an explicit instruction for abstention.
<table>
  <tr>
    <th rowspan="2">Backbone</th>
    <th colspan="6">LLAMA3 8B INSTRUCT</th>
    <th colspan="6">LLAMA2 7B CHAT</th>
    <th colspan="6">LLAMA2 13B CHAT</th>
    <th colspan="6">MISTRAL 7B INSTRUCT</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>F<sub>1</sub><sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
  </tr>
  <tr>
    <td colspan="25">NQ</td>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>57.26</td>
    <td>2.18</td>
    <td>50.22</td>
    <td>49.95</td>
    <td>50.23</td>
    <td>57.13</td>
    <td>0.09</td>
    <td>49.98</td>
    <td>49.97</td>
    <td>49.98</td>
    <td>57.14</td>
    <td>0.37</td>
    <td>50.00</td>
    <td>49.95</td>
    <td>50.00</td>
    <td>57.04</td>
    <td>0.25</td>
    <td>49.88</td>
    <td>49.85</td>
    <td>49.88</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>53.34</td>
    <td>2.14</td>
    <td>48.56</td>
    <td>48.29</td>
    <td>48.56</td>
    <td>54.78</td>
    <td>0.17</td>
    <td>47.94</td>
    <td>47.92</td>
    <td>47.94</td>
    <td>47.54</td>
    <td>0.48</td>
    <td>39.90</td>
    <td>39.84</td>
    <td>39.90</td>
    <td>55.03</td>
    <td>1.15</td>
    <td>48.21</td>
    <td>48.07</td>
    <td>48.21</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>71.36</td>
    <td>0.92</td>
    <td>62.50</td>
    <td>62.39</td>
    <td>62.50</td>
    <td>64.55</td>
    <td>0.15</td>
    <td>56.48</td>
    <td>56.46</td>
    <td>56.46</td>
    <td>66.66</td>
    <td>0.18</td>
    <td>58.33</td>
    <td>58.31</td>
    <td>58.33</td>
    <td>68.48</td>
    <td>0.09</td>
    <td>59.93</td>
    <td>59.93</td>
    <td>59.93</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>60.22</td>
    <td>5.05</td>
    <td>48.17</td>
    <td>38.36</td>
    <td>57.27</td>
    <td>38.73</td>
    <td>45.16</td>
    <td>26.73</td>
    <td>20.54</td>
    <td>42.11</td>
    <td>50.49</td>
    <td>46.62</td>
    <td>37.83</td>
    <td>29.70</td>
    <td>48.93</td>
    <td>61.86</td>
    <td>5.29</td>
    <td>52.85</td>
    <td>42.83</td>
    <td>59.70</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>64.06</td>
    <td>5.34</td>
    <td>53.53</td>
    <td>45.15</td>
    <td>60.90</td>
    <td>58.68</td>
    <td>41.09</td>
    <td>41.09</td>
    <td>51.88</td>
    <td>44.47</td>
    <td>54.08</td>
    <td>59.23</td>
    <td>42.61</td>
    <td>52.75</td>
    <td>44.55</td>
    <td>51.19</td>
    <td>56.29</td>
    <td>55.48</td>
    <td>46.63</td>
    <td>60.33</td>
  </tr>
  <tr>
    <td>ENTROPY</td>
    <td>64.06</td>
    <td>5.34</td>
    <td>53.53</td>
    <td>45.15</td>
    <td>60.90</td>
    <td>58.68</td>
    <td>41.09</td>
    <td>41.09</td>
    <td>51.88</td>
    <td>44.47</td>
    <td>54.08</td>
    <td>59.23</td>
    <td>42.61</td>
    <td>52.75</td>
    <td>44.55</td>
    <td>51.19</td>
    <td>56.29</td>
    <td>55.48</td>
    <td>46.63</td>
    <td>60.33</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>63.56</td>
    <td>5.24</td>
    <td>53.99</td>
    <td>43.82</td>
    <td>60.11</td>
    <td>48.82</td>
    <td>39.52</td>
    <td>37.55</td>
    <td>30.41</td>
    <td>45.66</td>
    <td>57.96</td>
    <td>46.34</td>
    <td>49.28</td>
    <td>40.27</td>
    <td>54.43</td>
    <td>61.46</td>
    <td>54.02</td>
    <td>55.80</td>
    <td>45.58</td>
    <td>59.56</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>69.27</td>
    <td>5.94</td>
    <td>59.46</td>
    <td>49.02</td>
    <td>65.09</td>
    <td>55.04</td>
    <td>47.26</td>
    <td>43.26</td>
    <td>34.47</td>
    <td>52.20</td>
    <td>62.44</td>
    <td>47.60</td>
    <td>53.66</td>
    <td>44.46</td>
    <td>58.19</td>
    <td>66.71</td>
    <td>55.51</td>
    <td>58.95</td>
    <td>48.32</td>
    <td>63.65</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td>72.06</td>
    <td><b>55.49</b></td>
    <td>62.95</td>
    <td>52.28</td>
    <td>67.51</td>
    <td>66.86</td>
    <td>47.52</td>
    <td>59.86</td>
    <td>51.22</td>
    <td>62.38</td>
    <td>68.62</td>
    <td>47.14</td>
    <td>61.63</td>
    <td>53.16</td>
    <td>63.76</td>
    <td>69.68</td>
    <td>56.47</td>
    <td>61.45</td>
    <td>50.61</td>
    <td>66.09</td>
  </tr>
  <tr>
    <td>CDA-m</td>
    <td><b>73.15</b></td>
    <td>55.47</td>
    <td>63.72</td>
    <td>53.16</td>
    <td><b>68.30</b></td>
    <td>69.99</td>
    <td>47.60</td>
    <td>62.28</td>
    <td>53.62</td>
    <td>64.81</td>
    <td>70.66</td>
    <td>48.12</td>
    <td>63.18</td>
    <td>54.46</td>
    <td><b>65.48</b></td>
    <td>71.00</td>
    <td><b>56.46</b></td>
    <td>62.30</td>
    <td>51.45</td>
    <td>67.03</td>
  </tr>
  <tr>
    <td colspan="25">HopQA</td>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>57.15</td>
    <td>1.19</td>
    <td>50.08</td>
    <td>49.93</td>
    <td>50.08</td>
    <td>57.14</td>
    <td>0.00</td>
    <td>49.98</td>
    <td>49.98</td>
    <td>49.98</td>
    <td>57.16</td>
    <td>0.10</td>
    <td>50.00</td>
    <td>49.98</td>
    <td>50.00</td>
    <td>56.99</td>
    <td>0.09</td>
    <td>49.88</td>
    <td>49.87</td>
    <td>49.88</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>55.78</td>
    <td>1.88</td>
    <td>48.89</td>
    <td>51.99</td>
    <td>48.89</td>
    <td>54.68</td>
    <td>0.04</td>
    <td>47.84</td>
    <td>47.84</td>
    <td>47.84</td>
    <td>54.60</td>
    <td>0.07</td>
    <td>47.76</td>
    <td>47.75</td>
    <td>47.76</td>
    <td>53.74</td>
    <td>0.20</td>
    <td>47.04</td>
    <td>47.01</td>
    <td>47.04</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>74.36</td>
    <td>0.57</td>
    <td>65.09</td>
    <td>65.02</td>
    <td>65.09</td>
    <td>69.27</td>
    <td>0.00</td>
    <td>60.60</td>
    <td>60.60</td>
    <td>60.60</td>
    <td>69.08</td>
    <td>0.07</td>
    <td>61.15</td>
    <td>61.14</td>
    <td>61.15</td>
    <td>72.72</td>
    <td>0.09</td>
    <td>63.64</td>
    <td>63.62</td>
    <td>63.64</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>66.88</td>
    <td>5.68</td>
    <td>56.23</td>
    <td>45.21</td>
    <td>63.55</td>
    <td>47.29</td>
    <td>48.55</td>
    <td>33.49</td>
    <td>26.18</td>
    <td>48.17</td>
    <td>51.71</td>
    <td>51.95</td>
    <td>43.02</td>
    <td>33.89</td>
    <td>55.10</td>
    <td>61.12</td>
    <td>54.66</td>
    <td>53.06</td>
    <td>42.55</td>
    <td>59.23</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>50.64</td>
    <td>48.62</td>
    <td>33.86</td>
    <td>28.60</td>
    <td>49.70</td>
    <td>58.85</td>
    <td>15.33</td>
    <td>15.29</td>
    <td>49.70</td>
    <td>52.33</td>
    <td>58.22</td>
    <td>14.36</td>
    <td>51.87</td>
    <td>49.89</td>
    <td>51.95</td>
    <td>63.36</td>
    <td>43.36</td>
    <td>56.36</td>
    <td>48.95</td>
    <td>57.70</td>
  </tr>
  <tr>
    <td>ENTROPY</td>
    <td>67.08</td>
    <td>5.44</td>
    <td>57.92</td>
    <td>46.94</td>
    <td>63.88</td>
    <td>59.00</td>
    <td>45.11</td>
    <td>52.44</td>
    <td>44.16</td>
    <td>55.59</td>
    <td>59.24</td>
    <td>45.77</td>
    <td>53.01</td>
    <td>44.49</td>
    <td>55.81</td>
    <td>63.31</td>
    <td>57.19</td>
    <td>56.55</td>
    <td>46.52</td>
    <td>62.40</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>65.18</td>
    <td>5.85</td>
    <td>66.21</td>
    <td>55.05</td>
    <td>65.55</td>
    <td>55.82</td>
    <td>57.53</td>
    <td>45.77</td>
    <td>42.04</td>
    <td>68.46</td>
    <td>54.16</td>
    <td>60.41</td>
    <td>50.05</td>
    <td>64.79</td>
    <td>74.43</td>
    <td>52.18</td>
    <td>68.20</td>
    <td>59.27</td>
    <td>69.94</td>
    <td>74.94</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>74.71</td>
    <td>6.29</td>
    <td>70.20</td>
    <td>58.36</td>
    <td>74.52</td>
    <td>73.29</td>
    <td>42.41</td>
    <td>66.96</td>
    <td>60.15</td>
    <td>67.82</td>
    <td>73.69</td>
    <td>58.81</td>
    <td>68.92</td>
    <td>59.42</td>
    <td>70.42</td>
    <td>76.94</td>
    <td>56.67</td>
    <td>69.98</td>
    <td>59.85</td>
    <td>72.49</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td>79.32</td>
    <td>62.59</td>
    <td>70.64</td>
    <td>58.78</td>
    <td>74.99</td>
    <td>74.09</td>
    <td>42.31</td>
    <td>67.00</td>
    <td>68.37</td>
    <td>73.66</td>
    <td>58.89</td>
    <td>68.92</td>
    <td>59.42</td>
    <td>70.42</td>
    <td>76.94</td>
    <td>56.67</td>
    <td>69.98</td>
    <td>59.85</td>
    <td>72.49</td>
    <td>72.49</td>
  </tr>
  <tr>
    <td>CDA-m</td>
    <td><b>80.93</b></td>
    <td>65.66</td>
    <td>72.74</td>
    <td>60.40</td>
    <td><b>77.87</b></td>
    <td>73.47</td>
    <td>52.10</td>
    <td>67.11</td>
    <td>68.04</td>
    <td>73.12</td>
    <td>53.46</td>
    <td><b>65.82</b></td>
    <td>56.09</td>
    <td><b>68.52</b></td>
    <td>76.95</td>
    <td><b>57.06</b></td>
    <td>68.38</td>
    <td>57.84</td>
    <td>72.21</td>
    <td>72.21</td>
  </tr>
</table>

Table 1: Experimental results on three different datasets. For each dataset, the best method is highlighted in bold, and the second-best method is underlined. CDA(-m) outperforms all the baselines across different metrics.

**Self-Ask** prompts the model with \( \mathcal{T}_c(\cdot) \) and further verifies the generation (Kadavath et al., 2022). Predictions verified as “unknown” are abstained.

**Context-aware Decoding (CAD)** amplifies contextual influence by gauging \( d_t^c = d_t^r + w_t^c \) (\( d_t^r - d_t^p \)) with a fixed \( w_t^c \) during decoding (Shi et al., 2024b).

**Entropy** measures the entropy (Eq. 5) of the generated tokens when prompted with \( \mathcal{T}_c(\cdot) \) (Huang et al., 2025). We measure four different variants — first-token, average, maximum, and minimum entropy — and report the first-token entropy with the best performance. If the measure exceeds a pre-defined threshold, the prediction is deemed uncertain and thus abstained.

**Adaptive Contrastive Decoding (ACD)** follows Eq. 2 where \( w_t^c = 1 - \frac{H_t^c}{H_t^c + H_t^r} \) (Kim et al., 2024b).

**ACD with Abstention (ACD-A)** expands ACD to perform abstention where \( w_t^c = 1 - \frac{H_t^c}{H_t^c + H_t^r + H_t^a} \) and \( w_t^a = 1 - \frac{H_t^a}{H_t^c + H_t^r + H_t^a} \) following Eq. 4.

**First Step Branching (FSB)** compares the first-token entropy \( H_1^p, H_1^c, \) and \( H_1^a \) when prompted with \( \mathcal{T}_p(\cdot), \mathcal{T}_c(\cdot), \) and \( \mathcal{T}_a(\cdot) \), respectively. We select the most certain method and continue the generation with the selected method.

**5.4 Main Results**

The main results are presented in Table 1.

Methods not accounting for abstention fail to handle unanswerable queries. Methods such as CONTEXT, CAD, and ACD exhibit near-zero F1<sub>abs</sub>, indicating their inability to handle unanswerable queries. Moreover, the negligible gap between their accuracy and coverage further confirms their incapability to abstain.

Incorporating abstention enhances the handling of unanswerable queries. ABSTAIN and SELF-ASK exhibit biased abstentions, resulting in low accuracy and high coverage. ACD-A and ENTROPY perform abstention to some extent, but they struggle to balance between accurate generation and abstention. FSB emerges as the strongest among the baseline, effectively addressing (un)answerable queries. Overall, incorporating abstention does provide the model with the ability to abstain, but the baselines fail to effectively balance the trade-off
Figure 5: The accuracy and coverage for all the scenarios regarding both knowledge. CDA(-m) effectively balances between correct predictions and abstentions, especially in the presence of irrelevant contexts (C=0).

between accurate generation and appropriate abstention.

CDA(-m) exhibits superior performance across all datasets. CDA(-m) outperforms all the baselines on F1_{abs} and RS, properly abstaining unanswerable queries. Moreover, its effective handling of answerable queries exhibits the highest F1_{ans} and the second-best accuracy following ACD.

6 Ablation Study

This section presents ablation studies of CDA(-m). Unless otherwise specified, all experiments are conducted on LLAMA3 8B INSTRUCT with the testbed from Section 3. Methods capable of abstention, including ABSTAIN, SELF-ASK, ENTROPY, FSB, and ACD-A, are utilized for comparison. Further details and results are in Appendix C.

6.1 Analysis of Different Scenarios

Figure 5 depicts the accuracy (in blue) and coverage (in gray) for each scenario in the NQ dataset. Note that the main objective is to balance between accurate generation when relevant knowledge is present (\( \mathcal{P}=1 \) or \( \mathcal{C}=1 \)) and abstention when such knowledge is absent (\( \mathcal{P}=0 \) and \( \mathcal{C}=0 \)). Most baselines exhibit over-abstention, particularly with irrelevant context. These methods fail to utilize relevant parametric knowledge, resulting in either biased abstention or incorrect generation. In contrast, CDA(-m) robustly leverages proper knowledge while maintaining balanced abstention capability.

6.2 Ablation on Momentum Weight

In this section, we investigate the impact of momentum weight \( \alpha \) on CDA-m. Figure 6 displays F1_{ans} and F1_{abs} of \( \alpha \) from 0.0 to 1.0, along with the best-performing baselines in the NQ dataset. We can observe that F1_{abs} remains stable while F1_{ans} improves when applying momentum. As intended, momentum reduces hallucinations while preserving abstention capabilities. Overall, CDA-m outperforms the strongest baselines regardless of \( \alpha \).

We further conduct case studies of how this find-

Figure 6: F1_{ans} and F1_{abs} according to different \( \alpha \) values. Applying momentum significantly improves F1_{ans}.

<table>
  <tr>
    <th>Query</th>
    <th>Irrelevant Context</th>
    <th>Ground-truth Answer</th>
  </tr>
  <tr>
    <td>What is the full form of IB board</td>
    <td>The International Bank for Reconstruction and Development (IBRD) is an international financial institution that offers loans to ...</td>
    <td>International Baccalaureate</td>
  </tr>
</table>

(a) Input query, irrelevant context, and ground-truth answer.

(b) Incorrect CDA output (\( \alpha = 0.0 \))

(c) Correct CDA-m output (\( \alpha = 0.7 \))

Figure 7: Example generation of CDA(-m) with an irrelevant context. Irrelevant context contains phrases that are similar to the ground-truth answer, making it easier for the model to hallucinate. The weights of CDA shift to the irrelevant context, resulting in incorrect generation. On the other hand, the momentum applied to CDA-m mitigates abrupt shifts in attention, resulting in correct generation.
<table>
  <tr>
    <th>Dataset</th>
    <th>Method</th>
    <th>F1<sub>abs</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
  </tr>
  <tr>
    <td rowspan="2">NQ</td>
    <td>CDA</td>
    <td>72.06</td>
    <td>55.49</td>
    <td>62.95</td>
    <td>52.28</td>
    <td>67.51</td>
  </tr>
  <tr>
    <td>w/o calibration</td>
    <td>59.35</td>
    <td>52.06</td>
    <td>47.89</td>
    <td>38.04</td>
    <td>56.79</td>
  </tr>
  <tr>
    <td rowspan="2">HotpotQA</td>
    <td>CDA</td>
    <td>78.71</td>
    <td>62.50</td>
    <td>70.20</td>
    <td>58.36</td>
    <td>74.52</td>
  </tr>
  <tr>
    <td>w/o calibration</td>
    <td>66.94</td>
    <td>56.39</td>
    <td>56.42</td>
    <td>45.43</td>
    <td>63.55</td>
  </tr>
  <tr>
    <td rowspan="2">TriviaQA</td>
    <td>CDA</td>
    <td>80.39</td>
    <td>65.66</td>
    <td>72.35</td>
    <td>60.01</td>
    <td>76.67</td>
  </tr>
  <tr>
    <td>w/o calibration</td>
    <td>67.56</td>
    <td>57.24</td>
    <td>56.46</td>
    <td>45.32</td>
    <td>64.17</td>
  </tr>
</table>

Table 2: The effects of applying calibration. We can observe significant degradation without calibration.

ing relates to the momentum weights. Figure 7a demonstrates an example of a question and an irrelevant context. Figure 7b displays the generation result of CDA and the weights measured for the knowledge at every decoding step. CDA initially assigns more weight to relevant parametric knowledge, generating the correct span up to “International Bank”. However, the model’s attention shifts toward contextual knowledge, incorporating incorrect information from the context span “International Bank for Reconstruction and Development.” We presume that the phrase “International Bank” from the irrelevant context causes this shift in weight. In contrast, Figure 7c demonstrates that CDA-M, leveraging momentum, mitigates abrupt shifts in attention toward irrelevant knowledge, resulting in accurate generation.

Notably, CDA-M does not persistently focus on a single knowledge source. A case where CDA-M fully utilizes both knowledge appropriately is shown in Figure 8. Figure 8a displays a case where a relevant context is provided to the model. We can observe from Figure 8b that CDA-M initially focuses on the relevant parametric knowledge, and over time, it transitions to incorporate contextual knowledge, producing richer and more nuanced answers.

6.3 Effect of Calibration

CDA(-M) leverages calibrated uncertainty measures to quantify the relevance of different knowledge. To evaluate the effect of calibration, we modify Eq. 4 to employ non-calibrated measure by directly setting \( r_t^p = \mathcal{H}_t^p \) and \( r_t^c = \mathcal{H}_t^c \). As shown in Table 2, this change significantly degrades the performance across all metrics by up to 14 points. Directly utilizing uncalibrated measures yields sub-optimal results, thus highlighting the importance of the additional calibration step in CDA(-M).

<Query>
The oligodynamic effect is a phenomenon that describes
</Query>
<Relevant Context>
The oligodynamic effect (from Greek oligos “few”, and dynamis “force”) is a biocidal effect of metals, especially heavy metals, that occurs even in low concentrations ...
</Relevant Context>
<Ground-truth Answer>
A biocidal effect of metals
</Ground-truth Answer>
(a) Input query, relevant context, and ground-truth answer.

![Correct CDA-M output (α = 0.7)](page_1012_1012_388_246.png)
(b) Correct CDA-M output (\( \alpha = 0.7 \))

Figure 8: Example generation of CDA-M for a relevant context. CDA-M initially focuses on the parametric knowledge, and the attention shifts to incorporate contextual knowledge, generating richer output.

6.4 Comparison with Training-based Methods

Training models to abstain has demonstrated notable performance. Hence, we compare CDA-M with instruction-tuning (Ouyang et al., 2022; Yang et al., 2024), which explicitly trains the model to abstain when necessary.

Experimental Setting The training data are labeled according to parametric and contextual knowledge. Following the procedure described in Section 3, we verify whether the model possesses relevant knowledge for each training sample. Samples with relevant knowledge are labeled with the ground-truth answer \( y \), while samples without any relevant knowledge are labeled with a pre-defined abstention response \( y_{abs} \) (e.g., “unknown”). The model is then trained to generate the label given \( T_c(c, x) \). For evaluation, we utilize the instruction-tuned model to generate an output given \( T_c(c, x) \).

Experimental Results Table 3 presents the in-domain (IND) and out-of-domain (OOD) results of instruction-tuning. CDA-M consistently outperforms instruction-tuning, even in IND scenarios. Furthermore, while training often tailors the model to specific domains, resulting in significant performance drops in OOD settings, CDA-M demonstrates superior generalization capabilities. This robustness makes CDA(-M) a more applicable solution for practical scenarios.
<table>
  <tr>
    <th rowspan="2">Target (→)</th>
    <th colspan="3">NQ</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">TriviaQA</th>
  </tr>
  <tr>
    <th>F1_abst</th>
    <th>F1_abst</th>
    <th>RS</th>
    <th>F1_abst</th>
    <th>F1_abst</th>
    <th>RS</th>
    <th>F1_abst</th>
    <th>F1_abst</th>
    <th>RS</th>
  </tr>
  <tr>
    <td>NQ</td>
    <td>66.37</td>
    <td>43.87</td>
    <td>61.12</td>
    <td>64.43</td>
    <td>48.79</td>
    <td>59.65</td>
    <td>67.31</td>
    <td>49.00</td>
    <td>62.14</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>64.86</td>
    <td>42.73</td>
    <td>59.80</td>
    <td>74.06</td>
    <td>57.57</td>
    <td>68.76</td>
    <td>65.26</td>
    <td>50.69</td>
    <td>61.17</td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>65.15</td>
    <td>46.37</td>
    <td>57.70</td>
    <td>66.68</td>
    <td>49.82</td>
    <td>60.58</td>
    <td>67.60</td>
    <td>53.68</td>
    <td>61.84</td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td><b>73.15</b></td>
    <td><b>55.47</b></td>
    <td><b>63.72</b></td>
    <td><b>79.32</b></td>
    <td><b>62.59</b></td>
    <td><b>70.64</b></td>
    <td><b>80.93</b></td>
    <td><b>65.66</b></td>
    <td><b>72.74</b></td>
  </tr>
</table>

Table 3: The results of instruction-tuning. In-domain results, the best results, and the second-based results are highlighted. CDA-M displays superior performance across all the scenarios.

![Bar chart showing average Reliability Score (RS) in RAG settings. CDA(-M) outperforms all the baselines.](page_184_563_627_246.png)

Figure 9: Average Reliability Score (RS) in RAG settings. CDA(-M) outperforms all the baselines.

6.5 Evaluation on RAG Setting

To evaluate CDA in practical, real-world scenarios, we conduct experiments within the RAG setting.

Experimental Setting We utilize CONTRIEVER-MSMARCO (Izard et al., 2022) as a retriever, and the top-1 context is retrieved from the Wikipedia contexts.4 Unlike the controlled setting, where the presence of both knowledge is precisely estimated, the prior knowledge of the given query is unknown in the RAG setting. Since it is difficult to determine the answerability of the given query, we evaluate solely based on the Reliability Score (RS).

Experimental Results Figure 9 displays the average results in the RAG setting. Similar to the main experiments, methods without abstention capabilities demonstrate poor performance, while FSB and ENTROPY emerge as strong baselines. Overall, CDA(-M) outperform all the baselines, highlighting the effectiveness in the practical RAG setting.

6.6 Output Distribution Analysis

This section analyzes how the output distribution shifts from parametric and contextual distributions to the final distribution of CDA. Figure 10 displays the top-5 softmax probabilities along with their corresponding tokens for \( d^p_i \), \( d^c_i \), and \( d^a_i \) from Eq. 4. Figure 10a illustrates an output distribution for an answerable query. CDA successfully attends to the relevant contextual knowledge “Christina” and amplifies the probability from 96.53% to 98.24%, effectively mitigating the influence of the parametric knowledge. Furthermore, Figure 10b illustrates how CDA handles unanswerable queries. While both knowledge erroneously generates the token “Andie”, CDA successfully shifts the distribution towards abstention, preventing hallucinations.

![Top-5 probabilities and their corresponding tokens for parametric, contextual, and CDA distribution. CDA (a) amplifies the relevant knowledge for answerable queries while (b) shifting the distribution to abstain from unanswerable queries.](page_1012_563_382_246.png)

Figure 10: Top-5 probabilities and their corresponding tokens for parametric, contextual, and CDA distribution. CDA (a) amplifies the relevant knowledge for answerable queries while (b) shifting the distribution to abstain from unanswerable queries.

4Wikipedia dump from Dec. 2018.

7 Conclusion

This work addresses the challenge of generating reliable responses leveraging parametric and contextual knowledge when available, while abstaining when both are absent. To evaluate these scenarios, we construct a testbed based on the model’s approximated knowledge. Furthermore, we present Contrastive Decoding with Abstention (CDA), a novel, training-free decoding method that incorporates abstention in the generation process. CDA quantifies the relevance of both knowledge and dynamically attends to the relevant one. When no relevant knowledge is available, CDA guides the model to abstain. Through extensive experiments, CDA exhibits accurate generation when relevant knowledge is available and abstention otherwise, reducing the risks of hallucination.
Limitations

Our study acknowledges a few limitations that present opportunities for future research.

Computation Cost  Contrastive decoding inherently involves comparing multiple outputs, inevitably increasing the overall cost. CDA also requires additional computations, costing roughly double that of greedy decoding. However, CDA enables reliable generation through abstention, a capability that is enhanced through this additional computation. We believe that the increased cost is a reasonable trade-off for achieving a more dependable and safe model. Nonetheless, reducing the cost is an essential factor, which will be addressed as a primary objective in future work. A detailed analysis of the inference cost is provided in Appendix D.

Limitations in Task Scope  Our study primarily focuses on single-context scenarios, providing a relatively clear distinction between the presence and absence of knowledge, facilitating precise analysis. However, extending the scope to multi-context scenarios would be an important direction for future work. Additionally, this work focuses on short-form QA tasks, which are relatively easier to assess the knowledge usage. However, expanding the task to reasoning-intensive, long-form generation tasks would be a meaningful advancement.

Advanced Abstention  The current work primarily focuses on the model’s ability to simply express abstention, which is often lacking in user-friendliness. Future research could explore incorporating reasoning capabilities to explain the rationale behind abstention decisions.

Acknowledgement

This work was partly supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University), No.RS-2020-II201373, Artificial Intelligence Graduate School Program (Hanyang University), NO.RS-2021-II212068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)] and the Mid-Career Bridging Program through Seoul National University.

References

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadehgan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. 2021. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L Edelman. 2024. Distinguishing the knowable from the unknowable with language models. arXiv preprint arXiv:2402.03563.

Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhui Chen, and William Yang Wang. 2024. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6416–6432, Bangkok, Thailand. Association for Computational Linguistics.

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbutz, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric J Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Chenyu Zhang, Ruiqi Zhong, Sean O’Higeartigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwani, Yoshua Bengio, Danqi Chen, Philip Torr, Samuel Albanie, Tegan Maharaj, Jakob Nicolaus Foerster, Florian Tramèr, He He, Atossa Kasirzadeh, Yejin Choi, and David Krueger. 2024. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Research. Survey Certification, Expert Certification.

Stefan Buttcher, Charles LA Clarke, and Gordon V Cormack. 2016. Information retrieval: Implementing and evaluating search engines. Mit Press.

Hung-Ting Chen, Michael Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2292–2307, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
Roi Cohen, Konstantin Dobler, Eden Biran, and Gerard de Melo. 2024. I don’t know: Explicit modeling of uncertainty with an [IDK] token. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge-base of language models. arXiv preprint arXiv:2301.12810.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc.

Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kaiklhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050–5063, Bangkok, Thailand. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Romina Etezadi and Mehrnoush Shamsfard. 2023. The state of the art in open domain complex question answering: a survey. Applied Intelligence, 53(4):4124–4144.

Ruitao Feng, Xudong Hong, Mayank Jobanputra, Mattes Warning, and Vera Demberg. 2024a. Retrieval-augmented modular prompt tuning for low-resource data-to-text generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14053–14062, Torino, Italia. ELRA and ICCL.

Shangbin Feng, Chan Young Park, Yuhai Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11737–11762, Toronto, Canada. Association for Computational Linguistics.

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024b. Don’t hallucinate, abstain: Identifying LLM knowledge gaps via multi-LLM collaboration. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14664–14690, Bangkok, Thailand. Association for Computational Linguistics.

Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1–13, Hong Kong, China. Association for Computational Linguistics.

Kang He, Yinghan Long, and Kaushik Roy. 2024. Prompt-based bias calibration for better zero/few-shot learning of language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12673–12691, Miami, Florida, USA. Association for Computational Linguistics.

Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2025. Look before you leap: An exploratory study of uncertainty analysis for large language models. IEEE Transactions on Software Engineering, pages 1–18.

Gautier Izard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Preprint, arXiv:2112.09118.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou. 2024. On large language models’ hallucination with regard to known facts. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1041–1053, Mexico City, Mexico. Association for Computational Linguistics.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.

Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684–5696. Online. Association for Computational Linguistics.

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.

Vladimir Karpukhin, Barlas Ouz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.

Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. Realtime QA: What’s the answer right now? In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Hyuhng Joon Kim, Youna Kim, Cheonbok Park, Junyeob Kim, Choonghyun Park, Kang Min Yoo, Sang-goo Lee, and Taeuk Kim. 2024a. Aligning language models to explicitly handle ambiguity. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1989–2007, Miami, Florida, USA. Association for Computational Linguistics.

Minsu Kim and James Thorne. 2024. Epistemology of language models: Do language models have holistic knowledge? In Findings of the Association for Computational Linguistics: ACL 2024, pages 12644–12669, Bangkok, Thailand. Association for Computational Linguistics.

Youna Kim, Hyuhng Joon Kim, Cheonbok Park, Choonghyun Park, Hyunsoo Cho, Junyeob Kim, Kang Min Yoo, Sang-goo Lee, and Taeuk Kim. 2024b. Adaptive contrastive decoding in retrieval-augmented generation for handling noisy contexts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2421–2431, Miami, Florida, USA. Association for Computational Linguistics.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenneth Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liška, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2024. Mind the gap: assessing temporal generalization in neural language models. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA. Curran Associates Inc.

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12286–12312, Toronto, Canada. Association for Computational Linguistics.

Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6691–6706, Online. Association for Computational Linguistics.

Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.

Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online. Association for Computational Linguistics.

Sean O’Brien and Mike Lewis. 2023. Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.09117.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc.

Zexuan Qiu, Zijing Ou, Bin Wu, Jingjing Li, Aiwei Liu, and Irwin King. 2024. Entropy-based decoding for retrieval-augmented large language models. arXiv preprint arXiv:2406.17519.

Mahimai Raja, E Yuvaraaian, et al. 2024. A rag-based medical assistant especially for infectious diseases. In 2024 International Conference on Inventive Computation Technologies (ICICT), pages 1128–1133. IEEE.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association for Computational Linguistics.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36, pages 68539–68551. Curran Associates, Inc.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024a. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations.

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024b. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 783–791, Mexico City, Mexico. Association for Computational Linguistics.

Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khyathi Chandu. 2024. Selective “selective prediction”: Reducing unnecessary abstention in vision-language reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12935–12948, Bangkok, Thailand. Association for Computational Linguistics.

Elior Sulem, Jamaal Hay, and Dan Roth. 2022. Yes, no or IDK: The challenge of unanswerable yes/no questions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1075–1085, Seattle, United States. Association for Computational Linguistics.

Meiqi Sun, Wilson Yan, Pieter Abbeel, and Igor Moravcik. 2022. Quantifying uncertainty in foundation models via ensembles. In NeurIPS 2022 Workshop on Robustness in Sequence Modeling.

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2024. Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110, Bangkok, Thailand. Association for Computational Linguistics.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwijal Bhargava, Shruti
Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. 2024. The art of defending: A systematic evaluation and analysis of LLM defense strategies on safety and over-defensiveness. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13111–13128, Bangkok, Thailand. Association for Computational Linguistics.

Jonas Waldendorf, Barry Haddow, and Alexandra Birch. 2024. Contrastive decoding reduces hallucinations in large multilingual machine translation models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2539, St. Julian’s, Malta. Association for Computational Linguistics.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. 2024. Know your limits: A survey of abstenion in large language models. Preprint, arXiv:2407.18418.

Hongshen Xu, Zichen Zhu, Sitou Zhang, Da Ma, Shuai Fan, Lu Chen, and Kai Yu. 2024. Rejection improves reliability: Training LLMs to refuse unknown questions using RL from knowledge feedback. In First Conference on Language Modeling.

Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2024. Alignment for honesty. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.

Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. ToolSword: Unveiling safety issues of large language models in tool learning across three stages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2181–2211, Bangkok, Thailand. Association for Computational Linguistics.

Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Chang-sung Kang, Hongbo Deng, Chikashi Nobata, Jean Marc Langlois, and Yi Chang. 2016. Ranking relevance in yahoo search. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 323–332, New York, NY, USA. Association for Computing Machinery.

Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024a. R-tuning: Instructing large language models to say ‘I don’t know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7113–7139, Mexico City, Mexico. Association for Computational Linguistics.

Lingxi Zhang, Jing Zhang, Xirui Ke, Haoyang Li, Xinmei Huang, Zhonghui Shao, Shulin Cao, and Xin Lv. 2023. A survey on complex factual question answering. AI Open, 4:1–12.

Zhixin Zhang, Leiqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024b. Safety-Bench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15537–15553, Bangkok, Thailand. Association for Computational Linguistics.

Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah Smith. 2024a. Set the clock: Temporal alignment of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15015–15040, Bangkok, Thailand. Association for Computational Linguistics.

Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning.

Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. 2024b. Enhancing contextual understanding in large language models through contrastive decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4225–4237, Mexico City, Mexico. Association for Computational Linguistics.

Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhaq Chen. 2023. Context-faithful prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544–14556, Singapore. Association for Computational Linguistics.

A Details of Testbed Setting

In this section, we provide the details of the testbed setup process.
A.1 Dataset Details

Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), HotpotQA (Yang et al., 2018) are open-domain question answering datasets, structured to include a question, a short form answer, and a pre-defined context. The answer span, including the ground-truth answer, can be found within the context. Specifically, NQ is composed of information-seeking queries from the Google search engine, and the contexts are Wikipedia pages retrieved by Crowdworkers. HotpotQA is a multi-hop reasoning dataset comprising two entity-linked paragraphs from Wikipedia and questions collected from crowdworkers. Unlike its original setting, which includes distractor paragraphs, we use the processed version from MRQA (Fisch et al., 2019), where the distractors have been removed. TriviaQA utilizes question-and-answer pairs collected from trivia and quiz-league websites. We use the web version of TriviaQA from MRQA.

A.2 Data Preprocessing

The dataset consists of a query \( x_i \), a ground-truth answer \( y_i \), and a pre-defined context \( c_i \). To keep both inputs and outputs concise, we only utilize samples with lengths of the \( x_i \) and \( y_i \) limited to 50 and 10 words, respectively. In cases where \( x_i \) appears multiple times within the context \( c_i \), we extract multiple corresponding spans (\( c_i^1, ..., c_i^k \)) for the same query \( x_i \). Each resulting triplets \( \{(x_i, c_i^j, y_i), ..., (x_i, c_i^k, y_i)\} \) are included in the dataset \( \mathcal{D}_{\text{init}} \).

Relevant Knowledge Estimation Although the pre-defined context always contains the answer span, further validation is necessary since it is split into 100-word spans, which may not contain sufficient information to answer the query. To address this, we measure the sampling consistency by applying a temperature of 1.0 and generating \( n = 10 \) samples for each query. Each generated sample is compared with the ground-truth answer \( y \) to determine its correctness. The consistency rate is compared with a pre-defined threshold \( \eta \) set to 0.7. The knowledge is considered relevant if the consistency rate exceeds the threshold value. In other words, the model is deemed to have relevant knowledge for the input \( x \) if at least eight samples are correct out of ten generations.

<table>
  <tr>
    <th>Backbone</th>
    <th>Seed</th>
    <th>NQ</th>
    <th>HotpotQA</th>
    <th>TriviaQA</th>
  </tr>
  <tr>
    <td rowspan="3">LLAMA3 8B INSTRUCT</td>
    <td>1</td>
    <td>2,338</td>
    <td>3,614</td>
    <td>7,524</td>
  </tr>
  <tr>
    <td>2</td>
    <td>1,912</td>
    <td>3,400</td>
    <td>9,924</td>
  </tr>
  <tr>
    <td>3</td>
    <td>2,432</td>
    <td>3,462</td>
    <td>8,122</td>
  </tr>
  <tr>
    <td rowspan="3">LLAMA2 7B CHAT</td>
    <td>1</td>
    <td>1,462</td>
    <td>2,748</td>
    <td>13,702</td>
  </tr>
  <tr>
    <td>2</td>
    <td>1,758</td>
    <td>3,538</td>
    <td>14,034</td>
  </tr>
  <tr>
    <td>3</td>
    <td>2,088</td>
    <td>3,560</td>
    <td>14,648</td>
  </tr>
  <tr>
    <td rowspan="3">LLAMA2 13B CHAT</td>
    <td>1</td>
    <td>2,150</td>
    <td>3,508</td>
    <td>9,806</td>
  </tr>
  <tr>
    <td>2</td>
    <td>2,108</td>
    <td>3,978</td>
    <td>10,274</td>
  </tr>
  <tr>
    <td>3</td>
    <td>2,460</td>
    <td>3,912</td>
    <td>11,284</td>
  </tr>
  <tr>
    <td rowspan="3">MISTRAL 7B INSTRUCT</td>
    <td>1</td>
    <td>764</td>
    <td>1,110</td>
    <td>9,364</td>
  </tr>
  <tr>
    <td>2</td>
    <td>520</td>
    <td>1,624</td>
    <td>11,578</td>
  </tr>
  <tr>
    <td>3</td>
    <td>894</td>
    <td>1,438</td>
    <td>11,252</td>
  </tr>
</table>

Table 4: Number of samples for each dataset constructed for the testbed.

Irrelevant Knowledge Estimation For irrelevant context selection, we utilize SBERT (Reimers and Gurevych, 2019) embedding to measure the cosine similarity between the training set contexts and the relevant context. The context with the highest cosine similarity is selected as the irrelevant context candidate. This process is necessary to avoid selecting contexts that are overly unrelated to the query. Finally, we select contexts with a consistency rate of \( r = 0 \) to ensure they do not provide any unintended information or hints.

Final Dataset Construction To ensure a balanced dataset, we randomly sample equal number of sample with relevant (i.e., \( p_i = 1 \)) and irrelevant (i.e., \( p_i = 0 \)) parametric knowledge, matching the size of the smaller set. The final number of selected samples for each model is presented in Table 4. Note that the number of selected data points varies across models, reflecting the differences in the possessed knowledge.

A.3 Evaluation Details

For evaluation, we utilize the final dataset \( D = \{(x_i, c_i^+, c_i^-, y_i, p_i)\}_{i=1}^N \). Specifically, for \( x_i \in D \), we evaluate on both input pairs with relevant (\( x_i, c_i^+ \)) and irrelevant (\( x_i, c_i^- \)) contexts. Only the input (\( x_i, c_i^- \)) where \( p_i = 0 \) is deemed unanswerable. All other inputs are considered answerable by leveraging either parametric or contextual knowledge. For example, the input (\( x_i, c_i^+ \)) where \( p_i = 0 \) is answerable since the model can utilize the relevant context \( c_i^+ \), even though the model does not pose relevant parametric knowledge (i.e., \( p_i = 0 \)). (\( x_i, c_i^- \)) where \( p_i = 1 \) is also answerable since \( x_i \) can be answered by utilizing the model’s parametric knowledge (i.e., \( p_i = 1 \)). All the exper-
iments are averaged over three different random seeds. The full results of LLAMA3 8B INSTRUCT, LLAMA2 7B CHAT, LLAMA2 13B CHAT, and MISTRAL 7B INSTRUCT are reported in Table 10, Table 11, Table 12, and Table 13, respectively.

B Experiential Setting Details

In this section, we describe implementation details for the experiment settings.

B.1 Implementation Details of CDA(-m)

CDA(-M) utilize the templates \( \mathcal{T}_p(\cdot) \), \( \mathcal{T}_c(\cdot) \), and \( \mathcal{T}_a(\cdot) \) from Table 3. For the calibration, we set the query placeholder token \( \bar{x} \) to "[QUESTION]" and the context placeholder token \( \bar{c} \) to "[CONTEXT]". The output distribution \( \vec{d}_t^p \) and \( \vec{d}_t^c \) for null prompts are computed as follows.

\[
\begin{align*}
\vec{d}_t^p &= \operatorname{logit}_\theta(y_t \mid \mathcal{T}_p(\bar{x}, y_{<t})), \\
\vec{d}_t^c &= \operatorname{logit}_\theta(y_t \mid \mathcal{T}_c(\bar{c}, \bar{x}, y_{<t})),
\end{align*}
\]

For CDA-M, momentum is applied to each weight as follows:

\[
\begin{align*}
w_t^c &\leftarrow \alpha\ w_{t-1}^c + (1-\alpha)\ w_t^c, \\
w_t^p &\leftarrow \alpha\ w_{t-1}^p + (1-\alpha)\ w_t^p, \\
w_t^a &\leftarrow \alpha\ w_{t-1}^a + (1-\alpha)\ w_t^a,
\end{align*}
\]

where the momentum weight \( \alpha \) is set to 0.7.

B.2 Evaluation Details

The model is provided with a 2-shot demonstration from the training set and evaluated with greedy generation. For answerable queries, the prediction is considered correct if it contains the ground-truth answer (Mallen et al., 2023; Schick et al., 2023). Furthermore, the model is expected to appropriately abstain from generating hallucinations for unanswerable queries. Since there are various ways to abstain, we determine proper abstention by detecting the presence of any pre-defined abstention phrases in the model’s output (Amayuelas et al., 2024; Kim et al., 2024a). The pre-defined phrases are the following: [unknown answer, answer is unknown, unable to answer, no answer, cannot answer, don’t know, do not know]

B.3 Baselines

This section provides details of some baselines.

<table>
  <tr>
    <th>Answer the following question. Given the context, question, and the answer, is the question known or unknown? Answer only known or unknown.</th>
  </tr>
  <tr>
    <td>Context: <code>context</code><br>Question: <code>question</code><br>Answer: <code>initial generation</code><br>Is the question known or unknown? Answer only known or unknown.<br>Known or Unknown:</td>
  </tr>
</table>

Table 5: Verification template \( \mathcal{T}_v(\cdot) \) for SELF-ASK. With the generated answer, original question, and context, the model is prompted to verify whether the question is (un)known.

SELF-ASK uses \( \mathcal{T}_c(\cdot) \) from Table 3b for the initial generation \( \hat{y} \). The initial generation is "self-asked" to the identical model and is verified using the template \( \mathcal{T}_v(c, x, \hat{y}) \) from Table 5. The prediction is abstained if the model generates “unknown” as the verification result.

CAD computes the output distribution by amplifying the influence of the contextual knowledge as \( d_t^o = d_t^c + w_t^c\ (d_t^c - d_t^p) \). A fixed weight \( w_t^c \) controls the amount of contextual knowledge applied to the final output distribution. Following the original work (Shi et al., 2024b), we evaluate the performance with \( w_t^c \) set to 0.5 and 1.0 and report the best result.

ENTROPY measures the entropy of the generated tokens when prompted with \( \mathcal{T}_c(\cdot) \). Specifically, for a prediction \( \hat{y} = \{\hat{y}^1, ..., \hat{y}^L\} \) with L-tokens, we leverage the output distribution of each token (i.e., \( d^1, ..., d^L \)) to measure the token entropy (i.e., \( \mathcal{H}^1, ..., \mathcal{H}^L \)) following Eq. 5. We measure four different variants: first-token (\( \mathcal{H}^1 \)), average (\( \frac{1}{L} \sum_{l=1}^L \mathcal{H}^l \)), maximum (\( \max(\mathcal{H}^1, ..., \mathcal{H}^L) \)), and minimum (\( \min(\mathcal{H}^1, ..., \mathcal{H}^L) \)) entropy.

We compare the entropy measure with a threshold value to perform abstention. Specifically, if the entropy measure exceeds the threshold value, the prediction is considered uncertain and is abstained. We utilize the threshold value, which demonstrates the best Reliability Score (RS) from the training set. We report the first-token entropy, which yields the best performance as the main result, and the results of other variants are reported only in the Appendix.

FSB utilizes \( \mathcal{T}_p(\cdot) \), \( \mathcal{T}_c(\cdot) \), and \( \mathcal{T}_a(\cdot) \) to measure \( \mathcal{H}_p^o, \mathcal{H}_c^o, \) and \( \mathcal{H}_a^o \), respectively. We compare the entropy values at the first decoding step and select the prompting method with the lowest entropy (highest confidence). For example, if \( \mathcal{H}_1^a \) displays the highest confidence, the model continues to generate the same prediction as ABSTAIN.
Figure 11: F1ans and F1abs according to different \( \alpha \) values. Applying momentum significantly improves F1ans.

<table>
  <tr>
    <th>Dataset</th>
    <th>F1abs</th>
    <th>F1ans</th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
  </tr>
  <tr>
    <td>NQ</td>
    <td>59.35 (3.43)</td>
    <td>52.06 (1.18)</td>
    <td>47.89 (4.43)</td>
    <td>38.04 (3.87)</td>
    <td>56.79 (2.78)</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>66.94 (0.84)</td>
    <td>56.39 (0.21)</td>
    <td>56.42 (1.33)</td>
    <td>45.43 (1.29)</td>
    <td>63.55 (0.69)</td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>67.56 (1.27)</td>
    <td>57.24 (0.63)</td>
    <td>56.46 (1.19)</td>
    <td>45.32 (1.14)</td>
    <td>64.17 (0.99)</td>
  </tr>
</table>

Table 6: Average and standard deviation (in parentheses) of CDA without calibration.

C Details on Ablation Experiments

C.1 Ablation on Momentum Weight

Figure 11 demonstrates the results of F1ans and F1abs with \( \alpha \) values from 0.0 to 1.0 in NQ, HotpotQA, and TriviaQA. We can observe a consistent pattern across the datasets. The results indicate that incorporating momentum enhances the performance of F1ans while F1abs remains consistent.

C.2 Results of Calibration

Full ablation results regarding the effect of calibration are reported in Table 6. All the experiments are averaged over three different random seeds.

C.3 Implementation Details of Training-based Methods

In this section, we describe the implementation details of training-based methods. Besides instruction-tuning, we also utilize external verifier (Cobbe et al., 2021; Cohen et al., 2023) for comparison. We report the average performance over three different random seeds. The full results are displayed in Table 7.

<table>
  <tr>
    <th rowspan="2">Source</th>
    <th rowspan="2">Target</th>
    <th colspan="2">F1abs</th>
    <th colspan="2">F1ans</th>
    <th colspan="2">RS</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="3">NQ</td>
    <td>NQ</td>
    <td>64.86 (1.49)</td>
    <td>46.56 (3.17)</td>
    <td>58.03 (1.06)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>66.62 (1.90)</td>
    <td>52.36 (2.61)</td>
    <td>58.50 (1.16)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>65.07 (2.09)</td>
    <td>53.52 (3.07)</td>
    <td>57.39 (1.17)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">HotpotQA</td>
    <td>NQ</td>
    <td>61.46 (3.13)</td>
    <td>38.42 (8.06)</td>
    <td>54.60 (2.15)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>68.41 (0.22)</td>
    <td>56.92 (1.32)</td>
    <td>60.80 (1.12)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>60.09 (2.37)</td>
    <td>38.53 (1.19)</td>
    <td>53.28 (1.04)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">TriviaQA</td>
    <td>NQ</td>
    <td>63.30 (0.74)</td>
    <td>45.10 (5.07)</td>
    <td>55.55 (0.27)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>66.97 (0.19)</td>
    <td>54.88 (0.98)</td>
    <td>59.24 (0.26)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>65.31 (1.14)</td>
    <td>52.63 (2.56)</td>
    <td>58.13 (0.43)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(a) Results of external verifier.

<table>
  <tr>
    <th rowspan="2">Source</th>
    <th rowspan="2">Target</th>
    <th colspan="2">F1abs</th>
    <th colspan="2">F1ans</th>
    <th colspan="2">RS</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="3">NQ</td>
    <td>NQ</td>
    <td>66.37 (0.55)</td>
    <td>43.87 (2.02)</td>
    <td>61.12 (0.46)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>64.43 (0.42)</td>
    <td>48.79 (3.88)</td>
    <td>59.65 (0.42)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>67.31 (1.82)</td>
    <td>49.00 (4.74)</td>
    <td>62.14 (1.43)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">HotpotQA</td>
    <td>NQ</td>
    <td>64.86 (2.38)</td>
    <td>42.73 (1.49)</td>
    <td>59.80 (1.92)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>74.06 (0.76)</td>
    <td>57.57 (0.73)</td>
    <td>68.76 (0.81)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>65.26 (4.92)</td>
    <td>50.69 (2.06)</td>
    <td>61.17 (4.10)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">TriviaQA</td>
    <td>NQ</td>
    <td>65.15 (1.66)</td>
    <td>46.37 (4.00)</td>
    <td>59.70 (1.10)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>66.68 (0.13)</td>
    <td>49.82 (3.52)</td>
    <td>60.58 (0.25)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>67.00 (1.73)</td>
    <td>53.68 (2.92)</td>
    <td>61.84 (0.70)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(b) Results of instruction-tuning.

Table 7: Average and standard deviation (in parentheses) of training-based methods over three different random seeds. In-domain results are highlighted in blue.

External Verifier  We utilize RoBERTa-base as an external verifier. The overall training process is as follows. First, we prompt the inference model (e.g., LLAMA3 8B INSTRUCT) with the prompt \( T_c(c, x) \) and generate a prediction \( \hat{y} \). Then, we measure the correctness of \( \hat{y} \) by comparing it with the ground-truth answer \( y \). The label for the verifier \( \bar{y} \) is assigned based on the correctness of \( \hat{y} \). Specifically, we assign \( \bar{y} = 1 \) when \( \hat{y} \) is correct and \( \bar{y} = 0 \) when \( \hat{y} \) is incorrect. We feed \( T_c(c, x, \hat{y}) \) into the verifier and pass the [CLS] token through a single MLP layer for the final prediction. The verifier is trained to predict \( \bar{y} \) given \( T_c(c, x, \hat{y}) \). During inference, we first generate \( \hat{y} \) from the inference model. Then, we pass \( T_c(c, x, \hat{y}) \) to the verifier to predict the correctness of the prediction \( \hat{y} \). Samples classified as incorrect by the verifier are abstained.

Instruction-tuning  For instruction-tuning, we first re-label the training data based on the model’s knowledge. Specifically, we follow the same knowledge estimation process from Section 3 and estimate the parametric and contextual knowledge
of the given training sample. If the model possesses at least one relevant knowledge for the given \( x \), it is labeled with the ground-truth answer \( y \). However, if the model does not have any relevant knowledge, \( x \) is labeled with a pre-defined abstention response \( y_{abs} \) (e.g., “unknown”). The model is then trained to generate the label given \( \mathcal{T}_c(c, x) \). For evaluation, we utilize the instruction-tuned model for the prediction with \( \mathcal{T}_c(c, x) \) as the input.

Training Details For instruction-tuning, we employ QLoRA (Dettmers et al., 2023) from Huggingface PEFT library (Mangrulkar et al., 2022) with \( r = 4 \) and \( alpha = 16 \) for efficient training. We select the model with the best Reliability Score (RS) performance in the validation set from the learning rates [1e-3, 5e-4, 1e-4, 5e-5, 1e-5, 5e-6, 1e-6] and training epochs [2, 3, 4, 5]. All results are averaged over three different random seeds. Figure 7 displays the full results of the external verifier and instruction-tuning in both in-domain (IND) and out-of-domain (OOD) scenarios. Overall, instruction-tuning displays better performance than the external verifier in IND scenarios. However, both methods exhibit a notable degradation in OOD inputs, limiting their generalizability.

C.4 Evaluation on RAG Setting

Table 8 presents the Reliability Score (RS) results across all the datasets and backbones. Results of ENTROPY variants (average, maximum, minimum entropy) are also reported. CDA(-M) consistently outperform the baselines, underscoring their effectiveness in practical scenarios.

D Computation Cost Analysis

Contrastive decoding enables the model to utilize various knowledge and abilities by leveraging multiple output distributions. However, the process incurs additional costs, which is also an inherent limitation of CDA. In this section, we analyze the computation cost of CDA in detail.

Let the lengths of the template, context, and query be \( L_t, L_c, \) and \( L_q \), respectively. The computational cost for a single inference (e.g., CONTEXT) is proportional to the square of the total input length \( O((L_t + L_c + L_q)^2) \). CDA performs five inferences of contextual prompting \( O((L_t + L_c + L_q)^2) \), parametric prompting \( O((L_t + L_q)^2) \), abstention prompting \( O((L_t + L_c + L_q)^2) \), contextual null prompting \( O((L_t + 2)^2) \), and parametric null prompting \( O((L_t + 1)^2) \). While this may appear computationally heavy, in practice, the template and the query are relatively short, while the context is typically long, making the overall computation

<table>
  <tr>
    <th>Method</th>
    <th>NQ</th>
    <th>HopotQA</th>
    <th>TriviaQA</th>
  </tr>
  <tr>
    <th colspan="4">LLAMA3 8B INSTRUCT</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>33.35</td>
    <td>31.00</td>
    <td>65.30</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>28.86</td>
    <td>28.24</td>
    <td>59.32</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>38.40</td>
    <td>34.05</td>
    <td>73.91</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>50.74</td>
    <td>50.71</td>
    <td>76.66</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>45.67</td>
    <td>39.61</td>
    <td>64.91</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td>50.84</td>
    <td>50.24</td>
    <td>76.20</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td>48.46</td>
    <td>47.67</td>
    <td>72.12</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td>49.40</td>
    <td>48.34</td>
    <td>74.24</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td>44.76</td>
    <td>43.58</td>
    <td>68.61</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>53.66</td>
    <td>51.07</td>
    <td>79.14</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>51.80</td>
    <td>50.35</td>
    <td>75.58</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td><b>54.33</b></td>
    <td><b>51.77</b></td>
    <td><b>80.62</b></td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td><b>54.32</b></td>
    <td><b>51.80</b></td>
    <td><b>80.67</b></td>
  </tr>
  <tr>
    <th colspan="4">LLAMA2 7B CHAT</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>30.11</td>
    <td>28.28</td>
    <td>61.01</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>27.20</td>
    <td>25.86</td>
    <td>54.44</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>33.02</td>
    <td>30.04</td>
    <td>67.35</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>22.06</td>
    <td>20.02</td>
    <td>70.70</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>31.72</td>
    <td>33.99</td>
    <td>63.12</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td>41.94</td>
    <td>44.71</td>
    <td>71.87</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td>42.88</td>
    <td>45.01</td>
    <td>70.12</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td>41.36</td>
    <td><b>45.69</b></td>
    <td>71.88</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td>41.30</td>
    <td>41.65</td>
    <td>65.67</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>37.36</td>
    <td>39.58</td>
    <td>73.13</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>33.56</td>
    <td>35.05</td>
    <td>70.55</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td><b>47.09</b></td>
    <td><b>45.07</b></td>
    <td><b>73.18</b></td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td><b>46.40</b></td>
    <td><b>45.02</b></td>
    <td><b>73.90</b></td>
  </tr>
  <tr>
    <th colspan="4">LLAMA2 13B CHAT</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>31.15</td>
    <td>30.57</td>
    <td>66.81</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>28.40</td>
    <td>27.59</td>
    <td>61.80</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>35.65</td>
    <td>32.73</td>
    <td>72.09</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>41.54</td>
    <td>24.54</td>
    <td>47.11</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>34.15</td>
    <td>33.68</td>
    <td>67.73</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td>48.28</td>
    <td>44.31</td>
    <td>74.56</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td>47.65</td>
    <td>43.31</td>
    <td>73.88</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td>48.61</td>
    <td>43.93</td>
    <td>75.11</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td>43.73</td>
    <td>39.88</td>
    <td>70.18</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>49.88</td>
    <td><b>45.28</b></td>
    <td>72.79</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>48.30</td>
    <td>41.62</td>
    <td>65.78</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td>51.42</td>
    <td>44.06</td>
    <td>77.89</td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td><b>51.95</b></td>
    <td><b>44.73</b></td>
    <td><b>78.74</b></td>
  </tr>
  <tr>
    <th colspan="4">MISTRAL 7B INSTRUCT</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>30.68</td>
    <td>29.00</td>
    <td>62.40</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>28.60</td>
    <td>26.08</td>
    <td>56.09</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>32.42</td>
    <td>28.91</td>
    <td>66.24</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>49.11</td>
    <td>41.99</td>
    <td>73.00</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>39.34</td>
    <td>38.78</td>
    <td>67.94</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td>48.65</td>
    <td>46.82</td>
    <td>71.69</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td>46.84</td>
    <td>46.69</td>
    <td>71.54</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td>48.38</td>
    <td>47.06</td>
    <td>71.57</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td>42.58</td>
    <td>42.76</td>
    <td>66.19</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>49.05</td>
    <td><b>47.32</b></td>
    <td>70.26</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>48.42</td>
    <td>47.14</td>
    <td>70.39</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td><b>49.36</b></td>
    <td>46.94</td>
    <td>72.86</td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td><b>49.51</b></td>
    <td><b>47.13</b></td>
    <td><b>73.22</b></td>
  </tr>
</table>

Table 8: Results of Reliability Score (RS) across all the scenarios in the RAG setting. CDA(-m) consistently outperform all the baselines.
<table>
  <tr>
    <th>Dataset</th>
    <th>CONTEXT</th>
    <th>CDA</th>
    <th>CDA-M</th>
  </tr>
  <tr>
    <td>NQ</td>
    <td>258.69</td>
    <td>622.81 (\( \times 2.41 \))</td>
    <td>645.78 (\( \times 2.50 \))</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>194.87</td>
    <td>486.82 (\( \times 2.50 \))</td>
    <td>535.67 (\( \times 2.75 \))</td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>188.92</td>
    <td>482.65 (\( \times 2.55 \))</td>
    <td>461.87 (\( \times 2.44 \))</td>
  </tr>
</table>

Table 9: Total computation time (in seconds) for LLAMA3 8B INSTRUCT generating 100 samples. We can observe that the total computation time of CDA(-M) is roughly two times that of CONTEXT.

roughly \( 2 * O(L_c^2) \). Consequently, the additional cost required in CDA is **approximately twice that of a single inference**.

To validate these estimates, we conduct an experiment on LLAMA3 8B INSTRUCT using 100 randomly selected samples from all the datasets. We compare the total generation time of CONTEXT with CDA in seconds on a single RTX 3090 GPU. Table 9 demonstrates the overall inference time. Empirical results indicate that the inference time of CDA does not exceed three times that of a single inference.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">NQ</th>
    <th colspan="4">HotpotQA</th>
    <th colspan="4">TriviaQA</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td></td>
    <td>57.26 (0.04)</td>
    <td>2.18 (0.05)</td>
    <td>50.22 (0.08)</td>
    <td>49.95 (0.08)</td>
    <td>50.23 (0.08)</td>
    <td>57.15 (0.04)</td>
    <td>1.19 (0.04)</td>
    <td>49.93 (0.04)</td>
    <td>50.08 (0.05)</td>
    <td>57.29 (0.04)</td>
    <td>2.24 (0.12)</td>
    <td>50.23 (0.12)</td>
    <td>49.95 (0.17)</td>
    <td>50.23 (0.17)</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td></td>
    <td>55.34 (0.96)</td>
    <td>2.14 (0.79)</td>
    <td>48.56 (0.89)</td>
    <td>48.29 (0.89)</td>
    <td>48.56 (0.89)</td>
    <td>55.78 (0.43)</td>
    <td>1.88 (0.71)</td>
    <td>48.89 (0.38)</td>
    <td>51.99 (0.49)</td>
    <td>48.89 (0.29)</td>
    <td>55.83 (0.40)</td>
    <td>0.65 (0.36)</td>
    <td>48.87 (0.37)</td>
    <td>48.79 (0.34)</td>
    <td>48.87 (0.37)</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td></td>
    <td>71.36 (1.18)</td>
    <td>0.92 (0.35)</td>
    <td>62.50 (1.06)</td>
    <td>62.39 (1.06)</td>
    <td>62.50 (1.06)</td>
    <td>74.56 (0.57)</td>
    <td>0.57 (0.17)</td>
    <td>65.09 (0.50)</td>
    <td>65.02 (0.50)</td>
    <td>65.09 (0.50)</td>
    <td>76.79 (0.38)</td>
    <td>3.09 (1.56)</td>
    <td>67.38 (0.44)</td>
    <td>66.99 (0.26)</td>
    <td>67.39 (0.44)</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td></td>
    <td>60.22 (1.85)</td>
    <td>52.05 (1.18)</td>
    <td>48.17 (3.52)</td>
    <td>38.36 (2.94)</td>
    <td>57.27 (2.94)</td>
    <td>66.88 (5.63)</td>
    <td>56.88 (5.63)</td>
    <td>56.23 (4.51)</td>
    <td>45.21 (3.13)</td>
    <td>63.55 (1.29)</td>
    <td>67.46 (0.71)</td>
    <td>57.31 (1.35)</td>
    <td>56.21 (1.34)</td>
    <td>45.07 (1.28)</td>
    <td>64.10 (1.06)</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td></td>
    <td>57.23 (0.87)</td>
    <td>48.11 (0.35)</td>
    <td>43.91 (0.48)</td>
    <td>35.06 (0.46)</td>
    <td>53.70 (0.64)</td>
    <td>50.64 (0.78)</td>
    <td>48.63 (0.30)</td>
    <td>33.86 (0.57)</td>
    <td>26.80 (0.49)</td>
    <td>49.70 (0.51)</td>
    <td>52.40 (0.28)</td>
    <td>48.38 (0.39)</td>
    <td>36.73 (0.34)</td>
    <td>29.01 (0.06)</td>
    <td>50.61 (0.06)</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td></td>
    <td>64.06 (2.23)</td>
    <td>53.34 (1.11)</td>
    <td>55.53 (2.18)</td>
    <td>45.15 (2.00)</td>
    <td>60.90 (1.96)</td>
    <td>67.08 (5.86)</td>
    <td>54.64 (0.21)</td>
    <td>57.92 (0.89)</td>
    <td>46.94 (0.95)</td>
    <td>63.88 (0.25)</td>
    <td>66.17 (0.71)</td>
    <td>57.26 (0.36)</td>
    <td>56.50 (0.82)</td>
    <td>45.33 (0.74)</td>
    <td>63.35 (0.50)</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td></td>
    <td>54.91 (4.80)</td>
    <td>48.12 (1.44)</td>
    <td>41.79 (5.43)</td>
    <td>33.19 (4.65)</td>
    <td>52.31 (3.70)</td>
    <td>58.86 (3.14)</td>
    <td>50.94 (0.86)</td>
    <td>44.05 (4.69)</td>
    <td>35.09 (3.95)</td>
    <td>55.72 (2.54)</td>
    <td>47.00 (10.45)</td>
    <td>29.53 (2.98)</td>
    <td>23.13 (9.55)</td>
    <td>45.57 (7.60)</td>
    <td>45.57 (6.67)</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td></td>
    <td>55.80 (7.06)</td>
    <td>49.76 (2.57)</td>
    <td>43.06 (8.61)</td>
    <td>34.38 (6.92)</td>
    <td>53.67 (5.49)</td>
    <td>61.44 (3.31)</td>
    <td>52.39 (1.69)</td>
    <td>46.88 (4.10)</td>
    <td>37.29 (3.84)</td>
    <td>58.12 (2.84)</td>
    <td>33.49 (6.09)</td>
    <td>44.95 (1.31)</td>
    <td>20.42 (4.48)</td>
    <td>15.88 (3.58)</td>
    <td>39.65 (3.38)</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td></td>
    <td>54.94 (0.38)</td>
    <td>26.45 (1.03)</td>
    <td>47.21 (0.58)</td>
    <td>42.42 (0.44)</td>
    <td>48.45 (0.58)</td>
    <td>56.44 (0.47)</td>
    <td>25.65 (0.37)</td>
    <td>49.25 (0.54)</td>
    <td>44.80 (1.41)</td>
    <td>50.18 (0.25)</td>
    <td>57.42 (0.30)</td>
    <td>24.73 (5.82)</td>
    <td>50.57 (0.44)</td>
    <td>46.44 (1.29)</td>
    <td>51.28 (0.65)</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td></td>
    <td>69.27 (0.61)</td>
    <td>54.94 (0.69)</td>
    <td>59.64 (1.98)</td>
    <td>49.02 (2.08)</td>
    <td>65.09 (0.86)</td>
    <td>74.89 (0.21)</td>
    <td>58.51 (0.91)</td>
    <td>66.21 (0.96)</td>
    <td>55.05 (0.72)</td>
    <td>70.55 (0.81)</td>
    <td>77.02 (0.81)</td>
    <td>59.84 (0.96)</td>
    <td>68.55 (1.07)</td>
    <td>57.24 (1.27)</td>
    <td>72.62 (0.68)</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td></td>
    <td>63.56 (0.14)</td>
    <td>52.46 (0.19)</td>
    <td>53.99 (1.72)</td>
    <td>48.32 (1.67)</td>
    <td>60.11 (0.36)</td>
    <td>65.88 (0.71)</td>
    <td>54.67 (0.17)</td>
    <td>57.41 (0.92)</td>
    <td>46.80 (0.95)</td>
    <td>62.63 (0.60)</td>
    <td>66.67 (0.65)</td>
    <td>56.73 (0.65)</td>
    <td>58.81 (0.65)</td>
    <td>57.88 (0.79)</td>
    <td>63.87 (0.39)</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td></td>
    <td>72.06 (0.42)</td>
    <td><b>55.49 (1.46)</b></td>
    <td>62.95 (0.23)</td>
    <td>52.87 (0.30)</td>
    <td>67.51 (0.17)</td>
    <td>78.71 (1.05)</td>
    <td>62.50 (0.51)</td>
    <td>70.20 (1.07)</td>
    <td>58.36 (1.03)</td>
    <td>74.52 (0.96)</td>
    <td>80.39 (1.65)</td>
    <td><b>65.67 (1.65)</b></td>
    <td>72.35 (2.33)</td>
    <td>60.01 (2.43)</td>
    <td>76.67 (1.60)</td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td></td>
    <td><b>73.15 (0.34)</b></td>
    <td>55.47 (0.12)</td>
    <td><b>63.72 (0.33)</b></td>
    <td>53.16 (0.55)</td>
    <td><b>68.30 (0.12)</b></td>
    <td><b>79.32 (0.91)</b></td>
    <td><b>62.59 (0.52)</b></td>
    <td><b>70.64 (0.99)</b></td>
    <td><b>58.78 (0.97)</b></td>
    <td><b>74.99 (0.85)</b></td>
    <td><b>80.93 (1.63)</b></td>
    <td><b>65.66 (1.67)</b></td>
    <td><b>72.74 (2.30)</b></td>
    <td><b>60.40 (2.39)</b></td>
    <td><b>77.07 (1.59)</b></td>
  </tr>
</table>

Table 10: Average and standard deviation (in parentheses) of the results over three different random seeds in LLAMA3 8B INSTRUCT. The best result is highlighted in bold, and the second-best result is underlined.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">NQ</th>
    <th colspan="4">HotpotQA</th>
    <th colspan="4">TriviaQA</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td></td>
    <td>57.13 (0.05)</td>
    <td>0.09 (0.13)</td>
    <td>49.98 (0.04)</td>
    <td>49.97 (0.04)</td>
    <td>49.98 (0.04)</td>
    <td>57.14 (0.00)</td>
    <td>0.00 (0.00)</td>
    <td>49.98 (0.01)</td>
    <td>49.98 (0.01)</td>
    <td>57.17 (0.01)</td>
    <td>0.26 (0.01)</td>
    <td>50.02 (0.01)</td>
    <td>49.99 (0.01)</td>
    <td>50.02 (0.01)</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td></td>
    <td>54.78 (0.07)</td>
    <td>0.17 (0.12)</td>
    <td>47.94 (0.05)</td>
    <td>47.94 (0.05)</td>
    <td>47.94 (0.05)</td>
    <td>47.84 (0.04)</td>
    <td>0.47 (0.33)</td>
    <td>47.84 (0.30)</td>
    <td>55.11 (0.30)</td>
    <td>0.25 (0.13)</td>
    <td>48.22 (0.13)</td>
    <td>48.19 (0.11)</td>
    <td>48.22 (0.12)</td>
    <td>48.22 (0.12)</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td></td>
    <td>64.55 (1.38)</td>
    <td>0.15 (0.11)</td>
    <td>56.46 (1.20)</td>
    <td>56.46 (1.20)</td>
    <td>56.46 (1.20)</td>
    <td>69.27 (0.78)</td>
    <td>0.00 (0.00)</td>
    <td>60.60 (0.68)</td>
    <td>60.60 (0.68)</td>
    <td>60.60 (0.68)</td>
    <td>60.60 (0.68)</td>
    <td>60.60 (0.68)</td>
    <td>60.60 (0.68)</td>
    <td>60.60 (0.68)</td>
    <td>63.72 (0.43)</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td></td>
    <td>38.73 (8.61)</td>
    <td>45.16 (1.83)</td>
    <td>26.73 (7.70)</td>
    <td>20.54 (6.13)</td>
    <td>42.11 (5.03)</td>
    <td>47.29 (8.00)</td>
    <td>26.39 (7.77)</td>
    <td>23.15 (8.91)</td>
    <td>17.41 (5.69)</td>
    <td>19.19 (5.98)</td>
    <td>20.33 (2.03)</td>
    <td>8.41 (3.75)</td>
    <td>25.67 (3.45)</td>
    <td>56.27 (3.85)</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td></td>
    <td>56.81 (1.47)</td>
    <td>10.95 (1.13)</td>
    <td>50.17 (1.34)</td>
    <td>48.66 (1.22)</td>
    <td>58.58 (1.29)</td>
    <td>17.35 (0.29)</td>
    <td>52.19 (0.29)</td>
    <td>49.70 (0.41)</td>
    <td>52.33 (0.41)</td>
    <td>57.93 (0.41)</td>
    <td>8.97 (1.06)</td>
    <td>51.05 (0.15)</td>
    <td>49.84 (0.02)</td>
    <td>51.09 (0.02)</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td></td>
    <td>58.00 (0.17)</td>
    <td>41.09 (0.18)</td>
    <td>51.98 (0.14)</td>
    <td>44.58 (0.33)</td>
    <td>59.00 (0.67)</td>
    <td>45.11 (0.92)</td>
    <td>54.66 (0.87)</td>
    <td>55.51 (0.95)</td>
    <td>60.21 (0.10)</td>
    <td>47.84 (0.42)</td>
    <td>53.99 (0.66)</td>
    <td>45.00 (0.66)</td>
    <td>57.10 (0.66)</td>
    <td>57.10 (0.66)</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td></td>
    <td>54.30 (2.15)</td>
    <td>40.06 (0.65)</td>
    <td>46.38 (2.96)</td>
    <td>38.63 (3.03)</td>
    <td>50.07 (2.87)</td>
    <td>44.54 (2.85)</td>
    <td>39.15 (2.32)</td>
    <td>30.93 (1.73)</td>
    <td>47.99 (0.72)</td>
    <td>56.46 (0.66)</td>
    <td>44.75 (1.35)</td>
    <td>48.08 (1.43)</td>
    <td>39.48 (0.74)</td>
    <td>52.93 (0.74)</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td></td>
    <td>54.96 (1.27)</td>
    <td>40.96 (1.20)</td>
    <td>47.82 (7.10)</td>
    <td>43.34 (1.02)</td>
    <td>38.94 (1.19)</td>
    <td>42.71 (2.27)</td>
    <td>29.66 (12.32)</td>
    <td>23.15 (10.62)</td>
    <td>41.41 (7.57)</td>
    <td>56.94 (3.75)</td>
    <td>49.30 (6.61)</td>
    <td>40.49 (5.73)</td>
    <td>54.13 (5.63)</td>
    <td>54.13 (3.24)</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td></td>
    <td>38.59 (2.18)</td>
    <td>21.56 (5.37)</td>
    <td>48.25 (0.54)</td>
    <td>44.60 (1.65)</td>
    <td>48.91 (0.13)</td>
    <td>57.30 (0.31)</td>
    <td>22.53 (0.97)</td>
    <td>50.56 (0.65)</td>
    <td>46.93 (0.65)</td>
    <td>51.05 (0.65)</td>
    <td>57.01 (0.65)</td>
    <td>17.77 (3.78)</td>
    <td>50.22 (0.06)</td>
    <td>47.48 (0.70)</td>
    <td>50.49 (0.17)</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td></td>
    <td>55.04 (3.59)</td>
    <td>47.26 (0.84)</td>
    <td>43.26 (4.01)</td>
    <td>34.47 (3.60)</td>
    <td>52.20 (2.57)</td>
    <td>65.63 (2.31)</td>
    <td>53.82 (0.93)</td>
    <td>55.73 (4.11)</td>
    <td>45.37 (4.07)</td>
    <td>62.04 (2.22)</td>
    <td>69.50 (2.21)</td>
    <td>51.88 (1.57)</td>
    <td>60.41 (4.05)</td>
    <td>50.59 (4.28)</td>
    <td>64.78 (2.23)</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td></td>
    <td>48.82 (4.58)</td>
    <td>39.52 (7.69)</td>
    <td>37.55 (4.90)</td>
    <td>30.41 (4.68)</td>
    <td>45.66 (3.35)</td>
    <td>57.79 (3.06)</td>
    <td>51.11 (1.22)</td>
    <td>47.90 (4.90)</td>
    <td>38.19 (4.56)</td>
    <td>55.61 (2.64)</td>
    <td>61.21 (1.78)</td>
    <td>50.52 (1.43)</td>
    <td>53.09 (3.82)</td>
    <td>43.46 (3.89)</td>
    <td>58.18 (1.73)</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td></td>
    <td><b>66.86 (2.51)</b></td>
    <td>47.52 (1.36)</td>
    <td>59.86 (2.94)</td>
    <td>51.22 (3.55)</td>
    <td>62.38 (2.01)</td>
    <td>73.29 (3.80)</td>
    <td>42.41 (1.08)</td>
    <td>66.05 (1.08)</td>
    <td>67.82 (1.08)</td>
    <td><b>73.70 (1.08)</b></td>
    <td>51.29 (1.08)</td>
    <td><b>67.29 (1.08)</b></td>
    <td>58.43 (1.08)</td>
    <td>69.06 (1.08)</td>
    <td></td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td></td>
    <td><b>69.99 (1.40)</b></td>
    <td>47.60 (1.32)</td>
    <td><b>62.28 (2.12)</b></td>
    <td>53.62 (1.21)</td>
    <td><b>64.81 (1.23)</b></td>
    <td><b>74.09 (1.34)</b></td>
    <td><b>42.31 (1.13)</b></td>
    <td><b>67.50 (1.08)</b></td>
    <td><b>68.37 (1.08)</b></td>
    <td><b>73.47 (1.08)</b></td>
    <td><b>52.10 (1.21)</b></td>
    <td><b>67.11 (1.08)</b></td>
    <td><b>58.40 (1.08)</b></td>
    <td><b>69.00 (1.08)</b></td>
    <td></td>
  </tr>
</table>

Table 11: Average and standard deviation (in parentheses) of the results over three different random seeds in LLAMA2 7B CHAT. The best result is highlighted in bold, and the second-best result is underlined.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">NQ</th>
    <th colspan="4">HotpotQA</th>
    <th colspan="4">TriviaQA</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>(57.14, 0.37)</td>
    <td>(50.00, 0.49)</td>
    <td>(50.95, 0.50)</td>
    <td>(57.16, 0.10)</td>
    <td>(50.00, 0.49)</td>
    <td>(49.98, 0.50)</td>
    <td>(57.15, 0.47)</td>
    <td>(50.03, 0.47)</td>
    <td>(49.97, 0.50)</td>
    <td>(50.03, 0.03)</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>(45.57, 0.48)</td>
    <td>(39.90, 0.38)</td>
    <td>(39.84, 0.39)</td>
    <td>(54.60, 0.07)</td>
    <td>(47.76, 0.47)</td>
    <td>(47.75, 0.55)</td>
    <td>(58.1, 0.32)</td>
    <td>(48.84, 0.32)</td>
    <td>(48.80, 0.48)</td>
    <td>(48.84, 0.04)</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>(66.66, 0.18)</td>
    <td>(58.33, 0.58)</td>
    <td><b>58.31</b></td>
    <td>(58.33, 0.69)</td>
    <td>(80.07, 0.61)</td>
    <td><b>61.15</b></td>
    <td><b>61.14</b></td>
    <td>(61.15, 0.42)</td>
    <td>(72.49, 0.42)</td>
    <td>(63.43, 0.63)</td>
    <td>(63.38, 0.63)</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>(50.49, 0.46)</td>
    <td>(37.83, 0.29)</td>
    <td>(29.70, 0.48)</td>
    <td>(49.83, 0.57)</td>
    <td>(51.95, 0.43)</td>
    <td>(33.89, 0.55)</td>
    <td>(51.50, 0.48)</td>
    <td>(46.74, 0.35)</td>
    <td>(35.11, 0.27)</td>
    <td>(27.52, 0.47)</td>
    <td>(47.88, 0.47)</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>(59.37, 0.20)</td>
    <td>(52.98, 0.49)</td>
    <td>(49.89, 0.53)</td>
    <td>(52.31, 0.58)</td>
    <td>(42.16, 0.51)</td>
    <td>(48.99, 0.51)</td>
    <td>(58.09, 1.20)</td>
    <td>(51.53, 0.51)</td>
    <td>(49.85, 0.51)</td>
    <td>(51.59, 0.14)</td>
    <td>(51.59, 0.14)</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td>(59.23, 0.45)</td>
    <td>(42.61, 0.11)</td>
    <td>(52.75, 0.08)</td>
    <td>(44.85, 0.59)</td>
    <td>(55.19, 0.24)</td>
    <td>(45.47, 0.53)</td>
    <td>(53.01, 0.44)</td>
    <td>(55.81, 0.60)</td>
    <td>(48.31, 0.54)</td>
    <td>(53.42, 0.57)</td>
    <td>(52.27, 0.57)</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td>(56.18, 1.25)</td>
    <td>(42.85, 0.54)</td>
    <td>(47.54, 1.91)</td>
    <td>(39.22, 0.52)</td>
    <td>(52.18, 0.66)</td>
    <td>(48.18, 0.48)</td>
    <td>(39.01, 0.53)</td>
    <td>(53.37, 0.57)</td>
    <td>(57.58, 0.46)</td>
    <td>(49.57, 0.46)</td>
    <td>(54.25, 0.46)</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td>(57.69, 0.62)</td>
    <td>(41.64, 0.46)</td>
    <td>(50.81, 1.10)</td>
    <td>(42.96, 0.53)</td>
    <td>(53.59, 0.54)</td>
    <td>(44.75, 0.45)</td>
    <td>(45.49, 0.37)</td>
    <td>(57.21, 0.51)</td>
    <td>(57.75, 0.47)</td>
    <td>(47.09, 0.50)</td>
    <td>(51.88, 0.42)</td>
    <td>(54.80, 0.42)</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td>(55.30, 0.40)</td>
    <td>(24.55, 0.92)</td>
    <td>(48.25, 0.59)</td>
    <td>(44.04, 0.49)</td>
    <td>(49.05, 0.57)</td>
    <td>(15.07, 0.50)</td>
    <td>(48.39, 0.50)</td>
    <td>(50.76, 0.56)</td>
    <td>(57.44, 0.14)</td>
    <td>(49.57, 0.47)</td>
    <td>(49.78, 0.47)</td>
    <td>(49.78, 0.47)</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>(62.44, 1.68)</td>
    <td>(47.60, 0.54)</td>
    <td>(53.66, 2.21)</td>
    <td>(44.46, 1.31)</td>
    <td>(58.19, 0.48)</td>
    <td>(68.58, 0.41)</td>
    <td>(60.41, 0.50)</td>
    <td>(64.79, 0.66)</td>
    <td>(52.08, 0.56)</td>
    <td>(49.81, 0.61)</td>
    <td>(61.91, 0.18)</td>
    <td>(61.91, 0.18)</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>(57.96, 0.80)</td>
    <td>(46.34, 0.27)</td>
    <td>(49.28, 1.55)</td>
    <td>(40.27, 0.59)</td>
    <td>(54.43, 0.62)</td>
    <td>(52.13, 0.54)</td>
    <td>(54.03, 0.43)</td>
    <td>(59.16, 0.58)</td>
    <td>(58.42, 0.49)</td>
    <td>(49.72, 0.49)</td>
    <td>(39.54, 0.35)</td>
    <td>(56.61, 0.56)</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td>(68.62, 0.65)</td>
    <td>(47.14, 1.79)</td>
    <td>(61.63, 1.14)</td>
    <td>(53.16, 0.94)</td>
    <td>(63.76, 0.02)</td>
    <td>(73.69, 0.25)</td>
    <td>(56.81, 0.19)</td>
    <td><b>68.98</b></td>
    <td><b>59.53</b></td>
    <td><b>70.44</b></td>
    <td>71.08</td>
    <td>54.14</td>
    <td>64.08</td>
    <td>54.78</td>
    <td>66.55</td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td>(70.66, 0.63)</td>
    <td>(48.12, 2.06)</td>
    <td><b>63.18</b></td>
    <td>(54.46, 1.15)</td>
    <td><b>65.48</b></td>
    <td>(73.66, 0.65)</td>
    <td><b>68.92</b></td>
    <td>(59.42, 0.53)</td>
    <td><b>70.42</b></td>
    <td><b>73.12</b></td>
    <td><b>53.46</b></td>
    <td><b>65.82</b></td>
    <td><b>56.09</b></td>
    <td><b>68.52</b></td>
  </tr>
</table>

Table 12: Average and standard deviation (in parentheses) of the results over three different random seeds in LLAMA2 13B CHAT. The best result is highlighted in bold, and the second-best result is underlined.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">NQ</th>
    <th colspan="4">HotpotQA</th>
    <th colspan="4">TriviaQA</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
    <th>F1<sub>ans</sub></th>
    <th>F1<sub>abs</sub></th>
    <th>RS</th>
    <th>Acc.</th>
    <th>Cov.</th>
  </tr>
  <tr>
    <td>CONTEXT</td>
    <td>(57.04, 0.07)</td>
    <td>(0.25, 0.36)</td>
    <td>(49.88, 0.06)</td>
    <td>(49.88, 0.06)</td>
    <td>(56.99, 0.04)</td>
    <td>(49.88, 0.04)</td>
    <td>(49.87, 0.04)</td>
    <td>(49.88, 0.04)</td>
    <td>(57.13, 0.03)</td>
    <td>(50.00, 0.03)</td>
    <td>(49.96, 0.03)</td>
    <td>(50.00, 0.03)</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>(55.03, 0.25)</td>
    <td>(1.15, 0.26)</td>
    <td>(48.21, 0.24)</td>
    <td>(48.07, 0.21)</td>
    <td>(48.21, 0.46)</td>
    <td>(53.74, 0.15)</td>
    <td>(47.04, 0.40)</td>
    <td>(47.01, 0.40)</td>
    <td>(54.71, 0.07)</td>
    <td>(47.89, 0.07)</td>
    <td>(47.82, 0.47)</td>
    <td>(47.89, 0.47)</td>
  </tr>
  <tr>
    <td>ACD</td>
    <td>(68.48, 2.18)</td>
    <td>(0.00, 0.00)</td>
    <td>(59.93, 0.00)</td>
    <td>(59.93, 0.00)</td>
    <td>(72.72, 0.09)</td>
    <td>(63.64, 0.09)</td>
    <td>(63.62, 0.09)</td>
    <td>(63.64, 0.09)</td>
    <td>(75.01, 0.05)</td>
    <td>(65.62, 0.05)</td>
    <td>(65.62, 0.05)</td>
    <td>(65.62, 0.05)</td>
  </tr>
  <tr>
    <td>ABSTAIN</td>
    <td>(61.86, 8.11)</td>
    <td>(53.29, 1.45)</td>
    <td>(52.85, 9.08)</td>
    <td>(42.83, 6.00)</td>
    <td>(59.70, 2.58)</td>
    <td>(51.12, 0.83)</td>
    <td>(54.66, 3.88)</td>
    <td>(53.06, 3.89)</td>
    <td>(59.23, 1.88)</td>
    <td>(51.14, 1.94)</td>
    <td>(48.21, 2.11)</td>
    <td>(48.21, 2.11)</td>
  </tr>
  <tr>
    <td>SELF-ASK</td>
    <td>(62.52, 0.80)</td>
    <td>(49.03, 0.39)</td>
    <td>(57.49, 0.69)</td>
    <td>(47.86, 0.23)</td>
    <td>(51.33, 0.27)</td>
    <td>(43.36, 0.27)</td>
    <td>(56.36, 0.27)</td>
    <td>(48.95, 0.13)</td>
    <td>(57.70, 0.21)</td>
    <td>(62.18, 0.21)</td>
    <td>(57.19, 0.15)</td>
    <td>(57.19, 0.15)</td>
  </tr>
  <tr>
    <td>ENTROPY (first-token)</td>
    <td>(61.91, 0.37)</td>
    <td>(56.29, 1.15)</td>
    <td>(55.48, 0.66)</td>
    <td>(46.33, 0.51)</td>
    <td>(63.31, 0.06)</td>
    <td>(60.07, 1.91)</td>
    <td>(57.19, 1.51)</td>
    <td>(45.65, 1.50)</td>
    <td>(59.28, 0.71)</td>
    <td>(52.83, 0.53)</td>
    <td>(52.83, 0.53)</td>
    <td>(48.50, 0.53)</td>
  </tr>
  <tr>
    <td>ENTROPY (average)</td>
    <td>(60.03, 0.32)</td>
    <td>(50.49, 2.07)</td>
    <td>(52.81, 0.52)</td>
    <td>(43.54, 0.85)</td>
    <td>(60.98, 0.56)</td>
    <td>(55.02, 0.33)</td>
    <td>(54.19, 0.50)</td>
    <td>(45.32, 0.50)</td>
    <td>(59.28, 0.17)</td>
    <td>(52.83, 0.20)</td>
    <td>(52.83, 0.20)</td>
    <td>(48.50, 0.20)</td>
  </tr>
  <tr>
    <td>ENTROPY (max)</td>
    <td>(61.73, 0.66)</td>
    <td>(56.68, 2.22)</td>
    <td>(55.73, 1.54)</td>
    <td>(44.92, 0.86)</td>
    <td>(63.44, 0.27)</td>
    <td>(61.65, 0.26)</td>
    <td>(57.50, 0.44)</td>
    <td>(45.58, 0.44)</td>
    <td>(62.94, 0.35)</td>
    <td>(61.16, 0.26)</td>
    <td>(56.79, 0.26)</td>
    <td>(45.06, 0.26)</td>
  </tr>
  <tr>
    <td>ENTROPY (min)</td>
    <td>(56.45, 0.23)</td>
    <td>(23.04, 0.96)</td>
    <td>(49.40, 0.33)</td>
    <td>(45.56, 0.50)</td>
    <td>(50.01, 0.55)</td>
    <td>(55.86, 0.26)</td>
    <td>(29.63, 0.46)</td>
    <td>(44.67, 0.46)</td>
    <td>(50.13, 0.72)</td>
    <td>(57.01, 0.17)</td>
    <td>(17.86, 0.50)</td>
    <td>(50.63, 0.50)</td>
  </tr>
  <tr>
    <td>FSB</td>
    <td>(66.71, 3.35)</td>
    <td>(55.51, 1.18)</td>
    <td>(58.95, 3.89)</td>
    <td>(48.32, 2.90)</td>
    <td>(63.65, 1.15)</td>
    <td>(44.71, 1.17)</td>
    <td>(23.86, 1.28)</td>
    <td>(23.86, 0.49)</td>
    <td>(9.54, 0.63)</td>
    <td>(9.54, 0.63)</td>
    <td>(9.54, 0.63)</td>
    <td>(9.54, 0.63)</td>
  </tr>
  <tr>
    <td>ACD-A</td>
    <td>(61.46, 1.38)</td>
    <td>(54.02, 1.66)</td>
    <td>(55.80, 1.23)</td>
    <td>(45.58, 0.68)</td>
    <td>(59.56, 0.65)</td>
    <td>(61.02, 0.46)</td>
    <td>(50.93, 0.46)</td>
    <td>(48.10, 0.35)</td>
    <td>(58.95, 0.79)</td>
    <td>(51.62, 0.10)</td>
    <td>(56.81, 0.10)</td>
    <td>(47.48, 0.10)</td>
    <td>(59.48, 0.10)</td>
  </tr>
  <tr>
    <td>CDA</td>
    <td>(69.68, 2.26)</td>
    <td>(56.47, 1.31)</td>
    <td>(61.45, 2.31)</td>
    <td>(50.61, 2.36)</td>
    <td>(66.09, 1.93)</td>
    <td>(76.58, 1.97)</td>
    <td>(58.43, 1.68)</td>
    <td>(75.76, 1.11)</td>
    <td>(71.83, 1.62)</td>
    <td>(56.35, 1.62)</td>
    <td>(67.57, 1.62)</td>
    <td>(71.21, 1.62)</td>
    <td>(71.21, 1.62)</td>
  </tr>
  <tr>
    <td>CDA-M</td>
    <td><b>71.00</b></td>
    <td>(56.46, 1.50)</td>
    <td><b>62.30</b></td>
    <td>(51.45, 2.39)</td>
    <td><b>67.03</b></td>
    <td><b>76.94</b></td>
    <td><b>56.67</b></td>
    <td><b>69.98</b></td>
    <td><b>59.85</b></td>
    <td><b>72.49</b></td>
    <td><b>76.95</b></td>
    <td><b>57.06</b></td>
    <td><b>68.38</b></td>
    <td><b>57.84</b></td>
    <td><b>72.21</b></td>
  </tr>
</table>

Table 13: Average and standard deviation (in parentheses) of the results over three different random seeds in MISTRAL 7B INSTRUCT. The best result is highlighted in bold, and the second-best result is underlined.

Reviewer paper summaries: The paper introduces Contrastive Decoding with Abstention (CDA), a novel, training-free decoding method designed to enhance the reliability of large language models (LLMs) by enabling them to abstain from generating responses when neither parametric (pre-trained) nor contextual (input-based) knowledge is sufficient to produce accurate answers. CDA evaluates the relevance of knowledge sources through calibrated uncertainty measures, dynamically determining whether to generate an answer or abstain. The method improves upon traditional contrastive decoding (CCD) by explicitly addressing unanswerable scenarios, mitigating hallucination risks in high-stakes applications. Extensive experiments on multiple datasets and LLMs demonstrate that CDA achieves a superior balance between accurate responses and abstention.

The authors propose to add to an existing decoding method (CCD), the ability to consider abstaining from answering a question.
CCD weights the next decoded token by the logits it gets from prompting with a context (a passage) and without a context (this estimates parametric knowledge).
The authors propose to weigh in another factor of the logit given a prompt that explicitly allows the model to abstain.

The authors show large gains in comparison to other methods that use different decoding techniques or direct prompting for getting the answer.

While results look good I think the approach is not very useful as it requires 3 different inferences for each token in the response (with context, without context, with abstaining instructions). 
Also, the proposed solution is very targeted to QA setups, while LLMs can be used for many different tasks.
In addition, the novel contribution here is relatively small on top of the existing decoding technique (CCD), which is extended here.
Summary of strengths: 1. This paper offers a brand new perspective of helping LLMs abstain during the decoding time. This insight can help researchers to better understanding the role of parametric and contextual knowledge.
2. CDA is a training-free method, which make it practical and ease of integration with grey-box LLMs.
3. CDA strikes an balance between generating accurate answers and appropriately abstaining when knowledge is insufficient, outperforming baselines in reliability.

* Results are strong

* Decoding interventions are under explored and authors do nice work in that direction

* The paper overall is easy to follow
Summary of weaknesses: 1. CDA requires 5 times decoding time (context-free prompt, context-based prompt, refusal prompt and their null prompts), which makes it computationally expensive, potentially limiting its scalability.
2. It seems that CDA relies on base model's calibration to estimate uncertainty. However, modern LLMs are known to be miscalibrated [1], which may deteriarate the performance of CDA.
3. Some strong baselines are not included in the paper. Please consider adding more baselines like self-consistency or ask-for calibration [1].

[1] Feng S, Shi W, Wang Y, et al. Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration[J]. arXiv preprint arXiv:2402.00367, 2024.

* The novel contribution here is relatively small on top of the existing decoding technique (CCD), which is extended here. 

* The approach is not very useful as it requires 3 different inferences for each token in the response (with context, without context, with abstaining instructions). I do think it is worth exploring, but it is not very compelling to actual serving of LLMs. It is much more compelling to teach the model to abstain when needed through training. 

* There are many ways to measure the certainty of a model given an answer -- some probe the model for that (https://aclanthology.org/2023.emnlp-main.220.pdf) and some take entropy measurements (https://arxiv.org/pdf/2002.10689). It is lacking authors didn't use these kind of baselines for detecting when the model should abstain.
Comments, suggestions and typos: No major typos.

* not clear what figure 3b depicts
* T_p, T_c and T_a should be defined early on (this means describing them in words and referencing the tables with the actual prompts)
* how do you measure abstaining? How do you measure if the predicted answer is equal to the gold?
Overall assessment scores: 3

3
Metareview score: 4
Metareview: This paper is about giving LLMs the possibility to abstain from answering a question if it cannot retrieve the necessary knowledge.
Summary of reasons to publish: The proposed method is training-free and improves precision considerably.