Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations

## Abstract

Abstract A major barrier towards the practical deployment of large language models (LLMs) is their lack of reliability. Three situations where this is particularly apparent are correctness, hallucinations when given unanswerable questions, and safety. In all three cases, models should ideally abstain from responding, much like humans, whose ability to understand uncertainty makes us refrain from answering questions we don’t know. Inspired by analogous approaches in classification, this study explores the feasibility and efficacy of abstaining while uncertain in the context of LLMs within the domain of question-answering. We investigate two kinds of uncertainties, statistical uncertainty metrics and a distinct verbalized measure, termed as In-Dialogue Uncertainty (InDU) . Using these uncertainty measures combined with models with and without Reinforcement Learning with Human Feedback (RLHF), we show that in all three situations, abstention based on the right kind of uncertainty measure can boost the reliability of LLMs.
By sacrificing only a few highly uncertain samples we can improve correctness by 2% to 8%, avoid 50% hallucinations via correctly identifying unanswerable questions and increase safety by 70% up to 99% with almost no additional computational overhead.

## 1 Introduction

Large Language Models (LLMs) such as ChatGPT (OpenAI et al., ), Gemini (Team et al., ), LLaMA (Touvron et al., ), and Vicuna (Chiang et al., ) have shown remarkable potential across a spectrum of real-world applications. However, a major barrier towards their increased practical deployment is their lack of reliability. Language models often answer simple questions incorrectly, and make up answers when they do not know — a phenomenon known as hallucination. Moreover, they can be manipulated into providing harmful or unsafe responses when prompted suitably, which leads to safety issues.

In all three situations – when answers are incorrect, unknown, or unsafe – we ideally seek the language model to refrain from answering a question, a concept termed abstention. In machine learning, abstention from prediction has traditionally been enabled by uncertainty.
Pioneering work by Chow () proposes that classifiers should refrain from making a prediction when they are uncertain. A corpus of literature has further explored methods for implementing this notion effectively in classification tasks (El-Yaniv et al., ; Geifman and El-Yaniv, , ). In this study, we explore the feasibility and efficacy of analogous approaches – abstaining when uncertain – in the context of large language models, particularly within the domain of question-answering.

How can we effectively assess uncertainty in large language models (LLMs)? Neural network classifiers typically predict outcomes by rounding a probability vector over classes. Hence, classification uncertainty is naturally measured by how spread out these probabilities are, often quantified through entropy. Similarly, when provided with a context or question, LLMs generate a probability vector for each token, suggesting that uncertainty can be evaluated by measuring some form of entropy over these probabilities. We call this type of uncertainty statistical uncertainty. However, unlike classifiers, LLMs also exhibit a different form of uncertainty due to their text output. They may opt for responses such as "I don’t know" or incorporate hedge-words like "perhaps" or "maybe" in their answers. We refer to this form of uncertainty as In-Dialagoe Uncertainty. Given that this form closely mirrors human expressions of uncertainty, it becomes crucial to examine this particular aspect of uncertainty in LLMs particularly after Reinforcement Learning through Human Feedback (RLHF), which is known to encourage more human-like responses.

In this study, we investigate the potential of uncertainty-based abstention to enhance performance in question answering tasks. Specifically, we analyze three distinct scenarios. First, we consider correctness, wherein the model provides incorrect responses in standard question answering tasks. Second, we explore the phenomenon of hallucinations in the context of unanswerable questions, where the questions posed lack a correct answer, leading the model to fabricate responses, if it does not abstain from answering them. Finally, we explore safety concerns arising when the model is presented with questions that may lead to unsafe or harmful answers. Across all three scenarios, we examine two different types of uncertainties, namely statistical uncertainty and In-Dialogue Uncertainty, and examine the impact of RLHF on these
uncertainties.

Figure: Figure 1: Abstention based on the right form of uncertainty improves correctness, hallucinations and safety in LLMs.

Specifically, our findings are as follows:

- •
For correctness, we find that abstaining using statistical uncertainty can improve correctness across a range of question-answering tasks from 2% up to 8%. Additionally, although RLHF alters the predictive distribution of the LLMs, it still preserves the model’s uncertainty awareness and can be used to enhance correctness.
- •
For hallucination under unanswerable questions, we find that thresholding by In-Dialogue Uncertainty (InDU) serves as an effective measure of abstention, and is capable of detecting approximately 50% of unanswerable questions, thereby mitigating hallucinations.
- •
For safety, we show that abstaining using statistical uncertainty on RLHF fine-tuned models can lead to a significant improvement in safety by filtering out between 70% and 99% of unsafe responses.

Our conclusions suggest that uncertainty based abstention can prove effective in improving the reliability of language models, potentially unlocking applications in higher-stakes domains where enhanced correctness, minimal risk of hallucinations, and high safety is paramount.

## 2 Related Work

Uncertainty quantification has become a prominent field of research across various machine learning domains, including Natural Language Processing (NLP). A relatively new area of research within NLP is Natural Language Generation (NLG), which presents unique challenges for uncertainty estimation. Researchers such as Jiang et al. (); Malinin and Gales (); Kuhn et al. (); Huang et al. () have adapted probability-based methods to NLG tasks. Jiang et al. () and Malinin and Gales () compute predictive entropy by focusing on token-wise conditional probabilities, thereby measuring lexical confidence. In contrast, Kuhn et al. () propose a more sophisticated approach that estimates uncertainties using semantic likelihood probabilities associated with the meanings of text, as opposed to standard sequence likelihoods.

Another research direction involves prompting or finetuning models to express uncertainty. Kadavath et al. (), for instance, allow LLMs to assess their own uncertainty based on their response to a given prompt. This is achieved by sampling candidate answers from an LLM and feeding them back into the model to predict the uncertainty of these samples. Lin et al. () finetune a model to express discrete uncertainty values via predefined labels, while Xiong et al. () prompt the model to express its uncertainty as an integer from 0 to 100. Conversely, Tian et al. () use specifically designed prompts to make the model judge its own uncertainty and respond with a predefined list of words indicating confidence. However, these approaches entail additional computational effort and can be sensitive to distribution shifts (Kuhn et al., ). Moreover, having to explicitly ask for uncertainty estimates does not come natural to humans and is thus suboptimal for chat-based applications.
Another approach is using external tools to verify additional evidence for uncertain responses (Srinivasan et al., ). The limitation of course is computational in that they use 3 external models.

Model awareness of its knowledge boundaries has been a longstanding challenge in machine learning, and recent efforts have begun to address this issue in the field of NLG. Slobodkin et al. () delve into the behavior of LLMs when faced with unanswerable questions, finding evidence of unanswerability encoding directly in the embeddings. Complementing this, present a unique dataset composed of unanswerable questions across five diverse categories, along with their answerable counterparts. Their research demonstrates that self-knowledge can be further improved by in-context learning and instruction tuning. Amayuelas et al. () contribute to this field by curating a dataset featuring new unanswerable questions and devising a semantic evaluation method to quantify the uncertainty of the responses.
However, these studies do not take into account uncertainty measures for distinguishing between answerable and unanswerable questions, a strategy that has proven effective in other machine learning domains (Hendrycks and Gimpel, ; Liang et al., ; Liu et al., ; Hendrycks et al., ).

Beyond enhancing the helpfulness of LLMs, RLHF finetuning is also employed to induce safety into these models (Touvron et al., ; Bai et al., ). However, it is widely recognized that the guardrails established via RLHF finetuning can be bypassed through adversarial attacks, particularly by crafting specialized prompts (Ganguli et al., ; Kour et al., ; Zhu et al., ). While there has been research on identifying harmful prompts and responses using a separate model (Inan et al., ), we use uncertainty metrics and their ability to predict unsafe responses in RLHF finetuned models solely via investigating the model’s outputs.

## 3 Uncertainty Estimation

### 3.1 Statistical Uncertainty Measures

In this study, our focus is on uncertainty measures that can be directly extracted from the model without the need for specific prompting or finetuning, making them suitable for chat-based scenarios. These probabilistic measures are widely used and have been proven to be highly effective in recent research (Kuhn et al., ; Malinin and Gales, ). These methods compute a model’s confidence directly based on the probability distribution of the prediction.
There are two types of methods: single inference, where confidence is derived directly from the response by summing the negative log likelihoods, and multiple inference, where multiple responses are sampled from the model to estimate the predictive or semantic entropy of the output distribution. We employ a singular inference method, specifically negative log-likelihood, alongside two distinct multiple inference techniques: predictive entropy and semantic entropy.

#### 3.1.1 Negative Log-Likelihood

For generative LLM tasks the negative log likelihood $NLL(x)$ of a sequence $s$ given an input prompt $x$ can be calculated as follows:

$$ $NLL(x)=-\log p(s\mid x)=-\sum_{l=1}^{L}\log p(s_{l}\mid s_{<l},x)$ (1) $$

where $p(s_{l}|s_{<l})$ denotes the token-wise conditional probability for the $l^{\prime}th$ generated token $s_{l}$ and the set of previous tokens $s_{<l}$.

#### 3.1.2 Predictive Entropy

Uncertainty can be quantified using predictive entropy, defined as $H(Y\mid x)=-\int p(y\mid x)\log p(y\mid x)dy$, where $Y$ is the output random variable with realization $y$.
In practice, we need to draw discrete samples from the model’s output distribution. Unlike (Malinin and Gales, ), who use ensembles of models to obtain samples, we follow the approach of Kuhn et al. () and sample $N$ sequences from a single model to estimate the predictive entropy (PE) as follows:

$$ $PE(x)\approx-\frac{1}{N}\sum_{i=1}^{N}\log p(s_{i}\mid x)$ (2) $$

#### 3.1.3 Semantic Entropy

In contrast to calculating entropy over the likelihoods of each sequence, Kuhn et al. () introduced semantic entropy, which is computed over the likelihoods associated with meaning clusters. This process necessitates the aggregation of responses into meaning or semantic clusters, achieved through the concept of bi-directional entailment using the Deberta-large model (He et al., ). The number of semantic clusters can be interpreted as the diversity of meanings present in the output distribution. Given $C$ as the number of meaning clusters and $p(c_{j}\mid x)$ as the likelihood for the $j-th$ meaning cluster $c_{j}$, the semantic entropy (SE) can be calculated as follows:

$$ $SE(x)\approx-\frac{1}{C}\sum_{j=1}^{C}\log p(c_{j}\mid x).$ (3) $$

### 3.2 In-Dialogue Uncertainty

Hedging is a widely used strategy in human language to convey uncertainty and has been extensively studied (Ferson et al., ; ; ; Theil et al., ; Raphalen et al., ; Ulinski et al., ). While our objective is not to devise a perfect metric for measuring degrees of hedging in natural language, we are primarily interested in whether LLMs inherently utilize hedging as a means to implicitly signal uncertainty, without the need for explicit prompting. We also aim to explore whether this verbalized implicit uncertainty gives further insights into uncertainty awareness of LLMs besides statistical measures. To this end, we employ a straightforward and interpretable method that quantifies the number of hedge words in a response. A higher count of hedge words in a response indicates a higher level of In-Dialogue Uncertainty (InDU), and conversely, a lower count suggests less uncertainty. For our experiments, we employ the same hedge word list as used by .

### 3.3 Assessing the quality of uncertainty measures

To assess the quality of an uncertainty measure we gauge how well uncertainty can distinguish the correctness of a response.
In line with previous work (Kuhn et al., ; Lin et al., ; Band et al., ), we compute the area under the receiver operator characteristic (AUROC) for each uncertainty measure.
AUROC captures how well uncertainty ranks correct versus incorrect responses. We use AUROC as prior work (Kuhn et al., ) confirmed AUROC serves as a more suitable gauage of uncertainty quality compared to calibration measures such as the Brier score for free-form question answering. We compute AUROC as a gauge of quality for hallucination and safety settings in a similar fashion with the corresponding ground truth label. For example, for safety the ground truth label corresponds to whether a responses is safe or unsafe.

Next to assess how well abstention based on uncertainty can improve correctness, we compute Accuracy-Rejection Curves (ARC)
(Lin et al., ). ARC shows the change in accuracy when a model abstains for low-confidence responses given an uncertainty threshold.
To gauge the overall quality across thresholds, we compute the Area Under the Accuracy-Rejection Curve (AUARC).
We compute ARC and AUARC for both hallucinations and safety in a similar fashion by assessing how abstention based on uncertainty can reduce hallucinations or boost the safety of model responses.

## 4 Experimental Setup

We divide our experiments based on 3 settings: correctness, hallucinations and safety. Correctness refers to scenarios were the model is required to answer questions truthfully in the case of answerable questions. Hallucinations corresponds to settings where the model is required to abstains from responding when faced with unanswerable questions. Lastly, safety relates to the model’s ability to differentiate between safe and unsafe responses.

### 4.1 Correctness Settings

##### Models

Throughout this study, we utilize models from the Llama2 family Touvron et al. () since these models are open source and let us directly access log probabilities. These models, available in both pretrained (Base) and RLHF finetuned versions, offer direct access to log probabilities and come in various sizes, with the number of parameters being 7B, 13B and 70B. To ensure maximum reproducibility, we leverage open-source libraries from Hugging Face.

##### Datasets

We employ a diverse range of Q&A datasets. These include closed book question-answering datasets such as TriviaQA (Joshi et al., ) and SciQA (Auer et al., ), as well as the open book conversational question-answering dataset CoQA (Reddy et al., ), which provides a supporting paragraph to aid in answering a question. We also utilize StrategyQA (Geva et al., ), an implicit reasoning dataset requiring Yes or No answers, and GSM8K (Cobbe et al., ), a mathematical dataset comprising grade school math word problems created by human problem writers.

##### Determining correctness

To derive a ground truth label, we use fuzzy exact match. Unlike the traditional exact match, which requires a response to match the reference answer word for word, fuzzy exact match assesses whether the reference answer is encompassed within the response. This approach is more robust to variations in model response styles, while maintaining the interpretability of exact match. We show the validity of fuzzy exact match by comparing it with human evaluations in Appendix G.

### 4.2 Hallucination Settings

##### Models

We use the same models as we use in correctness settings.

##### Datasets

We employ the SelfAware dataset () where the model’s objective is to distinguish between answerable and unanswerable questions. This dataset is divided into two subsets: unanswerable questions from five diverse categories and their semantically closest answerable counterparts. This setup facilitates a fair evaluation of the model’s ability to discern when it does not know the correct answer.

##### Determining hallucinations via unanswerable questions

We examine the model’s ability to differentiate between answerable and unanswerable questions, using this distinction as our ground truth labels for computing AUROC.

### 4.3 Safety settings

##### Models

To evaluate the impact of different finetuning methods on uncertainty awareness in safety settings, we utilize both the pretrained and RLHF finetuned versions of Llama2. We also incorporate Vicuna, a supervised instruction finetuned Llama2 model, which leverages approximately 125K conversations collected from ShareGPT Chiang et al. ().

##### Datasets

While numerous adversarial datasets are designed to elicit unsafe and harmful responses from models, only a few have reportedly been successful with Llama2 models. Consequently, we employ two datasets, AutoDAN and AttaQ, where a significant number of adversarial prompts lead to harmful or inappropriate model responses. The creators of AttaQ (Kour et al., ) utilize clustering methods that consider both the semantic similarity of adversarial prompts and the potential harm of the model’s responses. AutoDAN (Zhu et al., ), on the other hand, ensures readability by combining a malicious user request with an adversarial prompt.

##### Determining response safety

To determine whether responses are safe or unsafe, we employ two metrics: one based on Llama Guard (Inan et al., ) and the other one based on the keyword-based approach outlined in Zhu et al. (); Zou et al. (). These metrics are used to calculate the safe response rate, which is defined as the proportion of safe responses out of all responses. The results presented in the main paper are based on the keyword-based approach, while additional results for Llama Guard are provided in the Appendix Table 6. We also assess the validity of both methods in relation to human evaluations in the Appendix H.

## 5 Results

Can uncertainty reveal when a model should abstain from responding with incorrect, hallucinated, or unsafe answers?
To explore this question, we evaluate statistical and In-Dialogue forms of uncertainty for pretrained and RLHF finetuned models
across three scenarios eliciting incorrect, hallucinated, or unsafe answers.

First for correctness, we find statistical uncertainty for both pretrained and RHLF models can reveal incorrect responses.
Second for hallucinations, we find In-Dialogue uncertainty can distinguish answerable from unanswerable questions, particularly for RLHF finetuned models.
Third for safety, we show statistical uncertainty for RLHF models can surface unsafe responses.
Finally, we operationalize the right form of uncertainty for each scenario to determine when the model should abstain.
In doing so, we show uncertainty—at a minimal computational cost—can boost correctness, reduce hallucinations, and improve safety in language models.

### 5.1 Statistical uncertainty can indicate when to abstain from responding incorrectly

Can uncertainty distinguish correct from incorrect model responses?
In Table 1, we compare statistical uncertainty measures across nine Q&A datasets.
Specifically we measure AUROC as an indicator of how well a particular uncertainty measure ranks correct responses.
We find that statistical measures of uncertainty are able to discriminate correct from incorrect responses. For RLHF fine-tuned language models semantic entropy performs the best on the majority of the datasets with an average AUROC of 0.69. In-Dialogue Uncertainty on the other hand cannot effectively differentiate between correct and incorrect answers with an average AUROC of 0.53. For the reasoning based GSM8K dataset it is notable that both, statistical uncertainty measures as well as In-Dialogue Uncertainty, exhibit close to random detection performance.

Can statistical measures also distinguish incorrect responses for RHLF finetuned models?
To study this, we compare the same statistical measures of uncertainty as well as diversity of responses for RLHF finetuned models (measured by the number of semantic sets). As shown in Figure 2, we find that RLHF finetuning leads to less diversity and notably higher confidence for model responses. These findings align with prior work indicating that RLHF finetuning increases misscalibration in LLMs OpenAI et al. (); Kadavath et al. (); Zhao et al. (); Tian et al. (); He et al. (); Zhou et al. (). Nevertheless, our findings demonstrate that RHLF finetuning actually preserves how well uncertainty can distinguish incorrect from correct responses.

##### Improving correctness by abstaining with statistical uncertainty.

Given uncertainty can effectively distinguish incorrect responses, can we use uncertainty to improve the overall accuracy on QA tasks? We analyze Accuracy-Rejection Curves (ARCs), which measure accuracy on QA tasks given an uncertainty threshold for abstaining, in Figure 3 a. Our analysis reveals that by rejecting the 5% most uncertain samples on TriviaQA based on semantic entropy, we can boost the accuracy of the RLHF finetuned model from 84.4% to 86.0%.
With a higher rejection rate of 25% we can increase the accuracy further by 8.2% to 92.6%. We show additional results for SciQA in Appendix H. These results underscore how uncertainty, at a minimal computation cost, can improve models’ accuracy on QA tasks.

**Table 1: Statistical uncertainty metrics are better suited to abstain from responding incorrectly. This table depicts AUROCs w.r.t. uncertainty metrics for Q&A questions from various datasets. Both RLHF finetuned models and pretrained models yield comparable AUROCs. Despite RLHF finetuning leading to increased miscalibration, these results suggest that it maintains uncertainty awareness, crucial for discerning when to abstain in instances of incorrect answers.**
| Data | Mode | AUROC w.r.t. |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Statistical Measures | InDU |  |  |  |  |
|  |  | Predictive Entropy | Semantic Entropy | Neg Log-Likelihood |  |
| TriviaQA | Base | 0.76 | 0.78 | 0.61 | 0.56 |
| SciQA | Base | 0.69 | 0.73 | 0.50 | 0.56 |
| CoQA | Base | 0.80 | 0.89 | 0.61 | 0.60 |
| StrategyQA | Base | 0.57 | 0.52 | 0.54 | 0.52 |
| GSM8K | Base | 0.50 | 0.46 | 0.50 | 0.47 |
| TriviaQA | RLHF | 0.82 | 0.85 | 0.80 | 0.51 |
| SciQA | RLHF | 0.72 | 0.74 | 0.66 | 0.54 |
| CoQA | RLHF | 0.75 | 0.78 | 0.70 | 0.43 |
| StrategyQA | RLHF | 0.64 | 0.63 | 0.61 | 0.64 |
| GSM8K | RLHF | 0.51 | 0.43 | 0.60 | 0.54 |

Figure: (a) Uncertainty

Figure: (a) Correctness (Statistical Uncertainty)

### 5.2 In-Dialogue uncertainty can reduce hallucinations for unanswerable questions

Next we shift our focus to the common scenario in-dialogue-based chat where the model encounters questions it cannot answer, and therefore should abstain. We conduct an experiment to discern whether LLMs can differentiate between answerable questions, such as "What is a transformer architecture?" and logically unanswerable questions, such as "What color iPhone did Einstein prefer?".

In Table 2 we show predictive entropy and semantic entropy offer limited signal for distinguishing unanswerable questions in both pretrained and RLHF filetuned models.
However, In-Dialogue Uncertainty as measured by AUROC is much more effective for distinguishing unaswerable questions.
While In-Dialogue Uncertainty is more useful in both models with or without RLHF,
we find RLHF finetuning enhances In-Dialogue Uncertainty awareness for unanswerable questions as shown in Table 2.
We further probe this capability by illustrating RHLF results in far more hedge words in responses for unanswerable vs. unanswerable questions as shown in Appendix Figure 8 across various model sizes.
These results suggests RLHF can reveal through In-Dialogue uncertainty when a question is unanswerable.

##### Reducing hallucinations for unanswerable questions with In-Dialogue uncertainty.

We now use In-Dialogue uncertainty to indicate when a model should abstain to reduce hallucinations for unanswerable questions. We show a simple filter for abstention based on In-Dialogue uncertainty can significantly reduce hallucinations for unanswerable questions. For example, In-Dialogue uncertainty can filter responses for nearly 50% of unanswerable questions at a cost of incorrectly refusing only 10% of answerable questions. We show the full receiver operating characteristic curve for all thresholds in Figure 3 b.

**Table 2: In-Dialogue uncertainty (InDU) is better suited to distinguish between answerable questions and unanswerable questions in contrast to statistical uncertainty measures. This table depicts AUROCs w.r.t. uncertainty metrics for answerable and unanswerable questions from the SelfAware dataset. RLHF finetuning leads to higher In-Dialogue Uncertainty AUROCs compared to the base model.**
| Data Configuration for SelfAware | Mode | AUROC w.r.t. |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Statistical Measures | InDU |  |  |  |  |
|  |  | Predictive Entropy | Semantic Entropy | Neg Log-Likelihood |  |
| Answerable & unanswerable | Base | 0.64 | 0.60 | 0.62 | 0.69 |
| Answerable & unanswerable | RLHF | 0.59 | 0.60 | 0.48 | 0.75 |

### 5.3 Statistical uncertainty can improve the safety of RLHF models

Here we examine models’ capacity to abstain from providing unsafe responses using uncertainty.
We evaluate model responses to
adversarial malicious prompts designed to elicit harmful responses using two open-sourced datasets: AutoDAN (Zhu et al., ) and AttaQ (Kour et al., ). To ensure a fair comparison, we compare three models based on Llama2 with 7B parameters:
pretrained only, instruction finetuned, and RLHF finetuned.

We find the pretrained model has the lowest safe response rates (7.4% for AttaQ and 9.0% for AutoDAN), followed by the supervised instruction finetuned model with response rates of 54.1% for AttaQ and 11.5% for AutoDAN. Finally, we find
the Llama2 RLHF finetuned model exhibits the highest safe response rate of 92.1% for AttaQ and 92.5% for AutoDAN prompts. These results confirm instruction tuning and RLHF can improve the safety of model responses.

Next we analyze the predictive performance of uncertainty measures from the differently finetuned models in identifying unsafe responses, measured via AUROC. As shown in Table 3, the pretrained Llama2 model exhibits random performance in identifying unsafe responses. Surprisingly, the supervised instruction finetuned model, although yielding a higher safe response rate, also mostly yields poor performance on both datasets. The RLHF finetuned Llama2 model on the other hand exhibits probability vectors that carry valuable information for properly identifying unsafe responses. As expected, In-Dialogue Uncertainty performs rather randomly, as while hedge words are used by the model to indicate unanswerable questions, they do not indicate unsafe responses. Our experiments demonstrate that RLHF finetuning not only aligns the model to safety, but also notably enhances its uncertainty awareness w.r.t. safety.

**Table 3: Only RLHF finetuning leads to well performing AUROCs w.r.t. all statistical uncertainty metrics. Evaluation of models using different alignment methods in distinguishing safe from unsafe responses on two adversarial datasets (AttaQ and AutoDAN). The RLHF finetuned Llama2 is evaluated against its pretrained version and a supervised instruction tuned variant, Vicuna. The AUROC is used to assess the model’s ability in distinguishing safe from unsafe responses using a given uncertainty metric.**
| Data |  | AUROC w.r.t. |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Mode | Predictive Entropy | Semantic Entropy | Neg Log-Likelihood | InDU |  |
| AttaQ | Base | 0.57 | 0.53 | 0.43 | 0.48 |
| AttaQ | Inst. Tune | 0.48 | 0.45 | 0.44 | 0.50 |
| AttaQ | RLHF | 0.78 | 0.77 | 0.78 | 0.51 |
| AutoDAN | Base | 0.48 | 0.24 | 0.54 | 0.52 |
| AutoDAN | Inst. Tune | 0.50 | 0.33 | 0.68 | 0.67 |
| AutoDAN | RLHF | 0.96 | 0.94 | 0.99 | 0.41 |

##### Improving safety with statistical uncertainty in RLHF finetuned models.

We investigate how well uncertainty can determine when a model should abstain to improve safety.
To do so, we compute the accuracy-rejection curve (ARC) where accuracy corresponds to whether a response is safe in Figure 3. For the RLHF model we observe that by abstaining to answer only the 10% most uncertain samples w.r.t. negative log-likelihood the proportion of safe responses for AttaQ improves from 92.1% to 95.1% and for AutoDAN even from 92.5% to 99.4%. We present additional results for ARC in the Appendix Figure 5. Furthermore, in Appendix 10 when investigating the ROC curve we find that across both adversarial datasets uncertainty can filter a majority of unsafe responses. For AutoDAN 99% of unsafe responses can be filtered out with only 10% falsely refused responses. For AttaQ, 70% of unsafe responses can be filtered while sacrificing 30% falsely refused responses. These results represent even further enhancements upon an already high-performing RLHF finetuned model in terms of safety. Overall these findings demonstrate abstaining via statistical uncertainty can dramatically boost the safety of RLHF model responses.

## 6 Discussion

Uncertainty can be complicated in large language models. On the one hand, similar to classification, there is statistical uncertainty induced by the vector of token probabilities – together with its multiple versions, such as predictive and semantic entropy. On the other, there is In-Dialogue Uncertainty induced by the presence of hedge words in responses. In addition, there are different forms of the models – with or without RLHF – that have different kinds of uncertainty profiles.

The main message of our study is that despite these complications, selecting the right kind of uncertainty together with the right form of the model can lead to better abstention decisions for question answering, in the process promoting correctness and safety, and reducing hallucinations for unanswerable questions. The form of uncertainty and the form of the model differ from one use-case to the next – statistical uncertainty works for correctness and safety, while in-dialogue uncertainty works for unanswerable questions. Our study underscores the importance of uncertainty in the context of LLMs, and we hope that it will contribute to the recognition of uncertainty as a critical factor in working with LLMs, as has been the case in other fields of machine learning for a long time (Hendrycks and Gimpel, ; Liang et al., ; Liu et al., ; Hendrycks et al., ).

## Appendix

## Appendix A Accuracy-Rejection Curves (ARCs) for Correctness and Safety Scenarios

To provide deeper insights into the impact of uncertainty scores on the performance and safety of models, we present further Accuracy-Rejection Curves (ARCs) in the following sections. Generally, ARCs measure the accuracy on a subset of the dataset that is retained when samples are progressively rejected based on an uncertainty measure, from the most uncertain to the least uncertain. A higher Area Under the Accuracy-Rejection Curve (AUARC) indicates a more effective uncertainty measure, as it can better differentiate between incorrect and correct samples, or between safe and unsafe samples.

### A.1 Correctness Settings

In the context of correctness settings, accuracy is defined as the ratio of correct samples to all samples in the remaining dataset. In Figure 4 we present ARC curves for both TriviaQA and SciQA, comparing the pretrained and RLHF finetuned models. For TriviaQA, the AUARC is identical for both models at 0.94. For SciQA, the RLHF finetuned model outperforms the pretrained model with a score of 0.83 versus 0.77. As we reject more samples, the accuracy increases, demonstrating that the use of uncertainty measures to reject samples effectively enhances the model’s performance. This approach is particularly beneficial when the goal is to deploy a highly accurate model. While such a model may abstain more frequently, when it does provide an answer, it is more likely to be accurate.

Figure: (a) TriviaQA - Llama2 Pretrained

### A.2 Safety Settings

In safety settings, the accuracy in the ARC is defined as the ratio of safe samples to all samples in the remaining dataset. We use the keyword based safety metric to measure whether a response is safe or unsafe. We present ARC curves for AttaQ and AutoDAN, using the Llama2 70b RLHF finetuned model in Figure 5. The AUARC for AttaQ is 0.97, and for AutoDAN, it is near perfect at 0.99. Similar to the correctness setting, we observe that as more samples are rejected, the proportion of safe responses in the remaining dataset increases. This indicates that uncertainty metrics can be effectively employed to enhance the safety of models. In situations where particularly safe models are essential and a few unanswered questions can be tolerated, using uncertainty measures proves highly effective, as they require little to no additional computational effort.

Figure: (a) AttaQ - Llama2 RLHF finetuned

## Appendix B Additional results for Correctness Settings

This section presents additional results featuring more uncertainty metrics for fact-based Q&A settings. The datasets and prompting schemes used are identical to those in the main text. Figure 6 provides additional results for mean predictive entropy and mean negative log-likelihood, while Figure 7 presents the AUROC w.r.t. predictive entropy, semantic entropy and negative log-likelihood. These additional findings corroborate the results in the main paper, confirming that RLHF finetuning consistently leads to higher predictive confidence and less diversity in responses. However, the model’s uncertainty awareness, as measured by AUROC, remains intact.

Figure: (a) Mean Predictive Entropy

Figure: (a) AUROC w.r.t predictive entropy

## Appendix C Additional results for Hallucination Settings

Here we show further results for hallucination settings. Our findings in Figure 8 reveal that responses to unanswerable questions contain a higher average number of hedge words, indicating greater In-Dialogue Uncertainty, compared to responses to answerable questions. This pattern holds true across models of varying sizes and both with and without RLHF fine-tuning, although the difference in the number of hedge words in responses is slightly less pronounced for pretrained models. However, when it comes to mean statistical measures over samples, there is hardly any difference between answerable and unanswerable questions, which adversely impacts the AUROC as previously demonstrated. Moreover, the Receiver Operating Characteristic (ROC) curve with respect to In-Dialogue Uncertainty, as depicted in Figure 9, illustrates that InDU can effectively identify approximately 50% of unanswerable questions. This capability comes with a trade-off of incorrectly refusing only 10% of questions that could have been answered.

Figure: Figure 8: In-Dialogue Uncertainty (InDU) shows a much higher gap between answerable and unanswerable questions than statistical measures. Statistical uncertainty measures exhibit almost the same average scores for answerable and unanswerable questions. In contrast, In-Dialogue Uncertainty, based on the number of hedge words, shows a clear difference, with responses to unanswerable questions containing twice as many hedge words as responses to answerable questions. These results remain largely consistent regardless of model size and whether or not RLHF finetuning is applied.

Figure: Figure 9: In-Dialogue Uncertainty enables the filtering of 50% of unanswerable questions at the cost of incorrectly refusing 10% of answerable questions. This figure illustrates the Receiver Operating Characteristic (ROC) curve for distinguishing between answerable and unanswerable questions using In-Dialogue Uncertainty.

## Appendix D Additional Results for Safety Settings

We further investigate the Receiver Operating Characteristic (ROC) curves of all three models, which are based on Llama2 with 7B parameters: pretrained only, instruction finetuned, and RLHF finetuned. The models are evaluated on two datasets: AutoDAN [Zhu et al., ] and AttaQ [Kour et al., ]. In Figure 10 we find that for the RLHF finetuned model on AutoDAN almost 100% of unsafe responses can be filtered out with almost no falsely refused responses. For AttaQ, 70% of unsafe responses can be filtered while sacrificing 30% falsely refused responses.

Figure: (a) AutoDAN

## Appendix E Combining Hallucination & Correctness

Here we combine the hallucination with the correctness setting to inverstigate whether the model can even differentiate between answerable questions ("What is a transformer architecture?") that are answered incorrectly due to a lack of specific knowledge, and logically unanswerable questions ("What color iPhone did Einstein prefer?"). As shown in Table 4, we measure the AUROC between falsely answered and unanswerable questions and find that statistical uncertainty measures perform no better than random. Surprisingly, In-Dialogue Uncertainty performs nearly as well on this task as in distinguishing answerable from unanswerable questions. A more detailed analysis in Table 5 reveals that statistical uncertainty measures are, on average, very similar for incorrectly answered questions and unanswerable questions. Only In-Dialogue Uncertainty is twice as high for unanswerable questions than for incorrectly answered questions. This demonstrates that LLMs are capable of distinguishing not only between answerable and unanswerable questions but also between wrongly answered questions and unanswerable questions. However, instead of expressing this distinction through log probabilities, LLMs convey it through the use of hedge words.

**Table 4: In-dialogue verbalized uncertainty can distinguish between incorrectly answered questions and unanswerable questions, a scenario where statistical measures show no indicative power. This table depicts AUROCs w.r.t. uncertainty metrics for incorrectly answered questions and unanswerable questions from the SelfAware dataset. RLHF finetuning leads to higher in-dialogue verabalized uncertainty AUROCs compared to the base model.**
| Data Configuration for SelfAware | Mode | AUROC w.r.t. |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Statistical Measures | InDU |  |  |  |  |
|  |  | Predictive Entropy | Semantic Entropy | Neg Log-Likelihood |  |
| Answerable (False) & unanswerable | Base | 0.50 | 0.47 | 0.59 | 0.66 |
| Answerable (False) & unanswerable | RLHF | 0.46 | 0.47 | 0.44 | 0.72 |
| Answerable & unanswerable | Base | 0.64 | 0.60 | 0.62 | 0.69 |
| Answerable & unanswerable | RLHF | 0.59 | 0.60 | 0.48 | 0.75 |

**Table 5: Our In-Dialogue Uncertainty (InDU) measure shows a notable difference between incorrectly answered and unanswerable questions. This table presents the mean uncertainty measures for both answerable and unanswerable questions. While the statistical uncertainty measures exhibit only a slight increase for unanswerable questions compared to incorrectly answered questions, the mean number of hedge words for unanswerable questions is more than twice that for incorrectly answered questions.**
| Data Configuration for SelfAware | Mode | Mean |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Statistical Measures | InDU |  |  |  |  |
|  |  | Predictive Entropy | Semantic Entropy | Neg Log-Likelihood |  |
| Answerable (False) | RLHF | 4.63 | 6.96 | 11.2 | 1.92 |
| Unanswerable | RLHF | 4.20 | 6.62 | 10.6 | 4.79 |
| Answerable (False) | Base | 10.34 | 10.60 | 25.3 | 1.75 |
| Unanswerable | Base | 9.69 | 9.90 | 28.0 | 4.41 |

## Appendix F Further Insights into In-Dialogue Uncertainty

To gain deeper insights into the In-Dialogue Uncertainty, we examine the hedge words used in the model’s responses. Figure 11 compares incorrect responses from the pretrained with those from the RLHF finetuned model. We find that the RLHF finetuned model exhibits greater diversity in expressing verbalized uncertainty within dialogues. Furthermore, Figure 12 reveals a significant difference in the number of hedge words used in responses to answerable and unanswerable questions. Specifically, a large proportion of responses to answerable questions contain no hedge words, whereas responses to unanswerable questions typically include at least one hedge word.
The mean number of hedge words in responses to answerable questions is 1.67 (with a standard deviation of 2.3). In contrast, the mean number of hedge words in responses to unanswerable questions is nearly three times higher, at 4.79 (with a standard deviation of 4.45).
These findings suggest that the natural use of hedge words in model responses can serve as an indicator of whether a question is answerable or unanswerable, as evidenced by the high AUROC values reported in the main paper.

Figure: (a) Pretrained

Figure: (a) Answerable questions

## Appendix G Validity of Fuzzy Exact Match

To evaluate the correctness of model responses, we derive a ground truth label via utilizing fuzzy exact match. This method, unlike the traditional exact match metric that necessitates a word-for-word match between the response and the reference answer, determines if the reference answer is contained within the response. This approach is more resilient to variations in model response styles, while preserving the interpretability inherent in exact match.

We validate the effectiveness of fuzzy exact match by comparing it with human evaluations on 200 samples each from TriviaQA and SciQA.
We juxtapose the accuracy derived from these human evaluations with that obtained from fuzzy exact match, as well as the widely used exact match and F1 score metrics based on RougeL.
For TriviaQA, the accuracy based on human evaluations is 87.5%, closely mirrored by the 85.0% accuracy of fuzzy exact match. In contrast, the accuracies based on exact match and F1 score are significantly lower, at 14.5% and 27.8%, respectively. A similar trend is observed for SciQA, with human evaluations yielding an accuracy of 77.0%, fuzzy exact match 65.0%, and exact match and F1 score only 2.5% and 7.0% respectively. These findings indicate that the commonly used metrics of exact match and F1 score diverge significantly from human evaluations when assessing the correctness of responses. On the other hand, fuzzy exact match aligns much more closely with human evaluations and is still equally interpretable. Consequently, we adopt fuzzy exact match as our metric for correctness throughout our study.

## Appendix H Validity of Safety Metrics

We employ two metrics to compute the safe response rate of models given a dataset: one is based on Llama Guard [Inan et al., ], and the other utilizes the keyword-based approach described in Zhu et al. [], Zou et al. []. The safe response rate is defined as the ratio of safe responses to all responses.
To validate these metrics, we compare them with human evaluations on the AutoDAN dataset [Zhu et al., ]. Each response is assessed by three human judges, and a majority vote determines the safety of the response. The safe response rate for human evaluations is 92.5%, identical to the keyword-based metric, as the two sets are the same. However, Llama Guard yields a safe response rate of 100% on this dataset.
We further conduct a pairwise t-test for Llama Guard, resulting in a p-value of
$8.35\mathrm{e}{-10}$. Consequently, we use the keyword-based metric in the main paper and present results for Llama Guard on AttaQ in the appendix in Table 6.

**Table 6: Only RLHF finetuning leads to well performing AUROCs w.r.t. uncertainty metrics, underscoring that pretraining and supervised instruction tuning are insufficient for inducing safety-related uncertainty awareness into the model. Safety of responses is calculated with Llama Guard. Evaluation of models using different alignment methods in distinguishing safe from unsafe responses on AttaQ. The RLHF finetuned Llama2-7b is evaluated against its pretrained version and a supervised instruction tuned variant, Vicuna. The AUROC is used to assess the model’s ability in distinguishing safe from unsafe responses.**
| Data |  | AUROC w.r.t. |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Mode | Predictive Entropy | Semantic Entropy | Neg Log-Likelihood | VIDU |  |
| AttaQ | Base | 0.56 | 0.53 | 0.50 | 0.46 |
| AttaQ | Inst. Tune | 0.44 | 0.45 | 0.29 | 0.35 |
| AttaQ | RLHF | 0.79 | 0.52 | 0.98 | 0.92 |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: - To be clear, in these experiments, the model might not actually abstain, right? You would have to calculate these metrics and then hide the response from the user if it were deemed unacceptable, right?
- It was hard to tell if there was a proposed training or inference method from reading the intro. It took me a while to realize that this was more of an analysis paper, showing how these metrics could be used for filtering model outputs.
- Sec. 3 probably doesn’t need to take up as much space is it currently does (people should know what NLL is), but at the same time it could give more understanding into the metrics (computing actual entropy over samples is hard, so you compute predictive entropy).
- L.377 typo “unanswerable vs. unanswerable”
- L.471 “abstaining to answer”→ “abstaining from answering”
soundness: 4
strengths: - Important: The core idea of the paper is sensible, relating uncertainty metrics to abstention in order to improve factuality and safety of model responses.
- Important: The experiment design is sound and the chosen metrics are reasonable. The experiments include many relevant datasets for measuring knowledge, hallucination, and safety.
- Of some importance: The paper is fairly well-written and easy to follow.
summary: The paper studies how token-level and semantic uncertainty metrics on generated LLM text relate to accuracy on knowledge-intensive tasks, hallucination on unanswerable questions, and response safety on adversarial/malicious question datasets. Among the uncertainty metrics explored is one based on on counting hedge words in model responses. Experiments show that these uncertainty metrics are useful for model abstention in order to improve correctness of model generations, reduce hallucination, and increase response safety (at the cost of an increased abstention rate). Experiments are conducted across many relevant datasets using Llama 2 models.
weaknesses: - Very important: The novelty of the work is quite limited in my view. The high level conclusion that uncertainty is useful for abstention has already been thoroughly explored. What I see as this paper’s contributions beyond this observation are: (1) measuring in-dialogue uncertainty can help with abstention, specifically for factuality/hallucination; (2) uncertainty can help with safety, as it turns out that responses on AutoDAN-like datasets are more likely to be unsafe if they are uncertain. I don’t think the paper claims much beyond this. So a further issue with the novelty here is that (1) has already been shown, more or less, in https://arxiv.org/abs/2405.21028. The (2) result is interesting but I do not think it is a large enough result for a full paper, and it is not explored in much depth beyond one paragraph in this paper.
- Important: The measurement of in-dialogue uncertainty, even if useful, is a heuristic that does not feel particularly generalizable, especially compared to other model-based measurements of in-dialogue confidence.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: - How can we practically account for all possible hedge words for every use case? Some prompts might even require responses to include hedge words; seems like a lot of finetuning and engineering effort to incorporate this uncertainty metric
- I'm not sure I agree with hallucinations being only considered for unanswerable questions. LLMs definitely hallucinate in other situations. How extendable are these findings?
- Statistical uncertainty metrics perform at more or less the same level. What should the reader take away from all these results?
soundness: 3
strengths: - Well-written paper with a clear walkthrough over the different problems and uncertainty metric considerations
- Interesting idea of using in-dialogue uncertainty as a measure of response uncertainty
- Clear description of experiments, metrics, and results; strong scientific method
summary: Paper shows that abstention based on different measures of uncertainty for different types of prompts works well. Specifically, for correctness and safety, statistical uncertainty-based abstention helps improve correctness and reduce unsafe responses. For hallucinations, abstention based on in-dialogue uncertainty (coined by authors as the inclusion of phrases such as "I don't know" in model responses) helps reduce hallucinations.
weaknesses: - It should not come as a surprise that using uncertainty metrics helps LLMs abstain when they should not engage with the prompt, as shown in Kadavath (2022) and multiple other papers cited in the related works. The core contributions of this paper can be boiled down to the introduction InDU (which also was inspired by an existing paper by Islam (2020)) and when to use each kind of uncertainty, both of which seem more fitting for, e.g., an appendix in Kadavath's paper, especially since this reads more like a survey paper of implementation details than novel ideas or concepts
- Minor: various typos such as "In-Dialogoe" in Introduction, Islam et al. without year in 3.2

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - It would be worth discussing the tradeoff between abstention and usability further. 
- In-Dialogue Uncertainty is given an acronym but the acronym isn't used. It's also misspelled on L053.
soundness: 2
strengths: - **Well written and clearly organized**: the paper is easy to follow, the writing is clear, and the questions being tested are clear. 
- **In-dialogue uncertainty metric is new**: As far as I can tell, past work has not proposed counting the number of hedge words as a method of confidence estimation. 
- **Sufficient datasets examined**: The authors do a good job of testing multiple datasets to make their point.
summary: This paper focuses on abstention and uncertainty in LLMs, benchmarking how useful different uncertainty estimates are across three broad tasks.
These tasks are correctness, unanswerable questions, and safety. 
Correctness is evaluated against standard QA data (TriviaQA, SciQA, CoQA, StrategyQA, and GSM8K).
Unanswerable vs answerable questions are sourced from the SelfAware dataset and SciQA. 
Adversarial examples are sourced from AttaQ and AutoDAN. 
The authors examine negative log-likelihood, predictive entropy, semantic entropy, and In-Dialogue Uncertainty, which is the number of hedge tokens present in the output. 
All experiments were run on Llama2.
Across different tasks, the authors find that different uncertainty estimates lead to better or worse calibration, with no one method consistently outperforming the others. 
The authors show that thresholding uncertainty scores can lead to better correctness, safety, and less hallucination on unanswerable questions.
weaknesses: - **Limited novelty:** The novelty of the paper is pretty limited. From the abstract/intro, it seems like the main contribution of the paper is in showing that abstention based on uncertainty can improve results. This result is not new (see next point about missing related work). Moreover, the primary methodological novelty in this work is In-Dialogue Uncertainty, which is a fairly small contribution and does not consistently provide benefits in all settings. The Discussion presents a more nuanced view of the contribution (i.e. framing this paper as a survey of confidence estimation methods and showing that there isn't one method that consistently does well.) This framing would have been more novel but then I would have expected to see more different uncertainty estimation methods tested. 
- **Missing related work:** This paper misses a large chunk of the related work on abstention and confidence estimation from the last 2 years, focusing on older work. Examples:
	- https://arxiv.org/pdf/2407.18418
	- https://arxiv.org/abs/2308.13387
	- https://arxiv.org/abs/2311.09677
	- https://arxiv.org/abs/2404.00474
	- https://arxiv.org/abs/2405.21028
	- https://arxiv.org/abs/2401.06730
	- https://aclanthology.org/2024.naacl-long.301/

- **Outdated models**: It's not clear why the authors only conduct experiments on Llama2, when there are many newer and more performant models available (even in the same family). To make a strong claim about when different estimation methods work and don't work, I would have expected to see more open-source models tested. 
- **No unified method**: one way this paper could have been made more compelling is if it presented a unified estimation method/recipe that worked well across settings. Currently, the paper does not have any such unified method.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See above.
soundness: 2
strengths: 1. The motivation behind this work is compelling and relevant for the deployment and real-world application of LLMs. However, while the authors highlight the benefits of combining RLHF with uncertainty to enhance performance, reviewers suggest that additional validation experiments, particularly in the areas of hallucination and safety, would strengthen the claims.
2. The paper underscores the importance of uncertainty for abstention, demonstrating that incorporating uncertainty can improve various aspects of model performance.
summary: The authors investigate the potential of uncertainty-based abstention to improve performance in question-answering tasks, specifically focusing on correctness, hallucinations, and safety scenarios. They analyze two types of uncertainty—statistical uncertainty and in-dialogue uncertainty—and examine the effects of RLHF on these uncertainties.
weaknesses: 1. Although the reviewers appreciate the study's motivation, they raise concerns regarding the experimental setup. For instance, in the hallucination settings, tests are only conducted on the SelfAware dataset. It would be beneficial to include additional datasets to more comprehensively evaluate the method's effectiveness in reducing hallucinations, especially given that current approaches primarily rely on Retrieval-Augmented Generation (RAG) [1].

2. In the safety setting, the reviewers are interested in seeing how the uncertainty mechanism performs across a broader range of evaluation datasets. For example, PKU-SafeRLHF [3] provides safe, decoupled preferences and red-teaming prompts; how does the proposed approach perform on safety measures in these rigorous evaluations via case by case gpt-4 evaluation?

3. The reviewers are not fully convinced by the claim that "our experiments demonstrate that RLHF fine-tuning not only aligns the model with safety but also enhances its uncertainty awareness in relation to safety." RLHF alone does not guarantee model safety, particularly when the preference data distribution is uncertain. For instance, the GPT-4 technical report highlights that while RLHF helps align model responses with user intent, models may still exhibit brittle or undesired behaviors on both safe and unsafe inputs, especially when labeler instructions during reward model data collection are underspecified. Reviewers suggest that the authors provide a more detailed discussion on this aspect and include comparisons with models specifically designed for safety alignment, such as RLCD [2] and Safe RLHF [3].

4. Regarding evaluation, the authors rely primarily on statistical measures, such as keyword-based approaches. However, this static evaluation method may fall short of detecting nuanced harmful responses, such as those involving emotional abuse. Additionally, Llama Guard’s performance drops in non-OOD (Out-of-Distribution) scenarios. Reviewers recommend including case-by-case GPT-4 evaluations to directly assess the safety of two responses, providing a more granular safety evaluation.

[1] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection  
[2] RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment  
[3] PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: "we recommend that practitioners differentiate between different types of questions before they decide whether to use statistical uncertainty or verbalized uncertainty for abstention..." Could you explain why the experiment you conducted supports this claim?
soundness: 3
strengths: - Studied topic is timely and important. 
- Paper outlines multiple applications of the proposed method.
- Paper is well-written and easy to follow. 
- Experiments test diverse uncertainty measures
- Experiments consider different model sizes and variants.
summary: This paper proposes uncertainty-based methods for abstaining from providing incorrect, hallucinated, or unsafe answers. It considers probabilistic uncertainty (log likelihood, entropy, and semantic uncertainty) as well as verbal uncertainty. Experiments with Llama2 models across various question-answering and adversarial-prompting benchmarks demonstrate that (1) the considered uncertainty measures contain information about whether an answer is incorrect, hallucinated, or unsafe, and (2) abstention based on these measures is effective.
weaknesses: - There exist more sophisticated approaches to these problems but they are not compared empirically [1, 2]. [1] build classifiers based on hidden representations, and [2] constructs ensembles using LLM prompting. Authors mention the weaknesses of prompting and fine-tuning in related work but do not demonstrate them through experiments. In fact, the experiments do not seem to concern distribution shift so there is no reason not to compare with those methods. 
- Related work is missing some recent work showing similar results (e.g., [1,2,3]).
- Experiment sections mostly discuss observations but does not attempt to explain the observed phenemena. 
- Some parts of the experiment section are unclear or can be further improved. Specifically, in figure 3, "statistical uncertainty" should be replaced with a specific measure and model (e.g., entropy). It is also missing model names. The plots need to have baseline curves to clearly illustrate improvements. 

[1] https://arxiv.org/abs/2304.13734

[2] https://arxiv.org/abs/2402.00367

[3] https://arxiv.org/abs/2402.13213

Meta-review: PC is entering meta-review on behalf of SAC/AC:

The reviewers did not believe that this paper was a strong contribution given the limited novelty of the work, and lack of anchoring in the field.

Decision: Reject