On Calibration of LLM-based Guard Models for Reliable Content Moderation

## Abstract

Abstract Large language models (LLMs) pose significant risks due to the potential for generating harmful content or users attempting to evade guardrails. Existing studies have developed LLM-based guard models designed to moderate the input and output of threat LLMs, ensuring adherence to safety policies by blocking content that violates these protocols upon deployment. However, limited attention has been given to the reliability and calibration of such guard models. In this work, we empirically conduct comprehensive investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. Our findings reveal that current LLM-based guard models tend to 1) produce overconfident predictions, 2) exhibit significant miscalibration when subjected to jailbreak attacks, and 3) demonstrate limited robustness to the outputs generated by different types of response models. Additionally, we assess the effectiveness of post-hoc calibration methods to mitigate miscalibration. We demonstrate the efficacy of temperature scaling and, for the first time, highlight the benefits of contextual calibration for confidence calibration of guard models, particularly in the absence of validation sets. Our analysis and experiments underscore the limitations of current LLM-based guard models and provide valuable insights for the future development of well-calibrated guard models toward more reliable content moderation. We also advocate for incorporating reliability evaluation of confidence calibration when releasing future LLM-based guard models 1 1 1 Our code is publicly available at https://github.com/Waffle-Liu/calibration_guard_model .

## 1 Introduction

Recent advancements in Large Language Models (LLMs) have facilitated the development of powerful conversation systems, leading to the deployment of LLM-based chatbots in various real-world applications (Brown, ; Anil et al., ; Touvron et al., ; Dubey et al., ). However, these systems pose substantial risks due to the potential for malicious exploitation of powerful LLMs by generating harmful content (Wang et al., ). Consequently, addressing these risks has become an urgent and critical task. One promising strategy is to regulate LLMs during their training phase. Existing researches primarily focus on designing alignment algorithms through preference optimization (Ouyang et al., ; Rafailov et al., ), implementing adversarial training (Mazeika et al., ), or employing machine unlearning to remove harmful knowledge from the models (Chen & Yang, ; Liu et al., ). These approaches aim to control text generation and prevent undesired outputs. Despite these significant efforts to enhance LLM safety during training, red-teaming still makes efforts to expose vulnerabilities, including jailbreak attacks that successfully bypass the safety constraint and elicit harmful responses from LLMs, highlighting the risks of future, unseen threats (Zou et al., ; Liu et al., ; Chao et al., ). Therefore, in addition to training-time interventions, it is equally vital to implement test-time measures, such as constraint inference (Xu et al., ), and establish effective test-time guardrails through content moderation, particularly when deploying LLMs in real-world settings.
Content Moderation serves the critical function of monitoring both user inputs and model outputs during conversations. Typically, guard models are designed to assess whether user inputs and LLM outputs comply with safety regulations, and either reject user queries or block model responses when content violating safety protocols is detected. This approach remains effective even when LLMs have been compromised by previously unseen jailbreak attacks. Current state-of-the-art guard models, which are typically built on LLMs, demonstrate strong performance across various benchmarks (Inan et al., ; Ghosh et al., ; Han et al., ; Zeng et al., ). However, these guard models primarily emphasize the classification performance but overlook the predictive uncertainty of harmfulness predictions, therefore failing to assess the reliability of these models’ predictions. This oversight is crucial because guard models may occasionally make erroneous decisions, potentially allowing unsafe content to bypass moderation, especially when encountering non-trivial domain shifts, despite their strong in-domain performances. Therefore, quantifying the predictive uncertainty and confidence in model predictions is essential to assessing the trustworthiness of guard models, enabling more reliable decision-making in high-risk scenarios that may arise during conversations after model deployment.

Figure: Figure 1: An overview of LLM-based guard models for content moderation. Guard models monitor the input and output during conversations between the User and LLM (Agent), providing a binary prediction followed by a specific unsafe content category if unsafe content is detected. The instruction examples for prompt classification and response classification from LLama-Guard are detailed in the right yellow boxes respectively.
Refer to caption: x1.png

In this work, we examine the reliability of existing open-source guard models by focusing on their confidence calibration. Specifically, we empirically assess the calibration performance by commonly used expected calibration error (ECE) for two key tasks: user input (prompt) classification and model output (response) classification with binary labels. To conduct a systematic evaluation, we examine 9 models across 12 datasets. Our experimental results reveal that, despite achieving strong performance, most existing guard models exhibit varying levels of miscalibration. Additionally, our findings show that current LLM-based guard models:

- •
tend to make overconfident predictions with high probability scores.
- •
remain poorly calibrated under adversarial environments, exhibiting higher ECE in adversarial prompt classification, even when the SOTA guard model achieves high F1 scores.
- •
display inconsistent ECE across different types of response models, demonstrating weak robustness to variations in response model types.

These observations highlight critical challenges in improving the reliability of guard models in real-world deployments. Consequently, we are motivated to improve the confidence calibration of guard models, focusing on post-hoc calibration methods to avoid additional computational costs of training new guard models. We explore the impact of bias calibration methods on confidence calibration for the first time, discovering that contextual calibration proves impressively effective for prompt classification, while conventional temperature scaling remains more beneficial for response classification. Lastly, we identify miscalibration issues stemming from prediction vulnerabilities induced by single tokens and misaligned classification objectives, highlighting the limitations of instruction-tuned LLM-based guard models. We stress the importance of reliability evaluation and advocate for the inclusion of confidence calibration measurement in the release of future new guard models.

## 2 Related Work

Content Moderation. A substantial body of research has been devoted to the detection of hateful and toxic content in human-generated text from online platforms, such as social media (Schmidt & Wiegand, ; Lees et al., ). Various API services, including Perspective (Lees et al., ), OpenAI (Markov et al., ), Azure, and Detoxify, provide black-box content moderation tools for online texts. However, content moderation in LLMs specifically addresses the detection of both LLM input and output during conversations within deployed applications, such as chat assistants. This task poses unique challenges due to the distribution shift in conversation content generated by LLMs, which differs from previous human-generated online texts. Recent advancements in LLM content moderation have been achieved through the fine-tuning of LLMs, as seen in models such as LLama-Guard1/2/3 (Inan et al., ), BeaverDam (Ji et al., ), Aegis (Ghosh et al., ), MD-Judge (Li et al., ), WildGuard (Han et al., ), and ShieldGemma (Zeng et al., ). Notably, models like Llama-Guard, Aegis, and WildGuard support the detection of both user inputs and model outputs, while others do not due to differing training objectives. Additionally, adversarial cases are addressed by Harmbench (Mazeika et al., ) and RigorLLM (Yuan et al., ), with Harmbench specifically fine-tuning LLama2 and Mistral to evaluate the success rate of adversarial attacks by identifying undesirable content in model outputs. Furthermore, Nemo proposes programmable guardrails that provide dialogue management capability using a user-friendly toolkit (Rebedea et al., ). Our work focuses on quantifying the predictive uncertainty and evaluating the reliability of LLM-based guard models by their calibration levels.

Calibration of LLMs. Confidence calibration is a critical aspect in developing reliable and trustworthy language models (Nguyen & O’Connor, ; Guo et al., ; Minderer et al., ). In the context of LLMs, prior research has explored calibration in question-answering tasks (Jiang et al., ) and has empirically examined calibration during the pre-training and alignment stages (Chen et al., ; Zhu et al., ). Studies such as Lin et al. (); Mielke et al. (); Xiong et al. () have investigated uncertainty estimation through verbalized confidence, and Kadavath et al. () demonstrated improved calibration of larger models when handling multiple choice and true/false questions given appropriate formats. Another line of research addresses the calibration of biases stemming from in-context samples, instruction templates, sample ordering, and label distribution (Zhao et al., ; Zhou et al., ; Liu & Wang, ; Fei et al., ). These bias calibration techniques indirectly influence the prediction confidence by altering the linear decision boundary (Zhou et al., ), yet they are not designed for explicit confidence calibration. In contrast, our work specifically addresses the challenge of confidence calibration in instruction-tuned guard models for content moderation tasks.

## 3 Preliminary

LLM-based Guard Models. Given the user input text $\mathbf{X}$ and the corresponding response $\mathbf{R}=f(\mathbf{X})$ generated by a deployed LLM $f(*)$, the task of the LLM-based guard model $g(*)$ is to classify the user input $p_{g}(\mathbf{Y}|\mathbf{X})$, or the LLM output $p_{g}(\mathbf{Y}|\mathbf{X},\mathbf{R})$(^2^22Note that we ignore the instruction context $\mathrm{C_{inst}}$ in our all following notations for simplicity where they should be $p_{g}(\mathbf{Y}|\mathbf{X};\mathrm{C_{inst}})$ and $p_{g}(\mathbf{Y}|\mathbf{X},\mathbf{R};\mathrm{C_{inst}})$ instead.) These tasks are referred to as prompt classification and response classification, respectively. For the predicted label $\mathbf{Y}$, most existing LLM-based guard models initially perform binary classifications $y^{b}$ to determine whether the user input $\mathbf{X}$ or model response $\mathbf{R}$ is safe. If the binary classification result indicates the input or the response $y^{b}$ is unsafe, the guard model $g(*)$ then proceeds with a multiclass classification to categorize the specific type $y^{c}$ by $p_{g}(y^{c}|\mathbf{X},y^{b})$ or $p_{g}(y^{c}|\mathbf{X},\mathbf{R},y^{b})$ where the categories $c$ are pre-defined in a taxonomy. These classification tasks in LLM-based guard models are carried out in an autoregressive generation manner, and Figure 1 illustrates examples of the prompt and response classification instructions used in LLama-Guard.

Confidence Calibration. A model is considered perfectly calibrated if its predicted class $\hat{y}$ and the associated confidence $\hat{p}\in[0,1]$ satisfy $P(\hat{y}=y|\hat{p}=p)=p,\forall p\in[0,1]$, where $y$ is the ground-truth class label for any given input. This implies that higher confidence in a prediction should correspond to a higher chance of its prediction being correct. However, since $P(\hat{y}=y|\hat{p}=p)$ can not be directly calculated with finite sample size, existing approaches employ binning-based divisions on finite samples and utilize the Expected Calibration Error (ECE) as a quantitative metric to assess the model’s calibration (Naeini et al., ). Assuming that confidence is divided into $M$ bins with equal interval $1/M$ within the range $[0,1]$, the ECE is defined as

$$ $ECE=\sum_{m=1}^{M}\frac{|B_{m}|}{N}\left|Acc(B_{m})-Conf(B_{m})\right|,$ (1) $$

$$ $Acc(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\mathbf{1}(\hat{y}_{i}=y_{i}), \quad Conf(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\hat{p}_{i}$ (2) $$

where $B_{m}$ represents the set of samples falling within the interval $(\frac{m-1}{M},\frac{m}{M}]$, $\hat{y}_{i}$ and $y_{i}$ are the predicted and ground truth classes, respectively, and $\hat{p}_{i}$ is the model’s predicted probability. However, existing instruction-tuned LLM-based guard models do not directly output the probability of each class. Instead, the probability of class $c_{i}$ is derived from the output logits $z_{\mathcal{V}(c_{i})}$ of the corresponding target label token $\mathcal{V}(c_{i})$, where $\mathcal{V}(*)$ is the verbalizer. Re-normalization is then applied over the set of target label tokens as follows,

$$ $p(y=c_{i}|\mathbf{X},\mathbf{R})=\frac{e^{z_{\mathcal{V}(c_{i})}}}{\sum_{c_{i} }e^{z_{\mathcal{V}(c_{i})}}}$ (3) $$

where $\mathbf{R}$ is empty for prompt classification. Specifically, for binary classification tasks, the target label tokens could simply be “safe / unsafe”, “harmful / unharmful”, or “yes / no”, depending on the specific instructions utilized in different guard models.

## 4 Calibration Measurement of LLM-based guard models

To systematically evaluate the calibration of existing open-source LLM-based guard models across public benchmarks, we conduct an analysis of 9 models on 12 publicly available datasets. We take the prompt classification and response classification as two primary tasks in our investigation. Due to the variability in safety taxonomies across different guard models and datasets, it is challenging to directly compare performance on multiclass prediction tasks. Therefore, our evaluation emphasizes binary classification (safe/ unsafe) for both prompt and response classifications, allowing for a more consistent and fair comparison across guard models. Moreover, binary classification is a critical precursor to multiclass predictions, as an incorrect binary prediction could result in the dissemination of undesired content to users, increasing the associated risk. Thus, binary classification holds particular importance in ensuring the reliability and safety of these systems.

### 4.1 Experimental Setup

Benchmarks. To assess calibration in the context of binary prompt classification, we evaluate performance using a range of public benchmarks, including OpenAI Moderation (Markov et al., ), ToxicChat Test (Lin et al., ), Aegis Safety Test (Ghosh et al., ), SimpleSafetyTests (Vidgen et al., ), XSTest (Röttger et al., ), Harmbench Prompt (Mazeika et al., ) and WildGuardMix Test Prompt (Han et al., ). For the response classification, we utilize datasets containing BeaverTails Test (Ji et al., ), SafeRLHF Test (Dai et al., ), Harmbench Response (Mazeika et al., ), and WildGuardMix Test Response (Han et al., ). For all datasets, we report the ECE as the primary metric for calibration assessment, alongside the F1 score for classification performance. Detailed statistics of each dataset can be found in Appendix A.1.

LLM-based Guard Models. Existing LLM-based guard models vary in their capabilities, with some supporting both prompt and response classification, while others specialize in response classification, based on their instruction-tuning tasks. For prompt classification, we evaluate Llama-Guard, Llama-Guard2, Llama-Guard3, Aegis-Guard-Defensive, Aegis-Guard-Permissive, and WildGuard (Inan et al., ; Ghosh et al., ; Han et al., ). In the case of response classification, we additionally assess Harmbench-Llama, Harmbench-Mistral, and MD-Judge-v0.1 (Mazeika et al., ; Li et al., ). API-based moderation tools are excluded from our evaluation due to the nature of their black-box models, which output scores that cannot be simply interpreted as probability. More details can be found in Appendix A.2.

Table: Table 1: ECE (%) $\downarrow$ performances of prompt and response classification on existing public benchmarks. We bold the best average result and underline the second-best average result for both prompt and response classification.

Figure: Figure 2: Confidence distributions (First row) and reliability diagrams (Second row) of Llama-Guard, Llama-Guard3, Aegis-Guard-P, and WildGuard on the WildGuardMix Test Prompt set.
Refer to caption: x2.png

### 4.2 Main Results

#### 4.2.1 General Evaluation on public benchmarks

We begin by conducting a comprehensive evaluation of both prompt and response classifications for the existing guard models on public benchmarks. The ECE results for both tasks are presented in Table 1. Our experimental findings indicate that existing guard models exhibit significant miscalibration in both prompt and response classifications. Among the models evaluated, WildGuard demonstrates the lowest average ECE for prompt classification, achieving $14.4\%$, while MD-Judge achieves the lowest average ECE for response classification, at $11.4\%$. However, despite the relatively better performances, both Wildguard and MF-Judge exhibit average ECE values exceeding $10\%$, which is typically considered as poor calibration and underscores the need for further improvements. Additionally, each model displays a substantial variance in ECE across different datasets, suggesting unreliable predictions.

Finding 1: Existing guard models tend to make overconfident predictions. To further investigate, we visualize the confidence distributions and present the corresponding reliability diagrams in Figure 2. Additional results for other datasets, models as well as response classification can be found in Appendix A.4. The analysis reveals that for models such as LLama-Guard, Llama-Guard3, and WildGuard, the majority of predictions exhibit confidences between $90\%$ and $100\%$, indicating overconfident predictions along with high ECE. While Aegis-Guard-P shows a less extreme confidence distribution compared to the other models, the proportion of predictions with confidence greater than $90\%$ is still noticeably higher than those with lower confidence, further reflecting the trend of overconfidence.

#### 4.2.2 Evaluation under jailbreak attacks

Figure: Figure 3: F1 (%) $\uparrow$ and ECE (%) $\downarrow$ performances of prompt and response classification on Harmbench-adv set.
Refer to caption: x3.png

Table 1 reveals considerable variability of ECE for different guard models when handling harmful requests on the HarmbenchPrompt set. To further investigate the reliability of these guard models in adversarial environments involving dangerous jailbreak attacks, we extend our evaluation to the Harmbench-adv set. This dataset, which serves as a validation set for fine-tuning Llama2-variant classifiers in Harmbench, includes user inputs generated from various types of jailbreak attacks, such as GCG and AutoDAN, leading to a significant distribution shift from typical user input. In this evaluation, we utilize the adversarial user inputs and their corresponding responses and report the F1 and ECE results for each guard model in Figure 3.

Finding 2: Miscalibration in prompt classification is more pronounced than in response classification under jailbreak attacks. The results demonstrate that the ECE for prompt classification is generally higher than that of response classification, indicating that guard models tend to be more reliable when classifying model responses under adversarial conditions. We conjecture that this may be due to the more considerable distribution shift in adversarial prompts than that in model responses. Additionally, while WildGuard achieves SOTA performance with an F1 score of $92.8\%$ in prompt classification, its ECE score remains high at $34.9\%$, highlighting concerns about the reliability of its predictions in real-world deployment.

Table: Table 2: F1 (%) $\uparrow$ and ECE (%) $\downarrow$ performances of response classification on Harmbench-adv set across 10 different response models.

#### 4.2.3 Evaluation of Robustness to diverse response models

While the ECE for response classification under adversarial environments appears relatively lower in Figure  3, it remains important to investigate whether each guard model consistently maintains reliability when classifying responses generated by different response models. This is crucial because response models are often aligned differently during post-training so they may have different output distributions and produce different responses to jailbreak attacks. To this end, we continue our calibration evaluation under jailbreak attacks, shifting our focus to response classification. Specifically, we employ the same Harmbench-adv set and divide it according to the response model type. After filtering out subsets with a small sample size, we retain 10 subsets containing responses from Baichuan2, Qwen, Solar, Llama2, Vicuna, Orca2, Koala, OpenChat, Starling, and Zephyr. Each subset consists of outputs from a specific response model. Detailed information on the statistics for each subset is provided in Appendix A.1. The F1 and ECE results are reported in Table 2.

Finding 3: Guard models exhibit inconsistent reliability when classifying outputs from different response models. The results in Table 2 reveal significant variance in both F1 and ECE across different response models. This suggests potential limitations in the training of guard models that rely on responses from a single model. For example, Aegis-Guard models are trained using responses from Mistral, and Llama-Guard models are trained using responses from internal LLama checkpoints. In contrast, Harmbench-Llama, Harmbench-Mistral, and Wildguard are trained using responses from a more diverse set of models, leading to improved generalization across different output distributions of response models.

## 5 Improving the Calibration of LLM-based Guard Models

Empirical evidence has demonstrated the miscalibration of current LLM-based guard models, necessitating efforts to improve their reliability through calibration techniques. In this section, we focus on post-hoc calibration methods to circumvent the computational expense associated with training new guard models, reserving training-time calibration approaches for further investigation.

### 5.1 Calibration Techniques

Temperature Scaling (Guo et al., ). Temperature scaling (TS) is a widely employed confidence calibration method for traditional neural networks. By introducing a scalar parameter $T>0$ on the output logits, the output distribution can either be smoothed ($T>1$) or sharpened ($T<1$). Specifically, the calibrated confidence is computed as:

$$ $\hat{p}(y=c_{i}|\mathbf{X},\mathbf{R})=\frac{e^{\frac{z_{\mathcal{V}(c_{i})}}{ T}}}{\sum_{c_{i}}e^{\frac{z_{\mathcal{V}(c_{i})}}{T}}}$ (4) $$

It is important to note that applying temperature scaling does not affect the maximum value of the softmax function, and thus does not alter accuracy performance. The parameter $T$ is typically optimized on a held-out validation set with respect to the negative log-likelihood. However, in the context of the LLM content moderation task, validation sets may not always be available, posing a significant challenge, particularly when addressing in-the-wild user inputs or responses from unknown models. Besides temperature scaling, most conventional calibration methods similarly rely on validation sets to determine parameters, rendering them impractical in such scenarios. As such, we exclusively take temperature scaling as an instance for its simplicity and efficacy.

Contextual Calibration (Zhao et al., ). Contextual calibration (CC) is one type of matrix scaling technique to address contextual bias in LLMs, with the key advantage of requiring no validation set. This method estimates test-time contextual bias by using content-free tokens such as “N/A”, space, or empty tokens. The calibrated prediction is then computed as follows:

$$ $\hat{\mathbf{p}}(y|\mathbf{X},\mathbf{R})=\mathbf{W}\mathbf{p}(y|\mathbf{X}, \mathbf{R})$ (5) $$

where $\mathbf{W}=\mathrm{diag}(\mathbf{p}(y|[N/A]))^{-1}$. Although the original purpose of contextual calibration differs from confidence calibration, the utilized vector scaling modifies model predictions and impacts confidence levels as well, warranting its consideration for confidence calibration.

Batch Calibration (Zhou et al., ). Batch calibration (BC) is also a type of matrix scaling approach. The rationale behind batch calibration is to estimate contextual bias from a batch of $M$ unlabeled samples drawn from the target domains $P(x)$ or $P(x,r)$, rather than from context-free tokens as in contextual calibration. Specifically, batch calibration applies a transformation on the original prediction, which can be interpreted as a linear transformation in the log-probability space,

$$ $\log\hat{\mathbf{p}}(y|\mathbf{X},\mathbf{R})=\log\mathbf{p}(y|\mathbf{X}, \mathbf{R})-\log\mathbf{b}$ (6) $$

where $\mathbf{b}$ is computed in a content-based manner by $\mathbf{b}=-\mathrm{E}_{x\sim P(x)}[\mathbf{p}(y|x)]\approx-\frac{1}{M}\sum_{i
=1}^{M}\mathbf{p}(y|x^{(i)})$ for prompt classification or $\mathbf{b}=-\mathrm{E}_{x,r\sim P(x,r)}[\mathbf{p}(y|x,r)]\approx-\frac{1}{M}
\sum_{i=1}^{M}\mathbf{p}(y|x^{(i)},r^{(i)})$ for response classification. Note that batch calibration requires a batch of unlabeled samples to estimate the contextual prior during test time.

### 5.2 Calibration Results

We apply the calibration methods discussed in Section 5.1 to both prompt classification and response classification for each guard model. For temperature scaling, we utilize the XSTest set as the validation set to optimize the temperature due to its relatively small size. This optimized temperature value is then applied across all other datasets, as individual validation sets are not available for all examined datasets. For contextual calibration, we estimate the contextual bias using a space token. For batch calibration, we assume access to the full test set and estimate the contextual bias using the entire test set as a batch. The resulting calibration performance is reported in Table 3. Details regarding the implementation can be found in Appendix A.3, along with additional calibration results in adversarial environments in Appendix A.5.

Table: Table 3: ECE (%) $\downarrow$ performance comparison of different calibration techniques. For each guard model, we report the original calibration results in the first row and the rest results using TS: Temperature Scaling, CC: Contextual Calibration, BC: Batch Calibration, in the following three rows. We bold the best average result among different calibration techniques for each guard model in both prompt and response classification.

Contextual calibration proves more effective for prompt classification and temperature scaling benefits response classification more. Empirical results indicate that contextual calibration outperforms other methods in prompt classification, delivering improved calibration for the majority of guard models, with the exception of WildGuard. Additionally, temperature scaling effectively reduces the ECE and demonstrates particular effectiveness, despite being optimized on a validation set with a potentially different distribution from the target dataset. This finding further confirms the shared overconfident predictions across datasets and validates that proper temperature values can smooth the overconfident prediction distribution, thereby mitigating miscalibration. Furthermore, temperature scaling shows greater efficacy in response classification which often involves multiple sentences of both user inputs and model responses. In such cases, contextual calibration struggles to accurately estimate contextual prior, resulting in unstable or even degraded calibration performance. Moreover, it is noteworthy that batch calibration underperforms compared to contextual calibration for most models in prompt classification, as well as some models in response classification. We conjecture that this could be attributed to significant label shifts in the test datasets, leading to biased contextual prior estimation and diminished calibration effectiveness. However, no single method fully resolves the miscalibration issues, indicating the complexity of achieving reliable safety moderation across different deployment scenarios.

## 6 Discussion

In this section, to further understand why miscalibration of guard models happens, and how it manifests in prompt and response classification, we conduct two further investigations and point out the limitation and weak robustness of instruction-tuning LLM-based guard models.

Table: Table 4: Probability (%) of “safe” for prompt classification when input is set as a space token or “unsafe” token, and response classification when input and model output are set as a space token.

Prediction Vulnerability Induced by a Single Token. We analyze two specific scenarios by assessing model predictions when the user input consists of a space token or an “unsafe” token and both the user input and model response consist of a space token, respectively. Results of the probability of the input being classified as “safe” are reported in Table 5. The results demonstrate that many guard models exhibit high confidence in predicting “safe” for a space token input. However, the introduction of a single “unsafe” token without further context can cause many guard models to confidently predict “unsafe”. This finding underscores the persistent contextual bias in guard models revealing their limitations even after instruction-tuning. More extensive robustness evaluations of guard models are thus essential for future research.

Misaligned Classification Objectives. We further investigate guard models in the LLama-Guard family capable of both prompt and response classification, focusing on the accuracy of predictions when the model response is set to a content-free token. Specifically, we sample 100 “safe” user inputs and 100 “unsafe” user inputs from the WildGuardTest set and replace all model responses with a space token. We report the average probability of classifying the response as “safe” for using “safe” and “unsafe” user inputs separately in Table 5. The results indicate that the model is more likely to predict the responses as “unsafe” when user inputs (prompt) are unsafe, even when model responses are content-free and should logically be predicted as “safe”. This suggests that the model prediction is heavily influenced by the user input and the guard models act like conducting prompt classification even when response classification should be done. Such behavior can result in unreliable predictions and increased miscalibration.

## 7 Conclusion

In this work, we have systematically examined the uncertainty-based reliability of LLM-based guard models by assessing their calibration levels across various benchmarks. Our analysis reveals that despite their promising performance in content moderation, these models tend to make overconfident predictions, exhibit significant miscalibration under adversarial environments, and lack robustness to responses generated by diverse LLMs. To mitigate miscalibration, we explore several post-hoc calibration techniques. Our results show that contextual calibration proves particularly effective for prompt classification and temperature scaling improves response classification performance more. Our findings underscore the importance of uncertainty-based reliability and advocate for incorporating confidence calibration evaluation in the development and release of future LLM-based guard models.

## Ethical Statement and Broader Impacts

Our work examines the reliability and confidence calibration of existing LLM-based guard models. Despite enhanced confidence calibration for certain guard models, it is essential to emphasize that confidence calibration should not be the sole criterion for determining the suitability of a guard model for deployment. Guard models could potentially make incorrect predictions, particularly when dealing with texts in the wild. We thus advocate for a more holistic reliability evaluation that integrates uncertainty-based confidence calibration with assessments of model robustness and overall performance, and in certain cases, even human-involved factors tailored to specific scenarios.

The broader implications of our study have the potential to drive future research towards the development of better-calibrated guard models. Our insights also contribute to the design of more effective post-hoc calibration techniques, the incorporation of calibration optimization during instruction tuning, and the synthesis of diverse and high-quality data aimed at enhancing both calibration and robustness in the future. Furthermore, our work potentially provides valuable guidance on selecting classifiers that can ensure consistent and reliable evaluations of attack success rate (ASR) in determining whether a jailbreak attack has succeeded.

## Reproducibility Statement

In support of reproducibility, our submission includes the full code implementation, along with comprehensive instructions in the README.md file detailing the steps required to install the necessary environments and run our experiments. Additionally, we provide all relevant information regarding publicly available datasets and models, enabling interested researchers can replicate the results presented in this paper.

## Appendix A Appendix

### A.1 Dataset Details

In this section, we briefly describe the public datasets we examined and show the statistics in Table 6.

**Table 6: Statistics of datasets we used.**
| Dataset | # Prompt | # Response |
| --- | --- | --- |
| safe/unsafe | safe/unsafe |  |
| Prompt Classification |  |  |
| OpenAI Moderation | 1158/522 | - |
| ToxicChat Test | 2491/362 | - |
| Aegis Safety Test | 126/233 | - |
| SimpleSafetyTests | 0/100 | - |
| XSTest | 250/200 | - |
| Harmbench Prompt | 0/239 | - |
| WildGuardMix Test Prompt | 971/754 | - |
| Response Classification |  |  |
| BeaverTails | - | 894/1106 |
| SafeRLHF | - | 1000/1000 |
| Harmbench Response | - | 326/270 |
| WildGuardMix Test Response | - | 1440/285 |

OpenAI Moderation (Markov et al., ). This dataset contains 1680 prompts with labels for 8 unsafe categories including sexual, hate, violence, harassment, self-harm, sexual/minors, hate/threatening, and violence/graphic. Each category label is a binary flag.

ToxicChat Test (Lin et al., ). We use the test split of the new version toxicchat0124, involving 2853 user prompts collected from the Vicuna online demo(^3^33https://lmarena.ai/), each annotated with binary toxicity labels.

Aegis Safety Test (Ghosh et al., ). This dataset is built on the prompts from HH-RLHF and responses generated by Mistral-7B-v0.1 with human annotations. We utilize the prompt-only subset, with a size of 359, from the test split of the dataset. The absence of the turn-level split of utterances during the conversation makes it infeasible for response classification evaluation. This dataset covers 13 unsafe content categories according to NVIDIA’s content safety taxonomy including Hate/Identity Hate, Sexual, Violence, Suicide and Self Harm, Threat, Sexual Minor, Guns/Illegal Weapons, Controlled/Regulated substances, Criminal Planning/Confessions, PII, Harassment, Profanity, Other. The “Needs Caution” category is also involved for uncertain cases.

SimpleSafetyTests (Vidgen et al., ). This dataset involves 100 manually-crafted harmful prompts with topics in Suicide, Self-Harm and Eating Disorders, Physical Harm and Violence, Illegal and Highly Regulated items, Scams and Fraud, Child Abuse.

XSTest (Röttger et al., ). This dataset contains 250 safe prompts and 200 unsafe prompts. Safe prompts use similar language to unsafe prompts or mention sensitive topics but they are clearly safe and should not be refused. Binary labels are provided in this dataset.

Harmbench Prompt (Mazeika et al., ). This dataset is designed for robustness to jailbreak attacks with prompts for eliciting harmful outputs from LLMs. We use the “standard” and “copyright” subsets, with a total size of 239, from the test split of the dataset in our evaluation for LLM-based guard models on prompt classification. The topics of unsafe prompts include Cybercrime & Unauthorized Intrusion, Chemical & Biological Weapons/Drugs, Copyright Violations, Misinformation & Disinformation, Harassment & Bullying, Illegal Activities, General Harm.

Harmbench Response (Mazeika et al., ). This dataset refers to a variant of the validation set used for fine-tuning Llama2-variant from Harmbench, which consists of 602 responses generated by various models and jailbreak attacks. We use the pairs of their vanilla prompts and model responses with human labeling for response classification, resulting in a set of 596 pairs.

Harmbench-adv (Mazeika et al., ). This dataset refers to the original validation set with a size of 602 for fine-tuning Llama2-variant from Harmbench. We term it “Harmbench-adv” to differentiate it from “Harmbench Response” given that adversarial prompts from diverse attack methods are involved in the Harmbench-adv set. Adversarial prompts could be very different from vanilla ones. We further split this dataset in terms of the type of response models and retain 10 subsets with statistics in Table 7.

BeaverTail Test (Ji et al., ). We utilize the test split of this dataset with 33.4k prompt-response pairs, which contain manually annotated labels for model response harmfulness. The prompts are modified from the HH-RLHF and Safety-Prompts, while the responses are generated with the Alpaca-7B model. The 14 harm categories involve Animal Abuse, Child Abuse, Controversial Topics & Politics, Discrimination & Stereotype & Injustice, Drug Abuse & Weapons & Banned Substance, Financial Crime & Property Crime & Theft, Hate Speech & Offensive Language, Misinformation Regarding ethics & laws & safety, Non-Violent Unethical Behavior, Privacy Violation, Self-Harm, Sexually Explicit & Adult Content, Terrorism & Organized Crime, Violence & Aiding and Abetting & Incitement. We use a subset of 2k size randomly sampled from the original test split to reduce the evaluation cost.

SafeRLHF Test (Dai et al., ). The dataset shares the prompts with the BeaverTails dataset and generates responses from Alpaca-7B, Alpaca2-7B, and Alpaca3-8B. The 19 harm categories include Endangering National Security, Insulting Behavior, Discriminatory Behavior, Endangering Public Health, Copyright Issues, Violence, Drugs, Privacy Violation, Economic Crime, Mental Manipulation, Human Trafficking, Physical Harm, Sexual Content, Cybercrime, Disrupting Public Order, Environmental Damage, Psychological Harm, White-Collar Crime, Animal Abuse. We use a subset of 2k size randomly sampled from the original test split to reduce the evaluation cost.

WildGuardMix Test (Han et al., ). This dataset contains 1725 samples with synthetic, in-the-wild user-LLM interactions and annotator-written data. Responses to synthetic and vanilla prompts are generated using a suite of LLMs. We consider the prompt harmfulness and response harmfulness annotations in our evaluations. WildGuardMix Test Prompt and WildGuardMix Test Response refer to the prompts data and prompt+response pairs data for prompt and response classification, respectively.

Table: Table 7: Statistics of subsets across 10 different response models.

### A.2 Model Details

We summarize the hugging face model cards of 9 LLM-based guard models we examined in Table 8. Note that we do not assess the series of ShieldGemma(^4^44https://huggingface.co/google/shieldgemma-2b) models given that they only support classification for a single policy per inference, making public datasets infeasible for evaluation due to the policy difference.

**Table 8: Hugging Face Model Cards for examined LLM-based guard models**
| Guard Models | Hugging Face page |
| --- | --- |
| Llama-Guard-7B | [meta-llama/LlamaGuard-7b](https://huggingface.co/meta-llama/LlamaGuard-7b) |
| Llama-Guard2-8B | [meta-llama/Meta-Llama-Guard-2-8B](https://huggingface.co/meta-llama/Meta-Llama-Guard-2-8B) |
| Llama-Guard3-8B | [meta-llama/Llama-Guard-3-8B](https://huggingface.co/meta-llama/Llama-Guard-3-8B) |
| Aegis-Guard-Defensive-7B | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) |
| Aegis-Guard-Permissive-7B | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0) |
| Harmbench-Llama2-13B | [cais/HarmBench-Llama-2-13b-cls](https://huggingface.co/cais/HarmBench-Llama-2-13b-cls) |
| Harmbench-Mistral-7B | [cais/HarmBench-Mistral-7b-val-cls](https://huggingface.co/cais/HarmBench-Mistral-7b-val-cls) |
| MD-Judge-v0.1-7B | [OpenSafetyLab/MD-Judge-v0.1](https://huggingface.co/OpenSafetyLab/MD-Judge-v0.1) |
| WildGuard-7B | [allenai/wildguard](https://huggingface.co/allenai/wildguard) |

### A.3 Implementation Details

We use Pytorch and Huggingface Transformers in our implementation. We run all evaluations on a single NVIDIA A40 GPU (48G). We use $M=15$ bins as in Guo et al. () for all our ECE evaluations. For temperature scaling, we optimize the $T$ within the range from (0, 5]. For batch calibration, we set the batch size as the size of the entire test set by default following  Zhou et al. (). For prompt classification, we keep the original prompt lengths for most datasets except OpenAI Moderation where we truncate a few samples with extremely long lengths to avoid the out-of-memory error. We keep the maximum length as 1800. For response classification, we keep the original prompt length for all datasets and set the maximum response length as 500.

### A.4 More Reliability Diagrams

We report the full reliability diagrams with corresponding confidence distributions of all 9 models we examined for both prompt classification and response classification on the WildGuardMix Test Prompt set, shown in Figure 6, the WildGuardMix Test Response set, shown in Figure 6, 6, the Harmbench Prompt set, shown in Figure 9, and the Harmbench Response set, shown in Figure 9, 9. The full results illustrate that existing LLM-based guard models exhibit varying levels of miscalibration. Models including LLama-Guard, LLama-Guard2, LLama-Guard3, WildGuard, Harmbench-Llama, and Harmbench-Mistral tend to make overconfident predictions, while Aegis-Guard-D, Aegis-Guard, and MD-Judge-v0.1 are not.

Figure: Figure 4: Confidence distributions (First row) and reliability diagrams (Second row) on the WildGuardMix Test Prompt set.
Refer to caption: x4.png

Figure: Figure 7: Confidence distributions (First row) and reliability diagrams (Second row) on the Harmbench Prompt set.
Refer to caption: x7.png

### A.5 More Calibration Results under Adversarial Conditions

We conduct more calibration experiments using temperature scaling, contextual calibration, and batch calibration on the Harmbench-adv set to mitigate the miscalibration discussed in Section 4.2.2, and Section 4.2.3. We keep the same settings and implementations as those in the main text. The ECE results are presented in Table 9 and Table 10, respectively. These additional empirical findings indicate the same conclusion as in the main text, that contextual calibration proves impressively effective on prompt classification, while temperature scaling benefits more on response classification.

Table: Table 9: ECE (%) $\downarrow$ performance comparison of different calibration techniques. For each guard model, we report the Origin: original results, TS: Temperature Scaling, CC: Contextual Calibration, BC: Batch Calibration. We bold the best average result among different calibration techniques in both prompt and response classification.

Table: Table 10: ECE (%) $\downarrow$ performance comparison of different calibration techniques. For each guard model, we report the original calibration results in the first row and the rest results using TS: Temperature Scaling, CC: Contextual Calibration, BC: Batch Calibration, in the following three rows. We bold the best average result among different calibration techniques for each guard model.

### A.6 Instruction Prompts

Instruction prompts for all LLM-based guard models we examined can be found in our submitted code implementation or their Huggingface model cards in Table 8.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: + Were the guard models sampled from or is it using greedy decoding?
soundness: 2
strengths: + This is an extensive and comprehensive evaluation of a variety of calibration methods, models, and datasets. It clearly took a lot of effort.
+ Overall, this highlights the poor calibration of guard models and  could be good motivation for more consistent methods. The eval harness could also be the building block for an evaluation suite to test some improved methods down the line.
summary: This work examines how calibration can affect and potentially improve LLM-based guard models. The study finds most guard models are poorly calibrated, especially under jailbreaking attacks, but that off the shelf calibration methods don't seem to provide consistent calibration benefits (at least not as tested).
weaknesses: + The effect sizes for some of the calibration methods on some datasets were rather small and it’s a bit unclear what the potential variance here is. It would be great to have a table 3 with confidence intervals, though given the size of the table this might be a heavy lift. This becomes more important if there was sampling at test time and less important if there was greedy decoding (see question below).
+ It’s not clear that batch calibration didn’t work as well because of the selection of the batch of unlabeled samples or other choices. I don’t think this is a major issue, but should be called out more prominently as a limitation. Similarly for other decisions and methods. This is done to some extent, but a standalone limitations section might be warranted. Overall there seem to be some major caveats for design decisions as to the generalizability of the takeaways from the calibration method study.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1: For results on jailbreak prompts, the author said that "The results demonstrate that the ECE for prompt classification is
generally higher than that of response classification, indicating that guard models tend to be more reliable when classifying model responses under adversarial conditions. " However, it's not clear whether the guard model is vulnerblae due to the jailbroken nature of the prompts, or due to some suprrious correlations (eg length, patterns). It will be great if the authors can explain or design experimetns to ablate such factors.

Q2: It's interesting to see the variability in guard model performance across different response models (Table 2). However, it would be more insightful to understand the causes of these discrepancies. For example, why do all the models perform particularly poorly with Llama2's responses? Is there a qualitative or quantitative explanation for this?

Q3: Regarding the calibration results in Table 3, the improvements appear relatively modest (e.g., at most around 2% ECE reduction). It would be helpful to contextualize how significant these improvements are. Additionally, it seems that contextual calibration (CC) and batch calibration (BC) sometimes degrade performance. Understanding the reasons behind this would provide valuable insights.

Q4: Most of the evaluated models in Table 2 are open-weight models, so it’s unclear how these findings would transfer to proprietary models like ChatGPT, Gemini, and Claude.
soundness: 3
strengths: The paper conducts an in-depth analysis of LLM-based guard models and explores potential design improvements.
summary: The paper examines the reliability and calibration of guard models based on LLMs used in content moderation. The authors highlight that while these models achieve strong classification performance, they often produce overconfident and poorly calibrated predictions, particularly when faced with adversarial attacks like jailbreaks. Through an empirical evaluation of nine guard models across 12 benchmarks, the study identifies significant calibration issues, such as overconfidence and inconsistent robustness across different response models. To address these challenges, the paper explores post-hoc calibration techniques, demonstrating the effectiveness of temperature scaling for response classification and contextual calibration for prompt classification. The findings underscore the importance of improving model calibration to enhance the reliability of guard models in real-world content moderation scenarios.
weaknesses: Overall, I think the paper does a good job of presenting the "what"—namely, the findings and results—but it would benefit from delving deeper into the "why," or the reasons behind these observations (Sec 6 has some "understanding" results, but seems to be distantly related). Without this, the paper feels more like a dataset and benchmark track submission (which ICLR does not specifically have) rather than a main track paper.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What are some of the limitations of this experimental setup? How do these limitations affect the resultant outputs and findings?
2. Have you tried the setup on a larger model that is being prompted to act as a guardrail? It would be interesting as a comparison point to these guardrail-specific models.
soundness: 3
strengths: - Figure 1 is well thought out and easy to follow. sets the stage well
- the experimental setup is solid, with a variety of benchmarks that are used in industry. In particular, the use of benchmarks for which typical statistics are provided for these guardrail models is smart. 
- the breadth of guardrail models used is admirable
summary: The proposed study conducts investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. The resultant findings are that these guard models are overconfident, are miscalibrated with respect to jailbreak attacks, and have limited robustness.
weaknesses: - discussion of limitations is lacking, would be interesting to see where the pitfalls of this approach are and how they could be improved.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - More discussion on the generalizability of the techniques.
- Can you show the statistics for false positives and false negatives as well? Would be useful to know that which models are showing what types of over-confidence behavior.
- What is the “unsafe” token used for experiments in the discussion section?
- Could you provide more explanation or intuition on the calibration trade-offs across models? Why certain methods are better for response classification while some work better for prompt classification. 
- How does the size / training data characteristics affect calibration? It would be better to understand why certain methods work better for certain scenario.
soundness: 3
strengths: - The paper points out an important yet under-studied problem of (over) confidence in LLM safety guard models and analyzes the confidence calibration problem. 
- The evaluation covers a wide range of models and benchmarks for evaluation. 
- The work finds that lightweight approach like contextual calibration can be effective mechanisms for improving confidence calibration. 
- The paper is written clearly and the overall flow is easy to follow.
summary: This papers conducts an analysis on the reliability and calibration of LLM-based guard models for safety moderation. The findings reveal that these guard models tend to be overconfident in predictions, show miscalibration when subjected to jailbreak attacks, and different response models also have different calibration performance. Based on these insights, the authors also propose some easy-to-implement methods to improve calibration.
weaknesses: - The proposed calibration techniques does not show strong improvement on the ECE metrics, and in some cases even make the ECE score higher (Table 3). There is no statistical significance tests (eg. Multiple runs, variance, hypothesis testing) to show that the methods are indeed beneficial. 
- CC is primarily designed for binary or few-class settings, and the framework appears to be challenging to extend to a multi-class setup. The influence of a content-free token might not translate well to all classes, especially if some classes are rare or highly specialized. It will also be harder to interpret and apply, because the baseline bias captured from a content-free input could vary inconsistently across datasets. 
- The assumptions for the BC method are not realistic in actual LLM setting. It also may inadvertently adapt to an adversarial distribution shift. 
- For temperature scaling, the authors used XSTest as the validation set for optimization. However, since XSTest is a dataset for estimating over-refusal, there is likely bias in the temperature optimized on it. 
- The authors do not discuss the distribution of safe/ unsafe prompts in the dataset being studied. The ECE metric could be affected by dataset imbalance.

Meta-review: This is a solid empirical paper measuring present-day SotA LLM-based guardrails' (that classify prompts into e.g. jailbreak attempt vs not, and classify responses into e.g. toxic content or not) uncertainty quantification and calibration.  All reviewers appreciated just how comprehensive that task of benchmarking was, and to a lesser extent appreciated some of the proposed fixes to move these methods more toward being well-calibrated.  That latter half received less support from the reviewers, but the lift and value of the descriptive study is high.  While ICLR does not have a specific Datasets & Benchmarks style of track, the conference does have a history of supporting empirical-only work (even moreso than e.g. NeurIPS and ICML).

Decision: Accept (Poster)