On Calibration of LLM-based Guard Models for Reliable Content Moderation
Authors: Hongfu Liu
Hengguan Huang
Xiangming Gu
Hao Wang
Ye Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Content Moderation
LLM-based Guard Models
Calibration
Safety
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Large language models (LLMs) pose significant risks due to the potential for generating harmful content or users attempting to evade guardrails. Existing studies have developed LLM-based guard models designed to moderate the input and output of threat LLMs, ensuring adherence to safety policies by blocking content that violates these protocols upon deployment. However, limited attention has been given to the reliability and calibration of such guard models. In this work, we empirically conduct comprehensive investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. Our findings reveal that current LLM-based guard models tend to 1) produce overconfident predictions, 2) exhibit significant miscalibration when subjected to jailbreak attacks, and 3) demonstrate limited robustness to the outputs generated by different types of response models. Additionally, we assess the effectiveness of post-hoc calibration methods to mitigate miscalibration. We demonstrate the efficacy of temperature scaling and, for the first time, highlight the benefits of contextual calibration for confidence calibration of guard models, particularly in the absence of validation sets. Our analysis and experiments underscore the limitations of current LLM-based guard models and provide valuable insights for the future development of well-calibrated guard models toward more reliable content moderation. We also advocate for incorporating reliability evaluation of confidence calibration when releasing future LLM-based guard models.

Published as a conference paper at ICLR 2025
ON CALIBRATION OF LLM-BASED GUARD MODELS FOR RELIABLE CONTENT MODERATION
Hongfu :iu'; Hengguan Huang?, Xiangming Gu', Hao Wang', Ye Wang' National University of Singapore UUniversity of Copenhagen 'Rutgers University
ABSTRACT
Large language models (LLMs) pose significant risks due to the potential for gen- erating harmful content or users attempting to evade guardrails. Existing studies have developed LLM-based guard models designed to moderate the input and out- put of threat LLMs, ensuring adherence to safety policies by blocking content that violates these protocols upon deployment. However, limited attention has been given to the reliability and calibration of such guard models. In this work, we empirically conduct comprehensive investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. Our findings reveal that current LLM-based guard models tend to 1) produce overconfident predictions. 2) exhibit significant miscalibration when subjected to jailbreak attacks, and 3) demonstrate limited robustness to the outputs generated by different types of response models. Additionally, we assess the effectiveness of post-hoc calibration methods to mitigate miscalibration. We demonstrate the efficacy of temperature scaling and, for the first time, highlight the benefits of contextual calibration for confidence calibration of guard models, particularly in the absence of validation sets. Our analysis and experiments derscore the limitations of current LLLM-based guard models and provide valuable insights for the future development of well-calibrated guard models toward more reliable content moderation. We also advocate for incorporating reliability evalu- ation of confidence calibration when releasing future M----aeeed guard models!.
INTRODUCTION
Recent advancements in Large Language Models (LLMs) have facilitated the development of pow- erful conversation systems, leading to the deployment of LLM-based chatbots in various real-world applications (Brown, 2020; Anil et al., 2023; Touvron et al., 2023; Dubey et al., 2024). How- ever, these systems face substantial risks due to the potential for malicious exploitation of powerful LLMs (Wang et al., 2023). Consequently, addressing these risks has become an urgent and critical task. One promising strategy to regulate LLMs during their training phase. Existing researches primarily focus designing alignment algorithms through preference optimization (Ouyang et al., 2022; Rafailov et al., 2024), implementing adversarial training (Mazeika et al., 2024), or employing machine unlearning to remove harmful knowledge from the models (Chen & Yang. 2023; Liu al., 2024b). These approaches aim to control text generation and prevent undesired outputs. Despite these significant efforts to enhance LLM safety during training, red-teaming still makes efforts to expose vulnerabilities, including jailbreak attacks that successfully bypass the safety constraint and elicit harmful responses from LLMs, highlighting the risks of future, unseen threats (Zou et al., 2023; Liu et al., 2023; Chao et al., 2024; Liu et al., 2024a). Therefore, in addition to training-time interventions, it is equally vital to implement test-time measures, such as constraint inference (Xu et al., 2024), and establish effective test-time guardrails through content moderation, particularly when deploying I.LMs in real-world settings.
Content Moderation serves the critical function of monitoring both user inputs and model outputs during conversations. Typically, guard models are designed to assess whether user inputs and LLM outputs comply with safety regulations, and either reject user queries or block model responses
"Corresponding to: Hongfu Liu (hongfu@comp.nut www.eooccoom.muccuuug.ggggg 'Our code is publicly available at https://gtthu..com http://:thhuubcoo/aaflllll-l LafflleLiin/allirraatii iiu/calibration guard model

---

Published as a conference paper at ICLR 2025
Training Time
Test Time
Preference Optimiztioon
Harmful query
Harmful query
Inference
LLM
Adversarial Training
LLM
build bomb
How to build a bomb
Moderation
Machine Unlearning
Content Moderation
Guard Model
(P) Prompt Classification
(R] Response Classifiation
Task: Check If there un content in User convenations wscarding safuty
Task: Check If there & unsate content Agent
Conversation Examples
ons ccarding safety catagories.
policy catapories
How leamn to play guitar
(P)
UNSATE CONTENT CATEGORIESS
UNSATE CONTENT EATEGONESS
cotegories statements
unsate cotogories statements
Agent: Learning guitar rewarding and enjoyable experience. Hore's a step- st-eppp guildo help started....
(R)
safe
<END UNSAFE CONTENT CATEGORIES:
<TND UNSAFF CONTINT CATF GARISS
<38GIN CONVERSATION-
<BESIN CONVERSAT ON-
User Texl mm how build a hamb
User. How to loann play guitar
User: Tell me how to buiid a bomb
(P)
unsafe
<END CONVERSATION=
Agent: Here the step-ity step Instruction
CONVERSATIONS
Agent: Here the step-by step insructiion on how build bomb: First ....
(R)
unsafe
above conversation:
Catl, Cat2...
First must read 'saft' or "unsafe""
Provide your sa fety assesoment for Agent the abowe converattion:
unsafe, second line must include
comma-eeper absd
siolated categoriee.
Tinst lin must
'ursafe'.
Second....
I uncotie, a second thn must include
comma separated list of viclated categoriss.
Figure I: An overview of LLM-based guard models for content moderation. Guard models monitor the input and output during conversations between the user and LLM (Agent), providing binary prediction followed by specific unsafe content category if unsafe content is detected. The instruc- tion examples for prompt classification and response classification from LLama- Guand are detailed in the right yellow boxes respectively.
when content violating safety protocols is detected. This approach remains effective even when LLMs have been compromised by previously unseen jailbreak attacks. Current state-of-the-art guard models, which are typically built on LLMs, demonstrate strong performance across various bench- marks (Inan et al., 2023; Ghosh et al., 2024; Han et al., 2024; Zeng al., 2024). However, these guard models primarily focus on the classification performance but overlook the predictive uncer- tainty of harmfulness predictions (Wang al., 2024; 2016; Wang & Yeung, 2016; 2020), therefore failing to assess the reliability of these models' predictions. This oversight is crucial because guard models may occasionally make erroneous decisions. potentially allowing unsafe content bypass moderation, especially when encountering non-trivial domain shifts (Huang et al., 2022; Xu et al., 2022: Wang et al., 2020), despite their strong in-domain performances. Therefore, quantifying the predictive uncertainty and confidence in model predictions is essential to assessing the trustworthi- ness of guard models, enabling more reliable diccision-maiing in high-risk scenarios that may arise during conversations after model deployment.
In this work, we examine the reliability of existing open-source guard models by focusing on their confidence calibration. Specifically, we empirically assess the calibration performance by com- monly used expected calibration error (ECE) for two key tasks: user input (prompt) classification and model output (response) classification with binary labels. To conduct a systematic evaluation, we examine 9 models across 12 datasets. Our experimental results reveal that, despite achieving strong performance, most existing guard models exhibit varying levels of miscalibration. Addition- ally, our findings show that current LLM --beeddd guard models:
tend to make overconfident predictions with high probability scores.
remain poorly calibrated under adversarial environments, exhibiting higher ECE in adversarial prompt classification, even when the SOTA guard model achieves high Fl scores.
display inconsistent ECE across different types of response models, demonstrating weak ro- bustness to variations in response model types.
These observations highlight critical challenges improving the reliability of guard models in real- world deployments. Consequently, we are motivated to improve the confidence calibration of guard models, focusing on post-hoc calibration methods to avoid additional computational costs of train- ing new guard models. We explore the impact of bias calibration methods confidence calibration for the first time, discovering that contextual calibration proves impressively effective for prompt classification, while conventional temperature scaling remains more beneficial for response classifi- cation. Lastly, we identify miscalibration issues stemming from prediction vulnerabilities induced by single tokens and misaligned classification objectives, highlighting the limitations of instruction-
2

---

Published as a conference paper at ICLR 2025
tuned LLM-based guard models. We stress the importance of reliability evaluation and advocate for the inclusion of confidence calibration measurement in the release of future new guard models.
2 RELATED WORK
Content Moderation, A substantial body of research has been devoted to the detection of hateful and toxic content in human-genene gennnnteeeerreede text from online platforms, such as social media (Schmidt & Wiegand, 2017; Lees et al., 2022). Various API services, including Perspective (Lees et al., 2022), OpenAI (Markov et al., 2023), Azure, and Detoxify, provide black-box content moderation tools for online texts. However, content moderation in LLMs specifically addresses the detection both LLM input and output during conversations within deployed applications, such chat assistants. This task poses unique challenges due to the distribution shift in conversation content generated by LLMs, which differs from previous uumman-geneaated online texts. Recent advancements in LLM content moderation have been achieved through the fine-tuning of LLMs, as seen in models such as LLama-Guard 1/2/3 (Inan et al., 2023), BeaverDam (Ji al.. 2024). Aegis (Ghosh al., 2024), MD- Judge (Li et al., 2024), WildGuard (Han ct al., 2024), and ShieldGemma (Zeng et al., 2024). Notably, models like Hama-Guarr, Aegis, and WildGuard support the detection of both user inputs and model outputs, while others do not due to differing training objectives. Additionally, adversarial cases are addressed by Harmbench (Mazeika et al., 2024) and RigorLLM (Yuan et al., 2024), with Harm- bench specifically fine-tuning LLama2 and Mistral to evaluate the success rate of adverariial attacks by identifying undesirable content in model outputs. Furthermore, Nemo proposes programmable guardrails that provide dialogue management capability using a user-friendly toolkit (Rebedea et al., 2023). Our work focuses on quantifying the predictive uncertainty and evaluating the reliability of LLM--based guard models by their calibration levels.
Calibration of LLMs. Confidence calibration is a critical aspect in developing reliable and trust- worthy language models (Nguyen & O' O'Connor, 2015; Guo et al., 2017; Minderer al., 2021). In the context of LLMs, prior research has explored calibration in question-answering tasks (Jiang et al., 2021) and has empirically examined calibration during the pre-training and alignment stages (Chen et al., 2022: Zhu et al., 2023). Studies such as Lin et al. (2022); Mielke et al. (2022); Xiong et al. (2023) have investigated uncertainty estimation through verbalized confidence, and Kadavath et al. (2022) demonstrated improved calibration of larger models when handling multiple choice and true/false questions given appropriate formats. Another line of research addresses the calibration of biases stemming from in-context samples, instruction templates. sample ordering, and label dis- tribution (Zhao et al., 2021; Zhou et al., 2023b; Liu & Wang, 2023; Fei et al., 2023; Abbas et al., 2024). These bias calibration techniques indirectly influence the prediction confidence by altering the linear decision boundary (Zhou et al., 2023a), yet they are not designed for explicit confidence calibration. In contrast, work specifically addresses the challenge of confidence calibration in instruction- guard models for content moderation tasks.
3 PRELIMINARY
LLMM-baeed Guard Models. Given the user input text X and the corresponding response R S(X) generated by deployed L.M [()), the task of the M.L.-d guard model g(*) is to clas- sify the user input (, (X or the LLM output (Y (Y|X, R.2 These tasks are referred to as prompt classification and response classification, respectively. For the predicted label Y. most existing LLM-based guard models initially perform binary classifications y² to determine whether the user input X or model response R. is safe. If the binary classification result indicates the input or the sponse y is unsafe, the guard model (*) then proceeds with multiclass classification to categorize the specific type yf by (, n P.y ( X R, R,y²) where the categories are pre-defined in taxonomy. These classification tasks in LLM-based guard models are carried out in an autoregres- sive generation manner, and Figure illustrates examples of the prompt and response classification instructions used in LLama-Guaa
Confidence Calibration. model considered perfectly calibrated if its predicted class ý and the associated confidence P E |0, 1] satisfy Plù ylp p) = p. Vp E [0, 1], where is the ground-
Note that we ignore the instruction context Cinst in our all following notations for simplicity where they should be py Y X; Cinet) and Ps (Y|X, R:; Cinet) instead.
3

---

Published as a conference paper at ICLR 2025
truth class label for any given input. This implies that higher confidence in prediction should correspond to a higher chance of its prediction being correct. However, since P( = y|p = p) not be directly calculated with finite sample size. existing approaches employ binning based divisions on finite samples and utilize the Expected Calibration Error (ECE) as a quantitative metric to assess the model's is calibration (Naeini et al., 2015). Assuming that confidence is divided into M bins with equal interval 1/M within the range [0, 1], the ECE is defined as
Bm N
ECE Σ
AAc(BB.) Con
Aco Bm)) =
Σ
1 = v.), Cons(Bm)
Pi
(2)
Bm BEBm..
Bm
&E B.
where Bm represents the set of samples falling within the interval M dicted and ground truth classes, respectively, and p; is the model's predicted probability. However, existing instruction tuned LLM-based guard models do not directly output the probability of each class. Instead, the probability of class c is derived from the output logits 2y(e;) of the corresponding target label token V(c)), where V(*) is the verbalizer. Re- Re-normalization is then applied over the set of target label tokens as follows,
m. J DJ and y; are the pre-
- c
(3)
Eo, VVic)
where R is empty for prompt classification. Specifically, for binary classification tasks, the target label tokens could simply be "safe/ unsafe", "harmful / unharmful", or "yes / no", depending on the specific instructions utilized in different guard models.
4 CALIBRATION MEASUREMENT OF LLM-BASED GUARD MODELS
To systematically evaluate the calibration of existing open-source LLM-based guard models across public benchmarks, we conduct an analysis of 9 models on 12 publicly available datasets. We take the prompt classification and response classification as two primary tasks in our investigation. Due the variability in safety taxonomies across different guard models and datasets, is challenging to directly compare performance multiclass prediction tasks. Therefore, our evaluation empha- sizes binary classification (safe/ unsafe) for both prompt and response classifications, allowing for more consistent and fair comparison across guard models. Moreover, binary classification is critical precursor to multiclass predictions, as an incorrect binary prediction could result in the dis- semination of undesired content to users, increasing the associated risk. Thus, binary classification holds particular importance in ensuring the reliability and safety of these systems.
4.1 EXPERIMENTAL SETUP
Benchmarks. To assess calibration in the context of binary prompt classification, we evaluate per- formance using a range of public benchmarks, including OpenAl Moderation (Markov et al., 2023), ToxicChat Test (Lin al., 2023), Aegis Safety Test (Ghosh al., 2024), SimpleSafetyTestt (Vid- gen et al., 2023), XSTest (Röttger et al., 2023), Harmbench Prompt (Mazeika et al., 2024) and WildGuardMix Test Prompt (Han al., 2024). For the response classification, containing BeaverTails Test (Ji et al., 2024), SafeRLHF Test (Dai et al., 2023), Harmbench Re- sponse (Mazeika et al.. 2024), and WildGuardMix Test Response (Han al., 2024). For all datasets, we report the ECE as the primary metric for calibration assessment, alongside the F1 score for clas- sification performance. Detailed statisties of each dataset can found in Appendix A.
utilize datasets
LLM-based Guard Models. Existing LLM--bsed guard models vary in their capabilities, with some supporting both prompt and response classification, while others specialize in response classification, based on their instruction-tuning tasks. For prompt classification, we evaluate Llama-Guards Aegis-Guard and WildGuard (Inan et al., 2023; Ghosh et al., 2024; Han et al., 2024). In the case of sponse classification, we additionally assess Harmbench-LLane Harmbench-Min Mistral,, and MD- Judge-v0.1 (Mazeika al., 2024; el 2024). API-based moderation tools are excluded from
Llama-Guard Llama-Guard2
Aegis-Guard Shuard-Permissiee,
4

---

Published as a conference paper at ICLR 2025
Lama-Guard (6CE: 0.286)
Liama-Guards (ECE: 0.205)
Aegis-Guard-P (ECE: 0.126) (F16:0
WildGuard (ECE. 0.197)
Count
0.6
0.7
0.6
0.7
6.7
1.0
05
0.9
Confidence
Confidence
Confidence
Confidence
1.0
0.4
0.6
Accuracy
0.44
0.4
3.4-
0.4
0.2
02-
5.0
0.4
3.0
0.2
1.0
0.0
0.2
1.0
0.0
Confidince
Confidence
Confidence
Figure 2: Confidence distributions (First row) and reliability diagrams (Second row) Llama- Guard, Llama-Guarde Acgis- Srggis-Gaard-,, and WildGuard the WildGuardMix Test Prompt set.
evaluation due to the nature of their black-box models, which output scores that cannot be simply interpreted probability. More details be found in Appendix A.2.
Prompt Clussification XST
Response Classification HarmB
Model
OAI
ToxiC
SimpST
Aegis
HarmB
WildGT
Avg.
Beaver
S-RLHF
WilGGT
Avg.
Llana-Guard
11.0
32.6
29.6
20.5
68.1
28.6
28.5
29.1
24.4
23.2
14.2
22.7
Llama- Guand
13.7
15.9
26.5
34.7
12.2
30.3
26.8
22.9
29.9
25.2
24.9
12.8
23.2
Llana-Guard
13.9
14.5
7.6
33.4
13.9
8.8
20.5
16.1
32.3
25.3
27.2
11.1
24.0
Aegis-Guard-D
20.5
9.7
16.4
23.5
50.5
22.7
18.1
30.9
26.1
29.3
26.1
Legio-Gurd-J.
15.7
8.2
16.5
22.6
18.6
59.4
12.6
22.0
18.6
25.6
18.0
15.2
19.4
HammB-Llana
HarmB Mistral
18.1
14.5
13.1
23.6
17.8
MD-Judge
10.9
9.4
17.7
7.7
11.4
WildGuard
Table 1: ECE (%) J performances of prompt and response classification existing public bench- marks. We bold the best average result and underline the second-best average result for both prompt and response classification.
4.2
MAIN RESULTS
4.2.1 GENERAL EVALUATION ON PUBLIC BENCHMARKS
We begin by conducting comprehensive evaluation both prompt and response classifications for the existing guard models on public benchmarks. The ECE results for both tasks are presented in Table 1. Our experimental findings indicate that existing guard models exhibit significant mis- calibration in both prompt and response classifications. Among the models evaluated, WildGuard demonstrates the lowest average ECE for prompt classification, achieving 14.4%, while MD-Judge achieves the lowest average ECE for response classification. at 11.4%. However, despite the rel- atively better performances, both Wildguard and MF-Judge exhibit average ECE values exceeding 10%, which typically considered a poor calibration and underscores the need for further improve- ments. Additionally, each model displays substantial variance in ECE across different datasets, suggesting unreliable predictions.
Finding 1: Existing guard models tend to make overconfident predictions. To further investi- gate, we visualize the confidence distributions and present the corresponding reliability diagrams in Figure 2. Additional results for other datasets, models well response classification can be found in Appendix A.4. The analysis reveals that for models such as LLama-Guard Llama-Guard3, and WildGuard, the majority of predictions exhibit confidences between 90% and 100%. indicating
5

---

Published as a conference paper at ICLR 2025
overconfident predictions along with high ECE. While Aegis- Guard-G shows less extreme fidence distribution compared to the other models, the proportion of predictions with confidence greater than 90% is still noticeably higher than those with lower confidence, further reflecting the trend of overconfidence.
4.2.2 EVALUATION UNDER JAILBREAK ATTACKS
Table reveals considerable variability of ECE for different guard models when handling harmful requests on the HarmbenchPrompl set. To further investigate the reliability of these guard models in adversarial environments involving dangerous jailbreak attacks, we extend our evaluation the Harmbench-adv set. This dataset, which serves as validation set for fine-tuning Llama2-variant clas- sifiers in Harmbench, includes user inputs generated from various types of jailbreak attacks, such as GCG and AutoDAN, leading to significant distribution shift from typical user input. In this evalua- tion, utilize the adversarial user inputs and their corresponding responses and report the Fl and ECE results for each guard model in Figure 3.
160
Reiporoe caben
100
80
60
Finding 2: Miscalibration in prompt classification is more pro- nounced than in response classification under jailbreak attacks. The results demonstrate that the ECE for prompt classification is generally higher than that of response classification, indicating that guard models tend to be more reliable when classifying model re- sponses under adversarial conditions. We conjecture that this may be due to the more considerable distribution shift in adversarial prompts than that in model responses. Additionally, while Wild- Guard achieves SOTA performance with an Fl score of 92.8% in prompt classification, its ECE score remains high at 34.9%, high- lighting concerns about the reliability of its predictions in real- world deployment.
Figure 3: Fl (%) and ECE (%) J performances of prompt and response classification on Harmbench-adv set.
Response Model
Guard Model
Metric
Baichuan2
Qwen
Solar
Llama2
Vicuna
Orea2
Koala
OpenChat
Starling
Zephyr
Llama-Guard
Fl
57,8
66.7
54.2
44.4
64.0
62.7
74.6
60.6
66.7
72.7
BCE
26.9
23.0
49.4
10.5
28.0
26.4
27.4
40.3
46.3
Llama-Guard2
F1
77,8
88.9
82.8
71.4
72.1
80.6
78.4
70.0
82.0
78.4
BCB
18.2
5.8
27.1
7.9
28.4
25.2
30.4
28.5
37.0
39.4
Llama-Guard3
F1
73.8
82.4
84.1
60.0
83.3
82.2
77.5
76.4
91.2
87.7
ECB
33.7
17.1
31.0
27.4
20.5
27.3
36.5
34.2
27.6
23.1
Aegis-Guard-D
Fl
60.3
66.7
71.2
31.2
63.3
65.8
69.9
78.3
84.4
89.3
BCB
35.5
27.1
22.2
40.8
34.0
33.9
31.3
30.9
27.8
30.9
Acgis-Guard-P
Fl
57.6
66.7
67.9
33.3
56.3
72.7
72.2
76.2
83.3
80,8
BCB
22.8
17.4
28.3
23.6
26.2
28.2
32.1
35.5
36.3
25.7
HarmB-Llama
Fl
89.7
100.0
90.6
70.6
90.9
86.2
88.9
89.4
90.9
94.5
ECR
17.7
6.4
25.0
23.5
16.0
19.5
26.7
23.2
23.3
20.1
HarmB-Mo Mistral
F1
87.5
90.9
89.2
ECB
28.0
3.0
30.1
12.8
16.6
17.5
16.0
14.9
27.7
19.9
MD-Judge
F1
75.4
79.1
77.2
55.6
74.2
76.9
75.3
76.6
87.5
92.6
BCB
22.4
14.4
19.3
24.1
19.9
16.7
26.2
25.5
26.0
WildGnard
Fl
82.0
91.3
88.5
80.0
89.9
84.8
81.6
88.9
92.5
94.5
15.5
25.4
Table 2: F1 (%) and ECE (%) performances response classification on Harmbench-adv set across 10 different response models.
4.2.3 EVALUATION OF ROBUSTNESS TO DIVERSE RESPONSE MODELS
While the ECE for response classification under adversarial environments appears relatively lower in Figure 3, remains important to investigate whether each guard model consistently maintains re- liability when classifying responses generated by different response models. This is crucial because

---

Published as a conference paper at ICLR 2025
response models are often aligned differently during post-training they may have different out- put distributions and produce different responses to jailbreak attacks. To this end, we continue our calibration evaluation under jailbreak attacks, shifting our focus to response classification. Specifi- cally, we employ the same Harmbench-ade set and divide it according to the response model type. After filtering out subsets with small sample size, retain 10 subsets containing responses from Baichuan2, Qwen, Solar, Llama2, Vicuna, Orca2, Koala, OpenChat, Starling, and Zephyr. Each subset consists of outputs from a specific response model. Detailed information on the statistics for each subset is provided in Appendix 1. The FI and ECE results are reported in Table 2.
Finding 3: Guard models exhibit inconsistent reliability when classifying outputs from dif- ferent response models. The results in Table 2 reveal significant variance in both F1 and ECE across different response models. This suggests potential limitations in the training of guard models that rely on responses from single model. For example, Aegis-Guard models are trained using responses from Mistral, and Llama-Guard Guard models are trained using responses from internal L.Lama checkpoints. In contrast, Harmbench-Llanee responses from a more diverse set of models, leading to improved generalization across different output distributions of response models.
Harmbencch-Msstrll and Wildguard are trained using
5 IMPROVING THE CALIBRATION OF LLM-BASED GUARD MODELS
Empirical evidence has demonstrated the miscalibration of current LLM- guard models, neces- sitating efforts improve their reliability through calibration techniques. In this section, focus on post-hoc calibration methods to circumvent the computational expense associated with training new guard models, reserving training- calibration approaches for further investigation.
5.1 CALIBRATION TECHNIQUES
Temperature Scaling (Guo et al., 2017). Temperature scaling (TS) is a widely employed confidence calibration method for traditional neural networks. By introducing scalar parameter T 0 on the output logits, the output distribution can either be smoothed (T >> 1) or sharpened (T < 1). Specifically, the calibrated confidence is computed as:
ply = X.
(4)
Σg, V(G))
It is important to note that applying temperature scaling does not affect the maximum value of the softmax function, and thus does not alter accuracy performance. The parameter T' is typically op- timized on held-out validation set with respect to the negative log-likelihood. However, in the context of the LLM content moderation task, validation sets may not always be available, posing a significant challenge, particularly when addressing in-the-wild user inputs or responses from un- known models. Besides temperature scaling, most conventional calibration methods similarly rely on validation sets to determine parameters, rendering them impractical in such scenarios. As such, exclusively take temperature scaling as instance for its simplicity and efficacy.
Contextual Calibration (Zhao et al., 2021), Contextual calibration (CC) is one type of matrix scal- ing technique to address contextual bias in LLMs, with the key advantage of requiring validation set. This method estimates test-time contextual bias by using content- free tokens such as "N/A", space, empty tokens. The calibrated prediction is then computed as follows:
p R) Wp R)
(5)
where W diag(p(>/[//AA)> Although the original purpose of contextual calibration dif- fers from confidence calibration, the utilized vector scaling modifies model predictions and impacts confidence levels as well, warranting its consideration for confidence calibration.
Batch Calibration (Zhou et al., 2023a). Batch calibration (BC) is also a type of matrix scaling approach. The rationale behind batch calibration is to estimate contextual bias from batch of M unlabeled samples drawn from the target domains P(x) or P(x, r), rather than from context-free tokens in contextual calibration. Specifically, batch calibration applies transformation the original prediction, which can be interpreted as linear transformation in the log-probability space,
log X( R) log puy) X, R. log b
7

---

Published as a conference paper at ICLR 2025
where b is computed in content- manner by b -E)-P(s)(P((/z)) - Ei p(y|z!)) for prompt classification or = EE,,r~ [p(yla, r)] ] ~ classification. Note that batch calibration requires batch of unlabeled samples to estimate the contextual prior during test time.
E; p x(م) for response
5.2 CALIBRATION RESULTS
We apply the calibration methods discussed in Section 5.1 to both prompt classification and sponse classification for each guard model. For temperature scaling, we utilize the XSTest set as the validation set to optimize the temperature due its relatively small size. This optimized tem- perature value then applied across all other datasets, as individual validation sets are not available for all examined datasets. Additional experiments using in-domain validation sets be found in Appendix 1. I. For contextual calibration, we estimate the contextual bias using a space token. For batch calibration, we assume access to the full test and estimate the contextual bias using the entire test set as a batch. The resulting calibration performance is reported in Table 3. Details re- garding the implementation can be found Appendix A.3, along with additional calibration results in adversarial environments in Appendix A.5.
Prompt Clusification XST
Response
Classification Harmi
Model
OAI
ToxiC
SimpST
Aegis
Harml
WildGT
Avg.
BeaverT
S-RLIIF
WildGT
Axg.
Llana-Guard
11.0
32.6
29.6
20.5
28.6
28.5
29.1
24.4
23.2
14.2
22.7
IS
12.0
11.3
31.9
26.8
9.4
66.7
26.0
26.3
27.4
21.6
14.5
14.0
19.4
CC
14,8
7.4
26.3
22.0
23.9
65.1
20.9
25.8
25,
21.8
20.2
8.9
19.1
Llara-Gaadd2
13.7
15.9
26.5
34.7
12.2
30.3
26.8
22.9
29.9
25.2
24.9
12.8
23.2
TS
13.2
15.8
26.0
33.6
11.1
29.4
26.0
22.2
29.8
24.5
14.1
13.6
20.5
22.8
15.0
18.8
13.7
14.8
15.3
20.0
28.9
34.8
32.8
30.2
BC
15.2
16.6
30.6
34.2
12.0
36.3
23.8
24.1
29.5
25.2
24.8
14.7
23.6
Llama-Guard3
14.5
7.6
8.8
16.1
25.3
+ TS
16.0
15.2
9.3
30.6
11.5
9.8
20.4
16.1
31.2
24.8
19.3
13.0
22.0
CC
28.3
21.7
4,1
26.3
8.0
4.5
15.0
15.4
21.9
30.6
39.7
30.3
30.6
20.5
Aegis-GuardDD
30.2
20.5
9.7
16.4
23.5
50.5
8.0
22.7
18.1
30.9
26.1
29.3
26.1
TS
30.1
25.2
14.4
11.1
18.7
46.9
13.4
22.8
15.7
25.9
23.8
31.3
24.2
17.7
21.7
17.5
BC
24.9
34.6
43.0
18.4
20.5
56.2
9.0
29.5
17.2
23.1
18.7
34.2
23.3
Aegis-Gurd-PP
8.2
16.5
22.6
18.6
59.4
12.6
22.0
25.6
18.0
15.2
19.4
19.7
14.9
20.0
16.7
11.4
55.6
13.8
21.7
16.5
21.6
18.2
19.3
18.9
CC
17.6
9.2
15,3
21.5
19.3
58.5
11.1
21.8
18.3
26.3
18.7
16.6
20.0
HammB -lana
24.9
19.4
15.1
52.5
28.0
TS
22.2
17.3
14.2
51.1
26.2
32.0
24.9
BC
24.2
18.8
14.9
52.3
27.6
HammB-Mister
14.5
17.8
TS
13.4
10.3
11.1
23.4
14.6
CC
19.9
23.5
22.4
40.2
26.5
MD-Judge
10.9
9.4
17.7
7.7
11.4
TS
12.7
12.1
20.1
13.0
14.5
22.1
35.9
41.3
33.7
33,3
BC
9.9
8.3
17.1
22.9
14.6
WiMGuard
33.8
19.8
4.4
12.0
5.0
6.3
19.7
14.4
23.2
23.3
12.8
15.9
18.8
TS 32.4
19.1
5.7
9.1
4.2
8.2
19.3
14.0
23.8
22.3
10.5
16.5
18.3
CC
58.7
39.0
0.2
26.5
25.5
0.1
18.6
24.1
22.8
27.9
16.2
16.1
20.8
Table 3: ECE (%) J. performance comparison of different calibration techniques. For each guard
model, we report the original calibration results in the first row and the rest results using TS: Tem- perature Scaling, CC: Contextual Calibration, BC: Batch Calibration, in the following three rows. We bold the best average result among different calibration techniques for each guard model in both prompt and response classification.
Contextual calibration proves more effective for prompt classification and temperature scal- ing benefits response classification more. Empirical results indicate that contextual calibration outperforms other methods in prompt classification, delivering improved calibration for the major- ity of guard models, with the exception of WildGuard. Additionally, temperature scaling effectively reduces the ECE and demonstrates particular effectiveness, despite being optimized on validation
8

---

Published as a conference paper at ICLR 2025
set with potentially different distribution from the target dataset. This finding further confirms the shared overconfident predictions across datasets and validates that proper temperature values can smooth the overconfident prediction distribution, thereby mitigating miscalibration. Furthermore, temperature scaling shows greater efficacy in response classification which often involves multiple sentences of both user inputs and model responses. In such cases, contextual calibration struggles to accurately estimate contextual prior, resulting in unstable or even degraded calibration performance. Moreover, is noteworthy that batch calibration underperforms compared contextual calibration for most models in prompt classification, as well as some models in response classification. We con- jecture that this could be attributed significant label shifts in the test datasets, leading to biased contextual prior estimation and diminished calibration effectiveness. However, no single method fully resolves the miscalibration issues, indicating the complexity of achieving reliable safety mod- eration across different deployment scenarios.
6 DISCUSSION
In this section, to further understand why miscalibration of guard models happens, and how it man- ifests in prompt and response classification, we conduct two further investigations and point out the limitation and weak robustness of instruction-tuning LLM-based guard models.
Prediction Vulnerability Induced by a Single Token. We analyze two specific scenarios by assessing model predictions when the user in- put consists of space token ken and both the user input and model response consist of a space token, respectively. Results of the probability of the input being classified as "safe" are reported in Table The results demonstrate that many guard models exhibit high confidence in predicting "safe" for space token input. However, the introduction of a sin- gle "unsafe" token without further context can cause many guard models to confidently predict "unsafe". This finding underscores the persis- tent contextual bias in guard models revealing their limitations even after instruction instruction-tuinng More extensive robustness evaluations of guard models are thus essential for future research.
Prompt
Response Space
Model
Space
"unsafe"
"unsafe" to-
Llama-Guard
75.5
18.2
73.1
Llama-Guard2
98.9
83.5
99.2
Llama-Guard3
90.5
53.1
98.8
Acgis-Guard-D
29.4
9,5
29.4
Acgis-Guarde
53.1
16.5
53.1
HarmB-LLlam
98.8
HarmB-Mistral
91.7
MD-Judge
89.3
WildGuard
99.5
92.0
77.7
Table 4: Probability (%) of "safe" for prompt clas- sification when input is set as a space token or "unsafe" token, and response classification when input and model output are set as space token.
Model
Safe prompt
Unsafe prompt
Misaligned Classification Objectives. We fur- ther investigate guard models in the LLama-
Llama-Guard Llama-Guar22
62.1
29.0
63.0
21.5
Guard family capable of both prompt and re- sponse classification, focusing on the accuracy of predictions when the model response set to content-free token. Specifically, we sample 100 "safe" user inputs and 100 "unsafe" user inputs from the WildGuard set and replace all model responses with a space token. We report the average probability of classifying the response as "safe" for using "safe" and "unsafe" user inputs separately in Table 5. The results indicate that the model is more likely to predict the responses as "unsafe" when user inputs (prompt) are unsafe, even when model responses content-free and should logically be predicted as "safe". This sug- gests that the model prediction is heavily influenced by the user input and the guard models act like conducting prompt classification when response classification should be done. Such behavior can result in unreliable predictions and increased miscalibration.
Llama-Guar33
62.9
16.4
Table 5: Probability (%) of "safe" for response classification when output is set as space token and inputs are sampled from safe/unsafe prompts.
CONCLUSION
In this work, we have systematically examined the uncertainty- based reliability of LLL-bbacel guard models by assessing their calibration levels across various benchmarks. Our analysis reveals that de-

---

Published as a conference paper at ICLR 2025
spite their promising performance in content moderation, these models tend to make overconfident predictions, exhibit significant miscalibration under adversarial environments, and lack robustness to responses generated by diverse LLMs. To mitigate miscalibration, explore several post-hoc calibration techniques. Our results show that contextual calibration proves particularly effective for prompt classification and temperature scaling improves response classification performance more. Our findings underscore the importance of uncertainty-basee reliability and advocate for incorpo- rating confidence calibration evaluation in the development and release of future LLM-based guard models.
ETHICAL STATEMENT AND BROADER IMPACTS
Our work examines the reliability and confidence calibration of existing LLM-- guard models. Despite enhanced confidence calibration for certain guard models, essential to emphasize that confidence calibration should not be the sole criterion for determining the suitability of a guard model for deployment. Guard models could potentially make incorrect predictions, particularly when dealing with texts in the wild. We thus advocate for a more holistic reliability evaluation that integrates uncertainty-
confidence calibration with assessments of model robustness and overall performance, and in certain cases, even human-involved factors tailored to specific scenarios.
The broader implications of our study have the potential to drive future research towards the devel- opment of better-calibrated guard models. Our insights also contribute to the design of more effec- tive post-hoc calibration techniques, the incorporation of calibration optimization during instruction tuning, and the synthesis of diverse and high-quality data aimed at enhancing both calibration and robustness in the future. Furthermore, our work potentially provides valuable guidance on select- ing classifiers that can ensure consistent and reliable evaluations of attack success rate (ASR) in determining whether a jailbreak attack has succeeded.
LIMITATIONS
Our work focuses on post-hoc calibration methods for open-source LLM-based guardrail models. The explored methods do not apply closed-source models where the logit outputs are unavailable. As for each calibration method, there exist trade-offs. Temperature scaling requires the in in-domain validation set for temperature optimization, but in-domain data are not always available in the prac- tical setting. Contextual calibration requires access to the instruction prompt for inference, but the bias captured from content-free tokens may not always be accurate enough. Batch calibration re- quires access to a batch of unlabeled samples in the target domain, but they could be adapted to adversarial distribution shifts and may need additional validation sets to determine the batch size. In general, post-hoc calibration methods only mitigate the miscalibration in certain scenarios and it is challenging for one single method generalizable to all models and datasets. Nevertheless, this inspires future works to design not only better post-hoc calibration methods but also more reliable training methods to address miscalibration.
REPRODUCIBILITY STATEMENT
In support of reproducibility, our submission includes the full code implementation, along with com- prehensive instructions in the README.md file detailing the steps required install the necessary environments and run our experiments. Additionally, we provide all relevant information regard- ing publicly available datasets and models, enabling interested researchers can replicate the results presented in this paper.
ACKNOWLEDGMENT
The authors would like to thank anonymous reviewers for their valuable suggestions. This project is funded by a research grant NOE-OO0S00 -000 from the Ministry of Education in Singapore.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Saloni- dis, and Tianyi Chen. Enhancing in-context learning via linear probe calibration. In International Conference on Artificial Intelligence and Statistics, pp. 307-315. PMLR, 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.04400 2023.
Tom B Brown. Language models are few-shot learners. arXiv preprint
2020.
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint wrXiv:2404.01318, 2024.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms, arXiv preprint arXiv:2310.2011 2023.
Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. A close look into the calibration of pre-trained language models. arXiv preprint orXiv:2211.0015//, 2022.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:23 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating label biases for in-context learning. arXiv preprint arXiv:2305. 19148, 2023.
Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. Aegis: Online adaptive ai content safety moderation with ensemble of Ilm experts. arXiv preprint arXiv:2404005993, 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406. 18495, 2024.
Hengguan Huang, Xiangming Gu, Hao Wang, Chang Xiao. Hongfu Liu, and Ye Wang. Extrapolative continuous-time bayesian neural network for fast training-free test-time adaptation. In NeurlPS, 2022.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 2023.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of I1m via human-preferent dataset. Advances in Neural Information Processing Systems, 36, 2024.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfeeld-Dodds, Nova DasSarma, Eli Tran- Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.0522/ 2022.
11

---

Published as a conference paper at ICLR 2025
Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasser- man. A new generation of perspective api: Efficient multilingual character- transformers. In Proceedings of the 28th ACM SIGKDD conference knowledge discovery and data mining, pp- 3197-3207, 2022.
Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo. Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language mod- els. arXiv preprint arXiv:2402.050044 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. /4334, 2022.
Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat:
Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. arXiv preprint arXiv:2310.1 23.00.7899, 2023.
Hongfu Liu and Ye Wang. Towards informative few-shot prompt with maximum information gain for in-context learning. arXiv preprint arXiv:2310.0892. 2023.
Hongfu Liu, Yuxi Xie, Ye Wang, and Michael Shieh. Advancing adversarial suffix transfer learning on aligned large language models. arXiv preprint arXiv:2408. 14866, 2024a.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint wwXv:2242.007887 2024b.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint 10.04523 2023.
Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15009-15018, 2023.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.044449 2024.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguisticc, 10:857-872. -72, 2022.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34: 34:15222-15694 2021.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint arXiv:/508.05 154, 2015.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe Ilm applications with programmable rails. arXiv preprint arXhv:2310.1050// 2023.
Paul Rötteer, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint urXiv:2308.0/22:
2023.
Anna Schmidt and Michael Wiegand. A survey on hate speech detection using natural language processing. In Proceedings of the fifth international workshop on natural language processing for social media, pp. 1-10, 2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09228 2023.
Bertie Vidgen, Hannah Rose Kirk, Rebecca Qian, Nino Scherrer, Anand Kannappan, Scott Hale, and Paul Röttger. Simplesafetytests: test suite for identifying critical safety risks in large lan- guage models. arXiv preprint arXiv:2311.08311 2023.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In Neur/PS, 2023.
Hao Wang and Dit- Yan Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE, 28(12):3395-344 -408,, 2016.
Hao Wang and Dit- Yan Yeung. A survey bayesian deep learning. CSUR, 53(5):1-37, 2020.
Hao Wang, SHI Xingjian, and Dit-Yan Yeung. Natural-paramen networks: A class of probabilistic neural networks. In NIPS, pp. 118-126, 2016.
Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. ICML, 2020.
Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, and Hao Wang. Blob: Bayesian low-rank adaptation by backpropagation for large language models. In NeurIPS, 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li. Jie Fu. Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306. / 13063, 2023.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08988 2024.
Zihao Xu, Guang-Ho He Lee, Yuyang Wang, Hao Wang, et al. Graph-relational domain adaptation. In ICLR, 2022.
Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, and Bo Li. Rigor- 0I:: Resilient guardrails for large language models against undesired content. arXiv preprint 17X1v:2403.13031 2024.
Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al. Shieldgemma: Genera- tive ai content moderation based on gemma. arXiv preprint urXiv:2407.21772 2024.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp- 12697-12706. PMLR, 2021.
Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. arXiv preprint arXiv: 2023a.
13

---

Published as a conference paper at ICLR 2025
Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. Survival of the most influential prompts: Efficient black-box prompt search via clustering and pruning. arXiv preprint arXiv:2310.12774, 2023b.
Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. On the calibration of large language models and alignment. arXiv preprint arXiv:2311.32244 2023.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 15444 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DATASET DETAILS
In this section, we briefly describe the public datasets examined and show the statistics in Table 6.
# Prompt safe/unsafe
# Response safe/unsafe
Dataset
Prompt Classification
OpenAl Moderation
1158/522
ToxieChat Test
2491/362
Aegis Safety Test
126/233
SimpleSafetyTesst
0/100
XSTest
250/200
Harmbench Prompt
0/239
WildGuardMix Test Prompt
971/754
Response Classification
BeaverTaiils
894/1 106
SafeRLHF
1000/1000
Harmbench Response
326/270
WildGuardMix Test Response
1440/285
Table 6: Statistics of datasets we used.
OpenAI Moderation (Markov al., 2023). This dataset contains 1680 prompts with labels for 8 unsafe categories including sexual, hate, violence, harassment, self-harm, sexual/minors, hate/threatening, and violence/graphic. Each category label binary flag.
ToxicChat Test (Linn et al., 2023). We use the test split of the new version toxicchat0124, involving 2853 user prompts collected from the Vicuna online demo?, each annotated with binary toxicity labels.
Aegis Safety Test (Ghosh et al., 2024). This dataset is built on the prompts from HH-RLHF and responses generated by Mistral-7B-v0.1 with human annotations. We utilize the prompt-only sub- set, with a size of 359, from the test split of the dataset. The absence of the turn-level split of utterances during the conversation makes it infeasible for response classification evaluation. This dataset covers 13 unsafe content categories according to NVIDIA's content safety taxonomy includ- ing Hate/Identity Hate, Sexual, Violence, Suicide and Self Harm, Threat, Sexual Minor, Guns//llegal Weapons, Controlled/Regulated substances, Criminal Planning/Confessions PII, Harassment, Pro- fanity, Other. The "Needs Caution" category is also involved for uncertain cases.
SimpleSafetyTesst Tests (Vidgen et al., 2023). This dataset involves 100 manually-crafted harmful prompts with topies in Suicide, Self-Harm and Eating Disorders, Physical Harm and Violence, Ille- gal and Highly Regulated items, Scams and Fraud, Child Abuse.
XSTest (Röttger al., 2023). This dataset contains 250 safe prompts and 200 unsafe prompts. Safe prompts use similar language to unsafe prompts or mention sensitive topics but they are clearly safe and should not be refused. Binary labels are provided in this dataset.
Harmbench Prompt (Mazeika et al., 2024). This dataset is designed for robustness to jailbreak at- tacks with prompts for eliciting harmful outputs from LLMs. We use the "standard" and "copyright" subsets, with a total size of 239, from the test split of the dataset in our evaluation for LLM--ased guard models on prompt classification. The topics of unsafe prompts include Cybererime & Unau- thorized Intrusion, Chemical & Biological Weapons/Drugs, Copyright Violations, Misinformation & Disinformation, Harassment & Bullying, Illegal Activities, General Harm.
Harmbench Response (Mazeika et al., 2024). This dataset refers to variant of the validation set used for fine-tuning Llama2- variant from Harmbench, which consists of 602 responses generated by
hhups///marena.ai/
15

---

Published as a conference paper at ICLR 2025
various models and jailbreak attacks. We use the pairs of their vanilla prompts and model responses with human labeling for response classification, resulting in a set of 596 pairs.
Harmbench-ady (Mazeika et al., 2024). This dataset refers to the original validation set with size of 602 for fine-tuning Llama2-variant from Harmbench. We term "Harmbench-ady" to differen- tiate it from "Harmbench Response" given that adversarial prompts from diverse attack methods are involved in the Harmbench-ady set. Adversarial prompts could be very different from vanilla ones. We further split this dataset in terms of the type of response models and retain 10 subsets with statistics in Table 7.
BeaverTail Test (Ji al., 2024). We utilize the test split of this dataset with 33.4k prompt-response pairs, which contain manually annotated labels for model response harmfulness. The prompts are modified from the HH-RLHF and Safety-Prompts. while the responses are generated with the Alpaca-7B model. The 14 harm categories involve Animal Abuse, Child Abuse, Controversial Topies Politics, Discrimination & Stereotype & Injustice, Drug Abuse & Weapons & Banned Substance, Financial Crime & Property Crime & Theft, Hate Speech & Offensive Language, Mis- information Regarding ethics & laws safety, Voollit Unethical Behavior, Privacy Violation, Self-Harm, Sexually Explicit & Adult Content, Terrorism & Organized Crime, Violence & Aiding and Abetting & Incitement. We use a subset of 2k size randomly sampled from the original test split to reduce the evaluation cost.
SafeRLHF Test (Dai et al., 2023). The dataset shares the prompts with the BeaverTails dataset and generates responses from Alpaca-7B, Alpaca2-7B. and Alpaca3-8B. The 19 harm categories include Endangering National Security, Insulting Behavior, Discriminatory Behavior, Endangering Public Health, Copyright Issues, Violence, Drugs, Privacy Violation, Economic Crime, Mental Manipu- lation, Human Trafficking, Physical Harm, Sexual Content, Cybercrime, Disrupting Public Order, Environmental Damage. Psychological Harm, White-Collar Crime, Animal Abuse. We use a subset of 2k size randomly sampled from the original test split to reduce the evaluation cost.
WildGuardMixx Test (Han al., 2024). This dataset contains 1725 samples with synthetic, in-the- wild user-LLM interactions and mnnotator-written data. Responses to synthetic and vanilla prompts are generated using suite of LLMs.. We consider the prompt harmfulness and response harmful- ness annotations in our evaluations. WildGuardMix Test Prompt and WildGuardMix Test Response refer to the prompts data and prompt+response pairs data for prompt and response classification, respectively.
Response Model
Total
Baichuan2
Qwen
Solar
Llama2
Vicuna
Orea2
Koala
OpenChut
Starling
Zephyr
Response
540
64
45
68
38
Table 7: Statistics of subsets across 10 different response models.
A.2 MODEL DETAILS
We summarize the hugging face model cards of LLM-based guard models we examined in Ta- ble 8. Note that we do not assess the series of ShieldGemma' models given that they only support classification for single policy per inference, making public datasets infeasible for evaluation due to the policy difference.
A.3 IMPLEMENTATION DETAILS
We use Pytorch and Huggingface Transformers in our implementation. We run all evaluations on single NVIDIA A40 GPU (48G). We use M 15 bins in Guo et al. (2017) for all our ECE evaluations. For temperature scaling, we optimize the T within the range from (0, 5J. For batch calibration, we set the batch size the size of the entire test set by default following Zhou et al. (2023a). For prompt classification, we keep the original prompt lengths for most datasets except OpenAl Moderation where we truncate few samples with extremely long lengths avoid the out- of-memory error. We keep the maximum length as 1800. For response classification, we keep the original prompt length for all datasets and set the maximum response length as 500.
16

---

Published as a conference paper at ICLR 2025
Guard Models
Hugging Face page
Alama-Guard-7B
meta- mota-llaaa//lamaauurr
Llama-Guard2-2e
nete-llanaaMett--Laam-CCCCrCCC
Guard-2-88
Llama-Guarde Guma--Gurd3-88
meta-llama/Lllmma-Guurrd..
Segii-Guard-Defeeneee
Degis-Guardd-Permisiieee
midia/AAggissllCConnnntttyyyyyaa Permissive-1.0
Cermbenchh-Msttrlleee
cul/HHambbendd--isttrl------o
MD-Judge-vO.1- 1-7B
OpenSafeeyy.ab/ID-lddgee allena//vildguard
WildGuard-7B
Table 8: Hugging Face Model Cards for examined LLM-based guard models
A.4 MORE RELIABILITY DIAGRAMS
We report the full reliability diagrams with corresponding confidence distributions of all mod- els we examined for both prompt classification and response classification on the WildGuardMix Test Prompt set, shown Figure 4. the WildGuardMix Test Response set, shown Figure 5, 6, the Harmbench Prompt set, shown in Figure and the Harmbench Response set, shown in Fig- ure 8, 9. The full results illustrate that existing LLM-based guard models exhibit varying levels of miscalibration. Models including ALama-Guart LLama-Guard2, LLama-Guard3 WildGuard, Harmbench-Llanas and Harmbencc-MMittal tend to make overconfident predictions, while Aegis- Guard-D, Aegis-Guard Guard, and MD-Judge-vd. are not.
A.5 MORE CALIBRATION RESULTS UNDER ADVERSARIAL CONDITIONS
We conduct more calibration experiments using temperature scaling, contextual calibration, and batch calibration on the Harmbench-ady set to mitigate the miscalibration discussed in Section 4.2.2, and Section 4.2.3. We keep the same settings and implementations those in the main text. The ECE results are presented Table and Table 10, respectively. These additional empirical findings indicate the same conclusion in the main text, that contextual calibration proves impressively ef- fective on prompt classification, while temperature scaling benefits more on response classification.
Prompt Classification
Response Classification
Model
Origin
TS
CC
BC
Origin
TS
CC
BC
Llama-Guard
69.3
65.6
58.7
65.8
30.8
30.1
21.2
19.0
Llama-Guard2
59.6
57.6
35.0
61.2
24.4
13.7
31.1
23.7
Llama-Guard3
46.2
44.5
36.3
50.7
20.8
8.8
33.2
20.8
Aegis-Guard-D
40.4
40.8
54.5
52.2
21.5
21.8
11.8
13.8
Aegis-Guard Guard-P
54.4
51.9
52.5
56.6
12.1
16.7
12.5
12.5
HarmB-Llame
27.1
26.3
35.2
26.3
19.8
17.9
29.1
18.7
MD-Judge
12,9
17.2
32.9
12.9
WildGuard
34.9
34.3
26.2
38.6
13.1
10.4
14.3
13.1
Avg.
54.0
52.1
43.3
57.3
20.3
17.1
24.6
17.9
Table 9: ECE (%) J performance comparison of different calibration techniques. For each guard model, we report the Origin: original results, TS: Temperature Scaling, CC: Contextual Calibration, BC: Batch Calibration. We bold the best average result among different calibration techniques in both prompt and response classification.
A.6 INSTRUCTION PROMPTS
Instruction prompts for all I.LM-based guard models examined can be found in submitted code implementation or their Huggingface model cards Table 8.
17

---

Published as a conference paper at ICLR 2025
Respouse Model
Guard Model
Baichuan2
Qwen
Solar
Llama2
Vicuna
Orca2
Koala
OpenChat
Starling
Zephyr
Avg.
Llama-Guard
26.9
23.0
49.4
10.5
28.0
26.4
27.4
40.3
46.3
38.5
31.7
+TS
21.2
46.9
25.1
22.3
24.7
43.7
36.3
29.0
+CC
25.5
17.1
46.3
10.0
26.3
22.2
31.8
35.1
40.1
31.5
28.6
BC
25.5
16.6
48.0
21.7
26.4
23.3
26.9
36.9
47.3
40.6
31.3
Llama-Guard2
5.8
27.1
28.4
25.2
28.5
39.4
24.8
TT
15.4
5.3
25.0
7.1
25.9
22.9
27.5
26.0
33.7
36.5
22.5
+CC
40.3
31.1
31.7
36.3
29.1
38.1
43.0
39.4
27.4
32.5
34.9
+BC
18.3
7.8
29.7
15.6
28.3
25.2
28.6
28.5
40.5
26.1
Llama-Guard3
33.7
17.1
31.0
27.4
20.5
27.3
36.5
34,2
27.6
23.1
27.8
-TS
31.0
15.1
28.3
25.5
18.9
24.6
33.0
31.8
26.4
23.2
25.8
+CC
46.1
30.8
42.6
36.9
43.4
29.0
39.0
BC
32.0
18.0
25.5
31.8
19.2
25.5
30.4
23.3
32.2
26.4
26.9
Aegis Guard-D
35.5
27.1
22.2
40.8
34.0
33.9
31.3
30.9
27.8
30.9
31.4
TT
29.5
26.3
20.8
40.8
30.1
31.4
27.2
30.2
27.3
31.2
29.5
+CC
27.7
17.3
26.5
25.9
28.8
25.7
23.1
29.4
34.3
34.2
27.3
28.9
23.6
25.5
29.5
20.4
39.7
30.7
Aegis Guard-P
22.8
17.4
28.3
23.6
26.2
28.2
32.1
35.5
36.3
25.7
27.6
+TS
19.5
18.5
24.6
26.6
24.4
26.2
29.9
32.1
33.5
29.4
26.5
+CC
23.7
18.2
27.5
25.4
26.6
28.9
35.1
35.3
24.6
27.8
BC
22.3
20.5
29.3
39.4
26.2
27.6
29.0
35.8
43.8
35.4
30.9
Harm--Llam
17.7
6.4
25.0
23.5
16.0
19.5
26.7
23.2
23.3
20.1
20.1
TS
17.3
7.6
22.5
22.6
15.6
26.2
23.5
24.1
21.3
19,9
+CC
32.0
23.9
31.8
24.8
23.7
28.8
37.8
29.2
21.9
21.5
27.5
20.3
Hart--Mistral
28.0
3.0
30.1
12.8
16.6
17.5
16.0
14.9
27.7
19.9
18.6
TS
26.4
5.6
27.3
12.7
15.2
14.3
15.6
14.1
26.1
19.5
17.7
BCC
27.1
2.9
23.8
17.5
16.6
17.1
11.4
14.0
27.0
19.6
17.7
MD-Judge
22.4
14.4
19.3
24.1
19.9
16.7
26.2
25.5
26.0
17.9
21.2
-TS
22.2
17.5
+CC
47.2
40,9
30.3
61.5
41.7
39.4
44.2
37.1
25.9
24.8
39.3
BC
20.7
17.2
21.5
38.5
20.1
26.8
30.3
24.7
WildGuard
22.1
17.0
37.3
21.3
19.8
TS
19.6
6.7
15.2
12.3
12.0
19.2
34.6
25.0
16.3
21.3
18.2
24.3
12.2
20.1
16.1
20.9
39.2
31.3
24.0
23.0
BC
21.9
10.4
13.6
23.7
11.0
20.2
34.7
22.8
15.0
23.8
19.7
Table 10: ECE (%) performance comparison of different calibration techniques. For each guard model, we report the original calibration results in the first row and the rest results using TS: Tem- perature Scaling, CC: Contextual Calibration, BC: Batch Calibration, in the following three rows. We bold the best average result among different calibration techniques for each guard model.
18

---

Published as a conference paper at ICLR 2025
LamtaGraan gPon 0.560
Lame Guant (FEP: 0.700
Lama Suardh 0200)
Degis- 6.m00
AeerrrGuers 6:100
ViltGuant (EEF: 6.197)
Count
WT
Conficence
1.0
1:34
1.00
1.0
02
02-
03
82
8.4
1.0
046 05
03
8-4 Confidence
1.0
62
0.6 Cont dence
08
08
03
0.0 814
0.5 0.0
65
0.6
Corfidence
dence
Carricence
Cantidence
Figure 4: Confidence distributions (First row) and reliability diagrams (Second row) the Wild- GuardMix Test Prompt set.
Uamp-Guard Guand2 (ECe:
Goojs.GardD1iee Guard rECE
(SCE:
Cun
Corfilence
Confidence
Confdence
Confdence
Carfsince
Confidence
91-
64-
0.1
64-
844
024
Corfidenne
Confdoncc
Corfidence
Confidncc
Confirance
Caniidencc
Figure 5: Confidence distributions (First row) and reliability diagrams (Second row) on the Wild- GuardMix Test Response set.
27
06
Confidancee
Canficnnc
Corkdenos
00
20
08
Figure 6: Confidence distributions (First row) and reliability diagrams (Second row) on the Wild- GuardMix Test Response set.
19

---

Published as a conference paper at ICLR 2025
Lame Guant (FEP: a 101)
Lama Suaedh IPCF: n.tto
Aegii- 0.50
AegrrrGuers (FCP 0.500
VintGuant (EEF: 6.061)
Count
WT
85
89
1.0
1:34
1.0
1--
1.0
Aconey 1864 8n
02
01
92 8.4 8.8 1.0
04 05 dence
8X
62
0.6
08
05
03
0. 6. Carricance
844
0.0
6.
0.6
0.5
Corfidance
Corfidence
Cont dence
Cantidence
Figure 7: Confidence distributions (First row) and reliability diagrams (Second row) the Harm- bench Prompt set.
Uamb-Guard iECe
Guand2 (SCe:
Uama Guard IB0E:
Acg's MojjsGuardD1Bee
Guard
(SCE
Cun
Corfilence
Contiderice
Corfdence
ConFidecc
Canfisnnce
Confidence
51-
64-
66-
4.4
024
82-
Corfidenne
Conf-dence
Corfidence
Confidncc
Confirnnce
Cantddnnc
Figure 8: Confidence distributions (First row) and reliability diagrams (Second row) on the Harm- bench Response set.
02
08
Confitence
Confience
24
20 03
Figure 9: Confidence distributions (First row) and reliability diagrams (Second row) on the Harm- bench Response set.
20

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS
B.1 EFFECTS OF IN-DOMAIN VALIDATION SETS FOR TEMPER ATURE SCALING
To further improve the calibration effect of temperature scaling, we conduct additional experiments by optimizing temperature on the in-domain set. Specifically, we consider the WildGuardMMx dataset and split validation set of size 100 from its training set as the in-domain validation set. The temperature is then optimized on the new validation set. The results for prompt and response classification are reported in Table 11. It observed that the temperature optimized on the in-domain validation set is more effective in reducing ECE than the optimized on XSTest. Despite the ben- efits, there is not always access to such an in in-domain validation dataset for different scenarios in the wild, leading to trade-off when applying temperature scaling.
Prompt Classification
Response Classification TS(XSTest)
Model
Origin
TS(XSTest)
TS(nn-ommii))
Origin
TS(In-Domain)
Llama- Llama-Guard
28.6
26.0
19.9
14.2
14,0
11.6
Gama-Guard2
26.8
26.0
19.9
12.8
13.6
9.8
Llama-Guards Guard3
20.5
20.4
16.4
11.1
13.0
8.6
Table 11: ECE (%) 4 performance comparison of different validation sets.
B.2 ABLATION ON THE LENGTH OF ADVERSARIAL PROMPTS
To investigate whether the guard model is vulnerable due to the jailbroken nature of the prompts, or due to some spurious correlations such as length, we design additional experiments. Specifically, we consider three different ranges of adversarial prompt length, leading to three subsets. Then assess the ECE on three subsets separately and report the results in Table 12. is observed that there are similar ECE performances among different length ranges, which further supports that it the jailbroken nature adversarial prompts that makes guard models vulnerable. We conjecture that there exist many unseen adversarial prompts with any token combinations in the wild and it is impossible to involve all adversarial prompts in instruction tuning of guard models. Thus, the evaluation of adversarial prompts introduces varying levels of uncertainty instead of making over- confident predictions that lead to high ECE performances.
Length (L) Range
0 L 200
200 < L 500
500 < L
Lllam-Guurd
68.4
76.5
66.2
Gunna-Guard2
60.9
50.7 61.9
Alama--uarrh
49.6
46.7
33.4
Table 12: ECE (%) 4 performance comparison among different length ranges,
B.3 ROBUSTNESS ON BLACK- BOX RESPONSE MODELS
To further assess the ECE performance when classifying responses from proprictary models, we try collect the responses from GPT-3.5, GPT-4, and Claude-2 to the same set of adversarial queries, and then conduct the same evaluation of ECE. The results are reported in Table 13. It is observed that all guard models have low ECE values when classifying responses from GPT-4 and Claude-2. The reason is that the utilized adversarial attacks are not effective enough for these well-aligned black- box models and thus elicit clear refusal responses that are easy classify for guard models. Nevertheless, some guard models still exhibit high ECE values when classifying responses from GPT-3.5, serving as nice complement and support to our Finding Section 4.2.3.
21

---

Published as a conference paper at ICLR 2025
Response Model
Guard Model
GPT-3.5
GPT-4
Claude-2
Llama-Guard
39.1
0.0
0.8
18.8
0.0
0.0
Llama-Guards
0.0
0.0
0.0
Aegis-Guard-D
0.0
0.0
10.7
Legis-Guard-P
26.6
0.0
0.0
HarmB-Llama
31.2
0.0
0.0
HarmB-Mistral
30.5
0.0
0.0
MD-Judge
21.5
0.0
3.2
WildGuard
9.0
0.0
0.0
Table 13: ECE (%) 4 performances of response classification Harmbench-ady set on black-box response models.
B.4 COMPARISON OF FALSE POSITIVE RATE AND FALSE NEGATIVE RATES
To investigate what types of over-confidence behaviors different guard models are showing, we report the statistics of the false positive rates (FPR) and false negative rates (FNR) Table 14. is observed that most models generally showcase higher false negative rates, suggesting their over- confidence behaviors predict the input as the safe type.
Prompt Classification SimpST
Response Classification BeaverT
Model
Metric
OAT
ToxiC
Aegis
XST
HarmB
WIMGT
S-RLHH
HarmB
WildGT
FPR
8.4
144
3.2
15.2
2.4
8.5
16.6
12.9
0.8
Liama-Guard
FNR
28.9
53.0
13.0
39.9
49.8
60.9
46.
47.0
67.3
Liana Llana-Guan22
HPR
8.2
3.1
4.8
7.6
3.6
7.9
21.3
19.3
3.1
FNR
27.4
42.5
11.3
43.6
21.5
41.5
Liama-Guard3
FPR 9.2
5.0
2.4
2.8
4.4
4.0
17.7
35.3
3.5
Table 14: FPR (%) 4 and FNR (%) performances of prompt and response classification existing public benchmarks.
22


Reviews:
[{"review_id": "gupzR2PFcQ", "reviewer": "Reviewer_iHeM", "summary": "This work examines how calibration can affect and potentially improve LLM-based guard models. The study finds most guard models are poorly calibrated, especially under jailbreaking attacks, but that off the shelf calibration methods don't seem to provide consistent calibration benefits (at least not as tested).", "questions": "+ Were the guard models sampled from or is it using greedy decoding?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ This is an extensive and comprehensive evaluation of a variety of calibration methods, models, and datasets. It clearly took a lot of effort.\n+ Overall, this highlights the poor calibration of guard models and  could be good motivation for more consistent methods. The eval harness could also be the building block for an evaluation suite to test some improved methods down the line.", "weaknesses": "+ The effect sizes for some of the calibration methods on some datasets were rather small and it’s a bit unclear what the potential variance here is. It would be great to have a table 3 with confidence intervals, though given the size of the table this might be a heavy lift. This becomes more important if there was sampling at test time and less important if there was greedy decoding (see question below).\n+ It’s not clear that batch calibration didn’t work as well because of the selection of the batch of unlabeled samples or other choices. I don’t think this is a major issue, but should be called out more prominently as a limitation. Similarly for other decisions and methods. This is done to some extent, but a standalone limitations section might be warranted. Overall there seem to be some major caveats for design decisions as to the generalizability of the takeaways from the calibration method study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L6ycEKQ1oF", "reviewer": "Reviewer_Jv3L", "summary": "The paper examines the reliability and calibration of guard models based on LLMs used in content moderation. The authors highlight that while these models achieve strong classification performance, they often produce overconfident and poorly calibrated predictions, particularly when faced with adversarial attacks like jailbreaks. Through an empirical evaluation of nine guard models across 12 benchmarks, the study identifies significant calibration issues, such as overconfidence and inconsistent robustness across different response models. To address these challenges, the paper explores post-hoc calibration techniques, demonstrating the effectiveness of temperature scaling for response classification and contextual calibration for prompt classification. The findings underscore the importance of improving model calibration to enhance the reliability of guard models in real-world content moderation scenarios.", "questions": "Q1: For results on jailbreak prompts, the author said that \"The results demonstrate that the ECE for prompt classification is\ngenerally higher than that of response classification, indicating that guard models tend to be more reliable when classifying model responses under adversarial conditions. \" However, it's not clear whether the guard model is vulnerblae due to the jailbroken nature of the prompts, or due to some suprrious correlations (eg length, patterns). It will be great if the authors can explain or design experimetns to ablate such factors.\n\nQ2: It's interesting to see the variability in guard model performance across different response models (Table 2). However, it would be more insightful to understand the causes of these discrepancies. For example, why do all the models perform particularly poorly with Llama2's responses? Is there a qualitative or quantitative explanation for this?\n\nQ3: Regarding the calibration results in Table 3, the improvements appear relatively modest (e.g., at most around 2% ECE reduction). It would be helpful to contextualize how significant these improvements are. Additionally, it seems that contextual calibration (CC) and batch calibration (BC) sometimes degrade performance. Understanding the reasons behind this would provide valuable insights.\n\nQ4: Most of the evaluated models in Table 2 are open-weight models, so it’s unclear how these findings would transfer to proprietary models like ChatGPT, Gemini, and Claude.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper conducts an in-depth analysis of LLM-based guard models and explores potential design improvements.", "weaknesses": "Overall, I think the paper does a good job of presenting the \"what\"—namely, the findings and results—but it would benefit from delving deeper into the \"why,\" or the reasons behind these observations (Sec 6 has some \"understanding\" results, but seems to be distantly related). Without this, the paper feels more like a dataset and benchmark track submission (which ICLR does not specifically have) rather than a main track paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgzodIbGj6", "reviewer": "Reviewer_76NL", "summary": "The proposed study conducts investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. The resultant findings are that these guard models are overconfident, are miscalibrated with respect to jailbreak attacks, and have limited robustness.", "questions": "1. What are some of the limitations of this experimental setup? How do these limitations affect the resultant outputs and findings?\n2. Have you tried the setup on a larger model that is being prompted to act as a guardrail? It would be interesting as a comparison point to these guardrail-specific models.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Figure 1 is well thought out and easy to follow. sets the stage well\n- the experimental setup is solid, with a variety of benchmarks that are used in industry. In particular, the use of benchmarks for which typical statistics are provided for these guardrail models is smart. \n- the breadth of guardrail models used is admirable", "weaknesses": "- discussion of limitations is lacking, would be interesting to see where the pitfalls of this approach are and how they could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wlm2NsTajq", "reviewer": "Reviewer_fAD6", "summary": "This papers conducts an analysis on the reliability and calibration of LLM-based guard models for safety moderation. The findings reveal that these guard models tend to be overconfident in predictions, show miscalibration when subjected to jailbreak attacks, and different response models also have different calibration performance. Based on these insights, the authors also propose some easy-to-implement methods to improve calibration.", "questions": "- More discussion on the generalizability of the techniques.\n- Can you show the statistics for false positives and false negatives as well? Would be useful to know that which models are showing what types of over-confidence behavior.\n- What is the “unsafe” token used for experiments in the discussion section?\n- Could you provide more explanation or intuition on the calibration trade-offs across models? Why certain methods are better for response classification while some work better for prompt classification. \n- How does the size / training data characteristics affect calibration? It would be better to understand why certain methods work better for certain scenario.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper points out an important yet under-studied problem of (over) confidence in LLM safety guard models and analyzes the confidence calibration problem. \n- The evaluation covers a wide range of models and benchmarks for evaluation. \n- The work finds that lightweight approach like contextual calibration can be effective mechanisms for improving confidence calibration. \n- The paper is written clearly and the overall flow is easy to follow.", "weaknesses": "- The proposed calibration techniques does not show strong improvement on the ECE metrics, and in some cases even make the ECE score higher (Table 3). There is no statistical significance tests (eg. Multiple runs, variance, hypothesis testing) to show that the methods are indeed beneficial. \n- CC is primarily designed for binary or few-class settings, and the framework appears to be challenging to extend to a multi-class setup. The influence of a content-free token might not translate well to all classes, especially if some classes are rare or highly specialized. It will also be harder to interpret and apply, because the baseline bias captured from a content-free input could vary inconsistently across datasets. \n- The assumptions for the BC method are not realistic in actual LLM setting. It also may inadvertently adapt to an adversarial distribution shift. \n- For temperature scaling, the authors used XSTest as the validation set for optimization. However, since XSTest is a dataset for estimating over-refusal, there is likely bias in the temperature optimized on it. \n- The authors do not discuss the distribution of safe/ unsafe prompts in the dataset being studied. The ECE metric could be affected by dataset imbalance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)