On Calibration of LLM-based Guard Models for Reliable Content Moderation

Authors: Hongfu Liu, Hengguan Huang, Xiangming Gu, Hao Wang, Ye Wang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Content Moderation, LLM-based Guard Models, Calibration, Safety

Abstract:
Large language models (LLMs) pose significant risks due to the potential for generating harmful content or users attempting to evade guardrails. Existing studies have developed LLM-based guard models designed to moderate the input and output of threat LLMs, ensuring adherence to safety policies by blocking content that violates these protocols upon deployment. However, limited attention has been given to the reliability and calibration of such guard models. In this work, we empirically conduct comprehensive investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. Our findings reveal that current LLM-based guard models tend to 1) produce overconfident predictions, 2) exhibit significant miscalibration when subjected to jailbreak attacks, and 3) demonstrate limited robustness to the outputs generated by different types of response models. Additionally, we assess the effectiveness of post-hoc calibration methods to mitigate miscalibration. We demonstrate the efficacy of temperature scaling and, for the first time, highlight the benefits of contextual calibration for confidence calibration of guard models, particularly in the absence of validation sets. Our analysis and experiments underscore the limitations of current LLM-based guard models and provide valuable insights for the future development of well-calibrated guard models toward more reliable content moderation. We also advocate for incorporating reliability evaluation of confidence calibration when releasing future LLM-based guard models.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730648227673

thread_type:
official_review

<thread_object>
creator: reviewer_fad6

<content>
summary:
This papers conducts an analysis on the reliability and calibration of LLM-based guard models for safety moderation. The findings reveal that these guard models tend to be overconfident in predictions, show miscalibration when subjected to jailbreak attacks, and different response models also have different calibration performance. Based on these insights, the authors also propose some easy-to-implement methods to improve calibration.

strengths:
- The paper points out an important yet under-studied problem of (over) confidence in LLM safety guard models and analyzes the confidence calibration problem. 
- The evaluation covers a wide range of models and benchmarks for evaluation. 
- The work finds that lightweight approach like contextual calibration can be effective mechanisms for improving confidence calibration. 
- The paper is written clearly and the overall flow is easy to follow.

weaknesses:
- The proposed calibration techniques does not show strong improvement on the ECE metrics, and in some cases even make the ECE score higher (Table 3). There is no statistical significance tests (eg. Multiple runs, variance, hypothesis testing) to show that the methods are indeed beneficial. 
- CC is primarily designed for binary or few-class settings, and the framework appears to be challenging to extend to a multi-class setup. The influence of a content-free token might not translate well to all classes, especially if some classes are rare or highly specialized. It will also be harder to interpret and apply, because the baseline bias captured from a content-free input could vary inconsistently across datasets. 
- The assumptions for the BC method are not realistic in actual LLM setting. It also may inadvertently adapt to an adversarial distribution shift. 
- For temperature scaling, the authors used XSTest as the validation set for optimization. However, since XSTest is a dataset for estimating over-refusal, there is likely bias in the temperature optimized on it. 
- The authors do not discuss the distribution of safe/ unsafe prompts in the dataset being studied. The ECE metric could be affected by dataset imbalance.

questions:
- More discussion on the generalizability of the techniques.
- Can you show the statistics for false positives and false negatives as well? Would be useful to know that which models are showing what types of over-confidence behavior.
- What is the “unsafe” token used for experiments in the discussion section?
- Could you provide more explanation or intuition on the calibration trade-offs across models? Why certain methods are better for response classification while some work better for prompt classification. 
- How does the size / training data characteristics affect calibration? It would be better to understand why certain methods work better for certain scenario.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730663986766

thread_type:
official_review

<thread_object>
creator: reviewer_76nl

<content>
summary:
The proposed study conducts investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. The resultant findings are that these guard models are overconfident, are miscalibrated with respect to jailbreak attacks, and have limited robustness.

strengths:
- Figure 1 is well thought out and easy to follow. sets the stage well
- the experimental setup is solid, with a variety of benchmarks that are used in industry. In particular, the use of benchmarks for which typical statistics are provided for these guardrail models is smart. 
- the breadth of guardrail models used is admirable

weaknesses:
- discussion of limitations is lacking, would be interesting to see where the pitfalls of this approach are and how they could be improved.

questions:
1. What are some of the limitations of this experimental setup? How do these limitations affect the resultant outputs and findings?
2. Have you tried the setup on a larger model that is being prompted to act as a guardrail? It would be interesting as a comparison point to these guardrail-specific models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730754243793

thread_type:
official_review

<thread_object>
creator: reviewer_jv3l

<content>
summary:
The paper examines the reliability and calibration of guard models based on LLMs used in content moderation. The authors highlight that while these models achieve strong classification performance, they often produce overconfident and poorly calibrated predictions, particularly when faced with adversarial attacks like jailbreaks. Through an empirical evaluation of nine guard models across 12 benchmarks, the study identifies significant calibration issues, such as overconfidence and inconsistent robustness across different response models. To address these challenges, the paper explores post-hoc calibration techniques, demonstrating the effectiveness of temperature scaling for response classification and contextual calibration for prompt classification. The findings underscore the importance of improving model calibration to enhance the reliability of guard models in real-world content moderation scenarios.

strengths:
The paper conducts an in-depth analysis of LLM-based guard models and explores potential design improvements.

weaknesses:
Overall, I think the paper does a good job of presenting the "what"—namely, the findings and results—but it would benefit from delving deeper into the "why," or the reasons behind these observations (Sec 6 has some "understanding" results, but seems to be distantly related). Without this, the paper feels more like a dataset and benchmark track submission (which ICLR does not specifically have) rather than a main track paper.

questions:
Q1: For results on jailbreak prompts, the author said that "The results demonstrate that the ECE for prompt classification is
generally higher than that of response classification, indicating that guard models tend to be more reliable when classifying model responses under adversarial conditions. " However, it's not clear whether the guard model is vulnerblae due to the jailbroken nature of the prompts, or due to some suprrious correlations (eg length, patterns). It will be great if the authors can explain or design experimetns to ablate such factors.

Q2: It's interesting to see the variability in guard model performance across different response models (Table 2). However, it would be more insightful to understand the causes of these discrepancies. For example, why do all the models perform particularly poorly with Llama2's responses? Is there a qualitative or quantitative explanation for this?

Q3: Regarding the calibration results in Table 3, the improvements appear relatively modest (e.g., at most around 2% ECE reduction). It would be helpful to contextualize how significant these improvements are. Additionally, it seems that contextual calibration (CC) and batch calibration (BC) sometimes degrade performance. Understanding the reasons behind this would provide valuable insights.

Q4: Most of the evaluated models in Table 2 are open-weight models, so it’s unclear how these findings would transfer to proprietary models like ChatGPT, Gemini, and Claude.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731476628707

thread_type:
official_review

<thread_object>
creator: reviewer_ihem

<content>
summary:
This work examines how calibration can affect and potentially improve LLM-based guard models. The study finds most guard models are poorly calibrated, especially under jailbreaking attacks, but that off the shelf calibration methods don't seem to provide consistent calibration benefits (at least not as tested).

strengths:
+ This is an extensive and comprehensive evaluation of a variety of calibration methods, models, and datasets. It clearly took a lot of effort.
+ Overall, this highlights the poor calibration of guard models and  could be good motivation for more consistent methods. The eval harness could also be the building block for an evaluation suite to test some improved methods down the line.

weaknesses:
+ The effect sizes for some of the calibration methods on some datasets were rather small and it’s a bit unclear what the potential variance here is. It would be great to have a table 3 with confidence intervals, though given the size of the table this might be a heavy lift. This becomes more important if there was sampling at test time and less important if there was greedy decoding (see question below).
+ It’s not clear that batch calibration didn’t work as well because of the selection of the batch of unlabeled samples or other choices. I don’t think this is a major issue, but should be called out more prominently as a limitation. Similarly for other decisions and methods. This is done to some extent, but a standalone limitations section might be warranted. Overall there seem to be some major caveats for design decisions as to the generalizability of the takeaways from the calibration method study.

questions:
+ Were the guard models sampled from or is it using greedy decoding?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732306622163

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Revision

comment:
We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a **Paper Revision** including additional results and illustrations:

- **Limitation Section** (Page 10): summarize the limitations of each calibration method and the trade-off to be considered when applying them
- **Table 11** (Page 20): calibration effects of temperature scaling using different validation sets
- **Table 12** (Page 20): effects of adversarial prompt length on ECE evaluation
- **Table 13** (Page 21): robustness to responses from proprietary models
- **Table 14** (Page 21): False Positive Rate and False Negative Rate statistics
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a solid empirical paper measuring present-day SotA LLM-based guardrails' (that classify prompts into e.g. jailbreak attempt vs not, and classify responses into e.g. toxic content or not) uncertainty quantification and calibration.  All reviewers appreciated just how comprehensive that task of benchmarking was, and to a lesser extent appreciated some of the proposed fixes to move these methods more toward being well-calibrated.  That latter half received less support from the reviewers, but the lift and value of the descriptive study is high.  While ICLR does not have a specific Datasets & Benchmarks style of track, the conference does have a history of supporting empirical-only work (even moreso than e.g. NeurIPS and ICML).