BingoGuard: LLM Content Moderation Tools with Risk Levels

Authors: Fan Yin, Philippe Laban, XIANGYU PENG, Yilun Zhou, Yixin Mao, Vaibhav Vats, Linnea Ross, Divyansh Agarwal, Caiming Xiong, Chien-Sheng Wu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM, safety guardrail, content moderator

Abstract:
Malicious content generated by large language models (LLMs) can pose varying degrees of harm. 
Although existing LLM-based moderators can detect harmful content, they struggle to assess risk levels and may miss lower-risk outputs. 
Accurate risk assessment allows platforms with different safety thresholds to tailor content filtering and rejection. In this paper, we introduce per-topic severity rubrics for 11 harmful topics and build BingoGuard, an LLM-based moderation system designed to predict both binary safety labels and severity levels. 
To address the lack of annotations on levels of severity, we propose a scalable generate-then-filter framework that first generates responses across different severity levels and then filters out low-quality responses. Using this framework, we create BingoGuardTrain, a training dataset with 54,897 examples covering a variety of topics, response severity, styles, and BingoGuardTest, a test set with 988 examples explicitly labeled based on our severity rubrics that enables fine-grained analysis on model behaviors on different severity levels. Our BingoGuard-8B, trained on BingoGuardTrain, achieves the state-of-the-art performance on several moderation benchmarks, including WildGuardTest and HarmBench, as well as BingoGuardTest, outperforming best public models, WildGuard, by 4.3\%. Our analysis demonstrates that incorporating severity levels into training significantly enhances detection performance and enables the model to effectively gauge the severity of harmful responses. Warning: this paper includes red-teaming examples that may be harmful in nature.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730113712803

thread_type:
official_review

<thread_object>
creator: reviewer_6tpt

<content>
summary:
The paper introduces an open moderation tool called BingoGuard which aims to predict both harms and severity of risks. Their model is trained with synthesized data generated by authors. Specifically, the authors first define the risk and severity levels taxonomy before building the data, and apply over-generation then filtration procedure to construct the synthetic data. Their main focus of the data generation pipeline is to 1) diversify the data and 2) collecting hard data that the model might fail to classify. For the first goal, they collected diverse prompts from different sources considering the topics, and do generate-then-refine approach to construct diverse responses across different risk severity levels. For the second goal, they applied active learning, identifying hard responses among the generated candidate responses that their initial models fail to classify. Experimental results, including ablations, demonstrated their models' superiority.

strengths:
Paper is clearly written and easy to follow. It clearly presented the problem of detecting the risk severity of model responses, which have not been addressed by prior works. 

If the data and model weights are openly released, these resources will be meaningful for the field for further studies. But I can't find the resources in this submission, and I believe the authors will make them public -- let me know if I am missing those.

Their idea of building the new dataset is intuitive and effective. Their approach of training specialized models to elicit harmful responses is practical and their experimental results show the effectiveness of training models on these generated data.

Their ablation studies also help understanding the effectiveness of their suggested approach.

weaknesses:
Some of the benchmark results are missing: for example, WildGuard paper (which is the key baseline of this paper) showed results also showed results on SimpleSafetyTests, Harmbench, and WildGuardTest prompts for query classification tasks, and XSTest, Safe-RLHF responses for response classification tasks. This paper already tested on XSTest prompts for query classification and WildGuardTest, Harmbench responses for response classification -- which can be weird if some other numbers are missing. Providing more numbers from these benchmarks will help proving the significance of the proposed model.

questions:
My question is related to the weakness section: could you provide more numbers on these benchmarks: SimpleSafetyTests, Harmbench, WildGuardTest for prompt classifications and XSTest, Safe-RLHF for response classification tests?

I am willing to change the rating if this concern is addressed well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495724731

thread_type:
official_review

<thread_object>
creator: reviewer_jmyf

<content>
summary:
This paper studies the classification of harmful content and proposes to include risk levels in addition to binary labels for harmful/benign (5 risk levels from benign to extreme risk/harm). It also includes types of response dimensions, such as attitude (positive vs negative) or intent (education vs malicious). Several datasets to support these ideas are introduced: a training dataset covering several high-level topics of harms, along with response severity and style and a corresponding test set. The new datasets are used to train new harm detection classifiers that are compared to state of the art, observing a few percentage improvement.

strengths:
I find the severity levels and response styles useful.
The datasets, assuming they will be made available, could be useful for futher research.

weaknesses:
Some aspects are not well explained. For example, I didn't understand how the different response styles are incorporated. They seem like a great idea, but couldn't find details on whether they are also predicted, whether they are used in any way in training or testing.

questions:
1. What is the most challenging aspect of the infrastructure that you used to generate the data?
2. How much human oversight was included in the dataset creation?
3. How are the response styles included in the framework? Do you classify the style of a response?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651903199

thread_type:
official_review

<thread_object>
creator: reviewer_osfd

<content>
summary:
This work introduces a new taxonomy with severity levels for unsafe content detection. The authors build a dataset and train a "moderator" that they test on their benchmark and public benchmarks. I think this is a complete, end-to-end work.

strengths:
* I find this work interesting. The unsafe/safe content detection problem is hard and I think adding "severity levels" is a nice idea. The new model performs well and I appreciate the ablation showing more detailed insights. I can see this model and the dataset possibly useful in the future.

* While the model is simple, I can see that a lot of thought went into building a good dataset.

* I appreciate that the authors did not just test on their benchmark but also used other standard public benchmarks. I think the comparison makes sense.

* Finally, the appendix is very detailed and useful to actually understand the annotation scheme.

weaknesses:
I don't see strong weaknesses here. One could argue the taxonomy and severity levels are subjective (there's overlap in Levels 3 and 4). This might need more discussion and disclaimers but I wouldn't call it a weakness. Still, would appreciate a more in-depth discussion about this. We have many taxonomies for safety out there and, as mentioned before, there is a degree of subjectivity in what counts as unsafe.

questions:
* Can we see topic classification results on the test set? I understand we can't compute this for different datasets/methods but I'm just curious about that part's performance.

* Did the authors compare with OpenAI's content moderation API? I see GPT-4o in the tables but wonder if this could be useful (might have missed this).

* Line 383: "We call them BingoGuard-Instruct-8B and BingoGuard-3B, respectively" - took me a second to understand which the base models were. Would suggest making naming more explicit (BingoGuard-Phi?)

* Could you compute Krippendorff's alpha for the annotators?

* Line 378: Typo.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730835766902

thread_type:
official_review

<thread_object>
creator: reviewer_ybkr

<content>
summary:
This paper introduces BingoGuard, an LLM-based moderation system designed to predict both binary safety labels and severity levels. Using a generate-then-filter data generation framework, this work creates BingoGuardTrain and BingoGuardTest two datasets with explicitly labeled risk levels that enable fine-grained analysis on model safety behaviors. With BingoGuardTrain, this work trains BingoGuard-8B that achieves the state-of-the-art performance on several moderation benchmarks, outperforming best public models, WildGuard, by 4.3%. This work also releases useful data and model resources for LLM safety moderation.

strengths:
- The paper proposes a novel task for classifying unsafe model responses with 5 risk levels.
- I appreciate the effort for releasing useful training & evaluation data and model resources.
- In addition to the superior performance on the proposed new task, their released moderation tool also achieves SOTA performance on a variety of existing prompt & response safety classification benchmarks.

weaknesses:
- The justification for the importance and real-world impact of the proposed response risk level classification task could be strengthened. Often, it’s essential to assess the risk level of the harmful prompt itself. For example, prompts with a lower risk level (such as "write a program to scrape publicly available government documents," which is only mildly risky) might not need to be prioritized in refusal training. However, current practices focus on training models to reject any harmful prompts outright, avoiding harmful responses at all risk levels. This raises questions about how a fine-grained risk classification for responses would practically enhance safety moderation practices.

- The authors mention that Level 0 is classified as "safe" responses, while Levels 1-4 represent varying degrees of risk (L88-89). However, based on the risk level definitions provided in Figure 2, it's unclear whether Levels 1 and 2 genuinely present practical risks. For instance, under the topic "Privacy Invasion," a Level 1 risk includes "Educational discussions on privacy threats, such as definitions of privacy attacks, historical events, potential consequences, or news and statistics about privacy issues." A Level 2 risk includes "Content featuring fictional or general information on privacy breaches, such as references to movies involving privacy leakage, cyberattacks without explicit details, or general instructions." These definitions of risk levels do not really pose concrete enough harms in reality. Therefore, it remains unclear whether the Level 1 and Level 2 results in Table 1 accurately reflect the binary classification ability of these safety moderation tools across different risk levels. It is possible that these responses have risks too low to be classified as genuinely risky in binary classification setups.

- Since harmful responses with varying risk levels are generated by models fine-tuned on existing examples, the diversity of these synthetically generated responses remains unclear.

questions:
- How are finer-grained risk levels in model responses useful for safety alignment in practice?
- Could you explain a little more on the risk level definitions, and how do you imagine we should deal with model response with very low risk level? How reliable do you believe the evaluation of low-level risks using binary classification is in Table 1, given that some responses may potentially present risks too low to be considered practically significant?
- What measures have been taken to ensure diversity in synthetically generated harmful responses, especially when these examples are derived from existing training data? Also, how diverse are the queries used in BingoGuardTrain/BingoGuardTest?
- Seems like all queries in BingoGuardTrain are compiled from existing data resources. What do you think is the reason why BingoGuard surpasses the other baselines on the query classification task? Is it mainly because BingoGuard uses a stronger base model compared to others?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces BingoGuard, a moderation system for LLM safety that predicts binary safety labels and risk severity levels using a generate-then-filter framework. The authors construct BingoGuardTrain and BingoGuardTest, two datasets labeled with fine-grained risk levels to enable detailed evaluation. The system achieves state-of-the-art performance. Strengths include the novel focus on severity classification, the clear experimental setup, and the promise of releasing valuable datasets and models. Weaknesses include not very rigorous definitions of risk levels, the subjectivity of severity levels (especially between Levels 3 and 4), incomplete benchmarking comparisons, and limited discussion on the real-world applicability of fine-grained severity classification. Despite these limitations, the paper’s contributions to safety moderation and its practical potential justify acceptance.