$R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning

Authors: Mintong Kang, Bo Li
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM guardrail model, content moderation

Abstract:
As large language models (LLMs) become increasingly prevalent across various applications, it is critical to establish safety guardrails to moderate input/output content of LLMs and ensure compliance with safety policies. Existing guardrail models, such as OpenAI Mod and LlamaGuard, treat various safety categories (e.g., self-harm, self-harm/instructions) independently and fail to explicitly capture the intercorrelations among them. This has led to limitations such as ineffectiveness due to inadequate training on long-tail data from correlated safety categories, susceptibility to jailbreaking attacks, and inflexibility regarding new safety categories.
To address these limitations, we propose $R^2$-Guard, a robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning. Specifically, $R^2$-Guard comprises two parts: data-driven guardrail models and reasoning components. The data-driven guardrail models provide unsafety probabilities of moderated content on different safety categories.
We then encode safety knowledge among different categories as first-order logical rules and embed them into a probabilistic graphic model (PGM) based reasoning component. The unsafety probabilities of different categories from data-driven guardrail models are sent to the reasoning component for final inference. We employ two types of PGMs: Markov logic networks (MLNs) and probabilistic circuits (PCs), and optimize PCs to achieve precision-efficiency balance via improved graph structure. We also propose different methods to optimize the weights of knowledge. To further perform stress tests for guardrail models, we employ a pairwise construction method to construct a new safety benchmark TwinSafety, which features principled categories and presents new challenges for moderation. We show that $R^2$-Guard is effective even given unrepresentative categories or challenging jailbreaking prompts. We demonstrate the effectiveness of $R^2$-Guard by comparisons with eight strong guardrail models on six standard moderation datasets, and demonstrate the robustness of $R^2$-Guard against four SOTA jailbreaking attacks. $R^2$-Guard significantly surpasses SOTA method LlamaGuard by 12.6% on standard moderation datasets and by 59.9% against jailbreaking attacks.
We further reveal that $R^2$-Guard can effectively adapt to safety category updates by simply editing the PGM reasoning graph.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1729701791739

thread_type:
official_review

<thread_object>
creator: reviewer_t9hr

<content>
summary:
R2-Guard is a framework that enhances safety of LLMs. Unlike existing models treat safety categories independently, R2-Guard captures the 
relationships between them by integrating first-order logical rules into PGM, including MLN + PC. allow the system to infer unsafety probabilities through a reasoning process that combines safety rules. This method strengthens the model's ability to detect unsafe content across diverse categories and increases its resistance to jailbreak attacks. Another innovation is TwinSafety benchmark, which tests guardrail models on complex safety challenges like intent-hiding and double entendres. Evaluations show that R2-Guard outperforms eleven state-of-the-art guardrail models across six safety benchmarks, with a notable 30.4% improvement over LlamaGuard on the ToxicChat dataset and a 59.5% improvement in resisting jailbreak attacks.

strengths:
1. R2-Guard uses PGMs to explicitly capture relationships between safety categories, enabling more accurate moderation of complex unsafe content.
2. It significantly outperforms state-of-the-art models, showing a 59.5% improvement in resisting jailbreak attacks through logical inference and rule-based reasoning.
3. R2-Guard can adapt to new safety categories by simply modifying its reasoning graph, without retraining, making it highly adaptable for evolving safety needs.

weaknesses:
While R2-Guard demonstrates flexibility in adapting to new safety categories by modifying the reasoning graph, it cannot cover all possible types of unsafe content by itself. Its effectiveness is limited by the categories and logic rules predefined in the system, which means that it may not detect emerging or unforeseen forms of unsafe behavior unless explicitly updated. This reliance on pre-specified rules requires ongoing maintenance to ensure comprehensive coverage.

questions:
1. How does R2-Guard handle ambiguous or context-dependent cases of unsafe content that don’t fit neatly into the predefined safety categories?
2. Does R2-Guard have any mechanism to detect entirely new or emerging types of unsafe content that aren’t covered by its predefined safety categories and rules?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730516090385

thread_type:
official_review

<thread_object>
creator: reviewer_t55i

<content>
summary:
This paper proposes R2-guard, a new guardrail mechanism for LLMs based on logical reasoning with probabilistic graphical models (PGMs).
The key benefit of this R2-guard is that its decision-making is more interpretable than existing methods.
R2-guard first computes the probability that the input contains some known categories of harm (e.g., 40% hate speech, 80% violence, etc.).
These category-specific probabilities are then passed to a PGM with hard-coded rules (e.g., "self-harm implies unsafe") and learned rule weights, which compute the probability that the input is unsafe.
R2-guard is shown to outperform a number of existing benchmarks and generalizes well to unseen unsafety category combinations.
The authors additionally present an evaluation benchmark called TwinSafety.

strengths:
This paper presents an innovative method for rule-based guardrails that combines newer techniques like LLMs with classical ones like Markov Logic Networks and Probabilistic Circuits.
The PGM component is particularly nice, as a hard-coded rule structure gives developers an interpretable metric with which to evaluate content.
The evaluations are well done, and the new benchmark of TwinSafety should be valuable to the LLM defense community.
Overall, I believe that this paper makes a solid contribution to the improvement of LLM safety.

weaknesses:
I found the presentation of R2-guard to be technically dense, even though (in my opinion) the high-level idea is simple.
I think it would be of much benefit to this work and the community if the presentation is simplified.
For example:
* A simplified version of Figure 1 could be put in Section 1 to showcase the high-level idea.
* In Section 3.1, it would be helpful to demonstrate an execution of the example text "In her moments ...".

These changes could help better communicate the main idea to a short-attentioned reader and also support a more dedicated reader by walking through an example.

questions:
It would be good if the authors included some discussion about what kinds of safety rules R2-guard might have trouble modeling.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692706749

thread_type:
official_review

<thread_object>
creator: reviewer_dyjw

<content>
summary:
Existing LLM guardrails treat different categories of safety failures independently. In contrast, R2-Guard proposes a reasoning-enabled LLM guardrail that can perform additional reasoning on top of predictions from category-specific guardrails. This reasoning is done through a probabilistic graph model that is grounded using manually curated first-order logical rules. The paper explores different probabilistic graph model architectures as well as supervised and semi-supervised approaches to training them. A new safety benchmark is also proposed, testing resilience to more granular jailbreaks (for example, at phrase level) and new hybrid categories of harm. R2-Guard is shown to be effective across a range of models and safety benchmarks, against a variety of jailbreak attacks. Additionally R2-Guard is also efficient despite the additional probabilistic graph model component, having similar latency to existing LLM-based guardrails.

strengths:
1. The paper is generally well-written and also quite original in its use of logical rules through PGMs to enhance guardrail effectiveness. 
2. R2-Guard is shown to be more effective than most existing guardrails on standard safety benchmarks, and also proves to be an effective defense against some performant jailbreak attacks.
3. R2-Guard is adaptable: new safety categories can be added to the guardrail relatively easily, through a new categorical classifier for the category along with some additions to the set of logical rules used by the PGM, although it is unclear if the PGM needs to be retrained.
4. R2-Guard is efficient, having marginally higher latencies than existing LLM guardrails. This makes it practical for real-world usecases.

weaknesses:
1. The ruleset passed to the PGM is quite small. How are these rules created? If they are manually curated, have other alternative rulesets also been explored? It is also surprising that such a simplistic ruleset (boiling down to hypernym and hyponym relationships) results in large performance gains. This could be discussed further.
2. I also found the pseudolearning approach to training the PGM somewhat confusing. The data for training the PGM is curated using the ensemble approach: if the maximal score from the category-specific classifiers is greater than 0.5, the sample is treated as harmful. Why then does the resulting PGM outperform the ensemble approach on safety benchmarks? 
3. The TwinSafety section is very lacking in details. What does "pairwise construction" mean? The examples provided in the paper also do not look like typical harmful prompt queries (for example, ": It is illegal for children to take drugs. However, adults can do so since they control themselves well") I would suggest a human annotator study verifying the quality of this dataset. 
4. For the experimental baselines, how are the categorical models trained? Why is Llama-2-7b used for the chain of thought baseline? GPT-4 is generally accepted to be much better aligned with human preferences as a guardrail.
5. Why is R2-Guard nearly perfect when combating jailbreaks? How is the model trained for Section 5.2? If it is trained on real data that contains examples of prompts with these jailbreak attacks already applied to them, it might be unfair to other baselines. For example, with GCG, there is the same suffix attached to each prompt. If GCG-applied prompts are used in training, the guardrail can simply learn to ignore this suffix. 
6. R2-Guard seems dependent on strong category-specific guardrails for its performance. Some analysis where the performance of these guardrails is compared against R2-Guard performance for each corresponding category would help strengthen the paper, and identify where R2-Guard improves performance.

questions:
1. There is a typo on line 212: "realted"
2. More details should be provided regarding the training data for R2-Guard in each experiment. In Section 5.3.1, is the R2-Guard using an MLN or PC?
3. Section 5.3.3 needs more details as well. Is the PGM retrained after each category of harm is added, or is the set of logical rules simply expanded?
4. Why does having only direct rules for the PGM improve performance? Is this equivalent to learning dynamic ensembling weights? How well does a manually-tuned ensemble of categorical classifiers perform compared to R2-Guard?
5. Ensemble logic is used to train R2-Guard with pseudo learning, yet the resulting model outperforms the ensemble-based approach used to train it. This requires more discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732267059410

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summary

comment:
We thank all the reviewers for their valuable comments and feedback! We are glad that the reviewers find our work solid and novel with sound empirical results. Based on the reviews, we have made the following updates to further improve our work.

1. We added more details on the evaluation setup and TwinSafety dataset in Sections 4 and 5, following the suggestion of Reviewer DyjW.

2. We added more analysis and empirical evidence of the advantage of $R^2$-Guard over ensemble learning, following the suggestion of Reviewer DyjW.

3. We improved the writing, such as providing an abstract overview figure and including illustrations of a running example in Section 3.1 for better understanding, following the suggestion of Reviewer t55i and Reviewer DyjW.

4. We included more discussions on the limitation of $R^2$-Guard in Section 5.3.3 and Appendix C, following the comment of Reviewer t55i.

5. We included more discussions on the application of $R^2$-Guard to open-world scenario in Section 5.3.3 and Appendix A.6, following the comment of Reviewer t9hR.

The updates in the revised manuscript are shown with highlighted color.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel approach to language model safety that combines probabilistic graphical models with traditional category-specific guardrails. The system implements additional reasoning through a PGM grounded in first-order logical rules, enabling it to capture relationships between different safety categories. The approach demonstrates superior performance compared to existing guardrails across multiple safety benchmarks, including significant improvements in resisting various types of attacks.

The authors also introduce a new benchmark designed to test guardrail resilience against more granular attacks and hybrid categories of harm. Despite its additional PGM component, the system maintains efficiency, showing similar latency to existing guardrails.

The work presents several technical strengths, including an innovative combination of classical PGM techniques with modern guardrails, offering an interpretable approach to content moderation through explicit logical rules. The system demonstrates strong empirical results across multiple benchmarks and shows significant improvements in resisting various attacks. It also shows strong potential for real-world applications due to its efficient performance, adaptability to new safety categories without full retraining, and interpretable decision-making process through explicit rules.

However, the paper has some limitations. It lacks detailed discussion of the process for creating and selecting logical rules, exploration of alternative rulesets, and justification for why such a simple ruleset yields significant improvements. There are also clarity issues regarding the pseudo-learning approach for PGM training and the relationship between ensemble-based training and final performance. The benchmark documentation could be improved with clearer methodology description and validation of dataset quality.

Despite these limitations, the paper merits acceptance based on its strong technical merit, practical impact, and value to the research community. The system presents a novel and effective approach to safety, combining classical and modern techniques innovatively. It shows immediate practical value through its efficiency, adaptability, and strong performance. While the paper would benefit from revisions addressing documentation and analysis gaps, its core contributions are significant enough to warrant acceptance.