Title: RSafe: Incentivizing proactive reasoning to build robust and adaptive  LLM safeguards
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

safety

moderation
Abstract: Large Language Models (LLMs) continue to exhibit vulnerabilities despite deliberate safety alignment efforts, posing significant risks to users and society. To safeguard against the risk of policy-violating content, system-level moderation via external guard models—designed to monitor LLM inputs and outputs and block potentially harmful content—has emerged as a prevalent mitigation strategy. Existing approaches of training guard models rely heavily on extensive human curated datasets and struggle with out-of-distribution threats, such as emerging harmful categories or jailbreak attacks. To address these limitations, we propose RSafe, an adaptive reasoning-based safeguard that conducts guided safety reasoning to provide robust protection within the scope of specified safety policies. RSafe operates
in two stages: (1) guided reasoning, where it analyzes safety risks of input content through policy-guided step-by-step reasoning, and (2) reinforced alignment, where rule-based RL optimizes its reasoning paths to align with accurate safety prediction. This two-stage training paradigm enables RSafe to internalize safety principles to generalize safety protection capability over unseen or adversarial safety violation
scenarios. During inference, RSafe accepts user-specified safety policies to provide enhanced safeguards tailored to specific safety requirements. Experiments demonstrate that RSafe matches state-of-the-art guard models using limited amount of public data in both prompt- and response-level harmfulness detection, while achieving superior out-of-distribution generalization on both emerging harmful category and jailbreak attacks. Furthermore, RSafe provides human-readable explanations for its safety judgments for better interpretability. RSafe offers a robust, adaptive, and interpretable solution for LLM safety moderation, advancing the development of reliable safeguards in dynamic real-world environments. Our code is available at https://anonymous.4open.science/r/RSafe-996D.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents RSafe, a two-stage, reasoning-based guard model designed to improve robustness and adaptability of LLM safety moderation. In Stage 1 (Guided Reasoning), RSafe elicits explicit, policy-aware step-by-step reasoning (enclosed in <think>…</think> tags) over a user-specified safety taxonomy; in Stage 2 (Reinforced Alignment), it applies rule-based reinforcement learning (GRPO) with format and accuracy rewards to refine reasoning paths for correct safety judgments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. By grounding decisions in explicit, user-specified safety policies, RSafe generalizes beyond fixed training taxonomies to novel harmful categories and adversarial jailbreaks, a capability lacking in traditional guard models.\n\n\n2. The enforced <think>…</think> schema yields verifiable reasoning traces at both train and test time.\n\n\nWeaknesses\n1. At its core, RSafe applies reinforcement learning with two straightforward reward components—format compliance and classification accuracy. While effective, this combination follows established training patterns with minimal adaptation. \n\n2. The guided reasoning and RL are only conducted on Qwen2.5-7B. Weaker backbones may yield shallow rationales, limiting safety gains. A controlled comparison with GuardReasoner is needed.\n\n3. The mechanism by which dynamic safety categories are integrated into training is described only at a high level. Critical implementation details are omitted. Providing a complete account of data curation and training schedules would improve reproducibility and allow readers to gauge practical overhead.\n\n4. The adaptive mode presumes users can accurately define relevant safety taxonomies; in practice, overlooking critical policy categories (e.g., emergent threats) may degrade performance. No guidance or fallback is provided when policies are mis-specified.\n\nQUESTIONS:\nSee weaknesses part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RSafe, a system‑level guard model for LLMs that (i) adds an explicit reason‑then‑predict stage (“Guided Reasoning”) and (ii) aligns this behaviour with a lightweight rule‑based reinforcement‑learning procedure (“Reinforced Alignment”, based on GRPO). This exhibits generalisation to emerging harmful categories and jailbreak attacks (WildGuardTest). At inference time, users can supply arbitrary safety policies; RSafe reasons over the supplied categories without retraining. The rationale is also always returned, enabling interpretability in the guard model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nThe primary strength seems to be that RSafe addresses, through a unique RL objective, is the need for guardrail models that adapt with different safety policies and user-specified safety taxonomies by accepting arbitrary user‑supplied safety taxonomies at runtime. This drop‑in policy swap is done without retraining; a capability missing in prior guard models. This can generalize to various safety violation scenarios, such as intellectual property, which are more out-of-distribution. Furthermore, the reasoning is done for each safety policy.\n\n**Weaknesses:**\n- The experiments do not test the adaptiveness of the policy, despite this being the primary advantage that is proclaimed by RSafe -- see \"Questions\" and \"Limitations\" for more experiments I'd like to see done. While there was an evaluation on WildGuardTest, which shows increased performance on out-of-distribution adversarial examples, this is distinct from being able to show that safety policies can change and RSafe can adapt to different safety needs.\n- The experiments shown do not demonstrate a significant performance improvement over other open-source guardrail models.\n\nQUESTIONS:\nThe primary claim of the paper is its ability to accept various user-supplied safety taxonomies, **allowing developers to include and exclude safety policies at will**. This is mentioned as one of the most significant contributions of the paper on page 3. However, the experimental results do not necessarily investigate this claim in depth, beyond finding improved performance on a pre-established adversarial examples dataset.\n\nMany questions I would like to see answered are around providing experimental evidence to strengthen or weaken this claim. Primarily:\n1. One evaluation that would be interesting would be to see how RSafe handle entirely idiosyncratic, edge-case, or otherwise highly specific user‑defined safety rules, particularly when compared with GPT‑4o or Claude.\n2. How does RSafe work when certain safety metrics are excluded (e.g. are users/developers able to exclude copyright if they do not care about this specific risk, and \"remove refusals\" related to copyright?). Can it delineate and clearly change safety taxonomies compared with other baseline guard options?\n3. Is there possibility to expand the testing of robustness beyond WildGuardTest (Section 4.2), toward evals that demonstrate that it can follow instructions with respect to novel safety criteria?\n\nBoth positive and negative results would strengthen my confidence and rating.\n\nLIMITATIONS:\nThe main limitation has already been mentioned above -- centering around addressing the claim that the safety policy of RSafe can be dynamically aligned with task-specific requirements.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RSafe – an adaptive safeguard for large language models that first performs policy-guided, safety reasoning and then applies rule-based reinforcement learning to align those reasoning paths with correct safety judgments. This two-stage paradigm enables the model to generalize across different safety principles rather than merely memorize labels, letting it moderate both prompts and responses under user-specified policies. Experiment results demonstrate its effectiveness across different benchmarks compared to baselines such as Llama Guard and ShieldGemma.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper introduces an adaptive safeguard that couples policy-guided, step-by-step safety reasoning with rule-based policy optimization, producing a method that is both promising and empirically effective.\n\n2. The manuscript is clear and well structured: the problem statement, algorithmic details, and experimental protocols are presented in a logical sequence, making the work easy to follow.\n\n3. The evaluation is comprehensive. It spans several model scales, safety benchmarks, and jailbreak settings, and the authors provide both quantitative metrics and qualitative analyses that support their claims.\n\n**Weaknesses**\n\n1. The proposed formulation is closely related to the GuardReasoner framework. The principal difference is that GuardReasoner relies on DPO, whereas the present work adopts GRPO with verifiable rewards such as accuracy and format compliance. This change, while sensible, may appear incremental and therefore limits the novelty of the contribution.\n\n2. In the robustness experiment on unseen safety categories, the “Criminal Planning / Confessions” topic can overlap with categories already covered by Llama-Guard (violent crime) and ShieldGemma (dangerous content). To better demonstrate adaptiveness to truly novel topics, I suggest (i) choosing a domain unrelated to safety—e.g., a keyword-mentioning or mathematics discussion task—and (ii) adding an LLM-as-judge baseline (such as GPT-4o) to evaluate safeguard reliability when judgment is delegated to an external model.\n\nQUESTIONS:\nPlease respond to the weaknesses above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-06"}

comments: SUMMARY:
One natural approach to model safety is training/constructing an unsafe input/output filter. This paper proposes training the input/output filter using RL instead of the standard fine-tuning approach utilized by prior works. Furthermore, they enable adaptive classification by allowing users to specify content policies in natural text.

STRENGTHS AND WEAKNESSES:
Strengths
1. The paper framing/motivation is incredibly clear. The writing is also easy to follow.
2. The results/baselines are clearly described, as well as showing string gains in the WildGuardTest setting.

Weaknesses
1. The method lacks some novelty. For example, training an input/output filter and supporting adaptive policies has been introduced in prior works such as LlamaGuard. The main contribution seems to be limited to utilizing RL
2. For the WildGuardTest evaluation, don't the guard models also admit adaptive variants by specifying policies? Do these methods match performance?
3. [Minor] Is it possible to report averaged numbers as well for Table 1 for easier comparison?

QUESTIONS:
Asked in weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-07