Title: Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Safe Reinforcement Learning

Reinforcement Learning with Human Feedback

Large Language Model

Safety Alignment
Abstract: Balancing helpfulness and safety (harmlessness) is a critical challenge in aligning large language models (LLMs). Current approaches often decouple these two objectives, training separate preference models for helpfulness and safety, while framing safety as a constraint within a constrained Markov Decision Process (CMDP) framework. This paper identifies a potential issue when using the widely adopted expected safety constraints for LLM safety alignment, termed "safety compensation'', where the constraints are satisfied on expectation, but individual prompts may trade off safety, resulting in some responses being overly restrictive while others remain unsafe. To address this issue, we propose **Rectified Policy Optimization (RePO)**, which replaces the expected safety constraint with critical safety constraints imposed on every prompt. At the core of RePO is a policy update mechanism driven by rectified policy gradients, which penalizes the strict safety violation of every prompt, thereby enhancing safety across nearly all prompts. Our experiments demonstrate that RePO outperforms strong baseline methods and significantly enhances LLM safety alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe objective of this paper is align large language models (LLMs) to balance helpfulness and safety (i.e., harmlessness). Recent progress on safety alignment, represented as Safe RLHF or SACPO, has been made by solving constrained policy optimization problems under the CMDP framework. The authors identified an issue in the previous works that used expected safety constraints, potentially leading to overly conservative LM policies or safety violations. The authors propose Rectified Policy Optimization (RePO) that defines a problem with critical safety constraints imposed on every prompt. Their experiments show that RePO outperforms strong baseline methods and significantly enhances LLM safety alignment.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- This paper is well-written and organized. I found this paper is easy to follow.\n- The key ideas behind the problem formulation of RePO are well-explained. \n- The empirical evaluation is conducted in a standard manner of safety alignment research.\n- The empirical results are very strong. I see that RePO exhibits large improvements compared to SafeRLHF and SACPO.\n- The supplementary materials are nice. Please consider releasing the code as well.\n\n### Weaknesses\n\n- While I think Theorem 2 is correct, I am wondering why the authors present bounds with respect to $V^r$ and $V^c$ rather than $R$ and $C$. I consider that it is fairly ok to present Theorem 2, but this theorem does not lead to the theoretical guarantee that the policy obtained by RePO is critically safe.\n\n- It is unclear what is the relation between $R$ and $V^r$ or that between $R$ and $r$.\n\nQUESTIONS:\nQ1: Could you tell me the reason why Theorem 2 is characterized by $V^r$ and $V^c$, rather than $R$ and $C$? Why does Theorem 2  demonstrates RePO’s ability to guarantee safety while maintaining optimality in the sense of Problem (4)? Perhaps, the authors want to argue that Problem (4) is a special case with $\\gamma=1$ and horizon is equal to 1?\n\nQ2: Regarding the following statements in lines 197-199, could you elaborate more? I agree with the following arguments, but I consider that the Slater's condition holds in most cases; thus, strong duality is also usually met. Could you tell me some scenarios where the Slater’s condition does not hold?\n\n> Note that our RePO algorithm is different from the primal-dual methods Dai et al. [7], Huang et al. [16], Wachi et al. [41], which rely on the strong duality of CMDP with the expected constraints [ 27 ]. The property of strong duality is very likely to fail in CMDP with strict constraints where the Slater’s condition does not hold.\n\nLIMITATIONS:\n- If I understand correctly, there is no discussion regarding limitations, while the authors mentions that \"We discuss the future work in conclusion\" in line 517.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on training LLMs to generate helpful and harmless responses. Building on the SafeRLHF training framework, the authors particularly examine the constrained policy optimization phase and address the issue of safety compensation, which arises from the original CMDP formulation with expected safety constraints. To tackle this, they propose replacing the expected cost-based safety constraints with critical safety constraints, based on a rectified safety metric. Experimental results, compared against state-of-the-art baselines, show that the proposed method, RePO, significantly improves model safety.\n\nSTRENGTHS AND WEAKNESSES:\n**[Strengths]**\n\nThis paper addresses the important task of maintaining both helpfulness and harmlessness during LLM fine-tuning. The authors identify and demonstrate a key issue in the SafeRLHF framework, providing a strong motivation for their work. They focus on the constrained policy optimization phase, propose a rectified constraint metric, and support it with both theoretical analysis and a practical algorithm. The effectiveness of the proposed approach is further validated through comprehensive experimental evaluations.\n\n**[Weaknesses]**\n\nPlease refer to the Questions and Limitations below.\n\nQUESTIONS:\n1. The paper states that \"$\\lambda$ is not a Lagrange multiplier used in the traditional primal-dual method, but rather a non-decreasing rectified penalty.\" I would appreciate a more detailed clarification of how $\\lambda$ is defined in your method, and why it diverges from the classical use of Lagrange multipliers in constrained optimization.\n\n2. Furthermore, how is the value of $\\lambda_{max}$ chosen in practice? It would be helpful to understand its influence on the training dynamics and model performance, as well as any sensitivity analysis conducted around this hyperparameter.\n\n3. In Section 3.2, it is unclear whether the cost preference setup follows that of SafeRLHF. The notational reuse suggests that the same structure is adopted, but it is not explicitly stated whether the preferences are defined over cost or reward. Given that SafeRLHF defines preferences in terms of cost, if your method does the same, I strongly encourage you to revise the section to make this distinction clearer. Reusing the same notation for both reward and cost preferences leads to unnecessary confusion.\n\n4. The XS-Test experiments are introduced in the main paper without sufficient context. Readers unfamiliar with this benchmark may struggle to understand the purpose of the experiments and how to interpret the results until they reach the appendix. I recommend including a brief description of XS-Test in the main text to make the paper more self-contained.\n\n5. While the paper checklist claims that the authors have discussed the limitations of the work, no such discussion appears in the main paper. A single vague sentence about future work does not fulfill the requirement for a limitation discussion.\n\n6. Additionally, I was unable to locate the discussion of broader impacts, which is mentioned in the paper checklist that exists in the appendix. In case I missed it, could you please direct me to the section where broader impacts are discussed?\n\n7. Writing and Notation Suggestions: \n> (a) Please ensure that all symbols are clearly introduced upon first use, such as, the function $C_\\psi(\\cdot,\\cdot)$ in line 110, as well as symbols like $\\eta$, $\\alpha$, $\\rho$, appear without clear definitions. (b) In line 112, is it a typo as you are referring to a cost function? (c) Is it a typo of $V_\\rho^c$ in Theorem 2? (d) The symbols $\\phi$ and $\\psi$ were used to define both the reward and cost models, as well as the critic networks' parameters. This can lead to confusion and should be resolved through more distinct notation.\n\nLIMITATIONS:\nThis work can be viewed as an extension of SafeRLHF aimed at addressing the safety compensation problem. I appreciate the authors' efforts, as well as the theoretical and empirical support provided. However, the paper lacks discussion on how this SafeRLHF framework compares with DPO-like approaches in similarly safety-constrained settings (e.g., SACPO). Both SafeRLHF and RePO require maintaining multiple auxiliary models in addition to the policy model, such as reward and cost models, as well as two critic networks, which significantly increases computational overhead. Despite RePO's superior performance, I believe a discussion of these limitations, along with potential directions for future work, would be a valuable addition to the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper studies reinforcement learning formulations for safety alignment. It starts with highlighting the limitations of the expected safety constraint adopted in previous work: safe responses can compensate for unsafe responses, leading to suboptimal outcome. The paper then introduces the "critically safe" formulation which imposes a safety constraint on individual prompt-response pairs. The paper then introduced the rectified policy optimization algorithm to solve for the introduced problem. The presented method is shown to outperforms previous formulations in safety.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The introduced formulation is sound and well-motivated.

2. The presented algorithm is shown to yield significant empirical results over baselines.

Weaknesses:

1. The presented algorithm results in significantly higher over-refusal rate (20.4% vs. 6.8%), which is concerning by itself. Besides, the paper needs to provide some intuition or (qualitative) analysis to explain that behavior.

2. The paper lacks a discussion of the stability and scalability of the algorithm. It was only applied to a 3B and a 7B model in the experiments. 

3. It is not clear why the paper introduces $y_0^{(i)}$ in line 109. It was never used again! Also, line 112 has a typo "can be get".

QUESTIONS:
Please see weaknesses above.

LIMITATIONS:
No. The paper needed to discuss the scalability, cost, and stability of the presented algorithm compared to previous work. Also, over-refusal seems to be a major problem of the algorithm.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies \"safety compensation\" as an issue where satisfying safety constraints on average allows some LLM responses to be unsafe while others are overly restrictive. To address this, the authors propose Rectified Policy Optimization (RePO), an algorithm that imposes critical safety constraints on every prompt instead of relying on an overall expected value. RePO works by using rectified policy gradients to penalize any strict safety violation, thereby focusing on improving helpfulness only when safety is guaranteed for a given prompt-response pair. Experiments were conducted by fine-tuning Alpaca-7B and Llama3.2-3B models. The results showed that RePO outperformed baseline methods, enhancing LLM safety alignment by increasing the proportion of safe outputs while maintaining helpfulness. For the Alpaca-7B model, RePO achieved a 96.08% safety rate in model-based evaluations and a 90.04% rate in GPT-4 evaluations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper is well-motivated, addressing a clearly defined problem termed \"safety compensation\". It effectively argues that models can be \"expected safe\" by satisfying safety constraints on average, yet fail to be \"critically safe\" by still producing harmful content for individual prompts\n- The paper provides a theoretical foundation for its method, presenting theorems that connect the proposed algorithm to the goal of achieving critical safety. The authors support these claims with proofs and provide detailed experimental settings in the appendices, which contributes to the clarity and potential reproducibility of the work.\n\n**Weaknesses:**\n- My main concern is the Incremental novelty of this paper. The proposed RePO algorithm could be viewed as an incremental improvement from the paper ([SafeRLHF](https://arxiv.org/pdf/2310.12773)). The core mechanism involves separating the training batch into \"safe\" and \"unsafe\" subsets and applying the cost-related penalty term only to the \"unsafe\" samples. While the paper provides a theoretical rationale for this \"rectified\" design, the implementation is a modification of the objective function within a framework very similar to that used by its primary baseline, SafeRLHF.\n- The paper's central motivation rests on the concept of \"safety compensation\". However, the evidence supporting the prevalence of this issue is primarily based on a single set of experiments illustrated in Figure 1 and a hypothetical example. The argument could be significantly strengthened by including more diverse quantitative evidence or qualitative analysis, such as examples of actual model outputs, to more concretely demonstrate the problem in practice.\n- The paper's empirical validation relies heavily on assets from a single line of prior work, SafeRLHF. Specifically:\n  - The fine-tuning process uses prompts from the PKU-SafeRLHF dataset.\n  - The reward and cost signals are generated by the beaver-7B-v1.0 models.\n  - SafeRLHF is used as the main baseline for comparison.\n  \n  While this critique may be beyond the scope of this paper's evaluation, it is worth noting that as a close follow-up to SafeRLHF, the proposed method inherits its predecessor's methodological inflexibility. The reward-cost formulation requires preference pairs for helpfulness and harmlessness to be annotated on the same set of prompts—a criterion that, to my knowledge, perhaps only the BeaverTails and PKU-SafeRLHF datasets currently meet. Consequently, testing the efficacy of the proposed RePO method on other preference datasets is challenging due to this inherited incompatibility with the SafeRLHF formulation. While the authors perform robustness checks using an \"unseen internal evaluator\" and an \"external OOD dataset\", the core findings remain situated within a narrow experimental ecosystem, which may not fully reflect the method's generalizability.\n\nQUESTIONS:\nNo.\n\nLIMITATIONS:\nAlthough the justification provided by the authors: \"We discuss the future work in conclusion.\" in the Limitation section of the checklist, very little about the method's limitations has been discussed. I kindly suggest that the authors be more upfront about the weakness of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-06"}