Title: RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine Unlearning
Abstract: The widespread deployment of Large Language Models (LLMs) trained on massive, uncurated corpora has raised growing concerns about the inclusion of sensitive, copyrighted, or illegal content. This has led to increasing interest in LLM unlearning: the task of selectively removing specific information from a model without retraining from scratch or degrading overall utility.
However, existing methods often rely on large-scale forget and retain datasets, and suffer from unnatural responses, poor generalization, or catastrophic utility loss.
In this work, we propose $\textbf{R}$einforcement $\textbf{U}$n$\textbf{LE}$arning ($\textbf{RULE}$), an efficient framework that formulates unlearning as a refusal boundary optimization problem. RULE is trained with a small portion of forget set and synthesized boundary queries, using a verifiable reward function that encourages safe refusal on forget-related queries while preserving helpful responses on permissible inputs.
We provide both theoretical and empirical evidence demonstrating the effectiveness of RULE in achieving targeted unlearning without compromising model utility. Experimental results show that, with only 12\% forget set and 8\% synthesized boundary data, RULE outperforms existing baselines by up to $17.4\%$ forget quality and $16.3\%$ naturalness response while maintaining general utility, achieving $\textit{forget-retain Pareto Optimality}$. Remarkably, we further observe that RULE improves the $\textit{naturalness}$ of model outputs, enhances training $\textit{efficiency}$, and exhibits strong $\textit{generalization ability}$, generalizing refusal behavior to semantically related but unseen queries.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new LLM unlearning method: RULE, which applies reinforcement learning such as GRPO in LLM unlearning training. By optimizing against a reward considering the unlearning effectiveness based on final text response, this method effectively removes unwanted knowledge and maintain LLM itself.\nThe experiments show that it outperforms previous methods in improving unlearning performance and maintaining answer naturlness.\n\nSTRENGTHS AND WEAKNESSES:\n> Strengths: \n* Applying GRPO-like RL in LLM unlearning is reasonable to me.\n* The experiment performance is strong. Compared to previous unlearning methods, RULE achieves high language naturalness and unlearning performance.\n\n> Weakness:\n* RULE needs a boundary dataset synthesis, which currently is implemented by prompting GPT-4o to replace name entities in the original question/answer pairs. I'm concerned that this procedure introduces noises on other name entities that should be retained in the final model. This phenomenon is also highlighted in a previous work [1].\n* RULE heavily relies on the reward design since it is involves RL training. The current reward implementation is mainly a regex string match function as shown in Appendix C. I'm concerned that there may be some form of reward hacking phenomenon, which is very common in previous RL literature.\n* I'm not fully convinced by the claim about efficienc improvement. Since RULE is a RL training method, which involves large-scale online response generation process, and takes long time in training. For example, from my personal experience, it takes around 5 min for one step of RL training on 8 H-100 for a 7B LLM. Currently the efficiency is compard through training data size, which is not fair in claiming efficiency from my point of view.\n\n* Minor:\n\t* Appendix seems not using the Neurips template.\n\t* Some of the figures are hard to read, e.g., Figure 2 (a) the red performance difference text is hard to differentiate agains background bar, and Figure 3 x/y axis texts are too small.\n\n[1] Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective https://arxiv.org/pdf/2407.16997\n\nQUESTIONS:\n* Previous RL literature mentions length growing in math, what happens to the unlearning training? Does it slow down the generation for final LLM?\n* Some previous works also involve synthesizing additional retain data [1,2], what's the difference between the data synthesized in this work? \n\n[1] Who’s Harry Potter? Approximate Unlearning in LLMs https://arxiv.org/pdf/2310.02238\n\n[2] Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference https://proceedings.neurips.cc/paper_files/paper/2024/hash/171291d8fed723c6dfc76330aa827ff8-Abstract-Conference.html\n\nLIMITATIONS:\nPlease see the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RULE, a new method for unlearning in large language models (LLMs). RULE leverages a sample-efficient, reinforcement learning (RL)-based training pipeline designed to enforce high-quality refusal responses for inputs similar to the forget set, while maintaining appropriate replies for content intended to be retained.\n\nThe training pipeline consists of several steps:\n1. Teaching Refusal Behavior: Since refusal behavior is uncommon in most pre-trained models, the first step is to train the model to effectively refuse certain requests.\n2. Constructing a High-Quality Refusal Boundary: This involves generating a boundary set using a SOTA model (GPT-4o). The boundary set contains samples that should be retained but closely resemble the forget set, helping to delineate when to refuse.\n3. Reinforcement Learning Training: The model undergoes on-policy RL training. The reward mechanism encourages refusal of responses related to the forget set, including relevant disclaimers (e.g., “I cannot provide an answer relating to this copyrighted work of Stephen King”). For the boundary set, the model aims to preserve responses close to those of the original model, with similarity assessed using the ROUGE-L metric.\n\nThe authors' findings highlight that RULE delivers superior refusal quality for the forget set compared to baseline methods, even with substantially less data. While this comes at the expense of reduced retention quality, empirical results suggest that RULE strikes an efficient balance between forgetting and retaining content. Despite these significant benefits, the pipeline suffers from general complexity, with the construction of the boundary set serving as a crucial insufficiently explored step.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Principled Decision Boundary: Enforcing a decision boundary effectively enhances the forget-retain trade-off.\n2. Meaningful Refusal Answers: The approach of providing semantically meaningful refusal answers is superior to uncontrolled responses.\n3. Theoretical Foundation: The method’s effectiveness is theoretically supported, as detailed in the appendix.\n4. High-Quality Writing: The manuscript is well-written and comprehensive in form.\n5. Reproducibility: The availability of code and a detailed appendix facilitates easy reproduction of the work.\n\nMajor Weaknesses\n1. Dependency on Manual Annotation or LLM: The need for manual annotation or LLM-based generation for the boundary set introduces potential confounding factors, with limited exploration of the LLM-PROMPT pair's impact on RULE performance.\n2. Training Data and Optimization Guidance: The paper lacks clear guidance on the quantity of training data and the number of optimization iterations required for the Rejection-Steering phase, which are critical for pipeline effectiveness.\n3. The method is generally complex and involves an ample design space for all proposed components, making it potentially brittle at scale. \n\nMinor Weaknesses\n1. Opaque Qualitative Evaluation: Utilizing an LLM as a judge for qualitative assessments reduces interpretability.\n2. Limited Similarity Metrics: Relying solely on ROUGE-L for similarity assessment without reference to the model's latent space may limit the effectiveness of the reward signal. For example, it is unclear what the reward should be for answering with the same content as the base model, but in a different language.\n3. Scalability of Refusal Templates: The linear growth of predefined refusal templates with additional languages necessitates either manual translations or automated translation tools.\n4. Lack of Computational Cost Analysis: I could not find an indication of how RULE’s computational cost scales with the forget set size and model size. The appendix provides raw values for the number of GPU hours required for each phase.\n5. No Exact Unlearning Baseline: The paper does not compare with exact unlearning, which involves re-training the model with the forget set completely excluded.\n\nTypos:\n\n1. Line 117: “making it difficult to ensure the utility” -> “making it difficult to ensure the utility of the model.”\n2. Line 144: “We introduce a set of boundary set” -> “We introduce a boundary set.”\n\nQUESTIONS:\n1. Have you considered how the effectiveness of RULE might vary depending on the LLM used to construct the boundary set? For example, what differences emerge when using:\n- Reasoning models from online vendors\n- Local models\n- Models specifically fine-tuned to create high-quality boundary sets\n2. Additionally, how does the cost-benefit trade-off of RULE shift when factoring in the inference costs associated with the LLM used for constructing the boundary set?\n3. Regarding refusal steering, what would be the optimal size for the forget set, and how many iterations does this process typically require compared to the main RL loop? Does this vary with the model size?\n4. When considering the overall pipeline, how does it scale relative to the sizes of both the forget set and the model itself?\n5. During experimentation, did you explore simpler methods for various components? \n- For instance, instead of synthetically generating the boundary set, you could use sentence embeddings to identify similar pairs between the forget and retain sets, using that retain subset as the boundary set.\n- Alternatively, leveraging the model’s hidden state to assess similarity to the original model—or to specific entities for the forget set—could offer valuable insights. For example, using \"Washington\" as a metonym may not be effectively captured by the current approach, but could be when using embeddings.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper addresses the critical problem of targeted unlearning in Large Language Models (LLMs), where specific information must be removed from a model without compromising its overall utility. The authors propose a novel framework, Reinforcement UnLEarning (RULE), which formulates unlearning as a boundary optimization task solved via reinforcement learning. RULE leverages a small forget set and synthesized boundary queries, guided by a reward function that balances safe refusal and helpfulness.

STRENGTHS AND WEAKNESSES:
Strength:

1.	The authors have innovatively explored the application of reinforcement learning in the context of unlearning.
2.	The authors’ method shows a notable improvement compared to the baseline.
3.	The paper provides both theoretical and empirical evidence that RULE achieves a superior trade-off between forgetting target information and preserving general model utility. 
4.	The paper demonstrates remarkable data efficiency, using "only 12% forget set and 8% synthesized boundary data" compared to baselines that require 100% of both forget and retain datasets. This makes it more practical for real-world scenarios where large, cleanly partitioned datasets are often unavailable.

Weakness:

1. The reward design proposed by the authors appears to suffer from very sparse rewards. Have the authors attempted to address this issue?
2. In the Step 3 of Appendix B, The conditions for applying the martingale inequality do not appear to have been proven to hold.

QUESTIONS:
See weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02