Title: Explainable Reinforcement Learning from Human Feedback to Improve Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: explainable reinforcement learning from human feedback

alignment
Abstract: A common and effective strategy for humans to improve an unsatisfactory outcome in daily life is to find a cause of this outcome and correct the cause. In this paper, we investigate whether this human improvement strategy can be applied to improving reinforcement learning from human feedback (RLHF) for alignment of language models (LMs). In particular, it is observed in the literature that LMs tuned by RLHF can still output unsatisfactory responses. This paper proposes a method to improve the unsatisfactory responses by correcting their causes. Our method has two parts. The first part proposes a post-hoc explanation method to explain why an unsatisfactory response is generated to a prompt by identifying the training data that lead to this response. We formulate this problem as a constrained combinatorial optimization problem where the objective is to find a set of training data closest to this prompt-response pair in a feature representation space, and the constraint is that the prompt-response pair can be decomposed as a convex combination of this set of training data in the feature space. We propose an efficient iterative data selection algorithm to solve this problem. The second part proposes an unlearning method that improves unsatisfactory responses to some prompts by unlearning the training data that lead to these unsatisfactory responses and, meanwhile, does not significantly degrade satisfactory responses to other prompts. Experimental results demonstrate that our algorithm can improve RLHF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to further improve the effectiveness of RLHF methods after they are finished. It focuses on improving the performance on the part of the validation dataset where the model did not perform well. This is done by first identifying the training data (in the binary preference dataset) which are most similar to each problematic prompt-response pair, then performing some unlearning in the reward model and some fine-tuning for the policy model. Experiments show overall better win rates of the proposed approach over existing RLHF approaches (PPO, REMAX) separately on two datasets. Performance tradeoffs are observed on the non-problematic part of the validation dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-presented with clear problem definitions and good visualisations. The mathematical formulations are also very easy to follow.\n2. The methodology sections are sound and well-explained.\n\nWeaknesses:\n1. Clarity and Significance: The paper claims that the proposed approach is explainable. However, discussions on explainability are very shallow. The most relevant part might be lines 138-144, but these motivations hint that the method is aiming to explain why the reward model assigns a high reward to the policy’s response, rather than to explain why the policy LLM generates such a response. There are also no real evaluations for explainability, as all evaluations are regarding win rates. Therefore, using the word explainable to describe this paper is not appropriate.\n\n2. Clarity and Significance: Motivations and assumptions are lacking for the method. As mentioned in the paper, LLM post-training involves SFT, learning a reward model, and running RL. However, the proposed approach deems that the cause of unsatisfactory LLM behaviour lies in the preference training data for the reward model. This dataset is usually not directly seen by the LLM itself, so it is questionable to ignore the other data sources where the reasons why the LLM unsatisfactory behaviour might exist. This is a fundamental aspect to consider, but it is not discussed in the paper. (While in the experiments the three steps use the same dataset, lacking such discussions will also harm the generalisability, hence the significance of the proposed approach.)\n\n3. Quality: Writing (apart from the mathematical formulations, those are good) is often too simplistic or unconvincing. For example, in abstract and intro, discussions on *cause* are very vague; in Line 25, one should not expect a model to become all perfect after some training; in Preliminary section, high-level descriptions for the equations behind RM and RL would be helpful. Conclusions and related works are very incomplete; future works and limitations are not well-discussed.\n\n4. Quality and Significance: Evaluations are thin and a bit too narrow. \n(1) Only 2 datasets and 3 small models are involved. \n(2) The choices of the open-source reward models used to evaluate LLM responses are not well-justified – neither is state of the art, especially the deberta model. It’d be good to refer to the rewardbench leaderboard to find reasonably sized and good ones. While GPT-4 is also used, it is suggested that the order in which the candidate responses are presented be randomised – currently in the code, it’s always showing the response by the proposed method first and then the baseline response (if I’m not mistaken). \n(3) The scope of evaluation is narrow, as only the win rate metric is used on only one dataset in each experiment. It is not clear whether the proposed RLHF approach worsens the LLMs’ general performance, e.g., on other benchmarks, especially as some unlearning is involved.\n\nQUESTIONS:\n1. It is claimed that the method finds the causes of why the policy model generated unsatisfactory responses. What is your definition of a cause here? There are the research fields on explainable AI and causality / actual causes. How does your definition fit in? \n2. Why does identifying relevant examples in the preference dataset, which was only used to train the reward models and not directly the policy model, counts as explaining the unsatisfactory generations?\n3. Regarding the approach for separating the evaluation dataset into satisfactory and unsatisfactory subsets – how valid is setting a threshold using a reward model for this purpose? In line 608 you mentioned “the score lower bound of the satisfactory group is usually above the score upper bound of the unsatisfactory group” which does not seem clear. My intuitions are that it is difficult to compare the scalar outputs of the reward models for responses across different input prompts, because the prompts themselves will influence the magnitude of the scores?\n4. How's the reward models' performance after unlearning?\n\nLIMITATIONS:\nAll parts about explainability should be revisited with clear discussions on motivations and assumptions. The evaluations are not comprehensive enough to support the relevant contributions made in the paper. See detailed comments on weaknesses and questions. I think those are essential aspects to elaborate on in the paper's main text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method , named Explainable Reinforcement Learning from Human Feedback (XRLHF), which operates as a post-training step for RLHF and is inspired by the human strategy of identifying and correcting the cause of an undesirable outcome.\n\nThe paper's primary contributions are the novel formulation of explaining unsatisfactory responses by identifying causative training data and the two-phase unlearning method to improve the model without complete retraining.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper introduces an innovative two-stage \"explain-and-correct\" framework to identify and fix unsatisfactory responses from language models trained with RLHF. As a post-training method, it offers a highly practical way to improve models that are already deployed or expensive to retrain.\n2. The method is technically sound. It creatively formulates the \"explanation\" step as a constrained optimization problem and designs a tailored \"unlearning\" strategy for the RLHF pipeline, which first corrects the reward model and then fine-tunes the policy model.\n3. The paper is clearly written and well-organized. Figure 1, in particular, intuitively illustrates the entire workflow, which greatly aids reader comprehension.\n\n**Weaknesses:**\n\n1. The essence of Explainable RLHF (XRLHF) is a data filtering mechanism, but the proportion of data being filtered has not been well-studied through ablation.\n2. The process of using a reward model to filter the dataset D inherently introduces bias.\n3. Regarding Table 4, on which types of satisfactory samples does the model's performance degrade? Is this trade-off controllable?\n4. The evaluation is only conducted on an in-distribution test set. Does the unlearning process affect the model's generalization performance on other benchmarks (e.g. ApacaEval)?\n5. There is no analysis subsection dedicated to the data filtered by the explainable method. Could there be an experiment that uses a reward model to analyze the score distribution of the rejected and chosen answers for the unsatisfactory samples?\n\nQUESTIONS:\n1. Regarding Table 4, on which types of satisfactory samples does the model's performance degrade? Is this trade-off controllable?\n2. Does the unlearning process affect the model's generalization performance on other benchmarks (e.g. ApacaEval)? And what about performance of XRLHF on Llama3 / Qwen2？\n3. Could there be an experiment that uses a reward model to analyze the score distribution of the rejected and chosen answers for the unsatisfactory samples? In my opinion, this is a prerequisite for XRLHF to be effective.\n4. Doesn't it seem a bit strange that Algorithm 1 in the paper named XRLHF doesn't contain the RLHF (unlearning) process?\n\nIf these concerns can be addressed, I will be glad to increase my rating.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper presents a method to identify the causes of low-quality responses in RLHF settings. The core idea is to locate the most relevant preference pairs in the reward model training data that are associated with unsatisfactory outputs, and then perform unlearning on these data. Experimental results demonstrate the method’s effectiveness on dialogue generation and summarization tasks.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper is clearly written and easy to follow.

2. The problem addressed is highly interesting and important: identifying the root causes of unsatisfactory model outputs and mitigating them through unlearning. The proposed solution is well-motivated and carefully designed.

Weaknesses:

The base model is too weak, and models from two years ago are not sufficiently convincing given the current progress in large language models. Since policy unlearning based on negative gradient updates has already been widely used, the minimum-distance-based iterative training data selection is the main contribution of this paper. However, its effectiveness needs to be demonstrated on more recent and advanced models through broader and more up-to-date experiments.

QUESTIONS:
1. It is necessary to conduct experiments on more recent models (at least LLaMA-3.1 or Qwen-2.5).

2. This method should also get a better reward model, so its effectiveness need to be validated on reward model benchmarks such as RewardBench, PPE-Preference, RMB, RM-Bench, and JudgeBench.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces XRLHF, an approach that aims to further improve an LLMs adherence to preferences *after* initial preference fine-tuning using a dataset of \"unsatisfactory responses\".\nThe method starts by identifying preferences from the training data that encourage these unsatisfactory responses (which they show is done by training data that can be convexly combined to closely match the unsatisfactory sample) and then proposes an approach to unlearn these problematic preferences.\nThe unlearning is done in two steps:\nFirst, the reward model that was used in the initial preference fine-tuning is trained with inverse gradients to unlearn the undesirable preferences.\nThen, the language model is fine-tuned on this updated reward model, but only on the unsatisfactory responses under the constraint of keeping the KL-divergence on another set of satisfactory responses small.\n\nThe authors empirically evaluate this approach on two datasets (chat and summarization) and with two RLHF backbones. They show that with 500 validation samples (labeled to be satisfactory or unsatisfactory; around 25-30% of them are unsatisfactory) they can significantly improve LLM performance. Both labeling and evaluation is done by an external reward model (different from the one used in the initial RLHF training, likely with greater capability).\n\nThe key contributions are (1) an approach to identify training preference data that encourages a response; (2) a preference data unlearning approach; (3) a sample-efficient secondary preference fine-tuning approach (the combination of (1) and (2)).\n\nSTRENGTHS AND WEAKNESSES:\nIn the following, I discuss strengths and weaknesses on the dimensions of quality, clarity, significance, and originality. Weaknesses are marked with a **W**, minor weaknesses with an **M**, notes that do not impact my score with **N**, and strengths with an **S**. Weaknesses are numbered for reference purposes.\n\n## Quality\n\n- **W1** My understanding is that your key contribution is a data-efficient method to further improve language models after initial fine-tuning. The empirical evidence supporting this data-efficiency is somewhat lacking, however, as there are no comparisons to any other approaches. Possible alternatives include:\n\n\t- Use direct unlearning methods to unlearn D_u in an SFT manner (I consider this the most important baseline, maybe the most important of all the weaknesses).\n\n\t- Add 500 (size of D) more samples to the initial RLHF dataset. This is not entirely equivalent to what you're doing, since RLHF is based on pairwise comparisons and you require binary labels. I believe it is quite comparable in effort, however. Alternatively perform an additional round of RLHF with only 500 samples (heavily KL-regularized / small learning rate / one epoch).\n  \n  To properly understand the effectiveness of your approach, I think it would be important to compare to baselines like these. I understand that this is computationally expensive and I believe it would be sufficient to do it on one algorithm/dataset combination. Still, some baseline on how large the improvement of \"traditional methods\" with 500 additional data points would be would be interesting and important to evaluate your approach.\n\n- **W2** It would be interesting to see a comparison to a baseline that does a full retraining run without the \"bad\" preference data. As you say, this is more computationally expensive than your method. It would still be valuable to see how it compares in terms of performance, at least in one experimental condition (one backbone, one dataset).\n\n- **W3** On line 3678 you claim that since win rates on D are smaller than those on D_u, this implies degradation on D \\ D_u. This is not necessarily true, win rates would also be smaller if performance on D \\ D_u remains unchanged. Can you report performance on D \\ D_u separately to quantify the degradation?\n\n- **W4** (This is lower priority, considering that is likely a significant effort).\nYou claim that a major differentiator of your approaches from other approaches aiming for alignment with noisy preference data is that it also works with misleading preferences that are valid for some prompts but not for others. I am not entirely convinced that this is practically relevant for many cases (as the prompt is part of the RM training data and a sufficiently a well-trained RM should be able to distinguish cases where the preference is relevant and cases where it is not). Either way, it would be possible to empirically compare your method to alternative approaches. PerpCorrect (your [14]) seems particularly relevant here, as it does not need any additional data. Your method could be combined with PerpCorrect, however it is imaginable that the validation dataset would not contain sufficient \"bad\" examples after applying PerpCorrect, rendering your method ineffective. I would therefore like to see how the performance of PerpCorrect compares to the performance of your method (standalone and/or combined with PerpCorrect).\n\n## Clarity\n\n- **S** The paper is generally well-written and clear. I especially enjoyed the clarity of the preliminaries section.\n\n- **W5** It is not clear to me how the quadratic program in equation 2 is solved in practice.\n\n- **W6** Are the weights of the convex combination entirely ignored in the unlearning phase? If so, is this reasonable? This should be discussed.\n\n- **W7** As far as I understand it, the \"Remark on general reward cases\" is only valid if you consider the representation part to be frozen and only train the last linear layer during fine-tuning. Is that the case?\n\n- **W8** You use reward models for evaluation and labeling. I think this is okay for research purposes, but not realistic in practice: If you had a sufficiently good reward model available, you could just train on it directly. This should be discussed in the paper.\n\n- **W9** Why is it desirable that each datapoint of the composition is close to the \"bad\" training data point? Wouldn't it be better to have the smallest possible decomposition instead?\n\n- **W10** Doesn't minimizing the LL of The example in Table 8 then encourage the Khaki response, which is not necessarily better? The core problem is the misunderstanding of the prompt.\n\n- **M1** In the worst case in Algorithm 1, it may happen that the entire dataset has to be considered. Does it have any benefit for the worst-case complexity to iterate over the dataset starting from the closest element?\n\n- **M2** Why are there no examples of no XRLHF in D.4? That would be more interesting than PPO/ReMax comparison.\n\n- **M3** You claim that the unlearning reduces the influence of S (line 244). Would it not be more correct to say that it *inverts* the influence?\n\n- **M5** The claim on line 142 (policy generates response -> response must have had high reward) is not necessarily always true, as RLHF training uses only limited optimization and a KL constraint. It's still \"true enough\" for your purposes, but the claim should be slightly weakened.\n\n- **M6** I was initially confused by line 186 (I think there are more instances of this) claiming to find the convex combination that is closest to the projected target. I assumed this meant the actual combination is close, which confused me as an exact match is ensured by the projection. What is actually meant is that each individual member of the decomposition is close. This becomes clear in the formula, but should be clarified in the text.\n\n- **M7** On line 227 it is implied that there exists a single unique subset that \"leads the policy model\" to make an unsatisfactory response. Similarly later a causal relationship is implied. Both of these claims should be weakened.\n\n- **N** The introduction discusses the method in quite some level of detail. That is not a problem, but it left me confused on two points: (1) How is it ensured that the decomposition exists? (Later clarified, projection); (2) How is such a decomposition even possible, as preference data differs in format from prompt-response pairs (also later explained). If this detailed description of the method remains in the introduction, those two points should be briefly clarified. In general, I think discussing early that you mostly work on the difference of response features would clarify things a bit.\n\n- **N** You claim that \"This idea is in the same spirit as DPO\" (line 232). The same sentence would apply to RLHF, however. So I do not understand which value it adds.\n\n- **N** The name XRLHF is used for the first time in section 6 and never formally introduced.\n\n## Significance\n\nI think the approach to generate preference-data based explanations and to unlearn preference data is significant. The combination of these two, potentially leading to a data-efficient secondary preference fine-tuning approach, is not evaluated and compared with (simpler) alternatives sufficiently thoroughly to determine its significance, however.\n\n## Originality\n\nThe work provides original methods to the best of my knowledge. My familiarity with the relevant literature is somewhat limited, however. The authors describe related work and clearly distinguish themselves from it.\n\nQUESTIONS:\nPlease see the numbered weaknesses, focusing on the major ones (**WX**). Most of them are actionable, i.e., can be considered questions, although many of the minor ones (**MX**) do not require an individual response. Actual questions regarding the understanding of the paper are listed as weaknesses in the clarity section. The minor points have little impact on my score.\n\nI think the contribution is technically solid (leading to my score on the acceptance side), but further clarification and more thorough evaluation would improve it. I consider W1, W7, and W9 to be of particular importance. Note that it is not necessary to conduct *all* experiments I suggest in the Quality section, but *some* comparison with baselines, particularly natural ones like direct unlearning, would greatly improve the paper.\n\nLIMITATIONS:\n- **W11** The discussion of the limitations is slightly limited. One major limitation, which I think is overlooked, is that as models get better, the fraction of unsatisfactory responses should get smaller. The rate of unsatisfactory responses in the paper seems high, likely due to the small models used. A lower fraction would require a larger validation set, making the method less applicable as models get better.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}