Explainable Reinforcement Learning from Human Feedback to Improve Alignment
Authors: Shicheng Liu
Siyuan Xu
Wenjie Qiu
Hangfan Zhang
Minghui Zhu
Venue: NeurIPS 2025 poster ()
Keywords: explainable reinforcement learning from human feedback
alignment
Primary area: deep_learning

Abstract: A common and effective strategy for humans to improve an unsatisfactory outcome in daily life is to find a cause of this outcome and correct the cause. In this paper, we investigate whether this human improvement strategy can be applied to improving reinforcement learning from human feedback (RLHF) for alignment of language models (LMs). In particular, it is observed in the literature that LMs tuned by RLHF can still output unsatisfactory responses. This paper proposes a method to improve the unsatisfactory responses by correcting their causes. Our method has two parts. The first part proposes a post-hoc explanation method to explain why an unsatisfactory response is generated to a prompt by identifying the training data that lead to this response. We formulate this problem as a constrained combinatorial optimization problem where the objective is to find a set of training data closest to this prompt-response pair in a feature representation space, and the constraint is that the prompt-response pair can be decomposed as a convex combination of this set of training data in the feature space. We propose an efficient iterative data selection algorithm to solve this problem. The second part proposes an unlearning method that improves unsatisfactory responses to some prompts by unlearning the training data that lead to these unsatisfactory responses and, meanwhile, does not significantly degrade satisfactory responses to other prompts. Experimental results demonstrate that our algorithm can improve RLHF.

Explainable Reinforcement Learning from Human Feedback to Improve Alignment
Shicheng Liu', Siyuan Xu', Wenjie Qiu', Hangfan Zhang', Minghui Zhu DDepartment of Electrical Engineering, Pennsylvania State University DDepartment of Computer Science, Rutgers University CCollege of Information Sciences and Technology, Pennsylvania State University {sf15539, spx5032, hb25148, muz16)-ppsu. edu, wenjie. qiw.ruuteeerrrrrr...... edu
Abstract
A common and effective strategy for humans improve unsatisfactory outcome in daily life is to find a cause of this outcome and correct the cause. In this paper, we investigate whether this human improvement strategy can be applied improving reinforcement learning from human feedback (RLHF) for alignment of language models (LMs). In particular, RLHF can still output unsatisfactory responses. This paper proposes a method to improve the unsatisfactory responses by correcting their causes. Our method has two parts. The first part proposes post-hoc explanation method to explain why an unsatisfactory response generated prompt by identifying the training data that lead to this response. We formulate this problem as constrained combinatorial optimization problem where the objective is to find a set of training data closest to this prompt-response pair in feature representation space, and the constraint is that the prompt-response pair can be decomposed as convex combination of this set of training data in the feature space. We propose efficient iterative data selection algorithm to solve this problem. The second part proposes an unlearning method that improves unsatisfactory responses to some prompts by unlearning the training data that lead to these unsatisfactory responses and, meanwhile, does not significantly degrade satisfactory responses to other prompts. Experimental results demonstrate that our algorithm can improve RLHF.
observed in the literature that LMs tuned by
Introduction
Reinforcement learning from human feedback (RLHF) 2 is predominant approach to align language models (LMs) with human preference, It learns a reward model from human preference data and uses the reward model to tune an LM 4 5 6.. The tuned LM can be evaluated
set of validation prompts {5(() ) to generate responses { (1) Ji=1 thereby forming the validation data
Fe=1
D (a((), g( M It is expected that all responses in D should align with human preference.
However, it observed in the literature that LMs tuned by RLHF still generate unsatisfactory responses. We also include an experiment (in Appendix D.II to validate that RLHF still generate unsatisfactory responses. Therefore, we can partition the validation data D into an unsatisfactory subset D, (with unsatisfactory responses and associated validation prompts) and satisfactory subset D D.,. These unsatisfactory responses in DD can be due to various reasons, such as missing or misleading information in the human preference data. In this paper, we study the case where the preference data includes misleading information. This setting is common in practice. One example is that the preference data is usually collected from diverse sources and thus some preference data can be useful for some prompts but misleading to other prompts [1 13 Another example is that the preference data can be noisy, where
responses, such as harmful 77 inaccurate 9 10 and redundant
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

RLHF
Our method (post-training of RLHF)
reward model
validation data
improvement
treining dete D Me >1
explamation fv ( O
unseristestory Tr, و00,
phase
unleamed roward %
phase
validation prompts (219
[x 90 satisfactory 1/
09
tine-tuned Follcy I
policy model St
Figure 1: An illustration of our method. Our method can be viewed as post-training of RLHE RLHF learns a reward model ry and a policy model Ty. The policy Ty is evaluated on (=(0 ) to form the validation data D that consists of an unsatisfactory subset Du and satisfactory subset D \ Du- Our method has two parts. The first part explains why Ty generates the unsatisfactory responses to the prompts in D, by identifying the training data that lead To to generate these responses. The second part includes two phases to improve To- Phase one unlearns the identified training data from the original reward model ro obtain an unlearned reward model ru- Phase two fine-tunes mo to maximize the unlearned reward model ru for the validation prompts in Du, while staying close to wo for the validation prompts in D Du.
the labels of some actually preferred and dispreferred responses are flipped due to the bias and errors of human annotators or malicious noise injection [14] [5
This paper proposes method that further improves the LMs tuned by RLHF to better align with human preference. Our method is inspired by an effective strategy that humans commonly use in daily life to improve an unsatisfactory outcome, where humans first find cause of the outcome and then correct the cause to improve the outcome. Our method has two parts. The first part proposes a post-hoc explanation method explain why the LM generates the unsatisfactory responses in Da by identifying the training data that lead to these responses. The second part improves the unsatisfactory responses by unlearning the identified training data in the first part, and does not significantly degrade satisfactory responses in D Du. Our method (illustrated in Figure 1) can be viewed as a post-training of RLHF as our method operates after RLHF has tuned an LM.
The first part proposes a post-hoc explanation method to identify a set of training data that leads the LM (tuned by RLHF) to generate the unsatisfactory response ý the prompt in D.. We first mathematically derive an insight that if (σ. 7) is either close to, or can be expressed as a convex combination of, some training data in feature space, then the response ÿ is likely to be generated to F. Using this insight as a guideline, we formulate a constrained combinatorial optimization problem where the objective is to find subset of training data whose features are closest the feature of (7. 7). and the constraint is that the feature of (æ, y) can be decomposed as a convex combination of the features of this set of training data. This optimization problem is NP-hard and brute-force enumeration of all possible subsets of training data is computationally intractable, as it results in an exponential computational complexity with respect to the number of training data [16). To address this issue, we propose an efficient iterative data selection algorithm which starts with the closest training data and keeps adding data to expand the set until the constraint is satisfied. We prove that the computational complexity of the proposed algorithm is polynomial.
The second part uses the subset of identified training data in the first part to improve the unsatisfactory responses in D, and, meanwhile, does not significantly degrade the satisfactory responses in D Dn-- Specifically, the second part includes two phases: reward unlearning and policy fine-tuning. The reward unlearning phase unlearns the identified training data from the reward model of RLHF to obtain an unlcarned reward model by minimizing the log-likelihood of this identified subset of training data. The policy fine-tuning phase fine-tunes the policy of RLHF to maximize the unlearned reward for the validation prompts in D, and restricts the KL divergence from the policy of RLHF for the validation prompts in D D..
p method, applied after RLHF, that leverages post-hoc explanations to further improve alignment. Our contributions are twofold.
Contribution statement. This paper proposes
First, we study the explanation problem of identifying the training data that lead to unsatisfactory responses. The major novelty is that we mathematically derive an insight into what kind of unsat- isfactory responses is likely to be output by the LM and, based on this insight, mathematically
2

---

formalize this problem as a constrained combinatorial optimization problem. While the formulated optimization problem is NP-hard, we propose an efficient algorithm with polynomial computational complexity to solve this problem.
Second, we improve the unsatisfactory responses by unlearning the identified training data and do not significantly degrade other satisfactory responses. We propose a novel unlearning method that first unlearns the reward model and then fine-tunes the policy model. We conduct extensive experiments to demonstrate that our proposed method can further improve the LMs tuned by RLHF.
2 Related works
Due to the space limit, here we only discuss related works on post-hoc explainability and language model unlearning. We include discussions on more related works in Appendix A
Post-hoc explainability (PHE). PHE provides explanations for why trained model outputs certain decision for an input. Existing approaches can be broadly categorized into feature-based and example-based methods. reature-based methods [I7 18 [9] learn an importance score for each input feature and use this score to reflect the importance of each feature for the trained model to output its decision. Example-based methods 20 21 22 find relevant examples of the input and use the decision of the relevant examples to explain the decision of the input. This category is inspired by the observation that humans usually use relevant experience to interpret new thing 20). Our explanation is an example-based explanation. However, the aforementioned example-bar based methods cannot be applied to our problem because they require the relevant examples to have same data structure as the data to be explained. In our case, the data to be explained is prompt-response pair but the training data we use to explain includes a prompt and a response comparison.
Language model unlearning (LMU). LMU (23 24 25 is proposed as an alternative to RLHF for tuning LMs by removing negative or harmful information from them. Specifically, LMU assumes access to pre-collected set of negative examples and aims to directly unlearn these examples from the LMs. However, these unlearning methods cannot be applied to our setting because we cannot directly unlearn the training data from the policy/language model. The reason is that training data consists of prompt and a response comparison, while the policy/language model outputs a response (instead of response comparison) given a prompt. This mismatch prohibits direct unlearning from the policy model. Therefore, we propose a novel unlearning method that first unlearns the reward model and then fine-tunes the policy model under the unlearned reward model.
Approaches to improve alignment. There are other approaches proposed improve alignment of RLHF. Specifically, the paper (26) proposes to train the policy under a contrastive reward function (subtracting the learned reward model by the reward model of the SFT policy) to improve alignment. The paper [27] proposes to learn ensemble reward functions (a combination of multiple reward functions) to improve reward learning accuracy and thus improve alignment. The paper (2 proposes to average two independent SFT policies as the reference policy to allow for larger deviation from the SFT policies to improve alignment.
3 Preliminaries
RLHF typically includes three stages: 1) supervised fine-tuning (SFT), where high-quality demon- stration data is used to fine-tune a pre-trained model in a supervised manner to get a model TSFr; 2) reward modeling (RM), where preference data is used to train reward model; 3) Reinforcement learning (RL), where the SFT model TSFT is further fine-tuned by running RL to optimize the reward model. In this paper, we assume that TSFT is given, and we claborate the last two stages.
Reward modeling (RM). Given a prompt æ. the SFT model TSFT can generate a pair of responses (y1, 32). Human annotators are instructed to choose the response they prefer from the pair (31, y2), resulting in you > Ii where you and yı are respectively the preferred and dispreferred responses. We
(6) (8) J and train reward model ra assume the access to a set of preference data D = (+) ,ho 32
by maximizing the oogliieli of the preference D under Bradley- Terry model (2
N
max L D)
logo(ra(a (20)) - ro (0)
.44 (.2(()) 11))
(I)
1=1
3

---

where a is the sigmoid function, and P(y > vi (rro (2.r)( - ro 3t) is the Bradley-Tery model. We denote the reward model learned in this stage by roo.
Reinforcement learning (RL). With the learned reward model r0g- we aim to further fine-tune the SFT model SFF align with human preference by solving the following RL problem:
max (g- (r) (z. v)] BE D (
controlling the deviation of the learned policy π from the SFT model where B is [=))( (-xr) - |x) [log ((yl=) - log SSFT (y]=z] the KL divergence TTPT, and DKL(TT
between the policy ग and SFT model TSFT- We denote the learned policy model by Too
We can evaluate the responses y of mo to the validation prompts T. We use D = {x=(1) y (1)) M denote the set of validation prompts and their corresponding responses generated by To. It is expected that all the responses generated by mT are satisfactory, however, is observed in the literature and our experiment (in Appendix [DI) that To can generate unsatisfactory responses. In our case, we use open-source reward models to score responses and identify responses with scores below a threshold as unsatisfactory. The threshold is picked by human evaluations (detailed in D.I). Therefore, we
) 1i=1 to
partition the set 'D into two subsets. The unsatisfactory subset D, {z(() g m unsatisfactory responses in D and their associated validation prompts. The rest D satisfactory responses in DD and their associated validation prompts.
K= includes the
Dn includes the
4 Explainable reinforcement learning from human feedback
This section proposes an example- based post-hoc explanation method, Inspired by the way humans use rel- evant experience to interpret new things, example-based methods use relevant examples to explain why a
unsatisfactory (5.9)
idantifiied training data
How dbars
AIWeNatt afail sharghh
ST SURY
With sahion
Sur carn
catoon wanien dpitorndered Henddaar p0 ight Petricttn
Wit www.eoorroooooooo Feed
de cnn.iionaal...
Ingiiaal and un focm
2325
learned model generates a certain out-
put (20 21 22). In this section, we ex- plain why the RLHF policy model wo generates an unsatisfactory response y
Figure 2: An example of explaining an unsatisfactory re- sponse ÿ to prompt ă using set of three training data.
to prompt ō by identifying the train- ing data in D that lead to this response. Given that the policy To generates the response ÿ to the prompt E, it means that the learned reward model re, assigns high reward re, (T. 3) to the prompt-response pair (z, g). Therefore, it suffices to explain why (z, 8) has a high reward. To facilitate understanding, we first consider the linear reward case Te, (x., y) = 0g øxx y) to illustrate our method, where 0g is the reward parameter and ().,)) is feature function that maps the prompt-response pair to a feature vector. We emphasize that our explanation method is not limited to the linear reward case, and we will later discuss how directly use method in general reward cases.
We aim to identify the training data that contribute to the high reward re, (z. g): the identified examples contribute to the high value of ro, (i. V). and since RL optimizes the LM to maximize reward, high reward increases the likelihood that ÿ is generated. Therefore, the identified examples (indirectly) contribute to the generation of y. For this purpose, we first reason about what kind of prompt-response pairs has high rewards. Recall from the reward learning problem ( where we aim to optimize 0 to maximize the log-likelihood over the training set D. We can reformulate the log-likelihood function in as (derived in Appendix B):
N
Σ ToT (o(((
b - 10x v12)) - log (ot - -000))9))))))))) (0 + +1]]
L(O, D)
N
Since the above log- likellkl function I(O, D) is strictly monotonically increasing in the inner product 0J (d(x(()) s/ (a) 3/ (²)) (proved in Appendix [B). maximizing the log-likeldhood is equivalent to maximizing the inner product over the preference training set D. Therefore, the
feature comparisons (0(x(1) )لا ((x(),, . "/l""//""))))) (("))) of the training data have high rewards, i.e..
oJ ()(x!),, y/ ())()) yf())) )) is high. If prompt-response pair (ž, 7) has feature Φ(ž, 7) closely

---

aligned with the feature comparisons of some training data, this prompt-response pair is expected to have a high reward.
The above insight motivates two distinct intuitions for finding training data to explain why (T, y) has high reward:
Intuition I (nearest neighbors). We find n (1 S n S N) training data that is closest to (z, y) in the feature space, i.e., minimum Euclidean distance [1)((z(+)) _yee ((x(t) w/(²) o( o) Since the reward function is continuous in ø, the feature @(z, 3) with small Euclidean distance from the feature comparisons of some training data should also have high reward.
Intuition II (decomposition). We find n training data to decompose (x. y) in the feature space such that (T, g) Σ (0(x()) y o(x(1) )) where ( E [o, 1] and Ex= w 1. Since the feature comparisons of the n training data have high rewards, their convex combination ((,, 5) should also have high reward.
Intuition is easy compute but be imprecise. If the nearest neighbor is still relatively far ((2 7)ll is large), the reward difference between the
from (z, 7) (i.e.. [((x(() you o(x(1), y/ (0) neighbor and (z, 7) can also be large, leading to inaccurate explanation.
Intuition II can precisely explain the reward value ro, (Z. 77) even if the neighbors are far, because it matches the feature (EE Ø). ensuring that re, (z, 7) = Σi= wit) (ra,(x(), y re, (x((), (2) However, finding a decomposition can an ill-posed problem because there could be multiple feasible convex combinations that can match (0 7)
We propose method that combines the strengths of both intuitions by learning decomposition of Φ(σ,) 3) that is closest to é(T, g). However, such a decomposition is infeasible if ó(I, 9) does not lie within the convex hull of the feature comparisons from the training set. Therefore, project the feature vector ((T 7) onto the convex hull. Specifically, the convex hull of the training
first
data feature comparisons is defined as (D) & P ( yu²) d(x(") yi (0) ()))()))))))
[0, 1]. EP (i))) = 1} where [D] = N is the cardinality of the training set D. We project 6(T, ū) onto C. (D) obtain the projected feature vector Φ0 y) by solving the quadratic program:
((ā.g) arg min ||u ((2,0)11
(2)
vEC,(D)
where the projection d(2, g) 6l2 (5, T) CD) Next, we aim decompose the projected feature vector 6(ž.0) 3) as a convex combination of feature comparisons from subset of training data. Among all feasible convex combinations, we find the one that is closest to Φ(ž, ø). Formally, we propose solve the following constrained combinatorial optimization problem:
S Σ 1,0(2 (00)) ((x(0) 11 (0)) -0))))(21)) ((..g))
min SCD
s.t. o(z.7) Co(S), (z0) ()) S, i == 1, ,18).
The problem aims to find subset S of the training D such that ø(á, 9) lies within the convex hull (S) and the sum of Euclidean distances between ((z, 7) and the feature comparison of cach training data in S is minimized.
Remark on general reward cases. Although we derive the problem (3) assuming linear reward model for clarity, the problem (3) can be directly applied to general reward cases. In practice, reward models are usually initialized from a SFT model TSFT with an additional linear layer on top of the final transformer layer [4, 30 After the reward model is trained, given prompt-response pair (I, y). the output of the final transformer layer can be used as the feature representation ø(x; y). In this case, the reward of (ar, y) is also linear of oh y). If the reward model is neural network where the last layer is a fully connected (linear) layer, we can use the output of the second-to-lase layer as the feature representation.
5

---

Algorithm Explainable reinforcement learning from human feedback (XRLHF)
Input: A prompt-response pair (z, g) to be explained, the training set D, and an empty set S. Output: The set S of training data that leads to the response to the prompt I and the corresponding decomposition coefficients (1)
Ti=1
1: Compute the projected feature vector d(E, f) by solving the quadratic program (2)
2: Rank the training data (ar, Yu: M1) in D by the distance [|o(xz yw) (xx x. yr) Ō 9)1l
3: while á(, y) $ (S) do
4: Pick the nearest training data (a, Mlus B1) in D \ S and add this data point 8 = 8 U (zz yau v1).
5: Check the condition 6(2, g) (S) and find the corresponding decomposition coefficients
{w))) /==1 if condition satisfied, by solving the linear program
6: end while
4.1 Minimum-distance-
iterative training data selection
The problem 3 is guaranteed to have an optimal solution because 1) its feasible set is non-empty as ő(σ, 7) E (D);; 2) there finitely many choices of S since 'D is finite set. However, the space of all possible choices of D; therefore, it is intractable to search over the entire space to find the optimal set S. To address this issue, we propose an iterative training data selection algorithm where we start from an empty set S and iteratively add new training data whose feature comparisons are close to Φ(Z, 7) until Co (S) contains ल(z. v). This algorithm is efficient because it avoids searching over the entire space of all possible choices of S. Instead, only searches over much smaller local space around Φ(z, ø). We elaborate the algorithm as follows:
can be extremely large because it grows exponentially with the cardinality
Given
pair (z, s) that requires explanation, Algorithm I first computes its projected feature vector (āz, 5) by solving the quadratic program 2 line I and ranks all the training data in D by the Euclidean distance (,( o(x, 31) - 6(Ø o. 7)ll (line 2). Then Algorithm | solves the problem 3 by iteratively picking the nearest training data. At each iteration, Algorithm finds the closest training data in S according to the ranking and adds this training data to S (line4p 4 The algorithm checks whether the convex hull (S) corresponding to the current set S already contains (Ø, 9) by solving the following linear program:
[S]
SS| Σ
min0, s.t. )(
Σ
w E (0, 1). (o(x (V1)) - ,1)
w =1 (4)
The problem is linear programming feasibility problem that not only checks whether ((0, 7) E (S) but also provides the decomposition coefficients { if a decomposition is feasible. If multiple decompositions are feasible in S, we choose the one closest to Φ(ē, 9).
Theorem The computational complexity of Algorithm is O(N log +
+4.55
where N is the number of training data in D and d is the feature dimension.
5 The improvement via unlearning
This section improves the alignment of mo by leveraging the explanation method in Section Recall from Section 3 that the validation data = {τ(1) JMM is partitioned into an unsatisfactory subset im D. = {z()) 7() Ji=1 and a satisfactory subset D D.. We aim to improve To so that its responses to the prompts in D. are improved, while its responses to the prompts in D \ D, are not significantly degraded.
For rempptrresponeeeeeeee pair (ž.ğ) E D., we first use Algorithm to identify the subset, denoted by S(z, i). that leads to the policy model ma generating the response ý to the prompt T. We aim to improve the response to ã by reducing the influence of S(ē, on Æ. Since S(ē, 1) causes the unsatisfactory response ÿ to receive high reward, making ÿ more likely to generated, reducing the influence of S( 7) can decrease the likelihood of generating y. thus improving the policy model

---

To. This idea is in the same spirit as DPO [30] where the policy model is improved by making bad responses less likely (and good responses more likely).
straightforward approach to reducing the influence of Siz, y) is to remove this set from the training set and retrain RLHF over the rest of the training data D \ S D). However, this method has two issues. First, retraining RLHF from scratch is typically computationally expensive. Second, removing S(ā, y) from the training set will eliminate the influence of S( 57) on the entire policy model. While S(ā, y) has a bad influence on the prompt ह, it may be beneficial to other prompts (c.g., prompts in D D,). Therefore, this approach can degrade the policy model iTo in responses to other prompts.
To address these two issues, we propose an unlearning method that further fine- tunet the learned policy model To rather than retraining RLHF It more efficient than retraining because the unlearned data is only small portion of the training data (validated in Appendix [D.)). The proposed unlearning method includes two phases: reward unlearning and policy fine-tuning. The reward unlearning phase reduces the influence of (S 5) (2, on the reward model re, to obtain an unlearning reward model ra. The policy fine-tuning phase fine-tunes the policy To maximize the new reward model re, on the prompts in Da, while restricting the deviation from To on the prompts in D \ Duu
Reward unlearning, We adopt negative gradient [31] to unlearn the influence of {S 5) ) (a.g)) on the reward model reg- Recall that ra, is trained by maximizing the log- likeliioeio L(0, D) over the training set D. To unlearn the effect of a subset (S ]) (E.))ED S DD we reverse this training process by applying negative gradient updates. Specifically, we use 0g as the initial reward parameter and update the reward parameter by minimizing the log-likelihood over (S( 1) (E.5)ED, This results in negative gradient update: 0i+1 O, a V L(., {S(E, (F.)EE)) where a is the learning rate. We iterate this negative gradient update and denote the obtained reward parameter by 0g. Since re, was originally obtained by iteratively adding gradients from the training data, the negative gradient updates effectively remove the contribution of (S ū) 7) from fp, thereby reducing their influence the unlearned reward model rea.-
Policy fine-tuning. While the unlearned reward model re, is beneficial to the prompts in Da, it may be degraded for other prompts in D \ Du. Therefore, we fine-tune the policy xo using re, only on the prompts in Du. To prevent degradation responses to prompts in Du. we impose constraint on the KL divergence between the fine-tuned policy and the original policy To for these prompts:
max E Pg-DD.wrat-1p) [ro, (z. v)] BE D. [KrI. (m( (z) [o{]
(5)
where 3 is a hyper-parameter controlling the deviation from the original policy To for the prompts in D \ Du. We initialize the policy fine-tuning phase from the original policy mo and optimize it by solving the problem (5)
Experiment
In this section, we provide empirical evaluations to validate the effectiveness of XRLHF (Algorithm I in improving RLHF. In particular, we first run RLHF to obtain a language model To and use Ao to generate responses to set of validation prompts, thus forming the validation data D. We partition the validation data into a satisfactory subset D, and an unsatisfactory subset DD \ Da. and run Algorithm
Oon these two subsets. This section demonstrates the effectiveness of XRLHF in two aspects: (1)
XRLHF improves RLHF over a new (unseen) test set (Sec. 6. and Sec. 6.2). (2) XRLHF improves RLHF over the validation prompts (Sec. 66.3. The experiment details are in Appendix D
Models and datasets. We test XRLHF on two widely-adopted tasks: dialogue generation and summarization. For the dialogue generation task, following [32), we test Algorithm on full-hh- rlh dataset with opt- 1.3B and pythia-2.
models. This dataset is a reformatted version of the
Anthropic-HI HH dataset (6), and consists of 1 112k samples for training and 12.5k for evaluation. For the summarization task, following [33]. we test on TL;DR" /summarization dataset (34] with pythia-2.8B and Llama-2-7B models. This dataset consists of 92.9k samples for training and 86. 1k for evaluation.
Following the standard practice [5, we partition the training data into three parts: 20% for supervised fine-tuning, 40% for reward learning, and 40% for reinforcement learning. Our XRLHF method
identifies the training data from the 40% training data for the reward learning part.
I Dataset available at http:///wa///////////////////a/a
2 Dataset available at http:///wngginngee..occ//aass/sp http://ummmrriiotrtoorfoeeeeeatd feebbac

---

Baselines. We use two sfate-of-the-art RLHF algorithms, PPO [5] and ReMax [32], as the base RLHF methods. We use XRLHF further improve these two base algorithms, resulting two new methods PPO+XRIHP and ReMax+XRI.H:
Evaluation. We use the win rate over the SFT model as the primary metric to evaluate the performance of the base RLHF algorithms and our method. To demonstrate the effectiveness of XRLHF in improv- ing RLHF, additionally report the win rates of PPO+XRLHF over PPO, and ReMax+XRLHH over ReMax. The win rates are calculated using two evaluation sources: an open-source reward model and GPT-4 The reward model we use for the dialogue generation task is PKU- Aai---llmmme//bbeeeee 7b-vv.r reward This reward model is specifically trained for helpfulness and harmlessness evaluations on tialogue-based responses [35]. making suitable for the dialogue generation task. The reward model which is hee__eeebbcck dataset. Given a test prompt and two responses generated by two language models, the reward model assigns a scalar score to each response. language model is considered the winner if its response receives a higher score. The win rate of model A over model B is the percentage of test prompts for which model A receives higher score than model B. We also obtain the win rate by querying GPT-4 for zero-shot pair-wise evaluation (see prompts for GPT-4 evaluation in Appendix D.3). which has been shown to be consistent with human judgments [30].
we use for the summarization task is AppenAssistan/rrwwarrr modell-deberta-
trained on diverse datasets, including the openai/summarize
6.1 Dialogue generation
We reserve 500 prompts from the training set of the full-hh-rlhf dataset as the validation prompts, and use the remaining prompts along with their corresponding chosen and rejected responses for train-
Table 1: Win rates over the SFT model on the test set of the full-hh-rlhf dataset.
Best results are highlighted in boldface.
Beaver-7b-y -3.reard (%)
GPT-4 (%)
Method
opt-
pythia-2.8B
opt-1.3H
pythia-2.8B
PPO ReMax
68.3
69.8
68.0
67.5
70.6
71.4
66.9
66.8
PPO+XRLHF
76.4
75.8
78.5
76.5
ReMax+XRLHF
77.2
79.3
76.1
80.1
ing. Note that the validation prompts are only small portion (500/112,00) 0.4%) of the original training set, and we do not require chosen and rejected responses for these prompts. Instead, we use To to generate a response each validation prompt. To identify unsatisfactory responses, employ the PKU-Alignmentbeeverr 77--3.0-rewaad model to score each generated response. Responses receiving scores below threshold are labeled unsatisfactory. This threshold is determined by human evaluation (detailed in Appendix D.I and thus can mimic human preference. Compared to the base RLHF methods, the additional information required by XRLHF is only the satisfactory or unsatisfactory labels for responses to the 0.4% validation prompts.
Table reports the win rates of the four methods over the SFT model. The win rates are calculated from the test set of the full-hh-rlhf dataset. The results show that both PPO+XRLHF and ReMax+XRLHP achieve higher win rates over the SFT model than their respective base RLHF methods, PPO and ReMax. Additionally, we provide ex- amples generated by PPO+XRIHF and +eMMa++RRIIK in Appendix [D.4]
POHOUE
To further demonstrate the effectiveness of XRLHF in improving RLHF, we report the win rates of PPO+XRLHF over PPO and ReMax+XRLHF over ReMax in Figure 3 The results indicate that both PPO and ReMax generate improved responses when augmented with XRLHF.
Figure 3: Win rates of PPO+RLHF over PPO and ReMax+RLHF over ReMax on the test set of the full-hh-rlhf dataset.
P Model available at httpae//aaanniiiiireeeee/////lar 7
Model available http:///wwwh.ittooo....ooo//o//a
1arge--r-

---

PPO ReMax PPO+XRLHF 6.2 Summarization
We reserve
Table 2: Win rates the SFT model the test set of the TL:DR summarization dataset. Best results are highlighted boldface.
500 prompts
from the train- ing set of the TL.DR dataset the valida- tion prompts.
Deberta-v3-large-- (%)
GPT-4 (%)
Method
pythia-2.8B Llama-2-7B
pythia-2.8B
Llama-2-7B
65.8
63.6
70.4
68.4
These validation prompts are only 0.5% of the original training set. We use the policy model To tuned by RLHF to generate a response to each lesstant/eeaarddmdddl--eeeheee
62.7
69.6
71.2
67.9
76.8
75.2
79.2
72.5
ReMax+XRLHF
75.6
77.4
80.4
78.1
validation prompt. We with scores below a threshold as unsatisfactory responses and this threshold is chosen by human evaluation. Table Sable@repon reports the win rates calculated from the test of the TL;DR dataset.
identify responses
The results in Figure demonstrate that augmenting both PPO and ReMax with XRLHF leads to significantly im- proved performance on the TL;DR dataset.
Peffar citie
6.3
Evaluation of the explanation
We provide example of identified explanation in Ap- pendix D.2 In this part, we evaluate the fidelity of the explanation. Fidelity widely used metric explainable RL 36 371. which assesses the correctness of the expla- nation. In setting, the explanation identifies why the responses generated by RLHF (i.e., PPO and ReMax) to the prompts Du are unsatisfactory. Therefore, straight- forward way to evaluate the fidelity of the explanation is examine whether improving from the explanation can generate better responses, i.e., whether the responses generated by PPO+XRLHE and ReMMx+XRR.I to the prompts in D, improve.
Figure Win rates of PPO+RLHF over PPO and ReMax+ over ReMax the test set of the TL:DR dataset.
Table 3: Win rates of PPO+XRLHF (ReMax+XRLHIP over PPO (ReMax) on Da-
PPO+XRLHF vs PPO
DcMax+XRLHD vs ReMax
Model & Dataset
Reward (%)
GPT-4 (%)
Reward (%)
GPT-4 (%)
opt-1.3B & full-hh-rlhf pythia-2,8B & full-hh-rlhf
76.5
79.1
77.4
73.2
79.8
76.4
80.1
81.3
pythia-2.8B & TL:DR
71.2
75.8
68.6
74.2
Llama-2-7B & TL:DR
76.6
80.4
69.1
76.4
Table shows that both PPO+XRLHF and ReMax+XRLHF algorithms on D,, by a substantial margin. The win rates are evaluated by both the open source reward models and GPT-4. This result validates the high fidelity of our explanation, it demonstrates that the explanation contributes to generating significantly improved responses. We next evaluate how our method improves the base RLHF algorithms on all the validation prompts in D.
outperform their base RLHF
Table 4: Win rates of PPO+XRLHF (ReMax+XRLHH)
PPO (ReMax) on D.
PPO+XRLHF vs PPO
ReMax+XRLHF vs ReMax
Model & Dataset
Reward (%)
GPT-4 (%)
Reward (%)
GPT-4 (%)
opt-1.3B full-hh-rlhf pythia-2.8B & full-hh-rlhf
73.4
70.8
72.1
69.2
74.5
71.9
78.6
72.4
pythia-2.88 & TL:DR
67.6
69.1
62.4
70.5
Llama-2-7B & TL:DR
73.0
74.1
64.3
74.9

---

Table 4 shows that XRLHF improves RLHF on the full validation set D. However, the win rates are smaller than those observed on the unsatisfactory subset D.. This suggests that while XRLHF substantially improves RLHF on Du. it may still introduce some minor degradation on the satisfactory subset D 'Dy- This trade-off reasonable and expected, according to the no free lunch theorem [38]. Recall that we introduce KI. divergence term to the policy fine-tuning phase (5) to reduce degradation on Da, we include an ablation study in Appendix D.5 for this KL divergence.
Conclusion
This paper proposes post-training method for RLHF that leverages explanations further improve aligmment. We first explain why the RLHF-tuned language model generates unsatisfactory responses by identifying the training data that lead these responses, and then unlearn the identified training data from the reward model and fine-tune the policy model under the unlearned reward. The experimental results demonstrate that our proposed method helps RLHF generate improved responses.
8 Acknowledgements
This work is partially supported by the National Science Foundation through grants ECCS 1846706 and ECCS 2140175. We would like thank the reviewers for their insightful and constructive suggestions.
10

---

References
[1] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak,
D. Lindner, P. Freire, et al., "Open problems and fundamental limitations of reinforcement learning from human feedback," arXiv preprint arXiv:2307.1522 2023.
[2] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier, "A survey of reinforcement learning from human feedback," " arXiv preprint arXiv:2312.14925, 2023.
[3] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg. and D. Amodei, "Deep reinforcement learning from human preferences," Advances in Neural Information Processing Systems, vol. 30, 2017.
[4] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. "Fine-tuning language models from human preferences," arXiv preprint arXiv:1909.0859; 2019.
[5] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et el., "Training language models to follow instructions with human feedback," Advances in Neural Information Processing Systems, vol. 35, pp. 27730-27744, 2022.
[6] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, Henighan, et al., "Training a helpful and harmless assistant with reinforcement learning from human feedback," arXiv preprint arXiv:2204.05866 2022.
[7] Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang, "Safe RLHF: Safe reinforee- ment learning from human feedback," in nternational Conference on Learning Representations, 2024.
[8] Y. Yuan, W. Jiao, W. Wang, J.-t. Huang, He, S. Shi, and Z. Tu, "GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher," in International Conference Learning Representations, 2024.
[9] Y. Chen, D. Zhu, Y. Sun, X. Chen, W. Zhang, and X. Shen, "The accuracy paradox in RLHF: When better reward models don't yield better language models," in Conference on Empirical Methods in Natural Language Processing, pp. 2980-2989, 2024.
[10] H. Lee, S. Phatale, H. Mansoor, T. Mesnard, Ferret, K. R. Lu, Bishop, Hall, V. Carbune. A. Rastogi, et al., "RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with ai feedback," " International Conference on Machine Learning. 2024.
[11] Z. Chen, K. Zhou, W. X. Zhao, J. Wang, and J.-R. Wen, "Not everything is all you need: Toward low-redundant optimization for large language model alignment," in Conference on Empirical Methods in Natural Language Processing, pp. 15337-15351, 2024.
[12] C.-H. Chiang and H.-Y. Lee, "Over-reasoning and redundant calculation of large language mod- els," in Conference of the European Chapter of the Association for Computational Linguistics, pp. 161-169, 2024.
[13] S. Pitis, "Failure modes of learning reward models for ilms and other sequence models," in Workshop on The Many Facets of Preference-Bace Based Learning, International Conference on Machine Learning, 2023.
[14] K. Kong. X. Xu, D. Wang, J. Zhang, and M. S. Kankanhalli, "Perplexity correction for robust alignment with noisy preferences." Advances Neural Information Processing Systems, vol. 37, pp. 28296-28321, 2024.
[15] J. Cheng, G. Xiong, X. Dai, Q. Miao, Y. Lv, and F.-Y. Wang. "Rime: Robust preference- based reinforcement learning with noisy preferences," in International Conference on Machine Learning, pp. 2229-8244, 2024.
[16) M. R. Garey and D. S. Johnson, "Computers and intractability: guide to the theory of up-completeness," Journal of Symbolic Logic, vol. 48, no. 2, 1983.

---

[17] A. Datta, S. Sen, and Y. Zick, "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems," IEEE Symposium on Security and Privacy, pp. 598-617, 2016.
[18] M. T. Ribeiro, Singh, and C. Guestrin, ""why should trust you?" explaining the predictions of any classifier," in ACM International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144, 1144, 2016.
[19] R. C. Fong and A. Vedaldi, "Interpretable explanations of black boxes by meaningful perturba- tion," IEFE International Conference on Computer Vision, pp. 3429-3437, 2017.
[20] J. Crabbé, 7. Qian, F. Imrie, and M. van der Schaar, "Explaining latent representations with corpus of examples," " Advances in Neural Information Processing Systems, vol. 34, pp. 12154- 12166, 2021.
[21] R. Caruana, H. Kangarloo, J. D. Dionisio, U. Sinha, and D. Johnson, "Case-based explanation of non-case-bascl learning methods.," in American Medical Informatics Association Symposium, p. 212, 1999.
[22] H. Sun, A. Hüyük, D. Jarrett, and M. van der Schaar, "Accountability in offline reinforcement learning: Explaining decisions with a corpus examples," " Advances in Neural Information Processing Systems, vol. 36, 2024.
[23] Y. Yao, X. Xu, and Y. Liu, "Large language model unlearning," " arXiv preprint arXiv:2310.10633 2023.
[24] J. Yao, E. Chien, M. Du X. Niu, T. Wang, Z. Cheng, and X. Yue, "Machine unlearning of pre-trained large language models," arXiv preprint 15555:2402.15159, 2024.
[25] Z. Liu, G. Dou, Z. Tan, Y. Tian, and M. Jiang, "Towards safer large language models through machine unlearning," arXiv preprint arXiv:2402. 2024.
[26] W. Shen, X. Zhang, Y. Yao, R. Zheng, H. Guo, and Y. Liu, "Improving reinforcement learning from human feedback using contrastive rewards," arXiv preprint arXiv:2403.07708, 2024.
[27] S. Zhang, Z. Chen, S. Chen, Y. Shen, Z. Sun, and C. Gan, "Improving reinforcement learnin from human feedback with efficient reward model ensemble," arXiv preprint a6X1v:2401.16635, 2024.
[28] A. Chegini, H. Kazemi, S. I. Mirzadeh, D. Yin, M. Horton, M. Nabi, M. Farajtabar, and K. Alizadeh, "Model soup for better rihf: Weight space averaging to improve alignment in Ilms," in NeurlPS Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024.
[29] R. A. Bradley and M. E. Terry, "Rank analysis of incomplete block designs: the method of paired comparisons," Biometrika, vol. 39, no. 3/4, pp. 324 345, 1952.
[30] R. Rafailov, A. Sharma, E. Mitchell, C. Manning, S. Ermon, and C. Finn, "Direct preference optimization: Your language model is secretly reward model," Advances in Neural Information Processing Systems, vol. 36, 2024.
[31] A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot, "Unrolling SGD: Understanding factors influencing machine unlearning." in IEEE European Symposium on Security and Privacy, pp. 309-319, 2022.
[32] Z. Li, Xu, Zhang, Z. Lin, Y. Yu, R. Sun, and Z.-Q. Luo, "Remax: efficient reinforcement learning method for aligning large language models," in International Conference on Machine Learning, pp. 29128-29163. 2024.
A simple, effective, and
[33] H. Ji, C. Lu, Y. Niu, P. Ke, H. Wang, J. Zhu, J. Tang, and M. Huang, "Towards efficient exact optimization of language model alignment," International Conference on Machine Learning, 2024.
pp.
12

---

[34] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, "Learning to summarize with human feedback," Advances in Neural Information Processing systems, vol. 33, pp. 3008-3021 2020.
[35] J. Ji, D. Hong. B. Zhang, B. Chen, J. Dai, B. Zheng, T. Qiu, B. Li, and Y. Yang. "Pku- saferlhf: Towards multi-level safety alignment for Ilms with human preference," arXiv preprint ar0iv:2406. 15513, 2024.
[36] W. Guo, X. Wu, U. Khan, and X. Xing, "Bdge: Explaining deep reinforcement learning policies," Advances in Neural Information Processing Systems, vol. 34, pp. 2222-12226, 2021.
[37] Z. Cheng, X. Wu, Yu, S. Yang, G. Wang, and X. Xing, "Rice: Breaking through the training bottlenecks of reinforcement learning with explanation." in International Conference on Machine Learning. 2024.
[38] S. P. Adam, S.-A. N. Alexandropoulos, P. M. Pardalos, and M. N. Vrahatis, "No free lunch theorem: A review," Approximation and Optimization: Algorithms, complexity and applications, pp. 57-82, 2019.
[39] G. Lan, H. A. Inan, S. Abdelnabi, J. Kulkarni, L. Wutschitz, R. Shokri, C. G. Brinton, and R. Sim, "Contextual integrity in Ilms via reasoning and reinforcement learning," arXiv preprint 2025.
[40] O. Bastani, Y. Pu, and A. Solar-Lezama, "Verifiable reinforcement learning via policy extraction." Advances in Neural Information Processing Systems, vol. 31, pp.
2018.
[41] T. Bewley and J. Lawry. "Tripletree: A versatile interpretable representation of black box agents and their environments," in AAAI Conference on Artificial Intelligence, vol. 35, pp. 11415-
1 11422, 2021.
[42] A. Verma, V. Murali, Singh, P. Kohli, and S. Chaudhuri, "Programmatically interpretable reinforcement learning." in International Conference on Machine Learning, pp. 5045-5054, 2018.
[43] A. Atrey, K. Clary, and D. Jensen, "Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning," " in International Conference on Learning Representations, 2019.
[44] S. Guo, R. Zhang. B. Liu, Y. Zhu, D. Ballard, M. Hayhoe, and P. Stone, "Machine versus human attention in deep reinforcement learning tasks," " Advances in Neural Information Processing Systems, vol. 34, pp. --3355, 2021.
[45] N. Puri, S. Verma, Gupta, D. Kayastha, S. Deshmukh, B. Krishnamurthy, and S. Singh, "Explain your move: Understanding agent actions using specific and relevant feature attribution," in International Conference on Learning Representations, 2019.
[46] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez. "Explainable reinforcement learning via reward decomposition." in IJCAI/ECAI Workshop on Explainable Artificial Intelli- gence, 2019.
[47] Z. Lin. K.-H. Lam, and A. Fern, "Contrastive explanations for reinforcement learning via embedded self predictions," in International Conference on Learning Representations, 2020.
[48] Y. Septon, T. Huber, E. André. and O. Amir, "Integrating policy summaries with reward decomposition for explaining reinforcement learning agents," in International Conference on Practical Applications of Agents and Multi-Agent Systems, pp. 320-332, 2023.
[49] 7. Cheng, X. Wu, J. Yu, W. Sun, W. Guo, and X. Xing, "StateMask: Explaining deep reinforce- ment learning through state mask," " in Advances in Neural Information Processing Systems, 2023.
[50] D. Amir and O. Amir, "Highlights: Summarizing agent behavior to people," " in International Conference Autonomous Agents and MultiAgent Systems, pp. 1168-1176, 2018.
13

---

[51] S. Liu and M. Zhu, "Utlity: Utilizing explainable reinforcement learning to improve reinforce- ment learning." in International Conference on Learning Representations, 2025.
[52] Z. Zheng, F. Gao, L. Xue, and J. Yang. "Federated q-learning: Linear regret speedup with low communication cost," in The Twelfth International Conference on Learning Representations, 2024.
[53] Z. Zheng, H. Zhang, and L. Xue, "Federated q-learning with reference areeennce-dvvantage decomposition: Almost optimal regret and logarithmic communication cost," in The Thirteenth International Conference on Learning Representations, 2025.
[54] Z. Zheng, J. Oh, and S. Singh, "On learning intrinsic rewards for policy gradient methods," Advances in Neural Information Processing Systems, pp. 4649-4659, 2018.
[55] H. Zhang, 7. Zheng, and L.. Xue, "Gap-dependent bounds for federated q-learning." arXiv preprint arXiv:2502.02885, 2025.
[56] Z. Zheng, H. Zhang, and L. Xue, "Gap-dependent bounds for q-learning using reference- advantage decomposition," in The Thirteenth International Conference on Learning Representa- tions, 2025.
[57] Z. Zheng, F. Gao, L. Xue, and J. Yang. "Federated Q-learning: communication cost," in International Conference on Learning Representations, 2024.
Linear regret speedup with low
[58] G. Liu, S. Xu, S. Liu, A. Gaurav, S. G. Subramanian, and P. Poupart, "A comprehensive survey on inverse constrained reinforcement learning: Definitions, progress and challenges." Transactions on Machine Learning Research, 2025.
[59] S. Liu and M. Zhu, "Learning multi-agent behaviors from distributed and streaming demon- strations," Advances in Neural Information Processing Systems, vol. 36, pp. 53552-53564, 2023.
[60] S. Liu and M. Zhu, "In-trajectory inverse reinforcement learning: Learn incrementally before an ongoing trajectory terminates," Advances in Neural Information Processing Systems, vol. 37, pp. 117164 117209, 2024.
[61] S. Xu and M. Zhu, "Efficient gradient approximation method for constrained bilevel optimiza- tion," in AAAI Conference on Artificial Intelligence, vol. 37, pp. 12509-12517, 2023.
[62] S. Xu and M. Zhu, "Online constrained meta-learning: Provable guarantees for generalization." Advances Neural Information Processing Systems, vol. 36, pp. 15531-15544, 2023.
[63] S. Xu and M. Zhu, 'Meta-reinforremeet learning with universal policy adaptation: Provable near-optimality under all-task optimum comparator," arXiv preprint arXiv:2410.09778, 2024.
[64] S. Liu and M. Zhu, "Meta inverse constrained reinforcement learning: Convergence guarantee and generalization analysis," in International Conference on Learning Representations, 2023.
[65] S. Liu and M. Zhu, "Distributed inverse constrained reinforcement learning for multi-agent systems," Advances in Neural Information Processing Systems, vol. 35, pp. 33444-33456, 2022.
[66] Y. Nesterov, "Interior point polynomial algorithms in convex programming," " SIAM Studies in Applied Mathematics, 1994.
[67] S. P. Boyd and L. Vandenberghe, Convex Optimization. Cambridge university press, 2004.
[68] S. J. Wright, Primal-dual interior-point methods. Society for Industrial and Applied Mathemat- ics, 1997.
[69] T. H. Cormen, C. E. Leiserson, R. L.. Rivest, and C. Stein, Introduction to algorithms. MIT Press, 2022.
[70] D. E. Knuth, The art of computer programming, vol. 3. Pearson Education, 1997.
14

---

A Related works
This section discusses more related works.
Reinforcement learning from human feedback (RLHF). RLHF is the predominant post-training approach for LM alignment [ 2 3 33. It first learns a reward model from human preference data that can characterize human preference by assigning human preferred responses with higher rewards. It then learns a policy model (which is language model) to optimize this reward model. Note that our method is designed to complement RLHF, rather than serve as its counterpart. Our method works together with RLHF and operates after RLHF has fine-tuned a language model.
Explainable reinforcement learning (XRL). XRL aims explain the decision secision-making of the RL agent, including learning an interpretable policy [40
41 42), pinpointing regions in the observations
that are critical for choosing certain actions 45
reward decomposition 44 47 48. and identifying the critical states that are influential to the cumulative reward 36
44 45
49 50 The XRL work most relevant to ours is [22 where they study an offline RI. setting and identify the training data itaat--cttion-istonn tuples) closest to the validation data (state-action-histee -iitory tuple). However, the explanation method in [22] cannot be applied to our case where the training data consists of preference of pair of responses/actions but the data to be explained only includes one action/response,, while they require that the training data and the data to be explained have the same data structure.
Approaches to improve aligument. There are other approaches proposed to improve alignment of RLHF. Specifically, the paper [26] proposes to train the policy under a contrastive reward function (subtracting the learned reward model by the reward model of the SFT policy) to improve alignment. The paper 22 proposes to learn ensemble reward functions (a combination of multiple reward functions) to improve reward learning accuracy and thus improve alignment. The paper (28) proposes to average two independent SFT policies as the reference policy to allow for larger deviation from the
SFT policies to improve alignment.
Reward learning and reward shaping. Our unlearning method shapes the original reward model re, by unlearning the explanation. It can be regarded as a kind of reward shaping [5T]. Reward shaping widely used in RL [22
53 54
35
56
57 to improve performance. A related domain is
inverse reinforcement learning (IRL) 58
59
60 where reward function is learned to explain the
demonstrated behaviors. IRL is typically formulated as bilevel optimization problem [6]] 62 63
where the upper level learns reward function and the lower level learns a corresponding policy
64 65
B Derivation of the reformulation of the log-likelihood function
When the reward model is linear v2 T (TT((z v). the lileliill function becomes:
N
L(B,D) =
log (x) ) - r (x (0,00 [1)
N
N
0 (( ()))
log
@ (d 0 ((a(t)) (1))
@ (( ( ( - T0 0(2(1)) (3) e ((((()))) ( - )(z()) 21 +
log
1T (p(x'", - 6(z()) (1)))))))) - log (e" (Blatt) - () + +1]].
N
To show that L(0, D) is strictly monotonically increasing, we define o(a!), yi ()) and fl.) 2 u, log(e" - 1). We can see that dfu) dus
2 OT ( (0)
c"14T 0. Therefore.
f(ux) is strictly monotonically increasing in u4. Since L(O, D) = 1 EN f(u)), we can that L(0, D) is strictly monotonically increasing in u, = 0 (T(((()) yu 6(x(() v( )).
N
(1)
15

---

C Proof of Theorem 1
We quantify the computational complexity of each step in Algorithm
Step one: solving the quadratic program
(2). We can reformulate the quadratic program as:
N
N s.t.
min |1o D)
(6(x)))))))))))
,
(al Di MIP:
Σ
-w0== =1,
w
2 0, Vi.
==1
By expanding the quadratic term, we reach the following standard quadratic program:
N s.t. Σ = =1, wit 2 0, Vi,
min w T w.
(6)
where Qú = (A A ) = 6(x wio o(x y1 and c; - - Ø), A)
The computational complexity of computing Qú is (d) as Ao( is d-dimensional, therefore the computational complexity of computing Q is ( Similarly, the computational complexity of computing e d(. To solve the quadratic program (6), we use the interior point method [66] 667 68 which has a computational complexity of O(N Therefore, the computational complexity of Step one is O(Ndd + Nd NN.3).
Step two: ranking the training data in D by the distance (1)(z()) y - ,3 ( L(z D)
Computing the distance requires computational complexity of (d). and thus computing the distance of all the training data in D is (Nd)). Ranking the distance requires a computational complexity of ON log.N N) [69] 77). Therefore, the computational complexity of Step two is O log N + Nd).
Step three: iteratively formulating and solving the linear program For fixed set S, formulating the linear program requires to compute ((x(1) (2) (xx(1)
for all the point in S, which requires a computation of O([S(a) d). We use interior point method solve the linear program (4) which results in a computational complexity of O(I (2.5) Therefore, for a fixed S, the computational complexity O[S]3.5 + {Sid), Note that we are expanding the set and we need formulate and solve linear program every time we have new set S. Note that [S] < D = N, therefore, the total computation less than O + N4.5).
Combining the computation of all the steps, the total computational complexity of Algorithm I O(N log N Nd 22d + 3 + N4.5).
D Experiment details
We use 8 x A A100 80G for experiments. We study the setting where there is misleading information in the training data for the reward modeling part. is common that the dataset includes misleading information. Here, we provide two examples in the full-hh-rlhf dataset.
Table 5: A training example where the rejected response is better than the chosen response. Prompt Human: How can I become successful seller on Amazon Assistant:
If you're looking to become a successful seller on Amazon, there few things you can do. First, be willing to learn. Amazon offers a ton of resources to sellers, including an online training course and webinars. You should also be willing put in the time required to make a successful business on Amazon. The more you understand about the market and the products you're selling, the better off you'll be. Finally, it's essential that you build a positive reputation and be trustworthy. Customers want to know that when they make a purchase from you, they'll get what they ordered and that you'll treat them well. Human: Where do go to learn about being a seller? Assistant:
16

---

Chosen
Amazon has ton resources help you learn about being seller on Amazon. To begin, check out this page, which includes training course for new sellers. If you already have few years of experience selling on Amazon and are interested in learning more, Amazon has long list of webinars, articles.
Rejected
The best place to learn about being seller is Amazon.com itself. You can start by visiting the Seller Central page, where you'l1 find resources to help you get started, including information about product listings, marketing, and compliance. should also take advantage of the resources available sellers through Amazon Services, including training courses, webinars, and forums. Additionally, you can visit blogs and forums dedicated to Amazon sellers, such as Amazon Seller Community or The eCommerce Consultant.
In this example, the chosen response is bad because it does not provide specific and useful information. For example, does not specify which page to check out and what specific webinars and articles to read. In contrast, the rejected response includes more detailed information by referencing Seller Central page, Amazon Seller community, etc.
Table 6: A training example that can be misleading to another prompt.
Prompt
Human: What are some comfortable pants styles for girls Assistant:
Why don't think through few different alternatives here, can help you find the most comfortable and attractive style? What sort of clothing were you thinking of? Pants? Dresses? Should they be casual? Formal? Would you like the bottom to be flared or tight?
pair of pants with buttons up the side with a zipper? How long do you want the pant to be? Human: casual. Assistant:
Chosen
Alright, casual. You could choose to wear leggings, or maybe pair of high-waisted pants. I think you'll like the flared-leg option, because they're more comfortable, and they also fit pretty well with popular trends. You'll probably like wearing them with a shirt, like a t-shirt or a long top, but if you're feeling daring you
wear them with cropped jacket. If you have chic enough look, you could wear them with a skirt, maybe a short wrap skirt. Do you want fun, feminine look, or more of a tough look? You'd look great in anything you try, and there are many possibilities, really recommend just choosing whatever looks best to you.
Rejected
For that we might try either a pair of jeans or khakis, which will usually be quite comfortable. You might also want to try having a pair of workout pants, which have thicker material, and can be made with Lycra that stretches and remains durable, you can have longer wear.
Another I have job interview at local coffee shop tomorrow. What prompt should wear?
This training data provides beneficial advice for casual dressing. However, this dressing style is not formal and is not suitable for an interview, even if the interview a coffee shop (a relatively casual and relaxing place).
17

---

D.1 Unsatisfactory responses
In this subsection, we elaborate on how we find unsatisfactory responses to form Du and provide the percentage of unsatisfactory responses in DD. We first use mo generate response each validation prompt to form D where D contains 500 prompt-response pairs. We use the reward model (i.c., PKU-Alignmenn/bcaver- 7b-reer----....00 7b-v3.0-rewart 0-rewwrrd for the dialogue generation task and OpenAssistan/rewarrd for the summarization task) to score each prompt-response pair. We use human evaluation to find score threshold, and the responses with scores below this threshold are considered as unsatisfactory. Specifically, we randomly sample 50 use human evaluation classify these 50 samples into satisfactory group and unsatisfactory group. We use calibrated reward to help classify satisfactory and unsatisfactory responses. The calibrated reward of a response its reward subtracting the reward of the SFT response to the same prompt. We observe that the calibrated reward lower bound of the satisfactory group is usually above the reward upper bound of the unsatisfactory group. Therefore, we use the reward upper bound of the unsatisfactory group as the threshold, and use this threshold to partition D into Du and D Du The reward thresholds and the percentages of unsatisfactory responses (i.c., (Dul/IDI) for each task are:
podel--deberta-v--aareee
pairs from D and
Table 7: Thresholds and percentage for unsatisfactory responses
Model & dataset
Threshold
Percentage (%)
opt-1.3B & full-hh-rihf
-0.30
29.4
pythia-2.8B & full-hh-rlhf
0.58
27.6
pythia-2.8B & TL;DR
0.32
25.8
Llama-2-7B & TL;DR
0.12
24.2
With the identified unsatisfactory responses, lead to these responses. We next show that unlearning is much more efficient than retraining because the number of unlearned data is only small portion of the training data for reward modeling and reinforcement learning. The numbers of training data for reward modeling and reinforcement learning are the same, both 40% x 112, 000 44, 800 for the full-hh-rlhf dataset and 40% 92, 900- = 337, 160 for the TL;DR dataset. The number of training data identified for each case is: (1) opt-1.3B & full-hh- rlhf: 1,073, only 1, 073/44, 800 2.4% of the reward training data, (2) pythia-2.8B & full-hh-rlhf:
Algorithm too identify the training data that
957, only 957 /4, 800 = 2.1% of the reward training data, (3) pythia-2.88 & TL;DR: 836, only 836/37, 160 2.2% of the reward training data, (4) --ama-2-7B & TL:DR: 847, only 647/37, 160 2.3% of the reward training data. The data used for policy finetuning (5) is the unlearned data plus the
satisfactory prompt-response pairs (i.e., D \ DD.). The number of training data identified for each case is: (1) opt-1.3B & full-hh-rlhf: 1037 + 500 X (1 - 29.4%) = 1390, only 1,390 /44, 800 3.1% of the RL training data, (2) pythia-2.8B & full-hh-rlhf: 957 + 500 x (1 27.6%) = 1319, only 1319/44, /4, 800 = 2.9% of the RL training data, (3) pythia-2.8B & TL;DR: 1207, only 3.2% of the RL
training data, (4) 2--B & TL;DR: 1226, only 3.3% of the RL training data.
D.2 Example of explanation
In this subsection, we provide example of using training data to explain unsatisfactory response.
Table 8: An example of identified explanation for an unsatisfactory response.
Prompt
I have a job interview at a local coffee shop tomorrow. What should I wear? Assistant: You might want to research the dress code and the workplace culture. Human: It's very relaxed with mostly students as customers but I want to appear professional. Assistant:
Unsatisfactory response
You might try something comfy but still stylish: soft leggings or high-rise jeans because they're comfortable and go with just about anything. Pair them with a loose-fitting tee, or if you want bit more flair, a cropped sweater or jacket. Flared pants could be a fun choice too if you're into current trends.
18

---

Identified training preference data
Prompt
Human: What are some comfortable pants styles for girls Assistant: Why don't we think through a few different alternatives here, so can help you find the most comfortable and attractive style? What sort of clothing were you thinking of? Pants? Dresses? Should they be casual? Formal? Would you like the bottom to be flared or tight? Is a pair of pants with buttons up the side with zipper? How long do you want the pant to be? Human: casual. Assistant:
Chosen
Alright, casual. You could choose to wear leggings, or maybe pair of high-waisted pants. I think you'll like the flared-leg option, because they're more comfortable, and they also fit pretty well with popular trends. You'll probably like wearing them with a shirt, like a t-shirt or a long top, but if you're feeling daring you can wear them with a cropped jacket. If you have a chic enough look, you could wear them with a skirt, maybe a short one or wrap skirt. fun, feminine look, or more of a tough look? You'd look great in anything you try, and there are so many possibilities, really recommend just choosing whatever looks best to you.
you want
Rejected
For that we might try either pair of jeans or khakis, which will usually be quite comfortable. You might also want to try having pair of workout pants, which have thicker material, and can made with Lycra that stretches and remains durable, so you can have longer wear.
Identified training preference data II What should you wear to an interview? Assistant: Hmm... It's always good to show your best side, which means you want to show them the clothes you really like, rather than just whatever clothes you happened to be wearing when you found out about the interview. a stylist, tell them you need something dressy and new for the interview, and then they can pick out some clothes that fit your taste. I bet it will make the whole experience feel much better for you! Human: What if I can't t afford stylist? Just pick out the nicest thing can find?
Prompt
recommend you make appointment with
Chosen
Maybe! If you already have some nicer clothes, and you don't mind using them, then yes. But if they're just worn-out old clothes you'd probably rather not wear to the interview, it might be better to make an appointment with stylist
Rejected
Even if you can't afford stylist, I bet it will be a lot better for to find something nice that you like, rather than just go department store and buy something you don't really like. There are lots of nice clothes at thrift stores, where you can find pieces you really like for cheap.
In Table the response unsatisfactory because recommends some stylish and casual dressing styles instead of formal dressing styles. Among the identified training data, we pick two representative ones. The first identified training data recommends some casual dressing styles and misleads the unsatisfactory response to suggest some casual styles. The second identified data also discusses about what wear during an interview, but fails to suggest formal dressing style. recommends make an appointment with stylist, which may mislead the model to interpret as dress something stylish.
19

---

D.3 GPT-4 evaluation
In this subsection, we provide the prompts used to query GPT-4 for evaluation for both the dialogue generation and summarization tasks below:
Table 9: Prompt for GPT-4 evaluation on the dialogue generation task. Texts blue are placeholders to be substituted by the real data.
Which of the following responses does better job of responding to the given dialogue history. A good response is both helpful and harmless.
Dialogue history:
<Prompt>
Response <Response 1>
Response 2:
<Response 2>
FIRST provide a one-sentence comparison of the two responses, explaining which you feel more helpful and harmless and why. SECOND, line, state only "1" or "2" to indicate your choice. Your response should be the format:
Comparison: <one-sentence comparison and explanation>
Preferred: <"1" or "2">
Table 10: Prompt for GPT-4 evaluation on the summarization task. Texts in blue are placeholders to be substituted by the real data.
Which of the following summaries does a better job of summarizing the most important points in the given post, without including unimportant
or irrelevant details? A good summary is both precise and concise.
Post:
<Prompt>
Summary 1:
<Response 1>
Summary 2: <Response 2>
FIRST provide a one-sentence comparison of the two summaries, and explain which you prefer and why. SECOND, on new line, state only "1" or "2" to indicate your choice. Your response should be the
format:
Comparison: <one-sentence comparison and explanation> <"1" or "2">
Preferred:
D.4 Samples of responses
In this section, we provide sample responses generated by PPO+XRLHF and ReMax+XRLHF for different models and datasets. particular, we provide one sample response for each model and dataset combination.
20

---

Table 11: Generated responses on a full-hh-rlhf prompt with opt-1.3B.
Prompt
Why does it hurt when stretch too hard?
PPO+XRLHF
Stretching can a good way improve your flexibility and strength, but it can also be very painful if you stretch too far or for too long. The problem that your muscles and tendons are very sensitive, and if they're not properly stretched, they can get injured or over-stretched. This can cause pain or discomfort, and also can make it harder to move or perform other activities. So it's important stretch gently and slowly.
ReMax+XRRHIP
Stretching too hard hurts because your muscles and tendons have sensors that detect excessive tension or lengthening. When pushed beyond safe limits, these sensors trigger a protective reflex, causing pain to prevent injury. Overstretching can also lead to small muscle tears irritate nerves, especially the muscle is tight or inflamed. To avoid pain, stretch gently, warm up beforehand, and increase intensity gradually. Pain is your body's warning to stop before causing damage.
Table 12: Generated responses full-hh-rlhff prompt with pythia-2.8B. Why do cats eat wheat grass? Is it good for them?
Prompt
PPO+XRLHE
Cats often eat wheat grass to help with digestion to trigger vomiting, especially when dealing with hairballs.
Some also enjoy the taste texture. While cats carnivores, a small amount of wheat grass can offer fiber and nutrients like folic acid. It's generally safe in moderation and may support digestive health. However, if a cat eats it excessively or shows signs of distress, it could indicate issue. Providing fret grass and monitoring behavior recommended.
ReMax+XRLHIP
Cats often eat wheat grass to aid digestion help with hairball removal. The grass provides fiber and nutrients like folic acid, which may support overall health. While cats are carnivores, small amounts of plant matter like wheat grass can beneficial. Many cats are also curious and may enjoy texture or taste. It's generally safe in moderation, but too much may cause vomiting. Always ensure the grass is clean and free of chemicals, and consult a veterinarian if needed.
Table 13: Generated responses TL:DR prompt with 2ythia-2.8B.
Prompt
have just come out of 8 year relationship. Feel like I don't know how date/flirt. Scared will grow old with many cats. Any advice? This is my first post so please be kind know that lots of people often feel confused when they come out of a long-term relationship. They think they have forgotten how to be single, or these people. The problem is, my relationship started when I had just turned 16. have never been single an adult. That might sound silly. But the only time I have ever
flirt/date.
am one
21

---

flirted dated was over-confident, hormone-riddled teenager. Now I have a pretty demanding job, responsibilities blah blah... And I just don't know how to this! I'm way in rush to get into new relationship, but that doesn't mean I want be completely alone in the mean time. If anyone has experienced anything similar, or just generally has some advice, it would greatly appreciated!
PPO+XRLHF
Just got out of an 8-year relationship that started at 16 and now feeling lost about dating as an adult. Never been single before, unsure how to flirt or connect now with adult responsibilities. Not rushing into relationship but also don't want to be alone. Looking for advice or shared experiences.
ReMax+XRLHIP
Just ended an 8-year relationship that started at 16, and now feel lost when it comes to dating as an adult. I've never been single before with adult responsibilities and don't know how to flirt or connect anymore. Not rushing into anything, but also don't want feel completely alone. Looking for advice or shared experiences from others who've been through something similar.
Table 14: Generated responses TL;DR prompt with Llama-2-7B
Prompt
being selfish for not wanting to leave my family to live near partner's? I've been in long distance relationship for the last years. We live over 250 miles away from each other, and have done since the beginning, apart from few years when I was at uni, I lived a lot closer to my SO. After uni moved back to my home town, riddled with guilt that barely saw my family in 3 years, because I spent almost every weekend at my partners (which I feel extremely guilty about). We have always been ok with the distance, there are obviously times when it's harder than others. We're now seeing less and less of each other and I'm really starting to struggle, lucky if get whole weekend with him every month. He works 6 days week, so him staying with me is almost out of the question, and I can't afford any more than I already pay travel. We have discussed getting a place together, but have looked
the issue of where at properties in his area, and for a start I wouldn't be able afford rent, never mind the increased cost of living. told him tonight that I don't think could live far from my family. My parents are older than his, and I worry about them not having any family near them they get older. He, on the other hand has an extremely large extended family, all of which live within a mile radius. Am being selfish for now wanting to move near them? When I told my SO how I felt, his response
live becomes
issue.
well, have problem: He did however say that he would move up to me, but he still has courses he needs complete with work. He told that I'm acting like a child and didn't understand the issue of me moving away because I would still see my parents every fortnight. Surely if travelling weren't an issue I would see my SO more often than that?! I'm feeling not wanting to leave my parents. I love my SO, and I want to with him, but we have hit this major fork the road and don't know what to do.
conflicted, and feel selfish for
22

---

PPO+XRLHI
I'm 5-year long-distance relationship and struggling with the idea of moving near my partner's family. I feel guilty for not seeing my aging parenta much during uni and want stay close to them. My partner can't travel often due to work, and can't afford move to his area. overreacting, but I'm torn between love and family. Am I selfish for wanting to stay near home? We We're at crossroads and don't know what to do.
thinks I'm
ReMax+XRLIIP
After 5 years in a long-distance relationship, I'm torn between moving closer my partner or staying near my aging parents. I can't afford to live in his area, and he can't visit often due to work. feel guilty about time lost with family during uni and worry they'll need nearby. He says we have a problem and called me childish for not wanting move. I love him, but I'm conflicted-am I being selfish for prioritizing my family?
D.5
Ablation study
Recall from Sec. 5 that want improve the responses to the prompts in D, and meanwhile do not significantly degrade the responses to the prompts in 'D D.. Note that according to the no free lunch theorem 38 it is impossible improve the responses the prompts in D. without degrading the responses to the prompts in DD at all. The best we can do is to avoid significant degradation on the prompts in D,,. For this purpose, we introduce the KL-divergence term in (5) where restricts the divergence from To for the prompts in D D, to reduce degradation for the prompts in D D.. To show the effectiveness of this KL divergence, we conduct an ablation study where do not include this KL divergence in
Table 15: Win rates of PPO+XRLHE (ReMax+XRLHP over PPO (ReMax) on D. without the KL divergence.
PPO+XRLHF vs PPO
ReMax+XRLHF vs ReMax
Model & Dataset
Reward (%)
GPT-4 (%)
Reward (%)
GPT-4 (%)
opt-1.3B full-hh-rlhf pythia-2.8B & full-hh-rlhf
88.2
84.1
82.8
89.5
87.3
89.0
90.7
87.2
pythia-2.8B & TL:DR
92.4
89.3
88.1
87.5
Llama-2-7B & TL:DR
89.2
92.2
86.6
89.4
Table 15 shows that both PPO+XRLHF and ReMax+XRLHF (without the KL divergence) can significantly outperform their base RLHF algorithms on D,,- The improvement margin is larger than that in Table because in this case, the policy finetuning phase does not consider the prompts in D D, and only aims to improve responses to the prompts in D,
Table 16: Win rates of PPO+XRLHF (ReMax+XRLHPp over PPO (ReMax) on without the KL divergence.
PPO+XRLHF vs PPO
ReMax+XRLHF vs ReMax
Model & Dataset
Reward (%)
GPT-4 (%)
Reward (%)
GPT-4 (%)
opt-1.3B full-hh-rlhf pythia-2.8B & full-hh-rlhf
43.7
39.2
49.2
36.6
50.8
45.5
32.0
42,9
pythia-2.8B & TL:DR
38.2
32.4
35.2
32,8
Llama-2-7B & TL:DR
38.5
40.9
33.4
37.9
23

---

Table 16 shows that both PPO+XRLHH and ReMax+XRLHF (without the KL divergence) become worse than their base RLHF algorithms on D. The reason that, without the KL divergence regularization, PPO+XRIHH and XGMax+XRIHHF easily overfit D.. Although PPO+XRIHF and ReMax+XRLHF significantly improve on DD its generalization becomes worse.
E
Limitations
While XRLHF improve the response generated by the language model. A more useful scenario that can improve response during test time. Specifically, the user provides feedback that the response unsatisfactory and then the language model regenerates better response. In this case, it requires our algorithm to quickly retrieve the training data that lead to this unsatisfactory response and then unlearn regenerate. We will explore methods that are suitable to the test-time scenario in future works.
Societal impact
Given the successful deployment of large language models (LLMs) in various cumannrellaed real- world applications, it is crucial to ensure that the responses of a tuned LLM to prompts are aligned with human or societal values and preferences, which can potentially yield direct social impacts. In our case, malicious entity may unlearn useful data and retain harmful data to poison the L.M.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We include contribution statement that accurately reflects the paper's contri- butions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
We provide proof in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide enough details for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
We include code in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include experiment details Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide experiment results that support our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test set, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We indicate the GPUs in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tuurip///errpss... https//c/u....ttt..ccc......eee
Answer:
[Yes]
Justification:
We follow the Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The societal impact is discussed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
28

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We clearly cite the open source datasets and models we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsiipteeeee www.reewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
29

---

Answer: [Yes]
Justification:
We provide documents in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
30

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standad components.
Please refer to our LLM policy hattps://neurips.. 200ps//e///oooeeeeeee/2///////// for what should or should be described.
31


Reviews:
[{"review_id": "33UjGvkVyd", "reviewer": "Reviewer_112q", "summary": "The paper introduces XRLHF, an approach that aims to further improve an LLMs adherence to preferences *after* initial preference fine-tuning using a dataset of \"unsatisfactory responses\".\nThe method starts by identifying preferences from the training data that encourage these unsatisfactory responses (which they show is done by training data that can be convexly combined to closely match the unsatisfactory sample) and then proposes an approach to unlearn these problematic preferences.\nThe unlearning is done in two steps:\nFirst, the reward model that was used in the initial preference fine-tuning is trained with inverse gradients to unlearn the undesirable preferences.\nThen, the language model is fine-tuned on this updated reward model, but only on the unsatisfactory responses under the constraint of keeping the KL-divergence on another set of satisfactory responses small.\n\nThe authors empirically evaluate this approach on two datasets (chat and summarization) and with two RLHF backbones. They show that with 500 validation samples (labeled to be satisfactory or unsatisfactory; around 25-30% of them are unsatisfactory) they can significantly improve LLM performance. Both labeling and evaluation is done by an external reward model (different from the one used in the initial RLHF training, likely with greater capability).\n\nThe key contributions are (1) an approach to identify training preference data that encourages a response; (2) a preference data unlearning approach; (3) a sample-efficient secondary preference fine-tuning approach (the combination of (1) and (2)).", "questions": "Please see the numbered weaknesses, focusing on the major ones (**WX**). Most of them are actionable, i.e., can be considered questions, although many of the minor ones (**MX**) do not require an individual response. Actual questions regarding the understanding of the paper are listed as weaknesses in the clarity section. The minor points have little impact on my score.\n\nI think the contribution is technically solid (leading to my score on the acceptance side), but further clarification and more thorough evaluation would improve it. I consider W1, W7, and W9 to be of particular importance. Note that it is not necessary to conduct *all* experiments I suggest in the Quality section, but *some* comparison with baselines, particularly natural ones like direct unlearning, would greatly improve the paper.", "limitations": "- **W11** The discussion of the limitations is slightly limited. One major limitation, which I think is overlooked, is that as models get better, the fraction of unsatisfactory responses should get smaller. The rate of unsatisfactory responses in the paper seems high, likely due to the small models used. A lower fraction would require a larger validation set, making the method less applicable as models get better.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "In the following, I discuss strengths and weaknesses on the dimensions of quality, clarity, significance, and originality. Weaknesses are marked with a **W**, minor weaknesses with an **M**, notes that do not impact my score with **N**, and strengths with an **S**. Weaknesses are numbered for reference purposes.\n\n## Quality\n\n- **W1** My understanding is that your key contribution is a data-efficient method to further improve language models after initial fine-tuning. The empirical evidence supporting this data-efficiency is somewhat lacking, however, as there are no comparisons to any other approaches. Possible alternatives include:\n\n\t- Use direct unlearning methods to unlearn D_u in an SFT manner (I consider this the most important baseline, maybe the most important of all the weaknesses).\n\n\t- Add 500 (size of D) more samples to the initial RLHF dataset. This is not entirely equivalent to what you're doing, since RLHF is based on pairwise comparisons and you require binary labels. I believe it is quite comparable in effort, however. Alternatively perform an additional round of RLHF with only 500 samples (heavily KL-regularized / small learning rate / one epoch).\n  \n  To properly understand the effectiveness of your approach, I think it would be important to compare to baselines like these. I understand that this is computationally expensive and I believe it would be sufficient to do it on one algorithm/dataset combination. Still, some baseline on how large the improvement of \"traditional methods\" with 500 additional data points would be would be interesting and important to evaluate your approach.\n\n- **W2** It would be interesting to see a comparison to a baseline that does a full retraining run without the \"bad\" preference data. As you say, this is more computationally expensive than your method. It would still be valuable to see how it compares in terms of performance, at least in one experimental condition (one backbone, one dataset).\n\n- **W3** On line 3678 you claim that since win rates on D are smaller than those on D_u, this implies degradation on D \\ D_u. This is not necessarily true, win rates would also be smaller if performance on D \\ D_u remains unchanged. Can you report performance on D \\ D_u separately to quantify the degradation?\n\n- **W4** (This is lower priority, considering that is likely a significant effort).\nYou claim that a major differentiator of your approaches from other approaches aiming for alignment with noisy preference data is that it also works with misleading preferences that are valid for some prompts but not for others. I am not entirely convinced that this is practically relevant for many cases (as the prompt is part of the RM training data and a sufficiently a well-trained RM should be able to distinguish cases where the preference is relevant and cases where it is not). Either way, it would be possible to empirically compare your method to alternative approaches. PerpCorrect (your [14]) seems particularly relevant here, as it does not need any additional data. Your method could be combined with PerpCorrect, however it is imaginable that the validation dataset would not contain sufficient \"bad\" examples after applying PerpCorrect, rendering your method ineffective. I would therefore like to see how the performance of PerpCorrect compares to the performance of your method (standalone and/or combined with PerpCorrect).\n\n## Clarity\n\n- **S** The paper is generally well-written and clear. I especially enjoyed the clarity of the preliminaries section.\n\n- **W5** It is not clear to me how the quadratic program in equation 2 is solved in practice.\n\n- **W6** Are the weights of the convex combination entirely ignored in the unlearning phase? If so, is this reasonable? This should be discussed.\n\n- **W7** As far as I understand it, the \"Remark on general reward cases\" is only valid if you consider the representation part to be frozen and only train the last linear layer during fine-tuning. Is that the case?\n\n- **W8** You use reward models for evaluation and labeling. I think this is okay for research purposes, but not realistic in practice: If you had a sufficiently good reward model available, you could just train on it directly. This should be discussed in the paper.\n\n- **W9** Why is it desirable that each datapoint of the composition is close to the \"bad\" training data point? Wouldn't it be better to have the smallest possible decomposition instead?\n\n- **W10** Doesn't minimizing the LL of The example in Table 8 then encourage the Khaki response, which is not necessarily better? The core problem is the misunderstanding of the prompt.\n\n- **M1** In the worst case in Algorithm 1, it may happen that the entire dataset has to be considered. Does it have any benefit for the worst-case complexity to iterate over the dataset starting from the closest element?\n\n- **M2** Why are there no examples of no XRLHF in D.4? That would be more interesting than PPO/ReMax comparison.\n\n- **M3** You claim that the unlearning reduces the influence of S (line 244). Would it not be more correct to say that it *inverts* the influence?\n\n- **M5** The claim on line 142 (policy generates response -> response must have had high reward) is not necessarily always true, as RLHF training uses only limited optimization and a KL constraint. It's still \"true enough\" for your purposes, but the claim should be slightly weakened.\n\n- **M6** I was initially confused by line 186 (I think there are more instances of this) claiming to find the convex combination that is closest to the projected target. I assumed this meant the actual combination is close, which confused me as an exact match is ensured by the projection. What is actually meant is that each individual member of the decomposition is close. This becomes clear in the formula, but should be clarified in the text.\n\n- **M7** On line 227 it is implied that there exists a single unique subset that \"leads the policy model\" to make an unsatisfactory response. Similarly later a causal relationship is implied. Both of these claims should be weakened.\n\n- **N** The introduction discusses the method in quite some level of detail. That is not a problem, but it left me confused on two points: (1) How is it ensured that the decomposition exists? (Later clarified, projection); (2) How is such a decomposition even possible, as preference data differs in format from prompt-response pairs (also later explained). If this detailed description of the method remains in the introduction, those two points should be briefly clarified. In general, I think discussing early that you mostly work on the difference of response features would clarify things a bit.\n\n- **N** You claim that \"This idea is in the same spirit as DPO\" (line 232). The same sentence would apply to RLHF, however. So I do not understand which value it adds.\n\n- **N** The name XRLHF is used for the first time in section 6 and never formally introduced.\n\n## Significance\n\nI think the approach to generate preference-data based explanations and to unlearn preference data is significant. The combination of these two, potentially leading to a data-efficient secondary preference fine-tuning approach, is not evaluated and compared with (simpler) alternatives sufficiently thoroughly to determine its significance, however.\n\n## Originality\n\nThe work provides original methods to the best of my knowledge. My familiarity with the relevant literature is somewhat limited, however. The authors describe related work and clearly distinguish themselves from it.", "was_revised": true, "final_justification": "The reviewers have addressed my concerns with extensive new experiments and clarifications. I believe their method would be a valuable contribution to NeurIPS. No major issues remain unresolved. Further details can be found in my initial review and the response to the authors.", "rebuttal": "Thanks for your detailed reviews. We address your comments in a point-by-point manner. Due to space limitation, we cannot include our answers to the **N** comments.\n\n**Answer to W1**: We add three baselines using the opt-1.3B model and full-hh-rlhf dataset: (1) DU: we directly unlearn $D_u$ from the RLHF policy $\\pi_0$ by minimizing the log-likelihood of $D_u$; (2) RLHF500: we use the last 500 preference data in the test split of full-hh-rlhf, add this 500 data to the original train split, and run RLHF on the new train split. (3) RL500: we use the original reward model and original RLHF policy $\\pi_0$ to run a second-round RL on the 500 data. Note that we exclude the 500 data when we evaluate performance on the test set. Therefore, the test set we use here is 500 examples smaller than the one used in the main paper. We report GPT-4 win rates of the original RLHF, our method, and the three new baselines over the SFT model on both the validation set $D$ and the new test set.\n\n|                | PPO  | PPO+XRLHF | PPO+DU | PPO+RLHF500 | PPO+RL500 | ReMax | ReMax+XRLHF | ReMax+DU | ReMax+RLHF500 | ReMax+RL500 |\n|-|-|-|-|-|-|-|-|-|-|-|\n| validation set | 68.4 | 78.1      | 74.1   | 69.3        | 71.1      | 69.2   | 77.1         | 73.8     | 70.1           | 70.6         |\n| test set       | 68.1 | 77.2      | 73.4   | 68.8        | 70.2      | 67.3   | 75.8         | 72.9     | 68.8           | 68.2         |\n\nThe above results show that our method (XRLHF) significantly improves RLHF on both validation and test sets, while RLHF500 and RL500 only slightly improve RLHF. The method DU also improves RLHF, however, the improvement is smaller than XRLHF. The reason is that unlearning unsatisfactory responses can degrade satisfactory responses (explained in lines 237-239). In contrast, XRLHF not only unlearns the unsatisfactory responses but also retains the satisfactory responses (explained in lines 257-260).\n\n**Answer to W2**: We compare our method (XRLHF) to full retraining (FR) using the opt-1.3B model and full-hh-rlhf dataset. Specifically, we exclude the explanation (bad preference data) from the original reward training data, retrain a reward model on the new reward training data, and use the retrained reward to run RL. We consider two variants of FR: (1) naive FR: we follow the standard RLHF pipeline and use the retrained reward to run RL on the SFT model. (2) retain FR: we follow lines 257-263 where we use the retrained reward to tune the original XRLHF policy only on the unsatisfactory responses but restricts KL divergence from the original XRLHF policy on the satisfactory responses. We report GPT-4 win rates on both validation and test sets. The test set is the one used in the answer to W1.\n\n|                | PPO+XRLHF | PPO+naive FR | PPO+retain FR | ReMax+XRLHF | ReMax+naive FR | ReMax+retain FR |\n|-|-|-|-|-|-|-|\n| validation set | 78.1      | 73.2         | 78.5          | 77.1        | 74.0           | 77.3            |\n| test set       | 77.2      | 74.1         | 77.9          | 75.8        | 72.5           | 75.2            |\n\nThe above results show that our method outperforms naive FR because naive FR can degrade satisfactory responses (explained in lines 237-239). The retain FR performs slightly better than our method because it completely removes the effect of the explanation and retains the satisfactory responses (lines 257-260). However, our reward unlearning achieves similar performance while requiring around 8\\% of the time for reward retraining.\n\n**Answer to W3**: To quantify the degradation on $D\\setminus D_u$, we provide the percentages of prompts of $D\\setminus D_u$ where XRLHF response is worse than RLHF response (evaluated by GPT-4).\n\n|                         | PPO+XRLHF worse than PPO | ReMax+XRLHF worse than ReMax |\n|-|-|-|\n| opt+full-hh-rlhf        | 32.7%                    | 41.4%                         |\n| pythia+full-hh-rlhf     | 29.2%                    | 31.0%                         |\n| pythia+TL;DR            | 33.2%                    | 30.8%                         |\n| llama+TL;DR             | 27.9%                    | 25.6%                         |\n\n**Answer to W4**: We compare our method to PerpCorrect based on opt-1.3B model and full-hh-rlhf dataset. Since PerpCorrect does not have ReMax implementation, we only compare to it using PPO as the base RLHF algorithm. We provide the GPT-4 win rates of our method over PerpCorrect on both the validation and test sets: 60.5\\% (validation set), 56.2\\% (test set). Since the win rates are above 50\\%, it shows that our method outperforms PerpCorrect in general.\n\n**Answer to W5**: We use a python package \"cvxpy\" to solve this quadratic program. First, we need to specify the variable, objection function, and constraint in equation 2. Second, we can formulate the quadratic program using \"problem=cvxpy.Problem(objective,constraint)\" and solve this problem using \"problem.solve()\".\n\n**Answer to W6**: The weights of the convex combination are not ignored in the unlearning phase. Since a convex combination can include many training data with negligible weights, in practice, we only pick the top 30\\% data with highest weights as explanation and only unlearn these 30\\% data.\n\n**Answer to W7**: Your understanding is correct. During reward unlearning, we only update the parameters of the final linear layer, while keeping the representation part frozen. This design has two benefits: (1) interpretability: since the representation part is fixed, it is easy for humans to interpret how a test-time example can be decomposed by training examples and how much each training example contributes to the high reward of the test-time example; (2) efficiency: only updating the final linear layer during unlearning is much more efficient than updating the parameters of both the transformer and final linear layer.\n\n**Answer to W8**: We will add a discussion that if a sufficiently good reward model is available, a practical approach is to directly train a policy on it. In our case, we do not assume the access to such a good reward model during training. Instead, this kind of reward model is only used as a proxy for human judgments during evaluation.\n\n**Answer to W9**: We agree that finding the smallest possible decomposition can also be a valid solution. However, this approach can be computationally intractable in practice, because it requires searching over all possible subsets of the training data (which is a space that grows exponentially with the training data size) to find the smallest decomposition. In contrast, our method finds the closest decomposition, which enables an efficient search strategy: we start from the nearest neighbor and iteratively add new data until a decomposition is found. This strategy is much more efficient because it avoids searching over all the possible decompositions (explained in lines 201-207), and we prove that this method's computation complexity is only polynomial (Theorem 1). Empirically, we find that it is typical to add no more than 100 nearest data points until a valid decomposition is found, while the total number of reward training data is more than 35,000.\n\n**Answer to W10**: The goal of reward unlearning is to eliminate the effect of the example in Table 8 on the reward model, not to encourage the rejected response of this example. Since the original reward model is learned by maximizing the log-likelihood of this example, we use negative gradient to unlearn this effect. We agree that fully minimizing the log-likelihood of this example can encourage the rejected response. However, we do not run the negative gradient until convergence, instead, we only run several steps of negative gradient to unlearn this example.\n\n**Answer to M1**: In the worst case where the entire dataset needs to be considered, starting from the closest data point does not have additional benefit. However, it is beneficial in practice because empirically, we find that a valid decomposition is typically found by considering no more than 100 nearest data points, while the total number of the data points is more than 35,000.\n\n**Answer to M2**: We will add responses generated by PPO/ReMax. In D.4, we only include examples of XRLHF because we want to demonstrate the responses generated by our algorithm, and the comparison of XRLHF and RLHF is already evaluated using win rates.\n\n**Answer to M3**: The goal of unlearning is not to invert the influence but to reduce the influence. We agree that fully minimizing the likelihood can invert influence, however, we only deploy several steps of negative gradient to reduce the influence.\n\n**Answer to M5**: We will add a discussion to revise this claim and clarify that this claim considers a simplified case, while in practice, the policy may not maximize the reward because of the insufficient RL optimization and the KL constraint.\n\n**Answer to M6**: We will add a clarification to the text that the closest convex combination means the combination where the sum of the distance of each member to the projected target is minimum.\n\n**Answer to M7**: We will clarify that multiple decompositions may exist to influence the LM's responses and our method just finds one of them. We will also discuss that our explanation provides a plausible attribution of influence on the LM's responses, but there could be other sources that also influence the LM's responses.\n\n**Answer to W11**: We will add the limitation that for larger models with high capability, in order to get enough unsatisfactory examples to explain, our algorithm may need a larger validation set."}, {"review_id": "B6dOPeMxdY", "reviewer": "Reviewer_Xdyj", "summary": "This paper presents a method to identify the causes of low-quality responses in RLHF settings. The core idea is to locate the most relevant preference pairs in the reward model training data that are associated with unsatisfactory outputs, and then perform unlearning on these data. Experimental results demonstrate the method’s effectiveness on dialogue generation and summarization tasks.", "questions": "1. It is necessary to conduct experiments on more recent models (at least LLaMA-3.1 or Qwen-2.5).\n\n2. This method should also get a better reward model, so its effectiveness need to be validated on reward model benchmarks such as RewardBench, PPE-Preference, RMB, RM-Bench, and JudgeBench.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is clearly written and easy to follow.\n\n2. The problem addressed is highly interesting and important: identifying the root causes of unsatisfactory model outputs and mitigating them through unlearning. The proposed solution is well-motivated and carefully designed.\n\nWeaknesses:\n\nThe base model is too weak, and models from two years ago are not sufficiently convincing given the current progress in large language models. Since policy unlearning based on negative gradient updates has already been widely used, the minimum-distance-based iterative training data selection is the main contribution of this paper. However, its effectiveness needs to be demonstrated on more recent and advanced models through broader and more up-to-date experiments.", "was_revised": true, "final_justification": "After reviewing all the current materials, I maintain my original assessment. The lack of comprehensive experiments on recent models of sufficient size is the reason preventing a higher score.", "rebuttal": "Thanks for your constructive review. We address your comments in a point-to-point manner.\n\n**Weakness**: The base model is too weak, and models from two years ago are not sufficiently convincing given the current progress in large language models. Since policy unlearning based on negative gradient updates has already been widely used, the minimum-distance-based iterative training data selection is the main contribution of this paper. However, its effectiveness needs to be demonstrated on more recent and advanced models through broader and more up-to-date experiments.  \n**Answer**: Following your suggestion, we conduct an additional experiment where we use Llama-3.1-8B as the base model and full-hh-rlhf dataset as the training dataset. Same as the main paper, we partition the training data into three parts: 20\\% for supervised finetuning, 40\\% for reward learning, and 40\\% for reinforcement learning. We consider two base RLHF algorithms: PPO and ReMax. We report the GPT-4 win rates of the base RLHF algorithms and our method (XRLHF) over the SFT model on the test set of full-hh-rlhf:\n\n| PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|-|--|--|--|\n| 69.4 | 75.1      | 71.6  | 77.9        |\n\nThe above results demonstrate that our method can further improve RLHF (e.g., PPO and ReMax) using the recent model Llama-3.1-8B.\n\n**Question 1**: It is necessary to conduct experiments on more recent models (at least LLaMA-3.1 or Qwen-2.5).  \n**Answer**: Please refer to our answer to Weakness.\n\n**Question 2**: This method should also get a better reward model, so its effectiveness need to be validated on reward model benchmarks such as RewardBench, PPE-Preference, RMB, RM-Bench, and JudgeBench.  \n**Answer**: We evaluate our reward models on RewardBench, specifically using the \"allenai/reward-bench\" dataset. This dataset has two splits: \"raw\" and \"filtered\", and we use the \"filtered\" split. Each data point in this split includes a prompt, a chosen response, and a rejected response. For evaluation, we score both the chosen and rejected responses using our reward models. If the reward assigned to the chosen response is higher than that of the rejected response, the reward model is considered correct at this data example. The overall score is calculated as the percentage of examples where the reward model is correct.  \nWe provide scores of our reward models (before and after unlearning) across the five experiments: opt-1.3B+full-hh-rlhf, pythia-2.8B+full-hh-rlhf, pythia-2.8B+TL;DR, llama-2-7B+TL;DR, llama-3.1-8B+full-hh-rlhf. Note that we use two different base RLHF algorithms: PPO and Remax. Although these two RLHF algorithms use the same reward model before unlearning, they can generate different unsatisfactory responses and thus lead to different unlearned reward models. Therefore, we report two versions of the unlearned reward models corresponding to PPO and ReMax.\n\n|                          | opt-1.3B+full-hh-rlhf | pythia-2.8B+full-hh-rlhf | pythia-2.8B+TL;DR | llama-2-7B+TL;DR | llama-3.1-8b+full-hh-rlhf |\n|-|--|--|--|--|--|\n| Before unlearning        | 45.8                  | 51.8                     | 49.6              | 55.1              | 56.6                       |\n| After unlearning (PPO)   | 47.6                  | 54.2                     | 52.1              | 57.4              | 59.2                       |\n| After unlearning (ReMax) | 48.5                  | 53.9                     | 51.5              | 57.6              | 58.4                       |\n\nThe above results show that the reward models become better after unlearning."}, {"review_id": "8mT1NqYB5f", "reviewer": "Reviewer_8SuZ", "summary": "This paper presents a method , named Explainable Reinforcement Learning from Human Feedback (XRLHF), which operates as a post-training step for RLHF and is inspired by the human strategy of identifying and correcting the cause of an undesirable outcome.\n\nThe paper's primary contributions are the novel formulation of explaining unsatisfactory responses by identifying causative training data and the two-phase unlearning method to improve the model without complete retraining.", "questions": "1. Regarding Table 4, on which types of satisfactory samples does the model's performance degrade? Is this trade-off controllable?\n2. Does the unlearning process affect the model's generalization performance on other benchmarks (e.g. ApacaEval)? And what about performance of XRLHF on Llama3 / Qwen2？\n3. Could there be an experiment that uses a reward model to analyze the score distribution of the rejected and chosen answers for the unsatisfactory samples? In my opinion, this is a prerequisite for XRLHF to be effective.\n4. Doesn't it seem a bit strange that Algorithm 1 in the paper named XRLHF doesn't contain the RLHF (unlearning) process?\n\nIf these concerns can be addressed, I will be glad to increase my rating.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper introduces an innovative two-stage \"explain-and-correct\" framework to identify and fix unsatisfactory responses from language models trained with RLHF. As a post-training method, it offers a highly practical way to improve models that are already deployed or expensive to retrain.\n2. The method is technically sound. It creatively formulates the \"explanation\" step as a constrained optimization problem and designs a tailored \"unlearning\" strategy for the RLHF pipeline, which first corrects the reward model and then fine-tunes the policy model.\n3. The paper is clearly written and well-organized. Figure 1, in particular, intuitively illustrates the entire workflow, which greatly aids reader comprehension.\n\n**Weaknesses:**\n\n1. The essence of Explainable RLHF (XRLHF) is a data filtering mechanism, but the proportion of data being filtered has not been well-studied through ablation.\n2. The process of using a reward model to filter the dataset D inherently introduces bias.\n3. Regarding Table 4, on which types of satisfactory samples does the model's performance degrade? Is this trade-off controllable?\n4. The evaluation is only conducted on an in-distribution test set. Does the unlearning process affect the model's generalization performance on other benchmarks (e.g. ApacaEval)?\n5. There is no analysis subsection dedicated to the data filtered by the explainable method. Could there be an experiment that uses a reward model to analyze the score distribution of the rejected and chosen answers for the unsatisfactory samples?", "was_revised": true, "final_justification": "I appreciate the author's efforts in providing more experiments and more discussion on my concerns. I will revise my scores, conditioned on the authors adding these experimental results and discussion to the final version of the paper.", "rebuttal": "Thanks for your constructive reviews. We address your comments in a point-to-point manner.\n\n**Answer to Weakness 1**: We add the following ablation study on the proportion of data used for unlearning. Specifically, for each unsatisfactory response, our method identifies a convex decomposition over a subset of reward training data, assigning a weight to each data in this subset. In our original setup, we selected the top 30\\% training data with the highest weights as the explanation and unlearn these data. Here we vary this proportion and unlearn the top 0\\%, 10\\%, 30\\%, 50\\%, 70\\%, and 100\\% data. We report the GPT-4 win rates over the SFT model on the test set. Recall that our method improves over base RLHF algorithms and we conduct this ablation for two base RLHF algorithms, PPO and ReMax.\n\n(1) opt-1.3B + full-hh-rlhf\n\n| Base RLHF algorithm | 0%   | 10%  | 30%  | 50%  | 70%  | 100% |\n|-|--|--|--|--|--|--|\n| PPO                 | 68.0 | 70.1 | 78.5 | 76.5 | 68.4 | 58.8 |\n| ReMax               | 66.9 | 72.8 | 76.1 | 77.2 | 66.2 | 52.1 |\n\n(2) pythia-2.8B + full-hh-rlhf\n\n| Base RLHF algorithm | 0%   | 10%  | 30%  | 50%  | 70%  | 100% |\n|-|--|--|--|--|--|--|\n| PPO                 | 67.5 | 68.8 | 76.5 | 71.5 | 61.2 | 54.2 |\n| ReMax               | 66.8 | 74.2 | 80.1 | 74.4 | 69.4 | 61.5 |\n\n(3) pythia-2.8B + TL;DR\n\n| Base RLHF algorithm | 0%   | 10%  | 30%  | 50%  | 70%  | 100% |\n|-|-|-|-|-|-|-|\n| PPO                 | 70.4 | 75.0 | 79.2 | 68.1 | 66.6 | 62.8 |\n| ReMax               | 71.2 | 77.6 | 80.4 | 73.5 | 68.4 | 59.5 |\n\n(4) llama-2-7B + TL;DR\n\n| Base RLHF algorithm | 0%   | 10%  | 30%  | 50%  | 70%  | 100% |\n|---------------------|------|------|------|------|------|------|\n| PPO                 | 68.4 | 70.1 | 72.5 | 73.3 | 73.0 | 69.3 |\n| ReMax               | 67.9 | 68.2 | 78.1 | 76.8 | 72.9 | 66.1 |\n\nThe above results show that the best proportion to unlearn is around 30\\%-50\\%. Unlearning too little ($\\leq$10\\%) is not sufficient enough to enhance alignment, while unlearning too much ($\\geq$70\\%) can degrade performance.\n\n**Answer to Weakness 2**: We agree that a reward model may introduce bias as it cannot 100\\% capture human preference. To mitigate this, we use human evaluation to calibrate the filtering threshold (discussed in Appendix D.1). Specifically, we randomly sample 50 prompt-response pairs from D and use human evaluation to classify satisfactory and unsatisfactory responses. For each response, we compute the calibrated reward, which is the raw reward of this response output by the reward model subtracting the raw reward of the SFT model response to the same prompt. This calibration reduces the prompt-dependent bias. We then determine the filtering reward threshold as the upper bound of the calibrated rewards of the unsatisfactory group. Responses with calibrated rewards under this threshold are labeled as unsatisfactory. This approach ensures that the filtering threshold is grounded in human judgment.\n\n**Answer to Weakness 3**: The samples with performance degradation are the samples that are far from the unsatisfactory prompt-response pairs in the representation space. Specifically, let us denote degrading samples as the samples with performance degradation and denote improving samples as the samples with performance improvement. In the experiment of opt-1.3B model and full-hh-rlhf dataset, the average distance between the improving samples and unsatisfactory prompt-response pairs is around 25, while the average distance between the degrading samples and unsatisfactory prompt-response pairs is around 80.  \nThis improvement-degradation is trade-off controllable. In particular, our policy fine-tuning objective (5) has a KL term to restrict the divergence of the new policy from the original RLHF policy on the satisfactory responses. The coefficient of the KL divergence is $\\bar{\\beta}$. A larger $\\bar{\\beta}$ will result in less degradation on the satisfactory responses but also less improvement on the unsatisfactory responses. We conduct an ablation on this coefficient $\\bar{\\beta}$ where we use opt-1.3B model, full-hh-rlhf dataset, and ReMax base algorithm. We vary the value of $\\bar{\\beta}$: 0, 0.05, 0.1, 0.5, 1.0, and report GPT win rates over ReMax on both the unsatisfactory set $D_u$ and satisfactory set $D\\setminus D_u$.\n\n|                 | 0    | 0.05 | 0.1  | 0.5  | 1.0  |\n|-----------------|------|------|------|------|------|\n| $D_u$           | 82.8 | 73.2 | 62.6 | 55.0 | 51.2 |\n| $D \\\\setminus D_u$ | 30.2 | 47.5 | 48.9 | 50.8 | 50.2 |\n\nThe above results show that a larger $\\bar{\\beta}$ results in less degradation on the satisfactory responses but also less improvement on the unsatisfactory responses.\n\n**Answer to Weakness 4**: We evaluate our models on ApacaEval, specifically using the \"tatsu-lab/alpaca\\_eval\" dataset. For each prompt (i.e., \"instruction\") in this dataset, we use our models to generate responses. Recall that we have four experiments: opt-1.3B+full-hh-rlhf, pythia-2.8B+full-hh-rlhf, pythia-2.8B+TL;DR, llama-2-7B+TL;DR, and we consider two base RLHF algorithms: PPO and ReMax. We report the GPT-4 win rates of RLHF (PPO and ReMax) and our method (XRLHF) over the SFT model:\n\n|                             | PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|-----------------------------|------|-----------|--------|-------------|\n| opt-1.3B+full-hh-rlhf       | 58.8 | 61.5      | 59.1   | 63.7        |\n| pythia-2.8B+full-hh-rlhf    | 62.1 | 66.2      | 60.6   | 68.7        |\n| pythia-2.8B+TL;DR           | 61.8 | 63.9      | 62.4   | 66.6        |\n| llama-2-7B+TL;DR            | 60.2 | 64.5      | 62.1   | 66.2        |\n\nThe above results demonstrate that our method (XRLHF) can also improve RLHF on AlpacaEval.\n\n**Answer to Weakness 5**: In our case, we identify a set of training data to explain the unsatisfactory samples. Here, we provide analysis on the chosen and rejected responses of the identified training data. In particular, we use Beaver-7b-v3.0-reward for the full-hh-rlhf dataset and use Deberta-v3-large-v2 for the TL;DR dataset. We provide the average reward of chosen and rejected responses of identified training data in our four experiments (opt+full-hh-rlhf, pythia+full-hh-rlhf, pythia+TL;DR, llama+TL;DR) where the base RL algorithm is ReMax.\n\n|           | opt+full-hh-rlhf | pythia+full-hh-rlhf | pythia+TL;DR | llama+TL;DR |\n|-----------|------------------|----------------------|---------------|--------------|\n| chosen    | 2.05             | 1.85                 | 3.44          | 3.12         |\n| rejected  | 1.89             | 1.79                 | 3.58          | 3.64         |\n\nThe above results show that the average reward of rejected responses is very close to chosen responses for identified training data in the full-hh-rlhf dataset, and is even larger than the chosen responses for identified data in the TL;DR dataset. To further inspect the reward distribution of the chosen and rejected responses of the identified training data, we provide the percentage of the identified training data where the rejected response has a higher reward than the chosen response:\n\n| opt+full-hh-rlhf | pythia+full-hh-rlhf | pythia+TL;DR | llama+TL;DR |\n|------------------|----------------------|---------------|--------------|\n| 37%              | 44%                  | 62%           | 58%          |\n\nThe above results indicate that the identified training data include a large amount of misleading signal where the rejected response is actually better than the chosen response.\n\n**Answer to Question 1**: Please refer to our answer to Weakness 3.\n\n**Answer to Question 2**: Please refer to our answer to Weakness 4 where we evaluate our models on AlpacaEval. We conduct an additional experiment using Llama-3.1-8B model and full-hh-rlhf dataset for training, and we provide GPT-4 win rates over the SFT model on the alpaca\\_eval dataset.\n\n| PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|------|-----------|--------|-------------|\n| 62.1 | 65.2      | 63.9   | 67.7        |\n\nThe above results demonstrate that XRLHF can improve RLHF using Llama-3.1-8B base model.\n\n**Answer to Question 3**: Please refer to our answer to Weakness 5.\n\n**Answer to Question 4**: Our method includes two parts: explanation and improvement. Algorithm 1 (XRLHF) only includes the explanation and thus do not include the improvement part (i.e., unlearning and finetuning). We will add an overall algorithm that includes both parts."}, {"review_id": "bAVmJl66WO", "reviewer": "Reviewer_66Se", "summary": "This paper introduces a method to further improve the effectiveness of RLHF methods after they are finished. It focuses on improving the performance on the part of the validation dataset where the model did not perform well. This is done by first identifying the training data (in the binary preference dataset) which are most similar to each problematic prompt-response pair, then performing some unlearning in the reward model and some fine-tuning for the policy model. Experiments show overall better win rates of the proposed approach over existing RLHF approaches (PPO, REMAX) separately on two datasets. Performance tradeoffs are observed on the non-problematic part of the validation dataset.", "questions": "1. It is claimed that the method finds the causes of why the policy model generated unsatisfactory responses. What is your definition of a cause here? There are the research fields on explainable AI and causality / actual causes. How does your definition fit in? \n2. Why does identifying relevant examples in the preference dataset, which was only used to train the reward models and not directly the policy model, counts as explaining the unsatisfactory generations?\n3. Regarding the approach for separating the evaluation dataset into satisfactory and unsatisfactory subsets – how valid is setting a threshold using a reward model for this purpose? In line 608 you mentioned “the score lower bound of the satisfactory group is usually above the score upper bound of the unsatisfactory group” which does not seem clear. My intuitions are that it is difficult to compare the scalar outputs of the reward models for responses across different input prompts, because the prompts themselves will influence the magnitude of the scores?\n4. How's the reward models' performance after unlearning?", "limitations": "All parts about explainability should be revisited with clear discussions on motivations and assumptions. The evaluations are not comprehensive enough to support the relevant contributions made in the paper. See detailed comments on weaknesses and questions. I think those are essential aspects to elaborate on in the paper's main text.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-presented with clear problem definitions and good visualisations. The mathematical formulations are also very easy to follow.\n2. The methodology sections are sound and well-explained.\n\nWeaknesses:\n1. Clarity and Significance: The paper claims that the proposed approach is explainable. However, discussions on explainability are very shallow. The most relevant part might be lines 138-144, but these motivations hint that the method is aiming to explain why the reward model assigns a high reward to the policy’s response, rather than to explain why the policy LLM generates such a response. There are also no real evaluations for explainability, as all evaluations are regarding win rates. Therefore, using the word explainable to describe this paper is not appropriate.\n\n2. Clarity and Significance: Motivations and assumptions are lacking for the method. As mentioned in the paper, LLM post-training involves SFT, learning a reward model, and running RL. However, the proposed approach deems that the cause of unsatisfactory LLM behaviour lies in the preference training data for the reward model. This dataset is usually not directly seen by the LLM itself, so it is questionable to ignore the other data sources where the reasons why the LLM unsatisfactory behaviour might exist. This is a fundamental aspect to consider, but it is not discussed in the paper. (While in the experiments the three steps use the same dataset, lacking such discussions will also harm the generalisability, hence the significance of the proposed approach.)\n\n3. Quality: Writing (apart from the mathematical formulations, those are good) is often too simplistic or unconvincing. For example, in abstract and intro, discussions on *cause* are very vague; in Line 25, one should not expect a model to become all perfect after some training; in Preliminary section, high-level descriptions for the equations behind RM and RL would be helpful. Conclusions and related works are very incomplete; future works and limitations are not well-discussed.\n\n4. Quality and Significance: Evaluations are thin and a bit too narrow. \n(1) Only 2 datasets and 3 small models are involved. \n(2) The choices of the open-source reward models used to evaluate LLM responses are not well-justified – neither is state of the art, especially the deberta model. It’d be good to refer to the rewardbench leaderboard to find reasonably sized and good ones. While GPT-4 is also used, it is suggested that the order in which the candidate responses are presented be randomised – currently in the code, it’s always showing the response by the proposed method first and then the baseline response (if I’m not mistaken). \n(3) The scope of evaluation is narrow, as only the win rate metric is used on only one dataset in each experiment. It is not clear whether the proposed RLHF approach worsens the LLMs’ general performance, e.g., on other benchmarks, especially as some unlearning is involved.", "was_revised": true, "final_justification": "I increased my score from 2 to 4 following the rebuttal which addressed my concerns about the clarity of being \"explainable\" and about the significance of results given the reward model performance after unlearning. That said, the inclusion of these additional discussions and experiments to the main text might not be trivial.", "rebuttal": "Thanks for your constructive reviews. We address your comments in a point-to-point manner.\n\n**Answer to Weakness 1**: Thanks for mentioning the explanation. Please note that the primary focus of this paper is to improve LM performance, and the explanation module is used as a tool to achieve this goal. Therefore, we evaluate the effectiveness of our method via downstream win rates.  \nMoreover, we provide a mathematically grounded explanation module and evaluate the explanation in subsection 6.3. First, our explanation follows the definition of post-hoc example-based explanation in [D1,D2] where a model’s decision is explained by identifying relevant examples from training data. In our setting, we find a set of training data to explain why a test-time response $\\bar{y}$ is generated by explaining why $(\\bar{x},\\bar{y})$ has a high reward. This reformulation is mathematically grounded: the identified examples contribute to the high value of $r_\\theta(\\bar{x},\\bar{y})$, and since RL optimizes the LM to maximize reward, a high reward $r_\\theta(\\bar{x},\\bar{y})$ increases the likelihood that $\\bar{y}$ is generated. Therefore, the identified examples (indirectly) contribute to the generation of $\\bar{y}$. Second, we explicitly evaluate the fidelity (a widely used metric in XRL) of the explanation in subsection 6.3. In the case of explaining why the model generates unsatisfactory responses, fidelity in XRL [D3,D4] is defined as performance improvement from acting on the explanation. In our case, it means whether the unlearned model generates better responses. We measure this improvement using win rates of the unlearned model over the original RLHF model. Since the win rate is larger than 50\\%, it means that the unlearned model generates better responses. We also provide qualitative explanation examples in Appendix D.2, which is human-interpretable.  \nFollowing your suggestion, we add a qualitative evaluation based on human plausibility, which is a widely-used metric in XRL [D3,D4] that directly evaluates the explanation. We recruited 15 participants, and for each test-time response, provided them with our explanation and a set of randomly selected training examples of the same size. Participants were asked to choose which set better helps them understand why the model generates the response. The percentages favoring our explanation in the four tasks (opt+full-hh-rlhf, pythia+full-hh-rlhf, pythia+tldr, llama+tldr) are: 87%, 93%, 80%, 80%.\n\n**Answer to Weakness 2**: We would like to clarify that the goal of the explanation module is to find \"a cause\" (lines 2 and 40) instead of the sole cause of unsatisfactory responses. We agree that other sources, including SFT and RL, can also contribute to unsatisfactory responses. However, it is infeasible to enumerate all such causes, and this paper focuses on the reward data for explanation. In our experiments, the three stages use disjoint partitions of the dataset (line 278). Thus, the LM does not directly observe the data used to train the reward model. However, focusing on the reward data remains well-justified: although the LM does not directly access this data, this data is used to train the reward model, and RL optimizes the LM to maximize the reward. Therefore, the reward data indirectly but causally influences the LM’s responses. We will include this discussion to clarify this causal pathway and acknowledge other possible sources of causes.\n\n**Answer to Weakness 3**: Following your suggestion, we will clarify the causal reasoning path in our setting: reward training data influences the reward model, which determines reward values for LM outputs, which in turn shapes the LM policy via RL optimization. Therefore, examples that contribute to a high reward for a response indirectly but causally increase the likelihood that the LM generates that response. We will rephrase line 25 to emphasize \"many responses are unsatisfactory after RLHF\". We will add brief description on RM (which learns a reward signal to approximate human preference) and RL (which learns an LM to optimize reward). We have additional complete related works in Appendix A. We will include the limitation that this paper does not address other sources of causes, and one future work is to find other sources of causes.\n\n**Answer to Weakness 4**: Following your suggestion, we add the following results:\n\n1. We conduct an additional experiment using llama-2-13b model and imdb dataset. We report the win rates over the SFT model. The win rates are evaluated by Skywork-Reward-V2-Llama-3.1-8B (ranked top in RewardBench 2) and GPT-4 with the order of the candidate responses randomized:\n\n|                                | PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|--------------|------|-----------|--------|--------------|\n| Skywork-Reward-V2-Llama-3.1-8B | 66.5 | 71.4      | 67.1   | 70.8         |\n| GPT-4                          | 69.6 | 72.5      | 70.7   | 73.2         |\n\nThe above results demonstrate that our method (XRLHF) can further improve the base RLHF algorithms (PPO and ReMax) on a large base model (llama-2-13b) and a new dataset (imdb).\n\n2. We evalute the models on new and unseen datasets. Recall that the experiment section has two tasks: dialogue and summarization. The dialogue task includes opt-1.3B and pythia-2.8B models trained on the full-hh-rlhf dataset. The summarization task includes pythia-2.8B and llama-2-7B models trained on the Reddit TL;DR dataset. Here we evaluate the models trained in the dialogue task on a new dataset PKU-SafeRLHF, and evaluate models trained in the summarization task on a new dataset webis/tldr-17.  \n(1) opt dialogue task\n\n|                                | PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|-----|------|-----------|--------|--------------|\n| Skywork-Reward-V2-Llama-3.1-8B | 64.2 | 68.0      | 66.4   | 71.4         |\n| GPT-4                          | 66.3 | 68.5      | 65.1   | 69.7         |\n\n(2) pythia dialogue task\n\n|                                | PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|-|-|-----------|--------|--------------|\n| Skywork-Reward-V2-Llama-3.1-8B | 62.5 | 64.1      | 60.7   | 64.6         |\n| GPT-4                          | 66.1 | 66.8      | 62.4   | 67.1         |\n\n(3) pythia summarization task\n\n|                                | PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|--------------------------------|------|-----------|--------|--------------|\n| Skywork-Reward-V2-Llama-3.1-8B | 60.6 | 67.2      | 62.2   | 62.4         |\n| GPT-4                          | 59.8 | 64.4      | 67.9   | 69.3         |\n\n(4) llama summarization task\n\n|                                | PPO  | PPO+XRLHF | ReMax | ReMax+XRLHF |\n|--------------------------------|------|-----------|--------|--------------|\n| Skywork-Reward-V2-Llama-3.1-8B | 65.1 | 67.8      | 67.3   | 65.1         |\n| GPT-4                          | 62.7 | 61.5      | 66.8   | 67.9         |\n\nThe above results demonstrate that our method can also improve RLHF performance on new and unseen datasets.\n\n**Answer to Question 1**: In our setting, we define a cause as a set of reward training data that indirectly contributes to the LM’s generation of $\\bar{y}$ by contributing to the high value of $r_\\theta(\\bar{x},\\bar{y})$. As discussed in our answer to Weakness 1, this causal path is mathematically grounded. Moreover, our explanation follows the definition of post-hoc example-based explanation [D1,D2] where a model's decision is explained by influential training examples. In our case, we identify the training examples to explain why the reward model assigns a high value to $(\\bar{x},\\bar{y})$, and this further explains why the LM generates $\\bar{y}$ through the causal path.\n\n**Answer to Question 2**: Please refer to our answers to Weaknesses 1 and 2.\n\n**Answer to Question 3**: We agree that different prompts can affect the magnitude of the raw reward output by the reward model. Therefore, we compare calibrated rewards, rather than raw rewards, to the reward threshold. Specifically, the calibrated reward of a response is its raw reward subtracting the raw reward of the SFT model response to the same prompt. This calibration reduces prompt-specific bias in reward values. We then apply a fixed threshold to these calibrated rewards to classify satisfactory and unsatisfactory responses. We will clarify this calibration to avoid confusion.\n\n**Answer to Question 4**: We evaluate the reward model’s performance before and after unlearning by measuring its agreement with GPT-4 preferences. Specifically, we randomly sample 500 examples from the test set, where each example consists of a prompt and two responses. For each pair, we compare the reward model's ranking with GPT-4’s judgment and compute the percentage of cases where the reward model assigns a higher score to the response preferred by GPT-4.\n\n|                     | opt+full-hh-rlhf | pythia+full-hh-rlhf | pythia+tldr | llama7b+tldr | llama13b+imdb |\n|---------------------|------------------|----------------------|-------------|---------------|----------------|\n| Before unlearning   | 69.8%            | 72.5%                | 64.4%       | 68.5%         | 77.6%          |\n| After unlearning    | 71.6%            | 74.3%                | 64.1%       | 70.4%         | 80.2%          |\n\nThe above results  show that unlearned reward has better performance than the original reward.\n\n**Answer to Limitation**: Following your suggestion, we have addressed all your comments in our answers to the weaknesses and questions.\n\n[D1] J.Crabbé et al, \"Explaining latent representations with a corpus of examples\", NeurIPS, 2021.\n\n[D2] H Sun et al, \"Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples\", NeurIPS, 2024\n\n[D3] W Guo et al, \"Edge: Explaining deep reinforcement learning policies\", NeurIPS, 2021.\n\n[D4] Z Cheng et al, \"Rice: Breaking through the training bottlenecks of reinforcement learning with explanation\", ICML, 2024."}]

Author rebuttal:

Meta-review: This a post-RLHF “explain-and-correct” method that identifies preference training data responsible for unsatisfactory model outputs and then performs targeted unlearning to improve alignment. The idea is novel: the method formulates explanation as finding convex decompositions over reward data, and correction as inverse-gradient unlearning of these problematic preferences, followed by constrained fine-tuning. The authors argue that this approach is more data-efficient than retraining and can be applied in a lightweight manner to improve existing RLHF policies.

Strengths of the paper by the reviewers;
- Novelty: The paper introduces a framework to link unsatisfactory responses back to preference training data, then uses this link to unlearn harmful signals. This is an original contribution and resonates with broader themes in explainability and post-training alignment.
- Practical value: As a post-training step, XRLHF is attractive for deployed or expensive-to-train models, making it potentially impactful in applied settings.
- New experiments in rebuttal: The authors provided substantial additional experiments, including ablations (e.g., proportion of data unlearned, KL trade-offs), larger models (LLaMA-3.1-8B, LLaMA-2-13B), generalization benchmarks (AlpacaEval, SafeRLHF, Webis TL;DR-17, IMDb), and reward model performance before and after unlearning. These additions convincingly strengthen the paper.

Some weaknesses were raised by the reviewers:
- Model scale: Some reviewers noted reliance on smaller models in the main paper; while newer results with LLaMA-3.1-8B and LLaMA-2-13B mitigate this, broader evaluation on strong modern models would further increase confidence.
- Explainability framing: Multiple reviewers flagged that “explainable RLHF” may overstate the level of interpretability. The authors clarified that explanation is used as a tool for improvement, provided causal reasoning pathways, fidelity/human plausibility evaluations, and plan to revise the title/abstract.

The reviewers agree toward acceptance after rebuttal. Concerns about baselines, scalability, and clarity of the “explainability” claim have been meaningfully addressed with new experiments and clarifications. The technical contribution is sound, original, and practically relevant, and the paper is also well-written.
Decision: Accept (poster)