CPPO: Continual Learning for Reinforcement Learning with Human Feedback
Authors: Han Zhang
Yu Lei
Lin Gui
Min Yang
Yulan He
Hui Wang
Ruifeng Xu
Venue: ICLR 2024 poster (2024)
Keywords: lifelong learning
reinforcement learning
human feedback
proximal policy optimization
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a new method to continually learn from human preferences based on proximal policy optimization, in which sample-wise weights are introduced to adjust policy learning and knowledge retention.

Abstract: The approach of Reinforcement Learning from Human Feedback (RLHF) is widely used for enhancing pre-trained Language Models (LM), enabling them to better align with human preferences. Existing RLHF-based LMs however require complete retraining whenever new queries or feedback are introduced, as human preferences may differ across different domains or topics. LM retraining is often impracticable in most real-world scenarios, due to the substantial time and computational costs involved, as well as data privacy concerns. To address this limitation, we propose Continual Proximal Policy Optimization (CPPO), a novel method that is able to continually align LM with dynamic human preferences. Specifically, CPPO adopts a weighting strategy to decide which samples should be utilized for enhancing policy learning and which should be used for solidifying past experiences. This seeks a good trade-off between policy learning and knowledge retention. Our experimental results show that CPPO outperforms strong Continuous learning (CL) baselines when it comes to consistently aligning with human preferences. Furthermore, compared to PPO, CPPO offers more efficient and stable learning in non-continual scenarios.

Published as a conference paper at ICLR 2024
CPPO: CONTINUAL LEARNING FOR REINFORCEMENT LEARNING WITH HUMAN FEEDBACK
Han Zhang 12, Yu Lei , Lin Gui Min Yang', Yulan He*, Hui Wang?, Ruifeng Xu
1 Harbin Institute of Technology (Shenzhen)
2 Peng Cheng Laboratory
3 King's College London
4 Shenzhen Institutes of Advanced Technology
5 Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies
ABSTRACT
The approach of Reinforcement Learning from Human Feedback (RLHF) widely used for enhancing pre-trained Language Models (LM), enabling them to bet- ter align with human preferences. Existing RLHF-based LMs however require complete retraining whenever new queries feedback are introduced, as human preferences may differ across different domains or topics. LM retraining of- ten impracticable in most real- world scenarios, due to the substantial time and computational costs involved, as well as data privacy concerns. To address this limitation, we propose Continual Proximal Policy Optimization (CPPO), novel method that is able to continually align LM with dynamic human preferences. Specifically, CPPO adopts weighting strategy to decide which samples should be utilized for enhancing policy learning and which should be used for solidifying past experiences. This seeks good trade-off between policy learning and knowledge retention. Our experimental results show that CPPO outperforms strong Contin- uous learning (CL) baselines when it comes consistently aligning with human preferences. Furthermore, compared to PPO, CPPO offers more efficient and stable learning non-continual scenarios.
INTRODUCTION
Recent studies (Sticnnon et al., 2020 Bai et al., 2022a Ouyang et al.) 2022) have shown that Reinforcement Learning from Human Feedback (RLHF) can significantly enhance language models by aligning them with human intention. RLHF uses human preferences as a reward signal to fine-tune language models with the Proximal Policy Optimization (PPO) algorithm. The RLHF-basse model can effectively generate answers preferred by humans for tasks that lack standardized solutions, such summarization(Stiennn
al. 2020), translation Kreutzer Kreuuzer et al. 2018), and dialogue(Jaques et al. 2020), without over-optimizing metrics such as ROUGE(Lin 2004 or BLEU((Pping et al. (2022).
In real-world applications, learning continuously changing human preferences is more practical than learning invariable human preferences. For example. the progression from the onset of the COVID-19 virus in human society to widespread infections and finally to achieving herd immunity has seen corresponding changes in government policies and human perspective. An Al agent that keeps pace with the times should exhibit behavior that aligns with current government policies and human understanding preferences at different stages, rather than remaining static.
However, traditional alignment methods((Siennon et al. 2020 Ouyang et al. 2022), lack flexibility for continual learning (CL) of human preferences. Recent approach (Bai et al. [022a] tackles these problems by periodically retraining the Preference Model (PM) and policy based on both new and historical data, it might be inefficient and impractical due to the involved concerns of computational cost and data privacy.
In this paper, propose (CL), aiming to optimize two conflicting objectives: preserving old knowledge and acquiring new
practical approach by enhancing RLHF with continual learning
Corresponding authors: Yu Lei (leiy01 @ pcl.ac.en) and Ruifeng Xu (xuruifeng @ edu..ed...n)...

---

Published as a conference paper at ICLR 2024
knowledge (Rolnick al. (2119). This leads to long-standing challenge known as the stability- plasticiry dilenma (Abraham & Robins 2005). Moreover, due to the vast action space (vocabulary) of LMs, the RLHF algorithms (c.g.. PPO) usually suffer from the issues of inefficiency and instability during training (Ramamurthy et al. 2022). To tackle these challenges, we attempt to seek a good tradeoff between policy learning and knowledge retention with stable learning by designing sample- wise weighting strategy over the rollou samples. Our weighting strategy is motivated by the fact that a desired policy should always generate high-reward results with high probabilities.
Specifically, we first categorize the rollout samples into five types according to their rewards and generation probabilities, as shown in Figure We then assign each rollout sample with a policy learning weight a and a knowledge retention weight 8, in the following way. 1) For a high- performance sample, we assign a high and a high (3, order to consolidate the knowledge of this sample. 2) For high-variance or overfitting sample, we assign high a and low 8, so as to learn more knowledge of this sample and force the new policy to be different from the old one in generating such a sample. 3) For noisy sample, we assign low and low 3 to decrease its impact on learning. 4) For a normal sample, we make no changes.
Based on the above weighting strategy, we develop novel PPO-based method, named continual proximal policy optimization (CPPO). CPPO implements the weighting strategy in two different ways: heuristic and learnable, resulting in two different CPPO methods (see Section 3.2 for details). The heuristic approach sets the weight with linear gain decay according to strategy. The learnable approach converts the strategy into several inequality constraints and learns the best weight by optimizing the Lagrange function.
Reward
High Variance
High Performance
Normal
Noisy
Overtitiing
Experimental results on real-world summarization datasets demonstrate that our proposed CPPO methods significantly outperform the PPÒ re-training methods and the strong CL baselines, in both CL and non-CL settings (detailed in Appendix FT. Furthermore, addi- tional experiments in both settings verify the superior training stability of CPPO compared to the original PPO algorithm.
Generation probability
Figure Five types of the rollout are uti- lized in our method.
2 PRELIMINARY AND TASK FORMULATION
PPO algorithm (Schulman et al. 2017 utilizes the clipped surrogate objective with a learned state- value function, and the entropy bonus (Mnih et al. (2066) is added to the original reward. The total objective is approximately maximized in each iteration step i 1,2, ..., I (in the NLP scene, step-i denotes the generation of the i-th token):
L LCLIPVVF (0) E.LL c- F
(1)
where e is the coefficient, and L is squared-error loss (Vo(8;) Vi The clipped policy Ta(a; ss)
learning objective is: LÇLIS min(r; A,, clip(r,(0), 1 -A,), where (0) Re [s.] is the probability ratio, e is the clip hyperparameter, s, is the i-th state, A₁ is the truncated version of generalized advantage estimation.
Task Formulation: In this paper, we propose the task of continually learning human preferences under offline continual learning setting BBessillka et al.) 2020). Formally. we consider task sequence of = (Ti .2....> to continually learn a policy model on the corresponding human
In this context, stability refers to the retention of previously acquired knowledge, which is different from the training stability mentioned later. Plasticity, the other hand, refers the ability to adapt knowledge through policy learning.
"In the context of RLHF, a rollout, also known as a trajectory or episode, entails generating response sequence, such as summary, to given conversation prompt, starting from particular state (i.e. the initial prompt). The responses generated during the rollout then used to update the policy network.
2

---

Published as a conference paper at ICLR 2024
preference datasets HIF (HF1, HH,,, and prompt datasets S {Si, S...... For each task Ti(t = 1.,...)). the policy Ty is initialized by T--1 and then is trained against the reward model r where the reward model is learned on F The initial policy xo is the SFT model, namely, To = #SFT- Let æ = (s, a) denote the prompt & and answer a pair. The final objective is to learn a policy model my that maximizes the overall reward on all of the learned human preferences:
max ΣT Saaaa [ra(s,a)
3 CONTINUAL PROXIMAL POLICY OPTIMIZATION
3.1 MOTIVATION AND THEORETICAL ANALYSIS
To optimize the objective 2 in the CL paradigm, the key is to balance the tradeoff between policy learning and knowledge retention, i.e., to learn policy क, that not only fits current task / but also retains the knowledge of previous tasks. This is typically accomplished by maximizing T average reward and meanwhile minimizing the difference between ny and πT-1 by KL-based knowledge distillation (aallanis et al. 2019):
max Eg Spamoo 8) [Te(s, a)] - E.ES,,, (P= || (als) 0
where Pa., (als) denotes the probability that policy Ta generates the answer a to the prompt s. However, in the RLHF setting, we argue that a more effective way achieve policy learning is to maximize the rewards of the results that To has a high probability to generate. This is because LMs usually have a vast action space (vocabulary size) and adopt a sampling strategy such as beam search that favors high-prou ggg-probability generative results. For knowledge retention, on the other hand, it is more important to make Ty retain TT-13 's certain knowledge that generates high-reward outputs rather than all.
To accomplish the above ideas, we propose a theoretically desirable objective for continual RLHF at task T;:
max (s.)En (s, a) - DxL (a|s) || PP._((ass)
where, DJ = (() S, no (-1s), Pa. (als) > na (a]s)] + Lon[Pr (a|s)]} and D2 {(s, a)(s S ~ (=)1(- a) Halre(s, a)] + ko (r, }}l} denote the sets of samples with high generation probability and high rewards, respectively. μ and denote the mean and standard deviation respectively, and k is a hyperparameter. It is important to note that here we use (s(8,a) instead of (8,1(8,a). Since the reward model is continually learneded, we assume (t-1(8, a) R re(8, a) when 8 E S and a ~ (-(ss)). To simplify notation, the subsequent sections of the paper use instead of a).
The KL divergence term requires a significant amount of memory to store the probability distribution of each token across the vast vocabulary. To tackle this problem, we incorporate low computational knowledge retention penalty term (0) = (log Pro (ax) log (x,_)(x4)²² We compute the L2 distance of the log generation probability of true tokens instead of the KL divergence of the entire vocabulary's 's probability distribution. We find the former is effective for knowledge retention and
needs NOT to save the vocabulary's 's probability distribution in the memory
We introduce TD, (z) and (x) to denote the indicator functions of the sets of D2 and D2, respec- tively. By introducing the actor-critic version, the clipped ratio, and the entropy bonus, we claim that (4) can be written to (the derivation is detailed in Appendix Section B
J' (0) = - CLIP+ID,KK-VP. (0) = [E[I,, CLIP
(5) Compared with objective Eq. (1), (5) introduces the learning weights LD, (x), I (z), and the LKK loss. Unfortunately, is still impractical directly optimize the objective, since the training
(0) L
(o) - KR -V- LVF
'In our task, the reference model generates 512 summaries (max 50 tokens) in one rollout, The vocabulary size is nearly 5e+4. If we use FP16 to save the logits or proability tensor, it takes about 502 Bit/le9- 1. 1.28GB of memory. However, computing I. only needs to save the probability of true tokens, which takes only 512*50#2 Bit/le9 2.56F-05GB of memory.
3

---

Published as a conference paper at ICLR 2024
samples in D1 and D2 seldom as indicated by the Cantelli Inequation P(X
μ| + ko]
1/(1+ + k²). To make Eq.(5) easy to optimize, we generalize the indicator functions ID, (x) and In, to positive real-valued functions (zz) and ((z), which gives each sample non-zero learning weight.
3.2 WEIGHITNGG STRATEGY
Our method utilizes sample-wis wippe-wiseee balance weights a(x) and (1x) to regulate the policy learning and knowledge retention processes, aiming to find balance between knowledge retention and policy learning. The final objective is:
(0) Li -.-CIIBBKRVE (0)
(6)
E. (0)
[
for task 1 2,,,..., T. Next, propose weighting strategy for balancing policy learning and knowledge retention.
3.2.1 BALANCING POLICY LEARNING AND KNOWLEDGE RETENTION
To simplify the expression, we define the operator Fl.] = nn[1] - [ ] and operator G|-] = |u[-] + ko[-) [-]- As shown Figure I and Table I we classify the rollout samples into rollout types based on the joint distribution of (Pr,. no (z), R(x)) If me (z) R(x) is outside the discriminant interval (F[-), GI-I), it considered as high or low. Now, we detail each rollout type and corresponding weight strategy.
Righ--eeffrrmacc sample: If both no (x) and R(x) are high, it indicates that the old policy has high confidence to generate æ which gets high reward, implying that is al- ready performing well. In this case, we ask the new policy to enhance both policy learning and knowledge retention.
Table 1: The determining condition of rollout type and corresponding weight strategy to balance policy learning and knowledge retention. We monitor the generating probability РT. (a)) and the corresponding reward score R(x). The rollout type of sample depends on whether
the Pad (z) and R(x) fall in or outside the discriminant interval (FI,GG))
Rollout Type
Determining Condition
Weight Strategy
P1
High-performance
P P., (e) G s.
R(x) G
a(z) f
B(x)
Overfitiing
P (.. 2 G
R(z) < FR
a(zi f
B(a)
73
High-variance
P (z, S FP.,
R(z) GR
a(z) f
B(x)
Overfitting sample: A high
P4
Noisy
P., F(.
R(z) FR
a(z) J
B(x) 4
75
Normal
P (z) or R(x) c (F.G)
Px, (x) with a low R(x) in- dicates that the old policy is likely overfitting (due to high probability) the biased sample (due to low reward score). We aim to reduce the generation proba- bility of the biased sample x, which can be achieved through policy learning. However, knowledge retention will maintain the high probability of the biased sample x. Therefore, we enhance policy learning and slow down knowledge retention.
High-variance sample: If P (xr) is low while R(x)) is high, suggests that sample has high variance. Due to the low P (I), the likelihood of generating x next time is low. To achieve stable (low variance) performance, we aim increase the generation probability of sample æ. which can be accomplished through policy learning. However, knowledge retention will maintain a low generation probability. Therefore, we enhance policy learning and slow down knowledge retention.
Noisy sample: If both P PT. (x) and R(x) are low, sample x is considered noisy data which may lead to overoptimization against the PM (Gao et al., 2022). Therefore, we slow down both knowledge retention and policy learning.
Normal sample: If least one of PTo (x) and R(x)) falls within the discriminant interval, we consider it normal condition and do not alter the learning process.
'Cantelli's 's inequality (also called the Chebyshev- Canttell inequality and the one-sided Chebyshev inequality) version of Chebyshev's inequality for one-sided tail bounds.
4

---

Published as a conference paper at ICLR 2024
Variance
High Performance
Variance
9023
Hig Perfon
Noisy
Noisy
Ove ritting
Px(
Ri-)
Overfiting
Rix)
(a) Surface of heuristic a(x)
(b) Surface of heuristic 2(x)
Figure 2: The surfaces of heuristic weights. The weights equal to 1 when rollout samples fall in the normal zone.
3.2.2 HOw DETERMINE BALANCE WEIGHTS?
The above weight strategies constitute several inequality constraints (x()) and ((x), shown Table Determining balance weights requires finding feasible solution that satisfies those constraints. provide two methods to determine balance weights including the heuristic weight method and the learnable weight method.
Table 2: The constraint of weights and heuristic weights.
ID
Constraint of o(a))
Constraint of (8())
Heuristic a(x)
Heuristic ((ar)
a(er,) a(x,,) <
((x,,) ((x,))
min (ub,
Pr, (z) (e)-u(Pre, p(Pre.
min(ub, Ríx) [R]]
a(xr,) a(xr,) <
A(z,,) (((r)) min(ub,
P., (c) (a)-el No
max(10, 2 Ris)-
a(rr,) a(xr,) <
A(xr.) A(rr,)) min(ub, Pt, (2) "MPPte))
max(I), 2
As
a(tr,) a(x,)
8(xr.) B(xr,)) < max((1b2 2 Pao (M/al)
max(Ib, R(z)
kn|
Ts
Byneen [a(x))
Baner (pxx)
Heuristic a(x) and ((x): If Pf, (x) or R(x) fall within the discriminant interval, the balance weights are set to 1. If they are further away from the discriminant interval, the weights will linearly increase or decrease (depending on the rollout type). We can plot the surfaces of ((x) and B(x) 3D coordinate systems, shown in Figure 2 The heuristic weights a<xx) and ((x)) for given sample can be calculated by the formula presented Table
Learnable (x) and ((x): Heuristic (xx) and (xx) lack enough adaptation ability the dynamic learning process. Hence, we propose the learnable balance weights to automatically balance policy learning and knowledge retention. We learn 2N parameters for each rollout batch in which the LM generates responses, the 2N parameters can be discarded before the next rollout batch.
Our goal is find set of weights that satisfy the constraints in Table 2 Unlike the typical optimization problem solved by the Lagrange Multiplier method, we do not need to minimize an additional objective function. should be noted that the optimization objective of CPPO in Eq 6 is not directly optimized using the Lagrange Multiplier method.
We employ more straightforward strategy. We construct unconstrained optimization objective by adding all the terms on the left side of the inequalities (in Table together:
Leoef (x) Brote- [()p(x) 1) + (z)(z) - 1) + (a(xx.) a(Tr,) + ((zr,) ((Tr)) a((r)) - a(x,,) ((x,,) ) - 8(x,g) + ( - (2xr,) + 3(xr)) - 3(xr))
(7)
+ *((rr)) a(xr,)))))))))))))))))))))))(a + ((x,)))) -((rr,))
where, axx) = (ub- - lb) sig(d!) + lb. 3(x) = (ub lb) ( lb, and sig is sigmoid function, lb and wh are lower and upper bound of (<x) and ((x)). We directly optimize Eq. using SGD find
5

---

Published as a conference paper at ICLR 2024
set of weights that satisfy the constraints. We set multiplier hyperparameter, and 7 0.1 is selected from {0.01. 0.1, 0.5, 1.0). For more hyperparameter sensitivity analysis experiments, please refer to Appendix [E.I] We found this simple idea is highly effective in our scenario. In Appendix E.. we analyze the time and memory required for SGD to find feasible solutions and found that it does NOT significantly increase the overall training time and memory.
EXPERIMENTS
We assess the performance of CPPO and baseline methods in the domain incremental learning (DIL) summary task. We also evaluate CPPO on non-continual learning tasks (Appendix Section
4.1 THE EXPERIMENTAL CONFIGURATION FOR CONTINUAL LEARNING FROM HUMAN PREFERENCES
Dataset and split: In accordance with previous research (Stiennon et al., 2020). we eval- uate our method using the Reddit TL:DR (Völske et al., 2017) dataset for summariza- tion. We use the human preference data provided by CarperA To the best of our knowledge, there are limited benchmark datasets proposed for evaluating continual RLHF methods. Consequently, we divide the Reddit TL:DR dataset based on domains into two parts, which are outlined in Table Each part corresponds
distinct alignment task.
Experiment settings: We evaluate CPPO un- der the DIL setting with two tasks, and the historical data is assumed inaccessible. This scenario is typical in real-world applica- tions, such as develop- ers continually learn- task-2
Table 3: The dataset is utilized for continual learning. The human feedback data used for training the reward model. The post (prompt) and summary (label) of Reddit TL;DR are used for SFT. The domain of "r others" includes 28 categories, such as books, travel, and cooking. It's worth noting that the summary (label) data is not used in the reinforcement learning (RL) process.
Task ID
Data
Data split
Train
Valid
Test
Domain
Human Feedback
part-1
52243
45148
r/ relationsshis
task-1
Reddit TL;DR
part-1
63324
3462
3539
r/ relationships
Human Feedback
part-2
40291
38481
r/ others
ing an open-Source RLHF model like vicuma(Chiann et al. 2023) in special domain (e.g., game) without permis-
Reddit TL;DR
part-2
53398
2985
3014
r/ others
sion to access the pre-training corpus. For each task, we employ a 1.3B gpt2-xl (Radford et al.. 2019) model with value head the reward model (RM). The RM is continually trained for 5 epochs on each task using the MAS(Aljundi et al. 2018) method. Since the policy is prone to over-optimize against the PM (Gao et al. (0022), we train 6.7B gptj (Wang & Komatsuzaki 2021) model as the reference PM (rPM) to measure the performance of alignment. The rPM is trained on entire human preferences data. We conduct experiments
evaluate the RM trained with and without MAS through accuracy and forgetting ratio (Chaudhry et al. 2018) (FR) of ac- curacy. The evaluation results of RM and rPM are shown in Table The accuracy is computed by counting the percentage of the reward scores of human-preferred responses that are higher than the reward scores of human-NGe Naman--OT--reeerrees responses Yuan et al., 2023). We initialize the SFT model from gpt2-s and train it on the Reddit TL,DR part-1 for 5 epochs. However, we do not perform the SFT process in task-2 as we observe no significant effects on performance.
Metries: We use the forgetting radio Chaudhry ct al. 2018) the ROUGE and reference PM score to measure the extent to which the old policy is forgotten. Notably, we consider the alignment (Ouyang et al., (022) as part of forgetting since it arises when the SFT model learns human preferences during the RL step. After learning all tasks, we evaluate the models on the entire test set using both reference PM score and ROUGE score. Table presents the metrics used to
Table 4: The evaluation results of RMs and rPM.
Reward Model
)((())((()) (
FR
RM,
0.7441
RM, w MAS RM, wo MAS 1PM
0.7203
0.7482
0.024
0.6971
0,7496
0.047
0.7624
0.7592
URI.
iattennn_____maaaae_ summa ri ze
compaT sons

---

Published as a conference paper at ICLR 2024
evaluate each task, as well as the final evaluation metric. A well rell-perfonning model is expected to achieve high scores in both the reference PM and ROUGE metrics.
Table 5: Metries for our tasks. (i 1, 2) denote the test data of Reddit TL:DR data part-i, and rPM (M,, (isst (i = 1,2) denote the reference PM score of model M, on dataset Difett
Metric
Definition
Task-1
reference PM Score on Task-1 (rPMS),,) 1)
(PM, (Df
Task
Aligument Tax (AT, J)
Rengee(Mgrr Mssrr,Dartt (oo
Task-2
reference PM Score on Taask-2 (rPMS,, 1)
rPM (M2. (n5aa)
Task
Score Forgetting Ratio (SFR,
(PM, Difaat (PMM(MM
(Df,,
Final eval
reference PM Score on entire test data (rPMS, 1)
xPM(Ma, 18g 11g(e))
4.2 RESULTS OF CONTINUAL LEARNING FROM HUMAN PREFERENCES
Table shows the results of continual learning from human preferences on the summary task. We ob- serve that CL methods, such as EWC (Kirkpatrick et al. (017) regularization policy consolidation (Kaplanis et al. 2019) can improve the training stability of the PPO method, thereby ensuring that the policy does not change too much with every policy gradient step. This leads to improved rPMS. Our method outperforms CL baselines by achieving the most significant enhancement in policy learning (fPMS) and possessing Backward Transfer (BWT) (Lopez-Paz & Ranzato 2017) capability (negative SFR). This is because our learning strategy is sample-adaptive and balances policy learning and knowledge retention. Additionally, CPPO performs better than Iterated RLHF because PPO is not stable enough in the learning process. We observed that during PPO training, the KL divergence and value prediction errors tend to increase suddenly, discussed in Section
Table 6: The main results of continual alignment on TL: DR dataset. For PPO (In order)", we directly finetune the RM, the novel data to obtain RM2, without using MAS regularization; and directly train the policy model Ma, against RM, to obtain May. For the Iterated (PHF) we retrain the RM2 and policy model Mr, on the combination of the Task-1 and Task-2 corpus. Methods in italics are trained against the continually learned (by MAS) reward models. Details of the implementation
can be found in Appendix G
Method
IPMS, (7)
Task-1 (Mr,)
ronge (7)
AT
#PMS; (?)
Task-2 (M,,))
rouge
SER
FPMS (?)
Final eval (M,,)
reuge (1)
Human
2,958
2,805
2.903
ChatGPT
3.298
0.197
3,189
0.191
3,242
0.153
SFT (Ia order)
1.99 10.130
0.245
10.006
1.545 1:006)
0.237 10.000
1.438 10.051
0.237
:0.000
SET tamulti-tskks)
1,244 10.041
0.254 10.011
1,536 2:0000
112334
20.00
1,505 100111
0.256
20.006
PPO orders'
2,629 40 183
0.195 30050
0.052
44.
2,546 6:0.201
0.151 +0.00
0.144 -00004
2,500 10,242
0.186
-0:15
Terated RLHF:
2620 10.185 185
0.196 10.50
0.052
10.044
2,732
6.0 163
0.211
10.001
0.061
-0.015
2,656 10.134
0.200
-0.00
PPO
2,699 30 183
0. 0,195 40.50
0.052
40.44
2,687
=01 126
0.184 +..017
0.000 -0.017
2.612 10.191
0.78P 20013
OLO+Onlindd22 Rer
2,758
2.701 10.205
2.730
0.196
PRO+EWC Airkparrick el 2017
2,857 20.122
0,201 20,045
0.017 =3099
2.825
0,175 =0.002
0.000
2,891
0.196 10.023
PPO
180
00.000
2,714
0.179 10011
20.006
2,824
0.179 20019
PRO-TRCE
2019
2,642
Kaapland
2,791
+00034
+0002
=0.017
2.616 10.177
0.20] =0011
Ramasonny aral, aral
10.041
#0.116
+0.001
=0.012
0.781
(Hfenrisnic)
+0.024
+0,023
3,099 20.155
0.179 2000
(Lears)
3.19 20.154
0.220 30.040
0.029 30.042
3 10.134
0.164 50.004
40.107 30.008
3.207 30.11
0,179 20006
4.3 ABLATION STUDY
We conduct an ablation study on our proposed CPPO method. To analyze the effect of the balance weights, we conduct experiments by setting either axx) or {(z) to 1. To analyze the effect of the knowledge retention penalty, we set (x) = The training curves of different weights are shown in Figure 3 and the evaluation results are presented in Table 7
We observe that the training process becomes unstable when setting ((x) to O. When setting a(x) to I reduces the rPMS, the noisy samples are learned together with normal samples without distinction, hence the reward increase is slower than CPPO. When setting 3(x) to increases the SFR, the overfitting samples, high- variance samples, and noisy samples are consolidated the in the knowledge retention process, hence the final reward value is lower than CPPO. The above experiments indicate that the sample-wise balance weights are helpful for both policy learning and knowledge retention.
7

---

Published as a conference paper at ICLR 2024
Table 7: Ablation study. PPO is a special case of CPPO (* = 1,8 = 0).
Task-1 rouge (1)
Task-2
Method
rPMS, (1)
AT (
rPMS2 (1)
rouge (1)
SFR ())
CPPO / Heuristic
3,020 +0.137
0.213 +0.024
0.035 -0023
2.978 +0.113
0.174 +0.019
-0.164 +00009
CPPO / Learn
3,180 10.154
0.220 10.40
0.028 00042
3.085 1:0.14
0.164 100024
-0.161 100008
PPO a = 1,8 0
2.629 =0.183
0.196
=0.500
0.052 =0.444
2.687 #0.126
0.184 +0.017
0.080 #0.017
CPPO/ =1
2.837 #0.124
0.196
+0.029
0.047
-0.41
2.745
+0.121
0.169
+0,020
-0.031 +0.010
CPPO/ |=
2.476 10.117
0.185
=0.211
0.063
=0.025
2.520 10.119
0.186
+0.017
0.051
#0.009
CPPO/ =
2.012 10.186
0.209
#0.022
0.038 0.045
2.436 10.141
0.174 10.021
0.142 100.155
reward/mean
policy/approx_Et kl
values/values _eror
betam)
betam1 CPPO(H)
betary
betar1 CРРО(H)
bbttar betar1 (PPDCH)
al phasl
(PPO(L))
pha==
CPPD(L)
al phas1
CPPOCL)
35
N
30
25
20
15
10
$0.00
10k
15k
20k
25k
204
25k
10k
25k
(a) reward
(b) approx_ki kl
(c) value errors
The curves of different weights in task-1. The knowledge retention weights penalty can improve the training stability of the PPO algorithm. However, setting ((x) = 1 slows down the increase of the reward compared with CPPO. On the other hand, the policy learning weights (xx can boost the increase of the reward compared with axx) = 1.
Figure 3:
4.4 STABILITY ANALYSIS
In this section, we analyze the stability of the CPPO, PPO, and PPO with the knowledge retention penalty. Previous work (Bai et al. 2022a) argues that small models are more prone to be unstable in PPO training. However, we find that CPPO can learn stably without the need for invalid-action et al.) 2022), even with small models. As shown in Figure 4 the vanilla PPO performers unstably on the new data distribution. PPO with knowledge retention penalty is more stable than PPO, but policy learning is slow. CPPO gets fast convergence reward score and shows stable performance on the KL divergence and value prediction. This is because the sample-wise learning strategy of CPPO restricts the learning of noisy samples.
reward/mean
policy/approx_K kl PPO - CPPO - Pno
values/values_ee aerorreeerorrrrrrrrr eerorr
PPO+KR
CAOO
PFO
PPO+KR
CPPO PPO
5
2
20
4.5
1.5
15
3.5
1
10
3
2.5
2
0
5k
10k
15k
20k
25k
10k
15k
20k
25k
5k
10k
15k
20k
25k
(a) reward
(b) approx_Ki kl
(c) value errors
Figure 4: Training process of Task-2. The PPO algorithm is unstable at 7k steps and is unable to continuously increase the reward score.
4.5 HUMAN EVALUATION ON REDDIT TL;DR
We train two gpt2-xl xx1 models using CPPO and PPO, respectively, and compare their summaries with those generated by humans and ChatGPT using a Likert Likee( 1932). The results are shown in Table 8 During the human evaluation, we observe that ChatGPT tends to generate longer summaries than humans and our models, but its performance remains stable across the test samples.
Although humans provide the best summaries, they still made mistakes, such as obfuscating important details. Our model achieves comparable performance with ChatGPT but still makes mistakes that the
8

---

Published as a conference paper at ICLR 2024
small model often makes, such repeating words and sentences. Due to the training inefficiency and instability, the performance of gpt2-xl trained by PPO is not satisfactory.
RELATED WORK
5.1 REINFORCEMENT LEARNING FROM HUMAN OR AI FEEDBACKS
Learning from human preferences has been studied in the game field (Bradley Knox & Stone 2008) and has
Table 8: Human evaluation on 100 posts from the Reddit TL;DR.
recently been introduced into the NLP domain, such
as dialogue systems (Li et al. 2023; Zhao et al. 2023
Method
Likert score
Improve
p-value
2024). Previous work (Stiennon et al. (0200) utilizes
PPO
4.3
the PPO algorithm to fine-tune a language model (LM)
CPPO
4.730
8.23%
0.037
for summarization and demonstrates that RLHF can
ChatGPT
4.760g1.01
8.92%
0.013
improve the LM's generalization ability, which serves
Human
4.900g1.034
12.13%
0.001
as the technology prototype for InstructGPT (Ouyang er al. 2022) and ChatGPT. Learning LMs from feedback can be divided into two categories: human
or feedback. Recent works such as HH-RLHF (Bai et al. (2022a) and InstructGPT (Ouyang et al.
2022) collect human preferences to train reward model and learn policy through it. ILF (Scheurer 2023) proposes to learn from natural language feedback, which provides more information per human evaluation. Since human annotation can be expensive, learning from AI feedback (RLAIF) (Bai et al. 2022b Perez et al., 2022] Ganguli et af. 2022) is proposed, but current methods are only effective for reducing harmless outputs, while helpful outputs still require human feedback.
ct al.
5.2 CONTINUAL LEARNING
Within the realm of continual learning, several noteworthy methodologies emerge, encompass- ing the regularization-busee approach, replay-bused techniques, optimization-based strategies, representation based methodologies, and Erchitecture-based innovations (Wang et al. 2023).
The Regularization Based Approach Kirkpatrick et al. 2017; Aljunnii et al. 2018 Chaudhry et al.
2018; Li & Hoiem| 2018 Castro et al. 2018) orchestrates the introduction of explicit regularization terms, thereby striving to strike a harmonious balance between the acquisition of new skills and the retention of past knowledge. The Replay-Based Approach aims to preserve and reuse past experiences to enhance model performance, which includes experience replay (Lin] 1992). generative replay or pseudo-rehearsal (Sun al., (2020) and feature replay(Liu al.) 2020). The Optimization-Ban Based Approuch navigates the terrain of continual learning through explicit design and manipulation of optimization programs. This includes techniques such as gradient projection(Lopez-
Ranzato 2017), and meta-learning (Javed & White 2019). The Representation- -Based Approach leverages the strengths of supervuee learning (SST.,(Galaa et al.) 2021) and large-scale pre-training (Meta et al., 2022) to enhance the quality of representations at both the initialization and continual learning stages. The Architecture-F Based Approach addresses inter-task interference by fashioning task-specific parameters. This approach can be dissected into three distinct paradigms: parameter allocation Serra ct al. 2018 model decomposition Ebrahimi et al. 2020), and modular networks(Rust et al. 2016).
6 CONCLUSION
In this work, we propose CPPO, which utilizes learning weights to balance policy learning and knowledge retention, with the aim of improving the PPO algorithm for continual learning from human preferences. CPPO is task-agnostic and model-agnostic method that does not significantly increase the time and space complexity of PPO. We evaluate CPPO on both the DIL task and three non-continual tasks and show that it outperforms strong continual learning baselines when continually aligning with human preferences. Additionally, CPPO improves the learning efficiency and training stability of PPO. Our experiments demonstrate the potential of our approach for efficient and stable continual learning from human preferences, which can have applications in various domains and tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their valuable suggestions to improve the quality of this work, and we express our sincere gratitude to Dr. Bin Liang for his invaluable guidance and con- structive feedback throughout the preparation of this manuscript. This research was supported in part by the National Key Research and Development Program China (2021ZD01 12905), the Major Key Proiect of PCL (PCI.2023A09-4)) the National Natural Science Foundation of China (62176076), the Guangdong Provincial Key Laboratory of Novel Security Intelligence Technolo- gies(2022B 12 1212010005), Natural Science Foundation of Guangdong (2033A1515012922) Shenzhen Foundational Research Funding (BCY12002008181024115000) and the UK Engineering and Physical Sciences Research Council (EP/X019063/1).
REFERENCES
Wickliffe Abraham and Anthony Robins. Memory retention - the synaptic stability versus plasticity dilemma. Trends in Neurosciences, 28(2):73-78, 2005. ISSN 0166-2236. doi: https://doi.
org/10..0066j,tins.220000 20w.. 12.003. URL https::////// /ttt:s::/www.r http::/wwwww.cieceereeeee..... www.sssiiiiiiiiieeee rect. com/ science/
Aaticceep11//00162230000000
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) forget. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss ). Proceedings of the European Conference on Computer Vision (ECCV), pp. 144-161, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01219-9
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan. Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dode
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt. Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer. Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton. Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodee Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Sunmarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https 7/ www.aannwhooooooooo//oooeoe
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. Continual lifelong learning in natural language processing: A survey, In Proceedings of the 28th International Conference
on Computational Linguistics, pp. 6523-6541. Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10.18653/v1 1tt/33//11222.oo00 www.ing....nn.........o main.574. BRLhttp:
//acl a wwwnnhhooo........oooo wttp//og..200... col - ma mai..aii...... inmaiii... 574
W. Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement.
In 2008 7th IEEE International Conference on Development and Learning, pp. 392-297, 2008 doi: (1109//EEIRRN.000...0
10

---

Published as a conference paper at ICLR 2024
Francisco M Castro, Manuel Marin-Jiménee Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233 248, 2018.
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez. lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with htt.RI//tttt///// / cuna mays.... org
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pp. 86-102. Springer, 2020.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XI 16, pp. 386-402 602-402.60 Springer, 2020.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8008. PMLR, 17-23 Jul 2022.
Jhair Gallardo, Tyler L Hayes, and Christopher Kanan. Self-supervised training enhances online continual learning. arXiv preprint 1rXiv:2103. 2021.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen,
Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez. Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and compre- hend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL httttp:://proceedinng neur wwtprriss.ccppppppee file/ 1eeeaaffeeeeSSscccc1113022224000
fi Fiflle/paape pspppppp//2215//
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1638-1649, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653//1/18--111 URL http:p/://.aaaamnoooooooooo/o///
1152
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 831-839, 2019.
11

---

Published as a conference paper at ICLR 2024
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 2736-2746, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v112022
ptatpps://cllantholoogy/ ant.aaatthoottooomooooooo org www.c--maan.nn1 URL //c////c///
2021 naac main. 218
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), P) pp. 3983-498500 Online, November 2020. Association for Computational Linguistics. doi: 10. 2020 www.emmannmmmm.mmmmmmmmmmmmm 327
http:///0000.mmmmai...m..m URL httpt::/aclaaaahlllloooo org
Khurram Javed and Martha White. Meta-learning representations for continual learning. In H. Wallach, H. Larochelle, Beygelzimer, d'Alché-Bue, E. Fox, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://prooeeedingg.. nwwrips. saapperrflleseeeeeeeeeereeeeeese wwwes//appr//0111/.. File/
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3242 3242-3251. PMLR, 09-15 Jun 2019. URL httph::t://// /attps::/procceddngg
mlr. 2zzess/v7//aape
19a..9911 .hmll
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13 2017. doi: 10. 10.1073/pnas. 161 1835114. URL https::/p/// /www///www/////////////////l// pnww www.:/paas..or.....ooooo. abs 10t.3/1...1111111 1073/pnas
1611835114
Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 3 (Industry Papers), pp. 92-105, New Orleans Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/111-300 URL https
Jiazheng Li, Runcong Zhao, Yongxin Yang, Yulan He, and Lin Gui. Overprompt: Enhaneing chatgpt through efficient in-context learning. Advances in Neural Information Processing Systems RO-FoMo workshop, 2023.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal, Mach. Intell., 40(12):2935 2947, dec 2018 ISSN 0162-8828. doi: (0.100//AAM.220::273338.. URL http::://doi.ore/ 1109/T.. 2017. 2773081
Rensis Likert. A technique for the measurement of attitudes. Archives of Psychology, 140:1-55, 1932.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barceloma, Spain. July 2004. Association for Computational Linguistics. URL
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn., 8(3-4):293- 3211, May 1992. ISSN 0885-6125. doi: 10. 10.1007/BF00992699
Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. Generative feature replay for class-increment learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 226-227, 2020.
12

---

Published as a conference paper at ICLR 2024
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Isabelle Guyon, Ulrkee von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems
30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA, USA, pp. 6467-6476 2017. htttttttts////tt http::::s//pppceeeiieioooooe ngs. neurinneurrii wwwrres...... ps. cc/ paper/ /tt.://.hh7715522222222222776663
wttp///dddbbbbbba...aaaaa.a. html
Abstract
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https
//aclanthology. org/ 11///P111-10 11-11111010 1015
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning, 2022. URL https:/ htttps://operrve. ew. net/
Forum? d=D 9F...
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https //proceedings mlr w.w.r.ppreseee. prwss / v48 120/8/mmnii httm//mnnhm.....mmm.mm...mae.m.a html
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL abs 2203 02155
https::/arxiy: wtwp:/iiw..rg//////00000
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/10073083.1773333 URL http://cclaattooooooor///222
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 3419-3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. URL BRL
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. 2022. URL htt.hhtts://// / //tp://aaivvvvv//a org/ abs 2210.211222 01241
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL
neuris http:///cccceednoo..eerrrreecspp filess Pwwpr.pppprr.....
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint a60iv:1606.0 04671, 2016.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert. distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01100 108, 2019. URL http: ptttt://arxtt org/ 91 01108
13

---

Published as a conference paper at ICLR 2024
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale, 2023.
Philemon Schöpf, Sayantan Auddy, Jakob Hollenstein, and Antonio Rodriguez-sanchez Hypernetwork-PPO for continual reinforcement learnin In Deep Reinforcement Learning Work- shop NeurIPS 2022, 2022. URL Utthttts:://ppeeeeee
net wwt.:/oommmm110::3>1>.......
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017. URL httt:://axivv. wwwp:iiiworr/aa.. abs/
1707. 06347]
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp- 4548 4557. PMLR, 2018.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distriuttion generalization: A survey. CoRR, abs/2108.13624, 2021. URL https://
anxiv.org/abs//100 13624
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. URL https:://axxiii http://www..rggb//////o////
01325
Fan-Keng Sun, Che Ho, and Hung-Yi Lee. LAMOL: language modeling for lifelong language learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. URL settp:://openeevee..
forum? -eni2dddSkkke Skgxnn
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein, TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59-63, Copenhagen, Denmark. September 2017. Association for Computational Linguistics. doi: 11116653/11/11-4455
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model.
/estps://githie http://hooooaoooooooooooo/o////a
jax May 2021.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application, 2023.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears, 2023.
Runcong Zhao, Wenjia Zhang, Jiazheng Li, Lixing Zhu, Yanran Li, Lin Gui, and Yulan He. Narrative- play: Interactive narrative understanding. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, Demonstration Track. Association for Computational Linguistics, 2023.
Runcong Zhao, Wenjia Zhang, Jiazheng Li, Lixing Zhu, Yanran Li, Yulan He, and Lin Gui. Nar- rativeplay: An automated system for crafting visual worlds in novels for role-playing. In AAAI Conference on Artificial Intelligence, 2024.
14

---

Published as a conference paper at ICLR 2024
A NOTATIONS
All of the notations used in this paper and their corresponding meanings are listed in Table 9
Table 9: Notations used in this paper, italic font denotes the CPPO-specific symbols
Notations
Corresponding Meanings
K1.... ((),
generation of the i-th token t-th task of CL a rollout sample i-th token of sample state-i: prompt parumeters of policy learned in tusk-t policy learned in task-t
0
P Pr.00
generation probubility of under m total objective of PPO CPPO
J(0)
I
clipped policy learning objective squared-error value loss
LVF
V(s))
value estimation by critic reward/ value discount coefficients advantage score of token rewurd model score of x the probability ratio clip hyperparumeter clip by +
A/7
A,
R(a)
r.(0)
cbp(s 1
coefficient of LVF
samples number per rollout batch
the threhold of times of standard variance
LRR
knowledge retention penalty
a(z)
weight of policy learning
B(z)
weight of knowledge retention the upper bound and lower bound of weights
ubilb
expectation Pc,
A
expectation R(ar)
P.J
standard variance of (x)
o
standard variance of R(z)
(
paraintters weight learning
(0)
objective of weight learning
B THE THEORETICAL ANALYSIS OF CPPO
The theoretical objective in Eq. is an intuitive implementation of our basic idea. Based it, derive a more practical objective in Eq. 6 Next, we will elaborate the relationship between the two and explain how we were inspired by Eq. and designed Eq. 6
Eq. Gis generalized version of Eq.
Let ID, (I) and Ip, (x) denote the indicator functions of the sets of D1 and D2, respectively. In Eq. 6 (x)) and ((x) can be any non- -egaaiite real-valued functions defined on the rollout set. We claim
that in Eq. 4 (x) and (ax)) are specialized as axx) = TD (x), (xx TD,(r)) Next, we provide the derivation step by step.
By introducing the actor-critic version, the clipped ratio, the objective Eq. can be written as the objective Eq.
1. We utilize notations ID, (x) and I [D, to rewrite the Eq 4 maxę E ID, (x) R(x)
E L (x) D (Pxo (x) P..., (x)).
2. Then we introduce the importance sampling like PPO, the above objective can written as
maxg Eg Ip. (x)
(a)
R(x) ID. Dx (P.. Te (x) P.. (x))
P.,-,
3. In the PPO method, the objective is to maximize the expectation of the advantage function instead of the reward value. By introducing the advantage function instead of the reward, the
above objective be written as maxg B TD, (w) P A(x) E ID (x).
P (x)
DKL. (P.,, P (x))
15

---

Published as a conference paper at ICLR 2024
4. By introducing the knowledge retention penalty, the above objective is written Pro
maxy B (x) P ( Bu JD I
5. In the CL task, the new policy is generally initialized by the old policy _T-1- In CPPO, we treat the Th-1 and ne as the reference model and policy model respectively. Then, we consider the actor-critic version, the clipped ratio, and the entropy bonus used in PPO[32], the above objective can be written as J(0) L'd, CLIP+IP, KR-VF
From the objective Eq. to the objective Eq.
Comparing (0) with ( ( (0) in Eq. 6 a(x) and [(x) are specialized as TD, (x) and TD (a). Hence, the objective Eq. can be treated as a special case of Eq.
C CASE STUDY OF SUMMARY TASK
Table 10 presents where CPPO improves the performance task-1 (r/relationships) after learning task-2 (r/others).
Table 10: Example responses of Reddit TL:DR.
The original POST (the domain is r/relationships:
TITLE: [20F] My [22M] SO never wants to spend time with me.
POST: My boyfriend and I have heen together off & for three years. We've had fair share of issues, but I love him think he loves me.
We work opposite shifts, and it really cuts into our time together. Majorly. On the few days we both have off, every other weekend, he chooses to hang out with his friends. He works with his friends on the same schedule, and they over every single day.
always feel like have to force him to do anything with me and I'm getting tired of it. always complain to him that would be nice we could spend time together. But, he just says over reacting. And that can hang out with him and his friends. The thing is, we haven't It spent day together just him and I in over year. I like his friends but just want day where we be together just us. I' sick of being ignored for his friends, or video games and don't know what do. TL;DR:
The summary after learning task-1 (r/relationships):
CPPO (rPMS: 1.75); My SO never wants to spend time with me. He works with his friends on the same schedule, and they are over every single day. always feel like I have to force him to do anything with me and I'm getting tired of it.
The summary after learning task-2 (r/others):
CPPO (rPMS: 1.82): My boyfriend prioritizes friends over time with me. Opposite work schedules strain time together. I'm tired of urging for attention. Secking advice for reconnection.
PPO(In order) (rPMS: 1.44): The boyfriend has been with his boyfriend for three years and doesn't want to spend time with his boyfriend anymore. The boyfriend's 's boyfriend been with him for three years and doesn't 'tt want to spend time with anymore.
PPO (rPMS:1 1.65:: The boyfriend and have been dating for three years and love him but he is not interested going out with his girlfriend.
After learning the task-2, the model generates an accurate and concise summary. It more clearly conveys the main issue and emotions in the post, along with the desire for advice. And it also gets higher rPMS (1.82 v.s. 1.75).
From the three summaries after learning task-2, it can be observed that PPO(In order) seems to exhibit a more noticeable knowledge forgetting, with a seeming lack understanding of the concept "boyfriend." This is due to the frequent occurrence of "boyfriend" in task-1 (t/relationships) and its almost absence task-2 (r/others), resulting in a case catastrophic forgetting. The PPO model still manages to convey the main essence of the text, but it overlooks some crucial details, such as "opposite work schedule" and "prioritizes friends over time with me". hence PPO lags behind CPPO in terms of rPMS value.
16

---

Published as a conference paper at ICLR 2024
D BASELINES
Supervise fine-tuning (SFT) directly learns the human-labeled summary through the cross-entropy loss.
Online 1.2Reg penalizes the updating of model parameters through 1.2 loss T,)(0) (0:-00 This regularization term mitigates the forgetting issue by applying a penalty for every parameter change.
EWC (Kirkpatrick et al. 2017) uses fisher information measure the parameter importance to old tasks, then slows down the update of the important parameters by L2 regularization.
MAS (Aljundi et al., 2018) computes the importance of the parameters of neural network in an unsupervised and online manner to restrict the updating of parameters the next task.
hatigge-dissiillii-nbbabseee method, which computes smoothed version of the current responses for the new examples at the beginning of each task, minimizing their drift during training.
LwF (Li & Hoiem 2018) is a
TFCL (Aljundi et al., 2019) proposes to timely update the importance weights of the parameter regularization by detecting plateaus in the loss surface.
PC (Kallanis et al. 2019 is inspired by the biologically plausible synaptic model and proposes to consolidate memory directly at the behavioral level by knowledge distillation, aiming mitigate catastrophic forgetting in the reinforcement learning context.
Hypernetwork-Poe is continual model-free RL method employing a hyper network to learn multiple policies in a continual manner by using PPO.
HN-PPO |Schöpf al.
2022
2022 NLPO learns mask out less relevant tokens in-context it trains via top-p sampling, which restricts tokens to the smallest possible set whose cumulative probability is greater than the probability parameter p (Holtzman et al. 2018).
NLPO (Ramamurthy al.
E DISCUSSION
E. HYPERPARAMETER SENSITIVE ANALYSIS
Due to the introduction of additional hyperparameters by CPPO, we conducted a sensitivity analysis of CPPO's hyperparameters. We conduct sensitivity analysis on five hyperparameters, including the threhold of times of standard variance k, the upper bound ub and lower bound 1b weights, the learning rate weights-Ir of CPPO Heuristic, and the multiplier As shown in Table [I] the analysis of experimental results shows that our method insensitive to the introduction extra hyperparameters.
Table I1: Hyperparameter sensitivity analysis of CPPO Heuristic and CPPO Learn
Hyper-Parammterrs
Task-1 rPMS; (1)
Task-2
Method
k/ u/ub/ /bb
rouge (1)
AT (L)
rPMS, (1)
rouge (f)
SFR ())
0.55/2.5
Heuristic
3,020 20.137
0,213 10.024
0.035 =0.023
2.978 +00.113
0.174 #0.019
-0.164 20.009
0.85 0.5
Heuristic 3.011 10.141
0.209 10.026
0.036
10.025
2.97 10.121
k: 0.85 -> 1.0
Heuristic
3.017 +0.132
0,214 +0,025
0.031
=0.031
2,891
+0.117
0.170
+-00211
.0.151 +0.010
ub:
Heuristic
2.982 10.124
0.205 10.011
:0.042
124
0.173
+0.023
+0.008
ub: 2.5 3.0
Heuristic
3.012 #00.133
0.205 +0.02
0.040
=0.051
2,941
10.115
0.171 +00.029
-0.166 10.013
lb: 0.5 0.1
Heuristic
3.011 10.166
0.221 40051
0.024
=0.041
2,809
115
0.167
10.019
-0. .0.162 10.011
lb: 0.5
Heuristic
2.997 20.152
0.219 10:31
0.026
=0.040
10.141
0.179
-0.161 10.010
weights-lr/7
Method
rPMS, (t)
rouge
AT (
rPMS, (t)
rouge (t)
SFR (L)
0.1
Learn
3,180
110.154
0.220 10.040
0.028
10.042
3,085
110.134
0.164 100024
-0.161 100000
weights 0.01 > 0.1
Learn
3.122 40.162
0.201 40.011
0.044 +0.44
2,824 +0.141 141
0.171
+00024
.0.155 +0.012
weights-Ir 0.01 0.5
Leam
3,141
0.209 10.53
0.036
=0.022
2.934 +0.125
0.170
-0.162 =00016
7: 0.1 121->00 0.01
Learn
3.042 10.137
0
10.046
0.034
20.054
2.89 10.151
0.168
#0.026
-0.161 10.020
2.892
0.174
-0.161
0,1 1.0
Learn
20.144
0,209 10.039
0.036 =0.051
2,967 10.129
0.172 =0.021
-0,169 =0.008
17

---

Published as a conference paper at ICLR 2024
E.2 COMPLEXITY ANALYSIS
In this section, we compare CPPO (learnable weights) with PPO in terms of time and memory occupation. The steps of CPPO are similar to PPO, except for the step of learning balance weights. By considering the time of the rollout step our reference, we demonstrate that the time required to learn the weights is negligible compared to the overall training process of CPPO and PPO. Figure 5 iillusrrates the time required for learning balance weights and the time for making rollouts during the training of gpt2-s and gpt2-xl. For gpt2-s training, the ratio between the time spent on learning balance weights (approximately 8s) and the time taken for rollout steps (around 400s) is 1/50. This ratio decreases to 1/200 when training gpt2-xl, due to the fact that the time for learning balance weights remains the same, while the time for making rollouts increases to 1600s. Hence, our method does not significantly increase the time complexity of PPO, especially for training large language models.
Time/ Learning weights
Time Rollout
CPPO / upt2-xl OPP gpt2-s
01910 / ppt2-x1 UPO got2-s
20
15
1030
500
5k
10k
15k
20k
25k
10k
15k
20k
25k
(a) Time of learning weights
(b) Time of rollout
Figure 5: Time of learning weights and time of making rollout.
For memory occupation, we record the GPU memory allocation, GPU utilization, and the process memory in the training process of PPO and CPPO. Figure 6 illustrates the comparison of the above metrics between PPO and CPPO. CPPO, which learns the balance weights and calculates the knowledge retention loss, leads to higher allocation of GPU memory and process memory compared to PPO. Nevertheless, the improvements in GPU memory and process memory are not particularly substantial.
GPU Memory Allocated (9) CPPO Policy
GPU Utilization (96)
Process Memory In Use (MB)
PPO Policy
CPPO Policy
-PP Policy
CPPO
PPO
CPPO RM
-- PPO RM
-- CPPO RM -- PPO
10700
100
100
10920
80
90
10500
10400
10300
Time (hours)
Time (hours)
Time (hours)
10200
15
(a) GPU Memory
(b) GPU Utilization
(c) Process Memory
Figure 6: GPU utilization and memory allocation when the algorithm runs for 15+ hours. Compared to the PPO method, our CPPO does not significantly utilize extra memory.
F TASKS FOR STATIC LEARNING
We compare PPO and CPPO on 3 static learning tasks, including random walks, sentiment text generation, and summary on CNN Daily Mail.
F. RANDOM WALKS
The Chsn et al. 2021) involves finding the shortest path on directed graph. The reward is based on how optimal the path is compared to the shortest possible (bounded in [0. IJ). Paths are represented strings of letters, with each letter corresponding to node in the graph. For CPPO or
18

---

Published as a conference paper at ICLR 2024
PPO, language model was fine-tuned to prediet the next token in sequence of returns-to (sum of future rewards), states, and actions.
F.2 SENTIMENT TEXT GENERATION
This task focuses on generating positive movie reviews by fine-tuning a pre-trained model on the IMDB dataset using a sentiment reward function. We consider the IMDB al. 2011) 1) dataset for the task of generating text with positive sentiment. The dataset consists of 25k training, 5k validation and 5k test examples of movie review text with sentiment labels of positive and negative. We utilize a sentiment classifier (Sanh et al., 2019) trained on pairs of text and labels as a reward model, which provides sentiment scores indicating how positive given piece of text is.
F.3 SUMMARY ON CNN DAILY MAIL
The dataset for this task comprises 287k training examples, 13k validation examples, and 11k test examples. We utilize meteor (Banejee & Lavie 2005) as the reward function. T5 is chosen as the base language model due to its pre-training a unified text-to-text framework and its ability to handle zero-shot capabilities.
F.4 EVALUATION ON NON-CONTINUAL LEARNING TASKS
We compare the performance of PPO and CPPO on three static learning tasks, including ran- domwalks et al. 2021), sentiment text generation (Ramamurthy et al. 2022) on IMDB(Huang al. 2021), and summarization on CNN Daily Mail (Hermann et al. 2015). As in the continual learn- ing setting, we initialize our model with a pre-trained model and compute the knowledge retention penalty using both the policy model and the pre-trained model. Experimental results demonstrate that CPPO outperforms PPO in static learning settings. We observe the instability of PPO on the sentiment text generation task, while CPPO can learn stably. As shown in Figure 7 CPPO outperforms the PPO algorithm on all three tasks, which is attributed to CPPO's s ability to enhance the learning of high-perffrmmacc, sariancr,e and overfitting samples while slowing down the learning of noisy samples.
reward/mean
reward/mean
reward/mean
CFPO FPO
CPPN
PPO
CPPO PP0
0.7
0.7
0.5
0.8
0.19
0.4
0.3
0.4
02
0.1
Stop
0.7
Stop
0.15
0.15
Stop
3k
64
8k
104
1k 2k
3k
5k
10k
1.5k
(a) Randomwalks
(b) IMDB
(c) CNN Daily Mail
Figure 7: Evaluation results on the test data during different training steps. a) The optimality scores in [O, 1], as compared to the shortest path. Positive sentiment scores provided by the distilbert trained on the IMDB dataset. c) METEOR (Metric for Evaluation of Translation with Explicit ORdering).
G DETAILS OF IMPLEMENTATION
The algorithm of CPPO presented in Algorithm 1 Step is for learning an RM continually; step is for computing balance weights; step 9 is for calculating CPPO loss; other steps are the same the PPO algorithm. Our implementation is based on the open source library trlx[] The experiments on the SHP dataset conducted in 4 Nvidia A100 GPUs with 80 GB of RAM. other experiments conducted 2 Nvidia Tesla V100 GPUs with 32 GB of RAM. To conserve GPU memory, we utilize CPU-Offloal and
techniques. We provide all hyperparameters used in both the PPO
and CPPO algorithms in Table 12
httpp:://git httpgggtihuuhoooo
19

---

Published as a conference paper at ICLR 2024
Algorithm 1: CPPO
input :SFT model MYFT, critic model C, reward model RtM, ppo epoches N, ppo steps S. query streams Qe = 1.2....,77 T..
:Aligend model My
Initialize actor Mo f- MSFT
for f 1. do
update RM new feedback by MAS;
for epoch 1,2.....N do
make actor M,-1 generate response O on prompts Q+:
compute generation probability P (z) of M,-1 on O₃-1, reward R(x) of response O--1 by R.M, state value evaluation ft-1 by critic C and advantage A--1
compute set of balance weights [[a(2), (xx))] E O for step = I.2....,S do
compute the CPPO loss by Equation (5); update model Mt by Adam optimizer;
end 11
end
13 end
Table 12: Hyperparameters of different tasks. Italic font denotes the CPPO-specific hyperparameters. For all tasks, we utilize the default PPO hyperparameters released by trlx.
CNN
Random walks
IMDB
Reddit
seq-length total-steps
612
1024
550
17200
10000
4000
25600
batch-size
12
100
128
8
model (huggingface)
googla/ttantt5-small
Carper wrandomwwaks
Ivwerra/gnt2-imdb
gpi2
num-lyyers-unfrozen
8
optimizer
adamw
udamm
adamw
udamw
1.00E-05
3.00E-04
1.00E-04
5.00E-06
betas
[0.9, 0.999]
[0.9. 0.95]
[0.9, 0.95]
[0.9, 0.999)
eps
1.00--08
1.00F-08
1.00F-08
1.00F-08
weight-decay
1.00E-06
1.00E-06
1.00E-06
1.00E-06
scheduler
cosine-annealing
cosine-anmealing
cosine-anncaling
cosine-annealing
T-max
17200
10000
4000
25600
eta-min
1.00E-06
3.00E-04
1.00E-04
5.00E-06
num-rollouts
128
128
chunk-size
12
128
128
ppo-epochs
4
4
init-kl-coef
0.05
0.05
0.05
0.1
target
6
horizon
10000
10000
10000
10000
gamma
0.99
1
lam
0.95
0.95
0.95
0.95
cliprange
0.2
0.2
0.2
clirrange-value
0.2
0.2
0.2
vf-coef
1.2
0.2
scale-reward
False
False
False
False
clirrange-reward
10
10
10
max-new-tokens
100
40
50
top-k
50
top-p
0.95
k
0.85
0.85
0.85
0.85
reg-coef
0.1
0.1
0.1
0.1
ub
2.5
2.5
2.5
2.5
0.5
0.2
0.5
0.5
weights-Ir
0.01
0.01
0.01
0.01
20

---

Published as a conference paper at ICLR 2024
Matrix-taskorder
02
01
34
53
9.
21
11.
6.2
34
10.3
CED
24.
373
35.
112
20
23
38A
452
33
23
64
3.
35
46
5.
33
73
369
360
300
45
0.5
30
83
30
74
91
285
74
44
64
84
X-8
36
00
231
2.7
2.
43
711
15.1
55
3.7
71
9.
52
100
211
73
6.8
63
80
10.
45
20
180
Figure 8: evaluation of 18 reward models. The i-th row represents the RM trained on domain the j-th column represents the RM tested on domain j. and the value indicates the accuracy degradation value, namely, Acc Acciest
H EVALUATION OF STANFORD HUMAN PREFERENCES BENCHMARK
We have established a novel continual RLHF benchmark using the Stanford Human Preferences Dataset (SHP) (Ethayarajh et al. (2022). We assess CPPO and baselines on this benchmark within the context of a 3-task sequence setting.
H.1 DOMATN INCREMENTAL. LEARNING SETTING
The policy is required to continuously learn from three segments of the SHP dataset. The SHP dataset comprises 18 domains, which we divide into parts based on the maximal errors of the out-of-distribution (OOD) (eneralizationnShen et al. 2021). We employ the SteamSHP-faat -aaaa-----fan---- model (Ethayarajh et 2022), developed by Stanford, as the reference preference model (rPM) for assessing responses SHP prompts. The SaammSS/P--aa---555 mode is trained on the combination of the SHP (all domains) and the HH-RLHH et al., 2022a) human preference data.
To create more challenging DII. tasks, we individually trained 18 reward models (based on Llama-7b) on domains of SHP data and evaluated each reward model on the test set of the 18 domains, resulting in accuracy difference matrix of size 18x 18 shown in Figure 3 In this matrix, the row coordinates represent the training domains, and the column coordinates represent the evaluation domains. The elements in the matrix represent the relative decrease in performance domain compared to the training domain (both evaluated on test sets from various domains). Based on this accuracy difference matrix, divided the 18 domains into 3 groups (each has domains). This division ensures that there will be a significant performance decrease, i.e., the largest error of OOD generalization, when evaluated on domains from different groups. For example, the RM trained in the first group (from "explainlikcimfive" to "changemyview") has large errors in the third group (from "legaladivce" to "vet"), Refer Table 13 for the distribution of samples across various domains.
the evaluation
21

---

Published as a conference paper at ICLR 2024
Table 13: Number of posts in the SHP dataset by subreddit.
subreddit
train
valid
total
All
Task ID
ALL
348718
18436
18409
385563
100.00%
explainlikeimfive
19592
1014
1070
21676
5.62%
askanthropology
3910
203
268
4381
1.14%
askcarguys
3227
159
117
3503
0.91%
Task
asksciencefiction
29382
1576
1987
32945
8.54%
asksocialscience
2706
147
188
3041
0.79%
changemyvier view
38173
1637
1836
41646
10.80%
askacademia
31450
2095
1708
35253
9.14%
askculinary
45710
2094
2563
50367
13.06%
askengineers
57096
3154
2638
62888
16.31 %
uskphilosophy
10307
608
677
11592
3.01%
askphysics
7364
409
587
8360
2.17%
askscience
13316
899
977
15192
3,94%
legaladvice
21170
1106
1011
23287
6.04%
askbaking
44007
2096
1544
47647
12.36%
askdocs
6449
315
455
7219
1.87%
Task
askhistorians
3264
113
164
3541
0.92%
askhr
8295
641
395
9331
2.42%
askvet
3300
170
224
3694
0.96%
H.2 EVALUATION METRIC FOR CONTINUAL LEARNING
Overall performance commonly assessed using average accuracy (AA) Chaudhhr et al. 2018 Lopez-Par & Ranzato 2017) and average incremental accuracy (AIA) (Douillard et al. [020 Hou et 2019)) In our evaluation framework, accuracy is replaced by the reference Preference Score, which ranges from 0 1. Let dæj [0, 1 represent the reference Preference Score evaluated the test set of the j-th task after incremental learning of the k-th task (j S k). The two metrics the SkS-th task are then defined as:
AAE
akji
(8)
ATA5
AA
(9)
where AA signifies the current overall performance, and AIA additionally captures the historical fluctuations.
Memory stability can be evaluated through forgetting measure (FM) et al. 2018 and backward transfer (BWT) (Lopez- Paz & Ranzato 2017). Regarding the former, the torgetting of task computed the difterence between its best performance achieved in the past and its current performance:
max (ai. an.)) Vj k. xE(
(10)
FM at the k-th task is the average forgetting of all old tasks:
FM,
h: - Σ f.k.
1
(1I)
Regarding the latter, BWT assesses the collective impact of learning the k-th task on all previous tasks:
BWT:
(ak. ass);
(12)
H.3 EVALUATION RESULTS
We utilize the Llama-7B as the backbone of the policy model and reward model. When training the reward model, we ine-tune the all of parameters. When training the RLHF model, we only fine-tune the top 8 layers.
22

---

Published as a conference paper at ICLR 2024
The experiments run for 3 random seeds, the final results are shown in Table 14
We found that using reward and policy models based on Llama-7B, as opposed to training with gpt2-xl -xl as the backbone, significantly improves the training stability. It is related to larger reward models being less susceptible to reward attacks et al. 2022).
we observe that the Iterated RLHF method has significant effect in preventing forgetting. Since it re-trains the reward model and policy model from scratch at each task, resulting in higher memory and computational complexity. CPPO, compared to PPO, shows noticeable improvement in overall performance, and it outperforms PPO in the Memory stability metric. This indicates that CPPO can effectively learning varying human preferences.
From Table 14
Table 14: Performance on DIL setting. The range of SteamSHP score is [0, 1].
Task
Task 2
Tusk 3
Overall performance
Memory stubility
Method
SteamSHP (t)
SteamSHPP (1)
SteamSHP
(T)
AIA (T)
BWT
FM (3)
SFT (In order)
0.806 10.0101
0.836 10.0103
0.853 10.0103
0.832
20.0011
0,837
10.0009
-0.022 10.0094
0.022 10.0094
SET (multi tuskss
0.831 #0.0206
0,847 +0.0155
0.858 +0.0114
0,845
#0.147
0.844
#0.0082
-0.006
#3.0183
0.009=0.0101
Iterated RLHF
0.869 10.0683
0.88 10.0190
0.887
40.042
0.879
20.088
0.874
20.0433
4.,0004
10.0186
0.003
00.062
PPO
0.853 10.057a
0,879 10.028
0.889 1000
0.874
10.0033
0.877
10,0578
-0.017 1:.0951
0.020 10 0327
CPPO (Heuristie)
0.864
0557
0.89 10.0448
0.894 1..0350
0.883
1.0.029
0.887
10.039]
-0.015 1.0.021
0.018
0300
CPPO (Learn)
0,872
20.0644
0.898 +0,0550
0.899 +:1.0342
0.89 20,0224
0,894
20,0089
-0,013
#:.0298
0.016
00.028
I LIMITATION: THE RISK OF
We have observed that both PPO and CPPO have the potential risk of achieving high rewards while generat- ing poor summaries. This issue is depicted in Figure 9 where the policy model tends to overoptimize against the RM when trained for 100k steps (390 epochs). Over time, the policy becomes excessively focused on max- imizing rewards without adequately considering the quality of the generated summaries. To address the risk of optimization, various strategies can be employed. One approach to train an additional RM evaluate the policy during training. This allows for evaluating the policy's s performance using an external objective metric, providing more robust measure of the sum- mary quality. Another strategy is to implement early stopping, where the training process is halted based on the quality of the generated summaries or other exter- nal metries. Instead of solely focusing on maximizing rewards, we prioritize the quality of the generated sum- marics. Training halted when the summary quality reaches a certain threshold or shows no further improvement. This approach ensures that the generated summaries not only maximize rewards but also maintain a high level of quality.
RMS (pp(2
(gp12-)
rMMB (tumwe))
2
100
200
300 400
eoochs
Figure 9: Overoptimize against the reward model. After training 100k steps, the RM score (RMS) on the test data has high bias compared with the rPMS.
Recent research (Gao et al. 2022) has noted an interesting observation regarding larger policy models. has been found that as the size of the policy models increases, they become less susceptible to over-optimization against the RM. This suggests that scaling up the model size can potentially alleviate the over-optimization issue by introducing more complexity and capacity into the policy model, making it harder for the model to excessively optimize solely for rewards without considering the summary quality.
In summary, mitigating the risk of over-optimization in PPO and CPPO can be achieved through strategies such as training additional reward models, implementing early stopping, and considering larger policy models. These measures aim to strike balance between achieving high rewards and generating high-quality summaries, ensuring that the models generalize well and produce reliable results even on unseen data.
23

---

Published as a conference paper at ICLR 2024
J BROADER IMPACT
The broader impact of our proposed CPPO method is significant for both researchers and practitioners in the field of NLP. By addressing the limitations of existing RLHF-based LMs, we enable the continual alignment of these models with human preferences, opening up new possibílities for their widespread adoption and deployment.
One important implication of our work is the reduction of time and computational costs associated with retraining LMs. In many real-world scenarios, complete retraining is impractical due resource constraints and data privacy. By introducing sample-wise weights and enhancing policy learning while retaining valuable past experiences. CPPO offers a more efficient and practical alternative. This efficiency allows practitioners to keep LMs up-to-date with evolving human preferences without incurring the substantial overhead of retraining, making them more accessible and applicable across a range of applications.
The practical implications of our work extend beyond research and development. Industries that heavily rely LMs, such as customer service, virtual assistants, and content generation, stand to benefit from the continual alignment provided by CPPO. The improved performance and adaptability personalized and effective interactions with users, enhancing satisfaction and overall user experience. Additionally, CPPO's ability to align with human preferences consistently enables the development of more inclusive and fair AI systems that better understand and respect diverse user needs and values.
of LMs enable
In summary, our CPPO method has broad implications for the NLP community and beyond. By addressing the challenges associated with RLHF-bbase LMs, our approach offers a practical and efficient solution for continually aligning with human preferences while reducing retraining costs and preserving data privacy. These advancements promote the wider adoption and responsible use of LMs various domains, leading to more personalized, inclusive, and trustworthy Al systems.
24


Reviews:
[{"review_id": "nk6uqpYdKA", "reviewer": "Reviewer_qeM7", "summary": "This work studies the problem of continual learning from humans, where the signal from humans takes the form of a preference. The main challenge of continual learning is retaining past knowledge, while still able to maximize reward on the newly acquired preferences. To stabilize the training procedure, this work proposes \"stable learning\", favoring actions that are both a) high probability under the existing model -- i.e. retaining past knowledge and b) high reward for the new preferences acquired -- i.e. conforming to the new preferences.\n\nThese two aspects are turned into \"knobs\" \\alpha and \\beta, which are used to weigh parts of the objective function to either increase the effects of learning new knowledge or retaining old knowledge. The values of \\alpha and \\beta are adjusted by collecting roll-out samples and classifying them into 5 categories, normal, high performing, ... noise. Where each category has an influence on the adjustments on these two knobs.\n\nEmpirical results are promising.", "questions": "please provide confidence intervals and t-tests for table 8. It would be good to show the proposed method is significantly better than PPO.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "## originality : fair\nThe paper proposes a straight-forward (but nonetheless novel) idea to use samples in the continual learning process to modify the learning rates, whereby adjusting the rate in which the model retains old knowledge and learns new knowledge. \n\n## Clarity : good \nThe approach is clearly explained.", "weaknesses": "## Quality : less than ideal.\nIt is unclear if the proposed method is actually better than the baselines from table 8, as there is no confidence intervals being computed, nor is there statistical tests being performed. Depending on the results, the authors might have to conduct additional human evaluations, so that the confidence intervals \"pulls apart\". \n\nTable 8 is the only evaluation that is conducted directly against humans, it is the only \"non-proxy\" evaluation of the proposed method -- directly asking humans to rate the responses -- rather than evaluating it indirectly through a reward model.\n\nAs such, the authors should look to conduct evaluations more in the form of direct human ratings, to give readers who are not too familiar with the details of RLHF (such as myself) confidence that the approach works well \"end to end\", when evaluated directly against humans.\n\n## Significance : unclear\nAs someone not directly in the RLHF community, I leave this to other seasoned reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "909UcG2mTz", "reviewer": "Reviewer_Loz4", "summary": "This paper looks at a more ambitious form of RLHF that does not require complete retraining from scratch when new feedback is introduced as a result of human preferences that may vary over time across domains and topics. The authors propose to modify the PPO algorithm leveraged within RLHF training by designing an algorithm to accomplish 4 key desiderata: \n\n1. Examples with high reward and high generation probability have a high policy learning weight and high knowledge retention weight. \n2. Examples with high reward and low generation probability or high generation probability and low reward have a high policy learning weight and low knowledge retention weight. \n3. Examples with low reward and low generation probability have a low policy learning weight and low knowledge retention weight. \n4. All other examples are learned with default weightings. \n\nThe authors then conduct experiments comparing their proposed method favorably during continual RLHF training to generic PPO as well as relevant baselines and ablations.", "questions": "1. What is the statistical significance of the reported results across random seeds? \n2. What problem formulation is equation 4 a solution to? \n3. The key difference with generic RLHF seems to be that the optimization of the two terms is only over a subset of the data in equation 4. Why does it help us to essentially throw away data? Or is this motivated as part of a computational constraint from some implicitly considered problem formulation that is not spelled out? \n4. The hard cutoff based on hyperparameter k seems a bit weird to me. Towards what metric would the hyperparameter k be optimized for if it can't be equation 4 itself? This gets even weirder for me when the hard cutoff is then relaxed in equation 6. I don't get why it was ever introduced to begin with. \n5. Why does equation 6 make equation 5 easier to optimize? I buy that equation 6 is more general than equation 5, but this is the key motivation for this sense and it is really not clear to me why this would be the case. Especially considering that focusing on only a subset of the data would require less computation, which I presume is a major constraint of the implicit problem formulation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- I like the central motivation of this paper as I believe it is an important setting that has been under-explored to date in the context of RLHF. \n- The experimental results seem pretty good, suggesting that there may at least be some strong insights coming out of this paper about the impact of extreme examples of the efficacy of continual training. \n- Focusing on different treatment of extreme examples is a bit outside the norm within the continual learning context that is generally focused on either experience replay or knowledge retention strategies coming from different motivations. This gives the paper a certain degree of originality, but I feel it also makes justification of the particular approach more important, which is an area where I find the paper to be lacking. \n- There are a number of useful charts and tables throughout the text that help readers get the main idea when they may get confused.", "weaknesses": "- The continual RL problem solved by this paper is never really formulated. Equation 4 is proposed as an objective, but it is never made clear what problem it solves. This makes it feel a bit arbitrary to me. For example, see recent work proposing formal definitions of the problem [1], [2]. \n- In general I find section 3 prior to section 3.2.1 to be very hard to follow. A number of things are referred to as theoretically justified or derived, but as far as I can tell this is not true in any meaningful sense. I think I would be much more open to the positioning of the contribution if the authors just started with the desiderata of section 3.2.1 and explained the reason for each intuitively maybe with the aid of a little bit of math for each one (something that is currently lacking in my view). The paper could be better positioned in my view more as an exploration of the impact of these various intuitions and their impact on continual RLHF. This is because, in my mind, the current discourse does not really deliver on proposing this technique as a theoretically justified approach for RL. \n- Improvements in Table 6, Fig 5, and Table 8 seem to be there but it is not clear how significant these are or exactly why this would generalize across domains. I didn’t notice any analysis of statistical significance, which is odd because PPO (and most RL methods) are known to have high run to run variance in general, so it is very hard to take RL results seriously without this. \n\nBecause I worry about statistical significance and generality of the results and don't believe the formulation is well justified as presented, I lean towards rejection at this time. \n\n[1] Khetarpal, Khimya, et al. \"Towards continual reinforcement learning: A review and perspectives.\" Journal of Artificial Intelligence Research 75 (2022): 1401-1476.\n\n[2] Kumar, Saurabh, et al. \"Continual learning as computationally constrained reinforcement learning.\" arXiv preprint arXiv:2307.04345 (2023).\n\nDetailed Comments on Equation 5:\n\nA particular issue with the start of section 3 is with equation 5 and the discussion around it. What do you mean when you say “By introducing the actor-critic version, the clipped ratio, and the entropy bonus, we claim that Eq.(4) can be improved to (the derivation is detailed in Appendix Section B)”? What does \"improved\" mean in this context? Also going through Appendix Section B, it seems apparent to me that this is not a derivation of any particular theory/lemma/proposition or fact, rather it is just a series of steps that are not explained in the main text. For example, the authors mention improvement when they say: “In the PPO method, the objective is to maximize the expectation of the advantage function instead of the reward value. Hence, we improve the above objective as” -- here the \"improvement\" is either empirical or related to the theory about bias/variance. It is mentioned again when they say: “we introduce the knowledge retention penalty instead of the true KL divergence, we discuss the reason in lines 134-137 in our paper. Here, the above objective is improved as:”  -- here the improvement is based on the authors own empirical observations and computational justification. The authors also write \"Then we introduce the importance sampling like PPO, the above objective can be written as\" -- importance sampling is a deep theoretical topic related to off-policy optimization while having issues related to variance to implement in practice. I find the analysis of and flippant discussion of this component of the algorithm to be entirely inadequate in the appendix. It is also not even discussed in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tBzYrZ55Dy", "reviewer": "Reviewer_za1s", "summary": "The paper proposes a new method, CPPO, to align language models with human preferences in the continual learning setting, exempting the need to do LM retraining. Based on PPO, CPPO introduces a new imitation loss term to imitate the old policy outputs, and a weighting strategy is applied to balance the policy gradient loss and the imitation loss. The weighting strategy can be implemented in a heuristic way or a learnable way through the Lagrange function. As a result, in the continual learning setting, the policy (Language model) can seek a good trade-off between new task performance and old knowledge retention. The experiments show that the method outperforms all baselines and is close to the ChatGPT in terms of several metrics such as reference preference model score.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The work aims to address a practical and important issue, language model alignment in a dynamic human preference setting. \n2. The paper is well-written. \n3. The main idea and the motivation behind it make sense.\n4. The evaluation is thorough with reasonable metrics and adequate baselines.", "weaknesses": "I wonder about the performance of 1) $M_{\\pi2}$ on the Task-1 test set and 2) $M_{\\pi1}$ on the Task-2 test set. The result of the first experiment can show how much knowledge the model retains for the first task. In addition, the second experiment is supposed to show a mediocre result to prove the two tasks have a clear difference and that the experiment setting is indeed a continual learning setting. The performance gap of $M_{\\pi1}$ and $M_{\\pi2}$ on Task-2 can reveal to what extent the two data distributions mismatch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNRokadasb", "reviewer": "Reviewer_CUAu", "summary": "This paper studies the problem in the continual learning problem in the RLHF module of training language models. The authors propose continual PPO (CPPO), which utilizes a sample-wise weighting strategy based on the vanilla PPO algorithm to balance policy learning and knowledge retention. Experimental results show that CPPO has a good continual learning ability in the domain incremental learning summary task.", "questions": "see weeknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper studies the important problem of reinforcement learning with human feedback (RLHF) in a continual manner due to the limitation of computational cost or data privacy, which may have a great impact to the LLM community.\n- The paper is well written and easy to follow.\n- CPPO outperforms multiple baselines in the proposed summarization benchmark.", "weaknesses": "- The major contribution of this paper is to propose the problem of RLHF in a continual manner, however, the motivation and necessity of this problem is lack of detailed explanation and support, hindering readers’ understanding of its value. The authors should introduce more related work that retrain or finetune the LM when new data arrive. Furthermore, the storage and computational cost of different approaches should be analyzed, for example, the retraining methods [1], replay-based methods [2] and regularization-based methods [3].\n- The experiment part is weak.\n    - In the main experiment to evaluate the continual learning ability of CPPO in NLP tasks, only one benchmark Reddit TL;DR summarization is considered. What’s more, the length of the continual task stream is only 2, which is not enough for a solid evaluation of continual learning ability compared to related work [4][5][6][7].\n    - The proposed CPPO method can also be applied to other continual reinforcement learning tasks other than RLHF. Simpler conventional control tasks like DoorGym [4][5] should be included to further validate the soundness of CPPO.\n- The mathematical symbols and spellings need to be checked, for example,\n    - $A_i$ in Eq.(3) was not mentioned before.\n    - explanation of Eq.(4), k is a hyperparameter → $k$ is a hyperparameter.\n    - In Eq.(6), coefficients should be $C_1, C_2$, not $r_1, r_2$.\n    - Experiment settings: lkie → like.\n\nReference:\n\n[1] Yuntao Bai, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arxiv 2022.\n\n[2] David Rolnick, et al. Experience Replay for Continual Learning. NeurIPS 2019.\n\n[3] James Kirkpatrick, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.\n\n[4] Yizhou Huang, et al. Continual model-based reinforcement learning with hypernetworks. ICRA 2021.\n\n[5] Philemon Schöpf, et al. Hypernetwork-PPO for continual reinforcement learning. NeurIPS 2022.\n\n[6] Samuel Kessler, et al. Same state, different task: Continual reinforcement learning without interference. AAAI 2022.\n\n[7] Anonymous, et al. Scalable Language Model with Generalized Continual Learning. Summited to ICLR 2024. https://openreview.net/forum?id=mz8owj4DXu.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)