Prediction and Control in Continual Reinforcement Learning
Authors: Nishanth Anand
Doina Precup
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
continual reinforcement learning
lifelong learning
never-ending learning
prediction
control
multi-task learning
complementary learning systems
TL;DR: We propose to decompose the value function into two components and learning them at different timescales in continual reinforcement learning.

Abstract: Temporal difference (TD) learning is often used to update the estimate of the value function which is used by RL agents to extract useful policies. In this paper, we focus on value function estimation in continual reinforcement learning. We propose to decompose the value function into two components which update at different timescales: a _permanent_ value function, which holds general knowledge that persists over time, and a _transient_ value function, which allows quick adaptation to new situations. We establish theoretical results showing that our approach is well suited for continual learning and draw connections to the complementary learning systems (CLS) theory from neuroscience. Empirically, this approach improves performance significantly on both prediction and control problems.

Prediction and Control in Continual Reinforcement Learning
Nishanth Anand
Doina Precup
School of Computer Science
School of Computer Science
McGill University and Mila nishanth. anand@mail. mww.all..mmm..c...........
McGill University, Mila, and Deepmind dprecupêcs ctapliss.mmmllll............eeee
Abstract
Temporal difference (TD) learning is often used to update the estimate of the value function which is used by RL agents extract useful policies. In this paper, we focus on value function estimation in continual reinforcement learning. We propose decompose the value function into two components which update different timescales: permanent value function, which holds general knowledge that persists over time, and a transient value function, which allows quick adaptation new situations. We establish theoretical results showing that our approach is well suited for continual learning and draw connections to the complementary learning systems (CLS) theory from neuroscience. Empirically, this approach improves performance significantly on both prediction and control problems.
1
Motivation
Deep reinforcement learning (RI.) has achieved remarkable successes in complex tasks, e.g Go [40 41 but in narrow setting, where the task is well-defined and stable over time. In contrast, humans continually learn throughout their life and adapt to changes in the environment and their goals. This ability to continually learn and adapt will be crucial for general AI agents who interact with people order to help them accomplish tasks. One possible explanation for this ability in the natural world is the existence of complementary learning systems (CLS): one which slowly acquires structured knowledge and the second which rapidly learns specifics adapting to the current situation (26]. In contrast, in RL, learning is typically focused how to optimize return, and the main estimated quantity is one value function (which then drives learning a policy). As result, when changes occur in the environment, RL systems whether to forget past predictions in order to learn new estimates or to preserve old estimates, which may be useful again later, and compromise the accuracy on
faced with the stabilityplastice
dilemma [9]:
task.
The
dilemma in RL agnostic to the type of function approximation used to estimate predictions, and it is not simply side effect of the function approximator having small capacity. In fact, the problem exists even when the RL agent uses table to represent the value function. To this, consider policy evaluation different due to different rewards and transition probabilities.
sequence of tasks whose true value function convergent (TD) algorithm [43) aggregates information from all value functions in the task distribution implicitly, thereby losing precision in the estimate the value function for the current situation. A tracking TD learning algorithm [46] learns predictions for the current task, overwriting estimates of past tasks and re-learning from scratch for each new task, which can require a lot of data. Using a separate function approximator for each task, which is a common approach [23), requires detecting the identity of the current task to update accordingly, which can be challenging and limits the number of tasks that can be considered.
"corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

As an alternative, we propose CLS-inspired approach to the stability-plastiiity dilemma which relies on maintaining two value function estimates: a permanent one, whose goal is accumulate "baseline" knowledge from the entire distribution of information to which the agent is exposed over time, and transient component, whose goal to learn very quickly using information that relevant to the current cireumstances. This idea has been successfully explored in the context of RI. and tree search in the game of Go [39], but the particular approach taken uses domain knowledge to construct features and update rules. Our approach is general, simple, online and model-free, and can be used in continual RL. Our method is also orthogonal to various advancements in the continual RL field, and therefore, it can be combined with other improvements. Our contributions are:
A conceptual framework for defining permanent and transient value functions, which can be used to estimate value functions;
RL algorithms for prediction and control using permanent and transient value functions;
Theoretical results which help clarify the role of these two systems in the semi-continual (multi-task) RI. setting; and
Empirical case studies of the proposed approaches in simple gridworlds, Minigrid JellyBeanWorfe World (JBW) [3], and MinAtar environments (5T]
2 Background
Let S be the set of possible states and A the set of actions. At each timestep I, the agent takes action A, E A in state S, according to its (stochastic) policy S + (is((A). As a consequence, the agent receives reward R4+1 and transitions to
state S1+1. Let G. = Lk=t k R be the discounted return obtained by following x from step & onward, with < T < the discount factor. Let (s) Ex[(G S. = s). This quantity can be estimated with function approximator parameterized by w, for example using TD learning [43]:
=>==
W++1 f- W, + aro; V (S)),
where or is the learning rate at time I, 6, = Re+1 7 (S (S.) is the TD error (see Sutton and Barto [45] for details).
In the control problem, the agent's goal is to find a policy, x², that maximizes expected returns. This can be achieved by estimating the optimal action-value function, q" = mar, where qa
E, [G[| = s, A,- = a). e.g. using Q-learning [50]
"Continual RL" is closely related to transfer learning 48 meta learning 37 multi-task learning
10.. and curriculum learning [35]. A thorough review distinguishing these flavours can be found in Khetarpal et al. [24] We use Continual RL to refer a setup in which the agent's environment (i.e. the reward or transition dynamics) changes over time. We use the term semi-continual RL for the case when the agent can observe task boundaries (this is similar to the multi-task setting, but does not make assumptions about the task distribution or task IDs). In continual RL, unique optimal value function may not exist, so the agent has to continually track the value function in order learn useful policies 46 2].
Our work focuses the idea of two learning systems: one learning the "gist" of the task distribution, slowly, the other which is used for tracking can quickly adapt to current circumstances. We instantiate this idea in the context of learning value functions, both for prediction and control, but the idea can be applied broadly (for e.g. policies, GVFs). The concept of decomposing the value function into permanent and transient components was first introduced in model-based RL. [39], for the single task of learning to play Go, and using samples drawn from the model to train the transient value function. In contrast, we focus on general continual RL, with samples coming from the environment.
In continual RL, most of the literature is focused on supplementing neural network approaches with better tools, such designing new optimizers to update weights 44
19
12
4
building
new architectures [2
32 [2 using experience replay to prevent forgetting 34
36
22 g promoting
plasticity explicitly [28 29
using regularization techniques from continual supervised learning
23 20 (7] Our method is agnostic to the nature of the function approximator used, and therefore,
complementary these advancements (and could in fact be combined with any of these methods).
which use neural networks as the main learning system and experience replay secondary system. Another similar flavor is to use the task
Our approach has parallels with fast and slow methods [3] 7

---

distribution to learn good starting point, as in distillation- based approaches 49 18 or MaxQInit T But the latter are multi-task RL methods and their use in continual RL is limited.
Defining Permanent and Transient Value Functions
Inspired by CLS theory. we decompose the value function estimated by the agent into two, indepen- dently parameterized components: one which slowly acquires general knowledge, the other which quickly learns local nuances but then forgets. We call these components permanent value function, (-(P) and transient value function, Vi(") The overall value function is computed additively:
V(PT (s) Ve (s) VT (s),
(2)
A similar decomposition can be done to the action-valus function:
(PPT) a) = 0! (s, + Q (s,
(3)
Other approaches to combining the two components are possible but additive decomposition is the simplest.
3.1 Permanent Value Function
Similar to the role of the neocortex in the brain, the permanent value function should capture general structure in the value functions from the tasks that the agent has seen in its lifetime. It should also provide good baseline predictions for any task that the agent faces, in order to allow it to learn good estimates with fewer updates. To achieve this in scenario where the agent goes through sequence of tasks and knows when the task changes, we might store all the states visited during the current task in buffer and using them when the task finishes, update the permanent value function follows:
0k+1 + 0% + (v(PP( vs
(Sk). (St).
(4)
where ā is the learning rate and is the iteration index of the buffer. The permanent estimate takes a small step towards the new estimate of the value function in the above update rule: bootstrapping at slower timescale. We analyze this idea further in Sec.
3.2 Transient Value Function
The transient value function should compute corrections top of the estimates provided by the permanent value function, in order to approximate more closely the true value function of the current task. We implement this intuition by updating the parameters of transient value function
Wr+1 f- W. arde V ( (St),
where a the learning rate, 8, Ro+1 + (V(T)(( - ((T)(SS) and V V.TT) is the gradient of transient value function time f. Eq. (5) semi-gradient update rule, in the sense used to describe TDDlearning, as the gradient is with respect to the current estimate only and not the target. Since the permanent and the transient value functions
independently parameterized,
the semi-gramn cemi-gradient ends up being just V Vif (S.). This results in learning only a part of the value function that is not captured by the permanent value function. The transient value function is reset decayed after transferring the new information into the permanent value function (c.g. at the moment
when the task changes), similar to the functioning of the hippocampus 26 33
3.3 Learning Algorithm
We can easily turn the updates above into algorithms: blueprint for the prediction case is shown in Algorithm [. and the control version is in Appendix 3 The initialization and resets are done appropriately based on the function approximation used.
In the above algorithm, we assumed that the agent can observe task boundaries, though not any additional information (like task id). We made this assumption to facilitate the theoretical analysis presented in the next section. A fully continual version of the algorithms is presented and evaluated empirically in Sec. 6
3

---

Note that this algorithm is a strict generalization of TD learning: instead of setting to when the task changes, if we were to match the difference between the updated permanent value function and the transient value function instead, we would obtain exactly the TD learning predictions. Another way to get TD is by ini- tializing 0 to match the initialization of the TD learning algorithm and updating w only without resetting. We explore this connection formally in the next section.
Algorithm 1 PT-TD learning (Prediction)
1: Initialize: Buffer B, 8, w
2: for t : 00 do
3: Store St in B
4: Observe R S
5: Update using Eq.
6: if Task Ends then
7: Update 0 using B and Eq. 4
8: Reset transient value function, w
4 Theoretical Results
9: Reset B
10: end if
In this section, we provide some theoretical results aiming to understand the update rules of permanent and transient value function separately, and then we study the semi-continual RL setting (because the fully continual RL setting is poorly understood in general from a theoretical point of view, and even the tools and basic setup are somewhat contentious). All results are for the prediction problem with tabular value function representation; We expect that prediction results with linear function approximation can also be obtained. We present theorems in the main paper and include proof's in appendices B The appendix also contains analytical equations describing the evolution of the parameters based on these algorithms.
11: end for
4.1 Transient Value Function Results
We provide an understanding of what the transient value function does in the first set of results
(Theorems 1-4). We carry out the analysis by fixing (PP) and only updating V(T) using samples from one particular task. We use V. y((TD) and v.(PT) to denote the value function estimates at time t learned using TD learning and Alg. respectively.
The first result establishes relationship between the estimates learnt using TD and the transient value function update rule in 55.
Theorem 1. Vo (TD) (P), (T) = 0 and the two algorithms train on the same samples using the same learning rates, then Vt, V, (PT) v. (TD)
The theorem shows that the estimates learnt by our approach match those learnt by TD-learning in a special case, proving that our algorithm is strict generalization of TD-leaming.
The next two theorems together shows that the transient value function reduces prediction error. First we show the contraction property of the transient value function Bellman operator.
Theorem 2.
The expected target in the transient value function update rule a contraction-mapple Bellman operator: (T)) = + reward for each state written in vector form and P, is the transition matrix for policy w.
- V(P) + tPp (T), where Ta is expected one-step
We get the Bellman operator by writing the expected target in matrix form. We now characterize the fixed point of the transient value function operator in the next theorem.
Theorem 3. The unique fixed point of the transient operator is (I P-) (()),
The theorem confirms the intuition that the transient value function learns only the part of the value function that is not captured in the permanent value function, while the agent still learns the overall value function: (PT) (P) + V(T) = v(P) + - (__-)Px)--a - v Uz.
The next theorem relates the fixed point of the transient value function to the value function of different Markov Reward Process (MRP)
Theorem 4. Let M be an MDP in which the agent is performing transient value function updates for Define Markov Reward Process (MRP), M, with state space S {(Se, At, Se+1). Vt), transition dynamics P(z' (Ta2|1,00 a') P(s"|ss,,a) a'), Vz, S. S.'s'ss' E S, Va' E A and reward function R(x, a²) = R(s, a) + +>V(P)(s') - (s)...). Then, the fixed point of Eq. 5) is the value function of π in M.

---

The proof follows by comparing the value function of the modified MRP with the fixed point of transient value function. The theorem identifies the structure of the fixed point of the transient value function in the form of the true value function of a different MRP. This result can be used to design an objective for the permanent value function updates; For instance, to reduce the variance of transient value function updates.
4.2 Permanent Value Function Results
We focus permanent value function in the next two theorems. First, characterize the fixed point of the permanent value function. Then, we connect the fixed point of the permanent value function to the jumpstart objective [48] [ which defined the initial performance the agent in the new task before collecting any data. The latter is used to measure the effectiveness of transfer in a zero-shot learning setup, and therefore, is crucial measure generalization under non-stationarity, We make an assumption on the task distribution for the following proofs:
Assumption 1. Let MDP M = (S, R, Pr: denote task т. Then, 2t is the value function of task and let I,, denote the expectation with respect the task distribution. We assume there are N tasks and task t is i..dd sampled according to pr-
Theorem 5. Following Theorem 2 under assumption / and Robbins-Moree step-size conditions, the sequence of updates computed by Eq. contracts to a unique fixed point TE, [U>].
We use the approach described in Example 4.3 from Bertsekas and Tsitsiklis Details are given in Appendix 1..21
(5) to prove the result.
Theorem 6. The fixed point of the permanent value function optimizes the jumpstart objective, min 1 vr 1-112]
4.3 Semi-Continun RL Results
We present two important results this section which prove the usefulness our approach value function retention, for past tasks, and fast adaptability new tasks. For the proofs, assume that time t. both v(TD) and (((P have converged to the true value function of task and the agent gets samples from task from timestep + onward.
The first result proves that the TD estimates forget the past after seeing enough samples from new task, while our method retains this knowledge through ((P)
||VTT
Theorem 7. Under assumption / there exists ko such that Wk 2 ko. E,
>
E, [|| ([()) 11 Vi.
The proof uses the sample complexity bound from Bhandari et al. 6 (Sec. 6.3, theorem 1). Details are in Appendix B
Next, we prove that our method has a tighter upper bound on errors compared with the TD learning algorithm in expectation.
||u, - v (PT) 2
Theorem 8. Under assumption 1 there exists Ko such that Vk I ko. E,
has a
l+b
tighter upper bound compared with E, [| (TD))
Vi.
L+k
The proof uses the sample complexity bound from Bhandari et al. (6) (Sec. 6.3, theorem 1) to get the desired result as detailed in Appendix B.3.2] As - oc, the estimates both the algorithms converge to the true value function and their upper bounds collapse to 0. This result doesn't 't guarantee low error at each timestep. but it provides an insight on why our algorithm has low error.
To further understand the behaviour of our algorithm on a new task, we derive analytical expressions for mean squared error similar to Singh and Dayan [42 for both our algorithm and TD-learning algorithm (see Appendix B.3.3 and B.3.4). Then, we use them to calculate the mean squared error empirically on a toy problem. We observe low error for our algorithm every timestep for various task distributions (details in Appendix C.2
5

---

Experiments: Semi-Continual Reinforcement Learning
We conducted experiments in both the prediction and the control settings on range of problems. Although the theoretical results handle the prediction setting, we performed control experiments to demonstrate the broad applicability of our approach. In this section, we assume that the agent is faced with sequence of tasks in which the transition dynamics are the same but the reward function changes. The agent knows when the task has changed. In the sections below, we focus on the effectiveness of our approach. Additional experiments providing other insights (effect of network capacity, sensitivity to hyperparameters, and effect of task distribution) are included in the Appendix and
(Sec.
Baselines: Because our approach is built on top of TD-Jearning and Q-learning, we primarily use the usual versions of TD learning and Q-learning baselines, which can be viewed as emphasizing stability. We also include a version these algorithms in which weights are reset when the task switches, which creates significant plasticity. We include more baselines for the continual RL experiments, which are described Sec. 6 The results for all algorithms are averaged across 30 seeds and shaded regions in the figures indicate 90% confidence interval We use the same procedure to optimize hyperparameters for all algorithms, detailed in Appendix (Sec.
5.1 Prediction experiments
We ran five experiments on 3 types of navigation tasks, with various function approximators. The policy being evaluated was uniformly random all experiments. We used root mean squared value error (RMSVE) (MSE) on the other tasks.
performance measure the online task and the expected mean squared error
Discrete grid: is a 5x5 environment depicted in Fig. la which has four goal states (one in each corner). The agent starts in the center and chooses an action (up, down, left or right). Each action moves the agent one step along the corresponding direction. Rewards are O for all transitions except those leading into goal state. The goal rewards change as detailed in Table C.I] so the value function also changes shown in Fig. C.I For the linear setting, each state represented using features: five for the row and five for the the same
column. The permanent and transient value functions features. For the deep RL experiment, instead of receiving state information, the agent receives 48x48 RGB images. The agent's location is indicated red, and goal states are colored in green shown in Fig. 1a Goal rewards multiplied by 10 to diminish the noise due to the random initialization the value function. For the tabular and linear setting, we run 500 episodes and change the goal rewards after every 50 episodes. For the deep RL experiments, goal rewards are switched every 100 episodes. The discount factor is 0.9 in all cases.
(a) Discrete grid. (b) Minigrid.
Figure I: Environments (PE).
run 1000 episodes and
(Modle
Inuge
Veas
Sricda
2,31 2002
Episca
Figure 2: Prediction results on various problems. Solid lines represent RMSVE on the current task and dotted lines represent MSE on other tasks. Black dotted vertical lines indicate task boundaries.
Continuous grid: is continuous version of the previous task. The state space is the square [0, x [0, 1]. with the starting state sampled uniformly from [0.45, 0.55] x [0.45, 0.55]. The regions within 0.1 units (in norm-l) from the corners are goal states. The actions change the state by 0. units along the corresponding direction. To make the transitions stochastic, we add noise to the new state,
We report standard distribution confidence intervals with z-score of 1,645. We use μ o. to represent the shaded region.

---

sampled uniformly from [-0.01, 0.01] X [-0.01, 0.01). Rewards are positive only for transitions entering a goal region. The goal reward for each task given in Table [ In the linear function approximation setting we convert the agent's state into feature vector using the radial basis function as described 455 and Appendix C.I.I We run 2000 episodes, changing goal rewards every 200 episodes. For evaluation, we sample 225 evenly spaced points in the grid and estimate their true values by averaging Monte Carlo returns from 500 episodes across 10 seeds for each task. Here, 0.99.
Minigrid: We used the four rooms environment shown in Fig. lb Each room contains one type of item and each item type has a different reward and color. Goal states are located in all corners. The episode terminates when the agent reaches a goal state. The agent starts in the region between cells (3,3) and (6,6) and can move forward, turn left, or turn right. The agent perceives 5x5 (partial) view in front of it, encoded one-hot. The rewards for picking items change as described Table C.2 and the discount factor is 0.9. We run 750 episodes and change rewards every 75 episodes.
We provide the training and architecture details for the two deep RL prediction experiments in Appendix (Sec. C.I.I and C.I.2) due to space constraints.
Observations: The results are shown in Fig. 2 Solid lines indicate performance on the task currently experienced by the agent (i.e., the online task) and the dotted lines indicate the performance on the other tasks. The reset variant of TD-leaming performs poorly because all learning progress is lost when the weights are reset, leading to the need for many samples after every task change. TD-learning also has high online errors immediately after a task switch, its predictions are initially biased towards the past task. As the value function is updated using the online interaction data, the error on the current task reduces and the error other tasks rises, highlighting the Ktaiilit-pplasiiiie problem when using a single value function estimate. Our algorithm works well on both performance measures. The permanent value function learns common information across all tasks, resulting in lower errors on the second performance measure throughout the experiment. Also, due to the good starting point provided by the permanent value function, the transient component requires little data to make adjustments, resulting in fast adaptation on the new task. In our method, the permanent component provides stability and the transient component provides plasticity, resulting in a good balance between the two. We also plotted the heatmap of the value estimates, shown in Fig. C to understand the evolution of the predictions for TT>-lernnig and our method which confirms the observations made so far.
5.2 Control experiments
To test our approach complement Q-learning 3 conducted tabular gridworld experiment and a Minigrid experiment using deep neural networks.
G1
Meged Crso
nos P1.00 purd
Owarri
(a) Toy grid.
(b) Tabular results.
(c) Minigrid.
(d) DRL results.
Figure 3: The environments and the corresponding results in the control setting. (b) Tabular results:
episodic returns are plotted. (d) Deep RL results: returns smoothened across 10 episodes is plotted.
Tabular experiment:
We use the 6x6 grid shown in Fig. 3a with two goals located in the top right corner. The agent starts in the bottom left corner and can choose to go up. down, left, or right. The action transitions the agent to the next state along the corresponding direction if the space is empty and keeps it in the same state if the transition would end up into wall obstacle. The action has the intended effect 90% of the time; otherwise, is swapped with one of the perpendicular actions. The agent receives reward of +1 and -1 for reaching the two goal states respectively, and the rewards swap periodically, creating two distinct tasks. The discount factor is 0.95. We run 500 episodes and change the task every 50 episodes.
Minigrid experiment: We use the two-room environment shown Fig. 3c the bottom row of the bottom room and there are two goal states located in the top room. The goal
The agent starts in
7

---

rewards are +5 and O respectively, and they flip when the task changes. We run 2500 episodes, alternating tasks every 500 episodes. The other details are as described in the previous section.Further details on the training and architecture are described in Appendix (Sec. C.1.3).
The results are shown in Fig. 3 The reset variant of Q-learming has to learn good policy from scratch whenever the task changes, therefore, it requires many samples. Since, the goal rewards are flipped from one task to another, Q-Jearning also requires many samples re-adjust the value function estimates. In our method, since the permanent component mixes action-values across tasks, the resulting bias is favorable, and makes fairly easy to learn the corrections required to obtain overall good estimates. So, the transient component requires comparatively few samples to find a good policy when the task changes.
Observations:
Continual Reinforcement Learning
So far, we assumed that the agent observes task bound- aries: This information is usually not available in con- tinual RL. One approach that has been explored in the literature is to have the agent infer, or learn to infer, the task boundaries through observations (e.g. see [T7 23)). ). However, these methods do not work well when task boundaries are not detected correctly, or simply do not exist. We will now build on the previous algorithm we proposed to obtain a general version, shown in Alg. 2 which is suitable for continual RL.
Algorithm 2 PT-Q-learning (CRL)
1: Initialize: Buffer B, 0, w, h; À
2: for 0 + oo do
3: Take action A,
4: Store S., Ar in
5: Observe R4+1, S1+1
6: Update w, using Eq. 8
7: if mod(t, k) = 0 then
8: Update using B and Eq.
Our approach maintains the permanent and transient components, but updates them continually, instead of waiting for task changes. The permanent component updated every l steps (or episodes) using the samples stored in the buffer B. And, instead of resetting the transient component, it is now decayed by factor À E [0. Here, & and À are both hyperparameters. When A = O and & the duration between task switches, obtain the previously presented algorithm as a special case. In general, k and À control the plbilittyplasticiiyy trade off. Lower À means that the transient value function forgets more, thus retaining more plasticity. Lower is introduces more plasticity in the permanent value function, as its updates are based on fewer and more recent samples. The algorithm still retains the idea of consolidating information from the transient into the permanent value function.
9: W++1 t- Aw
10: Reset B
11: end if
12: end for
The effect of k and A: To illustrate the impact of the choice of hyperparameters, we use the tabular task shown in Fig. [a| The details of the experiment are exactly as before, but the agent does not know the task boundary. We run the experiment for several combinations of & and À for the best learning rate. The results are shown in Fig. 4 As expected, we observe that the hyperparameters k and À have co-dependent effects to a certain degree. For small values of k, large values of A yield better performance, because the variance in the permanent value function is mitigated, and the transient value function in fact is more stable. For large values of k. the transient value function receives enough updates before the permanent value function is trained, to attain low prediction error. Therefore, the updates to the permanent value function effective, and the transient predictions can be decayed more aggressively. More surprisingly. if the frequency of the permanent updates is significantly higher or significantly lower than the frequency of task changes, the overall performance is marginally affected, but remains better than for Q-learning (indicated using black dotted line) for most and À values. In fact, for each k; there is at least one value for which the performance of the proposed algorithm exceeds the Q-learning baseline. So, in practice. we could fix one of these hyperparameters beforehand and tune only the second. We suspect that the two parameters could be unified into a single one, but this is left for future work.
6.1 Experiments
We test our algorithm on two domains: JellyBean World (JBW) 31 and MinAtar [5T]. We chose these domains because of the relatively lighter computational requirements, which still allow to
8

---

sweep a range of hyperparameters and to report statistical results averaged over 30 seeds. We include
architecture and further training details in Appendix (Sec. C.I.5 C.1.6).
Jelly Bean World is a benchmark to test continual RI. algorithms [3 The environment is two-dimensional infinite grid with three types of items: blue, red, and green, which carry different rewards for different tasks as detailed in Sec. C.I.5 In addition to reward non-stationarity. the environment has spatial non-stationarity as shown in Fig. Sa The agent observes an egocentric 1x11 RGB image and chooses one of the four usual navigation actions. We run 2.1M timesteps, with the task changing every 150k steps. The discount factor is 0.9.
Anaryate
2.43
Mexn
MinAtar is standard benchmark for testing single-task RI. algo- rithms [51]. We use the breakout, freeway, and space invaders envi- ronments. We run for 3.5M steps and setup a continual RL. problem, by randomly picking one of these tasks every 500k steps. We stan- dardize the observations across tasks by padding with Os when needed and we make all the actions available the agent. This problem has non-stationarity in transitions, rewards and observations making it quite challenging. The discount factor is 0.99. More details are included Appendix (Sec. C.I.6).
0.3
Figure 4: À and ls analysis.
Baselines: We compare our method with DQN, two DQN variants, and a uniformly random policy, For the first baseline, recently as a viable option for continual RI. [8] The second baseline, DQN with multiple heads, uses common trunk to learn features and separate heads learn Q-values for the three tasks. This requires knowing the identity of the current task to select the appropriate head. We use this baseline because it is standard approach for multi-task RL, and intuitively, it provides performance. This approach is not suitable if the number of tasks is very high, or if the problem is truly continual and not multi-task. For our approach, we use half the number of parameters that of DQN for both permanent and transient value networks to ensure the total number of parameters across all baselines are same hence the name PT-DQN--055x
use large experience replay buffer, approach which was proposed
upper bound on the
k
(a) JellyBeanWorld.
(b) JellyBean World Results.
MinAtar.
(d) MinAtar results.
Figure 5: (a) JellyBean World task. (b) Results: JellyBeanWorld. (c) MinAtar tasks. (d) Results: MinAtar.
Results are presented in Fig. 1eg-5lanc and the pseudocode of PT-DQN is included in Appendix C.I.2 For JBW experiment, we plot the reward obtained per timestep over 10k step window (reward rate) as a function of time. For MinAtar,
plot the return averaged over the past 100 episodes. Our method, PT-DQN-0.5x, performs better than all the baselines. The performance boost is large in the JBW experiment, where the rewards are completely flipped, and still favourable in the MinAtar experiment. DQN's performance drops over time when needs to learn conflicting Q-values, as seen in the JBW results. DQN also needs large number of samples overcome the bias in its estimates when the task changes. Surprisingly, muuli-header DQN performs poorly compared to our method, despite having the additional task boundary information. The weights of the trunk shared, and they vary continuously over time. Therefore, the weights at the head become stale compared to the features, which degrades performance. These findings may be different if the trunk were pre-trained and then fixed, with only the heads being fine-tuned, or if the networks were much bigger (which is however not warranted for the scale of our experiment). Augmenting DQN with a large experience replay is detrimental for general non-stationary problem, based on our experiments. This is because the samples stored in the replay buffer become outdated, using these samples hampers the agent's ability to track environment changes.
'Strictly speaking, DQN-base baselines use more parameters compared with ours because their target networks are 2x larger than the target network of the transient value function.

---

7 Discussion
Designing agents that can learn continually, from a single stream of experience, is crucial challenge in RL. In this paper, we took a step in this direction by leveraging intuitions from complementary learning systems, and decomposing the value function into permanent and transient components. The permanent component is updated slowly to learn general estimates, while the transient component computes corrections quickly to adapt these estimates to changes. We presented versions adapted to both semi and fully continual problems, and showed empirically that this idea can be useful. The theoretical results provide some intuition on the convergence and speed of our approach.
The non-stationarity in our experiments is much more challenging, including arbitrary reward changes, well dynamics and observation changes, compared to other works, which use specific structure, or rely on a curriculum of tasks (thereby limiting non non-stationarity). Our approach provided improvements in performance compared the natural TD-learning and Q-learning baselines.
Our approach performs fast and slow interplay at various levels. The permanent value function is updated a slower timescale (every le steps at the end of the task); The transient value function is updated at faster timescale (every timestep). The permanent value function is updated to capture some part of the value function from the agent's entire experience (generalization), which is a slow process; The transient value function adapts the estimates to a specific situation, which is a fast process. The permanent value function updated using smaller learning rate (slow), but the transient value function is updated using a larger learning rate (fast) [3]
7.1 Limitations and Future Work
While we focused on illustrating the idea of permanent and transient components with bbsed algorithms with discrete action spaces, the approach is general and could in principle be used with policy gradient approaches. One could use the proposed value function estimates to compute the gradient of the policy (represented as a separate approximator) in actor-critic methods. Another possibility is have "permanent" policy, which is corrected by transient component to adapt to the task at hand. These directions would be interesting to explore. Our idea can be extended to learning General Value Functions
47 too.
Our theoretical analysis is limited to piece-wise non-stationarity, where the environment is stable for some period of time. An interesting direction would be to extend our approach other types of non-stationarity, such as continuous wear and tear of robotic parts.
Another important future direction is to consider distinct feature spaces for the permanent and transient components, as done Silver al. [38 This would likely increase the ability to trade off stability and plasticity. Another direction to better control the stability and plasticity trade-off is by learning the frequency parameter & and the decay parameter À using meta objectives [[6]; We suspect there's a relation between the two parameters that can unify them into a single one.
Our approach can also be used bridge offline pre-training and online fine-tuning of RL agents. The permanent value function can be learnt using offline data and these estimates can be adjusted by computing corrections using the transient value function using online interaction data.
Acknowledgments and Disclosure of Funding
We are grateful to NSERC, CIFAR, and Microsoft Research grants for funding this research. We thank Jalaj Bhandari, Maxime Webartha, Matthew Reimer, Blake Richards, Emmanuel Bengio, Harsh Satija, Sumana Basu, Chen Sun, and many other colleagues at Mila for the helpful discussions; Will Dabney, Harm Van Seijen, Rich Sutton, Razvan Pascanu, David Abel, James McClelland and all of the 2023 Barbados workshop participants for their feedback the project; the anonymous reviewers for the constructive feedback on the final draft. This research was enabled in part by support provided by Calcul Quebec and the Digital Research Alliance of Canada.
10

---

References
[1] David Abel, Yuu Jinnai, Sophie Yue Guo, George Konidaris, and Michael Littman. Policy and value transfer in lifelong reinforcement learning. In International Conference Machine Learning, pages 20-29. PMLR, 2018.
[2] David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. arXiv preprint arXiv:2307.11045 2023.
[3] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general contin- ual learning method based on complementary learning system. arXiv preprint arXiv:2201. 126644, 2022.
[4] Eseoghene Ben-Iwhiwhu, Saptarshi Nath, Praveen K Pilly, Soheil Kolouri, and Andrea Soltog- gio. Lifelong reinforcement learning with modulating masks. arXiv preprint arXiv:2212.11110, 2022.
[5] Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynaicc programming. Athena Scientific, 1996.
[6] Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory, pages 1691- 1692. PMLR, 2018.
[7] Matthew Botvinick, Sam Ritter, Jane X Wang. Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5):408 422, 2019.
[8] Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Task- agnostic continual reinforcement learning: In praise of simple baseline. arXiv preprint 2rXiv:2205.14445, 2022.
[9] Gail A Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing, 37(1): 54-115, 1987.
[10] Rich Caruana. Multitask learning. Machine learning, 28(1):41-7) 1997.
[11] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for openai gym, 2018.
[12] Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint (rXiv:2108.06322 2021.
[13] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.0277) 2016.
[14] Mohamed Elsayed and A Rupam Mahmood. Utility-based perturbed gradient descent: An optimizer for continual learning. arXiv preprint arXiv:2302.0328) 2023.
(15) Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. International conference on machine learning, pages 1126-1135. PMLR, 2017.
[16] Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. Bootstrapped meta-learning. arXiv preprint arXiv:2109.04504 2021.
[17] João Gama. Indré Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
[18] Valentin Guillet, Dennis George Wilson, and Emmanuel Rachelson. Neural distillation state representation bottleneck reinforcement learning. Conference on Lifelong Learning Agents, pages 798-818. PMLR, 2022.

---

[19] Andrew Jacobsen, Matthew Schlegel, Cameron Linke, Thomas Degris, Adam White, and Martha White. Meta-descent for online, continual prediction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33/0, pages 3943-3950, 2019.
[20] Khurram Javed and Martha White. Meta-learning representations for continual learning, Ad- vances in Neural Information Processing Systems, 32, 2019.
[21] Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. In International Conference on Machine Learning, pages 2497-2506. PMLR, 2018.
[22] Christos Kaplanis, Claudia Clopath, and Murray Shanahan. Continual reinforcement learning with multi-ting melti-timescale replay. arXiv preprint arXiv:2004.075.0 2020.
[23] Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J Roberts. Same state, different task: Continual reinforcement learning without interference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36/7, pages 7143-7151, 2022.
[24] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual rein- forcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75: 1401-1476, 2022.
[25] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-3526 2017
et al.
[26] Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 2117)55 2016.
[27] Kevin Lu, Igor Mordatch, and Pieter Abbeel. Adaptive online planning for continual lifelong learning. arXiv preprini /9/2:0119/201/11 2019.
[28] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.095560, 2022.
[29] Exgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pages 16828-16847. PMLR, 2022.
[30] Yangehen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning sparse representations online. arXiv preprint arXiv:1911.0800 2019.
[31] Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint (xXi:20022.06366, 2020.
[32] Sam Powers, Eliot Xing, and Abhinav Gupta. Self-activating neural ensembles for continual reinforcement learning. In Conference on Lifelong Learning Agents, pages 683-704. PMLR, 2022.
[33] Blake A Richards and Paul W Frankland. The persistence and transience of memory, Neuron, 1776)1177 2017.
[34] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint 1111v:1810.119/00 2018.
[35] Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77-104 1997.
[36] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experi- ence replay for continual learning. Advances Neural Information Processing Systems, 32, 2019.
12

---

[37] Jürgen Schmidhuber. Evolutionary principles in seff-referential learning, or on learning how to learn: the meta-meta... hook. PhD thesis. Technische Universität München, 1987.
[38] David Silver, Richard Sutton, and Martin Müller. Reinforcement learning of local shape in the game of go. In IJCAI, volume 7, pages 1053-1058, 2007.
[39] David Silver, Richard S Sutton, and Martin Müller. Sample-base based learning and search with permanent and transient memories. In Proceedings of the 25th international conference on Machine learning, pages 968 003-975,22 2008.
[40] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484-48) 2016.
[41] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354-3555 2017.
[42] Satinder Singh and Peter Dayan. Analytical mean squared error curves for temporal difference learning. Machine Learning. 32(1):5-40, 1998.
[43] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.
[44] Richard S Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-dela. In AAAI, pages 171-176. San Jose, CA, 1992.
[45] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[46] Richard Sutton, Anna Koop, and David Silver. On the role of tracking in stationary envi- ronments. In Proceedings of the 24th international conference on Machine learning, pages 871-878, 2007.
[47] Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Systems-Volume 2, pages 761-768, 2011.
[48] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey, Journal of Machine Learning Research, 10(7), 2009.
[49] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
[50] Christopher John Comish Hellaby Watkins. Learning from delayed rewards. 1989.
[51] Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint 1903.0390 2019.
13

---

Appendix
A
Control algorithm
The action-value function can be decomposed into two components as:
(s, = 2 (s, a) + ( (s,a),
(6)
The updates for permanent value function are:
Ox+1 + 02 + T. (Q(Sk, A&) Q A&) Vo)) A&).
where ā is the learning rate. An update rule similar to Q-learning can be used for the transient value function:
W++1 We + agde V ( (S,, At),
(8)
where 6, Re+1 + max, (S (PP)) (St, At) the TD error.
Algorithm PT-Q-learning (Control)
Initialize: Buffer B, 0,
2: for :: 0 -> cc do
3: Take action A,
4: Store S,, A; in B
5: Observe Re+1, S.+1
6:
Update using Eq.
8
7:
if Task Ends then
8: Update 8 using B and Eq.
9: Reset transient value function, w
10: Reset B
11: end if
12: end for
Proofs
B.1
Transient Value Function Results
We carry out the analysis by fixing V(P) and only updating (T) using samples from one particular task. We use (-(D)) and V- (((T)) to denote the value function estimates at time / learned using TD learning and Alg. [ respectively.
Theorem B.1.1. ((DD) v(n) (IT) 0 and the two algorithms train the same samples using the same learning rates, then Vt, ((PT) (TD)
Proof. We use induction to prove this statement.
Base case: When 0, the PT-estimate of state Vo (P (s) = (() + () (s) And, the TD estimate of state is v) (TDD) equal.
(s) = (s) Therefore, both PT and TD estimates are
Induction hypothesis: PT and TD estimates are equal for some timestep & for all states. That is, (PPT V(P) V. ((())s) (s) v (T (x).
k
Induction step: PT estimates at timestep + 1
(PP
(s)= v- (T( =s) (TT) (s) + arti (BR+1 + (((())))41) (,+((())))) (s) ((
14

---

k (s) + ax+1 (Ra-1 + +7 (Sk+1) ((P (s)), ((TD) (S(+1) - ((TD) (s)).
N ()T a2+1 + v/2
(TD) 1+1 (s).
The penultimate step follows from the induction hypothesis completing the proof.
Theorem B.1.2. The expected target in the transient value function update rule is a contraction- mapping Bellman operator: ((T) Tr + +P, -VP) + -P-V(T)
Proof. The target for state at timestep & in the transient value function update rule is R (((()) - (s) + (S+t)) The expected target is:
((T))(((s) E, R441 (( - (s) + +V00)) =] = (8) + 7 (P, (P)) -
We get the operator by expressing the above equation in the vector form:
T(y(()) Tx PP, v(P) + PTVV(T)
For contraction proof, consider two vectors u, v E IRIS.
T 1 77))
||1= -P,
+ 7P, -(P)
+ -P)
vVP) +
00
-1777x 0)1
S ||u - -ll-vllax ullac
Therefore, the transient value function Bellman operator is a contraction mapping with contraction factor y-
Theorem B.1.3. The unique fixed point of the transient operator is - (P- V(M).
Proof. We get the fixed point, (T) by repeatedly applying the operator to a vector e:
V = -r + +PPP (P) v(P) + *PT7,
Ta + +P,,V(17 -v() + +>Pal1s +7p,V(0) - ((P) + +PPRU),
-. - t? + (TPp)22(P)) - v(P) (rPp)²u,
-Ya + + + (>Pp) + +... (xPp)'(()) - v(P) + (>Px'"'
(I-)Px) -r,
We applied the operator to v in steps 1 to 4 and used the property lim )' + 0 in the penultimate step yPP is sub-stochastic matrix. Many terms cancel out as it is telescoping sum.
Theorem B.1.4. Let M be an MDP in which the agent is performing transient value function updates for T. Define an MRP, M, with state space S {(St, Ar, St+1), Vt), transition dynamics P( P's"" "','ss'a'),, a'), Vx, xf S,4s' s" E &, Va' E A and reward function R(x, a.) R(s,a) (V (()) Then, the fixed point of Eq. is the value function of in M.
Proof. The state space of the new MDP M is made up of all possible consecutive tuples of (S,, S44) pairs The rewards in this MDP are defined as R(I,a') R(s,a) (V(P)(s') (s),,s, Vr š, A. The transition matrix P(x'|x, a') prob(='lz, a') prob(s', a', as s',a') aa)== =prob(s"|ss ss's''')))))) = P(s"|s', Is', a') using Markov property of M. Then, the value state as) is:
(x) = U (1, (s,a, = E, (2 ss),
15

---

where Ze - RR-1 is the returns starting from state X; (s. a, s'). Without loss of generality, consider sample trajectory generated according to π. 7k (Sp = s, Ao = a, R s', A1, R2, S2, A2, R....., RT, ST) where ST is the terminal state. The returns n of this trajectory is:
t =
* Rk+1
0=
7 (RE+1 + (V(P)(S -VPP)
y Rk+1 (50
= G (s).
Returns of every trajectory end up being the difference between the returns in the original MDP, G and the permanent value function. Most importantly, these returns only depend the state and it is the same for all a and s' choices following 8. The value of state x (8, a, 8') can then be simplified using this observation:
Uy as, s') =E, [Z, XX, (s, a, s'),
= FE+[Ge - (s))s = =, A, = S,1 = =
(s).
B.2 Permanent Value Function Results
We focus permanent value function in the next two theorems. First, characterize the fixed point of the permanent value function. We use assumption I on the task distribution for the following proofs:
Theorem B.2.1. Following Theorem 2 under assumption 7 and Robbins-Monro step-size conditions,
the sequence of updates computed by Eq. 4 contracts to a unique fixed point E, [vr].
Proof. The permanent value function is updated using Eq. (4) and samples stored in the buffer B at the end of each task. This update rule can be simplified as:
(s) - (P) + Ch
(V) (PP) (s)).
(()) av (s))
(3))
1=1
+ (s) (s) (s))
(1) - ((
where I. is the length the buffer, I is the indicator function, I.(s) is the number of samples for state in the buffer B, (s) Ls). The above update is performed once for every state in B at the end of each task. We assume that the buffer has enough samples for each state so that all states are updated. This can be ensured by storing all the samples in B for long but finite period of time. The updates performed to permanent value function until that timestep will not affect the fixed point since we are in the tabular setting.
Let the sequence learning rate (s) obey the conditions: D (s) oo and Dx a² 0.
We rewrite the PM update rule by separating the noise term:
v(P) + (P(P) (s) + (v(s)(V (s) ((()) (P))
16

---

(s) ( - (s) + (sP = (s) (() (s) - v/ ((P)(())))))) (8) + +
Let Fk denote the history of the algorithm. Then, (1) E|V (M) E[V E [b)]] =0
and under the assumption [) S A + BV (P) (s) (finite noise variance), we conclude that the permanent value function converges to E[V;"1 (s)]. But Elv, [s)] contracting towards
(s),, therefore, the permanent value function is contracting towards E (s))
Theorem B.2.2. The fixed point of the permanent value function optimizes the jumpstart objective, = arg min el s E 1-
Proof. The jumpstart objective function is given by J arg min _ - or vr11)) Let 1 1E ur 11112. We get the optimal point by differentiating K with respect and then equating
to 0:
1 E, I||u Uy (2]
VaK V.
Er Vu ||u - 1+12)
2
1 E. 12(u 1T)] )
2
u E, [1T]
0 = u E [r]]
E, [u>]
Therefore, the fixed point of the permanent value function also optimizes the jumpstart objective.
B.3 Semi-Continual RL Results
We assume that at time f, both (TTD) and (PT) have converged to the true value function of task i and the agent gets samples from task from timestep + onward for the following theorems.
Theorem B.3.1. Under assumption there exists ko such that Vk 2 Ku. E, V(TP) vz E. ([( vall2. Vi.
Proof. Let [|[(P) 1/ D;, which is a constant because we are updating transient value function only. Let ||u, - vili A., denote the distance between value functions of two tasks i and Clearly, Ar, A.. + 0 and A,, 0. And, let Erri" || denote prediction error with respect to the previous task at timestep / for TD estimates.
At timestep ((T)) ((TD) Us. The agent starts getting samples from task 7 from timestep t + 1. Using the result from Bhandari et al. [6]:
|VTP) 1r 1| < exp
-1 -22) }
4
((1 (1-
cxp
Ar.i0
since the smallest eigenvalue of the feature matrix 1 for tabular approximations, Let C'
exp - }. Now we can apply the above result recursively for timesteps to get
((TD)
Ur < C* A
17

---

Then the error with respect to the past task after k timesteps is,
Erri 2 Ar, C* C* A1, C*).
Taking the expectation with respect to r gives
E, [Errif
Σ pr PrErrite
PrAA,,(1-Ch)
21- (1-
Now will find timestep ky such that vk ko E, Er which - (()) [A = D.. Then,
Dj. Let k denote the timestep for
C+: [A.1] = Di,
(1 Chi) = Dr
E, [A,A
Es A Di
Jhe
Fr [Ari)
E A r D; E, [Ar,]]
h;
log C' log
So, after time / + ki, (P) is a better estimate of the value function of the past task i than the current TD estimates. Then, by letting ko = max, k; we have E, B.rri > D,, vi.
Note that the above result depends on the fact that k exists, which only possible when E, AA, D; O. We will now show that there exists at least one solution, the fixed point of the permanent value function (E, [v.)), which satisfies this constraint.
E+
||E, [o; 11 (Using Jensen's 's inequality)
|| Ex
[x+]11
Then,
For (rrs) Dj 10u [u=11=1 Dj 0.
The final step follows from the fact that D; = || - vall2 [|, v.ll2 for (() fixed point.
Next, we show that our method has tighter upper bound on errors compared with the TD learning algorithm in expectation.
Theorem B.3.2. Under assumption there exists ko such that Vk: s ka, E, ||0- v((PT) | has
tighter upper bound compared with E, ||,-- (TD))
Vi.
Proof. We assume that (() is converged to its fixed point, Er [ur] (Theorem complexity bounds from Bhandari al. [6] for TD learning algorithm, we get:
5
Using sample
E, || pr - (TD) 2+
< E,
exp
(1 (1
?w
T
2r
V
(TD)
(1
7W
exp
T
E- V
18

---

We know that our algorithm can be interpreted as TD learning with ((P) as the starting point (see Theorem Using this relationship, we can apply the sample complexity bound to get:
(PT) U++
{
(1 w)
|0+ -1
Er
c+k
S
Er
exp
2
(1 2
T ET | vi(2 2
exp
4
For & 0, we know that our algorithm has lower error in expectation (see Theorem 6 that is Therefore, the error bound for our algorithm is E, u. -
E,
|, V
tighter compared with that of TD learning algorithm. As k + oo, the right hand side of both the equations goes 0, which confirms that both algorithms converges to the true value function.
The theorem shows that the estimates for our algorithm lie within a smaller region compared to that of TD learning estimates. This doesn't guarantee low errors for our algorithms, but we can expect to be closer to the true value function.
The next theorems contain analytical MSE equations for TD- TD-Jearning and for our algorithm. Our analysis is similar to that of 42). but we don't i assume that rewards are only observed in the final transition. Analytical MSE equations allow calculating the exact MSE that an algorithm would incur at each timestep when applied to problem, instead of averaging the performance over several seeds, which is the first step towards other analyses such as bias-variance decomposition of value estimates, error bounds and bias- variance bounds, step-size adaptation for the update rule, etc. We use these equations to compare our method against TD-learning when varying the task distribution diameter (Sec.
We assume that the agent gets N i.i.d samples (s, r, s') from task T in each round, where is sampled according to the stationary distribution d,. Also, the agent accumulates errors from all the samples and updates its estimates at the end of the round Let Exeede denote the expectation with respect to random seeds, i.e., expectation with respect to the data generation process which generates the data that the agent observes over rounds and within a round.
Theorem B.3.3. Let:
m (T) Eseeds Esecds|Ve (V(TD)(s)) (s)],
Cou(V!TD) (s), (TD) (x)) = M((DD (s.x) - m - (LD) (s)m, (s)m((
(z),
where 2) (TTD) (s, = Eseeds 1V, (TD) (s) (1 (s)] We have:
(TD)
(s) (s) (TD) (s), 241 (s,x) x) E >x(D) (s,z) aΩ (TD)
(TD) (s) = =m (TD)
mi+1
((T
(s, x)) + 2 (TD) (s,2),
and, the mean squared error E, (TD) at timestep t
(T))
= TD)
2x (s) (s) 2 (TD) (TT (s,
St+1
o d (s)
(TD) ())
#ES
Proof.
m (s) = Escnds
1+1 (s)], (s]],
Exeeds ((TD) (s) +
Is.=o RB vV, ((( - V. v((
k=1
N
m, (TD)
(TD)
(s) + ta
E
Exeeds [s. (RA + 7 VV (((TD) (5%) -V; ((TD)(s)),
19

---

N
(TD) (s) + a
[l + vV V (S)) - vi(TD) (s))
= m
E
Bseede
k=1
NV
= (s)
= =m: (TD)
(s) + a
(s)) + vV, (S)) (s) (sedd, Ss =s]
Eseeds
k=1
N
= = +
Eseeds
(()) - 1 -V)
k=1
Independent quantities
m (T (s) + (s) + (P m. (s)),
-m (TD) (8) + aNd (Prm (T))
(T)) (8) - m (s)),
m (8) = m, (2D) (s) + a (TD) (8),
(TD)
where A (TT) (8) = (s)))) + (P (s) m, (s). We used the following results in the proof:
1. E seeds (8)) Raseds (dr (s)) Eseeds [F] = d (s))rx(())
2. E seeds dd (s) (T( (s)] = -lEseed (d (s)) Eseeeds (P-V(TD)
= =dd((s) Σ B Exeeds P ( V. (TD) (s'),
= (d+((s Σ (x( (s))
(P= (T))
3. E Exeeds ()) [s)] Exeeds (d (s)) Beeede (T)) [s) [e
(TD)
(s)m (D) (s).
Now consider ((TD) (8
1++1
(s,x) = E Exceds |uth (TD)) (
(z),,
1+1
1+1
N
v ((TD))s)
Is. RR + +Vi (T)) (i (s))V
= Esedds
(s) +
k=1
T Ra (((TD)())) (Xi) V (aa (1)) (il (00))
v.TrD) (x)
E seeds IV. (TD) (s)
(x)]
1s (Ri + 7V, (T(TD)(()) (aai (l.l)(1 (2))
Ve (s)
(xm) (
alEseeds
(9)
Ts RR + ((( (St) - 1 (lllaa (s))
+ +oEoneds
vy VITD (x)
(10)
+ a a²E Exeede
Σ Is RR + (V((D)) (5)) -1- v.
N
Is (Ri + VV, V.(TD) (0)))())))
(11)
20

---

Next, we will tackle each part separately. We will begin with Eq 9
N
Eseeds
Ve
(ST
Is
+ TV( 7V (x (2
-x (x)) (x))
Rseeds t V((D)(s)
(V (x)) - (x) ssedd]
Eseeds (V(TD) (8) (F) (x) (PaV (z) - ((TD)
NEsseds (Tr (z), + NNNBseeds (1))))
NTEseds d ( Vz ()((D)(()) (s)),
(t)( + (x)
(TD) (
(d) (s.
(d,(x)
(x)m) (TD) (s) + Σ Brceds (P=('| (TD) (x)))())))))) - xDTD) ()
(2)
(+) (s)
(s) +
Σ
(x', - Ex 2(T
(x)x()) (z)
( (s) (P D (z. s) -()
A similar simplification can be done to Eq. 10 which results in:
(s) (s) (x) (P, x! x(T'D) (x""))(r (a,
We will break down Eq. (T) into 9 parts and simplify each of them separately.
N =
Esceds
Us
Rx
1s =
Rs
(12)
N
N
+ Bloeed
Is RR
Is V
(13)
1=1
Eseeds
Sh Rs
Ist Vi (()
(14)
+ Y Eseeds
Σ Is (sh)
Σ Is Rd
(15)
k=l
+ 22 Beeds
Is. I (S)
Is (TD) (x1)
(16)
Us (TD))))
Us V( (TD)
(17)
(T))))))) (8)
Egyeds
Ts Ra
(18)
I Is Is (()
Is V. (TD) ((TD)(())
-Beoees
(19)
Is V (TT }
Eseeds
Σ Is Is (T)
(20)
21

---

Consider Eq.
12
N
N
N
Eseeds
Σ
= Rk
Σ
Ts
Ri
Boceds
B
Σ
= Ri
Σ
Tsi=a
Ri
seed
1=1
1=1
R=1
1=1
N
N
N
Eseeds
E
Rk
Σ
== Ri
sseed
+ Exeedo
E
Σ
Is -sm Rm Rm
sseed
Am
mm=1
Li
N
N
N Σ
Eseeds
E
Is
Rx
seed
1s ma
R
|seed
(s) Isedd, Sm =
A==
1=1. 19tk
m=m
N
N
N
Bseeds
Σ (s)
Σ (x)
+ -==
E Beeeds (s)((((( + +
1-1. lik
m=1
N
N
E Exceds (1) (c((s) rs))
1/k
N(N (z)d Ld (c((s) (s))
We have used the i..dd assumption to separate the terms. Later, used the indicator function to indicate the part which is non-zero. Also, we have used the definition of the second moment in the penultimate step.
We do a similar simplification to Eq.
now.
{ 8, V
Is, = Ri
(TD)
A=1
N
N
Exeeedss
Σ
Is Rs V2 (.(TT)((X))
y
Fseeds
Σ m==1
Ism - Rm V(TD) (X
k=1/=1 1yk
N
Beeeeed
Elss R (T (D (Xf)
71ll=x Eeceds [B[Ism= Ro V, vi(Di)
(sedd,
1
N (1
=7Beeeds
+
k=1
= >NN(N (xd (TD) (2)) + (1)) (2D1 (s)
We will simplify Eq.
14h now
Σ Is Rs: )
N 1 Us (D) (x)
Egeeds
22

---

N
Eseeds
ss 1. 5 Rr Ve (x)
Bseeds Σ
sm Ms... = Ru Ve (x)
N Σ
-sseeds
BlUs, -s R& ?
ReV (rr
+ N
Bseeds
[R] - Is Rm vi (x)]seed]]
k=
mmm
17
N
N
Reeeds
Σ
(1))
+ +T Needs (s)r ( (T)
N(N - (x) (TD) (x) + (d))) (TD)
The next expression Eq. 15) is similar to Eq.
(13 and therefore, we skip the simplification details.
N
Rd ]
? E.sseds
Σ Us
(S})
= NN(N - (8) (M (T) + -T.-, (dd (TD)
Now, we simplify (66).
N
N
7 2E Essedds
-s v
(S))
((DD)
Σ
Us V (x)))
1=1
N
N
2
-s (ITD) (5 (TD1(
+ + Booeds
Σ m=1
Us ul 1/ ((TD) VV (x())
1= 1yk
=
Ul (T) (X/)
N
+ 7 I
Exceds E[lss,,, Il..... Vi (TDD) (S (V(D)) ( (X%) [sedd],
m
Eseeds
Σ "0
1=1=1. Lg
N
+ > 2
Esceds (s) (P, (TT)
m=l
NN - (sd b Nd
+
We now consider the next piece Eq. 17
Eseeds
Σ Is.=oV
V(TD) (5h)
Σ Is
(T)
(z)
23

---

N
S
Us Ve (S
Boedd
Σ
1mm =s S. =x =V
v(TD)
(S
(e (5)
N Σ
Bseeds
- Us - Ve (( (S)),, (x))s
k=
l=l, le
E seeds =s =x Vi (TD)
( (c))s]
m=
Essenss
Σ
(DD (P
+ y
Esceds (da(t) (PaV (())
>N
((TD)
(TD)
(s, + - Nd (s) ((TD) (s,
Simplification of Eq.
18
is similar to Eq.
14
N
N
Is Ri )
Exeeds
Σ
S V (s)
(s)
Σ
-N(N 1)d, (s) (r) (z) (z)mL (TD) + Nd (s)r (s)m
Also, the simplification of Eq. 19 is similar to that of Eq. (17
[E Σ
N
BBeedds
Su
V (T)
Σ
S. Vi ((TD
= NN (sd (z)
Nd (TD) (s,
We will simplify the final part i.e., Eq.
20
Eseeds
Σ Ts
V
((TD)
(s)
Ts V ((T (2r)
k=1
Is I v(TD) (s)
Eseeds Σ Imm II V(TD) (T)
E Egseds
k=11=1
B I1 - (TD) (s) (x))
= EEseeds
k=1
1=1.
+ >11==p
E Egeeds EIIs (TD)
(x)]seed])
N
ΣΣ (")
Bxeeds (dx (s)),,,
Egeeds
+ 711l=m
k=1 1=1.
m=1
= N(N - (s) (TD) (8,
+ Nd (s)
24

---

We can put all these pieces together to get the desired result.
D.U (s,x) 2// (8, all ( (8,2) 2 (8,
n 1 (0,z) (x)(z) TT (T=())m (P, (P (I, -E E(TD) (8,
+ (d) (rr(s) ((
(s, -E/TD) (x.
0 (s, 2) N(N (0) Nd, (8) (c(s) + +
+>N(N - 1)d=(())44(())==
(z) - (1) (s)
(s)
- -NNN - (10
+ *N 100.0000000000001110000000000000
+>ZN(N -1210040001122171100000a111..111 (TD))
->N(N - -24404400000222222110000-2111111
-N(N 1444000000000m.2"001011111041000 (s,
->N -
(s, -1a) - Nd ()
+ NNN - (s)
We will now prove the final part of the theorem. The mean squared error 4 (TD) into bias and covariance terms:
can be decomposed
(TD) S(+1
(bias (3) + Cou1+1(5,5)).
(V=((b) - m m1+1 (8)) + (E(E)) (8,8) (
(TD)
(s) (P)))(
(s)(())) 20, (8) (5) (TD)
(5) + -141 ((TD))(s (8,8));
Σ (1)(s) n0, (m) (T)(() aa (T)) (s)) Σ (s)) + aD (8, (s,
Σ( (1)(5) 20 (s)m² (TD) ( (s) + >KTDI| (s. s))
+ Σ 20xt, ( (TD) (s)) ane (D 8) + a a²v (TD (s,s))
(TD)
2u ( (TD)
(s) + Ω (T)) (s, s)) + 2 Σ
v (TD) (s.)
$ES
SES
Theorem B.3.4. Let Egeede denote the expectation with respect to seeds. Let
m (PT) (s) Excede (Veede(V((17) (PT) (PT)
Cou(V)!
Vi
(((T) (x)) (T()) (s,x)) ml (s)ml (T)
where D (PT)
(8, Escads [V. (PT) v.( (s)). Then:
(PT) (s) mi (PT)
()) (s) aNd (
m
241 (PP))) (s,x) [(P)) (s, + aΩ (s, + 2 (PT) (s,z),
(PT) (s,
25

---

and, the mean squared error cPPT)
timestep L is
(PT)
< = 51
(TT)
(s) S2 ( (s,
SES
+ Ws PT) (8,8).
SES
Proof. The proof is similar to that of TT learning algorithm. We have:
(PT) (s) mi (T
(PT)) m (PT) (s)):
mi+1
IlPT) (s = (s.
(s,
(P (8, (d) (2)m
(x, - -x(PT) (s, (8,2))
+ (s) (s)mm
(,, 2 (z, s)
= - 10 31110:000000010000..../.0o.11110
+NN - 1 (,
(PP)) (s)
-N(N -
+NN(N - 14 (PT) (s)
+~2N(N -,,","4.44..1.1.11""""""""""1111
-NN(N 144444444010733,3777"01,------
(s,
-N(N (2)) \| (0)
-7N(N - -|
+ N(N (1)
+ N. (s.
The MSE recursive equation be obtained similar to that of TD expression. Therefore, we skip the details and give the final expression.
(PT) S1+1
(PT)
2v (s)
24 (P(PT)((, 8)) + a² Σ ( (8,
a
#ES
sES
C Experiments
Besides providing details for the experiments described the main paper, we answer additional interesting questions, such as:
What is the effect of task distribution on the performance for various methods (Sec. C.2
What is the effect of network capacity on the performance (Sec. C.3
What is the effect of hyperparameters on the performance (Sec. C.4
C.1 Additional Details
We begin by providing details of the experiments.
C.1.1 Policy evaluation grids
We modify the rewards of the goal states as described in Table CTT The true value function for each task is shown in Fig. C.I We convert the 5x5 discrete grid into 48x48 RGB image for the deep RL
26

---

Reward
Task ID
Top-Left
Top-Right
Bottom-Left
Bottom-
0
2
1
0
0
4
1
1
0
0
Table C.1: Rewards at goal states for various tasks.
Teak:
Tred:
64
5.04
5.05
1:18
3.33
5.53
0.53
0:28
0:18
0.00
0.04
0.09
0.08
0.08
0.08
0.08
0:48
0.38
0.38
0.48
0.01
5.05
2.
2:39
0.16
0.00
5.15
1B
16
1.
5.04
1:
3.33
5.93
0.0
5.03
2.
4:48
0.5
0.
0.46
0:22
0.00
0.0
0.0
0.03
0.33
0.53
0.
0.04
0.00
0.04
Figure C.I: Heatmap of the true value function for different tasks.
task. The agent's s location is indicated by colouring the its location in red and we use green squares to indicate goal states. At every timestep, the agent receives RGB image as an input. The action space and the action consequences are the same as discrete grid. We select best hyperparameters (learning rates) for each algorithm by choosing those values which results in the lowest area under the error curves as shown in Table C.6 and Fig. C..9 The hyperparameter search and the final reporting is performed by averaging the results on 30 different seeds. For our method, the buffer stores the samples from the latest task only to update the permanent value function.
For the first linear FA experiment, we use the same setup as the tabular setting except we encode each state in the grid as vector of 10 bits. The first 5 bits encodes the row of the state and the last bits encodes the column of the state. For example, the starting state (2,2) represented as 1,.00.0000,,110000
Both permanent value function and transient value function use same features to learn respective estimates.
For the second linear FA experiment, we use continuous version of the grid. The goal rewards for each task is same as the discrete counterpart as described in Table The true value function also different for each task and it is shown as a heatmap in Fig. C.2, We use RBFs to generate features as described in Sutton and Barto 45 We use 676 centers (order?) spaced equally in both and y directions across the grid with a variance (0.75/ (order 1))² and use threshold to convert the continuous vector into binary feature vector. Any feature value greater than 0.5 is set 1, while those features with values less than 0.5 is set to O. We search for the best learning rate from the same set of values as the discrete grid task. The hyperparameter curves are shown Fig. C.11
For the deep RL experiment, the rewards are multiplied by a factor of 10 to minimize the initialization effect of deep neural network on prediction. We used convolution neural network (CNN) to approximate the value function for all algorithms tested. The CNN consists of two conv layers with
10
1.3
10
24
64
64
24
04
12
17
11
33
34
24
Figure C.2: Heatmap of the true value function for different tasks.
27

---

8 filters, (2,2) stride, and relu activation in each layer. We use a max pooling layer in between the two conv layers with a stride The output of the second conv layer is flatted, and is followed by couple of linear layers. The fully connected network has 64 and hidden units in the respective layers. The first fully connected layer uses tanh activation and no activation is used in the final layer. For our method, we attach new head after the first two conv layers for permanent value function. This head also has two linear layers where the first layer has tanh activation. The entire network is trained by backpropagating the errors from the transient value function. But, the conv layers are not trained when updating the permanent value function. The transient value function and the TD learning algorithm are trained in online fashion using experience replay buffer of size 100k. The buffer is reset at the end of the task for all methods to reduce the bias from old samples. permanent value function is trained at the end of the task using the samples from the latest task. 100 gradient steps are taken during each update cycle. We use SGI) optimizer to update its weights. The best learning rate is picked by computing the area under the error curve set of values shown in Table C.7
The mean AUC curves of various choices of learning rates are shown Fig. C.12
28

---

6
Figure C.3: Predictions evolution across episodes.
C.1.2 Policy evaluation minigrid
We use 4 rooms task as shown in Fig. 1b The environment is grid task with each room containing one type of item. Fach item type different in terms of how it perceived and in terms of reward the agent receives upon collecting it. There are 4 goal cells, one in each room located at the corner. At
29

---

the beginning of the episode, the agent starts in one of the cells within the region spanning between cells (3,3) and (6,6). It has actions to chose from turn left, turn right, and move forward. Turn-based actions changes the orientation of the agent along the corresponding direction, whereas the move action changes the agent's location to the state in front of it. The agent can only observe a 5x5 view in front of it (partial view). Items are one-hot represented along the 3rd dimension to make the overall input space 5x5x5, To set up the continual learning problem, we change the rewards of the items from task-to-task as described in Table C.2 The evaluation policy is kept uniformly random for all states and for all the tasks, We use a discount factor of 0.9. We evaluate the performance for a total of 750 episodes and the tasks are changes after every 75 episodes.
Reward
Task ID
Blue
Red
Yellow
Purple
-1
1
1
0.5
3
1
1
1
-1
-1
1
0.5
-1
Table C.2: Rewards for items for various tasks.
We used a convolution neural network (CNN) to approximate the value function for all algorithm. The CNN consists of three conv layers with 16, 32, and 64 filters, (2,2) stride and relu activation in each layer. The output of the third conv layer is flatted, which is then followed by couple of linear layers with 64 and hidden unit. Tanh activation in the first linear layer and no activation is used in the last layer. We attach a head after conv layers for permanent value function. The architecture of this head is same as the TD learning head. The training method and buffer capacities are same as the discrete image grid task. We use SGD optimizer to update weights in all networks. The best learning rate is picked by computing the area under the error curve on a set of values shown Table C.8 and All the results reported are averaged across 30 seeds.
Fig. C.13
C.1.3 Control minigrid
We use two room environment as shown in Fig. 3c The agent starts in one of the bottom row states in the bottom room and there are two goal states located in the top room. In the first task, the goal rewards are +5 and 0 for the first and second goals respectively, and it is flipped for the second task. We alternate between two tasks every 500 episodes for total of 2500 episodes. The agent receives a one hot, 5x5 view (partial observability) in front of it at each timestep. It can take one of the three actions (move forward, turn left, turn right) at every step. The move action transitions the agent the state in front of it if there is no obstacle. The environment is deterministic and episodic, and we use discount factor of 0.99.
We use a CNN as a function approximator. The CNN has 3 conv layers with 16, 32, and 64 filters, (2.2) stride, and relu activations. It is followed by fully connected neural network with 128, 64, and 4 hidden units. The first layer is relu activated, the second layer is tanh activated, and the final layer is linear. We use target network and experience replay train the network. Target network's weights are updated every 200 timesteps. Experience replay buffer's capacity is capped to 100k and the stored samples are deleted when the task changes. We use batch size of 64 and update the network every timestep. We search for the best learning rate from a set of value shown in Table C.10 on 3 seeds and we pick the one that gave the highest area under the rewards curve. We then run on 30 seeds using that learning rate to report the final performance. The transient value function training procedure and details are identical to DQN. permanent value function uses a separate head following conv layers and is updated using Adam optimizer whenever the task is changed.
30

---

C.1.4 PT-DQN Pseudocode
Algorithm 4 PT-DQN Pseudocode (Continual Reinforcement Learning)
1: Initialize transient network buffer B. permanent network buffer D
2: Initialize permanent network 0, transient network w
3: Initialize target network Wtarget
4: for t: 0 + oc do
5: Take action A, according to e-greedy policy
6: Observe 1
7: Store (S,, A,, S(41))
8: Store (St, A.. (S,) A.)) in D
9: Sample batch of transitions (Sy. A,. Ry+1, Sy+1) from B
Compute target
y R5+1 + maxxx' ELA
Q (SJ+1, a'; Wtarget)) Q(())))))
(
11:
Perform gradient step (y (S Ay; w)) with respect
12:
Update target network every N steps, W Wtarged w if mod(t, k) = then
13:
14:
Sample batch of transitions (S,, Aj, (S, A,)) from D
Compute target y = ((P) (S, A,)
16:
Perform gradient step on (j Q() Ay; 0)) with respect to 0
17:
Decay transient network weights, w dw
18:
Clear permanent network buffer, D 0
19:
end if
20:
end for
C.1.5
JellyBean World (JBW)
Item
Blue
Green
Red
Color
[0, 0, 1.0]
[0, 1.0, 0]
[1.0, O, 0.0]
Intensity
Constant [-2.0]
Constant [-6.0]
Constant [-2.0]
Piecewise box [2, 100, 10, -5]
Zero
Piecewise box [150, 0, 100, 100)
Interactions
Zero
Zero
Zero
Piecewise box [150. 0, -100 - 100]
Zero
Piecewise box [2, 100, 10, -5]
Table C.3: Worll World environment parameters.
4000
24000
4000
(b) Cummulative rewards against timesteps averaged across seeds.
(a) Sample observation reecived by the agent.
Figure C.4: JellyBean World sample observation and performance curves for various decay values.
We test algorithms continual learning problem in the JBW testbed 31 and use neural networks as the function approximator. The environment is wwoddimensional infinite grid with three types of items: blue, red, and green. We induce spatial pon-stationarity by setting the parameters of the environment as described Table C.3 our configuration, three four similar items (red or blue) form tiny group and similar groups appear close to each other to form sea of red (or blue) objects as shown in Fig. 55 Therefore, the local observation distribution various parts of the environment different. At each timestep, the agent receives egocentric 1 11x11 RGB, 360 view
31

---

Green Bems
Prodfti EaTs
Negstivn terrs
2.5
2.5
1.0
1.0
0.5
0.5
Taak tches
PT.DON- 5x (AAP))
DGN
DON-muE hood
DGN longe suflor
Random
Figure C.5: Items collected by various algorithms during each task switches.
as an observation as shown in Fig. C.4a It has four actions to choose from: up, down, left, and right. Each action transitions it by one square along the corresponding direction. The rewards for green items, which are uniformly distributed, are set to 0.1, whereas the rewards for picking red and blue items alternate between -1 and +2 every 150k timesteps, therefore, inducing reward non-stationarity. We run the experiment for 2.1 million timesteps and train algorithms using a discount factor 0.9.
The DQN agent uses four layered neural network with relu activations in all layers except the last to estimate q-values. The number of hidden units are 512, 256, 128, and 4 respectively. The network is trained using Adam optimizer with experience replay and a target network. We use apsilon-greedd policy with e = 0.1 for exploration. The permanent and transient network architectures are identical to that of DQN but with half the number of parameters in each layer. This ensures that the total number of parameters for the baselines and our method is same. Transient network is also trained using Adam optimizer, whereas, permanent network is updated every 10k timesteps (k=10,000) using SGD optimizer. Target network's 's weights are updated every 200 timesteps. Experience replay buffer capacity is capped to 8000 as higher capacity buffer stores old training samples which affects the training when the environment is changing continuously. We use batch size of 64 and update the network every timestep. For our approach, experimented with three choices of À values (0.55, 0.75, and 0.95) and all the choices resulted a better performance than the DQN agent as seen in Fig. C.44) We the same exploration policy as the DQN agent. We flatten the observation into one long array, then stack the last four observation arrays to form an input to the neural network. The best learning rate for all algorithms million timesteps for a set of fixed learning rate values on 3 different seeds. We search for the best learning rate from set of value shown in Table C.12 on seeds and pick the one that gave the highest rewards. We then run on 30 seeds using that learning rate to report the final performance. The random agent picks one of the four actions according uniform probability each timestep. For the multi-head DQN baseline, we attach two output heads one for each task. The baseline knows the task IDs, so it chooses the appropriate head to select actions and train the network. The replay buffer is reset for this baseline whenever the task is changed. We use buffer capacity of 250k for the DQN baseline that is augmented with large buffer. The rest of the training and architecture details are same as the DQN agent.
chosen by comparing the total reward obtained in 1.5
C.1.6 MinAtar
This domain is standard benchmark for testing single task RL algorithms [51] We use breakout, freeway, and space invaders environments to setup a continual RL problem by randomly picking one of these tasks every 500k steps for total of 3.5M steps. We standardize the observation space across tasks by padding Os when needed and make all the actions available for the agent. This problem has non-stationarity in transitions besides the reward and observation signals making it challenging. We use discount factor of 0.99 for this experiment.
We used a CNN which has relu activated conv layer with 16 filters followed by a relu activated linear layer with 256 units, and a linear output layer for DQN. The architecture is same as the one used in the original paper [5I] The architecture same for other DQN variants but we add output layers for the headed DQN. For our method, we use the same architecture but 8 filter in the conv layer and 128 units the linear layer. Target network is used for all the methods which is updated every 1000 steps and we use experience replay of 100k for all methods but we use 500k sized experience replay for large replay based DQN baseline. We use Adam optimizer to train the networks in online fashion. For our method, permanent value function network is updated using SGD
32

---

optimizer every 50k steps. The best hyperparameters are chosen by running the experiment for 1.5M steps by switching the tasks every 500k steps on 3 seeds. The results of hyperparameter tuning are 1.19 and C.20 We make all 7 actions available to the agent at all times and the input to the agent is 10x10x 10x7 images.
detailed in Tables C.17 C.I8
C.2 What is the effect of task distribution on the performance for various methods?
Thoik dbenwerr
Theck cldl downeter 7.5
Taxk dameler 10
Expedted
Espoted
(meet) PT-TO (G4)2)
10
Figure C.6: Expected analytic MSVE on a task for various algorithms and e values.
We set up continual learning problem by creating tasks whose value functions are e (task diameter) distance from one another in norm-2. Each task has 7 states. For particular value, for a particular number of task switches, we run each algorithm 100 times and save the final estimates which form the initial point for the new task. We then sample task according to uniform distribution and use analytical MSE expression derived in the previous section to analyze the behaviour of PT-TD learning and the two versions of TD learning algorithm for 100 rounds. We then average the error across rounds and across different starting values to get a point in the plot shown in Fig. C.6
Setup:
Task details: We use the following Px for all tasks and for all task diameters (e):
0..
0.19
0.08
0.16
0.1
0.11
0.18
0.04
0.05
0.01
0.0
0.4
0.0
0.51
0.0
0.2
0.0
0.05
0.28
0.06
0.41
P.
0.17
0.14
0.09
0.26
0.15
0.19
0.01
0.16
0.16
0.16
0.24
0.17
0.0
0.11
0.25
0.02
0.24
0.24
0.08
0.05
0.11
0.0
0.25
0.19
0.44
0.05
0.0
0.07.
We then sample 7 points in R that are units from one another. These points form the true value function of the tasks. We then compute the reward vector for each task as r. (T) = (1-?Pa (T,(()) At the time of testing the algorithms, we add zero mean Gaussian noise with 0.1 standard deviation to these rewards to simulate stochasticity. We search from the learning rates shown in Table C.4 to pick the best one for each combination of and number of task switches. We use a learning rate of 0.01 while using analytical expressions.
We observe both TD learning algorithm and our approach has the same performance after seeing data from one task. But the error gap increases as the number of task switches increase. The low error in our method can be attributed to the presence of permanent value function which captures some portion of the value that is common for all tasks. transient value function then learns the remaining part with small number of samples resulting in low error overall. We also observe that the error gap between TD learning and our method gets wider as we increase task diameter, which demonstrates that our method is broadly applicable. The reset variant of the TD learning algorithm whose predictions are reset whenever task changes has the largest error emphasizing learning from scratch could be costly.
Observation:
Algorithm TD (without reset) PT-TD
LRs
[8e-1 1 5e-1 3e-1 Te- 5e-2 1e-2)
PV: [5e-1 3e-1 le-1 5e-2 le-2 5c-3)
TV: (8e-1 5e-1 3e-1 le-1 5e-2 le-2)
Table C.4: LRs considered to pick the best one.
33

---

[le-2, 1c-3, le-4, le-5, 1c-6, 1e-7] PV: (5e-1 3e-1 1 le-1 5e-2 le-2 5e-3) C.3 What is the effect of network capacity on the performance?
Setup: We use JBW task the continual RL setting to perform the ablation study, We run two additional experiments by varying the number of parameters in DNN for the DQN and method. In the first experiment, we use the same DNN architecture for all algorithms, therefore, our method has twice the number of parameters that of DQN. In the second experiment, capacity for the DQN variant making the total number of parameters same as ours, We keep the rest of the experimental setup same before and pick the best learning rate based on the AUC by searching over the values detailed in Table C.5 over seeds for 1.5M steps.
double the network
Algorithm
LRs
DQN (2x)
PT-DQN
TV: (8e-1 5e-1 3e-1 le-1 5e-2 1e-2)
Table C.5: Search space of learning rates.
Negehe
(a) Results the GellyBeanWond
(b) Items collected in each task.
Figure C.7: (a) Rewards accumulated per timestep across 10k step window is plotted against timesteps.
(b) Number of items collected within a task for each type (subplots use different scales along y-axis).
Pabe BYI
Nagine Ner
Results the JellyBeanWorlee
(b) Items collected in each task.
Figure C.8: (a) Rewards accumulated per timestep across 10k step window is plotted against timesteps. (b) Number of items collected within a task for each type (subplots use different scales along y-axis).
The results for ablation experiments are shown in Figures C.7 and C.8 We observe that our algorithm continues to perform better than the DQN agent in both the cases. The results also indicate that it is efficient to devote the available capacity in learning parts of the value function rather than learning the whole.
Observations:
In the second ablation experiment, the DQN agent with twice the number of parameters starts off learning slowly but catches up with our method after seeing enough data. fact, the DQN agent continues to perform similar to our method without any drop in performance further (we tested it by running the agent to 6M timesteps). This result highlights that our approach is beneficial in the big world small agent setup where the computation budget of the agent is very small compared to the complexity of the environment. When the agent's capacity is large relative to the complexity of the environment, there's 's additional benefit (neither there is any downside) to method. Since the world is much more complicated in comparison to the agent's capacity, using our method is preferable.
continue train
These experiments also provide partial answer to the question when does the plasticity problem arise in neural networks? We think that combination of learning algorithm (like DQN), small agent
34

---

capacity, and complex environment could give rise to the plasticity problem in neural networks. A thorough investigation is necessary confirm this hypothesis.
C.4 What is the effect of hyperaarameters on the performance?
In this section we analyze the sensitivity of additional hyperparameter in our method (learning rate to update permanent value function) and compare it with the sensitivity of TD and Q-learning algorithms on various environments.
Algorithm
LRs
TD (without reset)
18e-1, Se-1,, 3e-1, le- 5e-2, le-2] [8e-1, 5e--5----- 3e-1, le- 5e-1, le-2) Se-2,
TD (with reset)
PT-TD
PV: (1e-1, le-2, 5e-3, 1e-3)
TV: (8e-1, 5e-1, 3e-1, le-1, 5e-2, 1e-2)
Table C.6: LRs considered in the search space for the policy evaluation grid tasks.
PT-TD
0.34
0.24
0.24
MenAAC RNSVV
0.1
0.05
0.20
0.20
0.20
0.18
MeenAl
0.18
MeanAUC
0.18
0.001
0.98
0.98
0:14
0.12
0.12
0:12
0.
0.5
0.0
0.0
0.5
LRa
LRa
TVLRu
Figure C.9: Mean AUC RMSVE against learning rate for various algorithms.
TO
TD or redecs
PT-TO (sums)
0.25
0.25
0.25
0.23
0.23
MoanAUC RRMMS
0.23
0.30
0.18
0:18
PVLRs
0.18
0.18
6:18
0.05
0.01
0.10
0.50
0.10
0.001
0.0
022
0.4
0.5
0.6
0.0
0.2
0.4
0.6
0.6
0.0
0.
0.4
0.8
0.8
LRa
LRa
LV
Figure C.10: Mean AUC of RMSVE against learning rate for various algorithms.
For tabular and linear grid tasks, select best hyperparameters (learning rates) for each algorithm by choosing those values which results in the lowest area under the error curves as shown in Table C.6, Figures C.9 C.10 C.III The hyperparameter search and the final reporting is performed by averaging the results on 30 different seeds. For the deep RL task with discrete grid, we choose the best learning rate from the values shown in Table C.7 and the corresponding U curves shown in
C.12
For the minigrid policy evaluation task, we search for the best learning rates from the values
shown in Table C.8 and the corresponding U curves are shown in Fig. C.13
For the tabular control experiment, the learning rates used to search for the best value is listed in Table and the mean returns for various choices of these values plotted in Fig. C.14 for various algorithms. Similar table for the minigrid task (deep control) shown in Table C.10 and the mean returns for various choices of these values is shown in Fig. C.15 We use 30 seeds to pick the best learning rate for the tabular task but use 3 seeds to select the learning rates for the minigrid task.
The learning rate tables and the corresponding performance for the full continual RL experiments are shown in Fig. [C.66)and and Tables C.I1 C.12 C.17 C.18 C.19) and C.20 We use 30 seeds find the best hyperparmeters for the tabular task, but use seeds for JBW experiment and MinAtar.
35

---

TO
TD
PT-10 (c415)
0.1
0.M
0.06
0:33
0.33
0.33
0.83
0.001
0.30
0.33
0.33
0.32
0.32
0.22
0.0
0.5
0:31
0.0
0.0
LRs
LRz
LVR
Figure C.11: Mean AUC RMSVE against learning rate for various algorithms.
Algorithm
LRs [le-1, Ie-2, 3e-3, le-3]
Deep TD (without reset) Deep TD (with reset)
1e-1, le-2, 3e-3, le-3) PV: (le-3, 3e-4, le-4, 3e-5) TV: (le-2, Se-3, 3e-3, le-3)
Deep PT-TD
Table C.7: LRs considered to pick the best one for the Deep prediction experiment with discrete grid.
We find the best learning rate for each k, À pair by searching over the learning rates listed in Table CTI
We see that our algorithm performs better than their vanilla counterparts on all the problems tested for a broad range of hyperparameters. There are handful of (ā,, a) pairs that results in a better performance in each setting, which indicates that our method is not sensitive to hyperparameter selection. We also observe that all the cases ō is smaller than a, which fits into our intuition. Because, the permanent value function has to capture regularities in the value function estimate, needs to update slowly, whereas, the transient value function should be updated quickly to facilitate fast adaptation. The sensitivity of two new hyperparameters: X, for the fully continual RL setting is analyzed in the main paper (see Sec. 6.1 where concluded that our algorithm is robust to these values.
Observations:
Deep
Doep
(ours)
1.50
1.00
1.80
PV URt
MeeAUC AUC OTFNNSVE
Moon AUG CIRASSIE
0.001
8.47
1:40
1.40
0.0071
3e
Mewe AUD
0.00
1.00
0.00
0.01
0.55
0.00
0.01
0.05
0.002
0,006
0.10
Figure C.12: Mean AUC of RMSVE against learning rate for various algorithms on image grid.
36

---

Algorithm
LRs
TD (without reset)
[le-1, 3e-2, le-2, 3e-3, 1e-3] [le-1, 3e-2, le-2, 3e-3, le-3)
TD (with reset)
permanent value function
[3e-2, le-2, 3e-3, 1e-3, 3e-4]
transient value function
(3e-3, le-3, 3e-4, le-4, 3e-5, le-5, 3e-6, le-6]
Table C.8: Search space of learning rate for the policy evaluation minigrid task.
Doep TD
Dosp TD rosets
PT-Doop TD (ours)
LVLRs
RMEVE
2.40
2.40
RMBVE
2.40
0.001
2.30
2.30
0 0.0001
2.20
Mcan
2.20
2.00
3o-05
2.10
2.10
2.10
2.00
2.00
2.00
1.90
000
0.01
0.02
0.03
1.50
0:00
0.01
0.02
0.55
1.00
0.00
0.01
0.02
0.05
LRs
LRs
LRs
Figure C.13: Mean AUC RMSVE against learning rate for various algorithms on minigrid grid.
Algorithm
LRs
Q-learning
{8e- 5e- le- le- Se-2, le-2,
Se-3, Ie-3]
Q-learning (with reset)
{8e- 5e-1, le-1 Se-2, 1e-2, 5e-3, le-3) 18e-1, 5e-1,, 3e-1, le-1, 5e-2, 1e-2, 5e-3, 1e-3)
Permanent Value Function
Transient Value Function
{8e-1, 5e-1, le-1, 5e-2, 1e-2]
Table C.9: Search space of learning rates for the tabular control task.
learning
leeamingg
lcamiig
0.50
roturne
0.30
0.30
0.30
PV
0.8
0.05
0.5
0.01
0.
0.006
TVLRs
Figure C.14: Mean returns for various learning rates for the tabular control task.
Algorithm
LRs
O-learning
[3e-3, 1e-3, 3e-4, Ic-4, 3e-5]
Q-learning (with reset)
(le-2, 3e-3, le-3, 3e-4, le-4) [le-5, 3e-6, le-6, 3e-7)
Permanent Value Function
Transient Value Function
(3e-4, le-4, 3e-5, le-5)
Table C.10: Search space of learning rates.
redeza
(curs)
AUC Retrrns
2.50
2.50
Mawer Raatara
2,00
2.00
1.50
1.50
1.50
2.00
1.00
1.00
36-06
0.50
0.50
0.50
16:06
0.00
0:00
0.50
Figure C.15: Mean returns for various learning rates for the Minigrid control task.
37

---

08
02
82
00
-0.2
-32
-00
-0.6
Ratumm
60
0.
Dicaning
P10-eemin
04
Epardoa
Eptradus
Figure C.16: Performance plot of PT-Q-learning against Q-learning for k, pairs
Algorithm
LRs
Q-learning
[8e- 1, Se-1, le-1, 5e-2, le-2, 5e-3, le-3)
Permanent Value Function (0.5x)
(le-4, le-5, 1e-6, le-7, 1e-8)
Transient Value Function (0.5x)
(le-2, le-3, le-4, le-5)
Table C.11: Search space of learning rates for the main JBW experiment.
Algorithm Q-learning Permanent Value Function
LRs
(le-2, 1e-3, 1e-4, 1e-5) (le-4, le-5, 1e-6, 1e-7 (le-2, 1e-3, le-4, 1e-5]
Transient Value Function
Table C.12: Search space of learning rates for the main JBW ablation experiment.
A=0.55
A=0.75
A=0.95
TV-LR
lc-2
le-3
1c-4
lc-5
lc-2
1c-3
1c-4
lc-5
1e-2
1c-3
le-4
lc-5
PV-LR
le-4
0.031
0.223
0.126
0.103
0.035
0.257
0.154
0.154
0.038
0.217
0.199
0.159
le-5
0.031
0.252
0.071
0.051
0,030
0.261
0.077
0.068
0.040
0.220
0.201
0.096
le-6
0.041
0.242
0.070
0.061
0.034
0.260
0.090
0.083
0.030
0.234
0.207
0.084
lc-7
0.036
0.256
0.056
0.067
0.040
0.268
0.062
0.070
0.028
0.227
0.216
0.124
lc-8
0.031
0.247
0.051
0.069
0.041
0.261
0.069
0.069
0.037
0.232
0.213
0.080
Table C.13: JBW: Reward per step for various hyperparameters for PT-DQN. We highlight the values that is larger than DQN's best performance.
LRs
le-2
lc-3
lc-4
le-5
Returns
0.03
0.127
0.192
0.093
Table C.14: JBW: Reward per step for various hyperparameters for DQN.
LRs
le-2
le-3
le-4
1e-5
Returns
0.037
0.135
0.247
0.245
Table C.15: JBW: Reward per step for various hyperparameters for DON-multi-head.
38

---

LRs Returns
le-2
1e-3
le-4
le-5
0.025
0.055
0.055
0.031
Table C.16: JBW: Reward per step for various hyperparameters for DQN with large experience replay.
0=555
A=0.75
A=0.95
TV-LR
1e-3
le-4
le-5
le-3
le-4
1e-5
1e-3
le-4
le-5
PV-LR
le-7
14.41
17.38
7.89
13.87
19.07
10.55
9.50
16.12
12.16
le-8
15.29
17.35
7.79
14.02
20.54
10.72
9.79
16.71
12.48
le-9
15.54
17.51
8.02
12.63
19.54
10.90
8.57
16.48
11.76
Table C.17: MinAtar: AUC of the average return (in the previous 100 episodes) step for various hyperparameters for PT-DQN. We highlight the values that is larger than DQN's best performance.
LRs
le-3
lc-4
le-5
1e-6
Returns
7.10
12.90
13.14
5.90
Table C.18: MinAtar: AUC of the average return (in the previous 100 episodes) per step for various hyperparameters for DQN.
LRs
le-3
le-4
le-5
le-6 8.54
Returns
8.72
14.93
15.68
Table C.19: MinAtar: AUC of the average return (in the previous 100 episodes) per step for various hyperparameters for DQN-multi-head.
LRs
le-3
1e-4
le-5
1e-6
Returns
6.78
11.44
10.70
4.56
Table C.20: MinAtar: AUC of the average return (in the previous 100 episodes) per step for various hyperparameters for DQN with large experience replay.
39


Reviews:
[{"review_id": "v2YS23I2wP", "reviewer": "Reviewer_mFKc", "summary": "This work proposes a reinforcement learning method for prediction and control. The value function in reinforcement learning is decomposed into two terms: a permanent value function, and a transient value function. The permanent function contains the long term knowledge of the interplay between the agent and the environment, and the transient value function absorbs the quick information variations. This work uses temporal difference learning for updating the estimate of the value function estimation. The learning process is related to the complementary learning systems from neuroscience. Prediction experiments are performed for 5x5 discrete grid. The agent moves from the center and selects an action the goal states.  ", "questions": "In the experiments, the proposed method is compared with DQN. Is it possible to compare other competitive algorithms?", "limitations": "The limitations of this work are not explicitly given. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It is of great interest to extract valuable information from stream data. The development of organic transistor featuring a selective transition from short-term to long-term plasticity of the biological synapse. Catastrophe forgetting is a major challenge for machine learning. A possible solution for solving such a problem is very attractive. It is closely related to meta learning. ", "weaknesses": "The idea for separating long term and transient information is intuitive. It has been used in many machine learning algorithms such as LSTM.  It is not clear if a long term function is always available, especially for complicated interplay between an agent and its environment. The separation of long term and transient can be dynamic. \n\nThe settings for the experiments are very simple. It does not demonstrate the possibilities for real applications. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We address the main points below.\n\n> The settings for the experiments are very simple.\n\nWe designed the experiments in the paper to cover several aspects that we deemed important: environment non-stationarity, different function approximators (tabular, linear, non-linear), generalization ability, and ability to deal with large observation spaces (such as RGB images). We used domains that are typical in semi-continual RL and continual RL (Minigrid, JellyBeanWorld, and MinAtar). The JellyBeanWorld and MinAtar experiments in particular, while not being \"real applications\", do test scaling.\n\nThe type of experiments we ran is consistent with other papers in the field, but the tasks we used are harder than in earlier works. For instance, [1] uses catcher and flappy bird with a curriculum of tasks, while the non-stationarity in our setup is much more severe. [2] used only one continual RL problem - slippery ant. We test our algorithm on a diverse set of problems and function approximators.\n\nIf you consider these experiments insufficient, it would be very useful to know what additional aspects of our approach you would have found useful to test, or what environments suitable for continual learning you would suggest. Concrete suggestions of environments and/or experiments that can be carried out in an academic environment would be helpful.\n\n> The proposed method is compared with DQN. Is it possible to compare other competitive algorithms?\n\nOur approach to choosing baselines is to compare against the algorithms on which we are building. We used TD learning as the baseline for policy evaluation because our work builds on TD, and the transient memory is updated using a TD-like rule. Q-learning is used in the control experiments for the same reason. This approach is similar to other research papers like [1] and [2]. In [1], DQN with experience replay is compared against the proposed DQN with meta-experience replay. [2] uses PPO with backprop as a baseline for continual backprop. \n\n**Also, in the continual RL experiments in Section 6, we introduced 2 additional competitive baselines:** (a) DQN paired with a large experience replay buffer &mdash; an approach proposed recently as a viable option for continual RL [3]; (b) DQN with a common trunk to learn features and separate heads to learn Q-values for the different tasks &mdash; a common approach when task boundaries are observable [4].\n\nNote that our approach could be wrapped around other RL methods that leverage value functions, in which case it would make sense to compare against those methods.\n\n> The limitations of this work are not explicitly given.\n\nAs reviewer zZeF pointed out, our work is limited to methods which leverage a value function. Also, our theoretical analysis is limited to piece-wise non-stationarity, where the environment is stable for some period of time. An interesting direction would be to extend our approach to other types of non-stationarity, such as continuous wear and tear of robotic plants. We will add a paragraph in the discussion section to address this.\n\n> The idea for separating long term and transient information is intuitive. [...] The separation of long term and transient can be dynamic\n\nIntuitive ideas are often the ones that stand the test of time. In our case, we build on CLS neuroscience intuitions, and we deliberately try to do so in the simplest way possible. We agree that different ways of combining permanent and transient memories, perhaps conditioned on a task description or through meta-learning could be interesting to explore in the future. But we were very pleased that the simplest approach that we could conceive actually leads to strong results in challenging continual RL tasks.\n\n*References:*\n\n[1] Riemer, et al. \"Learning to learn without forgetting by maximizing transfer and minimizing interference.\" arXiv preprint arXiv:1810.11910 (2018).\n\n[2] Dohare, et. al. \"Continual backprop: Stochastic gradient descent with persistent randomness.\" arXiv preprint arXiv:2108.06325 (2021).\n\n[3] Massimo, et. al.. Task-agnostic continual reinforcement learning: In praise of a simple baseline. arXiv preprint arXiv:2205.14495, (2022).\n\n[4] Samuel, et. al. Same state, different task: Continual reinforcement learning without interference. AAAI Conference on Artificial Intelligence, (2022)."}, {"review_id": "dQvMeuDRwo", "reviewer": "Reviewer_27Kd", "summary": "The paper proposes a method to improve continual learning in reinforcement learning with non-stationary dynamics. It proposes to maintain two value functions: one that tracks information about the current task (transient) and one that maintains a general knowledge of the task distribution (permanent) in order to better tackle the non-stationarity of the environment.\n", "questions": "1. What do the authors mean when they say prior work “reflected the nature of that task” in line 41?\n2. What is the intuition for the update in Eq 4? Why does the update to the parameters of the P get scaled by V^PT? Similarly, what is the intuition for Eq 5? Why is \\delta purely in terms of the overall update, PT?\n3. The definition in Eqn 2 and 3 is that both memories are equally weighted and are just a sum. I am curious what the authors think about this equal weighting and the consequences of it?\n4. It is a little unclear to me why only the initialization condition is sufficient for Theorem 1 to hold?\n5. Shouldn’t the equation in Theorem 2 be what is in Equation 5 i.e. PT instead of P?\n6. It may be worthwhile to see how Continual backprop [1] can be incorporated into the proposed method.\n\n[1] Continual Backprop: Stochastic Gradient Descent with Persistent Randomness. Dohare et al.\n", "limitations": "See above questions", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The general idea is simple in terms of intuition and implementation. I like the idea of maintaining two different value functions operating different time-scales.\n2. Assuming soundness (see below), the experiments well-done and are insightful.", "weaknesses": "While the overall approach is intuitive, the specifics dont seem very intuitive to me and there is not much information given about it. For example, Eq 4 (see questions below) is not intuitive to me on how it was derived. Similarly, I have other questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We answer your questions below, please let us know if further clarification is needed.\n\n> What do the authors mean when they say prior work “reflected the nature of that task” in line 41?\n\nWe meant that the method presented in Silver et al [1] leverages domain knowledge about the game of Go, which is the task they consider, to derive the planning update rule.\n\n> What is the intuition for the update in Eq 4? Why does the update to the parameters of the P get scaled by $V^{PT}$? Similarly, what is the intuition for Eq 5? Why is $\\delta$ purely in terms of the overall update, PT?\n\nWe want the overall estimate, $V^{PT}$, to approximate well the state value function. In order to provide a good starting point for any given task, the permanent value function should \"mix\" value estimates from all tasks. This is achieved by taking a step towards the new estimate of $V^{PT}$ at a slow pace (either at the end of the task or every $k$ steps). This is Eq. (4), which is basically similar to a supervised learning update with $V^{PT}$ as the target. Theorems 6 and 7 show that this update has favourable theoretical properties. \n\nSince some part of the value is already captured by $V^P$, $V^T$ should compute only further corrections from it, which is implemented in Eq.(5). We agree that the use of the overall TD-error in this update is a bit surprising, but this falls out of the math. Theorems 2 and 3 prove that this is the correct update given $V^P$.\n\n> The definition in Eqn 2 and 3 is that both memories are equally weighted and are just a sum. I am curious what the authors think about this equal weighting and the consequences of it?\n\nOur choice was meant to be as simple as possible and was inspired by an earlier paper in model-based RL [1] as discussed in Sections 1 and 2. In general, emphasizing $V^P$ increases bias towards the past value functions observed, which can reduce variance. Emphasizing $V^T$ would have the opposite effect. Other choices, like linear or non-linear combinations, could be explored in future work. It may also be possible to meta-learn how to combine these estimates, or to use side information, such as a task description, to condition a more flexible combination scheme. However, in continual learning, it may be difficult to make good choices unless some information about future task distribution is available.\n\n> It is a little unclear to me why only the initialization condition is sufficient for Theorem 1 to hold?\n\nThe proof (included in the Appendix) contains the details. When the initial condition is met along with step-size and data stream requirements, the predictions of our algorithm match those learned using TD-learning. This is not surprising because the transient memory updates use TD-learning, but the semi-gradient is with respect to the transient component only instead of the overall estimate $V^{PT}$.\n\n> Shouldn't the equation in Theorem 2 be what is in Equation 5 i.e. PT instead of P?\n\nIndeed, the Bellman operator in Theorem 2 is obtained by using the target in Eq. 5. $\\gamma P_{\\pi} V^P$ and $\\gamma P_{\\pi} V^T$ add up to $\\gamma P_{\\pi} V^{PT}$ by definition.\n\n> It may be worthwhile to see how Continual backprop can be incorporated into the proposed method.\n\nOur method performs a trade-off between stability and plasticity in the value function, that is agnostic to the nature of the function approximator. Continual backprop is an optimizer explicitly designed to promote neuron-level plasticity in neural networks. In principle, if the value functions are implemented using neural networks, this optimizer can be used to improve plasticity further. We expect that using continual backprop would be especially useful to update the permanent memory network as it runs the risk of losing neuron-level plasticity when aggregating the value estimates from several tasks. This is an interesting direction to explore.\n\nWe will edit the paper to incorporate these clarifications as well, let us know if you have further comments or questions.\n\n*References:*\n\n[1] David Silver, Richard S Sutton, and Martin Müller. Sample-based learning and search with permanent and transient memories. In Proceedings of the 25th international conference on Machine learning, pages 968–975, 2008."}, {"review_id": "4CvT12g96N", "reviewer": "Reviewer_zZwF", "summary": "This paper proposes new algorithms for prediction and control in continual reinforcement learning. The new algorithms use permanent and temporary memories. The permanent memory captures a general value function for all the tasks, while the transient memory is used to adapt to the current tasks continually. The paper provides theoretical results for the prediction setting with a tabular learner. Additionally, the paper provides empirical results showing that their method outperforms vanilla TD-based algorithms in various environments using different types of function approximators.", "questions": "1. Line 88 says, \"... multi-task RL methods ...\" What is multi-task RL, and how is it different from continual RL? What limits the use of multi-task RL methods in continual RL? The second question should be answered in the main text.\n2. Figure 4d shows the performance of different versions of DQN  on Minigrid. Why is the performance of all the methods, particularly DQN(reset), so much worse during the second task? If the DQN is being reset at each task, it should perform similarly on Episodes 500 and 1000. Is this because the first goal state in all the runs is the same?\n3. What is PT-DQN-0.5x in Figure 5? 0.5x should be described in the main paper.\n4. Lines 733-742 describe the training details for the deep RL experiments. However, many of these choices seem arbitrary, or the paper does not explain the reasoning behind these choices. Why are the conv layers not updated when updating the permanent memory? Why isn't there a fully independent permanent value network? Wouldn't that be closer to the algorithms presented in the paper?\n5. Lines 803-805 say that the permanent memory was trained with SGD while the rest of the network was trained with Adam. Why this specific choice? I'm guessing the authors also tried Adam to update the permanent memory. Why didn't Adam work for permanent memory?\n6. The plots (like Figure C.13) showing the sensitivity curve will be better with a log scale for the x-axis.\n7. Why isn't there any effect of PM LR on the performance in the minigrid environment (Figure C.13c)? ", "limitations": "One limitation of this work is that it only proposes methods for value-based RL. It is unclear how the idea of permanent and transient memory can be extended to policy-gradient based RL. A discussion on this direction of future work will be helpful.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper directly tackles the continual reinforcement learning setting.\nThis is a challenging problem setting with few general algorithms and evaluation environments.\nThe paper provides prediction and control algorithms for the continual RL setting.\nTheoretical proofs of convergence are provided for the prediction setting with a tabular learner.\nThe extensive and sound empirical investigation shows that the proposed method can generally outperform existing methods in various environments and with different various function approximations, including Deep Networks.   ", "weaknesses": "The paper does not have any major weaknesses. I did not check the proofs, so I can not comment on the correctness of the proofs.\nHowever, there are a few minor issues that can be fixed to improve the paper.\n1. I'm confused about Equation 5, specifically line 112. Does it mean the \"gradient\" or the \"semi-gradient\" of the transient memory?\n2. What are the confidence intervals being reported? Line 222 says it is the 90% confidence interval, but is the bootstrapped confidence interval or confidence interval based on the standard error? And if it is based on the standard error, how do you know it is the 90% confidence interval?\n3. The continual deep RL algorithms used in the paper use standard deep RL tools like Experience replay and target networks. However, the exact use of these tools is not fully explained in the paper (or the appendix). This creates a lot of questions; for example, are there a total of three networks, target, transient and permanent? I suggest that the authors include a section in the appendix that gives the pseudocode of the PT-DQN algorithm. The lack of details about the PT-DQN algorithm makes it hard to replicate the results given in the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and positive comments on our paper. Please find our answers to your questions below:\n\n> I'm confused about Equation 5, specifically line 112. Does it mean the \"gradient\" or the \"semi-gradient\" of the transient memory?\n\nThis is a semi-gradient update rule, in the same sense used to describe TD-learning, as the gradient is with respect to the current estimate only and not the target. Since the permanent and the transient memories are independently parameterized, the semi-gradient ends up being just $\\nabla_w V_w^T(S_t)$.\n\n> What are the confidence intervals being reported?\n\nWe reported standard distribution confidence intervals with a z-score of 1.645 and a population size of 30 (seeds). We used $\\mu \\pm z \\frac{\\sigma}{\\sqrt{n}}$ to represent the shaded region, where $\\mu$ is the mean performance, $\\sigma$ is the standard deviation of the performance over the population, and $n$ is the number of seeds.\n\n> [...] pseudocode of the PT-DQN algorithm. [...]\n\nWe value reproducibility and we will release the code for all the experiments so others can replicate our results. We will also expand the appendix to include the pseudocode of the continual deep RL algorithms along with the details of the deep RL tools used in experiments to provide more clarity.\n\n> [...] What is multi-task RL [...]\n\nWe define the multi-task setting in lines 67-69, and we will expand this description. We use the term multi-task RL to mean that the agent is confronted with a distribution of RL problems (tasks) and that during learning, the agent is aware of the task ID and/or of whether the task has changed since its previous episode. In some cases, the agent can sample tasks from a distribution, thereby controlling the data it observes to a certain extent. Methods in multi-task RL are designed to make use of this additional information, which is not available in continual RL (for example, see DQN-multi-head baseline Section 6). These methods have certain limitations as described in lines 32-35 when applied to continual RL: they require the agent to first identify the task and update appropriate parameters, which limits the number of tasks and the kind of non-stationarity (only piece-wise) that can be considered.\n\n> [...] so much worse during the second task? [...]\n\nThe first and second goals and the task order are the same for all baselines and for all runs. The DQN agent took longer to solve the second task when we trained it on each task separately (see performance between 500-1000 episodes, and 1500-2000 episodes). We suspect this is because the first goal is closer to the start state (it is the goal on the left), and therefore easier for the agent to discover during random exploration. We are unsure why the gap is as large as it is, and we will investigate this further. Note that the other methods, which are not reset, can leverage some of the previous learning, and they adapt faster to this goal than the reset baseline.\n\n> What is PT-DQN-0.5x in Figure 5?\n\nWe will add details in the main paper. It is a version in which the permanent and the transient memory networks have respectively half the number of parameters used by the DQN network, to ensure the total number of parameters across all baselines is the same. Strictly speaking, DQN-based baselines use more parameters compared with PT-DQN-0.5x because their target networks are 2x larger than the target network of the transient memory.\n\n> [...] the paper does not explain the reasoning behind these choices. [...]\n\nSince the task boundaries are observed, we chose to train our method in a similar manner to multi-task RL methods (for example, see DQN-multi-head baseline). We only update the corresponding head to learn the permanent value because the features are already learned when updating the transient memory. We will clarify this in the text.\n\n> Why isn't there a fully independent permanent value network? [...]\n\nIndeed, we parameterize the permanent and transient memory networks independently in the continual RL experiments (see Section 6) to align with the algorithm introduced in the paper. In general, one may in fact want different network architectures for these, not just different parameters, but this goes beyond the scope of this paper.\n\n> [...] the permanent memory was trained with SGD while the rest of the network was trained with Adam. [...]\n\nThe permanent memory network should take a small step during each update in order to consolidate the long-term value function information. SGD's updates are more aligned with this intuition, which is why we preferred it. We also tested the Adam optimizer initially; the performance was poor with default hyperparameters. We expect Adam's performance could match or exceed SGD's when all its hyperparameters are fine-tuned, but we did not try that.\n\nWe used the standard DQN training procedure (experience replay, Adam optimizer, target network) for training the DQN baselines, and the transient memory network.\n\n> The plots [...] with a log scale for the x-axis.\n\nWe will incorporate this suggestion in the final version of the appendix.\n\n> Why isn't there any effect of PM LR [...]\n\nIn general, we observed a good performance for a wide range of PM LRs in various environments. In minigrid, the difference seems to be small compared with other environments; perhaps learning permanent values is easier in that environment.\n\n> Extension to policy-based RL\n\nWhile the paper focuses on value-based RL, there is a straightforward way to extend the work to actor-critic methods, by simply using the proposed value function estimates to compute the gradient of the policy (represented as a separate approximator). Another possibility is to have a \"permanent\" policy, which is corrected by a transient component to adapt to the task at hand. Both would be worth exploring in future work and we will reflect this in the discussion. \n\nThank you again for all your questions and suggestions to improve the paper. "}, {"review_id": "saWjngV6f0", "reviewer": "Reviewer_mEyq", "summary": "In this work, the author proposes a new continual reinforcement learning method, the PT-TD, which consists of two value networks, one learns the transient memories and the other learns the permanent memories. The author proves that in the semi-continual setting, the PT-TD algorithm will get better performance and lower forgetting, and the same result can also be found in the continual reinforcement learning setting.", "questions": "I think the formulation of the Theorem 8 is a little confusing. Maybe you can give out explicitly which network is using (P or T).\n\nCan your algorithm generalize to continuous action space? How should I learn the actor network at this time?\n\nI notice that in the continual reinforcement learning setting, the learning rate of T network is always higher than it of P network. According to [1], if it is the difference in the learning rate affect good performance?\n\n[1]Arani, E. et al. “Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System.” ArXiv 2021", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The algorithm of this article is well-proven and well-demonstrated. The author not only proves the PT-TD algorithm in a mathematical way but also gives out detailed and rigorous experiments. Also, the article is really well-written and easy to understand. All of the hyperparameters are studied. Meanwhile, it is appreciated that the author discussed semi-continual reinforcement learning and continual reinforcement learning separately.", "weaknesses": "First, even though the proof is very well, the algorithm itself is not very groundbreaking, or, the result is not very rewarding. Besides, the environments of the experiments are simple.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and positive comments on our paper. Please find our answers to your questions below:\n\n> 1. I think the formulation of the Theorem 8 is a little confusing. Maybe you can give out explicitly which network it is using (P or T).\n\nWe will update the statement in the main paper using the one provided in the appendix (Theorem B.3.4) to make it clear. Intuitively, theorem 8 states that the prediction errors on a new task are lower for our method compared with TD-learning. The theorem follows Algorithm 1 where we fix the parameters of the permanent network and update only the transient network using online data. The overall estimate is the sum of the predictions of the two networks.\n\n> Can your algorithm generalize to continuous action space? How should I learn the actor network at this time?\n\nQ-learning with continuous actions can be implemented by passing the observation and action as input to estimate the Q-values. The permanent and transient value functions could then be used as in the paper. Alternatively, the action space can be discretized to learn Q-values using the network architecture described in the paper. \n\nAnother possibility is to pursue policy gradient approaches. There is a straightforward way to extend the work to actor-critic methods, by simply using the proposed value function estimates to compute the gradient of the policy (represented as a separate approximator). Another possibility is to have a \"permanent\" policy, which is corrected by a transient component to adapt to the task at hand. Both would be worth exploring in future work and we will reflect this in the discussion.\n\n> I notice that in the continual reinforcement learning setting, the learning rate of T network is always higher than it of P network. According to [1], if it is the difference in the learning rate affect good performance?\n\nThe difference in learning rates is part of the answer, but other aspects of the algorithm play a bigger role:\n\n(a) The permanent and the transient memories are updated at different timescales. The permanent network is updated at a slower timescale (every $k$ steps or at the end of the task). The transient memory is updated at a faster timescale (every timestep), therefore, it absorbs errors immediately (fast);\n\n(b) The update rules of permanent and transient memories are different. The permanent memory is updated to capture some part of the value function from all tasks that the agent has seen in its lifetime (generalization), which is a slow process. The transient memory adapts to a specific situation, which is a fast process.\n\nThe fast and slow interplay on various levels results in good performance on non-stationary problems.\n\n> The algorithm itself is not very groundbreaking, or, the result is not very rewarding. Besides, the environments in the experiments are simple.\n\nWe deliberately tried to keep our approach as simple as possible and build on top of standard TD-learning and Q-learning algorithms. This is desirable as TD-learning and Q-learning have demonstrated generality and scalability on a broad range of problems over time, despite (or perhaps because) of being simple. \n\nRegarding the experiments, we used domains that are typical in semi-continual RL and continual RL (Minigrid, JellyBeanWorld, and MinAtar). The JellyBeanWorld and MinAtar experiments in particular, while not being \"real applications\", do test scaling.\nThe type of experiments we ran is consistent with other papers in the field, but the tasks we used are harder than in earlier works. For instance, [1] uses catcher and flappy bird with a curriculum of tasks, while the non-stationarity in our setup is much more severe. [2] used only one continual RL problem - slippery ant. Any further concrete suggestions for experiments would be useful.\n\n*References:*\n\n[1] Riemer, et al. \"Learning to learn without forgetting by maximizing transfer and minimizing interference.\" arXiv preprint arXiv:1810.11910 (2018).\n\n[2] Dohare, et. al. \"Continual backprop: Stochastic gradient descent with persistent randomness.\" arXiv preprint arXiv:2108.06325 (2021)."}]

Author rebuttal:
Thank you for your time and useful suggestions for improving our paper. We appreciate the positive feedback on the ideas, their connections to complementary learning systems and the theoretical analysis. We address specific questions below and would be happy to provide any further clarifications.

Meta-review: This paper focuses on continual RL and proposes to decompose the value function into two parts: persistent and transient memory. The proposal is backed up by theory and the authors also demonstrate the use of the proposed algorithm in several empirical settings.

The authors addressed most of the reviewers' concerns and reviewers also raised their scores. There were some complaints about the need for more large-scale experiments. But I think the work is interesting enough and worth publishing. I recommend an acceptance.

I encourage the authors to

1. Add a detailed limitation section to the paper.
2. A detailed pseudocode of the algorithm in the appendix.
3. Incorporate all the clarifications made during the rebuttal period.
Decision: Accept (poster)