SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Continuous grid: is a continuous version of the previous task. The state space is the square  $[0,1]\times [0,1]$ , with the starting state sampled uniformly from  $[0.45, 0.55]\times [0.45, 0.55]$ . The regions within 0.1 units (in norm-1) from the corners are goal states. The actions change the state by 0.1 units along the corresponding direction. To make the transitions stochastic, we add noise to the new state,
2We report standard distribution confidence intervals with a z-score of 1.645. We use  $\mu \pm z\frac{\sigma}{\sqrt{n}}$  to represent the shaded region.
sampled uniformly from  $[-0.01, 0.01] \times [-0.01, 0.01]$ . Rewards are positive only for transitions entering a goal region. The goal reward for each task is given in Table C.1. In the linear function approximation setting we convert the agent's state into a feature vector using the radial basis function as described in [45] and Appendix C.1.1. We run 2000 episodes, changing goal rewards every 200 episodes. For evaluation, we sample 225 evenly spaced points in the grid and estimate their true values by averaging Monte Carlo returns from 500 episodes across 10 seeds for each task. Here,  $\gamma = 0.99$ .
Minigrid: We used the four rooms environment shown in Fig. 1b. Each room contains one type of item and each item type has a different reward and color. Goal states are located in all corners. The episode terminates when the agent reaches a goal state. The agent starts in the region between cells (3,3) and (6,6) and can move forward, turn left, or turn right. The agent perceives a 5x5 (partial) view in front of it, encoded one-hot. The rewards for picking items change as described in Table C.2 and the discount factor is 0.9. We run 750 episodes and change rewards every 75 episodes.
We provide the training and architecture details for the two deep RL prediction experiments in Appendix (Sec. C.1.1 and C.1.2) due to space constraints.
**Observations:** The results are shown in Fig. 2. Solid lines indicate performance on the task currently experienced by the agent (i.e., the online task) and the dotted lines indicate the performance on the other tasks. The reset variant of TD-learning performs poorly because all learning progress is lost when the weights are reset, leading to the need for many samples after every task change. TD-learning also has high online errors immediately after a task switch, as its predictions are initially biased towards the past task. As the value function is updated using the online interaction data, the error on the current task reduces and the error on other tasks rises, highlighting the stability-plasticity problem when using a single value function estimate. Our algorithm works well on both performance measures. The permanent value function learns common information across all tasks, resulting in lower errors on the second performance measure throughout the experiment. Also, due to the good starting point provided by the permanent value function, the transient component requires little data to make adjustments, resulting in fast adaptation on the new task. In our method, the permanent component provides stability and the transient component provides plasticity, resulting in a good balance between the two. We also plotted the heatmap of the value estimates, shown in Fig. C.3, to understand the evolution of the predictions for TD-learning and our method which confirms the observations made so far.
5.2 Control experiments
To test our approach as a complement to Q-learning 3, we conducted a tabular gridworld experiment and a Minigrid experiment using deep neural networks.
(a) Toy grid.
(b) Tabular results.
(c) Minigrid.
Figure 3: The environments and the corresponding results in the control setting. (b) Tabular results: episodic returns are plotted. (d) Deep RL results: returns smoothened across 10 episodes is plotted.
(d) DRL results.
Tabular experiment: We use the 6x6 grid shown in Fig. 3a, with two goals located in the top right corner. The agent starts in the bottom left corner and can choose to go up, down, left, or right. The action transitions the agent to the next state along the corresponding direction if the space is empty and keeps it in the same state if the transition would end up into a wall or obstacle. The action has the intended effect  $90\%$  of the time; otherwise, it is swapped with one of the perpendicular actions. The agent receives a reward of  $+1$  and  $-1$  for reaching the two goal states respectively, and the rewards swap periodically, creating two distinct tasks. The discount factor is 0.95. We run 500 episodes and change the task every 50 episodes.
Minigrid experiment: We use the two-room environment shown in Fig. 3c. The agent starts in the bottom row of the bottom room and there are two goal states located in the top room. The goal
rewards are  $+5$  and 0 respectively, and they flip when the task changes. We run 2500 episodes, alternating tasks every 500 episodes. The other details are as described in the previous section. Further details on the training and architecture are described in Appendix (Sec. C.1.3).
**Observations:** The results are shown in Fig. 3. The reset variant of Q-learning has to learn a good policy from scratch whenever the task changes, therefore, it requires many samples. Since, the goal rewards are flipped from one task to another, Q-learning also requires many samples to re-adjust the value function estimates. In our method, since the permanent component mixes action-values across tasks, the resulting bias is favorable, and makes it fairly easy to learn the corrections required to obtain overall good estimates. So, the transient component requires comparatively few samples to find a good policy when the task changes.
6 Continual Reinforcement Learning
So far, we assumed that the agent observes task boundaries; This information is usually not available in continual RL. One approach that has been explored in the literature is to have the agent infer, or learn to infer, the task boundaries through observations (e.g. see [17, 23]). However, these methods do not work well when task boundaries are not detected correctly, or simply do not exist. We will now build on the previous algorithm we proposed to obtain a general version, shown in Alg. 2, which is suitable for continual RL.
Our approach maintains the permanent and transient components, but updates them continually, instead of waiting for task changes. The permanent component is updated every  $k$  steps (or episodes) using the samples stored in the buffer  $\mathcal{B}$ . And, instead of resetting the transient component, it is now decayed by a factor  $\lambda \in [0,1]$ . Here,  $k$  and  $\lambda$  are both hyperparameters. When
Algorithm 2 PT-Q-learning (CRL)
1: Initialize: Buffer B, θ, w, k, λ
2: for t:0→∞ do
3: Take action At
4: Store St, At in B
5: Observe Rt+1, St+1
6: Update wt using Eq. (8)
7: if mod(t,k) == 0 then
8: Update θ using B and Eq. (7)
9: wt+1← λwt+1
10: Reset B
11: end if
12: end for
$\lambda = 0$  and  $k$  the duration between task switches, we obtain the previously presented algorithm as a special case. In general,  $k$  and  $\lambda$  control the stability-plasticity trade off. Lower  $\lambda$  means that the transient value function forgets more, thus retaining more plasticity. Lower  $k$  introduces more plasticity in the permanent value function, as its updates are based on fewer and more recent samples. The algorithm still retains the idea of consolidating information from the transient into the permanent value function.
The effect of  $k$  and  $\lambda$ : To illustrate the impact of the choice of hyperparameters, we use the tabular task shown in Fig. 3a. The details of the experiment are exactly as before, but the agent does not know the task boundary. We run the experiment for several combinations of  $k$  and  $\lambda$  for the best learning rate. The results are shown in Fig. 4. As expected, we observe that the hyperparameters  $k$  and  $\lambda$  have co-dependent effects to a certain degree. For small values of  $k$ , large values of  $\lambda$  yield better performance, because the variance in the permanent value function is mitigated, and the transient value function in fact is more stable. For large values of  $k$ , the transient value function receives enough updates before the permanent value function is trained, to attain low prediction error. Therefore, the updates to the permanent value function are effective, and the transient predictions can be decayed more aggressively. More surprisingly, if the frequency of the permanent updates is significantly higher or significantly lower than the frequency of task changes, the overall performance is marginally affected, but remains better than for Q-learning (indicated using black dotted line) for most  $k$  and  $\lambda$  values. In fact, for each  $k$ , there is at least one  $\lambda$  value for which the performance of the proposed algorithm exceeds the Q-learning baseline. So, in practice, we could fix one of these hyperparameters beforehand and tune only the second. We suspect that the two parameters could be unified into a single one, but this is left for future work.
6.1 Experiments
We test our algorithm on two domains: JellyBeanWorld (JBW) [31] and MinAtar [51]. We chose these domains because of the relatively lighter computational requirements, which still allow us to
sweep a range of hyperparameters and to report statistical results averaged over 30 seeds. We include architecture and further training details in Appendix (Sec. C.1.5, C.1.6).
JellyBeanWorld is a benchmark to test continual RL algorithms [31]. The environment is a two-dimensional infinite grid with three types of items: blue, red, and green, which carry different rewards for different tasks as detailed in Sec. C.1.5. In addition to reward non-stationarity, the environment has spatial non-stationarity as shown in Fig. 5a. The agent observes an egocentric  $11 \times 11$  RGB image and chooses one of the four usual navigation actions. We run 2.1M timesteps, with the task changing every 150k steps. The discount factor is 0.9.
MinAtar is a standard benchmark for testing single-task RL algorithms [51]. We use the breakout, freeway, and space invaders environments. We run for 3.5M steps and setup a continual RL problem, by randomly picking one of these tasks every 500k steps. We standardize the observations across tasks by padding with 0s when needed
Figure 4:  $\lambda$  and  $k$  analysis.
and we make all the actions available to the agent. This problem has non-stationarity in transitions, rewards and observations making it quite challenging. The discount factor is 0.99. More details are included in Appendix (Sec. C.1.6).
Baselines: We compare our method with DQN, two DQN variants, and a uniformly random policy. For the first baseline, we use a large experience replay buffer, an approach which was proposed recently as a viable option for continual RL [8]. The second baseline, DQN with multiple heads, uses a common trunk to learn features and separate heads to learn Q-values for the three tasks. This requires knowing the identity of the current task to select the appropriate head. We use this baseline because it is a standard approach for multi-task RL, and intuitively, it provides an upper bound on the performance. This approach is not suitable if the number of tasks is very high, or if the problem is truly continual and not multi-task. For our approach, we use half the number of parameters as that of DQN for both permanent and transient value networks to ensure the total number of parameters across all baselines are same — hence the name PT-DQN-0.5x. $^3$
(a) JellyBeanWorld.
(b) JellyBeanWorld Results.
(c) MinAtar.
Figure 5: (a) JellyBeanWorld task. (b) Results: JellyBeanWorld. (c) MinAtar tasks. (d) Results: MinAtar.
(d) MinAtar results.
Results are presented in Fig. 5 and the pseudocode of PT-DQN is included in Appendix C.1.4. For JBW experiment, we plot the reward obtained per timestep over a 10k step window (reward rate) as a function of time. For MinAtar, we plot the return averaged over the past 100 episodes. Our method, PT-DQN-0.5x, performs better than all the baselines. The performance boost is large in the JBW experiment, where the rewards are completely flipped, and still favourable in the MinAtar experiment. DQN's performance drops over time when it needs to learn conflicting Q-values, as seen in the JBW results. DQN also needs a large number of samples to overcome the bias in its estimates when the task changes. Surprisingly, multi-headed DQN performs poorly compared to our method, despite having the additional task boundary information. The weights of the trunk are shared, and they vary continuously over time. Therefore, the weights at the head become stale compared to the features, which degrades performance. These findings may be different if the trunk were pre-trained and then fixed, with only the heads being fine-tuned, or if the networks were much bigger (which is however not warranted for the scale of our experiment). Augmenting DQN with a large experience replay is detrimental for general non-stationary problem, based on our experiments. This is because the samples stored in the replay buffer become outdated, so using these samples hampers the agent's ability to track environment changes.
${}^{3}$  Strictly speaking, DQN-based baselines use more parameters compared with ours because their target networks are  $2\mathrm{x}$  larger than the target network of the transient value function.
Designing agents that can learn continually, from a single stream of experience, is a crucial challenge in RL. In this paper, we took a step in this direction by leveraging intuitions from complementary learning systems, and decomposing the value function into permanent and transient components. The permanent component is updated slowly to learn general estimates, while the transient component computes corrections quickly to adapt these estimates to changes. We presented versions adapted to both semi and fully continual problems, and showed empirically that this idea can be useful. The theoretical results provide some intuition on the convergence and speed of our approach.
The non-stationarity in our experiments is much more challenging, including arbitrary reward changes, as well as dynamics and observation changes, compared to other works, which use specific structure, or rely on a curriculum of tasks (thereby limiting non-stationarity). Our approach provided improvements in performance compared to the natural TD-learning and Q-learning baselines.
Our approach performs fast and slow interplay at various levels. The permanent value function is updated at a slower timescale (every  $k$  steps or at the end of the task); The transient value function is updated at a faster timescale (every timestep). The permanent value function is updated to capture some part of the value function from the agent's entire experience (generalization), which is a slow process; The transient value function adapts the estimates to a specific situation, which is a fast process. The permanent value function is updated using a smaller learning rate (slow), but the transient value function is updated using a larger learning rate (fast) [3].
7.1 Limitations and Future Work
While we focused on illustrating the idea of permanent and transient components with value-based algorithms with discrete action spaces, the approach is general and could in principle be used with policy gradient approaches. One could use the proposed value function estimates to compute the gradient of the policy (represented as a separate approximator) in actor-critic methods. Another possibility is to have a "permanent" policy, which is corrected by a transient component to adapt to the task at hand. These directions would be interesting to explore. Our idea can be extended to learning General Value Functions [47] too.
Our theoretical analysis is limited to piece-wise non-stationarity, where the environment is stable for some period of time. An interesting direction would be to extend our approach to other types of non-stationarity, such as continuous wear and tear of robotic parts.
Another important future direction is to consider distinct feature spaces for the permanent and transient components, as done in Silver et al. [38]. This would likely increase the ability to trade off stability and plasticity. Another direction to better control the stability and plasticity trade-off is by learning the frequency parameter  $k$  and the decay parameter  $\lambda$  using meta objectives [16]; We suspect there's a relation between the two parameters that can unify them into a single one.
Our approach can also be used to bridge offline pre-training and online fine-tuning of RL agents. The permanent value function can be learnt using offline data and these estimates can be adjusted by computing corrections using the transient value function using online interaction data.
We are grateful to NSERC, CIFAR, and Microsoft Research grants for funding this research. We thank Jalaj Bhandari, Maxime Webartha, Matthew Reimer, Blake Richards, Emmanuel Bengio, Harsh Satija, Sumana Basu, Chen Sun, and many other colleagues at Mila for the helpful discussions; Will Dabney, Harm Van Seijen, Rich Sutton, Razvan Pascanu, David Abel, James McClelland and all of the 2023 Barbados workshop participants for their feedback on the project; the anonymous reviewers for the constructive feedback on the final draft. This research was enabled in part by support provided by Calcul Quebec and the Digital Research Alliance of Canada.
[37] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta... hook. PhD thesis, Technische Universität München, 1987.
[38] David Silver, Richard S Sutton, and Martin Müller. Reinforcement learning of local shape in the game of go. In *IJCAI*, volume 7, pages 1053–1058, 2007.
[39] David Silver, Richard S Sutton, and Martin Müller. Sample-based learning and search with permanent and transient memories. In Proceedings of the 25th international conference on Machine learning, pages 968-975, 2008.
[40] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
[41] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
[42] Satinder Singh and Peter Dayan. Analytical mean squared error curves for temporal difference learning. Machine Learning, 32(1):5-40, 1998.
[43] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.
[44] Richard S Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI, pages 171-176. San Jose, CA, 1992.
[45] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[46] Richard S Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. In Proceedings of the 24th international conference on Machine learning, pages 871-878, 2007.
[47] Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pages 761-768, 2011.
[48] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
[49] Yee Teh, Victor Bapat, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
[50] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
[51] Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176, 2019.
A Control algorithm
The action-value function can be decomposed into two components as:
$$
Q ^ {(P T)} (s, a) = Q _ {\theta} ^ {(P)} (s, a) + Q _ {\mathbf {w}} ^ {(T)} (s, a), \tag {6}
$$
The updates for permanent value function are:
$$
\theta_ {k + 1} \leftarrow \theta_ {k} + \bar {\alpha} _ {k} \left(\hat {Q} \left(S _ {k}, A _ {k}\right) - Q _ {\theta} ^ {(P)} \left(S _ {k}, A _ {k}\right)\right) \nabla_ {\theta} Q _ {\theta} ^ {(P)} \left(S _ {k}, A _ {k}\right). \tag {7}
$$
where  $\overline{\alpha}$  is the learning rate. An update rule similar to Q-learning can be used for the transient value function:
$$
\mathbf {w} _ {t + 1} \leftarrow \mathbf {w} _ {t} + \alpha_ {t} \delta_ {t} \nabla_ {\mathbf {w}} Q _ {\mathbf {w}} ^ {(T)} \left(S _ {t}, A _ {t}\right), \tag {8}
$$
where  $\delta_t = R_{t+1} + \gamma \max_a Q^{(PT)}(S_{t+1}, a) - Q^{(PT)}(S_t, A_t)$  is the TD error.
Algorithm 3 PT-Q-learning (Control)
1: Initialize: Buffer  $\mathcal{B}$ ,  $\theta$ , w  
2: for  $t:0\to \infty$  do  
3: Take action  $A_{t}$   
4: Store  $S_{t}$ ,  $A_{t}$  in  $\mathcal{B}$   
5: Observe  $R_{t + 1}, S_{t + 1}$   
6: Update w using Eq. (8)  
7: if Task Ends then  
8: Update  $\theta$  using  $\mathcal{B}$  and Eq. (7)  
9: Reset transient value function, w  
10: Reset  $\mathcal{B}$   
11: end if  
12: end for
B.1 Transient Value Function Results
We carry out the analysis by fixing  $V^{(P)}$  and only updating  $V^{(T)}$  using samples from one particular task. We use  $V_{t}^{(TD)}$  and  $V_{t}^{(PT)}$  to denote the value function estimates at time  $t$  learned using TD learning and Alg. 1 respectively.
Theorem B.1.1. If  $V_0^{(TD)} = V^{(P)}$ ,  $V_0^{(T)} = 0$  and the two algorithms train on the same samples using the same learning rates, then  $\forall t, V_t^{(PT)} = V_t^{(TD)}$ .
Proof. We use induction to prove this statement.
Base case: When  $t = 0$ , the PT-estimate of state  $s$  is  $V_0^{(PT)}(s) = V^{(P)}(s) + V_0^{(T)}(s) = V^{(P)}(s)$ . And, the TD estimate of state  $s$  is  $V_0^{(TD)}(s) = V^{(P)}(s)$ . Therefore, both PT and TD estimates are equal.
Induction hypothesis: PT and TD estimates are equal for some timestep  $k$  for all states. That is,  $V_{k}^{(PT)}(s) = V^{(P)}(s) + V_{k}^{(T)}(s) = V_{k}^{(TD)}(s)$ .
Induction step: PT estimates at timestep  $k + 1$  is
$$
\begin{array}{l} V _ {k + 1} ^ {(P T)} (s) = V ^ {(P)} (s) + V _ {k + 1} ^ {(T)} (s), \\ = V ^ {(P)} (s) + V _ {k} ^ {(T)} (s) + \alpha_ {k + 1} (R _ {k + 1} + \gamma (V ^ {(P)} (S _ {k + 1}) + V _ {t} ^ {(T)} (S _ {k + 1})) - V ^ {(P)} (s) - V _ {k} ^ {(T)} (s)), \\ \end{array}
$$
$$
\begin{array}{l} = V _ {k} ^ {(P T)} (s) + \alpha_ {k + 1} (R _ {k + 1} + \gamma V _ {k} ^ {(P T)} (S _ {k + 1}) - V _ {k} ^ {(P T)} (s)), \\ = V _ {k} ^ {(T D)} (s) + \alpha_ {k + 1} (R _ {t + 1} + \gamma V _ {k} ^ {(T D)} (S _ {t + 1}) - V _ {k} ^ {(T D)} (s)), \\ \boxed {= V _ {k + 1} ^ {(T D)} (s).} \\ \end{array}
$$
The penultimate step follows from the induction hypothesis completing the proof.
Theorem B.1.2. The expected target in the transient value function update rule is a contraction-mapping Bellman operator:  $\mathcal{T}^{(T)}V^{(T)} = r_{\pi} + \gamma \mathcal{P}_{\pi}V^{(P)} - V^{(P)} + \gamma \mathcal{P}_{\pi}V^{(T)}$
Proof. The target for state  $s$  at timestep  $t$  in the transient value function update rule is  $R_{t + 1} + \gamma V^{(P)}(S_{t + 1}) - V^{(P)}(s) + \gamma V^{(T)}(S_{t + 1})$ . The expected target is:
$$
\begin{array}{l} \mathcal {T} ^ {(T)} V ^ {(T)} (s) = \mathbb {E} _ {\pi} \left[ R _ {t + 1} + \gamma V ^ {(P)} \left(S _ {t + 1}\right) - V ^ {(P)} (s) + \gamma V ^ {(T)} \left(S _ {t + 1}\right) \mid S _ {t} = s \right], \\ = r _ {\pi} (s) + \gamma (\mathcal {P} _ {\pi} V ^ {(P)}) (s) - V ^ {(P)} (s) + \gamma (\mathcal {P} _ {\pi} V ^ {(T)}) (s). \\ \end{array}
$$
We get the operator by expressing the above equation in the vector form: