Title: Prediction and Control in Continual Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

continual reinforcement learning

lifelong learning

never-ending learning

prediction

control

multi-task learning

complementary learning systems
Abstract: Temporal difference (TD) learning is often used to update the estimate of the value function which is used by RL agents to extract useful policies. In this paper, we focus on value function estimation in continual reinforcement learning. We propose to decompose the value function into two components which update at different timescales: a _permanent_ value function, which holds general knowledge that persists over time, and a _transient_ value function, which allows quick adaptation to new situations. We establish theoretical results showing that our approach is well suited for continual learning and draw connections to the complementary learning systems (CLS) theory from neuroscience. Empirically, this approach improves performance significantly on both prediction and control problems.

Prediction and Control in Continual Reinforcement Learning

Abstract

Temporal difference (TD) learning is often used to update the estimate of the value function which is used by RL agents to extract useful policies. In this paper, we focus on value function estimation in continual reinforcement learning. We propose to decompose the value function into two components which update at different timescales: a permanent value function, which holds general knowledge that persists over time, and a transient value function, which allows quick adaptation to new situations. We establish theoretical results showing that our approach is well suited for continual learning and draw connections to the complementary learning systems (CLS) theory from neuroscience. Empirically, this approach improves performance significantly on both prediction and control problems.

1 Motivation

Deep reinforcement learning (RL) has achieved remarkable successes in complex tasks, e.g Go [40, 41] but in a narrow setting, where the task is well-defined and stable over time. In contrast, humans continually learn throughout their life and adapt to changes in the environment and in their goals. This ability to continually learn and adapt will be crucial for general AI agents who interact with people in order to help them accomplish tasks. One possible explanation for this ability in the natural world is the existence of complementary learning systems (CLS): one which slowly acquires structured knowledge and the second which rapidly learns specifics adapting to the current situation [26]. In contrast, in RL, learning is typically focused on how to optimize return, and the main estimated quantity is one value function (which then drives learning a policy). As a result, when changes occur in the environment, RL systems are faced with the stability-plasticity dilemma [9]: whether to forget past predictions in order to learn new estimates or to preserve old estimates, which may be useful again later, and compromise the accuracy on a new task.

The stability-plasticity dilemma in RL is agnostic to the type of function approximation used to estimate predictions, and it is not simply a side effect of the function approximator having small capacity. In fact, the problem exists even when the RL agent uses a table to represent the value function. To see this, consider policy evaluation on a sequence of tasks whose true value function is different due to different rewards and transition probabilities. A convergent temporal-difference (TD) algorithm [43] aggregates information from all value functions in the task distribution implicitly, thereby losing precision in the estimate of the value function for the current situation. A tracking TD learning algorithm [46] learns predictions for the current task, overwriting estimates of past tasks and re-learning from scratch for each new task, which can require a lot of data. Using a separate function approximator for each task, which is a common approach [23], requires detecting the identity of the current task to update accordingly, which can be challenging and limits the number of tasks that can be considered.

As an alternative, we propose a CLS-inspired approach to the stability-plasticity dilemma which relies on maintaining two value function estimates: a permanent one, whose goal is to accumulate "baseline" knowledge from the entire distribution of information to which the agent is exposed over time, and a transient component, whose goal is to learn very quickly using information that is relevant to the current circumstances. This idea has been successfully explored in the context of RL and tree search in the game of Go [39], but the particular approach taken uses domain knowledge to construct features and update rules. Our approach is general, simple, online and model-free, and can be used in continual RL. Our method is also orthogonal to various advancements in the continual RL field, and therefore, it can be combined with other improvements. Our contributions are:

A conceptual framework for defining permanent and transient value functions, which can be used to estimate value functions;

RL algorithms for prediction and control using permanent and transient value functions;

Theoretical results which help clarify the role of these two systems in the semi-continual (multi-task) RL setting; and

Empirical case studies of the proposed approaches in simple gridworlds, Minigrid [11], JellyBeanWorld (JBW) [31], and MinAtar environments [51].

2 Background

Let S be the set of possible states and A the set of actions. At each timestep t , the agent takes action A t ∈ A in state S t according to its (stochastic) policy π : S → Dist ( A ) . As a consequence, the agent receives reward R t +1 and transitions to a new state S t +1 . Let G t = ∑ ∞ k = t γ k -t R k +1 be the discounted return obtained by following π from step t onward, with 0 < γ < 1 the discount factor. Let v π ( s ) = E π [ G t | S t = s ] . This quantity can be estimated with a function approximator parameterized by w , for example using TD learning [43]:

where α t is the learning rate at time t , δ t = R t +1 + γv w ( S t +1 ) -v w ( S t ) is the TD error (see Sutton and Barto [45] for details).

In the control problem, the agent's goal is to find a policy, π ∗ , that maximizes expected returns. This can be achieved by estimating the optimal action-value function, q ∗ = max π q π , where q π = E π [ G t | S t = s, A t = a ] , e.g. using Q-learning [50].

"Continual RL" is closely related to transfer learning [48], meta learning [37, 15], multi-task learning [10], and curriculum learning [35]. A thorough review distinguishing these flavours can be found in Khetarpal et al. [24]. We use Continual RL to refer to a setup in which the agent's environment (i.e. the reward or transition dynamics) changes over time. We use the term semi-continual RL for the case when the agent can observe task boundaries (this is similar to the multi-task setting, but does not make assumptions about the task distribution or task IDs). In continual RL, a unique optimal value function may not exist, so the agent has to continually track the value function in order to learn useful policies [46, 2].

Our work focuses on the idea of two learning systems: one learning the "gist" of the task distribution, slowly, the other which is used for tracking can quickly adapt to current circumstances. We instantiate this idea in the context of learning value functions, both for prediction and control, but the idea can be applied broadly (for e.g. policies, GVFs). The concept of decomposing the value function into permanent and transient components was first introduced in model-based RL [39], for the single task of learning to play Go, and using samples drawn from the model to train the transient value function. In contrast, we focus on general continual RL, with samples coming from the environment.

In continual RL, most of the literature is focused on supplementing neural network approaches with better tools, such as designing new optimizers to update weights [44, 19, 12, 4, 14], building new architectures [21, 32], using experience replay to prevent forgetting [34, 36, 22, 8], promoting plasticity explicitly [28, 29], or using regularization techniques from continual supervised learning [25, 20, 30, 27]. Our method is agnostic to the nature of the function approximator used, and therefore, complementary to these advancements (and could in fact be combined with any of these methods).

Our approach has parallels with fast and slow methods [13, 7] which use neural networks as the main learning system and experience replay as a secondary system. Another similar flavor is to use the task

distribution to learn a good starting point, as in distillation-based approaches [49, 18] or MaxQInit [1]. But the latter are multi-task RL methods and their use in continual RL is limited.

3 Defining Permanent and Transient Value Functions

Inspired by CLS theory, we decompose the value function estimated by the agent into two, independently parameterized components: one which slowly acquires general knowledge, the other which quickly learns local nuances but then forgets. We call these components permanent value function , V ( P ) θ and transient value function , V ( T ) w . The overall value function is computed additively:

A similar decomposition can be done to the action-value function:

Other approaches to combining the two components are possible but additive decomposition is the simplest.

3.1 Permanent Value Function

Similar to the role of the neocortex in the brain, the permanent value function should capture general structure in the value functions from the tasks that the agent has seen in its lifetime. It should also provide good baseline predictions for any task that the agent faces, in order to allow it to learn good estimates with fewer updates. To achieve this in a scenario where the agent goes through a sequence of tasks and knows when the task changes, we might store all the states visited during the current task in a buffer and using them when the task finishes, to update the permanent value function as follows:

where α is the learning rate and k is the iteration index of the buffer. The permanent estimate takes a small step towards the new estimate of the value function in the above update rule: bootstrapping at a slower timescale. We analyze this idea further in Sec. 4.

3.2 Transient Value Function

The transient value function should compute corrections on top of the estimates provided by the permanent value function, in order to approximate more closely the true value function of the current task. We implement this intuition by updating the parameters of transient value function as:

where α is the learning rate, δ t = R t +1 + γV ( PT ) ( S t +1 ) -V ( PT ) ( S t ) and ∇ w V ( T ) w ( S t ) is the gradient of transient value function at time t . Eq. (5) is a semi-gradient update rule, in the same sense used to describe TD-learning, as the gradient is with respect to the current estimate only and not the target. Since the permanent and the transient value functions are independently parameterized, the semi-gradient ends up being just ∇ w V ( T ) w ( S t ) . This results in learning only a part of the value function that is not captured by the permanent value function. The transient value function is reset or decayed after transferring the new information into the permanent value function (e.g. at the moment when the task changes), similar to the functioning of the hippocampus [26, 33].

3.3 Learning Algorithm

We can easily turn the updates above into algorithms; a blueprint for the prediction case is shown in Algorithm 1, and the control version is in Appendix 3. The initialization and resets are done appropriately based on the function approximation used.

In the above algorithm, we assumed that the agent can observe task boundaries, though not any additional information (like a task id). We made this assumption to facilitate the theoretical analysis presented in the next section. A fully continual version of the algorithms is presented and evaluated empirically in Sec. 6.

Note that this algorithm is a strict generalization of TD learning: instead of setting w to 0 when the task changes, if we were to match the difference between the updated permanent value function and the transient value function instead, we would obtain exactly the TD learning predictions. Another way to get TD is by initializing θ to match the initialization of the TD learning algorithm and updating w only without resetting. We explore this connection formally in the next section.

4 Theoretical Results

In this section, we provide some theoretical results aiming to understand the update rules of permanent and transient value function separately, and then we study

Algorithm 1 PT-TD learning (Prediction)

1: Initialize: Buffer B , θ , w

2: for t : 0 →∞ do

3: Store S t in B

4: Observe R t +1 , S t +1

5: Update w using Eq. (5)

6: if Task Ends then

7: Update θ using B and Eq. (4)

8: Reset transient value function, w

9: Reset B

10: end if

11: end for

the semi-continual RL setting (because the fully continual RL setting is poorly understood in general from a theoretical point of view, and even the tools and basic setup are somewhat contentious). All our results are for the prediction problem with tabular value function representation; We expect that prediction results with linear function approximation can also be obtained. We present theorems in the main paper and include proofs in appendices B. The appendix also contains analytical equations describing the evolution of the parameters based on these algorithms.

4.1 Transient Value Function Results

We provide an understanding of what the transient value function does in the first set of results (Theorems 1-4). We carry out the analysis by fixing V ( P ) and only updating V ( T ) using samples from one particular task. We use V ( TD ) t and V ( PT ) t to denote the value function estimates at time t learned using TD learning and Alg. 1 respectively.

The first result establishes a relationship between the estimates learnt using TD and the transient value function update rule in Eq.(5).

Theorem 1. If V ( TD ) 0 = V ( P ) , V ( T ) 0 = 0 and the two algorithms train on the same samples using the same learning rates, then ∀ t, V ( PT ) t = V ( TD ) t .

The theorem shows that the estimates learnt by our approach match those learnt by TD-learning in a special case, proving that our algorithm is a strict generalization of TD-learning.

The next two theorems together shows that the transient value function reduces prediction error. First we show the contraction property of the transient value function Bellman operator.

Theorem 2. The expected target in the transient value function update rule is a contraction-mapping Bellman operator: T ( T ) V ( T ) = r π + γ P π V ( P ) -V ( P ) + γ P π V ( T ) , where r π is expected one-step reward for each state written in vector form and P π is the transition matrix for policy π .

We get the Bellman operator by writing the expected target in matrix form. We now characterize the fixed point of the transient value function operator in the next theorem.

Theorem 3. The unique fixed point of the transient operator is ( I -γ P π ) -1 r π -V ( P ) .

The theorem confirms the intuition that the transient value function learns only the part of the value function that is not captured in the permanent value function, while the agent still learns the overall value function: V ( PT ) = V ( P ) + V ( T ) = V ( P ) +( I -γ P π ) -1 r π -V ( P ) = v π .

The next theorem relates the fixed point of the transient value function to the value function of a different Markov Reward Process (MRP)

Theorem 4. Let M be an MDP in which the agent is performing transient value function updates for π . Define a Markov Reward Process (MRP), ˜ M , with state space ˜ S : { ( S t , A t , S t +1 ) , ∀ t } , transition dynamics ˜ P ( x ′ | x, a ′ ) = P ( s ′′ | s ′ , a ′ ) , ∀ x, x ′ ∈ ˜ S , ∀ s ′ , s ′′ ∈ S , ∀ a ′ ∈ A and reward function ˜ R ( x, a ′ ) = R ( s, a ) + γV ( P ) ( s ′ ) -V ( P ) ( s ) . Then, the fixed point of Eq.(5) is the value function of π in ˜ M .

The proof follows by comparing the value function of the modified MRP with the fixed point of transient value function. The theorem identifies the structure of the fixed point of the transient value function in the form of the true value function of a different MRP. This result can be used to design an objective for the permanent value function updates; For instance, to reduce the variance of transient value function updates.

4.2 Permanent Value Function Results

We focus on permanent value function in the next two theorems. First, we characterize the fixed point of the permanent value function. Then, we connect the fixed point of the permanent value function to the jumpstart objective [48, 1], which is defined as the initial performance of the agent in the new task before collecting any data. The latter is used to measure the effectiveness of transfer in a zero-shot learning setup, and therefore, is crucial to measure generalization under non-stationarity. We make an assumption on the task distribution for the following proofs:

Assumption 1. Let MDP M τ = ( S , A , R τ , P τ , γ ) denote task τ . Then, v τ is the value function of task τ and let E τ denote the expectation with respect to the task distribution. We assume there are N tasks and task τ is i.i.d. sampled according to p τ .

Theorem 5. Following Theorem 2, under assumption 1 and Robbins-Monro step-size conditions, the sequence of updates computed by Eq.(4) contracts to a unique fixed point E τ [ v τ ] .

We use the approach described in Example 4.3 from Bertsekas and Tsitsiklis [5] to prove the result. Details are given in Appendix B.2.1

Theorem 6. The fixed point of the permanent value function optimizes the jumpstart objective, J = min u ∈ R |S| 1 2 E τ [ ‖ u -v τ ‖ 2 2 ] .

4.3 Semi-Continual RL Results

We present two important results in this section which prove the usefulness of our approach in value function retention, for past tasks, and fast adaptability on new tasks. For the proofs, we assume that at time t , both V ( TD ) t and V ( PT ) t have converged to the true value function of task i and the agent gets samples from task τ from timestep t +1 onward.

The first result proves that the TD estimates forget the past after seeing enough samples from a new task, while our method retains this knowledge through V ( P ) .

The proof uses the sample complexity bound from Bhandari et al. [6] (Sec. 6.3, theorem 1). Details are in Appendix B.3.1.

Next, we prove that our method has a tighter upper bound on errors compared with the TD learning algorithm in expectation.

Theorem 8. Under assumption 1, there exists k 0 such that ∀ k ≤ k 0 , E τ [ ∥ ∥ ∥ v τ -V ( PT ) t + k ∥ ∥ ∥ 2 2 ] has a

The proof uses the sample complexity bound from Bhandari et al. [6] (Sec. 6.3, theorem 1) to get the desired result as detailed in Appendix B.3.2. As k →∞ , the estimates of both the algorithms converge to the true value function and their upper bounds collapse to 0 . This result doesn't guarantee low error at each timestep, but it provides an insight on why our algorithm has low error.

To further understand the behaviour of our algorithm on a new task, we derive analytical expressions for mean squared error similar to Singh and Dayan [42] for both our algorithm and TD-learning algorithm (see Appendix B.3.3 and B.3.4). Then, we use them to calculate the mean squared error empirically on a toy problem. We observe low error for our algorithm at every timestep for various task distributions (details in Appendix C.2).

5 Experiments: Semi-Continual Reinforcement Learning

We conducted experiments in both the prediction and the control settings on a range of problems. Although the theoretical results handle the prediction setting, we performed control experiments to demonstrate the broad applicability of our approach. In this section, we assume that the agent is faced with a sequence of tasks in which the transition dynamics are the same but the reward function changes. The agent knows when the task has changed. In the sections below, we focus on the effectiveness of our approach. Additional experiments providing other insights (effect of network capacity, sensitivity to hyperparameters, and effect of task distribution) are included in the Appendix (Sec. C.2, C.3, and C.4).

Baselines: Because our approach is built on top of TD-learning and Q-learning, we primarily use the usual versions of TD learning and Q-learning as baselines, which can be viewed as emphasizing stability. We also include a version of these algorithms in which weights are reset when the task switches, which creates significant plasticity. We include more baselines for the continual RL experiments, which are described in Sec. 6. The results for all algorithms are averaged across 30 seeds and shaded regions in the figures indicate 90% confidence intervals 2 . We use the same procedure to optimize hyperparameters for all algorithms, detailed in Appendix (Sec. C.4).

5.1 Prediction experiments

We ran five experiments on 3 types of navigation tasks, with various function approximators. The policy being evaluated was uniformly random in all experiments. We used root mean squared value error (RMSVE) as a performance measure on the online task and the expected mean squared error (MSE) on the other tasks.

Discrete grid: is a 5x5 environment depicted in Fig. 1a, which has four goal states (one in each corner). The agent starts in the center and chooses an action (up, down, left or right). Each action moves the agent one step along the corresponding direction. Rewards are 0 for all transitions except those leading into a goal state. The goal rewards change as detailed in Table C.1, so the value function also changes as shown in Fig. C.1. For the linear setting, each state is represented using 10 features: five for the row and five for the column. The permanent and transient value functions use the same features. For the deep RL experiment, instead of receiving state information, the agent receives 48x48 RGB images. The agent's location is indicated in red, and goal states are colored in green

(a) Discrete grid.

(b) Minigrid.

as shown in Fig. 1a. Goal rewards are multiplied by 10 to diminish the noise due to the random initialization of the value function. For the tabular and linear setting, we run 500 episodes and change the goal rewards after every 50 episodes. For the deep RL experiments, we run 1000 episodes and goal rewards are switched every 100 episodes. The discount factor is 0.9 in all cases.

0

50

100

150

200

250

300

Episodes

0.05

0.10

0.15

0.20

0.25

RMSVE

Discrete Grid (Tabular)

0

50

100

150

200

250

300

Episodes

0.05

0.10

0.15

0.20

0.25

RMSVE

Discrete Grid (Linear)

0

200

400

600

800

1000

1200

Episodes

0.31

0.32

0.33

RMSVE

0

100

200

300

400

500

Episodes

1

2

3

RMSVE

Image Grid (DNN)

0

100

200

300

400

Episodes

2.0

2.5

RMSVE

Gym-Minigrid (DNN)

TD

TD (reset)

PT-TD (ours)

Other Tasks

Continuous grid: is a continuous version of the previous task. The state space is the square [0 , 1] × [0 , 1] , with the starting state sampled uniformly from [0 . 45 , 0 . 55] × [0 . 45 , 0 . 55] . The regions within 0 . 1 units (in norm-1) from the corners are goal states. The actions change the state by 0 . 1 units along the corresponding direction. To make the transitions stochastic, we add noise to the new state,

sampled uniformly from [ -0 . 01 , 0 . 01] × [ -0 . 01 , 0 . 01] . Rewards are positive only for transitions entering a goal region. The goal reward for each task is given in Table C.1. In the linear function approximation setting we convert the agent's state into a feature vector using the radial basis function as described in [45] and Appendix C.1.1. We run 2000 episodes, changing goal rewards every 200 episodes. For evaluation, we sample 225 evenly spaced points in the grid and estimate their true values by averaging Monte Carlo returns from 500 episodes across 10 seeds for each task. Here, γ = 0 . 99 .

Minigrid: We used the four rooms environment shown in Fig. 1b. Each room contains one type of item and each item type has a different reward and color. Goal states are located in all corners. The episode terminates when the agent reaches a goal state. The agent starts in the region between cells (3,3) and (6,6) and can move forward, turn left, or turn right. The agent perceives a 5x5 (partial) view in front of it, encoded one-hot. The rewards for picking items change as described in Table C.2 and the discount factor is 0 . 9 . We run 750 episodes and change rewards every 75 episodes.

We provide the training and architecture details for the two deep RL prediction experiments in Appendix (Sec. C.1.1 and C.1.2) due to space constraints.

Observations: The results are shown in Fig. 2. Solid lines indicate performance on the task currently experienced by the agent (i.e., the online task) and the dotted lines indicate the performance on the other tasks. The reset variant of TD-learning performs poorly because all learning progress is lost when the weights are reset, leading to the need for many samples after every task change. TD-learning also has high online errors immediately after a task switch, as its predictions are initially biased towards the past task. As the value function is updated using the online interaction data, the error on the current task reduces and the error on other tasks rises, highlighting the stability-plasticity problem when using a single value function estimate. Our algorithm works well on both performance measures. The permanent value function learns common information across all tasks, resulting in lower errors on the second performance measure throughout the experiment. Also, due to the good starting point provided by the permanent value function, the transient component requires little data to make adjustments, resulting in fast adaptation on the new task. In our method, the permanent component provides stability and the transient component provides plasticity, resulting in a good balance between the two. We also plotted the heatmap of the value estimates, shown in Fig. C.3, to understand the evolution of the predictions for TD-learning and our method which confirms the observations made so far.

5.2 Control experiments

To test our approach as a complement to Q-learning 3, we conducted a tabular gridworld experiment and a Minigrid experiment using deep neural networks.

(a) Toy grid.

0

50

100

150

200

250

300

Episodes

0.50

0.25

0.00

0.25

0.50

Returns

Discrete Grid (Tabular)

Q-learning

Q-learning (reset)

PT-Q-learning (ours)

(b) Tabular results.

(c) Minigrid.

0

500

1000

1500

2000

2500

Episodes

0

1

2

3

4

Returns

Minigrid (DNN)

DQN

DQN (reset)

PT-DQN (ours)

(d) DRL results.

Tabular experiment: We use the 6x6 grid shown in Fig. 3a, with two goals located in the top right corner. The agent starts in the bottom left corner and can choose to go up, down, left, or right. The action transitions the agent to the next state along the corresponding direction if the space is empty and keeps it in the same state if the transition would end up into a wall or obstacle. The action has the intended effect 90% of the time; otherwise, it is swapped with one of the perpendicular actions. The agent receives a reward of +1 and -1 for reaching the two goal states respectively, and the rewards swap periodically, creating two distinct tasks. The discount factor is 0 . 95 . We run 500 episodes and change the task every 50 episodes.

Minigrid experiment: We use the two-room environment shown in Fig. 3c. The agent starts in the bottom row of the bottom room and there are two goal states located in the top room. The goal

rewards are +5 and 0 respectively, and they flip when the task changes. We run 2500 episodes, alternating tasks every 500 episodes. The other details are as described in the previous section.Further details on the training and architecture are described in Appendix (Sec. C.1.3).

Observations: The results are shown in Fig. 3. The reset variant of Q-learning has to learn a good policy from scratch whenever the task changes, therefore, it requires many samples. Since, the goal rewards are flipped from one task to another, Q-learning also requires many samples to re-adjust the value function estimates. In our method, since the permanent component mixes action-values across tasks, the resulting bias is favorable, and makes it fairly easy to learn the corrections required to obtain overall good estimates. So, the transient component requires comparatively few samples to find a good policy when the task changes.

6 Continual Reinforcement Learning

So far, we assumed that the agent observes task boundaries; This information is usually not available in continual RL. One approach that has been explored in the literature is to have the agent infer, or learn to infer, the task boundaries through observations (e.g. see [17, 23]). However, these methods do not work well when task boundaries are not detected correctly, or simply do not exist. We will now build on the previous algorithm we proposed to obtain a general version, shown in Alg. 2, which is suitable for continual RL.

Our approach maintains the permanent and transient components, but updates them continually, instead of waiting for task changes. The permanent component is updated every k steps (or episodes) using the samples stored in the buffer B . And, instead of resetting the transient component, it is now decayed by a factor λ ∈ [0 , 1] . Here, k and λ are both hyperparameters. When

λ = 0 and k the duration between task switches, we obtain the previously presented algorithm as a special case. In general, k and λ control the stability-plasticity trade off. Lower λ means that the transient value function forgets more, thus retaining more plasticity. Lower k introduces more plasticity in the permanent value function, as its updates are based on fewer and more recent samples. The algorithm still retains the idea of consolidating information from the transient into the permanent value function.

The effect of k and λ : To illustrate the impact of the choice of hyperparameters, we use the tabular task shown in Fig. 3a. The details of the experiment are exactly as before, but the agent does not know the task boundary. We run the experiment for several combinations of k and λ for the best learning rate. The results are shown in Fig. 4. As expected, we observe that the hyperparameters k and λ have co-dependent effects to a certain degree. For small values of k , large values of λ yield better performance, because the variance in the permanent value function is mitigated, and the transient value function in fact is more stable. For large values of k , the transient value function receives enough updates before the permanent value function is trained, to attain low prediction error. Therefore, the updates to the permanent value function are effective, and the transient predictions can be decayed more aggressively. More surprisingly, if the frequency of the permanent updates is significantly higher or significantly lower than the frequency of task changes, the overall performance is marginally affected, but remains better than for Q-learning (indicated using black dotted line) for most k and λ values. In fact, for each k , there is at least one λ value for which the performance of the proposed algorithm exceeds the Q-learning baseline. So, in practice, we could fix one of these hyperparameters beforehand and tune only the second. We suspect that the two parameters could be unified into a single one, but this is left for future work.

6.1 Experiments

We test our algorithm on two domains: JellyBeanWorld (JBW) [31] and MinAtar [51]. We chose these domains because of the relatively lighter computational requirements, which still allow us to

sweep a range of hyperparameters and to report statistical results averaged over 30 seeds. We include architecture and further training details in Appendix (Sec. C.1.5, C.1.6).

JellyBeanWorld is a benchmark to test continual RL algorithms [31]. The environment is a two-dimensional infinite grid with three types of items: blue, red, and green, which carry different rewards for different tasks as detailed in Sec. C.1.5. In addition to reward non-stationarity, the environment has spatial non-stationarity as shown in Fig. 5a. The agent observes an egocentric 11x11 RGB image and chooses one of the four usual navigation actions. We run 2.1M timesteps, with the task changing every 150k steps. The discount factor is 0 . 9 .

MinAtar is a standard benchmark for testing single-task RL algorithms [51]. We use the breakout, freeway, and space invaders environments. We run for 3.5M steps and setup a continual RL problem, by randomly picking one of these tasks every 500k steps. We standardize the observations across tasks by padding with 0s when needed

0.9

0.7

0.5

0.3

0.1

Decay parameter (

)

0.36

0.38

0.40

0.42

Mean Returns

Analysis of

k

and

k

1

5

10

25

100

Q

and we make all the actions available to the agent. This problem has non-stationarity in transitions, rewards and observations making it quite challenging. The discount factor is 0 . 99 . More details are included in Appendix (Sec. C.1.6).

Baselines: We compare our method with DQN, two DQN variants, and a uniformly random policy. For the first baseline, we use a large experience replay buffer, an approach which was proposed recently as a viable option for continual RL [8]. The second baseline, DQN with multiple heads, uses a common trunk to learn features and separate heads to learn Q-values for the three tasks. This requires knowing the identity of the current task to select the appropriate head. We use this baseline because it is a standard approach for multi-task RL, and intuitively, it provides an upper bound on the performance. This approach is not suitable if the number of tasks is very high, or if the problem is truly continual and not multi-task. For our approach, we use half the number of parameters as that of DQN for both permanent and transient value networks to ensure the total number of parameters across all baselines are same - hence the name PT-DQN-0.5x. 3

60

40

20

0

20

40

60

60

40

20

0

20

40

60

DQN

PT-DQN-0.5x (ours)

DQN-multi-head

DQN-large buffer

Random

Jellybean world

(sdxs

0 3

Reward per step

0.2

0.1

0.0

0.0

0.5

1.0

1.5

4

2

0

2

Slep9

4

(c) MinAtar.

DQN

PT-DQN-0.5x (ours)

DQN-multi-head

DQN-large buffer

Random

MinAtar results

30

Episodic return

20

10

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Steps

Figure 5: (a) JellyBeanWorld task. (b) Results: JellyBeanWorld. (c) MinAtar tasks. (d) Results: MinAtar.

Results are presented in Fig. 5 and the pseudocode of PT-DQN is included in Appendix C.1.4. For JBW experiment, we plot the reward obtained per timestep over a 10k step window (reward rate) as a function of time. For MinAtar, we plot the return averaged over the past 100 episodes. Our method, PT-DQN-0.5x, performs better than all the baselines. The performance boost is large in the JBW experiment, where the rewards are completely flipped, and still favourable in the MinAtar experiment. DQN's performance drops over time when it needs to learn conflicting Q-values, as seen in the JBW results. DQN also needs a large number of samples to overcome the bias in its estimates when the task changes. Surprisingly, multi-headed DQN performs poorly compared to our method, despite having the additional task boundary information. The weights of the trunk are shared, and they vary continuously over time. Therefore, the weights at the head become stale compared to the features, which degrades performance. These findings may be different if the trunk were pre-trained and then fixed, with only the heads being fine-tuned, or if the networks were much bigger (which is however not warranted for the scale of our experiment). Augmenting DQN with a large experience replay is detrimental for general non-stationary problem, based on our experiments. This is because the samples stored in the replay buffer become outdated, so using these samples hampers the agent's ability to track environment changes.

4

2

0

2

4

x11’

7 Discussion

Designing agents that can learn continually, from a single stream of experience, is a crucial challenge in RL. In this paper, we took a step in this direction by leveraging intuitions from complementary learning systems, and decomposing the value function into permanent and transient components. The permanent component is updated slowly to learn general estimates, while the transient component computes corrections quickly to adapt these estimates to changes. We presented versions adapted to both semi and fully continual problems, and showed empirically that this idea can be useful. The theoretical results provide some intuition on the convergence and speed of our approach.

The non-stationarity in our experiments is much more challenging, including arbitrary reward changes, as well as dynamics and observation changes, compared to other works, which use specific structure, or rely on a curriculum of tasks (thereby limiting non-stationarity). Our approach provided improvements in performance compared to the natural TD-learning and Q-learning baselines.

Our approach performs fast and slow interplay at various levels. The permanent value function is updated at a slower timescale (every k steps or at the end of the task); The transient value function is updated at a faster timescale (every timestep). The permanent value function is updated to capture some part of the value function from the agent's entire experience (generalization), which is a slow process; The transient value function adapts the estimates to a specific situation, which is a fast process. The permanent value function is updated using a smaller learning rate (slow), but the transient value function is updated using a larger learning rate (fast) [3].

7.1 Limitations and Future Work

While we focused on illustrating the idea of permanent and transient components with value-based algorithms with discrete action spaces, the approach is general and could in principle be used with policy gradient approaches. One could use the proposed value function estimates to compute the gradient of the policy (represented as a separate approximator) in actor-critic methods. Another possibility is to have a "permanent" policy, which is corrected by a transient component to adapt to the task at hand. These directions would be interesting to explore. Our idea can be extended to learning General Value Functions [47] too.

Our theoretical analysis is limited to piece-wise non-stationarity, where the environment is stable for some period of time. An interesting direction would be to extend our approach to other types of non-stationarity, such as continuous wear and tear of robotic parts.

Another important future direction is to consider distinct feature spaces for the permanent and transient components, as done in Silver et al. [38]. This would likely increase the ability to trade off stability and plasticity. Another direction to better control the stability and plasticity trade-off is by learning the frequency parameter k and the decay parameter λ using meta objectives [16]; We suspect there's a relation between the two parameters that can unify them into a single one.

Our approach can also be used to bridge offline pre-training and online fine-tuning of RL agents. The permanent value function can be learnt using offline data and these estimates can be adjusted by computing corrections using the transient value function using online interaction data.

Acknowledgments and Disclosure of Funding

We are grateful to NSERC, CIFAR, and Microsoft Research grants for funding this research. We thank Jalaj Bhandari, Maxime Webartha, Matthew Reimer, Blake Richards, Emmanuel Bengio, Harsh Satija, Sumana Basu, Chen Sun, and many other colleagues at Mila for the helpful discussions; Will Dabney, Harm Van Seijen, Rich Sutton, Razvan Pascanu, David Abel, James McClelland and all of the 2023 Barbados workshop participants for their feedback on the project; the anonymous reviewers for the constructive feedback on the final draft. This research was enabled in part by support provided by Calcul Quebec and the Digital Research Alliance of Canada.

References

David Abel, Yuu Jinnai, Sophie Yue Guo, George Konidaris, and Michael Littman. Policy and value transfer in lifelong reinforcement learning. In International Conference on Machine Learning , pages 20-29. PMLR, 2018.

David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. arXiv preprint arXiv:2307.11046 , 2023.

Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. arXiv preprint arXiv:2201.12604 , 2022.

Eseoghene Ben-Iwhiwhu, Saptarshi Nath, Praveen K Pilly, Soheil Kolouri, and Andrea Soltoggio. Lifelong reinforcement learning with modulating masks. arXiv preprint arXiv:2212.11110 , 2022.

Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming . Athena Scientific, 1996.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , pages 16911692. PMLR, 2018.

Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences , 23(5):408-422, 2019.

Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Taskagnostic continual reinforcement learning: In praise of a simple baseline. arXiv preprint arXiv:2205.14495 , 2022.

Gail A Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing , 37(1): 54-115, 1987.

Rich Caruana. Multitask learning. Machine learning , 28(1):41-75, 1997.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym, 2018.

Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325 , 2021.

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016.

Mohamed Elsayed and A Rupam Mahmood. Utility-based perturbed gradient descent: An optimizer for continual learning. arXiv preprint arXiv:2302.03281 , 2023.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. Bootstrapped meta-learning. arXiv preprint arXiv:2109.04504 , 2021.

João Gama, Indr˙ e Žliobait˙ e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR) , 46(4):1-37, 2014.

Valentin Guillet, Dennis George Wilson, and Emmanuel Rachelson. Neural distillation as a state representation bottleneck in reinforcement learning. In Conference on Lifelong Learning Agents , pages 798-818. PMLR, 2022.

Andrew Jacobsen, Matthew Schlegel, Cameron Linke, Thomas Degris, Adam White, and Martha White. Meta-descent for online, continual prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33/0, pages 3943-3950, 2019.

Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems , 32, 2019.

Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. In International Conference on Machine Learning , pages 2497-2506. PMLR, 2018.

Christos Kaplanis, Claudia Clopath, and Murray Shanahan. Continual reinforcement learning with multi-timescale replay. arXiv preprint arXiv:2004.07530 , 2020.

Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J Roberts. Same state, different task: Continual reinforcement learning without interference. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36/7, pages 7143-7151, 2022.

Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research , 75: 1401-1476, 2022.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences , 20(7):512-534, 2016.

Kevin Lu, Igor Mordatch, and Pieter Abbeel. Adaptive online planning for continual lifelong learning. arXiv preprint arXiv:1912.01188 , 2019.

Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.09560 , 2022.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning , pages 16828-16847. PMLR, 2022.

Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning sparse representations online. arXiv preprint arXiv:1911.08068 , 2019.

Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint arXiv:2002.06306 , 2020.

Sam Powers, Eliot Xing, and Abhinav Gupta. Self-activating neural ensembles for continual reinforcement learning. In Conference on Lifelong Learning Agents , pages 683-704. PMLR, 2022.

Blake A Richards and Paul W Frankland. The persistence and transience of memory. Neuron , 94(6):1071-1084, 2017.

Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 , 2018.

Mark B Ring. Child: A first step towards continual learning. Machine Learning , 28(1):77-104, 1997.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems , 32, 2019.

Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook . PhD thesis, Technische Universität München, 1987.

David Silver, Richard S Sutton, and Martin Müller. Reinforcement learning of local shape in the game of go. In IJCAI , volume 7, pages 1053-1058, 2007.

David Silver, Richard S Sutton, and Martin Müller. Sample-based learning and search with permanent and transient memories. In Proceedings of the 25th international conference on Machine learning , pages 968-975, 2008.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Satinder Singh and Peter Dayan. Analytical mean squared error curves for temporal difference learning. Machine Learning , 32(1):5-40, 1998.

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9-44, 1988.

Richard S Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI , pages 171-176. San Jose, CA, 1992.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. In Proceedings of the 24th international conference on Machine learning , pages 871-878, 2007.

Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 , pages 761-768, 2011.

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(7), 2009.

Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems , 30, 2017.

Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.

Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176 , 2019.

A Control algorithm

The action-value function can be decomposed into two components as:

The updates for permanent value function are:

where α is the learning rate. An update rule similar to Q-learning can be used for the transient value function:

where δ t = R t +1 + γ max a Q ( PT ) ( S t +1 , a ) -Q ( PT ) ( S t , A t ) is the TD error.

Algorithm 3 PT-Q-learning (Control)

B Proofs

B.1 Transient Value Function Results

We carry out the analysis by fixing V ( P ) and only updating V ( T ) using samples from one particular task. We use V ( TD ) t and V ( PT ) t to denote the value function estimates at time t learned using TD learning and Alg. 1 respectively.

Theorem B.1.1. If V ( TD ) 0 = V ( P ) , V ( T ) 0 = 0 and the two algorithms train on the same samples using the same learning rates, then ∀ t, V ( PT ) t = V ( TD ) t .

Proof. We use induction to prove this statement.

Base case: When t = 0 , the PT-estimate of state s is V ( PT ) 0 ( s ) = V ( P ) ( s ) + V ( T ) 0 ( s ) = V ( P ) ( s ) . And, the TD estimate of state s is V ( TD ) 0 ( s ) = V ( P ) ( s ) . Therefore, both PT and TD estimates are equal.

Induction hypothesis: PT and TD estimates are equal for some timestep k for all states. That is, V ( PT ) ( s ) = V ( P ) ( s ) + V ( T ) ( s ) = V ( TD ) ( s ) .

Induction step: PT estimates at timestep k +1 is

Appendix

The penultimate step follows from the induction hypothesis completing the proof.

Theorem B.1.2. The expected target in the transient value function update rule is a contractionmapping Bellman operator: T ( T ) V ( T ) = r π + γ P π V ( P ) -V ( P ) + γ P π V ( T ) .

Proof. The target for state s at timestep t in the transient value function update rule is R t +1 + γV ( P ) ( S t +1 ) -V ( P ) ( s ) + γV ( T ) ( S t +1 ) . The expected target is:

We get the operator by expressing the above equation in the vector form:

For contraction proof, consider two vectors u, v ∈ R |S| .

Therefore, the transient value function Bellman operator is a contraction mapping with contraction factor γ .

Theorem B.1.3. The unique fixed point of the transient operator is ( I -γ P π ) -1 r π -V ( P ) .

Proof. We get the fixed point, V ( T ) π , by repeatedly applying the operator to a vector v :

We applied the operator to v in steps 1 to 4 and used the property lim n →∞ ( γ P π ) n → 0 in the penultimate step as γ P π is a sub-stochastic matrix. Many terms cancel out as it is a telescoping sum.

Theorem B.1.4. Let M be an MDP in which the agent is performing transient value function updates for π . Define an MRP, ˜ M , with state space ˜ S : { ( S t , A t , S t +1 ) , ∀ t } , transition dynamics ˜ P ( x ′ | x, a ′ ) = P ( s ′′ | s ′ , a ′ ) , ∀ x, x ′ ∈ ˜ S , ∀ s ′ , s ′′ ∈ S , ∀ a ′ ∈ A and reward function ˜ R ( x, a ′ ) = R ( s, a ) + γV ( P ) ( s ′ ) -V ( P ) ( s ) . Then, the fixed point of Eq.(5) is the value function of π in ˜ M .

Proof. The state space of the new MDP ˜ M is made up of all possible consecutive tuples of ( S t , A t , S t +1 ) pairs. The rewards in this MDP are defined as ˜ R ( x, a ′ ) = R ( s, a ) + γV ( P ) ( s ′ ) -V ( P ) ( s ) , ∀ x ∈ ˜ S , ∀ a ′ ∈ A . The transition matrix ˜ P ( x ′ | x, a ′ ) = prob ( x ′ | x, a ′ ) = prob ( s ′ , a ′ , s ′′ | s, a, s ′ , a ′ ) = prob ( s ′′ | s ′ , a ′ ) = P ( s ′′ | s ′ , a ′ ) using Markov property of M . Then, the value of state x = ( s, a, s ′ ) is:

where Z t = ∑ ∞ k = t γ k -t ˜ R k +1 is the returns starting from state X t = ( s, a, s ′ ) . Without loss of generality, consider a sample trajectory generated according to π , τ k = ( S 0 = s, A 0 = a, R 1 , S 1 = s ′ , A 1 , R 2 , S 2 , A 2 , R 3 , . . . , R T , S T ) where S T is the terminal state. The returns Z τ k of this trajectory is:

Returns of every trajectory end up being the difference between the returns in the original MDP, G τ k and the permanent value function. Most importantly, these returns only depend on the state s and it is the same for all a and s ′ choices following s . The value of state x = ( s, a, s ′ ) can then be simplified using this observation:

B.2 Permanent Value Function Results

We focus on permanent value function in the next two theorems. First, we characterize the fixed point of the permanent value function. We use assumption 1 on the task distribution for the following proofs:

Theorem B.2.1. Following Theorem 2, under assumption 1 and Robbins-Monro step-size conditions, the sequence of updates computed by Eq.(4) contracts to a unique fixed point E τ [ v τ ] .

Proof. The permanent value function is updated using Eq. (4) and samples stored in the buffer B at the end of each task. This update rule can be simplified as:

where L is the length of the buffer, I is the indicator function, L ( s ) is the number of samples for state s in the buffer B , ˜ α k ( s ) = α k L ( s ) . The above update is performed once for every state in B at the end of each task. We assume that the buffer has enough samples for each state so that all states are updated. This can be ensured by storing all the samples in B for a long but finite period of time. The updates performed to permanent value function until that timestep will not affect the fixed point since we are in the tabular setting.

Let the sequence of learning rate ˜ α k ( s ) obey the conditions: ∑ ∞ n =1 ˜ α n ( s ) = ∞ and ∑ ∞ n =1 ˜ α 2 n ( s ) = 0 .

We can rewrite the PM update rule by separating the noise term:

Let F k denote the history of the algorithm. Then, E [ w k ( s ) |F k ] = E [ V ( PT ) τ ( s ) -E [ V ( PT ) τ ( s )]] = 0 and under the assumption E [ w 2 k ( s ) |F k ] ≤ A + BV ( P ) k 2 ( s ) (finite noise variance), we conclude that the permanent value function converges to E [ V ( PT ) τ ( s )] . But E [ V ( PT ) τ ( s )] is contracting towards E [ v τ ( s )] , therefore, the permanent value function is contracting towards E [ v τ ( s )] .

Theorem B.2.2. The fixed point of the permanent value function optimizes the jumpstart objective, J = arg min u ∈ R |S| 1 2 E τ [ ‖ u -v τ ‖ 2 2 ] .

Proof. The jumpstart objective function is given by J = arg min u ∈ R |S| 1 2 E τ [ ‖ u -v τ ‖ 2 2 ] . Let K = 1 2 E τ [ ‖ u -v τ ‖ 2 2 . We get the optimal point by differentiating K with respect to u and then equating it to 0:

Therefore, the fixed point of the permanent value function also optimizes the jumpstart objective.

B.3 Semi-Continual RL Results

We assume that at time t , both V ( TD ) and V ( PT ) have converged to the true value function of task i and the agent gets samples from task τ from timestep t +1 onward for the following theorems.

Theorem B.3.1. Under assumption 1, there exists k 0 such that ∀ k ≥ k 0 , E τ [ ∥ ∥ ∥ V ( TD ) t + k -v i ∥ ∥ ∥ 2 2 ] E τ [ ∥ ∥ V ( P ) -v i ∥ ∥ 2 2 ] , ∀ i .

glyph[negationslash]

>

Proof. Let ∥ ∥ V ( P ) -v i ∥ ∥ 2 2 = D i , which is a constant because we are updating transient value function only. Let ‖ v τ -v i ‖ 2 2 = ∆ τ,i denote the distance between value functions of two tasks i and τ . Clearly, ∆ τ,i = ∆ i,τ = 0 and ∆ i,i = 0 . And, let Err i,τ t = ∥ ∥ ∥ V ( TD ) t -v i ∥ ∥ ∥ 2 2 denote prediction error with respect to the previous task i at timestep t for TD estimates.

At timestep t , V ( PT ) = V ( TD ) = v i . The agent starts getting samples from task τ from timestep t +1 . Using the result from Bhandari et al. [6]:

since the smallest eigenvalue of the feature matrix ω = 1 for tabular approximations. Let C = exp { -(1 -γ 2 ) 4 } . Now we can apply the above result recursively for k timesteps to get

Then the error with respect to the past task after k timesteps is,

Taking the expectation with respect to τ gives

Now we will find a timestep k 0 such that ∀ k > k 0 , E τ [ Err i,τ t + k ] > D i . Let k i denote the timestep for which (1 -C k i ) E τ [∆ τ,i ] = D i . Then,

So, after time t + k i , V ( P ) is a better estimate of the value function of the past task i than the current TD estimates. Then, by letting k 0 = max i k i we have E τ [ Err i,τ t + k 0 ] > D i , ∀ i .

Note that the above result depends on the fact that k i exists, which is only possible when E τ [∆ τ,i ] -D i > 0 . We will now show that there exists at least one solution, the fixed point of the permanent value function ( E τ [ v τ ] ), which satisfies this constraint.

Then,

The final step follows from the fact that D i = ∥ ∥ V ( P ) -v i ∥ ∥ 2 2 = ‖ E τ [ v τ ] -v i ‖ 2 2 for V ( P ) fixed point.

Next, we show that our method has tighter upper bound on errors compared with the TD learning algorithm in expectation.

Proof. We assume that V ( P ) is converged to its fixed point, E τ [ v τ ] (Theorem 5). Using sample complexity bounds from Bhandari et al. [6] for TD learning algorithm, we get:

We know that our algorithm can be interpreted as TD learning with V ( P ) as the starting point (see Theorem 1). Using this relationship, we can apply the sample complexity bound to get:

For k = 0 , we know that our algorithm has lower error in expectation (see Theorem 6), that is E τ [ ∥ ∥ ∥ v τ -V ( P ) t ∥ ∥ ∥ 2 2 ] ≤ E τ [ ∥ ∥ ∥ v τ -V ( TD ) t ∥ ∥ ∥ 2 2 ] . Therefore, the error bound for our algorithm is tighter compared with that of TD learning algorithm. As k → ∞ , the right hand side of both the equations goes to 0 , which confirms that both algorithms converges to the true value function.

The theorem shows that the estimates for our algorithm lie within a smaller region compared to that of TD learning estimates. This doesn't guarantee low errors for our algorithms, but we can expect it to be closer to the true value function.

The next theorems contain analytical MSE equations for TD-learning and for our algorithm. Our analysis is similar to that of [42], but we don't assume that rewards are only observed in the final transition. Analytical MSE equations allow calculating the exact MSE that an algorithm would incur at each timestep when applied to a problem, instead of averaging the performance over several seeds, which is the first step towards other analyses such as bias-variance decomposition of value estimates, error bounds and bias-variance bounds, step-size adaptation for the update rule, etc. We use these equations to compare our method against TD-learning when varying the task distribution diameter (Sec. C.2).

We assume that the agent gets N i.i.d samples 〈 s, r, s ′ 〉 from task τ in each round, where s is sampled according to the stationary distribution d τ . Also, the agent accumulates errors from all the samples and updates its estimates at the end of the round t . Let E seeds denote the expectation with respect to random seeds, i.e., expectation with respect to the data generation process which generates the data that the agent observes over rounds and within a round.

Theorem B.3.3. Let:

where Σ ( TD ) t ( s, x ) = E seeds [ V ( TD ) t ( s ) V ( TD ) t ( s )] . We have:

and, the mean squared error ξ ( TD ) t at timestep t is

Proof.

where ∆ ( TD ) t ( s ) = r π ( s ) + γ ( P π m ( TD ) t )( s ) -m ( TD ) t ( s ) . We used the following results in the proof:

Now consider Σ ( TD ) t +1 ( s, x ) ,

Next, we will tackle each part separately. We will begin with Eq (9):

E seeds [ V ( TD ) t ( s ) ( N ∑ l =1 I S l = x ( R l + γV ( TD ) t ( X ′ l ) -V ( TD ) t ( x )) )] = E seeds [ V ( TD ) t ( s ) N ∑ l =1 E [( I S l = x ( R l + γV ( TD ) t ( X ′ l ) -V ( TD ) t ( x ))) | seed ] ] , = E seeds [ V ( TD ) t ( s ) N ̂ d π ( x )( ̂ r π ( x ) + γ ( ̂ P π V ( TD ) t )( x ) -V ( TD ) ( x ))] , = N E seeds [ ̂ d π ( x ) ̂ r π ( x ) V ( TD ) t ( s )] + γN E seeds [ ̂ d π ( x )( ̂ P π V ( TD ) t )( x ) V ( TD ) t ( s )] -N E seeds [ ̂ d π ( x ) V ( TD ) t ( x ) V ( TD ) t ( s )] , = Nd π ( x ) r π ( x ) m ( TD ) ( s ) + γNd π ( x ) E seeds [ ∑ x ′ ̂ P π ( x ′ | x ) V ( TD ) t ( x ′ ) V ( TD ) t ( x ) ] -Nd π ( x )Σ ( TD ) t ( s, x ) , = Nd π ( x ) ( r π ( x ) m ( TD ) t ( s ) + γ ∑ x ′ E seeds [ ̂ P π ( x ′ | x ) V ( TD ) t ( x ′ ) V ( TD ) t ( x )] -Σ ( TD ) t ( s, x ) ) , = Nd π ( x ) ( r π ( x ) m ( TD ) t ( s ) + γ ∑ x ′ P π ( x ′ | x )Σ ( TD ) t ( x ′ , s ) -Σ ( TD ) t ( s, x ) ) , = Nd π ( x ) ( r π ( x ) m ( TD ) t ( s ) + γ ( P π Σ ( TD ) t )( x, s ) -Σ ( TD ) t ( s, x ) ) . A similar simplification can be done to Eq. (10) which results in: (10) = Nd π ( s ) ( r π ( s ) m ( TD ) ( x ) + γ ( P π Σ ( TD ) t )( s, x ) -Σ ( TD ) t ( x, s ) ) . We will break down Eq. (11) into 9 parts and simplify each of them separately. (11) = E seeds [( N ∑ k =1 I S k = s R k )( N ∑ l =1 I S l = x R l )] (12) + γ E seeds [( N ∑ k =1 I S k = s R k )( N ∑ l =1 I S l = x V ( TD ) t ( X ′ l ) )] (13) -E seeds [( N ∑ k =1 I S k = s R k )( N ∑ l =1 I S l = x V ( TD ) t ( x ) )] (14) + γ E seeds [( N ∑ k =1 I S k = s V ( TD ) t ( S ′ k ) )( N ∑ l =1 I S l = x R l )] (15) + γ 2 E seeds [( N ∑ k =1 I S k = s V ( TD ) t ( S ′ k ) )( N ∑ l =1 I S l = x V ( TD ) t ( X ′ l ) )] (16) -γ E seeds [( N ∑ k =1 I S k = s V ( TD ) t ( S ′ k ) )( N ∑ l =1 I S l = x V ( TD ) t ( x ) )] (17) -E seeds [( N ∑ k =1 I S k = s V ( TD ) t ( s ) )( N ∑ l =1 I S l = x R l )] (18) -γ E seeds [( N ∑ k =1 I S k = s V ( TD ) t ( s ) )( N ∑ l =1 I S l = x V ( TD ) t ( X ′ l ) )] (19) + E seeds [( N ∑ k =1 I S k = s V ( TD ) t ( s ) )( N ∑ l =1 I S l = x V ( TD ) t ( x ) )] . (20)

Consider Eq. (12)

glyph[negationslash]

We have used the i.i.d assumption to separate the terms. Later, we used the indicator function to indicate the part which is non-zero. Also, we have used the definition of the second moment in the penultimate step.

We do a similar simplification to Eq. (13) now.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

We will simplify Eq. (14) now.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

The next expression Eq. (15) is similar to Eq. (13) and therefore, we skip the simplification details.

Now, we simplify (16), glyph[negationslash]

glyph[negationslash]

We now consider the next piece Eq. (17),

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Simplification of Eq. (18) is similar to Eq. (14).

Also, the simplification of Eq. (19) is similar to that of Eq. (17).

We will simplify the final part i.e., Eq. (20).

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

We can put all these pieces together to get the desired result.

We will now prove the final part of the theorem. The mean squared error ξ ( TD ) t +1 can be decomposed into bias and covariance terms:

Theorem B.3.4. Let E seeds denote the expectation with respect to seeds. Let

and, the mean squared error ξ ( PT ) t at timestep t is

Proof. The proof is similar to that of TD learning algorithm. We have:

The MSE recursive equation can be obtained similar to that of TD expression. Therefore, we skip the details and give the final expression.

C Experiments

Besides providing details for the experiments described in the main paper, we answer additional interesting questions, such as:

What is the effect of task distribution on the performance for various methods (Sec. C.2)?

What is the effect of network capacity on the performance (Sec. C.3)?

What is the effect of hyperparameters on the performance (Sec. C.4)?

C.1 Additional Details

We begin by providing details of the experiments.

C.1.1 Policy evaluation grids

We modify the rewards of the goal states as described in Table C.1. The true value function for each task is shown in Fig. C.1. We convert the 5x5 discrete grid into a 48x48 RGB image for the deep RL

1

2

3

4

5

1

2

3

4

5

0.0

0.09

0.22

0.48

0.0

0.04

0.09

0.18

0.33

0.53

0.05

0.09

0.16

0.27

0.39

0.04

0.09

0.18

0.33

0.53

0.0

0.09

0.22

0.48

0.0

Task: 1

1

2

3

4

5

1

2

3

4

5

0.0

0.48

0.22

0.09

0.0

0.53

0.33

0.18

0.09

0.04

0.39

0.27

0.16

0.09

0.05

0.53

0.33

0.18

0.09

0.04

0.0

0.48

0.22

0.09

0.0

Task: 2

1

2

3

4

5

1

2

3

4

5

0.0

0.04

0.05

0.04

0.0

0.09

0.09

0.09

0.09

0.09

0.22

0.18

0.16

0.18

0.22

0.48

0.33

0.27

0.33

0.48

0.0

0.53

0.39

0.53

0.0

Task: 3

1

2

3

4

5

1

2

3

4

5

0.0

0.53

0.39

0.53

0.0

0.48

0.33

0.27

0.33

0.48

0.22

0.18

0.16

0.18

0.22

0.09

0.09

0.09

0.09

0.09

0.0

0.04

0.05

0.04

0.0

Task: 4

0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

task. The agent's location is indicated by colouring the its location in red and we use green squares to indicate goal states. At every timestep, the agent receives a RGB image as an input. The action space and the action consequences are the same as discrete grid. We select best hyperparameters (learning rates) for each algorithm by choosing those values which results in the lowest area under the error curves as shown in Table C.6 and Fig. C.9. The hyperparameter search and the final reporting is performed by averaging the results on 30 different seeds. For our method, the buffer stores the samples from the latest task only to update the permanent value function.

For the first linear FA experiment, we use the same setup as the tabular setting except we encode each state in the grid as a vector of 10 bits. The first 5 bits encodes the row of the state and the last 5 bits encodes the column of the state. For example, the starting state (2,2) is represented as [0,0,1,0,0,0,0,1,0,0]. Both permanent value function and transient value function use same features to learn respective estimates.

For the second linear FA experiment, we use a continuous version of the grid. The goal rewards for each task is same as the discrete counterpart as described in Table C.1. The true value function is also different for each task and it is shown as a heatmap in Fig. C.2. We use RBFs to generate features as described in Sutton and Barto [45]. We use 676 centers (order 2 ) spaced equally in both x and y directions across the grid with a variance (0 . 75 / ( order -1)) 2 and use a threshold to convert the continuous vector into a binary feature vector. Any feature value greater than 0.5 is set to 1, while those features with values less than 0.5 is set to 0. We search for the best learning rate from the same set of values as the discrete grid task. The hyperparameter curves are shown in Fig. C.11.

For the deep RL experiment, the rewards are multiplied by a factor of 10 to minimize the initialization effect of deep neural network on prediction. We used a convolution neural network (CNN) to approximate the value function for all algorithms tested. The CNN consists of two conv layers with

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Task: 1

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Task: 2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Task: 3

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Task: 4

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

8 filters, (2,2) stride, and relu activation in each layer. We use a max pooling layer in between the two conv layers with a stride 2. The output of the second conv layer is flatted, and it is followed by a couple of linear layers. The fully connected network has 64 and 1 hidden units in the respective layers. The first fully connected layer uses a tanh activation and no activation is used in the final layer. For our method, we attach a new head after the first two conv layers for permanent value function. This head also has two linear layers where the first layer has tanh activation. The entire network is trained by backpropagating the errors from the transient value function. But, the conv layers are not trained when updating the permanent value function. The transient value function and the TD learning algorithm are trained in an online fashion using an experience replay buffer of size 100k. The buffer is reset at the end of the task for all methods to reduce the bias from old samples. permanent value function is trained at the end of the task using the samples from the latest task. 100 gradient steps are taken during each update cycle. We use SGD optimizer to update its weights. The best learning rate is picked by computing the area under the error curve on a set of values shown in Table C.7. The mean AUC curves of various choices of learning rates are shown in Fig. C.12.

1

2

3

4

5

1

2

3

4

5

After 50 episodes

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.05

0.23

0.57

0.0

0.01

0.05

0.15

0.3

0.53

0.02

0.05

0.13

0.24

0.36

0.01

0.05

0.15

0.28

0.53

0.0

0.05

0.19

0.54

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.05

0.23

0.57

0.0

0.01

0.05

0.15

0.3

0.53

0.02

0.05

0.13

0.24

0.36

0.01

0.05

0.15

0.28

0.53

0.0

0.05

0.19

0.54

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.05

0.23

0.57

0.0

0.01

0.05

0.15

0.3

0.53

0.02

0.05

0.13

0.24

0.36

0.01

0.05

0.15

0.28

0.53

0.0

0.05

0.19

0.54

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 100 episodes

0.0

0.01

0.06

0.12

0.0

0.0

0.02

0.07

0.1

0.12

0.0

0.02

0.08

0.1

0.11

0.0

0.02

0.06

0.09

0.12

0.0

0.01

0.05

0.13

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.48

0.13

-0.07

0.0

0.56

0.29

0.09

-0.04

-0.1

0.34

0.22

0.04

-0.05

-0.08

0.52

0.27

0.07

-0.04

-0.09

0.0

0.48

0.14

-0.08

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.49

0.19

0.06

0.0

0.56

0.3

0.15

0.06

0.02

0.34

0.24

0.12

0.05

0.03

0.52

0.29

0.13

0.05

0.02

0.0

0.49

0.19

0.04

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.51

0.23

0.09

0.0

0.57

0.32

0.18

0.09

0.06

0.36

0.26

0.15

0.09

0.08

0.53

0.31

0.16

0.08

0.05

0.0

0.51

0.22

0.07

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 150 episodes

0.0

0.12

0.11

0.11

0.0

0.12

0.11

0.1

0.08

0.1

0.1

0.11

0.11

0.08

0.08

0.12

0.1

0.09

0.08

0.1

0.0

0.11

0.09

0.11

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

-0.1

-0.07

-0.08

0.0

-0.06

-0.04

-0.03

-0.02

-0.04

0.1

0.04

0.03

0.09

0.13

0.42

0.22

0.15

0.26

0.46

0.0

0.4

0.26

0.44

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.03

0.03

0.03

0.0

0.06

0.07

0.07

0.07

0.05

0.21

0.16

0.14

0.17

0.21

0.53

0.32

0.25

0.34

0.55

0.0

0.51

0.36

0.54

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.05

0.06

0.04

0.0

0.1

0.11

0.09

0.09

0.06

0.25

0.19

0.17

0.18

0.22

0.56

0.35

0.27

0.36

0.56

0.0

0.54

0.38

0.55

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 200 episodes

0.0

0.1

0.09

0.09

0.0

0.11

0.1

0.09

0.08

0.09

0.13

0.13

0.13

0.12

0.12

0.21

0.18

0.16

0.16

0.19

0.0

0.21

0.18

0.2

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.43

0.27

0.5

0.0

0.44

0.22

0.15

0.26

0.43

0.08

0.04

0.01

0.05

0.1

-0.15

-0.1

-0.09

-0.08

-0.13

0.0

-0.17

-0.13

-0.17

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.53

0.35

0.59

0.0

0.55

0.32

0.24

0.34

0.52

0.22

0.17

0.14

0.17

0.21

0.07

0.08

0.07

0.08

0.06

0.0

0.04

0.04

0.03

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.53

0.36

0.6

0.0

0.56

0.33

0.25

0.35

0.53

0.24

0.19

0.17

0.19

0.24

0.09

0.11

0.1

0.11

0.09

0.0

0.06

0.08

0.06

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 250 episodes

0.0

0.18

0.16

0.21

0.0

0.2

0.16

0.15

0.17

0.19

0.16

0.14

0.14

0.14

0.15

0.18

0.14

0.12

0.13

0.17

0.0

0.17

0.14

0.16

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

-0.11

0.06

0.33

0.0

-0.17

-0.08

0.01

0.16

0.37

-0.11

-0.06

0.01

0.13

0.23

-0.14

-0.07

0.04

0.18

0.37

0.0

-0.1

0.08

0.32

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.07

0.22

0.53

0.0

0.04

0.08

0.16

0.33

0.56

0.05

0.08

0.15

0.26

0.38

0.04

0.08

0.16

0.32

0.53

0.0

0.07

0.21

0.48

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.1

0.26

0.57

0.0

0.06

0.11

0.19

0.36

0.58

0.07

0.1

0.17

0.28

0.4

0.04

0.09

0.17

0.32

0.54

0.0

0.07

0.21

0.48

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 300 episodes

0.0

0.16

0.18

0.28

0.0

0.17

0.13

0.15

0.22

0.27

0.13

0.12

0.15

0.19

0.21

0.15

0.12

0.14

0.2

0.25

0.0

0.15

0.16

0.24

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.37

0.03

-0.21

0.0

0.42

0.2

0.01

-0.14

-0.22

0.24

0.14

0.0

-0.11

-0.16

0.38

0.2

0.02

-0.12

-0.21

0.0

0.37

0.05

-0.16

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.53

0.21

0.07

0.0

0.59

0.34

0.16

0.08

0.04

0.36

0.26

0.15

0.08

0.05

0.53

0.32

0.16

0.08

0.04

0.0

0.52

0.21

0.07

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.53

0.22

0.08

0.0

0.59

0.34

0.18

0.1

0.07

0.36

0.26

0.16

0.1

0.08

0.53

0.32

0.17

0.09

0.06

0.0

0.52

0.22

0.08

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 350 episodes

0.0

0.24

0.19

0.23

0.0

0.26

0.2

0.16

0.17

0.22

0.19

0.18

0.15

0.14

0.16

0.23

0.19

0.15

0.16

0.2

0.0

0.23

0.17

0.2

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

-0.2

-0.14

-0.19

0.0

-0.19

-0.13

-0.08

-0.09

-0.15

-0.0

-0.0

-0.01

0.02

0.04

0.31

0.12

0.09

0.13

0.3

0.0

0.34

0.2

0.36

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.04

0.05

0.04

0.0

0.07

0.07

0.07

0.08

0.07

0.19

0.17

0.14

0.16

0.2

0.54

0.31

0.24

0.29

0.5

0.0

0.57

0.37

0.55

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.05

0.06

0.04

0.0

0.08

0.09

0.08

0.08

0.06

0.22

0.19

0.15

0.16

0.2

0.56

0.32

0.25

0.29

0.5

0.0

0.58

0.39

0.56

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 400 episodes

0.0

0.19

0.15

0.19

0.0

0.21

0.15

0.12

0.14

0.19

0.19

0.17

0.14

0.15

0.17

0.3

0.23

0.19

0.2

0.27

0.0

0.31

0.23

0.27

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.4

0.25

0.38

0.0

0.33

0.21

0.16

0.19

0.34

0.02

0.0

0.01

0.02

0.04

-0.23

-0.14

-0.11

-0.12

-0.19

0.0

-0.26

-0.18

-0.23

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.59

0.39

0.58

0.0

0.54

0.36

0.28

0.33

0.53

0.21

0.18

0.15

0.17

0.21

0.07

0.09

0.08

0.08

0.08

0.0

0.04

0.06

0.04

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.58

0.38

0.57

0.0

0.54

0.36

0.28

0.33

0.53

0.22

0.18

0.16

0.17

0.22

0.08

0.1

0.1

0.09

0.09

0.0

0.06

0.08

0.05

0.0

TD estimates

1

2

3

4

5

1

2

3

4

5

After 450 episodes

0.0

0.28

0.22

0.27

0.0

0.29

0.22

0.19

0.2

0.26

0.2

0.18

0.15

0.16

0.18

0.25

0.18

0.14

0.16

0.22

0.0

0.24

0.17

0.21

0.0

PV estimates

1

2

3

4

5

1

2

3

4

5

0.0

-0.21

-0.01

0.24

0.0

-0.24

-0.14

-0.0

0.13

0.33

-0.15

-0.09

0.0

0.11

0.2

-0.21

-0.1

0.01

0.16

0.28

0.0

-0.18

0.01

0.25

0.0

TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.07

0.21

0.51

0.0

0.04

0.08

0.19

0.33

0.6

0.05

0.09

0.15

0.26

0.38

0.04

0.08

0.15

0.32

0.51

0.0

0.07

0.18

0.47

0.0

PV+TV estimates

1

2

3

4

5

1

2

3

4

5

0.0

0.09

0.24

0.53

0.0

0.05

0.1

0.21

0.35

0.61

0.06

0.09

0.16

0.27

0.39

0.04

0.08

0.15

0.32

0.51

0.0

0.06

0.17

0.46

0.0

TD estimates

1

2

3

4

5

After 500 episodes

0.0

0.23

0.22

0.32

0.0

0.23

0.18

0.19

0.24

0.33

0.15

0.14

0.15

0.2

0.24

0.21

0.15

0.15

0.21

0.28

0.0

0.2

0.18

0.27

0.0

PV estimates

1

2

3

4

5

0.0

0.33

0.04

-0.22

0.0

0.3

0.16

-0.01

-0.15

-0.29

0.23

0.12

-0.01

-0.11

-0.18

0.37

0.19

0.04

-0.12

-0.24

0.0

0.34

0.06

-0.19

0.0

TV estimates

1

2

3

4

5

0.0

0.56

0.26

0.1

0.0

0.53

0.33

0.17

0.09

0.04

0.38

0.26

0.14

0.08

0.06

0.57

0.33

0.18

0.09

0.05

0.0

0.55

0.23

0.08

0.0

PV+TV estimates

1

2

3

4

5

0.0

0.56

0.26

0.11

0.0

0.52

0.33

0.18

0.1

0.06

0.37

0.25

0.15

0.1

0.08

0.56

0.33

0.19

0.1

0.07

0.0

0.54

0.24

0.09

0.0

TD estimates

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

C.1.2 Policy evaluation minigrid

We use 4 rooms task as shown in Fig. 1b. The environment is a grid task with each room containing one type of item. Each item type is different in terms of how it is perceived and in terms of reward the agent receives upon collecting it. There are 4 goal cells, one in each room located at the corner. At

the beginning of the episode, the agent starts in one of the cells within the region spanning between cells (3,3) and (6,6). It has 3 actions to chose from turn left, turn right, and move forward. Turn-based actions changes the orientation of the agent along the corresponding direction, whereas the move action changes the agent's location to the state in front of it. The agent can only observe a 5x5 view in front of it (partial view). Items are one-hot represented along the 3rd dimension to make the overall input space 5x5x5. To set up the continual learning problem, we change the rewards of the items from task-to-task as described in Table C.2. The evaluation policy is kept uniformly random for all states and for all the tasks. We use a discount factor of 0.9. We evaluate the performance for a total of 750 episodes and the tasks are changes after every 75 episodes.

We used a convolution neural network (CNN) to approximate the value function for all algorithm. The CNN consists of three conv layers with 16, 32, and 64 filters, (2,2) stride and relu activation in each layer. The output of the third conv layer is flatted, which is then followed by a couple of linear layers with 64 and 1 hidden unit. Tanh activation in the first linear layer and no activation is used in the last layer. We attach a head after conv layers for permanent value function. The architecture of this head is same as the TD learning head. The training method and buffer capacities are same as the discrete image grid task. We use SGD optimizer to update weights in all networks. The best learning rate is picked by computing the area under the error curve on a set of values shown in Table C.8 and Fig. C.13. All the results reported are averaged across 30 seeds.

C.1.3 Control minigrid

We use two room environment as shown in Fig. 3c. The agent starts in one of the bottom row states in the bottom room and there are two goal states located in the top room. In the first task, the goal rewards are +5 and 0 for the first and second goals respectively, and it is flipped for the second task. We alternate between two tasks every 500 episodes for a total of 2500 episodes. The agent receives a one hot, 5x5 view (partial observability) in front of it at each timestep. It can take one of the three actions (move forward, turn left, turn right) at every step. The move action transitions the agent to the state in front of it if there is no obstacle. The environment is deterministic and episodic, and we use a discount factor of 0.99.

We use a CNN as a function approximator. The CNN has 3 conv layers with 16, 32, and 64 filters, (2,2) stride, and relu activations. It is followed by a fully connected neural network with 128, 64, and 4 hidden units. The first layer is relu activated, the second layer is tanh activated, and the final layer is linear. We use target network and experience replay to train the network. Target network's weights are updated every 200 timesteps. Experience replay buffer's capacity is capped to 100k and the stored samples are deleted when the task changes. We use batch size of 64 and update the network every timestep. We search for the best learning rate from a set of value shown in Table C.10 on 3 seeds and we pick the one that gave the highest area under the rewards curve. We then run on 30 seeds using that learning rate to report the final performance. The transient value function training procedure and details are identical to DQN. permanent value function uses a separate head following conv layers and is updated using Adam optimizer whenever the task is changed.























C.1.4 PT-DQN Pseudocode

Algorithm 4 PT-DQN Pseudocode (Continual Reinforcement Learning)

1: Initialize transient network buffer B , permanent network buffer D

2: Initialize permanent network θ , transient network w

3: Initialize target network w target

4: for t : 0 →∞ do

5: Take action A t according to glyph[epsilon1] -greedy policy

6: Observe R t +1 , S t +1

7: Store ( S t , A t , R t +1 , S t +1 ) in B

8: Store ( S t , A t , Q ( P ) ( S t , A t ) ) in D

9: Sample a batch of transitions ( S j , A j , R j +1 , S j +1 ) from B

10: Compute target y = R j +1 + γ max a ′ ∈A ( Q ( P ) ( S j +1 , a ′ ; θ ) + Q ( T ) ( S j +1 , a ′ ; w target ) ) -Q ( P ) ( S j , A j ; θ )

11: Perform a gradient step on ( y -Q ( T ) ( S j , A j ; w ) ) 2 with respect to w

12: Update target network every N steps, w target = w

13: if mod ( t, k ) == 0 then

14: Sample a batch of transitions ( S j , A j , Q ( P ) ( S j , A j )) from D

15: Compute target ˆ y = Q ( P ) ( S j , A j ) + Q ( T ) ( S j , A j ; w )

16: Perform a gradient step on ( ˆ y -Q ( P ) ( S j , A j ; θ ) ) 2 with respect to θ

17: Decay transient network weights, w = λ w

18: Clear permanent network buffer, D = {}

19: end if

20: end for

C.1.5 JellyBeanWorld (JBW)



























(a) Sample observation received by the agent.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Episodes

1e6

0

50000

100000

150000

200000

250000

300000

350000

400000

Cumulative rewards

=

0.55

=

0.75

=

0.95

Q learning

(b) Cummulative rewards against timesteps averaged across 3 seeds.

We test algorithms on a continual learning problem in the JBW testbed [31] and use neural networks as the function approximator. The environment is a two-dimensional infinite grid with three types of items: blue, red, and green. We induce spatial non-stationarity by setting the parameters of the environment as described in Table C.3 . In our configuration, three to four similar items (red or blue) form a tiny group and similar groups appear close to each other to form a sea of red (or blue) objects as shown in Fig. 5a. Therefore, the local observation distribution at various parts of the environment are different. At each timestep, the agent receives a egocentric 11x11 RGB, 360 o view

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

2.5

Counts

×

10

2

Green items

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

2.5

Counts

×

10

4

Positive items

0

5

10

Task switches

0

1

2

3

4

5

Counts

×

10

3

Negative items

PT-DQN-0.5x (ours)

DQN

DQN-multi-head

DQN-large buffer

Random

as an observation as shown in Fig. C.4a. It has four actions to choose from: up, down, left, and right. Each action transitions it by one square along the corresponding direction. The rewards for green items, which are uniformly distributed, are set to 0 . 1 , whereas the rewards for picking red and blue items alternate between -1 and +2 every 150k timesteps, therefore, inducing reward non-stationarity. We run the experiment for 2.1 million timesteps and train algorithms using a discount factor 0.9.

The DQN agent uses four layered neural network with relu activations in all layers except the last to estimate q-values. The number of hidden units are 512, 256, 128, and 4 respectively. The network is trained using Adam optimizer with experience replay and a target network. We use epsilon-greedy policy with glyph[epsilon1] = 0 . 1 for exploration. The permanent and transient network architectures are identical to that of DQN but with half the number of parameters in each layer. This ensures that the total number of parameters for the baselines and our method is same. Transient network is also trained using Adam optimizer, whereas, permanent network is updated every 10k timesteps ( k =10,000) using SGD optimizer. Target network's weights are updated every 200 timesteps. Experience replay buffer capacity is capped to 8000 as higher capacity buffer stores old training samples which affects the training when the environment is changing continuously. We use batch size of 64 and update the network every timestep. For our approach, we experimented with three choices of λ values (0.55, 0.75, and 0.95) and all the choices resulted in a better performance than the DQN agent as seen in Fig. C.4b. We use the same exploration policy as the DQN agent. We flatten the observation into one long array, then stack the last four observation arrays to form an input to the neural network. The best learning rate for all algorithms are chosen by comparing the total reward obtained in 1.5 million timesteps for a set of fixed learning rate values on 3 different seeds. We search for the best learning rate from a set of value shown in Table C.12 on 3 seeds and we pick the one that gave the highest rewards. We then run on 30 seeds using that learning rate to report the final performance. The random agent picks one of the four actions according to uniform probability at each timestep. For the multi-head DQN baseline, we attach two output heads - one for each task. The baseline knows the task IDs, so it chooses the appropriate head to select actions and to train the network. The replay buffer is reset for this baseline whenever the task is changed. We use a buffer capacity of 250k for the DQN baseline that is augmented with a large buffer. The rest of the training and architecture details are same as the DQN agent.

C.1.6 MinAtar

This domain is a standard benchmark for testing single task RL algorithms [51]. We use breakout, freeway, and space invaders environments to setup a continual RL problem by randomly picking one of these tasks every 500k steps for a total of 3.5M steps. We standardize the observation space across tasks by padding 0s when needed and make all the actions available for the agent. This problem has non-stationarity in transitions besides the reward and observation signals making it challenging. We use a discount factor of 0.99 for this experiment.

We used a CNN which has 1 relu activated conv layer with 16 filters followed by a relu activated linear layer with 256 units, and a linear output layer for DQN. The architecture is same as the one used in the original paper [51]. The architecture is same for other DQN variants but we add 3 output layers for the multi-headed DQN. For our method, we use the same architecture but 8 filter in the conv layer and 128 units in the linear layer. Target network is used for all the methods which is updated every 1000 steps and we use an experience replay of 100k for all methods but we use 500k sized experience replay for large replay based DQN baseline. We use Adam optimizer to train the networks in online fashion. For our method, permanent value function network is updated using SGD

optimizer every 50k steps. The best hyperparameters are chosen by running the experiment for 1.5M steps by switching the tasks every 500k steps on 3 seeds. The results of hyperparameter tuning are detailed in Tables C.17, C.18, C.19, and C.20. We make all 7 actions available to the agent at all times and the input to the agent is 10x10x7 images.

C.2 What is the effect of task distribution on the performance for various methods?

⋂∐√⌋(̂]√√(̂]∐⌉˜√˜√(/









⋂∐√⌋(⋃glyph[arrowvertexdbl]]√̂[˜√











⋂∐√⌋(̂]√√(̂]∐⌉˜√˜√(/









⋂∐√⌋(⋃glyph[arrowvertexdbl]]√̂[˜√











⋂∐√⌋(̂]√√(̂]∐⌉˜√˜√(/









⋂∐√⌋(⋃glyph[arrowvertexdbl]]√̂[˜√











⋂〈

⋂〈({√˜√˜√}

∑⋂∖⋂〈({}√√√}

Setup: We set up a continual learning problem by creating 7 tasks whose value functions are glyph[epsilon1] (task diameter) distance from one another in norm-2. Each task has 7 states. For a particular glyph[epsilon1] value, for a particular number of task switches, we run each algorithm 100 times and save the final estimates which form the initial point for the new task. We then sample a task according to uniform distribution and use analytical MSE expression derived in the previous section to analyze the behaviour of PT-TD learning and the two versions of TD learning algorithm for 100 rounds. We then average the error across rounds and across different starting values to get a point in the plot shown in Fig. C.6.

Task details: We use the following P π for all tasks and for all task diameters ( glyph[epsilon1] ):

















We then sample 7 points in R n that are glyph[epsilon1] units from one another. These points form the true value function of the tasks. We then compute the reward vector for each task τ as r π ( τ ) = ( I -γ P π ) v π ( τ ) . At the time of testing the algorithms, we add a zero mean Gaussian noise with 0.1 standard deviation to these rewards to simulate stochasticity. We search from the learning rates shown in Table C.4 to pick the best one for each combination of glyph[epsilon1] and number of task switches. We use a learning rate of 0.01 while using analytical expressions.

Observation: We observe both TD learning algorithm and our approach has the same performance after seeing data from one task. But the error gap increases as the number of task switches increase. The low error in our method can be attributed to the presence of permanent value function which captures some portion of the value that is common for all tasks. transient value function then learns the remaining part with a small number of samples resulting in low error overall. We also observe that the error gap between TD learning and our method gets wider as we increase task diameter, which demonstrates that our method is broadly applicable. The reset variant of the TD learning algorithm whose predictions are reset whenever task changes has the largest error emphasizing learning from scratch could be costly.

C.3 What is the effect of network capacity on the performance?

Setup: We use JBW task in the continual RL setting to perform the ablation study. We run two additional experiments by varying the number of parameters in DNN for the DQN and our method. In the first experiment, we use the same DNN architecture for all algorithms, therefore, our method has twice the number of parameters as that of DQN. In the second experiment, we double the network capacity for the DQN variant making the total number of parameters same as ours. We keep the rest of the experimental setup same as before and we pick the best learning rate based on the AUC by searching over the values detailed in Table C.5 over 3 seeds for 1.5M steps.

DQN

PT-DQN-2x (ours)

Random

Jellybean world

0.4

(sdas 0 ssooe pebeee)

0.3

Reward per step

0.2

0.1

0.0

0.1

0.0

0.5

1.0

Steps

1.5

2.0

×16

(a) Results on the JellyBeanWorld.

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

2.5

Counts

×

10

2

Green items

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

2.5

Counts

×

10

4

Positive items

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

Counts

×

10

3

Negative items

PT-DQN-2x

DQN

Random

(b) Items collected in each task.

DQN (2x)

Random

PT-DQN

Jellybean worid

0.4

(averaged across 10000 steps)

0.3

Reward per step

0.2

0.1

0.0

0.1

0.0

0.5

1.0

Steps

1.5

2.0

X30²

(a) Results on the JellyBeanWorld.

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

2.5

Counts

×

10

2

Green items

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

2.5

Counts

×

10

4

Positive items

0

5

10

Task switches

0.0

0.5

1.0

1.5

2.0

Counts

×

10

3

Negative items

PT-DQN

DQN (2x)

Random

(b) Items collected in each task.

Observations: The results for ablation experiments are shown in Figures C.7 and C.8. We observe that our algorithm continues to perform better than the DQN agent in both the cases. The results also indicate that it is efficient to devote the available capacity in learning parts of the value function rather than learning the whole.

In the second ablation experiment, the DQN agent with twice the number of parameters starts off learning slowly but catches up with our method after seeing enough data. In fact, the DQN agent continues to perform similar to our method without any drop in performance as we continue to train further (we tested it by running the agent to 6M timesteps). This result highlights that our approach is beneficial in the big world - small agent setup where the computation budget of the agent is very small compared to the complexity of the environment. When the agent's capacity is large relative to the complexity of the environment, there's no additional benefit (neither there is any downside) to our method. Since the world is much more complicated in comparison to the agent's capacity, using our method is preferable.

These experiments also provide partial answer to the question - when does the plasticity problem arise in neural networks? We think that a combination of learning algorithm (like DQN), small agent

capacity, and a complex environment could give rise to the plasticity problem in neural networks. A thorough investigation is necessary to confirm this hypothesis.

C.4 What is the effect of hyperparameters on the performance?

In this section we analyze the sensitivity of additional hyperparameter in our method (learning rate to update permanent value function) and compare it with the sensitivity of TD and Q-learning algorithms on various environments.

0.0

0.2

0.4

0.6

0.8

LRs

0.10

0.12

0.14

0.16

0.18

0.20

0.22

0.24

Mean AUC of RMSVE

TD

0.0

0.2

0.4

0.6

0.8

LRs

0.10

0.12

0.14

0.16

0.18

0.20

0.22

0.24

Mean AUC of RMSVE

TD with resets

0.0

0.2

0.4

0.6

0.8

TV LRs

0.10

0.12

0.14

0.16

0.18

0.20

0.22

0.24

Mean AUC of RMSVE

PT-TD (ours)

PV LRs

0.1

0.05

0.01

0.005

0.001

0.0

0.2

0.4

0.6

0.8

LRs

0.10

0.12

0.15

0.18

0.20

0.23

0.25

Mean AUC of RMSVE

TD

0.0

0.2

0.4

0.6

0.8

LRs

0.10

0.12

0.15

0.18

0.20

0.23

0.25

Mean AUC of RMSVE

TD with resets

0.0

0.2

0.4

0.6

0.8

TV LRs

0.10

0.12

0.15

0.18

0.20

0.23

0.25

Mean AUC of RMSVE

PT-TD (ours)

PV LRs

0.1

0.05

0.01

0.005

0.001

For tabular and linear grid tasks, we select best hyperparameters (learning rates) for each algorithm by choosing those values which results in the lowest area under the error curves as shown in Table C.6, Figures C.9, C.10, C.11. The hyperparameter search and the final reporting is performed by averaging the results on 30 different seeds. For the deep RL task with discrete grid, we choose the best learning rate from the values shown in Table C.7 and the corresponding U curves are shown in C.12. For the minigrid policy evaluation task, we search for the best learning rates from the values shown in Table C.8 and the corresponding U curves are shown in Fig. C.13.

For the tabular control experiment, the learning rates used to search for the best value is listed in Table C.9 and the mean returns for various choices of these values are plotted in Fig. C.14 for various algorithms. Similar table for the minigrid task (deep control) is shown in Table C.10 and the mean returns for various choices of these values is shown in Fig. C.15. We use 30 seeds to pick the best learning rate for the tabular task but use 3 seeds to select the learning rates for the minigrid task.

The learning rate tables and the corresponding performance for the full continual RL experiments are shown in Fig. C.16 and Tables C.11, C.12, C.17, C.18, C.19, and C.20. We use 30 seeds to find the best hyperparmeters for the tabular task, but we use 3 seeds for JBW experiment and MinAtar.

0.0

0.2

0.4

0.6

0.8

LRs

0.31

0.32

0.32

0.33

0.33

0.33

0.34

0.34

0.35

Mean AUC of RMSVE

TD

0.0

0.2

0.4

0.6

0.8

LRs

0.31

0.32

0.32

0.33

0.33

0.33

0.34

0.34

0.35

Mean AUC of RMSVE

TD with resets

0.0

0.2

0.4

0.6

0.8

TV LRs

0.31

0.32

0.32

0.33

0.33

0.33

0.34

0.34

0.35

Mean AUC of RMSVE

PT-TD (ours)

PV LRs

0.1

0.05

0.01

0.005

0.001

We find the best learning rate for each k, λ pair by searching over the learning rates listed in Table C.11.

Observations: We see that our algorithm performs better than their vanilla counterparts on all the problems we tested for a broad range of hyperparameters. There are a handful of ( α, α ) pairs that results in a better performance in each setting, which indicates that our method is not sensitive to hyperparameter selection. We also observe that in all the cases α is smaller than α , which fits into our intuition. Because, the permanent value function has to capture regularities in the value function estimate, it needs to update slowly, whereas, the transient value function should be updated quickly to facilitate fast adaptation. The sensitivity of two new hyperparameters: λ, k for the fully continual RL setting is analyzed in the main paper (see Sec. 6.1) where we concluded that our algorithm is robust to these values.

0.00

0.01

0.02

0.03

LRs

0.80

1.00

1.20

1.40

1.60

1.80

Mean AUC of RMSVE

0.00

0.01

0.02

0.03

LRs

0.80

1.00

1.20

1.40

1.60

1.80

Mean AUC of RMSVE

0.002

0.004

0.006

0.008

0.010

TV LRs

0.80

1.00

1.20

1.40

1.60

1.80

Mean AUC of RMSVE

PT-Deep TD (ours)

PV LRs

0.001

0.0003

0.0001

3e-05

0.00

0.01

0.02

0.03

LRs

1.90

2.00

2.10

2.20

2.30

2.40

2.50

Mean AUC of RMSVE

0.00

0.01

0.02

0.03

LRs

1.90

2.00

2.10

2.20

2.30

2.40

2.50

Mean AUC of RMSVE

0.00

0.01

0.02

0.03

TV LRs

1.90

2.00

2.10

2.20

2.30

2.40

2.50

Mean AUC of RMSVE

PT-Deep TD (ours)

PV LRs

0.003

0.001

0.0003

0.0001

3e-05

0.0

0.2

0.4

0.6

0.8

LRs

0.00

0.10

0.20

0.30

0.40

0.50

Mean returns

Q-learning

0.0

0.2

0.4

0.6

0.8

LRs

0.00

0.10

0.20

0.30

0.40

0.50

Mean returns

Q-learning with resets

0.0

0.2

0.4

0.6

0.8

TV LRs

0.00

0.10

0.20

0.30

0.40

0.50

Mean returns

PT-Q-learning

PV LRs

0.8

0.5

0.3

0.1

0.05

0.01

0.005

0.001

Algorithm

LRs

0.000

0.001

0.002

0.003

LRs

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Mean AUC of Returns

DQN

0.000

0.002

0.004

0.006

0.008

0.010

LRs

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Mean AUC of Returns

DQN with resets

0.0000

0.0001

0.0002

0.0003

TV LRs

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Mean AUC of Returns

PT-DQN (ours)

PV LRs

1e-05

3e-06

1e-06

3e-07

0

100

200

300

400

500

Episodes

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Returns

k

=

1,

=

0.7

0

100

200

300

400

500

Episodes

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Returns

k

=

5,

=

0.7

0

100

200

300

400

500

Episodes

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Returns

k

=

10,

=

0.5

0

100

200

300

400

500

Episodes

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Returns

k

=

25,

=

0.3

0

100

200

300

400

500

Episodes

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Returns

k

=

100,

=

0.7

Q-learning

PT-Q-learning

Peer reviews:
comments: SUMMARY:
In this work, the author proposes a new continual reinforcement learning method, the PT-TD, which consists of two value networks, one learns the transient memories and the other learns the permanent memories. The author proves that in the semi-continual setting, the PT-TD algorithm will get better performance and lower forgetting, and the same result can also be found in the continual reinforcement learning setting.

STRENGTHS:
The algorithm of this article is well-proven and well-demonstrated. The author not only proves the PT-TD algorithm in a mathematical way but also gives out detailed and rigorous experiments. Also, the article is really well-written and easy to understand. All of the hyperparameters are studied. Meanwhile, it is appreciated that the author discussed semi-continual reinforcement learning and continual reinforcement learning separately.

WEAKNESSES:
First, even though the proof is very well, the algorithm itself is not very groundbreaking, or, the result is not very rewarding. Besides, the environments of the experiments are simple.

QUESTIONS:
I think the formulation of the Theorem 8 is a little confusing. Maybe you can give out explicitly which network is using (P or T).

Can your algorithm generalize to continuous action space? How should I learn the actor network at this time?

I notice that in the continual reinforcement learning setting, the learning rate of T network is always higher than it of P network. According to [1], if it is the difference in the learning rate affect good performance?

[1]Arani, E. et al. “Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System.” ArXiv 2021

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes new algorithms for prediction and control in continual reinforcement learning. The new algorithms use permanent and temporary memories. The permanent memory captures a general value function for all the tasks, while the transient memory is used to adapt to the current tasks continually. The paper provides theoretical results for the prediction setting with a tabular learner. Additionally, the paper provides empirical results showing that their method outperforms vanilla TD-based algorithms in various environments using different types of function approximators.\n\nSTRENGTHS:\nThe paper directly tackles the continual reinforcement learning setting.\nThis is a challenging problem setting with few general algorithms and evaluation environments.\nThe paper provides prediction and control algorithms for the continual RL setting.\nTheoretical proofs of convergence are provided for the prediction setting with a tabular learner.\nThe extensive and sound empirical investigation shows that the proposed method can generally outperform existing methods in various environments and with different various function approximations, including Deep Networks.\n\nWEAKNESSES:\nThe paper does not have any major weaknesses. I did not check the proofs, so I can not comment on the correctness of the proofs.\nHowever, there are a few minor issues that can be fixed to improve the paper.\n1. I'm confused about Equation 5, specifically line 112. Does it mean the \"gradient\" or the \"semi-gradient\" of the transient memory?\n2. What are the confidence intervals being reported? Line 222 says it is the 90% confidence interval, but is the bootstrapped confidence interval or confidence interval based on the standard error? And if it is based on the standard error, how do you know it is the 90% confidence interval?\n3. The continual deep RL algorithms used in the paper use standard deep RL tools like Experience replay and target networks. However, the exact use of these tools is not fully explained in the paper (or the appendix). This creates a lot of questions; for example, are there a total of three networks, target, transient and permanent? I suggest that the authors include a section in the appendix that gives the pseudocode of the PT-DQN algorithm. The lack of details about the PT-DQN algorithm makes it hard to replicate the results given in the paper.\n\nQUESTIONS:\n1. Line 88 says, \"... multi-task RL methods ...\" What is multi-task RL, and how is it different from continual RL? What limits the use of multi-task RL methods in continual RL? The second question should be answered in the main text.\n2. Figure 4d shows the performance of different versions of DQN  on Minigrid. Why is the performance of all the methods, particularly DQN(reset), so much worse during the second task? If the DQN is being reset at each task, it should perform similarly on Episodes 500 and 1000. Is this because the first goal state in all the runs is the same?\n3. What is PT-DQN-0.5x in Figure 5? 0.5x should be described in the main paper.\n4. Lines 733-742 describe the training details for the deep RL experiments. However, many of these choices seem arbitrary, or the paper does not explain the reasoning behind these choices. Why are the conv layers not updated when updating the permanent memory? Why isn't there a fully independent permanent value network? Wouldn't that be closer to the algorithms presented in the paper?\n5. Lines 803-805 say that the permanent memory was trained with SGD while the rest of the network was trained with Adam. Why this specific choice? I'm guessing the authors also tried Adam to update the permanent memory. Why didn't Adam work for permanent memory?\n6. The plots (like Figure C.13) showing the sensitivity curve will be better with a log scale for the x-axis.\n7. Why isn't there any effect of PM LR on the performance in the minigrid environment (Figure C.13c)?\n\nLIMITATIONS:\nOne limitation of this work is that it only proposes methods for value-based RL. It is unclear how the idea of permanent and transient memory can be extended to policy-gradient based RL. A discussion on this direction of future work will be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to improve continual learning in reinforcement learning with non-stationary dynamics. It proposes to maintain two value functions: one that tracks information about the current task (transient) and one that maintains a general knowledge of the task distribution (permanent) in order to better tackle the non-stationarity of the environment.\n\nSTRENGTHS:\n1. The general idea is simple in terms of intuition and implementation. I like the idea of maintaining two different value functions operating different time-scales.\n2. Assuming soundness (see below), the experiments well-done and are insightful.\n\nWEAKNESSES:\nWhile the overall approach is intuitive, the specifics dont seem very intuitive to me and there is not much information given about it. For example, Eq 4 (see questions below) is not intuitive to me on how it was derived. Similarly, I have other questions.\n\nQUESTIONS:\n1. What do the authors mean when they say prior work “reflected the nature of that task” in line 41?\n2. What is the intuition for the update in Eq 4? Why does the update to the parameters of the P get scaled by V^PT? Similarly, what is the intuition for Eq 5? Why is \\delta purely in terms of the overall update, PT?\n3. The definition in Eqn 2 and 3 is that both memories are equally weighted and are just a sum. I am curious what the authors think about this equal weighting and the consequences of it?\n4. It is a little unclear to me why only the initialization condition is sufficient for Theorem 1 to hold?\n5. Shouldn’t the equation in Theorem 2 be what is in Equation 5 i.e. PT instead of P?\n6. It may be worthwhile to see how Continual backprop [1] can be incorporated into the proposed method.\n\n[1] Continual Backprop: Stochastic Gradient Descent with Persistent Randomness. Dohare et al.\n\nLIMITATIONS:\nSee above questions", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work proposes a reinforcement learning method for prediction and control. The value function in reinforcement learning is decomposed into two terms: a permanent value function, and a transient value function. The permanent function contains the long term knowledge of the interplay between the agent and the environment, and the transient value function absorbs the quick information variations. This work uses temporal difference learning for updating the estimate of the value function estimation. The learning process is related to the complementary learning systems from neuroscience. Prediction experiments are performed for 5x5 discrete grid. The agent moves from the center and selects an action the goal states.

STRENGTHS:
It is of great interest to extract valuable information from stream data. The development of organic transistor featuring a selective transition from short-term to long-term plasticity of the biological synapse. Catastrophe forgetting is a major challenge for machine learning. A possible solution for solving such a problem is very attractive. It is closely related to meta learning.

WEAKNESSES:
The idea for separating long term and transient information is intuitive. It has been used in many machine learning algorithms such as LSTM.  It is not clear if a long term function is always available, especially for complicated interplay between an agent and its environment. The separation of long term and transient can be dynamic. 

The settings for the experiments are very simple. It does not demonstrate the possibilities for real applications.

QUESTIONS:
In the experiments, the proposed method is compared with DQN. Is it possible to compare other competitive algorithms?

LIMITATIONS:
The limitations of this work are not explicitly given.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06