SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\xi_ {t + 1} ^ {(P T)} = \xi_ {t} ^ {(P T)} + \alpha \sum_ {s \in \mathcal {S}} (- 2 v _ {\pi} (s) \Delta_ {t} ^ {(P T)} (s) + \Omega_ {t} ^ {(P T)} (s, s)) + \alpha^ {2} \sum_ {s \in \mathcal {S}} \Psi_ {t} ^ {(P T)} (s, s).
$$
Besides providing details for the experiments described in the main paper, we answer additional interesting questions, such as:
- What is the effect of task distribution on the performance for various methods (Sec. C.2)?
- What is the effect of network capacity on the performance (Sec. C.3)?
- What is the effect of hyperparameters on the performance (Sec. C.4)?
C.1 Additional Details
We begin by providing details of the experiments.
C.1.1 Policy evaluation grids
We modify the rewards of the goal states as described in Table C.1. The true value function for each task is shown in Fig. C.1. We convert the 5x5 discrete grid into a 48x48 RGB image for the deep RL
Task ID	Reward
Top-Left	Top-Right	Bottom-Left	Bottom-Right
1	0	1	0	1
2	1	0	1	0
3	0	0	1	1
4	1	1	0	0
Table C.1: Rewards at goal states for various tasks.
Figure C.1: Heatmap of the true value function for different tasks.
task. The agent's location is indicated by colouring the its location in red and we use green squares to indicate goal states. At every timestep, the agent receives a RGB image as an input. The action space and the action consequences are the same as discrete grid. We select best hyperparameters (learning rates) for each algorithm by choosing those values which results in the lowest area under the error curves as shown in Table C.6 and Fig. C.9. The hyperparameter search and the final reporting is performed by averaging the results on 30 different seeds. For our method, the buffer stores the samples from the latest task only to update the permanent value function.
For the first linear FA experiment, we use the same setup as the tabular setting except we encode each state in the grid as a vector of 10 bits. The first 5 bits encodes the row of the state and the last 5 bits encodes the column of the state. For example, the starting state (2,2) is represented as  $[0,0,1,0,0,0,0,1,0,0]$ . Both permanent value function and transient value function use same features to learn respective estimates.
For the second linear FA experiment, we use a continuous version of the grid. The goal rewards for each task is same as the discrete counterpart as described in Table C.1. The true value function is also different for each task and it is shown as a heatmap in Fig. C.2. We use RBFs to generate features as described in Sutton and Barto [45]. We use 676 centers (order $^2$ ) spaced equally in both x and y directions across the grid with a variance  $(0.75 / (order - 1))^2$  and use a threshold to convert the continuous vector into a binary feature vector. Any feature value greater than 0.5 is set to 1, while those features with values less than 0.5 is set to 0. We search for the best learning rate from the same set of values as the discrete grid task. The hyperparameter curves are shown in Fig. C.11.
For the deep RL experiment, the rewards are multiplied by a factor of 10 to minimize the initialization effect of deep neural network on prediction. We used a convolution neural network (CNN) to approximate the value function for all algorithms tested. The CNN consists of two conv layers with
Figure C.2: Heatmap of the true value function for different tasks.
8 filters, (2,2) stride, and relu activation in each layer. We use a max pooling layer in between the two conv layers with a stride 2. The output of the second conv layer is flatted, and it is followed by a couple of linear layers. The fully connected network has 64 and 1 hidden units in the respective layers. The first fully connected layer uses a tanh activation and no activation is used in the final layer. For our method, we attach a new head after the first two conv layers for permanent value function. This head also has two linear layers where the first layer has tanh activation. The entire network is trained by backpropagating the errors from the transient value function. But, the conv layers are not trained when updating the permanent value function. The transient value function and the TD learning algorithm are trained in an online fashion using an experience replay buffer of size 100k. The buffer is reset at the end of the task for all methods to reduce the bias from old samples. Permanent value function is trained at the end of the task using the samples from the latest task. 100 gradient steps are taken during each update cycle. We use SGD optimizer to update its weights. The best learning rate is picked by computing the area under the error curve on a set of values shown in Table C.7. The mean AUC curves of various choices of learning rates are shown in Fig. C.12.
Figure C.3: Predictions evolution across episodes.
C.1.2 Policy evaluation minigrid
We use 4 rooms task as shown in Fig. 1b. The environment is a grid task with each room containing one type of item. Each item type is different in terms of how it is perceived and in terms of reward the agent receives upon collecting it. There are 4 goal cells, one in each room located at the corner. At
the beginning of the episode, the agent starts in one of the cells within the region spanning between cells (3,3) and (6,6). It has 3 actions to chose from turn left, turn right, and move forward. Turn-based actions changes the orientation of the agent along the corresponding direction, whereas the move action changes the agent's location to the state in front of it. The agent can only observe a  $5 \times 5$  view in front of it (partial view). Items are one-hot represented along the 3rd dimension to make the overall input space  $5 \times 5 \times 5$ . To set up the continual learning problem, we change the rewards of the items from task-to-task as described in Table C.2. The evaluation policy is kept uniformly random for all states and for all the tasks. We use a discount factor of 0.9. We evaluate the performance for a total of 750 episodes and the tasks are changes after every 75 episodes.
Task ID	Reward
Blue (●)	Red (●)	Yellow(●)	Purple (●)
1	-1	1	1	1
2	1	1	0.5	1
3	1	1	1	-1
4	-1	1	0.5	-1
Table C.2: Rewards for items for various tasks.
We used a convolution neural network (CNN) to approximate the value function for all algorithms. The CNN consists of three conv layers with 16, 32, and 64 filters, (2,2) stride and relu activation in each layer. The output of the third conv layer is flattened, which is then followed by a couple of linear layers with 64 and 1 hidden unit. Tanh activation in the first linear layer and no activation is used in the last layer. We attach a head after conv layers for permanent value function. The architecture of this head is same as the TD learning head. The training method and buffer capacities are same as the discrete image grid task. We use SGD optimizer to update weights in all networks. The best learning rate is picked by computing the area under the error curve on a set of values shown in Table C.8 and Fig. C.13. All the results reported are averaged across 30 seeds.
C.1.3 Control minigrid
We use two room environment as shown in Fig. 3c. The agent starts in one of the bottom row states in the bottom room and there are two goal states located in the top room. In the first task, the goal rewards are +5 and 0 for the first and second goals respectively, and it is flipped for the second task. We alternate between two tasks every 500 episodes for a total of 2500 episodes. The agent receives a one hot, 5x5 view (partial observability) in front of it at each timestep. It can take one of the three actions (move forward, turn left, turn right) at every step. The move action transitions the agent to the state in front of it if there is no obstacle. The environment is deterministic and episodic, and we use a discount factor of 0.99.
We use a CNN as a function approximator. The CNN has 3 conv layers with 16, 32, and 64 filters, (2,2) stride, and relu activations. It is followed by a fully connected neural network with 128, 64, and 4 hidden units. The first layer is relu activated, the second layer is tanh activated, and the final layer is linear. We use target network and experience replay to train the network. Target network's weights are updated every 200 timesteps. Experience replay buffer's capacity is capped to  $100\mathrm{k}$  and the stored samples are deleted when the task changes. We use batch size of 64 and update the network every timestep. We search for the best learning rate from a set of value shown in Table C.10 on 3 seeds and we pick the one that gave the highest area under the rewards curve. We then run on 30 seeds using that learning rate to report the final performance. The transient value function training procedure and details are identical to DQN. permanent value function uses a separate head following conv layers and is updated using Adam optimizer whenever the task is changed.
C.1.4 PT-DQN Pseudocode
Algorithm 4 PT-DQN Pseudocode (Continual Reinforcement Learning)
1: Initialize transient network buffer  $\mathcal{B}$  , permanent network buffer  $\mathcal{D}$    
2: Initialize permanent network  $\theta$  , transient network w   
3: Initialize target network  $\mathbf{w}_{target}$    
4: for  $t:0\to \infty$  do   
5: Take action  $A_{t}$  according to  $\epsilon$  -greedy policy   
6: Observe  $R_{t + 1},S_{t + 1}$    
7: Store  $(S_{t},A_{t},R_{t + 1},S_{t + 1})$  in  $\mathcal{B}$    
8: Store  $(S_{t},A_{t},Q^{(P)}(S_{t},A_{t}))$  in  $\mathcal{D}$    
9: Sample a batch of transitions  $(S_{j},A_{j},R_{j + 1},S_{j + 1})$  from  $B$    
10: Compute target  $y = R_{j + 1} + \gamma \max_{a^{\prime}\in A}\left(Q^{(P)}(S_{j + 1},a^{\prime};\theta) + Q^{(T)}(S_{j + 1},a^{\prime};\mathbf{w}_{target})\right) - Q^{(P)}(S_{j},A_{j};\theta)$    
11: Perform a gradient step on  $(y - Q^{(T)}(S_j,A_j;\mathbf{w}))^2$  with respect to w   
12: Update target network every  $N$  steps,  $\mathbf{w}_{target} = \mathbf{w}$    
13: if mod(t,k) == 0 then   
14: Sample a batch of transitions  $(S_{j},A_{j},Q^{(P)}(S_{j},A_{j}))$  from  $D$    
15: Compute target  $\hat{y} = Q^{(P)}(S_j,A_j) + Q^{(T)}(S_j,A_j;\mathbf{w})$    
16: Perform a gradient step on  $(\hat{y} -Q^{(P)}(S_j,A_j;\theta))^2$  with respect to  $\theta$    
17: Decay transient network weights,  $\mathbf{w} = \lambda \mathbf{w}$    
18: Clear permanent network buffer,  $D = \{\}$    
19: end if   
20: end for
C.1.5 JellyBeanWorld (JBW)
Item Blue (●) Green (●) Red (●) Color [0, 0, 1.0] [0, 1.0, 0] [1.0, 0, 0.0] Intensity Constant [-2.0] Constant [-6.0] Interactions Piecewise box [2, 100, 10, -5] Zero Piecewise box [150, 0, -100, -100] Zero Interactions Zero Piecewise box [150, 0, -100, -100] Zero Piecewise box [2, 100, 10, -5]
Table C.3: JellyBeanWorld environment parameters.
Figure C.4: JellyBeanWorld sample observation and performance curves for various decay values.
We test algorithms on a continual learning problem in the JBW testbed [31] and use neural networks as the function approximator. The environment is a two-dimensional infinite grid with three types of items: blue, red, and green. We induce spatial non-stationarity by setting the parameters of the environment as described in Table C.3. In our configuration, three to four similar items (red or blue) form a tiny group and similar groups appear close to each other to form a sea of red (or blue) objects as shown in Fig. 5a. Therefore, the local observation distribution at various parts of the environment are different. At each timestep, the agent receives a egocentric  $11 \times 11$  RGB,  $360^{\circ}$  view
Figure C.5: Items collected by various algorithms during each task switches.
as an observation as shown in Fig. C.4a. It has four actions to choose from: up, down, left, and right. Each action transitions it by one square along the corresponding direction. The rewards for green items, which are uniformly distributed, are set to 0.1, whereas the rewards for picking red and blue items alternate between -1 and +2 every 150k timesteps, therefore, inducing reward non-stationarity. We run the experiment for 2.1 million timesteps and train algorithms using a discount factor 0.9.
The DQN agent uses four layered neural network with relu activations in all layers except the last to estimate q-values. The number of hidden units are 512, 256, 128, and 4 respectively. The network is trained using Adam optimizer with experience replay and a target network. We use epsilon-greedy policy with  $\epsilon = 0.1$  for exploration. The permanent and transient network architectures are identical to that of DQN but with half the number of parameters in each layer. This ensures that the total number of parameters for the baselines and our method is same. Transient network is also trained using Adam optimizer, whereas, permanent network is updated every 10k timesteps ( $k = 10,000$ ) using SGD optimizer. Target network's weights are updated every 200 timesteps. Experience replay buffer capacity is capped to 8000 as higher capacity buffer stores old training samples which affects the training when the environment is changing continuously. We use batch size of 64 and update the network every timestep. For our approach, we experimented with three choices of  $\lambda$  values (0.55, 0.75, and 0.95) and all the choices resulted in a better performance than the DQN agent as seen in Fig. C.4b. We use the same exploration policy as the DQN agent. We flatten the observation into one long array, then stack the last four observation arrays to form an input to the neural network. The best learning rate for all algorithms are chosen by comparing the total reward obtained in 1.5 million timesteps for a set of fixed learning rate values on 3 different seeds. We search for the best learning rate from a set of value shown in Table C.12 on 3 seeds and we pick the one that gave the highest rewards. We then run on 30 seeds using that learning rate to report the final performance. The random agent picks one of the four actions according to uniform probability at each timestep. For the multi-head DQN baseline, we attach two output heads - one for each task. The baseline knows the task IDs, so it chooses the appropriate head to select actions and to train the network. The replay buffer is reset for this baseline whenever the task is changed. We use a buffer capacity of  $250\mathrm{k}$  for the DQN baseline that is augmented with a large buffer. The rest of the training and architecture details are same as the DQN agent.
C.1.6 MinAtar
This domain is a standard benchmark for testing single task RL algorithms [51]. We use breakout, freeway, and space invaders environments to setup a continual RL problem by randomly picking one of these tasks every 500k steps for a total of 3.5M steps. We standardize the observation space across tasks by padding 0s when needed and make all the actions available for the agent. This problem has non-stationarity in transitions besides the reward and observation signals making it challenging. We use a discount factor of 0.99 for this experiment.
We used a CNN which has 1 relu activated conv layer with 16 filters followed by a relu activated linear layer with 256 units, and a linear output layer for DQN. The architecture is same as the one used in the original paper [51]. The architecture is same for other DQN variants but we add 3 output layers for the multi-headed DQN. For our method, we use the same architecture but 8 filter in the conv layer and 128 units in the linear layer. Target network is used for all the methods which is updated every 1000 steps and we use an experience replay of  $100\mathrm{k}$  for all methods but we use  $500\mathrm{k}$  sized experience replay for large replay based DQN baseline. We use Adam optimizer to train the networks in online fashion. For our method, permanent value function network is updated using SGD
optimizer every 50k steps. The best hyperparameters are chosen by running the experiment for 1.5M steps by switching the tasks every 500k steps on 3 seeds. The results of hyperparameter tuning are detailed in Tables C.17, C.18, C.19, and C.20. We make all 7 actions available to the agent at all times and the input to the agent is  $10 \times 10 \times 7$  images.
C.2 What is the effect of task distribution on the performance for various methods?
Figure C.6: Expected analytic MSVE on a task for various algorithms and  $\epsilon$  values.
Setup: We set up a continual learning problem by creating 7 tasks whose value functions are  $\epsilon$  (task diameter) distance from one another in norm-2. Each task has 7 states. For a particular  $\epsilon$  value, for a particular number of task switches, we run each algorithm 100 times and save the final estimates which form the initial point for the new task. We then sample a task according to uniform distribution and use analytical MSE expression derived in the previous section to analyze the behaviour of PT-TD learning and the two versions of TD learning algorithm for 100 rounds. We then average the error across rounds and across different starting values to get a point in the plot shown in Fig. C.6.
Task details: We use the following  $\mathcal{P}_{\pi}$  for all tasks and for all task diameters  $(\epsilon)$ :
$$
\mathcal {P} _ {\pi} = \left[ \begin{array}{l l l l l l l} 0. 1 8 & 0. 1 9 & 0. 0 8 & 0. 1 6 & 0. 1 & 0. 1 1 & 0. 1 8 \\ 0. 0 4 & 0. 0 5 & 0. 0 1 & 0. 0 & 0. 4 & 0. 0 & 0. 5 1 \\ 0. 0 & 0. 2 & 0. 0 & 0. 0 5 & 0. 2 8 & 0. 0 6 & 0. 4 1 \\ 0. 1 7 & 0. 1 4 & 0. 0 9 & 0. 2 6 & 0. 1 5 & 0. 1 9 & 0. 0 1 \\ 0. 1 6 & 0. 1 6 & 0. 1 6 & 0. 2 4 & 0. 1 7 & 0. 0 & 0. 1 1 \\ 0. 2 5 & 0. 0 2 & 0. 2 4 & 0. 2 4 & 0. 0 8 & 0. 0 5 & 0. 1 1 \\ 0. 0 & 0. 2 5 & 0. 1 9 & 0. 4 4 & 0. 0 5 & 0. 0 & 0. 0 7 \end{array} \right]
$$
We then sample 7 points in  $\mathbb{R}^n$  that are  $\epsilon$  units from one another. These points form the true value function of the tasks. We then compute the reward vector for each task  $\tau$  as  $r_{\pi}(\tau) = (I - \gamma \mathcal{P}_{\pi})v_{\pi}(\tau)$ . At the time of testing the algorithms, we add a zero mean Gaussian noise with 0.1 standard deviation to these rewards to simulate stochasticity. We search from the learning rates shown in Table C.4 to pick the best one for each combination of  $\epsilon$  and number of task switches. We use a learning rate of 0.01 while using analytical expressions.
**Observation:** We observe both TD learning algorithm and our approach has the same performance after seeing data from one task. But the error gap increases as the number of task switches increase. The low error in our method can be attributed to the presence of permanent value function which captures some portion of the value that is common for all tasks. transient value function then learns the remaining part with a small number of samples resulting in low error overall. We also observe that the error gap between TD learning and our method gets wider as we increase task diameter, which demonstrates that our method is broadly applicable. The reset variant of the TD learning algorithm whose predictions are reset whenever task changes has the largest error emphasizing learning from scratch could be costly.
Algorithm	LRs
TD (without reset)	{8e-1 5e-1 3e-1 1e-1 5e-2 1e-2}
PT-TD	PV: {5e-1 3e-1 1e-1 5e-2 1e-2 5e-3}
TV: {8e-1 5e-1 3e-1 1e-1 5e-2 1e-2}
Table C.4: LRs considered to pick the best one.
C.3 What is the effect of network capacity on the performance?
Setup: We use JBW task in the continual RL setting to perform the ablation study. We run two additional experiments by varying the number of parameters in DNN for the DQN and our method. In the first experiment, we use the same DNN architecture for all algorithms, therefore, our method has twice the number of parameters as that of DQN. In the second experiment, we double the network capacity for the DQN variant making the total number of parameters same as ours. We keep the rest of the experimental setup same as before and we pick the best learning rate based on the AUC by searching over the values detailed in Table C.5 over 3 seeds for 1.5M steps.
Algorithm	LRs
DQN (2x)	{1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7}
PT-DQN	PV: {5e-1 3e-1 1e-1 5e-2 1e-2 5e-3}
TV: {8e-1 5e-1 3e-1 1e-1 5e-2 1e-2}
Table C.5: Search space of learning rates.
(a) Results on the JellyBeanWorld.
(b) Items collected in each task.
Figure C.7: (a) Rewards accumulated per timestep across 10k step window is plotted against timesteps. (b) Number of items collected within a task for each type (subplots use different scales along y-axis).
(a) Results on the JellyBeanWorld.
Figure C.8: (a) Rewards accumulated per timestep across 10k step window is plotted against timesteps. (b) Number of items collected within a task for each type (subplots use different scales along y-axis).
**Observations:** The results for ablation experiments are shown in Figures C.7 and C.8. We observe that our algorithm continues to perform better than the DQN agent in both the cases. The results also indicate that it is efficient to devote the available capacity in learning parts of the value function rather than learning the whole.
In the second ablation experiment, the DQN agent with twice the number of parameters starts off learning slowly but catches up with our method after seeing enough data. In fact, the DQN agent continues to perform similar to our method without any drop in performance as we continue to train further (we tested it by running the agent to 6M timesteps). This result highlights that our approach is beneficial in the big world - small agent setup where the computation budget of the agent is very small compared to the complexity of the environment. When the agent's capacity is large relative to the complexity of the environment, there's no additional benefit (neither there is any downside) to our method. Since the world is much more complicated in comparison to the agent's capacity, using our method is preferable.
These experiments also provide partial answer to the question - when does the plasticity problem arise in neural networks? We think that a combination of learning algorithm (like DQN), small agent
capacity, and a complex environment could give rise to the plasticity problem in neural networks. A thorough investigation is necessary to confirm this hypothesis.
C.4 What is the effect of hyperparameters on the performance?
In this section we analyze the sensitivity of additional hyperparameter in our method (learning rate to update permanent value function) and compare it with the sensitivity of TD and Q-learning algorithms on various environments.
Algorithm	LRs
TD (without reset)	{8e-1, 5e-1, 3e-1, 1e-1, 5e-2, 1e-2}
TD (with reset)	{8e-1, 5e-1, 3e-1, 1e-1, 5e-2, 1e-2}
PT-TD	PV: {1e-1, 5e-2, 1e-2, 5e-3, 1e-3}
TV: {8e-1, 5e-1, 3e-1, 1e-1, 5e-2, 1e-2}
Table C.6: LRs considered in the search space for the policy evaluation grid tasks.
Figure C.9: Mean AUC of RMSVE against learning rate for various algorithms.
Figure C.10: Mean AUC of RMSVE against learning rate for various algorithms.
For tabular and linear grid tasks, we select best hyperparameters (learning rates) for each algorithm by choosing those values which results in the lowest area under the error curves as shown in Table C.6, Figures C.9, C.10, C.11. The hyperparameter search and the final reporting is performed by averaging the results on 30 different seeds. For the deep RL task with discrete grid, we choose the best learning rate from the values shown in Table C.7 and the corresponding U curves are shown in C.12. For the minigrid policy evaluation task, we search for the best learning rates from the values shown in Table C.8 and the corresponding U curves are shown in Fig. C.13.
For the tabular control experiment, the learning rates used to search for the best value is listed in Table C.9 and the mean returns for various choices of these values are plotted in Fig. C.14 for various algorithms. Similar table for the minigrid task (deep control) is shown in Table C.10 and the mean returns for various choices of these values is shown in Fig. C.15. We use 30 seeds to pick the best learning rate for the tabular task but use 3 seeds to select the learning rates for the minigrid task.