SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pitfall  $r_t = d + c_1 \cdot \max(0, x_t - x_{t-1})$ , where  $d = -1.0$  is the penalty for falling into a pit,  $x$  represents the x-axis coordinate of the agent,  $c_1 = 0.1$  scales the reward for traversing the corridor, and  $t$  is the time step.
Run and Gun  $r_t = c_1 \cdot k_t + c_2 \cdot \left\| l_t - l_{t-5} \right\|_2^2$ , where  $k$  is the number of enemies eliminated,  $l$  represents the coordinates of the agent's location,  $c_1$  is the reward for eliminating an enemy,  $c_2 = 0.001$  scales the reward granted for movement, and  $t$  is the time step.
Floor is Lava  $r_t = c_1 \cdot \mathbb{1}\{h_t = h_{t-1} \wedge h_{t-1} < h_{t-2}\} + c_2 \cdot \mathbb{1}\{h_t < h_{t-1}\}$ , where  $h$  is the agent's health,  $c_1 = 3$  is the reward for stepping on a platform from lava,  $c_2 = -0.1$  is the penalty for standing on lava, and  $t$  is the time step.
Arms Dealer  $r_t = P + c_1 \cdot w + c_2 \cdot d + c_3 \cdot \left\| l_t - l_{t-5} \right\|_2^2$ , where  $P = -0.01$  is a constant penalty to avoid passivity,  $w$  is the number of weapons acquired,  $d$  is the number of weapons delivered,  $c_1 = 15$  is the reward of acquiring a weapon,  $c_2 = 30$  is the reward for delivering a weapon,  $c_3 = 0.001$  scales the reward granted for movement, and  $t$  is the time step.
Hide and Seek  $r_t = F + c_1 \cdot \mathbb{1}\{h_t > h_{t-1}\} + c_2 \cdot \mathbb{1}\{h_t < h_{t-1}\}$ , where  $F = 0.01$  is a constant reward granted for surviving an iteration,  $h$  is the remaining health (increases when a health kit is picked up and decreases when damage is inflicted by an enemy),  $c_1 = 5$  is the reward granted for picking up a health kit,  $c_2 = -5$  is the penalty for receiving damage from an enemy, and  $t$  is the time step. It is relevant to note that picking up a health kit grants more health in comparison to the damage inflicted by the enemy if it were to occur in the same time step, meaning that the agent receives a positive reward.
In this section, we will discuss our experimental framework in detail.
C.1 Network Architecture
We use the same convolutional encoder head as in [44] with relu activations, followed by two fully connected layers with 256 neurons each. A Layer Normalization is applied after the first dense layer, with a tanh activation, known to work well with Layer Normalization [4]. The second dense layer is followed by a leaky ReLU activation (with  $\alpha = 0.2$ ). Both the actor and critic networks utilize the presented architecture. The actor network takes an observation as input and generates a Gaussian distribution of action probabilities corresponding to the action space. Each task has its own actor head responsible for producing these outputs. The critic network takes the same input and produces a scalar value for each task head. The network is equipped with multiple heads, each of which is responsible for outputting the action distribution for one of the tasks in the sequence. Their number of output heads is thus equal to the number of tasks in the corresponding sequence. In Appendix ??, we showcase experiments where only a single head is employed, and the one-hot encoded task id input is provided.
During preprocessing the observation from the environment is downscaled to an  $84 \times 84$  pixel RGB image, and 4 of such most recent frames are stacked before fed to the network. At the start of each task, the collected experience is emptied from the replay buffer and the agent uses random policy exploration for 10000 steps to collect experience. The entropy regularization coefficient  $\alpha$  is automatically tuned such that the standard deviation of the action distribution on every dimension matches a predefined value  $\alpha_{tgt}$ . When switching to a new task, we prioritize minimizing forgetting in the actor while allowing the critic to adapt freely. By default, we thus only regularize the actor, except for AGEM, where we found it slightly beneficial to also include the regularizing of the critic network. We use a learning rate decay, which is reset upon the start of a new task.
The agent undergoes sequential training on each task  $\mathcal{T}_i$  for a duration of  $\Delta = 200\mathrm{K}$  iterations each. After every 1000 iterations, we assess the policy's performance on each task within the sequence, conducting 10 episodes for evaluation. A buffer is maintained to store the  $50\mathrm{K}$  most recent trajectories, which are subsequently cleared from the buffer at the conclusion of a task. When calculating forgetting, we consider  $k = 10$  final performance measurements of a task.
All experiments are conducted on a computer with a 20-core i9-10900K CPU and 2 GeForce RTX 3090 GPUs.
C.3 Hyperparameters
We adopt most of the hyperparameters from the original SAC algorithm [20] and Continual World [70]. The extensive list is brought in Table 4.
D Task Recollection
The individual tasks in our benchmark may vary in overall complexity and exhibit different potential for transfer between each other. In this section, we therefore examine the difficulty of maintaining performance on tasks within the Cross-Objective sequence. Our analysis in Table 5 reveals that the task Pitfall, closely followed by Hide and Seek, exhibit the highest levels of forgetting across
Table 4: Overview of the hyperparameters used in our experiments.
Hyperparameter	Value
Shared by all algorithms
optimizer	Adam
initial learning rate	10-3
learning rate decay	Linear
learning rate decay rate	0.1
learning rate decay steps per task	105
Adam β1, β2	0.9, 0.999
target action distribution std αtgt	0.089
hidden layer sizes	256, 256
activation convolution	relu
activation dense	tanh, lrelu
discount (γ)	0.99
polyak averaging interpolation factor (ρ)	0.995
frame stack	4
frame skip	4
batch size (n)	128
replay buffer size	50K
initial collect steps (binit)	104
steps per task	200K
update frequency (steps)	5000
gradient steps per update	50
number of evaluation episodes	10
regularization number of batches	10
regularization batch size	32
PackNet
retrain steps	104
gradient clipping norm	2·10-5
MAS
regularization coefficient	104
L2
regularization coefficient	105
EWC
regularization coefficient	250
AGEM
episodic memory per task	104
episodic batch size	128
VCL
regularization coefficient	1
ClonEx-SAC
exploration kind	best return
regularization coefficient	100
episodic memory per task	104
episodic batch size	128
Scenario	PackNet	MAS	AGEM	L2	EWC	Fine-tuning	VCL	ClonEx-SAC	Average
Pitfall	0.18 ± 0.01	0.60 ± 0.01	0.81 ± 0.06	0.04 ± 0.03	0.10 ± 0.01	0.75 ± 0.09	0.55 ± 0.04	0.00 ± 0.02	0.38 ± 0.31
Arms Dealer	0.00 ± 0.00	0.50 ± 0.14	0.63 ± 0.34	0.01 ± 0.01	0.08 ± 0.06	0.47 ± 0.08	0.63 ± 0.19	0.01 ± 0.01	0.29 ± 0.27
Hide and Seek	0.00 ± 0.01	0.25 ± 0.10	0.80 ± 0.07	-0.04 ± 0.07	0.14 ± 0.03	0.84 ± 0.05	0.76 ± 0.01	0.06 ± 0.04	0.35 ± 0.36
Floor is Lava	-0.18 ± 0.07	-0.03 ± 0.01	0.85 ± 0.04	0.01 ± 0.03	-0.04 ± 0.00	-0.00 ± 0.03	0.64 ± 0.21	-0.01 ± 0.02	0.15 ± 0.35
Chainsaw	-0.01 ± 0.00	-0.02 ± 0.02	0.93 ± 0.01	-0.08 ± 0.01	-0.02 ± 0.01	0.62 ± 0.08	0.61 ± 0.15	-0.07 ± 0.06	0.25 ± 0.38
Raise the Roof	-0.06 ± 0.10	-0.07 ± 0.02	0.97 ± 0.02	-0.13 ± 0.01	-0.01 ± 0.02	0.46 ± 0.01	0.89 ± 0.00	-0.02 ± 0.03	0.25 ± 0.43
Run and Gun	-0.01 ± 0.01	-0.02 ± 0.01	0.93 ± 0.03	-0.08 ± 0.02	-0.08 ± 0.02	0.20 ± 0.15	0.36 ± 0.08	0.00 ± 0.00	0.16 ± 0.32
Table 5: Forgetting of all methods on each individual task of the CO8 sequence, calculated as outlined in Equation 2. The results of tasks that were most forgotten by a given method are highlighted in bold. The final task of the sequence is omitted from the comparison, as no forgetting can have occurred.
all methods. However, it is important to note that this comparison may not accurately reflect the inherent difficulty of each task in terms of forgetting.
The ordering of tasks within the sequence can introduce a disparity, as the policy undergoes a greater number of weight updates based on experience gathered from later tasks before completing the entire sequence. This dynamic impacts the difficulty of remembering tasks at the beginning of the sequence. On the other hand, it increases the potential for backward transfer from future tasks, which could potentially enhance performance. However, we did not observe notable instances of backward transfer in our experiments. Consequently, we posit that tasks at the beginning may be slightly disadvantaged in the comparison due to forgetting.
Exploring different orders of sequences would be an interesting avenue for future research. By varying the order, we could better determine the complexity of different scenarios and their impact on performance. This approach would provide more nuanced insights into the difficulty of each task within the sequence. We have provided the functionality in COOM to randomize the tasks in a sequence. However, we have not conducted any experiments in this setting, as a proper evaluation would require numerous trials, which we lack the computational resources for.
Overall, our current analysis highlights the challenges in maintaining performance on the Pitfall and Hide and Seek tasks. However, further investigation is necessary to fully understand the complexities involved and to identify the most challenging scenarios within the Cross-Objective sequence.
E Memory and Time Consumption
(a) Elapsed Real Clock-Time
Figure 7: Time and Memory Consumption on the CD4 task sequence
(b) Memory Consumption
Our experiments indicate that replay-based methods consume more resources in terms of both time and memory compared to other baselines. Figure 7a depicts the elapsed real clock-time in hours. Most methods indicate an average runtime of 16-17 hours, whereas AGEM and Perfect Memory take around 20 and 25 hours to complete respectively. Figure 7b depicts memory consumption in gigabytes. Most methods exhibit an equal memory consumption of approximately 35GB, as the replay has a fixed size of 50K trajectories. AGEM uses slightly more memory due to storing selective episodic memory from previous tasks. Lastly, Perfect Memory clearly stands out in the comparison
with a consumption of over 500GB. This analysis can be valuable for when resources are a critical factor when selecting an appropriate CL method.
F Cross-Objective Challenge
The Cross-Objective Challenge (COC) is designed to be a challenge for future methods. In this section we will describe how the task sequence is constructed, and further expand how our baseline methods performed on the sequence. The following list outlines how each scenario is modified to obtain a harder version.
1. Pitfall
- The chance of a tile being a pit is raised from 0.4 to 0.7.
- The actor property APROP_Speed of the agent is decreased from 1.0 to 0.7.
2. Arms Dealer
- The size of the map is increased by  $50\%$ .
Instead of 5 weapons, only 1 is spawned at a time.
- Instead of a single weapon type (Chaingun), one of 7 weapons (BFG9000, Chaingun, Pistol, PlasmaRifle, Rocket Launcher, Shotgun, SuperShotgun) is randomly spawned.
3. Hide and Seek
- Number of initially spawned monsters is increased from 10 to 20.
- Number of initially spawned health kits is decreased from 20 to 10.
- The actor property APROP_Speed of the agent is decreased from 1.0 to 0.7.
4. Floor is Lava
- The change of a tile being a platform is decreased from 0.1 to 0.02.
- The change interval of the platforms is decreased from 150 to 100 game tics.
5. Chainsaw
- Number of initially spawned monsters is decreased from 20 to 5.
- The spawn interval of a new monster is decreased from 90 to 30 game tics.
- Monster health is increased from 1 to 50.
6. Raise the Roof
- The matching texture of the wall sections with switches (SW1BLUE1) is changed to (SW1GRAY1), so that it no longer differs from the rest of the wall.
7. Run and Gun
- Number of initially spawned monsters is decreased from 20 to 5.
- The spawn interval of a new monster is decreased from 90 to 30 game tics.
- Monster health is increased from 1 to 25.
8. Health Gathering
- The map layout from health_gathering is replaced with a more complex maze-like version health_gathering_supreme from ViZDoom [25].
- The spawn interval of a new health kit is increased from 30 to 60 game tics.
We will henceforth compare results between the CO8 and COC sequences. We depict the training curves of each individual task with its corresponding core success metric in Figure 8. We can observe a slight performance increase in each task on COC, but it significantly falls below the CO8 counterparts. Figure 9 indicates that PackNet, the best method in our previous benchmark evaluations, also stands out in COC. The continual evaluation results in Figure 10 further show how all displayed methods underperform on the hard COC.
Figure 8: Performance comparison of PackNet and MAS on individual tasks of CO8 and their complex counterparts in COC. Each graph only depicts the time steps during which the task was trained. COC remains unsolved for MAS and even PackNet, which was the best performing method on other sequences.
Figure 9: Comparison of average success between CO8 and COC.
Figure 10: Continual evaluation results of select methods on CO8 and COC.
G Sequence Length Comparison
Another way of assessing forward transfer would be to compare the average performance on a 4-task sequences with the corresponding  $2^{nd}$  half of its 8-task counterpart. The tasks and their order in this comparison are the same. The only difference here is that the 8-task sequence has had the potential to learn relevant competencies from previous tasks in its  $1^{st}$  half. If the  $2^{nd}$  half of the 8-task sequence has higher performance then this would be an indicator that knowledge from the initial tasks has been transferred forward to the later
ones. We present this comparison in Table 6. Across all methods, we report a  $4.04\%$  performance increase from CO4 to  $\mathrm{CO8^{2nd}}$  and  $12.8\%$  from CD4 to  $\mathrm{CD8^{2nd}}$ . A performance decrease only occurred in 3/12 of cases. We can indeed thus conclude that the  $1^{st}$  half of 8-task sequences fastens learning the  $2^{nd}$  half. We posit that the Cross-Domain setting has higher transfer due to the shared objective across tasks, meaning that learned behaviour can better bolster performance in further tasks.
Table 6: Average performance comparison of 4-task sequences with the  $2^{nd}$  half of the corresponding 8-task sequence.
Method	CD4	CD82nd	CO4	CO82nd
PackNet	0.92 ± 0.04	0.92 ± 0.03	0.87 ± 0.13	0.91 ± 0.04
MAS	0.55 ± 0.33	0.83 ± 0.16	0.72 ± 0.15	0.76 ± 0.23
AGEM	0.35 ± 0.24	0.43 ± 0.18	0.42 ± 0.25	0.35 ± 0.32
L2	0.71 ± 0.25	0.71 ± 0.12	0.80 ± 0.16	0.54 ± 0.14
VCL	0.46 ± 0.26	0.43 ± 0.17	0.40 ± 0.28	0.61 ± 0.26
Fine-tuning	0.59 ± 0.23	0.72 ± 0.18	0.50 ± 0.19	0.69 ± 0.21
Average	0.595	0.673	0.618	0.643
H Network Plasticity Experiments
In our experiments for network plasticity, we halve the number of steps for each task, reducing them to 100K, as we found it to be sufficient for convergence. The entire sequences thus lasts 4M and 8M environment iterations in CO4 and CO8 respectively. After a task is finished, we maintain the network weights and only reset the learning rate decay and replay buffer. Instead of measuring Success, we assess the individual performance metric of each task. We wish to observe whether the initial performance achieved by the agent on a given task deteriorates over successive trials when re-encountering it. We additionally depict the learning curves of the SAC agent on the CO8 task sequence in Figure 11.
Similar to the results on CO4 in the main paper (Figure 4), the attained peak performance remains stable throughout the iterations. However, on CO8, we can witness a discernible dip in performance across most tasks midway through the entire sequence, yet notably, this decline is followed by a resurgence towards the end. Further, on CO4 we can observe that the initial iteration of some tasks is
Figure 11: SAC does not exhibit a loss of network plasticity. Measuring the individual performance metric of each task across 5 seeds displays that repeated exposure to the same tasks throughout 10 iterations of the CO8 sequence does not hinder the capacity to reattain peak performance.
substantially lower than the subsequent runs. This, however, does not appear for the same tasks on CO8, because the agent has already learned to succeed in other tasks and is capable of transferring knowledge forward.
I Method Variations
Table 7 presents the full results of incorporating variations to our baseline methods. Figures 12 and 13 depict the average success curves of selects methods on the CO8 and CO4 task sequences respectively. PackNet stands the strongest in terms of stability across the variations, as not a single variations causes a substantial degradation of performance. Regularization-based methods stand out in our evaluations as the most volatile, as their performance is very likely to completely degrade when employed with variations. The variation that improved the performance of most methods is the Noise augmentation.
Figure 12: Performance comparison on the CO8 sequence of select methods between critic regularization and single output with the original setting.
Table 7: Method variation performance on CO8. Results higher than the default are highlighted in bold.
Method	Default	Conv	Shift	Noise	LSTM	Single Head	Critic Reg
PackNet	0.84 ± 0.07	0.72 ± 0.09	0.87 ± 0.04	0.88 ± 0.03	0.83 ± 0.06	0.68 ± 0.15	0.70 ± 0.12
MAS	0.58 ± 0.17	0.08 ± 0.07	0.48 ± 0.15	0.47 ± 0.15	0.01 ± 0.01	0.29 ± 0.18	0.25 ± 0.12
AGEM	0.30 ± 0.13	0.02 ± 0.00	0.40 ± 0.09	0.33 ± 0.02	0.48 ± 0.09	0.12 ± 0.11	0.28 ± 0.04
L2	0.73 ± 0.10	0.34 ± 0.05	0.75 ± 0.11	0.79 ± 0.08	0.01 ± 0.01	0.28 ± 0.10	0.28 ± 0.10
VCL	0.39 ± 0.15	0.05 ± 0.04	0.30 ± 0.10	0.29 ± 0.13	0.01 ± 0.00	0.31 ± 0.08	0.41 ± 0.06
ClonEx-SAC	0.89 ± 0.07	0.26 ± 0.01	0.89 ± 0.00	0.90 ± 0.06	0.02 ± 0.01	0.79 ± 0.11	0.87 ± 0.04
EWC	0.65 ± 0.13	-	-	-	-	0.32 ± 0.15	0.26 ± 0.17
Fine-tuning	0.44 ± 0.11	-	-	-	-	0.13 ± 0.13	-
Figure 13: Performance comparison on the CO4 sequence of select methods between image augmentation methods and and LSTM encoder with the original setting.
J Repeated Sequences
Figure 14: Average cumulative success off PackNet on CO16.
Figure 15: Continual evaluation of PackNet on each task of CO16.
In line with the approach used in [70], we have repeated our core sequences to generate 16-task sequences for both modalities. More tasks present a greater challenge in retaining previously acquired knowledge. Additionally, it enables a thorough evaluation of how methods leverage their acquired competencies when encountering previously learned tasks anew. Given the substantial computational cost associated with conducting experiments on the 16-task sequences, we have limited our evaluation to PackNet applied to CO16. We present a visual overview of the results in Figures 14 and 15.
K Cross-Domain Sequence
In this section, we provide a detailed description of the modifications implemented to compose the Cross-Domain sequences based on the default environment of the Run and Gun scenario. The specific modification types for each environment are displayed in Table 8.
Table 8: Environment modification types of Run and Gun
Environment	Obstacles	Green	Resized	Monsters	Default	Red	Blue	Shadows
Modification Type	Decorations	Textures	Target & Agent Size	Target Type	-	Textures	Textures	Target Rendering
Decorations: This modification involves adding various in-game elements (e.g., torches, pillars, barrels, lamps), to the game environment at fixed locations. These decorations serve as 1) distractions by having the agent considering them relevant, and 2) obstacles by impeding the agent's movement.
Textures: The textures used for various game assets, such as walls, floors, and ceilings, are altered in this modification. Different textures are applied to create distinct visual appearances for each task, further increasing the visual disparity across tasks.
Target & Agent Size: This modification involves adjusting the width and height of targets and the height of the agent within the game environment. Smaller or narrower targets can be more difficult to detect. Changing the height of the agent affects the view height and thus creates a visual shift of the perceived environment.
Target Type: In this modification, the visual appearance of targets is altered. We use various enemies from the original game and render them immobilized.
Target Rendering: This modification focuses on changing the visual rendering of the targets. For example, targets can be rendered with different visual effects or styles, making them visually more distinct or harder to detect and requiring the agent to adapt its perception and recognition abilities.
To implement these modifications, we utilize the SLADE3 tool, which enables us to customize various aspects of the DOOM game environment. Each modification type contributes to the visual disparity and complexity across tasks, challenging the agent's perception, decision-making, and adaptation capabilities, creating a comprehensive continual learning task sequence. In the following we provide a more comprehensive description of changes.
1. Obstacles
- A total of 25 elements named Tall Blue Torch, Tall Green Torch, and Tall Red Torch with respective in-game id's [44, 45, 46] are constructed at fixed locations of the map, which act as obstacles
- Wall texture A-CAM01
- Floor texture AQF022
- Ceiling texture DOGRMSC
3. Resized
- The actor properties  $APROP\_ScaleX$  and  $APROP\_ScaleY$  of enemies are randomized in the range of [0.3, 3.0]
- The actor property APROP(ViewHeight) of the agent is randomized in the range of [0.0, 70.0]
4. Monsters
- Replaced targets with immobile enemies: Arachnotron, Fatso, Cacodemon, Archville, Revenant
- The number of initially spawned enemies is set to 3 per type
- The actor property APROP_Invulnerable of the agent is set to True
5. Default
- The map layout originates from the ViZDoom [25] scenario health_gathering supreme
- Wall textures ICKWALL3
- Floor texture AQF051
- Ceiling texture FLAT19
- The surface no longer inflicts damage to the agent
- Stationary targets are spawned at random locations around the map
- The target type is CommanderKeen
The health of the target is set to 1
- The number of initially spawned targets is set to 20
- The spawn delay of every subsequent target is set to 30 game ticks
- The agent is equipped with a pistol and 200 bullets
6. Red
- Wall texture FIREWALL
- Floor texture CRACKLE4
- Ceiling texture DORED
7. Blue
- Wall texture FIREBLU2
- Floor texture FLAT14
Ceiling texture CEIL4_3
8. Shadows
- The actor property APROP_DrawStyle of targets is set to STYLE_Shadow
Sequence ordering In our benchmark, we focused on evaluating performance and transferability in the context of a fixed sequence ordering. We acknowledge that varying the order of the sequences could introduce additional dynamics and complexities to the continual learning (CL) process.
Different sequence orderings have the potential to impact the learning process by introducing variations in task difficulty, temporal dependencies, or interference patterns. Exploring multiple sequence orderings could provide valuable insights into the robustness and adaptability of CL methods across different scenarios.
However, due to practical constraints, such as limited computational resources and time, we were unable to thoroughly investigate the effects of sequence ordering in our experiments. Consequently, our benchmark considers only a single predefined ordering of the sequences.
Future research endeavors could delve into the impact of sequence ordering by systematically exploring different orderings and analyzing their effects on CL performance. This would enable a more comprehensive understanding of the challenges and opportunities associated with varying sequence orderings in continual learning scenarios.
Restricted Action Space In order to simplify the training process, enhance ease of learning, and bring more focus to the continual learning paradigm, rather than very complex RL challenges, we opted to restrict the total action space of the DOOM video game. By doing so, we only included actions that we deemed essential for effectively achieving the established objectives.