COOM: A Game Benchmark for Continual Reinforcement Learning
Authors: Tristan Tomilin
Meng Fang
Yudi Zhang
Mykola Pechenizkiy
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: benchmark
vizdoom
vision-based learning
embodied AI
continual learning
reinforcement learning
deep learning
simulation environment
TL;DR: A 3D benchmark for continual reinforcement learning with baseline evaluations of popular methods.

Abstract: The advancement of continual reinforcement learning (RL) has been facing various obstacles, including standardized metrics and evaluation protocols, demanding computational requirements, and a lack of widely accepted standard benchmarks. In response to these challenges, we present COOM ($\textbf{C}$ontinual D$\textbf{OOM}$), a continual RL benchmark tailored for embodied pixel-based RL. COOM presents a meticulously crafted suite of task sequences set within visually distinct 3D environments, serving as a robust evaluation framework to assess crucial aspects of continual RL, such as catastrophic forgetting, knowledge transfer, and sample-efficient learning. Following an in-depth empirical evaluation of popular continual learning (CL) methods, we pinpoint their limitations, provide valuable insight into the benchmark and highlight unique algorithmic challenges. This makes our work the first to benchmark image-based CRL in 3D environments with embodied perception. The primary objective of the COOM benchmark is to offer the research community a valuable and cost-effective challenge. It seeks to deepen our comprehension of the capabilities and limitations of current and forthcoming CL methods in an RL setting. The code and environments are open-sourced and accessible on GitHub.

COOM: A Game Benchmark for Continual Reinforcement Learning
Tristan Tomilin' Meng Fang21
Yudi Zhang'
Mykola Pechenizkiy'
Eindhhvenn University of Technology Uniiversity of Liverpool
{t www.tmmmi........ z.annig m. pechenizkiy) nl
Meng. FanggPangdliverpoo..oo
Abstract
The advancement of continual reinforcement learning (RL) has been facing various obstacles, including standardized metrics and evaluation protocols, demanding computational requirements, and a lack of widely accepted standard benchmarks. In response these challenges. we present COOM (Continual DOOM), continual RL benchmark tailored for embodied pixel-based lously crafted suite of task sequences set within visually distinct 3D environments, serving robust evaluation framework to assess crucial aspects of continual RI., such as catastrophic forgetting. knowledge transfer, and sample-efficies Following an in-depth empirical evaluation of popular continual learning (CL) methods, we pinpoint their limitations, provide valuable insight into the benchmark and highlight unique algorithmic challenges. This makes our work the first to benchmark image-based CRL in 3D environments with embodied perception. The primary objective of the COOM benchmark is to offer the research community a valuable and cost-effective challenge. It seeks to deepen our comprehension the capabilities and limitations of current and forthcoming CL methods in RL setting. The code and environments are open-sourced and accessible on GitHub
based RL. COOM presents a meticu-
learning.
Obstacles
Green
Resized
Monsters
Default
Blue
Shadows
(a) CDS Sequence. The agent is expected to locate the correct targets in the environment and eliminate them. The tasks are based single scenario (Rum and Gun), but the textures and in-game entities vary. The agent needs to effectively adapt its policy adjust the visual distortions.
Pitfall
Arms: Dealer
Hide and Seek
Floor is Lava
Chainsaw
Raise the Roof Run and Gun Health Gathering
(b) CO8 Sequence. In addition to visual disparities between scenarios, the agent is presented with novel diverse goals when exposed to task. These objectives often contrasting, e.g., whereas task requires seeking enemies, another favours avoiding them entirely.
Figure 1: COOM sequences (encompassing tasks ordered from left to right) for task-incremental
learning are composed of diverse environments built in ViZDoom (25). The agent perceives its surroundings from embodied perspective. The 4-task sequences (CD4 and CO4) are constituted by environments in the second half, whereas CD16 and CO16 repeat the original sequence.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
Deep reinforcement learning (RL) has made immense leaps forward over the past decade in the domain
of video games [68, 60, 29. robotic manipulation [28 30 72
50,
76). embodied AI [65] 12
63), and
22 5 63 However, this success has primarily resulted from fine-tuning agents solve specific tasks. As RL is increasingly applied to solving real-life problems in industry, healthcare, or robotics, situations arise where the environment and conditions are subject rapid change. Whereas humans are able to learn perform new similar tasks seamlessly, this competence is still predominantly absent in RL agents, who tend to exceedingly overfit to new tasks and forget all previously acquired competencies. A central goal of RL is thus to build systems that can master spectrum of skills in environments as noisy and diverse as the real world, while being capable of continuing to learn.
foremost on a variety of platforms intended for RL research [44
Continual learning (CL) is the ability to swiftly learn consecutive tasks while maintaining adequate performance on previously mastered problems, where they ought reemerge [49]. The CL paradigm emerges in RL when an agent proceeds to interact with an altered environment in the attempt to refine its policy 54 Whereas task boundaries in CL. might be gradual and smooth, we propose to consider learning a sequence of tasks with sudden transitions. Evaluating a CL agent usually involves several criteria: catastrophic forgetting, forward transfer and backward transfer (59]. CL research is particularly vital as it endorses the AI to thrive in encountered scenarios and conditions, which is much desired for AGI, compared to just solving individual problems.
Many prominent CL platforms are predominantly meant for supervised learning and require non- trivial adaptation for RL [47 38 [6]. Video games act ready- made simulation environments, which is why RL research platforms and benchmarks have widely been based on repurposed game engines 7 6 19 There are, however, relatively few visual-based environments for CRL compared to regular RI. Researchers must instead use ad-hoc adaptations of RL. benchmarks to achieve the desired setting. Proper virtual 3D simulation platforms and benchmarks are required to facilitate image-bun image-based CRL research. They need to be 1) lightweight, 2) well documented. 3) easy to install and use, 4) equipped with standardized metrics and baseline evaluations, and 5) computationally viable to run on small-scale systems and university-level budgets.
To this end, present COOM (Continual DOOM), a CRL benchmark for embodied basel learning on sequences of tasks with visually distinet 3D environments, designed to assess average performance, forgetting, and forward transfer. The visual modifications manifest in 1) wall and surface textures, 2) types, shapes, and sizes of in-game entities, and 3) modes to render objects. Compared to existing benchmarks for pixel-based learning oriented towards CL., COOM additionally includes a diverse set of objectives. To the best of our knowledge, this is the first benchmark specifically targeted towards CRL in complex 3D environments with differing objectives and visuals.
The contributions of our work are three-fold: 1) We assemble task sequences of different lengths, half with task dask-dependent objectives, an aspect often absent in previous works. We further include a very complex sequence, intended as a challenge to the RL community to be solved with more advanced future algorithms. We include demo of trained agent performing COOM tasks. 2) We design novel ViZDoom scenarios of two difficulties with contrasting visuals and dynamics to compose and publicty release COOM, Continual RL benchmark. 3) Following the CL evaluation principles from (70), we employ multiple well-known CL methods for baseline evaluations on our task sequences, assessing prominent CL. criteria. This makes our work the first to benchmark mase-haasdd CRL in 3D environments. We demonstrate how several methods fail to achieve the CL desiderata and how none come close to solving the complex sequence. We further elaborate on how the action distribution provides more comprehensive understanding of the agent's 's continual progression. We conclude that critic regularization and single output head architectures tend to diminish performance in CL.
2 Related work
Most recent eminent CL platforms and frameworks, such as Sequoia [47). Avalanche [38], and Continuum [6] learn from static data sets of fixed sizes (35), and are thus not predominantly intended for RL. In CRL, the data sequence consists of different environments, and data samples are obtained through the agent-environment interaction. Sequoia introduces metrics and baselines aimed at CRL but only consists of simple environments like based manipulation tasks from

---

Table 1: Comparison of existing Continual Reinforcement Learning benchmarks with COOM.
No. Task Sequences
Embodied View
Vision Transfer
Objective Transfer
Unified Metric
Benchmark
3D
Continual World 70
2
K
1.21.emmer 26
0
Jelly Bean World
0
CORA
63]
SIX
4
V x
JIX
VIX
CRLMaze 37
4
/
X
/
COOM
7
/
Morld (73), continuous control tasks from MuJoCo [66), and Monster Kong [62] Avalanche RL [39] introduces a library for CRL, but only includes direct support for basic Atari [43] and
Habitat v1.0 [58] environments, Moreover, it does not present any experimental results bascline methods.
Most prominent modern RL platforms and benchmarks do not offer fixed setting for CL [9 24
1
14 2) The Atari benchmark 77 has often been employed assess popular CL However, the games on the platform are deeply unrelated, lacking the potential for transfer. Deep.Mind Lab [6] provides a number of diverse 3D environments and MineRI. [1 incorporates many of the elements found in lifelong learning [64]. Both have widely been used for assessing CL methods [56] 26 55 64). However, neither include fixed task sequences or standardized CL metrics, requiring users to hand-pick existing tasks design new environments. This causes tedious experimental setup and provides no evident means comparison to other works. DeepMind Lab and ProcGen [3 them computationally expensive to run for CL. Platforms like A12Thor [29] and iGibson (36), based on the Unity3D game engine, good at replicating the complexity and visual fidelity of real world problems. However, they are computationally expensive to simulate, and thus often infeasible for low-budget computational setups.
51
75
methods 27
59
55
procedural content generation (PCG), which renders
Previous CRI benchmarks often exhibit certain deficiencies. CRLMaze 37 based on ViZDoom, presents non-stationary object-picking task, subject to visual changes. However, it lacks com- prehensive baseline evaluation the most popular CL methods, does not undertake to change the objective, and only modifies three attributes (light, textures, objects). Continual World (70) as- sembles robotic manipulation tasks from Meta- World 733 to evaluate CRL and Jelly Bean World uses PCG to generate 2D gridworld test bed. However, neither addresses embodied AI nor image-basee learning. L2Learner [23] creates a 3D PCG world to assess embodied agents, but lacks baseline evaluations and only consists of 5 tasks in single environment. CORA 53 aggregates environments from Atari (7]. ProcCien [3] MiniHack (57] and CHORES 29 into task sequences.
52
The first three environments all 2D, and the latter lacks diverse visuals. We refer readers Table
for the comparison. Numerous metrics have been employed throughout CRL research that attempt measure similar objectives [15 10 48). The abundance and incoherence of metrics can create
confusion in interpreting and disentangling results.
3 Preliminaries
Reinforcement Learning from Images We formulate embodied image-based learning par- tially observable Markov decision process (POMDP) [], which an agent interacts with an en- vironment over fixed horizon of discrete time steps T. A POMDP can be described as a tuple AS,A,, p, R, Q, O, 7). At each timestep f, the environment is in some state St E S. By taking an action 0, E A, the agent causes the environment to transition to another state st+1 with proba- bility p = (s 0z). The agent cannot observe the full state of the environment, but an observation 0₃
dependent on the action a; taken and new state St+1 reached with probability ( a). Ω is the high-dimensional set of oixel-observations. The agent receives reward A given a state and action. Finally, 7 (0. 1) is the discount factor determining how much immediate rewards are favoured over more distant rewards. The n-step return Ret+n time step defined as the discounted sum of rewards, Retti Ei T'rtti. The value function V (*(s) =El R[Rerls s. is the
F: = R(st; at), which is mapped by the reward function
3

---

Table 2: COOM scenarios. The properties determine 1) how the agent's performance is measured; 2) whether enemies exist; 3) does the agent has a weapon; 4) do items spawn on the ground; 5) number of iterations in episode; 6) which action can the agent execute apart from navigation: 7) what is randomized in the environment.
Scenario
Success Metric
Enemies
Weapon
Items
Max Steps
Execute
Stochasticity
Pitfall
Distance Covered
x
1000)
JUMP
Pitfall tile locations
Arms Dealer
Weapons Delivered
1000)
SPEED
Weapon and delivery locations
Hide and Seck
Frames Alive
V
2500
SPEED
Enemy behaviour, item locations
Lava
Frumes Alive
2500
SPEED
Platform locations
Chainsaw
Kill Count
2500
ATTACK
Enemy and agent spawn locations
the
Alive
USE
Agent spown location
Run and Gun
Kill Count
2500
ATTACK
Enemy and agent spown locations
Health Gathering
Frumes Alive
2500
SPEED
Health kit spawn locations
expected return from state when actions are selected accorded to a policy T[als). The action- value function QT a) [s.sss = 8, = a, is the expected return following action from state s. We aim to find a policy T(a()s)) that maximizes the cumulative discounted return E, Ei r'rela ~ (T((ss), st ~ p(-|s,, 0z), 81 p(-))
Continual Learning In the CL setting of this paper, we aim to learn a policy by training an agent sequentially on a fixed task sequence T = (T,...., Tn). The jth task is trained during the interval E [(i - A, i. AJ. where A is the fixed number of iterations per task, during which the agent can only interact with the given environment. The agent thus obtains trajectories Ti, T
from single task to learn from, during which the POMDP to solve becomes (S,, A, Pi,
COOM benchmark
The COOM (Continual DOOM) benchmark is based on ViZDoom 25 a flexible RL. research platform for learning from visual information, based the engine of the classical FPS video game Doom. ViZDoom has the benefit being very lightweight, enabling gameplay up to 7000 FPS single GPU. The benchmark comprised of scenarios built in the Action Code Script (ACS) language. Every scenario is orientated towards accomplishing particular objective, each having one or multiple aspects which make the environment stochastic. In Table 2 display the core properties of each scenario: 1) the metric for measuring agent performance, 2) are there enemies in the environment 3) is the agent equipped with weapon, 4) do items spawn on the ground, 5) how many iterations an episode lasts, 6) which is the execute action, and 7) what is randomized in the environment. A more detailed description of each scenario can be found in Appendix A The limitations of COOM are addressed in Appendix
4.1 Basic Setup
Observations
The agent is limited to only observing portion of its surroundings, having 90 degree horizontal field of view (FoV), which ranks lower than human vision [17] and modern cameras (74. The observation space is stacked frames of 160 120 pixels in a 4:3 resolution with 3 channels of 8-bit values in RGB, which represents the partially observed environment from the first person perspective of the embodied agent.
Action Space To facilitate the training of CL task sequence using single model, we re- quire a unified action space across environments. We thus limit the full actions from the original Doom game to suit the scenarios of our benchmark. We obtain multi-discrete action space A for all environments by finding the Cartesian Product of singular actions A = A1 X A x Ag. MOVE_FORRARD, NO-OP), A
where A₁
STURN_LEFT, TURN_RIGHT NO-OP), and A3 (ATTACK/ /USE/ SPEED, NO-OP}. The agent hence needs to choose one of [A] 12 actions per iteration. Note that the execute operation Ag scenario-dependent as indicated in Table
Rewards
The core component of the reward granted in each scenario directly tied with the success metric (c.g., performance in Run and Gun increases when shooting target, for which the

---

(a) Default environment of Floor Lava (left) and its hard counterpart (right), where only 5% of the surface tiles are platforms instead of the regular 30%. The agent is tasked to stand on the platforms, not the lava. The locations of all platforms are randomized after time interval, which is shorter in the harder version.
(b) Default environment of Arms Dealer (left) and its hard counterpart (right), where only weapon spawned at time instead of the regular The agent is tasked to pick up weapon and deliver it to the blue platform that only appears after pickup and immedi- ately disappears after delivery.
Figure 2: Constructing the hard COC sequence. We modify each CO8 task, leading to more sparse and elusive rewards, impose more complex CL challenge.
agent is also rewarded). We further utilize reward shaping for more granular dense feedback. A comprehensive description of reward functions is presented in Appendix
4.2 Task Sequences
We compose three lengths of sequences: 1) 8 unique tasks T (To, Tr), 2) including only the second half of the former T = (T4,...; T7} to streamline the experimental process, considering the computationally intensive nature of training RL agents, and 3) repeating the sequence T T + T (similar to (70)) allowing for revisiting each task in the same order. By default, the order of tasks in a sequence is predetermined. However, we provide the option for randomizing the sequence order, although this requires a substantially higher number of trials to ensure consistent results. This due 1) the performance of most continual learning methods being heavily dependent on the feature representation learned for the first task, and 2) the transferability of knowledge varying among different pairs of tasks. We distinguish between two sequence modalities.
Cross-Domain Sequence In the cross-domain setting, the agent is sequentially trained on modified versions of the same scenario. We select the Run and Gun scenario as the basis, as it best resembles the original Doom game, requiring the agent to navigate the map and eliminate enemies by firing a weapon. As depicted in Figure Ial the objective and map layout remain the same across tasks, whereas we modify the environment by 1) changing wall, ceiling and floor textures, 2) varying enemy size, shape and type, 3) randomizing the view height of the agent, and 4) adding obstructions which block the agent's movement. The modifications
inspired by the LevDoom generalization benchmark [67]. We henceforth refer to the cross-domain sequences as CD4, CD8 and CD16.
Cross-Ohjeettiv Sequence In addition to changing the visuals and dynamics within a scenario, we now design and employ new scenarios with novel objectives for consecutive tasks, as illustrated in Figure Tb This presents a more diverse challenge, as the goal might drastically change from locating and eliminating enemies (Run and Gun and Chainsaw) to running away and hiding from them (Hide and Seek). This introduces a concept in CL commonly known as Task Interference. Similarly, the scenario Floor is Lava requires the agent to stay at a bounded location for good performance, whereas scenarios Pitfall, Arms Dealer, Raise the Roof, and Health Gathering encourage constant movement. We equivalently refer to the sequences as CO4, CO8 and C016.
Cross- Pross-Obeeetive Challenge There are two main dogmas in the RL community that dictate how to make progress in solving real-world decision-making problems 42). The first is usually referred to as the generalizable agent view, in which focus should be attended towards the large-scale training of agents that solve diverse problems, hoping that along the way, a generalist agent will evolve. Our previously defined two task sequences endorse this philosophy. The second view, generally referred to as deployable RL, takes a more pragmatic view by seeking to design RL algorithms that solve particular difficult problems. To facilitate the advancement of the latter principle, we include task sequence our benchmark which serves as a challenge, aimed foremost to initially be solved. We posit that harder tasks are easier to forget in CL setting. To this end, we create hard counterpart
5

---

Table Results of Average Performance (AP), Forgetting (F) and Forward Transfer (FT) across 10 seeds. Extended results with 95% confidence intervals are presented in Tables 19 of Appendix The result of the best performing method is highlighted in bold.
CD4
CO8
Average
Method
AP
FT
AP
F
AP
FT
AP
F
FT
AP
F
FT
AP
F
FT
PackNet
0.92
0.00
0.40
0.87
0.01
-0.24
0.91
-0.01
0.19
0.82
-0.01
0.25
0.20
0.04 0.08
0.74
0.01 0.13
0.55
0.11
0.24
-0.04
0.82
0.14
0.25
0.04
0.02
0.23
AGEM
0.35 0.80 0.10
1142
0.80
0.03
0.30
0.86
0.17
0.30
0.84 0.23
0.02 0.11 0.02
0.28 0.68 0.11
L2
0.71
-0.28
0.00
0.87
0.07
0.71 -0.04 -0.32
0.10 0.00 0.00
-0.02 -0.23
EWC
0.69 0.00
-0.41
0.65
0.05 -0.77
0.76 -0.04 -0.55
0.65 0.05 -0.38
0.07 0,00 -0.01
0.56 0.00 -0.43
VCL
0.46 0.77
0.21
0.40
0.82
-0.57
0.36 0.73 0.04
0.39
0.64
0,20
0.03 0.16 0.05
0,33 0.62 -0.01
Fine tuning
0.59 0.63
0.32
0.50
0.71 00.01
0.45
0.75
0.28
0.44
0.48 0.23
0.02 0.10 0.02
0.40
0.53 0.17
ClonEx-SAr SAC
0.87 0.00 0.11
0.86
-0.03
-0,26
0.92
0.00
0.13
0.89
0.00 0.27
0.13 0.01 0.03
0.73 -0.01 0.06
Perfect Memory
for each task in CO8, illustrated in Figure 2 is further elaborated on in Appendix
We will further refer to this sequence COC, which
4.3 Evaluation Protocol
The agent is sequentially trained each task T; for A 200K iterations. After every 1000 iterations, we evaluate the policy on each task of the sequence for 10 episodes. 50K most recent trajectories are stored in the replay buffer, which is emptied at the end of each task by default. The Cross-Domain sequences are based on the Run and Gun scenario, which Kill Count the core measure of performance. In the Cross-Objectins goals across tasks. We are hence unable to directly use a single metric to adequately compare performance across tasks. As it would be tedious and convoluted compare the results between methods on each task from a Cross-Objective sequence individually, we define a unified metric success (score)
sequences, however, the agent is expected to achieve different
[0, 1J. where score represents the task metric (see Table 2 and fa is function for task i that transforms the score to success. The lower bound determined by the performance of random agent, and the upper bound f. (score) 1 by SAC trained until convergence.
f ((coor)) = 0 on Ti is
Average Performance P is the core evaluation metric in COOM and measured across the number of total sequence iterations T = [T| averaging the success of all tasks T at each time step t:
p P= + ΣTET E f(scoree).
Forgetting F quantifies the average decrease of success from the end training each task i to the end of the entire sequence T. For consistency, consider is last iterations of task:
F
E.T 1 E ((()cor((41))--)) + ΣØ= fi(coree--))
(2)
Note that we exclude the final task, as no forgetting can have occurred. In the case of negative values for forgetting (F < 0) we observe positive backward transfer.
Forward Transfer FT is measured by first finding the area under the training curve AUC, on task i for the CL. method and for the SAC baseline AUC, with
AUC; = 1 S.L Ji(scores )dl AUC, = 1 1 (score.)d
We can then first find the average normalized area between the curves across all tasks. Note that we do not consider the first task, as there is no knowledge to transfer forward from previous tasks:
FT
1 |T|
AUC;-AUCE 1-AUCh
5 Experiments
5.1 Setup
In this section, we assess the CL desiderata of baselines from three different families of popular CL methods on our task sequences. Regularization-basee order to maintain knowledge from previous tasks. 1.2 (27 adds a simple T22 penalty to achieve this. MAS (3] utilizes weighted penalty each parameter depending its importance to the network
methods constrain weight updates in

---

Pitfall
Dealer
Floor
Chainsaw
Raise
Gathering
1.0
0.8
MAS
Fine-tuning
Average Success
AGEM
VCL
0.6
L2
ClonEx-SAC
0.4
0.2
0.0
1e6
0.6
1.4
1.00
Pittall
Arms Dealer
Hide and Seek
ilooris Lava
Chainsaw
Raise the Roof
Run and Gun
Health Gathering
Success
0.75
0.50
0.25
0.00
1e6
0.6
1.0
1.4
Current Task Task Success
1.00
Pitfall
Armss Dealer
Hide and Seek
Floor is Lave
Chainsaw
Roise the Raof
Run and Gun
Health Gathering
M
0.75
M
mm
ClonEx-SAC
0.50
SAC
0.75
0.00
0.0
0.2 0.4
0.5
0.8
1.0 1.2
1.4
1.6
Figure 3: Visual analysis of CO8 results. 1) The average cumulative performance curves indicate a high disparity across baselines (top). 2) ClonEx-SAC effectively mitigates forgetting according to the continual evaluation curves of individual tasks (middle). The tasks are very diverse, as no notable performance is reached before exposure to the task itself. 3) ClonEx-SAC successfully transfers knowledge forward compared to the vanilla SAC, which trained on each task from scratch (bottom). The blue area between the curves indicates positive forward transfer and red represents its negative counterpart.
output. EWC [27] uses the Fisher information matrix to approximate the importance of each weight. VCL 45 uses variational inference to minimize the KI. divergence between the prior and posterior distribution of parameters. Structure-based methods preserve and update the network architecture to efficiently learn and adapt new tasks over time. PackNet [41] forbids any changes to parameters that are important for previous tasks by freezing the most relevant weights at the end of each task and pruning the rest. Rehearsal-based methods retain samples from previous tasks to constrain forgetting. Perfect Memory (70] refrains from resetting the buffer after a task and thus remembers interfere with previous tasks. ClonEx-SAC [71] retains some samples from previous tasks and performs behavior cloning based on them to reduce forgetting. We also include the naive Fine-tuning baseline, which simply continues regular weight updates on a new task. All the CL methods take the efficient and well-known Soft Actor-Critic (SAC) 20 as an RI training backbone. We follow the evaluation protocol outlined in Section 4.3 to measure performance of all baselines on CD4, CO4, CD8, CO8 and COC with fixed orders across 10 seeds controlling the pseudorandom nature of the environments. We refrain from using random sequence orders due to the necessity for numerous runs to ensure consistent evaluation, a demand that exceeds our available computational resources. Similarly, we exclude Perfect Memory on 8-task sequences due to the unfeasible memory requirements for storing all trajectories. We grant the agent full access the task identity both at training and test time, a CL setting coined as Incc-mnnreettal Learning [69]. We use 95% confidence intervals for all results. Our framework is discussed in finer detail in Appendix
everything. A-GEM (T] projects gradients from new samples as
5.2 Main Results
Performance PackNet, closely followed by ClonEx-SAC, triumphed across all sequences according to the results in 3 We conjecture PackNet's success on our benchmark to two key factors: 1) knowing the task identity during evaluation, and 2) short task sequences with distinct boundaries, as PackNet needs to assign fixed fraction of parameters to each task. is noteworthy that the top three best-performing methods (PackNet, ClonEx-SAC, and L2) originate from different families of CL,
7

---

Chainsaw
and
Health Gathering
600
40
KIHI Count
Fra.mex Alw
Kill Count
40
400
200
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
x110
X10
X10#
4 of Training steps (100K for Each Tasid
Figure 4: SAC does not exhibit loss of network plasticity on COOM. Measuring the individual performance of each task across 5 seeds displays that repeated exposure to the same tasks throughout 10 iterations of the CO8 sequence does not hinder the capacity to reattain peak performance.
indicating that each approach has strong potential for achieving excellent results in the benchmark. Although Perfect Memory performed well on the 4-task sequences, it required nearly twice as long run, and took up 16x more memory in the process. Memory consumption and walltime is further discussed in Appendix E The average success curves in Figure Bjilusstrae a significant disparity performance among our baselines. Despite an exhaustive exploration of hyperparameters, we were unable to attain satisfactory results with AGEM and VCI.. These methods ranked as the poorest performers in our benchmark, remarkably even falling below the naïve fine- fining approach. Comparing results across sequence lengths, almost all methods showed higher results in CO4 than CO8, whereas among the cross-domain sequences, the top four methods had a substantially higher performance the longer version. Performance related to sequence length is more extensively analysed in Appendix [ The performance difference between CD and CO sequences are not that staggering, indicating that visual perturbations alone, without changing objective in our embodied 3D environments, are sufficiently challenging for our CRL baselines.
Forgetting The top four basclines excel at preserving performance previously learned tasks, approaching near-perfeet perfectiffe.. However, out of these four, L2 and EWC have a substantially lower performance, indicating that regularization-bant based methods predominantly emphasize stability the expense of plasticity. Despite an extensive hyperparameter search for weaker regularization, we were not able to achieve more favorable aability-plasttcitty trade-off. In particular, lower regularization constant exhibited a loss of stability but no noticeable further gain in plasticity. In comparison to good stability, AGEM and VCL experience decline in average performance immediately after losing access to a learned task. In terms of forgetting particular tasks, Pitfall and Arms Dealer appear the hardest to remember given our fixed ordering of tasks in the CO sequence, analyzed in Appendix D
Transfer We attribute the core capacity for transfer in our benchmark environments navigation. Success in most tasks heavily relies on the ability to navigate effectively (e.g., repeatedly running into walls rarely contributes surroundings also holds sigmificant importance for most tasks (c.g.. detecting useful items to collect). The training success curve of ClonEx-SAC in Figure delineates the poor sample-efficiency of the B
high reward). Being able to identify objects and entities from their
vanilla SAC which is unable to attain comparable performance on several tasks by learning them from scratch. Regularization- based methods fall significantly below others in terms of utilizing acquired competencies to effectively learn future tasks. We did not observe notable backward transfer in our experiments.
5.3 Network Plasticity
Recent research has emphasized the importance of network plasticity in RL 40 46 33). To further enhance our analysis and provide a comprehensive perspective on the ability to adapt to new targets on our benchmark. Similar to the experimental design in []], we aim to investigate whether our base SAC agent experiences a decline in network plasticity when confronted with the Cross Objjectveee task setting. Intriguingly, unlike the trend observed in [1. our findings diverge given the learning curves of CO4 in Figure 4. where the agent cyclically repeats the sequence 10 times. The agent able to reach similar results in equal duration throughout the iterations without any noticeable deterioration. We therefore conclude that our base SAC agent does not exhibit a decline in network plasticity when subjected to our benchmark. An avenue for future research would be to investigate whether other
8

---

A/4l
Arso Doaeer beck Must slone
Cuukeon
febo Seccinaa
Nareral Den. Hoail oxlho
1020
600
220
400
75
420
1000
7200
5400
1600
Tieesaps
TURN,
TURNLARYT
TURN WEGH,
TURN LETT,
TURN
ZORWARD
TURN SIGHT, NEVE, FORWWRDD
TIR-LIFFT, WINT, JORWAAT
POLRE_ONRRAANC EXECUTE
TUNN, HRMMI, NUTE, NUTTE POPSHAND, Erecure
TUKCLLOP), MONO, MONE PORPLAHUL ERECUTE
Figure PackNet quickly adapts its policy to fit a new task when training on CO8. The stackplot (left) depicts the distribution of actions executed in the environment, For example, in Pitfall, PackNet highly favours moving forward while accelerating to reap the highest return. The histogram (right) indicates the mean number of action executions across the entire sequence. Actions constituting maximal sub-actions appear to be most optimal according to PackNet.
methods, such as model-based approaches, exhibit a similar pattern. The results of CO8 and further analysis is included in Appendix
5.4 Action Distributions
Since the agent has fixed multi-discrete action space across scenarios, we can visualize how the action distribution shifts over time training progresses to new tasks. We plot the actions executed by PackNet on CO8 in Figure PackNet's high performance suggests it has obtained an effective action distribution. We can observe how PackNet quickly manages to adjust its policy to select appropriate actions when presented with a new task. The action distribution indicates, that on, average executing multiple sub-actions in a single timestep more beneficial. In Appendix M we further visualize how our baselines are able to maintain the acquired action distribution on each given task.
5.5 Method Variations
Image Augmentation Using augmented images input has been recognized for its potential to enhance data diversity, mitigate catastrophic forgetting, and facilitate adaptation to changes in
data probability distribution (1]] We explore the impact of incorporating three established image
augmentation methods for RL to our selected baselines on COOM: 1) Random Convolution [34] randomly perturbs input observations. 2) Random Shift (31 pads each image with 4 pixels (repeating boundary pixels) and then applies random crop, yielding the original image shifted by +4 pixels, 3)
Random Noise
32
injects Gaussian noise into the image.
In our comparative analysis depicted in Figure 6 we can observe that random shifts and noise have minimal impact the performance of our baselines, whereas random convolutions significantly degrade all methods except PackNet. They lead to substantial imbalances in the data distribution for initial tasks, causing the policy converge local minimum from which it struggles to recover. This phenomenon especially pronounced regularization- based methods that rely on maintaining stable feature representations, which highly susceptible to disruption caused by random convolutions due to their rigid architecture and stringent regularization. Additionally, these perturbations affect the data distribution stored in the memory of rehearsal- methods, diminishing their effectiveness for future task learning. Our findings indicate that PackNet can better recover from a degenerate policy stuck in local minima, attributed freeing up redundant parameters after training on task. This suggests that architecture- based methods exhibit greater flexibility in the face of significant changes in data distribution.
LSTM We include LSTM encoder atop the CNN head in our model architecture to assess its impact on our baseline methods. Employing similar approach to a supervised setting has previously been shown to be beneficial [18). On our benchmark, however, this approach completely

---

Performance Increase
Convolution
Shift
LSTM
Single Regularizatiin
100%
50%
0%
TI
-50%
-100%
PackNet
MAS
AGEM
VCL
ClonEx-SAC
Figure 6: Performance comparison of potential method enhancements. Key observations: 1) While most baselines are unable learn a single task using an L.TTM encoder, AGEM reaps a substantial benefit, and PackNet remains indifferent. 2) Critic regularization has detrimental effect on CL, particularly on regularization based methods. 3) Methods performing well on COOM suffer less from only single head output. 4) Attributed to weight-pruning PackNet demonstrates superior recovery from a degenerate policy under an imbalanced data distribution with random convolutions.
deteriorates the performance of most methods, rendering them unable to learn anything meaningful. Strikingly, PackNet's 's performance remained consistent, whereas AGEM experienced a substantial increases. We hypothesize that, by virtue of capturing sequential patterns, LSTT encoder can facilitate more accurate and efficient retrieval relevant experiences from the episodic memory. This improved retrieval could enable AGEM to make better of past knowledge when confronted with new tasks. However, more in-depth analysis is required to confirm this.
Critic Regularization In the context of actor-critic frameworks like SAC, there is design decision regarding whether to regularize the critic. In some CL. methods, the critic is only used during training and only needs to store the value function for the current task, making forgetting less of concern. However, certain methods, such as replay-based approaches, may not easily accommodate framework where only the actor is regularized. We thus investigate critic regularization affects our baselines. Figure confirm the findings from [70], that regularizing the critic network has a harmful impact for CL. This suggests that is better to let the critic adapt freely the current task while prioritizing the minimizing of forgetting in the actor. However, this means that the critic needs to be trained from scratch, which not particularly efficient when tasks are repeated.
Output Heads In our default setup, we use a separate output head for each task, which is known to help prevent forgetting [70]. However, this approach inapplicable to dynamic scenarios with and unknown number of tasks. deuucctss how a single-heas head architecture impacts our baselines. All methods show decline in performance, however ClonEx-SAC, PackNet and VCL manage to better maintain their original performance. Extended results are presented in Appendix[]
6 Conclusion
Training proficient agents, who are able to continually adapt to learning new tasks in an altering environment, currently remains one of the greatest challenges in RL. To aid the community in grappling with this challenge, we presented and openly released COOM, a novel benchmark for assessing CL in visually distinct 3D environments with diverse objectives. COOM comprises seven task sequences of varying length, complexity, and modality, effectively demonstrating the strengths and weaknesses of popular continual learning methods. Our experimental evaluations reveal the impact of different variations to baseline methods, highlighting the importance of critic regularization and task-dependent output architectures. We visually illustrate how high-performing methods quickly adapt their policies and maintain learned action distributions across tasks within sequence. All the methods in our evaluations struggle with the complex COC sequence, making it valuable testing ground for future research, showcasing both learning challenges and transfer difficulties. COOM serves as crucial tool inspire the development of more capable algorithms and we expect it to facilitate the evaluation and comprehension of future continual reinforcement learning methods.
Acknowledgements
This work was conducted with the assistance of the Dutch national e-infrastructure, generously supported by the SURF Cooperative through grant number EINF-5755.
10

---

References
[1] Zaheer Abbas, Rosic Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023.
[2] Joshua Albrecht, Abraham Fetterman, Bryden Fogelman, Ellie Kitanidis, Bartosz Wróblewski, Nicole Seo, Michael Rosenthal, Maksis Knutins, Zachary Polizzi, James B Simon, et al. Avalon: A benchmark for generalization using procedurally generated worlds. arXiv preprint arXiv:2210.13417, 2022.
[3] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference computer vision (ECCV), pages 139-154, 2018.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: Nen::6607.65500 2016.
[5] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from aglti- autocurriclla. arXiv preprint arXiv:1909.0753 2019.
[6] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv: 161iv:161..0880,, 2016.
[7] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[8] Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
[9] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota with large scale deep reinforcement learning. arXiv preprint
2019.
[10] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pages 532-547, 2018.
[11] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint arXiv: /8/2.00420000 2018.
[12] Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. arXiv preprint
2019.
[13] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning. In International conference on machine learning, pages 2048-2056. PMLR, 2020.
[14] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, et al. Procthor: Large-scale embodied ai using procedural generation. arXiv preprint MxXiv:2206.06999 2022.
[15] Natalia Dodz-Rodr, Vincenzo Lomonaco, David Filliat, and Davide Maltoni. Don't t forget, there is more than forgetting: new metrics for continual learning. arXiv preprint arXiv:1810.13166 2018.
[16] Arthur Douillard and Timothée Lesort. Continuum: Simple management of complex continual learning scenarios. arXiv preprint arXiv:2102.06253, 2021.
[17] Kevin Fan, Jochen Huber, Suranga Nanayakkara, and Masahiko Inami. Spidervision: extending the human field of view for augmented awareness. In Proceedings of the 5th augmented human international conference, pages 1-8, 2014.

---

[18] Xin Guo. Yu Tian, Qinghan Xue, Panos Lampropoulos, Steven Eliuk. Kenneth Barner, and Xiaolong Wang. Continual learning long short term memory. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 18 1817-1822, 2020.
[19] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint 190. 2019.
[20] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 1 2018.
[21] Ya-nan Han and Jian-wei Liu. Online continual learning via the meta-learning update with multi-scale knowledge distillation and data augmentation. Engineering Applications of Artificial Intelligence, 13:104966, 2022.
[22] Max Jaderberg. Wojeiech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learn- ing. Science, 364(6443) -858 2019.
[23] Erik Johnson, Eric Q Nguyen, Blake Schreurs, Chigozie S Ewulum, Chace Ashcraft, Neil M Fendley, Megan M Baker, Alexander New, and Gautam K Vallabha. L2explorer: A lifelong reinforcement learning assessment environment. arXiv preprint arXiv:2203.07454, 2022.
[24] Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. arXiv preprint arXiv:1902.01377 2019.
[25] Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaskowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE conference on computational intelligence and games (CIG), pages 1-8. IEEE, 2016.
[26] Khimya Khetarpal and Doina Precup. Attend before you act: Leveraging human visual attention for continual learning. arXiv preprint arXiv: 180i: 09664, 2018
[27] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-335 2017.
et al.
[28] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11 (2(11): 1): 2013.
[29] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 1712.0544 2017.
[30] Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell. Reinforcement learning in roboties: Applications and real-world challenges. Robotics,
2013.
[31] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 2020.
[32] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 19884-198955 2020.
[33] Hojoon Lee, Hanseul Cho, Hyunseung Kim, Dachoon Gwak, Joonkee Kim, Jacgul Choo, Se- Young Yun, and Chulhee Yun. Enhancing generalization and plasticity for sample efficient reinforcement learning. arXiv preprint arXiv:2306. 107/ 2023.
[34] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization deep reinforcement learning. arXiv preprint arXiv: 1910.05366, 2019.
12

---

[35] Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Rodriguzz Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information fusion, 58:52-68, 2020.
[36] Chengshu Li, Fei Xia, Roberto Martin-Martin Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprini arXiv:2108.03272 2021.
[37] Vincenzo Lomonaco, Karan Desai, Eugenio Culurciello, and Davide Maltoni. Continual reinforcement learning 3d non-stationary environments. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 248 249, 2020.
[38] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido M Van de Ven, et al. Avalanche: an end-to-end library for continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3600-3610, 2021.
[39] Nicoló Lucchesi, Antonio Carta, Vincenzo Lomonaco, and Davide Bacciu. Avalanche rl: A continual reinforcement learning library. In International Conference on Image Analysis and Processing, pages 224-535. Springer, 2022.
[40] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. arXiv preprint mrXiv:2303.01486 2023.
[41] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 7765-7773, 2018. 2018.
[42] Shie Mannor and Aviv Tamar. Towards deployable rl-what's broken with research and potential fix. arXiv preprint 1xXi:2331.011220 2023.
[43] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 11X1v:131 2013.
[44] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-555 533, 2015.
[45] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint 17Xiv:1710.116222 2017.
[46] Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. arXiv preprint arXiv:2305. 15555, 2023.
[47] Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Pau Rodriguez, Matthew D Riemer, Julio Hurtado, Khimya Khetarpal, Ryan Lindeborg, Lucas Cecchi, Timothée Lesort, al. Sequoia: A software framework to unify continual learning research. arXiv preprint arXiv:2108.01005. 2021.
[48] Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKimney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv: 133600880 2019.
[49] German Parisi and Vincenzo Lomonaco. Online continual learning on sequences. In Re- cent Trends in Learning From Data: Tutorials from the INNS Big Data and Deep Learning Conference (INNSBDDL22019, pages 197-221. Springer, 2020.
[50] Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 20/8 IEEE international conference on robotics and automation (ICRA), pages 3803-3810, IEEE, 2018.
13

---

[51] Aleksei Petrenko, Erik Wijmans, Brennan Shacklett, and Vladlen Koltun. Megaverse: Simulat- ing embodied agents at one million experiences per second. In International Conference on Machine Learning, pages 8556-85666 8566. PMIR, 2021.
[52] Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint 20X::220020006300 2020.
[53] Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. Cora: Benchmarks, baselines, and metrics as platform for continual reinforcement learning agents. In Conference on Lifelong Learning Agents, pages 705-743. PMLR, 2022.
[54] Mark B Ring. Child: A first step towards continual learning. In Learning to learn, pages 261-292. Springer, 1998.
[55] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experi- ence replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[56] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 06006 04671, 2016.
[57] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:2 13202, 2021.
[58] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[59] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Trabska-Barwinnse Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 3528-4537.
PMLR, 2018
[60] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140- 140-1144, 2018
[61] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 14:25112266 2021.
[62] Norman Tasfi Pygame learning environment. PyGame-Learnin Learning- Environment) 2016.
https://giitub..commttaae
[63] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open- ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808 2021.
[64] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[65] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEFF/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
14

---

[66] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5033. IEEE, 2012.
[67] Tristan Tomilin, Tianhong Dai, Meng Fang. and Mykola Pechenizkiy. Levdoom: A benchmark for generalization on level difficulty in reinforcement learning. In 2022 IEEE Conference on Games (CoG), pages 72-79. IEEE, 2022.
[68] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprini urXiv:1708.04782) 2017.
[69] Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302..0045) 2023.
[70] Maciej Wotezyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciúski, and Piotr Milos. Contin- ual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 34:28496-28510, 2021.
[71] Maciej Wolczyk, Michat Zajac, Razvan Pascanu, Lukasz Kuciñski, and Piotr Mitoš. Disentan- gling transfer in continual reinforcement learning. Advances in Neural Information Processing Systems, 35:6304-6311 2022.
[72] Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 79-86.
IEEE, 2017.
[73] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1 100. PMIR, 2020.
[74] Zichao Zhang, Henri Rebecq, Christian Forster, and Davide Scaramuzza. Benefit large oield-of-view cameras for visual odometry. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 801-808. IEEE, 2016.
[75] Artem Zholus, Alexey Skrynnik, Shrestha Mohanty, Zoya Volovikova, Julia Kiseleva, Ar- Szlam, Mlec-Alexandre Coté, and Aleksandr I Panov. Iglu gridworld: Simple and fast environment for embodied dialog agents. arXiv preprint 2rXiv:2206.00142 2022.
[76] Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvu- nakool, János Kramár, Raia Hadsell, Nando de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv: 1802.00900994 2018.
15

---

A Scenario Descriptions
In this section, we provide a description for each scenario used in our benchmark. We outline and the reward functions.
Pitfall In this scenario, the agent is located in a very long rectangular corridor. The surface of the area is divided into squares, of which there are in each row. the beginning of an episode, each of those sections has a 30% chance of being turned into pit by lowering the height of the floor in the given area. The floor layout is randomized every episode, so there is no one optimal route to take. Falling into a pit instantly terminates the episode and should be avoided at all costs. Reaching the other end of the corridor also terminates the episode. The agent is tasked to traverse along the corridor as far possible and potentially reach the other end. Note that there is ~10 chance that a section of the corridor becomes untraversable due to the stochastic nature of the layout. The agent ought to identify safe path through the corridor and speedily navigate forwards. The agent can turn left and right, move forward, and accelerate. The agent is proportionally rewarded for how much has travelled forward in the corridor, and penalized for falling into pit.
Arms Dealer In this scenario, the agent has to pick up weapons from the ground in a square-shaped room and deliver them to a platform that appears at a random location after a weapon is obtained. The agent can carry multiple (the exact amount is dependent the delivery. A new weapon is spawned at a random location whenever one is picked up, i.c., there is always a constant number of weapons on the ground. A new platform is created for every weapon picked up. After a successful delivery, all the platforms disappear, and the agent has to start over by collecting weapons. The agent can turn left and right, move forward, and accelerate. The agent is rewarded for both obtaining a weapon and delivering it. Additionally, a small reward is granted for how much the agent moves in the environment. Finally, the agent is penalized for being idle.
the weapon) weapons before making
Hide and Seek In this scenario, the agent is randomly spawned in one of 20 possible locations within a maze-like environment. enemies are spawned at random locations at the beginning of an episode. The enemies can only inflict damage at close distance and thus constantly attempt to move closer to the agent. Health kits granting 25 hit points continually spawn in random locations at specified time intervals. The objective of the agent is survive by hiding from the enemies. The agent should identify the enemies and attempt to constantly move away from them while collecting the health kits when at low health. The agent can turn left and right, move forward, and run. The agent is rewarded for movement, surviving a frame and each health item collected, and penalized for having damage inflicted by enemies.
Floor is Lava In this scenario, the agent is located in rectangular room divided into 16x equal sized squares. The room is filled with lava, which infficts health point of damage if being stood upon. After every fixed time interval, each square section of lava has 20% chance of being changed to a platform, which no longer causes damage. The objective is to survive by minimizing the time spent standing in lava. The agent ought to identify the platforms and quickly navigate top of them as soon as they appear to avoid rumning out of health. The agent can turn left and right, move forward, and accelerate. The agent is rewarded for surviving a frame, stepping onto platform whilst having previously stood in lava, and penalized for standing in lava.
Chainsaw In this scenario, the agent is randomly spawned in one of 20 possible locations within a maze-like environment, and equipped with chainsaw. A fixed number of enemies are spawned at random locations at the beginning of an episode. Additional enemies will continually be added at random unoccupied locations after each time interval. The enemies are rendered immobile, forcing them to remain at their fixed locations. The goal of the agent is to find the enemies, walk up to melee distance from them and saw them in half. The agent can move forward, turn left and right, and use the chainsaw. The agent is granted a reward for moving in the environment and each enemy killed.
Raise the Roof this scenario, the agent is randomly spawned in one of 20 possible locations within maze-like environment. The room has very high ceiling at the beginning of an episode, which, however, starts to be slowly but constantly lowered. At specific locations of the area, there are switches on the walls that can be pressed to raise the ceiling back up a bit. After switch is pressed,
16

---

it disappears and can no longer be used. If the switches are not pressed with a high enough frequency. the ceiling will eventually crush the agent, which terminates the episode. The goal of the agent is thus to locate and press the switches to keep the ceiling high before the episodes timeouts. The agent can move forward, turn left and right, and activate a switch. The agent is rewarded for movement, pressing switch, and surviving for frame.
Run and Gun In this scenario, the agent randomly spawned in one of 20 possible locations within maze-like environment, and equipped with weapon and unlimited ammunition. A fixed number of enemies are spawned at random locations at the beginning of an episode. Additional enemies will continually be added at random unoccupied locations after each time interval. The enemies are rendered immobile, forcing them to remain at their fixed locations. The goal of the agent locate and shoot the enemies. The agent can move forward, turn left and right, and shoot. The agent is granted reward for each enemy killed.
Health Gathering In this scenario, the agent is trapped in a room with surface, which slowly but constantly decreases the agent's ' health. Health granting items continually spawn in random locations at specified time intervals. The default health item heals grants 25 hit points. Some environments contain poison vials, which inflict damage the agent instead of providing health. The objective is to survive. The agent should identify the health granting items and navigate around the map to collect them quickly enough to avoid running out of health. The agent can turn left and right, and move forward. A small reward is granted for every frame the agent manages to survive.
B Reward Shaping
In this section, we present all components constituting the auxiliary reward function of each scenario.
Raise the Roof Ti = F = +1 8 for surviving an iteration, s is the number of switches pressed. / represents the coordinates of the agent's location, C1 = 15 is the reward of pressing switch, co 0.001 scales the reward granted for movement, and is the time step.
4- - where F 0.01 is constant reward granted
Chainsaw Ti = Ci k c2 14-44-512 where is the number of enemies eliminated, represents the coordinates of the agent's 's location, ei the reward for eliminating an enemy, c2 0.001 scales the reward granted for movement, and / the time step.
Health Gathering Tt = F c1 1he > h where F 0.01 a constant reward granted for surviving an iteration, h is the remaining health (increases when health kit is picked up), = 15 scales the reward granted for picking up health kit, and is the time step.
Pitfall Ti d + e max (0, v2 14-1), where d 1.0 is the penalty for falling into pit, represents the x-axis coordinate of the agent, 4 = 0.1 scales the reward for traversing the corridor, and is the time step.
4- where k is the number of enemies eliminated, 1 represents the coordinates of the agent's location, 6₁ is the reward for eliminating an enemy, C2 0.001 scales the reward granted for movement, and / is the time step.
Run and Gun = ke C2 1
Floor is Lava Fe = c1 11 [h; = h Ah-1 < h-2) + C2 1 the < h4-1), where h is the agent's health, ei 3 is the reward for stepping on platform from lava, C2 -0.1 is the penalty for standing on lava, and t is the time step.
Arms Dealer r: = C w +2 d+ C3 11+ 114-4455 where P -0.01 is a constant penalty to avoid passivity, w is the number of weapons acquired, d is the number of weapons delivered, = 15 the reward of acquiring weapon, C2 is the reward for delivering a weapon, C3 = 0.001 scales the reward granted for movement, and is the time step.
17

---

Hide and Seek to = 1(h > h111)+ 1( < < where F 0.01 is a constant reward granted for surviving an iteration, h is the remaining health (increases when health kit is picked up and decreases when damage is inflicted by enemy), the penalty for receiving damage from an enemy, and the time step. It is relevant to note that picking up health kit grants more health in comparison to the damage inflicted by the enemy if were to occur in the same time step, meaning that the agent receives a positive reward.
5 is the reward granted for
picking up health kit, c
C Framework
In this section, we will discuss our experimental framework in detail.
C.1 Network Architecture
We use the same convolutional encoder head as in 44 with relu activations, followed by two fully connected layers with 256 neurons each. A Layer Normalization is applied after the first dense layer, with tanh activation, known to work well with Layer Normalization 4 The second dense layer is followed by leaky ReLU activation (with a 0.2). Both the actor and critic networks utilize the presented architecture. The actor network takes an observation as input and generates a Gaussian distribution of action probabilities corresponding to the action space. Each task has its own actor head responsible for producing these outputs. The critic network takes the same input and produces scalar value for each task head. The network is equipped with multiple heads, each of which is responsible for outputting the action distribution for one of the tasks in the sequence. Their number of output heads is thus equal to the number of tasks we showcase experiments where only single head is employed, and the one-hot encoded task id input is provided.
corresponding sequence. In Appendix ??,
C.2 Implementation Details
During preprocessing the observation from the environment is downscaled to an 84 x 84 pixel RGB image, and 4 of such most recent frames are stacked before fed the network. At the start of each task, the collected experience is emptied from the replay buffer and the agent uses random policy exploration for 10000 steps collect experience. The entropy regularization coefficient is automatically tuned such that the standard deviation of the action distribution on every dimension matches predefined value Otgt- When switching to a new task, we prioritize minimizing forgetting in the actor while allowing the critic to adapt freely. By default, we thus only regularize the actor, except for AGEM, where we found slightly beneficial to also include the regularizing of the critic network. We use learning rate decay, which is reset upon the start of a new task.
The agent undergoes sequential training each task T/ for duration of A 200K iterations each. After every 1000 iterations, we assess the policy's performance on each task within the sequence, conducting 10 episodes for evaluation. A buffer maintained store the 50K most recent trajectories, which are subsequently cleared from the buffer at the conclusion of a task. When calculating forgetting, we consider i 10 final performance measurements of a task.
All experiments are conducted on a computer with a 20-core 19-009 CPU and GeForce RTX 3090 GPUs
C.3 Hyperparameters
We adopt most of the hyperparameters from the original SAC algorithm 20 and Continual World 70 The extensive list is brought in Table
D
Task Recollection
The individual tasks in our benchmark may vary in overall complexity and exhibit different potential for transfer between each other. In this section, we therefore examine the difficulty of maintaining performance on tasks within the Cross- -oojs-Obeectiveeee sequence. Our analysis Table reveals that the task Pitfall, closely followed by Hide and Seek, exhibit the highest levels of forgetting across
18

---

Table 4: Overview of the hyperparameters used in experiments.
Hyperparameter
Value
Shared by all algorithms
optimizer
Adam
initial learning rate
10 10-3 3
learning rate decay
Linear
learning rate decay rate
0.1
learning rate decay steps per task
103
Adam 31, A2
0.9, 0.999
target action distribution std attg hidden layer sizes
0.089
256, 256
activation convolution
relu
activation dense
tanh, Irelu
discount (r)
0.99
polyak averaging interpolation factor (p)
0.995
frame stack
4
frame skip
4
batch size (n)
128
replay buffer size
50K
initial collect steps (binit) steps per task
10
200K
update frequency (steps)
5000
gradient steps per update number of evaluation episodes regularization number of batches
50
10
10
regularization batch size
32
PackNet
retrain steps
104
gradient clipping norm
2 10-5
MAS
regularization coefficient
104
L2
regularization coefficient
105
EWC
regularization coefficient
250
AGEM
episodic memory per task episodic batch size
10
128
VCL
regularization coefficient
1
ClonEx-SAC
exploration kind
best return
regularization coefficient
100
episodic memory per task episodic batch size
10
128
19

---

Scenario
PackNet
MAS
AGEM
L2
EWC
Fine-tuning
VCL
ClonEx-SAC
Average
Piifall
0.18
0,60
0.81
0.04.1m3
0,10
0.75 aum
0.55 aws
0.00 aom
0.38 .03
Arms Dealer
0.00 cue
0.50 2014
0.63 as
0.01 am
0.08 a3
0.47 oo
0.63 an
0.01 -m
0.29 .027
Hide and Seek
0.00
0.25
0.80
-0.04
a.
0.84 -ooe
0.76
0.16
0.35
Flour is Lava
-0.18 cer
.0.03 =08
0.85 an
0.01 22
-0.4 cmo
-0.00 am
0.64 a2
-0.01 40
0.15
Chainsaw
-0.01
-0.02
0.93
-0.08
-0.02-00
0.62
0.61-ais
-0.07
0.25
Raise the Roof
-0.06 c.3
-0.07 2-002
0.97 2
-0.13 smm
-0.01 2-408
0.46 =-40n
0.89 2m
-0.02 2:
0.25 2045
and Gun
-0.01
-0.02
0.93
-0.08
-0.08
2-402
0.20
0.36 400
0.00
0.16
Table 5: Forgetting of all methods on each individual task of the CO8 sequence, calculated as outlined
in Equation The results of tasks that were most forgotten by a given method are highlighted in bold.
The final task of the sequence is omitted from the comparison, as no forgetting can have occurred.
all methods. However, it is important to note that this comparison may not accurately reflect the inherent difficulty of each task in terms of forgetting.
The ordering of tasks within the sequence can introduce a disparity, as the policy undergoes a greater number of weight updates based on experience gathered from later tasks before completing the entire sequence. This dynamic impacts the difficulty of remembering tasks the beginning of the sequence. On the other hand, it increases the potential for backward transfer from future tasks, which could potentially enhance performance. However, we did not observe notable instances of backward transfer in our experiments. Consequently, we posit that tasks at the beginning may be slightly disadvantaged in the comparison due to forgetting.
Exploring different orders of sequences would be an interesting avenue for future research. By varying the order, we could better determine the complexity of different scenarios and their impact on performance. This approach would provide more nuanced insights into the difficulty of each task within the sequence. We have provided the functionality in COOM to randomize the tasks in sequence. However, we have not conducted any experiments in this setting, as a proper evaluation would require numerous trials, which we lack the computational resources for.
Overall, our current analysis highlights the challenges in maintaining performance on the Pitfall and Hide and Seek tasks. However, further investigation is necessary to fully understand the complexities involved and to identify the most challenging scenarios within the Cross-Objective sequence.
E Memory and Time Consumption
V0-
Chemary Consummgtion (Ux))
Calliteee.. 01
200
Recefee
VOL
ACEM
vcL
(a) Elapsed Real Clock-Time
(b) Memory Consumption
Figure 7: Time and Memory Consumption on the CD4 task sequence
Our experiments indicate that replay- based methods consume more resources in terms of both time and memory compared to other baselines. Figure 7a depicts the elapsed real clock-tima hours. Most methods indicate an average runtime of 16- 12-17hooee hours, whereas AGEM and Perfect Memory take around 20 and 25 hours to complete respectively. Figure 7b depicts memory consumption in gigabytes. Most methods exhibit an equal memory consumption of approximately 35GB, as the replay has a fixed size of 50K trajectories. AGEM uses slightly more memory due to storing selective episodic memory from previous tasks. Lastly, Perfect Memory clearly stands out in the comparison
20

---

with consumption of over 500GB. This analysis can be valuable for when resources are a critical factor when selecting an appropriate CL method.
Cross-Objective Challenge
The Cross Cross-Objest Objective Challenge (COC) is designed be challenge for future methods. In this section we will describe how the task sequence is constructed, and further expand how our baseline methods performed the sequence. The following list outlines how each scenario is modified to obtain a harder version.
1. Pitfall
The chance of a tile being pit is raised from 0.4 to 0.7.
The actor property APROP_ -Seed of the agent decreased from 1.0 to 0.7.
2. Arms Dealer
The size of the map is increased by 50%.
Instead of 5 weapons, only is spawned at a time.
Instead of single weapon type (Chaingun), one of weapons (BFG9000, Chaingun, Pistol, Pl PlasmaRifle RocketLauncher, Shotgun, SuperShotgun) is randomly spawned.
3. Hide and Seek
Number of initially spawned monsters is increased from 10 to 20.
Number of initially spawned health kits is decreased from 20 to 10.
The actor property SPROP_Spect of the agent decreased from 1.0 to 0.7.
4. Floor is Lava
The change a tile being platform is decreased from 0. to 0.02.
The change interval of the platforms is decreased from 150 to 100 game tics.
5. Chainsaw
Number of initially spawned monsters decreased from 20
The spawn interval of a new monster is decreased from 90 to 30 game tics.
Monster health is increased from to 50.
6. Raise the Roof
The matching texture of the wall sections with switches (SW1BLUE1) is changed to (SW1GRAY1), so that it no longer differs from the rest of the wall.
7. Run and Gun
Number of initially spawned monsters is decreased from 20 to 5.
The spawn interval of a new monster is decreased from 90 to 30 game tics.
Monster health is increased from to 25.
8. Health Gathering
The map layout from health health.gathherng is replaced with more complex maze-like
version health wethhhgtthrrinnnggppee supreme from ViZDoom 25
The spawn interval of a new health kit increased from 30 60 game tics.
We will henceforth compare results between the CO8 and COC sequences. We depict the training curves each individual task with corresponding core success metric in Figure X We can observe a slight performance increase in each task on COC, but it significantly falls below the CO8 counterparts. Figure indicates that PackNet, the best method our previous benchmark evaluations, also stands out in COC. The continual evaluation results in Figure 10 further show how all displayed methods underperform on the hard COC.
21

---

1e5
Weapons Delivered
600
600
Distance
20
Frames Alive
Frames
10
200
200
50
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
Chainsaw
Raise the Roof
Run and Gun
Health Gathering
Kill Count
Kill Count
Frames
400
40
400
0
150
150
Timesteps
PackNet (CO8)
L2 (CO8)
MAS (CO8)
PackNet (COC)
L2 (COC)
MAS (COC)
Performance comparison of PackNet and MAS individual tasks of CO8 and their complex counterparts in COC. Each graph only depicts the time steps during which the task was trained. COC remains unsolved for MAS and even PackNet, which was the best performing method on other sequences.
Figure 8:
CO8
Arms
Chainow
Roof
Feath
0.8
0.0
0.2
0.4
0.6
106
1.0
2.6
0.2
0.2
0.4
0.6
1.2
14
Timesteps
166
EWC
Fine-toning
ClerEsSAC
Figure 9: Comparison of average success between CO8 and COC.
22

---

Pittall
0.8
02
0.0
0.2
0.4
0.6
0.8
2.0
1.2
14
16
Arms Dealer
166
1.0
Succ8ss
0.4
0.0
14
and Seek
0.8
0.6
0.0
0.0
0.2
0.4
0.8
2.0
1.2
144
1.6
Floor Lava
166
1.0
0.8
0.4
0.2
0.4
Chainsaw
leb
1.0
0.6
0.4
0.0
Raise the Roof
leb
0.8
0.6
0.2
0.0
0.2
0.4
0.6
0.8
1.0
12
1.4
1.0
Run and Gun
165
1.0
Succois
0.4
1.0
Health Gathering
0.6
0.0
0.0
0.4
0.8
1.0
1.4
1.8
Timesteos
1c5
PackWet (CO8)
L2 ICDB)
EWC (CO8)
Pa (COCC
L2 (00C)
EWC (CDO)
Figure 10: Continual evaluation results of select methods CO8 and COC.
23

---

G Sequence Length Comparison
Another way of assessing forward transfer would be compare the av- erage performance on a 4-task se- quences with the corresponding 2nd half of its 8-task counterpart. The tasks and their order in this compar- ison are the same. The only differ- ence here is that the 8-task sequence has had the potential to learn relevant competencies from previous tasks in its 1st half. If the 2nd half of the 8- task sequence has higher performance then this would be an indicator that knowledge from the initial tasks has been transferred forward to the later ones. We present this comparison in Table 6 Across all methods, we report 4.04% performance increase from CO4 to CO82nd and 12.8% % from CD1 to CD82nd, A performance decrease only occurred in 3/12 of cases. We can indeed thus conclude that the 1s half of 8-task sequences fastens learning the 2nd half. We posit that the Cross-Domain setting has higher transfer due to the shared objective across tasks, meaning that learned behaviour can better bolster performance further tasks.
Table 6: Average performance comparison of 4-task se- quences with the 2nd half of the corresponding 8-task se- quence.
Method
CD4
CD8
C04
COg
PackNet
0.92 0.04
0.92 = 0.00
0.87 + 0.13
0.91 0:44
MAS
0.55 0.3
0.83 0.16
0.72 0.5
0.76 0.23
AGEM
0.35
0,43
0.42
0.35 0.32
L,2
0.71 0.25
0.71 0.12
0.80 1
0.54 2.14
VCL
0.46 + 0.88
0.43 0.17
0.40 = 0.28
0.61 0:30
Fine-tuning
0.59 022
0.72 0.18
0.50 0.11
0.69 0.21
Average
0.595
0.673
0.618
0.643
H Network Plasticity Experiments
In our experiments for network plasticity, we halve the number of steps for each task, reducing them to 100K. we found it to be sufficient for convergence. The entire sequences thus lasts 4M and 8M environment iterations in C04 and CO8 respectively. After a task is finished, we maintain the network weights and only reset the learning rate decay and replay buffer. Instead of measuring Success, we assess the individual performance metric of each task. We wish to observe whether the initial performance achieved by the agent on given task deteriorates over successive trials when re-encountering it. We additionally depict the learning curves of the SAC agent the CO8 task
sequence in Figure II
Similar to the results on CO4 in the main paper (Figure 4 the attained peak performance remains stable throughout the iterations. However, on CO8, we can witness discernible dip in performance across most tasks midway through the entire sequence, yet notably, this decline is followed by resurgence towards the end. Further, on C04 we can observe that the initial iteration of some tasks is
x10F
Seek
Viapprrs Dafoared
600
660
Famms Alvz
10
300
300
0.0
0.5
2.0
0.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
X10
x10*
x10*
x10"
Chainsew
Raise the Roof
Run and Gun
Health Gathering
Hoath KRs Cemaee
KBl COUIT
40
20
40
20
200
x10
of Training steps (1O0K for Each Thaki
Figure 11: SAC does not exhibit a loss of network plasticity, Measuring the individual performance metric of each task across 5 seeds displays that repeated exposure the same tasks throughout 10 iterations of the CO8 sequence does not hinder the capacity to reattain peak performance.
24

---

substantially lower than the subsequent runs. This, however, does not appear for the same tasks on CO8, because the agent has already learned to succeed other tasks and is capable of transferring knowledge forward.
I Method Variations
Table presents the full results of incorporating variations to our baseline methods. Figures 12 and 13 depict the average success curves of selects methods the CO8 and CO4 task sequences respectively. PackNet stands the strongest in terms of stability across the variations, as not a single variations causes substantial degradation of performance. Regularization-bet aaaed methods stand out in our evaluations as the most volatile, as their performance is very likely to completely degrade when employed with variations. The variation that improved the performance of most methods is the Noise augmentation.
PackNet
Gathering
1.0
Sacces
0.8
0.4
200
400
600
300
1000
1200
1400
1600
MAS
200
400
600
800
1000
1200
1400
1600
AGEM
0.2
0.0
200
460
10000
1400
1600
L2
3.0
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.
3.0
L000
1600
Oftic
Figure 12: Performance comparison on the CO8 sequence of select methods between critic regular- ization and single output with the original setting.
25

---

Table 7: Method variation performance on CO8. Results higher than the default are highlighted in
bold.
Method
Default
Conv
Shift
Noise
LSTM
Single Head
Critic Reg
PackNet
0.84 0.0
0.72 +0.0
0.87 0.04
0.88 +00
0.83 0.6
0.68 =0.15
0.70 e
MAS
0.58 =0.17
0.08 =0:07
0.48
0.47 0.15
0.01
0.29 20.18
0.25
AGEM
0.30 =0.13
0.02 =0.00
0.40 00.09
0.33 +002
0.48 0.09
0.12 =0.11
0.28 =004
L2
0.73 00.00
0.34 0.08
0.75 #0.11
0.79 0.088
0.01 0.00
0.28 2030
0.28 =0.00
VCL
0.39
0.05
0.30
0.29 0.13
0.01
0.31 2008
0.41
ClonEx-SAC
0.89 0.00
0.26 n.0
0.89 0.00
0.90 =0006
0.02 0.00
0.79 =0.11
0.87 =004
EWC
0.65 0.13
0.32 2015
0.26 20.17
Fine-tuning
0.44 6.11
0.13 =0.3
1.0
Chansaw
Oun
Health Gutholin
0.6
0.0
100
200
300
400
500
600
700
800
MAS
1.0
0.8
0.4
100
200
300
400
500
600
700
800
AGEM
1.0
0.8
0.4
0.0
200
1.0
0.2
100
200
300
400
500
600
700
800
VCL
0.8
0.4
0.0
1.0
0.6
0.2
100
200
300
400
500
600
700
800
Timesteps
Debutt
Convolition
Shit
Noise
LSTM
Figure 13: Performance comparison the C04 sequence of select methods between image augmen- tation methods and and LSTM encoder with the original setting.
26

---

Repeated Sequences
Pitfail
Arms
Hide
Hloor Chainsaw Raise
Health
Pitfall
Arma
Hide
Floor Chairsaw Raise
Health
1.0
0.8
Success
0.6
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Figure 14: Average cumulative success off PackNet on CO16.
1.0
0.4
0.0
1.0
2.0
3.0
1eb
Figure 15: Continual evaluation of PackNet on each task of CO16.
In line with the approach used in [70], have repeated core sequences to generate 16-task sequences for both modalities. More tasks present greater challenge retaining previously acquired knowledge. Additionally, it enables thorough evaluation of how methods leverage their acquired competencies when encountering previously learned tasks anew. Given the substantial computational cost associated with conducting experiments the 16-task sequences, have limited evaluation to PackNet applied to CO16. We present visual overview of the results in Figures 14 and 15
K Cross-Domain Sequence
In this section, provide detailed deseription of the modifications implemented to compose the sequences based on the default environment the Run and Gun scenario. The specific modification types for each environment are displayed in Table
Table 8: Environment modification types of Run and Gun
Environment
Obstacles
Green
Resized
Monsters
Default
Blue
Shadows
Modification Type
Decorations
Textures
Target
Agent Size
Target Type
Textures
Textures
Target Rendering
This modification involves adding various in-game elements (e.g., torches, pillars, barrels, lamps), to the game environment at fixed locations. These decorations serve as 1) distractions by having the agent considering them relevant, and 2) obstacles impeding the agent's movement.
Decorations:
Textures: The textures used for various game assets, such as walls, floors, and ceilings, are altered in this modification. Different textures are applied to create distinct visual appearances for each task, further increasing the visual disparity across tasks.
Target & Agent Size:
This modification involves adjusting the width and height of targets and the height the agent within the game environment. Smaller narrower targets can be more difficult detect. Changing the height of the agent affects the view height and thus creates a visual shift of the perceived environment.
27

---

Target Type: In this modification, the visual appearance of targets is altered. We use various enemies from the original game and render them immobilized.
This modification focuses on changing the visual rendering the targets. For example, targets be rendered with different visual effects styles, making them visually more distinct or harder to detect and requiring the agent to adapt its perception and recognition abilities.
Target Rendering:
To implement these modifications, utilize the SLADE3 tool, which enables us to customize various aspects the DOOM game environment. Each modification type contributes to the visual disparity and complexity across tasks, challenging the agent's perception, decision-maie making, and adaptation capabilities, creating a comprchensive continual learning task sequence. In the following provide more comprchensive description of changes.
1. Obstacles
total of 25 elements named Tall Blue Torch, Tall Green Torch, and Tall Red Torch with respective in in-game id's 's [44, 45, 46] are constructed at fixed locations of the map. which act as obstacles
2. Green
Wall texture A-CAMO1
Floor texture AQFO22
Ceiling texture DOGRMSC
3. Resized
The actor properties APROP_ PrroP_ScaleX and SPPROP_Scale" of enemies randomized in the range of [0.3, 3.0]
The actor property APROP -ieo_LeewWeiihht of the agent is randomized in the range of [0.0, 70.0]
4. Monsters
Replaced targets with immobile enemies: Arachnotron, Fatso, Cacodemon, Archvile, Revenant
The number of initially spawned enemies is set to 3 per type
The actor property APROP_Invulnerabl of the agent is set True
5. Default
The map layout originates from the ViZDoom scenario wealth_gahhrinnggsupeee
Wall textures ICKWALL3
Floor texture AQF05/
Ceiling texture FLATI9
The surface no longer inflicts damage to the agent
Stationary targets are spawmed at random locations around the map
The target type is CommanderKeen
The health of the target is set to
The number of initially spawned targets set to 20
The spawn delay of every subsequent target is set to 30 game ticks
The agent is equipped with pistol and 200 bullets
6. Red
Wall texture FIREWALL
Floor texture CRACKLE4
Ceiling texture DORED
7. Blue
Wall texture FIREBLU2
Floor texture FLATI4
Ceiling texture CEIL4_
8. Shadows
The actor property OPROP_RenderShyve of targets is set STYLE Shadow
28

---

L Limitations
Sequence ordering In our benchmark, we focused on evaluating performance and transferability in the context of a fixed sequence ordering. We acknowledge that varying the order of the sequences could introduce additional dynamics and complexities to the continual learning (CL) process.
Different sequence orderings have the potential to impact the learning process by introducing variations in task difficulty, temporal dependencies, interference patterns. Exploring multiple sequence orderings could provide valuable insights into the robustness and adaptability of CL methods across different scenarios.
However, due to practical constraints, such as limited computational resources and time, we were unable to thoroughly investigate the effects of sequence ordering in our experiments. Consequently, our benchmark considers only a single predefined ordering of the sequences.
Future research endeavors could delve into the impact of sequence ordering by systematically exploring different orderings and analyzing their effects on CL performance. This would enable more comprehensive understanding of the challenges and opportunities associated with varying sequence orderings in continual learning scenarios.
Restricted Action Space In order to simplify the training process, enhance ease of learning, and bring more focus to the continual learning paradigm, rather than very complex RL challenges, we opted to restrict the total action space of the DOOM video game. By doing so, we only included actions that we deemed essential for effectively achieving the established objectives.
In addition to maintaining the core actions required for navigating within the game environment, we kept single additional action called for each scenario, which we referred to as EXECUTE. For instance, in some scenarios, the action resulted in the agent accelerating, while in others, it corresponded to firing the weapon. This approach allowed us condense the action space even further, particularly in cross-objective sequences where certain actions would have otherwise been rendered unnecessary or redundant in some tasks.
However, this potentially confuses the agent both when selecting the best action during training, but also, when attempting maintain performance on learned tasks, in which performing an action previous task from the given space would result in a different action in-game compared to the one in the task being trained. On the other hand, allowing the agent have a broader range of actions its disposal may have the potential to enhance overall performance, especially when employing more sophisticated and powerful methods. wider variety of actions would enable the agent explore different strategies and potentially discover more effective approaches to achieving the objectives.
Sequence length
The number of tasks in our benchmark is relatively small, and they are not repeated. Introducing task repetition in continual learning (CL) benchmark can yield additional interesting observations. However, due to the extensive runtime requirements for low-level budget setups. we opted to avoid long sequences in this benchmark.
The dimensionality of the observation space and the rendering of the environment contribute to longer learning times for a single task. We established a fixed task length of 200K timesteps, as our observations indicated that this was sufficient for reaching convergence. In our limited experiments with longer task lengths, we found that no method exhibited notable performance improvement after this number of iterations nor managed learn anything meaningful beyond this point.
While longer task lengths and task repetition could provide valuable insights, the constraints of our experimental setup, including limited resources and time, prevented from exploring these aspects in-depth. However, future studies with more resources and computational power could investigate these factors and potentially uncover further insights in the context of continual learning benchmarks.
Task boundaries In our benchmark, we adopted the use of explicit task boundaries to define the transitions between different tasks. While this approach provides clear and well-defined setting for evaluation, it does come with certain limitations.
By relying on explicit task boundaries, we assume that the learner has prior knowledge or external cues to identify and differentiate between tasks. real-world scenarios, however, the presence of explicit task boundaries may not be readily available or easily discernible. Therefore, extending
29

---

the benchmark to incorporate task inference mechanisms would be a valuable direction for future research.
Task inference mechanisms enable the learner to autonomously detect task boundaries without explicit supervision. Incorporating such mechanisms in the benchmark would allow for more comprehensive evaluation of CL algorithms that possess the capability to infer task transitions.
Additionally, the current format of the benchmark does not explicitly test algorithms' ability to handle continuous distributional drift. The absence of continuous drift scenarios limits the assessment CL methods in dynamic and evolving environments. Future iterations of the benchmark could include settings that introduce continuous distributional drift, thereby enabling the evaluation algorithms that can adapt to changing data distributions over time.
M Action Distributions
Apart from visualizing the actions executed during training, is more relevant for the CL conundrum observe how particular method manages to preserve the obtained distribution after training on particular task has concluded and the sequence continues with new tasks being introduced. To this end, we depict the distribution of actions continuously performed by PackNet and L2 on each task within the CO8 task sequence in Figures 16 and 17
Given the experimental results in Table VCL in both average performance and forgetting. Observing the figures on action distributions, this manifests in VCL being noticeably incapable of maintaining the obtained distribution new tasks come along, whereas PackNet and L2 have minimal fluctuation after the policy has been adjusted for given task. We notice that although PackNet and L.2 both had relatively high performance the task sequence, the distributions for particular tasks can drastically vary. This suggests that there a multitude of possible policies for acquiring adequate results on a task. For instance, there are two clearly dominating actions which PackNet utilizes in Arms Dealer, whereas L2 has more variety in the selection of actions, with MOVE_FORWARD being the most used one. Interestingly, L,2 also has low variance in the distribution before a task has been encountered, whereas for PackNet it is more random. The L2 regularization after the first task seems to fix the distribution also for all other future tasks.
know that PackNet and L2 significantly outperformed
30

---

Pitrall
Pstal
1000
Number of Axtions
800
600
400
200
Arms Dealer
Number Actiors
800
600
400
200
200
400
600
800
1000
1200
1400
1600
and Seek
Number Actiors
800
600
200
200
400
600
800
1000
1200
1400
1600
Floor Lava
1000
Number of Actions
600
400
200
600
8000
1000
1200
1400
1800
Chainsaw
1000
Number of Actions
800
400
200
200
600
800
1000
1200
1600
Raise the Roof
Number Actions
800
600
200
200
400
600
800
1000
1200
1400
1600
1000
Number of Actions
600
400
200
200
400
600
800
1000
1200
1400
1600
Health Gathering
1000
Numbbe of Actions
800
400
200
1000
Timestops (K)
TURN_RIGHT
EXECUTE
TURN RIGHT, EXECUTE
TURN LEFT, EXECUTE
MOVE FORWARD
TURN MOVE FORWARD
MOVE FORWARD, EXECUTE
TURN RIGHT, MOVE FORWARD, EXECUTE
TURN LEFT, MOVE FORWARD, EXECUTE
Figure 16: Actions executed by PackNet on CO8.
31

---

Pitrall
Mstal
Chainsaw
1000
Number of Axtions
800
600
400
200
Arms Dealer
Number Actiors
800
600
400
200
200
400
600
800
1000
1200
1400
1600
and Seek
Number Actions
800
600
200
200
400
600
800
1000
1200
1400
1600
Floor Lava
1000
Number of Actions
600
400
200
600
8000
1000
1200
1400
1800
Chainsaw
1000
Number of Actions
800
400
200
200
600
800
1000
1200
1600
Raise the Roof
Number Actions
800
600
200
200
400
600
800
1000
1200
1400
1600
1000
Number of Actions
600
400
200
200
400
600
800
1000
1200
1400
1600
Health Gathering
1000
Numbbe of Actions
800
400
200
200
1000
Timestops (K)
TURN_RIGHT
EXECUTE
TURN RIGHT, EXECUTE TURN
TURN_LEFT LEFT, EXECUTE
FORWARD MOVE FORWARD, EXECUTE
TURN MOVE FORWARD
TURN RIGHT, MOVE FORWARD, EXECUTE
TURN LEFT, MOVE FORWARD, EXECUTE
Figure 17: Actions executed by L.2 on CO8.
32

---

Pitrall
Pstal
Hide
Health
1000
Number of Axtions
800
600
400
200
Arms Dealer
Number of Actiors
800
600
400
200
200
400
600
800
1000
1200
1400
1600
and Seek
Number of Actiors
800
600
200
200
400
600
B00
1000
1200
1400
1600
Floor Lava
1000
Number of Actions
600
400
200
600
800
1000
1200
1400
1800
Chainsaw
1000
Number of Actions
800
400
200
200
600
800
1000
1200
1600
Raise the Roof
Number Actions
800
600
200
200
400
600
800
1000
1200
1400
1600
1000
Number of Actions
600
400
200
200
400
600
800
1000
1200
1400
1600
Health Gathering
1000
Number of Actions
800
400
200
Timestops (K)
TURN_RIGHT
EXECUTE
TURN RIGHT, EXECUTE
TURN LEFT, EXECUTE
FORWARD
TURN MOVE FORWARD
MOVE FORWARD, EXECUTE
TURN RIGHT, MOVE FORWARD, EXECUTE
TURN LEFT, MOVE FORWARD, EXECUTE
Figure 18: Actions executed by VCL on CO8.
33

---

Extended Results and Plots
In Table 19
we present the main experimental results with 95% confidence intervals.
Sequence
PackNet
MAS
AGEM
1.2
EWC
VCL
Fine luning
ClonEx SAC
Perfect Menory
CD4
0.92 -am
0.55 -o
0.35 -012
0.71 +0
0.69 eate
0.16
0.59 -012
0.87 .03
0.89 o
C04
0.87
0.72
0.42
=0re
0.65 20.11
=..4
0.50
0.86
0.89
CDS
0.91 2015
0.82 2012
0.30
0.87 z01e
0.76
0.36 20.3
0.45 20.11
0.92 a.2
CO8
=.8
0.58
0.30
0.71
0.65
0.39
0.44
0.89
COC
0.20 2018
0.04 2012
0.02
0.10
0.07 2014
0.03 2012
0.02 aam
0.13 ancs
Average
0.74
0.54
(a) Performance
Sequence
PackNet
EWC
VCL
Fine-tuning
ClonEx-SAC
Perfect Memory
-0.02
0.00
0.63
-0.01
C04
0.01 20.0
0.24
0.80 =022
0.00
0.04 0.04:005
0.82 xmm
0.71 :0cs
-0.03=00
0.02
0.86
-0.03
-0.04
0.75
C08
-0.01 2002
0.17
0.84
-0.04 20.0
0.02 20:2
0.64 xmm
0.48 20.2
0.00 1001
COC
0.04
0.09
-.00
0.00
0.00 -000-00:
0,16
0.10
0.01
Average
0.01 20.00
0.23 20.8
0.68 20.0
-0.02 20.01
0.00 200
0.62 200
0.53 20.5
-0.01 =00)
(b) Forgetting
Sequence
PackNot
MAS
AGEM
L2
EWC
VCL
Fine-tuning
ClonEx-SAC
Perfect Memory
CD4
0.40 0.40 a016
0.11 2032
0.10
40.28 av2z
-0.41 a4.00
0.21 2029
0.32 a.11
0.11 aus
0.30 aus 0.03 202)
CO4
-0.24 2032
-0.04
0.03 20
-0.60
-0.77 20.13
-0.57
-0.01 20.5
-0,26 2059
0.19
.02)
0.07
-0.55 .0.16
sore
oms
CO8
0.25 kom
0.01 -0.00
0.23 -01
-0.32
-0.38 40.1
0,20 came
0.23 n0t
0.27 mm
40.00
-0.01
400
aum
Average
0.13 28.14
0.07 20.3
0.11 2011
-0.23 2016
-0.43 20.10
-0.01 2015
0.17 2010
0.06 000
(c) Transfer
Figure 19: Extended results of average performance, forgetting and forward transfer with 95% confidence intervals across 10 seeds. The result of the best performing method is highlighted in bold. In the of ties, the with the highest confidence in selected.
34

---

PackNet
Defnuit
Red
Shadowe
0.5
2
3
MAS
1e5
8.5
0.0
AGEM
165
1.0
0.0
1.0
3.0
EWC
1e5
Fine: tuniingnig
1a5
0.5
VCL
Ju5
0.5
CcCEE-SAC
1e5
1.0
SU000sS
0.0
Perfect Memory
165
1.0
3.0
Timesteps
165
Figure 20: Continual evaluation results on the CD4 sequence.
35

---

PackNot
Chahaw
Redf
Gun
Health Gatbering
0.5
MAS
8.5
0.0
AGEM
0.0
1.0
3.0
EWC
165
Finc Fun-ttuning
165
VCL
1u5
0.5
CscEX-SAC
1e5
0.0
Perfect Memory
1.0
0.0
Timesteps
165
Figure 21: Continual evaluation results on the C04 sequence.
36

---

PackNet
Obstaclet
Resized
Monserss
Defsult
Shadows
9.75
0.50
0.25
0.00
0.0
0.2
0.6
088 MAS
12
1.4
0.25
0.00
0.8
12
AGEM
1.00
3.00
02
0.8
1.0
12
166
1.00
0.75
0.50
0.2
3.8
1.0
1.4
EWC
166
1.00
0.75
0.50
0.25
0.0
02
0.4
0.8
5.8
1.0
1.2
3.4
1.6
Fine-tuning
1ef
0.50
0.25
0.0
0.2
0.8
1.4
VCL
1e6
0.25
0.00
02
1.4
ClonEx-SAC
1.00
0.00
Timesteps
Figure 22: Continual evaluation results on the CD8 sequence.
37

---

PackNet
Patall
Flsor
Chaiso
Health Gathering
0.75
0.50
0.25
0.00
0.0
0.2
0.4
0.6
088 MAS
1.4
0.25
0.00
12
1.00
3.00
0.2
1.0
12
1.00
0.75
0.50
0.
5.8
1.0
1.4
EWC
166
1.00
0.75
0.50
0.25
0.0
0.2
0.4
0.8
3.8
1.0
12
1.4
1.6
Fine-tuning
0.50
0.25
0.0
02
0.8
1.4
VCL
1e6
0.25
0.00
0.2
1.4
ClonEx-SAC
1.00
0.00
Timesteps
Figure 23: Continual evaluation results on the CO8 sequence.
38

---

PackMet
Pitfall
Arms Dealer
Hide and Seek
Floor is Lava
Chainsaw
Raise the Roof Run and Gun Health Gathering ng
0.0
MAS
1.0
1.0
Current Task Success
0.5
EWC
Fine tuning
0.0
VOL
1.0
0.0
ClonExSAC
1.0
0.0
0.2
0.4
0.5
6.8
1.0
1.2
1.6
186
Timesteps
Pack/et
Rinettrinng
VCL
DioExxSAc SAC
Figure 24: Forward transfer results the CO8 sequence.
39


Reviews:
[{"review_id": "tHK4bJqis6", "reviewer": "Reviewer_vkRe", "summary": "", "questions": "", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The writing is good and the narration is easy to follow and understand.\n2. The experiments are comprehensive and provide a base for future continual RL study.\n3. The proposed three task sequences are interesting, it is good to test how robust CRL methods are to different level of task interference.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "hb8aZ2pG0e", "reviewer": "Reviewer_u5Cf", "summary": "", "questions": "", "limitations": "Yes, the authors analysis the limitation of COOM in the Appendix.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper is well written, the details are clearly explained, and the experiments are sufficient and well organized.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "o3zDJV6wyq", "reviewer": "Reviewer_TUS9", "summary": "", "questions": "", "limitations": "Similar to the opportunities for improvement mentioned earlier, the paper lacks baselines specifically related to reinforcement learning.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper addresses the lack of comprehensive benchmarks in the subfield of continuous reinforcement learning, making a valuable contribution to ongoing research in this area.\nThe inclusion of various scenarios, visual differences, and difficulty levels within a single benchmark enables a wide range of experiments.\nThe paper thoroughly investigates and evaluates different types of continuous learning baselines.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sPLVXRKqYa", "reviewer": "Reviewer_5vZ2", "summary": "", "questions": "", "limitations": "Limitations are only discussed in the appendix, and there is no mention of them (or reference to the appendix) in the main paper. The authors discuss (but do not address) the limitations mentioned in Opportunity 1.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This is the first benchmark for continual pixel-based RL that I am aware of.\n2. Various approaches are evaluated, somewhat demonstrating the feasibility of handling some of the settings in the benchmark", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ijdA0Xnq4O", "reviewer": "Reviewer_eYzv", "summary": "", "questions": "", "limitations": "The author should further expand on the limitations instead of just a few words. ", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper presents the first benchmark dedicated to evaluating continual reinforcement learning (CRL) in complex 3D environments with diverse objectives and visuals. The authors created six unique VizDoom scenarios and applied multiple well-established continual learning (CL) methods to establish baseline performance on these tasks, consisting of 54 sequences. Through these experiments, the study offers a comprehensive examination of CL methods, assessing their effectiveness based on prominent CL criteria.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces a new benchmark for continual learning (CL) in RL based on ViZDoom along with evaluation of several baseline CL methods. Reviewers raised several good points during the discussion -- evaluating plasticity, inclusion of RL baselines, task sequence ordering and length, whether the benchmark is close to 'solved' already -- which the authors had reasonable responses to and the paper is indeed stronger / more complete as a result. I'd encourage the authors to incorporate these updates to the main paper. I also appreciate the thorough discussion on limitations in Appendix J. Overall, this is a solid contribution. I recommend acceptance.
Decision: Accept (Poster)