SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tristan Tomilin<sup>1</sup> Meng Fang<sup>2,1</sup> Yudi Zhang<sup>1</sup> Mykola Pechenizkiy<sup>1</sup>  
<sup>1</sup>Eindhoven University of Technology <sup>2</sup>University of Liverpool {t.tomilin,y.zhang5,m.pechenizkiy}@tue.nl Meng.Fang@liverpool.ac.uk
The advancement of continual reinforcement learning (RL) has been facing various obstacles, including standardized metrics and evaluation protocols, demanding computational requirements, and a lack of widely accepted standard benchmarks. In response to these challenges, we present COOM (Continual DOOM), a continual RL benchmark tailored for embodied pixel-based RL. COOM presents a meticulously crafted suite of task sequences set within visually distinct 3D environments, serving as a robust evaluation framework to assess crucial aspects of continual RL, such as catastrophic forgetting, knowledge transfer, and sample-efficient learning. Following an in-depth empirical evaluation of popular continual learning (CL) methods, we pinpoint their limitations, provide valuable insight into the benchmark and highlight unique algorithmic challenges. This makes our work the first to benchmark image-based CRL in 3D environments with embodied perception. The primary objective of the COOM benchmark is to offer the research community a valuable and cost-effective challenge. It seeks to deepen our comprehension of the capabilities and limitations of current and forthcoming CL methods in an RL setting. The code and environments are open-sourced and accessible on GitHub.
Deep reinforcement learning (RL) has made immense leaps forward over the past decade in the domain of video games [68, 60, 9], robotic manipulation [28, 30, 72, 50, 76], embodied AI [65, 12, 63], and foremost on a variety of platforms intended for RL research [44, 22, 5, 63]. However, this success has primarily resulted from fine-tuning agents to solve specific tasks. As RL is increasingly applied to solving real-life problems in industry, healthcare, or robotics, situations arise where the environment and conditions are subject to rapid change. Whereas humans are able to learn to perform new similar tasks seamlessly, this competence is still predominantly absent in RL agents, who tend to exceedingly overfit to new tasks and forget all previously acquired competencies. A central goal of RL is thus to build systems that can master a spectrum of skills in environments as noisy and diverse as the real world, while being capable of continuing to learn.
Continual learning (CL) is the ability to swiftly learn consecutive tasks while maintaining adequate performance on previously mastered problems, where they ought to reemerge [49]. The CL paradigm emerges in RL when an agent proceeds to interact with an altered environment in the attempt to refine its policy [54]. Whereas task boundaries in CL might be gradual and smooth, we propose to consider learning a sequence of tasks with sudden transitions. Evaluating a CL agent usually involves several criteria: catastrophic forgetting, forward transfer and backward transfer [59]. CL research is particularly vital as it endorses the AI to thrive in encountered scenarios and conditions, which is much desired for AGI, compared to just solving individual problems.
Many prominent CL platforms are predominantly meant for supervised learning and require nontrivial adaptation for RL [47, 38, 16]. Video games act as ready-made simulation environments, which is why RL research platforms and benchmarks have widely been based on repurposed game engines [7, 6, 19]. There are, however, relatively few visual-based environments for CRL compared to regular RL. Researchers must instead use ad-hoc adaptations of RL benchmarks to achieve the desired setting. Proper virtual 3D simulation platforms and benchmarks are required to facilitate image-based CRL research. They need to be 1) lightweight, 2) well documented, 3) easy to install and use, 4) equipped with standardized metrics and baseline evaluations, and 5) computationally viable to run on small-scale systems and university-level budgets.
To this end, we present COOM (Continual DOOM), a CRL benchmark for embodied pixel-based learning on sequences of tasks with visually distinct 3D environments, designed to assess average performance, forgetting, and forward transfer. The visual modifications manifest in 1) wall and surface textures, 2) types, shapes, and sizes of in-game entities, and 3) modes to render objects. Compared to existing benchmarks for pixel-based learning oriented towards CL, COOM additionally includes a diverse set of objectives. To the best of our knowledge, this is the first benchmark specifically targeted towards CRL in complex 3D environments with differing objectives and visuals.
The contributions of our work are three-fold: 1) We assemble 6 task sequences of different lengths, half with task-dependent objectives, an aspect often absent in previous works. We further include a very complex sequence, intended as a challenge to the RL community to be solved with more advanced future algorithms. We include a demo of a trained agent performing COOM tasks. 2) We design 8 novel ViZDoom scenarios of two difficulties with contrasting visuals and dynamics to compose and publicly release COOM, a Continual RL benchmark. 3) Following the CL evaluation principles from [70], we employ multiple well-known CL methods for baseline evaluations on our task sequences, assessing prominent CL criteria. This makes our work the first to benchmark image-based CRL in 3D environments. We demonstrate how several methods fail to achieve the CL desiderata and how none come close to solving the complex sequence. We further elaborate on how the action distribution provides a more comprehensive understanding of the agent's continual progression. We conclude that critic regularization and single output head architectures tend to diminish performance in CL.
Most recent eminent CL platforms and frameworks, such as Sequoia [47], Avalanche [38], and Continuum [16] learn from static data sets of fixed sizes [35], and are thus not predominantly intended for RL. In CRL, the data sequence consists of different environments, and data samples are obtained through the agent-environment interaction. Sequoia introduces metrics and baselines aimed at CRL but only consists of simple environments like state-based manipulation tasks from
Table 1: Comparison of existing Continual Reinforcement Learning benchmarks with COOM.
Benchmark	3D	No. Task Sequences	Embodied View	Vision Transfer	Objective Transfer	Unified Metric
Continual World [70]	✓	2	×	×	✓	✓
L2Learner [23]	✓	0	✓	✓	✓	×
Jelly Bean World [52]	×	0	×	✓	×	×
CORA [53]	✓/×	4	✓/×	✓/×	✓/×	✓
CRLMaze [37]	✓	4	✓	✓	×	✓
COOM	✓	7	✓	✓	✓	✓
Meta-World [73], continuous control tasks from MuJoCo [66], and Monster Kong [62]. Avalanche RL [39] introduces a library for CRL, but only includes direct support for basic Atari [43] and Habitat v1.0 [58] environments. Moreover, it does not present any experimental results on baseline methods.
Most prominent modern RL platforms and benchmarks do not offer a fixed setting for CL [19, 24, 13, 63, 61, 51, 75, 14, 2]. The Atari benchmark [7] has often been employed to assess popular CL methods [27, 59, 55]. However, the games on the platform are deeply unrelated, lacking the potential for transfer. DeepMind Lab [6] provides a number of diverse 3D environments and MineRL [19] incorporates many of the elements found in lifelong learning [64]. Both have widely been used for assessing CL methods [56, 26, 55, 64]. However, neither include fixed task sequences or standardized CL metrics, requiring users to hand-pick existing tasks or design new environments. This causes a tedious experimental setup and provides no evident means of comparison to other works. DeepMind Lab and ProcGen [13] use procedural content generation (PCG), which renders them computationally expensive to run for CL. Platforms like AI2Thor [29] and iGibson [36], based on the Unity3D game engine, are good at replicating the complexity and visual fidelity of real-world problems. However, they are computationally expensive to simulate, and thus often infeasible for low-budget computational setups.
Previous CRL benchmarks often exhibit certain deficiencies. CRLMaze [37], based on ViZDoom, presents a non-stationary object-picking task, subject to visual changes. However, it lacks a comprehensive baseline evaluation of the most popular CL methods, does not undertake to change the objective, and only modifies three attributes (light, textures, objects). Continual World [70] assembles robotic manipulation tasks from Meta-World [73] to evaluate CRL and Jelly Bean World [52] uses PCG to generate a 2D gridworld test bed. However, neither addresses embodied AI nor image-based learning. L2Learner [23] creates a 3D PCG world to assess embodied agents, but lacks baseline evaluations and only consists of 5 tasks in a single environment. CORA [53] aggregates environments from Atari [7], ProcGen [13], MiniHack [57], and CHORES [29] into task sequences. The first three environments are all 2D, and the latter lacks diverse visuals. We refer readers to Table 1 for the comparison. Numerous metrics have been employed throughout CRL research that attempt to measure similar objectives [15, 10, 48]. The abundance and incoherence of metrics can create confusion in interpreting and disentangling results.
Reinforcement Learning from Images We formulate embodied image-based learning as a partially observable Markov decision process (POMDP) [8], in which an agent interacts with an environment over a fixed horizon of discrete time steps  $T$ . A POMDP can be described as a tuple  $(S, \mathcal{A}, p, R, \Omega, O, \gamma)$ . At each timestep  $t$ , the environment is in some state  $s_t \in S$ . By taking an action  $a_t \in \mathcal{A}$ , the agent causes the environment to transition to another state  $s_{t+1}$  with probability  $p = \mathbb{P}(s_{t+1}|s_t, a_t)$ . The agent cannot observe the full state  $s_t$  of the environment, but an observation  $o_t \in \Omega$  dependent on the action  $a_t$  taken and new state  $s_{t+1}$  reached with probability  $O(o|s_{t+1}, a)$ .  $\Omega$  is the high-dimensional set of pixel-observations. The agent receives a reward  $r_t = R(s_t, a_t)$ , which is mapped by the reward function  $R: S \times \mathcal{A} \to \mathbb{R}$  given a state and action. Finally,  $\gamma \in [0,1)$  is the discount factor determining how much immediate rewards are favoured over more distant rewards. The  $n$ -step return  $R_{t:t+n}$  at time step  $t$  is defined as the discounted sum of rewards,  $R_{t:t+n} = \sum_{i=1}^{n} \gamma^i r_{t+i}$ . The value function  $V^\pi(s) = \mathbb{E}\big[R_{t:T}|s_t = s, \pi\big]$  is the
Table 2: COOM scenarios. The core properties determine 1) how the agent's performance is measured; 2) whether enemies exist; 3) does the agent has a weapon; 4) do items spawn on the ground; 5) number of iterations in an episode; 6) which action can the agent execute apart from navigation; 7) what is randomized in the environment.
Scenario	Success Metric	Enemies	Weapon	Items	Max Steps	Execute	Stochasticity
Pitfall	Distance Covered	X	X	X	1000	JUMP	Pitfall tile locations
Arms Dealer	Weapons Delivered	X	✓	✓	1000	SPEED	Weapon and delivery locations
Hide and Seek	Frames Alive	✓	X	✓	2500	SPEED	Enemy behaviour, item locations
Floor is Lava	Frames Alive	X	X	X	2500	SPEED	Platform locations
Chainsaw	Kill Count	✓	✓	X	2500	ATTACK	Enemy and agent spawn locations
Raise the Roof	Frames Alive	X	X	X	2500	USE	Agent spawn location
Run and Gun	Kill Count	✓	✓	X	2500	ATTACK	Enemy and agent spawn locations
Health Gathering	Frames Alive	X	X	✓	2500	SPEED	Health kit spawn locations
expected return from state  $s$ , when actions are selected accorded to a policy  $\pi(a|s)$ . The action-value function  $Q^{\pi}(s, a) = \mathbb{E}\big[R_{t:T}|s_t = s, a_t = a, \pi\big]$  is the expected return following action  $a$  from state  $s$ . We aim to find a policy  $\pi(a_t|s_t)$  that maximizes the cumulative discounted return  $\mathbb{E}_{\pi}\big[\sum_{t=1}^{T} \gamma^t r_t |a_t \sim \pi(\cdot|s_t), s_t' \sim p(\cdot|s_t, a_t), s_1 \sim p(\cdot)\big]$ .
Continual Learning In the CL setting of this paper, we aim to learn a policy  $\pi_{\theta}$  by training an agent sequentially on a fixed task sequence  $\mathcal{T} = \{\mathcal{T}_1,\dots ,\mathcal{T}_n\}$ . The  $i^{th}$  task is trained during the interval  $t\in [(i - 1)\cdot \Delta ,i\cdot \Delta ]$ , where  $\Delta$  is the fixed number of iterations per task, during which the agent can only interact with the given environment. The agent thus obtains trajectories  $\tau_{i,1},\ldots ,\tau_{i,\Delta}$  from a single task to learn from, during which the POMDP to solve becomes  $(S_i,A,p_i,r_i,\gamma)$ .
4 COOM benchmark
The COOM (Continual DOOM) benchmark is based on ViZDoom [25], a flexible RL research platform for learning from raw visual information, based on the engine of the classical FPS video game Doom. ViZDoom has the benefit of being very lightweight, enabling gameplay up to 7000 FPS on a single GPU. The benchmark is comprised of 8 scenarios built in the Action Code Script (ACS) language. Every scenario is orientated towards accomplishing a particular objective, each having one or multiple aspects which make the environment stochastic. In Table 2, we display the core properties of each scenario: 1) the metric for measuring agent performance, 2) are there enemies in the environment 3) is the agent equipped with a weapon, 4) do items spawn on the ground, 5) how many iterations an episode lasts, 6) which is the execute action, and 7) what is randomized in the environment. A more detailed description of each scenario can be found in Appendix A. The limitations of COOM are addressed in Appendix L.
4.1 Basic Setup
Observations The agent is limited to only observing a portion of its surroundings, having a 90 degree horizontal field of view (FoV), which ranks lower than human vision [17] and modern cameras [74]. The observation space  $S$  is 4 stacked frames of  $160 \times 120$  pixels in a 4:3 resolution with 3 channels of 8-bit values in RGB, which represents the partially observed environment from the first person perspective of the embodied agent.
Action Space To facilitate the training of a CL task sequence using a single model, we require a unified action space across environments. We thus limit the full actions from the original Doom game to suit the scenarios of our benchmark. We obtain a multi-discrete action space  $\mathcal{A}$  for all environments by finding the Cartesian Product of singular actions  $\mathcal{A} = A_1 \times A_2 \times A_3$ , where  $A_1 = \{\text{MOVE FORWARD}, \text{NO-OP}\}$ ,  $A_2 = \{\text{TURN_LEFT}, \text{TURN_RIGHT}, \text{NO-OP}\}$ , and  $A_3 = \{\text{ATTACK}/\text{USE/SPEED}, \text{NO-OP}\}$ . The agent hence needs to choose one of  $|\mathcal{A}| = 12$  actions per iteration. Note that the execute operation  $A_3$  is scenario-dependent as indicated in Table 2.
Rewards The core component of the reward granted in each scenario is directly tied with the success metric (e.g., performance in Run and Gun increases when shooting a target, for which the
Figure 2: Constructing the hard COC sequence. We modify each CO8 task, leading to more sparse and elusive rewards, to impose a more complex CL challenge.
agent is also rewarded). We further utilize reward shaping for more granular dense feedback. A comprehensive description of reward functions is presented in Appendix B.
4.2 Task Sequences
We compose three lengths of sequences: 1) 8 unique tasks  $\mathbb{T}^8 = \{\mathcal{T}_0,\dots ,\mathcal{T}_7\}$ , 2) including only the second half of the former  $\mathbb{T}^4 = \{\mathcal{T}_4,\ldots ,\mathcal{T}_7\}$  to streamline the experimental process, considering the computationally intensive nature of training RL agents, and 3) repeating the sequence  $\mathbb{T}^{16} = \mathbb{T}^8 +\mathbb{T}^8$  (similar to [70]) allowing for revisiting each task in the same order. By default, the order of tasks in a sequence is predetermined. However, we provide the option for randomizing the sequence order, although this requires a substantially higher number of trials to ensure consistent results. This is due to 1) the performance of most continual learning methods being heavily dependent on the feature representation learned for the first task, and 2) the transferability of knowledge varying among different pairs of tasks. We distinguish between two sequence modalities.
Cross-Domain Sequence In the cross-domain setting, the agent is sequentially trained on modified versions of the same scenario. We select the Run and Gun scenario as the basis, as it best resembles the original Boom game, requiring the agent to navigate the map and eliminate enemies by firing a weapon. As depicted in Figure 1a, the objective and map layout remain the same across tasks, whereas we modify the environment by 1) changing wall, ceiling and floor textures, 2) varying enemy size, shape and type, 3) randomizing the view height of the agent, and 4) adding obstructions which block the agent's movement. The modifications are inspired by the LevDoom generalization benchmark [67]. We henceforth refer to the cross-domain sequences as CD4, CD8 and CD16.
Cross-Objective Sequence In addition to changing the visuals and dynamics within a scenario, we now design and employ new scenarios with novel objectives for consecutive tasks, as illustrated in Figure 1b. This presents a more diverse challenge, as the goal might drastically change from locating and eliminating enemies (Run and Gun and Chainsaw) to running away and hiding from them (Hide and Seek). This introduces a concept in CL commonly known as Task Interference. Similarly, the scenario Floor is Lava requires the agent to stay at a bounded location for good performance, whereas scenarios Pitfall, Arms Dealer, Raise the Roof, and Health Gathering encourage constant movement. We equivalently refer to the sequences as CO4, CO8 and CO16.
Cross-Objective Challenge There are two main dogmas in the RL community that dictate how to make progress in solving real-world decision-making problems [42]. The first is usually referred to as the generalizable agent view, in which focus should be attended towards the large-scale training of agents that solve diverse problems, hoping that along the way, a generalist agent will evolve. Our previously defined two task sequences endorse this philosophy. The second view, generally referred to as deployable RL, takes a more pragmatic view by seeking to design RL algorithms that solve particular difficult problems. To facilitate the advancement of the latter principle, we include a task sequence in our benchmark which serves as a challenge, aimed foremost to initially be solved. We posit that harder tasks are easier to forget in a CL setting. To this end, we create a hard counterpart
Table 3: Results of Average Performance (AP), Forgetting (F) and Forward Transfer (FT) across 10 seeds. Extended results with  $95\%$  confidence intervals are presented in Tables 19 of Appendix N. The result of the best performing method is highlighted in bold.
Method	CD4	CO4	CD8	CO8	COC	Average
AP	F	FT	AP	F	FT	AP	F	FT	AP	F	FT	AP	F	FT	AP	F	FT
PackNet	0.92	0.00	0.40	0.87	0.01	-0.24	0.91	-0.01	0.19	0.82	-0.01	0.25	0.20	0.04	0.08	0.74	0.01	0.13
MAS	0.55	0.50	0.11	0.72	0.24	-0.04	0.82	0.14	0.25	0.58	0.19	0.01	0.04	0.09	0.02	0.54	0.23	0.07
AGEM	0.35	0.80	0.10	0.42	0.80	0.03	0.30	0.86	0.17	0.30	0.84	0.23	0.02	0.11	0.02	0.28	0.68	0.11
L2	0.71	0.01	-0.28	0.80	0.00	-0.60	0.87	-0.03	0.07	0.71	-0.04	-0.32	0.10	0.00	0.00	0.64	-0.02	-0.23
EWC	0.69	0.00	-0.41	0.65	0.05	-0.77	0.76	-0.04	-0.55	0.65	0.05	-0.38	0.07	0.00	-0.01	0.56	0.00	-0.43
VCL	0.46	0.77	0.21	0.40	0.82	-0.57	0.36	0.73	0.04	0.39	0.64	0.20	0.03	0.16	0.05	0.33	0.62	-0.01
Fine-tuning	0.59	0.63	0.32	0.50	0.71	-0.01	0.45	0.75	0.28	0.44	0.48	0.23	0.02	0.10	0.02	0.40	0.53	0.17
ClonEx-SAC	0.87	0.00	0.11	0.86	-0.03	-0.26	0.92	-0.03	0.13	0.89	0.00	0.27	0.13	0.01	0.03	0.73	-0.01	0.06
Perfect Memory	0.89	-0.01	0.30	0.89	0.02	0.03	-	-	-	-	-	-	-	-	-	-	-	-
for each task in CO8, as illustrated in Figure 2. We will further refer to this sequence as COC, which is further elaborated on in Appendix F.
4.3 Evaluation Protocol
The agent is sequentially trained on each task  $\mathcal{T}_i$  for  $\Delta = 200\mathrm{K}$  iterations. After every 1000 iterations, we evaluate the policy on each task of the sequence for 10 episodes.  $50\mathrm{K}$  most recent trajectories are stored in the replay buffer, which is emptied at the end of each task by default. The Cross-Domain sequences are based on the Run and Gun scenario, in which Kill Count is the core measure of performance. In the Cross-Objective sequences, however, the agent is expected to achieve different goals across tasks. We are hence unable to directly use a single metric to adequately compare performance across tasks. As it would be tedious and convoluted to compare the results between methods on each task from a Cross-Objective sequence individually, we define a unified metric success  $= f_{i}(score)\in [0,1]$ , where score represents the task metric (see Table 2) and  $f_{i}$  is a function for task  $i$  that transforms the score to success. The lower bound  $f_{i}(score) = 0$  on  $\mathcal{T}_i$  is determined by the performance of a random agent, and the upper bound  $f_{i}(score) = 1$  by SAC trained until convergence.
Average Performance  $P$  is the core evaluation metric in COOM and is measured across the number of total sequence iterations  $T = |\mathcal{T}| \cdot \Delta$  averaging the success of all tasks  $\mathcal{T}$  at each time step  $t$ :
$$
P = \frac {1}{| \mathcal {T} |} \frac {1}{T} \sum_ {\mathcal {T} _ {i} \in \mathcal {T}} \sum_ {t = 0} ^ {T} f _ {i} (s c o r e _ {t}). \tag {1}
$$
Forgetting  $F$  quantifies the average decrease of success from the end of training each task  $i$  to the end of the entire sequence  $T$ . For consistency, we consider  $k$  last iterations of a task:
$$
F = \frac {1}{| \mathcal {T} | - 1} \sum_ {i = 0} ^ {| \mathcal {T} | - 1} \frac {1}{m} \sum_ {j = 0} ^ {k - 1} f _ {i} \left(\operatorname {s c o r e} _ {(i + 1) \cdot \Delta - j}\right) - \frac {1}{m} \sum_ {j = 0} ^ {k - 1} f _ {i} \left(\operatorname {s c o r e} _ {T - j}\right). \tag {2}
$$
Note that we exclude the final task, as no forgetting can have occurred. In the case of negative values for forgetting  $(F < 0)$  we observe positive backward transfer.
Forward Transfer FT is measured by first finding the area under the training curve  $AUC_{i}$  on task  $i$  for the CL method and for the SAC baseline  $AUC_{i}^{b}$  with
$$
A U C _ {i} = \frac {1}{\Delta} \int_ {i - 1 \cdot \Delta} ^ {i \cdot \Delta} f _ {i} (s c o r e _ {t}) d t, A U C _ {i} ^ {b} = \frac {1}{\Delta} \int_ {0} ^ {\Delta} f _ {i} ^ {b} (s c o r e _ {t}) d t. \tag {3}
$$
We can then first find the average normalized area between the curves across all tasks. Note that we do not consider the first task, as there is no knowledge to transfer forward from previous tasks:
$$
F T = \frac {1}{| \mathcal {T} | - 1} \sum_ {i = 1} ^ {| \mathcal {T} |} \frac {A U C _ {i} - A U C _ {i} ^ {b}}{1 - A U C _ {i} ^ {b}}. \tag {4}
$$
5.1 Setup
In this section, we assess the CL desiderata of baselines from three different families of popular CL methods on our task sequences. Regularization-based methods constrain weight updates in order to maintain knowledge from previous tasks. L2 [27] adds a simple  $L_{2}$  penalty to achieve this. MAS [3] utilizes a weighted penalty on each parameter depending on its importance to the network
Figure 3: Visual analysis of CO8 results. 1) The average cumulative performance curves indicate a high disparity across baselines (top). 2) ClonEx-SAC effectively mitigates forgetting according to the continual evaluation curves of individual tasks (middle). The tasks are very diverse, as no notable performance is reached before exposure to the task itself. 3) ClonEx-SAC successfully transfers knowledge forward compared to the vanilla SAC, which is trained on each task from scratch (bottom). The blue area between the curves indicates positive forward transfer and red represents its negative counterpart.