SlowGuess/NeurIPS_2023_OCR huggingface dataset:
output. EWC [27] uses the Fisher information matrix to approximate the importance of each weight. VCL [45] uses variational inference to minimize the KL divergence between the prior and posterior distribution of parameters. Structure-based methods preserve and update the network architecture to efficiently learn and adapt to new tasks over time. PackNet [41] forbids any changes to parameters that are important for previous tasks by freezing the most relevant weights at the end of each task and pruning the rest. Rehearsal-based methods retain samples from previous tasks to constrain forgetting. Perfect Memory [70] refrains from resetting the buffer after a task and thus remembers everything. A-GEM [11] projects gradients from new samples as to not interfere with previous tasks. ClonEx-SAC [71] retains some samples from previous tasks and performs behavior cloning based on them to reduce forgetting. We also include the naive Fine-tuning baseline, which simply continues regular weight updates on a new task. All the CL methods take the efficient and well-known Soft Actor-Critic (SAC) [20] as an RL training backbone. We follow the evaluation protocol outlined in Section 4.3 to measure performance of all baselines on CD4, CO4, CD8, CO8 and COC with fixed orders across 10 seeds controlling the pseudorandom nature of the environments. We refrain from using random sequence orders due to the necessity for numerous runs to ensure consistent evaluation, a demand that exceeds our available computational resources. Similarly, we exclude Perfect Memory on 8-task sequences due to the unfeasible memory requirements for storing all trajectories. We grant the agent full access to the task identity both at training and test time, a CL setting coined as Task-Incremental Learning [69]. We use  $95\%$  confidence intervals for all results. Our framework is discussed in finer detail in Appendix C.
Performance PackNet, closely followed by ClonEx-SAC, triumphed across all sequences according to the results in Table 3. We conjecture PackNet's success on our benchmark to two key factors: 1) knowing the task identity during evaluation, and 2) short task sequences with distinct boundaries, as PackNet needs to assign a fixed fraction of parameters to each task. It is noteworthy that the top three best-performing methods (PackNet, ClonEx-SAC, and L2) originate from different families of CL,
Figure 4: SAC does not exhibit a loss of network plasticity on COOM. Measuring the individual performance of each task across 5 seeds displays that repeated exposure to the same tasks throughout 10 iterations of the CO8 sequence does not hinder the capacity to reattain peak performance.
of Training steps (100K for Each Task)
indicating that each approach has a strong potential for achieving excellent results in the benchmark. Although Perfect Memory performed well on the 4-task sequences, it required nearly twice as long to run, and took up 16x more memory in the process. Memory consumption and walltime is further discussed in Appendix E. The average success curves in Figure 3 illustrate a significant disparity in performance among our baselines. Despite an exhaustive exploration of hyperparameters, we were unable to attain satisfactory results with AGEM and VCL. These methods ranked as the poorest performers in our benchmark, remarkably even falling below the naïve fine-tuning approach. Comparing results across sequence lengths, almost all methods showed higher results in CO4 than CO8, whereas among the cross-domain sequences, the top four methods had a substantially higher performance on the longer version. Performance related to sequence length is more extensively analysed in Appendix G. The performance difference between CD and CO sequences are not that staggering, indicating that visual perturbations alone, without a changing objective in our embodied 3D environments, are sufficiently challenging for our CRL baselines.
Forgetting The top four baselines excel at preserving performance on previously learned tasks, approaching near-perfection. However, out of these four, L2 and EWC have a substantially lower performance, indicating that regularization-based methods predominantly emphasize stability at the expense of plasticity. Despite an extensive hyperparameter search for weaker regularization, we were not able to achieve a more favorable stability-plasticity trade-off. In particular, a lower regularization constant exhibited a loss of stability but no noticeable further gain in plasticity. In comparison to good stability, AGEM and VCL experience a decline in average performance immediately after losing access to a learned task. In terms of forgetting particular tasks, Pitfall and Arms Dealer appear the hardest to remember given our fixed ordering of tasks in the CO sequence, as analyzed in Appendix D.
Transfer We attribute the core capacity for transfer in our benchmark environments to navigation. Success in most tasks heavily relies on the ability to navigate effectively (e.g., repeatedly running into walls rarely contributes to a high reward). Being able to identify objects and entities from their surroundings also holds significant importance for most tasks (e.g., detecting useful items to collect). The training success curve of ClonEx-SAC in Figure 3 delineates the poor sample-efficiency of the vanilla SAC which is unable to attain comparable performance on several tasks by learning them from scratch. Regularization-based methods fall significantly below others in terms of utilizing acquired competencies to effectively learn future tasks. We did not observe notable backward transfer in our experiments.
5.3 Network Plasticity
Recent research has emphasized the importance of network plasticity in RL [1, 40, 46, 33]. To further enhance our analysis and provide a comprehensive perspective on the ability to adapt to new targets on our benchmark. Similar to the experimental design in [1], we aim to investigate whether our base SAC agent experiences a decline in network plasticity when confronted with the Cross-Objective task setting. Intriguingly, unlike the trend observed in [1], our findings diverge given the learning curves of CO4 in Figure 4, where the agent cyclically repeats the sequence 10 times. The agent is able to reach similar results in equal duration throughout the iterations without any noticeable deterioration. We therefore conclude that our base SAC agent does not exhibit a decline in network plasticity when subjected to our benchmark. An avenue for future research would be to investigate whether other
Figure 5: PackNet quickly adapts its policy to fit a new task when training on CO8. The stackplot (left) depicts the distribution of actions executed in the environment. For example, in Pitfall, PackNet highly favours moving forward while accelerating to reap the highest return. The histogram (right) indicates the mean number of action executions across the entire sequence. Actions constituting maximal sub-actions appear to be most optimal according to PackNet.
methods, such as model-based approaches, exhibit a similar pattern. The results of CO8 and further analysis is included in Appendix H.
5.4 Action Distributions
Since the agent has a fixed multi-discrete action space across all scenarios, we can visualize how the action distribution shifts over time as training progresses to new tasks. We plot the actions executed by PackNet on CO8 in Figure 5. PackNet's high performance suggests it has obtained an effective action distribution. We can observe how PackNet quickly manages to adjust its policy to select appropriate actions when presented with a new task. The action distribution indicates, that on, average executing multiple sub-actions in a single timestep is more beneficial. In Appendix M we further visualize how our baselines are able to maintain the acquired action distribution on each given task.
5.5 Method Variations
Image Augmentation Using augmented images as input has been recognized for its potential to enhance data diversity, mitigate catastrophic forgetting, and facilitate adaptation to changes in data probability distribution [21]. We explore the impact of incorporating three established image augmentation methods for RL to our selected baselines on COOM: 1) Random Convolution [34] randomly perturbs input observations, 2) Random Shift [31] pads each image with 4 pixels (repeating boundary pixels) and then applies a random crop, yielding the original image shifted by  $\pm 4$  pixels, 3) Random Noise [32] injects Gaussian noise into the image.
In our comparative analysis depicted in Figure 6, we can observe that random shifts and noise have minimal impact on the performance of our baselines, whereas random convolutions significantly degrade all methods except PackNet. They lead to substantial imbalances in the data distribution for initial tasks, causing the policy to converge to a local minimum from which it struggles to recover. This phenomenon is especially pronounced in regularization-based methods that rely on maintaining stable feature representations, which are highly susceptible to disruption caused by random convolutions due to their rigid architecture and stringent regularization. Additionally, these perturbations affect the data distribution stored in the memory of rehearsal-based methods, diminishing their effectiveness for future task learning. Our findings indicate that PackNet can better recover from a degenerate policy stuck in local minima, attributed to freeing up redundant parameters after training on a task. This suggests that architecture-based methods exhibit greater flexibility in the face of significant changes in data distribution.
LSTM We include an LSTM encoder atop the CNN head in our model architecture to assess its impact on our baseline methods. Employing a similar approach to a supervised CL setting has previously been shown to be beneficial [18]. On our benchmark, however, this approach completely
Figure 6: Performance comparison of potential method enhancements. Key observations: 1) While most baselines are unable to learn a single task using an LSTM encoder, AGEM reaps a substantial benefit, and PackNet remains indifferent. 2) Critic regularization has a detrimental effect on CL, particularly on regularization-based methods. 3) Methods performing well on COOM suffer less from only a single head output. 4) Attributed to weight-pruning, PackNet demonstrates superior recovery from a degenerate policy under an imbalanced data distribution with random convolutions.
deteriorates the performance of most methods, rendering them unable to learn anything meaningful. Strikingly, PackNet's performance remained consistent, whereas AGEM experienced a substantial increases. We hypothesize that, by virtue of capturing sequential patterns, an LSTM encoder can facilitate more accurate and efficient retrieval of relevant experiences from the episodic memory. This improved retrieval could enable AGEM to make better use of past knowledge when confronted with new tasks. However, more in-depth analysis is required to confirm this.
Critic Regularization In the context of actor-critic frameworks like SAC, there is a design decision regarding whether to regularize the critic. In some CL methods, the critic is only used during training and only needs to store the value function for the current task, making forgetting less of a concern. However, certain methods, such as replay-based approaches, may not easily accommodate a framework where only the actor is regularized. We thus investigate critic regularization affects our baselines. Figure 6 confirm the findings from [70], that regularizing the critic network has a harmful impact for CL. This suggests that it is better to let the critic adapt freely to the current task while prioritizing the minimizing of forgetting in the actor. However, this means that the critic needs to be trained from scratch, which is not particularly efficient when tasks are repeated.
Output Heads In our default setup, we use a separate output head for each task, which is known to help prevent forgetting [70]. However, this approach is inapplicable to dynamic scenarios with and unknown number of tasks. Figure 6 depicts how a single-head architecture impacts our baselines. All methods show a decline in performance, however ClonEx-SAC, PackNet and VCL manage to better maintain their original performance. Extended results are presented in Appendix I.
Training proficient agents, who are able to continually adapt to learning new tasks in an altering environment, currently remains one of the greatest challenges in RL. To aid the community in grappling with this challenge, we presented and openly released COOM, a novel benchmark for assessing CL in visually distinct 3D environments with diverse objectives. COOM comprises seven task sequences of varying length, complexity, and modality, effectively demonstrating the strengths and weaknesses of popular continual learning methods. Our experimental evaluations reveal the impact of different variations to baseline methods, highlighting the importance of critic regularization and task-dependent output architectures. We visually illustrate how high-performing methods quickly adapt their policies and maintain learned action distributions across tasks within a sequence. All the methods in our evaluations struggle with the complex COC sequence, making it a valuable testing ground for future research, showcasing both learning challenges and transfer difficulties. COOM serves as a crucial tool to inspire the development of more capable algorithms and we expect it to facilitate the evaluation and comprehension of future continual reinforcement learning methods.
This work was conducted with the assistance of the Dutch national e-infrastructure, generously supported by the SURF Cooperative through grant number EINF-5755.
[66] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[67] Tristan Tomilin, Tianhong Dai, Meng Fang, and Mykola Pechenizkiy. Levdoom: A benchmark for generalization on level difficulty in reinforcement learning. In 2022 IEEE Conference on Games (CoG), pages 72-79. IEEE, 2022.
[68] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Kuttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782, 2017.
[69] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023.
[70] Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłos. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 34:28496-28510, 2021.
[71] Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłos. Disentangling transfer in continual reinforcement learning. Advances in Neural Information Processing Systems, 35:6304-6317, 2022.
[72] Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 79-86. IEEE, 2017.
[73] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020.
[74] Zichao Zhang, Henri Rebecq, Christian Forster, and Davide Scaramuzza. Benefit of large field-of-view cameras for visual odometry. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 801-808. IEEE, 2016.
[75] Artem Zholus, Alexey Skrynnik, Shrestha Mohanty, Zoya Volovikova, Julia Kiseleva, Artur Szlam, Marc-Alexandre Coté, and Aleksandr I Panov. Iglu gridworld: Simple and fast environment for embodied dialog agents. arXiv preprint arXiv:2206.00142, 2022.
[76] Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564, 2018.
A Scenario Descriptions
In this section, we provide a description for each scenario used in our benchmark. We outline and the reward functions.
Pitfall In this scenario, the agent is located in a very long rectangular corridor. The surface of the area is divided into squares, of which there are 7 in each row. At the beginning of an episode, each of those sections has a  $30\%$  chance of being turned into a pit by lowering the height of the floor in the given area. The floor layout is randomized in every episode, so there is no one optimal route to take. Falling into a pit instantly terminates the episode and should be avoided at all costs. Reaching the other end of the corridor also terminates the episode. The agent is tasked to traverse along the corridor as far as possible and potentially reach the other end. Note that there is a  $\sim 10^{-5}$  chance that a section of the corridor becomes untraversable due to the stochastic nature of the layout. The agent ought to identify a safe path through the corridor and speedily navigate forwards. The agent can turn left and right, move forward, and accelerate. The agent is proportionally rewarded for how much it has travelled forward in the corridor, and penalized for falling into a pit.
Arms Dealer In this scenario, the agent has to pick up weapons from the ground in a square-shaped room and deliver them to a platform that appears at a random location after a weapon is obtained. The agent can carry multiple (the exact amount is dependent on the weapon) weapons before making the delivery. A new weapon is spawned at a random location whenever one is picked up, i.e., there is always a constant number of weapons on the ground. A new platform is created for every weapon picked up. After a successful delivery, all the platforms disappear, and the agent has to start over by collecting weapons. The agent can turn left and right, move forward, and accelerate. The agent is rewarded for both obtaining a weapon and delivering it. Additionally, a small reward is granted for how much the agent moves in the environment. Finally, the agent is penalized for being idle.
Hide and Seek In this scenario, the agent is randomly spawned in one of 20 possible locations within a maze-like environment. 5 enemies are spawned at random locations at the beginning of an episode. The enemies can only inflict damage at close distance and thus constantly attempt to move closer to the agent. Health kits granting 25 hit points continually spawn in random locations at specified time intervals. The objective of the agent is to survive by hiding from the enemies. The agent should identify the enemies and attempt to constantly move away from them while collecting the health kits when at low health. The agent can turn left and right, move forward, and run. The agent is rewarded for movement, surviving a frame and each health item collected, and penalized for having damage inflicted by enemies.
Floor is Lava In this scenario, the agent is located in a rectangular room divided into 16x16 equal sized squares. The room is filled with lava, which inflicts 1 health point of damage if being stood upon. After every fixed time interval, each square section of lava has a  $20\%$  chance of being changed to a platform, which no longer causes damage. The objective is to survive by minimizing the time spent standing in lava. The agent ought to identify the platforms and quickly navigate on top of them as soon as they appear to avoid running out of health. The agent can turn left and right, move forward, and accelerate. The agent is rewarded for surviving a frame, stepping onto a platform whilst having previously stood in lava, and penalized for standing in lava.
Chainsaw In this scenario, the agent is randomly spawned in one of 20 possible locations within a maze-like environment, and equipped with a chainsaw. A fixed number of enemies are spawned at random locations at the beginning of an episode. Additional enemies will continually be added at random unoccupied locations after each time interval. The enemies are rendered immobile, forcing them to remain at their fixed locations. The goal of the agent is to find the enemies, walk up to melee distance from them and saw them in half. The agent can move forward, turn left and right, and use the chainsaw. The agent is granted a reward for moving in the environment and each enemy killed.
Raise the Roof In this scenario, the agent is randomly spawned in one of 20 possible locations within a maze-like environment. The room has a very high ceiling at the beginning of an episode, which, however, starts to be slowly but constantly lowered. At specific locations of the area, there are switches on the walls that can be pressed to raise the ceiling back up a bit. After a switch is pressed,
it disappears and can no longer be used. If the switches are not pressed with a high enough frequency, the ceiling will eventually crush the agent, which terminates the episode. The goal of the agent is thus to locate and press the switches to keep the ceiling high before the episodes timeouts. The agent can move forward, turn left and right, and activate a switch. The agent is rewarded for movement, pressing a switch, and surviving for a frame.
Run and Gun In this scenario, the agent is randomly spawned in one of 20 possible locations within a maze-like environment, and equipped with a weapon and unlimited ammunition. A fixed number of enemies are spawned at random locations at the beginning of an episode. Additional enemies will continually be added at random unoccupied locations after each time interval. The enemies are rendered immobile, forcing them to remain at their fixed locations. The goal of the agent is to locate and shoot the enemies. The agent can move forward, turn left and right, and shoot. The agent is granted a reward for each enemy killed.
Health Gathering In this scenario, the agent is trapped in a room with a surface, which slowly but constantly decreases the agent's health. Health granting items continually spawn in random locations at specified time intervals. The default health item heals grants 25 hit points. Some environments contain poison vials, which inflict damage to the agent instead of providing health. The objective is to survive. The agent should identify the health granting items and navigate around the map to collect them quickly enough to avoid running out of health. The agent can turn left and right, and move forward. A small reward is granted for every frame the agent manages to survive.
B Reward Shaping
In this section, we present all components constituting the auxiliary reward function  $r$  of each scenario.
Raise the Roof  $r_t = F + c_1 \cdot s + c_2 \cdot \left\| l_t - l_{t-5} \right\|_2^2$ , where  $F = 0.01$  is a constant reward granted for surviving an iteration,  $s$  is the number of switches pressed,  $l$  represents the coordinates of the agent's location,  $c_1 = 15$  is the reward of pressing a switch,  $c_2 = 0.001$  scales the reward granted for movement, and  $t$  is the time step.
**Chainsaw**  $r_t = c_1 \cdot k_t + c_2 \cdot \left\| l_t - l_{t-5} \right\|_2^2$ , where  $k$  is the number of enemies eliminated,  $l$  represents the coordinates of the agent's location,  $c_1$  is the reward for eliminating an enemy,  $c_2 = 0.001$  scales the reward granted for movement, and  $t$  is the time step.
Health Gathering  $r_t = F + c_1 \cdot \mathbb{1}\{h_t > h_{t-1}\}$ , where  $F = 0.01$  is a constant reward granted for surviving an iteration,  $h$  is the remaining health (increases when a health kit is picked up),  $c_1 = 15$  scales the reward granted for picking up a health kit, and  $t$  is the time step.