SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In addition to maintaining the core actions required for navigating within the game environment, we kept a single additional action called for each scenario, which we referred to as EXECUTE. For instance, in some scenarios, the action resulted in the agent accelerating, while in others, it corresponded to firing the weapon. This approach allowed us to condense the action space even further, particularly in cross-objective sequences where certain actions would have otherwise been rendered unnecessary or redundant in some tasks.
However, this potentially confuses the agent both when selecting the best action during training, but also, when attempting to maintain performance on learned tasks, in which performing an action on a previous task from the given space would result in a different action in-game compared to the one in the task being trained. On the other hand, allowing the agent to have a broader range of actions at its disposal may have the potential to enhance overall performance, especially when employing more sophisticated and powerful methods. A wider variety of actions would enable the agent to explore different strategies and potentially discover more effective approaches to achieving the objectives.
Sequence length The number of tasks in our benchmark is relatively small, and they are not repeated. Introducing task repetition in a continual learning (CL) benchmark can yield additional interesting observations. However, due to the extensive runtime requirements for low-level budget setups, we opted to avoid long sequences in this benchmark.
The dimensionality of the observation space and the rendering of the environment contribute to longer learning times for a single task. We established a fixed task length of 200K timesteps, as our observations indicated that this was sufficient for reaching convergence. In our limited experiments with longer task lengths, we found that no method exhibited notable performance improvement after this number of iterations nor managed to learn anything meaningful beyond this point.
While longer task lengths and task repetition could provide valuable insights, the constraints of our experimental setup, including limited resources and time, prevented us from exploring these aspects in-depth. However, future studies with more resources and computational power could investigate these factors and potentially uncover further insights in the context of continual learning benchmarks.
Task boundaries In our benchmark, we adopted the use of explicit task boundaries to define the transitions between different tasks. While this approach provides a clear and well-defined setting for evaluation, it does come with certain limitations.
By relying on explicit task boundaries, we assume that the learner has prior knowledge or external cues to identify and differentiate between tasks. In real-world scenarios, however, the presence of explicit task boundaries may not be readily available or easily discernible. Therefore, extending
the benchmark to incorporate task inference mechanisms would be a valuable direction for future research.
Task inference mechanisms enable the learner to autonomously detect task boundaries without explicit supervision. Incorporating such mechanisms in the benchmark would allow for a more comprehensive evaluation of CL algorithms that possess the capability to infer task transitions.
Additionally, the current format of the benchmark does not explicitly test algorithms' ability to handle continuous distributional drift. The absence of continuous drift scenarios limits the assessment of CL methods in dynamic and evolving environments. Future iterations of the benchmark could include settings that introduce continuous distributional drift, thereby enabling the evaluation of algorithms that can adapt to changing data distributions over time.
M Action Distributions
Apart from visualizing the actions executed during training, it is more relevant for the CL conundrum to observe how a particular method manages to preserve the obtained distribution after training on a particular task has concluded and the sequence continues with new tasks being introduced. To this end, we depict the distribution of actions continuously performed by PackNet and L2 on each task within the CO8 task sequence in Figures 16 and 17.
Given the experimental results in Table 3, we know that PackNet and L2 significantly outperformed VCL in both average performance and forgetting. Observing the figures on action distributions, this manifests in VCL being noticeably incapable of maintaining the obtained distribution as new tasks come along, whereas PackNet and L2 have minimal fluctuation after the policy has been adjusted for a given task. We can notice that although PackNet and L2 both had a relatively high performance on the task sequence, the distributions for particular tasks can drastically vary. This suggests that there are a multitude of possible policies for acquiring adequate results on a task. For instance, there are two clearly dominating actions which PackNet utilizes in Arms Dealer, whereas L2 has more variety in the selection of actions, with MOVE_FORWARD being the most used one. Interestingly, L2 also has low variance in the distribution before a task has been encountered, whereas for PackNet it is more random. The L2 regularization after the first task seems to fix the distribution also for all other future tasks.
EXECUTE
MOVE FORWARD
MOVE_FORWARD, EXECUTE
TURN RIGHT EXECUTE
TURN RIGHT, MOVE FORWARD
TURN_RIGHT, MOVE_FORWARD, EXECUTE
TURN LEFT, EXECUTE
TURN LEFT, MOVE FORWARD
TURN_LEFT, MOVE_FORWARD, EXECUTE
Figure 16: Actions executed by PackNet on CO8.
TURN RIGHT EXECUTE
TURN RIGHT, MOVE FORWARD
TURN_RIGHT, MOVE_FORWARD, EXECUTE
Figure 17: Actions executed by L2 on CO8.
TURN LEFT EXECUTE
TURN LEFT, MOVE FORWARD
TURN_LEFT, MOVE_FORWARD, EXECUTE
NO-OP
MOVE FORWARD
MOVE_FORWARD, EXECUTE
Figure 18: Actions executed by VCL on CO8.
TURN RIGHT
TURN RIGHT, EXECUTE
TURN RIGHT, MOVE FORWARD
TURN_RIGHT, MOVE_FORWARD, EXECUTE
TURN LEFT
TURN LEFT, EXECUTE
TURN LEFT, MOVE FORWARD
TURN_LEFT, MOVE_FORWARD, EXECUTE
N Extended Results and Plots
In Table 19 we present the main experimental results with  $95\%$  confidence intervals.
Sequence	PackNet	MAS	AGEM	L2	EWC	VCL	Fine-tuning	ClonEx-SAC	Perfect Memory
CD4	0.92 ± 0.02	0.55 ± 0.17	0.35 ± 0.12	0.71 ± 0.13	0.69 ± 0.08	0.46 ± 0.14	0.59 ± 0.12	0.87 ± 0.03	0.89 ± 0.05
CO4	0.87 ± 0.07	0.72 ± 0.08	0.42 ± 0.13	0.80 ± 0.08	0.65 ± 0.11	0.40 ± 0.14	0.50 ± 0.10	0.86 ± 0.04	0.89 ± 0.06
CD8	0.91 ± 0.05	0.82 ± 0.12	0.30 ± 0.10	0.87 ± 0.06	0.76 ± 0.11	0.36 ± 0.13	0.45 ± 0.11	0.92 ± 0.03	-
CO8	0.82 ± 0.09	0.58 ± 0.17	0.30 ± 0.13	0.71 ± 0.13	0.65 ± 0.13	0.39 ± 0.18	0.44 ± 0.11	0.89 ± 0.07	-
COC	0.20 ± 0.08	0.04 ± 0.02	0.02 ± 0.01	0.10 ± 0.05	0.07 ± 0.04	0.03 ± 0.02	0.02 ± 0.01	0.13 ± 0.05	-
Average	0.74 ± 0.06	0.54 ± 0.11	0.28 ± 0.10	0.64 ± 0.09	0.56 ± 0.09	0.33 ± 0.12	0.40 ± 0.09	0.73 ± 0.05	-
(a) Performance
Sequence	PackNet	MAS	AGEM	L2	EWC	VCL	Fine-tuning	ClonEx-SAC	Perfect Memory
CD4	0.00 ± 0.00	0.50 ± 0.24	0.80 ± 0.06	-0.02 ± 0.00	0.00 ± 0.01	0.77 ± 0.08	0.63 ± 0.19	0.00 ± 0.00	-0.01 ± 0.01
CO4	0.01 ± 0.01	0.24 ± 0.13	0.80 ± 0.22	0.00 ± 0.00	0.04 ± 0.05	0.82 ± 0.09	0.71 ± 0.06	-0.03 ± 0.02	0.02 ± 0.01
CD8	0.01 ± 0.00	0.14 ± 0.12	0.86 ± 0.02	-0.03 ± 0.03	-0.04 ± 0.03	0.73 ± 0.10	0.75 ± 0.07	-0.03 ± 0.02	-
CO8	-0.01 ± 0.02	0.17 ± 0.03	0.84 ± 0.03	-0.04 ± 0.02	0.02 ± 0.02	0.64 ± 0.09	0.48 ± 0.02	0.00 ± 0.01	-
COC	0.04 ± 0.00	0.09 ± 0.03	0.11 ± 0.05	0.00 ± 0.01	0.00 ± 0.00	0.16 ± 0.06	0.10 ± 0.04	0.01 ± 0.00	-
Average	0.01 ± 0.00	0.23 ± 0.10	0.68 ± 0.03	-0.02 ± 0.01	0.00 ± 0.01	0.62 ± 0.06	0.53 ± 0.06	-0.01 ± 0.01	-
(b) Forgetting
Sequence	PackNet	MAS	AGEM	L2	EWC	VCL	Fine-tuning	ClonEx-SAC	Perfect Memory
CD4	0.40 ± 0.08	0.11 ± 0.32	0.10 ± 0.21	-0.28 ± 0.32	-0.41 ± 0.10	0.21 ± 0.29	0.32 ± 0.19	0.11 ± 0.05	0.30 ± 0.15
CO4	-0.24 ± 0.32	-0.04 ± 0.06	0.03 ± 0.10	-0.60 ± 0.37	-0.77 ± 0.13	-0.57 ± 0.30	-0.01 ± 0.15	-0.26 ± 0.09	0.03 ± 0.20
CD8	0.19 ± 0.25	0.25 ± 0.03	0.17 ± 0.21	0.07 ± 0.05	-0.55 ± 0.16	0.04 ± 0.08	0.28 ± 0.11	0.13 ± 0.05	-
CO8	0.25 ± 0.05	0.01 ± 0.10	0.23 ± 0.02	-0.32 ± 0.07	-0.38 ± 0.13	0.20 ± 0.06	0.23 ± 0.04	0.27 ± 0.10	-
COC	0.08 ± 0.01	0.02 ± 0.00	0.02 ± 0.00	-0.00 ± 0.01	-0.01 ± 0.00	0.05 ± 0.02	0.02 ± 0.00	0.03 ± 0.01	-
Average	0.13 ± 0.14	0.07 ± 0.10	0.11 ± 0.11	-0.23 ± 0.16	-0.43 ± 0.10	-0.01 ± 0.15	0.17 ± 0.10	0.06 ± 0.06	-
(c) Transfer
Figure 19: Extended results of average performance, forgetting and forward transfer with  $95\%$  confidence intervals across 10 seeds. The result of the best performing method is highlighted in bold. In the case of ties, the one with the highest confidence in selected.
Figure 20: Continual evaluation results on the CD4 sequence.
Figure 21: Continual evaluation results on the CO4 sequence.
Figure 22: Continual evaluation results on the CD8 sequence.
Figure 23: Continual evaluation results on the CO8 sequence.
Figure 24: Forward transfer results on the CO8 sequence.