SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CLS-ER [6] emulates the interplay between fast and slow learning systems by incorporating two supplementary semantic memories that aggregate the weights of the working model in a stochastic manner via an exponential moving average. That is, CLS-ER operates with three models: the working model, the stable model, and the plastic model. Conversely, in the TriRE framework, the architecture consists of two models: the working model and a stable model (referred to as the EMA model). Therefore, the similarity is that both methods use the concept of progressively aggregating the weights of the working memory as it sequentially learns tasks allowing us to consolidate the information efficiently.
However, there are two main differences. Firstly, CLS-ER uses two supplementary semantic memories, whereas TriRE only utilizes one supplementary memory. Nevertheless, our results outperform CLS-ER (with one EMA model, referred to as Mean-ER in the mentioned paper) in all three datasets, as shown in Table 1. Secondly, CLS-ER only uses experience replay to tackle catastrophic forgetting whereas our method harmoniously integrates all the families of methods in CL, i.e. weight regularization, parameter isolation, and experience replay. That is, CLS-ER only focuses on one aspect of biological learning whereas we effectively consider multiple aspects.
E Comparison with Recent Baselines
Table 6 provides a comparison between Foster [51], Memo [57], and TriRE in Class-IL. The experiments are conducted on Seq-CIFAR100 with buffer size 200 with 5 tasks and 3 random seeds. We caution that the implementation details are slightly different: TriRE uses reservoir sampling with ResNet-18 as a backbone while both Foster and Memo employ modified ResNet-18 with custom sampling methods. Foster entails a dynamic expansion and compression to accommodate new information. Similarly, Memo entails expansion in the later layers while preserving generic information in the earlier layers. On the other hand, TriRE promotes generalization through a combination of weight and function space regularization, selective forgetting, and relearning thereby producing superior performance across tasks. As can be seen, TriRE outperforms both Foster and Memo.
Table 6: Comparison with Foster and Memo on Seq-CIFAR100 with 5 tasks and buffer size 200.
Method	Foster	Memo	TriRE
Top-1 Accuracy (%)	40.48 ±0.53	43.57 ±0.44	43.91 ±0.18
As the CL training progresses, both Foster and Memo are constrained to accommodate new information in a limited number of parameters resulting in lower performance in later tasks. However, TriRE suffers from no such limitation resulting in superior performance (see Figure 9).
Figure 9: Task-wise performance on Seq-CIFAR100 with 5 tasks and buffer size 200. The models are assessed after completing each task (y-axis) to gauge how the progress of training impacts task performance (x-axis).
F Training Cost
We compare the training times of various CL models that were taken into account for this work. We consider one method from each family of CL models. Table 7 specifically lists the training times required to learn a single task (the first) for 3 epochs on an NVIDIA RTX 2080 Ti for Seq-CIFAR100 dataset with a buffer size of 200. We conducted the experiment for 3 epochs across all CL methods considering the three-phased approach in TriRE and to maintain the fairness of comparison.
Table 7: Training time comparison across various CL methods for three epochs on an NVIDIA RTX 2080 Ti for Seq-CIFAR100 dataset.
Method	SI	DER++	PNNs	TriRE	Phases of TriRE
Retain	Revise	Rewind
Training Time (sec)	~32	~72	~25	~75	~33	~30	~12
Table 7 shows that PNNs have the least training cost for our experimental setup. However, it is to be noted that in the case of PNNs, as the number of tasks increases, the model size also increases. This indicates that the training time also eventually increases towards the later tasks. Furthermore, we see that SI also costs less training time-wise. However, according to Table 1, the Class-IL accuracies of weight regularization methods mentioned are significantly lower for larger datasets. Interestingly,  $\mathrm{DER}++$  and TriRE exhibit similar training costs, yet our approach proves to be more effective in mitigating catastrophic forgetting and task interference. This can be attributed to our method's unique amalgamation of various CL method families, making it a more robust choice for CL tasks.
We also compared the training costs for each stage within TriRE's learning paradigm. We found that 'Retain' incurs the highest cost among the three, as it involves activation and weight pruning. The 'Revise' stage, where the joint distribution of past tasks and the current task is learned, follows. Finally, 'Rewinding' to a saved weight and learning for a few epochs to activate less active neurons requires significantly less training time compared to the other two stages.
G Datasets and Settings
We assess the effectiveness of our approach in two different types of CL scenarios: Class Incremental Learning (Class-IL) and Task Incremental Learning (Task-IL). In Task-IL and Class-IL, each task consists of a predetermined number of new classes that the model needs to learn. A CL model learns multiple tasks in sequence while being able to differentiate between all classes it has encountered so far. Task-IL is similar to Class-IL, but it has the advantage of having access to task labels during the inference process, making it one of the easiest scenarios.
To evaluate the performance of our method in Task-IL and Class-IL scenarios, we employ three different datasets: Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImageNet. These datasets are derived from CIFAR10, CIFAR100, and TinyImageNet, respectively. In Seq-CIFAR10, CIFAR10 is divided into five tasks, each task containing two classes. Similarly, in Seq-CIFAR100, CIFAR100 is divided into five tasks, each consisting of 20 classes. Lastly, in Seq-TinyImageNet, we partition TinyImageNet into ten tasks, each of which comprises 20 classes. These datasets are designed to introduce more challenging scenarios for a comprehensive analysis of various CL methods. By increasing the number of tasks or the number of classes per task, we can thoroughly examine the effectiveness of different CL approaches in handling different levels of complexity. Following [6], we used ResNet-18 as the backbone in all our experiments. The training process remains consistent for both Class-IL and Task-IL. To compare various state-of-the-art approaches, we present the average accuracy across all tasks encountered in Class-IL. According to Task-IL conventions, we take advantage of task identity and selectively deactivate neurons in the linear classifier that are not related to the current task.
Contrary to the common practice of using dense CL models, dynamic sparse methods take a different approach by starting with a sparse network and maintaining the same level of connection density throughout the learning procedure to incorporate sparsity into a CL model; it is necessary to disentangle interfering units to prevent forgetting and establish new pathways to encode new knowledge. This presents challenges when implementing batch normalization and residual connections for both the NISPA and CLNP methods. Consequently, these methods do not employ the ResNet-18 architecture. Instead, they opt for a simpler CNN architecture without 'skip connections' and batch normalization.