SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In summary, TriRE (Retatin-Revise-Rewind) involves iteratively applying the three phases mentioned above to each task within a lifelong learning setting. Our method effectively combines multiple biological phenomena and harnesses the advantageous characteristics provided by popular CL approaches. The step-by-step procedure is given in Algorithm 1.
Experimental Setup: We expand the Mammoth CL repository in PyTorch [9]. On the basis of Class-IL and Task-IL scenarios, we assess the existing CL techniques against the proposed one. Although the training procedure for Class-IL and Task-IL is the same, during inference, Task-IL has access to the task-id. We consider a number of rehearsal-based, weight regularization, and parameter-isolation approaches as useful baselines because TriRE necessitates experience rehearsal and model modularity. We use ResNet-18 [21] as the feature extractor for all of our investigations. In order to reduce catastrophic forgetting, we additionally offer a lower bound SGD without any support and an upper bound Joint where the CL model is trained using the full dataset.
Experimental Results: We compare TriRE with contemporary rehearsal-based and weight regularization methods in Class-IL and Task-IL settings. As shown in Table 1, TriRE consistently outperforms
Table 1: Comparison of prior methods across various CL scenarios. We provide the average top-1  $(\%)$  accuracy of all tasks after training.  $^{\dagger}$  Results of the single EMA model.
Buffer size	Methods	Seq-CIFAR10	Seq-CIFAR100	Seq-TinyImageNet
Class-IL	Task-IL	Class-IL	Task-IL	Class-IL	Task-IL
-	SGD	19.62±0.05	61.02±3.33	17.49±0.28	40.46±0.99	07.92±0.26	18.31±0.68
Joint	92.20±0.15	98.31±0.12	70.56±0.28	86.19±0.43	59.99±0.19	82.04±0.10
-	LwF	19.61±0.05	63.29±2.35	18.47±0.14	26.45±0.22	8.46±0.22	15.85±0.58
oEWC	19.49±0.12	68.29±3.92	-	-	7.58±0.10	19.20±0.31
SI	19.48±0.17	68.05±5.91	-	-	6.58±0.31	36.32±0.13
200	ER	44.79±1.86	91.19±0.94	21.40±0.22	61.36±0.35	8.57±0.04	38.17±2.00
DER++	64.88±1.17	91.92±0.60	29.60±1.14	62.49±1.02	10.96±1.17	40.87±1.16
CLS-ER†	61.88±2.43	93.59±0.87	43.38±1.06	72.01±0.97	17.68±1.65	52.60±1.56
ER-ACE	62.08±1.44	92.20±0.57	35.17±1.17	63.09±1.23	11.25±0.54	44.17±1.02
Co2L	65.57±1.37	93.43±0.78	31.90±0.38	55.02±0.36	13.88±0.40	42.37±0.74
GCR	64.84±1.63	90.8±1.05	33.69±1.40	64.24±0.83	13.05±0.91	42.11±1.01
DRI	65.16±1.13	92.87±0.71	-	-	17.58±1.24	44.28±1.37
TriRE	68.17±0.33	92.45±0.18	43.91±0.18	71.66±0.44	20.14±0.19	55.95±0.78
Figure 3: Comparison of TriRE against evolving architectures in terms of Task-IL accuracy on Seq-CIFAR100 dataset divided into 20 tasks. The graph reports the accuracy of individual tasks at the end of CL training.
rehearsal-based methods across most datasets, highlighting the significance of dynamic masking in CL. Although methods like  $\mathrm{Co}^2\mathrm{L}$  and ER-ACE excel on simpler datasets, they struggle with more challenging ones. The same applies to methods like DRI and GCR, which augment memory buffers through core-set and generative replay. Their performance, for instance, lags behind TriRE in Seq-TinyImageNet, where the buffer-to-class ratio is low. Retaining task-wise dynamic subnetworks and revising extracted subnetworks to preserve past knowledge significantly reduces task interference in TriRE. In essence, TriRE boosts generalization through weight and function space regularization, selective forgetting, and relearning, yielding superior performance across tasks.
As evident from Table 1, weight regularization methods such as LWF, oEWC, and SI perform miserably in both Class-IL and Task-IL settings across datasets. The reason being, these approaches encounter classes solely from the current task at any point in CL training. Therefore, they fail to discriminate between classes from different classes resulting in subpar performance. On the other hand, TriRE leverages samples from previous classes through experience rehearsal to learn discriminatory features across tasks. In addition, TriRE entails forming subnetworks through Retain and Revise resulting in modularity and reduced task interference between tasks.
Parameter isolation approaches minimize task interference in CL by creating distinct sub-networks either within a given model capacity or by dynamically growing the network. Figure 3 illustrates a comparison between methods trained on Seq-CIFAR100 with 20 tasks i.e., it depicts final accuracies
Figure 4: (Top) depicts the distribution of neuron activations across tasks in the last convolutional layer of the feature extractor and (Bottom) depicts the same for the last shortcut layer. The black cubes represent the extracted ones after Retain stage.
Figure 5: The effect of rewinding on Class-IL accuracy for all three datasets. The region from  $70\%$  to  $90\%$  of all epochs gives the best results consistently across datasets.
on  $1^{st}$ ,  $2^{nd}$ ,  $20^{th}$  task after training. Upon completing all tasks, TriRE achieves an average accuracy of  $80.85\%$ , surpassing the performance of the baselines considered. TriRE leverages the benefits of experience rehearsal, weight regularization, and function space regularization to learn compact and overlapping subnetworks, resulting in reduced task interference while maintaining scalability. In line with biological principles, TriRE incorporates selective forgetting and relearning mechanisms to activate less active neurons and enhance their receptiveness to learning subsequent tasks, thereby mitigating the risk of capacity saturation.
Task Interference: Figure 4 shows the changes in neuronal activity for the subset of neurons in the last convolutional layer (top) and the last shortcut layer (bottom) of ResNet-18 trained on 5 tasks in Seq-CIFAR100 dataset. The neurons that are blacked out are the most active neurons for each particular task after the Retain phase. For each task, it is observed that there are exclusive subnetworks forming on their own (horizontally) that capture task-specific information. However, with CL training, several neurons become generalizable by capturing information that can be shared across tasks. Therefore, there is neuron overlap between the extracted neurons across tasks (vertically). More so in the shortcut layer, since the number of parameters in these layers is low. TriRE optimally manages the model capacity vs. task modularity trade-off better by re-using neurons that can be shared across tasks while maintaining the modularity of knowledge in each task intact.
How Much to Rewind? In Figure 5, we examine Class-IL accuracy when the model is rewound to different points in the Retain stage in order to comprehend how rewinding affects the accuracy of the model. We observe that the accuracy of the inference model decreases if the model forgets too much and is rewound to an early stage in the training. This is in alignment with the observations made by [14] that rewinding to extremely early stages is not recommended for DNNs because the network has
Table 2: Comparison of the contribution of each phase in TriRE. Note that the combination of Revise alone or Revise & Rewind has not been considered, as it is not feasible without the Retain phase.
Retain	Revise	Rewind	Seq-CIFAR100	Seq-TinyImageNet
Class-IL	Task-IL	Class-IL	Task-IL
✓	✗	✗	38.01	66.23	11.54	40.22
✓	✓	✗	33.08	60.03	8.44	31.90
✓	✗	✓	43.03	72.09	16.25	48.89
✓	✓	✓	43.91	71.66	20.14	55.95
Table 3: Relative number of learnable parameters and corresponding memory footprint in Seq-CIFAR100 with varying number of task sequence.
Methods	Learnable Parameters (Million)	Memory Consumption (Million)
5 Tasks	10 Tasks	20 Tasks	5 Tasks	10 Tasks	20 Tasks
DER ++	1x	1x	1x	1x	1x	1x
EWC	1x	1x	1x	3x	3x	3x
TriRE	1x	1x	1x	6x	6x	6x
PNNs	27x	79x	240x	27x	79x	240x
not learned enough meaningful features by then to regain the lost accuracy. Additionally, we notice that accuracy also suffers when the model is rewound to a point extremely close to the end of training time. Rewinding to a very late point in the training phase close to convergence is not ideal because there is not enough time for relearning. Our experiments indicate that rewinding to between  $70\%$  and  $90\%$  of the training time results in the best accuracy.
Ablation Study: As explained previously, TriRE employs a three-stage learning paradigm to reduce task interference and improve weight reuse in CL. We seek to uncover how each of Retain, Revise, and Rewind in TriRE influence Class-IL and Task-IL accuracies in Seq-CIFAR100 and Seq-TinyImageNet datasets through Table 2. It can be seen that although Retain alone can extract the subnetworks containing the most active neurons and decrease task interference, it falls short in aspects of forward transfer and weight reuse. Similarly, Retain and Revise together can solidify the knowledge extracted from current and past tasks, but such a model suffers from capacity issues without the reactivation of less active neurons for future tasks. Likewise, Retain and Rewind together can encourage task-wise delimitation of knowledge and promote efficient usage of available networks, but lose out on the forward transfer introduced by the learning of joint distributions. Finally, analogous to the brain, it is evident that the harmony of all components is what achieves the best results in both datasets.
Memory and Computational Cost: We conduct a comparative analysis of the computational and memory overhead of TriRE in contrast to related works. Table 3 provides an analysis of the learnable parameters and memory required by TriRE in contrast to those of DER++, EWC, and PNNs. (Opting for one method from each individual family of CL methods). Firstly, similar to DER++ and EWC, TRiRE does not add any learnable parameters to the model. However, it is evident that PNNs have an infeasible amount of learnable parameters which gets progressively worse with longer task sequences. Secondly, the observed increase in memory consumption in TriRE can be attributed to several factors: (1) the application of multiple masking mechanisms for parameter isolation; (2) the incorporation of the Rewind phase necessitating weight retention from a previous epoch; and (3) the utilization of the Exponential Moving Average (EMA) model to enhance knowledge consolidation. All of these factors hold memory but do not add any learnable parameter to the training.
We introduced TriRE, a novel biologically inspired CL paradigm that entails experience rehearsal, scalable neurogenesis, and selective forgetting and relearning to effectively mitigate catastrophic forgetting in CL. Within each task, TriRE entails retaining the most prominent neurons for each task, revising the extracted knowledge of current and past tasks, and actively promoting less active neurons for subsequent tasks through rewinding and relearning. Analogous to multiple neurophysiological mechanisms in the brain, TriRE leverages the advantages of different CL approaches, thus significantly lowering task interference and surpassing different CL approaches when considered in isolation. For Seq-TinyImageNet, TriRE outperforms the closest rival in rehearsal-based baselines by  $14\%$ ,
surpasses the best parameter isolation baseline by  $7\%$ , and nearly doubles the performance of the best weight regularization method. Extending our method to CL scenarios oblivious to task boundaries and to few- and zero-shot learning settings are some of the future research directions for this work.
We proposed TriRE, a novel paradigm that leverages multiple orthogonal CL approaches to effectively reduce catastrophic forgetting in CL. As orthogonal CL approaches may not always be complementary, the selection of such approaches needs careful consideration in TriRE. In addition, having multiple objective functions naturally expands the number of hyperparameters, thereby requiring more tuning to achieve optimal performance. Therefore, additional computational complexity and memory overhead due to the staged approach and extensive hyperparameter tuning are some of the major limitations of the proposed method. For the same reason, we highlight that TriRE is not directed toward compute-intensive architectures such as vision transformers.
TriRE involves different stages of training within each task, requiring knowledge of task boundaries. In line with state-of-the-art methods in CL, each task entails a non-overlapping set of classes, and data within each task is shuffled to guarantee i.i.d. data. However, in the case of online learning where data streams and the distribution gradually shift, TriRE cannot be applied in its current form. Therefore, additional measures such as task-boundary approximation and modification to learning objectives are necessary to enable TriRE to work in such scenarios. Furthermore, traditional CL datasets considered in this work entail independent tasks and data points without intrinsic cumulative structure. As TriRE does not leverage structures learned in previously encountered tasks, structure learning forms another limitation of this proposed method. Reducing computational and memory overhead, extending to task-free CL scenarios with recurring classes, and leveraging intrinsic structures within underlying data are some of the future research directions for this work.
Inspired by the multifaceted learning mechanisms of the brain, we propose TriRE, which replicates the brain's ability to leverage multiple mechanisms for CL, enhancing the generalization capabilities of CL models across tasks. Its success not only encourages the exploration of neuro-inspired methods for deep neural networks, but also opens up opportunities to augment existing CL approaches by leveraging the advantages of competing strategies. By enabling models to learn continuously and adapt to new tasks, TriRE contributes to the responsible and ethical deployment of AI technologies, as models can improve and update their knowledge without requiring extensive retraining. This advancement has significant implications for various real-world applications and promotes the development of AI systems that can continually improve and adapt their performance.
Acknowledgement: The work was conducted while all the authors were affiliated with NavInfo Europe, Eindhoven, The Netherlands.
[1] Ali Abbasi et al. "Sparsity and heterogeneous dropout for continual learning in the null space of neural activations". In: Conference on Lifelong Learning Agents. PMLR. 2022, pp. 617-628.
[2] Wickliffe C Abraham and Anthony Robins. "Memory retention—the synaptic stability versus plasticity dilemma". In: Trends in neurosciences 28.2 (2005), pp. 73-78.
[3] James B Aimone, Janet Wiles, and Fred H Gage. "Potential role for adult neurogenesis in the encoding of time in new memories". In: Nature neuroscience 9.6 (2006), pp. 723-727.
[4] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. "Expert gate: Lifelong learning with a network of experts". In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, pp. 3366-3375.
[5] Rahaf Aljundi et al. "Memory aware synapses: Learning what (not) to forget". In: Proceedings of the European conference on computer vision (ECCV). 2018, pp. 139-154.
[52] Zhen Wang et al. "Continual learning through retrieval and imagination". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. 8. 2022, pp. 8594-8602.
[53] Zifeng Wang et al. "SparCL: Sparse Continual Learning on the Edge". In: Advances in Neural Information Processing Systems.
[54] John T Wixted. “The psychology and neuroscience of forgetting”. In: Annu. Rev. Psychol. 55 (2004), pp. 235–269.
[55] Jaehong Yoon et al. "Lifelong Learning with Dynamically Expandable Networks". In: International Conference on Learning Representations.
[56] Friedemann Zenke, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence". In: International conference on machine learning. PMLR. 2017, pp. 3987-3995.
[57] Da-Wei Zhou et al. “A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning”. In: The Eleventh International Conference on Learning Representations. 2022.
A Effect of Pruning Criteria
We seek to illustrate the effectiveness of different pruning criteria in TriRE. As explained in Section 3.1, the dense network is first pruned using k-WTA criteria, resulting in a subnetwork of the most activated neurons, and then this subnetwork is pruned using CWI criteria, resulting in a final extracted subnetwork at the end of Retain stage. Table 4 demonstrates the comparison of Class-IL accuracy between various pruning criteria, namely, magnitude-based, Fisher information-based, and CWI-based, across all three datasets. The idea behind magnitude pruning is that small valued weights impact the network's output less and can be safely pruned without significantly affecting performance. Fisher information-based pruning evaluates the importance of connections based on their contributions to the Fisher information matrix. Connections with low contributions, indicating less relevance or importance, are pruned or set to zero. However, both these criteria calculate the importance of weights within the current task, but do not consider the possibility of it being crucial for other tasks. On the other hand, CWI considers the significance of weights with respect to data saved in the rehearsal buffer as well, resulting in superior performance across all datasets.
Table 4: Comparison of the effect of various pruning criteria in TriRE on different datasets.
Dataset	Magnitude	Fisher Information	CWI
Seq-CIFAR10	65.09±0.83	64.40±0.43	68.17±0.33
Seq-CIFAR100	41.89±0.74	40.26±0.21	43.91±0.18
Seq-TinyImageNet	19.07±0.97	18.16±0.75	20.14±0.19
B Model Analysis
B.1 Task Recency Bias
In any CL setting, the model entails learning on a few or no samples from previous tasks while plenty of the most recent task [22]. This tilts learning toward the most recent task, resulting in decisions biased toward new classes and confusion among the old classes. However, the CL model should ideally have predictions distributed evenly across all tasks with the least possible recency bias. Figure 6 provides the confusion matrix for various CL models to evaluate the task recency bias. After training on Seq-CIFAR100 for 5 tasks with a buffer size of 200, the model is deemed to have correctly predicted the task label if it predicts any of the classes that make up the sample's true task label. As can be seen, ER and DER++ have a propensity to frequently classify the majority of samples as classes in the most recent task. However, TriRE's predictions are uniformly distributed across the diagonal. TriRE essentially decreases interference between tasks, captures task-specific information through extracted sub-networks, and produces the least recency bias.
Figure 6: Confusion matrix of different rehearsal-based CL models. Unlike ER and DER++, TriRE predictions are evenly distributed across the tasks with the least recency bias.
B.2 Stability-Plasticity Trade-off
A CL model is said to be stable if it can retain previously learned information, and plastic if it can effectively acquire new information. The stability-plasticity dilemma refers to an inherent trade-off
Figure 7: (Left) Stability-Plasticity Trade-off for CL models trained on Seq-CIFAR100 with 5 tasks. ER and DER++ are more plastic than stable leading to recency bias. TriRE maintains a better balance between stability and plasticity and achieves the highest trade-off amongst the baselines. (Right) Reliability diagram depicting model calibration: The red dashed line represents the ideal scenario. Compared to the other two methods, TriRE is better calibrated with the lowest ECE value. All models were trained on Seq-CIFAR100 with 5 tasks.
in which the CL model masters one of these aspects at the expense of the other. Sarfraz et al. [44] introduced a trade-off measure that serves as an approximation of how the model balances its stability and plasticity. Once the model completes the final task  $T$ , its stability ( $S$ ) is assessed by calculating the average performance across all preceding  $T - 1$  tasks as follows:
$$
S = \sum_ {i = 0} ^ {T - 1} A _ {T i} \tag {5}
$$
The plasticity of the model (P) is evaluated by computing the average performance of each task after its initial learning i.e.,
$$
P = \sum_ {i = 0} ^ {T} A _ {i i} \tag {6}
$$
Thus, the trade-off measure determines the optimal balance between the stability  $(S)$  and the plasticity  $(P)$  of the model. This measure is calculated as the harmonic mean of  $S$  and  $P$ .
$$
\text {T r a d e - o f f} = \frac {2 S P}{S + P} \tag {7}
$$
Figure 7 (Left) provides the stability-plasticity trade-off measure for different CL methods across different datasets for a buffer size of 200. ER and DER++ exhibit high plasticity, enabling them to rapidly adapt to new information. However, they lack the ability to effectively retain previously acquired knowledge. On the other hand, TriRE exhibits substantially high stability with low plasticity, resulting in a higher stability-plasticity trade-off.
B.3 Model Calibration
Ensuring the reliability of safety-critical CL systems necessitates the presence of a well-calibrated model. Calibration refers to the task of accurately predicting probability estimates that reflect the true likelihood of correctness. Miscalibration, on the other hand, refers to the disparity between confidence and accuracy expectations. To assess the degree of miscalibration in classification, the Expected Calibration Error (ECE) involves partitioning the predictions into bins of equal size and calculating the difference between the weighted average of accuracy and confidence within each bin. A lower ECE value indicates better calibration in the underlying models.
Figure 7 (Right) shows a comparison of different CL approaches using a calibration framework trained on Seq-CIFAR100 with a buffer size of 200. Well-calibrated CL systems accurately represent the true likelihood of accuracy (indicated by the red dashed line). Among the baselines, TriRE achieves the lowest ECE value and exhibits high calibration, demonstrating its effectiveness in minimizing task interference and reducing overconfidence in CL, thus enabling more informed decision making.
C Hyperparameter Selection
The hyperparameters required to replicate the results of TriRE can be found in Table 5. These hyperparameters were determined through a tuning process involving different random initializations and a small portion of the training set reserved for validation. All experiments were conducted using a batch size of 32 and trained for 50 epochs. TriRE was optimized using the Adam optimizer [24] implemented in PyTorch. Furthermore, the number of epochs allocated to each phase specified in Algorithm 1 was consistently set at a ratio of  $E_{1}:E_{2}:E_{3} = 3:1:1$ .
Table 5: Best hyperparameters of TriRE chosen for optimal performance on different datasets.
Dataset	η	η'	γ	λ	EMA Parameters	Rewind Percentile
μ	ζ
Seq-CIFAR10	0.0006	0.0001	0.4	0.06	0.999	0.18	0.9
Seq-CIFAR100	0.002	0.0001	0.2	0.04	0.999	0.12	0.9
Seq-TinyImageNet	0.002	0.0001	0.3	0.05	0.999	0.01	0.8
C.1 Hyperparameters Sensitivity Analysis
In order to showcase the robustness of TriRE to a choice of values for each hyperparameter, we conducted additional experiments by finetuning different hyperparameters. Specifically, we evaluated the performance of TriRE in the Seq-CIFAR100 dataset with a buffer size of 200 and 5 tasks. The results are visualized in the Figure 8. Our experimentation focused on adjusting hyperparameters such as sparsity, k-winner sparsity, and EMA model update frequency. Upon examining the graphs, it is evident that performance remains relatively stable across a range of values for these hyperparameters. This observation suggests that satisfactory results can be achieved without an exhaustive hyperparameter search.
Figure 8: Evaluation of performance across different values of Sparsity, K-Winner Sparsity, and Stable model update frequency in Seq-CIFAR100 with buffer size 200. As can be seen, the performance does not drop significantly for different choices of values for hyperparameters.
D TriRE vs. CLS-ER: Commonalities and Differences