SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Preetha Vijayan $^{1,*}$ , Prashant Bhat $^{2,3,*}$ , Bahram Zonooz $^{2,3,\dagger}$ , Elahe Arani $^{2,\dagger}$ , NavInfo Europe  $^{2}$  Eindhoven University of Technology (TU/e)  $^{3}$ TomTom preetha.vijayan@navinfo.eu, {p.s.bhat, e.arani, b.zonooz}@tue.nl
Continual learning (CL) has remained a persistent challenge for deep neural networks due to catastrophic forgetting (CF) of previously learned tasks. Several techniques such as weight regularization, experience rehearsal, and parameter isolation have been proposed to alleviate CF. Despite their relative success, these research directions have predominantly remained orthogonal and suffer from several shortcomings, while missing out on the advantages of competing strategies. On the contrary, the brain continually learns, accommodates, and transfers knowledge across tasks by simultaneously leveraging several neurophysiological processes, including neurogenesis, active forgetting, neuromodulation, metaplasticity, experience rehearsal, and context-dependent gating, rarely resulting in CF. Inspired by how the brain exploits multiple mechanisms concurrently, we propose TriRE, a novel CL paradigm that encompasses retaining the most prominent neurons for each task, revising and solidifying the extracted knowledge of current and past tasks, and actively promoting less active neurons for subsequent tasks through rewinding and relearning. Across CL settings, TriRE significantly reduces task interference and surpasses different CL approaches considered in isolation. $^{1}$
Continual learning (CL) over a sequence of tasks remains an uphill task for deep neural networks (DNNs) due to catastrophic forgetting of older tasks, often resulting in a rapid decline in performance and, in the worst-case scenario, complete loss of previously learned information [38]. Several approaches, such as parameter isolation [43, 4], weight regularization [56, 45], and experience rehearsal [41, 42, 9] have been proposed in the literature to address the problem of catastrophic forgetting in DNNs. Despite their relative success, these research directions have predominantly remained orthogonal and suffer from several shortcomings. Parameter isolation approaches suffer from capacity saturation and scalability issues in longer task sequences, while weight regularization approaches cannot discriminate classes from different tasks, thus failing miserably in scenarios such as class-incremental learning (Class-IL) [28]. In scenarios where buffer size is limited due to memory constraints (e.g., edge devices), rehearsal-based approaches are prone to overfitting on the buffered data [7]. As these research directions have rarely crossed paths, there is a need for an integrated approach to leverage the advantages of competing methods to effectively mitigate catastrophic forgetting in CL.
Catastrophic forgetting is a direct consequence of a more general problem, namely the stability-plasticity dilemma [2, 36]: the extent to which the CL model must be plastic to accommodate newly acquired knowledge and stable to not interfere with previously learned information [38]. In stark
*Equal contribution. †Equal advisory role.  
†Code is available at https://github.com/NeurAI-Lab/TriRE
Figure 1: TriRE consists of a three-phase learning paradigm that reduces task interference and drastic weight changes by using task modularity. In the Retain stage, the method selects and preserves the most active neurons and weights in a mask  $S_{t}$ , which is used in the subsequent Revise stage to finetune the joint distribution of current and past tasks along with a cumulative subnetwork mask  $S$ . The Rewind stage is responsible for reintroducing less active neurons to the learning process for future tasks by actively forgetting and relearning the non-sparse subnetwork.
contrast to DNNs, biological systems manage this dilemma better and are able to learn continually throughout their lifetime with minimal interference. CL in the brain is administered by a rich set of neurophysiological processes that encompass different kinds of knowledge, and conscious processing that integrates them coherently [17]. Empirical studies suggest that metaplasticity [27] and experience replay play a prominent role in memory consolidation in the brain [40, 15]. In addition, neurogenesis in the brain is crucial for the growth and restructuring necessary to accommodate new skills [3]. Neuromodulatory systems facilitate swift learning and adaptability in response to contextual changes induced by new stimuli or shifts in motivation [33]. Whereas context-dependent gating [34] and active forgetting [19] improve the separation between the representations of patterns belonging to different tasks. By simultaneously leveraging these processes, the brain exploits task similarity and exhibits positive forward transfer, rarely resulting in catastrophic forgetting.
Inspired by the biological underpinnings of the CL mechanisms in the brain, we propose 'REtain, REvise & REwind' (TriRE), a novel CL paradigm to mitigate catastrophic forgetting. Specifically, TriRE involves experience rehearsal, scalable neurogenesis, selective forgetting, and relearning to effectively mitigate catastrophic forgetting. Within each task, the proposed method consists of three stages: (i) Retain, where the most active neurons and their corresponding most important weights of the task are extracted and retained to avoid task interference and drastic weight changes, (ii) Revise, where the extracted network is finetuned to revise and solidify the current task as well as the joint distribution of the past tasks, (iii) Rewind, where the free neurons undergo active forgetting and relearning to promote the less active neurons back into the learning circuit for the next task as illustrated in Figure 1. TriRE effectively combines multiple mechanisms and leverages the advantages offered by different CL approaches.
We find that TriRE significantly reduces task interference and surpasses the aforementioned CL approaches across various CL scenarios. Specifically, TriRE outperforms rehearsal-based approaches in Seq-TinyImageNet for Class-IL scenario by almost  $14\%$ , even under low-buffer regimes, by promoting generalization through weight and function space regularization. Experience rehearsal enables discrimination of classes belonging to different tasks in TriRE, resulting in at least twice as good performance over weight regularization methods for the same dataset. Unlike parameter isolation approaches, TriRE is scalable and produces at least a  $7\%$  relative improvement compared to parameter isolation approaches in Seq-CIFAR100 Task-IL setting without requiring access to task identity at inference time.
Rehearsal-based Approaches: Prior works attempted to address the problem of catastrophic forgetting by explicitly storing and replaying previous task samples, akin to experience rehearsal in the brain. Experience rehearsal (ER) approaches [41, 42] maintain a fixed capacity memory buffer to
store data sampled from previous task distributions. Several approaches are built on top of ER to better preserve the previous task information: GCR [49] proposed a coreset selection mechanism that approximates the gradients of the data seen so far to select and update the buffer. DER++ [9] and CLS-ER [6] enforce consistency in predictions using soft targets in addition to ground-truth labels. DRI [52] uses a generative model to further support experience rehearsal in low buffer regimes. More recent works like TARC [8], ER-ACE [11] and  $\mathrm{Co}^2\mathrm{L}$  [12] focus on reducing representation drift right after task switch to mitigate forgetting through asymmetric update rules. Under low-buffer regimes and longer task sequences, however, these approaches suffer from overfitting on the buffered samples.
Weight Regularization Approaches: Catastrophic forgetting mainly emanates from large weight changes in DNNs when learning a new task. Therefore, weight-regularization methods seek to penalize the sudden changes to model parameters that are crucial for the previous tasks. Depending on the type of regularization, these approaches can be broadly categorized into prior- and data-focused approaches. Prior-focused approaches, such as elastic weight consolidation (EWC) [25], online EWC (oEWC) [45], memory-aware synapses (MAS) [5], and synaptic intelligence (SI) [56] employ prior information on model parameters and estimate the importance of parameters associated with previous tasks based either on the gradients of the learned function output or through Fisher's information matrix. On the other hand, data-focused methods, such as Learning without Forgetting (LwF) [30] instead perform knowledge distillation from models trained on previous tasks when learning on new data. Although weight regularization approaches do not require a memory buffer and are scalable, they only impose a soft penalty, thus failing to entirely prevent forgetting of previous task information.
Parameter Isolation Approaches: Parameter isolation has been predominantly done in two ways: either within a fixed capacity or by growing in size. In the former, dynamic sparse methods such as PackNet [32], CLNP [16], PAE [23], and NISPA [18] make use of DNN's over-parameterization to learn multiple tasks within a fixed model capacity. Similar to the brain, these models simultaneously learn both connection strengths and a sparse architecture for each task, thereby isolating the task-specific parameters. However, these methods suffer from capacity saturation in longer task sequences, limiting their ability to accommodate new tasks. In contrast, the latter methods, such as PNNs [39], Expert-gate [4] and DEN [55] expand in size, either naively or intelligently, to accommodate new tasks while minimizing forgetting. Although these approaches are extremely efficient in mitigating catastrophic forgetting, they do not scale well with longer task sequences, rendering them inapplicable in real-world scenarios.
Contrary to DNNs, the brain simultaneously exploits multiple neurophysiological processes, including neurogenesis [3], active forgetting [19], metaplasticity [27], experience rehearsal [40], and context-dependent gating [34] to continually acquire, assimilate, and transfer knowledge across tasks without catastrophic forgetting [26]. Inspired by how the brain exploits multiple mechanisms concurrently, we propose a novel CL paradigm, TriRE, that leverages the advantages of multiple aforementioned mechanisms to effectively mitigate catastrophic forgetting in CL.
CL problems typically comprise  $t \in \{1, 2, \dots, T\}$  sequential tasks, with  $c$  classes per task, and data that appear gradually over time. Each task has a task-specific data distribution associated with it  $(x_{t}, y_{t}) \in D_{t}$ . We take into account two well-known CL scenarios, class-incremental learning (Class-IL) and task-incremental learning (Task-IL). Our working model consists of a feature extractor network  $f_{\theta}$  and a single head classifier  $g_{\theta}$  that represents all classes of all tasks.
Sequential learning through DNNs has remained a challenging endeavor, since learning new information tends to dramatically degrade performance on previously learned tasks. As a result, to better retain information from past tasks, we maintain a memory buffer  $D_{m}$  that contains data from tasks previously viewed. Considering the desiderata of CL, we assume that the model does not have infinite storage for previous experience and thus  $|D_m|\ll |D_t|$ . To this end, we use loss-aware balanced reservoir sampling [10] to maintain the memory buffer. We update the working model,  $\Phi_{\theta} = g_{\theta}(f_{\theta}(.))$ , using experience rehearsal at each iteration by sampling a mini-batch from both  $D_{t}$  and  $D_{m}$  as follows:
$$
\mathcal {L} = \underbrace {\underset {(x _ {i} , y _ {i}) \sim D _ {t}} {\mathbb {E}} \left[ \mathcal {L} _ {c e} \left(\sigma \left(\Phi_ {\theta} \left(x _ {i}\right)\right), y _ {i}\right) \right]} _ {\mathcal {L} _ {t}} + \lambda \underbrace {\underset {(x _ {j} , y _ {j}) \sim D _ {m}} {\mathbb {E}} \left[ \mathcal {L} _ {c e} \left(\sigma \left(\Phi_ {\theta} \left(x _ {j}\right)\right), y _ {j}\right) \right]} _ {\mathcal {L} _ {e r}}, \tag {1}
$$
where  $\mathcal{L}_{ce}$  is the cross-entropy loss,  $\sigma(.)$  is the softmax function,  $\mathcal{L}_t$  is the task-wise loss and  $\mathcal{L}_{er}$  is the rehearsal-based loss. The objective in Eq. 1 encourages plasticity through the supervisory signal from  $D_t$  and increases stability through  $D_m$ . However, as CL training advances, model predictions carry more information per training sample than ground truths [7]. Hence, soft targets can be utilized in addition to ground-truth labels to better preserve the knowledge of the earlier tasks. Traditional methods to enforce consistency in predictions include using an exponential moving average (EMA) of the weights of the working model [6] or holding previous predictions in a buffer [9]. As the former result in better knowledge consolidation and decision boundaries, we use EMA of the working model weights to ensure consistency in predictions:
$$
\mathcal {L} _ {c r} \triangleq \underset {\left(x _ {j}, y _ {j}\right) \sim D _ {m}} {\mathbb {E}} \| \Phi_ {\theta_ {E M A}} \left(x _ {j}\right) - \Phi_ {\theta} \left(x _ {j}\right) \| _ {F} ^ {2}, \tag {2}
$$
where  $\Phi_{\theta_{EM,A}}$  is the EMA of the working model  $\Phi_{\theta}$  and  $\| .\| _F$  is the Frobenius norm. We update the EMA model as follows:
$$
\theta_ {E M A} = \left\{ \begin{array}{l l} \mu \theta_ {E M A} + (1 - \mu) \theta , & \text {i f} \zeta \geq \mathcal {U} (0, 1) \\ \theta_ {E M A}, & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
where  $\mu$  is the decay parameter and  $\zeta$  is the update rate. Finally, the EMA model acts as a self-ensemble of models with distinct task specializations for inference rather than the working model.
Notations: Let  $S_{t}$  be the extracted subnetwork mask corresponding exclusively to the current task and  $S$  be the cumulative dynamic network mask corresponding to the tasks learned so far. At the end of the training,  $S$  would contain the most active neurons and the best corresponding weights across all tasks. In the following, we describe in detail various components of TriRE learning paradigm.
3.1 Retain
In CL, typically, models from previous tasks are seen as initialization and are "washed out" by new updates from the current task, which causes CF [35]. However, the brain uses context-dependent gating [26] to selectively filter neural information based on the context in which it is presented, allowing the development of specialized modules that can be added or removed from the network without disrupting previously learned skills [37, 50]. Inspired by this, the Retain phase induces modularity in the model by training a hyper-network first and then extracting a subnetwork that is equivalently representational of the current task knowledge. This extracted subnetwork not only helps in creating task-wise specialized modules, but also helps the model preserve capacity for future tasks. Retention of this subnetwork is done using heterogeneous dropout of activations and weight pruning.
The Retain stage appears at the beginning of each task. At this stage, initially  $\{f_{\theta} \mid \theta \notin S\}$  is trained using a mini-batch of  $D_{t}$  and  $\{f_{\theta} \mid \theta \in S\}$  is trained using a mini-batch of  $D_{m}$ . This is to ensure that the weights not in the cumulative network learn the new task to maintain plasticity, while the weights in the cumulative network learn a combination of old and new tasks to maintain stability. At the convergence of this training, we perform activation pruning followed by weight pruning to extract  $S_{t}$  as shown in Figure 2.
Activation Pruning: This involves identifying and extracting neurons that contribute the most to the overall activation or output of the network. We monitor the frequency of neuron activations when a network is trained on a task. In essence, each neuron is given an activation counter that increases when a neuron's activation is among the top-k activations in its layer. We use these activation counts to extract the k-winner activations and retain them as the knowledge base for the current task. Heterogeneous dropout [1] is used to map the activation counts of each neuron to a Bernoulli variable, indicating whether the said neuron is extracted or dropped. This, essentially, leaves the less activated neurons free to learn the next tasks.
Weight Pruning: After retaining the most activated neurons for the task, we prune the less important connections corresponding to these neurons. In contrast to conventional methods, which only leverage weight magnitude or Fisher information for pruning, our method also takes into account the significance of weights with respect to data saved in the rehearsal buffer. Continual Weight Importance (CWI) [53] criteria ensure that we maintain: (1) weights of greater magnitude for output stability, (2) weights significant for the current task for learning capacity, and (3) weights significant for past data to prevent catastrophic forgetting.
Figure 2: Schematic representation of extraction of subnetwork at the end of Retain stage. The dense network is first pruned using k-WTA criteria, resulting in a subnetwork of the most activated neurons. This subnetwork is then pruned using CWI criteria, resulting in a final extracted subnetwork,  $S_{t}$ .
For the working model, the CWI of weight  $\theta$  is defined as follows,
$$
C W I (\theta) = \| \theta \| _ {1} + \alpha \| \frac {\delta \tilde {\mathcal {L}} _ {c e} (D _ {t} ; \theta)}{\delta \theta} \| _ {1} + \beta \| \frac {\delta \mathcal {L} _ {c e} (D _ {m} ; \theta)}{\delta \theta} \| _ {1} \tag {4}
$$
where  $\alpha$  and  $\beta$ , are coefficients that regulate the weight of current and buffered data, respectively. In addition,  $\tilde{\mathcal{L}_{ce}}$  denotes the single-head form of the cross-entropy loss, which only takes into account the classes relevant to the current task by masking out the logits of other classes.
At the end of this stage, we end up with a mask of the most activated neurons and their most important weights for the current task,  $S_{t}$ .
3.2 Revise
Although context-dependent gating is a helpful phenomenon for CL in tackling CF, there are other biological phenomena such as neuromodulation, metaplasticity, and neurogenesis among others which also contribute to the brain's ability to combat CF [26]. Neuromodulation, for instance, is used to decide whether a new feature is novel and unfamiliar (that is, creates a new memory) or common and familiar (that is, consolidates into an existing memory). Taking the cue from that, Revise stage mainly focuses on revising and solidifying the knowledge that the extracted subnetwork of the current task,  $S_{t}$ , and the cumulative subnetwork of past tasks,  $S$ , currently possess. It is clear that  $S_{t}$  is specialized on the current task and  $S$  is specialized on the past tasks. However, finetuning these two networks jointly can improve the knowledge overlap, and the features thus generated would be a better approximation of all the seen classes.
Firstly,  $\{f_{\theta} \mid \theta \notin (\mathcal{S} \cap \mathcal{S}_t)\}$  is trained with a mini-batch of  $D_t$  so that a part of the extracted network,  $\mathcal{S}_t$ , can be utilized to regain the performance for the current task. Subsequently,  $\{f_{\theta} \mid \theta \in (\mathcal{S} \cap \mathcal{S}_t)\}$  is trained with a mini-batch of  $D_m$  to preserve forward transfer and knowledge overlap between the past and the current tasks. The optimization learning rate is also considerably reduced at this stage compared to the Retain stage to prevent drastic weight changes in subnetworks, which in turn decreases the forgetting. This could be seen as an adaptation of the metaplasticity in the brain [26, 20] which refers to the ability to adjust the amount of plasticity in the network based on the current and future needs of the organism. At the end of this finetuning, the currently extracted subnetwork  $\mathcal{S}_t$  is merged with the cumulative extracted mask from past tasks,  $\mathcal{S}$ . The merging of the subnetworks can be seen as a process of integrating new neurons  $(\mathcal{S}_t)$  into the existing neural network  $(\mathcal{S})$ , similar to the way neurogenesis allows for the integration of new neurons into existing neural circuits to accommodate new memories.
3.3 Rewind
Evidence implies that the brain uses active forgetting as a tool for learning due to its limited capacity [47, 29]. Studies also suggest that although learning and memory become more difficult to access during the forgetting process, they still exist [54, 48]. There are still memory remnants in the brain that can be reactivated [31]. In this work, we define the rewinding of weights to an earlier state as active forgetting. We also make sure that rather than reinitializing the model back to random or very
Algorithm 1 Proposed Approach - TriRE
input: Data streams  $\mathcal{D}_t$  working model  $\Phi_{\theta} = g_{\theta}(f_{\theta}(.),$  EMA model  $\Phi_{\theta_{EMA}} =$ $g_{\theta_{EMA}}(f_{\theta_{EMA}}(.))$  , sparsity factor  $\gamma$  , learning rates  $\eta \gg \eta^{\prime}$  , Retain epochs  $E_1$  , Revise epochs  $E_2$  Rewind epochs  $E_3$    
1:  $S\gets \{\}$ $\mathcal{M}\gets \{\}$    
2: for all tasks  $t\in \{1,2,\dots,T\}$  do   
3: for epochs  $e_1\in \{1,2,\ldots E_1\}$  do  $\triangleright$  Retain   
4: for minibatch  $\{(x_i,y_i)\}_{i = 1}^B\in \mathcal{D}_t$  and  $\{(x_m,y_m)\}_{m = 1}^B\in \mathcal{M}$  do   
5: Update  $\{f_{\theta}\mid \theta \notin S\}$ $g_{\theta}$  with  $\eta$  on  $\{(x_i,y_i)\}_{i = 1}^B$  using Eq. 1 (Lt)   
6: Update  $\{f_{\theta}\mid \theta \in S\}$ $g_{\theta}$  with  $\eta$  on  $\{(x_m,y_m)\}_{m = 1}^B$  using Eqs. 1 (Ler) and 2   
7: if  $e = k$  then   
8: Save the weights  $\theta_{k}$    
9: Update  $\theta_{EMA}$  using Eq. 3   
10: Extract new subnetwork  $S_t$  with  $\gamma$  sparsity based on CWI from Eq. 4   
11: for epochs  $e_2\in \{1,2,\ldots E_2\}$  do  $\triangleright$  Revise   
12: for minibatch  $\{(x_i,y_i)\}_{i = 1}^B\in \mathcal{D}_t$  and  $\{(x_m,y_m)\}_{m = 1}^B\in \mathcal{M}$  do   
13: Finetune  $\{f_{\theta}\mid \theta \notin (\mathcal{S}\cap \mathcal{S}_t)\}$ $g_{\theta}$  with  $\eta^\prime$  on  $\{(x_i,y_i)\}_{i = 1}^B$    
14: Finetune  $\{f_{\theta}\mid \theta \in (\mathcal{S}\cap \mathcal{S}_t)\}$ $g_{\theta}$  with  $\eta^\prime$  on  $\{(x_m,y_m)\}_{m = 1}^B$    
15: Update  $\theta_{EMA}$    
16: Update cumulative set  $S = S\cup S_t$    
17: Reinitialize non-cumulative weights  $\{f_{\theta}\mid \theta \notin S\}$  with  $\theta_{k}$    
18: for epochs  $e_3\in \{1,2,\ldots E_3\}$  do  $\triangleright$  Rewind   
19: for minibatch  $\{(x_i,y_i)\}_{i = 1}^B\in \mathcal{D}_t$  do   
20: Update  $\{f_{\theta}\mid \theta \notin S\}$ $g_{\theta}$  with  $\eta$  on  $\{(x_i,y_i)\}_{i = 1}^B$    
21: Update  $\theta_{EMA}$  Update buffer M   
22: return model  $\Phi_{\theta}$  , model  $\Phi_{EMA}$
early weights, it is rebound to a point where the model has learned some features and has a generic perception of the objective closer to convergence (but not absolute convergence). To aid this, the weights from a later epoch  $k$  from the Retain phase are saved to be used in the Rewind stage.
Specifically, after the Retain and Revise steps, we rewind the weights belonging to non-cumulative subnetwork  $\{f_{\theta} \mid \theta \notin S\}$  back to the epoch  $k$  weights. Then the rewinded weights are finetuned for a few epochs using a mini-batch of  $D_{t}$ . This is helpful because studies [46, 13] show that in the human brain, less active neurons follow a 'use-it-or-lose-it' philosophy. Therefore, forgetting and relearning act as a warm-up for these less active neurons making them relevant again for the learning circuit and making them more receptive to learning the next task.