SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: Incremental learning with DSN.
Input:  $\{D_t\}_{t=1}^T$ , a hypernetwork  $\mathcal{H}$  with neurons  $n$  and associated parameters  $\theta$ , a cumulative mask  $m_{\leq 0}$ . Randomly initialize  $\theta$  and set  $m_{\leq 0} = 0$ ;  
for  $t = 1,2,\ldots,T$  do  
Initialize layer embeddings  $e_t = \{e_t^l\}_{l=1}^L$ ;  
Use  $e_t = \{e_t^l\}_{l=1}^L$  to obtain the mask  $m_t$  according to Eq.(3);  
Train the hypernetwork  $\mathcal{H}$  with  $\theta(n \odot m_t)$  on  $D_t$  according to Eq.(2) and Eq.(8);  
Obtain the optimal layer embeddings  $e_t$  and layer masks  $m_t^*$ ;  
Update the cumulative mask  $m_{\leq t}$  via Eq.(6);  
if  $t > 1$  then  
Measure the task similarity scores  $S_t$  via Eq.(10);  
Obtain the most similar task  $\argmax(S_t)$  with number of class  $C_{\argmax(S_t)}$ ;  
Compute the replay number of each class  $\{B_i\}_1^{C_{\argmax(S_t)}}$  according to the class-specific error rate;  
Initialize impression set  $\mathcal{I}_{\argmax(S_t)} \gets \{\}$ ;  
for  $c = 1:C_{\argmax(S_t)}$  do  
Set the concentration parameter  $\alpha^c = M_{\argmax(S_t)}^c$ ;  
for  $b = B_1,B_2,\dots,B_{C_{\argmax(S_t)}}$  do  
for  $i = 1:b$  do  
Sample  $\hat{o}_{\argmax(S_t)}^c \sim \text{Dir}(C_{\argmax(S_t)},\beta_b \times \alpha^c)$ ;  
Initialize  $\hat{x}_{\argmax(S_t)}^{c,i}$  to random noise and craft  $\hat{x}_{\argmax(S_t)}^{c,i}$  via Eq.(9);  
 $\mathcal{I}_{\argmax(S_t)} \gets \mathcal{I}_{\argmax(S_t)} \cup \hat{x}_{\argmax(S_t)}^{c,i}$ ;  
Merge mask for task  $\argmax(S_t)$ :  $\hat{m}_{\argmax(S_t)} = \max(m_{\argmax(S_t)}^*, m_t^*)$ ;  
Fine-tune the task  $\argmax(S_t)$  based on  $\mathcal{I}_{\argmax(S_t)}$  and  $\hat{m}_{\argmax(S_t)}$ ;  
if accuracy performance of task  $\argmax(S_t)$  is higher than before then  
update the mask  $m_{\argmax(S_t)}^*$ ;
Output: Optimal hypernetwork  $\mathcal{H}$  with parameters  $\theta^{*}$  and task-specific masks  $\{\pmb{m}_t^*\}_{1}^{T}$ .
3.4 Knowledge Transfer-enhanced Incremental Learning with DSN
Incremental Learning ( $t \geq 1$ ). For the first task, i.e.,  $t = 1$ , we randomly initialize the parameters  $\theta$  in the hypernetwork  $\mathcal{H}$ . For each task  $t$  including the first task, we start with randomly initializing a task-specific classifier (e.g., a fully-connected layer with a Softmax function) and the corresponding layer embeddings  $e_t$ . Then, we attempt to jointly learn the model parameters in the hypernetwork  $\mathcal{H}$  and task-specific masks of subnetworks regarding each task, where each task-specific mask determines which neurons will be used in the current task. Specifically, given a task  $t$ , we optimize the parameter  $\theta$  and its mask  $m_t$  with the objectives of Eq.(2) and Eq.(8) where the update of parameters will follow the rule of Eq.(7) for the purpose of addressing the CF problem. After the training, we likewise build the class similarity matrix  $M_t$  and update the cumulative mask  $m_{\leq t}$ .
Knowledge Transfer ( $t > 1$ ). To consolidate the possibly useful knowledge behind the newly learned task to the earlier tasks, we first measure the task similarity and then employ our date-free replay to transfer the knowledge to the most similar task. Notably, we could choose multiple similar tasks to transfer, but this would take more transferring time than it is worth due to the cost of memory replay. Since the unavailability of any real samples in the earlier tasks, it is impossible to use the naive solution for task similarity measurement, i.e., estimating the similarity between the instance distributions of two tasks. Fortunately, we can use the masks of tasks to measure the task similarities. For a current task  $t$ , we can compute the similarity scores  $S_{t}$  as follows,
$$
S _ {t} = \left\{\operatorname {c o s i n e} \left(\boldsymbol {m} _ {t}, \boldsymbol {m} _ {0}\right), \operatorname {c o s i n e} \left(\boldsymbol {m} _ {t}, \boldsymbol {m} _ {1}\right), \dots , \operatorname {c o s i n e} \left(\boldsymbol {m} _ {t}, \boldsymbol {m} _ {t} - 1\right) \right\}, \tag {10}
$$
where cosine is the cosine distance. As such, we can obtain the most similar task that owns the largest similarity score, i.e.,  $\arg\max(S_t)$ . After that, we can use the data-free memory replay to produce impression crafts as the input samples of the most similar task by optimizing the objective of Eq.(9). It is noted that we only use the impressions to determine whether we need to adjust the subnetwork architecture of the most similar task. Meanwhile, any parameter update in the hypernetwork will cause interference with the current task as well as all the previous tasks. Instead, we only allow the parameter update in the task-specific classifier (head) while freezing any parameters in the subnetwork.
Specifically, we first merge the optimal mask of current task  $t$  to the most similar task  $\operatorname{argmax}(S_t)$ , i.e.,  $\hat{m}_{\mathrm{argmax}(S_t)} = \max(m_{\mathrm{argmax}(S_t)}^*, m_t^*)$ . Then, we fine-tune the task  $\operatorname{argmax}(S_t)$  based on  $\hat{m}_{\mathrm{argmax}(S_t)}$  and make knowledge transfer possible. We will update the architecture of the subnetwork regarding task  $\operatorname{argmax}(S_t)$  if it performs better. Notably, the cumulative mask  $m_{\leq t}$  does not need to change due to the fact of  $\hat{m}_{\mathrm{argmax}(S_t)} \subseteq m_{\leq t}$ . The complete workflow is summarized in Algorithm 1.
Datasets. We employ four benchmark datasets for TIL problem as follows: Permuted MNIST (PMNIST) [3], Rotated MNIST (RMNIST) [21], Incremental CIFAR-100 [54, 55], and TinyImageNet [19]. PMNIST encompasses 10 variations of MNIST [56], wherein each task is transformed by a fixed permutation of pixels. RMNIST also comprises ten versions of MNIST, with each task rotated by a specific angle between 0 and 360 degrees. The original CIFAR-100 was divided into 20 tasks, each containing 5 different categories. TinyImageNet constitutes a variant of ImageNet [57], we construct twenty 10-way classification tasks for consistency in our experiments.
Baselines. We compare our DSN with the following representative baselines: SGD [58] utilizes a feature extractor without any parameter update and a learnable classifier to solve a series of tasks incrementally. EWC [3] purposefully uses Fisher information as a simple and effective standard baseline to alleviate CF. IMM [25] aims to penalize parameter modifications, yielding two variants, i.e., Mean-IMM and Mode-IMM. PGN [9] attempts to expand the task network by incorporating a fixed number of neurons. DEN [36] dynamically decides the number of new neurons via selective retraining and network split. RCL [21] is a reinforcement learning method that controls the scale of newly added neurons while rendering weights relative to the used neurons unchangeable. HAT [44] leverages a masking mechanism to prevent the optimization of old neurons. SupSup [22] solves the CF problem when sequentially learning tasks by discovering super masks (sub-networks). WSN [19] tries to learn the model parameters sequentially and select the optimal sub-network for each task.
Implementations and Evaluation Metrics. In accordance with the task-incremental continual learning framework, all methods used in the experiments employ a multi-head configuration. For two variants of MNIST, we adopt the experimental setup outlined in [44] in which we use a two-layer MLP, a multi-head classifier, and begin with 2000-2000-10 neurons for the first task. For the other datasets, we refer to previous works [19, 44] and use the modified model with three convolutional layers and two fully-connected layers. We follow previous studies [3, 12, 19] and evaluate all methods on three metrics: ACC, BWT, and Trans. ACC is a common CL metric that reports the average accuracy of all tasks validated on their respective test sets after training on all tasks. BWT measures how the model performance changes for the old tasks once the model is trained on the new task. Trans measures the proficiency to transfer knowledge from previously learned tasks to newer ones, indicating the usefulness of the acquired knowledge in facilitating new learning tasks. Notice that more details of the implementation setup are specified in Appendix C. The source codes are available at https://github.com/shanxiaojun/DSN.
Table 1: Performance comparison of the proposed method and baselines on four datasets.
Model	PMNIST	RMNIST	CIFAR-100	TinyImageNet
ACC(%)	BWT(%)	Trans(%)	ACC(%)	BWT(%)	Trans(%)	ACC(%)	BWT(%)	Trans(%)	ACC(%)	BWT(%)	Trans(%)
SGD	81.37	-24.52	-17.06	72.83	-25.32	-25.08	59.82	-24.09	-24.02	30.24	-19.12	-19.96
EWC	94.20	-0.32	-4.23	94.86	-0.73	-3.05	67.15	-8.61	-16.69	40.85	-5.24	-9.35
mean-IMM	80.10	-1.13	-18.33	88.81	-0.96	-9.10	56.08	0.23	-27.76	30.10	-3.21	-20.10
mode-IMM	93.13	-4.17	-5.30	89.48	-7.40	-8.43	61.22	-21.49	-22.62	32.26	-19.02	-17.94
PGN	91.89	0.00	-6.54	90.01	0.00	-7.90	53.84	-14.66	-30.00	24.47	-12.12	-25.73
DEN	91.96	-0.41	-6.47	91.53	-0.52	-6.38	59.32	-1.24	-12.79	33.86	-1.30	-3.88
RCL	92.28	0.00	-6.15	93.97	0.00	-3.94	61.77	0.00	-22.07	38.23	0.00	-11.79
HAT	97.10	0.00	-1.33	97.49	0.00	-0.42	71.23	0.00	-12.61	44.51	0.00	-5.69
SupSup	97.02	0.00	-1.41	97.15	0.00	-0.73	71.44	0.00	-12.40	43.22	0.00	-6.98
WSN	97.16	0.00	-1.27	97.32	0.00	-0.59	72.84	0.00	-11.00	45.96	0.00	-4.24
DSN	98.24	0.01	-0.19	97.73	0.02	-0.18	75.17	0.02	-8.67	46.56	0.04	-3.64
Overall Performance. To examine the influence of task mixture, we shuffled the tasks with five different seeds, resulting in five lists with different task orders. The averaged results over five runs are presented in Table 1, where the deviation results are reported in Appendix D. We have the following observations: First, not all CL solutions outperform the naive SGD method, even mean-IMM performs slightly worse on PMNIST than SGD. We consider that this observation may be due to the sequential
(a) RMNIST
(b) CIFAR100
Figure 2: The accuracy performance during entire incremental learning.
(c) TinyImageNet
weight penalty during knowledge transfer, which could significantly hinder knowledge transfer between old and new tasks. Second, we find that CL methods with network expandability such as PGN and RCL do not perform as well as EWC. This indicates that merely iterative network expansion, without network pruning or neuron selection, has the potential of incurring an over-parameterized risk [13, 59]. On the other hand, HAT, SupSup, and WSN adhere to the merit of network pruning and mainly seek to produce a sub-network by partially selecting some neurons (weights), resulting in higher accuracy performance. However, they can only facilitate forward knowledge transfer, yielding only narrow knowledge consolidation. In contrast, our DSN outperforms all the baselines on all metrics such as achieving the  $3.20\%$  improvements on CIFAR-100 regarding ACC. In addition, to show the superiority of DSN in incremental learning from the fine-grained aspect, Fig. 2 presents the averaged results as the number of tasks increases, we can clearly observe that DSN outperforms other baselines during the entire incremental learning process.
Forward Knowledge Transfer. The results of Trans in Table 1 demonstrate that DSN is substantially better overall due to the flexible neuron selection. Specifically, DSN performed on PMNIST and RMNIST shows only a slight degradation over the single-task learning scenario. This finding indicates that we can use more compact sub-networks to take on multiple tasks incrementally, which is more practical in future computing contexts, such as edge computing and low resource constraints.
(a) RMNIST
Figure 3: The accuracy performance of the first task in incremental learning.
Backward Knowledge Transfer. The experimental results on BWT indicate that SGD suffers from severe knowledge forgetting or negative knowledge transfer issues as it does not take any effort to avoid or alleviate the catastrophic forgetting problem in task incremental learning. Other traditional CL approaches, such as EWC and IMM, choose to confine the shift of model parameters to maintain the old task's performance while learning new tasks sequentially. However, the penalty of parameters still affects the ability of the model to recall old tasks, and these methods also confront severe catastrophic forgetting problems. In comparison with forget-free methods like RCL, HAT, and WSN (BWT=0), DSN can even promote the previous task performance on four datasets (BWT>0). There are two reasons for this observation. First, existing forgetting-free solutions concentrate on simply freezing the neurons or weights used in the previous tasks while making the architecture of the task network regarding each old task unchangeable throughout the entire incremental learning process. As a result, it offers us only barely forgetting or no-forgetting gains. Second, our DSN owning data-free
replay enables the possibility of actual backward transfer, i.e., transferring new knowledge to previous tasks. Fig. 3 shows the performance of the first task throughout the incremental learning. It is obvious that HAT and WSN can only maintain the model performance on the first task while they cannot promote the performance of the first task when incrementally receiving new tasks. This suggests that newly acquired knowledge can help address inadequate experience or knowledge of past tasks.
(a) PMNIST
(b) RMNIST
(c) TinyImageNet
Figure 4: The layer-wise neuron usage in incremental learning.
Capacity issues. Fig. 4 shows the capacity monitoring as the sequence of tasks emerges, where we only show the first two fully-connected layers of TinyImageNet for better visualization. From a macro perspective, we can find that the total usage (dashed line) of neurons increases incrementally as more task comes. But we discover that the used neurons for each task are gradually increased, which suggests that DSN prefers to reuse more neurons from earlier tasks when a new task arrives. Especially, DSN reuses more neurons from the past on RMNIST as the total usage increases slowly. The rationale behind this is that although different tasks have different angles of images, they do not change the goal of image classification and those that arrive in sequence are implicitly similar. Furthermore, we investigate the scale impact of our defined hypernetwork. Specifically, we vary the neuron number of each layer in the hypernetwork with different initial learning rates. As shown in Fig. 5, it is evident to observe that a larger over-parameterized hypernetwork enables higher accuracy performance. Interestingly, we find that a smaller learning rate brings us higher gains. We conjecture this phenomenon indicates that over-fitting problem could degrade the model performance.
(a) RMNIST
Figure 5: Hypernetwork capacity in incremental learning varying different learning rates.
(b) CIFAR-100
Efficiency Issue. We investigate the efficiency concern regarding our DSN and the representative baselines. First, Table 2 reports the runtime of all approaches including DSN. We first observe that solutions with network expandability take significantly more time, especially for RCL with reinforcement learning. The methods with parameter penalty spend less time while subnetwork selection methods such as WSN cost slightly higher. As for DSN, it costs more time but less than network expandability-based methods. There are two reasons: 1) recent mask mechanisms in recent subnetwork selection methods such as WSN are weight-wise, where the constructed binary masks have the same shape as the weights. As such, we can couple with them in a parallel manner. 2) DSN needs to craft the impressions regarding the old tasks, yielding a higher time cost. Nevertheless, Table 3 shows that DSN and HAT need less trainable masks than WSN due to the neuron-wise masks.
Sensitivity Analysis. We evaluate the impact of  $\eta$  that aims to hold more room for future tasks. As illustrated in Fig. 6, we can find that a larger  $\eta$  will leave more room but bring a slight performance drop.  $\eta = 0$  indicates that we do not penalize the number of activated neurons for each task. We can find that the capacity is full while the accuracy performance is the worst. We note that we evaluated the impacts of other key hyperparameters. However, due to space limitations, more details are described in Appendix D.
Table 2: Statistics for full training and inference time, where 'h' represents hours.
Dataset	SGD	EWC	mean-IMM	mode-IMM	PGN	DEN	RCL	HAT	SupSup	WSN	DSN
PMNIST	1.24h	1.36h	1.42h	1.38h	2.52h	41.54h	36.18h	1.58h	1.55h	1.65h	2.43h
RMNIST	1.17h	1.64h	1.82h	1.04h	2.12h	20.17h	14.01h	1.49h	1.32h	1.57h	2.18h
CIFAR-100	0.09h	0.11h	0.24h	0.12h	0.52h	9.08h	8.4h	0.13h	0.30h	0.37h	1.21h
TinyImageNet	0.28h	0.32h	0.63h	1.07h	1.32h	23.45h	21.17h	0.35h	0.74h	0.81h	1.54h
(b) TinyImageNet
Figure 6: The  $\eta$  impact on accuracy and network capacity.
Table 3: Total number of masks.
Dataset	HAT	SupSup
P-MNIST	4.4K	40.0M
R-MNIST	4.4K	40.0M
CIFAR-100	93.4K	90.1M
TinyImageNet	93.4K	90.1M
Dataset	WSN	DSN
P-MNIST	40.0M	4.4K
R-MNIST	40.0M	4.4K
CIFAR-100	90.1M	93.4K
TinyImageNet	90.1M	93.4K
Motivated by Lottery Ticket Hypothesis, we introduced a Data-free Subnetworks (DSN) approach for task incremental learning, aiming at enhancing knowledge transfer across sequentially arrived tasks. DSN consists of two components, i.e., neuron-wise mask and data-free replay. The former aims to find an optimal subnetwork using task-specific masks for a new arriving task while the latter attempt to craft the impressions for transferring the knowledge to the past tasks. DSN allows for more flexible knowledge transfer between old and new tasks, whereas backward knowledge transfer was usually ignored before. The experimental results on four datasets show the superiority of our proposed DSN. As part of our future work, we plan to investigate how to devise a knowledge-inspired mask mechanism to enhance knowledge transfer as more tasks arrive sequentially.
This work was supported in part by the National Natural Science Foundation of China (Grant No.62102326 and No.62072077), the Natural Science Foundation of Sichuan Province (Grant No.2023NSFSC1411 and No.2022NSFSC0505), and the Guanghua Talent Project.
[1] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(07):3366-3385, 2022.
[2] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109–165. Elsevier, 1989.
[3] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526, 2017.
[53] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
[54] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[55] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[56] Yann LeCun. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/, 1998.
[57] A Kirzhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, volume 25, pages 1097-1105, 2012.
[58] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
[59] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, pages 1-42, 2019.