SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} = \frac {1}{| \mathcal {B} _ {t} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {t} |} \mathcal {L} _ {c e} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {t}\right), y _ {i} ^ {t}; \mathcal {C} _ {n}\right) + \frac {1}{| \mathcal {B} _ {\mathcal {M}} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {\mathcal {M}} |} \mathcal {L} _ {r e p} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {\mathcal {M}}\right), y _ {i} ^ {\mathcal {M}}\right). \tag {7}
$$
It can be seen from (7) that this method removes the term  $\ell_n(\cdot)$  from (3) and only retains  $\ell_{ce}(\cdot ;\mathcal{C}_n)$ . In particular, loss in (7) is a special case of the loss in (4). More specifically, we can get the loss in (7) by setting  $\beta_{1} = 1$  and  $\beta_{2} = 0$  in (4). SSIL [1] has a similar form to that in (7) but uses a different  $\mathcal{L}_{rep}$  compared to ER-ACE. Due to the lack of  $\ell_n(\cdot)$ , the loss in (7) can only leverage the new classes' samples kept in memory to learn the objective of new/old class distinction. In experiments, we will show that setting  $\beta_{2} = 0$  in (4) performs worse than setting  $\beta_{2}\neq 0$  in (4).
Some existing methods also incorporate the idea of separating objectives in continual learning. However, these methods are primarily designed for task-aware problem. For instance, bilevel memory system with knowledge projection (BMKP) [37] requires the task identities to choose the corresponding knowledge representations during testing, making it unsuitable in the task-agnostic problem. Space decoupling (SD) [46] does not explicitly mention that it only considers task-aware problem, but its experiments completely follow some task-aware methods [35, 26], indicating its focus on task-aware problem.
Datasets We use three popular datasets for evaluation, including Seq-CIFAR10 [3], Seq-CIFAR100 [10], and Seq-TinyImageNet [22]. Seq-CIFAR10 consists of 5 disjoint tasks with each task having 2 classes and 10k training samples. Seq-CIFAR100 consists of 5 disjoint tasks with each task having 20 classes and 10k training samples. Seq-TinyImageNet consists of 10 disjoint tasks with each task having 20 classes and 10k training samples. The statistics of different datasets are given in Appendix. All the experiments are for the task-agnostic problem.
Table 1: Classification results which are averaged across 5 runs.
Keeping Extra Model	Seq-CIFAR10	Seq-CIFAR100	Seq-TinyImageNet
no	joint	91.86±0.26	70.10±0.60	59.82±0.31
finetune	19.65±0.03	17.41±0.09	8.13±0.04
Buffer Size	500	5120	500	5120	500	5120
no	SCR [27]	57.95±1.57	82.47±0.44	23.06±0.22	45.02±0.67	8.37±0.26	18.20±0.48
PCR [25]	65.74±3.29	82.58±0.42	28.38±0.46	52.51±1.61	11.88±1.61	26.39±1.64
MIR [3]	63.93±0.39	83.73±0.97	27.80±0.52	53.73±0.82	11.22±0.43	30.60±0.40
ER-ACE [8]	68.45±1.78	83.49±0.40	40.67±0.06	58.56±0.91	17.73±0.56	37.99±0.17
ER [9]	61.78±0.72	83.64±0.95	27.69±0.58	53.86±0.57	10.36±0.11	27.54±0.30
LODE (ER)	68.87±0.71	83.73±0.48	41.52±1.22	58.59±0.48	17.77±1.03	38.34±0.04
DER++ [7]	73.29±0.96	85.66±0.14	42.08±1.71	62.73±0.58	19.28±0.61	39.72±0.47
LODE (DER++)	75.45±0.90	85.78±0.40	46.31±1.01	64.00±0.48	21.15±0.68	40.31±0.03
yes	CLS-ER [5]	70.73±0.54	85.73±0.29	51.21±0.84	60.17±0.38	29.44±1.66	45.66±0.47
TAMiL [6]	74.25±0.31	84.82±1.77	50.62±0.23	63.77±0.43	27.83±0.41	43.00±0.56
iCaRL [33]	61.60±2.03	72.01±0.62	49.59±0.95	54.23±0.28	20.01±0.50	30.34±0.18
BIC [43]	52.63±2.46	79.98±1.49	37.06±0.60	60.43±0.61	29.82±0.88	37.60±0.23
SSIL [1]	64.31±0.89	71.72±1.47	41.61±0.37	57.53±0.52	16.80±0.71	40.06±0.58
ESMER [36]	71.48±0.98	79.19±0.68	52.37±0.87	63.99±0.13	30.97±1.12	44.07±0.52
LODE (ESMER)	74.53±0.95	85.34±0.41	55.06±0.35	65.69±0.33	32.15±0.17	46.40±0.46
Baselines We compare our method with many state-of-the-art replay-based continual learning methods, including incremental classifier and representation learning (iCaRL) [33], bias correction (BIC) [43], separated softmax for incremental learning (SSIL) [1], experience replay (ER) [9], maximally interfere retrieval (MIR) [3], dark experience replay++ (DER++) [7], supervised contrastive replay (SCR) [27], proxy-based contrastive replay (PCR) [25], experience replay with asymmetric cross entropy (ER-ACE) [8], error sensitive modulating experience replay (ESMER) [36], complementary learning system experience replay (CLS-ER) [5], and task-specific attention modules in lifelong learning (TAMiL) [6]. For CLS-ER, we follow the existing method [6] and implement it with a single exponential moving average model. We also include two methods without continual learning, joint and finetune, in the comparison. Here, joint denotes the method which learns all the tasks jointly while finetune denotes the method which learns all the tasks sequentially without any memory. The accuracy of joint can be treated as the accuracy upper-bound and the accuracy of finetune can be treated as the accuracy lower-bound. Among the methods we mentioned above, some methods only maintain a single learning model to perform continual learning, while others require an extra model in memory for knowledge integration or distillation. Since keeping more models requires more memory, and memory cost is an important metric in continual learning [30], we group different methods by whether keeping extra model (refer to Table 1) to make a more fair comparison.
Architecture and Training Details We follow existing continual learning works [7, 6] and use standard ResNet18 as the neural network architecture in all the experiments unless otherwise stated. The experiments are built on top of the mammoth [7] continual learning repository in PyTorch like many existing works [7, 6]. We use stochastic gradient descent (SGD) to optimize the parameters. The batch size and replay size are set to 32 to follow the existing continual learning works [7, 6]. We also follow existing methods [7, 5] to set memory as 500 and 5120 for all the datasets. The hyperparameters are selected through a small validation set. For the experiments of all the methods on all the datasets, we apply random crops and horizontal flips to both newly coming samples and buffered (saved) samples like existing works [7, 5]. For each of our experiments, we report the average and standard deviation of the mean test accuracy of all the tasks across 5 runs with different seeds. More details of training and hyperparameters for different methods are given in Appendix.
4.2.1 Accuracy
Table 1 shows the results of different methods on different datasets. Here, 'Keeping Extra Model' represents whether a method needs to keep an extra model for continual learning, as described
Number of Tasks (a) Seq-CIFAR10
Number of Tasks (b) Seq-CIFAR10
Number of Tasks (c) 4 Tasks in Total
Number of Tasks (d) 10 Tasks in Total
Figure 3: (a) and (b) show the variation of the accuracy for different methods on Seq-CIFAR10 and Seq-CIFAR100. (c) and (d) show the variation of accuracy on Seq-CIFAR100 with different number of tasks.
(a) Seq-CIFAR10
Figure 4: (a) and (b) show the variation of the accuracy for different  $\rho$ . (c) and (d) show the variation of the accuracy for different  $C$ .
(b) Seq-CIFAR100
(c) Seq-CIFAR10
(d) Seq-CIFAR100
in Section 4.1. We combine our LODE with ER, DER++ and ESMER, which can be written in the form of (1). We use LODE (ER), LODE (DER++) and LODE (ESMER) to denote them, respectively. Our experimental results confirm that the integration of our LODE method improves the performance of each combined method. For example, when compared to DER++ with buffer size 500, LODE (DER++) exhibits a  $2.16\%$  improvement on Seq-CIFAR10, a  $4.23\%$  improvement on Seq-CIFAR100 and a  $1.87\%$  improvement on Seq-TinyImageNet. Notably, LODE (DER++) achieves the best performance on Seq-CIFAR10. LODE (ESMER) achieves the best performance on Seq-CIFAR100 and Seq-TinyImageNet.
Figure 3 (a) and Figure 3 (b) show the variation of accuracy on Seq-CIFAR10 and Seq-CIFAR100 after the learning of each task. As we can see, LODE improves DER++ and ESMER at the end of each task. Figure 3 (c) and Figure 3 (d) show the results on Seq-CIFAR100 with different numbers of tasks. When there are 4 tasks in Seq-CIFAR100, each task consists of 25 exclusive classes. Similarly, when there are 10 tasks in Seq-CIFAR100, each task consists of 10 exclusive classes. As we can see, when the number of tasks varies, LODE still gives improvements on different methods. In Appendix, we show the results of more methods on Seq-CIFAR100 with different numbers of tasks.
4.2.2 Hyperparameter Analysis
We analyze the hyperparameters in LODE. We choose LODE (DER++) and LODE (ESMER) to analyze hyperparameters, as they represent the best performance of methods that retain or do not retain an extra model for continual learning, respectively.
We first vary the value of  $\rho$  in (5) to show its impact on the performance of the model. Figure 4 (a) and Figure 4 (b) give the analysis on Seq-CIFAR10 and Seq-CIFAR100. Note that when  $\rho = 0$ ,  $\beta_{2} = 0$  and the weight of  $\ell_{n}(\cdot)$  in (4) is always zero. At this time, the loss in (4) degenerates to the loss in (7) and the model fails to get satisfactory performance. When the value of  $\rho$  increases,  $\beta_{2}$  also increases and the performance of the model first increases and then decreases. This phenomenon is intuitively reasonable since a larger weight for  $\ell_{n}(\cdot)$  leads to more forgetting and thus influences the overall model performance.
We also vary the value of  $C$  in (5) to show its impact on the performance of the model. Figure 4 (c) and Figure 4 (d) show the results of varying  $C$  on Seq-CIFAR10 and Seq-CIFAR100, respectively. We can find that the performance of the model decreases significantly when  $C$  is close to zero. In
Table 2: Ablation study on Seq-CIFAR10 and Seq-CIFAR100.
LODE (DER++)	LODE (ESMER)
Seq-CIFAR10	Seq-CIFAR100	Seq-CIFAR10	Seq-CIFAR100
β1=C,β2=ρ|Cn|/|C0| (Ours)	75.45±0.90	46.31±1.01	74.53±0.95	55.06±0.35
β1=β2=ρ|Cn|/|C0|	71.18±0.80	37.49±1.79	73.41±0.40	45.64±0.87
β1=β2=C	73.80±0.72	42.08±1.71	73.08±0.81	52.37±0.87
β1=ρ|Cn|/|C0|,β2=C	73.19±0.15	40.79±0.12	72.38±0.24	51.86±0.35
Table 3: Classification results which are averaged across 5 runs in the online continual learning setting.
Keeping Extra Model	Seq-CIFAR10	Seq-CIFAR100	Seq-TinyImageNet
no	SCR [27]	69.49±3.02	36.09±0.82	20.04±1.24
PCR [25]	73.28±1.83	34.89±0.67	23.84±0.60
ER-ACE [8]	69.17±1.64	35.24±0.51	23.42±0.34
MIR [3]	71.10±1.59	35.08±1.32	20.64±1.17
ER [9]	67.93±2.04	34.40±1.13	21.14±0.72
LODE (ER)	69.63±1.41	36.91±1.38	24.31±0.82
DER++ [7]	72.30±0.99	34.72±1.51	20.40±1.02
LODE (DER++)	74.00±0.08	37.82±1.16	25.30±1.80
particular, the model gets the best performance when the value of  $C$  is between 1 and 5. Since  $\rho$  and  $C$  are highly different when the model achieves the best performance, decoupling the loss through (3) is necessary for continual learning.
We change the value of  $\beta_{1}$  and  $\beta_{2}$  to show the effectiveness of setting  $\beta_{1}$  and  $\beta_{2}$  through (5). We first set the value of  $\beta_{1} = \beta_{2}$  in (4) to remove the decoupling property. There are two possibilities to set  $\beta_{1} = \beta_{2}$ . The first possibility is to set  $\beta_{1} = \beta_{2} = C$  and the second possibility is to set  $\beta_{1} = \beta_{2} = \rho \frac{|\mathcal{C}_{n}|}{|\mathcal{C}_{o}|}$ . Table 2 shows the results of these two possibilities, which are significantly inferior to our method. This means that separating the two different objectives by decoupling the loss of the new task is necessary for the model to achieve good performance. In Table 2, we also show the result of a variant by exchanging the value of  $\beta_{1}$  and  $\beta_{2}$ , which means  $\beta_{1} = \rho \frac{|\mathcal{C}_{n}|}{|\mathcal{C}_{o}|}$  and  $\beta_{2} = C$ . We can find that the performance of this variant is still significantly inferior to our method.
4.2.4 Online Continual Learning
We also perform experiment for the online continual learning setting [3, 10] where the datasets of different tasks are concatenated to a non-stationary data stream. Since online continual learning is more challenging than offline continual learning, existing methods [11] usually use larger memory in online continual learning, especially for challenging datasets such as Seq-CIFAR100 and SeqTinyImageNet. Hence, we set the memory size to 5120 for all the datasets. To follow existing online continual learning methods [3, 10], the experimental settings we use here are different from those introduced in Section 4.1. These settings are introduced in Appendix.
Table 3 shows the results of different methods. We exclude those methods that have been implemented only in offline continual learning or those that have demonstrated lower performance in online continual learning in previous works [8, 25]. Similar to that in the offline continual learning setting, we can find that the integration of our LODE method also significantly improves the overall accuracy of each combined method. For example, when compared with DER++, LODE (DER++) exhibits a  $1.70\%$  improvement on Seq-CIFAR10, a  $3.1\%$  improvement on Seq-CIFAR100 and a  $4.9\%$  improvement on Seq-TinyImageNet.
In this work, we propose a new method called loss decoupling (LODE) for continual learning. Different from most existing replay-based methods which mixes the two different learning objectives together to learn the new task, LODE separates the two learning objectives for the new task by decoupling the loss of the new task. Experiments show that LODE can achieve a better trade-off between stability and plasticity than other baselines, and thus outperform other state-of-the-art replay-based methods across multiple datasets. Future work will extend LODE to other continual learning problem like task-aware problem and study the effectiveness of LODE for other types of tasks.
This work is supported by NSFC Project (No.62192783), National Key R&D Program of China (No.2020YFA0713901), and Fundamental Research Funds for the Central Universities (No.020214380108).
[1] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. SSIL: separated softmax for incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 824-833, 2021.
[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte-laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision, pages 139-154, 2018.
[3] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. arXiv preprint arXiv:1908.04742, 2019.
[4] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pages 11816-11825, 2019.
[5] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. In International Conference on Learning Representations, 2022.
[6] Prashant Shivaram Bhat, Bahram Zonooz, and Elahe Arani. Task-aware information routing from common representation space in lifelong learning. In International Conference on Learning Representations, 2023.
[7] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems, pages 15920-15930, 2020.
[8] Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022.
[9] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019.
[10] Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In Proceedings of the International Conference on Machine Learning, pages 1952-1961, 2020.
[11] Aristotelis Chrysakis and Marie-Francine Moens. Online bias correction for task-free continual learning. In International Conference on Learning Representations, 2023.
[46] Zhen Zhao, Zhizhong Zhang, Xin Tan, Jun Liu, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Rethinking gradient projection continual learning: Stability/plasticity feature space decoupling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3718-3727, 2023.
[47] Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5871-5880, 2021.
A Statistics of Different Continual Learning Datasets
We give detailed statistics on datasets in this section. Specifically, Table 4 shows the detailed information of three datasets, including Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImageNet. These datasets are constructed by one of the following datasets: CIFAR10, CIFAR100 or TinyImageNet. For each task,  $5\%$  of the training samples are divided into a validation set.
Table 4: Statistics on three datasets
Seq-CIFAR10	Seq-CIFAR100	Seq-TinyImageNet
Task Number	5	5	10
Input Size	3 × 32 × 32	3 × 32 × 32	3 × 64 × 64
Classes per Task	5	20	20
Training Samples	10000	10000	10000
Testing Samples	1000	1000	1000
B More Experimental Details
Table 5 provides the hyperparameters for different methods, which will be used for comparison.  $\rho$  is selected among four different values [0.01, 0.05, 0.1, 0.2] through the validation sets we split from the training sets. We do not carefully select hyperparameter  $C$  but set it as 1 for all the experiments to reduce the reliance of the model on hyperparameters.
It is important to note that both  $\mathrm{DER}++$  and ESMER have their own specific hyperparameters that may affect the performance of the model. To ensure a fair comparison, we maintain the specific hyperparameters of each method consistent when combined with our LODE, except for the exponential decay hyperparameter  $\alpha$  in ESMER. This means that we use the same hyperparameters settings as the original implementation when evaluating our combined models. For the exponential decay hyperparameter  $\alpha$  in ESMER, we find that setting it to the same value as ESMER (0.999) makes the model too conservative to learn the new task in LODE (ESMER). Therefore, we set it to 0.998 in order to make the model not so conservative. Furthermore, we follow the mammoth continual learning repository [7] to set the specific hyperparameters in each method or get them by the validation sets we split from the training sets.
Table 5: List of hyperparameters for different methods
Methods	Buffer	Split-CIFAR10	Split-CIFAR100	Split-TinyImageNet
C	ρ	C	ρ	C	ρ
LODE (ER)	500	1	0.1	1	0.01	1	0.05
5120	1	0.1	1	0.05	1	0.05
LODE (DER++)	500	1	0.05	1	0.1	1	0.1
5120	1	0.1	1	0.2	1	0.1
LODE (ESMER)	500	1	0.1	1	0.05	1	0.05
5120	1	0.1	1	0.2	1	0.1
The learning rate is determined by a grid-search. Specifically, the open-source code of the existing work DER [7] has conducted a grid-search to determine hyperparameters, including the learning rate, for a wide range of baselines. Since our experimental setting follows this work, the hyperparameters, including the learning rate, are kept consistent with the code of this work. For the baselines that are not included in the code of DER but follow the experimental settings of DER, we follow their open-source code to set the hyperparameters, such as learning rate and epoch, which are also determined by grid-search. For the methods not following the experimental setting of DER, we conduct a grid-search on validation datasets to determine the hyperparameters, including the learning rate. When combining our LODE with other methods, we keep the learning rate consistent with the original methods.
Table 6: Variation of the first task's accuracy on Seq-CIFAR100.
After Task 1	After Task 2	After Task 3	After Task 4	After Task 5
Remove ℓce(·;Cn)	87.40±0.22	54.28±3.41	56.18±4.08	50.77±2.57	43.92±0.87
Remove ℓn(·)	87.40±0.22	82.38±0.89	63.27±0.75	54.18±0.68	48.50±0.95
Table 7: Results of combining with other methods.
Seq-CIFAR100
PASS [47]	47.45±0.37
LODE (PASS)	50.49±0.22
B.2 Experimental Setting in Online Continual Learning
Datasets We use three popular datasets for evaluation, including Seq-CIFAR10 [3], Seq-CIFAR100 [10], and Seq-TinyImageNet [22]. We follow existing online continual learning methods to partition tasks across different datasets for the purpose of evaluation. Specifically, Seq-CIFAR10 consists of 5 disjoint tasks with each task having 2 classes and 10k training samples. Seq-CIFAR100 consists of 20 disjoint tasks with each task having 5 classes and 2.5k training samples. Seq-TinyImageNet consists of 20 disjoint tasks with each task having 10 classes and 5k training samples. All the experiments are for the task-agnostic problem.
Training Details For all the datasets, we use reduced-ResNet18 as our model architecture, which has been commonly used in prior online continual learning methods [8, 41]. Reduced-ResNet18 is a variant of the original ResNet18 that has fewer channel sizes, and fewer filters overall. We use stochastic gradient descent (SGD) to update the model for all the methods. We set memory sizes  $\mathbf{M} = 5120$  for all the datasets. Like the existing online continual learning works [8, 3], the replay size  $|\mathcal{B}_{\mathcal{M}}|$  is the same as the mini-batch size  $|\mathcal{B}_t|$  (fixed to 10). Data augmentation is the same as that in offline continual learning. Specifically, we use data augmentation such as random cropping and random rotation for both new mini-batch  $\mathcal{B}_t$  and old mini-batch  $\mathcal{B}_{\mathcal{M}}$ .
C.1 Combine with More Methods
LODE can be applied in an exemplar-free setting. We apply our LODE to a popular exemplar-free continual learning method called prototype augmentation and self-supervision (PASS) [47], which is designed for task-agnostic problems and utilizes a cross-entropy loss for learning new tasks. Table 6 shows the variation of the accuracy for the first task during the learning of subsequent tasks. We can find that removing  $\ell_{n}(\cdot)$  makes the model suffer from less forgetting than removing  $\ell_{ce}(\cdot; \mathcal{C}_n)$ . Table 7 shows the final results of PASS and LODE (PASS). We can find that LODE (PASS) outperforms PASS on Seq-CIFAR100.
C.2 TSNE Visualization
We present the TSNE visualization in Figure 5, which shows the learned representations of different models on Seq-CIFAR10. As we can see, removing loss  $\ell_{n}(\cdot)$  makes the representation of the old classes overlap less than removing loss  $\ell_{ce}(\cdot;\mathcal{C}_n)$ . Therefore, removing loss  $\ell_{n}(\cdot)$  makes the model suffer from less forgetting than removing loss  $\ell_{ce}(\cdot;\mathcal{C}_n)$ . We will add this figure in the final version of our paper. Thanks for the suggestion.
C.3 Plasticity of the Model
We also add a graph in Figure 6, which shows the accuracy of each new task during the training of Seq-CIFAR10. As we can see, removing either loss  $\ell_{ce}(\cdot ;\mathcal{C}_n)$  or loss  $\ell_n(\cdot)$  decreases the model's plasticity and leads to lower performance on the new task. Hence, both of these two losses are necessary for the learning of a new task.
(a) TSNE visualization of removing  $\ell_{ce}(\cdot ;\mathcal{C}_n)$  after learning task 3.
(b) TSNE visualization of removing  $\ell_n(\cdot)$  after learning task 3.
(c) TSNE visualization of removing  $\ell_{ce}(\cdot ;\mathcal{C}_n)$  after learning task 4.
(d) TSNE visualization of removing  $\ell_n(\cdot)$  after learning task 4.
(a) Seq-CIFAR10
Figure 6: (a) and (b) show the variation of the new task's accuracy on different datasets.
Figure 5: TSNE visualization
Number of Tasks (b) Seq-CIFAR100
C.4 More Comparison