Solving Continual Learning via Problem Decomposition
Authors: Gyuhak Kim
Changnan Xiao
Tatsuya Konishi
Zixuan Ke
Bing Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Continual learning
lifelong learning

Abstract: This paper is concerned with class incremental learning (CIL) in continual learning (CL). CIL is the popular continual learning paradigm in which a system receives a sequence of tasks with different classes in each task and is expected to learn to predict the class of each test instance without given any task related information for the instance. Although many techniques have been proposed to solve CIL, it remains to be highly challenging due to the difficulty of dealing with catastrophic forgetting (CF). This paper starts from the first principle and proposes a novel method to solve the problem. The definition of CIL reveals that the problem can be decomposed into two probabilities: within-task prediction probability and task-id prediction probability. This paper proposes an effective technique to estimate these two probabilities based on the estimation of feature distributions in the latent space using incremental PCA and Mahalanobis distance. The proposed method does not require a memory buffer to save replay data and it outperforms strong baselines including replay-based methods.

Under review as a conference paper at ICLR 2023
SOLVING CONTINUAL LEARNING VIA PROBLEM DECOMPOSITION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper is concerned with class incremental learning (CIL) in continual learn- ing (CL). CIL the popular continual learning paradigm which system re- ceives sequence of tasks with different classes in each task and is expected to learn to predict the class of each test instance without given any task related infor- mation for the instance. Although many techniques have been proposed to solve CIL, remains to be highly challenging due to the difficulty of dealing with catas- trophic forgetting (CF). This paper starts from the first principle and proposes novel method to solve the problem. The definition of CIL reveals that the prob- lem can be decomposed into two probabilities: within-task prediction probability and task-id prediction probability. This paper proposes an effective technique to estimate these two probabilities based on the estimation of feature distributions in the latent space using incremental PCA and Mahalanobis distance. The proposed method does not require memory buffer save replay data and it outperforms strong baselines including replay-based methods
INTRODUCTION
Continual learning (CL) is a learning problem where system leams and accumulates knowledge over time without forgetting the previous knowledge (Chen & Liu 2018). The key challenge the catastrophic forgetting (CF), which is phenomenon that the system corrupts the learned knowledge the past in learning a new task (MeCloskey & Cohen p989). This paper focuses on the challenging CL setting of class incremental learning (CIL.) (Rebuffi et al. 2017) in the offline (or batch) mode. In this setting, the system learns a sequence of classification tasks incrementally, where each task arrives with all its training data of set of classes. The resulting classifier a test instance among all the classes learned the process with no task information provided. The other popular setting of CL is task incremental learning (TIL), which builds separate model for each task and in testing, the test instance together with the task-id that the test instance belongs to provided so that the system can use the model of the specific task to classify the instance.
identify the class of
Existing approaches to CIL can be grouped into several categories. Regularization (Kirkpatrick let al. (017) or distillation (Li & Hoiem] 2016) tries not to change the parameters knowledge that are important to old tasks when learning the new task. Replay/memooy--bace approaches (Rebuffi al. 2017) save some old data and use them jointly with the new task data to learn the new task and to preserve/adjust the old knowledge. Parameter isolation approaches (Serra et al. 2018) expand the network or mask out the important parameters for old tasks (see Sec. [ for more details). Our approach is entirely different and is derived directly from the definition of the CIL setting.
Class incremental learning (CIL) learns a sequence of tasks 1,.., t, where each task é has a training data D = {(x)) by with æy E X (input space) and y E Y" (class label space). The class labels of tasks are disjoint, Y n y = @ for any * k Let X = - X and Y - Y The goal is to learn function f X + Y to predict the class label test case x.
Definition:
'The code is included in the Supplementary Material. In (Bang et al. 2021 tasks are considered to have shared classes. For instance, the system receives two datasets D and D consisting of classes {yI, and {y1, ga, y4). respectively. We define task I and 2 consisting of {v1, 32) and (23, )d), respectively, and consider the samples of shared label y1 additional training data for task This work does not consider this learning scenario. We leave for our future work.

---

Under review as a conference paper at ICLR 2023
As tasks have disjoint classes, the CIL probability of a sample having the jth class label v) of task i can be decomposed into two probabilities,
P(ul(=) = P(vjaz, (i)
The decomposition implies that there are two probabilities that define the CIL probability. The first probability on the right-hand-side (RHS) is the within-lask prediction (WTP) probability (or intra- task prediction probability) and the second probability on the RHS is the task-id prediction (TIP) probability (or inter-task prediction probability). Thus, system makes a correct CIL predictiion if it produces accurate within-task and task-id predictions.
We note that the WTP probability is exactly the prediction probability in a TIL problem. However, in TIL. the task-id is given in inference or testing. Thus, to solve the CIL problem, one can learn like TIL and then design mechanism Some existing works have taken this approach (Rajasegaran et al.) 2020 Abati et al.] 2020). but they perform poorly because their task-id predictors are very weak. However, these papers did not propose Eq. I We will discuss these and other related works in the related work section. In fact, the WTP probability can be improved from that given by TIL system too.
predict the task-id which the test instance belongs.
This paper proposes novel technique to estimate the two probabilities and an exemplar-free CIL system, called EWT (Estimation of WTP and TIP probabilities). EWT makes use of the highly effective hard-attentin
masking method HAT (Serra et al. 2018) for TIL to learn feature extractor for each task. HAT has almost no forgetting for TIL as it masks out the parameters and neurons learned for previous tasks. This ensures that the estimated probabilities are robust and are not af- fected by forgetting in incremental learning. Although we could directly use the probability of each class produced from each task as WTP probability, this approach is sub-optimal. We propose gen- crative approach to improve the estimation by considering possible noisy and/or out-of-distribution samples. This is done by fine-tuning the task classifiers using generated pseudo feature representa- tions for each class. The generation is done based on the Gaussian distributions in the latent feature space. The Gaussian distribution for each class is estimated incrementally using incremental Prin- ciple Component Analysis (iPCA). The TIP probability is estimated using Mahalanobis distance.
Our experiments demonstrate the effectiveness of the proposed method EWT using pre-trained transformer network that does not have information leak, i.c., it is trained using the ImageNet data with all classes that are similar to the classes in the experiment datasets removed. Both our sys- tem and the baselines fix the transformer and train adapter modules inserted at each transformer layer (Houlsby al. 2019). The experimental results shows that EWT outperforms the recent state- of-the- art baselines by large margins, including replay-based approaches.
2 RELATED WORK
Numerous techniques have been proposed for CL. We consider the five most relevant categories: exemplar- free, replay-based, generative methods, network-expansion Exemplar-free methods (saving no previous task data) often use regularization (Kirkpatrick et al.,
and parameter isolation.
2017 Zhu et al. 2021), knowledge distillation (Li & Hoiem| 2016), or orthogonal projection Zeng et al. 2019) to preserve previous important parameters (Zenke et al., 2017 Wang et al.. 2022 Our method is also exemplar-free and our CF prevention is based on task masking (Serra et al. 20188)
Replay-based CL has been widely studied in CIL. Different saving mechanisms (Rebuffi et al.
2017
Liu et al., 2020b Bang et al. 2022), replay strategies (Aljundi al. 2019). and regulariza- Castro et al. 2018 Chaudhry et al.) 2018 Buzzega et al. 2020 tions (Lopez-PPe Paz & Ranzato 2
Chaudhry et aal1)) have been used. The goal of these methods to balance the plasticity and
stability using the saved samples of previous tasks (Liu et al. 2021 Yan et al. 202 Our method does not save any samples and also performs much better than recent replay- based methods.
Generative methods (Shin et al. 2017] Ostapenko et al. 2019 Ayub & Wagner 2021) build genera- tors to generate pscudo-replay data to prevent forgetting. Lesort et al. (2018) studied the difficulties of the generative approach. We do not generate pseudo-replay samples similar to the raw data and thus do not have its problems. Our method generates feature vectors rather than raw data. Liu et al. (2020a) and Zhu et al. (2021) also generate feature vectors. They use the generated features for dis- tilling knowledge of previous tasks. However, we estimate the distributions of features to fine-tune the classifier and to compute task-id probability rather than for knowledge distillation.
2

---

Under review as a conference paper at ICLR 2023
(N Eal)
(a) Step
(b) Step 2
Figure I: Overview of the training process for task i. The dashed lines indicate that the gradient flow blocked while the solid lines indicate gradients are computed along the lines. (a) The network is trained in multi-head manner (one per task), in which task-specific parameters are trained based on hard-attention and used to effectively eliminate the interference between tasks. During training, the system dynamically estimates the distributions N (( ('(x'). The estimated distributions are used to generate pseudo feature vectors, and the network jointly trained with the pseudo feature vectors and the training data of task i. Since the network g'of' is trained to minimize the loss on both training data and the pseudo feature vectors, this process encourages feature vectors of I' to follow the desired distribution. (b) Given the distributions, the system fine-tunes a classifier 9 created by joining the multi-head classifiers g², S i. The fine- tuning done using pseudo feature vectors generated from the distributions.
(x(((()))) of feature vectors
Network expansion methods (Rusu et al.) 2016 Mchta et al. 2021: Yan et al. 2021 expand the network to preserve old parameters. TBP- WF Meehta al.) 2021 uses global weight factors for knowledge sharing and a Bayesian non con-paaaeetric approach for network expansion. It first finds the task-id in testing using Gaussian distributions and then uses the task-id to select the correct model for prediction. Our method is different as we do not expand the network or find the task-id, but directly compute the CIL probability. DER (Yan et al. 202T expands the network and also does pruning to reduce the network size. Our method does not expand the network.
Another popular branch continual learning is parameter isolation, which trains set of task spe- cific parameters. The methods are mostly designed for task incremental learning (TIL) as they require the task-id of each test instance to choose the correct task specific parameters. We leverage the hard attention masking (HAT) (Serra et al.) 2018) to prevent forgetting. However, our method is for CIL unlike the original HAT. Although there are attempts (von Oswald et al. 2020; Rajasegaran eet al.) 2020 Abati et al.) 2020 Henning et al.) 2021 to use the parameter isolation methods for CIL problem, they do not tackle CIL by problem decomposition problematically as we do. These
methods are much weaker than ours (see Sec. 4
Using a pre-trained model (c.g.. BERT, GPT-3, VIT. DeiT, or CLIP) has been a standard practice for CL in natural language processing (Ke et al. (02T). For image data, Ostapenko et al. (2022) studied using pre-trained models as foundation models for CL. SLDA (Hayes Kanan 2020) fixes the pre- trained feature extractor and fine-tune the classifier. L2P (Wang et al. (022) trains a prompt pool with a fixed feature extractor and Wu et al. (2022) fine-tunes replicate layers of a pre-trained model. Our method EWT also leverages pre-trained feature extractor, but we use adapters in the fixed feature extractor and trains only the adapters to learn new knowledge. Using the same pre-trained model. our method outperforms SLDA and L2P by large margin (see Sec. 4).
Different CIL settings have been studied as well. Blurry task (or task-free) is studied in online CIL (Buzzega et al. 2020) Bang et al. 2022), where the tasks boundaries are not clear as tasks change gradually. Our method is an offline method, where tasks are disjoint. As noted in footnote we split tasks by unseen classes rather than by datasets. We leave training with additional samples of previous tasks for our future work in the online CL setting.
3 PROPOSED METHOD
An overview of the training process of the proposed method is illustrated in Fig. Learning new task i involves two steps. Step 1 focuses on training the feature extractor. Specifically. it trains the task network g¹ o using both the training data X of task i and the pseudo feature vectors Z gen- erated from the Gaussian distribution N ((,,(f" (X')), (f (X')) of feature vectors for each class in the task (see Fig. I(a)). The Gaussian distribution is dynamically and incrementally estimated
3

---

Under review as a conference paper at ICLR 2023
using incremental Principle Component Analysis (iPCA) during training (see Sec. 3.1.1. Since the network is jointly trained with the training data and the generated features, which also depend on the values of feature extractor f', this encourages the feature vector to follow the distribution. This step has little forgetting as the training is done based on the hard-attention mechanism in Serra et al. (0188), which can protect/mask the parameters learned from previous tasks (see Sec. [.1.2p. Note that although f are task specific but they are all learned in the same network and there are a lot of parameter sharing. Step computes the two probabilities in Eq. based on the trained feature extractors f' and the Gaussian distribution for each class.
3.1 STEP 1: TASK TRAINING
We first discuss the detailed training process to learn a task i step which performs two functions: (i) estimating the distribution of feature vectors for each class in a task using incremental Princi- pal Component Analysis (iPCA), and (ii) training the feature extractor with hard attention masking in (Srraa et al. 2018) to prevent interference or CF in learning task i using the training data X' and the generated feature vectors Z based on the incrementally estimated distributions on the fly. As Z in the step training because we want to produce better distributions, which will be used in step 2.
explain above,
We train the network for task by minimizing the loss
Loc
Σ
logplyla: + Σ
logp(ylz,1)
B
(B.
(z,8)
where the first p on the right the softmax output (x))), the second p is the softmax output g²)z), B is a batch of training data, and Z is a batch of pseudo feature vectors generated from the Gaussian distributions (N Σe)} for each class e task i. The following sub-sections describe how to estimate the distributions of feature vectors and how to train the feature extractor without CF.
3.1.1 INCREMENTAL ESTIMATION OF GAUSSIAN DISTRIBUTIONS OF FEATURE VECTORS
We use multivariate Gaussian distribution to approximate the feature distribution for each class. The challenges in estimating the distribution of features in CL are: (i) the statistics (Je, Σg) need to be updated incrementally at each batch on the evolving feature extractor as using the whole data recompute the statisties after training computationally demanding and (ii) saving statisties of the Gaussian distributions is expensive as the feature vectors have a high dimension d. Therefore, take ideas from the algorithms developed for incremental Principal Component Analysis (iPCA) to approximate the covariance. In iPCA, only few (k. << d) principal vectors are saved and updated dynamically each batch without using any previous data. Since the following discussion is about estimating the distribution of feature vectors of each class, we remove the class indicator for simplicity in notation. Likewise, we also remove task index
Suppose we have seen n training samples so far. Denote the n samples of a class by X and the the feature vectors by f(X) = Z [z],... 2n.] from the feature extractor while minimizing Eq, 2 Denote the sample mean by μ Σ Zn Suppose that the system receives new batch X new of m instances. We obtain the feature vectors Znew = [zn+1, mn+ and update the mean by
j (nę + m/nnww/( ( + m))
(3)
where Mnew is the sample mean of the new batch. Denote the singular value decomposition (SVD) of the centered feature by UAV and Iz- 14], where is the transpose symbol. We approximate the covariance with le leading cigenvectors and eigenvalues follows
(n - 1) N U.A.U.
(4)
For simplicity, we denote the reduced matrices UR by U and A by A as the following discussions are based on the reduced matrices. Based on [Ross et al., 2008), the SVD can be updated for a
new set of data Znew as ÜÂVT snd Ivn IAA K] given the block matrix K [Znea
Anew /nm/((n + ()(nee 14]], and obtain
(n + ÜU'T
(5)
4

---

Under review as a conference paper at ICLR 2023
The derivation for Eq. | is given in Appendix D The statisties for the Gaussian distribution of class c is estimated dynamically with & (<< d) eigenpairs, and pseudo feature vectors can be drawn from the estimated distribution to train the classifier.
3.1.2 HARD ATTENTION MASKING
In training the network g6 f using the data of task i and the generated pseudo feature vectors, we employ the hard attention mask (Serra et al. 2018) to prevent forgetting in the feature extractor.
The hard attention mask af is trainable pseudo binary 0-1 vector at each layer / of task i. It is element-weie wise multiplied to the output of the layer as aj @ h4 and blocks (for value of 0) or unblocks (for value of 1) the information flow from neurons of adjacent layers. Neurons with value are important for the task and thus need to be protected while neurons with value 0 are not necessary for the task and can be freely modifed without affecting other tasks.
More specifically, we modify the gradients of parameters that are important in performing the pre- vious tasks , 1) during training task i so the important parameters for previous tasks are unaffected. The gradient of parameter Wkj.1 at kth row and jihh column of layer is modified as
Vukkz, (1 min
2 Vwokj. (
<i
where d is accumulated attentions over previous tasks and is 1 if the hard attention of neuron k at layer / is ever used by any previous task (see (Serra al. 2018) for details).
To encourage parameter sharing and sparsity in the number of activated masks, a regularization is introduced as C, = Σr, (1-a(_))))) Ex, (1 (1)) The final objective train comprehensive task network without forgetting is
C Lee + Cr:
where Lee is the cross-entropy loss in Eq.
3.2
STEP 2: COMPUTING THE TWO PROBABILITIES IN EQ.
We now discuss how compute the within-task prediction (WTP) probability and the task-id pre- diction (TIP) probability.
3.2.1 COMPUTING THE WTP PROBABILITY
We could use the softmax probability of each class in a task as the within task prediction (WTP) probability for the class. However, this method is not the best for computing the probability (see the experiment section) because those samples that may be outliers, noises, or other claasso-y cases are unlikely get accurate probabilities, which also affect the probabilities of those samples that are easy to classify. We propose to consider possible out-of-distribution (OOD) samples in each task. However, we do not have OOD data for each task to in learning the task. Since already computed the distributions of feature representations for each class in step 1, for each task we could use the generated data from the distributions of the other tasks as the OOD data for the task. Although we could consider the generated data from previous tasks as OOD data when training a new task, we cannot use the generated data from later task to update the model of an earlier task because we no longer have the data of the earlier task and even if we can use the generated data of this earlier task, updating its feature extractor can cause serious forgetting because the feature extractors for different tasks share many parameters in the hard attentions (Serra et al. 2018).
have
We propose simpler method. We build and fine-tune (see below) a separate linear classifier (with one input layer and one output layer) considering the classes from all tasks learned so far using the generated feature vectors for each class in each task from the feature distribution of the class estimated in step 1. The advantage of this approach is that using single combined model/classifier we can consider the OOD data for all tasks because for the classes of a task, the classes of all other tasks can be considered as OOD data for the task.
After training each task in step 1, we have the set of distributions { (NN be) of features of each class e of the task < t. We then fine-tune combined classifier g, which is the classifier cre-
5

---

Under review as a conference paper at ICLR 2023
ated by joining the parameters of each task's 's classifier g', using pseudo feature vectors Z generated from the distributions. This is illustrated in Fig. [[b)
Note that in step 1, each task network g' o f is trained independently without considering the other task networks. In this step, we consider the outputs of all the tasks together and fine-tune the combined classifier g. We minimize the cross entropy loss
Len =
Σ
logp(ulz)
(8)
(Z.D)EZ Z
where the probability is computed using the softmax (=)3:)";;;'")))) The WTP probability of class (which is our class label vj in Eq. | is
Pcce, i) = softmax (o
(9)
3.2.2 COMPUTING THE TIP PROBABILITY
We now compute the task-id prediction (TIP) probability for a given test sample æ. We make use of the distance between the feature vector [*) of and distribution ((u,, Σ.) of features estimated by the training data. This has been used as an effective measure for OOD detection (Lee) al. 2018). We define the covariance of the distribution of task i E' E.ee Ee/|C"), where C" the set of classes of task and Σg is the covariance matrix computed by the method discussed in Sec.B.1.1 with all the principal components. We discard the class covariance after the computation to save memory. Given a set of distributions {N (pe: >e)ee of task and a test instance æ, we define the following score of the feature f (z))
Sec. 3.1.1
(w>) - 1/ (M E'}}
(10)
where MD is the Mahalanobis distance of sample to the distribution N (ue, E¹). The higher the value, the further away the sample from the distributions of task
Finally, the TIP probability for task is defined as,
P()))) 5² (x)) Σ (x))
(1I)
Eq. 11 is justified as sample that closer to distribution more like belong to the distribution.
EXPERIMENT
Baselines. We compare the proposed EWT with 11 baselines among which five are exemplar-fre (i.e.. saving previous task data) methods and six are replay-based methods. The exemplar free methods are: HAT (Serra et al. 2018). OWM (Zeng et al. 2019). SLDA (Hayes & Kanan 2020), PASS (Zhu al. 2021b, and L2P (Wang al. 2022).. For the multi-head method HAT, make prediction by taking arg max over the concatenated logits from each task network as it works the best among all the considered prediction methods (refer Appendix C for details). The replay methods are: iCaRL (Rebuffi et al. 2017), A-GEM (Chaudhry et al. 2018), EEIL (Castro et al.) 2018). DER++ (Buzzega al. 2020). HAL (Cauddhry et al. (202 and DER without pruning (Yan et al. 2021). We could not run (Wu et al. (022) as no code was released. We also do not include the existing parameter isolation methods that deal with CIL problems as they are very weak
Datasets. We use four popular continual learning benchmark datasets. (1). CIFARIO (rrizhessky
& Hinton 2009). This is an image classification dataset consisting of 60,000 color images of size 32x32. among which 50,000 are training data and 10,000 are testing data. It has 10 different classes.
'HyperNet (von Oswald and SupSup (Wortsman al. (0020) finds via gradient update. They then make within-task prediction. SupSup. PR. and iTAML (Rajasegaran et al. (2020) assume the test instances come in batches and all samples sample on ResNet- 18, HyperNet, SupSup. PR and iTAML achieve 22.4, 118, 45.2 and 33.5 on 10 tasks of CIFAR100 100, respectively, which are much lower than 51.4 of the baseline iCaRI.. CCG (Abati et al. 2020) and IBP-WF (Mehta et al. 2021 do not provide code.
2020) and PR (Henning al. 2021) find the task-id via an entropy function
in batch belong to one task. When tested

---

Under review as a conference paper at ICLR 2023
(2). CIFAR100 Krizhevssyy & Hinton 2009). This dataset consists of 50,000 training images and 10,000 testing images with 100 classes. Each image is colored and of size 32x32. (3). Tiny- ImageNet (Le & Yang, 2013). This classification dataset has 200 classes with 500 training images of size 64x64 per class. The validation data has 50 samples per class. Since no label is provided for the test data, we use the validation set for testing in (Zhu et al.] 2021). (4). ImageNet380. We randomly selected 380 classes from the 389 classes, which are the remaining classes after removing those classes similar those in CIFAR and Tiny-ImageNet from the original 1,000 classes of the full ImageNet data (Russakovsky et al. 2015) for pre-trainee -re-training (see below). This dataset has about 1,300 color images per class. Similar to Tiny-ImageNet above, we use the validation set (50 images per class) for testing as its original test data has no label.
Backbone Architecture. We use the backhone architecture of transformer DeiT-S/16 Touvron er al. (021). We initially pre-train the network using 611 classes of ImageNet after removing 389 classes which are similar or identical to the classes of CIFAR and Tiny-ImageNet. To leverage the strong performance of the pre-trained model while adapting to new knowledge, we fix the feature extractor and append trainable adapter modules of fully-connected networks with one hidden layer at each transformer layer (Houlsby et al., 2019) except SLDA and L2P The number of neurons in each hidden layer is 64 for CIFÁRIO and 128 for other datasets. Note that all baselines and our method use the same architecture and the same pre-training model for fairness as using pre- trained model improves the performance (Ostapenko et al., 2022) (e.g.. DER improves from 65.2 to 73.3 on 10 tasks of CIFAR100 with pre-training on the same transformer architecture).
Note that we do not use the pre-trained models like CLIP (Radford et al. 2021) or others trained using the full ImageNet data due information leak both in terms of features and class labels because our experiment data have been used in training these pre-trained models. This leakage can seriously affect the results. For example, the L2P system using the pre- mre-trainig model trained using the full ImageNet data performs extremely well, but after those overlapping classes are removed in pre-training, its performances drop greatly. In Table 1, we can see that in fact quite weak.
Training Details. For saving eigenpairs. we follow the existing memory budget strategy in the replay-based method (Chaudhry et af., (019) for fairness. We fix the total number of eigenpairs saved in the CL process. After learning a new task, the system discards q eigenpairs with the smallest eigenvalues from each class of the previous tasks to accommodate k eigenpairs of each newly learned class. This strategy maintains & eigenvectors and the corresponding eigenvalues per class in the budget. Denote the budget size by I.
For CIFARIO, split the 10 classes into 5 tasks with 2 classes per task. The size of the hidden layer for the adapter module is 64 and the number of eigenpairs is 10 per class. We refer the experiment as C10-5T. The memory budget size [M| for eigenpairs is 100.
For CIFAR100, we conduct two experiments. We split the 100 classes into 10 and 20 tasks, where each task has 10 classes and 5 classes, respectively. We refer the experiments as C100-10T and C100-20T. We choose [M] = 1,000 for both experiments.
For Tmag-em we conduct two experiments. We split the 200 classes into 5 tasks with 40 classes per task and 10 tasks with 20 classes per task. We refer the experiments as T-5T and T-10T, respectively. We save 2,000 eigenpairs in total for both experiments.
For ImageNer380, we split the classes into 10 tasks with 38 classes per task and save 7,600 eigenpairs in total. We refer the experiment 1380-10T.
For all the experiments of our system, we find a good set of learning rates and the number of epochs via validation data made of 10% of the training data. We train our model for 15 epochs and use SGD with batch size of 128 and with momentum value 0.9 for step 1. For the experiments of CIFARIO and CIFAR100, we use learning rate of 0.05 and 0.01, respectively. For Tiny-ImageNet and ImageNet, we use learning rate 0.005. We train the classifier in step 2 for 35 epochs with SGD with the same batch size and learning rate as step 1. Following the random class order protocol of the existing methods (Rebuffi et al. 2017 Yan et al., (021), we randomly generate 5 different class orders for each experiment and report the average accuracy over the 5 random orders. For replay- based baselines, we follow Rebuffi et al. (201)). The systems use the memory buffer of size 200 for
*For SLDA and 1.2P, we follow the original papers. SL.DA finc-tunes only the classifier with fixed feature extractor and 1.2P trains learmalle prompts.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Average classification accuracy after the final task. 'XTT' means X number of tasks. Our system EWT and all baselines used the pre-trained network. The last column shows the average of each method over all datasets and experiments. We highlight the best results in each column in bold.
Method
C10-ST
C100-10T
C100-20T
T-ST
T-1OT
1380-10T
Average
HAT
59.
68.99::021
6..83+0.62
65.85=0.60
62.05-t0.55
01,20
68.21
OWM
41.69:1..4
21.3943.18
16.9844.44
24.55=2.448
17.52+3.45
0.26+0000
20.40
SLDA
88.64+0.055
67.82+0..05
67.80-0.05
57.93=0.05
57.93++0.06
65.78-0005
67.65
PASS
86.21=1.10
68.9011.94
66.77=1.18
61.03=0.38
58.3410.42
6..27
67.75
L2P
73.5914.15
61.72.10.81
53,84=1.59
59.12-0.96
54.09:1.144
47.89:3.24
58,38
iCaRL
87.55=10.99
68.90:10.47
69.15
53.13=1.04
51.88:2.36
62.23-10.66
65.47
A A-GEM
56.33+77777
25.21+4.00
21.99+4.01
30.53=3.99
21.90:5:52
30,38+10.02
31.06
EEIL
82.34113.13
68.08:0.51
63.79-0.66
53.34=0.54
50.3810.97
63.37=0.49
63.55
DER+
84.63+22.91
69.73+0.99
1..03+1.46
55.84-2.21
54.20+3.28
66.53+2.36
66.83
HAL
84.38+2.70
67.17+1.50
67.37+1.45
52.80+2.37
55.25+3.60
64,83+2.60
65.30
DER
86.79+1.20
73.30+00.8
72.00+0.57
59.57-0.89
57.18+1.40
69.19+1.36
69.67
EWT
87.6011.77
74.15:10.40
71.06:1..35
66.16=0.22
64.5910.17
74.30
72.98
CIFARIO, 2,000 for CIFARI00 and Imageena and 7,600 for ImageNet and save a set of raw training samples according the saving strategy the respective original papers) For the other baselines, we follow the experiment setups as reported in their official papers.
Evaluation Metries. We use two metries: average classification accuracy (ACA) and average for- getting rate. ACA after the last task A, Σ Af/t, where A; the accuracy of the model
on task ith data after learning task t. The average forgetting rate after task is J, t A - A.. This is also referred backward transfer in other literature (Lopez-Paz Ranzato 2017) the incremental classification accuracy (ICA) and ACA at each task in Appendix
We report
4.1 RESULTS AND COMPARISON
I shows the average classification accuracy after the final task. The last column Average indicates the average performance of each method over the ex- periments. Our proposed method EWT performs the best on average. We achieve 72.98% while the best baseline (DER) achieves 69.67%. The performance gap is even larger when we compare with non-replay based methods. The best exemplar-free method is HAT and it achieves 68. (8.21% % on average, which much lower than our method.
Average Classification Accuracy. Tab.
The baselines SLDA and 1,P are proposed to lever- age strong pre-trained feature extractor in the original papers. SLDA freezes the feature extractor and only fine-tunes the classifier. It performs well for the simple experiment C10-5T but is significantly poorer than EWT on other experiments. This because the fixed feature extractor does not adapt method updates the feature extractor via adapter mod-
RAT
BOWM
SLDA
Forgeting Rate
60
MAPSS
860 2P
30
knowledge. Our
DEA-GEM
ules new knowledge and is able learn more com- plex problems, L2P trains set of prompt embeddings. In the original paper, it uses feature extractor that was pre- ree-trained with ImageNet-211 21k which already includes the classes of the continual learning evaluation datasets. When we remove the classes similar to the datasets used
0 C10-ST
C100-101
C100-201
Figure Average forgetting rate. The lower the rate, the better the method is.
CL, its performance drops dramatically (58.38% on average over the 6 experiments) and much poorer than method EWT (72.98% on average).
Average Forgetting Rate (Backward Transfer). We compare the forgetting rate of each system after learning the last task in Fig The forgetting rates of the proposed method EWT are 5.26, 8.75,
Note that we save eigenvectors where each vector is in dimension of 384 whereas the replay- methods save raw inputs. For memory of size [.M] and dataset with C' classes, our method and the replay methods save l = JMIC pairs and raw inputs, respectively. after the last task. Thus, for C100-10T, EWT takes 384K elements for the eigenpairs while replay methods consumes 6. M elements. Refer Appdendix B for details.
8

---

Under review as a conference paper at ICLR 2023
The average classification accuracy by different variants of the proposed technique. The variant S1 indicates the model after step 1. The variant S1 + S2 indicates the model after step 2 and XX + TIP indicates the model with TIP of Sec. 3.2.2 applied prediction.
Table 2:
C10-5T
C100-10T
C100-20T
T-5T
T-10T
1380-10T
SI SI + S2
74.00113.92 93
67.76.10.49
59.9110.66
64.35.1..11
60,48.10.24
70.6610.91
2..05+2.22
71.22++0.44
65.25+0.39 66.11+0.72
61.95+0.47
71.95+0.79
S1 + TIP
82.83
718840065
66.8440.50
65 65.97:00744
63.49-0.37
73.45+1.14
S1 + S2 + TIP (EWT)
87.60.11.77
14.15
71.06.10.35
66, 66.16.10.28
64.59
74.30.10.66
Table 3: The accuracy performance and the number of saved eigenpairs on C100-10T, [M] = m
indicates that a total of m eigenvectors are saved with their corresponding eigenvalues.
M
100
500
1,000
1,500
2,000
C100-10T 73.4110.006
74.1010.62
74.15=0.40
7..25.10.06
73.87:10.22
and 8,85 on C1O-ST, C100-10T and C100-20T, respectively. iCaRL forgets less than ours on C10- ST and C100-20T as it achieves 4.95 and 8.31, respectively. However, iCaRL was not able adapt to new knowledge effectively as its accuracies are much lower than our method EWT on the same experiments. The average accuracy over the experiments of EWT is 72.98 while that of iCaRL. only 65.47. According to the forgetting rates, the best baseline (DER) adapts to new knowledge well, but it was not able to retain the knowledge as effectively as our method. Its forgetting rates 13.36, 15.92, and 15.48 on C10-5T, C100-10T, and C100-20T, respectively, and much larger than ours. This results in lower average performance of DER than EWT.
4.2 ANALYSIS AND ABLATION
Performances of Different Variant methods. Tab. 2 shows the performance gain by adding each proposed technique. The methods in the first (S1) and second rows (S1 S2) only produce the WTP probability without TIP probability since the TIP is not computed. Thus, we cannot decompose the CIL. probabbbyyy EWT. Instead, we make CII. prediction by taking arg max over the concatenated logits from each task classifier g', which better than the other considered prediction methods (refer to Appendix Cp. From Tab. 2 fine-tuning the classifier via the generated feature vectors in step already improves the performance from step 1 as shown in the second row (S1 + S2). On C10- ST, C100-10T, and C100-20T, S2 improves more than 3% from S1. When the proposed task-id prediction (TIP) is introduced, the performance also improves represented in the third row (SI + TIP). fact, this is slightly better than S1 + S2 without TIP which implies the effectiveness of the proposed problem decomposition for CIL. Combining all the proposed techniques together delivers the best performance as represented by the last row, which the full EWT.
Performance by the Number of Eigenpairs. Step and 2 are based on generating pseudo feature vectors from Gaussian distributions. Due to the memory consumption, we approximate the covari- ance by incremental PCA and save only |M| eigenvectors with the corresponding eigenvalues. This equivalent to saving |MI/ eigenpairs per class for a dataset of C classes when learning the last task. Tab. shows the model performance
C100-10T with different [M] sizes. With a single eigenvector per class (i.e., |M = 100), the model already achieves 73.41% accuracy. The perfor- mance increases with the size of M until |M| =2,000. The lower performance on 2,000 is because the less informative eigenpairs now generate noisy feature vectors.
5 CONCLUSION
This paper proposed an effective method to solve dlass-incremental learning (CIL) from the first principle. Based on the definition of CIL, it first decomposes the CIL prediction probability into two probabilities, within-task prediction (WTP) probability and task-id prediction (TIP) probabil- ity. Novel methods are designed to estimate these probabilities, which are based on an incremental PCA-based generative approach to fine-tune the multi-head task classifiers using a single head ap- proach and Mahalanobis distance, respectively. Experimental results show that the proposed EWT outperforms existing strong baselines by large margin.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Ehte- shami Bejnordi. Conditional channel gated networks for task-aware continual learning. CVPR, pp. 3931-3940, 2020.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelarrs, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Caccia. Online continual learning with maximal interfered retrieval. In NeurIPS, 2019.
Ali Ayub and Alan Wagner. {EEC}: Learning to encode and regenerate images for contin- ual learning. In International Conference on Learning Representations, 2021. URL https: //openreview. net. / forum? www.mmodowwaaaaaroooo
91cFU
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Wop Woo Ha, and Jonghyun Choi. Rainbow mem- ory: Continual learning with memory of diverse samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218-8221 2021.
Jihwan Bang, Hyunseo Koh, Seulki Park, Hwanjun Song, Jung- Woo Ha, and Jonghyun Choi. Online continual learning on a contaminated data stream with blurry task boundaries. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9275-9284, 2022.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA. Dark experience for general continual learning: strong, simple baseline. In NeurlPS, 2020.
Francisco M Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), 2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:/8/2.0044 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. 2019.
Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. Proceedings of the AAAI Conference on Artificial Intelligence, 3593) May 2021. URL htttittps:///o aaa wwrg/rnneeee php/
AAAT/article/// 16861
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Morgan & Claypool Publishers, 2018.
Tyler L Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear dis- criminant analysis. In CVPR Workshop Continual Learning, 2020.
Christian Henning, Maria Cervera, Francesco D' D'Angelo, Johannes Von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, Benjamin F Grewe, and João Sacramento. Posterior meta- replay for continual learning. Advances in Neural Information Processing Systems, 34, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICMI., 2019.
Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowl- edge transfer in continual learning. Advances in Neural Information Processing Systems. 34, 2021.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A and Others.
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):35213322 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report TR-2009, University of Toronto, Toronto., 2009.
10

---

Under review as a conference paper at ICLR 2023
Y. Le and X. Yang. Tiny imagenet visual recognition challenge, 2015.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. http://arzzwaor/ddd///8211111111 2018.
Zhizhong Li and Derek Hoiem. Learning Without Forgetting. In ECCV, pp. 2016.
Springer,
Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. Generative feature replay for class-ineremental learn- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 226--27,22 2020a.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. CVPR, 2020b.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In CVPR, 2021.
David Lopez-Pate Paz and Marc' Aurelio Ranzato. Gradient Episodic Memory for Continual Learning. In NeurIPS, pp. 6470-6479, 2017.
Michael McCloskey and Neal Cohen, Catastrophic interference in comnectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Nikhil Mehta, Kevin Liang, Vinay Kumar Verma, and Lawrence Carin. Continual learning using bayesian nonparametric dictionary of weight factors. In International Conference on Artificial Intelligence and Statistics, pp. 100-108. PMLR, 2021.
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11321-11329, 2019.
Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, and Laurent Charlin. Continual learning with foundation models: An empirical study of latent replay. Conference on Lifelong Learning Agents, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 103.0000, 2021.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. In CVPR, 2020.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, pp. 5543, 2017.
David A Ross, Jongwoo Lim, Ruei-Sung Lin, and Ming- Ming-Hsuan Yang. Incremental learning for robust visual tracking. IJCV, 77(1), 2008
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: arXiv:1606.0440 04671, 2016.
11

---

Under review as a conference paper at ICLR 2023
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548 4557. PMLR, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS. pp.
2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. ICLR, 2020.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022.
Mitchell Wortsman, Vivek R., R. Liu, A. Kembhavi, M. Rastegari, J. Yosinski, and A. Farhadi. Supermasks in superposition. In NeurIPS, 2020. UtL/tttpe://proceddng.. neurips.. http://f.......t....ooooooooaaa.
Tz- Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class ctass-incremental learning with strong pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9601-9610, June 2022.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014-3023, 2021.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continuous learning of context-dependent processing in neural networks. Nature Machine Intelligence, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pp. 3987-3995, 2017.
Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. pp. 5871-5880, June 2021.
A PSEUDO-CODE
We provide the pseudo-code for training and testing. Our comments start with symbol "//",
Algorithm Training (Step 1)
1: for training data D' of each task do
2: for each batch (X,, Yj) C D until converge do Obtain features Z, = f*(X,) and outputs g" (Z,)
3:
// Compute the mean and eigenpairs for features of each class in the task for each class e feature Z, C Z, do
4:
5:
Compute μe using Eq. 3 and the eigenpairs (U., Aa) using Eq. end for
6:
// Train the classifier with generated features and remember the distributions
7: Generate pseudo features Z from the distributions of the current task and obtain g¹ (Z)
8: Minimize Eq. 7 and update the parameters
9:
end for
10: end for
12

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Training (Step 2)
I/ After training task t, fine-tune the classifier
1: Construct y = Ig' .."'"""'') by concatenating the parameters of each task classifier g²
// Fine-tuning starts
2: for until converge do
3: Generate pscudo feature vectors Z from the distributions of all the learned classes
4: Minimize Eq. and update the parameters
5: end for
Algorithm Testing
Require:
["], and classifiers [g* "g",.....d 9 after learning task / // Obtain CIL probabilities of the classes corresponding each task
Test instance æ. task networks
1: for for each task S t do
2: Obtain WTP using Eq. and TIP using Eq. 11
3: Obtain the CIL probability p p(Yi))) = p( Y az, ()p(il=), where Y is the set of class labels
of task i
4: end for
// Concatenate the probabilities for full CIL probability and make a prediction
5: arg max D),((Ylaa [x),, where + is concatenation
Table 4: The size of the model (in entries) required for each method without the memory buffer.
Method
C10-5T
C100-101
C100-20T
T-5T
T-10T
1380-10T
HAT
24.1M
24.4M
24.7M
24.3M
24.4M
24.5M
OWM
26.6M
28.1M
28.1M
28.2M
28.2M
28.3M
SLDA
21.6M
21.6M
21.6M
21.7M
21.7M
21.7M
PASS
22.9M
24.2M
24.2M
24.3M
24.4M
24.6M
L2P
21.7M
21.7M
21.7M
21.8M
21.8M
21.8M
CCRRL
22.9M
24.1M
24.1M
24.1M
24.1M
24.2M
A-GEM
26.5M
31.4M
31.4M
31.5M
31.5M
31.6M
EEIL
22.9M
24.1M
24.1M
24.1M
24.1M
24.2M
DER+
22.9M
24.1M
24.1M
24.1M
24.1M
24.2M
HAL
22.9M
24.1M
24.1M
24.1M
24.
24.2M
DER
27.7M
45.4M
69.1M
33.6M
45,5M
45,5M
EWT
24.1M
24.4M
24.7M
24.3M
24.4M
24.5M
B REQUIRED MEMORY
We report the network sizes of the systems after learning the last task. We use an "entry" to denote parameter value required learn and inference for task.
All the systems except SLDA and L2P use the feature extractor DeiT-S/16 (Touvron et al. 2021 and adapter modules. The transformer consumes 21.6 millions (M) entries and the adapters take 1.2M and 2.4M entries for CIFAR10 and the other datasets. SLDA tne-ttnes only the classifier on top of the fixed pre-trained feature extractor as it does not have a protection mechanism. L2P uses prompt pool with 23k entries. Since each method requires method-speciff elements (e.g., task embedding for HAT), the number of entries required for each method different. The number of entries for each model is reported in Tab.
Our method saves the mean and eigenpairs to approximate the distribution of features for each class to draw pseudo feature vectors while the replay-based methods save the raw inputs to replay jointly with the current task data. As the number of eigenpairs and the number of saved inputs affect the performance,
budget M of size [M] each method can save. Since feature dimension is 384, the total entries required for saving the mean and eigenpairs for our method are 42.2k, 422.4k, 84 and 3. M. for CIFAR 10, CIFARIOM 100, Imag-em, and ImageNet380, respectively. The
13

---

Under review as a conference paper at ICLR 2023
Table The methods 1). 2), and 3) indicate entropy-based WTP prediction, softmax-based predic- tion, and logit-based prediction, respectively, as described Appendix C means the average value over the experiments.
The last column Average
Method
C10-ST
C100-10T
C100-20T
T-ST
T-10T
1380-10T
Average
HAT
!)
19.27=5.15
66.14=0.33
61.32++0.64
1..89+1.63
55.83+1.55
61.26++3.25
63.45
2)
79.27=5.15
68.96=0.29
61.83:10.66
65, 65.64:00.62
62. 12:1:.00,66
11.16:00.87
68.
3)
79.36=5.166
68.99-0.21
61.83:1062
65.85:10.60
62.05
71.20:10.99
68.21
S1
!)
73.71=4.06
65.05-0.71
59.74110.56
16.34
1..0
61.60:12.80
61.58
2)
73.71=4.06
67.89=0445
60.00+1..73
64.29
60.55:10.21
70.68:10.84
66.19
3)
74.00=3.93
67.76=0.49
59.91
64.35+:0.61
60.48:0.24
70.66::0.11
66.19
SI
S2
1)
76.55+4.10
67.25-0.43
63.90+0.71
1..7
53.66+0.82
59.40+1.67 1.67
62.58
2)
76.55=4.10
71.21-0.47
60.05
65.29-10.32
61.75110117
72.32:1.255
68.70
3)
84.05+2.24
71.22=0.44
66.11+0.72
65.25-+0.39
61.95+0047
71.95+0.79
70.09
sizes of raw inputs are 3#32*32, 3*64*64, and 3*224*224 (after resize) for CIFAR, Tiny-ImageNet, and ImageNet380. Thus, the total entries required for memory budget are 614.4k, 6.1 24.6M, and 1,144.0M for CIFARIO, CIFARIO0, Tiny-ImageNet, and ImageNet380.
Finally, our system saves the covariance matrices for computing TIP in Sec. 3.2.2 The covariances saved for each task. Since each covariance is in size 384x384, the total entries for this step are 737.3k, 1.5M, 2.9M, 737.3k. 1.5M, and 1.5M for C10-5T, C100-10T, C100-20T, T-5T, T-10T, and 1380- 1800-10T, respectively. The numbers are relatively small considering that some of the replay- based methods (e.g., iCaRL, HAL) require teacher model the same size as the training model for knowledge distillation.
C DIFFERENT PREDICTION METHODS
As HAT is designed for task incremental learning and does not provide a task-id prediction mecha- nism as the other parameter isolation methods such as HyperNet (von Oswald et al. 2020), we have tried different CIL. prediction methods. The reported values in Tab. are the results of the best one among the considered methods.
We considered three methods:
1) arg max p( lar, i), where the task-id i chosen based on the entropy values from each task network as HyperNet.
2) arg max (peY|[(([]a tained by taking softmax over logits ((() of task i. This is equivalent using equal proba- bility for task-id prediction (TIP) probability.
: t)), where the within-task prediction (WTP) probability is ob-
3) arg max[g* (f²) (
Tab. 5) shows the results each prediction method. Based on the result, the entropy- prediction performs the worst. The reason is that the entropy value from each task network is not informative as other values since it is not trained with entropy. The tiltmax-based and logit-based predictions not different average over the experiments. Since logit-based performance is the best, choose it as the CIL prediction method for HAT.
S1 and S1+S2 in Tab. in the main paper also do not have CIL prediction mechanism. We try the three prediction methods as HAT. The results are in Tab. []] We can observe similar behaviors in and S1+$2 HAT.
14

---

Under review as a conference paper at ICLR 2023
Table 6: Incremental classification accuracy. The last column shows the average of accuracies of each method over all the experiments. We highlight the best results in each column in bold.
Method
C10-ST
C100-10T
C100-20T
T-ST
T-1OT
1380-10T
Average
HAT
87.64112.63
19.26=1.113
73.91=0.68
74.26-00.6
72.61+0.69
19.40=1.47
77.84
OWM
56,00+33.46
1..10
12.58+1.58
45.18+0.33
35.75+2.21
5.95+-4.76
35,93
SLDA
93.5410.66
77.72-1..88
78.51:0.58
1..03=11.35
67.39+1811
69.66-0.02
75.48
PASS
89.03+7.13
77.01+2.44
76.42+1.23
67.12+6.26
67.33+3.63
74.76+2..33
75.28
1.2P
84.60+2.228
72.88+1.18
66.52+1.61
67.81+1.25
64.59+1.59
68.09+1..73
70.75
iCaLL
89.7411.63
76.50:13.56
77.06:12.36
61.36:16.21
63.56+3..8
73.71
73.65
A-GEM
68.19443.2
43.83++0.69
35.97=11.15
49.26-10664
39.58+3.32
50.16::6.63
47.83
BEIL
90.50=10.2
81.10=0.377
79.5411..9
66.63+10.4
66.54+0.61
15.0
76.57
DER+
89.01=6.29
80.64442.74
81.72++1.76
66.55-13.73
67.14+1.40
77.41-0.37
77.08
HAL
87.00=1.227
77.42
17.85
65.31=33.68
64,4811445
75.87=00400
74.65
DER
92.83+1.10
82.89
82.79+00.76
70.32+00.57
70,21+0.86
78.30-0067
79.56
EWT
13,20
82.57:0..69
80.52:10.85
74.27:10.70
73.87=1.00
81.24
80.94
D ADDITIONAL DERIVATION DETAILS
We have claimed that the orthonormal matrix Ü and diagonal matrix A obtained from Eq. in the main text are eigenvectors and eigenvalues of unnormalized sample covariance (n + m 1) Denote the previous sample mean by l4 and the eigenpairs of previous covariance Eoid by
(U, A2). Following Ross al. (2008), we provide more details about the claim. Since K
[Xnow Hmew /nm n + (Moe [4)] and UAUT
Eold.
ÚK = -ÜAV ÜÜAT
(12)
= [Vn 1UA KI[vn - I R)
(13)
(n U + RRT
(14)
(n 1)Eon (m 1)Enew
(15)
Yum
(Mnew (1) 12)
+ n m
(n + m 1)Σ
(16)
where the last derivation from Eq. Eq. 16 by Lemma 1 of Ross al. (2008).
INCREMENTAL CLASSIFICATION ACCURACY
In the main paper, we reported the average classification accuracy (ACA) after learning the last task. In this section, also report the incremental classification accuracy (ICA) over the learning process. ICA after task í. is defined as A Σi= A; where A; is ACA after learning task i. Tab. 6 shows ICA of our method EWT and the baselines. For the more challenging datasets (c.g.. Tiny- ImagemeeeNet and ImageNet), our system outperforms the baselines. SLDA is slightly better than our method on C10-5T and DER slightly better than EWT on C100-10T and 20T. However, their performances not consistent over different experiments. The average performance of our method over the experiments is 80.94 while the best performing baseline DER is 79.56. Fig. 3 shows the ACA at each task.
15

---

Under review as a conference paper at ICLR 2023
C10-ST
C100-10T
100
100
Accurecy (%)
Accuracy (%)
50
50
40
40
30
30
HAT
L2P
DEK
HAT
---
L2P
DER-+
20
OWM
ICERL
HAL
20
OWN
ICaRL
HAL
A-GEM
A-GEM
10
10
0
2
10
10
20
30
40
50
60
70
80
90
100
Number of Classes
Number of Classes
(a)
(b)
C100-20T
T-5T
100
100
Accurayy (%)
Aeeuracy (%)
50
40
40
30
HAT
L2P
DER
30
---
HAT
-
L2P
DER 1
20
OWM
iCaLL
HAL
20
OWN
iCaRL
HAL
10
10
PASS
EWT
REIL
BWT
10 29 35 as 55 65 75
80 90 95100
40
80
120
160
200
Number of Classes
Number of Classes
(c)
(d)
T-10T
1380-10T
100
100
90
Accuracy (%)
Aecuracy (%)
40
40
30
HAT
L2P
DER
30
HAT
-
L2P
DER-1
20
OWM
iCARL
HAL
20
OWM
iCaRL
HAL
10
10
PASS
EEIL
EWT
PASS
EEIL
EWT
20
40
60
140 160 Number of Classes
80
100 120
180 200
38
76
114 152 190 228 266 304 342 380
Number of Classes
(c)
(f)
Figure 3: Average classification accuracy after each task. The x-axis indicates the number of learned classes after each task. The systems with dashed lines are exemplar-free methods.
16


Reviews:
[{"review_id": "fL_MPwJ7KE2", "reviewer": "Reviewer_D1ii", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper decomposes the CIL problem into two prediction steps, which are within-task prediction and task-id prediction. Several strategies are designed to tackle these sub-problems, e.g. HAT, PCA, single head classifier fine-tuning. Experimental results show that the proposed EWT works better than existing baselines on pre-trained transformer network.", "summary_of_the_review": "Based on the above comments, I tend to reject this paper. The main reason is that the experimental settings are insufficient to prove the effectiveness of the method.", "strength_and_weaknesses": "Strength:\n1. This paper proposed a new perspective to decompose the CIL probability into WTH and TIP probability.\n2. For the above two sub-problems, the paper proposes a simple yet effective solution based on HAT, PCA, and features replay.\n3. The proposed EWT outperforms the existing methods without exemplar replay.\n\nWeaknesses:\n1. The pre-trained model used in the experiments still too strong for CIFAR and Tiny-/Sub-ImageNet.\n2. The proposed methods could be highly dependent on the good feature representation obtained by pre-trained model. If not,  some experiments under normal settings are needed to prove it.\n3. There is no quantitatively comparison of additional storage between EWT and other replay-based methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O2d4Dz54V_", "reviewer": "Reviewer_cHhV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers class incremental continual learning, in which disjoint subsets of classes are introduced one at a time. A multi-head architecture is used which shares a pre-trained feature extractor (backbone). Forgetting is addressed by using HAT (Serra et al., 2018) which applies a separate binary mask to the backbone for each task (subset of classes).\nThe authors decompose the prediction into task-id prediction (TIP) probability and within-task prediction (WTP) probability. Both are estimated by making use of class-specific approximations of the features distribution.\nThe resulting method is shown to outperform competitive baselines and the design decisions are motivated with ablation experiments.\n", "summary_of_the_review": "The paper present  interesting ideas and satisfactory experiments. Conversely, I am uncertain about the novelty of the method.\n", "strength_and_weaknesses": "I think that the method is comprised of insights which could be of interest to the community. Moreover, the experiment compare the method many baselines and conduct ablation studies.\nOne potential weakness is that the approach relies on a pre-trained backbone. Another issue is that the “Related Work” section does not describe and compare to the most closely related work, but rather just cites it.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IXayuZ8SDj", "reviewer": "Reviewer_JguF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed an approach for class incremental learning by leveraging the conditional of the labels given the data (covariates), which they decompose in terms of the labels given the task (consistent with task incremental learning) and the task given the data (task predictor), this without the need of a replay buffer or pseudo-rehearsal. The task prediction is realized incrementally via independent principal component analysis and quantified via Mahalanobis distance.", "summary_of_the_review": "The authors propose a conceptually simple approach for CIL based on incrementally estimated summaries over previous tasks via iPCA and the hard attention masking of HAT. The resulting approach is computationally efficient, leverages a pre-trained transformer encoder and delivers excellent performance. ", "strength_and_weaknesses": "The authors propose a relatively simple approach to address CIL tasks via estimating features for each task using iPCA and hard attention masking (i.e., the HAT core) to prevent catastrophic forgetting. The approach simple and intuitive, computationally efficient and leverages the hard attention mechanism in HAT as a means to mitigate catastrophic forgetting.\n\nAs a drawback, the authors seem to omit the existing literature in expansion-based approaches for continual learning. Further, expansion-based approaches have demonstrated excellent performance (at the time, so of which, do not consider transformer architectures) in terms of catastrophic forgetting at the expense of moderate parameter growth. For instance, the authors do not mention Mehta et al. 2021, a nonparametric network-expansion approach that also uses a Gaussian distribution and a likelihood similarity (with a Mahalanobis distance calculation at its core) to predict task labels. The latter, though probably will not be able to compete in terms of performance, underscores that the idea of using a summary in representation learning space and a similarity function as a means to identify tasks is not new, thus reducing the novelty of the proposed approach.\n\nThe experimental results are extensive and convincing, specially the fact that all baselines use the same architecture and that the transformer was trained to exclude classes similar to CIFAR and Tiny-ImageNet. The details of the experiments are clear and the ablation studies in Table 2 and 3 help understand the contribution of each component of the proposed approach.\n\nAs a minor note, Figure 2 is hard to see, so changing the visualization or simply replacing it by a Table will improve the presentation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IC6oeTCMGF", "reviewer": "Reviewer_vcxy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses continual learning where task IDs are not available.\nThe authors propose a learning method preventing catastrophic forgetting and providing task ID estimation capability.\nThe proposed method utilizes batches of features for each task generated from the generators obtained with incremental PCA. At this time, the OOD generator learned from past tasks is also used. The method estimates the ID for the current sample as that of the closest task in terms of the Mahalanobis distance from the mean and variance of the distribution of features of each task.\nExperimental results on multiple public datasets demonstrate that the proposed method performs better than existing methods.", "summary_of_the_review": "The clarity is low. The motivation and details of the proposed method are unclear.", "strength_and_weaknesses": "*Strength\n- Experimental results of the proposed method showed promising performance.\n\n*Weaknesses\n- The important assumption of why the class labels of tasks are disjoint is not well justified.\n- The problem setting is unclear. It is described in about four lines in the Definition paragraph but without a specific flow. It is difficult to understand what they actually do. For example, I cannot see when task IDs are estimated, when tasks are switched, tasks come in altogether; they come once, they don't come again, or they come repeatedly. Will all data be discarded each time, or will it be discarded for each task?\n- The motivation of Eq.2 is unclear. Why do the authors introduce pseudo feature vectors Z? The description of the method is also unclear. How can we generate y for Z? Where does B come from?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject