Class-Incremental Learning via Dual Augmentation
Authors: Fei Zhu
Zhen Cheng
Xu-yao Zhang
Cheng-lin Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: class-incremental learning
continual learning
catastrophic forgetting
augmentation

Abstract: Deep learning systems typically suffer from catastrophic forgetting of past knowledge when acquiring new skills continually. In this paper, we emphasize two dilemmas, representation bias and classifier bias in class-incremental learning, and present a simple and novel approach that employs explicit class augmentation (classAug) and implicit semantic augmentation (semanAug) to address the two biases, respectively. On the one hand, we propose to address the representation bias by learning transferable and diverse representations. Specifically, we investigate the feature representations in incremental learning based on spectral analysis and present a simple technique called classAug, to let the model see more classes during training for learning representations transferable across classes. On the other hand, to overcome the classifier bias, semanAug implicitly involves the simultaneous generating of an infinite number of instances of old classes in the deep feature space, which poses tighter constraints to maintain the decision boundary of previously learned classes. Without storing any old samples, our method can perform comparably with representative data replay based approaches.

Class-Incremental Learning via Dual Augmentation
Fei Zhu Zhen Cheng! Xu-Yao Zhang 1,2 Cheng-Lin Liu Liu. NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China Universitty of Chinese Academy of Sciences, Beijing, 100049, China
*Center for Excellence of Brain Science and Intelligence Technology, CAS (zhufei2015) chengzhen2019)8ia.. ac. cn, {xyz, liucl)@nlpr. (alpl)).pp........... ac.
Abstract
Deep learning systems typically suffer from catastrophic forgetting of past knowl- edge when acquiring new skills continually. In this paper, we emphasize two dilemmas, representation bias and classifier bias in class mnss-incremmntal learning, and present simple and novel approach that employs explicit class augmentation (classAug) and implicit semantic augmentation (semanAug) to address the two biases, respectively. On the one hand, we propose address the representation bias by learning transferable and diverse representations. Specifically, we investigate the feature representations in incremental learning based spectral analysis and present a simple technique called classAug, to let the model see more classes during training for learning representations transferable across classes. On the other hand, to overcome the classifier bias, semanAug implicitly involves the si- multaneous generating of infinite number of instances of old classes in the deep feature space, which poses tighter constraints to maintain the decision boundary of previously learned classes. Without storing any old samples, our method can perform comparably with representative data replay based approaches.
Introduction
Deep neural networks (DNNs) have enabled great success in many machine learning tasks, based on stationary, large-scale, computationally expensive, and memory-intense training data [1, 2, 3]. Yet the need the ability to acquire sequential experience in dynamic and open environments [4, 5, 6] poses serious challenge to modern deep learning systems, which only perform well on homogenized, balanced, and shuffled data [7]. Typically, DNNs suffer from drastic performance degradation of previously learned tasks after learning new knowledge, which is well--doeumeted phenomenon, known as catastrophic forgetting [8, 9, 10]. Recently, incremental learning (IL.), also referred to as lifelong learning continual learning, has received extensive attention [11, 12, 13, 14] enable DNNs to preserve and extend knowledge continually.
Many earlier studies focus different tasks, and needs the task identity for inference [11, 15, 16]. In this work, we consider more realistic and challenging setting of class- inssemnncrmm learning (Class-IL), where the model only has access to data of new classes at each stage and needs to learn a unified classifier that can classify all seen classes [13, 17, 18). Unfortunately, the learning paradigm of Class-II. will lead to two problems: representation bias and classifier bias, as shown in Figure 1. First, for representation learning, if the feature extractor is fixed after learning old classes, the learned representations could be preserved, but suffer from the lack of transfcrability for new classes; on the contrary, if we update the feature extractor on new classes, the updated representations would be no longer suitable for old classes. Consequently, the old and new classes would be easily overlapped in the deep feature space. We
musk-incrementll learning, which uses separate output layers for
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

old class data: unavailable new class data: sufficient
Note:
Pio
P
new class
nki
mismatch
bias
Basa
new data
funtre
feature extractor
classifier
match
Pan
Figure 1: Two inherent problems in Class-IL: representation bias and classifier bias.
denote this dilemma as the representation bias. Second, to distinguish new classes from old classes, the training loss is typically calculated on all classes. Without old training data, the class weights of old classes would be ill-updated and mismatched with the updated representation space. We denote this dilemma as the classifier bias. In this work, we investigate the learning of representation and classifier in incremental learning and propose simple and effective dual augmentation framework to overcome these two biases in Class-IL IIL without storing and replaying training data of old classes.
Learning Representation for Incremental Learning, Existing works typically regularize network parameters explicitly [11, 15, 16] or implicitly [12] to reduce the representation shift when learning new classes. In this paper, instead of asking how to keep previously learned representations unchanged, we investigate the following question:
Whul properties of learned representations could facilitate incremental learning?
We hypothesize that learning transferable and diverse representations is an important requirement for incremental learning. Intuitively, with such representations, it could be casier to find model to perform well on all tasks and improve both plasticity and stability, since different tasks would be closer in the parameters space. From spectral analysis viewpoint, investigate which components of feature representations are more transferable and less forgettable the incremental learning process. It is found that spectral components with large eigenvalues are less forgettable. Furthermore, we exploit this finding to propose a simple technique named classAug, which can enlarge the spectral components to introduce more diverse and transferable representations for incremental learning.
Learning Classifier for Incremental Learning. Recently, several works were proposed to alleviate the classifier bias in data replay based methods [18, 19, 20J. However, in non-exemplar based (i.e., without storing and replaying old data) Class-IL setting, the classifier bias is more serious and the above methods not be directly used. A straightforward way is storing instances of old classes in the deep feature space. However, this strategy is undesirable due to the limited memory resource and scalability. This work delves into the classifier learning for Class-IL and proposes implicit semantic augmentation (semanAug) approach to generate an infinite number of instances of old classes in the deep feature space by leveraging the distribution information. Seman is inspired by MCF [21] and ISDA [22], which have performed semantic augmentation for linear models and DNNs, respectively. However, both our way to leverage semantic augmentation and the motivation fundamentally differ from them [21, 22].
Contributions. (i) We provide new insights into the representation learning in incremental learning by analyzing the structural characteristics of the learned embedding space via spectral decomposition and find that spectral components with large eigenvalues are less forgettable and carry more transferable features. Based on this observation, we propose a simple and effective method of classAug to learn better embedding space for incremental learning. (fi) For classifier learning in incremental learning, we propose semanAug which implicitly involves simultaneous generating an infinite number of instances of old classes in the deep feature space to maintain the decision boundary of previously learned classes. (iii) Extensive experiments on benchmark datasets demonstrate the superior performance of our dual augmentation framework for the challenging scenario of Class-II..
2 Related Work
Incremental Learning. Diverse approaches have been proposed for incremental learning of DNNs. They can be roughly divided into three categories: regularization based, data replay based, and architecture based approaches. Regularization based methods focus weight regularization by

---

estimating and preventing the important network weights from changing [11, 15, 16). The difference among those methods is the way to compute the importance of the parameters. However, it is hard to design a reasonable metric to measure the importance of parameters, and it is known that regularization strategies show poor performance in Class-II. scenario [23, 24). Data replay based methods address both the representation bias and classifier bias straightforwardly by storing fraction of old data to jointly train the model with current data. With stored real samples, some works [17, 13, 25] use a distillation loss to prevent forgetting, while others [26, 27, 28] develop gradient-based regularization to make more efficient use of the rehearsal data. To avoid storing real data, another line of works generates aseudo-samples of all previous classes for replay using deep generative models [29, 30, 31, 32]. Nevertheless, storing real data is undesirable for resource- limited or privacy and safety concerning scenarios. Moreover, training big generative models for complex datasets is inefficient. Architecture based methods dynamically extend the network structure during the course of incremental learning [33, 34, 35, 36]. However, growing architecture is unfeasible for large numbers of tasks, and those methods are often impractical for Class-II..
Data Augmentation. Literature is rich data augmentation for improving the generalization of DNNs. Classical strategies commonly synthesize "positive" new samples a way that is consistent with the underlying data distribution of the original dataset [3]. Recent works show that label mixing based methods such as Mixup [37] and Cutmix [38] can greatly improve the generalization of DNNs. In complement to the input space augmentations mentioned above, some works have explored feature space augmentations which augment the learned representations in deep embedding space to enhance classifier performance. The intuition behind those works is that certain directions in the deep feature space correspond to meaningful semantic transformations [39, 40]. For instance, deep feature interpolation [40] leverages simple interpolations in the embedding space to achieve semantic augmentation. A recently proposed ISDA [22] performs semantic augmentation by estimating and leveraging the category-wise distribution of deep representations in an online manner. Despite the simplicity, ISDA shows its effectiveness in semi-supervised learning [22], contrastive learning [41], domain adaptation [42] and long-tailed recognition [43].
3 Dual Augmentation Framework for Class-Incremental Learning
We first formalize the problem of Class-IL, and then introduce the proposed classAug for repre- sentation learning and semanAug for classifier learning, respectively. Finally, we present the dual augmentation framework for Class-IL by combing the two augmentations.
Problem Definition. Typically, a Class-IL problem involves the sequential learning of T tasks that consist of disjoint classes sets, and the model has to classify all seen classes at any given point in training. At incremental step & E {1, .... T,, (x, y) E Dr denotes a training sample, where @ is an sample in the input space X' and y E C. is its corresponding label. C, is the class set of task t. To facilitate analysis, we represent the DNN based model with two components: a feature extractor and unified classifier. Specifically, the feature extractor fe X Z, parameterized by 6, maps the input x into a feature vector x = fo(x)) E R in the deep feature space Z; the unified classifier Up 2 CRC,, parameterized by sp. produces probability distribution g (z) as the prediction for x. Denote the overall parameters by e = ())
The general objective is to correctly classify test examples from all seen classes [44]. The key challenge of Class-IL is that data from previous tasks are assumed to be unavailable, which means that the best configuration of the model for all seen tasks must be sought by minimizing the predefined loss function L (e.g., cross-entropy) on current data Dj:
argmin B( (C (Jo(z)). v)). 0.
A widely used strategy to preserve old knowledge is knowledge distillation [45], which typically matches the current model with previous model response to current training data using the teacher- student framework [12, 13, 19].
3.1 Learning Representation with Class Augmentation
As we focus on non-exemplar based Class-II. we intentionally avoid storing training samples of old classes. To maintain the generalizability of the learned representations for old classes, existing
3

---

methods typically restrain the feature extractor from changing [11, 15, 16, 12]. However, this would lead to a trade-off between the plasticity and stability [5], and it would be hard to perform long-step incremental learning. Our high-level idea is to learn transferable and diverse representations to bridge the old and new classes a better feature space. To delve into this problem, we want to answer two questions: (1) Which part of feature representations tends to be forgotten in incremental learning? (2) How to facilitate the representation learning for incremental learning?
3.1.1 Analyzing Forgetting via Spectral Decomposition
In what follows, we explore which part of feature representations tends to be forgotten and may not be transferable across different tasks in incremental learning. To this end, we propose to quantify the sensitivity of the model to different directions in the deep feature space by measuring the similarity of the space before and after learning new tasks.
Formally, given a feature extractor oot trained on dataset Doid = {( yi y:) A new dataset Dnew that contains disjoint classes with Dold is used to update fo.old. and the updated feature extractor is denoted as fe new- For the samples in Dold, we can get two groups of deep features mapped by fod and fe.new. respectively. Using eigenvalue decomposition, we could respectively decompose the features mapped by original feature extractor (i.e., fo. (z;) as well as the features mapped by updated feature extractor (i.e., fe (x:)) to different directions following:
Σ [o(x))o(a)))) = Σ uj U
where Xy represents the eigenvalue with index j and u, is its eigenvector. d is the dimensionality of the feature space. Through spectral factorization in Eq. (2), we can represent the original and new representations with two groups of cigenvectors: (Mold,1, Mold,d and [Unew,1:... ..., Unew..)
Next, we investigate the forgetting or transferability of each direction. Shonkwiler [46] introduced the principal angles [47] to measure the similarity of two subspaces. However, it is unreasonable to treat all eigenvectors equally to calculate the principal angles, regardless their relative eigenvalues. Inspired by [48], we use corresponding angles, denoted by vi, to explore the distance between two subspaces in incremental learning:
Definition 1 (Corresponding Angle) Given two groups of cigenvectors: (uold,1..... Wold.d) J and [unew, 1 ... Une d corresponding angle represents the angle between two eigenvectors correspond- ing to the same eigenvalue value index. The cosine value of the corresponding angle is:
cos( (b)) (Hold.j) Unew,))
(3)
||uold, || [|
where Unld.j is the j-th eigenvectors with the j-th largest eigenvalue the old feature space, and similarly for Unew.j. Note that ||otd = 1 and |e 1. For IL, the meaning of "preserve old knowledge" refers to maintain the previously learned decision boundary among classes. At representation level, for an old class, the shape (i.e., covariance) of the distributions should not be changed too much. If an eigenvector direction only changes slightly after updating the feature extractor, the corresponding angle is small, and vice versa. Intuitively, the corresponding angle could capture the representation shift between the old and updated feature extractor during incremental learning, and reflect the forgetting along certain directions in the deep feature space.
Based on the metric defined above, we explore the forgetting of different directions in Class We use LWF-MC [12, 13] as baseline method and train ResNet-18 [1] on CIFAR- [49] using SGD in 2-step manner. Concretely, the model is first trained on the first 50 classes and then updated on the other 50 classes. Figure 2 (a) shows the absolute cosine values of corresponding angles between the old and new eigenvectors. We can observe that cigenvectors with larger eigenvalues produce larger similarity (small corresponding angles), which indicates those directions are more transferable and less forgettable across different tasks. On the contrary, the eigenvectors with small eigenvalues prefer to move after updating the model on new tasks, and could be regarded as forgettable directions.
Transferable and Diverse Representations. As demonstrated above, the directions with larger eigenvalues transfer better and suffer less forgetting. This thought-provokinn observation indicates that our learned representations should have the following properties: (1) Transferability: the eigenvalues of those several significant directions should be enlarged to transfer across tasks (or

---

yaues comesponding angres
eigenvectors 1S with larger eigenvalues
values comespondigg angles
everage of baass-wsedd results
representations for all classes
representations
class #0
eigannal asce
transfer
03
Baseline Mauu
Baseline
Maup
1.
daassuug
dlassuug
130
as 43
elgenivectors
eigenwectons
indox digenvesters
(8) cosine values of corresponding angles
(b) eigenvalues of representations
Figure 2: (a) Absolute cosine values of corresponding angles. (b) Distribution of eigenvalues for baseline, Mixup [37], LS [50], and our classAug training based models.
classes). (2) Diversity: the number of the directions with significant eigenvalues should be increased. Note that those properties are different from that in the common single-task learning scenario. Actually, reducing the number of directions with significant variance has been seen as form of feature compression [51], which is linked to generalization by information theory [52, 533. However, the usual concepts of generalization may not entirely be appropriate for II., since standard learning only aims to learn compact representations within training classes without considering new class generalizability, In IL, those less discriminative directions for the current task could capture useful representations for future tasks. A recent paper [54] has shown that strong compressed representations can actually hurt the generalization ability in the deep metric learning setting. Therefore, to reduce forgetting and enhance the transferability of the representations, it is important to enlarge the cigenvalues and increase the number of cigenvectors with significant variance.
3.1.2 Learning Representations via Class Augmentation
We now exploit our above analysis to propose a simple method for representation learning in Class- IL. Our key idea is to learn transferable and diverse representations by learning more classes at each incremental stage t. To do so, a direct way to introduce real classes from other datasets as auxiliary. However, it is unrealistic to always have access to other real classes, and which datasets should be used remains unknown. Therefore, we propose class augmentation (classAug) to augment the original classes by synthesizing auxiliary classes based on Dr. Concretely, inspired by Mixup [37], classAug randomly interpo- lates two samples
class a
and 2s from two different classes a and to generate a new sample new representing a new class:
e [0.44.0.6)
new a ah
Xx. + (1 A)zE,
(4)
Dr
1-1
where À is a random number of interpolation coefficient. For a k-claas problem, we can gen- erate k[k 1)/2 new classes using the above method, which can be further merged to m aux- iliary classes. As a result, the original k-class
augmented class
class
Figure 3:
Illustration of classAug.
problem in the current task extended to (k + m))clas problem. Moreover, we restrict the to be sampled from the interval of [0.4, 0.6]. to reduce the overlap between the augmented and original classes. At the end of each IL stage, the augmented class nodes in the classifier would be removed.
The proposed classAug is related to Mixup [37] which applies random interpolation on pair of training samples and the respective one-hot labels. However, the interpolated samples in Mixup are near original data, and the number of classes is not changed, but in our method, it increased. By learning to classify more classes in each stage t, the model could learn more transferable and diverse representations. Figure 2 (b) displays and compares the eigenvalues 2 of representations learned with different methods on the first 50 classes of CIFAR-100 It is obvious that the proposed classAug can enhance the value of eigenvalues significantly. and produce more directions with significant variance compared with other methods. On the contrary, Mixup and I Label- -mmothhnn (LS) [50] lead to significantly smaller eigenvalues for the several top eigenvectors, which represent more compact representations. Indeed, the compression effect of soft-label based methods has also been demonstrated in [51, 50]. As shown in Section 4.3, classAug can improve the performance of Class-IL significantly, while Mixup and LS have negative effect in our experiments.
Discussion.
To visualize the distribution clearly, do not include the largest eigenvalue in the figure.
5

---

3.2 Learning Classifier with Semantic Augmentation
As demonstrated in Section 1, classifier bias is another problem in Class-IL. When learning new classes, the previously learned decision boundary would suffer from catastrophic distortion and thus the test samples from old classes could be easily mapped to wrong classes. To overcome this issue, we propose semantic augmentation (semanAug) which leverages the distribution information (i.e., class mean and covariance) of old classes to regularize the learning of the classifier. Formally, for each old class k E (1.... Cota), we can generate M instances in the deep feature space from its distribution, i.e., Σ5; NN(p "[pu::2)) Ex); which y a non-negative coefficient. Then the generated instances of old classes and real instances of new classes in the deep feature space can be jointly fed to the classifier for minimizing cross-entropy loss:
The log
Cold
M Σ m=21
c Ek. +.b ECel e' Se mb
L
log
ECai efT the
Cold
M
(5)
Le,now: loss real features of new
Ci.uta: loss generated features old
where nt is the number of training samples in current task dataset Dr, Cold is the number of total old classes upon stage f, and Conl Cold + Ct is the number of all seen classes stage +0 [yp1.... Pp...l" E Reau x and b = [b,...., beauel E Reai are the weight matrix and bias vector of the last fully connected layer, respectively.
In Class-IL, the second term in Eq. (5), Lrood,. is computationally inefficient when M and Cold are large. In the following, we present an easy-to-compute way implicitly generate infinite instances in the deep feature space for old classes.
Upper bound of Liotd. Concretely, in the case of M + oc, the second term in Eq. (5):
Cald Bag
e opE#x-be
]
Cold
Call
Li,old
Cold
-log
uf c=1 eif?
the
Cold
1
Ez
log
Σ
(og - + (be-mb))
k=1
km
Cold
Call!
1&? -p? +a +00 -po
log
Big
Cold
kmm
cmm
Cold
Call
ee Me (b. bbo) 20
log
Cold
(6)
In above equation, Uo.k Se Pi:. The inequality is based on Jensen's 's inequality E[log (X)
<
(X), and the last equality is obtained by using the moment-generating function B[exX]
esn 2o' X ("), due to the fact that (4p: - ( ) + (be bb) is Gaussian random variable. As can be seen, Eq. (6) is an upper bound of original Le,atd, which provides an elegant and much efficient way to implicitly generate infinite instances in the deep feature space for old classes.
The old in Eq. (6) can be write in the common cross- cross-entropy loss form:
1 Cold
Lr.eemnnAu A Li,old Cold
log
ECul e ueut I0 Eavor
k=
Intuitively, Li.otd implicitly performs semantic transformations for μa based on Σg. To maintain the decision boundary, 7 should be smaller if the distribution of class near the decision boundary: instead, should be bigger if the distance is relatively far. We set 7 2 in our experiments. In addition, we can observe that when = 0, only the class means are used for knowledge retention.
Discussion. (1) Although the derivation of the upper bound in Eq. (6) is similar with ISDA [22], both our motivation and the way to leverage semanAug are different from ISDA. When learning new classes, we only apply semanAug for the class mean of each old class based on the memorized distribution information. While ISDA applies semanAug on all the training samples to improve generalization in standard supervised learning. In addition, a crucial step in ISDA is to estimate the mean and covariance matrix of each class in an online manner. Differently, seman Aug is naturally suitable for Class-II., since the distribution of old classes can be estimated with all training samples at the end of each learning stage. (2) Using previous class statistics for has also been explored in IL2M [55]. However, our method differs from IL2M in both the statistics information and the way to

---

old feature spece
legend
fixed
Di,ang
extractor
classiffer
distillation
Dt
new class 1
spacs
new class
augmented classes
augmented
cless
Figure 4: Illustration of our dual augmentation framework (II.2A) for Class-II.. On the hand, the training samples of new classes at current task are augmented via the proposed classAug. On the other hand, the distributions of old classes retained by semanAug in the deep feature space.
leverage them. First, The class statistics in 1L2M is the prediction score of the classifier, while ours is the class distribution statisties in the deep feature space. Second, IL2M uses the class statistics to calibrate the prediction of continual learner in a post-processing manner, while our method leverage the statistics automatically learn balanced classifier.
3.3
The Dual Augmentation Learning Framework
With AlassAug for representation bias and semanAug for classifier bias, Figure 4 describes the learning process of the dual augmentation framework (II.2).. We also the well-known knowledge distillation (KD) [19] for two reasons. Firstly, classAug and KD are complementary and focus on different aspect of learning representation. Secondly, KD can reduce the change of feature extractor, which is crucial for semanAug because it implicitly generate instances in the deep feature space from old distribution. The total learning objective each stage / is following:
L. Lt,new semannnn + BLe
(8)
where a and 3 are two hyper-parameterss. Ltnnew and L Stmaanndu are shown in Eq. (5) and Eq. (7), respectively. Le 1 Σ Il (xx) - Se. (ax))ll Note that Lt and eemenmanAug are applied to both the original and synthesized samples. Algorithm presents the pseudo code of 11.A.
Experiments
4.1 Evaluation Protocol
Datasets. We perform our experiments on CIFAR-100 [49] and Tiny-ImageNet [56]. A common setting is to train the model on half of classes for first task, and equal classes in the remaining incremental steps. Based on this, we split the CIFAR-100 dataset in different settings: 50 + x 10, 50+ 5, 40+22 20+ x 3. For instance, 50+ 10 x 5 represents that the first task contains 50 classes and there are 5 classes for the following 10 tasks. Similarly, the settings for Imageee are 100+5 x 20, 100+ 10x 10 and 100 20 x 5. Intuitively, more classes in each tasks requires the model to leam a harder problem for each task, while increasing the length of the task sequence challenges the model's retention.
Algorithm 1: IL2A: Dual augmentation algorithm
Randomly initialize O (0" "0 8 = 0:
foreach incremental stage / (1, ..., T) do
Input: model e data D, { 1
Output: model e
O' --1,
D Dr.ug (a/ v0 via classAug; add class nodes for augmented classes; if then
train Of by minimizing
else
L train O by minimizing Eq. (8); f- compute {1A, Σ} for each class in Dj;
St St-1 remove augmented class nodes in classifier;
Implementation Details. In our experi- ments, we follow [44] to utilize the ResNet-18 [1] as our base architecture, and train from scratch
7

---

CIFAR-103
103 (5 phoses)
CIFAR-100 (10 phoses)
CIFAR-100 (20 phases)
MAS
75
LwF -MC
15
s0aualay (%)
acaurecy (9)
MUG
L2Nours)
-- ---IRRI--MM
-CaRL NME
EEIL
LUCHR
10
123
7 001121344451771111210
number of taska
number lasks
number
Tiny-ImageNet (5 phases)
Tiny ImageNet (10 phases)
60-
Tiny-ImageNet (20 phases)
MAS
LWF-MC
accuracy (%6
-5-
eccuracy (36)
45-
accuracy (%)
MUC
1.2A/ours
25-
10-
-CRLL-ON
12
25-
EEILL
10-
LUDR
0031223381515177888
number
number
number of tasks
Figure 5:
Results of top-1 accuracy on CIFAR-100 and Tiny- ImageNet under different settings. Solid lines present methods that do not store old exemplars, dashed lines present data replay based methods.
in each experiment. All models are trained using Adam [57] optimizer with an initial learning rate of 0.001 for 100 epochs with the mini-batch size of 64. The learning rate is reduced by factor of 10 at 45 and 90 epochs. We use the same hyper-parameter value for all experiments. Specifically, we set a 10 and 3 10 in Eq. (8). The number of augmented classes (i.e. The number of augmented classes (i.e., m) depends on the number of (original) classes at current in- cremental step. Taking CIFAR-100 as an example, the m is 45 for 5 phases setting where each incremental step has 10 classes; and m is 10 for 10 phases setting where each incremental step has 5 classes. At the end of each incremental stage, evaluate the model all seen classes after removing the class nodes of the m augmented classes in the classifier. Our code is available at https://gi thw...tttttuttttg/ege//eee com/ wwaam/mmpressiioc www.siinnn002222222
Comparison Methods. Our method (IL.2A does not store any old samples for replay when learning new classes. Therefore, we first compare IL2A with several non-exemplar based approaches: MAS [16], LwF-MC [13], MUC [58], LwM [59]. In addition, we also compare with several exemplar based methods such as iCaRL [13], EEIL [18] and LUCIR [19]. Specifically, for the data replay based methods, we follow [13, 19] to store 20 samples for each class using 'herd' selection technique [13]. We report the average top-1 accuracy of all previously seen classes up to each incremental step t. For iCaRL, we respectively report its results of CNN predictions and -ote---maao----oeeeeeee classification, denoted as iCaRL-CNN and iCaRL-NME.
4.2 Experimental Results
Comparative results are shown in Figure 5. Firstly, we observe that our method performs much better than non-exemplar based methods such LwF-MC and MUC in the trend of accuracy curve under different settings. Particularly, the gap appears unbridgeable in the long-step Class-IL setting, e.g.. 10 phases and 20 phases. This suggests that only constraining old parameters does not suffice to prevent forgetting. We argue that this is partly due to the unaddressed classifier bias. When compared representative data replay based methods such as iCaRL, EEIL and LUCIR, our method remarkably shows strong performance without storing old samples.
Main Results.
The success of our method can contribute to the proposed classAug and semanAug. Specifically, classAug is applied to new classes of current task, which enables the model to learn more transferable and diverse representations for future classes and in turn, reduces the forgetting of old parameters when learning new classes. While semanAug is applied to old classes of previous tasks, which leverage the valuable distribution information of old classes to learn unified classifier to connect the classes from different tasks to each other.
Ablation Study. To evaluate the effect of each component in IL2A, we perform the ablation study and show the results of 10 phases setting (CIFAR-100) in Table 1. Specifically, the baseline denotes the method that does not generate pscudo-instance using semanAug, but only replays the class-mean
8

---

Table I: The effect of each component in IL2A.
Methocklncremental stage
1
4
8
Final
Knowledge Distillation
78.78
30.18
20.71
14.61
11.87
8.80
7.70
7.23
7.10
6.05
6.04
baseline
78.86
62.85
56.96
54.66
51.72
47.33
43.61
40.12
40.76
36.55
34.71
semanAug
79.16
69.14
60.68
58.18
54.77
50.89
48.45
46.29
46.97
44.38
42.09
+ classAug
79.72
68.30
64.15
60.15
56.21
52.61
51.48
46.48
46,36
43.63
41.56
+ classAug semanAug
81.08
74.54
66.28
63.89
58.80
54.97
51.32
48.64
49.74
47.05
45.07
of each old class when training new classes. By doing so, aim validate the effectiveness of semanAug compared with only replaying class-mean. In summary, we can observe that: (I) (2) SemanAug improves the performance of bascline from 34.71% to 42.09%. Those results indicate the effect of the distribution information for maintaining old knowledge Class-IL. (3) ClassAug also has remarkably effect on baseline, and (4) the performance can be further improved by combing with semanAug, which indicates that those two modules are complementary, Similar results are observed other settings of CIFAR-100 and Onn-IImgeNNet datasets. (5) As for the computational complexity, classAug involves input level sample mixing and the augmented samples are fed feature extractor. Differently, semanAug performs implicit old instance generation in the deep feature space. Therefore, semanAug is cheaper compared with classAug from the computation perspective.
Baseline improves the performance of KD significantly.
4.3 Further Analysis
ClassAug Improves both Plasticity and Stability in Class-IL. To analyze the effectiveness of classAug more concretely, we explore how it affects the new tasks accuracy (?) and average forgetting (L) (CIFAR-100, 100, 10 phases setting). Average forgetting [60] defined estimate the forgetting of previous tasks. The forgetting measure f& of the i-th task after training k-th task is defined as fi max (0e,i in Vi < k, in which am,n is the accuracy of task after training task m. tEI....k-1
task The average forgetting measure Fs is then defined as F5 E fi- Intuitively, accuracy can be viewed as the plasticity of the incremental learner and the average forgetting can be viewed as the stability of the incremental learner. Figure (a) and (b) report the results, from which that classAug simultaneously improves the task accuracy and reduces the average forgetting. Specifically, the significant improvement on new task accuracy implies that the model training with classAug is good initialization for the following tasks. Consequently, classAug is effective to improve the trade-off between plasticity and stability of a continual learner.
100
new lakk accuracy (%)
baselliee wi clessAug
as-
-B-
50-
better stability
finall aocuracy (%)
60-
baseline
classAug
Mixup
LS
forgetling (%)
so-
40
50-
s0
40
20-
baseline w/ classAug
30-
plasticity
20-
70
3 7 number of tasks
4
7
10:11
number of tasks
5 phases
10 phases
20 pheses
(a)
(b)
(c)
Figure 6: (a, b) ClassAug can simultaneously improve the new task accuracy and reduce the average forgetting. (c) Compared with classAug, Mixup and LS have negative effect for Class-IL.
Compare ClassAug with Other Regularizers. We compare the proposed classAug with Mixup and LS in Figure 6 (c), where the baseline (with semanAug) represents our IL2A without using classAug. As can be seen, Mixup and LS have negative effect the final accuracy. This phenomenon could be interpreted based on the analysis in Section 3.1.1 and Figure 2 (b). Specifically, those regularizers result in more compressed representations, damaging the transferability of the representations. Besides, the label smoothing strategy also affects the weights of old classes the classifier, thus increasing the classifier bias. Similar results have also been reported in [61].
Discussion of Covariance Matrix. our main experiments, we use the original covariance matrix for semanAug. However, storing the original covariance matrix might be inefficient when the

---

Table 2: OOD detection results. 1 indicates higher is better.
AUROC f
AUPR-In +
AUPR-Out 1
OOD
baseline
Mixup
classAug
baseline
Mixup
classAug
baseline
Mixup
classAug
MNIST
87.02
92,46
94.99
79.89
89.00
93.05
92.26
95.48
97.20
Fashion-MNNI -
90.28
93.37
94.40
86.18
89.11
92.43
94.26
96.19
96.78
LSUN
88.50
88.80
93,90
83,48
74.71
91.08
92.92
94.09
96.73
Tiny-ImageNet
88.49
84.96
93.92
83.84
64.02
91.77
92.70
92.19
96.55
Mean 88.57
89.90
94.30
83.35
79.21
92.08
93.04
94.49
96.81
matrix dimension is large. An alternative way is to only store the elements on the diagonal, which could greatly reduce the cost of memory. Figure 7 also reports the results of using the diagonal covariance matrix. Under different settings, using the original covariance matrix slightly better than the diagonal form. This is reasonable because the original covariance matrix stores more distribution information of old classes. However, using the diagonal covariance matrix would be more memory -ffinccreefffiintttt in practice.
ClassAug Improves Confidence Reliability, During continuous use of machine learning system in open- world applications, there are mainly three key steps [62]. The first step is dfitt--iiitiibuttin
70
original
diagonal
(OOD) detection [63], which requires the system to detect un- known samples from novel classes. The second step is to label the collected unknown samples by humans or automatic algorithms [64]. Finally, the system must scale and adapt incrementally to learn the novel classes, which is the Class-IL IL problem studied in this paper. Recently studies found that DNNs are overconfident for their predictions [63, 65], lacking the ability to detect samples from unknown classes. In sal-world applications, we expect a continual learner has good OOD detection ability.
accuracy
50-
40-
30
20
5 phases
10 phases
20 phases
Figure 7: Original v.s. diagonal covariance matrix. CIFAR- 100.
explore the OOD detection ability of the proposed classAug. Concretely, we train a ResNet-18 on CIFAR-10. and the test samples from CIFAR-IO are in-distribution. For OOD examples, we test on MNIST [66], Fashion-MNIST [67], LSUN (resized) [68] and Tiny-ImageNet (resized), As shown in Table 2, classAug noticeably improves the OOD detection performance of baseline [63] on commonly used metrics such as AUROC, AUPR- and AUPR-Out [63]. By recogmizing synthetic samples. DNNs could learn more robust and transferable representations which could be generalized to OOD samples. Moreover, as shown in Table 2, Mixup sometimes damages the performance of OOD detection, which further demonstrates the superiority of classAug.
Conclusion
In this paper, we propose a simple and effective dual augmentation framework to address the representation bias and classifier bias in Class-II. We first investigate the transferability (or forgetting) of representations via spectral decomposition, which motivates us to propose classAug that can learn propose
transferable, diverse and less compact representations for II.. Furthermore, semanAug to implicitly generate infinite instances of old classes in the deep feature space during jointly learning of the unified classifier. Experiments show that our method could achieve remarkable performance compared with state-of-the-art Class-IL methods, Future works will consider the dual augmentation framework for more challenging scenarios like Class-IL with distribution shift and OOD data, few-shot Class-IL. IL, and federated incremental learning.
Acknowledgements
This work has been supported by the National Key Research and Development Program under Grant No. 2018AAA0100400,, the National Natural Science Foundation of China (NSFC) grants U20A20223, 61633021, 62076236, 61721004, the Key Research Program of Frontier Sciences of CAS under Grant ZDBS-LY-7004, and the Youth Innovation Promotion Association of CAS under Grant 2019141.
10

---

References
[1] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
4.7
[2] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image datahase. CVPR, pages 248-255, 2009.
[3] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1097-1105, 2012. 1,3
[4] Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar. Learning nonstationary environments:
A survey, IEEE Computational Intelligence Magazine, 10(4):12-25, 2015. 1
[5] German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-77 71, 2019. 1,4
[6] Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Trans. Pattern Anal. Mach. Intell. 2021.
[7] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 2020.
[8] Ian J. Goodfellow, M. Mirza, Xia Da. Aaron C. Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradient-basee based neural networks. CoRR, 2014.
[9] M. McCloskey and N. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, pages 109-165, 1989.
[10] Robert M French. Interactive tandem networks and the scquential learning problem. Citeseer.
[11] J. Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, J. Veness, G. Desjardins, Andrei A. Rusu, K. Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, Demis Hassabis, C. Clopath, D. Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, pages 3521 - 3526, 2017. 1, 2, 3,4
[12] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., pages 2935-2947, 2018. 2, 3,4
[13] Sylvestre-ANve Alvise Rebuffi, A. Kolesnikov. Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 5533 5542,, 2017. 1,
[14] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, pages 524-540, 2020.
[15] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICMI., pages 3987-3995, 2017. 3,
[16] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, pages 139-154, 2018 1, 2, 3, 4,8
[17] Y. Wu, Yan-Jia Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental lcarning. In CVPR, pages 374-382, 2019. 1,
[18] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End End-to-end incremental learning. In ECCV, pages 233-248, 2018. 2,
[19] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and D. Lin. Learning unified classifier incremen- tally via rebalancing. In CVPR, pages 831-839, 2019. 2, 2, 7,8 8
[20] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu- Tao Xia. Maintaining discrimination and faimess in class incremental learning. In CVPR, pages
2020. 2
[21] Laurens Maaten, Minmin Chen, Stephen Tyree, and Kilian Weinberger. Learning with marginalized corrupted features. In ICML, pages 410-418, 2013. 2
[22] Yulin Wang, Gao Huang, Shiji Song, Xuran Pan, Yitong Xia, and Cheng Wu, Regularizing deep networks with semantic data augmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2021. 3,
11

---

[23] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: categorization and case for strong baselines. arXiv preprint
2018.
[24] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint UrXXv:190..07734 2019. 3
[25] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled oulputs distillation for small-tasks incremental learning. In ECCV, pages 86-102, 2020. 3
[26] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2018. 3
[27] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurlPS, 2017. 3
[28] Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhosciny. Efficient lifelong learning with a-gem. In ICLR, 2019. 3
[29] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NewrlPS, pages 2994-3003, 2017. 3
[30] Chenshen Wu, L.. Herranz, X. Liu, Wang, Joost van Weijer, and B. Raducanu. Memory replay gans: Learning to generate new categories without forgetting. In NeurlPS, pages 5962-5972, 2018. 3
[31] Xiang, Ying Fu, Pan and Hua Huang. Incremental learning using conditional adversarial networks. In ICCV, pages 6618 6627, 2019. 3
[32] Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental leamnnng In ICLR, 2018. 3
[33] Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1066.00 24100,0.4661,011666 3
[34] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In CVPR, pages 7765 7765-7773, 2018.
[35] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, pages 44488-455,222 2018. 3
[36] Jachong Yoon, Funho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandahle networks. In ICLR, 2018. 3
[37] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In ICLR, 2018 3,5
[38] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon You. Cutmix: Regularization strategy to train strong classificrs with localizable features. In ICCV, pages 6023-6032, 2019. 3
[39] Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. In ICMI., pages 552-560, 2013.
[40] Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In CVPR, pages 7064-7073, 2017. 3
[41] Qi Cai, Yu Wang, Yingwei Pan, Ting Yao, and Tao Mei. Joint contrastive learning with infinite possibilities. In NeurIPS, 2020. 3
[42] Shuang Li, Mixue Xie, Kaixiong Gong, Chi Harold Liu, Yulin Wang, and Wei Li. Transferable semantic augmentation for domain adaptation. arXiv preprint arXiv:2103. /2562, 2021. 3
[43] Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed visual recognition. arXiv preprint lrXiv:2133.1257) 2021.
[44] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. In NeurIPS, 2020. 3,7
[45] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 503.033 2015. 3
12

---

[46] Clayton Shonkwiler. Poincaré duality angles for riemannian manifolds with boundary. arXiv preprint
arXiv:0909.1963 1967 2009. 4
[47] Jianming Miao and Adi Ben-Isracl. On principal angles between subspaces in m. Linear algebra and its applications, 171:81- 171,8, 1992. 4
[48] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs, discriminability: Batch spectral penalization for adversarial domain adaptation. In ICML, pages 1081-1090, 2019. 4
[49] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009. 7.7
[50] Rafael Müller, Simon Korbllith, and Geoffrey E. Hinton. When does label smoothing help? In NeurlPS, pages 4696-4705, 2019. 5
[51] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, pages 6438-6447, 2019.
[52] Naftali Tishby and Noga 7aslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5,2015 2015.
[53] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 100.7703300001 2017. 5
[54] Karsten Roth, Timo Milbich, Samarth Sinha, Prateck Gupta, Bjöm Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance deep metric learning. In ICML, 2020.
[55] Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In ICCV, pages 583-592, 2019. 6
[56] Lcon Yao and John Miller. Tiny imagenet classification with convolutional neural networks. CS 231N.
[57] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
[58] Yu Liu, Sarah Parisot, Gregory G. Slabaugh, Xu Jia, Ales Leonardis, and Tinne Tuytelaars. More classifiers, less forgetting: A generic multi-classifier paradigm for incremental learning. In ECCV, pages 199-716, 2020.
[59] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, pages 5138-5146, 2019.
[60] Arslan Chaudhry, P. Dokania, Thalaiyasingam Ajanthan, and Tomr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. ECCV, pages 532-547, 2018.
[61] Sudhanshu Mittal, Silvio Galesso, and Thomas Brox. Essentials for class incremental learning. arXiv preprint arXiv:2102.09511 099177 2021.
[62] Yao Zhang, Cheng-Lin Liu, and Ching Y Suen. Towards robust pattern recognition: A review. Proceedings of the IEEE, 108(6):894 922, 2020. 10
[63] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and distri-iiitiitttio examples in neural networks. ICLR, 2017. 10
[64] Kai Han, Sylvestre-Alvise Rebufli, Sébastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Trans. Pattern Anal. Much. Intell..
2021. 10
[65] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modem neural networks. In ICML, pages 1321-1330, 2017. 10
[66] Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005.
[67] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708. 077477 2017. 10
[68] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans the loop. ArXiv, htw/1550603366. 2015. 10
13


Reviews:
[{"review_id": "knAkPtrbC8U", "reviewer": "Reviewer_2LZR", "summary": "The paper addresses two problems which occur in the class-incremental learning: representation bias and classifier bias. To address the representation bias, the authors propose to learn a less compact representation in each task. More concretely, they augment the number of existing classes by introducing random interpolations among real samples. Regarding classifier bias, the authors propose to generate an infinite number of features from of classes to maintain the decision boundary of previously learned classes. The insights into representation learning by analyzing the structural characteristics of the learned embedding space via spectral decomposition is interesting. The paper is relatively well written, although it lacks details in some aspects. The approach is moderately novel, being a combination of existing techniques. The experimental validation does not confirm that the proposed approach improve the current state of the art in class-incremental learning.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are my concerns:\n1. The problems addressed in the current paper, namely representation bias and classifier bias, are not depicted clearly enough in Figure 1. I would recommend to improve the clarity of the figure.\n2. Page 5, 'Feature Compression Perspective': what is the difference between the 'overall features' and 'learned features'? Why there is a difference in dimensionality in both cases?\n3. Figure 4: According to section 3.1.2, the extracted features from f_(t-1) and f_t are further compressed, right? This step is not depicted in the figure.\n4. An Algorithm to summarize all the steps of the approach would increase the clarity of the paper.\n5. There are several typos in the paper, but most of them in page 7.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The current work does not present a negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours reading + 1 hour filling the questionnaire"}, "extra_text": {}}, {"review_id": "bwa0wqkG0vt", "reviewer": "Reviewer_qrF1", "summary": "This paper proposes a class-incremental learning (CIL) method that addresses the representation-bias and classifier-bias observed in CIL. The goal is to learn transferable ans compact representations for incremental learning and leverage the distribution of old classes to avoid forgetting and to jointly learn a unified classifier. This paper also presents an interesting finding that regularization in CIL have a negative effect.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper explores important questions in CIL: i) which parts of the representations are tend to be forgotten, ii) how regularization impacts representation learning and iii) how to facilitate representation learning in CIL. To analyze the forgetting, they use spectral decompositions, obtain the eigenvalues and eigenvectors, and check the cosine similarity between eigenvectors. To learn the transferable representations, they propose to use class augmentations. To learn the classifier, semantic augmentation is used. Class augmentation and semantic augmentation are called as dual augmentation framework and it is the proposal of this paper. To the best of my understanding, this approach is novel and has not been used before. Besides this approach doesn't require examplers and replay.\n\nIt would be useful to clarify a few points. In section 3.1.1 it is mentioned that intuitively the corresponding angle could capture the representation shift between the old and updated feature extractors during incremental learning, and reflect the forgetting along different directions in feature space. It is not clear why intuitively this is correct. The complexity of dual augmentation framework is not analyzed in the paper. The experiments show that the results of no augmentation, only semantic augmentation, only class augmentation and their combination and it seems the combination of both augmentation outperforms the others by margin, but it is not clear how costly to run the combination, which augmentation is cheaper etc.\n \nThe experiments are performed on CIFAR-100 and Tiny-ImageNet. These datasets are called challenging but it is not explained in which sense they are challenging. To the best of my knowledge, they are standard benchmarking datasets for incremental learning and it would be useful to run additional experiments on real challenging datasets where there are data samples from different domains.   \n\nThe structure of the paper is nice, the problem, motivation and the goal is clear. But there are so many typos and grammatical mistakes that makes to follow the paper difficult. Some examples: p5, l161 two groups \"of\" eigenvectors, l170 firstly trained -> first trained, l185 simimlar -> similar, p7, l263 discribes -> describes, l266 foucs -> focus, diffrnt -> different, ... etc. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is not clear how this method is applicable to more challenging CIL problems where there is distribution shift and out-of-distribution data. It would be useful to add this to the discussion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "Fokc61LCd_", "reviewer": "Reviewer_cmzc", "summary": "This work tackles incremental image classification problem, and proposes two techniques, ClassAug and SemanticAug, to improve the overall accuracy over all classes, including old and new classes. Results on two benchmarks, including CIFAR-100 and Tiny-ImageNet, are presented to demonstrate the superior performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n  -\tThe method does not require to store example of old classes, and its performance is better than non-exemplar based methods and close to exemplar-based methods on two benchmarks.\n  -\tClassAug seems to be an original idea with supporting evidence from spectral analysis on primary eigenvector components in the per-class features\n\nWeakness\n  -\tOne issue I find with the paper writing is this paper is not self-contained. In Table 1, a minimal introduction to iCaRL and CCIL is not missing, which makes it difficult to get the idea in section 3.1 before reviewing prior papers. Similarly, L169, minimal introduction to LwF-MC is not included.\n  -\tThe proposed SemanticAug seems to be an incremental contribution as it resembles ISDA [24] a lot. While the discussions at L255 are valid, the adaptation of ISDA idea in incremental classification seems to be straight-forward.\n  -\tSome technical exposition is incomplete.\n    o\tFig 2 (b), when data augmentation is used, such as mixup, cutmix, it often requires more training epochs to learn a better model. Can you clarify whether results with data augmentation in Fig 2 (b) are obtained with more training epochs?\n    o\tL217, how many augmented classes (i.e. m) are added? Any data sampling technique is used to balance original new classes and augmented new classes? how many images are generated for each augmented class?\n    o\tL239, for each old class, do you fix the M deep features once they are generated from a normal distribution with fixed mean/covariance? Since backbone is also updated as more new tasks are added, the mean/covariance will be outdated.\n    o\tL292, what is “herd” selection technique?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The chance of having negative social impact is low. So it is ok authors do not address them.", "review_text": "", "extra_scores": {"ethical_concerns": "I could not find ethical concerns for the proposed method.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "FKK9vGnoleU", "reviewer": "Reviewer_9tuP", "summary": "The authors propose a non-replay class-incremental learning method that works by combining input mixup with pseudo-replay at feature level. The authors motivate this method by the means of a simple study on how feature space eigenvectors change while training. The proposed approach is evaluated against a collection of current continual learning methods and shows competitive performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nIn general, I found the technical core of this work to be very interesting:\n- I found semanAug to be a simple yet very interesting approach. Recent Continual Learning literature often disregards feature pseudo-rehearsal to focus instead on input replay or generation. According to the results shown by the authors, this approach is deserving of further exploration.  \n- The overall baseline proposed by the authors is very simple and yet manages to show remarkable performance, given its lack of a memory buffer. It is very remarkable to see it performing on par with memory-based methods, which up to now are generally regarded as the only viable solution for class-incremental learning.  \n\nHowever, this work is characterized by several weaknesses, that I feel need to be addressed to increase its potential:  \n- In the introduction, the authors make a distinction between classifier and representation bias. While the latter clearly refers to the mismatch between the features learned in past tasks vs the ones needed for new tasks, I cannot clearly understand what the authors mean by classifier bias. The definition provided at lines 32-33 seems to me a general description of the stability-plasticity dilemma, the issue at the core of continual learning and widely debated in literature [11, 17 in the paper]. I do not see this as a bias to be solved, but rather as a generally valid description of what happens when training a neural network on incremental data. Along the same line, the authors claim at line 128 that setting a trade-off between plasticity and stability hinders learning in the long run. Again, I see this as a choice that need to be made and that is not negative per se: whether it hinders training only depends on the produced amount of stability.\n- The authors claim at lines 44-45 that “learning diverse and transferable representation is an important requirement […] which has been ignored by previous works”. I do not find this statement to be accurate: finding suitable and transferable representations has always been one of the cornerstones of continual learning approaches [13, 28 in the paper, Joseph et al. 2020, Chaudhry et al. 2020]  \n- This paper adopts an unclear position towards Replay-based methods: on the one hand, the authors claim that they are deliberately targeting non-exemplar-based continual learning, on the other hand, they only adopt replay-based method for their preliminary experiment in section 3.1 and compare IL2A with several replay methods in section 4, so they are not limiting themselves to the aforementioned setting. Even though the method proposed by the authors requires retaining a previous snapshot of the model (which amounts to a memory footprint that may be comparable with replay methods), their non-exemplar-based proposal performing on par with rehearsal approaches is a significant result. However, I wonder if their proposed approach can be successfully coupled with replay and get to even better performance. Indeed, adding such a simple additional experiment could make the whole work even more relevant.\n- A continual learning work that shares some similarities with the authors’ proposal is IL2M (Belouadah et al., 2019), which also makes use of previous class statistics to rectify the prediction scores and prevent representation bias. I think that adding a direct comparison with how this principle is differently exploited in this work could provide a comprehensive picture for the readers.  \n- I found the experimental sections to be often unclear so I strongly suggest that the authors take further steps to improve it, with particular reference to the following points:  \n    - I could not understand the meaning of the numbers contained in Table 1. Are they average accuracy values or incremental accuracy values? Are these numbers referring to different experiment (50-split dataset, 60-split dataset, etc.) or do they refer to the same experiment? What dataset is being used?  \n    - What does “eigenvalues of covariance” mean at line 182? How exactly is the intrinsic dimension of features defined at line 188? When introducing a specific approach for evaluation, it could be useful to explicitly present it to the reader or at least introduce relevant references.  \n    - Running continual learning experiments is often a complex matter, since there exist different experimental settings which are often incompatible. In order to evaluate whether experiments are fairly conducted, since the authors did not submit the code for review, but state they will publish it upon acceptance, I need more details on the way experiments were conducted. What codebase was used for the experiments, a new one created from scratch or was it based on the code from other works? Are all results computed anew for all methods or taken from other works? How was hyper-parameter selection conducted for the competitors, were they tuned at all or were original hyperparameters used?  \n    - In the ablation study, I could not clearly understand how the baseline is constructed: how is the class-mean of old classes used to regularize the classifier (line 311)? Does this mean that only part of the semanAug loss is used? I could not understand this from the text.  \n    - How is the forgetting measure at line 324 defined? There are several instances of such measure in CL literature, so I would need a citation to understand which one the authors are referring to.  \n\nThe following minor issues did not affect my evaluation directly, but I list them for the authors to consider:  \n- I believe that adding variance values for the presented results (at least in the supplementary material) would be very useful to further simplify the understanding of the presented data.  \n- Since the denominator in equation 3 is 1, the equation could be simplified.  \n- In line 202, the authors define [56] “a very nice recent paper”, I would recommend keeping a neutral tone instead.  \n- The comic sans typeface was used in most presented figures, which is unusual in scientific papers. A common sans-serif font would probably be more appropriate.  \n- There are several typos that need fixing:  \nLine 39: learning, and → learning and  \nLine 49: is benefited from → benefits from  \nLine 62: to storing → to store / storing  \nLine 96: Classical strategies commonly synthetic → verb is missing  \nLine 99: in complementary → in complement  \nLine 129: our high idea → our high-level idea  \nLine 136: naturally rises → naturally raises  \nLine 140: How the regularization affect → How does the regularization affect  \nLine 143: are tend to be forgetting → tend to be forgotten  \nLine 149: is donated as → is denoted as  \nLine 161: two groups eigenvectors → two groups of eigenvectors  \nLine 185: We note that simimlar → We note that a similar  \nLine 236: tast samples → task samples  \nLine 236: wrong class → wrong classes  \nLine 237: high level idea to → high-level idea is to  \nCaption of figure 4: hans → hand  \nLine 266: differnt → different  \nLine 274: rest incremental step → remaining incremental steps  \nLine 300: bais → bias  \nLine 306: task → tasks  \nLine 310: method that using → method using  \nLine 311: and use the class → and class\n\n(Joseph et al., 2020) Meta-consolidation for continual learning, NeurIPS 2020  \n(Chaudhry et al., 2020) Continual Learning in Low-rank Orthogonal Subspaces, NeurIPS2020  \n(Belouadah et al., 2019) Il2m: Class incremental learning with dual memory, ICCV 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a theoretical work in continual learning classification. This is a very abstract field that does not elicit any immediate application. As such, I feel that no discussion on potential societal impacts is necessary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors for the additional clarifications provided in their rebuttal, which resolved most of the concerns raised by the reviewers. The contributions are relatively simple, but convincingly shown to be very effective compared to more sophisticated approaches recently proposed in continual learning. These results provide new insights and will benefit the NeurIPS community. However, the reviewers also provided a list of improvements that should be incorporated in the revised version. The paper was not self-contained at times and needs to be polished in order not to hamper the understanding. Exposition of the experiments would also benefit of a revision.
Decision: Accept (Poster)