Class-Incremental Learning via Dual Augmentation

1 Introduction . Deep neural networks ( DNNs ) have enabled great success in many machine learning tasks , based on stationary , large-scale , computationally expensive , and memory-intensive training data [ 1 , 2 , 3 ] . Yet the need of the ability to acquire sequential experience in dynamic and open environments [ 4 , 5 , 6 ] poses a serious challenge to modern deep learning systems , which only perform well on homogenized , balanced , and shuffled data [ 7 ] . Typically , DNNs suffer from drastic performance degradation of previously learned tasks after learning new knowledge , which is a well-documented phenomenon , known as catastrophic forgetting [ 8 , 9 , 10 ] . Recently , incremental learning ( IL ) , also referred to as lifelong learning or continual learning , has received extensive attention [ 11 , 12 , 13 , 14 ] to enable DNNs to preserve and extend knowledge continually . Many earlier studies focus on task-incremental learning , which uses separate output layers for different tasks , and needs the task identity for inference [ 11 , 15 , 16 ] . In this work , we consider a more realistic and challenging setting of class-incremental learning ( Class-IL ) , where the model only has access to data of new classes at each stage and needs to learn a unified classifier that can classify all seen classes [ 13 , 17 , 18 ] . Unfortunately , the learning paradigm of Class-IL will lead to two problems : representation bias and classifier bias , as shown in Figure 1 . First , for representation learning , if the feature extractor is fixed after learning old classes , the learned representations could be preserved , but suffer from the lack of transferability for new classes ; on the contrary , if we update the feature extractor on new classes , the updated representations would be no longer suitable for old classes . Consequently , the old and new classes would be easily overlapped in the deep feature space . We ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . denote this dilemma as the representation bias . Second , to distinguish new classes from old classes , the training loss is typically calculated on all classes . Without old training data , the class weights of old classes would be ill-updated and mismatched with the updated representation space . We denote this dilemma as the classifier bias . In this work , we investigate the learning of representation and classifier in incremental learning and propose a simple and effective dual augmentation framework to overcome these two biases in Class-IL without storing and replaying training data of old classes . Learning Representation for Incremental Learning . Existing works typically regularize network parameters explicitly [ 11 , 15 , 16 ] or implicitly [ 12 ] to reduce the representation shift when learning new classes . In this paper , instead of asking how to keep previously learned representations unchanged , we investigate the following question : What properties of learned representations could facilitate incremental learning ? We hypothesize that learning transferable and diverse representations is an important requirement for incremental learning . Intuitively , with such representations , it could be easier to find a model to perform well on all tasks and improve both plasticity and stability , since different tasks would be closer in the parameters space . From a spectral analysis viewpoint , we investigate which components of feature representations are more transferable and less forgettable in the incremental learning process . It is found that spectral components with large eigenvalues are less forgettable . Furthermore , we exploit this finding to propose a simple technique named classAug , which can enlarge the spectral components to introduce more diverse and transferable representations for incremental learning . Learning Classifier for Incremental Learning . Recently , several works were proposed to alleviate the classifier bias in data replay based methods [ 18 , 19 , 20 ] . However , in non-exemplar based ( i.e. , without storing and replaying old data ) Class-IL setting , the classifier bias is more serious and the above methods can not be directly used . A straightforward way is storing instances of old classes in the deep feature space . However , this strategy is undesirable due to the limited memory resource and scalability . This work delves into the classifier learning for Class-IL and proposes an implicit semantic augmentation ( semanAug ) approach to generate an infinite number of instances of old classes in the deep feature space by leveraging the distribution information . SemanAug is inspired by MCF [ 21 ] and ISDA [ 22 ] , which have performed semantic augmentation for linear models and DNNs , respectively . However , both our way to leverage semantic augmentation and the motivation fundamentally differ from them [ 21 , 22 ] . Contributions . ( i ) We provide new insights into the representation learning in incremental learning by analyzing the structural characteristics of the learned embedding space via spectral decomposition and find that spectral components with large eigenvalues are less forgettable and carry more transferable features . Based on this observation , we propose a simple and effective method of classAug to learn better embedding space for incremental learning . ( ii ) For classifier learning in incremental learning , we propose semanAug which implicitly involves simultaneous generating an infinite number of instances of old classes in the deep feature space to maintain the decision boundary of previously learned classes . ( iii ) Extensive experiments on benchmark datasets demonstrate the superior performance of our dual augmentation framework for the challenging scenario of Class-IL . 2 Related Work . Incremental Learning . Diverse approaches have been proposed for incremental learning of DNNs . They can be roughly divided into three categories : regularization based , data replay based , and architecture based approaches . Regularization based methods focus on weight regularization by estimating and preventing the important network weights from changing [ 11 , 15 , 16 ] . The difference among those methods is the way to compute the importance of the parameters . However , it is hard to design a reasonable metric to measure the importance of parameters , and it is known that regularization strategies show poor performance in Class-IL scenario [ 23 , 24 ] . Data replay based methods address both the representation bias and classifier bias straightforwardly by storing a fraction of old data to jointly train the model with current data . With stored real samples , some works [ 17 , 13 , 25 ] use a distillation loss to prevent forgetting , while others [ 26 , 27 , 28 ] develop gradient-based regularization to make more efficient use of the rehearsal data . To avoid storing real data , another line of works generates pseudo-samples of all previous classes for replay using deep generative models [ 29 , 30 , 31 , 32 ] . Nevertheless , storing real data is undesirable for resource-limited or privacy and safety concerning scenarios . Moreover , training big generative models for complex datasets is inefficient . Architecture based methods dynamically extend the network structure during the course of incremental learning [ 33 , 34 , 35 , 36 ] . However , growing architecture is unfeasible for large numbers of tasks , and those methods are often impractical for Class-IL . Data Augmentation . Literature is rich on data augmentation for improving the generalization of DNNs . Classical strategies commonly synthesize “ positive ” new samples in a way that is consistent with the underlying data distribution of the original dataset [ 3 ] . Recent works show that label mixing based methods such as Mixup [ 37 ] and Cutmix [ 38 ] can greatly improve the generalization of DNNs . In complement to the input space augmentations mentioned above , some works have explored feature space augmentations which augment the learned representations in deep embedding space to enhance classifier performance . The intuition behind those works is that certain directions in the deep feature space correspond to meaningful semantic transformations [ 39 , 40 ] . For instance , deep feature interpolation [ 40 ] leverages simple interpolations in the embedding space to achieve semantic augmentation . A recently proposed ISDA [ 22 ] performs semantic augmentation by estimating and leveraging the category-wise distribution of deep representations in an online manner . Despite the simplicity , ISDA shows its effectiveness in semi-supervised learning [ 22 ] , contrastive learning [ 41 ] , domain adaptation [ 42 ] and long-tailed recognition [ 43 ] . 3 Dual Augmentation Framework for Class-Incremental Learning . We first formalize the problem of Class-IL , and then introduce the proposed classAug for representation learning and semanAug for classifier learning , respectively . Finally , we present the dual augmentation framework for Class-IL by combing the two augmentations . Problem Definition . Typically , a Class-IL problem involves the sequential learning of T tasks that consist of disjoint classes sets , and the model has to classify all seen classes at any given point in training . At incremental step t ∈ { 1 , ... , T } , ( x , y ) ∈ Dt denotes a training sample , where x is an sample in the input space X and y ∈ Ct is its corresponding label . Ct is the class set of task t. To facilitate analysis , we represent the DNN based model with two components : a feature extractor and a unified classifier . Specifically , the feature extractor fθ : X → Z , parameterized by θ , maps the input x into a feature vector z = fθ ( x ) ∈ Rd in the deep feature space Z ; the unified classifier gϕ : Z → RC1 : t , parameterized by ϕ , produces a probability distribution gϕ ( z ) as the prediction for x. Denote the overall parameters by Θ = ( θ , ϕ ) . The general objective is to correctly classify test examples from all seen classes [ 44 ] . The key challenge of Class-IL is that data from previous tasks are assumed to be unavailable , which means that the best configuration of the model for all seen tasks must be sought by minimizing the predefined loss function L ( e.g. , cross-entropy ) on current data Dt : argmin θ , ϕ E ( x , y ) vDt [ L ( gϕ ( fθ ( x ) ) , y ) ] . ( 1 ) A widely used strategy to preserve old knowledge is knowledge distillation [ 45 ] , which typically matches the current model with previous model response to current training data using the teacherstudent framework [ 12 , 13 , 19 ] . 3.1 Learning Representation with Class Augmentation . As we focus on non-exemplar based Class-IL , we intentionally avoid storing training samples of old classes . To maintain the generalizability of the learned representations for old classes , existing methods typically restrain the feature extractor from changing [ 11 , 15 , 16 , 12 ] . However , this would lead to a trade-off between the plasticity and stability [ 5 ] , and it would be hard to perform long-step incremental learning . Our high-level idea is to learn transferable and diverse representations to bridge the old and new classes in a better feature space . To delve into this problem , we want to answer two questions : ( 1 ) Which part of feature representations tends to be forgotten in incremental learning ? ( 2 ) How to facilitate the representation learning for incremental learning ?

Summary: This paper proposes a class-incremental learning (CIL) method that addresses the representation-bias and classifier-bias observed in CIL. The goal is to learn transferable ans compact representations for incremental learning and leverage the distribution of old classes to avoid forgetting and to jointly learn a unified classifier. This paper also presents an interesting finding that regularization in CIL have a negative effect.