Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima

1 Introduction . Why study incremental few-shot learning ? Incremental learning enables a model to continually learn new concepts from new data without forgetting previously learned knowledge . Rooted from real-world applications , this topic has attracted a significant amount of interest in recent years [ 5 , 30 , 31 , 40 , 26 ] . Incremental learning assumes sufficient training data is provided for new classes , which is impractical in many application scenarios , especially when the new classes are rare categories which are costly or difficult to collect . This motivates the study of incremental few-shot learning , a more difficult paradigm that aims to continually learn new tasks with only a few examples . Challenges . The major challenge for incremental learning is catastrophic forgetting [ 14 , 28 , 35 ] , which refers to the drastic performance drop on previous tasks after learning new tasks . This phenomenon is caused by the inaccessibility to previous data while learning on new data . Catastrophic forgetting presents a bigger challenge for incremental few-shot learning . Due to the small amount of training data in new tasks , the model tends to severely overfit on new classes while quickly forgetting old classes , resulting in catastrophic performance . Current research . The study of incremental few-shot learning has just started [ 47 , 41 , 60 , 9 , 8 , 34 , 59 ] . Current works mainly borrow ideas from research in incremental learning to overcome the forgetting problem , by enforcing strong constraints on model parameters to penalize the changes of parameters [ 34 , 28 , 56 ] , or by saving a small amount of exemplars from old classes and adding constraints on the exemplars to avoid forgetting [ 40 , 20 , 4 ] . However , in our empirical study , we find ∗Equal contribution †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . that an intransigent model that only trains on base classes and does not tune on new tasks consistently outperforms state-of-the-art methods , including a joint-training method [ 47 ] that uses all encountered data for training and hence suffers from severe data imbalance . This observation motivates us to address this harsh problem from a different angle . Our solution . Unlike existing solutions that try to overcome the catastrophic forgetting problem during the process of learning new tasks , we adopt a different approach by considering this issue during the training of base classes . Specifically , we propose to search for flat local minima of the base training objective function . For any parameter vector in the flat region around the minima , the loss is small , and the base classes are supposed to be well separated . The flat local minima can be found by adding random noise to the model parameters for multiple times and jointly optimizing multiple loss functions . During the following incremental few-shot learning stage , we fine-tune the model parameters within the flat region , which can be achieved by clamping the parameters after updating them on few-shot tasks . In this way , the model can efficiently learn new classes while preserving the old ones . Our key contributions are summarized as follows : • We conduct a comprehensive empirical study on existing incremental few-shot learning methods and discover that a simple baseline model that only trains on base classes outperforms state-of-the-art methods , which demonstrates the severity of catastrophic forgetting . • We propose a novel approach for incremental few-shot learning by addressing the catastrophic forgetting problem in the primitive stage . Through finding the flat minima region during training on base classes and fine-tuning within the region while learning on new tasks , our model can overcome catastrophic forgetting and avoid overfitting . • Comprehensive experimental results on CIFAR-100 , miniImageNet , and CUB-200-2011 show that our approach outperforms all state-of-the-art methods and achieves performance that is very close to the approximate upper bound . 2 Related Work . Few-shot learning aims to learn to generalize to new categories with a few labeled samples in each class . Current few-shot methods mainly include optimization-based methods [ 12 , 23 , 32 , 39 , 45 , 46 , 55 ] and metric-based methods [ 13 , 19 , 44 , 49 , 52 , 58 , 57 , 53 ] . Optimization-based methods can achieve fast adaptation to new tasks with limited samples by learning a specific optimization algorithm . Metric-based approaches exploit different distance metrics such as L2 distance [ 44 ] , cosine similarity [ 49 ] , and DeepEMD [ 58 ] in the learned metric/embedding space to measure the similarity between samples . Recently , Tian et al . [ 48 ] find that standard supervised training can learn a good metric space for unseen classes , which echoes with our observation on the proposed baseline model in Sec . 3 . Incremental learning focuses on the challenging problem of continually learning to recognize new classes in new coming data without forgetting old classes [ 6 , 7 , 10 , 51 ] . Previous research mainly includes multi-class incremental learning [ 4 , 38 , 22 , 33 , 54 , 51 ] and multi-task incremental learning [ 21 , 31 , 42 ] . To overcome the catastrophic forgetting problem , some attempts propose to impose strong constraints on model parameters by penalizing the changes of parameters [ 28 , 1 ] . Other attempts try to enforce constraints on the exemplars of old classes by restricting the output logits [ 40 ] or penalizing the changes of embedding angles [ 20 ] . In this work , our empirical study shows that imposing strong constraints on the arriving new classes may not be a promising way to tackle incremental few-shot learning , due to the scarcity of training data for new classes . Incremental few-shot learning [ 47 , 41 , 60 , 9 , 8 ] aims to incrementally learn from very few samples . TOPCI [ 47 ] proposes a neural gas network to learn and preserve the topology of the feature manifold formed by different classes . FSLL [ 34 ] only selects few model parameters for incremental learning and ensures the parameters are close to the optimal ones . To overcome catastrophic forgetting , IDLVQC [ 8 ] imposes constraints on the saved exemplars of each class by restricting the embedding drift , and Zhang et al . [ 59 ] propose to fix the embedding network for incremental learning . Similar to the finding of Zhang et al. , we also discover that an intransigent model that simply does not adapt to new tasks can outperform prior state-of-the-art methods . Robust optimization . It has been found that flat local minima leads to better generalization capabilities than sharp minima in the sense that a flat minimizer is more robust when the test loss is shifted due to random perturbations [ 18 , 17 , 24 ] . A substantial body of methods [ 2 , 37 , 11 , 15 ] have been proposed to optimize neural networks towards flat local minima . In this paper , we show that for incremental few-shot learning , finding flat minima in the base session and tuning the model within the flat region on new tasks can significantly mitigate catastrophic forgetting . 3 Severity of Catastrophic Forgetting in Incremental Few-Shot Learning . 3.1 Problem Statement . Incremental few-shot learning ( IFL ) aims to continually learn to recognize new classes with only few examples . Similar to incremental learning ( IL ) , an IFL model is trained by a sequence of training sessions { D1 , · · · , Dt } , where Dt = { zi = ( xti , yti ) } i is the training data of session t and xti is an example of class yti ∈ Ct ( the class set of session t ) . In IFL , the base session D1 usually contains a large number of classes with sufficient training data for each class , while the following sessions ( t ≥ 2 ) only have a small number of classes with few training samples per class , e.g. , Dt is often presented as an N -way K-shot task with small N and K. The key difference between IL and IFL is , for IL , sufficient training data is provided in each session . Similar to IL , in each training session t of IFL , the model has only access to the training data Dt and possibly a small amount of saved exemplars from previous sessions . When the training of session t is completed , the model is evaluated on test samples from all encountered classes C = ⋃t i=1 Ci , where it is assumed that there is no overlap between the classes of different sessions , i.e. , ∀i , j and i 6= j , Ci ⋂ Cj = ∅ . Catastrophic forgetting . IFL is undoubtedly a more challenging problem than IL due to the data scarcity setting . IL suffers from catastrophic forgetting , a well-known phenomenon and long-standing issue , which refers to the drastic drop in test performance on previous ( old ) classes , caused by the inaccessibility of old data in the current training session . Unfortunately , catastrophic forgetting is an even bigger issue for IFL , because data scarcity makes it difficult to adapt well to new tasks and learn new concepts , while the adaptation process could easily lead to the forgetting of base classes . In the following , we illustrate this point by evaluating a simple baseline model for IFL . 3.2 A Simple Baseline Model for IFL . We consider an intransigent model that simply does not adapt to new tasks . Particularly , the model only needs to be trained in the base session D1 and is directly used for inference in all sessions . Training ( t = 1 ) . We train a feature extractor f parameterized by φ with a fully-connected layer as classifier by minimizing the standard cross-entropy loss using the training examples of D1 . The feature extractor f is fixed for the following sessions ( t ≥ 2 ) without any fine-tuning on new classes . Inference ( test ) . In each session , the inference is conducted by a simple nearest class mean ( NCM ) classification algorithm [ 36 ] . Specifically , all the training and test samples are mapped to the embedding space of the feature extractor f , and Euclidean distance d ( · , · ) is used to measure the similarity between them . The classifier is given by c ? k = argmin c∈C d ( f ( x ; φ ) , pc ) , where pc = 1 Nc ∑ i 1 ( yi = c ) f ( xi ; φ ) , ( 1 ) where C denotes all the encountered classes , pc refers to the prototype of class c ( the mean vector of all the training samples of class c in the embedding space ) , and Nc denotes the number of the training images of class c. Note that we save the prototypes of all classes in Ct for later evaluation . The baseline model outperforms state-of-the-art IFL and IL methods . We compare the above baseline model against state-of-the-art IFL methods including FSLL [ 34 ] , IDLVQC [ 8 ] and TOPIC [ 47 ] , IL methods including Rebalance [ 20 ] and iCarl [ 40 ] , and a joint-training method that uses all previously seen data including the base and the following few-shot tasks for training , for IFL . The performance is evaluated on miniImageNet , CIFAR-100 , and CUB-200 . We tune the methods re-implemented by us to the best performance . For the other methods , we use the results reported in the original papers . The experimental details are provided in Sec . 5 . As shown in Fig . 1 , the baseline model consistently outperforms all the compared methods including the joint-training method ( which suffers from severe data imbalance ) on every dataset3 . The fact that an intransigent model performs best suggests that • For IFL , preserving the old ( base classes ) may be more critical than adapting to the new . Due to data scarcity , the performance gain on new classes is limited and can not make up for the significant performance drop on base classes . • Prior works [ 47 , 8 , 34 , 20 , 40 ] that enforce strong constraints on model parameters or exemplars during fine-tuning on new classes can not effectively prevent catastrophic forgetting in IFL , indicating that actions may need to be taken in the base training stage .

Summary: This paper describes an approach to incremental few-shot learning based on finding a good starting configuration in parameter space that leads to more stable incremental few-shot learning with less forgetting. The authors propose to learn an approximately flat region on the initial training set and to then constrain few-shot incremental learning to remain within this flat region and thus minimize forgetting. Experimental results are given on a wide range of few-shot incremental benchmarks, including CIFAR-100, miniImageNet, and CUB-200.