Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima

1 Introduction . Why study incremental few-shot learning ? Incremental learning enables a model to continually learn new concepts from new data without forgetting previously learned knowledge . Rooted from real-world applications , this topic has attracted a significant amount of interest in recent years [ 5 , 30 , 31 , 40 , 26 ] . Incremental learning assumes sufficient training data is provided for new classes , which is impractical in many application scenarios , especially when the new classes are rare categories which are costly or difficult to collect . This motivates the study of incremental few-shot learning , a more difficult paradigm that aims to continually learn new tasks with only a few examples . Challenges . The major challenge for incremental learning is catastrophic forgetting [ 14 , 28 , 35 ] , which refers to the drastic performance drop on previous tasks after learning new tasks . This phenomenon is caused by the inaccessibility to previous data while learning on new data . Catastrophic forgetting presents a bigger challenge for incremental few-shot learning . Due to the small amount of training data in new tasks , the model tends to severely overfit on new classes while quickly forgetting old classes , resulting in catastrophic performance . Current research . The study of incremental few-shot learning has just started [ 47 , 41 , 60 , 9 , 8 , 34 , 59 ] . Current works mainly borrow ideas from research in incremental learning to overcome the forgetting problem , by enforcing strong constraints on model parameters to penalize the changes of parameters [ 34 , 28 , 56 ] , or by saving a small amount of exemplars from old classes and adding constraints on the exemplars to avoid forgetting [ 40 , 20 , 4 ] . However , in our empirical study , we find ∗Equal contribution †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . that an intransigent model that only trains on base classes and does not tune on new tasks consistently outperforms state-of-the-art methods , including a joint-training method [ 47 ] that uses all encountered data for training and hence suffers from severe data imbalance . This observation motivates us to address this harsh problem from a different angle . Our solution . Unlike existing solutions that try to overcome the catastrophic forgetting problem during the process of learning new tasks , we adopt a different approach by considering this issue during the training of base classes . Specifically , we propose to search for flat local minima of the base training objective function . For any parameter vector in the flat region around the minima , the loss is small , and the base classes are supposed to be well separated . The flat local minima can be found by adding random noise to the model parameters for multiple times and jointly optimizing multiple loss functions . During the following incremental few-shot learning stage , we fine-tune the model parameters within the flat region , which can be achieved by clamping the parameters after updating them on few-shot tasks . In this way , the model can efficiently learn new classes while preserving the old ones . Our key contributions are summarized as follows : • We conduct a comprehensive empirical study on existing incremental few-shot learning methods and discover that a simple baseline model that only trains on base classes outperforms state-of-the-art methods , which demonstrates the severity of catastrophic forgetting . • We propose a novel approach for incremental few-shot learning by addressing the catastrophic forgetting problem in the primitive stage . Through finding the flat minima region during training on base classes and fine-tuning within the region while learning on new tasks , our model can overcome catastrophic forgetting and avoid overfitting . • Comprehensive experimental results on CIFAR-100 , miniImageNet , and CUB-200-2011 show that our approach outperforms all state-of-the-art methods and achieves performance that is very close to the approximate upper bound . 2 Related Work . Few-shot learning aims to learn to generalize to new categories with a few labeled samples in each class . Current few-shot methods mainly include optimization-based methods [ 12 , 23 , 32 , 39 , 45 , 46 , 55 ] and metric-based methods [ 13 , 19 , 44 , 49 , 52 , 58 , 57 , 53 ] . Optimization-based methods can achieve fast adaptation to new tasks with limited samples by learning a specific optimization algorithm . Metric-based approaches exploit different distance metrics such as L2 distance [ 44 ] , cosine similarity [ 49 ] , and DeepEMD [ 58 ] in the learned metric/embedding space to measure the similarity between samples . Recently , Tian et al . [ 48 ] find that standard supervised training can learn a good metric space for unseen classes , which echoes with our observation on the proposed baseline model in Sec . 3 . Incremental learning focuses on the challenging problem of continually learning to recognize new classes in new coming data without forgetting old classes [ 6 , 7 , 10 , 51 ] . Previous research mainly includes multi-class incremental learning [ 4 , 38 , 22 , 33 , 54 , 51 ] and multi-task incremental learning [ 21 , 31 , 42 ] . To overcome the catastrophic forgetting problem , some attempts propose to impose strong constraints on model parameters by penalizing the changes of parameters [ 28 , 1 ] . Other attempts try to enforce constraints on the exemplars of old classes by restricting the output logits [ 40 ] or penalizing the changes of embedding angles [ 20 ] . In this work , our empirical study shows that imposing strong constraints on the arriving new classes may not be a promising way to tackle incremental few-shot learning , due to the scarcity of training data for new classes . Incremental few-shot learning [ 47 , 41 , 60 , 9 , 8 ] aims to incrementally learn from very few samples . TOPCI [ 47 ] proposes a neural gas network to learn and preserve the topology of the feature manifold formed by different classes . FSLL [ 34 ] only selects few model parameters for incremental learning and ensures the parameters are close to the optimal ones . To overcome catastrophic forgetting , IDLVQC [ 8 ] imposes constraints on the saved exemplars of each class by restricting the embedding drift , and Zhang et al . [ 59 ] propose to fix the embedding network for incremental learning . Similar to the finding of Zhang et al. , we also discover that an intransigent model that simply does not adapt to new tasks can outperform prior state-of-the-art methods . Robust optimization . It has been found that flat local minima leads to better generalization capabilities than sharp minima in the sense that a flat minimizer is more robust when the test loss is shifted due to random perturbations [ 18 , 17 , 24 ] . A substantial body of methods [ 2 , 37 , 11 , 15 ] have been proposed to optimize neural networks towards flat local minima . In this paper , we show that for incremental few-shot learning , finding flat minima in the base session and tuning the model within the flat region on new tasks can significantly mitigate catastrophic forgetting . 3 Severity of Catastrophic Forgetting in Incremental Few-Shot Learning . 3.1 Problem Statement . Incremental few-shot learning ( IFL ) aims to continually learn to recognize new classes with only few examples . Similar to incremental learning ( IL ) , an IFL model is trained by a sequence of training sessions { D1 , · · · , Dt } , where Dt = { zi = ( xti , yti ) } i is the training data of session t and xti is an example of class yti ∈ Ct ( the class set of session t ) . In IFL , the base session D1 usually contains a large number of classes with sufficient training data for each class , while the following sessions ( t ≥ 2 ) only have a small number of classes with few training samples per class , e.g. , Dt is often presented as an N -way K-shot task with small N and K. The key difference between IL and IFL is , for IL , sufficient training data is provided in each session . Similar to IL , in each training session t of IFL , the model has only access to the training data Dt and possibly a small amount of saved exemplars from previous sessions . When the training of session t is completed , the model is evaluated on test samples from all encountered classes C = ⋃t i=1 Ci , where it is assumed that there is no overlap between the classes of different sessions , i.e. , ∀i , j and i 6= j , Ci ⋂ Cj = ∅ . Catastrophic forgetting . IFL is undoubtedly a more challenging problem than IL due to the data scarcity setting . IL suffers from catastrophic forgetting , a well-known phenomenon and long-standing issue , which refers to the drastic drop in test performance on previous ( old ) classes , caused by the inaccessibility of old data in the current training session . Unfortunately , catastrophic forgetting is an even bigger issue for IFL , because data scarcity makes it difficult to adapt well to new tasks and learn new concepts , while the adaptation process could easily lead to the forgetting of base classes . In the following , we illustrate this point by evaluating a simple baseline model for IFL . 3.2 A Simple Baseline Model for IFL . We consider an intransigent model that simply does not adapt to new tasks . Particularly , the model only needs to be trained in the base session D1 and is directly used for inference in all sessions . Training ( t = 1 ) . We train a feature extractor f parameterized by φ with a fully-connected layer as classifier by minimizing the standard cross-entropy loss using the training examples of D1 . The feature extractor f is fixed for the following sessions ( t ≥ 2 ) without any fine-tuning on new classes . Inference ( test ) . In each session , the inference is conducted by a simple nearest class mean ( NCM ) classification algorithm [ 36 ] . Specifically , all the training and test samples are mapped to the embedding space of the feature extractor f , and Euclidean distance d ( · , · ) is used to measure the similarity between them . The classifier is given by c ? k = argmin c∈C d ( f ( x ; φ ) , pc ) , where pc = 1 Nc ∑ i 1 ( yi = c ) f ( xi ; φ ) , ( 1 ) where C denotes all the encountered classes , pc refers to the prototype of class c ( the mean vector of all the training samples of class c in the embedding space ) , and Nc denotes the number of the training images of class c. Note that we save the prototypes of all classes in Ct for later evaluation . The baseline model outperforms state-of-the-art IFL and IL methods . We compare the above baseline model against state-of-the-art IFL methods including FSLL [ 34 ] , IDLVQC [ 8 ] and TOPIC [ 47 ] , IL methods including Rebalance [ 20 ] and iCarl [ 40 ] , and a joint-training method that uses all previously seen data including the base and the following few-shot tasks for training , for IFL . The performance is evaluated on miniImageNet , CIFAR-100 , and CUB-200 . We tune the methods re-implemented by us to the best performance . For the other methods , we use the results reported in the original papers . The experimental details are provided in Sec . 5 . As shown in Fig . 1 , the baseline model consistently outperforms all the compared methods including the joint-training method ( which suffers from severe data imbalance ) on every dataset3 . The fact that an intransigent model performs best suggests that • For IFL , preserving the old ( base classes ) may be more critical than adapting to the new . Due to data scarcity , the performance gain on new classes is limited and can not make up for the significant performance drop on base classes . • Prior works [ 47 , 8 , 34 , 20 , 40 ] that enforce strong constraints on model parameters or exemplars during fine-tuning on new classes can not effectively prevent catastrophic forgetting in IFL , indicating that actions may need to be taken in the base training stage .

Summary: This paper proposes an incremental few-shot learning method focusing on training the base model that can easily learn new classes, preserving the classification capability for the old classes. When training the base model, the proposed method, Finding Flat Minima (F2M) finds the flat minima where the loss function does not change a lot even if a random noise is injected. When the novel classes are given, F2M updates the feature extractor and clamp the parameters to make the feature extractor resides in the flat minima region.