Subspace Regularizers for Few-Shot Class Incremental Learning

1 INTRODUCTION . Standard approaches to classification in machine learning assume a fixed training dataset and a fixed set of class labels . But for many real-world classification problems , these assumptions are unrealistic . Classifiers must sometimes be updated on-the-fly to recognize new concepts ( e.g . new skills in personal assistants or new road signs in self-driving vehicles ) , while training data is sometimes unavailable for reuse ( e.g . due to privacy regulations , Lesort et al . 2019 ; McClure et al . 2018 ; or storage and retraining costs , Bender et al . 2021 ) . Development of models that support few-shot class-incremental learning ( FSCIL ) , in which classifiers ’ label sets can be easily extended with small numbers of new examples and no retraining , is a key challenge for machine learning systems deployed in the real world ( Masana et al. , 2020 ) . As a concrete example , consider the classification problem depicted in Fig . 1 . A model , initially trained on a large set of examples from several base classes ( snorkel , arctic fox , meerkat ; Fig . 1a ) , must subsequently be updated to additionally recognize two novel classes ( white wolf and poncho ; Fig . 1b ) , and ultimately distinguish among all five classes ( Fig . 1c ) . Training a model to recognize the base classes is straightforward : for example , we can jointly optimize the parameters of a feature extractor ( perhaps a convolutional network parameterized by θ ) and a linear classification layer ( η ) to maximize the regularized likelihood of ( image , label ) pairs from the dataset in Fig . 1a : L ( θ , η ) = 1 n ∑ ( x , y ) log exp ( η > y fθ ( x ) ) ∑ y′ exp ( η > y′fθ ( x ) ) + α ( ‖η‖2 + ‖θ‖2 ) ( 1 ) But how can this model be updated to additionally recognize the classes in Fig . 1b , with only a few examples of each new class and no access to the original training data ? Naı̈vely continuing to optimize Eq . 1 on ( x , y ) pairs drawn from the new dataset will cause several problems . In the absence of any positive examples of those classes , performance on base classes will suffer due to catastrophic forgetting ( Goodfellow et al. , 2013 ) , while performance on novel classes will likely be poor as a result of overfitting ( Anderson & Burnham , 2004 ) . As a consequence , most past work on FSCIL has focused on alternative approaches that use nonstandard prediction architectures ( e.g. , Tao et al. , 2020b ) or optimize non-likelihood objectives ( e.g. , Yoon et al. , 2020 ; Ren et al. , 2019 ) . This divergence between approaches to standard and incremental classification has its own costs—state-of-the-art approaches to FSCIL are complicated , requiring nested optimizers , complex data structures , and numerous hyperparameters . When improved representation learning and optimization techniques are developed for standard classification problems , it is often unclear to how to apply these to the incremental setting . In this paper , we turn the standard approach to classification into a surprisingly effective tool for FSCIL . Specifically , we show that both catastrophic forgetting and overfitting can be reduced by introducing an additional subspace regularizer ( related to one studied by Agarwal et al . 2010 and Kirkpatrick et al . 2017 ) that encourages novel η to lie close to the subspace spanned by the base classes . On its own , the proposed subspace regularizer produces ordinary linear classifiers that achieve state-of-the-art results on FSCIL , improving over existing work in multiple tasks and datasets . Because of its simplicity , this regularization approach can be easily extended to incorporate additional information about relationships between base and novel classes . Using language data as a source of background knowledge about classes , we describe a variation of our approach , which we term semantic subspace regularization , that pulls weight vectors toward particular convex combinations of base classes that capture their semantic similarity to existing classes . This further improves accuracy by up to 2 % over simple subspace regularization across multiple tasks . These results suggest that FSCIL and related problems may not require specialized machinery to solve , and that simple regularization approaches can solve the problems that result from limited access to training data for both base and novel classes.1 1Code will be made publicly available . 2 BACKGROUND . A long line of research has focused on the development of automated decision-making systems that support online expansion of the set of concepts they can recognize and generate . An early example ( closely related to our learning-from-definitions experiment in Section 5 ) appears in the classic SHRDLU language grounding environment ( Winograd , 1972 ) : given the definition a steeple is a small triangle on top of a tall rectangle , SHRDLU acquires the ability to answer questions containing the novel concept steeple . Recent work in machine learning describes several versions of this problem in featuring more complex perception or control : Few-shot and incremental learning Few-shot classification problems test learners ’ ability to distinguish among a fixed set of classes using only a handful of labeled examples per class ( Scheirer et al. , 2012 ) . Most effective approaches to few-shot learning rely on additional data for pre-training ( Tian et al. , 2020 ) or meta-learning ( Vinyals et al. , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ; Yoon et al. , 2019 ) . One peculiarity of this evaluation paradigm is that , even when pre-trained , models are evaluated only on new ( few-shot ) classes , and free to update their parameters in ways that cause them to perform poorly on pre-training tasks . As noted by past work , a more realistic evaluation of models ’ ability to rapidly acquire new concepts should consider their ability to discriminate among both new concepts and old ones , a problem usually referred to as few-shot class-incremental learning ( FSCIL ) 2 ( Tao et al. , 2020b ) . FSCIL requires learners to incrementally acquire novel classes with few labeled examples while retaining high accuracy on previously learned classes . It combines the most challenging aspects of class-incremental learning ( Rebuffi et al. , 2017 ) task-incremental learning ( Delange et al. , 2021 ) , and rehearsal-based learning ( Rolnick et al. , 2019 ; Chaudhry et al. , 2019 ) , three related problems with much stronger assumptions about the kind of information available to learners . Existing approaches to this problem either prioritize novel class adaptation ( Ren et al. , 2019 ; Yoon et al. , 2020 ; Chen & Lee , 2021 ; Cheraghian et al. , 2021 ) or reducing forgetting in old classes ( Tao et al. , 2020b ) . Learning class representations Even prior to the widespread use of deep representation learning approaches , the view of classification as problem of learning class representations motivated a number of approaches to multi-class and multi-task learning ( Argyriou et al. , 2007a ; Agarwal et al. , 2010 ) . In few-shot and incremental learning settings , many recent approaches have also focused on the space of class representations ( Tao et al. , 2020a ) . Qi et al . ( 2018 ) initialize novel class representations using the average features from few-shot samples . Others ( Gidaris & Komodakis , 2018 ; Yoon et al. , 2020 ; Zhang et al. , 2021 ) train a class representation predictor via meta-learning , and Tao et al . ( 2020b ) impose topological constraints on the manifold of class representations as new representations are added . Alternatively , Chen & Lee ( 2021 ) models the visual feature space as a Gaussian mixture and use the cluster centers in a similarity-based classification scheme . Lastly , two concurrent works condition both old and new class representations at each session according to an auxiliary scheme ; graph attention network in Zhang et al . ( 2021 ) and relation projection in Zhu et al . ( 2021 ) . Our approach is related to Ren et al . ( 2019 ) , who proposes a nested optimization framework to learn auxiliary parameters for every base and novel class to influence the novel weights via regularization ; we show that these regularization targets can be derived geometrically without the need for an inner optimization step . Also related is the work of Barzilai & Crammer ( 2015 ) , which synthesizes the novel weights as linear combinations of base weights ; we adopt a regularization approach that allows learning of class representations that are not strict linear combinations of base classes . Learning with side information from language The use of background information from other modalities ( especially language ) to bootstrap learning of new classes is widely studied ( Frome et al. , 2013 ; Radford et al. , 2021 ; Reed et al. , 2016 ; Yan et al. , 2021 ) —particularly in the zero-shot learning and generalized zero-shot learning where side information is the only source of information about the novel class ( Chang et al. , 2008 ; Larochelle et al. , 2008 ; Akata et al. , 2013 ; Pourpanah et al. , 2Variants of this problem have gone by numerous names in past work , including generalized few-shot learning ( Schönfeld et al. , 2019 ) , dynamic few-shot learning ( Gidaris & Komodakis , 2018 ) or simply incremental few-shot learning ( Ren et al. , 2019 ; Chen & Lee , 2021 ) . 2020 ) . Specialized approaches exist for integrating side information in few-shot learning settings ( Schwartz et al. , 2019 ; Cheraghian et al. , 2021 ) . 3 PROBLEM FORMULATION . We follow the notation in Tao et al . ( 2020b ) for FSCIL : assume a stream of T learning sessions , each associated with a labeled dataset D ( 0 ) , D ( 1 ) , . . . , D ( T ) . Every D ( t ) consists of a support set S ( t ) ( used for training ) and a query set Q ( t ) ( used for evaluation ) . We will refer to the classes represented in D ( 0 ) as base classes ; as in Fig . 1a , we will assume that it contains a large number of examples for every class . D ( 1 ) ( and subsequent datasets ) introduce novel classes ( Fig . 1b ) . Let C ( S ) = { y : ( x , y ) ∈ S } denote the set of classes expressed in a set of examples S ; we will write C ( t ) = C ( S ( t ) ) and C ( ≤t ) : = ⋃ j≤t C ( S ( j ) ) for convenience . The learning problem we study is incremental in the sense that each support set contains only new classes ( C ( t ) ∩ C ( < t ) = ∅ ) 3 , while each query set evaluates models on both novel classes and previously seen ones ( C ( Q ( t ) ) = C ( ≤t ) ) . It is few-shot in the sense that for t > 0 , |S ( t ) | is small ( containing 1–5 examples for all datasets studied in this paper ) . Given an incremental learning session t > 0 the goal is to fine-tune existing classifier with the limited training data from novel classes such that the classifier performs well in classifying all classes learned thus far . FSCIL with a single session Prior to Tao et al . ( 2020b ) , a simpler version of the multi-session FSCIL was proposed by Qi et al . ( 2018 ) where there is only single incremental learning session after the pre-training stage i.e . T = 1 . This version , which we call single-session FSCIL , has been extensively studied by previous work ( Qi et al. , 2018 ; Gidaris & Komodakis , 2018 ; Ren et al. , 2019 ; Yoon et al. , 2020 ) . This problem formulation is the same as above with T = 1 : a feature extractor is trained on the samples from D ( 0 ) , then D ( 1 ) , then evaluated on samples with classes in C ( 0 ) ∪C ( 1 ) .

Summary: The paper proposes subspace regularization technique for incremental few-shot learning. The high-level idea is to find the basis vectors of the subspace spanned by the base classes, and then project the new classes into the subspace. The regularizer encourages the new class weights to be similar to the projected vectors with an L2 loss. Additionally, the paper also proposes to use semantic word vectors to perform regularization. Experimental results confirm that the proposed method is significantly better compared to Tao et al. (2020) and Chen & Lee (2021) either with an additional memory buffer or without, but it is unclear whether it is due to a better few-shot learning algorithm or simply a better pretrained checkpoint on the base classes. Moreover, the benefit of semantic regularization is not significant.