Year: 2021

Abstract:
Few-shot class incremental learning—the problem of updating a trained classifier to discriminate among an expanded set of classes with limited labeled data—is a key challenge for machine learning systems deployed in non-stationary environments. Existing approaches to the problem rely on complex model architectures and training procedures that are difficult to tune and re-use. In this paper, we present an extremely simple approach that enables the use of ordinary logistic regression classifiers for few-shot incremental learning. The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes. When combined with pretrained convolutional feature extractors, logistic regression models trained with subspace regularization outperform specialized, state-of-the-art approaches to few-shot incremental image classification by up to 22% on the miniImageNet dataset. Because of its simplicity, subspace regularization can be straightforwardly extended to incorporate additional background information about the new classes (including class names and descriptions specified in natural language); these further improve accuracy by up to 2%. Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.

heading: 1 INTRODUCTION
text: Standard approaches to classification in machine learning assume a fixed training dataset and a fixed set of class labels. But for many real-world classification problems, these assumptions are unrealistic. Classifiers must sometimes be updated on-the-fly to recognize new concepts (e.g. new skills in personal assistants or new road signs in self-driving vehicles), while training data is sometimes unavailable for reuse (e.g. due to privacy regulations, Lesort et al. 2019; McClure et al. 2018; or storage and retraining costs, Bender et al. 2021). Development of models that support few-shot class-incremental learning (FSCIL), in which classifiers’ label sets can be easily extended with small numbers of new examples and no retraining, is a key challenge for machine learning systems deployed in the real world (Masana et al., 2020).
As a concrete example, consider the classification problem depicted in Fig. 1. A model, initially trained on a large set of examples from several base classes (snorkel, arctic fox, meerkat; Fig. 1a), must subsequently be updated to additionally recognize two novel classes (white wolf and poncho; Fig. 1b), and ultimately distinguish among all five classes (Fig. 1c). Training a model to recognize the base classes is straightforward: for example, we can jointly optimize the parameters of a feature extractor (perhaps a convolutional network parameterized by θ) and a linear classification layer (η) to maximize the regularized likelihood of (image, label) pairs from the dataset in Fig. 1a:
L(θ, η) = 1 n ∑ (x,y) log exp(η>y fθ(x))∑ y′ exp(η > y′fθ(x)) + α ( ‖η‖2 + ‖θ‖2 ) (1)
But how can this model be updated to additionally recognize the classes in Fig. 1b, with only a few examples of each new class and no access to the original training data?
Naı̈vely continuing to optimize Eq. 1 on (x, y) pairs drawn from the new dataset will cause several problems. In the absence of any positive examples of those classes, performance on base classes
will suffer due to catastrophic forgetting (Goodfellow et al., 2013), while performance on novel classes will likely be poor as a result of overfitting (Anderson & Burnham, 2004).
As a consequence, most past work on FSCIL has focused on alternative approaches that use nonstandard prediction architectures (e.g., Tao et al., 2020b) or optimize non-likelihood objectives (e.g., Yoon et al., 2020; Ren et al., 2019). This divergence between approaches to standard and incremental classification has its own costs—state-of-the-art approaches to FSCIL are complicated, requiring nested optimizers, complex data structures, and numerous hyperparameters. When improved representation learning and optimization techniques are developed for standard classification problems, it is often unclear to how to apply these to the incremental setting.
In this paper, we turn the standard approach to classification into a surprisingly effective tool for FSCIL. Specifically, we show that both catastrophic forgetting and overfitting can be reduced by introducing an additional subspace regularizer (related to one studied by Agarwal et al. 2010 and Kirkpatrick et al. 2017) that encourages novel η to lie close to the subspace spanned by the base classes. On its own, the proposed subspace regularizer produces ordinary linear classifiers that achieve state-of-the-art results on FSCIL, improving over existing work in multiple tasks and datasets.
Because of its simplicity, this regularization approach can be easily extended to incorporate additional information about relationships between base and novel classes. Using language data as a source of background knowledge about classes, we describe a variation of our approach, which we term semantic subspace regularization, that pulls weight vectors toward particular convex combinations of base classes that capture their semantic similarity to existing classes. This further improves accuracy by up to 2% over simple subspace regularization across multiple tasks. These results suggest that FSCIL and related problems may not require specialized machinery to solve, and that simple regularization approaches can solve the problems that result from limited access to training data for both base and novel classes.1
1Code will be made publicly available.

heading: 2 BACKGROUND
text: A long line of research has focused on the development of automated decision-making systems that support online expansion of the set of concepts they can recognize and generate. An early example (closely related to our learning-from-definitions experiment in Section 5) appears in the classic SHRDLU language grounding environment (Winograd, 1972): given the definition a steeple is a small triangle on top of a tall rectangle, SHRDLU acquires the ability to answer questions containing the novel concept steeple. Recent work in machine learning describes several versions of this problem in featuring more complex perception or control:
Few-shot and incremental learning Few-shot classification problems test learners’ ability to distinguish among a fixed set of classes using only a handful of labeled examples per class (Scheirer et al., 2012). Most effective approaches to few-shot learning rely on additional data for pre-training (Tian et al., 2020) or meta-learning (Vinyals et al., 2016; Finn et al., 2017; Snell et al., 2017; Yoon et al., 2019). One peculiarity of this evaluation paradigm is that, even when pre-trained, models are evaluated only on new (few-shot) classes, and free to update their parameters in ways that cause them to perform poorly on pre-training tasks. As noted by past work, a more realistic evaluation of models’ ability to rapidly acquire new concepts should consider their ability to discriminate among both new concepts and old ones, a problem usually referred to as few-shot class-incremental learning (FSCIL)2 (Tao et al., 2020b).
FSCIL requires learners to incrementally acquire novel classes with few labeled examples while retaining high accuracy on previously learned classes. It combines the most challenging aspects of class-incremental learning (Rebuffi et al., 2017) task-incremental learning (Delange et al., 2021), and rehearsal-based learning (Rolnick et al., 2019; Chaudhry et al., 2019), three related problems with much stronger assumptions about the kind of information available to learners. Existing approaches to this problem either prioritize novel class adaptation (Ren et al., 2019; Yoon et al., 2020; Chen & Lee, 2021; Cheraghian et al., 2021) or reducing forgetting in old classes (Tao et al., 2020b).
Learning class representations Even prior to the widespread use of deep representation learning approaches, the view of classification as problem of learning class representations motivated a number of approaches to multi-class and multi-task learning (Argyriou et al., 2007a; Agarwal et al., 2010). In few-shot and incremental learning settings, many recent approaches have also focused on the space of class representations (Tao et al., 2020a). Qi et al. (2018) initialize novel class representations using the average features from few-shot samples. Others (Gidaris & Komodakis, 2018; Yoon et al., 2020; Zhang et al., 2021) train a class representation predictor via meta-learning, and Tao et al. (2020b) impose topological constraints on the manifold of class representations as new representations are added. Alternatively, Chen & Lee (2021) models the visual feature space as a Gaussian mixture and use the cluster centers in a similarity-based classification scheme. Lastly, two concurrent works condition both old and new class representations at each session according to an auxiliary scheme; graph attention network in Zhang et al. (2021) and relation projection in Zhu et al. (2021).
Our approach is related to Ren et al. (2019), who proposes a nested optimization framework to learn auxiliary parameters for every base and novel class to influence the novel weights via regularization; we show that these regularization targets can be derived geometrically without the need for an inner optimization step. Also related is the work of Barzilai & Crammer (2015), which synthesizes the novel weights as linear combinations of base weights; we adopt a regularization approach that allows learning of class representations that are not strict linear combinations of base classes.
Learning with side information from language The use of background information from other modalities (especially language) to bootstrap learning of new classes is widely studied (Frome et al., 2013; Radford et al., 2021; Reed et al., 2016; Yan et al., 2021)—particularly in the zero-shot learning and generalized zero-shot learning where side information is the only source of information about the novel class (Chang et al., 2008; Larochelle et al., 2008; Akata et al., 2013; Pourpanah et al.,
2Variants of this problem have gone by numerous names in past work, including generalized few-shot learning (Schönfeld et al., 2019), dynamic few-shot learning (Gidaris & Komodakis, 2018) or simply incremental few-shot learning (Ren et al., 2019; Chen & Lee, 2021).
2020). Specialized approaches exist for integrating side information in few-shot learning settings (Schwartz et al., 2019; Cheraghian et al., 2021).

heading: 3 PROBLEM FORMULATION
text: We follow the notation in Tao et al. (2020b) for FSCIL: assume a stream of T learning sessions, each associated with a labeled dataset D(0), D(1), . . . , D(T ). Every D(t) consists of a support set S(t) (used for training) and a query set Q(t) (used for evaluation). We will refer to the classes represented in D(0) as base classes; as in Fig. 1a, we will assume that it contains a large number of examples for every class. D(1) (and subsequent datasets) introduce novel classes (Fig. 1b). Let C(S) = {y : (x, y) ∈ S} denote the set of classes expressed in a set of examples S; we will write C(t) = C(S(t)) and C(≤t) := ⋃ j≤t C(S
(j)) for convenience. The learning problem we study is incremental in the sense that each support set contains only new classes (C(t) ∩ C(<t) = ∅)3, while each query set evaluates models on both novel classes and previously seen ones (C(Q(t)) = C(≤t)). It is few-shot in the sense that for t > 0, |S(t)| is small (containing 1–5 examples for all datasets studied in this paper). Given an incremental learning session t > 0 the goal is to fine-tune existing classifier with the limited training data from novel classes such that the classifier performs well in classifying all classes learned thus far.
FSCIL with a single session Prior to Tao et al. (2020b), a simpler version of the multi-session FSCIL was proposed by Qi et al. (2018) where there is only single incremental learning session after the pre-training stage i.e. T = 1. This version, which we call single-session FSCIL, has been extensively studied by previous work (Qi et al., 2018; Gidaris & Komodakis, 2018; Ren et al., 2019; Yoon et al., 2020). This problem formulation is the same as above with T = 1: a feature extractor is trained on the samples from D(0), then D(1), then evaluated on samples with classes in C(0) ∪C(1).

heading: 4 APPROACH
text: Our approach to FSCIL consists of two steps. In the base session, we jointly train a feature extractor and classification layer on base classes (Section 4.1). In subsequent (incremental learning) sessions, we freeze the feature extractor and update only the classification layer using regularizers that (1) stabilize representations of base classes, and (2) bring the representations of new classes close to existing ones (Sections 4.2-4.4).

heading: 4.1 FEATURE EXTRACTOR TRAINING
text: As in Eq. 1, we begin by training an ordinary classifier comprising a non-linear feature extractor fθ and a linear decision layer with parameters η. We choose η and θ to maximize:
L(η, θ) = 1 |S(0)| ∑ (x,y)∈S(0) log exp(η>y fθ(x))∑
c∈C(0) exp(η>c fθ(x))
− α ( ‖η‖2 + ‖θ‖2 ) (2)
As discussed in Section 5, all experiments in this paper implement fθ as a convolutional neural network. In subsequent loss formulations we refer to ‖η‖2 + ‖θ‖2 as Rprior(η, θ).

heading: 4.2 FINE-TUNING
text: Along with the estimated θ̂, feature extractor training yields parameters only for base classes ηy∈C(0) . Given an incremental learning dataset D(t), we introduce new weight vectors ηc∈C(t) and optimize
L(η) = 1 |S(t)| ∑ (x,y)∈S(t) log exp(η>y fθ̂(x))∑
c∈C(≤t) exp(η>c fθ̂(x))
− αRprior(η,0)− βR(t)old(η)− γR (t) new(η) . (3)
3This is the original setup established by Tao et al. (2020b). We will also present experiments in which we retain one example per class for memory replay following Chen & Lee (2021).
with respect to η alone. Eq. 3 features two new regularization terms, R(t)old and R (t) new. Rtold limits the extent to which fine-tuning can change parameters for classes that have already been learned:
R (t) old(η) = ∑ t′<t ∑ c∈C(t′) ‖ηt ′ c − ηc‖2 (4)
where ηt ′ c denotes the value of the corresponding variable at the end of session t ′. (For example, η0c refers to the weights for the base class c prior to fine tuning, i.e. after session t′ = 0.) As shown in Section 5.2, using Rold alone, and setting Rnew = 0, is a surprisingly effective baseline; however, performance can be improved by appropriately regularizing new parameters as described below.
Variant: Memory Following past work (Chen & Lee, 2021) which performs incremental learning while retaining a small “memory” of previous samples M , we explore an alternative baseline approach in which we append append S(t) in Eq. 3 with M (t). We define the memory at session t as M (t) = ⋃ (t′<t)M (t′) where M (t ′) ⊆ S(t′) and |M (t′)| = |C(t′)|. We sample only 1 example per previous class and we reuse the same example in subsequent sessions.

heading: 4.3 METHOD 1: SUBSPACE REGULARIZATION
text: Past work on other multitask learning problems has demonstrated the effectiveness of constraining parameters for related tasks to be similar (Jacob et al., 2008), lie on the same manifold (Agarwal et al., 2010) or even on the same linear subspace (Argyriou et al., 2007a). Moreover, Schönfeld et al. (2019) showed that a shared latent feature space for all classes is useful for class-incremental classification. Features independently learned for novel classes from small numbers of examples are likely to capture spurious correlations (unrelated to the true causal structure of the prediction problem) as a result of dataset biases (Arjovsky et al., 2019). In contrast, we expect most informative semantic features to be shared across multiple classes: indeed, cognitive research suggests that in humans’ early visual cortex, representations of different objects occupy a common feature space (Kriegeskorte et al., 2008). Therefore, regularizing toward the space spanned by base class weight vectors encourages new class representations to depend on semantic rather than spurious features and features for all tasks to lie in the same universal subspace.
We apply this intuition to FSCIL via a simple subspace regularization approach. Given a parameter for an incremental class ηc and base class parameters {ηj∈C(0)}, we first compute the subspace target mc for each class. We then compute the distance between ηc from mc and define:
R(t)new(η) = ∑ c∈C(t) ‖ηc −mc‖2 (5)
where mc is the projection of ηc onto the space spanned by {ηj∈C(0)}:
mc = P > C(0)ηc (6)
and PC(0) contains the orthogonal basis vectors of the subspace spanned by the initial set of base weights ηj∈C(0) . (PC(0) can be found using a QR decomposition of the matrix of base class vectors, as described in the appendix.)
Previous work that leverages subspace regularization for multitask learning assume that data from all tasks are available from the beginning (Argyriou et al., 2007b; Agarwal et al., 2010; Argyriou et al., 2007a). Our approach to subspace regularization removes these assumptions, enabling tasks (in this case, novel classes) to arrive incrementally and predictions to be made cumulatively over all classes seen thus far without any further information on which task that a query belongs to. Agarwal et al. (2010) is similar to ours in encouraging all task parameters to lie on the same manifold; it is different in that they learn the manifold and the task parameters alternately. Also related Simon et al. (2020) and Devos & Grossglauser (2019) model class representations over a set of subspaces (disjoint in the latter) for non-incremental few-shot learning.

heading: 4.4 METHOD 2: SEMANTIC SUBSPACE REGULARIZATION
text: The constraint in Eq. 5 makes explicit use of geometric information about base classes, pulling novel weights toward the base subspace. However, it provides no information about where within that
subspace the weights for a new class should lie. In most classification problems, classes have names consisting of natural language words or phrases; these names often contain a significant amount of information relevant to the classification problem of interest. (Even without having ever seen a white wolf, a typical English speaker can guess that a white wolf is more likely to resemble an arctic fox than a snorkel.) These kinds of relations are often captured by embeddings of class labels (or more detailed class descriptions) (Pennington et al., 2014).
When available, this kind of information about class semantics can be used to construct an improved subspace regularizer by encouraging new class representations to lie close to a convex combination of base classes weighted by their semantic similarity. We replace the subspace projection P>
C(0) ηc in
Eq. 5 with a semantic target lc for each class. Letting ec denote a semantic embedding of the class c, we compute:
R(t)new(η) = ∑ c∈C(t) ‖ηc − lc‖2 (7)
where
lc = ∑ j∈C(0) exp (ej · ec/τ)∑ j∈C(0) exp (ej · ec/τ) ηj (8)
and τ is a hyper-parameter. Embeddings ec can be derived from multiple sources: in addition to the class names discussed above, a popular source of side information for zero-shot and few-shot learning problems is detailed textual descriptions of classes; we evaluate both label and description embeddings in Section 5.
Schönfeld et al. (2019) also leverages label information on a shared subspace for few-shot incremental learning where they project both visual and semantic features onto a shared latent space for prediction in the single-session setting. In comparison, we re-use the base visual space for joint projection for multiple incremental sessions.
Baseline: Linear Mapping While the approach described in Eq. 7 combines semantic information and label subspace information, a number of previous studies in vision and language have also investigated the effectiveness of directly learning a mapping from the space of semantic embeddings to the space of class weights (Das & Lee, 2019; Socher et al., 2013; Pourpanah et al., 2020; RomeraParedes & Torr, 2015). Despite pervasiveness of the idea in other domains, this is the first time we are aware of it being explored for FSCIL. We extend our approach to incorporate this past work by learning a linear map L between the embedding space ej ∈ E and the weight space containing ηC(0) :
L∗ = min L ∑ j∈C(0) ‖ηj − L(ej)‖2 (9)
then set R(t)new = ∑ c∈C(t) ‖ηc − L∗(ec)‖2 . (10)
Concurrent work by (Cheraghian et al., 2021) also leverages side information for FSCIL where they learn a mapping from image space onto the label space to directly produce predictions in the label space. We provide comparisons in Section 5.

heading: 5 EXPERIMENTS
text: Given a classifier trained on an initial set of base classes, our experiments aim to evaluate the effect of subspace regularization (1) on the learning of new classes, and (2) on the retention of base classes. To evaluate the generality of our method, we evaluate using two different experimental paradigms that have been used in past work: a multi-session experiment in which new classes are continuously added and the classifier must be repeatedly updated, and a single-session setup (T = 1) in which new classes arrive only once. We use SGD as our optimizer to train all models. Details about experiment setups and results are discussed below. Additional details may be found in the appendix.

heading: 5.1 MULTI-SESSION
text: We follow the same setup established in Tao et al. (2020b) as well as Section 3: we first train a ResNet (He et al., 2016) network from scratch as the feature extractor on a large number of examples from base classes C(0) to obtain an initial classifier ηj∈C(0) . We then observe a new batch of examples S(t) and produce a new classifier defined by ηc∈C(≤t) . Finally, we evaluate the classifier according to top-1 accuracy in base and novel samples as well as their weighted average (Tao et al., 2020b; Chen & Lee, 2021). We use the miniImageNet dataset (Vinyals et al., 2016; Russakovsky et al., 2015) for our multi-session evaluation. miniImageNet contains 100 classes with 600 samples per class.
Table 1: Multi-Session FSCIL weighted average of accuracy (%) results on miniImageNet using an identical setup to Fig. 2 with memory distinction. We report the average results over 10 random splits of the data for incremental sessions 1, 2 and 8. ±M indicates 1 sample per class is kept (or not) in the memory to further regularize forgetting. Our Fine-tuning baseline is already superior to previous results. In both memory settings, our regularizers substantially outperform respective benchmarks for all 1-8 sessions. ∗Results are only estimates from the plot in the respective work. Bold indicates the highest.
Session 1 2 8 Model −M +M −M +M −M +M Tao et al. (2020b) 50.1 45.2 24.4 Chen & Lee (2021) 59.9 55.9 41.8 Fine-tuning 61.8 67.7 49.9 62.9 26.5 48.1 Subspace Reg. 71.7 72.9 66.9 67.8 46.8 48.8 +language
Cheraghian et al. (2021)* 58.0 53.0 39.0 Linear Mapping 72.6 73.2 67.1 68.0 46.9 50.0 Semantic Subspace Reg. 73.8 73.9 68.4 69.0 47.6 49.7
Feature extractor training In session t = 0, we randomly select 60 classes as base classes (|C(0)| = 60) and use the remaining 40 classes as novel classes. Reported results are averaged across 10 random splits of the data (Fig. 2). We use a ResNet-18 model that is identical to the one described in Tian et al. (2020). Following Tao et al. (2020b), we use 500 labeled samples per base class to train our feature extractor and 100 for testing.
Incremental evaluation Again following Tao et al. (2020b), we evaluate for a total of 8 incremental sessions 1 ≤ t ≤ 8 for miniImageNet. In each session, for S(t), we sample 5 novel classes for training and 5 samples from each class. Hence, at the last session t = 8, evaluation involves the entire set of 100 miniImageNet classes. We use GloVe embeddings (Pennington et al., 2014) for label embeddings in Eq. 7.
Results Fig. 2 and Table 1 show the results of multi-session experiments with and without memory. Session 0 indicates base class accuracy after feature extractor training. We compare subspace and language-guided regularization (linear mapping and semantic subspace reg.) to simple fine-
tuning (a surprisingly strong baseline). We also compare our results to three recent benchmarks: Tao et al. (2020b), Chen & Lee (2021) and Cheraghian et al. (2021).4
When models are evaluated on combined base and novel accuracy, subspace regularization outperforms previous approaches (by 22% (-M) and 7% (+M) at session 8); when semantic information about labels is available, linear mapping and semantic subspace regularization outperform Cheraghian et al. (2021) (Table 1). Evaluating only base sample accuracies (Fig. 2b), semantic subspace reg. outperforms others; compared to regularization based approaches fine-tuning is subject to catastrophic forgetting. Rold is still useful in regulating forgetting (Appendix B in appendix). The method of Chen & Lee (2021) follows a similar trajectory to our regularizers, but at a much lower accuracy (Fig. 2a). In Fig. 2c, a high degree of forgetting in base classes with fine-tuning allows higher accuracy for novel classes—though not enough to improve average performance (Fig. 2a). By contrast, subspace regularizers enable a good balance between plasticity and stability (Mermillod et al., 2013). In Table 1, storing as few as a single example per an old class substantially helps to reduce forgetting. Results from linear mapping and semantic subspace regularization are close, with semantic subspace regularization performing roughly 1% better on average. The two approaches offer different trade-offs between base and novel accuracies: the latter is more competitive for base classes and vice-versa.

heading: 5.2 SINGLE SESSION
text: In this section we describe the experiment setup for the single-session evaluation, (T = 1), and compare our approach to state-of-the-art XtarNet (Yoon et al., 2020), as well as Ren et al. (2019), Gidaris & Komodakis (2018) and Qi et al. (2018). We evaluate our models on 1-shot and 5-shot settings.5
miniImageNet and tieredImageNet For miniImageNet single-session experiments, we follow the the splits provided by Yoon et al. (2020). Out of 100, 64 classes are used in session t = 0, 20 in session t = 1 and the remaining for development. Following Yoon et al. (2020), we use ResNet-12 (a smaller version of the model described in Section 5.1). tieredImageNet (Ren et al., 2018) contains a total of 608 classes out of which 351 are used in session t = 0 and 160 are reserved for t = 1. The remaining 97 are used for development. While previous work (Ren et al., 2019; Yoon et al., 2020)
4Chen & Lee (2021) and Cheraghian et al. (2021) do not provide a codebase and Tao et al. (2020b) does not provide an implementation for the main TOPIC algorithm in their released code. Therefore, we report published results rather than a reproduction. This comparison is inexact: our feature extractor performs substantially better than Tao et al. (2020b) and Chen & Lee (2021). Despite extensive experiments (see appendix) on various versions of ResNet-18 (He et al., 2016), we were unable to identify a training procedure that reproduced the reported accuracy for session 0: all model variants investigated achieved 80%+ validation accuracy.
5Unlike in the preceding section, we were able to successfully reproduce the XtarNet model. Our version gives better results on the miniImageNet dataset but worse results on the tieredImageNet datasets; for fairness, we thus report results for our version of XtarNet on miniImageNet and previously reported numbers on tieredImageNet. For other models, we show accuracies reported in previous work.
separate 151 classes out of the 351 for meta training, we pool all 351 for feature extractor training. We train the same ResNet-18 described in Section 5.1. Additional details regarding learning rate scheduling, optimizer parameters and other training configurations may be found in the appendix.
Incremental evaluation We follow Yoon et al. (2020) for evaluation. Yoon et al. independently sample 2000D(1) incremental datasets (“episodes”) from the testing classes C(1). They report average accuracies over all episodes with 95% confidence intervals. At every episode, Q(1) is resampled from both base and novel classes, C(0) and C(1), with equal probability for both miniImageNet and tieredImageNet. We again fine-tune the weights until convergence. We do not reserve samples from base classes, thus the only training samples during incremental evaluation is from the novel classes C(1). We use the same resources for label embeddings and Sentence-BERT embeddings (Reimers & Gurevych, 2019) for descriptions which are retrieved from WordNet (Miller, 1995).
Results We report aggregate results for 1-shot and 5-shots settings of miniImageNet and tieredImageNet (Table 2). Compared to previous work specialized for the single-session setup without a straightforward way to expand into multi-session (Ren et al., 2019; Yoon et al., 2020), even our simple fine-tuning baseline perform well on both datasets—outperforming the previous stateof-the-art in three out of four settings in Table 2. Addition of subspace and semantic regularization improves performance overall but tieredImageNet 1-shot setting. Semantic subspace regularizers match or outperform linear label mapping. Subspace regularization outperforms fine-tuning in 5- shot settings and matches it in 1-shot. In addition to accuracy, we report a quantity labeled ∆ by Ren et al. (2019). ∆ serves as a measure of catastrophic forgetting, with the caveat that it can be minimized by a model that achieves a classification accuracy of 0 on both base and novel classes. We find that our approaches result in approximately the same ∆ in miniImageNet and worse in tieredImageNet than previous work.

heading: 6 ANALYSIS AND LIMITATIONS
text: What does regularization actually do? Fine-tuning results in prediction biased towards the most recently learned classes (top of Fig. 3) when no subspace regularization is imposed. Our experiments show that preserving the base weights while regularizing novel weights gives significant improvements over ordinary fine-tuning (bottom of Fig. 3)—resulting a fairer prediction over all classes and reducing catastrophic forgetting. In Table 1, Semantic Subspace Reg. results in 73.8% and 47.6% accuracies in the 1st and 8th sessions whereas, fine-tuning results in 61.8% and 26.5%, respectively,
even without any memory—suggesting that regularization ensures a better retention of accuracy. While the trade-off between accuracies of base and novel classes is inevitable due to the nature of the classification problem, the proposed regularizers provide a good balance between the two.
What are the limitations of the proposed regularization scheme? Our approach targets only errors that originate in the final layer of the model—while a convolutional feature extractor is used, the parameters of this feature extractor are fixed, and we have focused on FSCIL as a linear classification problem. Future work might extend these approaches to incorporate fine-tuning of the (nonlinear) feature extractor itself while preserving performance on all classes in the longer term.

heading: 7 CONCLUSIONS
text: We have described a family of regularization-based approaches to few-shot class-incremental learning, drawing connections between incremental learning and the general multi-task and zero-shot learning literature. The proposed regularizers are extremely simple—they involve only one extra hyperparameter, require no additional training steps or model parameters, and are easy to understand and implement. Despite this simplicity, our approach enables ordinary classification architectures to achieve state-of-the-art results on the doubly challenging few-shot incremental image classification across multiple datasets and problem formulations.

heading: A CODE AND DATASETS
text: Code will be made publicly available. We use miniImageNet (Vinyals et al., 2016) and tieredImageNet (Ren et al., 2018) datasets both are subsets of ImageNet dataset (Russakovsky et al., 2015). Use of terms and licenses are available through the respective sources.

heading: B ANALYSIS OF CATASTROPHIC FORGETTING
text: In Appendix B, we demonstrate the effectiveness of the regularization termRold in mitigating catastrophic forgetting.

heading: C RESULTS IN TABULAR FORM
text: In Table 4 and Table 5, we present the multi-session results in the main paper in the tabular form.

heading: D DETAILS OF FEATURE EXTRACTOR TRAINING
text: We use the exact ResNet described in Tian et al. (2020), the differences compared to the standard ResNet (He et al., 2016): (1) Each block (collection of convolutional blocks) is composed of three convolutional layers instead of two. (2) Number of blocks for ResNet-12 is 4 instead of 6 of the standard version, thus the total number of convolutional layers are the same. (3) Filter sizes are
[64,160,320,640] rather than [64,128,256,512], though the total number of filters is comparable since Tian et al. (2020) has less blocks. (4) There is Dropblock at the end of the last blocks.
Tian et al. (2020) provides a full visualization in Appendix and their code repository6 is easy to browse on which we base our own codebase. We observe that the previous work oftentimes use their slightly modified version of the standard ResNet. Ren et al. (2019) uses the ResNet-10 (Mishra et al., 2017) and ResNet-18 for for miniImageNet and tieredImageNet, respectively. XtarNet(Yoon et al., 2020) is originally based on a slightly modified version of ResNet-12 and ResNet-18 which we replaced with our version, improving their results for miniImageNet but not in tieredImageNet, thus we report improved results for miniImageNet and their results for tieredImageNet in the main paper.
D.1 DEFAULT SETTINGS
Unless otherwise indicated we use the following default settings of Tian et al. (2020) in our feature extractor training. We use SGD optimizer with learning starting at 0.05 with decays by 0.1 at epochs 60 and 80. We train for a total of 100 epochs. Weight decay is 5e-4, momentum is 0.9 and batch size is 64. As per transformations on training images, we use random crop of 84x84 with padding 8. We also use color jitter (brightness=0.4, contrast=0.4, saturation=0.4) and horizontal flips. For each run, we sample 1000 images from base classes and 25 images from each of novel classes. Our classifier does not have bias.
D.2 MULTI-SESSION miniIMAGENET
We re-sample the set of base classes (60 classes) 10 times across different seeds and train ten ResNet18 architectures. Each class has 500 training images. We follow the default settings for training.
D.3 MULTI-SESSION COMPARISON TO STANDARD RESNET-18
In Table 6 we provide validation set results for two types of ResNet-18’s: Tian et al. (2020) and He et al. (2016) across ten different seeds. Results show that use of Tian et al. (2020) does not incur unfair advantage over those who used He et al. (2016).
D.4 SINGLE-SESSION miniIMAGENET
We follow the default hyperparameters parameters Appendix D.1 except that for training, validation and testing we use the exact splits provided by Ren et al. (2019) also used by Yoon et al. (2020). There are 64 base, 16 validation and 20 testing classes provided (totaling 100). Training data consists of 600 images per base class. Dataset statistics are delineated in the Appendix of Ren et al. (2019) and downloadable splits are available here, courtesy of Ren et al. (2019).
D.5 SINGLE-SESSION tieredIMAGENET
tieredImageNet is first introduced by Ren et al. (2018). Same as above, we use the default parameters except that we train for a total of 60 epochs decaying the initial learning rate of 0.05 by 0.1 at epochs 30 and 45. Again, we use the same data as previous work available at the same link above. tieredImageNet is split into 351, 97 and 160 classes. Past work that use meta-learning Ren et al. (2019); Yoon et al. (2020) split 351 training classes into further 200 and 151 clases where the latter is used for meta learning. We pool all 351 for feature extractor training. At the end of feature extractor
6https://github.com/WangYueFt/rfs
training, we only keep the classifier weights for the first 200 classes to adhere to the evaluation scheme of 200+5 classes as past work.

heading: E DETAILS OF INCREMENTAL EVALUATION
text: E.1 QR DECOMPOSITION FOR SUBSPACE REGULARIZATION
To compute the orthogonal basis PC(0) for the subspace spanned by base classifier weights ηC(0) we use QR decomposition(Trefethen & Bau III, 1997):
[ PC(0) Q ′] [R 0 ] = η>C(0) (11)
E.2 MULTI-SESSION
For testing, we sample 1000 images from base classes and 25 images from each of novel classes. Testing images from a given class stay the same across sessions. Harmonic mean results take into account the ratio of base classes to novel classes in a given session. In this setting, there is no explicit development set (with disjoint classes than train and test) defined by previous work thus we use the first incremental learning session (containing 5 novel classes) as our development set.
Default settings We use the same transformations as in Appendix D.1 on the training images. We stop fine-tuning when loss does not change more than 0.0001 for at least 10 epochs. We use SGD optimizer. We repeat the experiments 10 times and report the average accuracy with standard deviation (95% confidence interval) in the paper.
Simple Fine-tuning We use learning rate of 0.002 and do not use learning decay. Weight-decay α is set at 5e-3. In order to limit the change in weights, we use different β’s for base and previously learned novel classes, where the former is 0.2 and the latter 0.1. We rely on the default settings otherwise.
Subspace Regularization Different from simple fine-tuning we use a weight decay of 5e-4. There is an additional parameter called γ in this setting controlling the degree of pulling of novel weights toward the subspace which we set to 1.0.
Semantic Subspace Regularization Different than simple subspace regularization, there is a temperature parameter used in the Softmax operation used in computation of lc’s which we set to 3.0.
Linear mapping regularization Same parameters as in subspace regularization are used except γ = 0.1. We formulate L as a linear layer with bias and we use gradient descent to train the parameters.
E.3 SINGLE-SESSION
In our 1-shot experiments unless fine-tuning converges by then, we stop at the maximum number of epochs at 1000. We sample 2000 episodes which includes 5 novels classes and 1-5 samples from each and report average accuracy. For testing, base and novel samples have equal weight in average per previous work (Yoon et al., 2020). SGD optimizer is used. For description similarity we use Sentence-BERT’s stsb-roberta-large.
miniImageNet Settings We use the same set of transformations on the training images as described in Appendix D.1. We first describe details of 1-shot setting. In 1-shot experiments we set the maximum epochs to 1000. For simple fine-tuning we use learning rate of 0.003, and weight decay of 5e-3. In Semantic Subspace Reg., we set temperature to 1.5. Both in Semantic Subspace Reg. and linear mapping γ = 0.005 and weight-decay is 5e-4. In subspace regularization, γ = 0.005 and weight-decay is set to 5e-5. Description similarity follows the same setup as Semantic Subspace Reg..
In 5-shot setting, we set β = 0.03 weight-decay to 5e-3 and learning rate to 0.002. For subspace regularization, Semantic Subspace Reg. and linear mapping we use γ = 0.03 and for description similarity we use γ = 0.01.
tieredImageNet Settings In 1-shot setting, fine-tuning uses learning rate of 0.003. Semantic Subspace Reg. has learning rate of 0.005, weight-decay 5e-3 and γ = 0.005. Subspace reg. and linear mapping use γ = 0.001.
In 5-shot setting, for simple fine-tuning we set lr = 0.001, weight-decay=5e-3, β = 0.3. For Semantic Subspace Reg. γ = 0.05 and β = 0.2 while others have γ = 0.03.
F VISUALIZATIONS
In Fig. 4 and Fig. 5 we depict principal components of classifier weights as well as semantic or subspace targets for novel weights.

heading: G COMPUTE
text: We use a single 32 GB V100 NVIDIA GPU for all our experiments.

Review summaries:
The paper proposes subspace regularization technique for incremental few-shot learning. The high-level idea is to find the basis vectors of the subspace spanned by the base classes, and then project the new classes into the subspace. The regularizer encourages the new class weights to be similar to the projected vectors with an L2 loss. Additionally, the paper also proposes to use semantic word vectors to perform regularization. Experimental results confirm that the proposed method is significantly better compared to Tao et al. (2020) and Chen & Lee (2021) either with an additional memory buffer or without, but it is unclear whether it is due to a better few-shot learning algorithm or simply a better pretrained checkpoint on the base classes. Moreover, the benefit of semantic regularization is not significant.

The authors tackle the problem of few-shot class incremental learning (FSCIL). In this setting, a learner is first trained on a set of base classes for which plenty of data is available. Then, it must learn from small sets of novel classes while retaining good performance in the whole sequence. In order to prevent the classifier from overfitting new classes, they authors propose to regularize novel class weights to keep them in the subspace of base classes, thus making the classifier rely on known features rather than new features that are probably spurious. In addition, they also show that it is possible to align new classes with base classes based on their similarity in a learned semantic space such as Glove. Experimental results on miniImagenet and tiered-Imagenet show that their method obtains better performance than previous state of the art.

This paper leverages a simple but effective regularization technique for few-shot incremental class learning. The assumption is that at the first stage we have a lot of data and can train a good feature extractor and a network that can classify all the classes to a good degree. Given a few examples of a new unseen class, the authors enforce that the weights of the final layer of the classifier are close to the space that is spanned by the current classifier's last layer weights. They achieve state-of-the-art results on this very challenging task.