Subspace Regularizers for Few-Shot Class Incremental Learning
Authors: Afra Feyza Akyürek
Ekin Akyürek
Derry Wijaya
Jacob Andreas
Venue: ICLR 2022 Poster (2022)
Keywords: few-shot class incremental learning
incremental learning
incremental classification
subspace regularization
manifold regularization
few-shot learning

Abstract: Few-shot class incremental learning---the problem of updating a trained classifier to discriminate among an expanded set of classes with limited labeled data---is a key challenge for machine learning systems deployed in non-stationary environments. Existing approaches to the problem rely on complex model architectures and training procedures that are difficult to tune and re-use. In this paper, we present an extremely simple approach that enables the use of ordinary logistic regression classifiers for few-shot incremental learning. The key to this approach is a new family of \textit{subspace regularization} schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes. When combined with pretrained convolutional feature extractors, logistic regression models trained with subspace regularization outperform specialized, state-of-the-art approaches to few-shot incremental image classification by up to 23\% on the \textit{mini}ImageNet dataset. Because of its simplicity, subspace regularization can be straightforwardly configured to incorporate additional background information about the new classes (including class names and descriptions specified in natural language); this offers additional control over the trade-off between existing and new classes. Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.

Published as a conference paper at ICLR 2022
SUBSPACE REGULARIZERS FOR FEW-SHOT CLASS INCREMENTAL LEARNING
Afra Feyza Akyürek Boston University akyurekêbu.. edd
Ekin Akyürek MIT CSAIL a kyurekêmi t. edu
Derry Tanti Wijaya Boston University wi jaya@bu. edu
Jacob Andreas MIT CSAIL. jda @mi it edu
ABSTRACT
Few-shot class incremental learning- the problem of updating a trained classifier to discriminate among an expanded set of classes with limited labeled data-is a key challenge for machine learning systems deployed in non-stationary environ- ments. Existing approaches to the problem rely on complex model architectures and training procedures that are difficult to tune and re-use. this paper, we present an extremely simple approach that enables the use of ordinary logistic re- gression classifiers for few-shot incremental learning. The key this approach is new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes. When combined with pretrained convolutional feature extractors, logis- tic regression models trained with subspace regularization outperform specialized, state-of-the-an approaches to few-shot incremental image classification by up 23% on the minilmageNet dataset. Because of its simplicity, subspace regular- ization can be straightforwardly configured to incorporate additional background information about the new classes (including class names and descriptions speci- fied in natural language); this offers additional control over the trade-off between existing and new classes. Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.
INTRODUCTION
Standard approaches to classification in machine learning assume fixed training dataset and fixed of class labels. But for many real-world classification problems, these assumptions are unreal- istic. Classifiers must sometimes be updated on-the-fly to recognize new concepts (e.g. new skills personal assistants or new road signs self-driving vehicles), while training data sometimes unavailable for reuse (e.g. due to privacy regulations, Lesort et al. 2019; McClure et al. 2018; or storage and retraining costs, Bender et al. 2021). Development of models that support few-shot clas-incremental learning (FSCIL), in which classifiers' label sets can be easily extended with small numbers of new examples and no retraining, is a key challenge for machine learning systems deployed in the real world (Masana et al., 2020).
As a concrete example, consider the classification problem depicted in Fig. 1. A model, initially trained large set of examples from several base classes (snorkel, arctic fox, meerkat; Fig. 1a), must subsequently be updated to additionally recognize two novel classes (white wolf and poncho; Fig. 1b), and ultimately distinguish among all five classes (Fig. Ic). Training a model to recognize the base classes is straightforward: for example, we can jointly optimize the parameters of a feature extractor (perhaps convolutional network parameterized by 0) and linear classification layer (7) to maximize the regularized likelihood of (image, label) pairs from the dataset in Fig. la:
exp(n, fo(x)) (z.v) Ey (ep(nnJo(())
L( n)
log
+ +
(1)
But how can this model be updated to additionally recognize the classes in Fig. Ib, with only a few examples of each new class and access the original training data?
Code for the experiments is released under http://gnhh...mmmoooyaa//u//aaaa

---

Published as a conference paper at ICLR 2022
(a) Base Clussifier Training
(b) Incremental Learning
(c) Testing
storlee
D
while wall
D
Q
Sutspece Regulerzzatie
(1 Semsrtii Subspace Regularizrdio
Pthits wall
(I1 I Linsar Mapping Regulartzation
Palii
sell
lnoebel
Psoockel
Tecedbel
snockel
Pens
Minerrkal
Recelun
Proodsat
Asseekot
Aahaa sale
ardtic for shilc
Parotic tox
Agoti
Marctic Box
Clasilfer Weights Space
Language Enabeödiing Space
Clastifee Weights Space
Languag Krshalding Spuce
Classifier Wrights Space
ClassiSe weight
Suleqaaze (xpresentative))
regelarizaiion dinection
Label eabbedling
product emibrrtyy
Figure 1: Few-shot class incremental learning: (a) base classifier is trained on a large dataset (D (b) This classifier is extended to also discriminate among a set of new classes with a small number of labeled examples (D (c) Models evaluated on lest that includes all classes (Q This paper focuses on extremely simple, regularization-bett
approaches to FSCIL, with and without side information from natural language: (i) We regularize novel classifier weights toward the shortest direction to the subspace spanned by base classifier weights. (ii) We regularize novel classifiers pulling them toward the weighted average of base classifiers where weights calculated using label/description similarity between novel and base class names or one-sentence descriptions. (iii) learn a linear mapping between word labels and classifier weights the base classes. Later, we project the novel label white wolf and regularize the novel classifier weight 1Juhite woll towards the projection.
Naïvely continuing to optimize Eq. 1 on (x, y) pairs drawn from the new dataset will cause several problems. In the absence of any positive examples of those classes, performance on base classes will suffer due to catastrophic forgetting (Goodfellow et al., 2013), while performance on novel classes will likely be poor as a result of overfitting (Anderson & Burnham, 2004).
As consequence, most past work on FSCIL has focused on alternative approaches that use non- standard prediction architectures (e.g., Tao et al., 2020b) or optimize non-likelihood objectives (e.g., Yoon et al., 2020; Ren et al., 2019). This divergence between approaches to standard and incremen- tal classification has its own costs stass-ttate-o--the--a approaches to FSCIL are complicated, requiring nested optimizers, complex data structures, and numerous hyperparameters. When improved repre- sentation learning and optimization techniques are developed for standard classification problems, it is often unclear to how apply these to the incremental setting.
In this paper. we turn the standard approach to classification into a surprisingly effective tool for FSCIL. Specifically, we show that both catastrophic forgetting and overfitting can be reduced by introducing an additional subspace regularizer (related to one studied by Agarwal et al. 2010 and Kirkpatrick al. 2017) that encourages novel 1) lie close to the subspace spanned by the base classes. On its own, the proposed subspace regularizer produces ordinary linear classifiers that achieve state-of-the-art results on FSCIL, improving over existing work in multiple tasks and datasets.
Because of its simplicity, this regularization approach can be casily configured to incorporate ad- ditional information about relationships between base and novel classes. Using language data as source of background knowledge about classes, we describe a variation of our approach, which we term semantic subspace regularization, that pulls weight vectors toward particular convex combi- nations of base classes that capture their semantic similarity to existing classes. Semantic subspace regularization results in comparable performance to subspace regularization ter preserves performance on existing classes at the expense of higher accuracy on newer classes. These results suggest that FSCIL and related problems may not require specialized machinery to solve, and that simple regularization approaches can solve the problems that result from limited access to training data for both base and novel classes.
average; it bet-
2

---

Published as a conference paper at ICLR 2022
2 BACKGROUND
A long line of research has focused on the development of automated decision-making systems that support online expansion of the set of concepts they can recognize and generate. An early example (closely related to our classic SHRDLU language grounding environment (Winograd, 1972): given the definition a steeple is small triangle on top of tall rectangle, SHRDLU acquires the ability to answer questions containing the novel concept steeple. Recent work in machine learning describes several versions of this problem in featuring more complex perception or control:
experiment in Section 5) appears in the
Few-shot and incremental learning Few-shot classification problems test learners' ability to dis- tinguish among fixed set of classes using only handful of labeled examples per class (Scheirer et al., 2012). Most effective approaches to few-shot learning rely on additional data for pre-training (Tian et al., 2020) or meta-learning (Vinyals et al., 2016; Finn et al., 2017; Snell et al., 2017; Yoon et al., 2019). One peculiarity of this evaluation paradigm is that, even when pre-trained, models are evaluated only on new (few-shot) classes, and free to update their parameters in ways that cause them to perform poorly on pre-training tasks. As noted by past work (Kuzborskij et al., 2013), a more realistic evaluation of models' ability to rapidly acquire new concepts should consider their ability to discriminate among both new concepts and old ones, problem usually referred to as few-shot inass-incremetall learning (FSCIL)? (Tao et al., 2020b).
FSCIL requires learners to incrementally acquire novel classes with few labeled examples while retaining high accuracy on previously learned classes. combines the most challenging aspects of class-incremental learning (Rebuffi et al., 2017) task-incremental learning (Delange et al., 2021), 1), and rehearsal- learning (Rolnick et al., 2019; Chaudhry et al., 2019), three related problems with much stronger assumptions about the kind of information available to learners Existing ap- proaches to this problem either prioritize novel class adaptation (Ren et al., 2019; Yoon et al., 2020; Chen & Lee, 2021; Cheraghian al., 2021) or reducing forgetting in old classes (Tao et al., 2020b).
Learning class representations Even prior to the widespread use of deep representation learn- ing approaches, the view of classification as problem of learning class representations motivated a number of approaches to multi-class and multi-task learning (Argyriou et al., 2007a; Agarwal et al., 2010). In few-shot and incremental learning settings, many recent approaches have also focused on the space of class representations (Tao et al., 2020a). Qi et al. (2018) initialize novel class repre- sentations using the average features from few-shot samples. Others (Gidaris Komodakis, 2018; Yoon et al., 2020; Zhang et al., 2021) train class representation predictor via meta-learning, and Tao et al. (2020b) impose topological constraints on the manifold of class representations as new representations are added. Alternatively, Chen & Lee (2021) models the visual feature space Gaussian mixture and use the cluster centers in a similarity- based classification scheme. Lastly, two concurrent works condition both old and new class representations at each session according to an auxiliary scheme; graph attention network in Zhang et al. (2021) and relation projection in Zhu et al. (2021).
Our approach is related to Ren et al. (2019), who uses a nested optimization framework to learn auxiliary parameters for every base and novel class inffuence the novel weights via regularization; we show that these regularization targets can be derived geometrically without the need for an inner optimization step. Also related is the work of Barzilai & Crammer (2015), which synthesizes the novel weights as linear combinations of base weights; we adopt regularization approach that allows learning of class representations that are not strict linear combinations of base classes. Moreover, Kuzborskij et al. (2013) study a class incremental learning setup where they increment the number of classes by one. Similar to ours, the parameters for the novel class is regularized towards a weighted combination of old class parameters while using as many examples from old classes there are from novel classes. In comparison, our approach does not require any examples from old classes.
Learning with side information from language The use of background information from other modalities (especially language) to bootstrap learning of new classes is widely studied (Frome et al.,
Variants of this problem have gone by numerous names in past work, including generalized few- few-shot learn- ing (Schönfeld et al., 2019), dynamic few-shot learning (Gidaris & Komodakis, 201 8) or simply incremental seo-shot learning (Ren et al., 2019; Chen & Lee, 2021).
3

---

Published as a conference paper at ICLR 2022
2013; Radford et al., 2021; Reed et al., 2016; Yan et al., 2021) particularly in the zero-shot learn- ing and generalized zero-shot learning where side information the only source of information about the novel class (Chang et al., 2008; Larochelle et al., 2008; Akata et al., 2013; Pourpanah et al., 2020). Specialized approaches exist for integrating side information few-shot learning settings (Schwartz al., 2019; Cheraghian et al., 2021).
3 PROBLEM FORMULATION
We follow the notation in Tao et al. (2020b) for FSCIL: assume a stream of T learning sessions, each associated with a labeled dataset D(0), D(1), ,(.,...., D(T), Every D consists support set ((t) (used for training) and query set Q(+) (used for evaluation). We will refer to the classes represented in D base classes; in Fig. la, we will assume that it contains large number of examples for every class. D(1) (and subsequent datasets) introduce novel classes (Fig. lb). Let C(S) = {y : (x, y) E S} denote the set of classes expressed in a set of examples S; we will write C(O = C(S(t) and ((S))) := Uمzt C(Sb)) for convenience. The learning problem we study is incremental in the sense that each support set contains only new classes (C(t) n C(st) 0y) while each query set evaluates models on both novel classes and previously seen ones (C(Q(()) = C(St)). few-shot in the sense that for > 0, | small (containing 1-5 examples for datasets studied in this paper). Given incremental learning session t 0 the goal is to fine-tune existing classifier with the limited training data from novel classes such that the classifier performs well in classifying all classes learned thus far.
FSCIL with single session Prior to Tao al. (2020b), simpler version of the multi-session FSCIL was proposed by Qi et al. (2018) where there is only single incremental learning session after the pre-training stage i.e. T 1. This version, which we call single-session FSCIL, has been extensively studied by previous work (Qi et al., 2018; Gidaris & Komodakis, 2018; 8; Ren et al., 2019; Yoon et al., 2020). This problem formulation is the same as above with T 1: a feature extractor is trained the samples from D(0), then D(1), then evaluated samples with classes in ( U (C))
4 APPROACH
Our approach to FSCIL consists of two steps. In the base session, we jointly train a feature extractor and classification layer on base classes (Section 4.1). subsequent (incremental learning) sessions, we freeze the feature extractor and update only the classification layer using regularizers that (1) stabilize representations of base classes, and (2) bring the representations of new classes close to existing ones (Sections 4.2-4.4).
4.1 FEATURE EXTRACTOR TRAINING
As in Eq. begin by training an ordinary classifier comprising a non-linear feature extractor fo and linear decision layer with parameters 7). We choose and 0 to maximize:
exp(17, fo(x)) exp(n fo(x)) a (1ln) + 10113)
C(n. 0) = Sto
log
(2)
(2.0)
(EC(O)
As discussed in Section 5, all experiments in this paper implement fo as convolutional neural network. In subsequent loss formulations we refer to ||>l ||0|2 as Rynior 0).
4.2 FINE-TUNING
Along with the estimated ò, feature extractor training yields parameters only for base classes 7.EC Given an incremental learning dataset D(1,, we introduce new weight vectors NeEcx
This is the original setup established by Tao et al. (2020h). We will also present experiments in which we retain example per class for memory replay following Chen & Lee (2021).
4

---

Published as a conference paper at ICLR 2022
and optimize
exp(n, fg(x))
((n) = |500|
log
Apriorr (7, 0) - -3 (1) - (n) (3)
Σ exp(ne fo(x))
CEC(SI)
with respect to ry alone. Eq. 3 features two new regularization terms, Rel and R Raud which also appears in past work (Kuzborskij et al., 2013), limits the extent to which fine-tuning can change parameters for classes that have already been learned:
R (17
E Σ
Il -
CEC()))
where v/ denotes the value of the corresponding variable at the end of session 1'. (For example, ne refers to the weights for the base class e prior fine tuning, i.c. after session t' 0.) As shown in Section 5.2, using Rold alone, and setting Haww = 0, is a surprisingly effective baseline; however, performance can be improved by appropriately regularizing new parameters as described below.
Variant: Memory Following past work (Chen & Lee, 2021) which performs incremental learn- ing while retaining a small "memory" of previous samples M. we explore an alternative baseline approach in which we append ((t) in Eq. 3 with M( We define the memory session as M(t) -w M(') where M( C s(E') and [M [C(e)) We sample only 1 example per previous class and we reuse the same example in subsequent sessions.
4.3 METHOD 1: SUBSPACE REGULARIZATION
Past work on other multitask learning problems has demonstrated the effectiveness of constraining parameters for related tasks to be similar (Jacob et al., 2008), lie on the same manifold (Agarwal et al., 2010) or even on the same linear subspace (Argyriou et al., 2007a). Moreover, Schönfeld ct al. (2019) showed that a shared latent feature space for all classes useful for class-iccremental classification. Features independently learned for novel classes from small numbers of examples are likely to capture spurious correlations (unrelated to the true causal structure of the prediction problem) result of dataset biases (Arjovsky al., 2019). In contrast, we expect most informative semantic features to be shared across multiple classes: indeed, cognitive research suggests that in humans' early visual cortex, representations of different objects occupy a common feature space (Kriegeskorte et al., 2008). Therefore, regularizing toward the space spanned by base class weight vectors encourages new class representations depend on semantic rather than spurious features and features for all tasks to lie in the same universal subspace.
We apply this intuition to FSCIL via a simple subspace regularization approach. Given a parameter for an incremental class Te and base class parameters {π] }, we first compute the subspace target m. for each class. We then compute the distance between 7e from me and define:
R(O)
Σ
Il
mell2
CEC(!)
where m. is the projection of onto the space spanned by {mjесto)::
(6) and Poco contains the orthogonal basis vectors of the subspace spanned by the initial set of base weights 7)Ec (Poro) can be found using QR decomposition of the matrix of base class vectors, as described in the appendix.)
m, = P Th
Previous work that leverages subspace regularization for multitask learning assume that data from all tasks are available from the beginning (Argyriou et al., 2007b; Agarwal et al., 2010; Argyriou et al., 2007a). Our approach to subspace regularization removes these assumptions, enabling tasks (in this case, novel classes) to arrive incrementally and predictions to be made cumulatively over all classes seen thus far without any further information which task that a query belongs to. Agarwal et al. (2010) is similar to ours in encouraging all task parameters to lie on the same manifold; is different in that they learn the manifold and the task parameters alternately. Also related Simon et al. (2020) and Devos & Grossglauser (2019) model class representations over a set of subspaces (disjoint in the latter) for non-incremental few-shot learning.
5

---

Published as a conference paper at ICLR 2022
4.4 METHOD 2: SEMANTIC SUBSPACE REGULARIZATION
The constraint in Eq. 5 makes explicit use of geometric information about base classes, pulling novel weights toward the base subspace. However, it provides no information about where within that subspace the weights for a new class should lie potentially causing interference with base classes. most classification problems, classes have names consisting of natural language words or phrases; these names often contain significant amount of information relevant to the classification problem of interest. (Even without having ever seen white wolf. typical English speaker can guess that a white wolf is more likely to resemble an arctic fox than snorkel.) These kinds of relations are often captured by embeddings of class labels (or more detailed class descriptions) (Pennington et al., 2014).
When available, this kind of information about class semantics can be used to construct an improved subspace regularizer by encouraging new class representations to lie close to convex combination of base classes weighted by their semantic similarity. We replace the subspace projection P Poroy Te in Eq. 5 with semantic target l. for each class. Letting ec denote semantic embedding of the class c, we compute:
R
Σ oECU
I Lell
where
exp (ej ee/T) 1e Σ
jEC E exp (ej cc/T) d
(8)
and r is a hyper-parameter. Embeddings e can be derived from multiple sources: in addition to the class names discussed above, popular source of side information for zero-shot and few-shot learning problems is detailed textual descriptions of classes; we evaluate both label and description embeddings in Section 5.
Schönfeld et al. (2019) also leverage label information shared subspace for few-shot incre- mental learning where they project both visual and semantic features onto a shared latent space for prediction in the single-session setting. In comparison, re-use the base visual space for joint projection for multiple incremental sessions.
Baseline: Linear Mapping While the approach described in Eq. combines semantic informa- tion and label subspace information, number of previous studies in vision and language have also investigated the effectiveness of directly learning mapping from the space of semantic embeddings to the space of class weights (Das & Lee, 2019; Socher al., 2013; Pourpanah al., 2020; Romera- Paredes Torr, 2015). Despite pervasiveness of the idea other domains, this is the first time we aware of it being explored for FSCIL. We extend our approach incorporate this past work by learning a linear map L between the embedding space ej E E and the weight space containing Igoo
L min Σ jEC(o)
|| I(e,)]1
then set
R(U)
Σ eEC(
|| - (e
(10)
new
Concurrent work by (Cheraghian et al., 2021) also leverages side information for FSCIL where they leam mapping from image space onto the label space to directly produce predictions in the label space. We provide comparisons in Section 5.
EXPERIMENTS
Given a classifier trained on an initial set of base classes, our experiments aim to evaluate the effect of subspace regularization (1) the learning of new classes, and (2) on the retention of base classes. To evaluate the generality of our method, we evaluate using two different experimental paradigms that have been used in past work: multi-session experiment in which new classes are continuously

---

We follow the same setup established in Tao et al. (2020b) as well as Sec- tion 3: we first train ResNet (He et al., 2016) network from scratch as the fea- ture extractor a large number of ex- amples from base classes (0)) to ob- tain an initial classifier T) We then observe new batch of examples Published as a conference paper at ICLR 2022
6) Winghed Average
b) Dene Somples
Novel Surgles
Losk Purtebi
Deupuee une Ioc.
Silepe 2og
Chon (312))
Vage
12
Seauri Siapue Dag.
Session
Socico
Figure 2: Multi-Session FSCII accuracy (%) results on miniImageeet. In the first session O. there total of 60 classes (base). Every session following the first one introduces novel classes with 5 labeled samples from each. Each session provides accuracy over all classes that were seen thus far. Weighted average is the weighted combination of novel and base accuracies with respect to the number of classes in each category. Error bars are standard deviation (95% CI). In accordance with Chen & Lee (2021) we preserve only one sample per class from previous classes and append them the support set during fine-tuning (+M variant). Regularization based approaches i.c. subspace regularization, semantic subspace regularization and linear mapping consistently outperform previous benehmark and fine fine-tuning in weighted average (a).
added and the classifier must be repeatedly updated, and a single-session setup (T' = 1) in which new classes arrive only once. As our optimizer, we use stochastic gradient descent for feature extractor training and gradient descent in incremental sessions to train all models. Details about experiment setups and results are discussed below. Additional details may be found in the appendix.
5.1 SESSION
Table Multi Multi-Session FSCIL weighted average of accuracy (%) results on minilmageNet using an identical setup to Fig. with memory distinction. We report the average results over 10 random splits of the data for incremental sessions 1, 2 and 8 + indicates 1 sample per class is kept (or not) in the memory to further regularize forgetting. Our Fine-tuning baseline already superior to previous results. In both memory settings, our regularizers substantially outperform respective benchmarks for all 1-8 sessions. 'Results only estimates from the plot in the respective work. Bold indicates the highest.
((t) and produce a new classifier de- fined by Mecoss) Finally, we evaluate the classifier according to top-/ accu- racy in base and novel samples as well as their weighted average (Tao et al., 2020b; Chen Lee, 2021). We use the minilmageNet dataset (Vinyals et al.,
Session Model
-M
+M
-M
+M
- M
+M
al. (2020h) Chen & Lee (2021)
50.1
45.2
24.4
59.9
55.9
41.8
Fine-tuning
61.8
67.7
49.9
62.9
26.5
Subspace Reg.
74.0
74.7
68.9
69.4
50.6
51.7
2016; Russakovsky et al., 2015) for our multi-session evaluation. minillmaeeeet contains 100 classes with 600 samples per class.
+ +anggunge
Cheraghian et al. (2021)*
58.0
53.0
39.0
Lincar Mapping
72.6
73.2
67.1
68.0
46.9
50.0
Semantic Suhspace Reg.
73.8
73.9
68.4
69.0
47.6
49.7
Joint-training
73.1
67.5
47.3
Feature extractor training In session L O. we randomly select 60 classes as base classes ((C(0) 60) and use the remaining 40 classes novel classes. Reported results are averaged across 10 random splits of the data (Fig. 2). We use a ResNet-18 model that is identical to the one described in Tian et al. (2020). Following Tao et al. (2020b), class to train our feature extractor and 100 for testing.
500 labeled samples per base
Incremental evaluation Again following Tao et al. (2020b), we evaluate for a total of 8 incre- mental sessions S t 8 for minilmageNet. In each session, for 5(*), we sample 5 novel classes for training and 5 samples from each class. Hence, at the last session t 8, evaluation involves the entire set of 100 minilmageNet classes. We use GloVe embeddings (Pennington et al., 2014) for label embeddings in Eq. 7.
7

---

Published as a conference paper at ICLR 2022
Joint-training In order to assess how our approaches compare to setting where memory is abun- dant and retraining is allowed, we train a separate model in each session from scratch. These models are trained the combination of all training examples including those from the base classes ce S(t') using the loss described in Eq. 2. Note that in this setup the memory requirement is substantially large (500 examples per each class in CXO,, and 5 for each of the subsequent classes ((s)'st) and the computation is more expensive.
Results Fig. and Table show the results of multi-session experiments with and without mem- ory. Session 0 indicates base class accuracy after feature extractor training. We compare subspace and sagguage-guiedd regularization (linear mapping and semantic subspace reg.) to simple fine- tuning (a surprisingly strong baseline) and joint-training. We also compare our results three recent benchmarks: Tao et al. (2020b), Chen & Lee (2021) and Cheraghian et al. (2021).*
When models are evaluated on combined base and novel accuracy, subspace regularization outper- forms previous approaches (by 23% (-M) and 10% (+M) at session 8); when semantic information about labels available, linear mapping and semantic subspace regularization outperform Cher- aghian et al. (2021) (Table 1). Evaluating only base sample accuracies (Fig. 2b), semantic subspace reg. outperforms others; compared to regularization based approaches fine-tuning is subject to catas- trophic forgetting. Rold is still useful in regulating forgetting (Table 3 in appendix). The method of Chen & Lee (2021) follows a similar trajectory to our regularizers, but at a much lower accu- racy (Fig. 2a). In Fig. 2c, a high degree of forgetting in base classes with fine-tuning allows higher accuracy for novel classes thasuh not enough to improve average performance (Fig. 2a). By con- trast, subspace regularizers provide good balance between plasticity and stability (Mermillod et al., 2013). In Table I, storing as few as single example per an old class substantially helps to reduce forgetting. Results from linear mapping and semantic subspace regularization are close, with mantic subspace regularization performing roughly 1% better on average. The two approaches offer different trade-offs between base and novel accuracies: the latter is more competitive for base classes and vice-versa. A similar argument can be made when comparing the subspace regularizer with the semantic counterpart where the latter more efficient base classes than the former. Last but not least, keeping as little one example per old class in the memory along with our subspace regular- izers sufficient to surpass joint-training-2 procedure which is extremely inefficient in terms of both space and computation.
5.2 SINGLE SESSION
In this section we describe the experiment setup for the single-session evaluation, (T' 1), and compare our approach to state-of-the-art XtanNet (Yoon et al., 2020), as well as Ren et al. (2019), Gidaris & Komodakis (2018) and Qi et al. (2018). We evaluate our models on I-shot and 5-shot settings.
minilmage Net and liered For minilmageNet single-session experiments, we follow the the splits provided by Yoon et al. (2020). Out of 100, 64 classes used in session 0, 20 in session & = 1 and the remaining for development. Following Yoon et al. (2020), we use ResNet-12 (a smaller version of the model described in Section 1). tieredImageNet (Ren et al., 2018) contains a total of 608 classes out of which 351 are used in session t = 0 and 160 are reserved for t = 1. The remaining 97 are used for development. While previous work (Ren et al., 2019; Yoon et al., 2020) separate 151 classes out of the 351 for meta training, we pool all 351 for feature extractor training.
*Chen & Lee (2021) and Cheraghian et al. (2021) do not provide codebase and Tao et al. (2020b) does not provide an implementation for the main TOPIC algorithm in their released code. Therefore, we report published results rather than a reproduction. This comparison is inexact: our feature extractor performs substantially better than Tao et al. (2020b) and Chen & Lee (2021). Despite extensive experiments (see appendix) on various versions of ResNet-18 18 (He et al., 2016), we were unable to identify a training procedure that reproduced the reported accuracy for session 0: all model variants investigated achieved 80%+ validation accuracy.
Unlliee in the preceding section, we were able to successfully reproduce the XtarNet model. Our version gives better results on the minilmageNet dataset but worse results on the fieredImageNet datasets; for fair- ness, we thus report results for our version of XtarNet minimageNet and previously reported numbers on rieredImageNet. For other models, we show accuracies reported in previous work.
8

---

Published as a conference paper at ICLR 2022
Table 2: minilmageNet 64+5-way and rieredllageNet 200+5-way single-session results. We follow previous work in reporting the average of accuracies of base and novel samples over all classes rather than weighted average. In addition accuracy, individual accuracies and joint accuracies of both base and novel samples averaged. Lower values of A are better. Bold numbers are not significantly different from the best result in cach column under paired t-test (p < 0.05 after Bonferroni correction). All results are averaged across 2000 runs.
report a quantity labeled A by Ren et al. (2019), which is the gap between
mineInageNet
HeredImaagNeet
Model
I-shot
S-sbot
1-shot
S-shot
Acc.
Acc.
A
Acc.
Acc.
4
Imprinted Networks (1) et 2018)
41.34 10.50
795.79%
46. in
-25 25%
40.83 20.45
-22.29%
53.87 abcc
17.18%
LwoF (Gidaris & Komodakis, 2018)
49.65
1e.6
-14.47%
59.66
10:55
-12.35%
53.42
10.56
-9.59%
63:22 10:50
-7.27%
Atiention Altractor Networks (Ron all, 2019)
54.95
-11.84%
63.04 40.30
-10.66%
56.11
-6.11%
65. 52
203t
-4,8%
XtarNet (Yoon 2020)
50.17
-13.2%%
69.51
-9.70%
-1.85%
09:58
-1.79%
Fine tunittunng
58.56 =0..5
-12.16%
66.54
+0.35
-13.77%
64.42
:0.35
-7.23%
72.59 20:54
-688%
Subspace Regularization
58.38 #9.92
-12.30%
68.88
#0.52
-10.74%
64.39
20.35
-7.23%
73,03
#0.54
-6.16%
longuage
Mapping
2:0.33
-12.83%
69.68
10.31
-10.40%
20.35
-7.31%
73.10
20:31
-6.16%
Semontic Subsquee Reg. (wl description)
$9.09
2:0:0
-12.385
68.46 in
-11. 11.700
64.49 10.25
-7.14%
72.94 10:01
-6.29%
Semantic Subsqace Reg. (wi label)
58,70 10..2
-12.24%
69.75 10.32
-10.48%
64.75 10:38
-7.22%%
73.51 1,:30
-6.08%
We train the same ResNet-18 described in Section 5.1. Additional details regarding learning rate scheduling, optimizer parameters and other training configurations may be found in the appendix.
Incremental evaluation We follow Yoon et al. (2020) for evaluation. Yoon al. independently sample 2000 D(1) incremental datasets ("episodes") from the testing classes ((L), They report aver- age accuracies over all episodes with 95% confidence intervals. At every episode, Q(1) is resampled from both base and novel classes, ()0) and (()), with equal probability for both minilmageNet and rigred We again fine-tune the weights until convergence. We do not reserve samples from base classes, thus the only training samples during incremental evaluation is from the novel classes C()), We use the same resources for label embeddings and Sentence-BERG embeddings (Reimers & Gurevych, 2019) for descriptions which are retrieved from WordNet (Miller, 1995).
Results We report aggregate results for 1-shot and 5-shots settings of minilmageNee and tieredImageNet (Table 2). Compared to previous work specialized for the single- session setup with- out a straightforward way to expand into multi-session (Ren et al., 2019; Yoon et al., 2020), even our simple fine-tuning baseline perform well on both datasets--ouppeffrmmigg the previous state- of-the-art in three out of four settings in Table 2. Addition of subspace and semantic regularization improves performance overall but tieredImageNet 1-shot setting. Semantic subspace regularizers match or outperform linear label mapping. Subspace regularization outperforms fine-tuning in 5- shot settings and matches it in 1 1-shot. In addition accuracy, we report quantity labeled A by Ren et al. (2019). A serves as a measure of catastrophic forgetting, with the caveat that it can be minimized by a model that achieves classification accuracy of on both base and novel classes. We find that our approaches result in approximately the same A in minilmageNet and worse in fieredImageNet than previous work.
6 ANALYSIS AND LIMITATIONS
What does regularization actually do? Fine-tuning results in prediction biased towards the most recently learned classes (top of Fig. 3) when no subspace regularization is imposed. Our experiments show that preserving the base weights while regularizing novel weights gives significant improve- ments over ordinary fine-tuning (bottom of Fig. 3)-resulting a fairer prediction over all classes and reducing catastrophic forgetting. In Table 1, Semantic Subspace Reg. results in 73.8% and 47.6% accuracies in the 1a and 8th sessions whereas, fine-tuning results in 61.8% and 26.5%, respectively, even without any memory- mummoy--suggesting that regularization ensures better retention of accuracy. While the trade-off between accuracies of base and novel classes is inevitable due to the nature of the classification problem, the proposed regularizers provide good balance between the two.
What are the limitations of the proposed regularization scheme? Our approach targets only errors that originate in the final layer of the model- while a convolutional feature extractor is the parameters of this feature extractor are fixed, and we have focused FSCIL as linear clas-
used,

---

Published as a conference paper at ICLR 2022
032899900000 www.
Muidd
Makia felepan Incleinkke
Maiid wendeeddge &apper Bepd sindee
Maild Scheom
Predacie
Pradihes
Figure 3: Simple fine-tuning (top) vs. subspace regularization (bottom) predictions without memory across the the x- and y-axes, we present predictions and gold labels
first four incremental sessions of minilmageNet. ranging from 0 to 79 where the first 60 are base classes. The number of classes grows by 5 every session starting from 60 up 80. Brighter colors indicate more frequent predictions. Note that simple fune-tuning entails bias towards the most recently learned classes (top row) whereas addition of subspace regularization novel weights remedies the aforementioned bias; resulting a fairer prediction performance for classes.
sification problem. Future work might extend these approaches to incorporate fine-tuning of the (nonlinear) feature extractor itself while preserving performance all classes in the longer term.
7 CONCLUSIONS
We have described a family of regularization-basee approaches to few-shot class-incremental learn- ing, drawing connections between incremental learning and the general multi-task and zero-shot learning literature. The proposed regularizers extremely theple-thyy involve only extra hyperparameter, require no additional training steps or model parameters, and are easy to understand and implement. Despite this simplicity, our approach enables ordinary classification architectures achieve state-of-the-arr results on the doubly challenging few-shot incremental image classification across multiple datasets and problem formulations.
few-ssssooo
ACKNOWLEDGMENTS
This work is supported in part by the U.S. NSF grant 1838193 and DARPA HR001 1880044 (the LwLL program). The U.S. Government is authorized to reproduce and distribute reprints for Gov- ernmental purposes. The views and conclusions contained this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA and the U.S. Government. Computing resources were provided by the MIT Supereloud (Reuther et al., 2018).
REFERENCES
Arvind Agarwal, Samuel Gerber, and Hal Daume. Learning multiple tasks using manifold regular- ization. In J. Lafferty, C. Williams, Shawe-Taylor, Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23, Curran Associates, Inc., 2010. 2, 3,
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute- based classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 819-826, 2013. 4
10

---

Published as a conference paper at ICLR 2022
D Anderson and K Burnham. Model selection and multi-model inference. Second. NY: Springer- Verlag, 63(2020):10, 2004.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Multi-task feature learning. In B. Schölkopf. J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007a. URL https://proceed: ngs www.gnnuuuueeee neurips. ct. paper/
3,5
Andreas Argyriou, Charles A Micchelli, Massimiliano Pontil, and Yiming Ying. spectral regu- larization framework for multi-task structure learning. In NIPS, volume 1290, pp. 1296. Citeseer, 2007b.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:/907.0289)) 2019. 5
Aviad Barzilai and Koby Crammer. Convex multi-task learning by clustering. In Artificial Intelli- gence and Statistics, pp. 65-73. PMLR, 2015.
Emily M Bender, Timnit Gebru, Angelina McMillan-Manl and Shmargaret Shmitchell. On dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, 2021.
Ming-Wey Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. Importance of semantic repre- sentation: Dataless classification. In AAAI, volume 2. pp. $30-835, 2008. 4
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning, arXiv preprint 1rXiv:1902.10486. 2019.
Kuilin Chen and Chi-Guhn Lee. Incremental few-shot learning via vector quantization in deep embedded space. In International Conference on Learning Representations, 2021. 3, 4, 5, 7, 8, 15
Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Semantic-aware knowledge distillation for few-shot class-incremental learning. arXiv preprint arXiv:2 10.. 103,0499,, 04059, 2021. 3, 4, 6, 7, 8, 15
Debasmit Das and CS George Lee. Zero-shot image recognition using relational matching, adap- tation and calibration. In 20/9 International Joint Conference on Neural Networks (IJCNN, pp- 1-8. IEEE, 2019. 6
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Arnout Devos and Matthias Grossglauser. Regression networks for meta-learning few-shot classifi- cation. arXiv preprint arXiv:1905. 1936:1995..3663, 2019. 5
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017. 3
Andrea Frome, Greg S Corrado, Jon Shlens. Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In C. J. C. Burges, L. Bot- tou, M. Welling, Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018. 3, 4, 8,9
Ian Goodfellow. Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient- based neural networks. arXiv preprint 112 2013. 2
11

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. 7, 8, 15, 16
Laurent Jacob, Francis Bach, and Jean- Yean-Philippe Vert. Clustered multi-task learning: A convex formulation. In Advances in Neural Information Processing Systems, 2008.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinsker et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017. 2
Nikolaus Kriegeskorte, Maricke Mur, Douglas Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky, Keiji Tanaka, and Peter A Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6): 11111126--1444 2008.
Ilja Kuzborskij, Francesco Orabona, and Barbara Caputo. From n to 1: Multiclass transfer incremental learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3358-3365, 2013. 3,5
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, volume pp. 3, 2008. 4
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Diaz Rodriguez. Continual learning for robotics: Definition, framework, learning strategies, op- portunities and challenges. Information Fusion, 58, 12 2019. doi: i0.f016/).nffuss22109999 inf.u.2019..1.00044
Mare Masana, Xialei Liu, Bartlomiej Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation. arXiv preprint arXiv:2010.15277, 2020. 1
Patrick McClure, Charles Y. Zheng. J. Kaczmarzyk, John Rogers-Lee, S. Ghosh, D. Nielson, P. Ban- dettini, and Francisco Pereira. Distributed weight consolidation: brain segmentation case study. In NeurlPS, 2018.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: Inves- tigating the continuum from catastrophic forgetting age-limited learning effects. Frontiers in psychology, 4:504, 2013. 8
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv: 2017. 15
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014. 6,
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, and Xi-Zhao Wang. A review of generalized zero-shot learning methods. arXiv preprint arXiv:201 1.066411, 2020. 4, 6
Hang Qi, Matthew Brown, and David Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5082-583288
2018. 3, 4, 8,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 103. 00000, 2021. 4
12

---

Published as a conference paper at ICLR 2022
S Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. icarl: Incremental classifier and repre- sentation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 49-58, 2016.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1908. 10084, 2019. 9
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. arXiv preprint arviv: 180360067, 2018. 8, 14, 16
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental few-shot learning with attention attractor networks. In Advances in Neural Information Processing Systems, 2019. 2, 3, 4, 8, 9, 15, 16
Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, et al. Interactive supercom- puting on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2018. 10
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learn- ing. In International conference on machine learning. pp. 2152 2161. PMLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 15(3):211-252 2015. doi: 10.1007/s1 (0777/11263-011 1263-015-0816-1 7,
Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7):
1757-1772, 2012.
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata, General- ized zero-shot learning via aligned variational autoencoders. Proceedings of the IEEF/VVF Conference Computer Vision and Pattern Recognition, 2019. 3, 5,
Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex M Bronstein. Baby steps towards few-shot learning with multiple semantics. arXiv preprint arXiv: /ativ:9906.. 199669990090 2019. 4
Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4136-4145, 2020. 5
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D Manning, and An- drew Y Ng. Zero-shot learning through cross- transfer. In Advances in Neural Information Processing Systems, 2013. 6
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. Topology Topology-preserving class- innremnnctmm learning. In European Conference on Computer Vision, pp. 254-270. Springer, 2020a. 3
13

---

Published as a conference paper at ICLR 2022
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang. Songlin Dong, Xing Wei, and Yihong Gong. Few- shot class- inaassincremetaall learning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22183-12192 2020b. 2,7,4778
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: good embedding is all you need? arXiv preprint arXiv:2003. 1 110220003.119,, 2020. 3, 15, 16
Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50. Siam, 1997. 16
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. arXiv preprint arXiv: 04080, 2016. 3, 7, 14
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972. 3
Kun Yan, Zied Bouraoui, Ping Wang, Shoaib Jameel, and Steven Schockaert. Aligning visual pro- totypes with bert embeddings for few-shot learning. In ICMR, 2021.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. In International Conference Machine Learning, pp- 7115-7123. PMLR, 2019. 3
Sung Whan Yoon, Do-Yeor Yeon Kim, Jun Seo, and Jackyun Moon. Xtamet: Learning to extract task- adaptive representation for incremental few-shot learning. International Conference on Ma- chine Learning, pp. 10852-10860. PMLR, 2020. 2, 3, 4, 8, 9, 15, 16, 17
Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng. Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12455-12464, 2021.
Kai Zhu, Yang Cao, Wei Zhai, Cheng, and Zheng-Jun Zha. Self-promoted prototype refine- ment for few-shot class-incrementl learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6801-6810, 2021.
A CODE AND DATASETS
Code is released under https: /ttps:///gttbbb com/ www//eeyaaayurrkk/eaaaaaaaaaaaae wttp://ywurrekeuupeee-e-eeee reg. We use minilmageNet (Vinyals et al., 2016) and tieredImageNet (Ren et al., 2018) datasets both are sub- sets of ImageNet dataset (Russakovsky et al., 2015). Use of terms and licenses are available through the respective sources.
B ANALYSIS OF CATASTROPHIC FORGETTING
In Table 3, we demonstrate the effectiveness of the regularization term Rata in mitigating catas- trophic forgetting.
Table 3: minilmageNet weighted average results across multiple sessions showcasing the usefulness of Rotd in reducing catastrophic forgetting across multiple sessions for -M setting. Higher accuracies are highlighted and results are averages over 10 random splits. Rota useful in combination with Subspace Reg. for all sessions while is more helpful in the long-run for Fine-tuning. Note that Subspace Reg. consistently outperforms Fine-tuning regardless the use of Rold
Model
0
2
4
6
8
Subspace Reg.
80.37
72.99
67.21
62.40
58.46
54.28
50.33
48,03
45.06
Subspace Reg. Roid
80.37
71.96
65.49
60.43
55.56
50.35
45.66
42.91
39.54
Fine-tuning
80.37
61.77
49.93
40.45
34.04
31.63
28.43
27.91
26.54
Fine-tuning no Roid
80.37
62.39
53.89
46.56
39.73
32.92
27.00
23.95
20.39
14

---

Published as a conference paper at ICLR 2022
Table minilmageNet +M results across multiple sessions in tabular form. Initial number of base classes is 60 and new classes are introduced at every session. Results are on the test set that grows with the increasing number of classes. the last session we evaluate over all 100 classes. Joint-training involves keeping all training data seen thus far in the memory and retrains feature extractor from scratch on the entirety base and novel classes.
Session
2
3
4
6
7
8
Model
Chen & Lee (2021)
64.77
59.87
55.93
52.62
49.88
47.55
44.83
43.14
41.84
Fine-tuning
80.37
67.69
62.91
59.52
56.87
54.37
51.92
50.26
48.13
Subspace Reg.
80.37
74.68
69.39
65.51
62.38
59.03
56.36
53.95
51.73
+language
Linear Mapping
80.37
73.24
67.96
64.50
61.28
57.68
54.64
52.25
50.00
Semantic Subspace Reg.
80.37
73.92
69.00
65.10
61.73
58.12
54.98
52.21
49.65
Joint-training
80.37
73.06
67.52
63.21
59.19
55.98
52.11
49.59
47.31
Table minilmageNet -M results across multiple sessions in tabular form. Initial number of base classes is 60 and 5 new classes are introduced at every session. Results are on the test set that grows with the increasing number of classes. The last session evaluated over all 100 classes. *Note that the entries for Cheraghian et al. (2021) only rough estimates from the visual plot provided in their published work.
Model
4
5
6
7
8
Tao al. (2020)
61.31
50.09
45.17
41.16
37.48
35.52
32.19
29.46
24.42
Fine-tuning
80.37
61.77
49.93
40.45
34.04
31.63
28.43
27.91
26.54
Subspace Regularization
80.37
74.01
68.92
64.37
61.35
58.22
54.99
53.15
50.58
+language
Cheraghian al. (2021)*
62.00
58.00
52.00
49.00
48.00
45.00
42.00
40.00
39.00
Lincar Mapping
80.37
72.65
67.11
63.47
59.82
55.44
51.42
49.64
46,90
Semantic Subspace Reg.
80.37
73.76
68.36
64.07
60.36
56.27
53.10
50.45
47.55
C RESULTS IN TABULAR FORM
In Table and Table 5, we present the multi-session results the main paper in the tabular form.
D DETAILS OF FEATURE EXTRACTOR TRAINING
We use the exact ResNet described in Tian et al. (2020), the differences compared the standard ResNet (He et al., 2016): (1) Each block (collection of convolutional blocks) is composed of three convolutional layers instead of two. (2) Number of blocks for ResNet- 12 is 4 instead of 6 of the standard version, thus the total number of convolutional layers are the same. (3) Filter sizes are 164,160,320,640 rather than 222..522,,,))), though the total number of filters is comparable since Tian et al. (2020) has less blocks. (4) There is Dropblock at the end of the last blocks.
Tian al. (2020) provides full visualization in Appendix and their code repository is easy to browse which we base our own codebase. We observe that the previous work oftentimes use their slightly modified version of the standard ResNet. Ren et al. (2019) uses the ResNet- 10 (Mishra et al., 2017) and ResNet-18 18 for for minilmageNet and tiered/mageeNet, respectively. XtarNet( Yoon et al., 2020) is originally based on a slightly modified version of ResNet-12 12 and ResNet-13 18 which replaced with version, improving their results for minilmageNett but not in tieredImageNet, thus we report improved results for minImmaeeeet and their results for tieredImageNot in the main paper.
https://gii thub. httpttt....////////tt////////eee
15

---

Published as a conference paper at ICLR 2022
Table minimaaeNeet validation accuracy with two ResNet-18 architectures with slight differences as listed in Appendix D. Overall performances are comparable.
Sced
Seod 2
Sced
Sood 4
Seed
Soed 6
Scod 7
Soed S.
Seed 9
Seed 10
Mccan
Our RetNet- (Tian 2020)
84.833
79.167
83.200
81,300
81.267
78.933
82.033
82,067
10000
82.367
81,6967
Standard ResNet-18 18 (Ho al, 2016)
83.333
80.100
83.867
81.333
80967
79.100
81.833
82.500
81.167
81.567
81,5767
D.1 DEFAULT SETTINGS
Unless otherwise indicated we use the following default settings of Tian al. (2020) in our feature extractor training. We use SGD optimizer with learning starting at 0.05 with decays by 0.1 at epochs 60 and 80, We train for total of 100 epochs. Weight decay 5e-4, momentum is 0.9 and batch size is 64. As per transformations on training images, we use random crop of 84x84 with padding We also use color jitter (brightness=0.4, contrast=0.4, saturation=0.44 and horizontal flips. For each run, we sample 1000 images from base classes and 25 images from each of novel classes. Our classifier does not have bias.
D.2 MULTI-SESSION MiniIMAGENET
We re-sa the set of base classes (60 classes) 10 times across different seeds and train ten ResNet- 18 models. Each class has 500 training images. follow default settings for training.
D.3 MULTI-SESSION COMPARISON TO STANDARD RESNET-18
In Table 6 we provide validation set results for two types of ResNet-188 18's: Tian et al. (2020) and He al. (2016) across different seeds. Results show that use of Tian et al. (2020) does not incur unfair advantage over those who used He et al. (2016).
D.4 SINGLE-SESSION MiRÍTMAGENET
We follow the default hyperparameters parameters Appendix D.1 except that for training, validation and testing we use the exact splits provided by Ren et al. (2019) also used by Yoon et al. (2020). There 64 base, 16 validation and 20 testing classes provided (totaling 100). Training data consists of 600 images per base class. Dataset statistics are delineated in the Appendix of Ren et al. (2019) and downloadable splits are available here, courtesy Ren et al. (2019).
D.5 SINGLE-SESSION tiered|MAGENET
tieredImageNet is first introduced by Ren et al. (2018). Same as above, we use the default parameters except that we train for total of 60 epochs decaying the initial learning rate of 0.05 by 0.1 at epochs 30 and 45. Again, we use the same data previous work available at the same link above. tigred is split into 351, 97 and 160 classes. Past work that use meta-learning Ren et al. (2019); Yoon et al. (2020) split 351 training classes into further 200 and 151 clases where the latter is used for meta learning. We pool all 351 for feature extractor training. the end of feature extractor training, only keep the classifier weights for the first 200 classes to adhere to the evaluation scheme of 200+5 classes as past work.
E DETAILS OF INCREMENTAL EVALUATION
E. QR DECOMPOSITION FOR SUBSPACE REGULARIZATION
To compute the orthogonal basis Pero; for the subspace spanned by base classifier weights No:,) use QR decomposition
CimppoiitioorTreftteeen Trefethen Bau III, 1997):
R
[Pote
Q']
l(xo)
(11)
16

---

Published as a conference paper at ICLR 2022
E.2 SESSION
For testing, we sample 1000 images from base classes and 25 images from each of novel classes. Testing images from given class stay the same across sessions. Harmonic mean results take into account the ratio of base classes to novel classes a given session. this setting, there is no explicit development set (with disjoint classes than train and test) defined by previous work thus we use the first incremental learning session (containing 5 novel classes) as our development set.
Default settings We use the same transformations as in Appendix D.1 on the training images. We stop fine-tuning when loss does not change more than 0.0001 for at least 10 epochs. We use SGD optimizer. We repeat the experiments 10 times and report the average accuracy with standard deviation (95% confidence interval) the paper.
Simple Fine-tuning We use learning rate of 0.002 and do not use learning decay. Weight-decay is set at 5e-3. In order limit the change in weights, we different I's for base and previously learned novel classes, where the former is 0.2 and the latter 0.1. We rely on the default settings otherwise.
Subspace Regularization Different from simple fine-tuning we use a weight decay 5e-4. There an additional parameter called y in this setting controlling the degree of pulling of novel weights toward the subspace which we set to 1.0.
Semantic Subspace Regularization Different than simple subspace regularization, there is a tem- perature parameter used in the Softmax operation used in computation of 1, 's which we set 3.0.
Linear mapping regularization Same parameters as in subspace regularization are used except y 0.1. We formulate L linear layer with bias and we gradient descent to train the parameters.
E.3 SINGLE-SESSION
In our 1-shot experiments unless fine-tuning converges by then, we stop at the maximum number of epochs 1000. We sample 2000 episodes which includes 5 novels classes and 1-5 samples from each and report average accuracy. For testing, base and novel samples have equal weight in average per previous work (Yoon al., 2020). SGD optimizer used. For description similarity Soutence-BERTT stab-ttstooeooo roberta-largee......
minilmageNet Settings We use the same set of transformations on the training images de- scribed in Appendix D.1. We first describe details 1-shot setting. In 1-shot experiments we set the maximum epochs to 1000. For simple fine-tuning cay of 5e-3. In Semantic Subspace Reg., we set temperature to 1.5. Both in Semantic Subspace Reg. and linear mapping 7 0.005 and weight-decay is 5e-4. In subspace regularization, 7 0.005 and weight-decay is set to 5e-5. Description similarity follows the same setup as Semantic Subspace Reg..
use learning rate of 0.003, and weight de-
In 5-shot setting, we set 3 = 0.03 weight-decay to 5e-3 and learning rate to 0.002. For subspace regularization, Semantic Subspace Reg. and linear mapping we use 7 0.03 and for description similarity we use = 0.01.
Imaged Settings In 1-shot setting, fine-tuning learning rate of 0.003. Semantic Sub- space Reg. has learning rate of 0.005, weight-decay 5e-3 and = 0.005. Subspace reg. and linear mapping use 0.001.
In 5-shot setting, for simple fine-tuning we set = 0.001, -ecght-decay=Se-3, 8 = 0.3. For Seman- tic Subspace Reg. 7 0.05 and = 0.2 while others have 7 0.03.
17

---

Published as a conference paper at ICLR 2022
Trajectory of Classifier Weights with Subspace Regularization using PCA
130
0.75
aikte SwSlT
0.30
025
hureel
feat2
0.00
achine
adit
abolae
Afica
-025
Hac Woagh
Nivel C' lio Weight
eodkal Staker
cldr Prnneei
Projectoon
Hisotaning trouc
alteetool Aeh
0.75
Jenotzn
(90
SeaneticlSah
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.35
1.00
1.25
feat
Figure 4: Clasifier weight space when subspace regularization is applied for minlmageeeet single-session 5-shot setting, First two principal components shown according to PCA. Red labels indicate novel classes while the black indicates base. The green crosses indicate the projection of the respective novel class weight to the base subspace. Note that unlike label/description similarity and linear mapping, subspace target is dynamic: it changes according its corresponding novel weights and vice versa.
Trajectory of Classifier Weights with Label Similarity using PCA
1.25
0.75
0.50
feat2
0.25
Caippry
Towr Class Wright
thaler
Navd Clas Weight
Jussco
Scauatic Taget
Pimookkmme tpock
Whorh frace
200
-0.50
Senaurio Stbsqqee Tage
-0.75
-0.5
feat
Figure 5: Clasifier weight space when Semantic Subspace Reg. is applied for minilmageNet session 5-shot setting, First two principal components are shown according to PCA. Red labels indicate novel classes while the black indicates base. The green crosses indicate the semantic target of the respective novel class. Note that semantic targets static: they don't change during fine-tuning. Notably, the semantic target for theater curtain falls closely to the class representation of the base class stage, dragging novel weight for theater curtain towards there. Same dynamic is visible for novel class crate and base barrel.
18

---

Published as a conference paper at ICLR 2022
F VISUALIZATIONS
In Fig. 4 and Fig. 5 we depict principal components of classifier weights as well as semantic or subspace targets for novel weights.
G COMPUTE
We single 32 GB V100 NVIDIA GPU for all our experiments.
19


Reviews:
[{"review_id": "ceMtY4fkQ9O", "reviewer": "Reviewer_oQAd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n=======\n* The method is technically sound, simple, and it improves the performance on multi-session FSCIL.\n* The text is well-written.\n* The authors provide enough details to reproduce their results and they promise to release the code public.\n\nWeaknesses\n==========\n* The improvement on the single-session setup is not clear. However, I value that the authors were honest about it. In fact, in this setup simple fine-tuning seems to be very effective. \n* There are some relevant works that were not cited such as [A,B,C,D,E].\n* Some previous work such as [C, Cheraghian et al.] evaluated their models on CUB and CIFAR. Could you clarify why you did not use the same evaluation as them?\n* The main contribution (aligning new weights with previously learned ones) is not new (the authors already cite previous work), but the way they do it seems new to me. It would be good if you clarified what exactly in your work is proposed by you and what is borrowed from previous literature.\n\n[A] Simon, Christian, et al. \"Adaptive subspaces for few-shot learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[B] Tao, Xiaoyu, et al. \"Topology-preserving class-incremental learning.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n[C] Tao, Xiaoyu, et al. \"Few-shot class-incremental learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[D] (Arxiv, Optional) Yan, Kun, et al. \"Aligning Visual Prototypes with BERT Embeddings for Few-Shot Learning.\" arXiv preprint arXiv:2105.10195 (2021).\n\n[E] Xing, Chen, et al. \"Adaptive cross-modal few-shot learning.\" Advances in Neural Information Processing Systems 32 (2019): 4847-4857.\n", "summary_of_the_paper": "The authors tackle the problem of few-shot class incremental learning (FSCIL). In this setting, a learner is first trained on a set of base classes for which plenty of data is available. Then, it must learn from small sets of novel classes while retaining good performance in the whole sequence. In order to prevent the classifier from overfitting new classes, they authors propose to regularize novel class weights to keep them in the subspace of base classes, thus making the classifier rely on known features rather than new features that are probably spurious. In addition, they also show that it is possible to align new classes with base classes based on their similarity in a learned semantic space such as Glove. Experimental results on miniImagenet and tiered-Imagenet show that their method obtains better performance than previous state of the art.", "summary_of_the_review": "The proposed method is simple and effective and the paper is well-written. On the other hand, the main contribution is small and I have some questions about the experimental setup and relationship with previous related work (see weaknesses). Overall, I think that the simplicity and effectiveness outweigh the negative aspects and **I would be inclined towards accept if the authors resolve my concerns and those of the other reviewers.**", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ScukpCGYSr1", "reviewer": "Reviewer_EoLF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- This paper attacks a very important and practical challenge. and the motivation of the proposed method is well presented.\n\n- The idea of regularizing the novel weight vector using the base ones to prevent the network from capturing spurious correlations seems interesting. \n\n**Questions**\n\n- I'm a bit confused by the regularization term imposed on the old class $R_{old}$. In the following session, since the feature extractor is fixed and I assume there are no available samples from the base classes, why the weight vectors of base classes are not fixed? And why this regularization term is important?\n\n- While better quantitative results are reported in the experiment section, it remains unclear to me why restricting the novel feature vectors to be in the subspace of base classes can improve the overall performance. This motivation opposes some prior work on subspace-motivated continual learning framework, e.g., [1]. My main concern is why this regularization will not cause interference to base and other incremental classes, as the classification boundaries are all restricted to the subspace defined by the base classes, and I do not see any terms used to encourage a robust subspace obtained by the base classes only.\n\n- Some visualizations, e.g. to the weight vectors are expected to further support the effectiveness of the proposed regularization terms, especially the semantic subspace regularization. \n\n- Missing discussions on some recent work, e.g., [2,3].\n\n**Minor**\n\nThis paper demands additional rounds of proofreading. \n\nNon-exclusive examples:\n\nPage 5 line 7 double 'append'\n\nSection 2 paragraph 3 'learners.Existing' missing space\n\nSection 3 paragraph 4 'Qi et al. (2018) initialize novel' -> 'initializes'\n\nSection D.1 line 2 'with learning starting' -> 'with a learning rate starting'\n\n[1] Continual Learning in Low-rank Orthogonal Subspaces, NeurIPS, 2020\n[2] Few-Shot Incremental Learning with Continually Evolved Classifiers, CVPR, 2021\n[3] Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning, CVPR, 2021", "summary_of_the_paper": "This paper introduces a new method of few-shot incremental learning with its regularization-based method motived by a subspace view.\nRegularization terms to the linear layers of incremental classes are proposed to encourage the novel weight vector to be in the subspace spanned by those of the base classes, and semantic information from the text domain can further be leveraged to guide the interpolation within the base-class subspace. ", "summary_of_the_review": "As a simple idea, more discussions and empirical supports are expected to make it more convincing. I'll raise my score if all the concerns are effectively addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EFW_AmYr8Hd", "reviewer": "Reviewer_Vrap", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper presents a simple model and strong experimental results.\n- The experimentation seems solid. It has evaluated on single and multi-session setups, with or without a memory buffer.\n\nWeaknesses:\n- It is unclear where the most benefit of the proposed model comes from. Based on Figure 2, it seems that Chen & Lee (2021) has a much worse checkpoint for base classes, and perhaps as a result the novel class learning is also not as good. It would be good if the authors can experiment both Tao et al. (2020) and Chen & Lee (2021) with the same pretrained checkpoint to showcase the real benefit in the few-shot learning phase.\n- The type of subspace regularization exists in non-incremental few-shot learning and I wonder if the authors can provide an empirical comparison on that. It is unclear why subspace regularization depends on the incremental aspect. See Simon et al. (2020), Devos & Grossglauser (2020), Yoon et al. (2019).\n- The semantic regularization technique provides a small gain on the multi-session benchmark and very marginal gain on the single session benchmark. This makes the contribution less focused and I suggest only include this in an additional study in the experiment section.\n- As pointed out by the authors, the current framework only aims at learning the linear classifier in a sequential manner, which limits the ways it can be applied.\n- If the number of samples is small (as is the case for few-shot learning sessions), then one could afford to do joint training at each session and store all the support examples. Such a joint training baseline should be provided for comparison (as was provided in Chen & Lee).\n\nReferences:\n- Simon et al. Adaptive Subspaces for Few-Shot Learning. CVPR 2020.\n- Devos and Grossglauser. Regression Networks for Meta-Learning Few-Shot Classification. ICML Workshop on Automated Machine Learning 2020.\n- Yoon et al. TAPNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learning. ICML 2019.\n", "summary_of_the_paper": "The paper proposes subspace regularization technique for incremental few-shot learning. The high-level idea is to find the basis vectors of the subspace spanned by the base classes, and then project the new classes into the subspace. The regularizer encourages the new class weights to be similar to the projected vectors with an L2 loss. Additionally, the paper also proposes to use semantic word vectors to perform regularization. Experimental results confirm that the proposed method is significantly better compared to Tao et al. (2020) and Chen & Lee (2021) either with an additional memory buffer or without, but it is unclear whether it is due to a better few-shot learning algorithm or simply a better pretrained checkpoint on the base classes. Moreover, the benefit of semantic regularization is not significant.", "summary_of_the_review": "In conclusion, my main concern is that this work may not have the same starting checkpoint when comparing to prior works and therefore it is hard to evaluate its empirical success in the few-shot learning phase. Another main concern is that the relation between subspace and incremental learning is unclear, since subspace regularization could also be applied to standard few-shot learning, and therefore comparisons to subspace methods on standard few-shot learning seems required. Other aspects of this work look solid to me. My initial evaluation of this paper is “weak reject”.\n\nAuthors' response on joint training addressed my concern. However, my other concern about comparison to Tao 2020 and Chen & Lee 2021 was not fully addressed. That being said, I understand there might be difficulty reproducing the numbers. Overall, I increased my rating from 5 to 6 to acknowledge the effort in responding my review comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9r4c5-BgF5T", "reviewer": "Reviewer_qioC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points**\nVery thorough experiments. I liked the visualizations in Figures 2 and 3. This gives a very good understanding of the extent of the model and how it performs. I also liked the idea of adding classes in each session. The method is novel and intuitive for solving this problem. Finally, the paper is easy to follow and well-structured.\n\n**Weak points**\nThis method does not work if the feature extractor cannot do a good job. An important part of training a new classifier is to learn the feature extractors. This approach misses this in both ways. First of all, when new classes are introduced, their features are not added to the feature extractor. Second, if the feature extractor does not focus on an important feature in a future class, the method is expected to fail.\n\n**Review** \nBased on strong points and weak points, I vote for acceptance of the paper. I think it would be great if the authors can explore the points I mentioned in the weak points, but still, the paper itself is an interesting idea with very good experiments. In addition the idea of using text for further improving the regularization kind of weighted my decision more toward acceptance. It shows that there are interesting ways to develop this method.", "summary_of_the_paper": "This paper leverages a simple but effective regularization technique for few-shot incremental class learning. The assumption is that at the first stage we have a lot of data and can train a good feature extractor and a network that can classify all the classes to a good degree. Given a few examples of a new unseen class, the authors enforce that the weights of the final layer of the classifier are close to the space that is spanned by the current classifier's last layer weights. They achieve state-of-the-art results on this very challenging task.\n \n", "summary_of_the_review": "The paper proposes a method for using regularization on the final layer of a classifier to improve the performance on novel unseen classes with very few examples. Then delves into the proposed idea and studies it with very good and justifiable experiments and visualizations. As a result, I vote for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a subspace regularization technique that encourages the new class weight vector to be in the subspace spanned by those of the base classes for few-shot class incremental learning. Even though similar techniques exist in few-shot learning literature, reviewers appreciate the simplicity of the method and thorough experiments. The authors have revised the paper to include missing references suggested by reviewers during the rebuttal. They were not able to add experiment comparisons to Tao et al. (2020) and Chen & Lee (2021) as requested by reviewer Vrap due to missing code release.  Please consider adding them in your draft later.
Decision: Accept (Poster)