Title: Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration
Venue: NeurIPS 2023
Accepted: yes
Keywords: Few-Shot Class-Incremental Learning

Continual Learning

Class-Incremental Learning
Abstract: Real-world scenarios are usually accompanied by continuously appearing classes with scare labeled samples, which require the machine learning model to incrementally learn new classes and maintain the knowledge of base classes. In this Few-Shot Class-Incremental Learning (FSCIL) scenario, existing methods either introduce extra learnable components or rely on a frozen feature extractor to mitigate catastrophic forgetting and overfitting problems.  However, we find a tendency for existing methods to misclassify the samples of new classes into base classes, which leads to the poor performance of new classes. In other words, the strong discriminability of base classes distracts the classification of new classes. To figure out this intriguing phenomenon, we observe that although the feature extractor is only trained on base classes, it can surprisingly represent the *semantic similarity* between the base and *unseen* new classes. Building upon these analyses, we propose a *simple yet effective* Training-frEE calibratioN (TEEN) strategy to enhance the discriminability of new classes by fusing the new prototypes (i.e., mean features of a class) with weighted base prototypes. In addition to standard benchmarks in FSCIL, TEEN demonstrates remarkable performance and consistent improvements over baseline methods in the few-shot learning scenario. Code is available at: https://github.com/wangkiw/TEEN

Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration

Abstract

Real-world scenarios are usually accompanied by continuously appearing classes with scarce labeled samples, which require the machine learning model to incrementally learn new classes and maintain the knowledge of base classes. In this Few-Shot Class-Incremental Learning (FSCIL) scenario, existing methods either introduce extra learnable components or rely on a frozen feature extractor to mitigate catastrophic forgetting and overfitting problems. However, we find a tendency for existing methods to misclassify the samples of new classes into base classes, which leads to the poor performance of new classes. In other words, the strong discriminability of base classes distracts the classification of new classes. To figure out this intriguing phenomenon, we observe that although the feature extractor is only trained on base classes, it can surprisingly represent the semantic similarity between the base and unseen new classes. Building upon these analyses, we propose a simple yet effective Training-frEE calibratioN (TEEN) strategy to enhance the discriminability of new classes by fusing the new prototypes ( i.e. , mean features of a class) with weighted base prototypes. In addition to standard benchmarks in FSCIL, TEEN demonstrates remarkable performance and consistent improvements over baseline methods in the few-shot learning scenario. Code is available at: https://github.com/wangkiw/TEEN

1 Introduction

Deep Neural Networks ( i.e. , DNNs) have achieved impressive success in various applications [16, 19, 36, 55, 54, 5], but they usually rely heavily on static and pre-collected large-scale datasets ( e.g. , ImageNet [11]) to achieve this success. However, the data in real-world scenarios usually arrive continuously. For example, the face recognition system is required to authenticate existing users, and meanwhile, new users are continually added [30]. The scene where the model is required to continually learn new knowledge and maintain the ability on old tasks is referred to as ClassIncremental Learning (CIL) [41]. The main challenge of CIL is the notorious catastrophic forgetting problem [14], where the model forgets old knowledge as it learns new ones.

Many methods have been designed to overcome catastrophic forgetting in CIL from different perspectives, e.g. , knowledge distillation [35], parameter regularization [22, 53], and network expansion [50, 47, 46]. These methods require new tasks to contain sufficient labeled data for supervised training. However, collecting enough labeled data in some scenarios is challenging, making the conventional CIL methods hard to deploy [30]. For example, the face recognition system can only collect very few facial images of a new user due to privacy reasons. Therefore, a more realistic and practical incremental learning paradigm, Few-Shot Class-Incremental Learning (FSCIL) [42], is

proposed to address the problem of class-incremental learning with limited labeled data. Figure 1a gives a detailed illustration of FSCIL.

In addition to the catastrophic forgetting problem [14], FSCIL will suffer the overfitting problem because the model can easily overfit the very few labeled data of new tasks. Previous works [57, 58, 61] have adopted prototype-based methods [40, 33] to conquer the limited data problem. These methods freeze the feature extractor trained on base classes when dealing with new classes and use the prototypes of new classes as the corresponding classifier weights. The frozen feature extractor can alleviate the catastrophic forgetting problem and the prototype classifier can circumvent the overfitting problem . However, the inherent problem of learning with few-shot data is challenging to depict precisely the semantic information of a new category with limited data, making prototypes of new classes inevitably biased [27]. An intuitive reason for the biased prototypes is that the feature extractor has not been optimized for the new classes. For example, the feature extractor trained on 'cat' and 'dog' can not precisely depict the feature of a new class 'bird' especially when the instances of 'bird' are incredibly scarce 2 . Many prototype adjustment methods [27, 57, 58, 15, 61, 1, 24] are dedicated to rectifying the biased representations of the new classes. These methods usually focus on designing complex pre-training algorithms to enhance the compatibility of representations [58, 59, 10], or complicated trainable modules that better adapt the representation of the new classes [57, 15, 61, 1], all of which rely

…

…

…

Class: 1

Class: 2

Class: 3

Class: 4

Class: 5

Class: 6

Class: 7

Session 1

Session 2

Base Classes

Novel Classes

(a) An illustration of FSCIL setting.

'Forest'

'Plain'

High

Low

'Plate'

'Porcupine'

'Cloud'

'Mountain'

'Bowl'

'Clock'

'Can'

'Bear'

'Otter'

Semantic Similarity

Novel Classes

Base Classes

Figure 1: (a) The base session contains sufficient samples of base classes for training. The incremental sessions ( i.e. , sessions after the base session) only contain few-shot samples of new classes. FSCIL aims to obtain a unified classifier over all seen classes. Notably, the data from previous sessions are unavailable. (b) For example, considering the new classes 'Plain', 'Plate' and 'Porcupine' in CIFAR100 [23], the corresponding most similar base classes selected by the similarity can depict the really semantic similarity. The three most similar base classes are selected by computing the cosine similarity between base prototypes and novel prototypes. The results show the feature extractor only trained on the base classes can also well represent the semantic similarity between the base and new classes.

on significant training costs in exchange for improved model performance.

However, we find that although existing methods perform well on the widely used performance measure ( i.e. , the average accuracy across all classes), they usually exhibit poor performance on new classes , which suggests that the calibration ability of existing methods could be improved. In other words, existing methods neglect the performance of new classes. A direct reason for this negligence is the current unified performance measure is easily overwhelmed by the dominated base classes ( e.g. , 60 base classes in the CIFAR100 dataset). However, the more recent tasks are usually more crucial in some real-world applications. For example, the new users in the recommendation system are usually more important and need more attention [45]. The important yet neglected new classes' performance inspires us to pay more attention to it.

To better understand the performance of current FSCIL methods [42, 6, 9, 58, 57], we explicitly evaluate and analyze the low performance of existing methods on new classes and empirically demonstrate the instances of new classes are prone to be predicted into base classes . To further understand this intriguing phenomenon, we revisit the representation of the feature extractor and find that the feature extractor trained only on base classes can already represent the semantic similarity between the base and new classes well. As shown in Figure 1b, although the novel classes are unavailable during the training of the feature extractor, the semantic similarity between the base and novel classes can also be well-represented. However, existing methods have been obsessed with designing complex learning modules and disregard this off-the-shelf semantic similarity .

Based on the above analysis, we propose to leverage the overlooked semantic similarity to explicitly enhance the discriminability of new classes. Specifically, we propose a simple yet effective TrainingfrEE prototype calibratioN ( i.e. , TEEN) strategy for biased prototypes of new classes by fusing the biased prototypes with weighted base prototypes, where the weights for the base prototypes are class-specific and semantic-aware . Notably, TEEN does not rely on any extra optimization procedure or model parameters once the feature extractor is trained on the base classes with a vanilla supervised optimization objective. Besides, the excellent calibration ability and training-free property make it a plug-and-play module.

Our contributions can be summarized as follows:

We empirically find that the lower performance of new classes is due to misclassifying the samples of new classes into corresponding similar base classes, which is intriguing and missing in existing studies.

We propose a simple yet effective training-free calibration strategy for new prototypes, which not only achieved a higher average accuracy but also improved the accuracy of new classes ( 10.02% ∼ 18.40% better than the runner-up FSCIL method ).

We validate TEEN on benchmark datasets under the standard FSCIL scenario. Besides, TEEN shows competitive performance under the few-shot learning scenario. The consistent improvements demonstrate TEEN's remarkable calibration ability.

2 Related Works

2.1 Class Incremental Learning

The model in the Class-Incremental Learning ( i.e. , CIL) scenario is required to learn new classes without forgetting old ones. Save representative instances in old tasks ( i.e. , exemplars) and replay them in new tasks is a simple and effective way to maintain the model's discriminability ability on old tasks [20, 2, 37]. Furthermore, knowledge distillation [17, 35, 25, 49] is widely used to maintain the knowledge of the old model by enforcing the output logits between the old model and the new one to be consistent. iCaRL [35] uses the knowledge distillation as a regularization item and replays the exemplars when learning the feature representation. Many methods follow this line and design more elaborate strategies to replay exemplars [29] and distill knowledge [21, 12]. Recently, model expansion [28, 50, 47, 46, 39, 60] have been confirmed to be effective in CIL. The most representative method [50] saves a single backbone and freezes it for each incremental task. The frozen backbone substantially alleviates the catastrophic forgetting problem.

2.2 Few-Shot Learning

The model in Few-Shot Learning ( i.e. , FSL) [48] scenario is required to learn new classes with limited labeled data. Existing methods usually achieve this goal either from the perspective of optimization [13, 31, 3] or metric learning [40, 43, 26, 27, 56]. The core thought of optimization-based methods is to equip the model with the ability to fast adaptation with limited data. The metric-based methods focus on learning a unified and general distance measure to depict the semantic similarity between instances. Besides, [51] proposes to use Gaussian distribution to model each feature dimension of a specific class and sample augmented features from the calibrated distribution. Based on these augmented features, [51] train a logistic regression and achieve competitive performance. However, TEEN can outperform [51] in most settings and without any training cost when recognizing new classes.

2.3 Few-Shot Class-Incremental Learning

The model in Few-Shot Class-Incremental Learning ( i.e. , FSCIL) [42] scenario is required to incrementally learn new knowledge with limited labeled data. Many existing methods are dedicated to designing learning modules to train a more powerful feature extractor [58] or adapting the representation of the instances of new classes [57, 9, 61, 1]. Besides, TOPIC [42] utilizes a neural gas network to alleviate the challenging problems in FSCIL. [8] adopts word embeddings as semantic information and introduces a distillation-based FSCIL method. IDLVQ [6] proposes to utilize quantized reference vectors to compress the old knowledge and improve the performance in FSCIL. However, all these

methods overlook the abundant semantic information in base classes and the poor performance in new classes. In this study, we aim to take a small step toward filling this gap. The most related work to TEEN is [1]. However, it relies on optimizing a regularization-based objective function to implicitly utilize the semantic information. As opposed to [1], TEEN takes advantage of the empirical observation and gets rid of the optimization procedure and is thus more efficient and effective.

3 Preliminaries

3.1 Definition and notations

̸

In FSCIL [42], we assume there exists T sessions in total, including a base session ( i.e. , the first session) and T -1 incremental sessions ( i.e. , sessions after the first session). We denote the training data in the base session as D 0 and the training data in the incremental sessions as {D 1 , D 2 , . . . , D T -1 } . For the training data D i in the i -th session, we further notate it with { ( x j , y j ) } N i j =1 and corresponding label space with C i . Note that only training data D i is available in the i -th session. Accordingly, the testing data and testing label space in session i can be denoted as D test i and C test i . To better evaluate the model's discriminability on all seen tasks, the testing label space C test i of i -th session contains all seen classes during training, i.e. , C test i = ⋃ i j =0 C j . An incremental session can also be denoted as a N -way K -shot classification task, i.e. , N classes and K labeled examples for each class. Note that the training label spaces between different sessions are disjoint, i.e. , for any i, j ∈ [0 , T -1] and i = j, C i ⋂ C j = ∅ .

Compared to conventional CIL, FSCIL only requires the model to learn new classes with limited labeled data. On the other hand, compared to conventional Few-Shot Learning (FSL), FSCIL requires the model to continually learn the knowledge of new classes while retaining the knowledge of previously seen classes . We introduce the related works on CIL, FSL and FSCIL in supplementary due to the space limitation.

The model in FSCIL can be decoupled into a feature encoder ϕ θ ( · ) with parameters θ and a linear classifier W . Given a sample x j ∈ R D , the feature of x j can be denoted as ϕ θ ( x j ) ∈ R d . For a N -class classification task, the output logits of a sample x j can be denoted as O j = W ⊤ ϕ θ ( x j ) ∈ R N where W ∈ R d × N .

3.2 Prototypical Network

ProtoNet [40] is a widely used method in few-shot learning problems. It computes the mean feature c k of a class k ( e.g. , class prototype) and uses the class prototype to represent the corresponding class:

Num k denotes the number of samples in class k . For a classification task with N classes, the classifier can be represented by the N prototypes, i.e. , W = [ c 1 , c 2 , . . . , c N ] . Following the [58, 57, 61, 1], we freeze the feature extractor ϕ θ trained on the base task and plug the class prototype into the classifier while dealing with a new class. The frozen feature extractor can alleviate catastrophic forgetting and the plug-in updating of the classifier can circumvent the overfitting problem relatively.

4 A Closer Look at FSCIL

In this session, we comprehensively analyze current FSCIL methods from a decoupled perspective. Although the previous updating paradigm of extractor-frozen and prototypes-plugged can achieve adequate average accuracy in all classes, there also exist some shortcomings in it. In this section, we empirically show that the current methods ( e.g. , [57, 58]) are generally not effective in new classes. Furthermore, we take a step toward understanding the reason for the low performance in new classes.

4.1 Understanding the reason for poor performance in new classes

To better understand the performance of existing methods, we first measure the performance by average accuracy on all classes, base classes and new classes, respectively. As illustrated in Figure 2,

our first observation is that the average accuracy across base classes is extremely higher than the accuracy in new classes . The inconsistent performance between the base and new classes is caused by the frozen feature extractor and biased new prototypes. The former forces the model to overfit base classes and the latter causes the model to underfit the new classes.

To determine the real cause of the low performance on new classes, we further investigate ' What base classes are the new classes incorrectly predicted into? ' and get our second observation.

Our second observation is that the prototype-based classifier misclassifies the new classes to their corresponding most similar base classes with high probability , i.e. , many instances of new classes are closer to their nearest base prototype than corresponding target prototypes

To verify this observation, we first analyze the detailed prediction results from a decoupled perspective. Specifically, we treat all base classes as a 'positive class' and all new classes as a 'negative class' and transform the FSCIL problem into a two-class classification task. We compute the false negative rate ( i.e. , FNR ) and false positive rate ( i.e. , FPR ) of each binary prediction task on each incremental session. The FNR and FPR in the confusion matrix is defined as follows:

































As shown in Table 1, the FPR is far greater than FNR illustrating new classes are generally misclassified into base classes but the base classes are generally misclassified into base classes. On the basis of this conclusion, we further explore the details of misclassification. To better illustrate the analysis, we define 'misclassified T o most similar B ase classes R atio' ( i.e. , TBR ) for new classes and 'misclassified T o most similar N ew classes R atio' ( i.e. , TNR ) for base classes. Specifically, considering the misclassified instances in base classes and new classes respectively, the TBR and

̅

𝑐𝑐

𝑛𝑛

𝑐𝑐

𝑛𝑛

𝑐𝑐

𝑛𝑛

𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏

𝑐𝑐

𝑛𝑛

Instances of new classes

Instances of base classes

Misclassfied instances

Calibration

𝑐𝑐

𝑛𝑛

𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏

(a) An qualitative illustration of the calibration

̅

𝑝𝑝

𝑛𝑛𝑏𝑏𝑛𝑛

𝑝𝑝 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 (b) An quantitative illustration of the calibration

Figure 3: (a) The ⃝ represents the samples of new classes and the △ represents the samples of base classes. The dotted lines between samples and prototypes ( i.e. , c n , c base n , ¯ c n ) represent the corresponding classification results. Blue and purple dotted lines represent samples that are correctly classified, and red dotted lines represent misclassification. The yellow circles ( i.e. , W → R samples) and green triangles ( i.e. , R → W samples) in the right figure are samples with prediction changes after calibration. (b) The detailed ratio (%) of base and new classes with regard to three types of samples( i.e. , UC samples, W → R samples, R → W samples). Only two incremental sessions ( i.e. , session 1 and session 2) of CIFAR100 are listed here for convenience. Please refer to the supplementary for results on more incremental sessions and more datasets.

TNR are defined as follows:

N n means the number of misclassified instances of new classes. M b is the number of instances misclassified into most similar base classes in N n samples. M n and N b have similar meaning. As shown in Table 2, the TBR is higher and the TNR is relatively low, which strongly underpins our second observation. To our knowledge, we are the first to explain the reason for the low performance of new classes in existing methods and observe that the samples of new classes are easily misclassified into the most similar base classes.

To summarize, we empirically demonstrate that existing methods perform poorly on new classes and find that this is because the model tends to misclassify new class samples into the most similar base classes. Combining existing analysis ( i.e. , the samples of new classes tend to be classified into the most similar base classes ) and observations ( i.e. , the well-trained feature extractor on base classes can also well represent the semantic similarity between the base and new classes ), we propose to use the frozen feature extractor as a bridge and calibrate the biased prototypes of new classes by fusing the biased new prototypes with the well-calibrated base prototypes.

5 Similarity-based Prototype Calibration

Based on the common assumptions in FSCIL [42], sufficient instances of base classes are available during the training of the feature extractor. We argue the sufficient data in the base session contains abundant semantic information ( i.e. , a sufficient number of classes) and the base prototypes are well-calibrated ( i.e. , a sufficient number of instances for each class). The above analysis leads to a natural question:

Can we leverage the well-calibrated prototypes in the base session for a new prototype calibration?

As the previous methods overlook the low performance of new classes caused by the corresponding biased prototypes, we propose to explicitly calibrate these biased prototypes with the help of the well-calibrated base prototypes. The off-the-shelf semantic similarity serves as a bridge between the base prototypes and the new ones. In the following sections, we introduce the details of fusing the well-calibrated base prototypes and ill-calibrated new prototypes to calibrate the biased ones. Afterwards, we analyze the effects of this training-free prototype calibration on new classes and base classes, respectively.

5.1 Fusing the biased prototypes with calibration item

We assume there exist B classes in the base session and C classes in an incremental session, e.g. , B = 60 , C = 5 in the CIFAR100 dataset. Without loss of generality, we only consider the base session and the first incremental session for simplification. Other incremental sessions can be obtained in the same way. Following the notations in section 3.1, the empirical prototype of i -th class can

be notated as c i . Therefore, the base prototypes are c b (0 ≤ b ≤ B -1) and the new prototypes are c n ( B ≤ n ≤ B + C -1) . As the base session contains sufficient classes and sufficient samples for each class, the model trained on the base session can capture the distribution of base classes and obtain well-calibrated prototypes for base classes. Generally, the empirical prototype of base classes can be regarded as approximately consistent with the expected class representation. However, due to the limited data in incremental sessions, the empirical prototypes of the novel classes are considered to be severely biased.

Based on the analysis of prototypes, we only calibrate biased prototypes in incremental sessions. Given a new class prototype c n ( B ≤ n ≤ B + C -1) , the calibrated new prototype ¯ c n can be notated by:

The calibration item ∆ c n is a component of base prototypes. The hyperparameter α controls the calibration strength of biased prototypes. Smaller α means the calibrated prototype reflects more of the original biased prototype, while larger α means the calibrated prototype heavily incorporates the base prototypes. Motivated by the observations in section 4.1, the similarity between well-calibrated base prototypes and ill-calibrated new prototypes contains auxiliary side information about the new classes. Therefore, we use weighted base prototypes to represent the calibration item ∆ c n and enhance the discriminability of biased prototypes. Specifically, we compute the cosine similarity S b,n between a new class prototype c n and a base prototype c b :

where τ ( τ > 0 ) is the scaling hyperparameter controlling the weight distribution's sharpness. The weight of a new prototype c n with respect to a base prototype c b is the softmax results over all base prototypes:

Finally, the calibration of biased prototypes of new classes can be formulated as follows:

Notably, the above prototype rectification procedure is a training-free calibration strategy because it does not introduce any learning component or training parameters. Figure 3a gives an intuitive description of the calibration effect.

5.2 Effect of calibrated prototypes

Intuitively, w b,n is larger when a new prototype c n and base prototype c b are more similar. Given a new prototype c n , we assume the most similar base prototype with c n is the base prototype c base n . Therefore, given a new prototype c n , ∑ B -1 b =1 w b,n c b ≈ c base n when the scaling hyperparameter τ is large enough. From the perspective of a biased prototype, a calibrated prototype ¯ c n will be aligned to its most similar base prototypes c n with a proper τ .

To further comprehend the effect of TEEN on the predictions of base and new classes respectively, we define three types of test samples according to whether the prediction results change after TEEN: with unchanged predictions ( i.e. , UC samples), with prediction going from right to wrong ( i.e. , R → W samples), with predictions going from wrong to right ( i.e. , W → R samples). We analyze the specifics of these three types of samples in detail.

Intuitively, some calibrated prototypes of new classes are aligned to base prototypes and reduce the discriminability of base prototypes. Oppositely, aligning the biased prototype to most similar base prototypes can calibrate the prediction of new classes. As shown in Table 3b, we observe the W → R samples are mainly from new classes and the R → W samples are mainly from base classes. Besides, extensive comparison results on the benchmark datasets ( i.e. , Table 3 and Figure 4) show that the negative effect of TEEN is negligible due to the significance of TEEN.















































(a)

mini

ImageNet

















































(b) CUB200















































(c) CIFAR100

6 Experiments

In this section, we first introduce the main experiment details of FSCIL in section 6.1, which include the implementation and performance detail. Subsequently, we introduce the FSL performance of TEEN in section 6.2, which consistently shows the effectiveness of TEEN. Finally, we evaluate TEEN by a comprehensive ablation study in section 6.3.

6.1 Main experimental details of FSCIL

6.1.1 Implementation Details

Datasets and baseline details: Following previous methods [42, 57, 58], we evaluate TEEN on CIFAR100 [23], CUB200-2011 [44], mini ImageNet [38]. We keep the dataset split consistent with existing methods [57, 58]. Notably, each benchmark dataset is divided into subsets containing nonoverlapping label space. For example, CIFAR100 is divided into 60 classes for the base session and the left 40 classes are divided into eight 5-way 5-shot few-shot classification tasks. To validate the performance of TEEN , we compare TEEN with popular CIL methods [35, 4, 18], FSL methods [26, 56, 43], and FSCIL methods [42, 57, 58]. Please refer to the supplementary material for more datasets and baseline details.

Training details: All experiments are conducted with PyTorch [32] on a single NVIDIA 3090. The training of the feature extractor uses vanilla cross-entropy loss as the objective function. It does not evolve any extra complex pretraining module [58, 59, 10, 57], thus making TEEN more efficient and elegant. In addition, we adopt the cosine similarity to measure the similarity between the instances and class prototypes. Following [42, 57, 58], we use ResNet20 [16] for CIFAR100, pre-trained ResNet18 [16] for CUB200 and randomly initialized ResNet18 [16] for mini ImageNet. All compared methods use the same backbone network and initialization for a fair comparison. We set α = 0 . 5 , τ = 16 for mini ImageNet and CUB200, α = 0 . 1 , τ = 16 for CIFAR100. We train the feature extractor on CUB200 with a learning rate of 0.004, batch size of 128, and epochs of 400. Please refer to the supplementary for more training details on CIFAR100 and mini ImageNet.

6.1.2 Comparison results

In this section, we conduct overall comparison experiments on different performance measures. These different performance measures focus on different aspects of the methods. For example, the widely-used average accuracy across all classes ( i.e. , AvgAcc ) is difficult to reflect the performance of new classes because the base classes take a large percentage of all classes ( e.g. , 60 base classes of 100 classes in CIFAR100). Following [58], the Harmonic mean ( i.e. , HMean ) is used to evaluate the balanced performance between the base and new classes. Besides the above performance measures, we also additionally evaluate the different methods of their performance on new classes ( i.e. , NAcc ). Following [42, 57, 58], we also measure the degree of forgetting by the P erformance D ropping Rate ( i.e. , PD ). The PD is defined as PD = Acc 0 -Acc -1 , i.e. , the average accuracy dropping between the first session ( i.e. , Acc 0 ) and the last session ( i.e. , Acc -1 ). The detailed comparison results of PD and AvgAcc are reported in Table 3, and the detailed comparison results of NAcc and HMean are reported in Table 4. These experimental results from different performance measures all demonstrate the effectiveness of TEEN. Besides, significantly lower FPR and TBR in Table 1 and Table 2 also verify the effective calibration of TEEN.

Notably, many previous state-of-the-art FSCIL methods ( e.g. , [57, 58]) usually design complex pretraining algorithms to enhance the extendibility of feature space, which may harm the discriminability of base classes. This elaborate pretraining stage may lead to an inconsistent but negligible performance in the base session. Besides, PD ↓ and ∆ PD in Table 3 are not affected by the pretrained results and also show TEEN outperforms previous state-of-the-art methods.

6.2 Comparison results of FSL

FSL can be approximated as measuring only the performance of the new classes in the first incremental session of FSCIL. Besides, FSL itself also faces the challenge of biased class prototypes due to the few-shot data. Therefore, we validate TEEN in the setting of FSL and report the compared results in Table 5. To ensure a fair comparison, we strictly followed the experimental setup of [51]. The comparison results in Table 5 show

that TEEN can easily outperform the previous state-of-the-art method [51] in several experimental settings. Notably, TEEN does not involve any additional training cost when recognizing new classes , and the only time cost lies in feature extraction. Therefore, once the sample features are extracted, the inference cost of TEEN can be considered negligible compared to previous methods that require heavy training.

6.3 Ablation Study

The influence of α and τ : Notably, the proposed TEEN does not involve additional training-based modules or procedures after pretraining on base classes. When TEEN incrementally learns new classes, only the scaling temperature τ and the coefficient of calibration item α need to be determined.























=8

=16

=32

=64

(a)

α

and

τ

w.r.t AvgAcc



























=8

=16

=32

=64

(b)

α

and

τ

w.r.t NAcc























k

=1

k

=3

k

=5

k

=7

k

=9



(c) SimTEEN w.r.t AvgAcc



























k

=1

k

=3

k

=5

k

=7

k

=9



(d) SimTEEN w.r.t NAcc

Specifically, we select τ from { 8 , 16 , 32 , 64 } and α from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } . Figure 5a and Figure 5b show how the hyper-parameters α and τ influence the average accuracy on all classes and new classes, respectively. It also demonstrates that a proper α can improve the performance of new classes, which confirms the base prototypes can benefit the performance of new classes. Compared with larger τ , relatively smaller τ can smooth the weight distribution. The smaller τ with stronger performance further confirm the effectiveness of utilizing the abundant semantic information from base classes.

The influence of similarity-based weight: To further validate the effectiveness of the semantic similarity between the base and new classes, we remove the Equation 6 and directly align the new prototypes to their corresponding K most similar base class prototypes, i.e. , ¯ c n = α c n + (1 -α ) ∑ K k =1 c k . We denote this Sim ple version of TEEN without similarity-based weight as SimTEEN. We select K from { 1 , 3 , 5 , 7 , 9 } . Figure 5c and Figure 5d show how the similarity-based weight ( i.e. , Equation 6) influences the average accuracy on all classes and new classes, respectively. Notably, as shown in Figure 5d, there is a significant drop, particularly on new classes, after removing the similarity-based weight. This phenomenon confirms that TEEN can achieve calibration of new prototypes with the help of well-represented semantic similarity between the base and new classes.

7 Conclusion

Few-shot class-incremental learning is of great importance to real-world learning scenarios. In this study, we first observe existing methods usually exhibit poor performance in new classes and the samples of new classes are easily misclassified into most similar base classes. We further find that although the feature extractor trained on base classes can not well represent new classes directly, it can properly represent the semantic similarity between the base and new classes. Based on these analyses, we propose a simple yet effective training-free prototype calibration strategy ( i.e. , TEEN) for biased prototypes of new classes. TEEN obtains competitive results in FSCIL and FSL scenarios.

Limitations: The FSCIL methods mentioned in this paper all select the base and new classes from the same dataset. In other words, current FSCIL methods assume the model pre-trains in the same domain, increasing the restrictions on pre-training data collection. Therefore, a more realistic scenario is to pre-train on a dataset that is independent of the subsequent data distribution and then perform few-shot class-incremental learning on a target dataset that we expect. The intricate problem of cross-domain few-shot class-incremental learning will be thoroughly investigated in our future works.

References

Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya, and Jacob Andreas. Subspace regularizers for few-shot class incremental learning. In ICLR , 2022.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurIPS , pages 11816-11825, 2019.

Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. In ICLR , 2019.

Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV , pages 233-248, 2018.

Wei-Lun Chao, Han-Jia Ye, De-Chuan Zhan, Mark E. Campbell, and Kilian Q. Weinberger. Revisiting meta-learning as supervised learning. CoRR , abs/2002.00573, 2020.

Kuilin Chen and Chi-Guhn Lee. Incremental few-shot learning via vector quantization in deep embedded space. In ICLR , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607, 2020.

Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Semantic-aware knowledge distillation for few-shot class-incremental learning. In CVPR , pages 2534-2543, 2021.

Ali Cheraghian, Shafin Rahman, Sameera Ramasinghe, Pengfei Fang, Christian Simon, Lars Petersson, and Mehrtash Harandi. Synthesized feature based few-shot class-incremental learning on a mixture of subspaces. In ICCV , pages 8641-8650, 2021.

Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta-learning approach for few-shot class incremental learning. In CVPR , pages 14146-14155, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV , pages 86-102, 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , pages 1126-1135, 2017.

[14]

Robert M. French and Nick Chater. Using noise to compute error surfaces in connectionist networks: A

novel means of reducing catastrophic forgetting.

Neural Computation

, 14(7):1755-1769, 2002.

Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR , pages 4367-4375, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR , abs/1503.02531, 2015.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR , pages 831-839, 2019.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR , pages 2261-2269, 2017.

David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI , pages 3302-3309, 2018.

Minsoo Kang, Jaeyoo Park, and Bohyung Han. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In CVPR , pages 16050-16059, 2022.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS , 114(13):3521-3526, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Anna Kukleva, Hilde Kuehne, and Bernt Schiele. Generalized and incremental few-shot learning by explicit learning and calibration without forgetting. In ICCV , pages 9000-9009, 2021.

Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV , pages 614-629, 2016.

Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In ECCV , pages 438-455, 2020.

Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In ECCV , pages 741-756, 2020.

Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In CVPR , pages 2544-2553, 2021.

[29]

Yaoyao Liu, Bernt Schiele, and Qianru Sun. RMM: reinforced memory management for class-incremental learning. In

NeurIPS

, pages 3478-3490, 2021.

Suresh Madhavan and Nitin Kumar. Incremental methods in face recognition: a survey. Artificial Intelligence Review , 54(1):253-303, 2021.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. CoRR , abs/1803.02999, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In

NeurIPS , pages 8024-8035, 2019.

Hang Qi, Matthew Brown, and David G. Lowe. Low-shot learning with imprinted weights. In CVPR , pages 5822-5830, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In CVPR , pages 5533-5542, 2017.

Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS , pages 91-99, 2015.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NeurIPS , pages 348-358, 2019.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV , 115(3):211-252, 2015.

Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML , pages 4548-4557, 2018.

Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NIPS , pages 4077-4087, 2017.

Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117 , 2023.

Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incremental learning. In CVPR , pages 12180-12189, 2020.

Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS , pages 3630-3638, 2016.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Feng Wang, Weike Pan, and Li Chen. Recommendation for new users with partial preferences by integrating product reviews with static specifications. In UMAP , pages 281-288, 2013.

[46]

Fu-Yun Wang, Da-Wei Zhou, Liu Liu, Han-Jia Ye, Yatao Bian, De-Chuan Zhan, and Peilin Zhao. Beef:

Bi-compatible class-incremental learning via energy-based expansion and fusion. In

ICLR

, 2023.

Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In ECCV , pages 398-414, 2022.

Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys , 53(3):1-34, 2020.

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR , pages 374-382, 2019.

Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. In CVPR , pages 3014-3023, 2021.

Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR , 2021.

Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning. In ICLR , 2023.

Yang Yang, Da-Wei Zhou, De-Chuan Zhan, Hui Xiong, and Yuan Jiang. Adaptive deep models for incremental learning: Considering capacity scalability and sustainability. In KDD , pages 74-82, 2019.

Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Heterogeneous few-shot model rectification with semantic mapping. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(11):38783891, 2021.

Han-Jia Ye, De-Chuan Zhan, Nan Li, and Yuan Jiang. Learning multiple local metrics: Global consideration helps. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(7):1698-1712, 2020.

Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR , pages 12203-12213, 2020.

Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In CVPR , pages 12455-12464, 2021.

Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In CVPR , pages 9036-9046, 2022.

Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan. Few-shot class-incremental learning by sampling multi-phase tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(11):12816-12831, 2023.

Da-Wei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. arXiv preprint arXiv:2305.19270 , 2023.

Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha. Self-promoted prototype refinement for few-shot class-incremental learning. In CVPR , pages 6801-6810, 2021.

Supplementary materials

A Discussion

A.1 Implicit Semantic Similarity

Notably, TEEN demonstrates that the off-the-shelf feature extractor is already capable of representing semantic similarity without the need for any additional training costs or auxiliary side information ( e.g. , the textual information of the class name [8]).

A.2 Efficiency

The only time cost in TEEN is the pre-training cost on the base classes. Furthermore, the training cost of the vanilla cross-entropy loss function is relatively small compared to other more complex training paradigms. Additionally, the calibration in TEEN does not evolve any training cost and update procedure. Therefore, the cost of incrementally learning new classes of TEEN can be considered negligible compared with existing methods.

A.3 Combination with better representation

In addition to combining with existing prototype-based methods, TEEN can also be combined with more powerful pre-trained models ( e.g. , self-supervised pre-trained model [7]). Notably, the recent popular vision-language models ( e.g. , CLIP [34]) can also be seen as a powerful pre-trained model and provide better representation.

B A Closer Look at FSCIL

In this section, we show the observations ( i.e. , Section 3 of the main paper) on more datasets. As shown in Table 10, the existing prototype-based FSCIL methods are extremely prone to misclassify new classes into base classes. Considering the misclassified instances of base classes, we quantitatively show that the samples of new classes are usually misclassified into base classes . These observations in the additional benchmark dataset are consistent with observations in the main paper, which further confirms the universality of our observation.

To further understand the aforementioned observation, we further analyze the question What base classes are the new classes incorrectly predicted into? and come to our second observation: The prototype-based classifier misclassifies the new classes to their corresponding most similar base classes with a high probability. The detailed analysis results in Table 11 further verify the correctness of our observation.

C Experiments

C.1 Details of experiments

Dataset details: Following previous methods [42, 57, 58], we evaluate TEEN on CIFAR100 [23], CUB200-2011 [44], mini ImageNet [38]. CIFAR100 [23] contains 100 classes and each class contains 500 images for training and 100 images for testing. The image size of CIFAR100 is 3 × 32 × 32 . The CUB200-2011 [44] is a widely-used fine-grained dataset and is also a benchmark dataset of few-shot image classification. It contains 200 classes and all images of these classes belong to birds. The mini ImageNet [38] are sampled from the raw ImageNet [11] and contains 100 classes.

In FSCIL, each benchmark dataset is divided into different subsets. Each subset contains specific classes and the label space of different subsets is nonoverlapping. Specifically, CIFAR100 is divided into 60 classes for the base session and the remaining 40 classes are divided into eight 5-way 5-shot few-shot classification tasks. CUB200 is divided into 100 base classes for the base session and the remaining 100 classes are divided into ten 10-way 5-shot few-shot classification tasks. mini ImageNet is divided into 60 base classes for the base session, and the remaining 40 classes are divided into eight 5-way 5-shot few-shot classification tasks. The splitting details ( i.e. , the class order and the selection of support data in incremental sessions) follow the previous methods [42, 57, 58]. Notably, no old samples are saved to assist in maintaining the discriminability of previous classes.

Baseline details: Following previous methods [42, 57, 58, 61], we compare TEEN with popular CIL methods, FSL methods, and FSCIL methods. For CIL methods, we select iCaRL [35], EEIL [4] and Rebalancing [18] as our baseline methods. For methods based on few-shot, we adopt DecoupledNegativeCosine [26], Decoupled-Cosine [43] and Decoupled-DeepEMD [56] as our baseline methods. For FSCIL methods, we adopt TOPIC [42], CEC [57] and FACT [58] methods as our baseline methods.

Training details: The training of the feature extractor uses vanilla cross-entropy loss as the objective function. In addition, we adopt the cosine similarity to measure the feature of instances to class prototypes. Following [42, 57, 58], we use ResNet20 [16] for CIFAR100, pre-trained ResNet18 [16] for CUB200 and randomly initialized ResNet18 [16] for mini ImageNet. All compared methods use the same backbone network and same initialization for a fair comparison.

For the hyperparameters setting, we set α = 0 . 5 , τ = 16 for mini ImageNet and CUB200, α = 0 . 1 , τ = 16 for CIFAR100. We train the feature extractor on CUB200 with a learning rate of 0.004,

batch size of 128, and epochs of 400. We train the feature extractor on CIFAR100 and mini ImageNet with a learning rate of 0.1 and batch size of 256. The cosine scheduler is used to adjust the learning rate. Before computing the cross-entropy loss, a widely-used temperature scalar is used to adjust the distribution of output logits. For example, the original output logits of instance x j are denoted as O j ∈ R C . The logits used to compute cross-entropy loss are denoted as O j τ o . The τ o is set to 16 for the CIFAR100 dataset and 32 for mini ImageNet and CUB200 datasets.

C.2 More comparison results

In the main paper, we compare different methods in overall performance measures: performance drop rate, average accuracy across all classes, harmonic mean accuracy and average accuracy on new classes. In this session, we report the detailed comparison results of the aforementioned performance measures on CIFAR100 [23] and CUB200 [44]. Besides the main results in Table 6 and Table 7, we also report the accuracy across new classes and harmonic mean accuracy in Table 8. The consistent performance improvement demonstrates the effectiveness of TEEN's calibration ability.

D Empirical Analysis

D.1 Effects of TEEN

In the main paper, we only show the effect of calibration prototypes on the CIFAR100 dataset. In this supplementary material, we report more detailed results in Table 9. The results further confirm our conclusion in the main paper: the proposed TEEN can well calibrate the biased prototype of new classes. Although TEEN leads to misclassifying some instances of base classes, the average accuracy across all classes is always very competitive.

D.2 Combination of TEEN with existing methods

Due to the flexibility of proposed TEEN , it can be seen as a plug-and-play module to calibrate the biased prototypes of new classes. For a more in-depth understanding of the TEEN , we combine TEEN with FACT [58] and show the performance on different performance measures in Figure 6, Figure 7 and Figure 8. We empirically find that combining the TEEN with FACT has little effect on the average accuracy across all classes ( i.e. , the left figures in Figure 6, Figure 7 and Figure 8). However, the TEEN can improve the average accuracy of new classes and the harmonic mean accuracy. In other words, the existing prototype-based methods ( i.e. , FACT) benefit the well-calibration effect of TEEN and achieve better-balanced performance between the base and new classes. We believe more methods ( e.g. , [52]) can benefit from the flexibility of TEEN.

E Broader impact

Our proposed method can significantly improve the accuracy of new classes in FSCIL and FSL scenarios, leading to a better understanding of the low performance of new classes. Furthermore, the analyses in this study are first proposed in the FSCIL and FSL fields, which point out that we should pay more attention to the performance of new classes . In addition to this, our method may also have potential heuristic effects on the expectation estimation of the distribution.































(a) Accuracy on

all

classes

































(b) Harmonic mean accuracy





























(c) Accuracy on

new

classes































(a) Accuracy on

all

classes





























(b) Harmonic mean accuracy





























(c) Accuracy on

new

classes

































(a) Accuracy on

all

classes





























(b) Harmonic mean accuracy





























(c) Accuracy on

new

classes

Peer reviews:
comments: SUMMARY:
The paper proposes a strategy called Training-frEE calibratioN (TEEN) for Few-Shot Class-Incremental Learning (FSCIL) scenario to enhance the discriminability of new classes by fusing the new prototypes with weighted base prototypes. TEEN demonstrates remarkable performance and consistent improvements over baseline methods in the few-shot learning scenario.

STRENGTHS:
The paper addresses the Few-Shot Class-Incremental Learning (FSCIL) scenario, which is a challenging and important problem in real-world scenarios.

The proposed strategy, TEEN, is simple yet effective and demonstrates remarkable performance and consistent improvements over baseline methods in the few-shot learning scenarios.

WEAKNESSES:
The paper demonstrates significant improvement on new classes. However, based on the experimental results, it appears that the False Negative ratio, which involves classifying base instances into incorrect classes, may increase. It would be beneficial to address how this problem is handled and how the performance can be balanced between base classes and new classes. Additional analysis on this topic would be appreciated.

As stated by the authors, pre-training on a dataset that is independent of the subsequent data distribution would be advantageous.

The paper lacks a comparison of the proposed method with state-of-the-art methods in other few-shot learning scenarios, such as few-shot domain adaptation or few-shot semi-supervised learning. It would be beneficial to include more analysis and comparisons with these methods.

QUESTIONS:
See the weakness section for more details

LIMITATIONS:
See the weakness section for more details
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-19

comments: SUMMARY:
This paper tackles the problem of few-shot class incremental learning based on prototypical network. Motivated by the observation that novel classes are easily misclassified as base classes, the authors propose a prototype calibration strategy. The calibrated prototype is a weighted sum of prototype computed from novel class support set and base classes semantically similar with the novel class. Experiments on miniImageNet, CUB and CIFAR demonstrate the superiority of the proposed method over previous ones.

STRENGTHS:
1. The paper is well motivated that the inferior performance of prototypical network results from the phenomenon that novel classes are easily misclassified as base classes. The phenomenon is well-studied via experiments.
2. The paper is well written and easy to follow.
3. The authors propose a simple but effective calibration strategy to improve the performance.

WEAKNESSES:
1. The paper introduces two hyper parameters which need to be exhaustively searched for every dataset.
2. There lacks cross-dataset experiments to show the effectiveness of the proposed method when transferring knowledge from one dataset to another.

QUESTIONS:
The semantic similarity based calibration is a straightforward and commonly used method used in the classification area. There may lack technical contribution to the community.

LIMITATIONS:
The limitation of cross-dataset evaluation is discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors work on the Few-Shot Class-Incremental Learning (FSCIL) scenario and propose the Training-frEE calibratioN (TEEN) strategy. This strategy enhances the discriminability of new classes by fusing the new prototypes with base prototypes. This approach is different from previous methods, which either introduce extra learnable components or rely on a frozen feature extractor.\n\nSTRENGTHS:\n1. The authors observed that the feature extractor, although only trained on base classes, can surprisingly represent the semantic similarity between the base and unseen new classes. Accordingly, they proposed a solution.\n2. The paper is easy to follow, with clear experimental details that aid reproducibility. The motivations are also presented clearly.\n\nWEAKNESSES:\n1. The issue of misclassification has already been observed and studied in previous few-shot learning works. Therefore, the poor performance of new classes is not surprising in the task of Few-Shot Class-Incremental Learning (FSCIL).\n2. Forming connections across samples from base and novel classes is not a new concept. For instance, FADI[1] discovered that a novel class may implicitly leverage the knowledge of multiple base classes to construct its feature space. It then builds a discriminative feature space for each novel class via association and discrimination steps.\n[1] Few-Shot Object Detection via Association and Discrimination, Yuhang Cao et al., NeurIPS 2021.\n\nQUESTIONS:\nI can understand that the prototype of a novel class is biased due to the lack of samples, and it can be easily misclassified as a base class. Intuitively, if the novel class is prone to be classified as the most similar base class, interpolation among such features could exacerbate this issue. But why is the use of weighted base prototypes to calibrate novel ones beneficial in enhancing the discriminability of such a prototype while reducing the discriminability of base prototypes (lines 217-218, 238-239)?. And could you please clarify the definition of discriminability?\n\nLIMITATIONS:\nAs mentioned in the weaknesses section, the proposed idea is not novel at all. It unfortunately is just a tiny incremental contribution which is rather insufficient for a publication in top tier conference such as NeurIPS.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a novel training-free calibration approach for few-shot class incremental learning. The authors make an observation that one main problem with FSCIL is that data of new classes can be easily mis-classified as base session classes. By utilizing the well-calibrated embeddings of base session classes to help embeedings of the new classes, the author improve the performance of FSCIL.

STRENGTHS:
1. The observation of new classes being confused as old classes is somewhat novel.
2. The proposed classifier-calibration method is interesting and novel.
3. The proposed method is simple yet effective.
4. Detailed ablation studies have been performed on the introduced hyper-parameters.

WEAKNESSES:
1. One of the reviewer's concern is in terms of the robustness of the method to hyper-parameter \alpha and \tau. Are the optimal hyper-parameters the same for different datasets or different incrementing procedures (i.e., how many classes per incremental session)?
2. In addition, the author should provide more detailed justification for the design of the method. Why use this simple linear interpolation for calibration? Are there any empirical observations or theory to support this design?

QUESTIONS:
1. I suggest the author provide more ablation study on robustness of hyper-parameter on more dataset.
2. I suggest the author provide more justification on the designing of the method (i.e., why this simple linear interpolation form). Such justification can be empirical visualizations, theoretical analysis, etc.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper closely looks at problems in prototypical networks and methods in the context of few shot class incremental learning. The authors observe semantic similarity between new and base prototypes along with new classes being misclassified regularly as semantically similar base classes. Towards this they use a training free calibration strategy to guide the new ill calibrated prototypes using the base prototypes. Essentially bridging the lack of training data and suppressing bias.\n\nSTRENGTHS:\n- The major strength of the paper lies in the simplicity of the method. Instead of compromising the base class performance they approach the problem from a new class orientation. Doing so relieves the method from the convoluted approaches of feature space reservation[1], meta learning schemes[2] or self supervision[3]. Instead the methods strength lies in utilising the uncompromised power of the base prototypes.    \n- The fact that almost all evaluation metrics one way or the other focus on the novel class performance and not the less interpretable average accuracy and performance decay rate makes the paper even more attractive. This shows commitment towards pushing the frontier in aspects previously underexplored and facilitating a fair and thorough comparison.  \n- The work is constructed clearly and simply. It seems obvious that the authors followed a train of thought. They end up convincing the reader on why each module is essential. It generally reads very smoothly, hopping from hypothesis to observations to results and repeats. For example section 3 which tries to “Understand the reason for poor performance in new classes” leads smoothly into the observations for that section which motivates the section on calibration (section 4) and this section takes a similar hypothesis-observation-result route.   \n- The fact that almost all evaluation metrics one way or the other focus on the novel class performance and not the less interpretable average accuracy and performance decay rate makes the paper even more attractive. This shows commitment towards pushing the frontier in aspects previously underexplored and facilitating a fair and thorough comparison.\n\nWEAKNESSES:\n- Session 4 uses a variety of terminology. Using a consistent term for each prototype could potentially read better. (For example ill-calibrated new prototypes, new prototypes, biased prototypes)  \n- In relation to prior work it seems only Section 1 is truly dedicated to the literature. And even then it seems the prior works that this paper branches out from are not discussed in appropriate detail. Prior methods from the 3 other papers from Table 1 and Table 2 are not detailed in any preceding or subsequent section. Their would be reason to do so as the method directly criticises (line 86-87 or line 190-191) prior works. Prefacing in better detail how previous methods mitigate biases could be essential in understanding the state of the literature and motivate TEEN.  \n- [4] show in “Simpleshot” that L2 normalisation of prototypes improves performance. This is also confirmed by [5] in their nearest neighbour method for few shot learning. The authors overlooked this crucial feature transformation which is commonly conducted in previous studies on Non Parametric methods for FSCIL for example the NoNPC method by Oh et al. [6] This omission somewhat limits the comprehensiveness of their findings despite the improved results given the prevalnece of the method.   \n- [7], [8] are missing comparisons. Therefore, the tables that compare the method to the current sota would should less prominent results.  \n\nMinor Remarks and Typos:  \n- 213: “rely” should be “relies”  \n- 2: “scarce”  \n- The use of tau parameter in equation (5) might be better understood if shifted to equation 6 as S_{b,n} is a scalar and in any case it seems more common that a temperature scaling parameter appears in the softmax. Would recommend to make that alteration.  \n- 73: “neglect” should be “negligence”  \n\n\n[1] Zhou, Da-Wei, et al. \"Forward compatible few-shot class- incremental learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n \t\t\t\t\t\t\t\t\n[2] Zhang, Chi, et al. \"Few-shot incremental learning with continually evolved classifiers.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n \t\t\t\t\t\t\t\t\n[3] Ahmad, Touqeer, et al. \"Few-shot class incremental learning leveraging self-supervised features.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n \t\t\t\t\t\t\t\t\n[4] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting nearest-neighbor classification for few- shot learning. arXiv preprint arXiv:1911.04623, 2019.\n \t\t\t\t\t\t\t\t\n[5] Wang, Guangpeng, and Yongxiong Wang. \"Self-attention network for few-shot learning based on nearest-neighbor algorithm.\" Machine Vision and Applications 34.2 (2023): 28.\n \t\t\t\t\t\t\t\t\n[6] Oh, J., & Yun, S.-Y. (2022). Demystifying the Base and Novel Performances for Few-shot Class-incremental Learning. http://arxiv.org /abs/2206.10596\n\n[7]  Peng, Can et al. Few-Shot Class-Incremental Learning from an Open-Set Perspective, ECCV 2022\n\n[8] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning. In International Conference on Learning Representations 2023\n\nQUESTIONS:\n- See weaknesses.   \n- Discussions and comparisons to the most recent sota are necessary\n\nLIMITATIONS:\nThe authors have adequately mentioned the limitations. And motivate the setting where the pretrain base classes being from different data distribution as the incremental new classes is actually a more realistic setting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}