A Bag of Tricks for Few-Shot Class-Incremental Learning
Authors: Shuvendu Roy
Chunjong Park
Aldi Fahrezi
Ali Etemad
Venue: Accepted by TMLR ()

Abstract: We present a bag of tricks framework for few-shot class-incremental learning (FSCIL), which is a challenging form of continual learning that involves continuous adaptation to new tasks with limited samples. FSCIL requires both stability and adaptability, i.e., preserving proficiency in previously learned tasks while learning new ones. Our proposed bag of tricks brings together six key and highly influential techniques that improve stability, adaptability, and overall performance under a unified framework for FSCIL. We organize these tricks into three categories: stability tricks, adaptability tricks, and training tricks. Stability tricks aim to mitigate the forgetting of previously learned classes by enhancing the separation between the embeddings of learned classes and minimizing interference when learning new ones. On the other hand, adaptability tricks focus on the effective learning of new classes. Finally, training tricks improve the overall performance without compromising stability or adaptability. We perform extensive experiments on three benchmark datasets, CIFAR-100, CUB-200, and miniIMageNet, to evaluate the impact of our proposed framework. Our detailed analysis shows that our approach substantially improves both stability and adaptability, establishing a new state-of-the-art by outperforming prior works in the area. We believe our method provides a go-to solution and establishes a robust baseline for future research in this area.

Published in Transactions on Machine Learning Research (09/2024)
A Bag of Tricks for Few-Shot Class-Incremental Learning
Shuvendu Roy1.2; Chunjong Park2, Aldi Fahrezit, Ali Etemad
Google Research
*Qucen's University, Canada
shuvendu. ryvvendu.ryyQqueensu.coo pej@google.com aldifahrezii@goglee.com ali. etemaddedeQuue
Reviewed on OpenReview: http:://ppaaariiee.eee/e///oooor
Abstract
We present a bag of tricks framework for few-shot class-incremental learning (FSCIL). which is a challenging form of continual learning that involves continuous adaptation to new tasks with limited samples. FSCIL requires both stability and adaptability, i.e, preserving proficiency in previously learned tasks while learning new ones. Our proposed bag of tricks brings together six key and highly influential techniques that improve stability, adaptability, and overall performance under a unified framework for FSCIT. We organize these tricks into three categories: stability tricks, adaptability tricks, and training tricks. Stability tricks aim to mitigate the forgetting of previously learned classes by enhancing the separation between the embeddings of learned classes and minimizing interference when learning new ones. On the other hand, adaptability tricks focus on the effective learning of new classes. Finally, training tricks improve the overall performance without compromising stability or adaptability. We perform extensive experiments on three benchmark datasets, CIFAR-100, CUB-200 200, and miniTMageNett, to evaluate the impact of our proposed framework. Our detailed analysis shows that our approach substantially improves both stability and adaptability, establishing a new state-of-the-arl by outperforming prior works in the area. We believe our method provides a go-to solution and establishes a robust baseline for future research in this area.
1 Introduction
Continual learning is machine learning paradigm that focuses on the ability of model to learn new knowledge, without forgetting what it previonsly learned. In real-world applications, machine learning models often encounter scenarios where they must adapt to novel classes with only a limited number of samples available for learning. This scenario has inspired the introduction of an exciting paradigm called Few-Shot Class Incremental Learning (FSCIT)) (Tao et al.] 20200. Existing literature (Tao el al., 2020: Zhou el al.j 2022a) has demonstrated that traditional continual learning approaches are ineffective in FSCIL, primarily due to the scarcity of labelled data during incremental learning sessions. This data limitation often leads to overfitling on the novel classes, resulting in well-known catastrophic forgetting issue.
Some prior works have linked the problem of catastrophic forgetting with high adaptability (or plasticity) during incremental training (Chi et al. 2022 Zhao et al.) 2021 Consequently, these approaches reduce the adaptability by utilizing an incremenntal frozen Tramework, where the encoder is trained only the base
session and remains frozen during the incremental sessions (Zhao et al. 2021 Tao al.
2020 Chi al.j
2022
Cheraghian et al.) 2021a). The resulting methods provide very high stability, but very little adaptability.
This phenomenon is often referred to the sabbiity-adaptabilie dilemma (Peng et al. 2022b Zhao et al.
2023 in the FSCI.. where high stability causes reduced adaptability and vice-versa.
In this work, we combine a collection of techniques under a bag of tricks framework with the goal of concurrently enhancing both the stability and adaptability of FSCIL. These tricks have never been explored
"This work was partially done when Shuvendu Roy was Intern Google Research. This work was partially done when Ali Etemad was Visiting Researcher at Google Research.

---

Published in Transactions on Machine Learning Research (09/2024)
together, and their impact stability, adaptability, and overall performance has not been studied under the same framework. We categorize the techniques into three main groups: (1) stability, (ii) adaptability, and (iit) training, Stability tricks improve the separation among 'base' classes within the learned embedding space, which minimizes interference with previously learned classes when introducing novel classes during the incremental session. This results greater stability and reduced lorgetting of already learned classes. These approaches include incorporating supervised contrastive loss (Khosla et al.) 2020); pre-assigning prototypes Yang et al., 2022), and including pseudo-classes (Lee et al., 2020) during training. On the other hand, adaptability tricks enhance the model's ability to learn novel classes in the incremental session with careful tuning of the encoder through incremental SubNet tuning (Kang et al. 2023). Finally, training tricks include a pre-training step (Chen et al. 2020), and adding a pre-text task (Gidaris et al. 2018) in order to help boost the overall performance without compromising either stability or adaptability. In summary, the key innovation of our framework is combining a set of modules (tricks), which improve stability, adaptability, and overall performance the same time. While we did not propose any new components, no prior work has explored such modules (Lricks) logether in the context of FSCIL under unified framework.
Extensive experiments three popular benchmark datasets, namely CIEAR100 (Krizhevsky et al. 2009 minJImageNet (Russakovsky et al. 2015) and CUB200 (Wah et al. 2011), demonstrate the effectiveness of our framework. Overall, the proposed framework improves the performance by 3.22%, 1.1%, and 2.0% on CIFAR-100, CUB-200, and minilmageNet, respectively. Incorporating these tricks not only surpasses the overall accuracy of existing FSCIL methods but also demonstrates improved stability (reduced forgetting), higher adaptability (improvements on novel classes), and better separation in the embedding space. Study on the representations learned under stability tricks demonstrates lower intra-class variance and higher inter-class distance between the learned classcs, which ensures lower forgetting and thus higher training stability and overall accuracy. Further study on adaptability tricks shows substantial improvement in the performance on new classes while retaining performance for the learned ones. We also present an ablation study to understand the impact of stability, adaptability, and training tricks on overall performance. This study highlights that stability tricks have the most significant impact, followed by adaptability tricks. Following prior works, we report the main results with 5-shot seiting experiments on commonly used benchmark datasets. Moreover, to study the behaviour of our method under data-scarce scenarios, we also experiment with 1-shot and 2-shot scttings. Additionally, our framework shows similar improvement for larger encoders like ViT, and larger Netss Finally, lo evaluate scalabilily with respect to nunber of classes, we also report our results on the large-scale ImageNet- 1K (Russakovsky et al. 2009] dataset with 1000 classes. Overall, we make the following contributions:
We present new framework for FSCIL that combines a bag of tricks to simultancously improve the stability, adaptability and overall performance of the model.
We conduct a thorough analysis of the behaviour and effectiveness of all tricks towards stability and adaptability in the context of FSCIL.
We significantly improve upon the state-of-the methods on FSCIL and establish robust baseline for future research in this area, including new evaluations low-shots and with a large number of classes.
2
Related Works
2.1 Class Incremental Learning
inassinncremltal learning (CIL) is continual learning paradigm that involves continuously learning novel classes while preserving the knowledge of previously learned ones (Masana et al. 2022). Existing literature on CIL can be broadly categorized into three main groups. The first group, commonly referred to as the replay-based method, stores past samples in a memory bank for rehearsal during incremental sessions to ensure retention of old knowledge (Rebuffi et al. 2017 Rolnick et al.) 2019). The second group is rogularization-bett focusing on preventing significant changes in the parameters to prevent forgetting (Li & Hoiem 2017 Liu et al. 2018 In contrast, the third group dynamically expands the network architecture to accommodate the
2

---

Published in Transactions on Machine Learning Research (09/2024)
learning of novel classes ZZuu et al. 2021a Shi et al. 2022). In practice, the model is trained over several sessions, with each session introducing novel classes to learn. In traditional CIT, each session includes an adequate number of labelled samples for every novel class, as well as the option to store some of the previous samples for rehearsal in future incremental sessions. Howover, as per the definition and problem sctup of FSCIL, neither of these assumptions holds.
2.2 Few-Shot Class-Incremental Learning
In real-world applications, assuming that an incremental session contains a large number of samples for each novel class is impractical (Tao et al. 2020). FSCIL (Tao et al., 2020 Hersche et al., 2022 Cheraghian ct al.] 2021b Liu et al. 2022) tackles this challenging scenario where the model needs to incrementally learn novel classes with only a few samples per class. Tt also assumes that samples from previous sessions are available, which causes privacy concerns in many domains. Consequently, none of the traditional CIL methods, mentioned in the previous section, perform well in the FSCIL selting (Zhou el al. 2022a Tao el al. 2020j. However, there have been developments in FSCIL literature, which can be discussed in two broad categorics. The first category trains the model only on the basc session, keeping the model frozen in the incremental session (Zhu et al. 2021b Shi et al. 2021 Zhang et al. 2021]; ; while the second group tunes the model the incremental sessions (Tao al.] 2020 Cheraghian et al. 2021a Dong et al. 2021 Zhao el. al. 2021).
The main idea of the first category (frozen encoder-based methods) is to ensure greater separability of base classes in the embedding space so that novel classes in the incremental step can easily fit in this space with minimal interference (Zhu et al. forward compatibility (Song et al. 2023). Prior works in the literature proposed different techniques aiming to enable forward compatibility. For instance, FACT (hhuu et al. 2022a) proposed to use virtual prototypes to force the embodding of different classes to be maximally separated while respecting the relative positioning of classes in the embedding space. SAVC (Song et al. (003) created virtual classes during the base session, and Yaagg et al., 2022) assigned random prototypes that are maximally separated from one another. These methods offer relatively high stability, but little adaptability for incremental training.
2021b: Shi et al.) 2021 Zhang et al. 2021). This is often referred to as
The second group of methods tunes the encoder in the incremental session to provide better flexibility for learning new tasks, thus providing better adaptability. For example, MgSvF' (Zhao et al.) 2021 strategically updated different components at different rates, effectively balancing the adaptation to new knowledge and the preservation of old ones. The exemplar relation distillation framework (Dong et al. 2021 constructed and updated an exemplar relation graph to facilitate the learning of novel classes. Soft.Net (Kang et al. 2023
proposed to utilize the concept of the lottery-ticket hypothesis to find sub-network of important parameters from the previous session, which is left frozen during incremental tuning with the rest of the paramcters. A common problem of this second group of methods is that adaptability comes at the cost of stability, That is, the model's performance in the old classes det.riiorates as it learns novel classes. Overall, there is a lack of balance between the stability and adaptability in existing FSCIL methods, which we aim to improve with our bag of tricks.
3 Method
3.1 Overview
In FSCIL, model is trained across T consecutive sessions, with each session introducing novel classes for the model to learn. Training data for each session t T is labelled, Dirain [[x,, w)} where 2; and Vi are the i-th sample and the corresponding label. In FSCIL, only the base scssion (first session) contains a sufficient amount of samples for effective training. Subsequent incremental sessions contain only a few samples per class, typically organized in an N-way K-shot format, containing K training samples for each of the N classes. By definition, each session exclusively contains samples from novel classos, meaning that the label space for each session (C*) is mutually exclusive with others. The performance of method is evaluatod after each session the test set Dt Peest: which contains samples from all classes encountered far. In our
3

---

Published in Transactions on Machine Learning Research (09/2024)
framework, include pre-training stage before the base training. Overall, the framework is divided into three training stages: pre-training, base training, and incremental training.
3.2 Baseline
We consider the incremental frozen framework as our baseline due to its proven effectiveness (Zhang et al, 2021 Shi et al. 2021 Song ct al. 2023) in addressing the data-scarce incremental learning scenario of FSCIL. In this framework, the model ((x) is trained only on the base session (Di using & standard cross-entropy loss, Leee (o(x;), y)- The model d(x) consists of an encoder, fo(x: R and classifier head W Rd The prediction can therefore be expressed )(x:) = W fo(z)) After training on the base session, the encoder fxx) remains frozen during the incremental sessions. To classify novel classes, the classifier W is expanded with the classífier weight for novel classes parameterized by the prototype of each class. A prototype is the average of the embeddings of all samples belonging to that class, w -E fo(xe,))
3.3 Stability Tricks
Incremental classes
Stability tricks in our frame- work revolve around the idea that. better separation of 'base' classes ensures improved stabil- ity in learning new classes the incremental sessions (Song
classes
et al. 2023). As illustrated in Figure I better separation of
Training without stability tricks
Training with stability tricks
base classes in the embedding space allows the novel classes to be placed in the embedding space without interfering with existing classes. This involves increasing the distance between classes (inter- class distance) while reducing the distance between samples within the same class (intra-class distance). This approach is also known as forward compatibility in FSCIL literature. Accordingly, incorporate three techniques that can effectively promote better stability: training with supervised contrastive loss, pre-assigning prototypes, and including pseudo-classes.
The intuition behind stability tricks. Better separation of base classes ensures stability in incremental learning.
Figure 1:
Supervised Contrastive Loss. While most of the existing literature on FSCIL use the standard cross- entropy loss for learning during the base session, some prior works (Song et al. 2023 have demonstrated that cross-entropy does not effectively separate classes in the embedding space. Some studies have indicated that the supervised contrastive loss (SupCon) (Khosla et al. 2020) exhibits better scparability in the embedding space. SupCon is a variation of the popular contrastive loss (Chen et al. 2020), which additionally includes class labels to guide representation learning in a supervised manner. Specifically, SupCon learns representations by pulling the samples (and their augmentations) the same class closer the embedding space while pushing the samples of different classes apart, resulting in a more scparable embodding space than cross-entropy. In other words, SupCon forces the representation of each sample be close to its corresponding class prototype (center of embedding), while pushing the prototypes away from one another. For batch of labelled samples {(x;, v:) b the SupCon loss can be represented as:
N
expl2 - 23/T) ΣN k= ezzpe =///T)
Suu
Σ
[iAss 1 (y-=w)) log
(1)
where = = fo(x), N is the batch-size, and Ny. is the number of positive samples from class yr. The indicator function denoted by 1y yields value of when indices and correspond to instances of the same class, 7 is a temperature parameter.
Pre-assigning Prototypes. In SupCon, prototypes are learned along with the optinization of the model. However, it was shown in Yang et al. (022 that 'pre-assigning' prototypes that are maximally separated
4

---

Published in Transactions on Machine Learning Research (09/2024)
from one another ensures improved separation in the embedding space. In Yang et al. (2022), the maximally separated prototypes were defined as simplex Equiangular Tight Frame (ETF) Paapynn et al. 2020) P, a geometric arrangement of K vectors in d-dimensional Euclidean space (Yang et al. 2022). Each vector in this space has an Euclidean norm of 1, with any pair of distinct vectors yielding an inner product of -1/(K - 1). This specific inner product value corresponds to the largest, possible angle between any two vectors in this space. Our implementation of the prototype pre-assignment slightly differs from (Yang et al. 2022) two ways. First, we do not rely solely on the cross-entropy loss but instead incorporate the SupCon loss introduced earlier. Second, we do not assign the ETF prototypes randomly at the very beginning of training. Instead, we train the encoder with the SupCon for few epochs and take advantage of the prototypes naturally formed by SupCon. Then, for each class, we assign the ETF prototype that is most closely aligned with its naturally formed protolype, and train the encoder to minimize the distance belween the learned and pre-assigned prototype as:
C LETF
Σ Pe we)? C
1
(2)
where, Pe and we are the assigned and learned prototypes of class and C is the total number of classes in the base session.
Including Pseudo-classes. Previous studies such as FACT (Zhou et al. 2022a) and SAVC (Song al. 2023) introduced the concept of integrating pseudo-classes during the base session to serve as placeholders in the embedding space for novel classes. In SAVC, the pscudo-classes were generated by a pre-defined transformation, which was considered a more fine-grained variant of the original classes. In our work, we lake a similar approach of including pseudo-classes from (Lee et al. 2020), , which employs hard augmentations to transform the semantics of the sample and consider it as a psoudo-class. Let, JF be a set of pre-defined (hard) augmentations for pseudo-class formation, and Ve. be a sample of class c. With the pseudo-class trick, consider the transformation of the image F(Fo,)) as instance of pseudo-class (C" M + e), where C is the total nunber of classes in base session, and M is the pseudo-class multiplication factor. Such pscudo-classes can be seen as a fine-gran fine-grained class derived from the original class. In our work, we use M= = 2, which doubles the total number of classes (including pseudo-classes) during base training. This trick does not. include any new loss function.
3.4 Adaptability Tricks
While stability tricks help the model retain the knowledge of base classes, they offer limited adaptability for the model to learn novel classes effoctively. Independent performance evaluation of base and novel classes at the end of training demonstrates that the performance of novel classes is substantially inferior to that of base classes (Song et al.] 2023). Consequently, the overall performance the end of training is predominantly influenced by the performance of the base classes. This underscores the need for FSCIL methods to euhance the model's adaptability on incremental sessions to improve the model's performance on novel classes. In this section, we discuss two tricks that provide more adaptability for the model: incremental fine-tuning and SubNet tuning, which we combinedly denote as Incremental SubNet Tuning.
Incremental SubNet Tuning. Fine-tuning is common practice in machine learning literature to lune a pre-trained model for a new task or setting. It is also widely used technique across conventional continual learning literature, predominantly with rehearsal-based techniques. However, fine-tuning in the context of FSCIL requires careful consideration of the training setup, since no data from the previous session is available for rehearsal during the current session. Consequently, tuning can cause catastrrophic forgetting of already learned knowledge. In our framework, we adopt the fine-tuning concept of Song et al. (2023), which utilizes a small learning rate tune certain portions of the pre-trained encoder, while keeping the rest of the encoder frozen. Specifically, we freeze the shallow layers of the network since shallower layers are known to capture domain-invariant features, whereas the deeper layers learn more fine-grained foatures.
While such an incremental fine-tuning approach provides more adaptability for learning novel classes, it may result in decreased stability. To deal with this issue, we combine this adaptability trick with the concept of
5

---

Published in Transactions on Machine Learning Research (09/2024)
SubNet tuning, which maintains the adaptability of incremental fine-tuning without substantial drop in stability. The concept is inspired by the Lottery-ticket hypothesis (Frankle & Carbin 2018), which states that there exists a subnetwork of dense network that performs as well as the whole network. In [Kan et al. (2023), this concept is utilized in the context of FSCIL to find sub-network the trained encoder that performs on par with the whole nerk on the base classes. Adopting this into our framework, after the training of the encoder (fo) on the base session, we find subset of network parameter that performs as well the whole network on the base gession.Speccficalll we follow Kang et al. (2023) extract SubNet. (subnetwork) mask m
m
min ME 10
(1:). (ffo(t)).))
(3)
71
where O an element-size dot-product operation. Here, L is the loss function for training the model on the base session, and m= is the optimal binary mask with the same size the network, for which the performance of the masked SubNet is comparable to that of the original network. During the incremental session, we freeze the sub-network to ensure the performance of the basc session and fine-tune the remaining parameters for learning the novel classes.
3.5 Training Tricks
Building upon the principles of stability and adaptability tricks discussed earlier, we introduce a set of training techniques that can further enhance overall performance without compromising either stability adaptability. These tricks include adding pre-training step and including an additional learning signal.
Additional Pre-training Step. Existing research in this field indicates that self-supervised praiiignn followed by supervised fine-tuning, consistently outperforms fully supervised training, particularly in scenarios with limited training data. In the context of FSCIL where data scarcity is significant challenge, leveraging self-supervised pre-training step has the potential to provide substantial benefits. As result, we introduce a contrastive self-supervised pre-training step before the training of the base session. The contrastive self-supervised loss is similar to the SupCon loss introduced earlier, except no label information is utilized. Accordingly, the contrastive pre-training loss can be represented as:
1
26 log
exp(zi [x(1)/T) Inzperp(Zz, Zx/T)
Leon
(4)
2b
26 2k=1
where, xtii is the index of the second augmented sample, and 1 is an indicator function that returns when A is not equal to i, and otherwise. is temperature parameter, and b is the batch size.
Including Additional Learning Signal. Following the intuition from the previous trick, we include another self-supervised learning signal, but this time while training on the base session rather than as a separate step. Existing literature on other data-scarce scenarios, such as semi-supervised learning, has shown that adding pre-text task along with the supervised learning helps the model learn better representation without overfitting to the small labelled set (Berthelot et al.) 2019 Roy & Etemaad 2023). To this end, we include rotation prediction task (Gidaris et al. 2018) that is shown to perform well with supervised learning (Berthelot et al.] ] Here, the basic idea is to apply rotation operation on the input image, and the task is to predict the amount of rotation applied to the image. In practice, a rotation module randomly samples one the following rotations and applies it to the image: 0%, 97,,0, 180", 270%. As a result, the rotation prediction task can be viewed as way classification task, represented as:
Lra H(r, Po. (r) Rotale((z:
(5)
Here, is the input image, Po. is the encoder that predicts the rotation, and H. is the cross-cntropy loss.

---

Published in Transactions on Machine Learning Research (09/2024)
4 Experiments and Results
4.1 Datasets and Implementation Details
Following the established protocol in the FSCIL literature, we conduct our experiments on three popular datasets: C1FAR100 (Krizhevsky et al. 2009), minimageNet (Rnssakovsky et al. 2015) and CUB200 (Wah et al. 2011). To ensure a fair comparison with prior works (Tao et al., 2020 Chi et al. 2022 on FSCIL, we use the same encoder (ResNct-18), and data split across the training sessions. Specifically, for CIFAR-100 use 60 classes for the base session and 40 classes for the incremental sessions. The conducted on a 5-way, 5-shot setting. In the case of CUB-200, we allocate 100 classes for the base session and another 100 classes for the incremental sessions, each containing ten classes (10-way, 5-shot). Further details on implementation and rrper-parameters are presented in A Appendix ST
and minillageNee,, incremental learning experiments
4.2 Evaluation Protocols
Following the standard evaluation protocol in FSCIL literature (Tao et al. 2020; Song et al. 2023 Kang ct al.] 2023), we report the model's accuracy after each incremental session. To further understand the properties the learned representations we investigate inter-class distance, intra-class distance, and class separation. Following, we define these properties.
Inter-class distance is the distance between the prototypes of any two classes in the embedding space. Given, two class prototypes wy and wy for classes i and j, we compute their inter-class distance as:
diter
cosine(ws, wy),
where cosine() is the cosine similarity between two vectors.
Intra-class distance indicates the average distance from a prototype to the samples belonging to that class. For class k; we compute the intra-class distance as:
cosine(zi, wk),
intra
Tik
where ny is the number of samples belonging to class k and 2; is the embedding of the 7th sample.
Class-separation determines how well the samples from one class are separated from other classes in the embedding space. For a dataset with C classes, class separation can be represented as: - dwithin dzotal- Here, dwithin is the average distance between samples the same classes, while deotal is the average distance between samples the embedding space. Accordingly, they are formulated as:
n.
The cosine(zo, C.n²
dwittin
(8)
and
Rg
nd
cosine(ze,s 2d.5)
diotal
(9)
C Th Thd
4.3
Main Results
In this section, we present the main results of our tricks by cumulatively adding them to the bascline. The results are presented in Table [ which include the category of the tricks, along with the stages at which the tricks are applied and the performance on CIFAR-100, CUB-200 and minilN. As outlined in Section 3.2 we adopt the incremental-frozen framework as the bascline for our study. As shown in Table [ the accuracy of this baseline for CIFAR-100, CUB-200, and minilmageNet datasets are 43.77%, 59.88%, and 45. 45.08%, respectively. To ensure the best performance for the baseline, we perform extensive hyper-parameter study
7

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Impact different tricks on the baseline for all datasets. Here, the tricks are added cumulatively to the baseline, and the 'Stage' indicates which stage of training the tricks are applied.
Category
Trick
Stage
CIFAR-100
CUB-200
minilN
Baseline
43.77
59.88
45.08
+ SupCon
50.16
60.38
48,90
Stability
ETF vector
Base
51.10
60.74
49.73
Pseudo-classes
Base
51.65
62.18
54.79
Adaptability
+ Inc. SubNet tuning
Incremental
58.12
63.10
57.85
Pre-training
Pre-training
58.34
63.30
58.01
Training
+ Additional signal
Base
58.55
63.60
59.11
Macciie
0.5
Maxchine
-Satee +baude
Camu Camuaaber aite Prghabllt
-Pheade
Camu Cumurattro Probaahltt
<Pyzuta Clases
Class supuratien
Accuraly
16
18
6.5
4.5
0.20
0.58
00
Wond
Intansclass Distanice
Intra-class Dnstance
Glaooss
Casses
(b)
(c)
(d)
Figure 2: Properties of stability tricks CIFAR-100. (a) Presents inter-class distance (the distance between class prototypes), which we aim to maximize for better stability during incremental training; (b) depicts the intra-class distance (the average distance of samples from the corresponding prototypes), which we aim to minimize for better stability; (c) presents the class separation degree (the overall separability of classes ranging between 0 and 1), which we aim to maximize; (d) presents the accuracy of Base, Novel, and Total classes.
presented in Appendix $3.1 As we later compare our results with prior works (Section 4.5) we observe that this baseline outperforms many of the prior studies, showing the robustness of our baseline.
Stability Tricks. Next, discuss the results of including stability tricks into the baseline, beginning with the addition of SupCon loss. The results from this study show substantial improvement for all datasets (Table D obtaining accuracies of 50.16%, 60.38%, and 48.90%, for CIFAR-100, CUB-200, and minilmageNeet respectively. As discussed earlier, a key element for ensuring high stability in FSCIL is to ensure increased separability in the embedding space (increased inter-class distance and reduced intra-class dispersion). Figures 2a and [2b] we plot the cumulative probability of inter-class and ratralsssss distances (defined
Section 4.2 for different tricks. We plot the cumulative probability of inter-class and intra-class distance (Song
et al. 2023 instcad of the average since the average distance be mislcading due to the presence of outliers, whereas the cumulative probability provides more robust and nuanced representation of the distribution of distances. As we observe from Figure 2a adding SupCon provides a large increase in the inter-class distance compared to training with cross-entropy loss in the baseline. At the same time, SupCon greatly reduces the intra-class distance compared to the cross-entron ceass-entrop--baae based baseline (illustrated in Figure 2bb. However, our findings diverge from those of [Song et al. (0223) which indicated that while supervised contrastive loss effectively reduces intra-class distances, it unexpectedly leads a reduction inter-class distances as well. The divergence in our findings may be attributed to implementation specifics. For example, SAVC utilized multi-crop augmentation, which do not use. Also, we adopted different set of determined through our study (Appendix S3.2).
We also show the overall class separation degree for different tricks in Figure 2c Class separation degree refers to the degree of distinctiveness or separability between different categories within the embedding space, which can be measured as defined in Scction 4.2 It typically ranges between and 1, with higher values indicating clearer boundaries and better separability between classes. As we observe, adding SupCon provides better class separation for both base and novel classes. Overall, the higher separation in the embedding space
8

---

Published in Transactions on Machine Learning Research (09/2024)
contributes to the large improvement in the overall accuracy. Analyzing the performance of both base and novel classes in Figure 2d we find improvements for both, with a 5.4% boost in base class accuracy and a 7.8% increase in novel class accuracy.
Next, we include our second stability trick pre- assigning ETF vectors as prototypes. As we see in Figure 2a] this trick further increases inter-class dis- tance (Figure 2ap over the previous trick. Although this trick does not further reduce the intra-class dis- tance (Figure 2bj, it increases the class scparation of novel classes (Figure 2ch, resulting in 2.1% improve- ment novel classes. Overall, including this trick increases the final performance to 51.10%, 60.74%, and 49. 49.73%, respectively.
Eno
50
Baseline
Stabilly Tricks
75-
+Stabiliy Tricks
+nc. SubNet, Tuning
+inc. SubNet Turing
80
14.
Accuracy
35
214
50-
To1ai
Base Clesses
Novel
2
Session
(a.
(b)
Finally, adding pseudo classes causes a further reduc- tion in the intra-class distance since it requires fitting twice the number of classes in the same amount of space. Although this trick reduces the inter-class distance, it provides an increase in the novel class separation. This results in a 1.20% improvement in novel classes and 0.6% in the final performance. Overall, find an improved accuracy of 51.21% %, 62.27%, and 55.82% for CIFAR-100, CUB-200, and minilmageNet, respectively. Here, we find the largest improvement of 5.06% on the minilmageNet dataset.
(c)
(d)
Figure 3: Properties of adaptability tricks CIFAR- 100. (a) Presents accuracy of Base, Novel, and Total classes; (b) presents the total accuracies after each session, which we aim to maximize; (c) and (d) depict t- SNE visualizations for stability and adaptability tricks, where incorporating adaptability tricks shows more separation. Here, 0-4 are base classes, and 5-6 are novel classes.
Adaptability Tricks As discussed earlier, stability tricks do not provide sufficient adaptability for the model perform well on novel classes. This is evident from the accuracy base and novel classes in Figure 2d where we see 49.6% difference in the performance on base and novel classes. Specifically, the performance on base and novel classes are 71.5% and 21.9%, respectively. This indicates that the model struggles to effectively learn the novel classes, demonstrating the need for adaptability tricks.
Our adaptability trick with Incremental Sub- Net Tuning provides adaptability by means of fine-tuning sub-network on the encoder in incremental sessions. As shown in Figure 3a, this trick improves accuracy on novel classes by 11.1%. We also show the accuracy after each scssion in Figure Bb From this figure, we observe comparable accuracies to that of stability tricks in the earlier sessions. However, Incremental SubNet Tuning shows better per- formance in later sessions when the number of novel classes increases. This again shows the importance of the adaptability trick for learning novel classes. Overall, this trick pro- vides an improved accuracy of 18%1 63.1% 63.1%, and 57.85% on the C00,R-100, CUB-200, and minilmageNet.
(a) Baseline.
(b) bag of tricks.
Figure 4: Confusion matrices for the baseline and our bag of tricks. The bascline performs well on the base session, but performance drops for novel classes. Our framework shows improved performance for both base and novel classes
9

---

Published in Transactions on Machine Learning Research (09/2024)
In Figure 3c we visualize t-SNE representation of the learned embeddings (for randomly selected subset of classes) for the stability tricks and the adaptability tricks. As observed in these figures, stability tricks alone provide sufficient space for the integration of novel classes (5 and 6), although it results in a slight overlap with base classes. In contrast, the incorporation of adaptability tricks creates more separable embeddings.
Training Tricks. Finally, we discuss the results of adding the training tricks. First, we investigate the impact of including the pre-training trick. With the pre-training trick with self-supervised learning, the performance the three datasets is increased to 58.34%, 63.30%, and 58.01%, respectively. Finally, we observe additional improvements with our last training trick of incorporating the additional learning signal, resulting in accuracies of 58.55%, 63.60%, and 59.11%, on CIFAR-100, CUB-200, and minilmageNet, respectively. To provide an overall evaluation, we present the confusion matrices for both the baseline and the complete bag of tricks in Figure For the baseline, noticeable dark points (correct predictions) are observed along the diagonal for the base classes (first 60 classes), but these become less prominent for the incremental classes. With the bag of tricks, we obscrve significant improvements for all classes, including novel ones.
4.4 Ablation Study
In this section, we present an ablation study on the stability, adaptability and training tricks on the CIFAR-100 dataset. The results of this study are presented in Table 2 It. is evident from the table that stability tricks have the highest individual impact removing which results in an 11.2% drop in performance. Furthermore, removing stability tricks with any of the other tricks drastically drops the performance, resulting in a final accuracy worse than the baseline. This is caused by the fact that without stability tricks, adaptability tricks alone overfit the new classes, and training tricks alone do not learn generalized features. We find the second important component be the adaptabil- ity tricks, which result in 2.7% drop in performance when removed.
Table 2: Ablation study.
Stab. Adap. Train. Acc
the performance of the model,
58.55
X
58.12
X
V
55.85
X
J
47.35
51.65
X
X
37.34
X
40.19
X
X
X
43.77
Performance with Larger Encoders. For all the experiments in this work, we followed the same problem setup and encoders used by all the prior works on FSCIL (Song et al., 2023 Kang et al. 2023). Our focus in this paper is to improve the stability, adaptability, and overall performance of FSCIL while following the same benchmarks and problem setups, as well as encoder backbones, to ensure fair comparison. However, the tricks in our framework are independent of the encoder's 's choice and can be easily adapted to any encoder, including ViT. To investigate the performance of our framework on larger encoders, in Table we show the results on ResNet-30, VIT-B/16 /16, and pre-trained (on ImageNet) VIT-B/16, VIT-B/32, and ViT-L,/14. The results from this study show that the proposed tricks not only transfer to larger encoders and ViTs, but the performance gains are even larger than those of smaller networks like ResNet-20. For instance, ResNet-50 shows a 20. 20.74% improvement compared to 14.78% in ResNet-20. Similarly, VTT-B/16 with pre-trained and randomily ialized encoders shows 19.58% and 15. 75% improvements over the baseline. Additionally, pre-trained VTT-B/32 3/2 and ViT-L/14 show 19.57% and 19.54% improvements over the baseline. This is due to the fact that without the stability and adaptability tricks from our framework, a larger encoder with more parameters is more prone overfitting, making it difficult to learn incremental classes in a few-shot learning setting. With our proposed framework, the larger encoders show a large improvement in performance.
Table 3: Performance of prior works and our framework for different encoder sizes on CIFAR-100 dataset.
Accuracy Ours
Encoder
Baseline
ResNet-20
43.77
58.55
ResNet-50
38.01
58.75
ViT-B/16
32.14
47.89
ViT-B/16 (pre-trained)
50.14
69.72
ViT-B/32 (pre-trained)
50.01
69.58
VIT-L/14 (pre-trained)
51.55
71.09
4.5 Comparison to Prior Works
Finally, we compare the results of our bag of tricks with the prior works on CIFAR-100 in Table 4 and present the results for CUB-200 and minilmageNet in Appendix Tables ST and 52 The tables present the
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table Comparison to prior works across the base and incremental sessions on CIFAR-100 for 5-way, 5-shot setting. The results for CUB-200 and minilmageNeet are presented in the Appendix.
Ace. in each session (%) f
Method
5
Baseline
70.05
65.37
60.59
56.72
53.38
50.53
48.07
45.87
43.77
Rebalancing (Hou et al.
2019
61.31
47.80
39.31
31.91
25.68
21.35
18.67
17.24
14.17
iCaRL (Rebuffi et al. 2017
61.31
46.32
42.94
37.63
30.49
24.00
20.89
18,80
17.21
TOPIC Tao et al. 2020
61.31
50.09
45.17
41.16
37.48
35.52
32.19
29.46
24.42
IDLVQ-C Chen & Lee
2020
64.77
59.87
55.93
52.62
49.88
47.55
44.83
43.14
41.84
FSLL (Mazumder et al.
2021
66.48
61.75
58.16
54.16
51.10
48.53
46.54
44.20
42.28
FSLL [Mazumder
2021
68.85
63.14
59.24
55.23
52.24
49.65
47.74
45.23
43.92
F2M Shi et al. [2021]
67.28
63.80
60.38
57.06
54.08
51.39
48.82
46,58
44.65
CEC Zhang al. 2021
73.07
68.88
65.26
61.19
58.09
55.57
53.22
51.34
49.14
MetaFSCII
(Chi et all 2022
74.50
70.10
66.84
62.77
59.48
56.52
54.36
52.56
49.97
CLOM
Zou et al. 2022
74.20
69.83
66.17
62.39
59.26
56.48
54.36
52.16
50.25
C-
C-FSCIL
(eesshe et al
2022
77.47
72.40
67.47
63.25
59.84
56.95
54.42
52.47
50.47
LIMIT
Zhou al. 2022 b
73.81
72.09
67.87
63.89
60.70
57.77
55.67
53.52
51.23
FACT
Zhou al.
2022a
74.60
72.09
67.56
63.52
61.38
58.36
56.28
54.24
52.10
SAVC
Song el al.
2023
79.85
73.70
69.37
65.28
61.91
59.27
57.24
54.97
53.12
ALICE
Peng
2022a
79.00
70.50
67.10
63.40
61.20
59,20
58.10
56.30
54.10
NoftNet
Kang et al.
2023
79.88
75.54
71.64
67.47
64.45
61.09
59.07
57.29
55.33
Ours
80.25
77.20
75.09
70.82
67.83
64.86
62.73
60.52
58.55
"We reproduced the results for SAVC the original paper does not provide the exact values for each session.
CIFAR-100
CUB200
minilmageNet
80-
70
70
Accuracy (96)
60-
60
50-
Accuracy
80
30
6
30
20
30
Session
Session
Ours
ICaRL
FACT
MetaFSCIL
Rebalancing
TOPIC
LIM
Softiee
SAVC
Figure 5: Comparison to prior works across CIFAR-100, CUB-200, and minilmageNNet datasets, demonstrating that our solution outperforms prior works.
results for the base session and the overall performance across incremental sessions. The first row in this table presents the results for our baseline. Among prior works, SoftNet held the previous state-of- state-of-the-art the-art for CIFAR-100 with final accuracy of 55.33%. Our framework achieves performance of 58.55%, marking 3.22% improvement over the current slate-of-the-arl, In Figure 5 plot the accuracy over the incremental sessions for CIFAR-100, CUB-200, and miniImageNet. For CUB-200, the previous state-of-the-art was held by SAVC with an accuracy of 62.50%, which our framework outperforms with an accuracy of 63.66 63.60%. Finally, the state-of-the-ee the-rrrtt on minilmageNet was also held by SAVC, with an accuracy of 57.11%, which our method outperforms by 2.0%, achieving an accuracy 19.11%.
Performance Across Different Shots. Following prior works in FSCIL (Zou et al. 2022a Tao et al. 2020), we report the main results in 5-shot selting. Nonetheless, We also investigate the performance on I-shot and 2-shot settings to evaluate how the model performs in data-scarce settings and compare the performance with two of the previous statc-of-the-art methods, SAVC and SoftNet. We also investigate the 10-shot setting to determine how an increase in data impacts performance. As seen from Figure D. in 1-shot learning, proposed framework outperforms prior works by larger margin than the 5-shot setting. This
11

---

Published in Transactions on Machine Learning Research (09/2024)
Table 5: Performance on the ImageNet-II dataset. We use 500 classes as base sessions and perform 50-way,
10-shot incremental learning.
Acc.
in each session (%) 1
Method
0
2
3
5
6
7
8
10
SAVC (Song et al. 2023)
37.74
35.94
34.45
33.16 31 30.93
29.80
28.83
28.1
27.37
26.12
NoftNet (Kang al.
2023
36.21
31.23
32.35
30.53
28.71
27.15
25.68
21.31
23.16
22.19
21.21
Ours
39.34
37.86
36.49
35.23
34.30
33.83
32.29
31.21
30.77
29.34
28.25
further shows the effectiveness of the bag of tricks in data-scarce settings, More experiments on 1-, 2-, and 10-shot setlings are discussed in Appendix S3.55
Performance ImageNet-1KK So far we have report the main results of our approach on, CIFAR-100, CUB-200, and minilmageNee, which are commonly used by prior works in this area. However, these datasets are relatively small terms the number of classes. To better understand the scalability of FSCIL in terms of the number of classes, we conduct an experiment on the ImageNet-1I (Russakovsky et al. 2009) dataset, which contains 1000 classes. For this study, we consider 500 randomly selected classes the base classes and report
80
Ours SeftNet
75
70-
Accuracy (%)
65
60
55
the results for 50-way, 10-shot incremental learning setting. The results of this study with our method and its comparison to prior methods are presented in Table 5 As seen from the results, our proposed framework outperforms prior works on the ImageNet-11K dataset by considerable margin. More specifically, our framework achieves 7.04% and 2.13% improvement over SoftNet and SAVC, respectively.
50
45
3
8
Session
Figure 6: Performance for different shots. Solid and dotted lines repre- sent 5- and 1-shot performances.
4.6 Time Complexity
In Table we discuss the time complexity of our framework using a single Nvidia RTX 2080 GPU
Table 6: Comparison of computational complexity with a
batch-size of 64.
in comparison to SAVC (Song et al. 2023 To
this end, report the throughput in frames per second (FPS) and the training time. During the training phase on the base session, the through- puts for SAVC and our framework are 630 FPS and 450 FPS, respectively. Nonetheless, during incremental training and inference, our frame- work with ResNet-18 is fast as SAVC. Con- sequently, once trained, our framework can be reliably deployed in real-world applications with the same inference time as previous saate-of-the-art while achieving enhanced performance.
Train
Method
Acc.
Infer.
Runtime.
base
inc.
SAVC
53.12
810
950
2150
6.1h
Ours (ResNet-18)
58
630
935
2150
6.8h
Ours (ResNet-50)
58.75
450
710
1800
8.1h
4.7 Discussion
According to the setup adopted in the literature for FSCIL [Sog et al. (203)); Kang et al. (2023) each of the new classes in the in- far. Therefore, incremental classes are generally not exactly identical to the base classes. However, some of the incremental classes can be similar to the base classes. For instance, in CIFAR-100, 'Orchid' class of flower that part of the base class, while 'Sun-flower' is another flower that is part of the incremental class. A few similar pairs of examples of base classes with similarities to incremental classes
Table 7: Performance comparison for base and incremental classes with high semantic similarity.
cremental session is different from the classes
Method
Base
Incremental
Baseline Ours
65.3%
19.7%
74.5%
30.4%
12

---

Published in Transactions on Machine Learning Research (09/2024)
are "Tion'-'Tiaar, "Bus'''Streetear" and "Shark'-Dophhin' respectively. To understand the effectiveness of the stability on closely related classes, we report the average accuracy on the above-mentioned base and incremental classes with and without our tricks below in Table 7 As we find from this experiment, even for semantically similar incremental classes, our proposed framework is capable of improving the performance of both the base classes (improved stability) and the incremental ones (improved adaptability).
5 Conclusion and Future Work
We present a bag of tricks framework that combines six effective tricks three distinct categories to improve the stability, adaptability, and overall performance of FSCIL. Stability tricks improve the separation among learned classes to facilitate the learning of new ones, resulting in large improvements on both the base and total classes. Adaptability tricks improve the performance on novel classes by providing more learning capability during incremental sessions. Finally, training tricks provide an additional boost to the final performance. While we did not introduce any new tricks, our contribution in this work lies in developing a framework that combines a set of tricks that jointly improve both stability and adaptability, which, as discussed in the literature review, is a difficult task since improved stability or adaptability often hampers the other aspect. Additionally, We provide extensive analysis of these tricks for a better understanding of their impact beyond the final accuracy, including the impact on stability and adaptability, class separation in the learned embedding space, and performance improvement base and novel classes. Furthermore, detailed study and comparison to prior works on different aspects that are not explored in the cxisting literature, including low-shot (1-, 2-shot) performance, results on a dataset with a large number of classes (InageNet- 1k), performance on larger encoders (e.g. ResNet-50, ResNet-101), 101), performance on fine-tuning pre-trained foundation models (ViT-B/16, ViT-B/32, and ViT-H/14), performance when new-class has similarity to a base classes (included in revised manuscript). We believe the unified framework and the extensive set of experiments will add value the further development of the challenging setting of FSCIL.
present
Though our framework provides adequate adaptability, the performance of the novel classes is still relatively lower compared to the base classes. While this phenomenon has also been widely reported in prior works, our framework was successfully able to reduce the gap between the performance of the base and incremental class from 55% in the literature to 45.2%. However, the performance the incremental class is still considerably lower than the base class, which can be a potentially interesting direction of inquiry for future work.
References
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR, 2019.
Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, pp. 233-248, 2018.
Kuilin Chen and Chi-Guhn Lee. Incremental few-shot learning via vector quantization in deep embedded space. In ICLR, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, pp. 1597 1607, 2020.
Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Semantic-aware knowledge distillation for few-shot class-incremental learning. In CVPR, pp. 2534 2543, 2021a.
Ali Cheraghian, Shafin Rahman, Samcora Ramasinghe, Pengfei Fang, Christian Simon, Lars Potersson, and Mehrtash Harandi. Senthesized feature based few-shot class-incremental learning on mixture of subspaces. In ICCV, pp. 8661-8677 2021b.
Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: meta-learning approach for few-shot class incremental learning. In CVPR, pp. 14166-14175, 2022.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong, Few-shot class-incremental learning via relation knowledge distillation. In AAAT, pp. 1255-1263, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Constrained few-shot class-incremental learning, In CVPR, pp. 9057 9067, 2022.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, pp. 831-839, 2019.
Haeyong Kang, Jachong Yoon, Sultan Rizky Hikmawan Madjid, Sung Ju Hwang, and Chang D Yoo. On the soft-subnetwork for few-shot class incremental learning. In ICLR, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NewrIPS, pp- 18661-18673, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Self-supervised label augmentation via input transformations. In TCML, pp. 5714-5724, 2020.
Zhizhong Li and Derek Hoiem. Learning without. forgelting. TPAMI, 40(12):2935 2947, 2017.
Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class-incremental learning via entropy-regularian data-free replay. In ECCV, pp. 146-162, 2022.
Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In ICPR, pp. 2262 22222 2018.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. TPAMI, 45(5):5513 5533, 2022.
Pratik Mazumder, Pravendra Singh, and Piyush Rai. Few-shot lifelong learning. In AAAI, pp. 2337 2345, 2021.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. National Academy of Sciences, 117(40):24652 24663, 2020.
Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C Lovell. Few-shot class-incremental learning from an open-set perspective. In ECCV, pp. 382-397, 2022a.
Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C Lovell. Few-shot class-incremental Icarning from an open-set perspective. In ECCV, pp. 382 227, 2022b.
Aiirestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pp. 2001-2010, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NewrIPS, 2019.
Shuvendu Roy and Ali Etemad. Scaling up semi-supervised learning with unconstrained unlabelled data. arXiv preprint ar Xtx:23330 01 01222, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Alexander C Berg, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. pp. 248-255, 2009.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. In IJC,, volume 115, pp. 211 252, 2015.
Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. In NeurIPS, pp. 6747-6761, 2021.
Yujun Shi, Kuangqi Zhou, Jian Liang, Zihang Jiang, Jiashi Feng, Philip HS Torr, Song Bai, and Vincent YF Tan. Mimicking the oracle: an initial phase decorrclation approach for class incremental learning. In CVPR, pp. 16722 16731, 2022.
Zeyin Song, Yifan Zhao, Yujun Shi, Peixi Peng, Li Yuan, and Yonghong Tian. Learning with fantasy: Semantic-aware virtual contrastive constraint for few-shot class-nnermmentl learning. In CVPR, pp. 24183-2418 2023.
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong, Few-shot class-ineremental learning. In IEEE TPAMI, pp. 12183 12192, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd birds-200-2011 dalaset. 2011.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired feasure- alignment for few-shot inssemmncreltt learning. In ICLR, 2022.
Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In CVPR, pp. 12455 124644, 2021.
Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, and Xi Li. Mgsvf: Multi-grained slow vs. fast framework for few-shot class-incremental learning. TPAMI, 2021.
Linglan Zhao, Jing Lu, Yunlu Xu, Zhanzhan Cheng, Dashan Guo, Yi Niu, and Xiangzhong Fang. Few-shot class-incremental learning via class-aware bilateral distillation. In CVPR, pp. 11838-11847, 2023.
Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In CVPR, pp. 9046 9056, 2022a.
Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xic, Shiliang Pu, and De-Chuan Zhan. Few-shot class-incremental learning by sampling multi-phase tasks. TPAMI, 2022b.
Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. In NewrIPS, pp. 14306 14306-14318, 2021a.
Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zue Zha. Self-promoted prototype refinement for 2021b.
few-shot class-incremental learning. In CVPR, pp.
Yixiong Zou, Shanghang Zhang, Yuhua Li, and Ruixuan Li. Margin- based few-shot class- incrementtt learning with class-level overfitting mitigation. NearIPS, pp. 77267-27277 2022.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Supplementary Material
In this section, we present more details on the implementation and discuss additional experimental results. Following is overview of the organization of the supplementary material.
ST
Datasets and Implementation Details
$2
Comparison to SOTA
Additional Results
53.1
Baseline
53.2 2
Stability Tricks
53.3
Adaptability Tricks
$3.4 4
Training Tricks
$3.5
Performance on Different Shots
S1 Datasets and Implementation Details
Following previous works (Tao al. 2020; Song et al. 2023 we evaluate our framework on three popular datasets: CIFAR100 (Krizhevsky et al. 2009), minilmageNeet [Russakovsky et al., 2015), and CUB200 (Wah) et al. 2011). CIFAR-100 is dalaset of T00 classes, where we use 60 classes (Following (Tao et al. 2020 in the base session and the remaining 40 classes in the incremental sessions. Each incremental session is formulated as a 5-way, 5-shot problem. CUB-200 is a dataset with 200 fine-grained categories. For this dataset, we use 100 classes (following (Tao et al. 2020)) in the base session and the remaining 100 classes in the incremental sessions, with 10 classes each session. Finally, miniImageNet is a subset of the popular Iet dataset that contains 100 classes. For this dataset, we 60 classes (following (Tao et al., 2020 for the base session and 5 classes per incremental session over sessions.
For the encoder, we use ResNet-18 by default for all datasets, similar to (Tao et al. 2020 Kang et al. 2023 We train the model with an SGD optimizer, momentum of 0.9, and a batch size of 64. The learning rate is set to 0.1 for CIFAR-100 100 and minilmageNet and 0.001 for CUB-200. For all experiments, the model is trained on an Nvidia RTX 2080 GPU.
S2 Comparison to State-of-the-art
In this section, we present the results for CUB-200 and minilmage.Net datasets and their comparison to prior works. As we observe from Table SS] our framework outperforms all existing works on CUB-200 and sots a state-of-the-at of 63.60%. Similarly, for minilmageNet in Table S2 our framework performs better than all prior works, showing a new best accuracy of 59.11%.
S3
Additional Results
S3.1
Baseline
In this section we discuss sensitivity studies on some of the key hyper-parameters of the bascline method, including the number of training epochs and learning rale. The sensitivily study on the training epochs is presented in Figure[S]] Our findings from this study show that the best results for CIFAR-100, CUB-200 and miniImageNet datasots are obtained for training 400, 80, and 80 epochs, respectively. The study on learning rate in Table S3 shows that the best performances are achieved for learning rate of 0.1 for CIFAR-100 100, and miniTmageNer datasets, and 0.001 for the CUB-200 dataset.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Table S1: Comparison to prior works across the base and incremental sessions on CUB-200.
Acc. in each session (%)
Method
5
Baseline
79.57
75.39
72.23
67.53
67.49
64.73
64.41
62.69
60.54
60.85
59.88
ICaRL Rebulli el al
2017
68.68
52.65
48.61
44.16
36.62
29.52
27.83
26.26
24.01
23.89
21.16
EEIL Castro et al.
2018
68.68
53.63
47.91
44.20
36.30
27.46
25.93
24.70
23.95
24.13
22.11
TOPIC Tao
2020
68.68
62.49
54.81
49.99
45,25
41.40
38.35
35.36
32.22
28.31
26.28
Rebalancing
cL 2019
68.68
57.12
44.21
28.75
26.71
25.66
24.62
21.52
20.12
20.06
19.87
SPPR (Zhu et al. 202
68.68
61.85
57.43
52.68
50.19
46.88
44.65
43.07
40.17
39.63
37.33
MetaFSCIL Chi ot al.
2022
75.90
72
68.78
64.78
62.96
59.99
58.30
36.85
54.78
53.82
52.64
F2M
Shi et al. 2021
81.07
78.16
75.57
72.89
70.86
68.17
67-01
65.26
63.36
61.76
60.26
CEC
Zhang al.
2021
75.85
71.94
68.50
63.50
62.43
58.27
57.73
55.81
54.83
53.52
52.28
FACT
Zhou
21224
75.90
73.23
70.84
66.13
65.56
62.15
61.74
59.83
58.41
57.89
56.94
LIMIT
Zhou cl
20226
75.89
73.55
71.99
67.42
63.61
62.40
61.35
59.91
58.66
Nef.
Kang cl a
2023
78.07
74.58
71.37
67.54
65.37
62.60
61.07
59.37
57.53
57.21
56.75
ALICE
Peng et a
2022a
77.40
72.70
70.60
67.20
65.90
63.40
62.90
61.90
60.50
60.60
60.10
SAVC
Song ct ml
2023
81.85
77 .2.
74.95
70.21
69.96
67.02
66.16
65.30
63.84
63.15 15
62.50
Ours
82.31
78.03
75.45
70.99
71.06
67.85
67.44
66.05
64.95
64.31
63.60
Table S2: Comparison to prior works across the base and incremental sessions on minilmageNNet.
Acc. in each session (%) f
Method
0
1
2
3
5
6
Baseline
69.08
64.4
60.22
57.08
53.8
50.88
48.42
46.54
45.08
iCaRL (Rebulfi et al. 2017
71.77
61.85
58.12
54.60
51.49
48,47
45.90
44.19
42.71
Rebalancing Hou et al. 2019
72.30
66.37
61.00
56.93
53.31
49.93
46.47
44.13
42.19
TOPIC
Tao et al.
2020
61.31
50.09
45.17
41.16
37.48
35.52
32.19
29.46
21.12
EEIL
Castro et 2018
61.31
16.58
44.00
37.29
33.14
27.12
24.10
21.57
19.58
FSLL
Mazumder et al.
2021
66.48
61.75
58.16
54.16
51.10
48,53
46.54
44.20
42.28
FSLL
+SL (Mazumder al.
2021
68.85
63.14
59.24
55.23
52.24
49.65
47.74
45.23
43.92
F2M
(Shi et al. [0021
72.05
67.47
63.16
59.70
56.71
53.77
51.11
49.21
47.84
CEC
Zhang
2021
72.00
66.83
62.97
59.43
56.70
53.73
51.19
49.24
47.63
MStal Chi eb al.
2022
72.04
67.94
63.77
60.29
57.58
55.16
52.90
50.79
49.19
C-FSCIL (Hersche et 2022
76.40
71.14
66.46
63.29
60.42
57.46
54.78
53.11
51.41
FACT
Zhou et al.
2022a
72.56
69.63
66.38
62.77
60.60
57.33
54.34
52.16
50.49
CLOM
Zou et al. 2022
73.08
68.00
64.16
60.41
57.41
54.29
51.54
49.37
48.00
LIMIT
Zhou a.. 2022b
72.32
68.47
64.30
60.78
55.07
52.70
50.72
Nef
Kang et al.
2023
79.77
75.08
70.59
66.93
64.00
61.00
57.81
55.81
54.68
ALICE
Peng et
2022a
80.60
70.60
67.40
64.50
62.50
60.00
57.80
56.80
55.70
SAVC
Song et al.
2023
81.12
76.14
72.43
68.92
66.48
62.95
59.92
58.39
57.11
Ours
84.3
79.59
75.49
71.4
68.45
65.129
62.2
60.52
59.11
S3.2 Stability Tricks
In this section, we discuss additional results on the stability tricks. In Figure S2 we study the sensitivity towards the mumber of epochs when training with the SupCon loss. This study shows that optimal performance with the SupCon loss is observed for a relatively larger number of epochs comparison to the baseline. More specifically, the best results for CIFAR-100, CUB-200, and miniImageNet are observed when trained for 500, 100, and 120 epochs, respectively.
Next, discuss the experiments pre-assigning prototypes. As discussed in Section 3.3 prototype after training for pre-defined number of epochs. In Table S4 we study the optimal epoch before assigning the protolype, defined as a factor of (otal epochs. For instance, a value of 0.1 means the prototypes are assigned after training for 10% of the total number of epochs. As we find from this table, assigning a prototype at the beginning of the training does not yield the best performance for any datasct. The best results for the CIFAR-100, CUB-200, and minilmageNet are obtained for the epoch factor of 0.1, 0.5, and 0.5, respectively.
assign the
17

---

Published in Transactions on Machine Learning Research (09/2024)
44.0
60.2
46
43.8
60.0
45
Accuracy (%)
43.6
Accuracy (%)
59.8
Accuracy (%)
59.6
43.4
59.4
43.2
59.2
43
43.0
59.0
42.8
58.8
42-
42.6
58.6
41
200
400
500
600
50
80
100
120
50
80
100
120
Epochs
Epochs
Epochs
(a) CIFAR-100
(b) CUB-200
(c) minilmageNNet
Figure S1: Sensitivity study on the number of training epochs for the baseline method.
Table S3: Sensitivity study on the learning rate for the baseline method.
LR
CIFAR-100
CUB-200
minilN
0.5
29.36
55.02
34.74
43.77
57.75
45.08
0.01
42.73
58.33
43.42
0.001
39.47
59.88
32.54
0.0001
19.63
50.17
16.01
49
50.1
60.35
50.0
60.30
Accuracy (%6
Accuracy (96)
60.25
Accuracy (9)
49.9
47
60.20
49.8
60.15
49.7
60.10
45
49.5
400
600
50
100
Epochs
Epochs
Epochs
(a) CIFAR-100
(b) CUB-200
(c) minilmageNet
Figure $2: Sensitivity study on the number training epochs for SupCon.
S3.3 Adaptability Tricks
In this section, we discuss the experiments on the adaptability tricks. As discussed in Section 3.5 only tune few layers of the encoder during the incremental fine-tuning, keeping the remaining layers frozen. Table S5 presents the results for fine-tuning different portions of the pre-trained encoder. As seen in this table, the best results are observed when only the last ResNet block turned, while the worst results are consistently observed for tuning the full encoder.
S3.4 Training Tricks
Finally, we discuss the experiments on the training tricks. In Table S6 different encoders all the datasets. As observe in this table, increasing the model size in framework
we show the results for training
18

---

Published in Transactions on Machine Learning Research (09/2024)
Table S4: Sensitivity study on the epoch factor for prototypes.
Epoch factor
CIFAR-100
CUB-200
MinilN
49.66
60.50
51.01
0.1
49.45
60.74
51.10
0.5
49.73
60.24
50.87
0.75
49.35
60.11
50.75
Table S5: Sensitivity study on tuning different portions of the encoder.
Tunable parameters
CIFAR-100
CUB-200
MinilN
Full encoder
49.11
59.16
51.19
Last ResNet block
52.51
62.27
55.82
Last 2 ResNet blocks
52.02
61.50
55.18
Table S6: Study different encoders.
Encoder
CIFAR-100
CUB-200
MinilN
ResNet-18
58.12
63.10
57.85
ResNet-34
58.44 68.44
63.17
57.90
ResNet-50
58.52
63.55
57.99
Res
57.95
63.19
57.55
Table S7: Comparison to prior works with different shots of data.
Acc. in each session (%) f
Method
0
3
4
5
8
1-shots
SAVC
Song al.
2023
79.85
71.29
64.11
57.08
53.45
50.45
48. 74
46.54
44.45
SoftNet
Kang et al.
2023
79.88
72.01
65.01
59.01
55.01
52.01
50.01
48.01
46.01
Ours
80.25
76.01
71.87
67.24
63.56
59.82
56.62
54.11
52.84
2-shots
SAVC
Song al.
2023
79.85
72.00
65.45
59.17
56.25
52.2
50.16
48.11
46.94
SoftNet
Kang ct al.
2023
79.88
73.02
66.01
60.01
56.01
53.01
51.01
49.01
47.01
Ours
80.25
77.63
73.071
68.53
64.51
60.67
58.41
55.98
54.15
5-shots
SAVC
Song
2023
79.85
73.70
69.37
66.28
61.91
59.27
57.24
54.97
53.12
SoftNet
Kang et al.
2023
79.88
75.54
71.64
67.47
64.45
61.09
59.07
57.29
55.33
Ours
80.25
77.20
75.09
70.82
67.83
64.86
62.73
60.52
58.75
10-shots
SAVC
Song 2023
79.85
75.45
72.18
68,26
65.66
62.15
60.27
58.19
56.01
SoftNet Kang 2023
79.88
77.28
73.15
69.34
66.14
63.38
61.27
59.33
57.14
Ours
80.25
78.10
76.39
72.02
69.03
65.96
64.00
61.92
60.25
improves the performance up to certain model size. Notably, we observe that the best performance for all the datasets is obtained with ResNet-50.
S3.5 Performance on Different Shots
In this section, we present additional results on different shots. More specifically, we present the results for 1-, 2-, 5-, and 10-shots on the CIFAR-100 dataset. The results this study are presented in Table [S7 As we see from this table, our framework outperforms prior works in data-scarce settings. Specifically, in 1-shot setting, our framework outperforms the previous state-of-the-art by a significant margin of 6.83%. In the 2-shot setting, the difference increases slightly to 7.14%. In 5-shot setting, the improvement over existing methods decreases to 3.42%. Finally, find boost in performance for all the methods when we increase the labelled samples to 10-shot setting, with our framework showing 3.11% 11% improvement over the SoftNet.
19


Reviews:
[{"review_id": "hMHCWTr4z2", "reviewer": "Reviewer_YMFp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper systematically explores a wide range of techniques for improving FSCIL performance.\n2. The authors provide detailed insights into the individual contributions of each trick and their impact on stability and adaptability.\n3. The proposed framework achieves significant performance improvements over previous methods.\n4. Exploring 1-shot and 2-shot scenarios demonstrates the framework's robustness under data scarcity.\n\nWeakness:\n1. I think the biggest weakness of the paper is that it uses models and benchmarks which are extremely outdated and almost toyish. Since pretrained models have come into the picture the field of continual learning has very dramatically and drastically changed. The few shot adaptation in most cases is trivial by just prompting the model. So in the light of the progress in the field since 2017 I find that the insight from the paper might not be applicable in the settings in which people mostly find them these days. For example, from my experience the insights from resnet model and MNIST/cifar tasks would not tranfer at all when we have models like ViT or even T5 kind of models. The paper does a really good job at the through analysis however my worry is that none of these insights would be useful.\n2. The paper primarily combines existing techniques. While the unified framework and analysis are valuable, the individual techniques are not novel.\n3. The explanation of the SubNet tuning technique could be further elaborated for better clarity.\n4. The paper acknowledges the higher computational cost but provides limited analysis or discussion on its implications. I see that in Table 6 they provide some FPS throughput however It would be great if they can provide wall clock time for the experiments where there are additional components involved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a unified framework that integrates eight highly influential techniques, categorized as stability, adaptability, and training tricks, to enhance both the stability (reducing forgetting of previously learned classes) and adaptability (effectively learning new classes) of FSCIL models. The authors conduct a thorough analysis of each trick's contribution towards stability and adaptability in the context of FSCIL, providing valuable insights into their individual and combined effects. This analysis sheds light on the complex interplay between these techniques and their impact on overall performance. They experiment with three benchmarks (CIFAR-100, CUB-200, and miniImageNet), the proposed framework demonstrably outperforms existing state-of-the-art methods, however I have concerns regarding the experimental setup used in the paper. The authors also study a low-shot settings (1-shot and 2-shot) to assess the framework's effectiveness under extreme data scarcity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add some experiments related to some newer models like ViT and compare with methods like \"Learning to Prompt for Continual Learning\" and followups.\n2. Provide more detailed explanation and justification for the specific SubNet extraction method used. Do they first optimize to find the subnetwork? or find the subnetwork in some other way and then just train it?\n3. Further investigation into adaptability: Given the persistent gap between performance on base and novel classes, the authors could further analyze the limitations of current adaptability tricks and explore potential avenues for bridging this gap.\n4. See Weakness section.", "broader_impact_concerns": "N\\A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BVUNUkxf5k", "reviewer": "Reviewer_R2Ls", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nThe paper is well-supported by numerous detailed figures that effectively illustrate the main concepts and enhance understanding. Additionally, the authors provide a thorough comparison with other existing methods, highlighting the framework’s advancements in the field.\n\n[Weaknesses]\n\n1.\tThe paper lacks clarity regarding the novelty of the proposed tricks. Many of the techniques discussed appear to have been previously utilized in other studies, making it difficult to ascertain the unique contributions of this work.\n2.\tThe research is exclusively conducted on image classification tasks using specific datasets. This specialization limits understanding of the framework’s applicability to other domains, such as text or audio processing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive framework aimed at enhancing both stability and adaptability in Few-Shot Class-Incremental Learning (FSCIL). The framework is categorized into three main strategies: stability tricks, adaptability tricks, and training tricks. Stability tricks are designed to minimize forgetting by improving class separation within the embedding space. Adaptability tricks facilitate the learning of new classes through targeted fine-tuning methods. Training tricks aim to boost overall model performance while maintaining stability and adaptability. The framework’s effectiveness is validated through extensive testing on CIFAR-100, CUB-200, and miniImageNet, where it demonstrates superior performance over existing methods and sets a robust baseline for future FSCIL research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tThe authors should delineate the innovative aspects of the tricks introduced in this paper, specifically highlighting how these adaptations or enhancements differ from or improve upon previous implementations.\n\n2.\tIn the conclusion, the paper discusses a limitation related to integrating multiple tricks within a single framework. It would be beneficial for the authors to elaborate on the challenges faced in this integration and how they impact the framework’s adaptability or scalability.\n\n3.\tFigure 1 presupposes that incremental classes differ from base classes. The authors should explore and discuss the effectiveness of stability tricks when incremental classes closely resemble or are identical to base classes. This investigation would greatly enhance understanding of the framework’s robustness and versatility.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LqbIpNDYOm", "reviewer": "Reviewer_rHCM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- Although each of the individual tricks considered in this paper is not novel, the idea of combining them into a single framework to improve the performance of FSCIL is sensible and has not been considered before.\n- The experiment results show the proposed approach has better accuracy than previous state-of-the-art methods such as SAVC and SoftNet.\n- The paper provides some in-depth analysis of the inter-class distance, intra-class distance, and class separation when adding the stability tricks on the CIFAR-100 dataset. The paper also visualizes the effects of the adaptability tricks on the class separation on this dataset. These analyses are helpful for understanding the proposed approach.\n\n**Weaknesses:**\n- One main weakness of the paper is the unclear presentation of the proposed approach. In the current manuscript, each of the tricks is just presented separately, and it is not clear how all these tricks can be combined into a single framework. For instance, in their original papers, the SupCon, ETF, and pseudo-classes all have different training losses. It is not clear how these losses can be combined into a single framework when these tricks are used together.\n- Related to the previous point, in the training tricks, both contrastive pre-training and additional learning signal are self-supervised learning steps performed before the base training session. However, it is not clear to me how these tricks can be used together (i.e., what would the training loss be when the two self-supervised learning methods are used together?).\n- Additionally, it is also not clear how the fine-tuning and SubNet tuning tricks can be combined, since they are all fine-tuning different parts of the network.\n- I am not very sure that the change from ResNet-18 to ResNet-50 could be qualified as a major trick. This change seems trivial and the accuracy improvements in the experiments are also not great.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to combine 8 different tricks to improve the performance of few-shot class-incremental learning (FSCIL). The tricks are grouped into 3 groups: stability, adaptability, and training techniques. The paper then performs experiments and some in-depth analysis on 3 benchmark datasets to evaluate the effectiveness of these tricks on FSCIL, showing they can achieve a new state-of-the-art accuracy for this problem.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest a major change in the presentation of the proposed approach, especially by consolidating the tricks and showing a full picture of how the tricks would be combined together. More specifically (*all these points are critical*):\n- The authors could provide a figure showing the order where the tricks are performed if all of them are used together. In this figure, we could use different colors for different categories of tricks.\n- I suggest to remove the larger encoder trick since it seems not a significant step.\n- For the adaptability tricks, I suggest to combine both incremental fine-tuning and SubSet tuning into a single trick (with some new name), since these two tricks are best used together, as shown in the experiment results (e.g., in figure 3).\n- The paper should add more details on the ETF and pseudo-classes tricks, especially their training losses and how these losses are combined together. These information could also be added into the overall figure above.\n- Some experiment results are not consistent. For example, in table 3, the accuracy without training tricks is 57.85, but this accuracy is 58.12 in table 1. Besides, further removing the adaptability tricks results in 54.79 in table 3 but 51.65 in table 1. There should be a throughout check to ensure consistency of the results and if possible, also including the error ranges of the results.\n\nBesides these major changes, there are other minor changes that need to be addressed. Specifically:\n- In Section 3.1, since $T$ is a number, each session $t$ should be in $\\\\{ 1, 2, \\ldots, T \\\\}$.\n- On page 4, the paragraph after Eq (1), $y_i = y_j$ should be in the subscript of the indicator function.\n- In Eq (2), the operator $\\odot$ should be defined.\n- In Eq (3) and the paragraph after that, the indicator function should be used consistently with that in Eq (1); for example, without (or with) the square brackets in the subscript. The temperature $\\tau$ should be introduced in Eq (1) as well.\n- In the legends of Figure 2, it should be \"ETF\" instead of \"EFT\".\n- In the paragraph right after the caption of figure 2, there should be a closing parenthesis after \"illustrated in Figure 2b\".\n- In Section 4.3, some references to Section 4.2 should be \"Section 4.2\" instead of \"Appendix 4.2\".\n- On page 8, the paragraph before the Adaptability Tricks paragraph, the numbers 51.21%, 62.27%, 55.82% are not consistent with Table 1.\n- In Figure 2, why do you choose to report cumulative probability instead of a simple average distance? Some explanations are needed here.", "broader_impact_concerns": "No concerns on broader impacts and ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a unified framework that integrates multiple techniques, termed "tricks," to improve Few-Shot Class-Incremental Learning (FSCIL). The proposed framework categorizes these techniques into stability tricks, adaptability tricks, and training tricks, each contributing to the overall enhancement of model performance, stability, and adaptability. The authors conduct extensive experiments on standard benchmarks, demonstrating significant performance improvements and providing detailed analyses of the effects of these tricks on the model's learning process.

The reviewers expressed concerns regarding the experimental setup, particularly the use of outdated models and benchmarks. In response, the authors have added more experiments, including those involving larger and more modern encoders like ViT. If possible, it would be beneficial for the authors to further strengthen the paper by including additional experiments that leverage pre-trained large models.
Decision: Accept (poster)