On the Soft-Subnetwork for Few-Shot Class Incremental Learning
Authors: Haeyong Kang
Jaehong Yoon
Sultan Rizky Hikmawan Madjid
Sung Ju Hwang
Chang D. Yoo
Venue: ICLR 2023 poster (2023)
Keywords: Soft-subnetwork
Few-shot class incremental learning (FSCIL)

Abstract: Inspired by Regularized Lottery Ticket Hypothesis, which states that competitive smooth (non-binary) subnetworks exist within a dense network, we propose a few-shot class-incremental learning method referred to as Soft-SubNetworks (SoftNet). Our objective is to learn a sequence of sessions incrementally, where each session only includes a few training instances per class while preserving the knowledge of the previously learned ones. SoftNet jointly learns the model weights and adaptive non-binary soft masks at a base training session in which each mask consists of the major and minor subnetwork; the former aims to minimize catastrophic forgetting during training, and the latter aims to avoid overfitting to a few samples in each new training session. We provide comprehensive empirical validations demonstrating that our SoftNet effectively tackles the few-shot incremental learning problem by surpassing the performance of state-of-the-art baselines over benchmark datasets.

Published as a conference paper at ICLR 2023
ON THE SOFT- SOFT-SUBNETWORK FOR FEWT CLASS INCREMENTAL LEARNING
Haeyong Kang, Jaehong Yoon, Sultan Rizky Madjid, Sung Ju Hwang, and Chang D. Yoo" Korea Advanced Institute of Science and Technology (KAIST) 291 Daehak-ro, Yuseong-gu, Daejeon (haeyong. kang, loang,aace jaehong. yoon, suulkyy, beblyyyysswwnng. s.hwang82,
ac. kr
ABSTRACT
Inspired by Regularized Lottery Ticket Hypothesis, which states that competitive smooth (non-binary) subnetworks exist within dense network, we propose a few- shot class- glass-incremental learning method referred as Soff-SubNetworss (SoftNet). Our objective is to learn sequence of sessions incrementally, where each session only includes a few training instances per class while preserving the knowledge of the previously learned ones. SoftNet jointly learns the model weights and adaptive non-binary soft masks at a base training session in which each mask consists of the major and minor subnetwork; the former aims to minimize catastrophic forgetting during training, and the latter aims to avoid overfitting to few samples in each training session. We provide comprehensive empirical validations demonstrating that our SoftNet effectively tackles the few-shot incremental learning problem by surpassing the performance of state-of-the-ard baselines over benchmark datasets. The public code is available at https: // ///////t// //g/tttul thub. com/ hheey o SoftNet-FSCIL.
INTRODUCTION
Lifelong Learning, or Continual Learning, is a learning paradigm to expand knowledge and skills through sequential training of multiple tasks (Thrun, 1995). According to the accessibility of task identity during training and inference, the community often categorizes the field into specific problems, such les-iincremental (Pfülb and Gepperth, 2019; Delange al., 2021; Yoon al., 2020; Kang et al., 2022), tlass-incrementll (Chaudhry et al., 2018; Kuzborskij et al., 2013; Li and Hoiem, 2017; Rebuffi et al., 2017; Kemker and Kanan, 2017; Castro et al., 2018; Hou et al., 2019; Wu et al., 2019), and fask-free continual learning (Aljundi et al., 2019; Jin et al., 2021; Pham et al., 2022; Harrison et al., 2020). While the standard scenarios for continual learning assume a sufficiently large number of instances per task, a lifelong learner for weal-world applications often suffers from insufficient training instances for each problem to solve. This paper aims to tackle the issue of limited training instances for practical Class Incremental Learning (CIL), referred to Few-Shot CIL (FSCIL) (Ren et al., 2019; Chen and Lee, 2020; Tao et al., 2020; Zhang et al., 2021; Cheraghian et al., 2021; Shi al., 2021).
However, there are two critical challenges in solving FSCIL problems: catastrophic forgetting and overfitting. Catastrophic forgetting (Goodfellow et al., 2013: Kirkpatrick et al., 2017) or Catastrophic Interference McCloskey and Cohen (1989) is a phenomenon in which a continual learner loses the previously learned task knowledge by updating the weights adapt to new tasks, resulting in significant performance degeneration since the scenario does not allow the model to revisit past task data. Recent works propose to mitigate catastrophic forgetting for class-nncrementl learning, often categorized in multiple directions, such as constraint-be -base (Rebuffi et al., 2017; Castro et al., 2018; Hou et al., 2018; 2019; Wu et al., 2019), memory-basee bbssed (Rebuffi et al., 2017: Chen and Lee, 2020; Mazumder et al., 2021; Shi et al., 2021), and architecture-based methods (Mazumder et al., 2021; Serra et al., 2018: Mallya and Lazebnik, 2018; Kang al., 2022). However, we note that catastrophic forgetting becomes further challenging
previous tasks. Such undesired knowledge drift is irreversible
"Corresponding Author.

---

Published as a conference paper at ICLR 2023
in FSCIL. Due to the small amount of training data for new tasks, the model tends to severely overfit to new classes and quickly forget old classes, deteriorating the model performance.
Mcanwhile, several works address overfitting issues for continual learning from various perspectives. NCM (Hou et al., 2019) and BiC (Wu et al., 2019) highlight the prediction bias problem during sequential training that the models OCS (Yoon et al., 2022) tackles the class imbalance problems for rehearsal- -based continual learning, where the number of instances at each class varies per task that the model would perform biased training on dominant classes. Nevertheless, these works do not consider the overfitting issues caused by training a sequence of few-shot tasks. FSLL (Mazumder et al., 2021) tackles overfitting for few-shot CIL by partially-splitting model parameters for different sessions through multiple substeps of iterative reidentification and weight selection. However, led to computationally inefficient.
prone to predict the data to classes in recently trained tasks.
To deploy a practical few-shot CIL model, we propose a simple yet efficient method named SoftNet, effectively alleviating catastrophic forgetting and overfitting Motivated by Lottery Ticket Hypoth- esis (Frankle and Carbin, 2019), which hypothesizes the existence of competitive subnetworks (winning tickets) within the randomly initialized dense neural network, we suggest a new paradigm for Few-shot CIL, named Regularized Lottery Ticket Hypothesis:
Regularized Lottery Ticket Hypothesis (RLTH). A aundomly-initaalin dense neural network contains regularized subnetwork that can retain the prior class knowledge while providing room to learn the new class knowledge through isolated training of the subnetwork.
Based on RLTH, we propose method, referred to Soft-SubNetworks (SoftNet), illustrated in Figure 1. First, SoftNet jointly learns the randomly initialized dense model (Figure 1 (a)) and soft mask m E [0, 1] pertaining to Soft-subnetwork (Figure (b)) on the base session training; the soft mask consists of the major part of the model parameters
1 and the minor ones < 1 where m = is obtained by the top-c% of model parameters and m < 1 is obtained by the remaining ones (100- top-c%) sampled from the uniform distribution. Then, we freeze the major part of pre-trained subnetwork weights for maintaining prior class knowledge and update the only minor part of weights for the novel class knowledge (Figure (c)).
We summarize our key contributions as follows:
This paper presents new masking- massed method, Soft-SubNetwork (SoftNett, that tack- les two critical challenges in the few-shot class incremental learning (FSCIL), known as catastrophic forgetting and overfitting.
Our SoftNet trains two different types of non-binary masks (subnetworks) for solving FSCIL, preventing the continual leamer from forgetting previous sessions and overfitting simultaneously.
We conduct a comprehensive empirical study on SoftNet with multiple class incremental learning methods. Our method significantly outperforms strong baselines on benchmark tasks for FSCIL problems.
2 RELATED WORK
Catastrophic Forgetting, Many recent works have made remarkable progress in tackling the challenges of catastrophic forgetting in lifelong learning. To be specific, Architecture-based ap- proaches (Mallya et al., 2018; Serrà et al., 2018; Li et al., 2019) utilize an additional capacity to expand (Xu and Zhu, 2018: Yoon et al., 2018) or isolate (Rusu et al., 2016) model parameters. thereby avoiding knowledge interference during continual learning: SupSup (Wortsman et al., 2020) allocates model parameters dedicated to different tasks. Very recently, Chen et al. (2021); Kang et al. (2022) shows the existence of sparse subnetwork, called winning tickets, that performs well on all tasks during continual learning. However, many subnetwork-basse ble with the FSCIL setting since performing task inference under data imbalances is challenging. FSLL (Mazumder et al., 2021) aims to search session-specific subnetworks while preserving weights for previous sessions for incremental few-shot learning. However, the expansion process comprises another series of retraining and pruning steps, requiring excessive training time and computational costs. On the contrary, our proposed method, SoftNet, jointly learns the model and task-adaptive
approaches are incompati-
2

---

Published as a conference paper at ICLR 2023
$-1
S-1
S-1
(a) Dense Initialization
(b) Training of SoftNet
(c) Update of SoftNet
Figure I: Incremental Soft-Subnetwork (SoftNet): (a) Dense Neural Network is randomly initialized for the base session (S-1) training (b) SoftNet is trained by major subnetwork Manjor = 1 (thick solid line) and minor Mominor U0, 1), and (c) SoftNet updates only few minor weights (thin solid line) new sessions (S).
smooth (i.e., non-binary) masks of the subnetwork associated with the base session while selecting an essential subset of the model weights for the upcoming session. Furthermore, smooth masks behave like regularizers that prevent overfitting when learning new classes.
Soft-subnetwork. Recent works with context-dependent gating of sub-spaces (He and Jaeger, 2018), parameters (Mallya and Lazebnik, 2018; He al., 2019; Mazumder et al., 2021), or layers (Serra et al., 2018) of single deep neural network demonstrated its effectiveness in addressing catastrophic forgetting during continual learning. Masse al. (2018) combines context-dependent gating with the constraints preventing significant changes in model weights, such as SI (Zenke et al., 2017) and EWC (Kirkpatrick et al., 2017), achieving further performance increases than using them alone, A flat minima could also be considered acquiring sub-spaces. Previous works have shown that flat minimizer is more robust to random perturbations (Hinton and Van Camp, 1993; Hochreiter and Schmidhuber, 1994; Jiang et al., 2019). Recently, Shi et al. (2021) showed that obtaining flat loss minima in the base session, which stands for the first task session with sufficient training instances, necessary to alleviate catastrophic forgetting in FSCIL. To minimize forgetring. they updated the model weights on the obtained flat loss contour. In our work, by selecting sub-networks (Frankle and Carbin, 2019; Zhou et al., 2019; Wortsman al., 2019; Ramanujan al., 2020; Kang al., 2022; Chijiwa et al., 2022) and optimizing the sub-network parameters in a sub-space, we propose new method preserve learned knowledge from base session on major subnetwork and learn new sessions through regularized minor subnetworks.
SOFT-SUBNETWORK FOR FEW-SHOT CLASS INCREMENTAL LEARNING
3.1 PROBLEM STATEMENTS
Various works have tried mitigate catastrophic forgetting problems in class incremental learning using knowledge distillation, revisiting a subset of prior samples, or isolating essential model param- eters retain prior class knowledge even after the model loses accessibility to them. However, as a few-shot class incremental learning scenario regards following tasks/sessions containing a small amount of training data, the model tends to severely overfit new classes, making difficult to fine- tune the previously trained model on a few samples. In addition, the fine-tuning process often leads to the catastrophic forgetting of base class knowledge. As result, regularization indispensable in the models to avoid forgetting and prevent the model from overfitting to new class samples by updating only the selected parameters for learning in the new session.
Few-shot Class Incremental Learning (FSCIL) aims to learn new sessions with only few examples continually, A FSCIL model learns a sequence T' training sessions {D),... .D'') }. where D { (a) 3.t is the training data of session and æ is an example of class y/ E O In FSCIL, the base session D usually contains a large number of classes with sufficient training data for each class. In contrast, the subsequent sessions (t > 2) will only contain small number of classes with few training samples per class, e.g., the tt session Df is often presented as a N-way K-shot task. In each training session I, the model can access only the training data Dt and few examples stored
3

---

Published as a conference paper at ICLR 2023
in the previous session. When the training of session t is completed, evaluate the model test samples from all classes O Ui=1 O', where O' N noiti 0 for vi, T.
Consider supervised learning setup where the T sessions arrive a lifelong learner f(;; O) parame- terized by the model weights 0 in sequential order. A few-shot class incremental learning scenario aims to learn the classes in a sequence of sessions without catastrophic forgetting. In the training session the model solves the following optimization procedure:
The Σ (.(f(2):)) 0). The
1
l minimize
where L, is classification loss like cross-entropy, and n, is the number of instances for session 4.
3.2 SUBNETWORK-BASER TRAINING FOR FEW-SHOT CLASS INCREMENTAL LEARNING
As lifelong learners often adopt over wer-paaraettrizee dense neural networks to allow resource freedom for future classes tasks, updating entire weights in neural networks for few-shot tasks is often not preferable and often yields the overfitting problem. To overcome the limitations in FSCIL, we focus on updating partial weights in neural networks when a new task arrives. The desired set of partial weights, named subnetwork, can achieve on-par or even better performance with the following motivations: (1) Lottery Ticket Hypothesis (Frankle and Carbin, 2019) shows the existence of subnetwork that performs well as the dense network, and (2) The subnetwork significantly downsized from the dense network reduces the size of the expansion of the solver while providing extra capacity to learn new sessions or tasks.
We first suggest the objective referred to as HardNet as follows: given dense neural network parameters 0, the binary attention mask m. describes the optimal subnetwork for session / such that [m,]] is less than the dense model capacity [0]. However, such binarized subnetworks m, E {0, } cannot adjust the remaining parameters in a dense network for future sessions while solving past task problems cost- and memory efficiently. In FSCIL, the test accuracy of the base session drops significantly when it proceeds to learn sequential sessions since the subnetwork of m = plays a crucial role in maintaining the base class knowledge. To this end, we propose a soft-subnetwork m, E [o, 1] instead of the binarized subnetwork. It gives more flexibility to fine-tune small part of the soft- subnetwork while fixing the rest to retain base class knowledge for FSCIL As such, we find the soft-subnetwork through the following objective:
m minimize 1 1
The Cf m.).vi)
The
(2)
subject |m S.c.
where session loss J [(f((] 0). vf). the subnetwork sparsity < |0] (used the selected proportion % of model parameters in the following section), and @ represents an element-wise product. In the following section, we describe how to obtain the soft-subnetwork m, using the magnitude-basee -based criterion (RLTH) while minimizing session loss jointly.
3.3 OBTAINING SOPT-SUBNETWORKE VIA COMPLEMENTARY WINNING TICKETS
Let each weight be associated with a learnable parameter we call weight score 8, which numerically determines the importance of the associated weight. In other words, we declare weight with higher score as more important. At first, we find a subnetwork 0* = O O m; of the dense neural network and then assign it as a solver of the current session The subnetworks associated with each session jointly learn the model weight f and binary mask m,. Given an objective Lt, we optimize 0 as follows:
0" m minimize (, m.; D.).
0,8
where m, is obtained by applying an indicator function 1, on weight scores s. Note (s(s) = if belongs top-c% scores and 0 otherwise.
In the optimization process for FSCIL, however, we consider two main problems: (1) Catastrophic forgetting: updating all m when training for new sessions will cause interference with
4

---

Published as a conference paper at ICLR 2023
the weights allocated for previous tasks; thus, we need to freeze all previously learned parameters @ m,-1; (2) Overfitting: the subnetwork also encounters overfitting issues when training an incremental task few samples, such, need to update few parameters irrelevant to previous task knowledge., i.e., @ m)-1).
To acquire the optimal subnetworks that alleviate the two issues, define soft-subnetwork by dividing the dense neural network into two parts-one is the major subnetwork majorr and another is the minor subnetwork Momioor- The defined soft-subnetwork follows as:
goft = majoor 2
minnors
where majoor is a binary mask and Miminoe ~ (O, 1) and @ represents an element- wise summation. As such, soft-mask is given as m E [0, 1| in Eq.3. In the all-experimental FSCIL setting, majaor maintains the base task knowledge t = while mimnoo acquires the novel task knowledge 2 Then, with base session learning rate a, the 0 is updated follows: 0 <- O Lo0 @ m. effectively regularize the weights the subnetworks for incremental learning. The subnetworks are obtained by the indicator function that always has gradient value of O; therefore, updating the weight scores with its loss gradient is impossible. To update the weight scores, we use Straight-throuug Estimator (Hinton, 2012; Bengio et al., 2013; Ramanujan al., 2020) in the backward pass. Specifically, ignore the derivatives of the indicator function and update the weight score f 8 de where sot for exploring the optimal subnetwork for base session training. Our Soft-subnetwork optimizing procedure is summarized in Algorithm 1. Once a single soft-subnetworks m,,n is obtained in the base session, then we use the soft- coft-subnetwork for the entire new sessions without updating.
(oc
[C @ mssot).
Algorithm 1
(SoftNet)
input (D' W- model weights 0, and score weights s, layer-wise capacity
1: // Training over base classes
Randomly initialize and
3: for epoch e 1,2, do
4: Obtain softmask M.soft of Mmmjjo and Mmminor U(0,) 1) each layer
5: for batch b. D' do
Compute Crase
mwat b.) by Eq.
7: 0
-
M
8:
s -- s
02
8a
④ m.att)
9: end for
10: end for
11: Incremental learning
12: Combine the training data D' and the exemplars saved in previous few-shot sessions
13: for epoch = 1,2, do
14: for batch b. DD' do
Compute L. (0 O Mmonn b,) by Eq.
16: 0 0 3 0C
26
mminor)
17: end for
18:
end
output model parameters 0, 8, and Msso.
INCREMENTAL LEARNING FOR SOFT-SUBNETWORK
We now describe the overall procedure of our poft-pruning-basee based incremental learning/inference method, including the training phase with normalized informative measurement in Section 4.1, as followed by the prior work (Shi et al., 2021), and the inference phase in Section 4.2.
4.1 INCREMENTAL SOFT-SUBNETWORK TRAINING
Base Training (t 1). In the base learning session, optimize the soft-subnetwork parameter (including a fulle-con layer as a classifier) and weight score s with cross-entropy loss jointly using the training examples of !
Incremental Training (t 2 2). In the incremental few-shot learning sessions (t 2 2). leveraged by Myoft fine-tune few minor parameters 0 M-minor of the soft-subnetworkk learn new classes.
5

---

Published as a conference paper at ICLR 2023
Table Classification accuracy of 8 on CIFAR-100 for 5-way 5-shot incremental learning. Underbar denotes the comparable results with FSLL (Mazumder et al., 2021). denotes the results reported from Shi al. (2021).
Method
sessions
The
cRT (Shi et al,, 2021)
65.18
63.89
60.20
57.23
53.71
50.39
48.77
47.29
45.28
iCaRL (Rebuffi at.,, 2017y"
66.52
57.26
54.27
50.62
47.33
44,99
43.14
39.49
-5.79
Rebalance (Hou et al., 2019)*
66.66
61.42
57.29
53.02
48.85
45.68
43.06
40.56
38.35
-6.93
FSLL (Mazunder
2021)"
65.18
56.24
54.55
51.61
49.11
47.27
45.35
43.95
42.22
-3.08
iCaRL (Rebuffi et al., 2017)
64.10
53.28
41.69
34.13
27.93
25.06
20.41
15.48
13.73
-31.55
Rebalance (Hou
2019)
53.05
43.96
36.97
31.61
26.73
16.78
13.54
-31.74
TOPIC (Cheraghian et al., 2021)
64.10
55.88
47.07
45.16
40.11
38
33.96
31.55
29.37
-15.91
F2M (Shi 2021)
64.71
62.05
59.01
55.58
52.55
48.08
46.28
44.67
FSLL (Mazunder ct al., 2021)
64,10
55.85
51.71
48.59
45.34
43.25
41.52
39.81
38.16
-7.12
HardNet,
60.77
56.95
53.53
50.40
-3.37
HardNet, 80%
69:65
64.60
60,59
56.93
53.60
50.80
48, 69
46.69
44.63
-0.65
HardNet,
62.75
59.09
55.92
53.03
46.31
SoftNet, 50%
69.20
64.18
60.01
56.43
53.11
50.62
48, 60
46.51
44.61
.0677
SoftNet, <= 80%
70.38
65.04
60.94
57.26
54.13
51.58
49.52
47.36
45.16
-0.12
SoftNet, 99%
72.62
67.31
63.05
59.39
56.00
53.23
51,06
48.83
46.63
+1.35
Since Miminoe < 1, the soft-subnetwork alleviates the overfitting few samples. Furthermore, instead of Euclidean distance (Shi et al., 2021), we employ metric-based classification algorithm with cosine distance to finetune the few selected parameters. In some cases, Euclidean distance fails to give the real distances between representations, especially when two points with the same distance from prototypes do not fall in the same class. In contrast, representations with a low cosine distance are located in the same direction from the origin, providing normalized informative measurement. We define the loss function as:
dat (cmmsof)))
L i fz = Σ>
16y BEDOEO
oog
(5)
EoLEO e d(po of (a;; 00mxaft))
where d ...)) denotes cosine distance, Pa is the prototype of class O
O' refers to all encountered classes, and D' UP denotes the union of the current training data D' and the exemplar set P = (p2 Pt-1 }, where Pt, (2ste < t) is the set of saved exemplars in session t..
-'n
Note that the prototypes of new classes are computed by Pa = N. E, (y;: (xx; 0 @ m and those of base classes are saved in the base session, and N₂ denotes the number of the training images of class o. We also save the prototypes of all classes in O for later evaluation.
4.2 INFERENCE FOR INCREMENTAL SOFT- SOPT-SUBNETWORK
In each session, the inference is also conducted by a simple nearest class mean (NCM) classification algorithm (Mensink al., 2013; Shi al., 2021) ) for fair comparisons. Specifically, all the training and test samples are mapped to the embedding space of the feature extractor f. and Euclidean distance d, (---) is used measure the similarity between them. The classifier gives the kth prototype index of = arg min.co dry (æ; 0 O m.oft), Po) as output.
5 EXPERIMENTS
We introduce experimental setups in Section 5. Then, we empirically evaluate our soft-subnetworks for incremental few-shot learning and demonstrate its effectiveness through comparison with state-of- the-art methods and vanilla subnetworks in the following subsections.
5.1 EXPERIMENTAL SETUP
Datasets. To validate the effectiveness of the soft- soft-subnetwork, we follow the standard FSCIL experi- mental setting. We randomly select 60 classes the base class and the remaining 40 new classes for CIFAR- 100 and minilmageNet. In each incremental learning session, we construct 5-way 5-shot tasks by randomly picking five classes and sampling five training examples for each class.

---

Published as a conference paper at ICLR 2023
Table 2: Classification accuracy of R on minilmageNet for 5-way 5-shot incremental learning. Underbar denotes the comparable results with FSLL Mazumder et al. (2021). denotes the results reported from Shi al. (2021).
Method
sessions
The
cRT (Shi et al., 2021)
67.30
64.15
60.59
57.32
54,22
51.43
48.92
46.78
44.85
iCaRL (Rebulli et al., 2017)"
67.35
59,91
55.64
52.60
49.43
46.73
44.13
42.17
40.29
-4.56
Rehalance (Hou et al., 2019)*
67.91
63.11
58.75
54.83
50.68
47.11
43.88
41.19
38.72
-6.13
FSLL (Mazurnder al., 2021y"
67.30
59.81
57.26
54.57
52.05
49.42
46.95
44,94
42.87
-1.11
iCaRL (Rebufii et al., 2017)
61.31
46.32
42.94
37.63
30.49
24.00
20.89
18.80
17:21
-27.64
Rebalance (Hou al., 2019)
61.31
47.80
39.31
31.91
25.68
21.35
18.67
17.24
14.17
-30.68
TOPIC (Cherughian et al., 2021)
61.31
50.09
45.17
41.16
37.48
35.52
32.19
29.46
24.42
-20.33
IDLVQ-C (Chen and Lee, 2020)
64.77
59.87
55.93
52.62
49.88
47.55
44,83
43.14
41.84
-3.01
F2M (Shi al. 2021)
67.28
63.80
60.38
57.06
$4.08
51.39
48.82
46.58
44.65
-0.20
FSLL (Mazunder al., 2021)
66.48
61.75
58.16
54.16
51.10
48.53
46.54
44.20
42.28
-2.57
HardNet c = 50%
63
60.37
$6.12
$3.17
50.17
47.74
45.34
43.35
42.13
-2.72
HardNet,
60.42
$7.09
$4,09
51.18
45.66
HardNet: e 100%
64.66
59.80
$3.70
52.82
50.01
47.30
45.17
43.34
42.09
-2.76
SoftNet, 50%
72.83
67.23
62.82
59.41
56.44
53.55
50.92
48.99
47.60
SoftNet, 80%
76.63
70.13
65.92
62.52
59.49
56.56
53.71
51.72
50.48
+5.63
SoftNet. c= -90%
77.00
70.38
65.94
62.45
59.32
56.25
53.76
51.75
50.39
+5.54
SoftNet, 97%
77.17
70.32
66.15
62.55
59.48
56.46
53,71
51.68
50.24
+5.39
Baselines. We mainly compare our SoftNet with architecture-based methods for FSCIL: FSLL (Mazumder et al., 2021) that selects important parameters for each session, and HardNet, repre- senting a binary subnetwork. Furthermore, we compare other FSCIL methods such as iCaRL (Rebuffi et al., 2017). Rebalance (Hou et al., 2019), TOPIC (Tao et al., 2020), IDLVQ-C (Chen and Lee, 2020), and F2M (Shi et al., 2021). We also include a joint training method (Shi et al., 2021) that uses all previously seen data, including the base and the following few-shot tasks for training as a reference. Furthermore, we fix the classifier re- re-training method long-tailed classification trained with all encountered data as the approximated upper bound.
(cRT) (Kang et al., 2019) for
Experimental details. The experiments are conducted with NVIDIA GPU RTX8000 on CUDA 11.0.
We also randomly split each dataset into multiple sessions. We run each algorithm ten times for each dataset and report their mean accuracy. We adopt ResNet18 (He et al., 2016) as the backbone network. For data augmentation, we use standard random crop and horizontal flips. In the base session training stage, we select top-e% weights at each layer and acquire the optimal soft-subnetworks with the best validation accuracy. In each incremental few-shot learning session, the total number of training epochs is 6, and the learning rate is 0.02. We train new class session samples using a few minor weights of the soft-subnetwork (Conv4x layer of ResNet18 and Conv3x layer of ResNet20) obtained by the base session learning. We specify further experiment details in Appendix A.
5.2 RESULTS AND COMPARTSONS
We compared SoftNet with the architecture- based methods - FSLL and HardNet. We pick FSLL as architecture- based baseline since it selects important parameters for acquiring old/new class knowledge. The architecture- based results on CIFAR-100 and minilmageNet are presented in Table and Table 2 respectively. The performances of HardNet show the effectiveness of the subnetworks that go with less model capacity compared to dense networks. To emphasize our point, we found that ResNet18, with approximately 50% parameters, achieves comparable performances with FSLL on CIFAR- 100 and minilmageNet. In addition, the performances of ResNet20 with 30% parameters (HardNet) are comparable with those of FSLL on CIFAR-100, as denoted in Appendix of Table 9 and Table 11, including performances (Figure 4 and Figure 5) and smoothness in t-SNE plots (Figure 6).
Experimental results are prepared to analyze the overall performances of SoftNet according to the sparsity and dataset as shown in Figure 2. As we increase the number of parameters employed by SoftNet, we achieve performance gain on both benchmark datasets. The performance variance of SoftNet's sparsity seems to be depending on datasets from the fact that the performance variance on CIFAR- 100 is less than that on minilmageNet. In addition, SoftNet retains prior session knowledge successfully in both experiments described in the dashed line, and the performances of SoftNet (c = 60.0%) on the new class session (8, 9) of CIFAR- 100 than those of SoftNet (c = 80.0%) depicted in the dashed-dot line. From these results, could expect that the best performances
7

---

Published as a conference paper at ICLR 2023
depend on the number of parameters and properties of datasets. We further result comparisons of HardNet and SoftNet in Appendix B.
Softhict c-80.0%
SoftNet with
Softhet with
Softhet with cm 30 0%
Softhet with cm30.0%
Overall Acc. (98)
Softhett c-20.0%
80
80
70-
50
50
40
20
20
Sessions
Sessions
(a) CIFAR-I00
(b) minilmageNet
Figure 2: Classification accuracy of SoftNet on CIFAR-100 and minilmageNet for 5-way 5-shot FSCIL: the overall performance depends on capacity and softness of subnetwork. Note that solid( ) dashed(- and dashed-dot(--- lines denote overall, base, and novel class performances respectively.
Our SoftNet outperforms the state-of-the art methods and cRT. which is used as the approximate upper bound of FSCIL (Shi al., 2021) 1) as shown Table and Table 2. Moreover, Figure represents the outstanding performances of SoftNet on CIFAR-100 and minilmageNet. SoftNet provides new upper bound on each dataset, outperforming cRT, while HardNet provides new baselines among pruning-based methods.
CIFAR-100
minilmageNet
70
60
Accuracy
Accuracy
50
40
30
Sessi ans
Sessions
Rebalance
HardNet (best)
(best)
Figure 3: Comparision of subnetworks (HardNet and SoftNet) with state-of the-hhteat art methods.
5.3 LAYER-WISH ACCURACY
In incremental few-shot learning sessions, we train new class session samples using a few minor weights minor of the specific layer. At the same, we entirely fix the remaining weights to investigate the best performances as shown in Table 3. The best performances involve fine-tuning at the Conv5x layer with 97%. It means features computed by the lower layer are general and reusable in different classes. On the other hand, features from the higher layer are specific and highly dependent on the dataset.
5.4 ARCIITECTURE-WIS ACCURACY
Depending on architectures, the performances of subnetworks vary, and the sparsity is also one another: ResNet18 tends to dense parameters, whereas ResNet20 tends to sparse parameters on CIFAR- 100 for 5-way 5-shot as shown in Table 4. We observed that the SoftNet with ResNet20 90% than ResNet18 this CIFAR-100 FSCIL setting. From these
has more sparse solution
8

---

Published as a conference paper at ICLR 2023
Table 3: Classification accuracy of ResNet18 on minilmageNet for 5-way 5-shot incremental learning. The layer-wise inspection with fixed с 97%. all denotes that all minor weights Meninor of the entire layers were trained while only conv*X trained.
Method
sussions
The gap with cRT
et al., 2021)
67.30
60.59
57.32
54.22
51.43
46.78
44.85
SoftNet, Conv2x
77.17
70.29
66.09
62.54
59.44
56.43
53.68
51.60
50.19
+5.34
SoftNet, Conv3x
77.17
70.30
66.05
62.51
59,42
56.44
53.70
51.59
50.15
+5.30
SoftNet, Conv4x
77.17
70.30
66.08
62.54
59,45
56.46
53,70
51.59
50.17
45,32
SoftNet, Conv5x
77.17
70.32
66.15
62.55
59.48
56.46
53.71
51.68
50.24
+5.39
SoftNet, All
77.17
65.09
55.25
45.92
38.20
33.37
29.52
27.66
25.64
-19.21
observations, our SoftNet could significantly impact deep neural network architecture search it helps search sparse and task-specific architecture.
Table 4: Classification accuracy of ResNet18, 20, 32, and 50 on CIFAR-100 for 5-way 5-shot FSCIL.
Method
sessions
The cRT
R8 cRT (Shi et al., 2021)
65.18
63.89
60.20
57.23
53.71
50.39
48.77
47.29
45.28
ResNet18, SoftNet, 70%
70.92
65.
61.00
57.25
54.09
51.37
49.29
47.03
44.90
SoftNet. e 90%
72.25
66.82
62. 63
58.98
55.64
52.77
50.71
48. 48.42
46. 15
+0.87
SoftNet, c
72.62
67.31
63.05
59.39
56.00
53.23
51.06
48.83
46.63
+1.35
ResNer20, SoftNet, 70%
70.38
66.
62.6
58.93
55.81
53.11
51,38
49.29
47.08
+1.80
SoftNet. e 90%
72.63
68.60
64.96
61.
57.98
55,32
53.48
51.46
49.20
+3.92
SoftNet,
71.78
67.79
36.86
60.07
57.05
54.32
52.34
50.28
48.11
+2.83
ReaNet32, SoftNet, 90%
75.47
70.84
66.84
63.01
59.69
56.86
54.75
52.70
50.36
+5.08
SoftNet, == 93%
75.35
71.22
67.25
63.25
60.05
57.24
55,16
53.01
50.76
+5,48
SofNNet, 95%
74.67
70.32
66.31
62.61
59.29
56.54
54.52
52,30
50.05
+4.77
ResNet50, SoftNet, 70%
76.20
71.82
67.90
64.17
60.91
57.89
55,72
53,20
50.87
+5.59
SoftNer, == 80%
78.20
73.32
69.22
65.43
62.09
59.08
56.80
54.45
52.18
+6,90
SoftNet, 90%
77.67
72.73
68.50
64.57
61.08
58.06
55.70
53.37
51.20
+5.92
5.5 DISCUSSIONS
Based on our thorough empirical study, we uncover the following facts: (1) Depending on architec- tures, the performances of subnetworks vary, and the sparsity is also another: ResNet18 tends to use dense parameters, while ResNet20 tends to use sparse parameters on CIFAR-100 FSCIL settings. This result provides the general pruning- wruning-based model with a hidden clue. (2) In general, fine-tuning strategies are essential in retaining prior knowledge and learning new knowledge. We found that performance varies depending fine-tuning Conv layer through the layer-wise inspection. Lastly, (3) from overall experimental results, the base session learning is significant for lifelong learners to acquire generalized performances in FSCIL.
CONCLUSION
Inspired by Regularized Lotlery Ticket Hypothesis (RLTH), which hypothesizes that smooth subnet- works exist within a dense network, we propose Sofl-SubNetworks (SoftNet); an incremental learning strategy that preserves the learned class knowledge and learns the newer ones. More specifically. SoftNet jointly learned the model weights and adaptive soft masks to minimize catastrophic forgetting and to avoid overfitting novel few samples in FSCIL. Finally, we compared a comprehensive empiri- cal study on SoftNer with multiple class incremental learning methods. Extensive experiments on benchmark tasks demonstrate how our method achieves superior performance over the state-of-the-ant class incremental learning methodologies. We also discovered how subnetworks perform differently under specified architectures and datasets through ablation studies. In addition, we emphasized the importance of fine-tuning and base session learning in achieving optimum performance for FSCIL. We believe that our findings could bring a monumental on deep neural network architecture search, both task task-specific architectures and utilization of sparse models.

---

Published as a conference paper at ICLR 2023
Acknowledgement. This work was supported by Institute for Information & communications Technol- ogy Promotion (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-01381 Develop- ment of Causal AI through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and partly supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. 2022-0-00184 Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics).
REFERENCES
Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya, and Jacob Andreas. Subspace regularizers for few-shot class incremental learning. arXiv preprint arXiv:2110.07059, 2021.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254- 11263, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR, 2013.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311 2018.
Francisco M Castro, Manuel Marín- Jiménzzz Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European conference computer vision (ECCV), pages 233-248, 2018.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pages 532-547, 2018.
Kuilin Chen and Chi-Guhn Lee. Incremental few-shot learning via vector quantization in deep embedded space. In International Conference on Learning Representations, 2020.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. Proceedings of the International Conference Learning Representations (ICLR), 2021.
Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Semantic-aware knowledge distillation for few-shot class-incremental learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2534-2543, 2021.
Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta-learning approach for few-shot class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14166-14177 2022.
Daiki Chijiwa, Shin's ya Yamaguchi, Atsutoshi Kumagai, and Yasutoshi Ida. Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks. In Advances in Neural Information Processing Systems, 2022. URL. https: /ttpp://////eeeveeeeeee net/forus 1d=r4__ppttttttp
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the International Conference on Learning Representations (ICLR). 2019.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation catastrophic forgetting in gradient-basse neural networks. arXiv preprint 1312v:1312.6211 2013.
10

---

Published as a conference paper at ICLR 2023
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. Advances in neural information processing systems, 33:17571-17581, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor-aided backpropa- gation. In International Conference on Learning Representations, 2018.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906.05201, 2019.
Michael Hersche, Geethan Karumaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Constrained few-shot class-incremental learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9057-9067, 2022.
Geoffrey Hinton. Neural networks for machine learning, 2012.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Lifelong learning via progressive distillation and retrospection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 437-452, 2018.
Saihui Hou, Xinyu Pan. Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 831-839, 2019.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912.02 2019.
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-basse editing of memory examples for online task-free continual learning. Advances in Neural Information Processing Systems, 34, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint 17Xiv:1910.09217. 2019.
Haeyong Kang. Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson Sung Ju Hwang. and Chang D Yoo. Forget-free continual learning with winning subnetworks, In International Conference on Machine Learning, pages 10734-10750.
PMLR, 2022.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspiree inspired model for incremental learning. arXiv preprint arXiv: 171. 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan. John Quan, Tiago Ramalho, Agnieszka -arnskaaaBBwiiskaa, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. 2017.
Ilja Kuzborskij, Francesco Orabona, and Barbara Caputo. From to n+ 1: Multiclass transfer incremental learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3358-3365, 2013.
Xilai Li. Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn grow: A continual structure learning framework for overcoming catastrophic forgetting. Proceedings of the International Conference Machine Learning (ICMI),, 2019.
11

---

Published as a conference paper at ICLR 2023
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-22)) 22
Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class- incremental learning via entropy-rese regularzee data-free replay. arXiv preprint arXiv:2207.11213. 2022.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for caassincrremenal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2544-2553 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 7763, 2018
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Nicolas Y Masse, Gregory D Grant, and David J Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 14464444110 2018.
Pratik Mazumder, Pravendra Singh, and Piyush Rai. Few-shot lifelong learning. arXiv preprint arXiv:2103.00991, 2021.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem, 1989.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes at near- zero cost. IEEE transactions on pattern analysis and machine intelligence, 35(11):2624 2637, 2013.
Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C Lovell. Few-shot class-incremental learning from an open-set perspective. In European Conference Computer Vision, pages 382-397. Springer, 2022.
B. Pfülb and A. Gepperth. A comprehensive, application-orientee study of catastrophic forgetting in DNNs. In International Conference on Learning Representations, 2019. URL https:// openreview. net/ forum?ormmmmmmmm Bidiiddkkklsgggg.g
Quang Pham, Chenghao Liu, and Steven HOL. Continual normalization:
Rethinking batch normaliza- tion for online continual learning. In International Conference on Learning Representations, 2022.
URL https: / opw.revieeeeeeeeeeeeeeeeee net forum? id-vwLLQ HwqhZ.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's 's hidden in randomly weighted neural network? In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental few-shot learning with attention attractor networks. Advances in Neural Information Processing Systems, 32, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 044,066606 2016.
Joan Serra, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548 4557. PMLR, 2018.
12

---

Published as a conference paper at ICLR 2023
Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. Advances in Neural Information Processing Systems, 34, 2021.
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12183-12192, 2020.
Sebastian Thrun. A Lifelong Learning Perspective for Mobile Robot Control. Elsevier, 1995.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. Advances in Neural Information Processing Systems, 32, 2019.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184, 2020.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374-382, 2019.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581-590. IEEE, 2020.
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. In International Conference Machine Learning, pages 10820-10830. PMLR, 2020.
Jaehong Yoon, Eunho Yang. Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In Proceedings of the International Conference Learning Representations (ICLR), 2018
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. URL https:/ //op//peeiiieeeeee ew net / forum? d= F9D- 5WNG4Nv.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 12455-12464, 2021.
Da-Wei Rei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan. Few-shot class- incremental learning by sampling multi-phase tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence 2022.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros. signs, and the supermask. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jue Jun Zha. Self-promoter prototype refinement for sho- class incremental learning. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 6801-6810. 2021.
13

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
We validate the effectiveness of the soft-subnetwork in our method on several benchmark datasets against various architecture-based methods for Few-Shot Class Incremental Learning (FSCIL). To proceed with the details of our experiments, we first explain the datasets and how we involve them in our experiments. Later, we detail experiment setups, including architecture details, preprocessing, and training budget.
A.1 DATASETS
The following datasets are utilized for comparisons:
CIFAR-100 CIFAR- 100, each class contains 500 images for training and 100 images for testing. Each image has size of 32 x 32. Here, we follow an identical FSCIL procedure in (Shi et al., 2021), where we divide the dataset into base session with 60 base classes and eight novel sessions with 5-way 5-shot problem each session.
minilmageNet minilmageNet consists RGB images from 100 different classes, where each class contains 500 training images and 100 test images of size 84 x 84. Originally proposed for few- shot learning problems, minilmageNee is part of a much larger ImageNet dataset. Compared with CIFAR- 100, the minilmageNet dataset is more complex and suitable for prototyping. The setup of minilmageNet is similar to that of 110. To proceed with our evaluation, we follow the procedure described in (Shi al., 2021), where incorporate 60 base classes and eight novel sessions through 5-way 5-shot problems.
CUB-200-2011 CUB-200-2011 contains 200 fine-grained bird species with 11, 788 images with varying images for each class. To proceed with experiments, we split the dataset into 6, 000 training images and 6, 000 test images as in (Tao al., 2020). During training, We randomly crop each image to be of size 224 x 224. We fix the first 100 classes as base classes, where we utilize all samples in these respective classes train the model. On the other hand, we treat the remaining 100 classes as novel categories split into ten novel sessions with 10-way 5-shot problem in each session.
A.2 EXPERIMENT SETUPS
We begin this section by describing the setups used for experiments in CIFAR-100 and minilmageNet. After that, we proceed with follow-up discussion on the configuration we employ for experiments involving the
dataset.
CIFAR-100 and minilmageNNe. For experiments in these two datasets, we using NVIDIA GPU RTX8000 on CUDA 11.0. We randomly split these two datasets into multiple sessions, as described in the previous sub-section. We run each algorithm ten times for experiments a fixed split and report their mean accuracy. We adopt ResNet18 (He et al., 2016) as the backbone network. For data augmentation, we use standard random crop and horizontal flips. During the training stage in the base session, we select top-c% weights at each layer and acquire the optimal soft-subnetworks with the best validation accuracy. For each incremental few-shot learning session, we train our model for six epochs with a learning rate is 0.02. We train new class session samples using a few minor weights of the soft-subnetwork (conv4x layer of ResNet18 and conv3x layer of ResNet20) obtained by learning at the base session.
both datasets with
CUB-200-2011. Besides experiments in the previous two datasets, we conducted an additional experiment on this dataset. We prepare this dataset following the split procedure described in the previous sub-section. We run each algorithm ten times and report their mean accuracy. We also adopt ResNet18 (He et al., 2016) as the backbone network and follow the same data augmentation in the previous two datasets. We follow the same base- buse-session training procedure as in the other two datasets. In each incremental few-shot learning session > 1, the total number of training epochs is 10, and the learning rate is 0.1. We train new class session samples using a few minor weights of the sobt-subbe (cony4x layer of ResNet18) obtained at the hase session.
14

---

Published as a conference paper at ICLR 2023
B RESULTS AND CONCLUSIONS
To expand upon the results of paper, conduct more experiments on various datasets mentioned in the previous section. We first display the full performance table with more capacity values c employed towards our method in Table 9 and Table 11. Next, we identify how choosing different architecture would impact the performance of our algorithm in Table 10. Furthermore, we analyze the performance of our method on the
200-2011 dataset in Table 7.
Through extensive experiments, we deduce the following three conclusions for incorporating our method in the few-shot class incremental learning:
Structure. We identified SubNetwork of ResNet18 and ResNet20 with varying capacities on CIFAR-100 for the 5-way 5-shot FSCIL setting as shown in Table and Table 10. First, according to both tables, our method performs better we use more parameters within our network. In addition, as denoted our paper, we see how effective subnetwork is by observing how HardNet, with only 50% of its dense capacity, achieves comparable performance to methods utilizing dense networks, while SoftNet can do the same with only 30% of its dense capacity. Furthermore, we argue that method is grchitecture-deendeen
Our observation from Table 10 shows that at ResNet18, our architecture performs the best at the maximum capacity of c = 99%, while at ResNer20, we achieve the optimum performance at 90%.
Comparisions of Hard and SoftNet. Furthermore, increasing the number of network parameters leads better overall performance in both subnetworks types, shown in Figure 4 and Figure 5, Subnetworks, in the form of HardNet and SoftNet, tend to retain prior (base) session knowledge denoted in dashed (----) line, and HardNet seems be able classify new session class samples without continuous updates stated in dashed-dot (--) line. From this, we could expect how much previous knowledge HardNet learned at the base session to help learn new incoming tasks (Forward Transfer). The overall performances of SoftNet are better than HardNet since SoftNet improves both base/new session knowledge by updating minor subnetworks. Subnetworks have broader spectrum of performances on minilmageNet (Figure 5) than on CIFAR-I00 (Figure 4). This could be an observation caused by the dataset complexity i.e., if the minilmagenet dataset is more complex or harder to learn for subnetwork or deep model as such subnetworks need more parameters to learn minilmageNet than the CIFAR- dataset.
Sctted
Satted with 6%
Satted C4
150
75
Sessions
Sessions
Sessions
(a) = 10.0%
(b) c 30.0%
(c) c= 80.0%
Figure 4: Performances of HardNet v.s. SoftNet on CIFAR-100 for 5-way 5-shot FSCIL: the overall performance depends on capacity and the softness of subnetwork. Note that solid( -), dashed and dashed-dot(---) lines denote overall, base, and novel class performances respectively.
Smoothness of SoftNet. As emphasized in Table 11, SoftNet has a broader spectrum of performances than HardNet on minilmageNet. 20% of minor subnet might provide a smoother representation than HardNet because the performance of SoftNet was the best approximately at 80%. From these results, we could expect that model parameter smoothness guarantees quite competitive results. To support the claim, we prepared the loss landscapes of a dense neural network, HardNet, and SoftNet on two Hessian eigenvectors (Yao et al., 2020) as shown in Fig. 7. We observed the following points through simple experiments:
From these results, we can expect how much knowledge the specified subnetworks can retain and acquire each dataset.
15

---

Published as a conference paper at ICLR 2023
Satted with Heritet with cath 6%
Satted with Harttet
Satied with cxt00%
Hormict with outh
Sessions
(a) c= 10.0%
(b) 30.0%
(c) 80.0%
Figure 5: Performances of HardNet SoftNNet minilmageNet for 5-way 5-shot FSCIL: the overall performance depends capacity and the softness of subnetwork. Note that solid( ), dashed(-- and dashed-dot(- - lines denote overall, base, and novel class performances respectively.
The loss landscapes of Subnetworks (HardNet and SoftNet) were flatter than those of dense neural networks.
The minor subnet of SoftNet helped find a flat global minimum despite random scaling weights in the training process.
Moreover, we compared the embeddings using t-SNE plots as shown in Figure 6. In t-SNE's 2D embedding spaces, the overall discriminative of SoftNet is better than that of HardNet in terms of base class set and novel class set. This 70% minor subnet affects SoftNet positively in base session training and offers good initialized weights in novel session training.
13-
10
56
-30
HardNet with 30.0% Session2
(b) SoftNet with 30.0% @ Scssion2
Figure 6: t-SNE Plots of NetdNet SoftNet on minilmage Net for 5-way 5-shot FSCIL: t-SNE plots represent the embeddings of the even-numbered test class samples and compare another. Note Sessionl Class (e:: ... 59) and Session2 Novel Class Set: [60.... 64].
Preciseness. Regarding ineined and small-sized CUB200-2011 FSCIL settings, HardNet also shows comparable results with the baselines, and SoftNet outperforms others as denoted Table In this FSCIL setting, we acquired the best performances of SoftNet through the specific parameter selections. As of now, our SoftNet achieves state-of-the-art results on the three datasets.
C CONVERGENCE OF SUBNETWORKS
Convergences of HardNet and SoftNet. To interpret the convergence of SoftNet, we follow the Lipschitz-continunn objective gradients (Bottou al., 2018): the objective function of dense networks R Rd IR is continuously differentiable and the gradient function of R, namely, VR: dRd + gd. Lipschitz continuous with Lipschitz constant 0, i.c.
(V)(0) - VR(0')|2 < 0| - 0'll for all (0, 0') C Rd
(6)
Following the same formula, we define the Lipschitz-comtinuous objective gradients of subnetworks follows:
16

---

Published as a conference paper at ICLR 2023
14
16
Los5a
Losse
64
64
64
64
(a) epoch=70, 10.0%
(b) epoch=100, 10.0%
Figure 7: Loss landscapes of DenseNet, HardNet, and SoftNet: Subnetworks provide more flat global minimum than dense neural networks. To demonstrate the loss landscapes, trained simple Suree-layered fully connected model (fc-4-25-30-3) on the Flower dataset (which three classification problem) for 100 epochs.
VRR O m) R @ m)ll2 S LII(0 0') @ ml| for all (0,0') C Rd.
(7)
where m is binary mask. In comparision of Eq. 6 and 7, we use the theoretical analysis (Ye et al., 2020) where subnetwork achieve faster rate R(0 @ m = ([1[[|ml]) at most. The comparison follows:
R(O m) VR(0' o m)
VR(O) VR(0')l2 S
[ 0') @ ml
0 0'1
The smaller the value is. the flatter the solution (loss landscape) has. The equation is established from the relationship R(O O m) & (0), where R" denotes the best possible loss achievable by
convex combinations all parameters despite ( 0') C m | 0'|]. Furthermore, we have
the following inequality if R((O Mihard
R(( O m.oft]l and ||maaral] < ||
V R C Mhard)
VR(O C Mihord la
IVR( m.oft)
RR m.off)ll
O 0') ② Miarall
0') @ mooftll
where the equality holds iff [marrd Hard-WSN, and SSNt-WSN as shown in Figure an example to support the inequality.
[ We prepare the loss landscapes Dense Network,
D ADDITIONAL COMPARISONS WITH CURRENT WORKS
Comparisons with SOTA. We compare SoftNet with the following state-of-art-methodss TOPIC class split (Tao et al., 2020) of three benchmark datasets CIFAR 100 (Table 5), minilmageNet (Table 6). and CUB-200-2011 (Table 7). We summarize the current FSCIL methods as follows:
CEC Zhang et al. (2021): The authors proposed a Continually Evolved Classifier (CEC) that employs graph model to propagate context information between classifiers for adaptation.
LIMIT Zhou et al. (2022): The authors proposed new paradigm for FSCIL based meta-learning by Learning Multi-phase Incremental Tasks (LIMIT), which synthesizes fake FSCIL tasks from the base dataset. Besides. LIMIT also constructs calibration module based on transformer, which calibrates the old class classifiers and new class prototypes into the same scale and fills in the semantic gap.
17

---

Published as a conference paper at ICLR 2023
MetaFSCIL Chi et al. (2022): The authors proposed bilevel optimization based meta-learning to directly optimize the network to leam how to learn incrementally in the setting of FSCIL. Concretely, They proposed to sample sequences of incremental tasks from base classes for training to simulate the evaluation protocol. For each task, the model is learned using a meta-objective to perform fast adaptation without forgetting. Furthermore, they proposed a bi-directional guided modulation to modulate activations and reduce catastrophic forgetting.
C-FSCIL Hersche et al. (2022): The authors proposed C-FSCIL, which is architecturally composed of a frozen meta-learned feature extractor, a trainable fixed-size fully connected layer, and rewritable dynamically growing memory that stores as many vectors as the number of encountered classes.
Subspace Reg. Akyürek al. (2021): The authors presented straightforward approach that enables using logistic regression classifiers for few-shot incremental learning. The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes.
Entropy-Reg Liu et al. (2022): The authors alternatively proposed using data-free replay to synthesize data by generator without accessing real data.
ALICE Peng et al. (2022): The authors proposed a method Augmented Angular Loss Incremental Classification or ALICE inspired by the similarity of the goals for FSCIL and modern face recognition systems. Instead of the commonly used cross-entropy loss, they proposed using the angular penalty loss to obtain well-clustered features in ALICE.
Table 5: Classification accuracy of ResNet18 on CIFAR-100 for 5-way 5-shot incremental learning with the same class split as in TOPIC (Cheraghian al., 2021). denotees the results reported from Shi et al. (2021). represents our reproduced results.
Method
sessions
The gap with
8
cRT Shi et al. (2021)'
12.28
69.58
65. 16
61.41
58.83
55.87
53.28
51.38
49.51
Joint-training
(20211)
72.28
59. 16.16
55.73
49.01
46.74
34
Baseline Shi et al. (2021)
72.28
68.01
64.18
60.56
57.44
54.69
52.98
50.80
48.70
.0.81
iCaRL Rebuffi et al. (2017)'
72.05
65.35
61.55
57.83
54.61
51.74
49.71
45.03
-4.48
Reholance Hou al. (2019)*
74.45
67.74
62.72
$7.
52.78
48.62
45.56
42.43
39.22
-10.29
FSLL Mazumder et (2021)"
72.28
53.21
51.77
50.93
iCaRL Rebufli et al. (2017)
64.10
53,28
41.69
34.13
27.93
25.06
20.41
15.48
13.73
-35.78
Rebalance Hou
(2019)
64.10
53.05
36.97
31.61
26.73
21.23
16.78
13.54
-35.97
TOPIC Cheraghian et (2021)
64.10
55.88
47.07
45.16
40.11 1
36.38
33.96
31.55
29.37
20.14
CBC Zhang (2021)
73.07
61.19
53.22
51.34
F2M Shi ot al. (2021)
71.45
68. 10
64.43
60.80
57.76
55,26
53.53
51.57
49.35
-0.16
LJMIT 7hou (2023)
73.81
72.09
67.87
63.89
60.
57.77
55.67
53.52
51.23
+1.72
MetaFSCII Chi et al. (2022)
74.50
70.10
66.84
62.77
59.48
56.52
54.36
52.56
49.97
+0.46
ALICE Pong
(2022)
79.00
70.50
67.10
63.40
61.20
59.20
58.10
$6.30
$4.10
+4.59
Entropy-Reg Liu et al. (2022)
74.40
70,20
66.54
62.51
59.71
56.58
54.52
$2.39
50.14
+0.63
C-FSCII Hersche ct al. (2022)
77.50
72.45
67.94
63.80
60.24
57.34
54.61
52.41
50.23
+0,72
FSLL Mazumder al. (2021)
64.10
55.85
51.71
48.59
45.34
43,25
41.52
39.81
38.16
-11.35
FSLL+SS Mazumder et (2021)
66.76
55.52
52.20
49.17
46.23
44.64
43.07
41 20
39.57
-9.94
HardNet, c- 50%
78.35
74.12
65.88
62.
59.56
57.
56.31
+4.81
HardNet, == 80%
79.27
75.38
71.11
66.68
63.32
60.06
58.16
56.40
54.31
+4,80
HardNet, c- 90%
79.22
74.77
70.89
66.41
62.90
59.48
58.10
56.13
$3.92
+4.41
SofNNer, 50%
79.88
75.54
71.64
67.47
64.45
61.09
59.07
57.29
55.33
+5.82
SoftNet, 80%
80.33
76.23
72.19
67.83
64.64
61.39
59.32
57.37
$4.94
+5.45
SofNNet 90%
79.97
75.75
71.76
67.36
64.09
60.91
59.07
56.94
54.76
+5.25
Leveraged by regularized backbone ResNet, SoftNet outperformed all existing current works on CIFAR 100 as shown in Table 5. On minilmageNet Table 6 and CUB-200-201 Table 7, the performances of SoftNet were comparable with those of ALICE and LIMIT, considering that ALICE used class/data augmentations and LIMIT added an extra multi-head attention layer.
Comparisions of SoftNet and AANet. Our SoftNet and AANet Liu et al. (2021) have proposed alleviating catastrophic forgetting in FSCIL and CIL, respectively. AANet consists of mesti- one residual block learns new knowledge while another fine-tunes to maintain the previously learned knowledge. Through the learnable scaling parameter for the linear combination of the multi- ResNet features, AANet showed outstanding performances in the CSIL setting. However, AANet tends to overfit since the ResNet block's 's parameters fully used to update few new class data in FSCIL.
18

---

Published as a conference paper at ICLR 2023
Table 6: Classification accuracy ResNet18 on minilmageNet for 5-way 5-shot incremental learning with the same class split as in TOPIC (Cheraghian et al., 2021). * denotes results reported from Shi et al. (2021). 1 represents our reproduced results.
Method
sessions 5
The guap with cKT
cRT Shi et al. (2021)*
72.08
63.06
56.57
51.81
49.86
Joint-training Shi ot al. (2021)"
72.08
67.31
62.04
58.51
54.41
51.53
48.70
45.49
443
-4.43
Baseline Shi al. (2021)
72.08
66.29
61.99
58.71
55.73
53.04
50.40
48.59
47.31
-1.0
iCaRI. Rehufii al. (2017)'
71.77
61.85
58. 12
54.60
51.49
48.47
45,90
44.19
42.71
-5.6
Rehalance Hou et (2019)*
72.30
66.37
61.00
56,93
53.31
49.93
46.47
44.13
42.19
-6.12
FSLL Mazumder et al. (2021)
72.08
59.04
53.75
51.17
49.11
47.21
45.35
44.06
43.65
-4.66
iCaRL Reboffi al. (2017)
61.31
46.32
42.94
37.63
30.49
24.00
20.89
18.80
17.21
-31.10
Rebalance Hou (2019)
61.31
39.31
25.68
17.24
-34.14
TOPIC Cherighian al. (202.)
61.31
50.09
45.
41.16
37.48
35.52
32.19
29. .6.4
24.42
-23,89
IDLVQ.C Chen
(2020)
64.77
59.87
49.88
CEC Zhang et al. (202))
72.00
66.83
62.97
59.43 443
56.70
53.73
51.19
49.24
47.63
-0.68
F2M Shi al. (2021)
22.05
67.47
63.16
59.70
56.71
53.77
51,11
49.21
47.84
-0.43
LIMIT Zhou et al. (2022)
73.81
72.09
67.87
63.89
60.70
57.77
55.67
53.52
51.23
+2.92
MetaFSCIL Chi ct al. (3022)
72:04
67.94
63.77
60.29
$7.58
55.16
52,90
50.79
49.19
+0.88
ALICE Peng et (2022)
80.60
70.60
62.50
60.00
57,80
56.80
55.70
+7.39
C-FSCIL Herscho et al. (2022)
76.40
71.14
66.46
63329
60.42
57.46
54.78
53.11
51.41
+3.10
Eutropy-Reg Liu al. (2022)
71.84
67.12
63
57.01
53.95
51.55
49.52
48.21
-0.10
Subspace Reg. Akyltrek et al. (2021)
80.37
71.69
66.94
62,53
58.90
55.00
51.94
49.76
46.79
-1.52
FSI. Mazumder al. (2021)
66.48
61,75
58.
54.16
51,10
48.53
46.54
44.20
42.28
-6.03
FSLL+SS Mazumder al. (2021)
68.85
63.14
59.24
55,23
52.24
49.65
47.74
45.23
43.92
-4.39
HardNet, c= 80%
78.70
72.55
68 26
64,45
61.74
58.93
55,99
54,09
52.74
+4.43
HardNet, c= 87%
79.17
75.05
65,43
62.61
59.31
56.73
54.69
53.47
+5.16
HardNet, c= 90%
79.15
72.03
68. 76
65.32.
62,00
58.21
56.52
53.66
$3.07
+4.76
Nef e- -80%
79.37
74.31
69.88
63,40
60.75
57.62
55.67
+6.03
Not.Net. e- -8%%
79.77
75.08
70.59
66.93
64.00
61.00
57.81
55.81
54.68
+6.37
SoftNet, 90%
79.72
74.25
70.00
66.35
63.19
60.04
57.36
55.38
54.14
+5.83
Table 7: Classification accuracy of ResNet18 CUB-200-2011 for 10-way 5-shot incremental learning (TOPIC class split Tao et al. (2020)). * denotes results reported from Shi et al. (2021). represents reproduced results.
Method
sessions
The gup with eRT
cRT Shi et al. (2021)'
77.16
74.41
71.31
68.08
65.57
63.08
62.44
61.29
60.12
$9.85
$9.30
Joint crint-rriining Shi al. (2021)
77.16
74.39
69.83
67.17
64.72
62.25
59.77
59.05
$7.99
57,81
56.82
-2.48
Baseline
(2021)
77.16
74.00
70.21
66.07
63.90
61.35
60..11
58.66
56.33
56. 12
55.07
-4.23
iCaRL Rebufii et al. (2017)* 7)*
75.95
60.90
57.65
$4.51
50.83
48.21
46:95
45.74
43.21
43.01
41.27
-18.03
Rehalance Hou al. (2019)*
77.44
58.10
50.15
44,
12
34.44
31.73
29.75
27.56
26.93
25. 30
-34.00
FSLL Mazumder et (2021))
77.16
71,85
66.53
59.90
58 58.01
57. .00
56.06
54.78
52,24
52.01
51.47
-7.83
iCaRL Rebulii et al. (2017)
68.68
52.65
48.61
44.16
36.6 62
29.52
27.83
26.26
24.01
23.89
21.16
-39.92
Rehalance Hou al. (2019)
68.68
57.12
44.21
28. 78.78
26.71
66
24.62
21.52
20.12
20.06
19.87
.41.21
TOPIC Chirraghaa et al. (2021)
68.66 68.68
62.49
$4.81
49.
45.25
41.40
38.35
35,36
32.23
28.31
26.28
-34.50
SPPR Zhu et al. (2021)
68.68
61.85
57.43
52.68
50.19
46.88
44.65
43.07
40.17
39.63
37.33
-21.77
CEC Zhung
(2021)
75.85
71.94
68.50
63.50
62.43
58.27
57.73
55.81
$4,83
$3.52
52.28
-7.02
F2M ct al. (2021)
77.13
73.92
70.27
66.37
64.34
69
60.52
59.38
57.15
56.94
55
-3.41
LIMIT Zhou et al. (2022)
75.89
73.55
71.99
68.14
67.42
63.61
62.40
61.35
59.91
58.66
$7.41
.1.99
MeraFSCII. Chi
(2022)
75.90
72.41
68.78
64.78
62.96
59.99
58,
56.85
54,78
53.82
52.64
-6.66
ALICE Peng et al. (2022)
77.40
72.70
70.60
67.20
65.90
40
62.90
61.90
60.55
600. 60
60.10
-0.92
Entropy-Rog Rog Liu et al. (2022)
75.90
72.14
68.64
63.76
62.58
59.11
57.82
55.89
54,92
53.58
52,39
-6.91
Marumderr
(2021)
72.77
69.33
65.51
62.66
61.10
58.65
57.78
$7.26
55.59
55, 39
54.21
-6.87
FSLL+SS Mazunder et al. (2021)
75.63
71.81
66.16
64.32
62.61
10
58.
58.70
56.45 45
$6.41
55.5 85.82
-5.26
HardNet. c = 88%
76.89
73.40
69.77
66.
64.00
60.98
59.56
58.05
56. 05
55.84
55.20
-4.10
HardNet, 90%
77.23
73.62
70.20
66.36
64.32
61.40
58.28
56.36
55,88
58.30
-4.00
HardNet, e= 9036
77.76
73.97
70.41
66.60
64.47
61.35
59.99
58.18
56.17
55.73
58.18
-4.12
SofiNet, = 88%
78.14
74.61
71.28
67.46
65. 14
62.59
60.84
59.17
57.41
57.12
56.64
-2.6
SohNet, 90%
78:07
74,58
71.37
67.54
65.37
62.60
61.07
57.53
81.21
56.75
-2.55
SonNet e 93%
78.11
74.51
71.14
62.27
65.14
62.27
60.77
59.03
57.13
56.77
56.28
-3.02
This point makes it difficult to train AANet on a few samples even though the performance at session 1 is comparable with NettNet as shown in Table 8.
19

---

Published as a conference paper at ICLR 2023
Table Classification accuracy of 8 on CIFAR-100 for 5-way 5-shot incremental learning with the same class split as in TOPIC (Cheraghian et al., 2021). denotes the results reported from Shi al. (2021). represents our reproduced results.
sessions
The gap with cRT
Method
cRT Shi al. (2021)"
72.28
69.58
65.16
61.41
58.83
55.87
53.28
51.38
49.51
AANet Liu (2021)1
79.05
67.52
62.33
56.10
51.92
45.92
45.92
48.38
47.21
-2.30
HardNet, 50%
78.35
74.12
70.13
65. 88
62.74
59.56
57.98
56.31
54.32
+4.81
HardNet, 80%
79.27
75.38
71.11
66.68
63.32
60.06
58.
56.40
54.31
+4.80
HardNet. 90%
79.22
74.77
70.89
66.41
62.90
59,48
58.10
56.13
53.92
+4.41
SoftNet, 50%
79.88
75.54
71.64
67.47
64.45
61.09
59.07
57.29
55.33
+5.82
SoftNet. c= 80%
80.33
76.23
72.19
67.83
64.64
61.39
59.32
57.37
54.94
+5.43
SoftNet,
79.97
71.76
67.36
39.07
56.94
+5.25
E LIMITATIONS AND FUTURE WORKS
Our method employs two sets subnetworks. One is the major subnetworks, whereas the other minor subnets. Since the former serve their duty retain the base session knowledge, once the major subnetwork is tuned, there could potential loss of previously acquired knowledge. Furthermore, explicitly divide SoftNet by the magnitude criterion. As result, when SoftNet parameters exposed, the essential parameters will be vulnerable to intentional attacks. could result in the leak of knowledge maintained by SoftNet. First, avoid tunning the major subnetwork issue, the new session learner should know the model sparsity for maintaining base session knowledge, Second, to address the leaking information issue, the binary mask should be encoded by compression method to reduce model capacity and protect the privacy of task knowledge. Moreover, in FSCIL tasks, SoftNet alleviates overfitting issues while effectively maintaining base-session performance. In future work, we consider expanding the model parameters to acquire a long sequence of incoming new class knowledge depending on the data or task size, i.e., CIL tasks.
20

---

Published as a conference paper at ICLR 2023
Table 9: Classification accuracy ResNet18 on CIFAR-100 for 5-way 5-shot incremental learning. Underbar denotes the comparable results with baseline. denotes the results reported from Shi et al. (2021).
sessions
The gap with
Method
<RT Shi et (2021)
65.18
63.89
60.20
57.23
53.71
50.39
48.77
47.29
45.28
Joint-training
(2021)
53
50.84
47.33
Baseline Shi et al. (2021)
65.18
61.67
58.61
55.11
51.86
49.43
47.60
45.64
43.83
-1.45
iCaRL Rebuffi et (2017)'
66.52
57.26
54.27
50.62
47.33
44.99
43.14
41.16
-5.79
Rebalance Hou al. (2019)*
66.66
61.42
$7.29
53.02
48.85
45.68
43.06
40.56
38.35
-6.93
FSLL Mazumder al. (2021)*
65.18
56.24
54.55
51.61
49.11
47.27
45.35
43.95
42.22
-3.08
iCaRL Robuffi et (2017)
64.10
53.28
41.69
34.13
27.93
25.06
20:41
15.48
13.73
-31,55
Rebalance Hou al. (2019)
64.10
53.05
43.96
36.97
31.61
26.73
21.23
16.78
13.54
-31.74
TOPIC Cheraghian
(2021)
64.10
55.88
47.07
45.16
40.11
36.38
33.96
31.55
29.37
-15.91
F2M Shi et al. (2021)
64.71
62.05
59.01
55.58
52.55
49.96
48.08
46.28
44.67
+0.61
FSLL Mazumder al. (2021)
64.10
55.85
$1.71
48.59
45,34
43.25
41.52
39.81
38.16
-7.12
HardNet, e 10%
28.97
27.71
26.08
24.68
23.34
22.02
21.12
20.48
19.50
225.78
HardNel, 20%
37.42
35.29
33.22
31.32
29.51
27.80
26.54
25,28
24.16
21.12
HardNet, e 30%
55.47
52.37
49.38
46.53
43.88
41.50
39.58
37.82
36.06
9.22
HardNet, 40%
57.52
53.85
50.62
47.74
44.90
42.64
40.76
38.95
37.07
-8.11
HardNet, e= 50%
64.80
60.77
56.95
53.53
$0.40
47.82
45.93
43.95
41.91
-3.37
HardNet, 60%
66.72
62.21
58.14
$4.60
51.47
48.88
46.67
44.67
42.66
-2.62
HardNet, e 70%
68.27
63.52
59.45
55 $9.89
$2.91
50.30
48.27
46.25
44.22
-1.06
HardNet, 80%
69.65
64.60
60.59
56.93
53.60
50.80
48.69
46.69
44.63
+0.55
HardNet, e 90%
70.85
65.84
61.59
57.92
$4,65
51.90
49.79
47.66
45.47
+0.19
HardNet, 93%
71.22
66.20
62.00
58.34
55,04
$2.34
50.22
48.07
46.04
+0.76
HardNet = 95%
71.73
66.31
62.17
58.44
54.98
$2.20
50.17
47.97
43.87
+0.59
HardNet, 97%
71.85
66.48
62.29
58.62
55,36
$2.55
50.60
48.43
46.22
+0.94
HardNet e 99%
71.95
66.83
62.75
59.09
55.92
53.03
50.78
48.52
46.31
+1.03
SoftNet, 10%
60.77
$3.62
50.51
47.67
39.58
-5.70
SofiNett, 20%
64.67
60.69
15.
53.77
$0.76
48.28
46.24
44,23
42.31
-2.97
SoftNet.
$4.69
$1.82
SofiNeet, 40%
67. 50
63.
59.29
55
$2.53
49.85
47.85
45.84
43.85
-1.33
SoftNet.
53.11
50.62
SofiNeet, 60%
69.15
63.68
59.54
56.05
$2.72
50. 10
48.20
46.18
44.15
-1.13
SoftNet. 70%
70.92
$7.25
$1.37
SofiNett, 80%
70.38
65.04
60.94
57.26
$4.13
$1.58
49.52
47.36
45.16
-0.12
SoftNet, 90%
72.25
66.82
62.63
58.98
55.64
52.77
50.71
48.42
46.15
+0.87
SofiNet, 93%
71.38
65.93
61.89
58.20
54.87
51.83
49.82
47.57
45.47
+0.19
SoftNet, 95%
72.23
66.94
62.56
58.84
55.65
52.74
50.61
48.47
46.27
+0,99
SofiNet, c= 97%
70.88
65.72
61.38
57.88
54.63
51.82
49.57
47.30
45.19
-0.09
SoftNet, 99%
72.62
67.31
63.05
59.39
56.00
53.23
51.06
48.83
46.63
+1.35
21

---

Published as a conference paper at ICLR 2023
Table 10: Classification accuracy of 8 v.s. ResNer20 CIFAR-100 for 5-way 5-shot FSCIL with varying capacity c. Underbar denotes the comparable results with baseline. denotes results reported from Shi al. (2021).
Method
sessions 5
The gap
7
9
with cRT
ResNert 8, cRT Shi et (2021)
65.
63.89
60.20
57.23
53.71
50.39
48.77
47.29
45.28
RosNett8, Joint-trining Shi et (2021)
65.18
61.45
57.36
53.68
50.84
47.33
44.79
42.62
40.08
-5.20
8, Baseline Shi et al. (2021)
65.18
61.67
58.0 61
55.11
51.86
49.43
47.60
45.64
43.83
-1.45
ResNet18, HardNet, e- 10%
28.97
27.71
RosNet18, HardNet, e 20%
37.42
35.29
33. 22
31.32
29.51
27.80
26, 06.54
25.28
24.
-21.12
ResNet18, HardNet, 30%6
55.47
52.37
46.53
43.88
41.50
39.58
37.82
RosNet18, HardNet. c= 10%
57.52
53.85
50.62
47.74
44.90
42.64
40.76
38.95
37.07
-8.21
ResNe188 HardNet, 60%
64.80
60.77
56. 95
53.53
50.40
47,82
45.93
43.95
41.91
-3.37
ResNett,, HardNet, 80%
66.72
48.86
46.67
ResNett8, HardNet, e 70%
68.27
63.52
59.45 45
55.89
52.91
50.30
48.27
46.25
44.22
-1.06
ResNet18, HardNet, 40%6
69.65
64.60
60.59
56.93
$3.60
50.80
69
46.69
44.63
RosNett8, HardNet, c= 90%%
70.85
65.84
59
57.92
54.65
$1,90
49.79
47.66
45.47
+0.19
ResNet8, HardNet, e 98%
71.22
66.20
62.00
S8.34
$5.04
52.34
50.22
48.07
46. 044
+0.76
RosNett8, HardNet, 96%
71.73
54.98
50.17
45.87
ResNet18, HardNet, e 97%
71.85
66.48
62.29
58.62
55, 36
$2.55
50. 60
48.43
46.22
+0.94
RosNet18, HardNot. 90%
71.95
66.83
62.75
$9.09
$5.92
$3.03
50.78
48.52
46.31
+1.03
RosNet18, SoftNet. 10%
60.77
57.02
53.62
50.51
67.67
45.14
43.32
41.6
39.58
-5.7
ResNe118, SoftNet, 21%
64.67
60.69
$7.15
$3.77
S0.
48.28
46. 24
44
42.31
-2.97
RosNet18. SoftNet.
49.12
ResNett8, SoftNet, 40%%
67.50 50
63.11
59. 29
55.61
$2.53
49.85
47.85
45 84
43.85
-1433
ResNe18,, SoftNet, RosNett8, SoftNet. 60%
69.20
64.18
60.01
56.43
$3.11
50.62
46.51
44.61
69.15
63.68
59.54
56.05
52.72
50.10
48.20
46.18
44.15
-1.13
ResNet18, SoftNet, 70%
70.92
65. 16
61 00
$7.25
$4.09
$1.37
49.29
47.03
44.90
-0.38
RosNet18. SoftNet.
70.38
60.94
ResNet18, SoftNet, 901%
72.25
66.82
62.63
58.98
55.64
$2.77
50.71
48.42
46. 15
+0.87
ResNet18, SoftNet, 935%
71.38
65.93
61.89
58.20
54.87
51.83
49.82
47.57
45.47
+0,19
ResNett8, SoftNet, 93%
72.2 23
66.94
62.56
58.84
55.65
52.74
50.61
48.47
46.27
+0,99
ResNet18, SoftNet, 97%
70.88
65.72
61.38
57.88
$4.63
51.82
49.57
47.30
45.19
-0.09
RosNet18, SoftNet,
72.62
63.05
56.00
51.06
46.63
ResNer201 HardNet, 107
$1.50
50.32
47.79
45.11
42.57
40.38
38
37.43
35.69
ResNer20, HardNet. 20%
61.83
58.24
55.17
51.92
49.07
46.68
45.02
43.31
41.40
-388
ResNe120, HardNet, 3076
66.68
62.81
59.
56. 12
$3.21
50.52
48.85
46.93
44.84
-0.44
ResNer20, HardNet,
64.17
60.72
ResNer20, HardNet, e= 50%
70.73
66.55 56
63. 10
59.59
58
$3.87
$1.86
49.72
47.4 47.46
+2.18
ResNer20, HardNet,
71.90
67.64
63.84
60.22
$7.06
54.30
52.53
50.53
48.34
+3.06
ResNe120, HardNet. == 70%
71.41
67.44
63.76
61.022
56.84
54.14
52.54
50.42
48.24
+2,96
ResNer20, HardNet, 8076
71.97
67.65
63.93
60.
$7.12
54.44
52.72
50.67
48.43
+3.15
ResNe120, HardNet,
ResNer20, HardNet, 93%
72.28
67.99
64.48
60.83
$7.64
16.16
53.27
$1.11
48.93
+3.65
ResNer20, HardNet, 95%
73.13
68.14
64.50
60.74
57.68
55.12
53.17
51.23
48.97
+3.69
ResNer20, HardNet, 97%
71.90
67.81
64.11
$7.14
54.41
74
50.71
48.0
+3.39
ResNer20, SoftNet, 10%
$3.13
49.73
46.85
44.01
41.5
39.45
37.84
36.30
34.62
-10.66
ResNer20, SoftNet,
60.15
43.75
ResNer20, SoftNet, 31%
64.65
$9.99
56.65
$3.45
50.42
48.02
46.48
44.61
42.4 43
-2.85
ResNer20, SoftNei,
66.77
62.55
59.35
55.63
$2.70
50.21
48. 54.54
46.60
44.50
ResNer20, SoftNet. 50%
68.20
64.21
60.78
57.15
54.20
51.53
49.84
47.92
45.84
+0.56
ResNer20, SoftNet, 60%
68.63
64.76
00
57.53
$4.56
52.11
50.34
48.41
46.30
+1.02
ResNer20, SoftNet,
51.38
ResNer20, SoftNet, 80%
70.87
66.47
62.85
59.28
$6.27
$3.84
52.01
50. 16
48.01
+2.73
ResNer20, SoftNei, 90%
73.63
68.60
64.96
61.25
57.98
55.32
53.48
51.46
49.20
+3,92
ResNer20, SoftNet, 93%
72.53
68.45
59
60.80
$7.48
54.1
51.03
48.83
+3.55
ResNer20, SoftNet, 933
72.58
68.30
64.57
60.83
57.67
54.86
53.11
51.21
49.06
+3.78
ResNer20, SoftNet,
72.80
68.46
64.61
60.90
$7.63
53.26
48.95
+3.67
ResNer20, SoftNet, 99%
71.78
67.79
63.86
60.07
$7.05
54.32
52.34
50.28
48.11
+2.83
22

---

Published as a conference paper at ICLR 2023
Table 11: Classification accuracy of R minilmageNee for 5-way 5-shot incremental learning. Underbar denotes the comparable results with baseline. denotes results reported from Shi et al. (2021).
sessions
The gap
Method
8
9
with cRT
(2021)
46.78
Joint-tmining Shi et al. (2021)
67.30
62.34
57.79
54.08
50.93
47.65
44.64
42.61
40.29
-4.56
Baseline (2021)
67,30
63.18
59.62
56.33
53.28
50.50
47.96
35.85
43.88
-0.97
iCaRL Rebulli (2017)"
67.35
59.91
55.64
52.60
49.43
46.73
44.13
42,17
40.29
-4.56
Rehalance Hou et al. (2019)*
67.91
63.11
58.75
54.83
50.68
47.11
43.88
41.19
38.72
-6.13
FSLL Mazunder al. (2021)"
67.30
59.81
57.26
54.57
$2.05
49.42
46.95
44,94
42.87
-1.11
iCaRI. Rehufli et al. (2017)
61.31
46.32
42.94
37.63
30.49
24.00
20.89
18.80
17.21
-27.64
Rebalance Hou al. (2019)
61.31
47.80
39.31
31,91
25.68
21.35
18.67
17.24
14.17
-30.68
TOPIC Cheraghian al. (2021)
61.31
50.09
45.17
41.16
37.48
35.52
32.19
29.46
24.42
-20.43
IDLVQ-C Chen and Lee (2020)
64.77
59.87
55.93
52.62
49.88
47.55
44.83
43,14
41.84
-3.01
F2M Shi et al. (2021)
67.28
63.80
60.38
57.06
54.08
51.39
48.82
46.58
44.65
-0.20
FSLL Mazumder al. (2021)
66.48
61.75
58.16
48.53
46.54
44.20
42.28
-2.57
HardNet, c = 10%
9,70
8.46
8.13
7.67
7.35
6.78
6.55
6.3
5.8
-39.05
HardNet,
15.79
HardNet 30%
44.08
40.66
38.24
36.06
33.65
31.84
30.25
29.01
28.02
-16.83
HardNet,
27.53
24.76
HardNet: 50%
13
60.37
56.12
53.17
50.17
47.74
45.34
43.35
42.13
-2.72
HardNet,
73.32
47.88
HardNet 70%
71.75
66.66
62.19
58.85
35.74
52.82
$0.14
48.45
47.01
+2.16
HardNet,
HardNet, 90%
68
59,80
55.70
52.82
50.01
47.30
45.
43.34
42.09
-2.76
HardNet,
HardNet, 95%
64.72
60.13
56.05
53.25
50.20
47.62
45. 45.11
43.40
42.33
-2.52
HardNet,
HardNet, 90%
67.28
17.
58.06
55.05
52.01
49.12
46.92
45.07
43.90
-0.95
SoftNet e= 10%
62.75
58,26
$3.80
50.82
47.668
44.86
42.05
39.86
38.15
-6.70
SofNNet 20%
68.33
62.76
58.60
55.25
$2.07
49.36
46.48
44.21
42.52
-2.33
SofNeet <= ==30%
72.20
66.92
62.56
59.16
56.07
53.10
50.59
48.46
47.03
+2.18
Sof.Nei 40%
72.58
67.34
62.91
59.65
$6.72
$4.00
51.46
49.39
48.11
+3.26
SoftNet, <= ==50%
72.83
67.23
62.82
59.41
56.44
53.55
50.92
48.99
47.60
+2.75
SofLNeL 60%
73.83
67.78
63.46
60.21
$7.27
54.42
51.74
49.94
48.57
+3.72
SoftNet, <= 70%
75.15
69.06
64.79
61.40
58.38
55.49
$2.87
50.89
49.69
+4,84
SofLNeL, 80%
76.63
70.13
65.92
62.52
$9.49
56.56
53.71
51.72
50.48
+5.63
SoftNet, <= <=90%
77.00
70.38
65.94
62.45
59.32
56.25
53.76
51.75
50.39
+5.54
SoftNet, 93%
73.97
67.39
63.35
51.61
49,71
48.45
+3.60
SofNet, e= 95%
76.22
69.64
65.22
61.91
58.84
55.75
53.07
51.18
49.84
+4.99
SoftNet, 97%
77.17
70.32
66.15
62.55
56.46
53.71
51.68
50.24
SofNet, <= 99%
76.80
69.79
65.44
62.01
58.87
55.94
53.21
51.25
50.04
+5.19
23


Reviews:
[{"review_id": "rJDhndHzRJ", "reviewer": "Reviewer_5XQs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Regularized Lottery Ticket Hypothesis inspired network to deal with the well-known catastrophic forgetting problem. By only updating partial weights in the network, the proposed method claims to have a good performance on both the classes in previous and current sessions.", "summary_of_the_review": "Despite good paper writing, the novelty and the experimental performance are the largest issues of this paper. Most of the baseline methods are from 2 years ago. An important baseline in FSCIL, CEC (Zhang et al., CVPR 2021), is even not compared with. \n\nSee weaknesses for details. ", "strength_and_weaknesses": "Strengths: \n\n1). The paper is well present. In particular, Fig 1 and Algorithm 1 are clear and informative. \n\n2). The method is simple and easy to follow.\n\nWeaknesses:\n\n1). From the comparison experiments, it is hard to distinguish whether the proposed method has a good performance since there are plenty of works for few-shot class-incremental learning that the paper fails to have a comparison with. \n\n- Few-shot incremental learning with continually evolved classifiers. (CVPR ‘21)\n- Few-shot class incremental learning by sampling multi-phase tasks. (TPAMI)\n- Subspace regularizers for few-shot class incremental learning. (ICLR ‘22)\n- Metafscil: A meta-learning approach for few-shot class incremental learning. (CVPR ‘22)\n- Constrained few-shot class-incremental learning. (CVPR ‘22)\n- Few-shot class incremental learning via entropy-regularized data-free replay. (ECCV ‘22)\n- Few-shot class-incremental learning from an open-set perspective. (ECCV ‘22)\n\nAs an ICLR 23 submission, most of the baseline methods in this paper are from 2 years ago. An important baseline in FSCIL, CEC (Zhang et al., CVPR 2021), is even not compared with. I want to note that the above-mentioned methods seem to outperform the proposed method in this paper by a large margin. \n\n2). The proposed method seems to have a better result than F2M in the final session, but with a much better results in the session 1, which may indicate that the proposed method cannot prevent the network from forgetting, which is opposite from what is claimed in the introduction.\n\n3). The proposed method sounds not that new to me. I found the authors fail to give a reference for AANet (Adaptive Aggregation Networks for Class-Incremental Learning, CVPR ‘21). However, from my own perspective, the proposed method shares a similar motivation with AANet since both want to alleviate the catastrophic forgetting by fixing some parameters inside the network. Though CIL and FSCIL are different benchmarks, AANet seems to perform better and have a capability of preventing catastrophic forgetting. The authors should discuss the comparison with AANet in details. \n\n\n-------After rebuttal-----\n\nThe authors offer detailed response and clarification. I am happy to increase the score, but I sitll have the concern about the similar motivation to AANet. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JOzZI2LrR0a", "reviewer": "Reviewer_5Go7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a brand-new few-shot class-incremental learning method called soft-subNetworks (SoftNet). The two main problems in few-shot class-incremental learning contains catastrophic forgetting and overfitting. The former one asks the new session training not to interfering the former session and the latter one asks only updating a few parameters irrelevant to previous tasks. Inspired by Regularized Lottery Ticket Hypothesis, the paper points out that the subnet can perform on-par or better than the whole network. Taking usage of this hypothesis, the authors split the network into a major subnetwork and a minor one. In the base classes session, soft-subnetwork parameters and weight score are learned. In the incremental learning session, minor parameters of the subnetwork are updated. The SoftNet surpasses the state-of-the-art baselines over datasets.", "summary_of_the_review": "The paper outstands for its novel and the ingenious subnetwork training strategy. Also, the surpassing experiment results on benchmarks convince its idea. However, the theoretical analysis is quite week so that the readers cannot get the clear knowledge of the mechanism of subnetwork. The theoretical analysis should include how the split of major and minor help the forgetting and overfitting. Overall, the advantage outweighs its disadvantage so it can be recommended to the community.", "strength_and_weaknesses": "Strengths:\n1.\tBorrowing idea from regularized lottery ticket hypothesis to incremental learning is a novel and ingenious idea for its both maintaining performance in subnetwork and solving overfitting.\n2.\tThe paper is well written for its clear logic, figures, equations, and organization.\n3.\tThe outstanding performance of SoftNet is convincing.\nWeaknesses:\n1.\tThe choice of top-c weights is handcraft without theory support. Just comparing results on overall accuracy is not enough. The authors can refer to the works in subnetwork to refine their analysis.\n2.\tThe random choice of parameter follows uniform distribution, but condensation argues that there are always some weights much important than the others. The authors should explain why they use uniform distribution or the future improvement in this prior.\n3.\tIs there any possibility of extending SoftNet to the other kind of incremental learning? Or is the SoftNet limited to only class-incremental learning?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-ge2vLhegsB", "reviewer": "Reviewer_ZoLy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by Regularized Lottery Ticket Hypothesis, which hypothesizes that smooth subnetworks exist within a dense network, this paper propose Soft-SubNetworks (SoftNet), an incremental learning strategy that preserves the learned class knowledge and learns the newer ones. The SoftNet jointly learns the model weights and adaptive soft masks to minimize catastrophic forgetting and to avoid overfitting novel few samples in Few Shot Class-Incremental Learning. Experiments are conducted on the CIFAR-100 and miniImageNet datasets under various settings.", "summary_of_the_review": "The paper tackles an important problem, proposing an incremental learning strategy that preserves the learned class knowledge and learns the newer ones.  Experiments are comprehensive and seem to show improved performance in two common benchmarks. \n\nI have carefully reviewed all the comments and responses. Though a reviewer show concerns about the inadequate comparison with SOTAs, I still consider that this paper provides an interesting and effective idea. I would like to keep my original rating.", "strength_and_weaknesses": "Strengths:\n\nS1) Few-shot Class Incremental Learning is an important and active topic in the community, which is addressed in this work.\n\nS2) The paper is generally well written and easy to follow.\n\nS3) The idea of jointly learns the model weights and adaptive soft masks to minimize catastrophic forgetting and to avoid overfitting novel few samples is interesting and makes sense. \n\nS4) The method significantly surpasses many State-Of-The-Art methods under various variants on the benchmarks.\n\nWeakness:\n\nW1) Some of the details are not fully discussed. For example in section 3.3, why the update scheme of $\\theta$ can effectively regularize the weights of the subnetworks for incremental learning.\n\nW2) Though the effectiveness of SoftNet is verified via extensive experiments. These experiments are conducted only on two datasets (CIFAR-100 and miniImageNet), which seem not adequate enough to show the robustness of the propsed method. Results on other popular benchmark datasets such as CUB200 may add to the convincingness of the SoftNet.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster