Warping the Space: Weight Space Rotation for Class-Incremental Few-Shot Learning
Authors: Do-Yeon Kim
Dong-Jun Han
Jun Seo
Jaekyun Moon
Venue: ICLR 2023 notable top 25% (2023)
Keywords: incremental few-shot learning
catastrophic forgetting
parameter space
weight space rotation

Abstract: Class-incremental few-shot learning, where new sets of classes are provided sequentially with only a few training samples, presents a great challenge due to catastrophic forgetting of old knowledge and overfitting caused by lack of data. During finetuning on new classes, the performance on previous classes deteriorates quickly even when only a small fraction of parameters are updated, since the previous knowledge is broadly associated with most of the model parameters in the original parameter space. In this paper, we introduce WaRP, the \textit{weight space rotation process}, which transforms the original parameter space into a new space so that we can push most of the previous knowledge compactly into only a few important parameters. By properly identifying and freezing these key parameters in the new weight space, we can finetune the remaining parameters without affecting the knowledge of previous classes. As a result, WaRP provides an additional room for the model to effectively learn new classes in future incremental sessions. Experimental results confirm the effectiveness of our solution and show the improved performance over the state-of-the-art methods.

Published as a conference paper at ICLR 2023
WARPING THE SPACE: WEIGHT SPACE ROTATION FOR CLASS-INCREMENTA FEW-SHOT LEARNING
Do-Yeon Kim', Dong-Jun Han Jun Seo Jaekyun Moon'
1 Korea Advanced Institute of Science & Technology. 2 Purdue Univirsity, LG Al Research dy wikkkkkkiiss..e. acc kr, han7620puraue edu, jun. seol ar jmoonkkaist.edi
ABSTRACT
Class-incremental few-shot learning, where new sets of classes are provided se- quentially with only a few training samples, presents great challenge due to catastrophic forgetting of old knowledge and overfitting caused by lack of data. During finetuning on new classes, the performance previous classes deteriorates quickly even when only a small fraction of parameters are updated, since the previous knowledge is broadly associated with most of the model parameters in the original parameter space. In this paper, we introduce WaRP, the weight space rotation process, which transforms the original parameter space into a new space so that we can push most of the previous knowledge compactly into only few important parameters. By properly identifying and freezing these key parameters the new weight space, we can finetune the remaining parameters without affecting the knowledge of previous classes. As a result, WaRP provides an additional room for the model to effectively learn new classes in future incremental sessions. Ex- perimental results confirm the effectiveness of our solution and show the improved performance over the state-of toffthh-art methods.
INTRODUCTION
Humans can easily acquire new concepts while preserving old experiences continually over the course of their life span. With a growing desire imitate such ability, incremental or continual learning has been brought into the spotlight in the Al community recently (Hung et al., 2019; Wortsman et al., 2020; Saha et al., 2021). Here, due to storage and privacy constraints, the training samples of previous tasks during the training process (Desai et al., 2021). The most challenging issue in this setup is to preserve the knowledge of previous tasks against catastrophic forgetting (Serra et al., 2018). More recently, an increasing need for such learning capability when dealing with rare data (e.g., military image, medical data, photos of rare animals) has encouraged many researchers to focus
impractical save all
more challenging setup, known as class-incremental few-shot learning (CIFSL). CIFSL, each task consists of only a few training samples, making the problem much harder must additionally handle the severe overfitting issue caused by lack of training data.
Prior works on CIFSL typically take the following two steps: (i) pretraining the model on the first task (base classes) and (ii) adapting the pretrained model to new classes (novel classes) in each training session (e.g., via finetuning), assuming that the first task contains a sufficiently large number of training samples. One recent work, F2M (Shi et al., 2021), tries to find the flat local minima during the pretraining stage and then finetunes the model within this flat area for learning novel classes so that the forgetting issue could be resolved. Another line of work, named FSLL (Mazumder al., 2021), mitigates the forgetting issue by keeping some important parameters frozen and finetunes only the remaining trainable parameters during the incremental learning sessions. Several other works adopt different strategies to learn new classes well during incremental sessions without forgetting (Tao al., 2020; Cheraghian et al., 2021a: a; Zhang et al., 2021; Kukleva al., 2021; Chen & Lee, 2021; Akyürek et al., 2022).
Motivation. One key strategy shared among some of these prior works is: model update performed in the original parameter space, which is defined here with the standard busis. To illustrate the
Corresponding author.

---

Published as a conference paper at ICLR 2023
Diase
Rotate axes
Standard basis
Liasn
New basis
WaRP
u112
W11
Flat direction K
11
1812
Figure I: Visualization of loss landscape base classes after pretraining. Left: red dashed arrow lines refer to directions standard basis. Finetuning along either of these directions at each incremental session can adversely affect the loss. In this original space, it is generally challenging to capture the flat directions directly. Right: weight space rotation with desired hasis (blue dashed lines) provides with flat direction K11: following this direction,
finetune the model without performance loss on the hase classes. The performance the novel classes are also preserved by accumulating and freezing the important parameters obtained at each session.
concept of standard basis in this context, consider example with 2 x 2 weight matrix W for specific layer. By defining Wiy as the (t, j parameter of W, we can unroll W as follows:
0
1
0
0 0
0 0
W w1l
+ 112
0
0
+
122
= Wne + W12C12 + W2 C21
+ W220222
Here, we denote {e11 G12, G21, 622) ) as standard basis, upon which the weight space is param- eterized by w = {1 W12, W21, W22) Neural networks are typically implemented in the weight space spanned by E, and thus the learning processes (e.g. gradient computation, parameter update) are performed with respect to the parameters in as shown in Figure 1. However, our empirical observation in Figure 2, which showcases the effect of simple finetuning in two different spaces, suggests that the accuracies of previous classes are extremely vulnerable to model finetuning in the original space, even when we finetune only a small fraction of parameters; finetuning only 3% of model parameters (freeze 97%) throughout the incremental sessions significantly degrades the performance base classes. This implies that the previous knowl- edge is more broadly associated with, to a certain extent, most of the model parameters in the original space. Motivated by this, we pose the following question: Can we find another basis, i.e., new weight parameter space, such that we can push most of the previous knowledge compactly into only few key parameters?
madragetee
pacce (97%
poce
Original Space (57% Teusel
Oniginal Space (3% meeze)
Session
Figure 2: Original vs. New space.
Contributions. In this paper, we introduce the concept of Weight space Rotation Process (WaRP) that provides a solution to the above question. By viewing the weight matrix of a neural network from a different perspective, WaRP transforms the original parameter space into new space. For any given orthonormal set of matrices, we can reparameterize the weight in (1) as
W
Ki + 112KC12 + 121 + T222K22
(2)
updatee
freezo
"update
updatte
where B [K11, K12, K21, Koz) newly constructed basis that consists of orthonormal matrices and Wij is the weight parameter that is reparameterized according to Kig. As shown in Figure 1, we can always rotate the axes by which the weight represented for an arbitrary basis B. Here, ú12 in (2) plays key role in determining the output of the layer, then (K11 K21,, K22) can be viewed as flat directions. Thus, finetuning can be performed along the flat directions by freezing the important parameter W12 in the new space, which effectively preserves the previous knowledge (Figure 2). Figure provides high-level descriptions WaRP and our finetuning process along the flat direction.
By introducing the concept of WaRP, we propose using the low-rankness property of activation, and redefine the model parameters this new space so that we can push most of the knowledge of base classes compactly into only a few parameters. Once we construct a new weight space, we identify and freeze the important parameters for keeping the knowledge of previous classes at the end of each training session, and finetune only the remaining
strategy to construct an appropriate basis B
2

---

Published as a conference paper at ICLR 2023
parameters in the next session without performance degradation on previous knowledge. With careful construction of new basis and proper criterion for identifying important parameters, we can preserve the knowledge of base classes after finetuming the model novel classes at each incremental session. The knowledge of novel classes can be also preserved by accumulating the important parameters and keeping them frozen. It turns out that WaRP is easy implement and the proposed score criterion is efficient to compute. Experimental results on various benchmarks show the improved performance of our method over the state-of-the-art baselines.
2 WEIGHT SPACE ROTATION
Before describing our solution for CIFSL,, we first introduce the general concept of WaRP proposed in this paper. To begin with, view the weight matrix W E TRmx= as linear combination of orthonormal matrices {eij Hig. where eij E IR=x" is defined as matrix having one for the (i, j)th element and zeros for others. We can rewrite W
m
W
Wij
where Wij denotes the (i. element of W which is the coefficient of eij. Since neural networks are parameterized by coefficients w == [way] Jú, practical deep learning implementations, typical learning operations such gradient descent are conducted on the weight space spanned by the standard basis E := [eyj... However, the learned knowledge of previous tasks is likely to be associated with most of the parameters in w (i.e., most directions in standard basis E); updating a small fraction of selected parameters could give rise to significant performance degradation on previous classes as shown in Figure 2, suggesting that the previous performance is highly vulnerable to finetuning in the original space.
In this paper, we extend the above linear algebraic interpretation of the weight matrix (based on standard basis E) using an arbitrary basis B = (Kij E R E [m]].] E [n]] that spans mR Specifically, let B be an arbitrary set of orthonormal matrices satisfying: (i) (X,, K() = Of K' for KC, K' B, and (ii) ||K]|| = /(K, KC) 1 for all K. E B, where || and (.. are the norm and inner product properly defined in IR x respectively. Since a vector space H. Rm** equipped with the inner product operation (A, B) == trace(A A B) (for A, B E H. is Hilbert space, one can always rewrite W H as linear combination of orthonormal matrices in B as follows:
m ΣΣ (W, Key)K.y
W
(4)
KEB
Note that this equation generalizes (3) to the case with an arbitrary basis B; the special case with Kay eij (i.e., B = E) reduces to the result in equation (3). Thus for any orthonormal set of matrices spanning RT*x,, we can always reparameterize the layer of the model with w := {(W, XC) JiCB in place of w. In the next section, we describe how WaRP can address forgetting in CIFSL.
3 PROPOSED ALGORITHM FOR CIFSL
Problem Setup and Notations. The main goal of CIFSL is to incrementally train the model on a new set of classes at each training session, while avoiding forgetting of old classes. Let Ta be the task given in k-th session with tuples of training sample and its class label, i.e., T = {= at No where N₁. denotes the number of samples in Tr. For simplicity, we omit super/subscripts of x and vt if these can be clearly referred from the context. By defining n, as the total number of training sessions, we assume that the corresponding sets of categories in each task, denoted by C....., Cn., are completely disjoint. As in previous works (Tao et al., 2020; Shi et al., 2021), we consider a setup where a sufficiently large amount training data are available in the first task Tí. In contrast, the subsequent tasks T2 ... Tr., consist of only small number of samples per class, e.g., 5 samples per class. We consider a practical setup where all training samples of previous tasks cannot be accessed during training due to privacy and storage constraints. We use the term 'task' and 'session' interchangeably throughout the paper. We let W2 E RTT x be the weight matrix of 1-th layer in the neural network. We discard the layer index I and use W mm=x for weight whenever the context obviates it. In denotes the n x n identity matrix, [n] denotes the set {1.... n. and {a:) or tett represents {a-li I) for some index set I.
1=
3

---

Published as a conference paper at ICLR 2023
Overview of Approach. Given the model trained the first task T1. our first step is to construct a new basis via WaRP on which most directions are flat. Once new basis is constructed in the first session, this new basis is fixed throughout the whole remaining (incremental) sessions. At the end of each session, we identify the important parameters based on the scoring criterion and freeze them. In the next session, we finetune only the remaining parameters, namely, the directions of flat axes, and the same process is repeated for the future tasks. Throughout the learning process, the important parameters are accumulated and kept frozen to preserve the knowledge of both novel and base classes.
In the following, we start by describing how to construct an appropriate basis via WaRP for CIFSL and how to implement it in practice (Sec. 3.1). Then we describe our criterion for identifying the important parameters in each incremental session (Sec. 3.2). Finally, we describe the overall procedure of our algorithm for CIFSI, with details (Sec. 3.3).
3.1 CONSTRUCTING NEW BASIS FOR WEIGHT SPACE ROTATION
To find an appropriate basis to reparameterize W E Rmxx, we start by expressing the basis by using a pair of unitary matrices. Given any two unitary matrices V E Rm U E R=*"" can be seen that the set B {Kij ;= V R=x"|i E [m].j [n|]} is orthonormal, where v, and u, denote the i-th and j-th column vectors of V and U, respectively. Thus from (4), we can rewrite W as
W ΣΣ (W KW = EE,,,,,EENN (W, (W. M.
V.y.
(5)
where Whj = (W.v.u)) In other words, W is reparameterized by {wiy). using the basis [v:u]]), hig constructed with arbitrary two unitary matrices V and U. Now the question is, how to design appropriate V and U make most of the elements in the basis be flat directions?
For specific layer I, we define a matrix +; = [0)( di(())).... ás )] where (x( denotes the activation of the 1-th layer for the (-th sample in the first task Ti. Then the singular value decomposition of covariance of activations is given by
L UNE,UT
(6)
where Σ₁ is diagonal matrix having singular values on its diagonal and U7 is the corresponding unitary matrix. Here, note that the activations of deep neural networks are known to have low-rankness property shown in (Suzuki et al., 2020b; Wang et al., 2021), i.e., singular values decrease rapidly and most of the singular values are close to O. Thanks to this low-rankness property, by adopting U Uj in (5) for the 1-th layer Wi and defining wily (W1, v.u.) ) we have
Wid((x) = ΣΣ v.u Φ1(x)
Re for most (6, j) pairs by low-rankness.
for any unitary matrix V and any sample x in Ti. Namely, the effect of activation Φr(x) is highly likely to be negligible in most directions in = (v.u, }as during forward propagation. In other words, most directions in B are flat and thus only few directions and their corresponding parameters are meaningful. So if we can properly identify the important parameters among (wij)e and freeze them, finetuning along the remaining directions (for which v.u, (xx) x 0) does rarely change the output Wz)r(x) for x in T. This implics that the feature embedding of previous classes can be preserved by freezing only small fraction of parameters in this new weight space.
Implementation. Note that the actual implementation of WaRP in (5) turns out to be simple in practice. By using the fact that V = Im and UUT = Ln, we have W = where W : V WU. From the definition above, Wyy is (i. --tt element of W. Thus to apply WaRP, we just multiply unitary matrices to W and reparameterize the layer with W while V, remain fixed. Throughout the algorithm, we set V = I,,. which we found be valid choice.
VWUT
Convolutional Case. Implementing WaRP for convolutional layers is almost the same with the case of fully connectdd layers except for flattening the feature maps and reshaping the weight/unitary matrices, to make the pipelines workable as in fully-connected layer. The detailed implementation for convolutional layer can be found in Appendix A.2.
4

---

Published as a conference paper at ICLR 2023
3.2 IDENTIFYING IMPORTANT PARAMETERS
In the new weight space constructed in Sec. 1. our scheme identifies the important parameters at the end of each training session and keep freezing them throughout the training process. The remaining trainable parameters (except the accumulated important parameters) are finetuned in the next session. In this subsection, we describe our strategy for selecting the important parameters in each session.
To identify the trainable parameters (i.e.. flat directions), at the end of each k-th session, we take the score criterion that is well compatible with our new space constructed by WaRP. By the chain rule, taking the derivative of the loss function, denoted by L, with respect to the parameter wig yields:
dL/dw, = dzi v.uj (1(
(8)
(a)
where dz, denotes the gradient of the loss with respect to the output of the layer 21 = Widh (1) for
any x in Tk- Interestingly, it turns out that the term (a) in equation (8) also appears in (7). In addition,
as shown in (8), the trend of the magnitude of / follows the tendency of (a). From these
observations, it can be seen that the gradient can indirectly capture the influence of finetuning within
each direction y on Tg. Motivated by this, we consider the following score criterion:
importance score of wh
Σ DED
dd (b)/d
where La (b) is the loss computed with batch b in Tk. and D is set that consists of Ng/ng with size ny in Tk- In words, the score is computed by summing up the magnitudes of the gradients using the samples in Tg. Here, if sly is small, then Vu (1(x)) is likely to be small where Awsy simulates the gradient with respect to W.y. Thus, finetuning with respect to this Wij (following the direction of v,u,) ) on new classes may not distort the feature embedding of all previous classes.
/ng batches
Remarks. We wish to highlight two points here. First, WaRP in Sec. 3.1 and the score criterion in Sec. 3.2 work in a highly complementary manner; if we the standard basis, the flat directions limited and thus the performance could be restricted regardless of the scoring criterion. If we use a different score criterion (e.g. weight magnitude done in (Mazumder et al., 2021)) other than (9) in the new weight space, we cannot fully exploit the low-rankness as in (8) and thus cannot properly identify flat directions. These results are confirmed via experiments in Sec. 5.3. Secondly, due the simplicity of implementing WaRP, we can rely on auto-grad computation of gradient with respect to the weight matrix; we can take the derivative with respect to W rather than with respect to Wiy one by one, to obtain the scores at once. Thus it is efficient to compute.
3.3 OVERALL PROCEDURE FOR CIFSL
Based WaRP and the proposed score criterion, we describe the overall procedure of our algorithm.
First Session (k: = 1). We pretrain the embedding network and the linear classifier Wel by minimiz-
ing the cross-entropy loss (Tx)(Tk) = Σm Exsc, 1 (ut=c) log "ereut e exp(xee.)) with
exp(2e.a)
k = 1, where 1 is an indicator function and %e,t denotes the output of classifier corresponding to class c. When pretraining done, we replace the weight vector of linear classifier Was corresponding to class e with the prototype (average of feature embeddings corresponding to class c), which we found is a good option for balancing the model performance on base classes and future novel classes.
Now we construct a new basis for each layer according to Sec. 3.1 and apply WaRP. Here, to alleviate the overfitting issue on novel classes during the remaining incremental sessions, we restrict the Icarnable space by applying WaRP only to the last few layers while keeping other layers fixed throughout the whole training process. Note that once we construct a new basis in the first session, this basis is fixed throughout the remaining sessions and thus model updates performed in the new/fixed space. See Appendix A.4 for discussions on validity of the new space for novel classes.
At the end of the first session, identify the important/rrainale parameters based the proposed importance score (9) in Sec. 3.2. We select the parameters with the scores up to top a x 100% as important parameters and let the remaining ones trainable parameters, where is the parameter
5

---

Published as a conference paper at ICLR 2023
keeping ratio, a hyperparameter defined in range [0, 1]. To indicate whether each element of Wi is
important or not, we introduce a 0-1 mask M, E R Rm x as follows:
(-th element of M, = {
s Sa
v(i,) E [mm] x [nu], V selected layers / (10)
otherwise"
where So is the score value corresponding top a x 100% of the scores of all parameters in all selected layers, i.e., in ( [mi].] E [ni], selected layers l}. If the (i, element of M, is 1,
the (i, j-th element of W is regarded as an important parameter.
Incremental Sessions (k 2 2). In the beginning of each incremental session k 2 2. compute the prototypes corresponding to new classes in C& and utilize them as a classifier along with the prototypes of the base classes. To preserve the knowledge of previous classes during finetuning, only update the unimportant parameters using the importance mask M, obtained from the previous session. We block the gradient computation with respect to important parameters at each layer / as
Wi <- sttpp-grad Wi O M.) + Wi O (1 M,)
(11)
and let W/ VWUT,, where stop stop.gradd grad(-) is an operation that detaches the input from the computational graph during loss computation, and @ denotes glement-wiss multiplication. Thus, gradient descent with respect to Wi in (11) leads to model update of the unimportant parameters only. In each incremental session k; we finetune the model Wi using the cross-entropy loss (TT(TT)) for each selected layer 7, while keeping the weights of the classifier fixed during finetuning.
After finetuning is finished on task Te. we also identify the important parameters for Ta to be considered for the next session; we obtain the importance mask MI, using criterion in (10) for given a. To preserve not only the knowledge of Te but also of classes in previous sessions, we accumulate important parameters identified far:
Mi + 1 - (1 Mi) @ - Mt,)-
(12)
All the knowledge encountered so far can be properly preserved by this accumulated mask.
Finally, when each k-th session is finished, the prediction is made on the test samples from all encoun- tered classes, i.c., for given input, the prediction Vieat is given by Preat arg max tt,c, Te',. The whole process described above is repeated until the last incremental session, i.e., for k = 2,3,... ng-
RELATED WORKS
Ilase-nneremental Few-shot Learning. There have been many attempts solve class-incremental few-shot learning where only a few labeled samples are available in each task (Cheraghian et al., 2021b; Zhu al., 2021; Zhou et al., 2022b; Cheraghian et al., 2021a). To address both catastrophic forgetting and overfitting issues in CIFSL, some prior works take meta-learning approaches (Gidaris Komodakis, 2018; Ren et al., 2019; Yoon et al., 2020). They typically mimic the inference stage during training so that the model can rapidly adapt to new classes. However, these works assume that only one set of classes is given during the incremental session and do not consider a scenario in which "more than one" sets of new classes the first work that considered more than one incoming sets of classes, providing a cornerstone for future emerging studies (Akyürek et al., 2022; Chen & Lee, 2021: Zhang et al., 2021; Mazumder et al., 2021; Shi et al., 2021; ; Zhou et al., 2022a). The authors of TOPIC adopted neural gas network to maintain the topology of feature space, preventing forgetting issue. FSLL (Mazumder al., 2021) addressed the overfitting issue by choosing a few trainable parameters to be updated for finetuning and prevented forgetting issue by imposing additional regularization to minimize the deviation from the pretrained model parameters. However, this additional regularization may limit the learnability of the model on new classes. Unlike many existing works, (Shi et al., 2021: Zhou et al., 2022a) turned their focus on preparing a desired pretrained model in advance suitable for future incremental sessions. Specifically, the authors of F2M (Shi et al., 2021) inject noise to model parameters during pretraining to find the flat local minima, and finetune the model within this flat area during incremental sessions. However, pretraining with noise injection can lead to a longer training time for the model to converge and induce a little compromise of local minima. Compared to existing works where the proposed algorithms are performed in the "original parameter space' (e.g. freezing/udating the parameters in standard basis as done in FSLL.),
consecutively provided. TOPIC (Tao et al., 2020) is
the performance of the pretrained model in return for the flatness

---

Published as a conference paper at ICLR 2023
Table I: Accuracy on minilmageNet dataset under 5-way 5-shot incremental few-shot learning setup.
Session
Method
4
7
Finetuning
72.99
67.79
63.71
60.17
56.82
53.18
48.36
43.43
39.60
iCaRL (Rebuffi et al. 2017)
71.77
61.85
58.12
54,60
51.49
48.47
45.90
44.19
42.71
Rebulance (Hou et al., 2019)
72.30
66.37
61.00
56.93
53.31
49.93
46.47
44.13
42.19
GPM (Saha al., 2021)
72.99
68.04
64.18
60.96
58.32
55.66
52.90
51.10
EEIL (Castro et al. 2018)
61.31
46.58
44,00
37.29
33.14
27.12
24.10
21.57
19.58
TOPIC (Tao et al. 2020)
61.31
50.09
45.17
41.16
37.48
35.52
32.19
29.46
24.42
FSLL (Mazumder al., 2021)
66.48
61.75
58.16
54.16
51.10
48.53
46.54
44,20
42.28
FSLL4SS (Mazumder et al., 2021)
68.85
63.14
59.24
55.23
52.24
49.65
47.74
45.23
43.92
CEC (Zhang et al., 2021)
72.00
66.83
62.97
59.43
56.70
53.73
51.19
49.24
47.63
F2M (Shi 2021)
72.05
67.47
63.16
59.70
56.71
53.77
51.11
49.21
47.84
WaRP (Ours)
72.99
68.10
64.31
61.30
58.64
56,08
53.40
51.72
50.65
our method explores the 'new parameter space" where the axes are aligned with many flat directions along which the model is stably updated, by reparameterzing the weight in equation (4). With careful construction of the new weight space, we can finetune along the flat directions by properly identifying and freezing the important parameters at each session, enjoying both advantages of F2M (Shi et al., 2021) and FSLL (Mazumder et al., 2021), without any dedicated process to find the flat region in advance or additional regularization on model parameters. Although not directly targeting CIFSL, there is another work (Liu et al., 2018) that reparameterizes the weight using rotation matrix for tackling continual learning. Its focus is mainly on a good diagonalization of the Fisher information matrix (FIM) that shows up in the objective function of (Kirkpatrick et al., 2017). In our work, however, the role of rotation is different in that we rotate the axes of loss landscape for identifying/utilizing the flat directions aligned with new basis by further exploiting the low-rankness property.
Utilizing -aw-Rankness Property of Activation. There are a few works that exploit the low- rankness property of activation in deep neural network (Suzuki et al., 2020b;a; Saha et al., 2021; Wang al., 2021). This suggests that the imputs are highly concentrated on the subspace spanned by only a few number of certain orthogonal vectors. (Suzuki et al., 2020b; Wang et al., 2021) confirm this property by showing the empirical distribution of singular values at some layers. The authors of (Suzuki et al., 2020a) proposed a network compression scheme by exploiting this property which brings on small number of degrees of freedom. In (Suzuki et al., 2020b), the authors derived the tight generalization bound of deep neural network with the help of high compressibility of the model induced by this property. Another line of works (Saha al., 2021; Wang et al., 2021) try to solve continual learning problem by utilizing the low-rankness property. At each learning session, they update the model parameters using the gradient orthogonal to the subspace which the previous tasks are concentrated on. Note that our work turns out to generalize the update process of (Saha et al., 2021; Wang et al., 2021) in that we freeze the parameters of the weight matrix in a column- manner, the gradient update of our method can be reduced to that of these works as special cases.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP& & DETAILS
Datasets. We evaluate our method on three benchmark datasets, CIFAR100 (Krizhevsky et al., 2019), minilmageNet (Vinyals al., 2016) and CUB200 (Wah et al., 2011), in the CIFSL setting. For both CIFAR 100 and minilmageNNet, we split the total of 100 classes into 60 classes (for base classes) and 40 classes (for novel classes). The 40 novel classes are divided into 8 different sets having categories each, and allocated to each incremental session following the 5-way 5-shot setting. For CUB200, we split the total of 200 classes into 100 for the base classes and the remaining 100 for the novel classes, where the 100 novel classes are divided into 10 different sets having 10 categories each, following the 10-way 5-shot setting in each incremental session. We follow the same split configuration proposed by TOPIC (Tao et al., 2020) in all datasets.
Model Architecture. For the embedding network, following the standard setup in (Tao et al., 2020; Mazumder et al., 2021; Shi et al., 2021), we use ResNet20 for CIFAR 100 and use ResNer18 (He et al., 2016) for the others. Additionally, to see how a wider model affects the ability of our method for making additional room for learning new classes, also conduct experiments with R8sNet18
'The ResNet18 model architecture for CIFAR100 dataset is adopted from https: / //gi chub.com/ buaggluupyyorre wtt.bbat///aattrrrrrrrrrreeae/ /mode s/reanet.
7

---

Published as a conference paper at ICLR 2023
Table 2: Accuracy on CUB200 dataset under 10-way 5-shot incremental few-shot learning setup.
Method
Session
3
5
7
8
10
11
Finetuning
77.74
68.29
51.62
23.36
13.05
iC-RI (Rehuffi et al., 2017)
75.95
60.90
$7.65
54.51
50.83
48.21
46.95
45.74
43.21
43.01
41.27
Rebalance (Hou 2019)
77.44
58,10
50.15
44.80
39.12
34.44
31.73
29.75
27.56
26.93
25,30
(Saha al. 2021)
77.74
70.49
64.51
62.03
59.10
56.02
EEIL (Castro ot 2018)
68.68
53.63
47.91
44,20
36.30
27.46
25.93
24.70
23.95
24.13
22.11
TOPIC (Tho al., 2020)
68.68
62.49
54,81
49,99
45.25
41.40
38.35
35,36
32.22
26.31
26.28
FSLL (Mazumder et al., 2021)
72.77
69.33
65.51
62.66
61.10
58.6
58.65
57.78
57.26
55.59
55.39
54.21
FSLL+S$ (Mazumder et al., 2021)
75.63
71.81
68. 16
64.32
62.61
60.
58.82
58.70
56.45
56.41
55.82
CBC (Phang al. 2021)
75.85
71,94
68.50
63.50
62.43
58.27
57.73
55.81
54.83
53.52
52.28
F2M (Shi et al., 2021)
77.13
73.92
70.27
66.37
64.34
61.69
60.52
59.38
57.15
56.94
$3.89
WaRP (Ours)
77.74
74.15
70.82
66,90
65.01
62.64
61.40
59.86
57.95
57.77
57.01
Table 3: Accuracy on CIFARIO0 dataset under 5-way 5-shot incremental few-shot learning setup.
Scssion
Method
2
8
9
Finetuning
80.31
75.43
70.82
66.26
62.21
59.08
56.43
53.58
$1.05
GPM (Saha al., 2021)
80.31
75.54
71.27
66.97
63.63
60.48
57.92
55.57
53.18
FSLL (Mazumder et al,, 2021)
80.24
74.32
69.03
64.75
61.37
58.56
56.48
54.47
$2.27
FSLL+SS (Mazumder al., 2021)
81,42
74.79
69.33
64.97
61.60
58.92
57.06
55.28
53.12
CEC (Zhang et al., 2021)
79.69
74.77
70.24
66.13
62.93
59.94
57.67
53.63
53.44
F2M (Shi al., 2021)
79.78
74.93
70.68
66.58
60.51
58.30
56.20
53.99
WaRP (Ours)
80.31
75.86
71.87
67.58
64.39
61.34
59.15
$7.10
$4.74
on CIFAR100, which is wider than ResNer20. We randomly initialize the network before training on base classes for both CIFAR100 100 and minilmageNet. For CUB200, we deploy the Pytorch's built-in pretrained model for initialization as existing works have done.
Evaluation & Design Parameter. At each session, we evaluate our method by measuring the accuracy on test samples from all encountered classes so far. We conduct 5 simulations under different random seeds and report average values. Regarding the hyperparameters, the parameter keeping ratio is set to be a = 0.1 for all experiments. We apply WaRP to the last or 2 resnet blocks while the remaining parameters including batchnorm layers are fixed after the pretraining stage in the first session. For computing the output of the classifier, we use the standard dot product in the beginning, and replace it with cosine similarity (Gidaris & Komodakis, 2018) after adopting the proiotypes as a classifier. More detailed hyperparameter settings can be found in Appendix A.
Baselines. We consider the following recent methods as baselines: iCaRL (Rebuffi et al., 2017). Rebalance (Hou et al., 2019), GPM (Saha et al., 2021), EEIL (Castro et al., 2018), TOPIC (Tao et al., 2020), FSLL (Mazumder et al., 30.1). CEC (Zhang et al., 2021) and F2M (Shi et al., 2021). )). The performance of these baselines are mostly taken from (Zhang et al., 2021; Shi et al., 2021) for fair comparisons. We also consider simple finetuning, which is equivalent to our method with 0 using a smaller learning rate.
5.2 MAIN EXPERIMENTAL RESULTS
We first report the evaluated accuracy of each scheme on minilmageNet, CUB200 and CIFAR100 for R8sNet18, in Tables 1, and 3, respectively. Due to space limitation, the result on CIFARI00 for ResNet20 is left to Appendix A.3. As can be seen from the results in Tables 1 and 2, WaRP consistently outperforms all the baselines in whole incremental sessions. More specifically, iCaRL. Rebalance and EEIL are far below WaRP, since all these methods are not originally developed for handling the lack of data in CIFSL. The naive finetuning method shows poor performance in all results as the model is highly overfitted new classes and forgets previous ones rapidly due to scarcity of data. It is worth noting that the performance gain of WaRP against one of SOTA methods, F2M, is 2.68% for minilmageNet and 1.12% for CUB200. For CIFAR100 dataset, since there are no prior results for ResNet1 8, we reproduced the results of the following SOTA methods: GPM, FSLL, CEC and F2M. In Table 3, the trend is consistent with the results on minilmageNet and CUB200, where the performance gains over the F2M and CEC are 0.75% and 1.30%, respectively.
To see the effect of finetuning in WaRP clearly, we compare our method with a simple and strong baseline which takes the prototype of each class as classifier for both base and novel classes. Since this baseline is equivalent to the case of our method with no finetuning, we can directly see the pure gains of finetuning in WaRP by comparing with this baseline. As seen from the results on CUB200 in Table 4, this prototype-based baseline already outperforms or is comparable to SOTA methods, which is consistent with the observation from F2M (Shi et al., 2021). We can see performance gains
8

---

Published as a conference paper at ICLR 2023
Table 4: Performance improvement of WaRP over prototype- based baseline on CUB200 dataset.
Method
Seasion
2
Baseline (Prototype)
77.74
73.88
70.40
66.45
64.40
61.88
60.46
58.85 89
56.93
56.48
55.46
WaRP (Ours)
77.74
74.15
70.82
66.90
65.01
62.64
61.40
59.86
57.95
57.77
57.01
Improvement
+0.27
+0.42
+0.45
+0.61
+0.76
+0.94
+0.97
+1.02
+1.29
+1.35
GUIR20
movimageleet
midtrageNet
80
88
75
100
Accurasy (N)
Easo WORF
Accuracy (%g
16
Essan FBLL
han Ea (WuhP
Accuresy
Wo
3.50
16
Novel Protsypp
Nisuel FSI
Stunekrt M5
48
I'wostype
2.41
2.30
20
45
16
10
11
6
8
Session
Sesaion
Session
(a) Accuracy on base/novel classes.
(b) Effect basis and score.
(c) Performance with varying a.
Figure 3: Further studies WaRP using CUB200 and min/ImageNet datasets.
of WaRP over this strong baseline coherently in all incremental sessions, where the gain increases as the session progresses.
5.3 FURTHER STUDIES ON WARP
Performance on Base and Novel Classes. In Figure 3a, we report the accuracies on base classes and novel classes separately. We compare our WaRP with FSLL (Mazumder et al., 2021), which also finetunes the model by selecting trainable parameters as ours, but in the original parameter space detined with the standard basis. We also report the performance of the prototype-basee baseline considered in Table 4. We have the following key observations in Figure 3a. First, for all sessions, the base class performance of WaRP sacrifices little compared the prototype baseline (0.61% drop in the last session) while achieving better result than FSLL (1.40% gain in the last session). This indicates that WaRP can effectively keep the knowledge of base classes throughout the learning process. Regarding the novel classes, WaRP has considerable gains compared to both FSLL (1.2% 2.8% gain) and the prototype- baseline (3%-5% gain), indicating that WaRP effectively provides additional room for learning new classes in the new space, compared to FSLL which may limit the learnability on new classes due to additional regularization. The overall results confirm the ability of WaRP for preserving the knowledge of previous classes and learning new classes.
Ablation on Choices of Basis/Score. To see how the proposed new basis and score criterion work in highly complementary manner, in Figure 3b, compare the performance of 4 different (basis, score) combinations: we consider standard basis and new basis for the basis candidates. For the score criterion, we consider weight score (Ws) and gradient score (Gs), which the magnitudes of weight and gradient, respectively. As we can see, if we do not apply our new space and score criterion simultancously, the performance extremely degrades as the session progresses while our method shows much better performance than other combinations.
Ablation on Varying a. In Figure 3c, consider ablation on varying a ranging from 0.01 to 0.50 to see the robustness of WaRP against the perturbing hyperparameter. For an extreme case with 0.01, the accuracy is low since most of model parameters are updated and thus the previous knowledge cannot be preserved. However, if is larger than a specific small threshold (0.05 in this case), WaRP consistently provides additional gain against the baseline, suggesting that the performance gain is robust to perturbing
CONCLUSION
In this paper, we introduced the concept of weight space rotation process (WaRP), general framework for reparameterizing the weight matrix of the neural network. By taking advantage of WaRP, we proposed a novel strategy for CIFSL that effectively learns new classes while preserving previous knowledge, alleviating forgetting/overfiting issues. Experimental results on various benchmarks confirmed the advantage of our method over the state-of-the-art Although we focused on CIFSL in this paper, the general concept of WaRP can be utilized in any deep learning applications where the knowledge of the model should be kept in few parameters, which leave for future work.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The detailed experimental setups including datasets, model architectures, evaluation metrics and hyperparameters are described in Sec. 5.1 and Appendix A.I. We also provide the source code in Supplementary Material. We run the simulations times with different random seeds using NVIDIA GeForce RTX 3090 GPU machine and report average values. The code is available at https: /tt/t/ttttubbb com/ Edwi www.winiiimmemmm n -IRP-CIFSI.
ACKNOWLEDGMENTS
This work supported by Samsung Electronics Co., Ltd. and by IITP funds from MSIT of Korea (No. 2020-0-00626).
REFERENCES
Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya, and Jacob Andreas. Subspace regularizers for fho- class incremental learning. In International Conference Learning Representations, 2022.
Francisco M Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European conference computer vision (ECCV), pp. 233-248, 2018.
Kuilin Chen and Chi-Guhn Lee. Incremental few-shot learning via vector quantization in deep embedded space. In International Conference on Learning Representations, 2021.
Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Semantic-aware knowledge distillation for few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2021a.
Ali Cheraghian, Shafin Rahman, Sameera Ramasinghe, Pengfei Fang, Christian Simon, Lars Peters- son, and Mehrtash Harandi. Synthesized feature based few-shot plass-incremente learning mixture of subspaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8661-861 16760, 2021b.
Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta-learning approach for few-shot class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 44166-14155. 2022.
Pradnya Desai, Phung Lai, NhatHai Phan, and My T Thai. Continual learning with differential privacy. In International Conference on Neural Information Processing, pp. 334-343. Springer, 2021.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4367-4375, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Constrained few-shot class inseemmaatmettal learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 9057-9067, 2022.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Ching Yi Hung, Cheng- Hao Tu, Cheng- En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances Neural Information Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2023
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3524 2017.
Grabska-Barwinskee Barwinska, et al. Overcoming
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10, cifar-100 (canadian institute for advanced research). URL: http://www. cs. toronto. edu/kriz/cifar: himl (Last accessed, 2019.
Anna Kukleva, Hilde Kuehne, and Bernt Schiele. Generalized and incremental few-shot learning by explicit learning and calibration without forgetting. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 9020-90299 2021.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruning based connection sensitivity. In International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Haman Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017.
Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class-incremental learning via entropy-regularizee data-free replay. In European Conference on Computer Vision, pp. 146-162. Springer, 2022.
Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting, In 2018 24th International Conference on Puttern Recognition (ICPR), pp. 2262-2268. IEEE, 2018.
Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Federico Perazzi, and Sun- Yuan Kung. Content-aware gan compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12156-111 2215-11216, 2021.
Pratik Mazumder, Pravendra Singh, and Piyush Rai. Few-shot lifelong learning. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 35, pp. 2337-2345, 2021.
Sylvestre-Alviss -Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental few-shot learning with attention attractor networks. Advances Neural Information Processing Systems, 32, 2019.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548-4557. PMLR, 2018.
Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. Advances in Neural Information Processing Systems, 34:6747-6761, 2021.
Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization emror. In IJCAI, 2020a.
Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura. Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network. In International Conference on Learning Representations, 2020b.
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incrent incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2219-1 2020.

---

Published as a conference paper at ICLR 2023
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 184-193, 2021.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184, 2020.
Sung Whan Yoon, Do- Yeon Kim. Jun Seo, and Jackyun Moon. Xtarnet: Learning to extract task- adaptive representation for incremental few-shot learning, In International Conference on Machine Learning, pp. 10852-10860. PMLR, 2020.
Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12465-12464, 2021.
Da-Wei Zhou, Fu- Yun Wang. Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9046-9056, 2022a.
Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De De-Chuan Zhan. Few-shot class- incremental learning by sampling multi-phase tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha. Self-promoted prototype refinement for sho-shot iaasemnncementll learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6801-6810, 2021.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1 HYPERPARAMETERS
Optimizer. We use SGD optimizer with momentum of 0.9 throughout the whole sessions for all our simulations.
First Session (I: = 1). We pretrain the model with the batch size of 128 in the first session for all datasets. The number of pretraining epochs is 210 for 8 on both CIFAR 100 and minilmageNet and 300 for ResNet20 on CIFAR100. Regarding ResNet18 on CUB200, as we adopt Pytorch built-in pretrained model as done in most of previous works (Tao et al., 2020; Zhang et al., 2021), we pretrain the model using a smaller learning rate of 0.01 following (Chen & Lee, 2021), and the total number of epochs is 120. The learning rate decays by 1/10 at last 90, 60, 30 epoch to the end of training (for example, the learning rate decays at 210, 240, 270 if the total training epochs 300).
Incremental Sessions (k 2 2). Following (Mazumder et al., 2021), we finetune the model with training epochs of 30 for all datasets/architecturss in WaRP experiments. We use the learning rate of 0.01 for ResNet18 and 0.001 for ResNer20. When searching the learning rate, we randomly split trainset of base classes into train/validation so that the validation set has a ratio of about 10% in total. We also randomly sample the data from trainset of novel classes (different from the 5-shots that use for training) and use them as validation set for novel classes. We search the learning rate within the grids of [1e-1, 1e-2, 1e-3) CUB200 for ResNet18 8 and on CIFAR 100 for ResNet20. As for the minilmageNet and CIFAR 100 for ResNet18, we follow the setting of CUB200.
Overall Settings for WaRP Algorithm. Throughout the whole datasets/architeetures we randomly sample 20 batches (20 x 128 = 2, 560 samples) from base classes and use them for computing SVD in the first session. For computing score criterion, we set the batch size in equation (9) to be n, 60 for base classes and n2 = 1 for new classes. As for the layers to be applied weight space rotation, we apply WaRP the last two resnet blocks (out of four blocks in total) for ResNet18 model and apply to the last one block (out of three blocks in total) for ResNet20 model. We do not apply WaRP to the nathh-oormalization layer. Note that we freeze all the layers, to which WaRP not applied, after completing the first session. The hyperparameter of keeping ratio a at each session is set to be 0.1. The temperature for cosine similarity (Gidaris & Komodakis, 2018) in classifier is set to a constant value of 16, following (Zhang al., 2021).
A.2 IMPLEMENTATION OF WARP FOR CONVOLUTIONAL LAYER CASE
We first denote the filter (kernel) of convolutional layer by W E R.mxxkk where & is size of kernel, m is the target number of output channels and is the number of input channels, and let the activation (input of the layer) be denoted by & E R where h and w are height and width of activation feature map respectively. For computing SVD of activation map, all the patches in the feature map, that the kernel slides across, are flattened as a vector form and all these flattened vectors are stacked to be 2-dimensional matrix. For example, if the kernel size is 3 and the stride and padding are I respectively, the shape of stacked version of activation matrix is (3 x 3 X n, h x w) and the resulting unitary matrix would be Uzdim E 99 Then after we reshape the filter W into 2-dimensional weight matrix as W2-dim E M the remaining process is the same with fully-comnected layer case; we reparameterize the layer with W2-dim V2Tti Wz-dim Uz-dim where Vz.dim E mmm and
d2 E Bank xnk? are unitary matrices.
During computing the loss, there are two options we can choose implement W VWUT The first one is just to compute Wz.tm = V2.iin W2din 2-dim and reshape Wz-tm to be the same with W E mmxxx at every loss computation. This approach is easy to implement but might be computationally inefficient. To pursuit slight computational efficiency, we take the second approach: configure this layer as three convolutions with filters of V.Om m W conn E I and U E Rud? x where Veon. W cony and Ucany are reshaped version of Vzdim, W2dim and Undim respectively.
13

---

Published as a conference paper at ICLR 2023
A.3 ADDITIONAL EXPERIMENTAL RESULTS
We report additional results on CIFAR 100 for ResNer20 as shown in Table 5. As ResNet20 is too narrow (e.g., the number of channels in the last layer is 64) make additional room that can be reaped by WaRP for learning new classes, the gain is rather minimal compared to the results for wider network of ResNet18 (the number of channels in the last layer is 512) as shown in Table 2, and 3. However, as can be seen in Table 6, WaRP still consistently provides gains in all sessions against prototype-basee based baseline which we considered in Sec. 5.2. Note that the prototype- baased baseline is strong in that this baseline already outperforms or is comparable to state-of-the-us methods, which is also confirmed in F2M (Shi et al., 2021). Moreover, the effectiveness of WaRP is not restricted by the dataset; result the same dataset, CIFAR 100, for ResNet18 18 achieves seen in Table 3 and reads appreciable gains compared to ResNet20 case as shown in Table 7. Thus, the limited gain in small network can be easily handled by replacing the layers of the model with wider layers in practical deployment of our method.
ofte-of-thhearr as can be
Table 8 and Figure 4 refers to the results on repeated experiments of Table 4 and Figure 3c different datasets respectively. As shown in Figure 4, WaRP consistently provides the gains against strong baseline (prototype) once the a is larger than certain threshold value, which suggests that the WaRP is fairly robust to perturbing hyperparameter value.
Table 5: Accuracy on CIFAR 100 dataset under 5-way 5-shot incremental fow-shot learning sctup for ResNet20.
Session
Method
3
5
7
9
Finetuming
74.21
69.81
65.61
61.59
58.53
55. 55.59
53.29
51.20
49.06
iCaRL (Rebuffi et al., 2017)
72.05
65.35
61.55
57.83
54.61
51.74
49.71
47.49
45.03
Rehalance (Hou al. 2019)
74.45
67.74
62.72
57.14
52.78
48.62
45.56
42.43
39.22
GPM (Saha et al., 2021)
74.21
69.84
65.66
61.69
58.53
55.56
53.29
51.27
49.16
EEIL. (Castro et al., 2018)
64.10
53.11
43.71
35.15
28.96
24.98
21.01
17.26
15.85
TOPIC (Tao al. 2020)
64.10
55.88
47.07
45.16
40.11
36.38
33.96
31.55
29.37
FSLL (Mazumder et al.. 202
64.10
55.85
51.71
48.59
45.34
43.25
41.52
39.81
38.16
FSLL+SS (Mazumder et al., 2021)
66.76
55,52
$2.20
49.17
46.23
44.64
43.07
41.20
39.57
CEC (Zhung et al,, 2021)
73.07
68.88
65.26
61.19
58.09
55.57
53.22
51,34
49.14
F2M (Shi et al., 2021)
71.45
68.10
64.43
60.80
57.76
55.26
53.53
51.57
49.35
WaRP (Ours)
74.21
69.96
65.86
61.92
58.74
55.79
53.50
51.51
49.33
Table 6: Performance impravement of WaRP over prototype-based baseline on CIFAR100 dataset for ResNet20.
Session
Method
2
3
5
6
7
8
Bascline (Prototype)
74.21
69.83
65.67
61.72
58.55
55.62
53.30
51.27
49.11
WaRP (Ours)
74.21
69.96
65.86
61.92
58.74
55.79
53.50
51.51
49.33
Improvement
+0.13
+0.19
+0.20
+0.19
+0.17
+0.20
+0.24
+0.22
Table 7: Performance improvement of WaRP over prototype-based baseline CIFAR100 dataset for ResNet18.
Session
Method
2
7
Baseline (Prototype)
80.31
75.54
71.22
67.02
63.73
60.65
58.30
56.11
53.84
WaRP (Ours)
80.31
75.86
71.87
67.58
64.39
61.34
59.15
57.10
54.74
Improvement
+0.32
+0.65
+0.56
+0.66
+0.69
+0.85
+0,99
+0,90
Additionally, we further compare our method with more recent works (Zhou et al., 2022b; Chi et al., 2022; Hersche et al., 2022; Liu et al., 2022) on miniImageeet dataset. Note that as (Hersche et al., 2022) adopted ResNet12 network for evaluating their method, we reproduced the results of (Hersche et al., 2022) by adopting ResNet18 to match the architecture. As shown in the table 9, we confirm our method outperforms all the baselines, except for (Zhou et al., 2022b), but still shows better performance in most sessions. Moreover, we found that (Akyürek et al., 2022) provided the results with ResNet1 2 architecture done in (Hersche al., 2022), thus also evaluate
14

---

Published as a conference paper at ICLR 2023
Table Performance improvement of WaRP over prototype- based baseline minilmageNeet dataset.
Session
Method
Baseline (Prototype)
72.99
68.07
64.09
60.83
58.06
55.38
52.68
50.88
49. 19.70
WaRP (Ours)
72.99
68.10
64.31
61.30
58.64
56.08
53.40
51.72
50.65
Improvement
+0.03
+0.22
+0.47
+0.58
+0.70
+0.72
+0.84
+0.95
CIFAR100
CIFAR100
CUIR0O
06
69.0
49-4
Accuracy (M))
46
0.36
0.12
69.3
0.15
Accuracy
0.01
54
46-
45
Session
(a) Performance with varying a CIFARI00 for ResNet18.
(b) Performance with varying a CIFARIOK 100 for ResNet20.
(c) Performance with varying CUB200.
Figure Performance with varying a on different datasstss/architeccurss.
method for ResNet12 architecture and compare with (Akyürek al., 2022; Hersche et al., 2022) on minilmageNet. As can be seen in the Table 10, our method shows better performance than the baselines which is consistent with the case for ResNer18.
Table 9: Accuracy on minilmageNee dataset under 5-way 5-shot incremental few-shot setup for ResNet18.
Session
Method
2
3
4
5
6
7
8
9
LIMIT (Zhou al., 2022b)
72.32
68.47
64.30
60.78
57.95
55.07
52.70
50.72
49.19
MetaFSCIL (Chi et al., 2022)
72.04
67.94
63.77
60.29
57.58
55,
52.90
50.79
49.19
C-FSCIL Mode (Hersche et al., 2022)
64.72
59.63
55.43
51.95
48.99
46.44
43.86
41.76
39.78
C-FSCIL Mode (Hersche et al., 2022)
64.73
59.68
55.41
52.04
49.31
46.65
44.11
42.09
40.15
C-FSCIL Mode (Hersche et al., 2022)
64.73
59.75
55.23
51.97
49.05
46.36
43.62
41.43
39.69
Entropy-reg (Liu et al., 2022)
71.84
67.12
63.21
59.77
57.01
53.95
51.55
49.52
48.21
WaRP (Ours)
72.99
68.10
64.31
61.30
58.64
56.08
53.40
51,72
50.65
Lastly, we compare with (Hersche et al., 2022) on CIFARI00 dataset for ResNet12 2 (we further evaluate our method CIFARIO0 100 for ResNet12 instead of ResNet18), as (Hersche al., 2022) also provided the results on CIFAR 100 for ResNet12. Table 11 indicates that our method also shows the improved performance compared to (Hersche et al., 2022) on CIFAR100, which is consistent with the results on minilmageNet.
A.4 VALIDITY OF THE NEW BASIS ON NOVEL CLASSES
As aforementioned, once the new basis is constructed in the first session, the new basis is fixed during the whole remaining sessions. The future incoming tasks are not used reconstruct or modify the basis to avoid unexpected behavior or statistical unreliability due to their lack of training samples. Here one may argue that the new basis is suitable only for preserving the knowledge of the base classes. Recall that we apply WaRP only to the last few layers while remaining the other layers fixed to resolve the overfitting issue. Here, since the activation of each layer depends not only on the task itself but also preceding layers, the important parameters identified in the new basis are shared among the base and novel classes, i.e., the novel classes are also aligned with the new basis to some extent. To figure out this, measure the keep ratio of important parameters that accumulated as the session progressed. As shown in Figure 5, although we set keep ratio a as 0.1 for all sessions, the measured accumulated ratio does grow by multiples of 0. 1. Rather, the
15

---

Published as a conference paper at ICLR 2023
Table 10: Accuracy minilmageNet dataset under 5-way 5-shot incremental few-shot setup for ResNet12.
Session
Method
5
6
7
8
9
Subspace-rog (Akyürek ct al., 2022)
80.37
73.92
69.00
65.10
61.73
58.12
54.98
52.21
49.65
C-FSCIL Mode (Hersche et al., 2022)
76.37
70.94
66.36
62.64
59.31
56.02
53.14
51.04
48.87
C-FSCIL Mode (Hersche et al., 2022)
76.45
71.23
66.71
63.01
60.09
56.73
53.94
52.01
50.08
C-FSCIL Mode (Hersche et al., 2022)
76.40
71.14
63.29
60.42
57.46
54.78
53.11
51.41
WaRP (Ours)
82.05
77.06
73.17
70.19
67.92
65.14
62.36
60.57
59.49
Table 11: Accuracy on CIFARI00 dataset under 5-way 5-shot incremental few-shot setup for R2sNet12.
Session
Method
5
6
7
8
C-FSCIL Mode (Hersehe et al., 2022)
77.47
72.20
67.53
63.23
59.58
56.67
53.94
51.55
49.36
C-FSCII, Mode (Hersche al., 2022)
77.50
72.45
67.94
63.80
60.24
57.34
54.61
52.41
50.23
C-FSCIL Mode (Hersche al., 2022)
77.47
72.40
67.47
63.25
59.84
56.95
54.42
52.47
50.47
WaRP (Ours)
79.30
75.27
71.49
67.60
64.55
61.90
59.89
57.82
55,49
accumulated ratio increases by some value smaller than a. As can be seen from the observation, the important parameters at each session somewhat overlap since the activation also depends on the preceding layers. This suggests that the new basis constructed in the first session would be aligned with the future tasks to a certain extent. Hence the new basis is also eligible for preserving the knowledge of the novel classes with our score criterion in (9).
CIFAR100 for ResNer18
CIFAR100 for ResNet20
Ratio of Importart Params
Ratio of Importart Params
0.18
0.22
0.16
0.18
0.14
Session minlmageNet
Session CUB200
0.25
Patio of Important Params
Patio of Important Params
0.18
6.18
0.14
0.12
0.1
Session
Figure 5: Accumulated keep ratios the session progresses various datasets.
Along with this observation, we further verify the ability to preserve the novel tasks by showing how the mode accuracy novel classes varies session grows. The table 12 and 13 show the accuracy on To and Ta respectively, i.c., the first and second incremental novel task. We evaluated on minilmageNet dataset.
Moreover, to single out the effectiveness WaRP in preserving the previous knowledge, we further evaluate the accuracies
certain novel task when the model knows task ID. In other words, the prediction made when the output logits corresponding to this novel task are only given. By doing so, can exclude the effect of forgetting induced by the interference from the output logits
16

---

Published as a conference paper at ICLR 2023
Table 12: Performance the first novel task 72 each incremental session minilmageNee dataset.
Session
Method
5
Prototype
21.52 27.16
20.68
19.88
19.28
18.92
18.24
17.84
17.04
WaRP (Ours)
25.48
24.32
23.92
23.52
22.84
22.12
21.04
Table 13: Performance the first novel task T3 each incremental session minilmageNet dataset.
Session
Method
3
4
5
6
Prototype
22.32
21.48
21.00
19.68
18.48
17.96
17.88
WaRP (Ours)
27.76
26.76
25.84
23.64
22.00
21.60
21.52
corresponding to other tasks, As can be seen in the table 14 and 15, the new basis constructed using base classes is also suitable for preserving the knowledge of the novel classes.
Table Performance on the first novel task Ta on miniImageNet dataset when task ID is known.
Session
Method
3
4
5
6
7
9
Prototype
58.92
58.92
58.92
58.92
58.92
58.92
58.92
58.92 60.48
WaRP (Ours)
60.60
60.76
60.68
60.56
60.36
60.40
60.40
A.5 COMPARISON WITH EXISTING NETWORK PRUNING SCHEMES
As our method identifies the important parameters, can compare our method against the network pruning schemes. We consider three network pruning methods: two for structured (channel/filter) pruning (Li al., 2017: Liu al., 2021) and one for unstructured pruning. SNIP (Lee et al., 2019). We adopted the methods of identifying parameters, including the score criterion and the structure of identification (e.g. whether identifying channel-wise or element- wise manner) proposed in these works. Although we set the ratio of important parameters a as 0.1 in WaRP experiments, during the reproducing the results on the methods from (Li er al., 2017: Liu al., 2021; 1; Lee al., 2019), we tuned the parameter of this ratio to get their best performances. We found that a = 0.9 ~ 0.95 is suitable for (Li et al., 2017: Liu et al., 2021; Lee et al., 2019) as all them still consider the original parameter space which exhibits the fundamental limitation shown in the Figure 2. The results are evaluated on minilmageNet dataset. As can be seen in the table 16, our method outperforms all of score criterion suggested in pruning schemes considered, which indicates that the space and its compatible score criterion are very effective in solving CIFSL.
A.6 COMPUTATIONAL COST
We first separate the computational aspects of our scheme into two parts and discuss them separately: (a) constructing new basis in the first session, (b) finetunin//ifference in the incremental sessions.
(a) After pretraining the model the first session, we construct new basis by using singular value de- composition (SVD). The computational complexity of SVD is known as O(n²) where the covariance of activation is in the shape of n x n. Although constructing the new basis appears to have somewhat large complexity due to the SVD, only need to construct it just the real time taken to construct the basis is not that significant compared to that of pretraining in the first session in practice. The table 17 shows the elapsed times for constructing the basis and pretraining with only 5 epochs on minilmageNet dataset. We tested on NVIDIA GeForce RTX 3090 GPU.
in the first session. Morcover,
(b) During finetuning in the new basis, the complexity induced in the layer, to which the WaRP is applied, increases as much when training with 3 layers due V and U multiplied on both sides of
17

---

Published as a conference paper at ICLR 2023
Table 15: Performance the first novel task Ts minilmageNet dataset when the task ID is known.
Session
Method
3
4
5
6
Prototype
69.40
69.40
69.40
69.40
69.40
69.40
69.40
WaRP (Ours)
71.04
70.72
70.80
70.64
70.64
70.52
70.44
Table 16: Comparison against various network pruning schemes on minilmageNet dataset.
Scssion
Method
2
3
4
7
Filter prune cL ct al., 2017) (ratio=0.90)
72.99
66.57
56.81
41.19
29.98
24.24
19.44
17.52
13.15
Filter prune (Li et al., 2017) (ratio=0.95)
72.99
67.15
62.07
55.77
49.75
44.88
38.39
36.17
33.14
Channel prune (l.iu et al., 2021) (ratio=0.90)
72.99
66.77
61.19
$6.78
49.34
45.39
40.37
Channel prune (Liu al., 2021) (ratio=0.95)
72.99
67.81
63.75
60.58
57.86
55.26
52.50
50.71
49.55
SNIP (Lee et al. 2019) (ratio=0,90) SNIP (Lee et al. 2019) (ratio=0.95)
72.99
66.98
62,42
59.16
56.52
54.11
51.40
49.63
48.44
72.99
67.65
62.41
58.38
55.48
53.04
50.38
48.59
47.48
WaRP (Ours)
72.99
68.10
64.31
61.30
58.64
56.08
$3.40
51.72
50.65
W in practical implementation. However, this difference does not that significantly slow down the finetuning the model is finetuned only with small iterations using only few labeled samples in the actual incremental session. The table 18 shows the clapsed times for finetuning in both new and original space on minilmageNet dataset (we report averaged time per each incremental session). We tested on NVIDIA GeForce RTX 3090 GPU.
Moreover, we would like to highlight that, after completing the finetuning, we recover this layer its original one, i... W = VWU so that the inference time complexity does not increase.
A.7 NOTES ON THE SPACE OF MATRIX
In equation (4), we deal with the "space of matrix" and generalize the linear algebraic concepts (which we are familiar with in space R for example) to this matrix space. Here, we define the inner product as (A, B) trace(A A B) in Sec. In fact, this inner product actually can be seen typical dot product in the vector space RR* for any & (i.e. a b- = >im agby for any vectors a, E RR where ag and b; i-th element of and respectively), if A, B. Rmxn are flattened as vector form. In other words, it can be easily seen that (A, B3 = Ataten Buatce where Attatenn Bianca E IR flattened version of and B respectively. Thus, once W and Kij are flattened, can easily interpret the equation (4) as the flattened version of W and Kaj are in the vector space Rum and the linear algebraic concepts we are familiar with in IR* such as linear combination, orthonormal vectors directly adopted to this equation.
18

---

Published as a conference paper at ICLR 2023
Table 17: Elapsed time for constructing new basis and pretraining the model with 5 epochs.
Construct basis
Pretrain with 5 epochs
Elapsed time (sec) 136.73
123.49
Table 18: Elapsed time for finetuning in and original space.
New space
Original space
Flapsed time (sec)
1.2661
0.8472
19


Reviews:
[{"review_id": "hP3N0uTyt5z", "reviewer": "Reviewer_Z3CK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to mitigate catastrophic forgetting in class-incremental few-shot learning (CIFSL). The key observation is that there may be many directions in the weight space that are flat with respect to the loss. If these directions can be identified, then updates may be freely made in these directions without affecting performance on the previous classes. In order to do this, a method called WaRP is proposed that utilizes the SVD of the network activations to construct a rotation such that the majority of directions are flat. For each subsequent incremental learning session, only a subset of the parameters with respect to the new basis are updated. The subset of parameters to be learned in each learning session are chosen based on a score criterion that relies on the magnitude of the gradients. Experiments on several standard CIFSL benchmarks demonstrate the improvements of the proposed WaRP method relative to baselines.", "summary_of_the_review": "Update post-rebuttal: The authors have addressed my remaining concerns with additional results and ablations. I will maintain my original rating.\n\n---\n\nOverall, this is a strong submission. It addresses a meaningful problem and provides an effective and elegant solution to tackle it. Experiments show the proposed WaRP method provides consistent accuracy improvements relative to baselines.\n", "strength_and_weaknesses": "Strengths\n- Writing is clear and contains several interesting insights distributed throughout the paper.\n- The method makes intuitive sense and should be easy to implement.\n- The experiments are complete and contain a set of ablations justifying design decisions.\n- The proposed WaRP method could have impact in a variety of transfer learning scenarios beyond CIFSL. Any setting where performance on previous tasks should be preserved can take advantage of the ideas behind WaRP.\n\nWeaknesses\n- The method seems to be primarily focused on preserving accuracy with respect to the base classes rather than the incrementally learned novel classes. In particular, the basis is computed only once after training on base classes, rather than updating after each session.\n- After many learning sessions, it seems plausible that there could be a shortage of trainable parameters.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O1TMszFRH_F", "reviewer": "Reviewer_1hev", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose an algorithm to efficiently learn new classes in an incremental setup with a few-shot dataset at every iteration. Towards this, they propose WaRP - which first changes the basis of the parameter/weight space of the base network and then uses SVD to find parameters with less importance in that space. At every round, a fraction of parameters which are found to be unimportant (with a metric defined in this paper) are fine-tuned to learn new classes while the important parameters are kept fixed. Important parameters found after each iteration are added to the original set of important parameters. On standard few-shot class incremental learning (FSCIL) benchmarks, WaRP outperforms existing methods. ", "summary_of_the_review": "Overall, I found the paper to be well written, clearly motivating the proposed method along with explaining each component in details and finally showcasing the efficacy of the method on standard benchmarks against current SOTA methods and via additional ablation studies. At this point, based on my opinion alone, I am leaning towards accepting this paper. I'd still like the authors to respond to my comments - especially regarding the channel pruning style baselines.", "strength_and_weaknesses": "**Strengths**\n* The narrative of the paper is crisp, easy to follow and enjoyable to read. Authors clearly motivate why changing the basis is important (instead of staying in the original parameter space) with an explanation in the two-dimensional space. After that, authors use the existing knowledge that activations reside in a low-dimensional sub-space to find unimportant parameters in this new basis and updates only those parameters for a given task while keeping the important parameters fixed. Every major component of the proposed algorithm is well motivated, clearly explained and intuitively makes sense. \n* Apart from only providing a more theoretical overview of the proposed method, authors also provide details on how the weight-space change of basis (re-parameterization) can be implemented for a fully-connected and convolutional layer. Authors also improve the computational cost of the method by organizing the parameters in a way such that standard automatic differentiation packages can be used.\n* Experimental results are performed on multiple FSCIL benchmarks and the proposed method outperforms existing methods (on-average, ~1 point over the SOTA method F2M). Authors also provide reasonable ablation studies to showcase the importance of various components including weight space re-parameterization, amount of forgetting with the amount of parameters modified etc. - overall, the empirical evaluations properly justify the efficacy of the algorithm.\n\n**Weaknesses/Clarifications needed**\n* For me, a major missing piece in this paper is the lack of comparison against methods from the domain of neural network pruning. Although the authors compare against existing methods from the literature, but for me, an important baseline to compare this method would be to take a pruning algorithm e.g. any channel pruning method for CNN, use that to identify which channels are unimportant after every round and only fine-tune those channels for a given task. Apart from channel pruning, other methods like layer pruning or structured pruning are also relevant to this discussion.\n* Modifying the basis by re-parameterization requires non-trivial amount of work specific to each neural network operator. Authors provide guidance on how to do it for Conv/FFN, but I'd like to understand if this process can be generalized and applied to other kind of operators - multi-head attention from a Transformers network architecture or a transposed convolution operator for example. This can potentially hinder the adaptability of this method to other type of tasks or architectures.\n* I'd like to see a plot on how many parameters are staying unimportant after every round - this will help us understand how does this method scale to more number of tasks.\n* Finally, some details on the overall computational cost increase (at the beginning to change the basis and then for every iteration) compared to doing a standard fine-tuning will help us understand the cost/performance trade-off better.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-pzRiaOn5Sn", "reviewer": "Reviewer_AjwX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces weight space rotation process into the few-shot class-incremental learning. Specifically, the proposed method can identify the important parameters and freeze them in the following sessions.", "summary_of_the_review": "The paper is well-written and has a clear motivation. But the experiments do not support the advantage of the proposed method. Comparison is not enough. Necessary discussion is lacking to support the novelty. ", "strength_and_weaknesses": "Strengths: \n\nThe paper is well-written. The motivation is clear and the method is technically correct. \n\nWeaknesses:\n\n1). In experiments, the comparisons with the previous methods are not enough. The following papers are not compared, but they seem to outperform the proposed method:\n\n[1] Few-shot class incremental learning by sampling multi-phase tasks. (TPAMI)\n\n[2] Subspace regularizers for few-shot class incremental learning. (ICLR ‘22)\n\n[3] Metafscil: A meta-learning approach for few-shot class incremental learning. (CVPR ‘22)\n\n[4] Constrained few-shot class-incremental learning. (CVPR ‘22)\n\n[5] Few-shot class incremental learning via entropy-regularized data-free replay. (ECCV ‘22)\n\n[6] Few-shot class-incremental learning from an open-set perspective. (ECCV ‘22)\n\nI notice that the authors give references for [2] (introduction) and [1] (related work), but I do not find a reasons to ignore them in the comparison in experiments.\n\n2). Fixing part of the network, though seems to be novel in FSCIL, but has been studied in the very similar task CIL (Adaptive Aggregation Networks for Class-Incremental Learning, CVPR ‘21). The motivations are very similar. It would be better to have a discussion instead of claiming “first work that makes changes to the parameter space itself”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-0PQMuQxQwd", "reviewer": "Reviewer_hFPG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to rotate the weight space in a way that newly acquired knowledge does not interfere with the previous one. In the rotated space, only few dimensions are important for previous tasks and thus, they can be frozen to encode new knowledge in the remaining weights. The proposed method outperforms all the considered the state-of-the-art baselines.", "summary_of_the_review": "The proposed method is sound and achieves good performance. My main concerns are about comparison with recent papers like [A], generality (why few-shot incremental and not class incremental, for instance), and the amount of novelty. Overall I think this is a good paper and I would be willing to raise my score after discussion.", "strength_and_weaknesses": "Strengths\n=======\n* The proposed method is sound and generalizes the methods of Saha et al., 2021 and Wang et al., 2021.\n* Results are state of the art\n* The authors provide ablations and code, which makes their work more reproducible\n\nWeaknesses\n=========\n* Given the generality of the method, it is not clear why it is only tested on class-incremental learning\n* Only papers from 2021 are considered for comparison, missing works like [A]\n* I found the notation complex and I had to read the paper multiple times to understand all the equations. I still have trouble with the relationship between Hilbert spaces and equation 4.\n\n[A] Hersche, Michael, et al. \"Constrained Few-shot Class-incremental Learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%