Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation
Authors: Yixiong Zou
Shanghang Zhang
Yuhua Li
Ruixuan Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Few-shot class-incremental learning
Class-level overfitting
Margin-based classification

Abstract: Few-shot class-incremental learning (FSCIL) is designed to incrementally recognize novel classes with only few training samples after the (pre-)training on base classes with sufficient samples, which focuses on both base-class performance and novel-class generalization. A well known modification to the base-class training is to apply a margin to the base-class classification. However, a dilemma exists that we can hardly achieve both good base-class performance and novel-class generalization simultaneously by applying the margin during the base-class training, which is still under explored. In this paper, we study the cause of such dilemma for FSCIL. We first interpret this dilemma as a class-level overfitting (CO) problem from the aspect of pattern learning, and then find its cause lies in the easily-satisfied constraint of learning margin-based patterns. Based on the analysis, we propose a novel margin-based FSCIL method to mitigate the CO problem by providing the pattern learning process with extra constraint from the margin-based patterns themselves. Extensive experiments on CIFAR100, Caltech-USCD Birds-200-2011 (CUB200), and miniImageNet demonstrate that the proposed method effectively mitigates the CO problem and achieves state-of-the-art performance.

Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation
Yixiong Zou', Shanghang Zhang", Yuhua Li and Ruixuan Li
School of Computer Science and Technology, Huazhong University of Science and Technology "School of Computer Science, Peking University Hyixiongz, idcliyuhua, rxli] edu. ?shanghang@pku. edu.
Abstract
Few-shot class-incremental learning (FSCIL) is designed to incrementally recog- nize novel classes with only few training samples after the (rree)training on base classes with sufficient samples, which focuses on both base-class performance and novel-class generalization. A well known modification to the base-class training is to apply margin to the base-class classification. However, dilemma exists that we can hardly achieve both good base-class performance and novel-class gen- eralization simultancously by applying the margin during the base-class training, which is still under explored. In this paper, we study the cause of such dilemma for FSCIL. We first interpret this dilemma lem from the aspect of pattern learning, and then find its cause lies in the easily- satisfied constraint of learning margin-based patterns. Based the analysis, we propose a novel margin-based FSCIL method to mitigate the CO problem by pro- viding the pattern learning process with extra constraint from the margin-based patterns themselves. Extensive experiments on CIFAR100, Caltech-USCD Birds- 200-2011 (CUB200), and minilmageNet demonstrate that the proposed method effectively mitigates the CO problem and achieves sute-of-the-ar performance.
class-level overfitting (CO) prob-
Introduction
With the development deep learning, deep neural networks gradually demonstrate superior perfor- mance the recognition of pre-defined classes with large amount of training data [T9, I0O]. How- ever, the model's s generalization capability on the downstream novel classes is much less explored and still needs to be improved [T
D]. To deal with this problem, the few-shot class- fass-inccrmmntal B0] comes into sight. FSCIL first (pre-)traing model on set of pre-defined classes (base classes), and then generalizes the model to the incremental novel classes with only few training samples, simulating human's 's ability continually learning novel concepts with only few examples, and emphasizing both the performance the pre-defined base classes and the generalization on the downstream novel classes.
learning (FSCIL) task IS
22
2X
However, dilemma is recently revealed [TK3 5L H. [ that better loss functions, which lead to higher performance on the pre-training data, could lead to worse generalization on the downstream tasks. As introduced by [16] and depicted in Fig. III similar phenomenon also exists in the FSCIL task
that a positive classification margin [U.. 25 & [T] applied to the classification of the base-class (pre-)tte )training could lead to higher base-class performance but lower novel-class performance, while a negative margin could result in lower base-class performance but increase the novel-class perfor- mance. Although this dilemma widely exists in the tasks involving novel-class generalization such as few-shot learning (FSL) and FSCIL,, only few works [[G]] tried to explore its cause, and can hardly be used to handle it. Due to space limitation, will provide extended related works in the appendix.
CCoresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

No Margin
Positive Margin
Negative Margin
Ours
Nove! Class Base Class
Figure 1: A dilemma exists between base-class performance and downstream novel-class general- ization. By applying positive margins, base-class features are better separated which indicates better base-class performance, but the novel-class features are confused which indicates lower novel-class generalization. In contrast, by applying negative margins, base-class features are confused but the novel-class features are better separated. In this paper, we study the cause of such dilemma for the few-shot caas-inncremenal learning task, and propose method to mitigate such dilemma to better separate both base and novel classes.
In this paper, we study the cause of the dilemma in the margin-based classification for the FSCIL problem from the the aspect of pattern learning. We find this dilemma can be understood as a class- level overfitting (CO) problem, which can be interpreted by the fitness of the learned patterns to each base class. The fitness determines how much the learned patterns are specific some base classes or shared among classes, making the learned patterns either discriminative (tend to overfit base classes) transferable (tend to underfit base classes) and causes the dilemma. Based on the interpretation, we discover the cause of the dilemma lies in the easily-satisfied constraint of learning shared or class-specific patterns. Therefore, we further design a novel FSCIL method to mitigate the dilemma of CO by providing the pattern learning process with extra constraint from margin- -aaged patterns themselves, improving performance on both base and novel classes as shown Fig. and achieving late-of-the-art performance in terms of the all-class accuracy. Our contributions are:
We interpret the dilemma of the margin-bases classification as a class-level overfitting problem from the aspect of pattern learning.
We find the cause of the class-level overfitting problem lies in the easily-satisfied constraint of learning shared or class-specific patterns.
We propose a novel FSCIL method mitigate the class-level overfitting problem based on the interpretation and analysis of the cause.
Extensive experiments on three public datasets verify the rationale of the model design, and show that we can achieve state-of-the-art performance.
2
Interpreting the Dilemma of Few-Shot Class-Incremental Learning
In this section, we first describe the Few-Shot Mlass-Incrementa Learning (FSCIL) task and the baseline model, and then conduct experiments to analyze the dilemma.
2.1 Task and Baseline Description
The FSCIL task aims to incrementally recognize novel classes with only few training samples. Ba- sically, the model is first (pre-)trainee -tranne on base session), then confronted with novel classes with limited training samples (a.k.a. incremental session), and finally required recognize test samples from all encountered classes.
base classes with sufficient training samples (a.k.a.
Specifically, given the base session dataset D {(x,, y:) Pe with the label space Yo, the model trained recognize all Yol classes from Yn by minimizing the loss
Σ (o
L(b(r)),
where L(---) typically a cross-entropy loss, o(-) the predictor which is composed of backbone
network f(.)) for feature extraction and linear classifier, represented as Φ(x) WTf((z where
Φ(x) R Nox1 W e RdxNo and f(x) pdx1, Typically, f(x) and the W Le normalized [28].

---

When the kth incremental session comes, the model needs to learn from its training data DR {(2,, Me) 1.) The weight of the classifier will be extended to represent the novel label space Y imported by this session, represented as W {wf, wh, ... w " U ... {wf,... w where wf denotes the weight of the classifier corresponding to the jth class of the ktth session.
A strong baseline [28] is to freeze model's parameters to avoid the catastrophic forgetting brought by the finetuning on novel-classes. For the incremental sessions (i.e., k 0), the average of the features extracted from the training data will be used as the classifier's 's weight [28] (a.k.a. prototype)
as w m E. f(;)), where n/ denotes the number of training samples in the class j for the session k. As this baseline focuses on the base-class training, in this paper, the term training, if not otherwise stated, refers to the base-class training. Finally, the performance of the kth session will be obtained by classifying the test samples from all Σi= [Yi] encountered classes.
2.2 Margin-Based Classification
well known modification to base-class training loss (Eq. II) is to integrate a margin IC X, 25
as
TT(m,, ((;))-m)
L(I v = -log (m, (()))mm) Ejjt, e=m))(xx))
(2)
where Wyy refers to the classifier weight for class ye, is typically set to 16.0 and m is the margin.
As analyzed in [I6]), empirically a dilemma exists that a positive margin could improve the base- class performance but harm the novel-class generalization, and reversely, negative margin could contribute to the novel-class performance but decrease that of the base classes as shown in Fig. MI Similar phenomenon has been observed in other works such as [[SS]] that a better loss function for the pre-training task could harm the generalization on downstream tasks.
2.3 Interpretation of Class-Level Overfitting from Pattern Learning View
Experiments are conducted on the 11FAR100 [DS]] dataset and reported in Fig. Z CIFARIO0 con- tains 100 classes in all. As split by [22]] 60 classes are chosen as base classes, and the remaining 40 classes (with 5 training samples in each class) are chosen as novel classes". Experiments are conducted on the last incremental session, where all 100 classes are involved.
From Fig. Q (left), we can see that as the margin increases, the base-class accuracy increases while the novel-class accuracy decreases, which is consistent with [T6]] and validates the dilemma exists. Compared with the well-known overfitting between the training and testing data, such dilemma, although all validated on testing data, is more like the overfitting to base classes instead of samples. Therefore, we term it as class-level overfitting (CO), Additionally, the balance is reached when no margin is added, i.e., FSCIL cannot be improved by simply applying the margin. For such dilemma, [16] gave an explanation by the degraded mapping from novel to base classes. However, it could hardly be used develop methods for handling such dilemma. In this section, we go a step further to explain this phenomenon from the aspect of pattern learning for developing methods to handle it.
A pattern denotes part of information that the model extracts from the input, which is finer- grained level of analyzing the model's 's behavior. As studied in the interpretability of deep nets [2%, []), each channel in the feature extracted by deep networks could correspond to a certain pattern of the inpuro. which can be viewed as to compose the base and novel classes [G]]. Therefore, we conduct experiments on feature channels to study the patterns learned by applying different margins.
2.3.1 Class-Level Overfitting Interpreted by Pattern Fitness to Base Classes
Pattern's fitness to each base class. We first evaluate the sparsity of the base-class patterns, which is measured by the L norm of each feature vector. As the extracted features are L22 normalized, the smaller the T. norm is, for each feature, the sparser the patterns with high activation are. Results are plotted in Fig. E (mid), where we can see consistent decrease in T, when the margin increases, which means the model needs less activated patterns (channels) to represent each base class. As the number of activated patterns decreases, the effecctiveness of each activated pattern must increase to account for the performance increase in the base-class pre-training in Fig. 0 (left). Therefore, we hold that as the margin increases, the patterns learned by the model could fit each base class better.
2 Pleasse refer to the appendix for details.
3

---

Performance by Accuracy
Sparsity and Top Activation
Moa-top activation
Ollerellss adivallin
Transferability Relation
0.06
similarily
45.0
75.0
6.0
5.0
0.5
0.10
0.04
40.0
70.0
4.0
0.5
0.05
0.12
35.0
65.0
3.0
0.4
0.00
30,0
60.0
2.0
0.4
0.00
-0.02
norgin
0.3 0.2 0.1 0.0 0.1 0.2 0.3
mangin
-0.3
-0.2-0.1
0.0 0.1
0.2
0,3
margin
0.3-0.2-0.1 0.0 0.1 0.2 0.3
Novel -d -*-Oveall -+- Base
+.-1I - MTA
- Trans -0- Relation
Figure 2: Left: Class-level overfitting exists between base and novel classes, and simply applying margins to the training can not help the overall performance. Mid: Pattern fits base classes more as the margin increases, making it more discriminative but less transferable. Right: Transferability of patterns decreases as the margin increases, pushing classes away from each other.
Pattern fitness measured by the template-matching score. To further verify the fitness increase, we view each pattern as semantic template and measure its matching score to each base class. As analyzed in [2_]] and (I]) each pattern can be understood as a template [[] for the model to match the input (so that each class would has its own set of templates for recognition), and the activation can be viewed as the matching score. Therefore, we could know how much all patterns fit (match) each class by finding the most important patterns for each class and compare their activation. As analyzed in [29]] and [Gi], patterns (channels) with higher weights in the classification layer are more important, and the most important ones dominate the model decisions. Therefore, given an input, we select its most important patterns by the top classification weights of its ground-truth class, and record the average activation on these patterns. The Mean value of such Top Activation across all samples is denoted as MTA in Fig. E (mid). As can be seen, as the margin increases, MTA increases consistently, which further verifies patterns' increase in fitting each base class.
Better pattern fitness, worse pattern transferability. As each pattern could fit correspond- ing base class better, its discriminability increases accordingly, but could it be transferred across classes? To answer it, test the transferability of patterns. Since classes are related (e.g., cat and tiger), transferable patterns activated in one class could also be activated in other classes (e.g., felid patterns). Therefore, we first find important patterns for each base class by the classification weights, then record activation of these patterns on other classes, and measure the transferability of patterns by the mean value of such aamer-class-activatinn seen, the transferability consistently decreases when the margin increases. Combine this result with Fig. 0 (mid), we hold that patterns tend to be less transferable when they fit each base class better.
The results plotted in Fig. Q (right). As can be
Discussion.
The fitness also reflects the how much the given pattern is specific to a base class. Imagine the extreme situation where each base-class only needs one pattern for representation, the fitness would reach its upper bound to make such pattern thoroughly specific to the corresponding class. Therefore, we interpret that the higher the margin is, the more specific (overfitting) the patterns each base class, which makes patterns more discriminative but less transferable. Mcanwhile, the lower the margin is, the more the patterns could be shared between classes (underfitting), making patterns more transferable but less discriminative. The CO dilemma lies in that patterns can hardly be both class-specific and shared among classes by simply applying the classification margin.
2.3.2 Inherent Class Relations Lead to the Change in Pattern's Base-Class Fitness
Pattern's fitness negatively influences class relations. In Fig. D (right), we also plot the class relations w.r.t the margins. The class relations are measured by the average of cosine similarities between every two classes' prototypes. As can be seen, the relation drops as the margin grows, in consistent with the trend of the patterns' transferability. This is rationale because the if two prototypes share some patterns, the activation of the corresponding channels will be similar, making the cosine similarity larger. As the transferability of patterns is negatively related to pattern's 's base- class fitness, we hold that the class relations are also negatively related to the base-class fitness.
Inherent class relations influence pattern's fitness. The margin applied the classification di- rectly modifies the decision boundary between every two classes, and the decision boundary is related the relationship between every two classes. Therefore, we study how the class relation influences the pattern's 's fitness to base classes. Specifically, given 60 base classes, for the model trained without margins, we first calculate the cosine similarity between every two different classes, which gives 60 x (60- / 1,750 relations denoted as Ro. which represents the inherent rela- tions between all classes. Similarly, we calculate 1, 1,750 relations for the model trained with positive and negative margins respectively, denoted as R Rpos and Rneg- Then we calculate Dpoz Rpos Ro

---

Class A
0.3
Class-specific inprove Pasitive
Cosine
0.1
Cosine
patiems
margin
Negative
Class Relation
Shared patteens
margin
-0.1
Closs-spcciiee
Inyore
Positive
500
Sorted Index
1000
1500
500
Sarted Index
1000
1500
pattems
margin
Figure 3: The change of class relations sorted by the inherent class relations. Left: negative margin. Right: positive margin.
Class
Intuitive interpretation of the pat- tern learning and inherent class relation.
Figure 4:
and Dneg = Rneg Rg. Finally, we sort Rg from small to large, and use the sorted index to arrange D Dreg and Poos. As plotted in Fig. &. the blue dots denote Daeg (left) or Pos (right), and the blue curve are dots fitted by multinomial functions. We can see that the change in class relations is pos- itively related to the inherent class relations for the negative margins, while is negatively related to the inherent class relations for the positive margins, especially for index larger than 1,000.
Since class relations are negatively related to pattern's fitness to base classes, the more the class relation increases, the less the patterns could fit the corresponding base class. Therefore, the results can be understood that the more inherently similar two classes are, by applying negative margin, the less the patterns could fit the given base class, i.e., the more the patterns are shared by given classes, making these classes' representations more similar; by applying positive margin, the more the patterns could the given base classes, making these classes' representations more dissimilar.
Conclusion and Discussion. Therefore, as shown in Fig. H. we interpret the pattern learning pro- cess as follows. Given a negative margin, the decision boundary between two classes are confused, making more samples fall into the overlapping region between two classes. This makes it possi- ble to learn patterns shared by these two classes and hinder the pattern from fitting the given base class, and makes patterns more transferable but less discriminative. The more similar inherently two classes are (i.e., larger overlapping region in Fig. B) the more shared patterns can be learned (c.g.. more patterns could be shared between cats and tigers than cats and air- air-planes), therefore making these classes' representations more similar. On the contrary, given a positive margin, the decision boundary between two classes should be well separated, pushing the model to learn patterns fitting (i.e., specific to) each class, which are more discriminative but less transferable. The more similar inherently two classes are, the harder the learning and the larger the training loss will be, therefore making the patterns fit each class more and making these classes' representations more dissimilar.
Mitigating the Dilemma of Few-Shot Class-Incremental Learning
In this section, we first analyze the cause of the dilemma in margin-based classification based on the above interpretation, then we propose our method (named as Class-Level Overfitting Mitigation, CLOM) to mitigate the CO dilemma based on the analysis, shown in Fig. B.
3.1 Analysis on the Cause of Class-Level Overfitting
Based on the above analysis, we find the learning of wwaeee-aaannnnnnne patterns loosely constrains the given pattern to be shared by both classes. However, such constraint is easily satisfied by ineffective patterns simple as edges or corners, which could lead to the low discriminability of negative-margin-basse patterns. Similarly, the learning of positive-margin-basee constrains the given pattern be specific to the given class. However, such constraint could be satis- fied by easily finding patterns sharing no information with other classes, such as finding some com- plex texture only specific the given class, which could lead the low transferability of positive- margin-basee based patterns. Such casy-constraint problems push patterns to be only class-specific or shared among classes, making pattern effective in one scenario ineffective in other ones.
patterns loosely
To verify the above claims, we analyze how simple or complex the learned patterns are. Inspired by [IE]] quantitatively measure the simplicityccomplextyy of patterns by the similarity between the extracted feature and the simplest feature (e.g., corner or edge features).
We first use the baseline model to train on CIFAR 100, and use the first convolutional layer as the simplest feature extractor (denoted as ssimpte), since many works (e.g., (27]) has shown that the
first convolutional layer tends to capture corners or edges. Then, we train models with different
5

---

WP
F(x)
Relation Mapping
W
X
m
mi
mi!!
Loss
g()
Wi
Relation Mapping
S()
fx)
W
may!
Lnng
Figure 5: Method (CLOM) framework. We construct the positive-margin-- based (PM, F())) feature from the vegative-margin-be based (NM, f()) feature, and map class relations to form a set of class- specific margins, which can effectively mitigate the dilemma of class-level overfitting by providing extra constraint to the NM/PM pattern learning through the learning of PM/NM patterns respectively.
margins, and use the backbone network for feature extraction (denoted as ftorget). After that, we extract feimple and ftarget features from all images base classes. Finally, we compare the CKA similarity [[]]]] for measuring the similarity between faimple and the ftarget-
For sanity check, first report the similarity between different layers within the baseline model.
Table 1: Sanity check for the CKA measure.
Starget CKA
Convl-outpur
Stagel-outphd Stage2-outpur second-last-Conv
hackhone-outpur
0.8876
0.5664
0.2097
0.1306
We can see that the shallower the layer is, the higher the CKA similarity would be, which means the more similar they are the feamples i.e. the patterns are simpler. more transferable but less discriminative [26]] Then, we report the comparison of the CKA similarity between fsimple and Starget (backbone feature) of the baseline model trained with margins.
Table 2: CKA between the fsimpte and baseline backbone features trained with different margins.
Margin
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.2
0.3
0.4
CKA 0.2432
0.2245
0.2010
0.1661
0.1510
0.1306
0.1149
0.0837
0.0642
0.0576
We can see that by applying negative margin, the CKA similarity clearly increases (even larger than that of the second last convolutional layer when margin < -0.3), showing that the backbone network captures patterns more similar to simplest ones such as edges or corners, which verifies that the model tends to learn simple patterns that are easily shared between classes. When applying positive margin, the captured patterns grow be more complex and tend to overfit base classes, making the CKA much smaller than baseline model's 's backbone-outpur, which verifies that the model tends to learn complex patterns that are easily specific to given base class.
Therefore, the key to mitigating the CO dilemma is to extra constrain the pattern learning process.
3.2 Mitigating Class-Level Overfitting by Providing Extra Constraint
based (NM) patterns more transferable and class-shared, while Positive- Margin- based (PM) patterns are discriminative and class-specific. These characteristics are similar to the behavior of features from the shallow and deep layers of deep networks [26,, 32). Generally, shallow-layer features encode low-level patterns that are easily shared by most objects therefore more transferable, such as edge corner, while the deep-layer features encode high-level patterns that are semantically related to only few object classes therefore more discriminative. Such simi- larity inspires to view PM patterns high-level patterns while view NM patterns as (relatively) low-level patterns, and build PM patterns from the NM patterns, just like building high-level fea- tures from low-level features in deep networks. As the learning of NM patterns is influenced by PM patterns, this design could benefit the learning of NM patterns by providing extra constraint from the learning of PM patterns, and vice versa, which could therefore handle the casy-constraint problem.
Specifically, given an feature extractor f(-), we add one more layer to construct the PM feature as
F(x.) ((f())))
(3)
where F(-) is the PM feature extractor, g()) is typically composed of a fully-connected layer, batch- normalization layer [[[2] and an activation layer. During training, two classification loss is applied to both F(-) and f((:) with positive and negative margins respectively, represented as

---

L(e;, Mi) = Line (22, + Lpoo(as; Mc)
(4)
(v f(x)) mmnes))mm)
=llog- e7(un, (aa)-mmes + Ejsn, erma
X log eiug F(a.) mpoc)) Entt. eu P(z,)
(5)
where wf E RR denotes the classifier weights for F-)) corresponding to class j. mneg denotes the negative margin for f(-), poss is the positive margin for F()), and is typically set to 1.0.
Given this modification, the NM patterns f()) are utilized to construct the discriminative PM pat- terns F()). Since g(.) is relatively too simple capture complex patterns, f(-) will be pushed to be discriminative (validated in Fig. DD. instead of casually learning ineffective patterns shared be- tween similar classes. Similarly, F-() are built from the transferable f(-) by the simple g(-), the transferability could be maintained by pushing PM patterns to fit the corresponding base class by transferable information, which improves the transferability of F(-) (validated in Fig. M
During testing, given the improved f()) and F(.), the final feature would be their concatenation.
3.3 Further Mitigation of Class- Overfitting by Integrating Class Relations
Based on the architecture design, we propose to further mitigate the CO problem through boosting the discriminability and tranferability of NM and PM patterns by integrating the class relations.
For easy understanding, we first introduce the modification to the classification for f(-), and later introduce that for F(-). To modify the margin-based classification from a single margin to margins related to class relations, we first move the margin from the ground-truth logit to all other logits as ", (2x)
Mx,, yk) -log ( + Ejtu, (wy ((:))+((A))))))
where is the adjacency matrix between all classes, m(-) maps the adjacency value between two classes margin. Given this modification, if the margin is set to the original fixed value as Eq. D the decision boundary between class Y4 and other classes remains the same, therefore the new loss in Eq. & could be approximation of the original loss in Eq. D
For measuring class relations, we choose to utilize the adjacency matrix of all classes. As both the feature and the classifier's 's weights are L2 normalized, the adjacency can be measured by the cosine similarity between every two class prototypes as Aij = cos Pý). where P. the prototype for class i, which is typically set to w; of the classifier.
If two classes identical, the cosine similarity would reach its upper bound to 1.0. Meanwhile, the average of class relations reflects a global margin that is effective for most classes. Therefore, we design to interpolate the margin from a global effective value to pre-defined upper bound value as
m(A.) = mave + Mupper mave
(Aij Aave),
(7)
1.0 Aave
where Mupper is a hyper- paperr-prrametr controlling the margin for the upper-bound class relation (i.e., two identical classes), Mave is another hyper-parameter controlling the margin for the average class relations, set to the same value as that in Eq, E. and Aave the average class relations calculated as Agve (1
زj= Yol
ΣkSj Ajk.
For F()), we adopt the same modification as I(), which replaces s()) with F(.) in Eq. R. and replaces A with A" in Eq, B where Al, = cos( cos(w, wf'). Therefore, the final training objective
T f(z))
a T F(zz)
L(x;, v:) -log o wwn 1ad +x Σ e Je,,) matd., A log y P(ue) + Σ ( FoaAitmpAAp
371.
where m.(.) and m.(.) are two interpolate functions as Eq. with positive and negative hyper- parameters (muppers m m and (muzper. Mave) respectively.
P In experiments (Fig. H) we find it beneficial to have Mupper < Mave and m mp geet which means for classes with higher similarities, the relation mapping module enables the model to learn more shared (transferable) patterns by applying smaller negative margins, and learn more class- specific (discrininative) patterns by applying larger positive margins. Moreover, due to the connec- tion between NM and PM patterns, the improved transferability in NM patterns would be transmitted to PM patterns and vice versa, validated in Fig. E which further mitigates the CO dilemma.
7

---

Table 3: Evaluation datasets.
Dutaset
Total Classes
Base Classes
Novel Classes
Incremental Sessions
Novel-Class Shot
Input Size
CIFARIOO
100
60
40
8
5
32 x 32
CUB200
100
100
10
224 224
minilmageNet
100
60
40
8
84 84
Table 4: Comparison with state- of teffthe-ate the-art works on the CUB200 dataset.
Method
SO
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
Finetune
68.68
43.70
25.05
17.72
18.08
16.95
15.10
10.06
8.93
8.93
8.47
Rebalancing [1]]]]]
68.68
57.12
44.21
28.78
26.71
25.66
24.62
21.52
20.12
20.06
19.87
iCaRL. [X]
68.68
52.65
48.61
44.16
36.62
29.52
27.83
26.26
24.01
23.89
21.16
ERIT.
68.66 08.68
53.63
47.91
44,20
36.30
27.46
25.93
24.70
23.95
24.13
22.11
TOPIC [1)]
68 68
62.49
54,81
49,99
45.25
41.40
38.35
35,36
32.22
28.31
26.26
Decoupled-NegCosine [
74.96
70.57
66.62
61.32
60.09
56.06
55.03
52.78
51.50
50.08
48.47
CEC [28]
75.85
71.94
68,50
63.50
62.43
58.27
57.73
55.81
54.83
53.52
52.28
FSLL+SS
75.63
71.81
68.16
64.32
62.61
60.10
58.82
58.70
56,45
56.41
55.82
FACT [50]]
75.90
73.23
70.84
66.13
65.56
62.15
61.74
59.83
58.41
57.89
56.94
IDLVQ-C [8]
77.37
74.72
70.28
67.13
65,34
63.52
62.10
61.54
59.04
58.68
57.81
CLOM (Ours)
79.57
76.07
72.94
69.82
67.80
65.56
63.94
62.59
60.62
60.34
59.58
After the base-session training, the feature extractor F()) and f()) will be applied to the training data of each session to obtain the extended classifier weight W = [w]. w² ( } U
{wf, w ) by averaging extracted features, and the final performance of each session will be obtained based on the classification of all the encountered classes' test samples.
4 Experiments
In this section, we first introduce the experiment settings, then compare the proposed method with the tfate-of-the-et methods, and finally report the ablation study for the effectiveness of each design.
4.1
Datasets
Datasets include 10FAR100
[5]. Caltech-UCSD Birds-200-2011 (CUB200) [24]] and mini-
ImageNet [23] as listed Tab. following the split [2 For details, please refer the appendix.
4.2
Implementation Details
The implementation is based on CEC's [28), and our code will be released For CIFAR100, we set d' (256256, set m002= set 0.5 and we have =0. and my 0022 For CUB200, we scale the learning rate of the backbone network to 10% of the global learning rate since the pre- training of the backbone is adopted [G0, [28], and set df to 8192. Then we have m.v.=-022 and mapper==0... 0025 and m m0 =0.3 and mupper==06. For minilmageNet,
set d' to 4096, and have Mave=-022 and 7upper and mare and meppper=0.2. =0.2. Please refer to appendix for details.
4.3 Comparison with the State-of-the-Art
Comparisons with the state-of-the-an works are listed in Tab. A and Fig. i. where we can achieve state-of-the-ar performance on three datasets. Specifically, we can first see that our method, as prototype-bast based method (c.g.. CEC [28]), could significantly outperform mantune-bbsed methods (c.g., iCaRL [[S]). This is because the few-shot training data could not provide sufficient informa- tion for novel-class learning, therefore directly freezing parameters novel classes could reduce the iitastrophic-forgettmnn problem brought by the finetuning. Our method also outperforms other prototype- based methods, this is because the core of the prototype- based methods lies in the met- ric learning [20]] As empirically proved by current works [21. 25,, H. IG), applying the margin- based classification could effectively improve the embedding space learned by mesric-based meth- ods. Since our method handles the difficulty of applying margin-based classification (i.e., class-level overfitting) to the FSCIL task, our method could beat these pototype-based ones. For detailed num- bers of 100 and miniImageNet, please refer to the appendix.
4.4 Ablation Study of Modules
The ablation study of each module is reported in Tab. 5L where g(.) denotes adding another layer, Margin denotes the applicant of margin-based classification, and Relation refers to the relation map-
hhaps//githbb.ccm//zlllleenooo

---

CIFAR100
minrimageNet
70.0
NogCosine
70.0
NogCosine
Accuracy
60.0
Cosine
Ascuracy
60.0
Cosine
+1.11%
DespEM
+0.37%
DeepEMD
50.0
50.0
CTC
CLOM
CLOM
40.0
40.0
2
4
2
4
7
Session ID)
Session IT)
Figure 6: Comparison with state-of-the-ee works on CIFARIO0 and minilmageNet.
Table Ablation study of modules on the last incremental session of three datasets.
CUB200
CIFARI00
minilmageNet
Method
Overall
Novel
Base
Overall
Novel
Base
Overall
Novel
Base
Baseline
57.78
45.97
79.48
47.02
37.40
72.32
46.58
31.02
72.33
+ .9-))
57.21
47.13
79.39
48.37
39.55
72.70
46.79
30.97
72.60
+ Margin
58.73
49.93
79.47
49.21
40.22
73.72
47.30
32.07
72.93
Rellaiio
59.58
50.89
79.57
50.25
41.17
74.20
48.00
33.60
73.08
ping of the margin. We study from three aspects: Base-class, Novel-class, and Overall accuracy of the last incremental session. From Tab. EL we can see that
Simply applying another layer cannot consistently improve the performance.
The designed architecture could mitigate class-level overfitring. Compared with experiments in Fig. @ where no Overall performance improvements can be obtained by simply adding margins, the performance here clearly improved by adding margins on the designed architecture. Moreover, the overall improvements originate from not only the improved Base performance, but also the boosted Novel performance, demonstrating the mitigation of class-level overfitting (CO) problem.
Relation mapping could further improve performance by mitigating class-level overfitting.
4.5 Verification of Class-Level Overfitting Mitigation
Mitigating the class-level overfitting by the architecture design. We first compare our method with the baseline method which applies the margin directly to the backbone feature Fig. D and &. so as to validate the mitigation of CO. In Fig. M we apply the negative margin to the backbone (NM) feature while applying no margin to the PM feature, and see the accuracy of the NM feature against the baseline. As can be seen, while keeping the improvements on novel classes (mid), our method (orange) maintains the base-class accuracy (right), in contrast to sharp decrease of the baseline (blue) when the margin decreases, which makes our NM feature significantly outperforms the baseline terms of the overall accuracy (left). Similarly, in Fig. & our PM feature also maintains novel classes when the margin increases. These experiments validate the
higher performance mitigation of the CO dilemma: by implicitly guiding the learning of the PM and NM features by each other, the transferability or discriminability drop on them can be mitigated respectively.
Further mitigating class-level overfitting by relation mapping. To validate the relation map- ping module, in Fig. Bl fixed the average margin (i.e., -02 in Fig. 1st and -0.--0.3 in Fig. 2ndd) and experiment with different upper margins (i.e., Mupper in Fig. B 1st and mupper in of
Fig. 2nd). Similar to Fig. M when conducting experiments on one branch, the hyper-paramedt the other branch are fixed. We can see the relation mapping module can indeed help the learning, as the upper margins are significantly different from the average margin. We also plot the NM and PM performance on base classes with negative margins in Fig. 3rd), and those on novel classes with positive margins in Fig. 9K 4th). We can see the NM and PM performance are improved simultane- ously, which validates the improvements brought by the extra guidance of each features. Therefore, the relation mapping module can also help to mitigate the class-level overfitting problem by building PM features with more transferable patterns, and vice versa. Moreover, it is interesting to find the performance is improved when Mupper < Mave and me > m which means for classes with higher similarities, the relation mapping module enables the model to learn more shared patterns by applying negative margins with larger absolute value, and learn more class-specific patterns by applying larger positive margins. Also, this is consistent with the results in Fig. B

---

All Classes
Novel Classes
Base Classes
50.0
44,0
76.0
Accurity
48.0
Accumcy
42.0
Accurccy
74,0
46.0
72.0
40.0
44.0
70.0
42.0
68.0
40.0
36,0
66.0
.0.4 -0.3
-0.2
-0.1
0.0
.0.4
-0.3
-02 -0.1 0.0
-0.4
-0.3
-0.2
-0.1
0.0
Baseline
Ours-NM
Baseline
Ours-NM
Baseline
Ours-NM
Figure 7: NM feature's S accuracy compared with the baseline with negative margins (CIFARI00).
All Classes
Novel Classes
Base Classes
50.0
40.0
75.0
Accaracy
48.0
Aocancy:
38.0
Accuracy
46.0
34.0
44.0
32.0
73.0
0.1
0.3
0.1
0.3
0.1
0.3
Baseline
Ours-PM
Baseline
Ouns.
Baseline -
Ours.PM
Figure 8: PM feature's 's accuracy compared with the baseline with positive margins (CIFAR
All Classes
Caassss
Novel Classcs
Bose Classes
51.0
60.0
43.0
80.0
50.0
Actuacy
59.5
Accuraay
Accoracy
49.0
59.0
37.0
780
58.5
4.2 -033 -044 -0.5 -0.6 -0.7 Mayger
0.3 0.4 U.5 US U.T 0.8 Maypeer
Mapper
mppeer
NMM-Novel - PM-Novel
-ND--Hase
PM Hase
Figure 9: Upper margins in the relation mapping module (Mupper: CIFAR100, M Mupper CUB200).
4.6
Mitigating the Easy-Constrain Problem
Following experiments in Tab. II also utilize the CKA similarity between the simplest feature simmpte and the NM/PM feature to validate the mitigation of the easy-constraint problem. By apply- ing our method, for the NM (or PM) patterns, we set Storget to f() (or F()) in Fig.5 and set the positive/negative margin to 0.3/-0.4, and report the CKA below.
Table 6: CKA between feemple and NM (left) PM (right) features trained with different margins.
Margin -0.5
-0.4
-0.3
-0.2
-0.1
0.0
Margin
0.0
0.1
0.2
0.3
0.4
CKA
0.1867
0.1779
0.1724
0.1638
0.1552
0,1427
0.1476
0.1439
0.1430
0.1214
0.1100
In Tab. E (left), compared with Tab. D. CKA decreases clearly compared with the baseline method, which validates that NM patterns learned by our method are more complex and less similar to edges and corners than the baseline method, verifying the mitigation of the easily-satisfied constraint prob- lem by extra supervision from the learning of PM patterns. Also, in Tab. B (right), compared with Tab. 2 CKA is much larger than that of the baseline method, which validates that the PM patterns are more similar to the simplest patterns than the baseline method, which makes less overfitting the base classes, verifying the mitigation by extra supervision from the learning of NM patterns.
5 Conclusion
In this paper, we focus on the dilemma in the margin-based classification for FSCIL. We first inter- pret the dilemma as a class-level overfitting problem from the aspect of pattern learning, then find the cause of this problem lies in the easily-satisfied constraint of learning shared or class-specific pat- terns. Based on the analysis, we design method (CLOM) to mitigate the dilemma by constructing PM patterns from NM patterns, and mapping class relations into class-specific patterns. Extensive experiments on three public datasets validate the effectiveness and outstanding performance of the proposed method.
Acknowledgements
This work is supported by National Natural Science Foundation of China under grants U1836204, U1936108, 62206102, Science and Technology Support Program of Hubei Province under grant 2022BAA046, and 2022 CCF-DiDi Gaiya Young Scholar Research Fund.
10

---

References
[1] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability deep visual representations. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6541-6549, 2017.
[2] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision, pages 233-248, 2018.
[3] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: we really need multiplications deep learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1468-1477 2020.
[4] Kuilin Chen and Chi-Guhn Lee. Incremental few-shor learning via vector quantization in deep embedded space. In Proceedings of the International Conference on Learning Representations, 2020.
[5] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discrim- inability: Batch spectral penalization for adversarial domain adaptation. In Proceedings of the International Conference on Machine Learning, pages 1081-1090. PMLR, 2019.
[6] Quan Cui, Bingehen Zhao, Zhao-Min Chen, Borui Zhao, Renjie Song, Jiajun Liang, Boyan Zhou, and Osamu Yoshie. bbseriminabllliy-ttaasfeaaaaaa trade-off: An information-theoreti perspective. arXiv preprint arXiv:2203.03871, 2022.
[7] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situ- ations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 3941 3941-3950, 2020.
[8] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular mar- gin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690 4699, 2019.
[9] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 141. Springer, 2020.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[11] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 831-839, 2019.
[12] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network train- ing by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning, pages
PMLR, 2015.
[13] Simon Kornblith, Ting Chen. Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? Proceedings of the Advances in Neural Information Processing Systems, 34, 2021.
[14] Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. International Conference on Muchine Learning, pages 3519-3529, PMLR, 2019.
[15] Alex Krizhevsky, Geoffrey Hinton, et Learning multiple layers of features from tiny images. 2009.
11

---

[16] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Nega- tive margin matters: Understanding margin few-shot classification. In Proceedings of the IEEF/CVF European Conference Computer Vision, pages 438-455. Springer, 2020.
[17] Pratik Mazumder, Pravendra Singh, and Piyush Rai. Few-shot lifelong learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2337-2345, 2021.
[18] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 2001 20100, 2017.
[19] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. In Proceedings of the International Conference on Learning Representations, 2015.
[20] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the International Conference on Neural Information Processing Systems, pages 4080-4090, 2017.
[21] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1398-6407, 2020.
[22] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Patlern Recognition, pages 12183-12192, 2020.
[23] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the International Conference on Neural Information Processing Systems, pages 3637-3645, 2016.
[24] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- uesd birds-200-2011 dataset. 2011.
[25] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5265-5274, 2018.
[26] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Proceedings of the International Conference on Neural Information Processing Systems, pages 3320-3328, 2014.
[27] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506 (tt.::1506.0679, 2015.
[28) Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incre- mental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 12455-12464, 2021.
[29] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2921-2929, 2016.
[30] Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. For- ward compatible few-shot class-incremental learning. arXiv preprint arXiv:2203.066 06953, 2022.
[31] Yixiong Zou, Shanghang Zhang, Ke Chen, Yonghong Tian, Yaowei Wang, and José MF Moura. Compositional few-shot recognition with primitive discovery and enhancing. In Proceedings of the ACM International Conference on Multimedia, pages 156-164, 2020.
[32] Yixiong Zou, Shanghang Zhang, Jianpeng Yu, Yonghong Tian, and José MF Moura. Revis- iting mid-level patterns for cross-domain few-shot recognition. In Proceedings of the ACM International Conference on Multimedia, pages 741-749, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
Experiments are conducted based on the code released by CEC (CVPR'2I), which
specified the random seed for re-implementation and comparison. Therefore, fol- low the same random seed to conduct experiments and compare with other methods without the error bar.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'e using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "xcXJKxRrtce", "reviewer": "Reviewer_9Up9", "summary": "This paper deals with the few shot incremental learning task. The authors identify the inferior trade-off between base- and novel-class as a class-level overfitting problem during the margin-based training, and propose a new margin-based learning method with extra constraint to overcome the problem. The experiments show the advantage of the proposed method on certain commonly used benchmarks. ", "questions": "It is better if the paper includes the extra experiments on face recognition datasets.\n\n", "limitations": "No obvious negative societal impact observed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nPros: Two branches for pos and neg margin setting, respectively, are new and interesting to me.\nCons: The class relation in the inter-class logit is not new, but is studied by many methods in the face recognition community.\n\nQuality\nPros: The overall quality of this paper is good. The method is well formulated, based on the solid observation and analysis. The experiments are clear to show the effectiveness.\nCons: It lacks the comparison with the similar methods in the face recognition community.\n\nClarity\nPros: The method and the experiments are well presented.\nCons: Missing definition of horizontal and vertical axes in Fig. 2.\n\nSignificance\nPros: The proposed method will be useful for practical application of incremental learning task.\nCons: -\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "x__z3CGl4xd", "reviewer": "Reviewer_cyf6", "summary": "This paper tackles Few-shot Class-incremental Learning (FSCIL) problem which is designed to incrementally recognize novel classes with only few training samples after the (pre-)training on base classes. A dilemma exists that we can hardly achieve both good base-class performance and novel-class generalization simultaneously. \n\nThis paper analyzes the effects of positive margin and negative margin loss to the novel and based classes and designs a new network in which negative margin loss is attached to the lower layer and positive margin loss is attached to the top layer. Further improvement method based on the adjacency matrix is also proposed. Extensive experiments on three datasets show the effectiveness of the proposed method. ", "questions": "- For the ablation study, how about the case when the positive margin loss is attached to the lower layer, and negative margin loss is attached to the higher layer? Is the proposed design of the loss function positions really better?  \n\n- For the experiment of Sec.4.5, which dataset is used? \n", "limitations": "The authors did not address the potentially negative societal impact of this work. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Enhancing the novel class generality while keeping the discriminative features for the base model with positive and negative margin losses, which are attached to lower and upper layers is an elegant solution. \n- Further improvement method based on the adjacency matrix of all classes is proposed. \n- Detailed analysis of the dilemma of FSCIL is shown. \n- Experimental results show that the proposed method outperforms SOTA methods on FSCIL. \n\nWeaknesses\n-  The proposed method is based on existing findings in the paper on the negative margin matters[15]. (Thought the method and analysis of this paper are more sophisticated.)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WD35hHxuo3J", "reviewer": "Reviewer_L5yy", "summary": "This paper studies margin-based few-shot class-incremental learning (FSCIL), which is motivated by an observation that positive (negative) margins adopted in base training can negatively influence the transferability (discriminability) of the representations. The authors explained this dilemma as a class-level overfitting problem from the perspective of pattern learning and claimed the cause of such class-level overfitting is due to the fact that the constraint of learning shared or class-specific patterns can be easily satisfied. To address this problem, the authors proposed simultaneously adopt positive and negative margin and learn positive-margin-features from negative-margin features. Experiments on a set of FSCIL datasets show the proposal can achieve STOA performance. ", "questions": "1. How to justify the claim about the dilemma is from the easily-satisfied constraint of learning shared (e.g., corner or edge features) or class-specific patterns (complex class-specific features that cannot generalise).\n2. How $m_{upper}$ and $m_{ave}$ are set?\n3. Why baseline method performs so well?\n4. the authors find large margin can lead to sparse patterns. Is this observation to some extent relevant to supervision collapse observed in [1], which described lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper Strength\n\n+ The connection between margins and pattern learning is interesting. It is an interesting observation that positive margin pushes the sparsity and fitness of patterns but harms the transferability of the patterns. This observation provides insight in understanding the generalisation of deep neural networks. \n+ Thorough experiments are conducted to evaluate the effectiveness of the proposed method. The results reveal that the method can simultaneously improve the performance of base and novel classes and thus is empirically proven to be able to overcome the dilemma of the margin-based classification. \n\nWeakness\n- There lacks evidence to support some important claims made in this paper. For example, the authors claimed the cause of the class-level overfitting problem lies in the easily-satisfied constraint of learning shared (e.g., corner or edge features) or class-specific patterns (complex class-specific features that cannot generalise). But how to verify this? Additional visualization seems necessary. \n- It makes sense to consider the class relations into class-specific margin. But the necessity to setting two key hyper-parameters $m_{upper}$ and $m_{ave}$ manually makes the method less elegantly. How these two parameters are determined? Is this method sensitive to such parameters?\n- From Table 3 it can be the baseline method performs very well even better than most of the comparing methods. More detailed explanation to the baseline and its performance should be added.   \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept