Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation
Authors: Donghyeon Baek
Youngmin Oh
Sanghoon Lee
Junghyup Lee
Bumsub Ham
Venue: NeurIPS 2022 Accept (2022)
Keywords: class-incremental learning
continual learning
incremental learning
semantic segmentation

Abstract: Class-incremental semantic segmentation (CISS) labels each pixel of an image with a corresponding object/stuff class continually. To this end, it is crucial to learn novel classes incrementally without forgetting previously learned knowledge. Current CISS methods typically use a knowledge distillation (KD) technique for preserving classifier logits, or freeze a feature extractor, to avoid the forgetting problem. The strong constraints, however, prevent learning discriminative features for novel classes. We introduce a CISS framework that alleviates the forgetting problem and facilitates learning novel classes effectively. We have found that a logit can be decomposed into two terms. They quantify how likely an input belongs to a particular class or not, providing a clue for a reasoning process of a model. The KD technique, in this context, preserves the sum of two terms ($\textit{i.e.}$, a class logit), suggesting that each could be changed and thus the KD does not imitate the reasoning process. To impose constraints on each term explicitly, we propose a new decomposed knowledge distillation (DKD) technique, improving the rigidity of a model and addressing the forgetting problem more effectively. We also introduce a novel initialization method to train new classifiers for novel classes. In CISS, the number of negative training samples for novel classes is not sufficient to discriminate old classes. To mitigate this, we propose to transfer knowledge of negatives to the classifiers successively using an auxiliary classifier, boosting the performance significantly. Experimental results on standard CISS benchmarks demonstrate the effectiveness of our framework.

Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation
Donghyeon Baek'
Youngmin Oh'
Sanghoo Lee!
Junghyup Lee"
Bumsub Ham
Yonsei University
Korea Institute of Science and Technology (KIST)
https://cvlab, yonsei wonsei.ac.ken ac htt.kk//ooooet///D//////
Abstract
Class-incremental semantic segmentation (CISS) labels each pixel of image with corresponding object/stuff class continually. To this end, it is crucial to learn novel classes incrementally without forgetting previously learned knowledge. Current CISS methods typically use knowledge distillation (KD) technique for preserving classifier logits, or freeze a feature extractor, avoid the forgetting problem. The strong constraints, however, prevent learning discriminative features for novel classes. We introduce CISS framework that alleviates the forgetting problem and facilitates learning novel classes effectively. We have found that a logit can be decomposed into two terms. They quantify how likely an input belongs to particular class not, providing a clue for a reasoning process of a model. The KD technique, in this context, preserves the sum of two terms (i.e., class logit), suggesting that each could be changed and thus the KD does not imitate the reasoning process. To impose constraints on each term explicitly, we propose new decomposed knowledge distillation (DKD) technique, improving the rigidity of a model and addressing the forgetting problem more effectively. We also introduce novel initialization method to train classifiers for novel classes. In CISS, the number of negative training samples for novel classes is not sufficient discriminate old classes. To mitigate this, we propose transfer knowledge of negatives to the classifiers successively using an auxiliary classifier, boosting the performance significantly. Experimental results on standard CISS benchmarks demonstrate the effectiveness of our framework.
Introduction
A general way of learning knowledge for neural networks is to tune network weights with examples for all object/scene classes hand. After finishing the learning process, the weights are normally fixed for inference, suggesting that the current learning paradigm is not flexible enough to handle novel classes unseen at training time. Fine-tuning the weights with additional examples for novel classes addresses the problem in part, but this causes catastrophic forgetting [19]. Namely, neural networks rather forget the previously learned knowledge in order to learn new information. Class-incremental learning (CIL) targets to learn novel object/scene classes continually using training samples for those classes only, while minimizing the forgetting problem. A key to CIL to design a learning method that balances between rigidity and plasticity of model [20]. On the one hand, network weights should not altered abruptly in learning new information from novel classes, in order to preserve the discriminative ability for old ones (i.e., rigidity), avoiding catastrophic forgetting. On the other
Cooresponning author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

hand, a strong rigidity rather distracts learning knowledge from novel classes. The network weights should thus be tuned accordingly (i.e. plasticity).
Class-incremental semantic segmentation (CISS) adopts a CII. paradigm for the task of semantic segmentation. CISS methods [5, 9, 21, 29] typically exploit softmax cross-entropy (CE) term along with knowledge distillation (KD) [14]. Although the CE term helps to learn novel classes, applying the softmax function all classes, including both old and novel ones, lowers class probabilities of old ones. This in turn prevents the model from preserving knowledge learned from old classes, resulting in catastrophic forgetting [1, 23]. The KD technique prevents changing network weights drastically, alleviating the forgetting problem. Recently, SSUL [6] proposes to use multiple binary cross- entropy (BCE) terms for individual novel classes separately. This approach handles the forgetting problem caused by the softmax function, but it is limited in the following: (1) A feature extractor is frozen in order to enforce the rigidity of the model. This strong constraint for preserving knowledge for old classes makes it hard to learn discriminative features for novel classes.
(2) An initialization technique for classifiers requires an off- offtthe-shelf saliency detector [15], which is computationally demanding,
In this paper, we present simple yet effective CISS framework that overcomes the aforementioned problems. To achieve better plasticity and rigidity for a CISS model, we propose to train a feature extractor, and introduce a decomposed knowledge distillation (DKD) technique. KD encourages a model to predict logits similar to the ones obtained from an old model. We have found that logits can be represented as the sum of positive and negative reasoning scores that quantify how likely and unlikely an input belongs to particular class, respectively. In this context, KD focuses on preserving the relative difference between positive and negative reasoning scores only, without considering the change of each score. The DKD technique imposes explicit constraints on each reasoning score, instead of class logits themselves, which is beneficial to improving the rigidity of a CISS model together with KD effectively. We also propose an initialization technique to train classifiers for novel classes effectively. Note that training samples for novel classes are available only for each incremental step, suggesting that classifiers for the novel classes are trained with a small number of negative samples. To address this, we propose to train an auxiliary classifier in a current step, and use it to initialize classifiers for novel classes in a next step. To this end, consider training samples of a current step as potential negatives for novel classes in a next step. We then train the auxiliary classifier, such that all pixels in current training samples as negative ones, transferring prior knowledge of negative samples to the next step for the classifiers of novel classes. Our initialization technique also does not require any pre-trained models, e.g., for saliency detection [6], in order to differentiate possibly negative samples. We demonstrate the effectiveness of our framework with extensive ablation studies on standard CISS benchmarks [11, 30J. We summarize main contributions of our work as follows:
We introduce a simple yet effective CISS framework that exploits a novel DKD and multiple BCE terms, achieving a good trade-off between rigidity and plasticity.
We present a novel initialization technique that encodes prior knowledge for negatives to train classifiers for novel classes effectively.
We achieve new state of the art on standard benchmarks for CISS [11, 30], and demonstrate the effectiveness of our approach through extensive experiments and ablation studies.
2 Related work
2.1 Class incremental image classification
Many CIL methods [2, 4, 10, 16, 27] have been proposed for image classification, attempting to preserve the discriminative ability for old classes. Since training samples of novel classes are available only in incremental steps, they typically adopt KD [14] technique to retain classifier logits for old classes. For example, the works of [8, 10] additionally apply the technique to intermediate feature maps from feature extractor. LwM [8] proposes an attention- based distillation loss to preserve visual information of old classes. PODNet [10] introduces a spatial- based distillation technique that encourages pooled feature maps between current and previous steps to be similar to each other. CIL methods [2, 4, 16, 27] exploiting an external memory have recently been introduced. They store subset of training samples for old classes for re-tinnni which is effective to alleviate catastrophic forgetting. Due to the data imbalance between old and novel classes, they generally use training

---

Step
Step 2++1
bicyde
F-1
Ld
4 (i)
F+
Initialize
F++1
car
G--1
14-1(())
C1:--1
bieyal
Bieyalo
Initialize
Bicyon
GO $B
person Ge
person
Ci
F
Lmboe
sheep
person
Initialize
G++1
f
F
Zc(3)
f
Lae
H
He+1
ignoneg
gradieat
He
zi(s)
(a) Training.
(b) Initialization.
Figure 1: Overview of our framework. (a) Our framework consists of feature extractor Fe. classifiers Ir, and an auxiliary classifier H4 at each step t. Given an input image, we extract feature map f, and obtain class logits 2, from corresponding classifiers G.. We train our model with four terms: mBCE (Lentce), KD (Lxa). DKD (Laa),, and AC (Lac) losses. Note that the feature extractor does not receive any gradients from the AC term for the auxiliary classifier. (b) In the next step + 1, we initialize classifiers for novel classes with the previous auxiliary classifier H4. The feature extractor F++1, a new auxiliary classifier Hz+1. and other classifiers are simply initialized with the counterparts from the step I. Best viewed in color.
tricks, such as re-training with balanced subset of training samples [4, 27] re-balancing classifier weights [2, 16], which is however computationally expensive and requires additional memory. Note that all the aforementioned methods [2, 4, 8, 10, 16, 27] exploit softmax CE term learn novel classes. Similar to ours, the seminal work of [255] uses BCE losses along with KD for CIL. Differently, we use novel DKD loss together with novel initialization technique specialized for CISS.
2.2 Class incremental semantic segmentation
Similar to class-incremental classification, CISS methods [5, 9] generally adopt the KD technique to alleviate catastrophic forgetting. For example, MiB [5] applies the technique to pixel-wise classification scores. PLOP [9] employs KD to intermediate feature maps, and introduces a local POD loss, specially designed for CISS, by extending the vanilla version in [10]. In addition, SDR [22] recently proposes to leverage contrastive learning, complementary to existing KD techniques, to separate feature clusters, making it easier to learn novel classes. These methods also exploit a softmax CE term to learn novel classes. Different from image classification. supervisory signals for CISS are given in pixel-level labels only for regions corresponding to novel classes. The regions for background and old classes thus unlabeled, suggesting that have limited information train classifiers for CISS using a softmax CE loss. To tackle this issue, several works [9, 22] propose to generate pseudo labels from old model provide auxiliary supervisory signals for the unlabeled regions. Following the work of [25], exploiting multiple BCE losses for CISS is recently introduced to train classifiers for novel classes individually [6]. This approach enables training model without auxiliary supervisory signals for unlabeled regions. however freezes a feature extractor to alleviate catastrophic forgetting, constraining the plasticity of classifiers for novel classes excessively, On the contrary, our method trains both feature extractor and classifiers, together with novel DKD and initialization techniques, achieving better trade-off in terms of plasticity and rigidity.
Approach
We train CISS model continually with training samples D, each training step i, where & e {I. and T total number of steps. The training dataset D, contains pairs of an image and a corresponding ground-truth mask We denote by C, and Ci.r-1 sets of novel and old object/stufff classes at the step f, respectively. Note that the sets are disjoint, i.e., C1a-1 nC, = Note also that regions for background and old classes are labeled unknown in current step t. Namely, a ground-tran -ooun label at position i, y()), is either one of the classes in C2 or the unknown class Cus L.e. y() C, U {cu). J. at the step
3

---

3.1 Overview
We show in Fig. an overview of our framework. For each step f, we train CISS model together with auxiliary classifier H4. Our framework mainly consists of feature extractor Fi and set of classifiers G predicting pixel-level semantic labels for previous and novel classes at the step & (Fig. ((a)). In next step 1, we initialize feature extractor Fz+11 and classifiers for old classes Cia with the previous ones from the step / to preserve knowledge for old classes. Classifiers for novel classes the step + and new auxiliary classifier H4+1 initialized with the previous one He (Fig. 1(b)).
We exploit four loss terms for training (Fig. I(a)): Multiple binary cross-entropy (mBCE), knowledge distillation (KD), decomposed knowledge distillation (DKD), and auxiliary classifier (AC) losses. The first three terms used to train CISS model every step, while the last is for the auxiliary classifier 24. Specifically, the mBCE term encourages the model to learn knowledge from novel classes. The KD and DKD terms, the other hand, help to preserve the discriminative ability for old classes. The AC term enables transferring knowledge of negatives to the next step for classifiers of novel classes.
3.2 Training
We train our framework using an objective as follows:
L = Lenine aLLad + B LLakd + Lax;
(1)
where Lmboce Lxa, Lakd, and Lue are mBCE, KD, DKD, and AC terms, respectively, balanced by the hyperparameters of a and B. the following, we describe each term in detail.
mBCE loss. Given an input image, we first obtain a feature map f, at a step We then compute class logits, 2, E #R# with classifiers Gr. where H and W are the height and width of the input image, respectively, and |- is the cardinality of given set. Concretely, the class logit is obtained by computing the dot product between features and weights for corresponding classifiers, followed by adding a bias:
(i, = (i) (e(C) + by(c),
where we denote by We(c) and by(c) weights and bias of a classifier for class c, respectively, and (i) is a feature at position i. To learn novel classes of C,, current CISS methods typically exploit a CE loss computing softmax probabilities w.r.t all classes, C, and C1:--1, at a step t. This could lower the softmax probabilities for the old classes C1:111, causing catastrophic forgetting [1, 23]. We instead exploit mBCE loss, and apply it to train classifiers for the novel classes C2 only as follows:
HW
Lmbee
1
Σ
Σ
x1y((1) pr 1 * cllog(1 c)(i,c)),
(3)
N1W
1=1 aEC,
where
ci 1 e (((0.))
(4)
and [:] an indicator function that outputs if the argument is true, and otherwise. Following [3, 28], we use weighting strategy with balance parameter of to handle the imbalance between two terms (3).
KD loss.
We adopt KD term to prevent our model from changing abruptly, mitigating the catastrophic forgetting problem, defined as follows:
HW Σ
Lu
Σ CEC
pI-1(i, -1----11111110001111
log(1 c(().c)). (5)
HW
2
where P:-1 is similarly computed as in (4) with Z,-1, i.e., class logits predicted by previous model the step / This term encourages our model to provide class logits similar to the ones obtained from the previous model, namely, (i, c) 24-1ii, (i, c), for old classes, e E C1st-1, at the step f.

---

dkd
(1,e)
2
(2,c)
1 (f)c)
3y (f,
(H,c)
(5)
R-1(6) (c-1(c)
Lxd
(c)
(i)
Wa-1(e)
Likd
(c)
(8)
positive element
negative element
Figure 2: Comparison of KD and DKD. KD uses logit 2,, while DKD exploits positive and negative reasoning scores, z and z, The DKD term encourages our model to output the reasoning scores of 2 and z similar to the ones of 2 and Zi-1. respectively, obtained from a previous model.
DKD loss. Based that the dot product is the sum of element-wise multiplication between vectors, we decompose the class logit in (2) as follows:
(i. = z (i.c) (1, (1.c).
(6)
where z; (i, e) is the sum of positive elements chosen from the result of element-wist multiplication
between (i) and we(c), and zi (i,c) is similarly defined using negative elements (Fig. 2 right). We omit the bias of the classifier for ease of notation. We call z (i, and z, (i,c) as positive and negative reasoning scores, respectively, which quantify how likely and unlikely an input belongs to the class c. In this context, the KD technique retains the relative difference between positive and negative reasoning scores only, suggesting that each reasoning score itself is not preserved (Fig. 2 left). For example, if one of the reasoning scores increases, the other one would decrease accordingly in order to maintain the sum of reasoning scores, i.e., the class logit. To address this problem, we propose a decomposed knowledge distillation (DKD) loss as follows:
Lak = Lidd + Likd
where
HW Σ Σ
1
tid
(i,c (1- (i,) p. (i.c)),
(8)
HW
cEC1a-1
and
(i,c) 1+ e (-- (i,e)
(9)
fis is similarly defined using the negative reasoning score 2, Note that Lid and Laka are anal- ogous to the KD term in (5), but they compute losses with the reasoning scores, z, and z sepa- rately, instead of exploiting a class logit z, directly. That is, the DKD term encourages our model to provide positive and negative reasoning scores similar to the ones obtained from a previous model, i.e., z," c) = ( c) and z, (i,c) x z explicit constraint on each reasoning score improves the rigidity of our model, enabling it to preserve the discriminative ability for old classes effectively.
(i. c), for the old classes e E CT.--1- This
The dataset D, at a step provides training images mainly depicting one of novel classes C, This suggests that the number of negative samples for the novel classes Ce, e-g., images containing objects for old classes C1a-1 as well, would not be sufficient in the dataset D,. We conjecture that training samples in the step i can serve as good negative examples for novel classes C.+1 in the next step & + 1, as C1.t n C++1 = 0. Based on this, we propose to exploit an auxiliary classifier H, to encode knowledge of the negatives for the novel classes C++1 in advance of the step + 1. To this end, we train the classifier H, with the AC term as follows:
AC loss.
HW
(p'
Σ (og(1-)(()).
(10)
HW
where =
= 1/(1 + e -0), and 2i E HW is a logit from the auxiliary classifier H.. That is, the classifier He is trained to classify all pixels in the images of D, as negatives for the next step t + 1. Note that training images at a step ł might contain objects/stuff for novel classes in the future. However, we assume that noisy training signals in the step can be compensated in the future by using sufficient training images for those objects/stuff classes. Note also that the AC term is used to train the auxiliary classifier H4 only, and thus a feature extractor does not receive any gradients from this term.
5

---

3.3 Initialization
At the beginning of step t + 1, we initialize our model, including a feature extractor and classifiers, using the model the step to transfer previously learned knowledge (Fig. 1(b)). Note that classifiers for novel classes are newly added to predict logits for corresponding classes at the step t + 1. Note also that the number of negative samples for novel classes might not be sufficient in a dataset D(+1- In this context, exploiting a random initialization technique for the new classifiers is not effective, degrading the discriminative ability of classifiers for the negatives. To address this, we propose to initialize new classifiers using an auxiliary classifier in a previous step, He. The auxiliary classifier H, contains prior knowledge of negatives for novel classes. Concretely, we initialize classifiers 9 in the step t + 1, as follows:
[ (W1, b (c) b.(c)
if E C++1 otherwise,
(werr(e)
(11)
where we denote by w/ and by weights and a bias of the auxiliary classifier, respectively. That is, the parameters of new classifiers for novel classes C2+1 are initialized with the ones from the previous auxiliary classifier He. and those for other classifiers are initialized with the counterparts from the previous step.
3.4 Inference
We predict pixel-level semantic labels at step t, using class probabilities pt, as follows:
Cog (argmax, pr(i,e) otherwise,
max Po(i, <
(i)
(12)
where 7 a threshold, empirically set to 0.5, and we denote by Ony background class. We assign class label to each pixel, only when one of class probabilities for the pixel is at least larger than the threshold r. Otherwise, the pixel is assigned as a background class.
4 Experiments
4.1 Implementation details
Datasets. We use PASCAL VOC [11] and ADE20K [30] datasets for evaluation. PASCAL VOC [11] consists of 10, 582 training and 1, 449 validation images for 20 object and background classes. ADE20K [30] provides 20, 210 and 2, 000 images for training and validation, respectively, with 150 object and stuff classes. Following the protocol in [5],
official validation splits for evaluation. We also exclude 20% of training sets, and use them to tune hyper-parameters.
Experimental protocols. We follow the experimental protocols in [5]. First, we evaluate our model for various incremental scenarios. Specifically, we split all object/stuff classes into base and novel ones. We train the model for base classes in an initial step, and update sequentially for novel classes in each of the following training steps. We denote by (No-N,, the incremental scenario, where N₂, and N, are the numbers of base and novel classes, respectively. For example, given 20 object classes in PASCAL VOC [11], for an incremental scenario of (15-1), we learn 15 base classes initially, and add a single novel class sequentially, which requires training steps in total. Second, we train our model under two configurations: Disjoint and overlapped settings. The disjoint setting uses a unique set of training samples for each training step. Training images in the set depict object/stufff classes belonging to one of categories to learn in current step. The disjoint setting however excludes the images they have any pixels regarding novel classes to be presented in the future. The overlapped setting leverages all training images that contain at least single instance of classes to learn in a current step. Note that the overlapped setting is more realistic in practice, since the disjoint setting assumes that novel classes to learn in the future known in advance at each training step. We perform experiments on PASCAL VOC on both disjoint and overlapped settings with incremental scenarios of (19-1), (15-5), and (15-1). For ADE20K [30], we evaluate our model under the overlapped setting only, with the scenarios of (100-50), (100-10), and (50-50), following [6, 9].

---

Table 1: Quantitative results on the validation split of PASCAL VOC [11] for disjoint and overlapped settings. All numbers are obtained by averaging results over five runs with standard deviations in parenthesis.
19-1 (2 steps)
15-5 (2 steps)
15-1 (6 steps)
mloU,
mloU,
hloU
mloU,t
mloUh
mloU,
hloU
mloUai
mloUp
mloU,
hloU
mloU,n
MiB [S]
69.60
25.60
37.43
67.40
71.80
43.30
54,02
64.70
46.20
12,90
20.17
37.90
SDR [22]
69.90
37.30
48.64
68.40
73.50
47.30
$7.56
67.20
59.20
12.90
21.18
48.10
PL.P [9]
75.37
38.89
51.31
73.64
71.00
42.82
53,42
64.29
57.86
13.67
22.12
46.48
SSUL [6]
77.38
22.43
34.78
74.76
76.44
45.60
57.12
73.97
32.15
44.82
64.01
Disjoint
RCIL [29]
75.00
42,80
54,50
67.30
66. 10
18.20
28.54
$4.70
Ours
77.43
43.56
55.72
75.81
77.56
54.13
63.76
71.98
76.34
39.36
51,92
67.54
(+b0.07)
(#22433
(+2.00
(:0015)
(#:0.26)
(#:087)
(#0.65)
(+00.5)
(=90555
(#22077
(d:1.89)
(#:0.82)
RECALL [18]
65.00
47.10
54.62
65.40
69.20
52.90
59.96
66.30
67.60
49.20
56.95
64.30
SSUL-M [6]
77.58
43.89
56.06
75.9%
76.47
48.55
59.39
69.83
76.46
43.37
55.35
68.58
77.62
56.86
65.63
76.64
77.71
55.43
64.70
72.40
77.25
48.20
59.36
70.33
Ours-M
(+0011))
(=1171)
(+1.1
1:0.177)
(st0.21)
(#:0.99)
(+0.52)
(=:0.29)
(=00220)
1.15)
(+b0866)
(=:0.25)
MiB
33.62
67.80
75.50
59.72
69.00
13.50
19.50
SDR [22]
69.10
32.60
44.30
67.40
75.40
52.60
61.97
69.90
44.70
21.80
29.31
39.20
PLOP 19]
75.35
37.35
49.94
73.54
75.73
51.71
61.46
70.09
65.12
21.11
31.88
54.64
Overlapped
SSUL [6]
77.73
29.68
42.96
75.44
77.82
50,10
60.96
71:22
77.31
36.59
49.67
67.61
RCI1 [29]
78.80
52.00
62.65
72. 40
70.60
23.70
35, .99
59, 40
Ours
77.76
41.45
54,03
76.03
78.83
58.23
66.98
73.93
78.09
42.72
55.21
69.67
(+-0.18)
(#2.91)
(+-2.49)
1++0.24)
(++0.233)
(+:0.45)
(+0.31)
(++0.21)
(=0.32)
(+ (+-1.58)
(++1.33)
(++0.49)
RECALL [18]
68.10
55.30
61.04
68.60
67.70
54.30
60,26
65.
67,80
50,90
58.15
64.80
SSUL-M [6]
77.83
49.76
60,71
76.49
78.40
55.80
65.20
73.02
78.36
49.01
60.30
71.37
77.98
57.66
66,27
77.01
79.13
68.63
74.72
78.84
52.36
62.91
72.53
Ours-M
(+00111)
(=:2229
(1:1.51)
(::0.14)
(sb0.25)
(#:0.42)
(b00255
(sb0.17)
(=00.1)
(d:7.67)
(d:1.22)
(st00.43
Joint
77.57
77.80
77.68
77.58
79.48
71.52
75.29
77.58
79.48
71.52
75.29
77.58
Table 2:
Quantitative results on the validation split of ADE20K [30] for an overlapped setting. All numbers are obtained by averaging results over five runs with standard deviations in parenthesis.
100-50 (2 steps)
100-10 (6 steps)
50.50 steps)
mioU;
mloU,
hloU
mloUiai
mloU,
mloU,
hloU
mloU,a
mloU,
mloli,
hloU
mloU,ll
MiB 15|
40.52
17.17
24.12
32.79
38.21
11.12
17.23
29.24
45,57
21.01
28.76
29.31
PLOP [9]
41.87
14.89
21.97
32.94
40.48
13.61
20.37
31.59
48.83
20.99
29.36
30.40
SSUL [6]
41.28
18.02
25.09
33.58
40.20
18:75
25.57
33.10
48.38
20.15
28.45
29.56
RCIL [29]
42.30
18.80
26.03
34.50
39.30
17.60
24.31
32.10
48.30
25,00
32.95
32,95
Ours
42.41
22.89
29.74
35.95
41.56
19.51
26.55
34,26
48.84
26.28
34.17
33,90
(20.42)
(1.0.37)
(1.0.40)
(1.0.38
(10.36)
(10.35)
(1.0.32)
(1.0.24)
(10.34)
(1.060)
(1:0.52)
(1:043
SSUL-M [6]
42.79
17.54
24.88
34.37
42,86
17.66
25.01
34.46
49.12
20.10
28.53
29.77
Ours-M
(=0.3)
42.43
(220.3)
22.95
29.79
(2:0.30)
(220.30
35.98
(10033)
41.74
(3:0.27)
20.11
(2:0.26
27.14
(2:0.25)
34.58
(1:0.78)
48.84
(2:0559)
26.31
(2:0.51)
34.19
(2:041)
33.92
Joint
43.16
30.03
35.42
38.81
43.16
30.03
35.42
38.81
49.35
33.44
39.86
38.81
Training. We use the DeepLabV3 [7] architecture using ResNet-101 [13] pretrained on Ima- geNet [26] as a backbone network. Following [5, 6, 9, 22], we adopt different training strategies for each dataset. For PASCAL VOC [11], we train our model with 60 epochs for both initial and incremental steps, with batch size of 32. We adopt polynomial learning rate scheduler, where learning rates are set to 0.001 and 0.0001 for initial and incremental steps, respectively. We em- pirically set y to 2 during an initial step and for others. For ADE20K [30], we train our model for 100 epochs with a batch size of 24. Following [6], we adopt the poly learning rate scheduler with a linear warm-up [12], where learning rates are set 0.0025 for an initial step and 0. 0.00025 for incremental ones, respectively. We set y to 35 for all training steps. For both datasets, we adopt the SGD optimizer with momentum of 0.9, and set and 3 to We implement our model using PyTorch [24] and train it with four NVIDIA RTX A5000 GPUs.
Evaluation metrics. Following [5, 6, 9, 22], we report mloU,, mIoU,, and mloUai scores, that is, antersection-over-unioo (IoU) scores averaged over base, novel and all classes, respectively. Simply averaging the IoU score over all classes (i.e., mloU,.) is not appropriate to evaluate the performance of CISS models, especially for the case that the number of novel classes is relatively small compared to that of base ones. Accordingly, we also report a harmonic mean (hloU) of mloU, and mloU, scores, which is less susceptible to the imbalance between base and novel classes.
7

---

80
60
40
60
ralou (96)
30
30
50
60
40
LoU (%)
40
LoU
20
30
10
20
5
4
5
Steps
Steps
Steps
Steps
Steps
(a) Base classes.
(b) Potted plant.
(c) Sheep.
(d) Sofa.
(e) Train.
Figure 3: mloU comparisons of state-of-the-art CISS methods [5, 6, 9] over training steps. We train
CISS models for 15 base classes initially, and add single novel class for each incremental step (i.e., 15-1 setting with steps). We show variations mloU scores for base classes (a) and for individual novel classes separately (b-e).
Input image.
Stop 1.
Step 2. potted plant)
Step 3. (sheeep)
Step 4. (soffi)
Step 5. (razin)
Step 6.
Ground truth.
Figure 4: Qualitative results for the 15-1 overlapped setting on PASCAL VOC [1J]
chair
dog. and
person belong to base classes
4.2 Results
We show in Tables and quantitative comparisons between ours and state-of-the-arr CISS meth- ods [5, 6, 9, 22] on PASCAL VOC[L]] and ADE20K [30], respectively. To better demonstrate the effectiveness of our approach, we also report results for joint training that serve as upper bounds. For fair comparison with SSUL-M [6] exploiting an external memory, we also report the results (Ours-M) obtained using previous training samples, following the official implementation provided by the authors.
We observe from Tables and 2 three things: (1) CISS methods using BCE term [6] to learn novel classes, including ours, perform better than other approaches [5, 9, 22] employing softmax CE term, demonstrating the drawback of the softmax function for CISS. (2) Among competitive methods without using an external memory, our method achieves a new state of the art for all cases in terms of mIoU,, and hloU scores. This suggests that ours preserves knowledge learned from base classes, while learning novel ones effectively, compared to other methods, achieving better compromise between rigidity and plasticity for CISS. We can also see that our method outperforms others in terms of a hloU score for all cases by significant margin. (3) The external memory provides complementary information, and this brings additional performance gains. Our method (Ours-M) clearly outperforms SSUL-M in terms of mloUan and hloU scores. Note that SSUL and SSUL-MM also exploit an off-the-shelf saliency detector [15], pretrained on additional training samples [17], which requires computational power and memory for training.

---

Table 3: Quantitative comparisons for variants of our method under the overlapped setting on PASCAL VOC [11]. All numbers are obtained by averaging results over five runs with standard deviations.
Baseline
Initialization
Cas
15-1 (6 steps)
(Cutee Lgg)
Random
Ours
add
miooU
mloU,
hloU
mioUja
76.04-1:.82
35,16:1.533
48.07=1
66.30:0..3
77.97:0.32
36.53:+1.18
49.74=1110
68.10:10:42
74.43+11.15
39.4141.511
51.53=1153
6..09
77 07.94
42.47+ 1.54
54.80=13 1.31
69.45:00551
75.92:+:1
40.27::
52.62=
67.43=
0..09+0.32
42.72::: 1.58
15.21:1::33
69.67:00.9
DXD
w/o
w/ DXT
w/ DKD
w/ TXX
1.0
0.5
250
500
750
1000
00
250
500
750
1000
250
500
750
1000
Iterattons
Iterations
Iterations
(a)
12,""
(b) 2
(c) ||
Z
Figure 5: Quantitative comparisons of our model trained with and without the DKD term. We plot values of | - .ll. I|z _, ll. and I|z, z over iterations. We obtain the results for the
single incremental step under 19-1 scenario on PASCAL VOC [11].
We also compare in Fig. 3 our method with the state of the art, including MiB [5]. PLOP [9] and SSUL [6]. over training steps in terms of mloU. We can see that method avoids catastrophic forgetting effectively, maintaining mloU scores for both base and novel classes over a number of steps. In contrast, other methods often fail preserve the mIoU scores of old classes in later steps. Except for the novel class at the fourth incremental step in Fig. 3(d), where SSUL [6] is slightly better than ours, our approach outperforms the state of the for all training steps by significant margin.
We show in Fig. qualitative results on the PASCAL VOC [11]. We can see that our method successfully learns novel classes incrementally without forgetting previously learned classes. More qualitative results can be found in the supplement.
4.3 Discussion
We show in Table 3 an ablation analysis of our approach. The baseline model in the first row uses mBCE and KD terms only. Note that the baseline already performs comparable to state-of-the-a methods [5, 6, 9, 22], confirming once more the significance of employing the BCE loss for CISS. The last row shows that exploiting the DKD loss along with our initialization technique achieves the best performance for all metrics. We further provide a detailed analysis for the two components in the following.
DKD. From the first and the second rows in Table 3, can see that the DKD term boosts the performance for both base and novel classes, as it helps to retain knowledge even after incremental steps. Moreover, the second row shows that adopting the DKD term performs significantly better, compared to freezing a feature extractor as in SSUL [6], in terms of mloU, score (See the result in Table 1). This suggests that our DKD technique is more effective for maintaining the rigidity of CISS model. We plot in Fig. 5 numerical values of | z |I. ||z, 21-1ll- and ||z, 2:-1ll over iterations, where || is the 1.2 norm. As positive and negative reasoning scores, 2, and 2, are similar to previous ones, 2,=, and z,_ respectively, ||z," - 2,= || and |z, z,-il becomes smaller. We can see from Figs. 5(a) and (b) that the model trained without the DKD term does not preserve the reasoning scores effectively. On the contrary, the DKD term prevents abrupt changes of the scores. Note that ||z, - 2.-1 quantifies the consistency of predictions before and after an incremental step, suggesting that minimizing ||z, zt-1l ll is also crucial to alleviate catastrophic forgetting. We can observe in Fig. 5(c) that the DKD term helps to further minimize ||z, 24-1]]- These results verify that the DKD term improves the rigidity of CISS model together with KD.

---

(c) w/ initializatton
(a) Input and ground truth.
(b) w/o initialization.
owerluveelle Figure 6: Visual comparisons of activation maps and segmentation results for the 15-1 overlapped setting of PASCAL VOC [11]. We show activation maps for frain and predictions using our model with and without the initialization technique. Our model learns traii class in the 4-th incremental step, after learning potted plant sheep. and sofa incrementally. car, buspperson classes.
person chair belong to base
Initialization. The first and third rows in Table 3 demonstrate that our initialization technique itself provides considerable performance gain for novel classes in terms of the mloU, score, verifying that properly initializing classifier weights for novel classes is crucial for CISS. The initialization technique alleviates the problem, caused by lack of negatives each incremental step, and provides strong prior knowledge to learn novel classes. Note that our model in the third row already gives competitive results compared SSUL [6] in terms of the mloU, score, even without exploiting an off-the-shelf saliency detector [15]. We provide in Fig. 6 visual comparisons of activation maps for a novel class (i.e., train)), and segmentation labels for all target classes (i.e., 15 base classes and the incremental ones of potted plant sheep, sofa train). We obtain the results using classifiers trained with and without our initialization technique. We can see that the classifiers without our initialization often activates incorrectly on background regions (Fig. 6(b) bottom) or previous classes (i.e.. bus in Fig. 6(b) top). This distracts classifiers for previous classes, resulting in incorrect semantic labels in the regions. On the contrary, the classifiers initialized with our technique successfully suppresses false class probabilities for those regions, providing better segmentation results.
Note that train an auxiliary classifier in advance of learning novel classes for incremental steps. This suggests that the initialization technique can be adopted only when segmentation models are trained from scratch, and thus pre-trale mre-trained off sffftf segmentation models could not be exploited directly. Other techniques including mBCE and DKD in Sec. 3.2 can be applied to the off-the-shelf models, which brings significant performance gains, compared to current CISS methods [5, 22, 29] (See Tables and 3).
Note also that the auxiliary classifier makes segmentation models aware of the fact that novel classes will be added in the future. This sereemnnt-wwar continual learning achieves a new state of the art on standard CISS benchmarks with negligible computational overhead for training, and it will provide novel paradigm for CIL.
Conclusion
We have presented a novel framework that shows good trade-off between rigidity and plasticity for class-incremental semantic segmentation (CISS). In particular, we have introduced a new learning paradigm, decompose to distill knowledge, to improve the rigidity, and have proposed a novel initialization technique to learn novel classes better. Finally, have shown that our framework achieves a new state of the art on standard CISS benchmarks.
paradi
Acknowledgments and Disclosure of Funding
This work was partly supported by Institute of Information & communications Technology Plan- ning & Evaluation (ITT)) grant funded by the Korea government (MSIT) No.RS-20022-0144352 Development of Fundamental Technology and Integrated Solution for fext-Geeerratin Automatic Artificial Intelligence System) and the KIST Institutional Program (Project No.2E31051-21 2E110051211200))
10

---

References
[1] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang. Hyojun Kim, and Taesup Moon. SS-IL: Separated softmax for incremental learning. In ICCV, 2021.
Eden Belouadah and Adrian Popescu. IL2M: Class incremental learning with dual memory. In ICCV, 2019.
[3 Sergi Caelles, Kevis-Kokiite Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool, One-shot vidco object segmentation. In CVPR, 2017.
[4] Francisco Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, 2018.
[5] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental lcarning in semantic segmentation. In CVPR, 2020.
[6] Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al. SSUL: Semantic segmentation with unknown label for exemplar-base basel class-incrementa learning. In NewrIPS, 2021.
[7] Jlang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 170.:0:070008888 2017.
[8] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, 2019.
[9] Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord. PLOP: Learning without forgetting for continual semantic segmentation. In CVPR, 2021.
[10] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020.
[11] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303-338 2010.
[12] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming Hc, Accurate, large minibatch sgd: Training imagenct preprint arxiv: 0eee)7766000 02677,, 2017.
hour. arXiv
[13] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 500.00231,, 2015.
[15] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji. Zhuowen Tu, and Philip HS Torr. Deeply supervised salient object detection with short connections. In CVPR, 2017.
[16] Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In CVPR, 2019.
[17] Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung- Yeung Shum. Learning detect salient object. IEEE TPAMI, 33(2):353-361 2010.
[18] Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanuttigh. RECALL: Replay-based continual learning semantic segmentation. In ICCV, 2021.
[19] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 24:109-165, 1989.
[20] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plastiiity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
[21] Umberto Michicli and Pictro Zanuttigh. Incremental learning techniques for semantic segmentation. In ICCVW, 2019.
[22] Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion repulsionnattracioonn of sparse and disentangled latent representations. In CVPR, 2021.

---

[23] Sudhanshu Mittal, Silvio Galesso, and Thomas Brox. Essentials for class incremental learning. In CVPRW, 2021.
[24] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[25]
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incre- mental classifier and representation learning. In CVPR, 2017.
[26] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJC,, 115())211-252 2015.
[27] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.
[28] Saining Xie and Zhuowen Tu. Holistically-nestel edge detection. In ICCV, 2015.
[29] Chang-Bin Zhang, Jia-Wen Xiao, Xialei Liu, Ying-Cong Chen, and Ming-Ming Cheng. Representation compensation networks for continual semantic segmentation. In CVPR, 2022.
[30] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes) See Section 4.3.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] But our code and models is available online.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 4.2.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "btRqR3UYMz", "reviewer": "Reviewer_vbQC", "summary": "This paper proposes a CISS framework which decomposes the prediction logit into two terms.  Then the DKD technique and an auxiliary classifier is adopted to improve the rigidity and transfer the knowledge of negatives, respectively. Experimental results demonstrate the effectiveness of the proposed framework.", "questions": "1. One contribution of this paper is to propose the decomposed knowledge distillation (DKD) loss, which distills the knowledge of the decomposition. And the decomposition values are obtained by decomposing the prediction logit into two scores (i.e., element-wise sum of positive or negative elements). The decomposition strategy is unconvincing, please provide the theoretical proof.\n2. In Equation 1, the hyperparameters and are necessary to the entire model. How are they selected in the experiments? Please provide the experiments about the hyperparameters. And dynamic selection will be a good choice to balance the hyperparameters.\n3. Both KD loss and DKD loss help the model retain the old knowledge, and is there any conflict between them? Please add the ablation study.\n4. Please compare the proposed algorithm with the state-of-arts method [1].\n5. Please add experiments on the cityscapes dataset [2].\n6. The current manuscript need to be carefully polished. \n\n[1] Chang-Bin Zhang, Jia-Wen Xiao, Xialei Liu, Ying-Cong Chen, Ming-Ming Cheng. Representation Compensation Networks for Continual Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.\n[2] M. Cordts, M. Omran, S. Ramos, T. Reheld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.", "limitations": "+The first limitation is whether the framework can still perform well on a large number of task. \n+The second limitation is that the hyperparameters in formula (1) are not dynamic.\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This paper proposes a new distillation method, DKD to address the forgetting problem. \n+ Extensive comparison experiments illustrate the effective perofrmance of the proposed model to some degree.\n+ The main idea of this paper is easy to understand.\n\nWeaknesses:\n+The contributions of this paper are somewhat limited, which cannot meet the requirements of NeurIPS acceptance.\n+The motivation of some parts of this paper (distillation and initialization) needs to provide more clear explanations. \n+It would be beeter to discuss how the backbone architecture affects continual learning performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "PiXF2LP-sG", "reviewer": "Reviewer_X4eN", "summary": "The paper proposes a technique for class-incremental learning in semantic segmentation. First, a knowledge distillation loss is proposed, decomposing positive and negative class logits into separate loss functions. Second, an initialization strategy for the classifiers of novel classes is proposed, which is relies on an auxiliary classifier. Evaluation on several standard benchmarks for incremental semantic segmentation shows the methods effectiveness.", "questions": "1. Could you state from where you adopted the knowledge distillation loss in Eq. 5? Also from SSUL [6]?\n\n2. Do the 5 different runs use different random seeds?\n", "limitations": "Limitations have been discussed to some degree in the supplementary. I think there are, however, two limitations which have not been clearly stated. The method requires modifications to the semantic segmentation model, which makes the method not applicable to off-the-shelf models (cf. page 5, ll. 181-185). Also, the assumption that new classes rarely appear in in old data (page 5, ll. 186-188) is true for Pascal VOC, but not for more complex scenes such as Cityscapes, where such classes would often just not be labeled. This might make this method rather applicable to simple scenes. I would be happy to hear the authors’ take on these thoughts.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The introduction and related work provide a clear motivation for the proposed method and overall discuss the contributions well w.r.t. to related works.\n\n2. Clear and extensive mathematical description of the method, which seems reproducible, although code is not provided in the submission.\n\n3. State-of-the-art results on the standard incremental semantic segmentation benchmarks.\n\n4. Nicely prepared figures and tables, clarity of the overall manuscript.\n\nWeaknesses:\n\n1. Lack of hyperparameter analysis: The methodology on how hyperparameters are determined is well-described, but I did not find an analysis in this work, on how changes in the method’s hyperparameters affect the performance.\n\n2. Applicability: The method requires a modification to the pre-trained model before the incremental steps. Therefore, the method is not applicable to off-the-shelf semantic segmentation models and from an applicability perspective rather comparable to replay-based approaches. I still think, the memory-footprint of the method is much smaller than other replay-based methods (e.g., SSUL), but I think the limitation should be clearly stated as it is a bit different from the standard incremental setting for semantic segmentation.\n\n3. Root cause of improved performance: It is a bit unclear how much of the improvement towards SOTA performance is due to better chosen hyperparameters (such as, e.g., larger batch size on better hardware) or how much is due to the contributions. This could be easily fixed by adding a self-simulated version of MiB [5] or SSUL-M [6] in the author’s simulation setting to the comparison in Tables 1 and 2, showing how the self-simulated version of these methods compare to the original numbers provided in the papers.\n\nOverall, I really like the approach taken in this work, the paper is very well written and although some weaknesses definitely exist, the overall value of the contributions has been shown. I would welcome a clear answer to the weaknesses in the rebuttal as some of them might be easy to address.\n\nMinor comments and suggestions:\n\n4. I think the related work Maracani et al. “RECALL: Replay-based Continual Learning in Semantic Segmentation,” ICCV 2021, should be discussed and added to the experimental comparison. The work is similar to the strongly related work SSUL [6] and therefore appears to be relevant also for this work.\n\n5. The conclusion mainly repeats the abstract. It would be beneficial to put a bit more focus on future works, larger impact of this method on the field of semantic segmentation, or limitations of the method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4zKKuLoDZN2", "reviewer": "Reviewer_HGuy", "summary": "The paper addresses incremental learning applied to semantic segmentation. There are two main contributions: adding an extra loss term aiming at preserving separately the positive and negative contribution values in the classification logit and learning an additional classifier for negative sample prediction that can be transferred to initialize the new classifier with incremented categories. Evaluation is done on two classical benchmarks: PASCAL VOC and ADE20k.\n", "questions": "I do not have any precise question, only a request for more justification of the two main contributions (see the weaknesses point).\n", "limitations": "Non applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper is clearly written, and mostly well motivated (see however my remarks below).\n\n- The ablation study, experiments and result analysis justify the approach on standard benchmarks.\n\n\nWeaknesses\n\n- Finding a good loss for class imbalance management is an old problem in ML (e.g., the focal loss that has been used in object detection or AUC maximization). The approach should have been compared, at least in principle, to solutions that have been proposed in the literature outside the field of incremental learning.\n\n- The auxiliary classifier learned to initialize the next incremental step seems to be the most original idea of the paper, and should have been better developed and studied. The impact  of initialization, as illustrated in a few examples in Figure 5, seems to be strong and would have deserved a more in depth analysis, and, possibly, a more conceptual justification. Somehow, I am disappointed that the focus has been on the loss addressing class imbalance – there are other ways to handle it, and this is a well known phenomenon of incremental learning – and not on the initialization step.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept