SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning
Authors: Sungmin Cha
Beomyoung Kim
YoungJoon Yoo
Taesup Moon
Venue: NeurIPS 2021 Poster (2021)
Keywords: Class Incremental Semantic Segmentation
Unknown Class
Tiny Memory
Catastrophic Forgetting

Abstract: We consider a class-incremental semantic segmentation (CISS) problem. While some recently proposed algorithms utilized variants of knowledge distillation (KD) technique to tackle the problem, they only partially addressed the key additional challenges in CISS that causes the catastrophic forgetting; \textit{i.e.}, the semantic drift of the background class and multi-label prediction issue. To better address these challenges, we propose a new method, dubbed as SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining several techniques tailored for semantic segmentation. More specifically, we make three main contributions; (1) modeling \textit{unknown} class within the background class to help learning future classes (help plasticity), (2) \textit{freezing} backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing \textit{tiny exemplar memory} for the first time in CISS to improve \textit{both} plasticity and stability. As a result, we show our method achieves significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough and extensive ablation analyses and discuss different natures of the CISS problem compared to the standard class-incremental learning for classification. The official code is available at https://github.com/clovaai/SSUL.


SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning
Sungmin Chal-2; Beomyoung Kim'; YoungJoon Yoo and Taesup Moon'
Department of Electrical and Computer Engineering, Seoul National University
NAVER AI Lab," Face, NAVER Clova
sungmin. cha@snu ac ac kr, (beomyoung kim, tsmoonlsnu. ac.
www.:o/avvcrrrp..oommmm com,
Abstract
This paper introduces solid of-te-of-thh-ar baseline for a class-incremeee incrementael seman- tic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Se- mantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (I) defining unknown classes within the background class to help learn future classes (help plasticity). (2) freezing backbone network and past classi- fiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-an baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https:///ii thw.b.ttbbb....m com caem//looaai//se
Introduction
Class incremental learning (CIL.) problem, in which a learner should incrementally learn newly arriving class objects while not "catastrophically" forgetting the past learned classes, is one of the fundamental, yet still open, problems in machine learning. After the seminal work. [27], most of the recent neural network-based CIL research has focused on the classification setting, and various approaches have been proposed to address the main challenge of the problem, the so-called plasticity- stability dilemma, e.g., 20 30
to name few.
The CIL framework has been extended more complex semantic segmentation tasks 24 motivated by the practical need in various applications such as autonomous driving. One of the key additional difficulties of the class-incremental semantic segmentation (CISS) problem lies in the semantic drift of the background class present in the incrementally arriving training data. Namely, the label 'background (BG)" assigned all the pixels not included in the current class object region. The "BG" pixels belong to three categories: future object classes that the model does not yet observes, past object classes that are already learned, and the true background.
'Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A few recent works attempted to address the above semantic drift issue by leveraging and modifying the wentedge-disiillationn (KD) 20 technique, popular for CIL in standard image classification. Namely, the initiative study 24 modified the KD for CISS straightforwardly. and [3] proposed strategy to incorporate the BG class probability in computing the cross-entropy and distillation losses. Furthermore, [7]. the current state-of-the-art utilized the pseudo-labeling of the BG pixels of the current task data with the model of the previous task using the cross-entropy loss, and it applies the multi-scale feature distillation scheme adopted from [8] However, we argue that above works only partially addressed the BG label issue. That is, [3] naively added the class probabilities to modify the cross-entropy and distillation losses making it hard to have ann-ggaieed learning of prediction probability for each class, each pixel. Moreover, 7 could only handle the BG class pixels to the past classes via weeudollabeling and lacked any mechanism for handling the future class case, one possible option for the the BG class. Consequently, their CII, segmentation performance, measured by mean crrrrectioo-ove-Unnieee (mloU), has been significantly lower than the upper-bound, the case of joint-training with all the labels.
This paper first identifies that the multi-label prediction of semantic segmentation is another critical challenge of CISS and proposes SSUL-M (Semantic Segmentation with Unknown Label with Memory) to address the challenge. Specifically, Our contributions are summarized as follows. First, we introduce an additional "Unknown" class label assigned to the objects in the background, detected by the off-the-shelf saliency-map detector. We let the base feature extractor distinguish the representations of the potential future class objects and the actual background region by augmenting the BG label with this additional class. Second, adopt the pscudo-labeling strategy in 77 and further augment the BG & Unknown class labels with the past class labels, but with two essential differences in concrete learning strategies to stabilize the classification scores and improve the precision of the prediction. One is using the separate sigmoid, instead of the softmax, output classifier for each class so that the model can learn the logit score in an absolute sense per class. The other is freezing the base feature extractor and the classifiers for past classes after initial learning to strictly maintain the past classes' knowledge. Third, we utilize an exemplar memory store tiny portion of training data, including past classes, as anchors and further improve the mloU. Note that using the exemplar memory is a standard practice for CIL in classification but has been overlooked in CISS. Moreover, we show that the memory helps improve the mloU for the current classes, in contrast to common belief that it is tool to prevent forgetting of past classes.
By integrating the above contributions, SSUL-M achieved the state-of-the-arr performance on popular benchmarks with significantly large margin over the recent baselines [3] 7). particularly when the number of incremental tasks gets larger. Furthermore, we conduct extensive ablation studies and both quantitative and qualitative analyses to convincingly highlight the strength of our method.
2 Related Work
considers the setting in which new class objects arrive incrementally and the model needs classify all the classes without storing all the past class data. It is well known that neural network- based CIL suffers from catastrophic forgetting [23], caused by the score bias toward the new classes due to the training data imbalance. Most CIL studies have focused on the classification tasks, and the exemplar-memory based methods combined with KD
Class Incremental I Learning (CIL.) CIL. 28 25
30 12 i 2 g achieved the state-of-the of-the-arttttttt performance by re-balancing the biased predictions of the classifier.
Class Incremental Semantic Segmentation (CISS) Contrary to the current trend of CIL for classification, CIL for semantic segmentation have only focused on the setting without utilizing the semplaa-memoor (24] B 77. (3] first addressed the semantic drift of BG label, and 7 proposed to use pseudo-labels to augment the BG label, all without exemplar memory. xxempaa-memory. To our knowledge, we firstly use exemplar- memooy and potential future classes in BG label to solve CISS.
Saliency Map Detection Salient object detection is a fundamental computer vision task that identifies the most visually distinctive objects an image. We use the off-the-shelf saliency-map detector to define the Unknown class in the BG label. The early salient object detection method, DRFT [3] conducts a multi-level segmentation with a random forest regressor to detect a salient object. Recent deep neural network based method exploits rich multi-scale feature maps with short
2

---

connections []] and pooling- based modules (2]]. The saliency information has widely been utilized on various tasks, For example, the weakly semantic segmentation approaches (29 3T 14 generate pseudo-labels filtering out the background regions using saliency map, and recent data augmentations 17 [6]
also exploit the saliency information to find the optimal mixing of mask.
3 Proposed Method
3.1 Notations and Problem Setting
We consider exactly the same setting as considered in 3 7. In CISS, the learning happens with t= 1....., . incremental tasks. For each incremental state we observe training dataset 'D, that consists of pairs ( yr), in which 2t E X' denotes an input image of size N, and y/ E 3 denotes the corresponding
ground-truth (GT) pixel labels. The label space ) = {ca} U C2 consists of the current classes in task c, C, and the dummy background class Cg. that assigned to all pixels that do not belong to C. Thus, the label can be assigned to the objects with the past classes Cl:t-1, the objects with the future classes C++1:1 or the true background pixels.
After learning task f, the semantic segmentation model fg is required to predict whether a pixel belongs to all the classes learned so far, Ct: = Cl:t-1 U C,, or the true background. As in other work, assume the classes in each C. are disjoint. Typically, the model is defined to be mapping fg X + N>("", in which y = Cu U with C. containing dummy labels. All previous work (777 simply set Cu {co.. but in our work, we also add the separate "Unknown" class label, Cu. to Ca, hence, set Cd {ca, Cu). Later, we show defining this additional label Cu our model output plays a critical role in improving the learning capability for future classes. The architecture of typically fully-convolutional network, which consists of a convolutional feature extractor, M4 followed by the final 1 x 1 classifier filters, {Ø') one for each output class in yr
The learning of fo is done in conjunction with the previous model f-11; X' + R N to prevent forgetting during incrementally updating the model. Determining which output classifier (e.g.. softmax sigmoid) to use for each pixel as well as how to transfer the knowledge of f/ få (e.g.. knowledge distillation or model freezing) are design choices, and we elaborate our choices more in details in the subsequent sections. Furthermore, we denote M as the exemplar memory, which can store a small number of samples from past training data, D1:t-1, and use it for learning Jo
Once the learning of f is done, the prediction for pixel of an input image is obtained by
g! arg max fe)
and the performance is measured by the mean intersection-ovee-unioo (mloU) metric for the classes in ) (Only in the evaluation phase, we merge c and cu for computing mIoU of the BG class.)
3.2 Two Additional Challenges of CISS
In addition to the typical reason for causing catastrophic forgetting in CIL for classification, i.e., the prediction bias due the data imbalance, note there two additional unique challenges in CISS.
The first challenge, as mentioned in the Introduction and in previous work B 7 comes from the semantic drift of the background (BG) label. Namely, the ground-truth label associated with the pixels of the object can change depending on the incremental state. For example, when a pixel labeled as "BG" at state I, it is possible that the corresponding pixel would be labeled as "offa" at state 1 (i.e., past class) or labeled as "dog" at state + 1 (i.e., future class), depending on the true object to which the pixel belongs. Therefore, naively learning with the "BG" target label for the pixel at state could cause either severe forgetting of the past class (i.e., hurting stability) or interfering the learning of the future class (i.e., hurting plasticity).
The second challenge stems from the fact that semantic segmentation is a multi-labe prediction problem. Namely, for given image, the segmentation model needs to output a set of classes, in contrast to the classification model which outputs only single class. Therefore, the precision of the prediction for each pixel becomes important in addition to the recall as is reflected in the mloU performance metric; iee, not only predicting a correct class for a pixel is important, but also not predicting a wrong class is important for the overall metric. That is, if the prediction for every pixel
3

---

Task é data
printt magin
Librron
Person
Unknown Class Modeling
r540 poollyonyyy Come to
Map
Saliency Map
Y
Pseudo Labeling
9
sampling
X2
Exemplar-memory
Freeze
f8
BCE
YE
Training
sampling
Training
Figure 1: Overall procedure of SSUL.. Given (æ,, y.) E Dr, the augmented label ÿt is first obtained by the "Unknown" class modeling and pseudo-labeling. Then, using ÿt as a target, we update fg with model freezing and BCE loss. The exemplar-memory is also updated with class-balanced sampling.
gets biased toward the current classes in Cr, the mloU's for the past classes Cl:t-1 as well as the current classes would significantly drop jointly. This point is exactly why the mloU's for the newly learned classes are very low in 7.7. Note this a stark difference with the classification, in which the accuracy of the current classes would remain high with the severe bias and forgetting, since it a single-label prediction problem.
To address above unique challenges of CISS, we devise our SSUL-M by carefully combining several ideas, which overall procedure outlined in Figure [ We now elaborate on our three main contributions in details: 1) label augmentation for BG class with Unknown class and pseudo-labele 2) stable score learning with model freeze and sigmoid output, and 3) usage of tiny exemplar memory with Mass-balanced sampling.
3.3 Contribution 1: Label Augmentation for BG Class
Here, we describe how we generate augmented target label Dr E (D {cb, Cus C2ag given training sample (æe, yr) in D, Recall that y: E (D))* {co, C.). C.J.A C.V.V.
Unknown class modeling In order to handle the case in which the potential future class objects labeled as BG, we propose to use a novel unknown class label, cu. that is defined to designate any foreground objects that not the current classes in C, Specifically, Figure ! we first apply an off-the-shelf salient object detector [T]] S > {0, 1} to the input the pixels if they salient (i.e., part of foreground object) and
depicted in the top part of
image 3t, which assigns otherwise. Then, we assign ca to the pixels that are BG-labeled but salient; namely, we set Út., = Cu if (yt. (a) A (S(x;); a critical role so that the base feature extractor can distinguish the representations of the potential future class objects and the true background, even before observing the class labels.
for pixel i. In experiments, show this Unknown class label plays
Eseudo-labeling Once augmenting with c, done, we further augment with pseudo-labels generated from the previous model f maintain the knowledge of the past classes, similarly as in 7 and as shown the middle part of Figure Namely, we denote Út-1,i 1, = arg max f6 (xx) as the prediction of f/ =t-1 for the i-th pixel and set ÿt,, #.-1, if
(yr, = A (5: 1,1 E CLiL-1) A (n, > T):
in which μ4; = maxegcua o af (x:)) stands for the confidence of prediction for ÿt-1, E C1:f-1 0,
where (7)) is the sigmoid function, and is threshold (set to 0.7). In words, assign the pseudo- labels (i.e,, the predictions from the previous model) to the pixels that are BG-labeled only when the predictions made to be the past object classes (excluding c and c.) with enough confidence.

---

Task t - 1
0.05
Freeze
00.33
x-1
π--1
0.51
Sigmoid
0.70
0.24
Freeze
jo 06
Training with (x.})
0.18
Inference with
Softmax
Frazen weights
Trainable weights
0.32
Noisy pseudo label
0.21
Freeze
1..75
Unknown
Sofa
10.35
Xt
y
0.60
0.44
Sigmoid
0.05
0.12
Freeze
10.03
0.51
Task t
0.00
Softmax
Figure 2: Comparison of the effect on the classification scores for sigmoid with binary cross-entropy (BCE) and softmax with cross-entropy (CE). For pixel with noisy label ÿt.; cs (i.e., BG label (black) when the true is "sofa" (green), see the red dashed circle at task (), even when the classifier for "sofa" (f cEchiti1) score for "sofa" (see the red dashed arrow), whereas the sigmoid would let the BG score rise only moderately (see the blue dashed arrow). Thus, the classification for the pixel could still become "sofa" for sigmoid, whereas softmax would make a false prediction as BG.
frozen, the softmax would cause the BG class score to rise above the
In summary, at the incremental state I, the augmented target label for the i-th pixel becomes
Vt.b if yt, Ce
U1-1,
if (3),, ct) A (#1-1, E (m²r))
Cu
(yr,, = cb) A (S(a)), = 1) ((0-11 E C.) v (n, ST)
else.
in which the pseudo-label is generated only for the incremental state 2. Figure shows concrete
example of augmentating y: to Dr. in which C- = {sofa) and C. {person).
3.4 Contribution 2: Stable Score Learning
We argue that simply using Dt target and training fi with softmax output layer would make the output scores too unstable as the incremental learning continues. The reason because the augmented labels are noisy and the softmax computes the prediction probability in relative way. As shown in our experiments, such instability significantly hurts the precision of the multi-label prediction. To that end, propose the following three strategies for the stable learning of output scores.
Model freezing Instead of updating the full model with ÿr at every state I, we freeze the convolu- tional feature extractor, htg.. after initial learning (t = 1) as well as the classifiers for the past classes, ( and only learn the x filters & &i and (d') Such strict model freezing certainly can prevent forgetting, but is
from the plasticity viewpoint. However, thanks
5

---

to the unknown class it turns out the feature extractor fy can roughly learn the representations for the potential future class objects present with BG label in D,. Thus, becomes sufficient to learn the decision boundaries for cb, Cu. and C2 on those representations at state 1.
Sigmoid with binary cross entropy loss For learning l',, & and (Ø') the choice of output layer and loss function becomes important to make the output score stable. A common choice is the softmax with cross-entropy loss, however, since the target labels for eæ and c. in Yt. are noisy, we observe the relative scoring of softmax could cause significant forgetting of past classes C1:t-1 To see this, let Sic 1 (ee Iee be the score for class c pixel When the softmax with cross-entropy (CE) is used, the gradient of the loss at Ste becomes dC (0)/Oscc Pic - 1 Dt.s J. in which Pie = exp( Ste) (tee exp(sie))). The issue occurs when yt. = c or Cu. while the true label for the pixel should be the past classes in Cl:t-1, (Thus, the pseudo-label 9/-11, missed the pixel, which could often happen.) Then, the gradient descent learning will force Stc, or Sic, to become much higher than {ste feeet 1, the scores obtained from the frozen classifiers for past classes. Thus, at test time, for a similarly confusing pixel, the model would tend to predict as or Ca. hence, causing the forgetting of past classes even though their classifiers are fixed.
Therefore, we instead use the sigmoid output with binary cross-entropy (BCE) loss independently for each class. In that case, the gradient of the loss Sic. becomes Of (0)/Osic (8ic) (c yt,},, hence, even for above noisy target label case, the scores 8ic, or Sic, will only increase to a certain absolute level regardless of other class score values, {8te) cech Thus, test time, similar pixel still may be predicted as past class in Ct-1 since the frozen past classifiers would still output considerable score this subtle difference between the loss functions is illustrated in Figure 2
Weight transfer from unknown class classifier Finally, for learning [e'). we initialize all the filters with & the classifier learned for the unknown class cu the previous state. The reasoning is that the classes in C. would have been labeled as Cu (as the potential future classes) in state t hence, such weight transfer from di would yield stable and fast learning of (Øe)
3.5 Contribution 3: Tiny Exemplar Memory
Using exemplar-memory to store small portion of past training data for CIL backed with theoretical finding 19 as well as strong empirical results (27 30 for classification.
Hence, we propose to use it for CISS as well with a tailored class-balanced sampling strategy. The main rationale of using the memory is to make sure to include at least one sample with correct GT label per each class in cl:t-1 in the training set for state t.
Algorithm 1: Pseudo code of SSUL-M
1: Require fo, M, T, Repechs, batch size K
2: fe t Initialize hap. Pess &c..
3: Initialize M
4: for t t- 1...., T'..,,'a do
Namely, even though the pseudo-label - can provide labels for the classes in Clit-1 it is also possible that the given image D, would never contain object cues for cla--1 In such a case, even with model freezing and stable score learn-
5: fb < Initialize [0'
6:
if t 1 then
7:
Freeze hap (0'loecur-
8:
(Φ' de')eEC. f- Weight Transfer(
9:
end if
ing, when the confidence for the new class is learned be high for pixel (potentially for an old class), the prediction for the pixel could get biased toward the new class, causing the for- getting of the old class. Therefore, by denoting
10: for n + 1,.... Nepochs do
Bn K/2 DD do
11:
for
BM K/2 .M
12:
M = [M], after learning incremental state t- 1,
13:
Bn, BM t- Label. Buabel.Au..re (Bn, B,) DGSGDBB, U BM, 10,,Cgon (gcc(
we sample M/ M/[c1=t-| data points from D,-1. 14:
and store them in the memory M by remov-
15:
end for
16: end for
ing equal number of samples per each class in
cl:-22 from M. In this way, M always contains
17: Update M
at least one sample from each class in C1:t-1, and we show the experiments that this class-
18: end for
19: return f6
balanced strategy is more helpful than random sampling [4] is done for CIL for classification.

---

3.6 Summary
We summarize our SSUL-M algorithm in Algorithm in which our contributions described in Section
Section 3.5 are denoted in the typewriter font. Namely, Label. Aug. stands for generating ý as in (1) for the selected batches B, and BM. "Freeze, Lecg and Weight Transfer" denote the methods for the stable score learning described in Section 3.4 and Update M denote the exemplar- memory maintenance with the class balanced sampling as mentioned in Section [35] Note we are constructing a mini-batch by sampling equal amount of data from Dr and M, hence, the samples in M act as anchor points to improve the precision of the predictions.
3.3
Experiments
4.1 Experimental Setting
Dataset We followed the experimental setting of 3 and evaluated our method using Pascal-VOO VOC 2012 [ and ADE20K [32 datasets. Originally, set experimental protocols, disjoint and overlapped, but we believe the latter is more realistic and challenging. Therefore, we evaluated on the overlapped setup only. Pascal VOC 2012 contains 20 foreground object classes and one background class, and ADE20K consists of 150 classes of both stuff and objects. We consider several incremental learning scenarios for each dataset, from the scenarios considered by the other baselines newly proposed challenging scenarios with larger number of incremental states. A more detailed description on the datasets is introduced in the Supplementary Material (S.M.).
We use the mean Intersection Union (mloU) as our evaluation metric, which computes the IoU for each class then computes the average over the classes. The IoU is defined as ToU = true-positive/ positivve + false-positee + fulse-negative).
Evaluation Metries
Implementation Details For all experiments. following other works 3], we use a Deeplab-
v3 segmentation network [5] with a ResNet-101 [IO] backbone pre-trained on ImageNet [6]. We optimize the network using SGD with initial learning rate of 10-2 and momentum value of 0.9 for CISS steps. Also, we set the learning rate schedule, data augmentation, and output stride of 16 following (5) for all experiments. For each incremental state I, we train the network for 50 epochs for Pascal VOC with a batch-size of 32 and 60 epochs for ADE20K with a batch-size of 24. We tune the hyperparameters using 20% of the training set validation set and report the final results the standard test set. For the exemplar memory, we utilized memory with a fixed size of |M| 100 for Pascal VOC and [M 300 for ADE20K. To highlight the effect of the report the results of the two versions of our method - SSUL (without memory) and SSUL-M (with memory). For the saliency-map detector to generate the Unknown class label, we employed DSS (II] pretrained on MSRA-B dataset [22]. which contains 5,000 labels for salient objects. The experiments were implemented in PyTorch [26] 1.7 with CUDA 10.1 and CuDNN 7 using two NVIDIA V100 GPUs, and all experiments were conducted with NSML experimental details are in the S.M.
memory,
15
framework. More information on the
Baselines For a representative of the general regularization-basee continual Icarning method, we select EWC [18] and LWC MM and applied them to each experimental setup of CISS. For CISS specific baselines, we compared with ILT (24). MiB [3] and PLOP (7). and the Joint Training (Joint) result is also given
upper bound. Note that PLOP 7 is the current tfte-offthe-art. We
reproduced the results of all baselines using the official code provided by the authors of (
4.2 Experimental results on benchmark dataset
Pascal VOC 2012 Following 7 3], evaluate method on four different scenarios, i.e., [10-1,
15-1, 15-5, and 19-1} as well as a more challenging scenario, (5-3). The numbers in each scenario denote the number of classes to be trained for each state. For example, VOC 5-3 means learning 5
classes at the base task (t = 1), and then incrementally learning 3 classes five times (hence, T = 6).
In Table I we observe our SSUL consistently outperforms the baselines with huge margin in all scenarios, even without using the exemplar-memory. Furthermore, the gap widens particularly for more challenging scenarios, namely, for the cases in which the base task has fewer classes and the number of tasks is larger. Note although MiB E and PL.OP (7] show robustness for simple tasks scenarios (19-1 and 15-5), their performance are rapidly dropped in more challenging and practical
7

---

Table Experimental results on Pascal VOC 2012.
VOC
10-1 fusks)
VOC
15-1 tasks)
VOC 5-3 tasks)
VOC
19-1 tasks)
VOC
15:5 (2 tusks)
Method
0-10
11-20
0-15
16-20
all
6-20
0-19
all
0-15
16-20
EWC 18
0.30
4.30
1:30
26.90
14.00
26.30
24.30
35.50
27.10
6.90
1LT 24
7.15
3.67
5.50
8.75
7.99
8.56
22.51
31.66
29.04
67.75
10.88
65.05
67.08
39.23
60.45
MiB 8
12.25
13.09
12.65
34.22
13.30
29.29
57.10
42.56
46.71
71.43
23.59
69.15
76.37
49.97
70.08
PLOP 2
44.03
15.51
30.45
65.12
21.11
54.64
17.48
19.16
18.68
75.35
37.35
73.54
75.73
51.71
70.09
SSUL
71.31
45.98
59.25
77.31
36.99
67.61
72.44
50.67
56.89
77.73
29.68
75.44
77.82
50.10
71.22
78.36
71.37
77.83
76.49
55.80
Joint
78.41
76.35
77.43
79.77
72.35
77.43
76.91
77.63
77.43
77.51
77.04
77.43
79.77
72.35
77.43
90
70
itou (ai))
60
micu tall
50
40
40
SSUL
PLOP
SSUL
SSUL-M
30
16-20
MIB
40
15
Number of leerned claszers
16
37
18
15
20
0-15
Classes
16-20
20 42
60 80
Memary size
100
120
180
(a)
(b)
(c)
Figure 3: (a): mloU evaluation on VOC 15-1, (b): mloU distributions for 20 different class-orderings for VOC 15-1, (c): mIOU on VOC 15-1 with varying memory size IM
scenarios (10-1, 15. More specifically, Figure shows the mloU evolution for 15-1 scenario each incremental step, and the baselines suffers from significant drop of mloU, as the new classes are incremenaally learned. In contrast, SSUL improves mloUs for both base (0-15) and newly learned (16-20) classes significantly, showing much slower drop of mloU. These results show that long as the label augmentation of BG class with Unknown class and pseudo-labels is properly done, our stable score learning very effective for CISS. Particularly, we observe that model freeze. which is believed to be not effective in CIL, is much more effective than the KD used in other baselines.
further strengthens SSUL significantly, particularly for the newly learned classes (i.e., for t 2 2). For example, in VOC 15-1, by storing only ~ 6 images per class in M, the mloU for classes 16-20 improved about 12%. This confirms our intuition that the samples in the memory act as "anchors" to improve the precision of the predictions, hence, prevent forgetting. We can also clearly observe this improvement in Figure 4 As an additional experiment, we also conducted the experiment on the case of further reducing the number of base classes, such 5-1 and 2-1, and again observe that SSUL and SSUL-M surpass other baselines. A detailed result on this additional experiment can be found also in the S.M.
Furthermore, observe SSUL-M, which
To verify the robustness of each method on various class orderings, experimented on the 20 differ- ence orderings on VOC 15-1 scenario, as proposed in [7]. Figure 3(b) plots the mloU distributions for different methods, and we clearly observe both SSUL and SULL-M not only achieve higher mloUs, but also have smaller variations compared to the baselines. In addition, we note SULL achieves x1.5 faster training time compared PLOP due to network freezing.
Qualitative analysis In Figure we visualized the qualitative results for four images from VOC 15-1 task. We observe that PI.OP partly maintains knowledge learned from the base task (bird and person), however, it becomes fatal in forgetting the classes learned the intermediate steps, such as plant and sheep. In addition, PLOP frequently produces many false-positive predictions, lowering the mloU of several classes. (E.g., see bird Step 5.) On the other hand, we observe SSUL and SSUL-M both maintains the previously learned classes with high stability and effectively learns new classes. For example, the sheep class is accurately learned at Step 3 and not forgotten afterwards. Moreover, observe SULL-M achieved further improvement of both plasticity and stability SULL, especially by reducing the false-positive predictions.
ADE20K Unlike VOC 2012, ADE20K is densely labeled for both stuff and thing with 150 classes. means that the class definition in ADE20K clearer therefore, naturally make reduce the concern about the semantic drift of BG label. To make efficient of this prior knowledge of dataset,

---

Table 2: Experimental results on ADE20K.
ADE 100-5 (11 tasks)
ADE
100-10 (6 tasks)
ADE
100-50 tasks)
ADE
50-50 (3 tasks)
Method
0-100
101-150
all
0-100
101-150
0-100
101-150
ull
0-50
511-150
all
ILT
0.08
1.31
0.49
0.11
3.06
1.09
18.29
14.40
17.00
3.53
12.85
9.70
MIB g
36.01
5.66
25.96
38.21
11.12
29.24
40.52
17.17
32.79
45.57
21.01
29.31
PLOP 7
39.11
7.81
28.75
40.48
13.61
31.59
41.87
14.89
32.94
48.83
20.99
30.40
SSUL
39.94
17.40
32.48
40.20
18.75
33.10
41.28
18.02
33.58
48.38
20.15
29.56
SSUL-M
42.86
17.78
34.56
42.86
17.66
34,46
42.79
17.54
34.37
49.12
20.10
29.77
Joint
44,30
28.20
38,90
44.30
28.20
38.90
44,30
28.20
38,90
51.10
33.30
38.90
Step 1
Step2 2
3lep3
Step4
Step
Step 6
Ground Truth
0:15
16.
101
SSUIIS
PLOP
Figure 4: Qualitative results of SSUL-M, SSUL and PLOP on VOC 15-1 scenario.
consider unlabeled pixels as unknown class without using the saliency detector and enlarge the size of memory .M| 300. We evaluated our method in four different scenarios, i.e., (100-5, 100-10, 50-50, 100-50) as in 7 B]. Table 2 again shows that SSUL achieves superior performance in the more challenging tasks (100-5, (00-10) than other methods. We believe that, this result demonstrates SSUL is also quite effective in CISS for densely labeled dataset, without any extra saliency detector. The qualitative analysis on ADE20K is provided in the S.M.
4.3 Ablation Study
Ablation study on proposed components of SSUL Here, we analyze the effect of each proposed component of SSUL on VOC 15-1 scenario. Table 3 compares the mloU of each ablation case, without memory, and the first low shows the result of SSUL with full components. Firstly, when the Unknown class is removed, we clearly observe that the mloU's 's of both the first task (0-15) and subsequent classes (16-20) decrease. Hence, this clearly demonstrates the advantage of using the Unknown class label, in terms of increasing both the plasticity and stability. Secondly, note that model freezing has a significant impact on the performance of CISS. Specifically. it not only prevents catastrophic forgetting on the first task, but also plays critical role to learn new classes well. Finally, we observe when Softmax+CE instead of Sigmoid+BCE is used (the last ablation), the forgetting of the first task (0-15 15) drops more significantly despite model freezing. We believe this confirms our intuition Section 3.4 on why Sigmoid+BCE can lead more stable score learning.
Saliency-map detector and weight transfer The first two columns in Table shows the ablation study on saliency-may detector for Unknown class and weight transfer. We compare the result of a default neural network-bact saliency map extractor, DSS [II] with ground-truie truth and a random forest-based DRFI [13. We observe the differences of the mIoU's among them are quite small, therefore, we believe that the quality of saliency map is not a significant factor of our method. The ablation study on weight transfer demonstrates weights for new classes initialized with t--1 is most effective to learn it. This result may at first look counter-intuitive since learning the final linear layer &e should not sensitive on the initialization. However, given the noisy pseudo-label for learning, a quicker convergence of Φ using the warm-start weight Φ5 would effectively prevent the forgetting of past class, which could be caused by fitting the noisy pseudo-label for many epochs. More details on the effect of weight transfer is proposed in S.M.

---

Table 3: Ablation study for SSUL VOC 15-1 about PL (pseudo labeling), Unknown (unknown class modeling & weight transfer), Freeze (model freezing), and Sigmoid+BCE (stable score learning).
15-1 tasks)
PL
Unknown
Freeze
Sigmoid+BCE
0-15
16-20
/
77.31
36.59
67.61
x
/
73.42
21.79
61.12
X
/
53.56
14.48
44.25
/
X
61.42
22.97
52.26
Table 4: Ablation study for saliency- map detector (left), weight transfer (mid), and memory sampling
(right) on VOC 15-1.
Saliency-maa detector
Weight transfer
Memory sampling
Methods
0-15
16-20
all
Methods
0-15
16-20
Methods
0-15
16-20
all
DRFT [13
76.46
32.53
66.00
random d
0%
73.73
23.99
61.89
random
78.61
38.87
69.15
DSS [1]
77.31
36.59
67.61
73.29
23.70
61.48
cluss-bulanced
78.36
49.01
71.37
ground-truth
78.42
40.41
69.36
77.31
36.59
67.61
Memory size and sampling rule The right column row of Table shows result the mloU on two sampling rules. We observe that, compared random sampling, proposed bass-balanced sampling achieves better mloU, particularly for the newly learned classes. We believe the reason is dlss-balanced sampling ensures at least one sample per class, therefore. prevents the forgetting of minority classes than random sampling, which can miss certain classes M. Finally, Figure 33c) shows the dependency of SSUL-M the memory size. It illustrates that tiny exemplar-ment memoyy for CISS significantly helps to increase the mloU for the newly learned classes (16-20) than the base classes (0-15), Moreover, we observe that after sufficiently large M the mloU performance becomes robust.
Concluding Remarks and Limitation
We proposed new class-incren maassinncremetal learning method SSUL-M (Semantic Segmentation with Un- known Label with Memory) for semantic segmentation. In order to address two additional challenges of CISS, we made three main contributions - label augmentation with Unknown class labels, stable score learning, and tiny exemplar memory. They all were convincingly shown to be very effective in various CISS scenarios and our SSUL.-M significantly outperformed baselines.
While promising, we admit our work also has some limitations as follows. First, our unknown class modeling may not be satisfactory for some "stuff" segmentation tasks since saliency-maps are mainly targeted for "things" (or objects). Second, our model freezing may harm plasticity and cause the model suffer from learning new classes especially when they are not captured by the unknown class label during the base task. Together with our insights from the analyses of SULL-M, we believe attempting address above limitations would lead fruitful future research directions for the CISS problem.
Acknowledgement
This work was done while Sungmin Cha did a research internship at NAVER AI Lab. This work also supported in part by the New Faculty Startup Fund from Scoul National University. NRF Mid-Career Research Program [NRF-202 222
1ITP grant [NO.2021 -0-01343, Ar- tificial Intelligence Graduate School Program (Scoul National University)] funded by the Korea government(MSTT), and and SNU-NAVER Hyperscale AI Center. The authors thank NAVER Smart Machine Learning (NSML) team for the GPU support. Taesup Moon also thanks the support from Automation and Systems Research Institute (ASRI) at Seoul National University.
10

---

References
[1] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. arXiv preprint arXiv:2003. 13947, 2020.
[2] Francisco M Castro, Manuel J Marin-Jimééne Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference computer vision (ECCV), pages 233-248, 2018.
[3] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9233-9242, 2020.
[4] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiay episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019.
[5] Liang- Chigh Chen, George Papandreou, Florian Schrofff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706. 1706008777,,,77,, 2017.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. lcce, 2009.
[7] Arthur Douillard, Yifu Chen, Arnaud Dapogay, and Matthieu Cord. Plop: Learning without forgetting for continual semantic segmentation. arXiv preprint arXiv:201 1..20111111990 11390, 2020.
[8] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCH 2020-16th European conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XX, volume 12365, pages 86-102. Springer, 2020.
[9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[11] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip HS Torr. Deeply supervised salient object detection with short connections. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 3203-3212, 2017.
[12] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang. and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 831-839, 2019.
[13] Huaizu Jiang, Jingdong Wang, Zejian Yuan, Yang Wu, Nanning Zheng, and Shipeng Li. Salient object detection: A discriminative regional feature integration approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2083 20000, 2013.
[14] Beomyoung Kim, Sangeun Han, and Junmo Kim. Discriminative region suppression for reakky-supervised semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1754-1761, 2021.
[15] Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. Nsml: Meet the mlaas platform with a real-world case study, arXiv preprint 18Xiv:1810.09957, 2018
[16] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pages 5275-5285. PMLR, 2020.
[17] JangHyyun Kim, Wonho Choo. Hosan Jeong. and Hyun Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In International Conference on Learning Representations, 2021.
[18] James Kirkpatrick, Razvan Pascanu. Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- -arbskkaaar Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):551-3322 2017.
11

---

[19] Jeremias Knoblauch, Hisham Husain. and Tom Diethe. Optimal continual learning has perfect memory and is np-hard. In International Conference Machine Learning, pages 5327-5337. PMLR, 2020.
[20] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935 2944,, 2017.
[21] Bang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Jiashi Feng, and Jianmin Jiang. A simple pooling-based design for real-time salient object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3917-3919 3926, 2019.
[22] Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung- Yeung Shum. Learning detect salient object. IEEE Transactions on Pattern analysis and machine intelligence. 33(2):353-367, 2010.
[23] Michael McCloskey Neal Cohen. Catastrophic interference connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[24] Umberto Michieli and Pietro Zanultigh. Incremental learning techniques for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[25] German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026 8037, 2019.
[27] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 2001-2010, 2017.
[28] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint 11Xiv:9904007734. 2019.
[29] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial crasing: A simple classification semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1568 1576, 2017.
[30] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 374-382, 2019.
[31] Qi Yao and Xiaojin Gong. Saliency guided self-attention network for weakly and semi supersiie semantic segmentation. IEEE Access, 8: 044413114423 2020.
[32] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633 641, 2017.
12


Reviews:
[{"review_id": "ku6Ec24HL_Z", "reviewer": "Reviewer_xEzA", "summary": "This paper proposes a method, Semantic Segmentation with Unknown Label with Memory (SSUL-M), to solve a class-incremental semantic segmentation (CISS) problem. The authors make three main contributions. First, they introduce an additional “Unknown” class label to distinguish the potential future class and the true background. Second, they adopt the pseudo-labeling strategy and freeze backbone network and past classiﬁers with binary cross-entropy loss to tackle catastrophic forgetting. Third, an exemplar memory is utilized to improve both plasticity and stability. Extensive experiments and ablation studies are conducted to justify these proposed components and contributions. Results show that SSUL-M signiﬁcantly outperformed baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tThis paper is well structured and clearly written. The authors introduce the proposed method logically and describe their contributions clearly. The experimental details are provided and look technically sound and reproducible.  \n2.\tThis work aims to tackle the challenges that cause catastrophic forgetting in CISS. To this end, the authors make three main contributions which are quite novel and each contribution has been proved effective by adequate ablation studies. Experimental results on two datasets show that the proposed method achieves signiﬁcantly better performance than the baselines.\n3.\tFigure 1 is not illustrated clearly. It would be better if the authors explain this figure clearly in its caption so that readers can quickly understand the meaning of each module or symbol in this figure at a glance. Additionally, there are some typos and grammar mistakes. I list several of them here: “SSUL and SSUL-M both maintains … and effectively learns” in page 8, “The ﬁrst two columns in Table 4 shows” in page 9, etc.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not describe the limitations in the paper and stated no expected potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5H"}, "extra_text": {}}, {"review_id": "kLM9ijcMP6U", "reviewer": "Reviewer_zwbF", "summary": "This paper proposes novel a class-incremental learning method, named semantic segmentation with unknown label with memory. The proposed method addresses two additional challenges in class-incremental learning and proposes three contributions: label augmentation, stable score learning, and tiny exemplar memory.  The proposed method achieves impressive performance compared with other methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " This paper proposes a novel class-incremental semantic segmentation setting. In addition, the writing of this paper is clear and easy to understand. The experiments are sufficient to prove the effectiveness of the proposed paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1 There are some related methods missed, such as [1][2][3].\n2 The proposed method employs three strategies to alleviate catastrophic forgetting. These strategies are often employed in other incremental methods. Thus, the contributions are limited in my opinion.\n3 Compared with the ground truth, the visualization results are terrible. Thus,  I want to see some analysis of satisfied cases and terrible cases.\n[1] Class-incremental learning for semantic segmentation re-using neither old data nor old labels. ITSC2020.\n[2] Modeling the background for incremental learning in semantic segmentation. CVPR2020.\n[3] Incremental Learning Techniques for Semantic Segmentation. ICCVW2019", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Two hours "}, "extra_text": {}}, {"review_id": "chpo5KwOxdz", "reviewer": "Reviewer_eHGE", "summary": "The paper presents a class-incremental learning strategy for semantic segmentation. The main contributions of this work include two aspects: first, it introduces an external saliency map predictor to better model the thing classes vs the stuff background; second, the method adds a class-balanced memory to overcome the forgetting in CIL. The proposed method is evaluated on two benchmarks with the overlapped setup and achieves better performance than the prior approaches.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper developed a well-engineered strategy for class incremental segmentation, which integrates several well-motivated components to reduce catastrophic forgetting. \n- The proposed method achieves strong results on two benchmarks under the overlapped setup, which validates its efficacy. It also includes a detailed ablative study on the main components. \n- The paper is most clearly written and easy to follow. \n\n\nCons:\n- The overall novelty of this work is a bit incremental. While the main components are clearly motivated, they are largely borrowed from the prior work in weakly-supervised segmentation (foreground saliency), class-incremental learning (memory), and [7] (pseudo-labels). The beneficial effects of those modules seem to be well understood. \n\n- There are several concerns on the technical aspects of the method:\n1) While adopting object saliency maps can help the thing classes, this would have a negative impact on the stuff classes. For semantic segmentation, this is restrictive for the general cases where we have to incrementally segment both the thing and stuff classes. This also implicitly introduces additional training data (saliency) which may not available for certain application domains. \n2) The model freezing step is also rather restrictive, which leads to weak plasticity for the model. In particular, the performance of the convolutional feature extractor can degrade significantly if the initial class set is small. It is unclear how well such a strategy can generalize to different CIL setups. \n3) How are the hyperparameters of the method determined, such as \\tau in pseudo labeling? \n4) Line 189: if the sigmoid output is adopted in training, how does this method calibrate the classifier scores for the inference in Sec 3.1? \n\n- The experimental evaluation seems a bit lacking and the conclusions are not fully convincing: \n1) The evaluation is only conducted on the overlapped setup, which is insufficient. The method should be tested on the disjoint setup too. The overlapped setup allows the method to learn features for future classes based on the saliency mask, which would favor the proposed strategy.\n2) How well the method would perform if the classes include both thing and stuff categories? \n3) Using external saliency maps seems to be unfair to other methods, as it requires additional training data. Such additional information can also help improve the prior methods. \n4) In the ablative study, it seems that adding the Sigmoid+BCE hurts the overall performance. It is unclear if this is a necessary component as Sec 3.4 claims.\n\nPost-rebuttal: The author's reply addressed many of my initial concerns and I would like to raise my rating accordingly. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Should add discussion on its limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2~3"}, "extra_text": {}}, {"review_id": "95dcVP8RtAJ", "reviewer": "Reviewer_8KnD", "summary": "This paper provided a recipe of methods to improve the incremental learning on semantic segmentation that empirically improved the performance on multiple datasets compared to other SOTA methods. Specifically, both pseudo labels from previous models and unknown classes from saliency detectors are adopted in model training; Backbone networks are frozen and Sigmoid + BCE are used to avoid forgetting; finally an exemplar dataset is used here to balance the training data in later times and hence further boost the performance. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written in general except a few questions that may be worth considering below. The originality of the work is moderate. Adding additional label sources and/or training data, however not exact, can usually improve deep models’ performance. The paper found an interesting and valid way to do so under the incremental learning setup. Using Sigmoid + BCE is another interesting application under this setting and the explanation makes sense.  Exemplar memory throws more data into the workflow -  more often than not, it will work if doing it right. \n\nA few question that may need further clarification: \n\nWith model freezing and stable score learning, in theory there would not be any forgetting for the foreground classes in t - 1? The explanation of having examples to avoid severe forgetting of any foreground classes seems unconvincing in Line 208. \n\nMore detailed explanation on the effectiveness of weight transfer would be helpful for the audience to appreciate the design better. In particular, after model freezing with only BCE in training, one would think that initialization matters less given the shallow structure. Why such a big difference as shown in the ablation study? \n\nThe ablation study is fascinating and considering the work is a recipe of multiple steps, it is worth more discussion and careful look to shed more light on the importance of each step. For example, \n\nWithout freezing, why also such a big drop on new classes as well?\n\nIt is argued earlier in the paper (3.4) that model freezing “can roughly learn the representations for the potential future class objects…” in the unknown class. However, in the ablation study, the model performs really well even without the unknown classes. In fact, the performance is still higher than most of the SOTA methods. What is the underlying reason here? Does the composition of data have any influence here as well? \n \nAny chance we can show the ablation study without PL? It is helpful to understand how important it is to have PL even if the model is frozen. \n\nIt would be interesting to see the impact of each step with respect to the complexity of the problem, i.e. reduce the initial number of classes. For example, if we increase the complexity of the problem, is there a certain point that model freezing starts to hurt the performance?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative societal impact as far I know.\n\nThe limitation of this work has not been fully discussed in the paper. Specifically, the overall method relies on the success of the saliency detector to label the unknown classes. If a new class is not able to be picked up by saliency detectors, or even the new class is simply not of “object-like” but rather “area-like”, I’d expect the proposed method to have a hard time to accommodate the new classes, especially with model freezing. \n\nAdditionally, the benefit of model freezing has a positive effect with the underlying assumption that the initial data are enough for the model to learn good features, which may not be valid in other setup of datasets. Many things can go south here, e.g. new classes are not included in the previous unknown classes, learnt feature space bounded by the initial dataset, etc.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hrs"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes an effective method for class-incremental semantic segmentation.
The different components integrated to achieve this goal have limited novelty per se,
but are combined nicely and result in a system that works well, particularly under the non-disjoint setting.
Authors are encouraged to include a discussion on limitations in the final version of the paper.
Decision: Accept (Poster)