SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning
Year: 2021

Abstract:
This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.

heading: 1 Introduction
text: Class incremental learning (CIL) problem, in which a learner should incrementally learn newly arriving class objects while not “catastrophically” forgetting the past learned classes, is one of the fundamental, yet still open, problems in machine learning. After the seminal work, [27], most of the recent neural network-based CIL research has focused on the classification setting, and various approaches have been proposed to address the main challenge of the problem, the so-called plasticitystability dilemma, e.g., [20, 30, 12, 1, 2, 8], to name a few.
The CIL framework has been extended to more complex semantic segmentation tasks [24, 3, 7], motivated by the practical need in various applications such as autonomous driving. One of the key additional difficulties of the class-incremental semantic segmentation (CISS) problem lies in the semantic drift of the background class present in the incrementally arriving training data. Namely, the label ‘background (BG)” is assigned to all the pixels not included in the current class object region. The “BG” pixels belong to three categories: future object classes that the model does not yet observes, past object classes that are already learned, and the true background.
∗Equal contribution.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A few recent works attempted to address the above semantic drift issue by leveraging and modifying the knowledge-distillation (KD) [20] technique, popular for CIL in standard image classification. Namely, the initiative study [24] modified the KD for CISS straightforwardly, and [3] proposed a strategy to incorporate the BG class probability in computing the cross-entropy and distillation losses. Furthermore, [7], the current state-of-the-art, utilized the pseudo-labeling of the BG pixels of the current task data with the model of the previous task using the cross-entropy loss, and it applies the multi-scale feature distillation scheme adopted from [8]. However, we argue that above works only partially addressed the BG label issue. That is, [3] naively added the class probabilities to modify the cross-entropy and distillation losses making it hard to have fine-grained learning of prediction probability for each class, each pixel. Moreover, [7] could only handle the BG class pixels to the past classes via pseudo-labeling and lacked any mechanism for handling the future class case, one possible option for the the BG class. Consequently, their CIL segmentation performance, measured by mean Intersection-over-Union (mIoU), has been significantly lower than the upper-bound, the case of joint-training with all the labels.
This paper first identifies that the multi-label prediction of semantic segmentation is another critical challenge of CISS and proposes SSUL-M (Semantic Segmentation with Unknown Label with Memory) to address the challenge. Specifically, Our contributions are summarized as follows. First, we introduce an additional “Unknown” class label assigned to the objects in the background, detected by the off-the-shelf saliency-map detector. We let the base feature extractor distinguish the representations of the potential future class objects and the actual background region by augmenting the BG label with this additional class. Second, we adopt the pseudo-labeling strategy as in [7] and further augment the BG & Unknown class labels with the past class labels, but with two essential differences in concrete learning strategies to stabilize the classification scores and improve the precision of the prediction. One is using the separate sigmoid, instead of the softmax, output classifier for each class so that the model can learn the logit score in an absolute sense per class. The other is freezing the base feature extractor and the classifiers for past classes after initial learning to strictly maintain the past classes’ knowledge. Third, we utilize an exemplar memory to store a tiny portion of training data, including past classes, as anchors and further improve the mIoU. Note that using the exemplar memory is a standard practice for CIL in classification but has been overlooked in CISS. Moreover, we show that the memory helps improve the mIoU for the current classes, in contrast to a common belief that it is a tool to prevent forgetting of past classes.
By integrating the above contributions, SSUL-M achieved the state-of-the-art performance on popular benchmarks with a significantly large margin over the recent baselines [3, 7], particularly when the number of incremental tasks gets larger. Furthermore, we conduct extensive ablation studies and both quantitative and qualitative analyses to convincingly highlight the strength of our method.

heading: 2 Related Work
text: Class Incremental Learning (CIL) CIL [28, 25] considers the setting in which new class objects arrive incrementally and the model needs to classify all the classes without storing all the past class data. It is well known that neural network-based CIL suffers from catastrophic forgetting [23], caused by the score bias toward the new classes due to the training data imbalance. Most CIL studies have focused on the classification tasks, and the exemplar-memory based methods combined with KD [30, 12, 1, 2, 8] achieved the state-of-the-art performance by re-balancing the biased predictions of the classifier.
Class Incremental Semantic Segmentation (CISS) Contrary to the current trend of CIL for classification, CIL for semantic segmentation have only focused on the setting without utilizing the exemplar-memory [24, 3, 7]. [3] first addressed the semantic drift of BG label, and [7] proposed to use pseudo-labels to augment the BG label, all without exemplar-memory. To our knowledge, we firstly use exemplar-memory and potential future classes in BG label to solve CISS.
Saliency Map Detection Salient object detection is a fundamental computer vision task that identifies the most visually distinctive objects in an image. We use the off-the-shelf saliency-map detector to define the Unknown class in the BG label. The early salient object detection method, DRFI [13], conducts a multi-level segmentation with a random forest regressor to detect a salient object. Recent deep neural network based method exploits rich multi-scale feature maps with short
connections [11] and pooling-based modules [21]. The saliency information has widely been utilized on various tasks. For example, the weakly-supervised semantic segmentation approaches [29, 31, 14] generate pseudo-labels filtering out the background regions using saliency map, and recent data augmentations [17, 16] also exploit the saliency information to find the optimal mixing of mask.

heading: 3 Proposed Method

heading: 3.1 Notations and Problem Setting
text: We consider exactly the same setting as considered in [3, 7]. In CISS, the learning happens with t = 1, . . . , T incremental tasks. For each incremental state t, we observe a training dataset Dt that consists of pairs (xt,yt), in which xt ∈ X denotes an input image of size N , and yt ∈ YNt denotes the corresponding ground-truth (GT) pixel labels. The label space Yt = {cb} ∪ Ct consists of the current classes in task t, Ct, and the dummy background class cb, that is assigned to all pixels that do not belong to Ct. Thus, the cb label can be assigned to the objects with the past classes C1:t−1, the objects with the future classes Ct+1:T , or the true background pixels. After learning task t, the semantic segmentation model f tθ is required to predict whether a pixel belongs to all the classes learned so far, C1:t = C1:t−1 ∪ Ct, or the true background. As in other work, we assume the classes in each Ct are disjoint. Typically, the model is defined to be a mapping f tθ : X → RN×|Y
t|, in which Yt = Cd ∪ C1:t with Cd containing dummy labels. All previous work [3, 7] simply set Cd = {cb}, but in our work, we also add the separate “Unknown” class label, cu, to Cd, hence, set Cd = {cb, cu}. Later, we show defining this additional label cu in our model output plays a critical role in improving the learning capability for future classes. The architecture of f tθ is typically a fully-convolutional network, which consists of a convolutional feature extractor, htψ, followed by the final 1× 1 classifier filters, {φtc}c∈Yt , one for each output class in Yt.
The learning of f tθ is done in conjunction with the previous model f t−1 θ : X → RN×|Y t−1| to prevent forgetting during incrementally updating the model. Determining which output classifier (e.g., softmax or sigmoid) to use for each pixel as well as how to transfer the knowledge of f t−1θ to f t θ (e.g., knowledge distillation or model freezing) are design choices, and we elaborate our choices more in details in the subsequent sections. Furthermore, we denoteM as the exemplar memory, which can store a small number of samples from past training data, D1:t−1, and use it for learning f tθ. Once the learning of f tθ is done, the prediction for pixel i of an input image x is obtained by
ŷti = argmax c∈Yt f tθ,ic(x),
and the performance is measured by the mean intersection-over-union (mIoU) metric for the classes in Yt. (Only in the evaluation phase, we merge cb and cu for computing mIoU of the BG class.)

heading: 3.2 Two Additional Challenges of CISS
text: In addition to the typical reason for causing catastrophic forgetting in CIL for classification, i.e., the prediction bias due to the data imbalance, we note there are two additional unique challenges in CISS.
The first challenge, as mentioned in the Introduction and in previous work [3, 7], comes from the semantic drift of the background (BG) label. Namely, the ground-truth label associated with the pixels of the object can change depending on the incremental state. For example, when a pixel is labeled as “BG” at state t, it is possible that the corresponding pixel would be labeled as “ sofa” at state t− 1 (i.e., past class) or labeled as “dog” at state t+ 1 (i.e., future class), depending on the true object to which the pixel belongs. Therefore, naively learning with the “BG” target label for the pixel at state t could cause either severe forgetting of the past class (i.e., hurting stability) or interfering the learning of the future class (i.e., hurting plasticity).
The second challenge stems from the fact that semantic segmentation is a multi-label prediction problem. Namely, for a given image, the segmentation model needs to output a set of classes, in contrast to the classification model which outputs only a single class. Therefore, the precision of the prediction for each pixel becomes important in addition to the recall as is reflected in the mIoU performance metric; i.e., not only predicting a correct class for a pixel is important, but also not predicting a wrong class is important for the overall metric. That is, if the prediction for every pixel
gets biased toward the current classes in Ct, the mIoU’s for the past classes in C1:t−1 as well as the current classes would significantly drop jointly. This point is exactly why even the mIoU’s for the newly learned classes are very low in [3, 7]. Note this is in a stark difference with the classification, in which the accuracy of the current classes would remain high even with the severe bias and forgetting, since it is a single-label prediction problem.
To address above unique challenges of CISS, we devise our SSUL-M by carefully combining several ideas, of which overall procedure is outlined in Figure 1. We now elaborate on our three main contributions in details: 1) label augmentation for BG class with Unknown class and pseudo-labels, 2) stable score learning with model freeze and sigmoid output, and 3) usage of tiny exemplar memory with class-balanced sampling.

heading: 3.3 Contribution 1: Label Augmentation for BG Class
text: Here, we describe how we generate an augmented target label ỹt ∈ (Yt)N = {cb, cu, C1:t}N given a training sample (xt,yt) in Dt. Recall that yt ∈ (Yt)N = {cb, Ct}N . Unknown class modeling In order to handle the case in which the potential future class objects are labeled as BG, we propose to use a novel unknown class label, cu, that is defined to designate any foreground objects that are not the current classes in Ct. Specifically, as depicted in the top part of Figure 1, we first apply an off-the-shelf salient object detector [11] S : X → {0, 1}N to the input image xt, which assigns 1 to the pixels if they are salient (i.e., part of a foreground object) and 0 otherwise. Then, we assign cu to the pixels that are BG-labeled but salient; namely, we set ỹt,i = cu if (yt,i = cb)∧ (S(xt)i = 1) for pixel i. In our experiments, we show this Unknown class label plays a critical role so that the base feature extractor can distinguish the representations of the potential future class objects and the true background, even before observing the class labels.
Pseudo-labeling Once augmenting with cu is done, we further augment with pseudo-labels generated from the previous model f t−1θ to maintain the knowledge of the past classes, similarly as in [7] and as shown in the middle part of Figure 1. Namely, we denote ŷt−1,i = argmaxc∈Yt−1 f t−1 θ,ic (xt) as the prediction of f t−1θ for the i-th pixel and set ỹt,i = ŷt−1,i if
(yt,i = cb) ∧ (ŷt−1,i ∈ C1:t−1) ∧ (µi > τ),
in which µi = maxc∈C1:t−1 σ(f t−1 θ,ic (xt)) stands for the confidence of prediction for ŷt−1,i ∈ C1:t−1 where σ(·) is the sigmoid function, and τ is a threshold (set to 0.7). In words, we assign the pseudolabels (i.e., the predictions from the previous model) to the pixels that are BG-labeled only when the predictions are made to be the past object classes (excluding cb and cu) with enough confidence.
In summary, at the incremental state t, the augmented target label for the i-th pixel becomes
ỹt,i =  yt,i if yt,i ∈ Ct ŷt−1,i if (yt,i = cb) ∧ (ŷt−1,i ∈ C1:t−1) ∧ (µi > τ) cu if (yt,i = cb) ∧ (S(xt)i = 1) ∧ {(ŷt−1,i ∈ Cd) ∨ (µi ≤ τ)} cb else,
(1)
in which the pseudo-label is generated only for the incremental state t ≥ 2. Figure 1 shows a concrete example of augmentating yt to ỹt, in which Ct−1 = {sofa} and Ct = {person}.

heading: 3.4 Contribution 2: Stable Score Learning
text: We argue that simply using ỹt as a target and training f tθ with softmax output layer would make the output scores too unstable as the incremental learning continues. The reason is because the augmented labels are noisy and the softmax computes the prediction probability in a relative way. As shown in our experiments, such instability significantly hurts the precision of the multi-label prediction. To that end, we propose the following three strategies for the stable learning of output scores.
Model freezing Instead of updating the full model with ỹt at every state t, we freeze the convolutional feature extractor, hψ , after initial learning (t = 1) as well as the classifiers for the past classes, {φc}c∈C1:t−1 , and only learn the 1 × 1 filters φtcb , φ t cu and {φ t c}c∈Ct . Such strict model freezing certainly can prevent forgetting, but is counter-intuitive from the plasticity viewpoint. However, thanks
to the unknown class cu, it turns out the feature extractor hψ can roughly learn the representations for the potential future class objects present with BG label in D1. Thus, it becomes sufficient to learn the decision boundaries for cb, cu, and Ct on those representations at state t. Sigmoid with binary cross entropy loss For learning φtcb , φ t cu and {φ t c}c∈Ct , the choice of output layer and loss function becomes important to make the output score stable. A common choice is the softmax with cross-entropy loss, however, since the target labels for cb and cu in ỹt are noisy, we observe the relative scoring of softmax could cause significant forgetting of past classes C1:t−1. To see this, let sic = [f tθ(xt)]ic be the score for class c at pixel i. When the softmax with cross-entropy (CE) is used, the gradient of the loss at sic becomes ∂LtCE(θ)/∂sic = pic − 1{c = ỹt,i}, in which pic = exp(sic)/( ∑ c′ exp(sic′)). The issue occurs when ỹt,i = cb or cu, while the true label for the pixel should be the past classes in C1:t−1. (Thus, the pseudo-label ŷt−1,i missed the pixel, which could often happen.) Then, the gradient descent learning will force sicb or sicu to become much higher than {sic}c∈C1:t−1 , the scores obtained from the frozen classifiers for past classes. Thus, at test time, for a similarly confusing pixel, the model would tend to predict as cb or cu, hence, causing the forgetting of past classes even though their classifiers are fixed.
Therefore, we instead use the sigmoid output with binary cross-entropy (BCE) loss independently for each class. In that case, the gradient of the loss at sic becomes ∂LtBCE(θ)/∂sic = σ(sic)−1{c = ỹt,i}, hence, even for above noisy target label case, the scores sicb or sicu will only increase to a certain absolute level regardless of other class score values, {sic}c∈C1:t−1 . Thus, at test time, a similar pixel still may be predicted as a past class in Ct−1 since the frozen past classifiers would still output a considerable score — this subtle difference between the loss functions is illustrated in Figure 2.
Weight transfer from unknown class classifier Finally, for learning {φtc}c∈Ct , we initialize all the filters with φt−1cu , the classifier learned for the unknown class cu at the previous state. The reasoning is that the classes in Ct would have been labeled as cu (as the potential future classes) in state t− 1, hence, such weight transfer from φt−1cu would yield stable and fast learning of {φ t c}c∈Ct .

heading: 3.5 Contribution 3: Tiny Exemplar Memory
text: Using exemplar-memory to store a small portion of past training data for CIL is backed with a theoretical finding [19] as well as strong empirical results [27, 30, 2, 1, 4] for classification.
Algorithm 1: Pseudo code of SSUL-M 1: Require fθ,M, T , nepochs, batch size K 2: fθ ← Initialize hψ , φcb , φcu 3: InitializeM 4: for t← 1, ..., T do 5: f tθ ← Initialize {φtc}c∈Ct 6: if t 6= 1 then 7: Freeze hψ, {φtc}c∈C1:t−1 8: {φtc}c∈Ct ← Weight Transfer(φt−1cu ) 9: end if 10: for n← 1, ..., nepochs do 11: for Bn
K/2∼ Dt do 12: BM
K/2∼ M 13: B̃n, B̃M ← Label.Aug.(Bn, BM) 14: θ ←SGD(B̃n ∪ B̃M,LtBCE(θ)) 15: end for 16: end for 17: UpdateM 18: end for 19: return f tθ
Hence, we propose to use it for CISS as well with a tailored class-balanced sampling strategy. The main rationale of using the memory is to make sure to include at least one sample with correct GT label per each class in C1:t−1 in the training set for state t.
Namely, even though the pseudo-label ŷt−1 can provide labels for the classes in C1:t−1, it is also possible that the given image xt would never contain object cues for C1:t−1. In such a case, even with model freezing and stable score learning, when the confidence for the new class is learned to be high for a pixel (potentially for an old class), the prediction for the pixel could get biased toward the new class, causing the forgetting of the old class. Therefore, by denoting M = |M|, after learning incremental state t−1, we sample M/|C1:t−1| data points from Dt−1, and store them in the memory M by removing equal number of samples per each class in C1:t−2 fromM. In this way,M always contains at least one sample from each class in C1:t−1, and we show in the experiments that this classbalanced strategy is more helpful than random sampling [4] as is done for CIL for classification.

heading: 3.6 Summary
text: We summarize our SSUL-M algorithm in Algorithm 1, in which our contributions described in Section 3.3∼ Section 3.5 are denoted in the typewriter font. Namely, Label.Aug. stands for generating ỹt as in (1) for the selected batches Bn and BM, “Freeze, LtBCE, and Weight Transfer” denote the methods for the stable score learning described in Section 3.4, and UpdateM denote the exemplarmemory maintenance with the class balanced sampling as mentioned in Section 3.5. Note we are constructing a mini-batch by sampling equal amount of data from Dt andM, hence, the samples in M act as anchor points to improve the precision of the predictions.

heading: 4 Experiments

heading: 4.1 Experimental Setting
text: Dataset We followed the experimental setting of [3] and evaluated our method using Pascal-VOC 2012 [9] and ADE20K [32] datasets. Originally, [3] set two experimental protocols, disjoint and overlapped, but we believe the latter is more realistic and challenging. Therefore, we evaluated on the overlapped setup only. Pascal VOC 2012 contains 20 foreground object classes and one background class, and ADE20K consists of 150 classes of both stuff and objects. We consider several incremental learning scenarios for each dataset, from the scenarios considered by the other baselines to newly proposed challenging scenarios with larger number of incremental states. A more detailed description on the datasets is introduced in the Supplementary Material (S.M.).
Evaluation Metrics We use the mean Intersection-over-Union (mIoU) as our evaluation metric, which computes the IoU for each class then computes the average over the classes. The IoU is defined as IoU = true-positive/(true-positive + false-positive + false-negative).
Implementation Details For all experiments, following other works [7, 3], we use a Deeplabv3 segmentation network [5] with a ResNet-101 [10] backbone pre-trained on ImageNet [6]. We optimize the network using SGD with an initial learning rate of 10−2 and a momentum value of 0.9 for all CISS steps. Also, we set the learning rate schedule, data augmentation, and output stride of 16 following [5] for all experiments. For each incremental state t, we train the network for 50 epochs for Pascal VOC with a batch-size of 32 and 60 epochs for ADE20K with a batch-size of 24. We tune the hyperparameters using 20% of the training set as a validation set and report the final results on the standard test set. For the exemplar memory, we utilized memory with a fixed size of |M| = 100 for Pascal VOC and |M| = 300 for ADE20K. To highlight the effect of the exemplar-memory, we report the results of the two versions of our method — SSUL (without memory) and SSUL-M (with memory). For the saliency-map detector to generate the Unknown class label, we employed DSS [11] pretrained on MSRA-B dataset [22], which contains 5,000 labels for salient objects. The experiments were implemented in PyTorch [26] 1.7 with CUDA 10.1 and CuDNN 7 using two NVIDIA V100 GPUs, and all experiments were conducted with NSML [15] framework. More information on the experimental details are in the S.M.
Baselines For a representative of the general regularization-based continual learning method, we select EWC [18] and LWC [20]-MC and applied them to each experimental setup of CISS. For CISS specific baselines, we compared with ILT [24], MiB [3] and PLOP [7], and the Joint Training (Joint) result is also given as an upper bound. Note that PLOP [7] is the current state-of-the-art. We reproduced the results of all baselines using the official code provided by the authors of [7].

heading: 4.2 Experimental results on benchmark dataset
text: Pascal VOC 2012 Following [7, 3], we evaluate our method on four different scenarios, i.e., {10-1, 15-1, 15-5, and 19-1} as well as a more challenging scenario, {5-3}. The numbers in each scenario denote the number of classes to be trained for each state. For example, VOC 5-3 means learning 5 classes at the base task (t = 1), and then incrementally learning 3 classes five times (hence, T = 6).
In Table 1, we observe our SSUL consistently outperforms the baselines with huge margin in all scenarios, even without using the exemplar-memory. Furthermore, the gap widens particularly for more challenging scenarios, namely, for the cases in which the base task has fewer classes and the number of tasks is larger. Note although MiB [3] and PLOP [7] show robustness for simple 2 tasks scenarios (19-1 and 15-5), their performance are rapidly dropped in more challenging and practical
scenarios (10-1, 5-3, 15-1). More specifically, Figure 3(a) shows the mIoU evolution for 15-1 scenario at each incremental step, and the baselines suffers from a significant drop of mIoU, as the new classes are incrementally learned. In contrast, SSUL improves mIoUs for both base (0-15) and newly learned (16-20) classes significantly, showing much slower drop of mIoU. These results show that as long as the label augmentation of BG class with Unknown class and pseudo-labels is properly done, our stable score learning is very effective for CISS. Particularly, we observe that model freeze, which is believed to be not effective in CIL, is much more effective than the KD used in other baselines.
Furthermore, we observe our SSUL-M, which uses exemplar-memory, further strengthens SSUL significantly, particularly for the newly learned classes (i.e., for t ≥ 2). For example, in VOC 15-1, by storing only 5 ∼ 6 images per class inM, the mIoU for classes 16-20 improved about 12%. This confirms our intuition that the samples in the memory act as “anchors” to improve the precision of the predictions, hence, prevent forgetting. We can also clearly observe this improvement in Figure 4. As an additional experiment, we also conducted the experiment on the case of further reducing the number of base classes, such as 5-1 and 2-1, and we again observe that our SSUL and SSUL-M surpass other baselines. A detailed result on this additional experiment can be found also in the S.M.
To verify the robustness of each method on various class orderings, we experimented on the 20 difference orderings on VOC 15-1 scenario, as proposed in [7]. Figure 3(b) plots the mIoU distributions for different methods, and we clearly observe both SSUL and SULL-M not only achieve higher mIoUs, but also have smaller variations compared to the baselines. In addition, we note SULL achieves ×1.5 faster training time compared to PLOP due to network freezing.
Qualitative analysis In Figure 4, we visualized the qualitative results for four images from VOC 15-1 task. We observe that PLOP partly maintains knowledge learned from the base task (bird and person), however, it becomes fatal in forgetting the classes learned in the intermediate steps, such as plant and sheep. In addition, PLOP frequently produces many false-positive predictions, lowering the mIoU of several classes. (E.g., see bird in Step 5.) On the other hand, we observe SSUL and SSUL-M both maintains the previously learned classes with high stability and effectively learns new classes. For example, the sheep class is accurately learned at Step 3 and not forgotten afterwards. Moreover, we observe SULL-M achieved further improvement of both plasticity and stability over SULL, especially by reducing the false-positive predictions.
ADE20K Unlike VOC 2012, ADE20K is densely labeled for both stuff and thing with 150 classes. It means that the class definition in ADE20K is clearer therefore, it naturally make reduce the concern about the semantic drift of BG label. To make efficient use of this prior knowledge of dataset, we
consider unlabeled pixels as unknown class without using the saliency detector and enlarge the size of memory to |M| = 300. We evaluated our method in four different scenarios, i.e., {100-5, 100-10, 50-50, 100-50} as in [7, 3]. Table 2 again shows that SSUL achieves superior performance in the more challenging tasks (100-5, 100-10) than other methods. We believe that, this result demonstrates SSUL is also quite effective in CISS for a densely labeled dataset, without any extra saliency detector. The qualitative analysis on ADE20K is provided in the S.M.

heading: 4.3 Ablation Study
text: Ablation study on proposed components of SSUL Here, we analyze the effect of each proposed component of SSUL on VOC 15-1 scenario. Table 3 compares the mIoU of each ablation case, without memory, and the first low shows the result of SSUL with full components. Firstly, when the Unknown class is removed, we clearly observe that the mIoU’s of both the first task (0-15) and subsequent classes (16-20) decrease. Hence, this clearly demonstrates the advantage of using the Unknown class label, in terms of increasing both the plasticity and stability. Secondly, note that model freezing has a significant impact on the performance of CISS. Specifically, it not only prevents catastrophic forgetting on the first task, but also plays a critical role to learn new classes well. Finally, we observe when Softmax+CE instead of Sigmoid+BCE is used (the last ablation), the forgetting of the first task (0-15) drops more significantly despite model freezing. We believe this confirms our intuition in Section 3.4 on why Sigmoid+BCE can lead to more stable score learning.
Saliency-map detector and weight transfer The first two columns in Table 4 shows the ablation study on saliency-map detector for Unknown class and weight transfer. We compare the result of a default neural network-based saliency map extractor, DSS [11] with ground-truth and a random forest-based DRFI [13]. We observe the differences of the mIoU’s among them are quite small, therefore, we believe that the quality of saliency map is not a significant factor of our method. The ablation study on weight transfer demonstrates weights for new classes initialized with φt−1cu is most effective to learn it. This result may at first look counter-intuitive since learning the final linear layer φtc should not be sensitive on the initialization. However, given the noisy pseudo-label for learning, a quicker convergence of φtc using the warm-start weight φ t−1 cu would effectively prevent the forgetting of past class, which could be caused by fitting the noisy pseudo-label for many epochs. More details on the effect of weight transfer is proposed in S.M.
Memory size and sampling rule The right column row of Table 4 shows the result of the mIoU on two sampling rules. We observe that, compared to random sampling, our proposed class-balanced sampling achieves better mIoU, particularly for the newly learned classes. We believe the reason is class-balanced sampling ensures at least one sample per class, therefore, it prevents the forgetting of minority classes than random sampling, which can miss certain classes inM. Finally, Figure 3(c) shows the dependency of SSUL-M on the memory size. It illustrates that tiny exemplar-memory for CISS significantly helps to increase the mIoU for the newly learned classes (16-20) than the base classes (0-15). Moreover, we observe that after sufficiently large |M| the mIoU performance becomes robust.

heading: 5 Concluding Remarks and Limitation
text: We proposed a new class-incremental learning method SSUL-M (Semantic Segmentation with Unknown Label with Memory) for semantic segmentation. In order to address two additional challenges of CISS, we made three main contributions — label augmentation with Unknown class labels, stable score learning, and tiny exemplar memory. They all were convincingly shown to be very effective in various CISS scenarios and our SSUL-M significantly outperformed baselines.
While promising, we admit our work also has some limitations as follows. First, our unknown class modeling may not be satisfactory for some “stuff” segmentation tasks since saliency-maps are mainly targeted for “things” (or objects). Second, our model freezing may harm plasticity and cause the model suffer from learning new classes especially when they are not captured by the unknown class label during the base task. Together with our insights from the analyses of SULL-M, we believe attempting to address above limitations would lead to a fruitful future research directions for the CISS problem.

heading: Acknowledgement
text: This work was done while Sungmin Cha did a research internship at NAVER AI Lab. This work was also supported in part by the New Faculty Startup Fund from Seoul National University, NRF Mid-Career Research Program [NRF-2021R1A2C2007884], IITP grant [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)] funded by the Korea government(MSIT), and and SNU-NAVER Hyperscale AI Center. The authors thank NAVER Smart Machine Learning (NSML) team for the GPU support. Taesup Moon also thanks the support from Automation and Systems Research Institute (ASRI) at Seoul National University.

Review summaries:
The paper presents a class-incremental learning strategy for semantic segmentation. The main contributions of this work include two aspects: first, it introduces an external saliency map predictor to better model the thing classes vs the stuff background; second, the method adds a class-balanced memory to overcome the forgetting in CIL. The proposed method is evaluated on two benchmarks with the overlapped setup and achieves better performance than the prior approaches.  

This paper proposes a method, Semantic Segmentation with Unknown Label with Memory (SSUL-M), to solve a class-incremental semantic segmentation (CISS) problem. The authors make three main contributions. First, they introduce an additional “Unknown” class label to distinguish the potential future class and the true background. Second, they adopt the pseudo-labeling strategy and freeze backbone network and past classiﬁers with binary cross-entropy loss to tackle catastrophic forgetting. Third, an exemplar memory is utilized to improve both plasticity and stability. Extensive experiments and ablation studies are conducted to justify these proposed components and contributions. Results show that SSUL-M signiﬁcantly outperformed baselines.

This paper proposes novel a class-incremental learning method, named semantic segmentation with unknown label with memory. The proposed method addresses two additional challenges in class-incremental learning and proposes three contributions: label augmentation, stable score learning, and tiny exemplar memory.  The proposed method achieves impressive performance compared with other methods.