SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

1 Introduction . Class incremental learning ( CIL ) problem , in which a learner should incrementally learn newly arriving class objects while not “ catastrophically ” forgetting the past learned classes , is one of the fundamental , yet still open , problems in machine learning . After the seminal work , [ 27 ] , most of the recent neural network-based CIL research has focused on the classification setting , and various approaches have been proposed to address the main challenge of the problem , the so-called plasticitystability dilemma , e.g. , [ 20 , 30 , 12 , 1 , 2 , 8 ] , to name a few . The CIL framework has been extended to more complex semantic segmentation tasks [ 24 , 3 , 7 ] , motivated by the practical need in various applications such as autonomous driving . One of the key additional difficulties of the class-incremental semantic segmentation ( CISS ) problem lies in the semantic drift of the background class present in the incrementally arriving training data . Namely , the label ‘ background ( BG ) ” is assigned to all the pixels not included in the current class object region . The “ BG ” pixels belong to three categories : future object classes that the model does not yet observes , past object classes that are already learned , and the true background . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A few recent works attempted to address the above semantic drift issue by leveraging and modifying the knowledge-distillation ( KD ) [ 20 ] technique , popular for CIL in standard image classification . Namely , the initiative study [ 24 ] modified the KD for CISS straightforwardly , and [ 3 ] proposed a strategy to incorporate the BG class probability in computing the cross-entropy and distillation losses . Furthermore , [ 7 ] , the current state-of-the-art , utilized the pseudo-labeling of the BG pixels of the current task data with the model of the previous task using the cross-entropy loss , and it applies the multi-scale feature distillation scheme adopted from [ 8 ] . However , we argue that above works only partially addressed the BG label issue . That is , [ 3 ] naively added the class probabilities to modify the cross-entropy and distillation losses making it hard to have fine-grained learning of prediction probability for each class , each pixel . Moreover , [ 7 ] could only handle the BG class pixels to the past classes via pseudo-labeling and lacked any mechanism for handling the future class case , one possible option for the the BG class . Consequently , their CIL segmentation performance , measured by mean Intersection-over-Union ( mIoU ) , has been significantly lower than the upper-bound , the case of joint-training with all the labels . This paper first identifies that the multi-label prediction of semantic segmentation is another critical challenge of CISS and proposes SSUL-M ( Semantic Segmentation with Unknown Label with Memory ) to address the challenge . Specifically , Our contributions are summarized as follows . First , we introduce an additional “ Unknown ” class label assigned to the objects in the background , detected by the off-the-shelf saliency-map detector . We let the base feature extractor distinguish the representations of the potential future class objects and the actual background region by augmenting the BG label with this additional class . Second , we adopt the pseudo-labeling strategy as in [ 7 ] and further augment the BG & Unknown class labels with the past class labels , but with two essential differences in concrete learning strategies to stabilize the classification scores and improve the precision of the prediction . One is using the separate sigmoid , instead of the softmax , output classifier for each class so that the model can learn the logit score in an absolute sense per class . The other is freezing the base feature extractor and the classifiers for past classes after initial learning to strictly maintain the past classes ’ knowledge . Third , we utilize an exemplar memory to store a tiny portion of training data , including past classes , as anchors and further improve the mIoU . Note that using the exemplar memory is a standard practice for CIL in classification but has been overlooked in CISS . Moreover , we show that the memory helps improve the mIoU for the current classes , in contrast to a common belief that it is a tool to prevent forgetting of past classes . By integrating the above contributions , SSUL-M achieved the state-of-the-art performance on popular benchmarks with a significantly large margin over the recent baselines [ 3 , 7 ] , particularly when the number of incremental tasks gets larger . Furthermore , we conduct extensive ablation studies and both quantitative and qualitative analyses to convincingly highlight the strength of our method . 2 Related Work . Class Incremental Learning ( CIL ) CIL [ 28 , 25 ] considers the setting in which new class objects arrive incrementally and the model needs to classify all the classes without storing all the past class data . It is well known that neural network-based CIL suffers from catastrophic forgetting [ 23 ] , caused by the score bias toward the new classes due to the training data imbalance . Most CIL studies have focused on the classification tasks , and the exemplar-memory based methods combined with KD [ 30 , 12 , 1 , 2 , 8 ] achieved the state-of-the-art performance by re-balancing the biased predictions of the classifier . Class Incremental Semantic Segmentation ( CISS ) Contrary to the current trend of CIL for classification , CIL for semantic segmentation have only focused on the setting without utilizing the exemplar-memory [ 24 , 3 , 7 ] . [ 3 ] first addressed the semantic drift of BG label , and [ 7 ] proposed to use pseudo-labels to augment the BG label , all without exemplar-memory . To our knowledge , we firstly use exemplar-memory and potential future classes in BG label to solve CISS . Saliency Map Detection Salient object detection is a fundamental computer vision task that identifies the most visually distinctive objects in an image . We use the off-the-shelf saliency-map detector to define the Unknown class in the BG label . The early salient object detection method , DRFI [ 13 ] , conducts a multi-level segmentation with a random forest regressor to detect a salient object . Recent deep neural network based method exploits rich multi-scale feature maps with short connections [ 11 ] and pooling-based modules [ 21 ] . The saliency information has widely been utilized on various tasks . For example , the weakly-supervised semantic segmentation approaches [ 29 , 31 , 14 ] generate pseudo-labels filtering out the background regions using saliency map , and recent data augmentations [ 17 , 16 ] also exploit the saliency information to find the optimal mixing of mask . 3 Proposed Method . 3.1 Notations and Problem Setting . We consider exactly the same setting as considered in [ 3 , 7 ] . In CISS , the learning happens with t = 1 , . . . , T incremental tasks . For each incremental state t , we observe a training dataset Dt that consists of pairs ( xt , yt ) , in which xt ∈ X denotes an input image of size N , and yt ∈ YNt denotes the corresponding ground-truth ( GT ) pixel labels . The label space Yt = { cb } ∪ Ct consists of the current classes in task t , Ct , and the dummy background class cb , that is assigned to all pixels that do not belong to Ct . Thus , the cb label can be assigned to the objects with the past classes C1 : t−1 , the objects with the future classes Ct+1 : T , or the true background pixels . After learning task t , the semantic segmentation model f tθ is required to predict whether a pixel belongs to all the classes learned so far , C1 : t = C1 : t−1 ∪ Ct , or the true background . As in other work , we assume the classes in each Ct are disjoint . Typically , the model is defined to be a mapping f tθ : X → RN×|Y t| , in which Yt = Cd ∪ C1 : t with Cd containing dummy labels . All previous work [ 3 , 7 ] simply set Cd = { cb } , but in our work , we also add the separate “ Unknown ” class label , cu , to Cd , hence , set Cd = { cb , cu } . Later , we show defining this additional label cu in our model output plays a critical role in improving the learning capability for future classes . The architecture of f tθ is typically a fully-convolutional network , which consists of a convolutional feature extractor , htψ , followed by the final 1× 1 classifier filters , { φtc } c∈Yt , one for each output class in Yt . The learning of f tθ is done in conjunction with the previous model f t−1 θ : X → RN×|Y t−1| to prevent forgetting during incrementally updating the model . Determining which output classifier ( e.g. , softmax or sigmoid ) to use for each pixel as well as how to transfer the knowledge of f t−1θ to f t θ ( e.g. , knowledge distillation or model freezing ) are design choices , and we elaborate our choices more in details in the subsequent sections . Furthermore , we denoteM as the exemplar memory , which can store a small number of samples from past training data , D1 : t−1 , and use it for learning f tθ . Once the learning of f tθ is done , the prediction for pixel i of an input image x is obtained by ŷti = argmax c∈Yt f tθ , ic ( x ) , and the performance is measured by the mean intersection-over-union ( mIoU ) metric for the classes in Yt . ( Only in the evaluation phase , we merge cb and cu for computing mIoU of the BG class . ) 3.2 Two Additional Challenges of CISS . In addition to the typical reason for causing catastrophic forgetting in CIL for classification , i.e. , the prediction bias due to the data imbalance , we note there are two additional unique challenges in CISS . The first challenge , as mentioned in the Introduction and in previous work [ 3 , 7 ] , comes from the semantic drift of the background ( BG ) label . Namely , the ground-truth label associated with the pixels of the object can change depending on the incremental state . For example , when a pixel is labeled as “ BG ” at state t , it is possible that the corresponding pixel would be labeled as “ sofa ” at state t− 1 ( i.e. , past class ) or labeled as “ dog ” at state t+ 1 ( i.e. , future class ) , depending on the true object to which the pixel belongs . Therefore , naively learning with the “ BG ” target label for the pixel at state t could cause either severe forgetting of the past class ( i.e. , hurting stability ) or interfering the learning of the future class ( i.e. , hurting plasticity ) . The second challenge stems from the fact that semantic segmentation is a multi-label prediction problem . Namely , for a given image , the segmentation model needs to output a set of classes , in contrast to the classification model which outputs only a single class . Therefore , the precision of the prediction for each pixel becomes important in addition to the recall as is reflected in the mIoU performance metric ; i.e. , not only predicting a correct class for a pixel is important , but also not predicting a wrong class is important for the overall metric . That is , if the prediction for every pixel gets biased toward the current classes in Ct , the mIoU ’ s for the past classes in C1 : t−1 as well as the current classes would significantly drop jointly . This point is exactly why even the mIoU ’ s for the newly learned classes are very low in [ 3 , 7 ] . Note this is in a stark difference with the classification , in which the accuracy of the current classes would remain high even with the severe bias and forgetting , since it is a single-label prediction problem . To address above unique challenges of CISS , we devise our SSUL-M by carefully combining several ideas , of which overall procedure is outlined in Figure 1 . We now elaborate on our three main contributions in details : 1 ) label augmentation for BG class with Unknown class and pseudo-labels , 2 ) stable score learning with model freeze and sigmoid output , and 3 ) usage of tiny exemplar memory with class-balanced sampling .

Summary: The paper presents a class-incremental learning strategy for semantic segmentation. The main contributions of this work include two aspects: first, it introduces an external saliency map predictor to better model the thing classes vs the stuff background; second, the method adds a class-balanced memory to overcome the forgetting in CIL. The proposed method is evaluated on two benchmarks with the overlapped setup and achieves better performance than the prior approaches.