SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

1 Introduction . Class incremental learning ( CIL ) problem , in which a learner should incrementally learn newly arriving class objects while not “ catastrophically ” forgetting the past learned classes , is one of the fundamental , yet still open , problems in machine learning . After the seminal work , [ 27 ] , most of the recent neural network-based CIL research has focused on the classification setting , and various approaches have been proposed to address the main challenge of the problem , the so-called plasticitystability dilemma , e.g. , [ 20 , 30 , 12 , 1 , 2 , 8 ] , to name a few . The CIL framework has been extended to more complex semantic segmentation tasks [ 24 , 3 , 7 ] , motivated by the practical need in various applications such as autonomous driving . One of the key additional difficulties of the class-incremental semantic segmentation ( CISS ) problem lies in the semantic drift of the background class present in the incrementally arriving training data . Namely , the label ‘ background ( BG ) ” is assigned to all the pixels not included in the current class object region . The “ BG ” pixels belong to three categories : future object classes that the model does not yet observes , past object classes that are already learned , and the true background . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A few recent works attempted to address the above semantic drift issue by leveraging and modifying the knowledge-distillation ( KD ) [ 20 ] technique , popular for CIL in standard image classification . Namely , the initiative study [ 24 ] modified the KD for CISS straightforwardly , and [ 3 ] proposed a strategy to incorporate the BG class probability in computing the cross-entropy and distillation losses . Furthermore , [ 7 ] , the current state-of-the-art , utilized the pseudo-labeling of the BG pixels of the current task data with the model of the previous task using the cross-entropy loss , and it applies the multi-scale feature distillation scheme adopted from [ 8 ] . However , we argue that above works only partially addressed the BG label issue . That is , [ 3 ] naively added the class probabilities to modify the cross-entropy and distillation losses making it hard to have fine-grained learning of prediction probability for each class , each pixel . Moreover , [ 7 ] could only handle the BG class pixels to the past classes via pseudo-labeling and lacked any mechanism for handling the future class case , one possible option for the the BG class . Consequently , their CIL segmentation performance , measured by mean Intersection-over-Union ( mIoU ) , has been significantly lower than the upper-bound , the case of joint-training with all the labels . This paper first identifies that the multi-label prediction of semantic segmentation is another critical challenge of CISS and proposes SSUL-M ( Semantic Segmentation with Unknown Label with Memory ) to address the challenge . Specifically , Our contributions are summarized as follows . First , we introduce an additional “ Unknown ” class label assigned to the objects in the background , detected by the off-the-shelf saliency-map detector . We let the base feature extractor distinguish the representations of the potential future class objects and the actual background region by augmenting the BG label with this additional class . Second , we adopt the pseudo-labeling strategy as in [ 7 ] and further augment the BG & Unknown class labels with the past class labels , but with two essential differences in concrete learning strategies to stabilize the classification scores and improve the precision of the prediction . One is using the separate sigmoid , instead of the softmax , output classifier for each class so that the model can learn the logit score in an absolute sense per class . The other is freezing the base feature extractor and the classifiers for past classes after initial learning to strictly maintain the past classes ’ knowledge . Third , we utilize an exemplar memory to store a tiny portion of training data , including past classes , as anchors and further improve the mIoU . Note that using the exemplar memory is a standard practice for CIL in classification but has been overlooked in CISS . Moreover , we show that the memory helps improve the mIoU for the current classes , in contrast to a common belief that it is a tool to prevent forgetting of past classes . By integrating the above contributions , SSUL-M achieved the state-of-the-art performance on popular benchmarks with a significantly large margin over the recent baselines [ 3 , 7 ] , particularly when the number of incremental tasks gets larger . Furthermore , we conduct extensive ablation studies and both quantitative and qualitative analyses to convincingly highlight the strength of our method . 2 Related Work . Class Incremental Learning ( CIL ) CIL [ 28 , 25 ] considers the setting in which new class objects arrive incrementally and the model needs to classify all the classes without storing all the past class data . It is well known that neural network-based CIL suffers from catastrophic forgetting [ 23 ] , caused by the score bias toward the new classes due to the training data imbalance . Most CIL studies have focused on the classification tasks , and the exemplar-memory based methods combined with KD [ 30 , 12 , 1 , 2 , 8 ] achieved the state-of-the-art performance by re-balancing the biased predictions of the classifier . Class Incremental Semantic Segmentation ( CISS ) Contrary to the current trend of CIL for classification , CIL for semantic segmentation have only focused on the setting without utilizing the exemplar-memory [ 24 , 3 , 7 ] . [ 3 ] first addressed the semantic drift of BG label , and [ 7 ] proposed to use pseudo-labels to augment the BG label , all without exemplar-memory . To our knowledge , we firstly use exemplar-memory and potential future classes in BG label to solve CISS . Saliency Map Detection Salient object detection is a fundamental computer vision task that identifies the most visually distinctive objects in an image . We use the off-the-shelf saliency-map detector to define the Unknown class in the BG label . The early salient object detection method , DRFI [ 13 ] , conducts a multi-level segmentation with a random forest regressor to detect a salient object . Recent deep neural network based method exploits rich multi-scale feature maps with short connections [ 11 ] and pooling-based modules [ 21 ] . The saliency information has widely been utilized on various tasks . For example , the weakly-supervised semantic segmentation approaches [ 29 , 31 , 14 ] generate pseudo-labels filtering out the background regions using saliency map , and recent data augmentations [ 17 , 16 ] also exploit the saliency information to find the optimal mixing of mask . 3 Proposed Method . 3.1 Notations and Problem Setting . We consider exactly the same setting as considered in [ 3 , 7 ] . In CISS , the learning happens with t = 1 , . . . , T incremental tasks . For each incremental state t , we observe a training dataset Dt that consists of pairs ( xt , yt ) , in which xt ∈ X denotes an input image of size N , and yt ∈ YNt denotes the corresponding ground-truth ( GT ) pixel labels . The label space Yt = { cb } ∪ Ct consists of the current classes in task t , Ct , and the dummy background class cb , that is assigned to all pixels that do not belong to Ct . Thus , the cb label can be assigned to the objects with the past classes C1 : t−1 , the objects with the future classes Ct+1 : T , or the true background pixels . After learning task t , the semantic segmentation model f tθ is required to predict whether a pixel belongs to all the classes learned so far , C1 : t = C1 : t−1 ∪ Ct , or the true background . As in other work , we assume the classes in each Ct are disjoint . Typically , the model is defined to be a mapping f tθ : X → RN×|Y t| , in which Yt = Cd ∪ C1 : t with Cd containing dummy labels . All previous work [ 3 , 7 ] simply set Cd = { cb } , but in our work , we also add the separate “ Unknown ” class label , cu , to Cd , hence , set Cd = { cb , cu } . Later , we show defining this additional label cu in our model output plays a critical role in improving the learning capability for future classes . The architecture of f tθ is typically a fully-convolutional network , which consists of a convolutional feature extractor , htψ , followed by the final 1× 1 classifier filters , { φtc } c∈Yt , one for each output class in Yt . The learning of f tθ is done in conjunction with the previous model f t−1 θ : X → RN×|Y t−1| to prevent forgetting during incrementally updating the model . Determining which output classifier ( e.g. , softmax or sigmoid ) to use for each pixel as well as how to transfer the knowledge of f t−1θ to f t θ ( e.g. , knowledge distillation or model freezing ) are design choices , and we elaborate our choices more in details in the subsequent sections . Furthermore , we denoteM as the exemplar memory , which can store a small number of samples from past training data , D1 : t−1 , and use it for learning f tθ . Once the learning of f tθ is done , the prediction for pixel i of an input image x is obtained by ŷti = argmax c∈Yt f tθ , ic ( x ) , and the performance is measured by the mean intersection-over-union ( mIoU ) metric for the classes in Yt . ( Only in the evaluation phase , we merge cb and cu for computing mIoU of the BG class . ) 3.2 Two Additional Challenges of CISS . In addition to the typical reason for causing catastrophic forgetting in CIL for classification , i.e. , the prediction bias due to the data imbalance , we note there are two additional unique challenges in CISS . The first challenge , as mentioned in the Introduction and in previous work [ 3 , 7 ] , comes from the semantic drift of the background ( BG ) label . Namely , the ground-truth label associated with the pixels of the object can change depending on the incremental state . For example , when a pixel is labeled as “ BG ” at state t , it is possible that the corresponding pixel would be labeled as “ sofa ” at state t− 1 ( i.e. , past class ) or labeled as “ dog ” at state t+ 1 ( i.e. , future class ) , depending on the true object to which the pixel belongs . Therefore , naively learning with the “ BG ” target label for the pixel at state t could cause either severe forgetting of the past class ( i.e. , hurting stability ) or interfering the learning of the future class ( i.e. , hurting plasticity ) . The second challenge stems from the fact that semantic segmentation is a multi-label prediction problem . Namely , for a given image , the segmentation model needs to output a set of classes , in contrast to the classification model which outputs only a single class . Therefore , the precision of the prediction for each pixel becomes important in addition to the recall as is reflected in the mIoU performance metric ; i.e. , not only predicting a correct class for a pixel is important , but also not predicting a wrong class is important for the overall metric . That is , if the prediction for every pixel gets biased toward the current classes in Ct , the mIoU ’ s for the past classes in C1 : t−1 as well as the current classes would significantly drop jointly . This point is exactly why even the mIoU ’ s for the newly learned classes are very low in [ 3 , 7 ] . Note this is in a stark difference with the classification , in which the accuracy of the current classes would remain high even with the severe bias and forgetting , since it is a single-label prediction problem . To address above unique challenges of CISS , we devise our SSUL-M by carefully combining several ideas , of which overall procedure is outlined in Figure 1 . We now elaborate on our three main contributions in details : 1 ) label augmentation for BG class with Unknown class and pseudo-labels , 2 ) stable score learning with model freeze and sigmoid output , and 3 ) usage of tiny exemplar memory with class-balanced sampling .

Summary: This paper proposes a method, Semantic Segmentation with Unknown Label with Memory (SSUL-M), to solve a class-incremental semantic segmentation (CISS) problem. The authors make three main contributions. First, they introduce an additional “Unknown” class label to distinguish the potential future class and the true background. Second, they adopt the pseudo-labeling strategy and freeze backbone network and past classiﬁers with binary cross-entropy loss to tackle catastrophic forgetting. Third, an exemplar memory is utilized to improve both plasticity and stability. Extensive experiments and ablation studies are conducted to justify these proposed components and contributions. Results show that SSUL-M signiﬁcantly outperformed baselines.