Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation
Authors: Zekang Zhang
Guangyu Gao
Zhiyuan Fang
Jianbo Jiao
Yunchao Wei
Venue: NeurIPS 2022 Accept (2022)
Keywords: Incremental Learning
semantic drift
catastrophic forgetting
regional objectness

Abstract: Incremental or continual learning has been extensively studied for image classification tasks to alleviate catastrophic forgetting, a phenomenon in which earlier learned knowledge is forgotten when learning new concepts. For class incremental semantic segmentation, such a phenomenon often becomes much worse due to the semantic shift of the background class, \ie, some concepts learned at previous stages are assigned to the background class at the current training stage, therefore, significantly reducing the performance of these old concepts. To address this issue, we propose a simple yet effective method in this paper, named Mining unseen Classes via Regional Objectness (MicroSeg). Our MicroSeg is based on the assumption that \emph{background regions with strong objectness possibly belong to those concepts in the historical or future stages}. Therefore, to avoid forgetting old knowledge at the current training stage, our MicroSeg first splits the given image into hundreds of segment proposals with a proposal generator. Those segment proposals with strong objectness from the background are then clustered and assigned new defined labels during the optimization. In this way, the distribution characterizes of old concepts in the feature space could be better perceived, relieving the catastrophic forgetting caused by the semantic shift of the background class accordingly.  We conduct extensive experiments on Pascal VOC and ADE20K, and competitive results well demonstrate the effectiveness of our MicroSeg. Code is available at \href{https://github.com/zkzhang98/MicroSeg}{\textcolor{orange}{\texttt{https://github.com/zkzhang98/MicroSeg}}}.

Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation
Zekang Zhang'; Guangyu Gao Zhiyuan Fang',
Jianbo Jiao",
Yunchao Wei
School of Computer Science, Beijing Institute of Technology
School of Computer Science, University of Birmingham
3 WEI Lab, Institute of Information Science, Beijing Jiaotong University
Beijing Key Laboratory of Advanced Information Science and Network zkzhang www.9999tooto...ooooom com
Abstract
Incremental or continual learning has been extensively studied for image classifica- tion tasks alleviate catastrophic forgetling, a phenomenon that earlier learned knowledge forgotten when learning new concepts. For class incremental se- mantic segmentation, such phenomenon often becomes much worse due the background shift, i.e., some concepts learned at previous stages are assigned to the background class at the current training stage, therefore, significantly redue- ing the performance of these old concepts, To address this issue, we propose a simple yet effective method in this paper, named Mining unseen Classes via Regional Objectness for Segmentation (MicroSeg). Our MicroSeg is based on the assumption that background regions with strong objectness possibly belong those concepts in the historical or fulure stuges. Therefore, avoid forgetting old knowledge at the current training stage, our MicroSeg first splits the given image into hundreds of segment proposals with proposal generator. Those segment pro- posals with strong objectness from the background are then clustered and assigned newly-defined labels during the optimization. In this way, the distribution char- acterizes of old concepts in the feature space could be better perceived, relieving the catastrophic forgetting caused by the background shift accordingly. Extensive experiments on Pascal VOC and ADE20K datasets show competitive results with ffate-of-the-eee well validating the effectiveness of the proposed MicroSeg. Code is available at https:/ //gi/ titth..... com/ htta///kzangg8//iicooom crooee....
Introduction
Deep learning based methods have made significant achievements various recognition tasks, while assuming the data distribution to be fixed or stable and the training samples are independently and identically distributed [18]. However, practical scenarios, the data always presents continuous stream without stable distribution. Under these scenarios, the old knowledge will be easily interfered with or even totally forgotten by continuously acquiring new knowledge, which generally considered to be catastrophic forgetting [3, 24, 28]. To tackle this problem, many approaches were proposed to incrementally learn new concepts over time but without forgetting old knowledge, especially for the classification task, i.e., Class-Incremental Learning (CIL) [17, 19].
Meanwhile, Class-Incremental Semantic Segmentation (CISS), aiming predict a pixel- wiie mask for an image in CIL scenarios, is crucial for applications like autonomous driving and robotics.
Work done during internship WEI Lab of Beijing Jiaotong University. Conresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

MicroSeggoons
SSUL
Proposal
Pred.
Saliency
Pred
Micrateg M SSUL-M SOLL
mloU(%%)
19-1/2 stpp 15-5( stepo) 5-36 stepo) 2-2110 stepo)
(a)
(h)
(a) Comparison of our proposed MicroSeg and the sfate-of-the-arr SSUL [5]. Saliency detection only points out the most important area of an image, leaving the rest unseen classes result in semantic shift and lead to incorrect predictions. The proposed proposal based approach attends each pixels within the sample, including both things and stuff. (b) Performance comparison of prior works when the number of incremental step increases (mloU < 45% has been dropped out).
Figure 1:
Nevertheless, besides catastrophic forgetting of old concepts in CIL, CISS involves an additional key issue of background shift in the image [4], making CISS rather challenging. Specifically, background shift means that the class of background (all other classes that are not learned at the moment) is shifting over the learning steps. As result, such background shift will confuse the model and make it more difficult to distinguish different unseen classes, which generally include the true background (dummy) class, the historical classes, and future classes.
The problems of catastrophic forgetting and background shift in CISS have been studied in recent works [4, 5, 11 1J. MiB [4] proposed distillation-based framework to address the background shift issue. PLOP [11] re-models the background labels with pscudo-labeling w.r.t. predictions of the old model for historical classes. However, these prior works did not particularly address the future classes issue that leads to background shift, while instead treating those future classes as background. These future classes will not only interfere with old knowledge due to catastrophic forgetting, but also be constantly introduced as current classes to be accurately identified. The state-of-the-an method SSUL [5] tried to address the future class issue by introducing unknown class according to saliency detection. However, as shown in Fig. 1 (a), as saliency only highlights a few regions within an image, most unseen classes still remain in the background and result in background shift. As shown in Fig. 1 (b), with the number of incremental steps increasing, the performance of the aforementioned methods has a dramatic performance drop, suggesting that the background shift issue still apparently exists, especially for the challenging setting of more learning steps. Whereas for our approach (MicroSeg-M in Fig. 1 (b)), it outperforms prior works at each step. Furthermore, if considering the incremental learner lies in a lifelong learning process, the number of unseen classes would be infinite and diverse with endless learning steps. Thus, the unseen classes in each learning step become even more non- negligiie,,,,, which interfere not only with maintaining concepts of historical classes, but also with the learning of current classes.
To address the above-mentioned issues in CISS, especially the unseen classes, we propose a simple yet effective approach for Mining unseen Classes via Regional Objectness for the Segmentation task (MicroSeg). MicroSeg first generates set of segment proposals for input samples with a proposal generator. Each proposal is binary mask pointing out segment with strong objectness in an image. This mask is also class-agnostic and can generalize even to unseen categories. With the assumption of background regions with strong objectness belong to concepts either in the historical or future steps, the proposals from the background can be confidently clustered and re-modeled as unseen classes. Following that, proposal-basee -asee mask classification [8, 9] applied to label each proposal with current classes and unseen classes. Moreover, to better cache the feature diversity in loose future classes into particular unseen classes, multiple sub-classes are explicitly defined and clustered according to contrastive loss. As shown in Fig. (a), the background shift issue gets alleviated with the discovery of unseen classes, showing the effectiveness of the proposed MicroSeg in reducing semantic ambiguity within categories.
2

---

Extensive experimental analysis on Pascal VOC2012 and ADE20K show the effectiveness and competitive performance of the proposed MicroSeg. MicroSeg consistently achieves higher perfor- mance across all benchmark datasets, on all incremental learning settings, e.g., an average of 3% improvements over the state-of-the-art method on Pascal VOC2012 and 2% on ADE20K, Particularly, MicroSeg outperforms prior works in the more challenging setting of long-term incremental scenarios, e.g., the performance gain of 6.2% % on Pascal VOC 2012.
2 Related work
Class incremental learning (CIL). Class incremental learning (CIL) is incremental learning bound to classification tasks. Due to the catastrophic forgetting, some approaches consider CIL as a trade-off' between defying forgetting and learning from new samples of network. The earlier work to deal with catastrophic forgetting is mainly about parameter isolation [2, 23, 33, 34, 35, 40], which assigns isolated models parameters for each task, while the number of parameters will always grow without upper bound with task increases. Replay-basee approaches [10, 22, 30, 31, 36] were proposed by maintaining a sampler of data for future training. Later, regularization based methods [1, 17, 19, 29, 41], such knowledge distillation [15] appears, which rely on teacher model for transferring knowledge of previous categories to current models.
Semantic segmentation. In semantic segmentation, deep models have achieved outstanding perfor- mance. FCN [21] and U-Net [32] replace fully connected layers with convolution layers to preserve the location information of pixels. The following methods [16, 25, 38, 39, 44] are mainly based on FCN and enlarge the perceptual field, making it more suitable for semantic segmentation tasks that require intensive annotation. DeepLab series methods [6, 7] contains an Atrous Spatial Pyramid Pool- ing (ASPP) for semantic segmentation using dilated convolution to capture multi-scale information. Meanwhile, MaskFormer [9] regards the semantic segmentation as a mask classification task and introduces the transformer [37] to obtain a better performance. Specifically, they first generate mask proposals and then perform mask classification with the queries generated by the transformer.
Class incremental semantic segmentation (CISS). Previous work has made attempts at semantic segmentation in incremental learning scenarios. Modeling-thee Beeiiigg-hh-Bakgruuuuuu (MiB) [4] first clarifies the background shift in CISS, and proposed a distillation-bl work of Sparse and Disentangled Representations (SDR) [27] exploited a latent space to reduce forgetting, especially stored prototypes (class centroids) to recall the previous categories. Meanwhile, Douillard et al. [11] proposed the approach of PLOP to utilize the pseudo-label of the background pixels in the current step with the previously learned model as supervision. Furthermore, the SSUL [S] introduces an additional unknown class label to unseen objects in the background by an off-the-shelf saliency-mag mma detector, preventing the model from forgetting by freezing model parameters.
framework in CIL. The subsequent
Method
3.1 Problem Definition
We follow the common definition of Class Incremental Semantic Segmentation (CISS) as in the previous work [5, [1], but also give a more clear definition on the background class and unseen classes. In CISS, there are a series of incremental learning steps, as t = 1....,T T, Let Dr denotes
the dataset for the tu learning step, and C, the class set of samples in dataset D.. For any pair (x,, yt) within D,, x = { and y: = 13 denote the input image and its ground-truth mask with Q pixels, respectively. Unlike previous approaches that specifically define background class, we argue that in a lifelong incremental learning process with annotations of both stuff and thing, all pixels that not belong to current class will be learned in the future. Therefore, in each learning step, the class set can be expressed as C, U {cu). ). where C, denotes the classes to be learned currently, and Cu denotes special class consisting of all the historical and future classes. eu can also be considered as the background class from the perspective of each current learning step, and varies for each learning step, where the background shifi lies.
The CISS model fe,o with parameters at the 4th learning step, aims at assigning each pixel of a with the probability for each class. The model fú, includes convolutional feature extractor and
3

---

Model stop
Proposal Reorganization
Unseen Classes Assembly
Clacsifeer
prediction
Foature
Feature Extrector
Proposal dassification branch
step
Avarage
Pooling
Classifier
Propesel Generator
Beorganisation
Unseen Clavses Assembry
Figure 2: Overall architecture of the proposed MicroSeg.
classifier (to classify each category within the union of CI. = U 1 C, and the unseen class cu. i.e., C = Ci=t U Cu The model after the learning step gives a prediction for all classes, within historical classes C1a-1- Specifically, the predicted result can be expressed as ÿJ = arg max,ec J (œ). Thus, catastrophic forgetting happens due to the lack historical and future classes supervision in C..
3.2 Overview of the MicroSeg
Fig. 2 illustrates the overall framework of our proposed method. We design two-branch structure with the proposed Mining unseen Classes via Regional Objectness (Micro) mechanism. Given an input, the upper branch in Fig. 2 performs dense prediction over the whole image, while the lower branch splits into segment proposals by proposal generator and classifies them into corresponding categories. In particular, introduce the micro mechanism in the optimization process of both branches, to address the background shift issue.
Proposal generator. The proposal generator aims to generate a group of binary masks P (0, Nx Nx where N is the number of proposals and H, W denote height and width. Unlike previous semantic segmentation methods, we split the task into class-agnostic proposal generation by Mask2Former [8] and proposal classification. In this study, all proposals are disjoint, and each pixel belongs and only belongs to one proposal.
The dense prediction branch. The CISS model fú,e is composed of feature extractor 9t.0, and a classifier he.e,- For convenience, we abbreviate Je. as f == h o g(-). Note that the proposal structure and parameters of g(.). but
classification branch and dense prediction branch share the with different designs of /t(-), i.e., (-(-) (-) and ha(-). The dense prediction branch performs conventional semantic segmentation, i.e., Pa ha(g(x)) where Pa indicates the prediction scores (logits).
The proposal classification branch. In this branch, masked average pooling (MAP) [42] used to generate prototype (centroid) pr NNCC of each proposal, where C' is the number of channels of feature map. The logits p IR NxC] of all proposals are generated by classifying pr, where [C] = [..tt {cu)) denotes number C. Then p reorganized to the original shape and the prediction scores are assigned accordingly. Formally, the proposal classification branch can be expressed as:
Pp MMA()
where Reg(-) is the Proposal Reorganization shown in Fig. 2, and can be implemented by matrix multiplication, detailed Supplementary Materials. The proposal classification branch focuses on objectness aggregation within proposal, aiming at consistent representations in a particular proposal.

---

3.3 Mining Unseen Classes via Regional Objectness
Due to the background shift issue (i.e., background class has unclear and unstable semantics during the training steps) mentioned above, the unseen class here refers to all the classes exclude the current foreground class. As mentioned in the observation in Sec. 1, this problem was not well explored in prior works, due to the ignorance of unseen classes. The proposed MicroSeg, on the other hand, can discover the unseen area with the proposal classification branch. Based on the assumption that background regions with strong objectness possibly belong to those concepts in the historical or future steps, we further remodel the supervision label and mining unseen classes via regional objectness, namely, Micro mechanism. Mirco mechanism clusters proposals with strong objectness in unseen classes, and assigns them with new defined labels during the optimization.
Label remodeling. To capture the knowledge learned in the past learning step t 1, we extract the sample on f1-10 during inference to get the pseudo label of mask ýt-1 and the corresponding prediction score map 8.-1- Formally:
9 arg max f(-1,0(z),
Ciit-1
(2)
8t-1 max (())) Ci.--1
where (.) denotes Sigmoid function. With the ground truth mask y, = (y.,) current step, we augment the supervision label Dr, of pixel according to the following rules:
ye,a
where Vr E C.
yr.
1t- where Vt, = Gu A A
Cp
others ( or Yr, = Cu C1=--1
where t represents threshold for Út-1, Ca denotes future classes within the unseen class Cu. the symbol A represents the co-taking of conditions and! \ means the difference of the sets.
Micro mechanism. The micro mechanism is introduced to characterize the various semantics in unseen classes during the optimization and alleviate the background shift issue. Even the future classes Cú include several classes of stuff or things, so we represent it as a set of K classes to better cache the feature diversity those loose future classes. Formally, we represent Ga = {cr,e) where each Ca.& can be viewed as a cluster center of all the future classes. We impose both supervised and unsupervised constraints (detailed in Sec. 3.4) on ea during the optimization. For the supervised part, the prediction scores of class ca are assembled with the summary of the logits of ca.::
K
Pee (x)
(4)
k=
where P& (z) fiee E "xWW is the the prediction scores of class ca.k- H and W denote the height and width of image samples respectively.
We argue that ca captures more features with different spatial characteristics in unseen classes. This prevents potential classes from being incorrectly classified into historical classes C1,--14 which reduces the impact historical knowledge and the collapse of class predictions. Besides, equipped our method with the memory sampling strategy [5], resulting in MicroSeg-M -M. With rehearsal of samples from historical learning steps, catastrophic forgetting is alleviated significantly.
3.4 Objective Function
MieroSeg constrains the two branches via Binary Cross-Entropy (BCH) loss, calculated with aug- mented supervision label from Eq. 3. Specifically, LBCE is defined as:
1 C
CACE
( ci) log(o(p))) + (1 1 =cil) log log(1-
C
1=1
i=1
(5)
Q
(1]0r, = caJ log(o () 1 =
col) =
(v'
1=1
5

---

where pi means the predicted logits of the in pixel to the 4n class in C, and Pi, corresponds to in Eq. 3. We note BCE loss of two branches as BCE and BCE.
Benefiting from the alass-ggosttic proposals, the proposal classification able to discover the unseen classes in CIL. Meanwhile, the dense prediction is still helpful when the supervised label is complete, i.e. the first learning step. The reason is that there is no historical class the very initial learning step, where catastrophic forgetting is not yet happened. The dense prediction branch is also beneficial to learn a robust and versatile feature extractor on base classes, which vital for the incremental settings, where historical samples are not available in the latter learning steps. The output of the whole model is from the proposal classification branch. The loss for this part is then defined as:
LECE + LICE ift= [
LRCE
others
We also apply another constraint to enhance the unseen classes modeling, i.e., the contrastive loss [13], to capture the diverse concepts in cai
1 K
1
K log
exp(pi ps)
L. =
Σ/= exp(p: PT)
where denotes the inner product. In Micro mechanism, we represent future classes ca explicitly- defined multiple sub-classes to better cache the feature diversity in those loose future classes. To avoid model degradation caused by ca. characterizing the same features, we use unsupervised contrastive loss to force them to capture the diverse concepts, thus ensuring the effectiveness of the Micro mechanism.
Finally, the overall loss function is defined as below with hhper--arameeee parameter balance the two terms:
L = LBCE + fe.
4 Experiments
4.1 Experimental Setups
Dataset. We have evaluated our approach on datasets of Pascal VOC 2012 [12] and ADE20K [43]. Pascal VOC 2012 contains 10,582 training images and 1449 validation images, with total of 20 foreground classes and one background class. ADE20K contains 20,210 training images and 2,000 validation images with 100 thing classes and 50 stuff classes.
Protocols. Following the conventions of previous work [5, 11J, we evaluate our approach with overlapped experimental setup, which is more realistic and challenging than the disjoint setting studied by [4, 27]. For the incremental scenario, we apply the same representation in the previous works [4, 27]. For example. the 15-1 scenario of Pascal VOC 2012 (including 20 foreground classes) takes steps train the model. i.e., 15 classes are used in step and one class for each step in steps 2 to 6. More details of protocols are shown in the supplementary materials.
Implementation details. Following the common practice [4, 5, 27J, we use DeepLabv3 [6] and ResNet- 101 [14] pretrained on ImageNet as the segmentation network, and the output stride is 16. We train the network using the SGD with learning rate of 10-2 on Pascal VOC2012, and 5 10-3 for all steps on ADE20K. The momentum and weight decay are 0.9 and 10-2 on both datasets. The batch size is 16 for Pascal VOC 2012, and 12 for ADE20K. Data augmentation from [11] is applied to all the images. MicroSeg freezes the parameters of the feature extractor in the learning steps t > 1, following [5]. Our experiments are implemented with Py PyTorch on two NVIDIA GeForce RTX 3090 GPUs We choose Formrrormer [8] pre-trained on MS-COCO [20] to generate N = 100 class-agnostic proposals. Note that, for fair comparisons, the proposal generator not fine-tuned on any benchmark dataset. The setting of Micro mechanism is K = 5 for Pascal VOC 2012 and K = 1 for ADE20K. Thus, all pixels of samples in ADE20K can be potential foreground. Hyper-parameters 0.7 and X = 1 are set for all experiments. With the same setting of memory sampling strategy as [5], we set the [ 100 for Pascal VOC 2012 and |M| = 300 for ADE20K to evaluate MicroSeg-Mr M. More details are shown in the supplementary materials.

---

Table 1: Comparison with state-of-the methods on Pascal VOC 2012.
Method
0-10
VOC 10-1 (11 steps) all
11-20
VOC
0-2
2-18
(10
stepai all
0-15
VOC15.1 (6 stepe)
16-20
all
VOC
0-5
5.3 (ts(pp))
6-30
all
VOC
0-19
19-1 (2 stepe)
20
all
0-15
VOC
15-5 (2 steps)
16-30
all
LuR-MC [19]
4.7
5.9
4.9
3.5
4.7
4,5
6.4
3.4
6.9
20.9
36.7
24.7
64.4
13.3
61.9
58.1
35.0
52.3
[6]
3.7
5.5
5.8
5.0
8.0
22.5
31.7
29.0
67.8
10.9
65.1
67.1
29,2
60.5
13.1
13.5
57.1
42.6
71.4
23.6
76.4
50.0
17.0
12.1
57.4
52.6
PLOP [1]]
44.0
15.5
30.5
24.1
11.9
31.1
54.6
17.5
19.2
18.7
75.4
374
73.5
75.7
51.7
70.1
SSULISI
71.3
46.0
59.3
62.4
42.5
45.3
77.3
36.6
67.5
72.4
50.7
56.9
71.7
29.7
75.4
77.8
50.1
71.2
SSUL-MI M[51
74.0
53.2
61.1
58.8
45.8
47.6
78.4
19.0
71.4
71.3
53.2
58.4
77.8
49.8
76.5
78.4
55.8
73.0
MicnSeg (Ours)
72.6
48.7
61.2
61.4
40.6
43.5
80.1
16.8
69.8
77.6
59.0
64.3
78.8
14.0
75.7
80.4
52.8
73.8
Micnoeg- M (Ours)
77.2
57.2
67.7
50.9
52.2
81.3
52.5
74.4
74,8
60.5
64.6
79.3
62.9
76.5
82.0
59.2
76.6
79.6
75.0
80.7
80.9
75.0
90
M
10UL-M
80
90
FL.o
80
70-
mloU(%)
mlo(((%)
60
70
70-
50
60
60
30
50-
50
2
10
40
learning steps
learning steps
asclbads
1001-M
(a)
(b)
Figure 3: (a) mloU visualization on Pascal VOC 2012 15-1, (b) mloU visualization on Pascal VOC 2012 2-2, (c) mloU on Pascal VOC 2012 in scenario 15-1, averaged over 20 different training orders.
Baselines. We evaluate our MicroSeg and MicroSeg-M, with the comparisons with some classical methods in CIL (e.g., 8., Lwf-MC [19] and ILT [26]), as well as the state-of-the-art art methods in CISS, including MiB [4], SDR [27], PLOP [11]. and SSUL [5]. For the methods of CIL, we applied them to (mloU) as the evaluation metric for all the experimental evaluation and comparisons. In our experiment, the result of each experiment setting shows in three columns, i.e., mIoU of classes learned in learning step
each experimental setup of CISS. We use the commonly used mean Intersection-ors ceeenn-vve--nneee
(base classes), in /. 2 (novel classes), and all classes to now, respectively. Besides comparing with prior methods, we also provides the experimental results of joint training, i.e., training all classes offline. This setting is usually regarded as an upper bound of corresponding incremental scenario [4, 19].
4.2 Experimental Results
Experiments on Pascal VOC 2012. Besides the widely discussed scenarios of 19-1, 15-1, and 15-5 in previous works, we verified the performance of MicroSeg in several more challenging incremental scenarios, e.g., scenarios of 5-3 (6 steps), 10-1 (11 steps), 2-2 (10 steps). These long-term incremental scenarios is more meaningful, and closer to situations in reality. Tab. shows the final results after all learning steps in each incremental scenario. From the experiments, we can see that our approach has a very clear advantage over previous methods for all incremental scenarios. Even in the without-memory setting, MicroSeg shows performances over the state-of-the-arr methods in almost all incremental scenarios. MicroSeg is obviously superior to SSUL. with memory-free by the mIoU improvements of 2% to 3%, in the short-term incremental scenarios, egg., 19-1 and 15-5. While for the long-term scenarios, MicroSeg outperforms SSUL by a wider margin of up to over 7% improvement on mloU, Not only that, MicroSeg-M (i.e., MicroSeg with memory sampling of past samples) further strengthens the performances, and undoubtedly outperforms all prior methods in all scenarios.
Meanwhile, Fig. (a) shows the change of average mIoU for all past classes under the scenarios of 15-1, with the training step progressing. In the first step, the performances of all methods are similar, but the mloU starts to decrease as the learning steps increase because of the forgetting of the past classes. However, the mioU of the previous work drops significantly, while MicroSeg effectively slows down the drop. By ensuring similar semantic segmentation capabilities to previous work, it demonstrates that MicroSeg exactly benefits incremental learning and outperforms prior works.
7

---

Table 2: Comparison with state-of-the-att methods on ADE20K.
ADE 100-5 (11 steps)
ADE
100-10 (6 steps)
ADE
100-50 (2 steps)
ADE
50-50 (3 steps)
Method
0-100
101-150
all
0-100
101-150
all
0-100
101-150
all
0-50
51-150
all
[26]
0.1
1.3
0.5
0.1
3.1
1.1
18.3
14.4
17.0
3.5
12.9
9.7
MiB
36.0
5.7
38.2
40.5
17.2
32.8
29.3
PLOP
39.1
7.8
28.8
40.5
13.6
31.6
41.9
14.9
32,9
48.8
21.0
30.4
SSUL[S]
39.9
17.4
32.5
40.2
18.8
33.1
41.3
18.0
33.6
48.4
20.2
29.6
SSUL-M[5]
42.9
17.8
34.6
42.9
17.7
34.5
42.8
17.5
34.4
49.1
20.1
29.8
MicroSeg (Ours)
20.5
33.8
41.5
21.6
34,9
40.2
18.8
33.1
48.6
24.8
MicroSeg M (Ours)
43.6
22.4
36.6
43.7
22.2
36.6
43.4
20.9
35,9
49.8
22.0
31.4
Joint
43.8
28.9
38.9
43.8
28.9
38,9
43.8
28.9
38,9
50.7
32.8
38.9
Step
Step
Step
Step
Step 5
Step
image
16 plant.
shoup
train
GT
SSUL-M
Tralir
1aa
15-1
label
MarrSSgg-M
MicoSag-M SSUL-M
bike
bareil
bilee
table
2-2
image
Step
Step 3
Step
Step
Step
Step 10
3
Figure 4: Qualitative analysis of MicroSeg Pascal VOC 2012. Color-shaded boxes with semantic category and its index indicate the learned class during the corresponding step.
Fig. (b) shows the similar analysis on the long-term scenario of 2-2. The effect of MicroSeg has shown
challenging settings. Fig. (c) shows the performance of MicroSeg with 20 random learning orders of 15-1 incremental scenario. Compared with prior approaches, our approach performs better result with lower variance. The result illustrates that the robustness of our approach with various task settings outperforms prior methods.
significant advantage in such
Experiments ADE20K. On the ADE20K dataset, have provided experimental results with multiple incremental scenarios, which have various numbers of learning steps, and we extensively compared MicroSeg with prior methods in these setting. In Tab. 2, as in the case of Pascal VOC 2012, our MicroSeg has achieved a compelling enough performance without any samples sampling memory cost. Moreover, our MicroSeg-M extends this advantage and outperforms the previous otate-of-the-art on all incremental scenarios. Further, the performance of our approach is very close to the offline training on base classes, and has significant improvement prior works on novel classes.

---

image
GT
pred.
cluster
image
GT
pred.
cluster
Figure 5: Qualitative performance of the proposed Micro mechanism. The first and second columns sampled imput images and ground truth. The third column shows the predictions of the last learning step. For the last column, distinct colors denotes different clusters of unseen class, i.e, Ca.k mentioned in Sec. 3.3.
Qualitative analysis of MicroSeg. Fig. 4 shows qualitative analysis of our approach compared with SSUL-M, the current state-of-the-ar method. Results are visualized of each learning step on multiple incremental scenarios of Pascal VOC 2012. First, we evaluated the typical scenario of 15-1 with large number of base classes, to assess the plasticity the model, i.e., the ability to learn new classes. Regarding the samples set of [dog, cat, sofa) (the first and third rows), (dog, cut) belongs to the base classes while (sofa) is learnt step 4. contrast to SSUL.-M, MicroSeg-M M extract feature of [sofa] well, and performs complete segmentation of the novel class sofa. The other samples set (the second and fourth rows) an image of TV (learned in the last learning step). The visualization indicated that, starting from step 4, SSUL-M tends to segment the images incorrectly into current foreground classes, i.e., sofa and train, which are similar only in low-level semantic features (e.g., similar square shapes). Our approach, the other hand, better learns the semantic features of the novel class by regional objectness, avoids the wrong segmentation, and segments correctly and completely after learning the class TV.
We further evaluate the model stability (i.e., ability to maintain old knowledge while learning new concepts) the more challenging long-term scenario of 2-2. First SSUL-M incorrectly classifies bird in the image as bike at step before learning bird. After learning class bird, SSUL-MM products an acceptable result in step 3. However, in the subsequent learning step, the prediction results become more and more chaotic. Meanwhile, our MicroSeg has stronger stability, even long-term incremental scenario. MicroSeg maintains consistent results in multiple steps, and mitigates the negative effect noise on the prediction with Micro mechanism.
Qualitative analysis of the Micro mechanism Fig. shows the qualitative results of the proposed Mícro mechanism. For the 'cluster' column, distinct colors denotes different clusters of unseen class, ie., Ca.& motioned in Sec. 3.3. It can be seen from the figure, the background of samples are clustered well and reasonably with the Micro mechanism, even without supervision by ground truth. The mining of the unseen class not only mitigates the impact of the historical knowledge issue, but also benefits learning new concepts.
4.3 Ablation Study
Effect of components in MicroSeg. Tab. shows the contributions of two components of our approach, includ- ing proposal classification branch and Micro mechanism. The first row refers to the baseline and the last row stands for MicroSeg. The ablation studies are done on VOC 15-1. We will analyze them separately. The pro- posal classification branch improves the mloU in the base class and the novel class, thus leading to an overall performance improvement.
Table 3: Ablation study of components in the proposed method on VOC 15-1, with DP (dense prediction branch), Proposal (proposal classification branch), and Micro (Micro mechanism). Numbers in brackets are the improvements over baseline (the first row).
VOC 15-1 (6 tasks)
DP
Proposal
Micro
0-15
16-20
all
X
69.1
14.0
56.0
/
x
/
76.8
31.2
65.9 (49.9)
X
77.8
32.2
66.9 (+10.9)
/
/
/
80.1
36.8
69.8 (+13.8)

---

The usage of proposal classification branch increases both stability and plasticity, which are the most critical elements in incremental learning. As for Micro mechanism, observe the first and second rows of the table, the application of Micro mechanism decreases the influence of background shift, significantly improves the result. Due to the design of the Micro mechanism, the model generates more reasonable augmented labels and decreases the influence of background shift, strengthening the ability for acquiring novel classes and maintaining historical concepts. Finally, by combining the two components, MicroSeg can get better performance, 13.8% % over the baseline.
Effect of dense prediction branch. Tab. 4 shows the effect of dense prediction branch. As we discussed Sec. 3.4, this branch only used when learning step t = 1. Hence, the constraint of dense prediction only applied when the model is trained on the base classes. The result shows the comparison between the experiment of applying proposal classification branch only and using both branches. can be observed that at the first learning step, dense prediction branch benefits the training on base classes. When all training steps are done, mloU of novel classes is significantly improved. Dense prediction branch enables the model to be trained better the base classes. Since MicroSeg applies remodeled labels composed of pseudo label, the quality of predictions of past classes becomes vital to the results. Thus, it increases the performance of the novel classes by improving the reliability of the remodeled label.
Table 4: Ablation study of the dense prediction branch VOC 15-1. DP: dense prediction branch, Proposal: proposal classification branch.
step=1 base
step=6
Setting
0-15
16-20
all
Proposal
82.2
77.6
28.0
65.8
Proposal+DP
83.3
80.1
36.8
69.8
Ablation of Sypee-parametent of future classes К and threshold r in lebol-nem...ling. The results show that in most cases, our approach is not sensitive syper-parameters.
Tab. 5 illustrates the influence of hyper--ppaaeeee parameters: the number
Table 5: Search of rrper-parameters: the number future classes K, and threshold T in label- remodeling. All experiment are conducted VOC 15-1.
K
0-15
16-20
all
0-15
16-20
all
78.6
30.9
67.2
0.1
79.7
30.7
68.3
79.7
34.0
68.8
0,3
81.1
31.3
69.2
80.1
36.8
69.8
0.5
80.8
33.1
69.4
80.4
32.8
69.1
0.7
80.1
36,8
69.8
80.1
33.1
68.9
0.9
80.0
32.5
68.7
Conclusions
In this study, we propose effective method, MicroSeg, aiming class incremental semantic segmentation problems by mining unseen classes via regional objectness. We first introduce proposals into class incremental semantic segmentation, and design proposal-guie -gsiddd segmentation branch. To tackle background shift, we propose Micro mechanism, clustering and remodeling unseen classes, capture diverse category features. Experiments demonstrate the effectiveness of our method. MicroSeg and MicroSeg-N M achieve remarkable performance, especially on long-term incremental scenarios, outperforming prior ofate-of-the-aee class incremental semantic segmentation methods.
Limitations and societal impact. Although our proposed Micro mechanism effectively mitigates background shift, catastrophic forgetting still exists the learning step increases. As simple baseline, we hope our work to inspire following-up research towards more challenging scenarios in incremental learning, e.g., few base classes and long-term settings.
Acknowledgment. This work was supported in part by the National Key R&D Program China 12001 1DDD0112100 12100), the National Natural Science Foundation of China (No. 61972036), the Fundamental Research Funds for the Central Universities (No. K22RC00010).
10

---

References
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proc. European Conference on Computer Vision, pages 139-154, 2018.
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. pages 3366-3375, 2017.
[3] Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. Advances in neural information processing systems, 13, 2000.
[4] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 9233-9242, 2020.
[5] Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al. Ssul: Semantic segmentation with unknown label for xemplar-based class-iccrementll learning. Advances in neural information processing systems, 34, 2021.
[6] -lang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Recognition and Machine Intelligence, 40(4))834-848, 2017.
[7] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint wxXiv:770.005587 2017.
[8] Bowen Cheng, Ishan Misra, Alexander Schwing, Alexander Kirillov, and Rohit Gird- har. Masked-attention mask transformer for universal image segmentation. arXiv preprint 12.11:21 2021.
[9] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34, 2021.
[10] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proc. IEEE International Conference on Computer Vision, pages 2021.
[11] Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord. Plop: Learning without forgetting for continual semantic segmentation. In Proc. IEEE Conference Computer Vision and Pattern Recognition, pages 4040-4050, 2021.
[12] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal on Computer Vision, 88(2):303-338, 2010.
[13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 9729-9738. 2020.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[15] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint 1503:02533333 2015.
[16) Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. Proc. IEEE International Conference on Computer Vision, pages 603-612,

---

[17] Heechul Jung. Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122 2016.
[18] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(7553):436 444, 2015.
[19] Zhizhong Li and Derek Hoiem. Learning without forgetting. JEEE Transactions on Pattern Recognition and Machine Intelligence,
-9447 2017.
[20] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Proc. European Conference on Computer Vision, pages 740-755, 2014.
[21] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proc. IEEE Conference on Computer Vision and Paltern Recognition, pages 3431-3440, 2015.
[22] David Lopez-Pat Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[23] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. pages 7765-7773.2115
[24] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[25] Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannaneh Hajishirzi. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. In Proc. European Conference Computer Vision, pages 552-568, 2018.
[26] Umberto Michieli and Pietro Zanuttigh. Incremental learning techniques for semantic segmen- tation. In Proc. IEEE International Conference Computer Vision Workshops, pages 0-0, 2019.
[27] Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion- attraction of sparse and disentangled latent representations. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 11 1114-1124, 2021.
[28] Robi Polikar, Lalita Upda, Satish S Upda, and Vasant Honavar. Learn++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 31(4):497-508, 2001.
[29] Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In Proc. IEEE International Conference on Computer Vision, pages 1320- 1328, 2017.
[30] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[31] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Expe- rience replay for continual learning, Advances in neural information processing systems, 32, 2019.
[32] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. International Conference on Medical Image Comput- ing and Computer-Assisted Intervention, pages 234-241 2015.
[33] Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE Transactions Pattern Recognition and Machine Intelligence, 22(3):651-633 2018.
12

---

[34] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins. Hubert Soyer. James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks, arXiv preprini arXiv: (6666666677,, 2016.
[35] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proc. International Conference on Machine Learning, pages 4548-4557, 2018.
[36] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
(38) Panqu Wang. Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell. Understanding convolution for semantic segmentation. In Proc. IEEE/CVE Winter Conference Applications of Computer Vision, pages 1451-1460, 2018.
[39] Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, and Yizhou Yu. Fastfen: Re- thinking dilated convolution in the backbone for semantic segmentation. arXiv preprint arXiv: / 1816, 2019.
[40] Ju Xu and Zhanxing Zhu. Reinforced continual learning. Advances in neural information processing systems.
[41] Junting Zhang. Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Blass-incremental learning via deep model consolidation. In Proc. IEEF/CVF Winter Conference Applications of Computer Vision, pages 1131-1140, 2020.
[42] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics, 50(9):3855- 3865, 2020.
[43] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 633-641, 2017.
[44] Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In Proc. IEEE International Conference on Computer Vision, pages 593-602, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We introduce the main claims our study in the Abstract and Introduction.
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations and future work in Conclusions.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss societal impact in Conclusions.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read and obey them faithfully.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] Code and instructions are included in Supplement materials.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Training details described Experiments.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We provide experimental results of evaluating our method with twenty random category orders.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Computing resources described in Implementation Details, in the section of Experiments.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] The proposal generator in our method is based on mask2former.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The datasets we used in our work are open source.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "bloyOSjfIy", "reviewer": "Reviewer_3V6U", "summary": "This paper proposes a new method for class-incremental segmentation. Based on the assumption that background regions with strong objectness possibly belong to those concepts in the historical or future stages, this paper introduces a proposal generator for generating multiple segment proposals. Also, some tricks including label remodeling and the new loss functions are designed. The proposed method gains good performance on Pascal VOC and ADE20K. ", "questions": "Please refer to the weaknesses part for details.\n\nI cannot recommend accepting this work at its current version considering the high standards of NeurIPS. However, considering the interesting motivation and the relatively novel method, I suggest the authors to carefully revise the paper. I believe it would be a good work if the writing is clearer.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I appreciate the motivation of this work, which is well presented in the introduction. Also, I believe the proposed method makes sense and is somehow novel. \n\nMy main concern of this work is the writing. I suggest the authors to reconstruct this paper in a clearer form. Many parts of this paper are quite confusing and hard to understand. Also, it lacks a lot of details, so that it is hard for others to follow. Below are some of the weaknesses:\n\n1. This paper proposes a proposal generator. However, the detailed structure of the generator is not clear. \n\n2. In Line 142 to Line 143, 'Then $P$  is *reorganized* to the original shape and the prediction scores are assigned accordingly'. I can understand this after checking Figure 2. However, I think it may be better if the authors illustrate this step more clearly by words or equations. \n\n3. Why the contrastive loss (Equation 7) is used for optimization? The motivation and prupose are not clear. \n\n4. What's the value of $\\tau$ in Equation 3? It is not illustrated in the implementation details. \n\n5. Some ablation experiments are missed. For example, how the settings of hyper-parameters $N$, $K$ and $\\tau$  affect the performance?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IC0WdkM92H_", "reviewer": "Reviewer_FY3k", "summary": "This work address the \"semantic shift of the background class\" problem where some concepts learned at previous stages are wrongly assigned to the background class at the current training stage, in Class incremental semantic segmentation task. The authors first splits the given image into hundreds of segment proposals with a proposal generator pretrained on MSCOCO. Then the segment proposals with strong objectness from the background are clustered and assigned newly-defined labels during the optimization. Combining such supervision into a conventional dense prediction framework, the proposed method could relieve the catastrophic forgetting caused by the semantic shift of the background class accordingly. Experiments on Pascal VOC and ADE20K datasets show competitive results with state-of-the-art.", "questions": "Please see the weaknesses.\n\n------------------------------------------\nThe authors have addressed my major concerns. I would stick to my original score.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The motivation is clearly presented and the method is straightforward to address the \"semantic shift of the background class\" problem where some concepts learned at previous stages are wrongly assigned to the background class at the current training stage, in Class incremental semantic segmentation task.\n\n2. The paper is well written and easy to follow.\n\n3. The overall performance of the proposed method is good and sufficient to verify the effectiveness of the proposed method.\n\n\nWeakness:\n\n1. The claim in Line 175~176 is not validated which it is valuable to see whether the proposed method could prevents potential classes from being incorrectly classified into historical classed.\n\n2. In Tab. 1, for VOC 2-2 (10 tasks) and VOC 19-1 (2 tasks) MicroSeg gets inferior performance compared with SSUL, the reason should be explained. It also appear in ADE 100-50 (2 tasks) in Tab. 2.\n\n3. The proposed method adopts a proposal generator pretrained on MSCOCO which aggregates more information. Is it fair to compared with other methods? Besides, could the proposed technique propmotes existing Class incremental semantic segmentation methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "BpKJv8eLxs-", "reviewer": "Reviewer_uMp6", "summary": "The authors propose a novel way to cluster the unseen classes present in the background using K different background components and a pre-trained class-agnostic mask proposal generator (Mask2Former). The results show a clear improvement from pixel-wise methods, which is further improved by the micro mechanism.\n", "questions": "- The disjoint setting as it is described would allow future classes unlabeled in the background contrary to the definition of disjoint in MiB [4]. Is this an actual change of the setting or just a mistake in the description?\n- Are the joint results on ADE20k based on the proposed architecture, including the Mask2Former or for the base DeepLabv3 model (the results are identical to those reported in [4], which seems low if the Mask2Former also is used since they achieve 47.7 using ResNet50 by training on ADE20k)?\n- Please specify which pre-trained weights that are used for the ResNet-101 network for facilitating reproducibility\n- How does the proposed split of segment proposals and classification differ from [8]? And is the splitting of the task into segment proposal and classification novel for semantic segmentation or for CISS?\n- Does the performance of the proposal generator differ between classes present and absent in MS-COCO? \n- Is the contrastive loss only applied to the background? And if so, is this the background before or after pseudo-labeling?\n- Which losses are applied to the baseline in Tab 3? It seems to perform better than all methods except for SSUL by only using pseudo-labels?", "limitations": "Yes, though there would be room to further discuss the importance of the pre-training of the proposal generation and possible limitations of it when studying data which differs more than COCO/Pascal/ADE20k.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clear improvement in the benchmarks over current SOTA\n- Interesting approach for mining unseen classes and perform pseudo clustering of these\n- Good comparison of how stable the performance is under different class orders on Pascal (Fig 3c).\n\nWeaknesses:\n\nSome claims of novelty seems exaggerated and require revision/clarification, i.e.\n- Experimental settings:\n  - The different settings that are \"introduced\" are not new, i.e. \n    - Pascal VOC 10-1 and ADE20k 100-5 were used in PLOP [11]\n    - Pascal VOC 10-1 + 5-3 and ADE20k 100-5 were all used in SSUL [5] \n  - \"RECALL: Replay-based Continual Learning in Semantic Segmentation\" (ICCV2021) has not been referenced in the paper though they also proposed the 10-1 setting as well as 10-10 and 10-5.\n- Mask2Former:\n  - Partly unclear which parts of the proposal generation and classification that differs from Mask2Former for semantic segmentation.\n\nNo ablation study of how the number of components in the Micro mechanism affects the results, which would be interesting since this is the main contribution of the paper. The choice of K=1 for ADE20k seems conter-intuitive, since ADE20k contains much more classes than Pascal and clustering these into a single component would enforce similar feature representations for all unseen classes.\n\nIt would have been interesting to see how the performance varies for different class orders on ADE20k since the class-frequency decreases drastically between stuff and things and as the original index increases. (As done in MiB and SDR)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9fajU1YZK6A", "reviewer": "Reviewer_Kd2M", "summary": "The paper proposes MicroSeg for class incremental semantic segmentation. The region proposal generator is used to detect the potential objects with strong objectness in the background. The these identified proposals are grouped into a new defined label during the optimization. Experiments are performed on both Pascal VOC and ADE20K to validate the idea.", "questions": "I will consider to raise the rating if my concerns in the above weakness section can be well addressed.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1.The paper has a good motivation, where incremental semantic segmentation is a meaningful to explore and the discussed background shift problem is also difficult for existing models.\n\n2.The experiments show an obvious improvement to existing methods. Table 3 shows the effect of Proposal and Micro respectively.\n\n3.The paper is organized well with good writing and figures.\n\n**Weakness**:\n\n1.The tech contribution of MicroSeg is very limited. Region proposal network for class-agnostic detecting novel objects is already widely used, such as [a].\n\n2.SSUL uses the off-the-shelf saliency-map detector to detect unseen classes while the paper uses the pretrained Mask2Former to produce the region proposals. This may introduce an unfair comparison in both data and model size. Mask2former is additionally trained on COCO and has much larger parameters than the off-the-shelf detector. What if SSUL also adopts Mask2Former to detect unseen classes? Or what if SSUL can generate the object proposals in an unsupervised way w/o additional lots of data or heavy model, such as [b].\n\n3.Missing experiments of using other region proposals networks instead of using Mask2Former, such as the RPN in Mask R-CNN? Will it influence the final model performance a lot?\n\n4.Missing the speed comparison and model parameters with other methods. What are the model sizes of the proposed MicroSeg combined with Mask2Former?\n\n[a] Gu, Xiuye, et al. \"Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2022.\n\n[b] Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity. CVPR, 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "21KaTFRC6KD", "reviewer": "Reviewer_rFyJ", "summary": "This paper proposes a novel framework called MicroSeg for the catastrophic forgetting problem caused by the semantic shift of the background class. This paper pioneered the idea of considering unknown categories that may occur in the future in advance in background learning. More specifically, the idea is quite interesting with the assumption that background regions with strong objectness possibly belong to those concepts in the historical or future stages.  Experiment results show promising results on Pascal VOC and ADE20K datasets.", "questions": "1. In the Label Remodeling part (i.e., Equation 3), what is the meaning of symbols, such as ^ and \\ ?\n2. The Proposal Generator has the ability to generate a binary map. How is it trained? Is it a pretrained model from [8]? ", "limitations": "The authors have well addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Overall the paper is well written, well illustrated, and subsequently pleasant to read.\n2. The method itself is clear and proposed contributions are really intuitive.\n3. The contributions and experiments are decent and clearly shows improvement over the state-of-the-art. Ablations are okay.\n\nWeakness\n1. The author proposes pre-learning for future categories, so the labeling of unknown categories that may appear in the future will be the key to this paper. However, Label Remodeling in Section 3.3 is not explained clearly. See the the Questions part for details.\n2. There are few descriptions of MicroSeg-M in the paper. It just shows that it uses the memory sampling strategy [5]. In my opinion, simply migrating other methods to this task may not be a contribution. If authors have no special design on MicroSeg-M, I suggest to remove it, since MicroSeg itself is interesting enough.\n3. There are several formatting errors in the article. For instance, there are no full stops on lines 42 and 66. The authors are desired to read the paper thoroughly and revise them carefully.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept