Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel framework called MicroSeg for the catastrophic forgetting problem caused by the semantic shift of the background class. This paper pioneered the idea of considering unknown categories that may occur in the future in advance in background learning. More specifically, the idea is quite interesting with the assumption that background regions with strong objectness possibly belong to those concepts in the historical or future stages.  Experiment results show promising results on Pascal VOC and ADE20K datasets.
strengths_and_weaknesses: Strengths
1. Overall the paper is well written, well illustrated, and subsequently pleasant to read.
2. The method itself is clear and proposed contributions are really intuitive.
3. The contributions and experiments are decent and clearly shows improvement over the state-of-the-art. Ablations are okay.

Weakness
1. The author proposes pre-learning for future categories, so the labeling of unknown categories that may appear in the future will be the key to this paper. However, Label Remodeling in Section 3.3 is not explained clearly. See the the Questions part for details.
2. There are few descriptions of MicroSeg-M in the paper. It just shows that it uses the memory sampling strategy [5]. In my opinion, simply migrating other methods to this task may not be a contribution. If authors have no special design on MicroSeg-M, I suggest to remove it, since MicroSeg itself is interesting enough.
3. There are several formatting errors in the article. For instance, there are no full stops on lines 42 and 66. The authors are desired to read the paper thoroughly and revise them carefully.
questions: 1. In the Label Remodeling part (i.e., Equation 3), what is the meaning of symbols, such as ^ and \ ?
2. The Proposal Generator has the ability to generate a binary map. How is it trained? Is it a pretrained model from [8]? 
limitations: The authors have well addressed the limitations.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes MicroSeg for class incremental semantic segmentation. The region proposal generator is used to detect the potential objects with strong objectness in the background. The these identified proposals are grouped into a new defined label during the optimization. Experiments are performed on both Pascal VOC and ADE20K to validate the idea.
strengths_and_weaknesses: **Strengths**:

1.The paper has a good motivation, where incremental semantic segmentation is a meaningful to explore and the discussed background shift problem is also difficult for existing models.

2.The experiments show an obvious improvement to existing methods. Table 3 shows the effect of Proposal and Micro respectively.

3.The paper is organized well with good writing and figures.

**Weakness**:

1.The tech contribution of MicroSeg is very limited. Region proposal network for class-agnostic detecting novel objects is already widely used, such as [a].

2.SSUL uses the off-the-shelf saliency-map detector to detect unseen classes while the paper uses the pretrained Mask2Former to produce the region proposals. This may introduce an unfair comparison in both data and model size. Mask2former is additionally trained on COCO and has much larger parameters than the off-the-shelf detector. What if SSUL also adopts Mask2Former to detect unseen classes? Or what if SSUL can generate the object proposals in an unsupervised way w/o additional lots of data or heavy model, such as [b].

3.Missing experiments of using other region proposals networks instead of using Mask2Former, such as the RPN in Mask R-CNN? Will it influence the final model performance a lot?

4.Missing the speed comparison and model parameters with other methods. What are the model sizes of the proposed MicroSeg combined with Mask2Former?

[a] Gu, Xiuye, et al. "Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2022.

[b] Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity. CVPR, 2022.

questions: I will consider to raise the rating if my concerns in the above weakness section can be well addressed.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work address the "semantic shift of the background class" problem where some concepts learned at previous stages are wrongly assigned to the background class at the current training stage, in Class incremental semantic segmentation task. The authors first splits the given image into hundreds of segment proposals with a proposal generator pretrained on MSCOCO. Then the segment proposals with strong objectness from the background are clustered and assigned newly-defined labels during the optimization. Combining such supervision into a conventional dense prediction framework, the proposed method could relieve the catastrophic forgetting caused by the semantic shift of the background class accordingly. Experiments on Pascal VOC and ADE20K datasets show competitive results with state-of-the-art.
strengths_and_weaknesses: Strength:

1. The motivation is clearly presented and the method is straightforward to address the "semantic shift of the background class" problem where some concepts learned at previous stages are wrongly assigned to the background class at the current training stage, in Class incremental semantic segmentation task.

2. The paper is well written and easy to follow.

3. The overall performance of the proposed method is good and sufficient to verify the effectiveness of the proposed method.


Weakness:

1. The claim in Line 175~176 is not validated which it is valuable to see whether the proposed method could prevents potential classes from being incorrectly classified into historical classed.

2. In Tab. 1, for VOC 2-2 (10 tasks) and VOC 19-1 (2 tasks) MicroSeg gets inferior performance compared with SSUL, the reason should be explained. It also appear in ADE 100-50 (2 tasks) in Tab. 2.

3. The proposed method adopts a proposal generator pretrained on MSCOCO which aggregates more information. Is it fair to compared with other methods? Besides, could the proposed technique propmotes existing Class incremental semantic segmentation methods.

questions: Please see the weaknesses.

------------------------------------------
The authors have addressed my major concerns. I would stick to my original score.
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel way to cluster the unseen classes present in the background using K different background components and a pre-trained class-agnostic mask proposal generator (Mask2Former). The results show a clear improvement from pixel-wise methods, which is further improved by the micro mechanism.

strengths_and_weaknesses: Strengths:
- Clear improvement in the benchmarks over current SOTA
- Interesting approach for mining unseen classes and perform pseudo clustering of these
- Good comparison of how stable the performance is under different class orders on Pascal (Fig 3c).

Weaknesses:

Some claims of novelty seems exaggerated and require revision/clarification, i.e.
- Experimental settings:
  - The different settings that are "introduced" are not new, i.e. 
    - Pascal VOC 10-1 and ADE20k 100-5 were used in PLOP [11]
    - Pascal VOC 10-1 + 5-3 and ADE20k 100-5 were all used in SSUL [5] 
  - "RECALL: Replay-based Continual Learning in Semantic Segmentation" (ICCV2021) has not been referenced in the paper though they also proposed the 10-1 setting as well as 10-10 and 10-5.
- Mask2Former:
  - Partly unclear which parts of the proposal generation and classification that differs from Mask2Former for semantic segmentation.

No ablation study of how the number of components in the Micro mechanism affects the results, which would be interesting since this is the main contribution of the paper. The choice of K=1 for ADE20k seems conter-intuitive, since ADE20k contains much more classes than Pascal and clustering these into a single component would enforce similar feature representations for all unseen classes.

It would have been interesting to see how the performance varies for different class orders on ADE20k since the class-frequency decreases drastically between stuff and things and as the original index increases. (As done in MiB and SDR)

questions: - The disjoint setting as it is described would allow future classes unlabeled in the background contrary to the definition of disjoint in MiB [4]. Is this an actual change of the setting or just a mistake in the description?
- Are the joint results on ADE20k based on the proposed architecture, including the Mask2Former or for the base DeepLabv3 model (the results are identical to those reported in [4], which seems low if the Mask2Former also is used since they achieve 47.7 using ResNet50 by training on ADE20k)?
- Please specify which pre-trained weights that are used for the ResNet-101 network for facilitating reproducibility
- How does the proposed split of segment proposals and classification differ from [8]? And is the splitting of the task into segment proposal and classification novel for semantic segmentation or for CISS?
- Does the performance of the proposal generator differ between classes present and absent in MS-COCO? 
- Is the contrastive loss only applied to the background? And if so, is this the background before or after pseudo-labeling?
- Which losses are applied to the baseline in Tab 3? It seems to perform better than all methods except for SSUL by only using pseudo-labels?
limitations: Yes, though there would be room to further discuss the importance of the pre-training of the proposal generation and possible limitations of it when studying data which differs more than COCO/Pascal/ADE20k.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new method for class-incremental segmentation. Based on the assumption that background regions with strong objectness possibly belong to those concepts in the historical or future stages, this paper introduces a proposal generator for generating multiple segment proposals. Also, some tricks including label remodeling and the new loss functions are designed. The proposed method gains good performance on Pascal VOC and ADE20K. 
strengths_and_weaknesses: I appreciate the motivation of this work, which is well presented in the introduction. Also, I believe the proposed method makes sense and is somehow novel. 

My main concern of this work is the writing. I suggest the authors to reconstruct this paper in a clearer form. Many parts of this paper are quite confusing and hard to understand. Also, it lacks a lot of details, so that it is hard for others to follow. Below are some of the weaknesses:

1. This paper proposes a proposal generator. However, the detailed structure of the generator is not clear. 

2. In Line 142 to Line 143, 'Then $P$  is *reorganized* to the original shape and the prediction scores are assigned accordingly'. I can understand this after checking Figure 2. However, I think it may be better if the authors illustrate this step more clearly by words or equations. 

3. Why the contrastive loss (Equation 7) is used for optimization? The motivation and prupose are not clear. 

4. What's the value of $\tau$ in Equation 3? It is not illustrated in the implementation details. 

5. Some ablation experiments are missed. For example, how the settings of hyper-parameters $N$, $K$ and $\tau$  affect the performance?


questions: Please refer to the weaknesses part for details.

I cannot recommend accepting this work at its current version considering the high standards of NeurIPS. However, considering the interesting motivation and the relatively novel method, I suggest the authors to carefully revise the paper. I believe it would be a good work if the writing is clearer.
limitations: NA
review_title: null

Meta-review: metareview: Most of the reviewers pointed out that the motivation of the method is clear, and the method is novel and interesting. The proposed method is also effective on multiple benchmarks. One of the reviewer has concerns about the choice of a parameter (K), and another reviewer has concerns about details of the method. AC admits that these points need to be further improved, but thinks these points can be clarified in the camera ready version. Thus, considering the novelty and efficacy of the method, AC recommends accept for this paper, yet suggests the authors to carefully take the reviewers' comments into account when preparing the final version.
confidence: Less certain
Decision: Accept