Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a class-incremental learning strategy for semantic segmentation. The main contributions of this work include two aspects: first, it introduces an external saliency map predictor to better model the thing classes vs the stuff background; second, the method adds a class-balanced memory to overcome the forgetting in CIL. The proposed method is evaluated on two benchmarks with the overlapped setup and achieves better performance than the prior approaches.  
main_review: Pros:
- The paper developed a well-engineered strategy for class incremental segmentation, which integrates several well-motivated components to reduce catastrophic forgetting. 
- The proposed method achieves strong results on two benchmarks under the overlapped setup, which validates its efficacy. It also includes a detailed ablative study on the main components. 
- The paper is most clearly written and easy to follow. 


Cons:
- The overall novelty of this work is a bit incremental. While the main components are clearly motivated, they are largely borrowed from the prior work in weakly-supervised segmentation (foreground saliency), class-incremental learning (memory), and [7] (pseudo-labels). The beneficial effects of those modules seem to be well understood. 

- There are several concerns on the technical aspects of the method:
1) While adopting object saliency maps can help the thing classes, this would have a negative impact on the stuff classes. For semantic segmentation, this is restrictive for the general cases where we have to incrementally segment both the thing and stuff classes. This also implicitly introduces additional training data (saliency) which may not available for certain application domains. 
2) The model freezing step is also rather restrictive, which leads to weak plasticity for the model. In particular, the performance of the convolutional feature extractor can degrade significantly if the initial class set is small. It is unclear how well such a strategy can generalize to different CIL setups. 
3) How are the hyperparameters of the method determined, such as \tau in pseudo labeling? 
4) Line 189: if the sigmoid output is adopted in training, how does this method calibrate the classifier scores for the inference in Sec 3.1? 

- The experimental evaluation seems a bit lacking and the conclusions are not fully convincing: 
1) The evaluation is only conducted on the overlapped setup, which is insufficient. The method should be tested on the disjoint setup too. The overlapped setup allows the method to learn features for future classes based on the saliency mask, which would favor the proposed strategy.
2) How well the method would perform if the classes include both thing and stuff categories? 
3) Using external saliency maps seems to be unfair to other methods, as it requires additional training data. Such additional information can also help improve the prior methods. 
4) In the ablative study, it seems that adding the Sigmoid+BCE hurts the overall performance. It is unclear if this is a necessary component as Sec 3.4 claims.

Post-rebuttal: The author's reply addressed many of my initial concerns and I would like to raise my rating accordingly. 
limitations_and_societal_impact: Should add discussion on its limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method, Semantic Segmentation with Unknown Label with Memory (SSUL-M), to solve a class-incremental semantic segmentation (CISS) problem. The authors make three main contributions. First, they introduce an additional “Unknown” class label to distinguish the potential future class and the true background. Second, they adopt the pseudo-labeling strategy and freeze backbone network and past classiﬁers with binary cross-entropy loss to tackle catastrophic forgetting. Third, an exemplar memory is utilized to improve both plasticity and stability. Extensive experiments and ablation studies are conducted to justify these proposed components and contributions. Results show that SSUL-M signiﬁcantly outperformed baselines.
main_review: 1.	This paper is well structured and clearly written. The authors introduce the proposed method logically and describe their contributions clearly. The experimental details are provided and look technically sound and reproducible.  
2.	This work aims to tackle the challenges that cause catastrophic forgetting in CISS. To this end, the authors make three main contributions which are quite novel and each contribution has been proved effective by adequate ablation studies. Experimental results on two datasets show that the proposed method achieves signiﬁcantly better performance than the baselines.
3.	Figure 1 is not illustrated clearly. It would be better if the authors explain this figure clearly in its caption so that readers can quickly understand the meaning of each module or symbol in this figure at a glance. Additionally, there are some typos and grammar mistakes. I list several of them here: “SSUL and SSUL-M both maintains … and effectively learns” in page 8, “The ﬁrst two columns in Table 4 shows” in page 9, etc.

limitations_and_societal_impact: The authors did not describe the limitations in the paper and stated no expected potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper provided a recipe of methods to improve the incremental learning on semantic segmentation that empirically improved the performance on multiple datasets compared to other SOTA methods. Specifically, both pseudo labels from previous models and unknown classes from saliency detectors are adopted in model training; Backbone networks are frozen and Sigmoid + BCE are used to avoid forgetting; finally an exemplar dataset is used here to balance the training data in later times and hence further boost the performance. 

main_review: The paper is well written in general except a few questions that may be worth considering below. The originality of the work is moderate. Adding additional label sources and/or training data, however not exact, can usually improve deep models’ performance. The paper found an interesting and valid way to do so under the incremental learning setup. Using Sigmoid + BCE is another interesting application under this setting and the explanation makes sense.  Exemplar memory throws more data into the workflow -  more often than not, it will work if doing it right. 

A few question that may need further clarification: 

With model freezing and stable score learning, in theory there would not be any forgetting for the foreground classes in t - 1? The explanation of having examples to avoid severe forgetting of any foreground classes seems unconvincing in Line 208. 

More detailed explanation on the effectiveness of weight transfer would be helpful for the audience to appreciate the design better. In particular, after model freezing with only BCE in training, one would think that initialization matters less given the shallow structure. Why such a big difference as shown in the ablation study? 

The ablation study is fascinating and considering the work is a recipe of multiple steps, it is worth more discussion and careful look to shed more light on the importance of each step. For example, 

Without freezing, why also such a big drop on new classes as well?

It is argued earlier in the paper (3.4) that model freezing “can roughly learn the representations for the potential future class objects…” in the unknown class. However, in the ablation study, the model performs really well even without the unknown classes. In fact, the performance is still higher than most of the SOTA methods. What is the underlying reason here? Does the composition of data have any influence here as well? 
 
Any chance we can show the ablation study without PL? It is helpful to understand how important it is to have PL even if the model is frozen. 

It would be interesting to see the impact of each step with respect to the complexity of the problem, i.e. reduce the initial number of classes. For example, if we increase the complexity of the problem, is there a certain point that model freezing starts to hurt the performance?


limitations_and_societal_impact: No negative societal impact as far I know.

The limitation of this work has not been fully discussed in the paper. Specifically, the overall method relies on the success of the saliency detector to label the unknown classes. If a new class is not able to be picked up by saliency detectors, or even the new class is simply not of “object-like” but rather “area-like”, I’d expect the proposed method to have a hard time to accommodate the new classes, especially with model freezing. 

Additionally, the benefit of model freezing has a positive effect with the underlying assumption that the initial data are enough for the model to learn good features, which may not be valid in other setup of datasets. Many things can go south here, e.g. new classes are not included in the previous unknown classes, learnt feature space bounded by the initial dataset, etc.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes novel a class-incremental learning method, named semantic segmentation with unknown label with memory. The proposed method addresses two additional challenges in class-incremental learning and proposes three contributions: label augmentation, stable score learning, and tiny exemplar memory.  The proposed method achieves impressive performance compared with other methods.
main_review:  This paper proposes a novel class-incremental semantic segmentation setting. In addition, the writing of this paper is clear and easy to understand. The experiments are sufficient to prove the effectiveness of the proposed paper.
limitations_and_societal_impact: 1 There are some related methods missed, such as [1][2][3].
2 The proposed method employs three strategies to alleviate catastrophic forgetting. These strategies are often employed in other incremental methods. Thus, the contributions are limited in my opinion.
3 Compared with the ground truth, the visualization results are terrible. Thus,  I want to see some analysis of satisfied cases and terrible cases.
[1] Class-incremental learning for semantic segmentation re-using neither old data nor old labels. ITSC2020.
[2] Modeling the background for incremental learning in semantic segmentation. CVPR2020.
[3] Incremental Learning Techniques for Semantic Segmentation. ICCVW2019
review_title: null

Meta-review: The paper describes an effective method for class-incremental semantic segmentation.
The different components integrated to achieve this goal have limited novelty per se,
but are combined nicely and result in a system that works well, particularly under the non-disjoint setting.
Authors are encouraged to include a discussion on limitations in the final version of the paper.
Decision: Accept (Poster)