Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses incremental learning applied to semantic segmentation. There are two main contributions: adding an extra loss term aiming at preserving separately the positive and negative contribution values in the classification logit and learning an additional classifier for negative sample prediction that can be transferred to initialize the new classifier with incremented categories. Evaluation is done on two classical benchmarks: PASCAL VOC and ADE20k.

strengths_and_weaknesses: Strengths

- The paper is clearly written, and mostly well motivated (see however my remarks below).

- The ablation study, experiments and result analysis justify the approach on standard benchmarks.


Weaknesses

- Finding a good loss for class imbalance management is an old problem in ML (e.g., the focal loss that has been used in object detection or AUC maximization). The approach should have been compared, at least in principle, to solutions that have been proposed in the literature outside the field of incremental learning.

- The auxiliary classifier learned to initialize the next incremental step seems to be the most original idea of the paper, and should have been better developed and studied. The impact  of initialization, as illustrated in a few examples in Figure 5, seems to be strong and would have deserved a more in depth analysis, and, possibly, a more conceptual justification. Somehow, I am disappointed that the focus has been on the loss addressing class imbalance – there are other ways to handle it, and this is a well known phenomenon of incremental learning – and not on the initialization step.


questions: I do not have any precise question, only a request for more justification of the two main contributions (see the weaknesses point).

limitations: Non applicable
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a CISS framework which decomposes the prediction logit into two terms.  Then the DKD technique and an auxiliary classifier is adopted to improve the rigidity and transfer the knowledge of negatives, respectively. Experimental results demonstrate the effectiveness of the proposed framework.
strengths_and_weaknesses: Strengths:
+ This paper proposes a new distillation method, DKD to address the forgetting problem. 
+ Extensive comparison experiments illustrate the effective perofrmance of the proposed model to some degree.
+ The main idea of this paper is easy to understand.

Weaknesses:
+The contributions of this paper are somewhat limited, which cannot meet the requirements of NeurIPS acceptance.
+The motivation of some parts of this paper (distillation and initialization) needs to provide more clear explanations. 
+It would be beeter to discuss how the backbone architecture affects continual learning performance.
questions: 1. One contribution of this paper is to propose the decomposed knowledge distillation (DKD) loss, which distills the knowledge of the decomposition. And the decomposition values are obtained by decomposing the prediction logit into two scores (i.e., element-wise sum of positive or negative elements). The decomposition strategy is unconvincing, please provide the theoretical proof.
2. In Equation 1, the hyperparameters and are necessary to the entire model. How are they selected in the experiments? Please provide the experiments about the hyperparameters. And dynamic selection will be a good choice to balance the hyperparameters.
3. Both KD loss and DKD loss help the model retain the old knowledge, and is there any conflict between them? Please add the ablation study.
4. Please compare the proposed algorithm with the state-of-arts method [1].
5. Please add experiments on the cityscapes dataset [2].
6. The current manuscript need to be carefully polished. 

[1] Chang-Bin Zhang, Jia-Wen Xiao, Xialei Liu, Ying-Cong Chen, Ming-Ming Cheng. Representation Compensation Networks for Continual Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[2] M. Cordts, M. Omran, S. Ramos, T. Reheld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
limitations: +The first limitation is whether the framework can still perform well on a large number of task. 
+The second limitation is that the hyperparameters in formula (1) are not dynamic.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a technique for class-incremental learning in semantic segmentation. First, a knowledge distillation loss is proposed, decomposing positive and negative class logits into separate loss functions. Second, an initialization strategy for the classifiers of novel classes is proposed, which is relies on an auxiliary classifier. Evaluation on several standard benchmarks for incremental semantic segmentation shows the methods effectiveness.
strengths_and_weaknesses: Strengths:

1. The introduction and related work provide a clear motivation for the proposed method and overall discuss the contributions well w.r.t. to related works.

2. Clear and extensive mathematical description of the method, which seems reproducible, although code is not provided in the submission.

3. State-of-the-art results on the standard incremental semantic segmentation benchmarks.

4. Nicely prepared figures and tables, clarity of the overall manuscript.

Weaknesses:

1. Lack of hyperparameter analysis: The methodology on how hyperparameters are determined is well-described, but I did not find an analysis in this work, on how changes in the method’s hyperparameters affect the performance.

2. Applicability: The method requires a modification to the pre-trained model before the incremental steps. Therefore, the method is not applicable to off-the-shelf semantic segmentation models and from an applicability perspective rather comparable to replay-based approaches. I still think, the memory-footprint of the method is much smaller than other replay-based methods (e.g., SSUL), but I think the limitation should be clearly stated as it is a bit different from the standard incremental setting for semantic segmentation.

3. Root cause of improved performance: It is a bit unclear how much of the improvement towards SOTA performance is due to better chosen hyperparameters (such as, e.g., larger batch size on better hardware) or how much is due to the contributions. This could be easily fixed by adding a self-simulated version of MiB [5] or SSUL-M [6] in the author’s simulation setting to the comparison in Tables 1 and 2, showing how the self-simulated version of these methods compare to the original numbers provided in the papers.

Overall, I really like the approach taken in this work, the paper is very well written and although some weaknesses definitely exist, the overall value of the contributions has been shown. I would welcome a clear answer to the weaknesses in the rebuttal as some of them might be easy to address.

Minor comments and suggestions:

4. I think the related work Maracani et al. “RECALL: Replay-based Continual Learning in Semantic Segmentation,” ICCV 2021, should be discussed and added to the experimental comparison. The work is similar to the strongly related work SSUL [6] and therefore appears to be relevant also for this work.

5. The conclusion mainly repeats the abstract. It would be beneficial to put a bit more focus on future works, larger impact of this method on the field of semantic segmentation, or limitations of the method.

questions: 1. Could you state from where you adopted the knowledge distillation loss in Eq. 5? Also from SSUL [6]?

2. Do the 5 different runs use different random seeds?

limitations: Limitations have been discussed to some degree in the supplementary. I think there are, however, two limitations which have not been clearly stated. The method requires modifications to the semantic segmentation model, which makes the method not applicable to off-the-shelf models (cf. page 5, ll. 181-185). Also, the assumption that new classes rarely appear in in old data (page 5, ll. 186-188) is true for Pascal VOC, but not for more complex scenes such as Cityscapes, where such classes would often just not be labeled. This might make this method rather applicable to simple scenes. I would be happy to hear the authors’ take on these thoughts.
review_title: null

Meta-review: metareview: This submission deals with incremental semantic segmentation. The authors propose two contributions: 1/ a new knowledge distillation loss based on two separate loss functions for positive and negative class logits; 2/ a dedicated initialization strategy for the classifiers of novel classes. They present strong results on several standard benchmarks for incremental semantic segmentation.  
This submission received initial diverging ratings. Reviewers have raised important concerns about the strategy and some missing experiments. After rebuttal, the active reviewers appreciated the answers and additional experiences provided. Following discussions, the final scores of active reviewers have increased and are clearly positive on this submission.  

On the whole, the novelty and the interest of the proposal stand out clearly. The AC agrees that the strengths in this case outweigh the weaknesses. Authors are encouraged to consider all comments for their final version.
confidence: Less certain
Decision: Accept