Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper describes an approach to Incremental Semantic Segmentation (ISS) of images. The authors provide a detailed and critical analysis of the Calibrated Cross Entropy (CCE) and Calibrated Knowledge Distillation (CKD) losses from the MiB approach [2], and from this analysis derive a new calibrated loss (called ALI) that combines the stability benefits of CCE with the forward transfer benefits of CKD. The authors additionally propose a three-stage incremental learning approach that first fits to a new task using ALI and (uncalibrated) CE and KD losses, then fits a rotation matrix to align memorized features with the new feature space, and finally fine-tunes the new task classifier on the current task samples and the newly rotated and memorized features from previous tasks. An extensive experimental evaluation is given on PASCAL VOC and ADE20K.

strengths_and_weaknesses: # Strengths

+ **Clarity**: The paper is generally very well-written and the main contributions are clear and understandable. The paper was a pleasure to read and I am convinced it adds something interesting to the discussion on incremental learning. 
+ **Motivation and technical development**: Similarly, the main technical motivations and developments are very clearly described, and the gradient analysis of CCE and CKD (as well as the proposed ALI) losses are quite compelling. The proposed approach is described in detail and I feel like the results presented would be easy to reproduce from the technical descriptions in the paper.
+ **Experimental evaluation**: The experimental evaluation is also quite convincing -- although see *Weaknesses* and questions below for some issues. The proposed approach demonstrates clear advantages over the state-of-the-art, both in terms of accuracy and memory efficiency. 

# Weaknesses

+ **Steps**: Though the paper is well-written, there are quite a few moving parts and I feel that the generic use of "Step 1" and "Step 2" does some disservice to putting all of the pieces together into a coherent, global picture of the proposed approach. The approach seems more clearly articulated in **three** stages (names improvised): New Task Acquisition (currently Stage 1), Drift Compensation (via rotation), and maybe Current Task Fine-tuning.
  
+ **Detailed comparison with CKD + CCE**: This seems like a very important comparison as one of the main novelty claims is improvement over MiB which proposed these calibrated losses. The only direct comparison with these seems to be in Fig. 1, and thus gets kind of forgotten when arriving at the experimental evaluation. Moreover, the terminology is somewhat unclear: the proposed approach, I think, should be CE+KD+ALI (eq. 10).

+ **Ablations**: The feature alignment rotation is not ablated, moreover it would be interesting to see the ablations on ADE20K in attempt to understand why replay adds so little.

questions: + **Improvement of S2 over S1**: on ADE20K the improvement of adding replay (and rotation and fine-tuning) is very marginal compared to that seen on PASCAL. This is also evident in the poor improvement (or even deterioration as noted in the paper) of SSUL-M with respect to SSUL. Do you have an explanation for this? Why would replay perform so poorly in some cases compared to others?

+ **mIoU scores over all**: I found the discussion of the drawbacks of the mIoU over *all* categories a bit hard to parse. Are the columns in Tables 3-4 the mIoU over all categories, as commonly reported in the literature? This should maybe be made more clear to facilitate comparison with existing work.

limitations: The authors include a thoughtful discussion of technical limitations in the Supplementary Material. I think something could and should be said about potential risks of incremental learning as well -- for example, a discussion of the dangers of biases becoming *baked in* to incrementally-learned representations, and the difficulty of eliminating them.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the question of incrementally adding new categories in a semantic segmentation problem. This is a variant of the more standard class incremental learning problem where data are annotated with the new categories only in each incremental session and may contain unannotated regions from seen or unseen categories.

The proposed approach makes two contributions: definition of a new term in the loss for regularization that takes into consideration the various types of data, and alignment of the memorized features from previous sessions to the new features using a category-specific rotation estimated from a criterion combining discriminating capacity and similarity with previous features. The approach is evaluated on two standard benchmarks in this field (PASCAL VOC and ADE20k).

strengths_and_weaknesses: Strengths

- Good and thorough discussion of loss terms in order to handle the various types of data and annotation state.  
- Nice proposition to incrementally adapt memorized features using global alignment by rotations.
The ablation study, experiments and result analysis justify the approach on standard benchmarks.

Weaknesses
- Incremental work. Evolution of classical concepts of incremental learning: knowledge distillation and data replay.
- The maximal number of incremental stages is limited to 5: other papers (SSUL [3]) use more stages (until 11). Since the number of incremental steps is a critical parameter of incremental learning, in general, it would be fair to assess the approach on similar grounds.

questions: - It was not clear where the performances of competing approaches (SSUL, PLOP, etc.) come from. Did you re-implement the methods or take the number from the papers? Are the backbones identical?

- How can it be justified that the group of rotations, which are global isometries, is a good set of transformations ensuring feature alignment? How do the features behave when the number of incremental stages is large (I suspect that new features not only isometrically shift but may also be rearranged in a non-uniform way)? Maybe a T-SNE plot would give some idea about what is happening during sessions.

limitations: Non applicable
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time.
To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques are effective, there is still lack of theoretical support. Replay-based methods propose to memorize a small set of images for previous categories. They achieve state-of-the-art performance at the cost of large memory footprint. The author propose in this paper a novel ISS method, dubbed ALIFE, that provides a better compromise between accuracy and efficiency. To this end, it first  show an in-depth analysis on the calibration techniques to better understand the effects on ISS. Based on this, it then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones. It also present a feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements significantly. Since a feature extractor is changed continually, memorized features should also be updated at every incremental stage. To handle this, it introduce category specific rotation matrices updating the features for each category separately. The paper demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks.
strengths_and_weaknesses: Strengths of this paper are as follows:
1. The paper shows an in-depth analysis of probability calibration methods widely used for ISS, and introduces ALI that enables our model to better learn new categories, while maintaining the knowledge for previous categories.
2. The paper present a novel replay strategy using category-specific rotation matrices, which helps to alleviate catastrophic forgetting for previous categories with much less memory requirements than replaying raw images.
3. Extensive experiments demonstrate the effectiveness of our approach to using ALI and replaying  features with rotation matrices. It set a new state of the art on standard ISS benchmarks.

Weakness of this paper are as follows:
1. The proposed method  requires more memory than other similar methods [1, 4, 9] that do not adopt the experience replay.
2. The proposed method could not achieve the best performance on benchmark in Table 4, though the author claims that the proposed method need far less memory. But is it possible to increate the memory to achieve the best performance?
3. in line 137, the classifier weight is learnable, right? or it's fixed.
4. in line 147, how does the labeled regions R_new mean? is any region of image labelled and other regions not labelled? why there is labelled region and unlabelled regions? is it because the labelled regions are previous categories indicated region?
5. in line 264, in the fine-tuning step, is the rotation matrices fixed or still learnable?

 
questions: questions are put in Strengths and Weakness.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method for class-incremental learning in semantic segmentation. First, in a theoretical analysis of the gradients induced by commonly used loss functions are analyzed and several shortcomings are pointed out. As a way to mitigate these problem, a new knowledge distillation loss dubbed adaptive logit regularizer (ALI) and a feature replay strategy are proposed. Experimental evaluation shows the superiority of ALI over previous (non-replay-based) approaches in several experimental settings and competitive performance compared to other replay-based approaches, while having a much lower memory footprint.
strengths_and_weaknesses: Strengths:

1. Strong motivation of the method in the introduction and through the theoretical analysis of previously used loss functions. 

2. Well-written related work section, which is informative in putting the paper into the larger context of incremental semantic segmentation. The novelty of the method is clearly stated and discussed w.r.t. other works.

3. Section 3.2 provides an overall very clear and informative mathematical description and analysis of de-facto standard loss functions in incremental semantic segmentation.

4. The experimental results are strong compared to previous methods and appear to be SOTA for non-replay-based methods.

Weaknesses:

1. CKD analysis: In the analysis of the CKD loss, it is not always clear to me, what the desired gradient behavior is vs. the observed behavior. Specifically, it would help, if on page 5, ll. 180-196, this would be clearly stated. Afterwards, it is also not completely clear to me how ALI solves these issues. This could be added in page 5, ll. 197-208 with reference to the desired behavior in the previously mentioned text part.

2. Loss application of ALI: It is not quite clear to me, why you use the vanilla CE and KD losses in Eq. 10 and also why you apply them specifically in these regions. I think this should be motivated in Section 3.2 and supported by an ablation study later on. In particular an ablation showing the experimental comparison with the commonly used CKD and CCE losses would help to better understand the benefits of ALI. Otherwise, also other factors such as better chosen hyperparameters could be the reason for the observed improvements through ALI.

3. Hyperparameter sensitivity. The method seems to contain a lot of hyperparameters, which even have to be determined for every incremental stage separately (cf. Table B in the appendix). I think this might limit the usability of the method, as it is computationally very expensive to run a grid search for each new incremental step. Maybe the authors could elaborate on this point?

4. Experimental setting: The approaches SSUL and RECALL compare on Pascal-VOC on the 10-1 setting and show particularly strong results in this setting. I think it would be better to evaluate this replay-based approach also in the most challenging settings introduced by other replay-based methods in incremental semantic segmentation.
 
5. Hyperparameter ablation studies: The influence of the different hyperparameters of the method is not shown. Table B of the appendix shows that different hyperparameters are chosen for each setting, so it would be good to know the influence of these hyperparameters on the final performance.

Overall, I really like the approach taken in this work. I also feel, the contributions contain a lot of value to the incremental and continual learning community. I would like to raise my rating, if my concerns are addressed satisfactory in the rebuttal.

Minor comments and suggestions:

6. Page 2, ll. 44-45: I think the CCE and CKD losses are indeed theoretically motivated. Rather than saying there is lack of theoretical support, I would rather rephrase to saying that the conducted theoretical analysis points out shortcomings which are addressed in this work.

7. Page 4, ll. 146: There might be a superscript t missing in the symbol R_new.
 
8. Page 4, ll. 162-166: Although it might appear trivial, it would be helpful to remind the reader that optimization is carried out in negative direction of the gradients shown in Table 1.
 
9. It would help a lot to state the dimensions of the features used for replay, mentioned on page 6, ll. 223-225 and in Eq. 12.
 
10. Eq. 12: The correlation score and the item index s have the same symbol, which is a bit confusing. Also, is the dot operator a scalar product between two feature vectors? Maybe this could be clarified.

11. I think the bold-face highlighting in Table 3 and 4 is a bit misleading. I would propose to compare replay-based methods and non-replay-based methods. In this sense, you would still have SOTA results for non-replay-based methods and competitive results among replay-based methods. But it seems unfair to compare Ours-S2 with, e.g., MiB or PLOP.

questions: 1. Eq. 9: I am not quite sure how this equation is derived from Table 2. Could the authors elaborate?
2. Have you varied the random seed between the three experiments that you average over?
3. Did you determine all method hyperparameters by cross-validation, in particular during the mentioned grid search mentioned in Appendix B?

limitations: Limitations have been discussed to some degree in the supplementary. However, I feel the hyperparameter sensitivity might be an additional limitation.
review_title: null

Meta-review: metareview: This work deals with incremental semantic segmentation. The authors propose a three-step incremental learning approach. They provide an in-depth analysis of the probability calibration methods widely used for the ISS, and introduce an interesting proposal for incrementally adapting the memorized features using global alignment by rotations. They show strong results on standard benchmarks for incremental segmentation, including the ablation study.  

The rebuttal provides valuable insight, and the questions raised by the reviewers have been convincingly answered by the authors.  
On the whole, the reviewers converged positively, the novelty and the interest of the proposal stand out clearly.  
Authors are encouraged to consider all comments for their final version.

confidence: Certain
Decision: Accept