Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a weakly supervised approach that leverages semantic relations to transfer the previously learned semantic class knowledge into the new class. The main motivation is the similar class often has closer distance in the feature space. Extensive experiments are conducted to validate this method.
strength_and_weaknesses: Strength:
1) The idea of using the semantic distance as a guidance is straightforward and has been verified effective. 
2) The experimental evaluation is sufficient and reasonable.

Weakness:
1) The proposed method could not deal with the case that new class has rare similarity with existing classes. In addition, how to calculate the semantic distance is not clear and need more discussion.
2) The overall method is based on WILSON, which also weaken the contribution.
clarity,_quality,_novelty_and_reproducibility: 1) The paper writing and organization is good, which is clear enough for understanding. 
2) The writing quality and experimental evaluation is good.
3) Novelty and Reproducibility is ok.
summary_of_the_review: Overall, this is a good paper with a novel class incremental learning that using semantic priors. Based on the strengths and the weakness, currently, the reviewer suggests a positive rating to this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a weakly-supervised incremental semantic segmentation strategy, which starts from a base learning task with fully-annotated images and then incrementally learns novel classes with image-level labels only. To tackle this problem, it introduces a pseudo label generation method for the weakly supervised semantic segmentation (WSSS) at each incremental stage, which infers the pseudo labels of the new classes based on the model prediction of the old classes and the semantic similarity between new & old categories in a word embedding space. The proposed WSSS loss is then integrated into an existing weakly-supervised incremental semantic segmentation framework,  WILSON,  in order to build a full pipeline.  The authors evaluate the method on the PASCAL VOC dataset and COCO-to-VOC benchmark with comparisons to prior methods.   
strength_and_weaknesses: Strengths:
- The problem setup of this work, which requires only weak annotation in incremental semantic segmentation, is useful in practice but less explored. 
- The idea of exploring semantic similarity for forward transfer seems novel for the task of weakly-supervised incremental segmentation. 
- The paper is mostly well-written and easy to follow.    

Weaknesses:
- The overall novelty of the proposed framework is limited. It heavily relies on the existing framework WILSON and the main contribution is an improved WSSS loss integrated within each incremental step. 
- The assumption on the background class is problematic for incremental segmentation. This work seems to assume a static background class, which is not true in this task. Due to this background drifting issue, the proposed pseudo label generation strategy seems less effective: In equation (2), the bkg class would have a high similarity to previous bkg class which includes novel classes. This would largely suppress the new classes and conflict with the CAM loss. To make this work, it seems to require a good base segmentation network to generate foreground masks, which is difficult for the incremental setup. 
- The experimental results are mixed or marginally improved compared to the WILSON framework. On some task settings of the VOC benchmark, the overall improvements are around 1% and on some task settings of the COCO-to-VOC benchmark, the proposed method is inferior to the baseline. 
- The experimental evaluation settings are limited. All of them start from a base learning task with many classes (15 or 60) and incrementally learn a smaller number of classes (5 or 20). It should be evaluated on more challenging settings with fewer base classes and more novel classes. Also, the evaluation on larger datasets in ISS, such as ADE, should be added.     
clarity,_quality,_novelty_and_reproducibility: See above for the evaluation of the clarity and novelty.
summary_of_the_review: The proposed weakly supervised ISS method seems less convincing due to the limitations on its novelty and pseudo-label generation, mixed performance, and the lack of sufficient experimental evaluation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new loss for weakly supervised (image-level labels) class-incremental segmentation that enables taking into account semantic relations between past classes and new classes. The proposed approach can be integrated to existing semantic image segmentation approaches that used a localizer module. In the paper, they integrate the loss to the recent WILSON framework. Experiments on Pascal VOC and COOC-to-Voc with different incremental settings are promising and show the interest of the approach.


strength_and_weaknesses: **Strengths** 

+ Weakly supervised class-incremental segmentation is very relevant and important and only a few approaches have been proposed in the literature
+ The idea of using semantic relations between past classes and new classes makes sense.
+ The paper is easy to follow and well-structured.
+ Experiments and comparison to the state of the art are promising.

**Weaknesses** 

+ My main concern is related to the novelty of the approach which just consists in adding new semantic losses in the WILSON framework and is thus marginal.
+ Moreover, the choice and design of the semantic losses are very simple and could be more discussed. For instance, how to take into account stronger relations between classes than just embedding similarity ones? For instance, in semantic segmentation part-whole relations or spatial relations between classes could also be of great interest and the proposed approach can not differentiate between the different types of relations.
+ Since the claim of the paper is to use conceptual relations between past and new classes, I would appreciate more experiments focusing on these specific points: for instance a setting with explicit relations between the different incremental steps and various configurations: specialization relations, generalization relations, mereo-topological relations... 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow.

As I stated earlier, the novelty is only marginal: integration of a simple semantic loss into the existing WILSON framework.

The paper seems to be easily reproducible.
summary_of_the_review: The paper tackles the relevant and important weakly supervised (image-level labels) class-incremental segmentation problem. It proposes to integrate a semantic loss term into an existing framework that improves the state-of-the-art but the novelty of the proposed approach is only marginal and important component of the proposed approach such as the notion of semantic priors is not deeply studied.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The proposed approach address the problem of class-incremental for weakly supervised semantic image segmentation. The goal of this paper is to leverage the semantic relations among the labels to improve the weakly supervised learning for semantic segmentation. The idea is to generalize the WILSON approach for weakly semantic image segmentation. A new loss relation aware semantic prior loss (RASP) is developed to encourage the forward transfer across classes within the learning sequence. The proposed loss is validated in continuous learning setting.
strength_and_weaknesses: Pros

The approach naturally generalized the prior works on image-classification to the weakly semantic image segmentation, which is novel and inspiring. I think approaching the problem from the loss perspective is reasonable.

The proposed approach seems to be working well on the proposed settings. The proposed approach shows good results in the settings of effect of memory and few-shot. 

Cons

The proposed approach is relative not incremental on top of the WILSON project. As the main idea in this paper is to add the RASP loss on top of the WILSON framework. However, it does not seem that the additional RASP loss adds the big advantages to the WILSON framework. More experiments should be provided to justify if the additional useful signals are provided from the localizer. Specifically, we need to see improvements of the slice and dice metric improvements of the new classes once we have applied the RASP loss and use the information from the old class. The Baseline in that setting is that we are not using RASP loss, but just run CRF-like approach. 

Another question is about the RASP loss. There is no guarantee that the localizer for old tasks is providing the positive impact to the new tasks, unless we have made some careful choice on the old tasks and new tasks, which is not possible in the real-world use cases. How would one use the proposed approach in practice?

One design choice needs to be justified. In particularly, the proposed approach chooses to treat the new class region as background and build the proposed approach on a localizer module trained with weak annotations. In contrast, there are multiple approaches in literatures in salient region detection that shows promising results in segmenting the background, and plenty of the approaches using super-pixels. It is not clearly why the proposed approach pick up the route, which seems to be bias towards to give better thing-like object instance segmentation. The proposed technical choice might not work well on the stuff-like objects instances, which has extended boundaries. These are missed out in the discussion.

The experiments are mostly focus on the easy datasets such as Pascal and COCO. For the proposed to work, I would like to see how the proposed approach work on a more challenging dataset. The simulated experiments on new and old classes seem to be cherry picked.
clarity,_quality,_novelty_and_reproducibility: Clarity
The paper does do well in present the core idea of the proposed approach. However, it is not very clear about the key difference between the proposed approach and the one from WILSON. 

Quality
The paper is presented in a good quality. The experiments have also demonstrated the effectiveness of the proposed approach in several continuous learning setting. 

Novelty
The paper has a incremental contribution on top of the WILSON work.


Reproducibility
The paper seems to be easily reproduced.
summary_of_the_review: On the positive side, I think the proposed approach has good spirit as it has shown clear improvements on class-incremental setting for weakly semantic image segmentation. The proposed approach has also touched the drawbacks of the WILSON framework, and added the new loss RASP which encourages the forward transfer across classes within the continuous learning setting. However, the paper does not seem to show clear improvements. The aggregated metric improvements on the benchmarks do not guarantee that the improvements are from the proposed RASP loss. There are other technical design choices are not well justified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary of Contribution

This paper describes a method for weakly-supervised class-incremental learning for continual semantic image segmentation (called RaSP). The authors propose to use image-level labels and pseudo-labels inferred using the previous model (and similarity of new classes and old measured in an embedding space) to perform class-incremental semantic segmentation. The proposed approach is integrated into the WILSON class-incremental semantic segmentation model for testing. Experimental results are given on PASCAL VOC and MS-COCO for a variety of class-incremental learning scenarios.

# Strengths

+ **Relevance**: The problem of weakly-supervised, incremental semantic image segmentation is an important problem that has relatively little coverage in the existing continual learning literature. Considering longer task sequences is also of relevant interest. The experimental evaluation demonstrates the effectiveness of the proposed integration of semantic similarity into WILSON for most considered scenarios.

+ **Clarity and Reproducibility**: The reviewers are nearly unanimous in their praise of the clarity of technical presentation and believe the results to be reproducible.

# Weaknesses

+ **Novelty**: The main contribution of RaSP is the introduction of semantic similarity maps between new and old classes, which is a relatively minor intervention to the localization prior in Eq. (4) of WILSON (Cermelli et al., 2022). The authors argued during the discussion phase that RaSP is a "plug-and-play" module, easily integrable in any weakly-supervised semantic segmentation model. To effectively claim this, additional experiments integrating RaSP with other state-of-the-art methods would be needed. As presented, it is a minor modification to WILSON.

+ **Marginal Improvements**: While integrating RaSP into WILSON indeed improves performance, most significantly on longer task sequences, these longer task sequences lack broader comparison with the state-of-the-art -- which has been evaluated mostly on 2-task sequences, for which the improvements of RaSP are less significant.

# Summary

This is a well-written paper that makes a contribution to incremental, weakly-supervised semantic image segmentation. However, the incremental nature of the main contribution is not enough to meet the bar for acceptance at ICLR. While the extension to consideration of longer task sequences is appreciated, considering WILSON and WILSON+RaSP in the comparative performance analysis is limits this as a contribution.

justification_for_why_not_higher_score: This is a well-written paper with a crisp narrative and convincing (in some aspects) empirical evaluation. However, the incremental nature of the contribution, combined with limitations of the experimental evaluation, make it fall short of the bar for acceptance in its current form.
Decision: Reject