IPSeg: Image Posterior Mitigates Semantic Drift in Class-Incremental Segmentation

Authors: Xiao Yu, Yan Fang, Yunchao Wei, Yao Zhao
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: incremental learning, semantic segmentation

Abstract:
Class incremental learning aims to enable models to learn from sequential, non-stationary data streams across different tasks without catastrophic forgetting. In class incremental semantic segmentation (CISS), the semantic content of the background class changes across incremental phases, which is known as \textbf{semantic drift}. Our research identifies two severe issues within semantic drift: separate optimization and noisy semantics, which significantly degrade CISS performance. Based on this insight, we propose a simple yet effective method, \textbf{I}mage \textbf{P}osterior and Semantics Decoupling for \textbf{Seg}mentation (IPSeg), designed to address these challenges through two specific mechanisms. First, IPSeg leverages image posterior probabilities as guidance to resolve the separate optimization issue. Second, IPSeg utilizes semantics decoupling to effectively handle noisy semantics and tailor the learning strategies for different types of knowledge. Experiment results on the Pascal VOC 2012 and ADE20K datasets demonstrate superior performance compared to previous state-of-the-art approaches, particularly in more realistic and challenging long-term scenarios. Furthermore, IPSeg exhibits excellent properties in terms of both learning plasticity and memory stability.

Decision: reject
Reviewer scores: 8, 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730585629313

thread_type:
official_review

<thread_object>
creator: reviewer_fu4w

<content>
summary:
To address the challenging Class incremental semantic segmentation (CISS), this paper proposes Image Posterior to attack the noisy semantics and Semantic Decoupling to cope with separate optimization, which both of these two issues refer to semantic drift and background shift problems in CISS. The proposed Image posterior supports with guidance to calibrate the error prediction and amplify the scale of correct prediction, hence solving the semantic drift issue when the model faces ambiguous categories between the past task and the current task. Semantic Decoupling implemented by Memory Bank manner is used to decouple the pure background, unknown foreground, past class pixels, and target class pixels. The authors employ a saliency estimator and filtering trick to improve the memory buffer while the memory buffer is an image-level corpus.

strengths:
1. The paper writing is kind of good.

2. Supporting image posterior within the image-level to guide the CISS task sounds reasonable to resolve the semantic drift and works like error calibration.

3. The final CISS experimental results seem competitive and the ablative studies are extensive.

weaknesses:
1. Decoupling the background into pure background and unknown foregrounds (utilizing the saliency model), I believe, has been widely studied. The decoupling implementation sounds complicated to understand in the main paper, but I think the authors could provide a more clear presentation of this part. I think the authors should make detailed comparisons with existing works, like Ssul.

2. Not sure how the authors construct the memory buffer with image-level samples. This plays an important role in this paper to obtain good guidance.

3. The figure presentations are inconsistent, Figure shows good quality while figures 2/4/5/6/7/8 display a blur effect. These should be presented by pdf, etc.

questions:
Please refer to my weaknesses part. Moreover, the authors shall further make some comparisons in terms of training efficacy with the previous method, since they use a memory buff and constitute a two-branches architecture.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615957570

thread_type:
official_review

<thread_object>
creator: reviewer_k5js

<content>
summary:
This paper addresses the challenges of class incremental semantic segmentation (CISS), where models learn from sequential tasks with changing background semantics, a phenomenon known as semantic drift. The authors identify two key issues—separate optimization and noisy semantics—that significantly hinder CISS performance. To tackle these issues, they propose Image Posterior and Semantics Decoupling for Segmentation (IPSeg), which employs two main mechanisms: (1) Image Posterior Guidance: IPSeg uses image-wise posterior probabilities to guide pixel-wise predictions, mitigating separate optimization issues. (2) Semantics Decoupling: Noisy semantics are split into two groups, stable, static semantics and dynamic, temporary semantics, each handled by separate branches with distinct life cycles. Experiments on the Pascal VOC 2012 and ADE20K datasets show that IPSeg outperforms existing approaches, particularly in challenging long-term scenarios, and demonstrates improved balance between learning plasticity and memory stability.

strengths:
## 1. Strong Motivation

The paper presents a well-defined and convincing motivation for addressing Separate Optimization, a challenge where earlier-trained task heads can produce disproportionately higher scores compared to later-trained heads for visually similar categories. This issue is thoroughly analyzed, with compelling visual support in Figures 1, 5, and 6.

## 2. Comprehensive Analysis

The authors provide a detailed ablation of each proposed component, supported by robust quantitative and qualitative analyses. This thorough exploration effectively demonstrates the contribution of each component to the overall method’s performance.


## 3. State-of-the-Art Performance

The proposed method, IPSeg, demonstrates state-of-the-art performance on the VOC2012 and ADE20K datasets, with results that convincingly surpass existing approaches.

## 4. Clear and Structured Presentation

The paper is well-written and carefully structured. The problem setting and motivation are clearly articulated, and each proposed component is thoroughly explained and ablated, enhancing the paper’s clarity and rigor.

weaknesses:
Most of my concerns raised are addressed within the paper’s appendix, which provides comprehensive additional supportive analysis.

## Recommentation
I conclude that this paper is solid and compelling, meeting the high standards expected for ICLR. My initial recommendation is to Accept. I will finalize the rating after a discussion with the authors and other reviewers.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659751434

thread_type:
official_review

<thread_object>
creator: reviewer_wqer

<content>
summary:
The paper proposes IPSeg, an innovative framework for addressing semantic drift in Class-Incremental Semantic Segmentation (CISS). IPSeg leverages two main strategies: (1) Image Posterior (IP) guidance to mitigate errors from independent task optimization, and (2) Permanent-Temporary Semantics Decoupling to handle noisy semantics. These mechanisms allow IPSeg to retain past knowledge while learning new classes, achieving significant improvements in segmentation performance on Pascal VOC and ADE20K benchmarks.

strengths:
1. This approach enhances pixel-level accuracy by leveraging global image-level predictions. Specifically, by introducing the image posterior guidance mechanism, IPSeg aims to mitigate the separate optimization issue in CISS.
2. The decoupling of stable and temporary semantic components is well-conceived and allows the model to handle both static background and dynamic foreground objects across incremental steps.
3. IPSeg outperforms various state-of-the-art methods across different incremental segmentation scenarios.

weaknesses:
1. The issue of separate optimization has been investigated by [2]. In that paper, the scale inconsistency issue "earlier incremental task heads may have larger output scales than the later heads, especially in similar classes" was alleviated by logit manipulation. Therefore, the claim that "separate optimization does not attract any attention" is inaccurate.
2. The introduction of additional components like the image posterior branch and decoupled semantics increases model complexity. This may hinder its applicability for real-time or resource-constrained applications. Efficiency comparisons, such as computational complexity, iterations required for convergence, and finetuning time cost, should be conducted.
3. Salient object detector is used to find out the foreground regions, which will incur additional computational costs. Moreover, it brings additional information and is unfair to other competing methods that do not use it.
4. The memory buffer may raise scalability and privacy concerns, especially when scaling to larger datasets and storing sensitive user data.
5. This paper missed several SOTA methods [1-2] in experimental comparison.

[1] Yang, Ze, et al. "Label-guided knowledge distillation for continual semantic segmentation on 2d images and 3d point clouds." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Kim, Beomyoung, Joonsang Yu, and Sung Ju Hwang. "ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

questions:
1. The permanent branch aims to learn dummy labels representing “pure” background and unknown objects. However, the definition of unknown objects are ever-changing. For instance, "cat" can be an unknown class in step 1, and then a target class in step 2, and afterwards become a past seen class in step 3. I don't see the rationale why it is called the permanent branch. Generally, only consistent class definitions across all steps can be regarded as permanent semantics.
2. How are the final prediction maps generated from the permanent and temporary branches? The detail procedures should be elaborated.
3. The previous-class image labels are not available at the current step. How to supervise the image posterior in each step?
4. It seems like temporary branches predict pure background and other foreground regions in addition to the target classes at the current step. Does it contribute to the final prediction or just the target class scores are used to multiply by the corresponding confidence score from the image posterior branch?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721631692

thread_type:
official_review

<thread_object>
creator: reviewer_zxha

<content>
summary:
This paper indicates two key issues within semantic drift, separate optimization, and noisy semantics, for class-incremental segmentation. The authors propose a method called IPSeg, including image posterior probabilities and semantics decoupling. This paper conducts extensive experiments on replay-based and non-replay-based scenarios and shows significant improvement under extremely long incremental learning steps.

strengths:
This paper identifies two issues in the semantic drift for class-incremental segmentation and proposes two components for each problem, respectively. Extensive experiments demonstrate the effectiveness of the proposed method.

weaknesses:
The writing and clarification could be simplified in Section 3.3 and Section 3.4, especially for Equation 5. In Figure 2, the targets of the permanent and temporal branches are not clear.

questions:
- This paper uses image posterior probability to rectify the class prediction. However, the subnetwork for image-level classification is trainable during different steps. I wonder why the classification subnetwork does not suffer from catastrophic forgetting. Is there any evaluation of the performance of the classification subnetwork during multiple steps?
- In Figure 2, the feature extractor is frozen.  What is the pre-trained knowledge of the backbone from, the first training step or the ImageNet? 
- The mechanism of image posterior probability is similar to the way used in transformer-based mask prediction, like Incrementer [1]. Is there any comparison and discussion with it? 
- Because this work modifies the inference procedure of the mask, please report the training and inference costs, such as training time and memory.
- This method uses salient maps derived from an off-the-shelf model as one of the targets. The salient maps are suitable and useful for the VOC dataset because most objects can be detected by the salient model. However, for the more challenged dataset, ADE20K, many semantic regions can not be identified by the salient model, such as grassland in the supplementary material Figure 9. Please study the effect of the salient maps supervision and the quality of the salient maps on the ADE20K dataset. A suggestion: you may replace the salient maps with the results of the SAM model [2].
- In Table 2, why is the replay-based IPSeg (33.6) worse than data-free-based PLOP+NeST (34.8)? Is there any insight or explanation?

[1] Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class, CVPR 2023
[2] Segment Anything, ICCV 2023
</content>
</thread_object>

</thread>
<thread>
creation date:
1732293730690

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all Reviewers

comment:
We thank all reviewers for their great efforts and constructive comments, which help us further improve our work. This official comment contains two part: 1. The analysis on training and inference cost. 2. The summary of our manuscript modification.

---

## 1. Analysis on training and inference cost 
Here, we provide a comprehensive analysis of the model parameters, training, and inference costs. We test and report the results of IPSeg, SSUL-M and CoinSeg-M with Swin-B on the VOC 15-1 setting. We set *image_size=512x512, epochs=50, and batch_size=16* in training and *image_size=512x512* for inference. All results are run on RTX 3090 GPU.

1. **Model Parameters**: Using the thop tool, we analyze and compare the trainable parameters for these methods. The increase in parameter sizes is similar across them, with an average of $3.84M$ additional parameters per step. Additionaly, IPSeg has $29.72M$ parameters more than SSUL-M due to the additional image posterior branch.

| Step | 0 | 1 | 2 | 3 | 4 | 5 |
| :-------- | :--------: | :--------: | :--: | :--: | :--: | :--: |
| IPSeg     | 135.92 M    | 139.76 M   | 143.60 M | 147.66 M| 151.28 M| 155.12 M|
| SSUL-M     |  106.20 M   |110.03 M     | 113.89 M|117.95 M | 121.56 M| 125.40 M |
| CoinSeg-M | 107.02 M | 111.15 M | 115.29 M | 119.42 M | 123.55 M | 127.68 M |


2. **Training cost**: The training time and GPU Memory usage of these methods are shown in the table below. Due to the introduced image posterior branch, IPSeg needs more training cost compared with SSUL-M but less than CoinSeg-M.

| Method | Time | GPU usage |
| :-------- | :--------: | :--------:|
| IPSeg     | 9h 14min     | 21.1G     |
| SSUL-M     | 7h 13min    | 19.4G     |
| CoinSeg-M | >15h         |21.3G      |

3. **Inference**: The inference speed (FPS), flops, and cost are shown in the table below. The inference speed of IPSeg ($27.3$ FPS) is slightly lower than SSUL-M ($33.7$ FPS) and similar to CoinSeg-M ($28.2$ FPS). Due to the proposed image posterior branch, the model's floating-point operations ($137.1$ GFLOPs) are higher than the baseline ($94.9$ GFLOPs), and with an approximately $1$ GB increase in GPU usage. 

| Method         | FPS      | FLOPs | GPU usage |
| :-------- | :--------: | :--:    | :--:        |
| IPSeg    |     27.3 |137.1G |       6.2G|
| SSUL-M   |     33.7 |94.9G  |       5.3G|
| CoinSeg-M|     28.2 |96.3G  | 5.6G      |

Overall, IPSeg introduces an additional image posterior branch with slight increases in model parameters, training and inference costs but brings great performance improvement. It is a worthwhile trade-off between performance and cost.

---

## 2. Manuscript Modification
To provide clearer insight into the revisions we make to our paper and the experiments conducted in response to the reviewers' feedback, we summarize the changes during the rebuttal period as follows：

### Additional analyses:
* We conduct a quantitative evaluation of image poserior branch on different settings. The results show that image posterior branch has excellent resilience against catastrophic forgetting. (reviewer zxHa Q1)
* We provide a comprehensive analysis of the model parameters, training, and inference costs of IPSeg compared with previous works. The results show that IPSeg introduces slight increases in model parameters, training and inference costs but brings great performance improvement. (reviewer zxHa Q4 and reviewer wQeR W2 and reviewer FU4w Q4)
* We conduct ablation study on different salient map. The results show that the default saliency map struggles with identifying "Stuff" classes, and high-quality salient map can obtain better performance. (reviewer zxHa Q5)

### Clarification:
* More details on the process to get the final prediction maps are provided in L269-L275. (reviewer wQeR Q2)
* More details on the construction of memory buffer are provided in the appendix. (reviewer FU4w Q2)
* Additional SOTA methods comparisons are added in Table 1 and Table 2. (reviewer wQeR W5)
* Table 2 is reorganized and the results of "IPSeg w/o M" are added for comprehensive and fair comparison. (reviewer wQeR W5, latest revision)

### Correction: 
* A typo is fixed in Equation 3. ($\phi_{1:T}$ --> $\phi_{0:T}$)
* The statement in L156 is revised by adding previous work. (reviewer wQeR W1)
* All figures are replaced with PDF versions for better presentation. (reviewer FU4w Q3)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses key issues in the task of class-incremental semantic segmentation, focusing on two main issues i.e., semantic drift and separate optimization that hinder performance. The proposed IPSeg method introduces two primary innovations: (1) Image Posterior Guidance, which leverages image-level classification to rectify segmentation errors and mitigate separate optimization, and (2) Permanent-Temporary Semantics Decoupling, aimed at distinguishing stable background and dynamic instance semantics. Extensive experiments on Pascal VOC and ADE20K datasets demonstrate that IPSeg achieves notable performance gains over SOTA methods, especially in long-term incremental learning scenarios.

Strengths:
The paper clearly identifies and addresses the issues of semantic drift and separate optimization, which are critical for improving CISS. IPSeg’s use of image posterior probabilities to guide segmentation is reasonable and is supported by extensive ablation studies and performance benchmarks. The experimental results consistently show that IPSeg outperforms baseline and competing methods across different datasets and settings. The authors' thorough responses to reviewer questions demonstrate the method’s robustness, particularly in scenarios with memory buffers. While the abstract may be somewhat confusing, the overall writing is generally clear in showing the main ideas and implementation.

Weaknesses:
IPSeg introduces several modules to address the observed issues, however, their entanglement makes it difficult to isolate the contributions of each component, limiting the clarity of the paper’s novelty and potential to inspire future work. The overall approach, though effective, is not entirely novel. Experimental results indicate that IPSeg performs well with a memory buffer; however, its performance without memory (IPSeg w/o M) is less competitive, lagging behind methods like LGKD+PLOP in data-free settings (noted by Reviewer wQeR). This discrepancy raises doubts about the core hypothesis that image-level classification mitigates forgetting better than segmentation. Additionally, the computational cost of IPSeg is higher than that of some baseline methods. Reviewers also highlighted the complexity of certain sections (e.g., semantic decoupling), suggesting that these areas could benefit from simplification and clearer comparisons with existing methods tackling similar challenges.

Decision:
The submission received mixed ratings, with two positive and two negative reviews. Reviewer k5js provided positive feedback but their comments were general, thus I slightly down-weighted of their evaluation in the final decision-making. Although the paper presents strong experimental results across multiple datasets, significant concerns remain regarding the novelty of IPSeg’s design and the empirical validation of its core components. As a result, the paper in its current form may not be accepted this time.