Title: Continual Gaussian Mixture Distribution Modeling for Class Incremental Semantic Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Class Incremental Semantic Segmentation; Semantic Segmentation; Class Incremental Learning; Gaussian Mixture Models
Abstract: Class incremental semantic segmentation (CISS) enables a model to continually segment new classes from non-stationary data while preserving previously learned knowledge. Recent top-performing approaches are prototype-based methods that assign a prototype to each learned class to reproduce previous knowledge. However, modeling each class distribution relying on only a single prototype, which remains fixed throughout the incremental process, presents two key limitations: (i) a single prototype is insufficient to accurately represent the complete class distribution when incoming data stream for a class is naturally multimodal; (ii) the features of old classes may exhibit anisotropy during the incremental process, preventing fixed prototypes from faithfully reproducing the matched distribution. To address the aforementioned limitations, we propose a Continual Gaussian Mixture Distribution (CoGaMiD) modeling method. Specifically, the means and covariance matrices of the Gaussian Mixture Models (GMMs) are estimated to model the complete feature distributions of learned classes. These GMMs are stored to generate pseudo-features that support the learning of novel classes in incremental steps. Moreover, we introduce a Dynamic Adjustment (DA) strategy that utilizes the features of previous classes within incoming data streams to update the stored GMMs. This adaptive update mitigates the mismatch between fixed GMMs and continually evolving distributions. Furthermore, a Gaussian-based Representation Constraint (GRC) loss is proposed to enhance the discriminability of new classes, avoiding confusion between new and old classes. Extensive experiments on Pascal VOC and ADE20K show that our method achieves superior performance compared to previous methods, especially in more challenging long-term incremental scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper integrates Gaussian mixture distribution for class-incremental segmentation, which tackles the challenge of performance degradation due to overfitting to new classes.\nThe paper proposes modeling the latent class feature distribution as a GMM. It also introduces a dynamic adjustment strategy that iteratively updates GMM parameters based on feature representations from the current model, and a Gaussian-based representation constraint loss that encourages separation between features of new and old classes.\n\nSTRENGTHS AND WEAKNESSES:\n-- Strengths\n\nThe contrastive learning-inspired loss is a reasonable and effective way to preserve class separability during incremental learning, preventing overlap between similar old and new classes.\n\nThe dynamic adjustment strategy is a reasonable design to handle continual drift during incremental steps.\n\nExperiment results on Pascal VOC and ADE20K datasets are promising, especially in challenging long-term incremental settings.\n\n\n-- Weaknesses\n\nAssuming latent features follow a fixed GMM with K=3 components per class is strong. While it may work empirically, it is recommended to at least provide more discussion.\n\nSome mathematical notations (e.g., subscript/superscript usage) are not properly explained. It would improve the paper if the authors included a summary table of all symbols and notations.\n\nIt is not clearly explained how the segmentation network leverages or integrates the GMM features during classification. It is recommended to provide either diagrams or mathematical formulations specifying the corresponding inputs and outputs, and/or any special layer architecture if used.\n\nQUESTIONS:\nPlease refer to the items listed in the weakness section. It would be helpful if the authors could provide an explanation to improve the clarity of the paper.\n\nLIMITATIONS:\nLimitations are discussed at the end of the paper.\nThis paper does not introduce potential negative socialtal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Continual Gaussian Mixture Distribution (CoGaMiD) modeling method for class-incremental semantic segmentation task, which considers the distribution characteristics of class prototypes in Gaussian Mixture Models (GMMs) and updates the prototype distribution of old classes through the Dynamic Adjustment (DA) strategy to solve the problem of distribution shift of old-class prototypes caused by extractor updates. The author conducted extensive experiments on the Pascal VOC and ADE20K datasets to validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problems with the single-prototype-based approach mentioned by the author are reasonable, and the idea proposed by the author to store the distribution of class features is intuitive.\n2. The author's observation on the problem of class feature shift caused by the update of the extractor is insightful.\n3. Strong performance is achieved on several benchmarks.\n\n\n\nWeaknesses:\n1. Is the form of $ L_{grc} $ inconsistent with what the author stated in line 212? The optimization goal of $ L_{grc} $ is more like shortening the distance between new classes and their most similar old counterparts. And there might be some typos, e.g., $ \\mu_{n_o} $ in Eq. 10.\n2. Based on the calculation of the geometric centroid of the old class according to Eq. 9, how to ensure a sufficiently clear decision boundary if some classes are widely distributed in the feature space? Will this also cause overlap?\n3. Is the method of dynamically adjusting the distribution of old classes also considered a form of forgetting?\n4. It is better to clarify how the prototype is used in the continual learning process and the specific setting of the baseline in the ablation studies, which will be more friendly for readers to follow.\n\nQUESTIONS:\nSee weaknesses. I will modify my rate based on the author's responses to the questions in the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoGaMiD, a novel method for Class Incremental Semantic Segmentation (CISS). The core idea is to address the limitations of recent prototype-based approaches that cannot represent the complete features of each category. The method models the feature distribution of each class with a Gaussian Mixture Model (GMM) instead of a single prototype. The additional Dynamic Adjustment overcomes the shifting feature distribution of previous classes. Gaussian-based Representation Constraint (GRC) Loss maintain discriminability between new and old classes. The experiments on Pascal VOC 2012 and ADE 20K demonstrate the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well written and easy to understand. \n- The experiments are solid and the performance is impressive. \n- The application of the Gaussian Mixture Model (GMM) balances the quality of features and the external space of buffer. \n\nWeaknesses: \n\n- Why is the Joint result of CNN different from previous methods? The previous work [1][2] shows that the Joint result of Deeplab-V3 with ResNet-101 is 77.4. \n- Limited intuitive experiment results showing the anisotropy problem. \n- Lack of experiments on computational and memory cost with previous exemplar-based methods. \n\n[1] Cermelli, Fabio, et al. \"Modeling the background for incremental learning in semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Park, Gilhan, et al. \"Mitigating Background Shift in Class-Incremental Semantic Segmentation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\nQUESTIONS:\n- Is this method effective for Mask2Former-based  structures compared to [1][2][3]?\n- Is there any visualization result (t-SNE or segmentation result) to show anisotropy in detail?\n\nLIMITATIONS:\nThe authors have provided a discussion of limitations in Appendix A.6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an novel framework, dubbed CoGaMiD, for class-incremental semantic segmentation. CoGaMiD utilizes a Gaussian Mixture Model to generate features for previously learned classes that may be inaccessible or scarcely available during incremental learning phases. These generated features facilitate training a segmentation model and play a crucial role in alleviating the issue of forgetting prior knowledge of old classes. Additionally, the authors propose a dynamic adjustment strategy to refine these generated features with real features collected during the incremental steps. They also introduce a Gaussian-based representation constraint loss aimed at learning discriminative features for new classes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- CoGaMiD uses GMMs to generate features for previously learned classes, assisting a segmentation model in mitigating forgetting.\n\n**Weaknesses**\n\n- The authors utilize Gaussian Mixture Models (GMMs) to generate features of previously learned classes for the purpose of updating a segmentation model. In recent work, STAR [6], features of old classes are generated from Gaussian-distributed random variables collected from earlier stages. The Gaussian-based Representation Constraint (GRC) bears similarity to the similarity-aware discriminative loss in STAR, as both aim to ensure the means of Gaussians for new classes are distinct from those of old classes. The specifics of the refined knowledge distillation loss $L_{kd}$ in Eq. 11 are unclear. Is this referring to the effectiveness of the Old-Class Features Maintaining (OCFM) loss in STAR? Given these considerations, CoGaMiD appears to be closely related to STAR, with the primary distinction being the use of a Gaussian mixture model as opposed to a single Gaussian model, which suggests that CoGaMiD might be seen as an extension of STAR. It would be beneficial to clearly delineate the differences between CoGaMiD and STAR. Additionally, to demonstrate the effectiveness of using multiple Gaussians, a key distinction from STAR, it would be valuable to present results for different values of K, ranging from 1 to 3, for both the baseline and the final models.\n- The performance observed when training with all samples simultaneously, referred to as 'joint,' represents an indirect upper boundary for the incremental model. In Tables 1 and 2, the results for the 'joint' method appear to be higher than those for DKD [1], a recent approach in CISS which utilizes binary cross-entropy loss, similar to CoGaMiD. To ensure a fair comparison and to highlight the effectiveness of the proposed methods without modifications to the architecture or minor techniques, it would be advantageous to include the reproduced results of some baseline models in Tables 1 and 2\n\nQUESTIONS:\nPlease read the questions in the weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}