Continual Gaussian Mixture Distribution Modeling for Class Incremental Semantic Segmentation
Authors: Guilin Zhu
Runmin Wang
Yuanjie Shao
Wei dong Yang
Nong Sang
Changxin Gao
Venue: NeurIPS 2025 poster ()
Keywords: Class Incremental Semantic Segmentation; Semantic Segmentation; Class Incremental Learning; Gaussian Mixture Models
Primary area: applications
TL;DR: We propose a Continual Gaussian Mixture Distribution (CoGaMiD) modeling method for class incremental semantic segmentation.

Abstract: Class incremental semantic segmentation (CISS) enables a model to continually segment new classes from non-stationary data while preserving previously learned knowledge. Recent top-performing approaches are prototype-based methods that assign a prototype to each learned class to reproduce previous knowledge. However, modeling each class distribution relying on only a single prototype, which remains fixed throughout the incremental process, presents two key limitations: (i) a single prototype is insufficient to accurately represent the complete class distribution when incoming data stream for a class is naturally multimodal; (ii) the features of old classes may exhibit anisotropy during the incremental process, preventing fixed prototypes from faithfully reproducing the matched distribution. To address the aforementioned limitations, we propose a Continual Gaussian Mixture Distribution (CoGaMiD) modeling method. Specifically, the means and covariance matrices of the Gaussian Mixture Models (GMMs) are estimated to model the complete feature distributions of learned classes. These GMMs are stored to generate pseudo-features that support the learning of novel classes in incremental steps. Moreover, we introduce a Dynamic Adjustment (DA) strategy that utilizes the features of previous classes within incoming data streams to update the stored GMMs. This adaptive update mitigates the mismatch between fixed GMMs and continually evolving distributions. Furthermore, a Gaussian-based Representation Constraint (GRC) loss is proposed to enhance the discriminability of new classes, avoiding confusion between new and old classes. Extensive experiments on Pascal VOC and ADE20K show that our method achieves superior performance compared to previous methods, especially in more challenging long-term incremental scenarios.

Continual Gaussian Mixture Distribution Modeling for Class Incremental Semantic Segmentation
Guilin Zhu
Runmin Wang
Yuanjie Shao'
Weidong Yang l**
Nong Sang!
Changxin Gao
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University Science and Technology. School of Information Science and Engineering, Hunan Normal University fgzhu, shaoyuanjie, Yangud, nsang, cgao Auhustthht....m edu. en, runninwang@hunnu. edu.
Vanggudd,
Abstract
Class incremental semantic segmentation (CISS) enables model to continually segment new classes from non-stationary data while preserving previously learned knowledge. Recent top-performing approaches are prototype-based methods that assign a prototype to each learned class to reproduce previous knowledge. However, modeling each class distribution relying on only single prototype, which remains fixed throughout the incremental process, presents two key limitations: (i) a single prototype is insufficient to accurately represent the complete class distribution when incoming data stream for class is naturally multimodal; (ii) the features old classes may exhibit anisotropy during the incremental process, preventing fixed prototypes from faithfully reproducing the matched distribution. To address the aforementioned limitations, propose Continual Gaussian Mixture Distribution (CoGaMiD) modeling method. Specifically, the means and covariance matrices of the Gaussian Mixture Models (GMMs) estimated model the complete feature distributions of learned classes. These GMMs are stored to generate pseudo- features that support the learning of novel classes incremental steps. Moreover, we introduce a Dynamic Adjustment (DA) strategy that utilizes the features of previous classes within incoming data streams update the stored GMMs. This adaptive update mitigates the mismatch between fixed GMMs and continually evolving distributions. Furthermore, Gaussian-bais based Representation Constraint (GRC) loss is proposed to enhance the discriminability of new classes, avoiding confusion between new and old classes. Extensive experiments on Pascal VOC and ADE20K show that our method achieves superior performance compared to previous methods, especially more challenging long-term incremental scenarios. The souree code is available at httph: htmp::://githun /tt/ggittuu..... com/ zhu gl ux/CoGaMiD
Introduction
Semantic segmentation has witnessed remarkable advancements over the decades, particularly with the emergence of deep learning. wide range of architectures, such as convolution neural networks (CNNs) [23] 18 and Transformers [12 26). have been developed to tackle the challenge of semantic segmentation. Traditional semantic segmentation networks acquire the ability handle predefined number of semantic categories by conducting a one-time training process. However, in realistic applications, the trained segmentation model is expected to learn new concepts continuously under the situation that the previously labeled data are not available due to privacy or legal reasons.
Co-corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fine-tuning the old model with new data results in overfitting the new classes quickly while forgetting learned old classes, termed catastrophic forgetting (29
To alleviate this problem, equipping segmentation models with the ability to continually learn new classes from the non-stationary data while preserving previously learned classes has emerged as a pivotal task, known as class incremental semantic segmentation (CISS), Classically, studies in
CISS explore various techniques, such as knowledge distillation [30 44 34], pseudo-labeling
13 50 4. contrastive learning 48
55 49 and proposal generation 52
53 to maintain a balance
between learning new knowledge and preserving old knowledge. Although impressive, these methods
still suffer from catastrophic forgetting, leading to performance degradation in the old classes, especially in the challenging long-tern incremental scenarios.
34 13
55 52]
have adopted a prototype-based technique to incorporate knowledge from previous classes into the new model, achieving good balance between stability and plasticity. In comparison to previous exemplar-replay methods [28] 34 [ these methods effectively
Recent advances in CISS 60
mitigate privacy concerns and storage costs by maintaining a single prototype for each learned class. Although prototype-based replay methods have demonstrated remarkable performance, certain limitations still persist. Firstly, accurately estimating the distribution of each learned class is essential for preserving previously acquired knowledge. However, a single prototype, represented by the mean of the features and the corresponding variances, is insufficient to precisely capture the complete class distribution when the data streams for a class is inherently multimodal. Secondly, the distributions of learned classes may exhibit anisotropy. since the features appear shift in various directions due to the continual updates of model parameters during the incremental process [16]. Nevertheless, the class prototypes are stored in an unaltered manner, which prevents the faithful reproduction of the corresponding distributions in the feature space.
To this end, we propose a novel method, called Continual Gaussian Mixture Distribution (CoGaMiD) modeling, to address the limitations of current features replay solutions from a prototype perspective. Specifically, after each training step, the feature extractor is frozen to extract class features from current training data. The distribution of each learned class is estimated using an independent Gaussian mixture model with multiple Gaussian components. Utilizing these GMMs, CoGaMiD generates more accurate pseudo-feature, that represent the distributions of old classes to support the new model in distinguishing between novel and old classes. During the incremental learning process, the features of old classes may exhibit anisotropy due to the model training, portion of old class distributions will deviate from the saved GMMs, leading to the issue of mismatch. To address this, Dynamic Adjustment (DA) strategy is developed to ensure that the generated pseudo-features can continuously adapt to the distribution anisotropy by updating the stored GMMs with features of previous classes in current data steams. Additionally, maintaining discriminative representations essential for accurately learning and estimating the distribution of novel classes. Therefore, we introduce a Gaussian- based Representation Constraint (GRC) loss to push the features of new classes from the centroids of the similar old ones. The contributions of this work are summarized as:
We present the Continual Gaussian Mixture Distribution (CoGaMiD) modeling, a method that incrementally learns novel classes by generating pseudo-features of learned classes using GMMs. Compared to single prototype-based CISS methods, our approach more effectively reproduces the precise distribution of old knowledge by leveraging the inherent ad advantages of GMMs in modeling distributions from multimodal perspective.
We designed a novel Dynamic Adjustment (DA) strategy to enable the generated pseudo- features continuously adapt to the anisotropic features during the incremental process. Moreover, a Gaussian-basee Representation Constraint (GRC) loss is proposed to maintain discriminative distance between new and old classes.
Through extensive experiments on PASCAL VOC 15 and ADE20K 57 we demonstrate the otate-of-the-art performance of our method across various CISS scenarios, particularly in more challenging long long-term incremental scenarios.

---

2 Related Work
2.1 Class Incremental Learning
Class Incremental Learning (CIL) aims to overcome the limitations of traditional model training by continuously aequiring new concepts while retaining previously learned knowledge. The main challenge that CIL systems will encounter here is catastrophic forgetting [29]. Existing methods primarily address this problem through three main approaches: regularization, replay, and param- eter isolation. In regularization-base
methods, additional regularization terms [21] 39 36 or
knowledge distillation 45 (5 14
introduced prevent significant changes to the knowledge
of previously learned classes in the new model. Replay-based methods store [35] 27 2 or generate [4 [9 a small sample set of previous tasks and replay them when learning the new tasks. Parameter
isolation methods [43 25 20 assign dedicated parameter modules to new tasks, while keeping the
parameters of old tasks fixed be protected from forgetting.
2.2
Gaussian Mixture Models in CIL
A series of studies have investigated the use of Gaussian Mixture Models (GMMs) in the CIL task. pre-assigns multiple virtual embeddings using GMMs to prepare the model for future classes.
[58]
54 introduces GMMs to automatically and simultancously estimate both prototype number and employ GMMs to generate class-specific images. However, their applications have been largely restricted to simpler datasets, such as MNIST [24] [22 proposes the integration of gradient-basee based GMMs with a continual learning framework by using GMMs as classifiers In contrast to the above methods, are the first to leverage the natural advantages of GMMs to preserve learned knowledge while effectively learning new knowledge for CISS tasks.
prototypes.
2.3 Class Incremental Semantic Segmentation
Class Incremental Semantic Segmentation (CISS) is first introduced by 330. which utilizes standard knowledge distillation to alleviate catastrophic forgetting. 4 addresses the unique issue of back- ground shift by designing unbiased function. Since then, various methods have focused on the aforementioned main challenges in CISS. Distillation-basee methods develop the varied distillation
terms [38] 44 (55) to maintain old class knowledge. The pscudo-label strategy [13 3 50 32] is
proven to effectively mitigate background shift. Exampler-based methods employ memory selection mechanism [6]] or generative model [28] to replay old class samples in incremental steps. Some
methods employ additional information, such as saliency maps and mask proposals [5 53 to
reduce semantic ambiguity in background regions.
leverage stored prototypes, where each old class is represented by single prototype, to generate pseudo-features of old classes. Although achieving remarkable performance, the use of a single prototype for class may not be sufficient to capture the complete distribution of that class, because the real-world data often exhibit multimodal characteristics. Meanwhile, the former 7 ignores the issue of mismatch between unchanged prototypes and anisotropic distribution of old classes during incremental processes. The latter [60] alleviates this problem by updating the centroids of the prototypes, but still suffers from limited robustness due the unchanged variances (37. In contrast, precise distribution of each old class using a GMM with natural multiple attributes, and adapts to anisotropic distributions through dynamic GMM updates, striking advantage.
Additionally, prototype-based replay methods 60
method continuously estimates the
3 Method
3.1
Problem Definition
In the class incremental semantic segmentation (CISS) task, model is expected to learn classes over multiple steps, and we assume that there are steps. each step I, the model M comprises a feature extractor I' and a classifier g¹. The dataset D' for training the model consists set of pairs (xr', )²),, where x² is a input image and y is the corresponding segmentation label. Noting that y only cxhibits the labels in current classes C² while all other class (i.e. learned classes clit-1 and
3

---

Input Images D'
Dense Features
Gauxsian Mixture Distribution Modeling
Dynamic Adjustment
Feature Extractor
Stored GMMMs
M-Stip
E-Supp
(8)
Update
Feature Extractor
Dicribution of Old Classes
Iuput Imagea D
Gaussian-based Representation Constraint
Ldd
Lgne
Generate
f
g'
Laveg
Connod
Podina
Repel
Classifer
Figure 1: Overview of our proposed CoGaMiD. (a) Given the dataset D' at step t. after conventional training, we estimate the distribution of classes at this step using the corresponding GMMs, which are optimized by the EM algorithm. (b) The features of old classes, obtained by the current model using old class masks, is applied in the Dynamic Adjustment (DA) strategy update parameters of the stored GMMs. Meanwhile, a Gaussian-basel Representation Constraint (GRC) loss is proposed to maintain the discriminative distance between new classes and old ones.
future classes C++1:T) ) are labeled as background C.. Once the incremental step is completed, the model is required to perform segmentation for all seen classes Cla
3.2 Overview of the Proposed Method
We present an overview our proposed CoGaMiD in Fig. I. After training step f, we freeze the feature extractor I and estimate the distribution of each class in the current set of classes C² using independent multivariate Gaussians based on the training dataset DD. The estimated distributions are then stored as GMMs, denoted as G In the next step + 1, given batch of data, both the previous frozen feature extractor ff and the current trainable feature extractor ft++1 are used simultancously to obtain dense feature representations. Following (5 6 52), the knowledge distillation loss is adopted as a fundamental supervision to preserve old class knowledge in the feature space. The pseudo-feature of old classes, generated by stored GMMs (i.e., g ... G¹), combined with the output features of f serve as input to the classifier g" 1+ The centroids of the new classes are obtained by masked average pooling of the output from f These centroids
then used to compute the representation distances between them and the geometric centroids of the stored GMMs, forming the Gaussian- baased Representation Constraint (GRC) loss. Additionally, we update the stored GMMs using the old class features extracted from the current model at intervals of several training epochs. We refer to this process the Dynamic Adjustment (DA) strategy, which enables the stored GMMs adapt the anisotropic representations of old classes caused by parameter updates in the current model.
3.3 Gaussian Mixture Distribution Modeling and Generation
As previously stated, single prototype in existing CISS methods the complete distribution of each class, particularly when the feature space is inherently multimodal.
inadequate for capturing 60

---

In this situation, the efficiency of the greudo-feature replay strategy is limited. Therefore, we propose to assign GMM for each learned foreground class and store those GMMs for generating pseudo-features in subsequent steps.
Specifically, after training each step 4, the feature extractor f is froze and utilized to extract foreground features from the training dataset D'. Subsequently, Gaussian Mixture Models (GMMs) are employed to estimate the distribution of the current classes C' in the feature space. assumed that the features distribution of each class follows a mixture distribution consisting of K Gaussian components. We denote the k-th mixture component (W Στ). where μ3 D D-dimensional mean vector while Σg is D x D covariance matrix. The feature distribution of each class can be modeled by linear combination of K mixture components:
pa = Σ p(kee me)p(arle Mos Ed)
Σ TaW Hek: Eck), k=1
(I)
where p(k|c; n.) is the prior probability belonging to k-th Gaussians, i.e. Σ Tekk = 1, and cT= Me: Σe} represents the parameters of GMMs. The multimodal nature of GMMs enables the accurate estimation of feature distributions, thereby providing a significant advantage over prototypes that assume unimodality for each class.
Our goal is to find the optimal parameters of the GMMs for each class, i.e., {Øtee= by maximizing the log likelihood over the features this class in the current training dataset D
No
K
arg max
log
p(a,, klc; Φ.),
where N₂ denotes the number of features for class c. We employ the EM SespectationMMaxiiiaan algorithm, which consists of two main steps: the E-step (Expectation step) and the M-step (Max- imization step). to iteratively approximate the parameters of GMMs until convergence to a local optimum. In the E-step. given feature sample I,, the posterior probability generated by k-th Gaussian component is computed as:
Tekk Nax;; Meke Eeck)
Yikk
Σ/=1 Tej N(x;
Mags Eej)
In the M-step, the parameters of k-th Gaussian component are updated based on the above probability:
1
N
EN- No
(a
Ma))( Hcck) 1
Tak
Tike
Mek
Eek
Σ 7/
Next, in order generate the aseudo-features with stored GMMs in the next step need to confirm the proportion of each class. For a foreground class c in the current training dataset Dt, which comprises image-label pairs (ar', yy"), have:
n
kxw
N Σ
5( c].
Here 5(-) is the indicator function, and 0 R denotes the downsampled label aligned with the spatial size of features. With the stored parameters GMMs, { and the statisties, we can generate the pscudo-features of each class for reproducing in the next step by: F. S(d No), where S(.) denotes the Gaussian sampling function. The obtained pecudo-featuree of old classes uniformly assigned in each training iteration, enabling the classifier to capture discriminative characteristics between new and learned classes in the feature space.
3.4 Dynamic Adjustment Strategy
Precise pseudo-features generated by the GMMs are essential for refining the decision boundary. They effectively provide negative samples to support the learning the new classes. During the incremental training process, the model parameters should be updated to accommodate new classes.
5

---

Algorithm 1: Dynamic Adjustment Strategy
Input: Training dataset D extractors f" and f"-1, stored GMMs {ør (Pelem training interval
Output: Updated GMMs [0=] while the end of each interval do
Freeze extractors f" and ft+1; for Ta'yy', ED*+1 do
Compute the mask maps based Eq.
Obtain old class features F. from extractor f++1 based on Eq. end
Initialize GMMs with {Ø {o for с in 1 do Sample peeudo-features Fe =((0% (102%.N0 Nc); while not converged do
10
11
Estimate responsibility 7tk Σi Kay W(lF Fe (e)
Tel Ne F. F. F.Hla E.d) E.t)
/+ The E-step
Update the parameters {Tes Me 2o. by Eq.
/+ The M-step
13
end
/. - aa Me, Doj;
14
15
end
end
In this situation, the features of old classes, obtained from the trainable feature extractor, may exhibit anisotropy [16] 37], wherein the features shift in various directions. This anisotropy results mismatch between the fixed distributions obtained by stored GMMs and the continually evolving distributions of old classes. Consequently, a Dynamic Adjustment (DA) strategy developed to address this issue.
At intervals of each training epochs in step + 1, we keep the extractor f frozen to obtain the features of old classes Clat from the current training dataset Di+1, Firstly, we obtain the masks of old classes by combining predictions of the old model, p', with the current ground truth label +l:
pp,
E ( A (1++1 cc)
6
0, otherwise
where о denotes the background class. Then, given the input images in Di+1, obtain the features of the old classes based on above masks:
F
(F++1 x 8(mi c)...., Ft+1 8(m, c}}-
Here, Ft+1 denotes the features extracted by fi from the n-th input image, and m represents the downsampled mask aligned with the spatial size of the features. We believe that the old class features extracted from the current model intuitively contain the anisotropic distribution, which cannot be accurately reproduced by the stored GMMs. Therefore, we utilize them to scasonably update the stored GMMs using EM algorithm. With continuously updated GMMs, we are able to generate matched pseudo-features that immediately adapt the anisotropic distribution of old classes, facilitating the learning of new classes. The main procedure is summarized in algorithm [
3.5 Gaussian-based Representation Constraint
Above continual Gaussian mixture distribution modeling method addresses the issue of overfitting to new classes classifiers. However, the feature extractor may generate ambiguous features for new and old classes with similar semantics. This ambiguity leads to an overlapping decision boundary, confusing the learning of classifiers. Additionally, the ambiguous features of the new class also adversely affect the subsequent Gaussian mixture modeling. Our goal is to maintain sufficiently discriminative distance berween similar classes in the latent space, Based on the above consideration, introduce Gaussian based Representation Constraint (GRC) loss.

---

Table Quantitative comparison on Pascal VOC 2012 between our method and previous CNN-based based methods (top half) and Transformer-basee methods (bottom half) under the overlapped setting. means results from our re-implementation. $ represents the results are from [53].
Method
VOC 15-1 (6 steps)
VOC
(6 steps)
(11 steps) all
2-2 steps) all
VOC
(20 steps)
0-15
16-20
all
0-5
6-20
all
0-10
11-20
0-2
3-20
0-1
2-20
all
MiB!
35.1
13.5
29.7
57.1
42.6
46.7
12.3
13.1
12.7
41.7
26.0
28.2
38.5
8.1
11.0
PLOP 13]
65.1
21.1
54.6
41.1
23.4
25.9
44.0
15.5
30.5
24.1
11.9
13.7
12.4
11.9
4.7
GSC 10
72.1
24.4
60.8
32.7
30.1
30.9
50.6
17.3
34.7
SSUL 5
77.3
36.6
67.6
72.4
50.7
56.9
71.3
46.0
59.3
62.4
42,5
45.3
52.6
27.5
29.9
DKD
78.1
42.7
69.7
69.6
53.5
58.1
73.1
46.5
60.4
60.5
45.8
47.9
56.1
24.6
27.6
FWF 46
77.7
32.7
67.0
61.7
42.2
47.7
71.5
30.3
51.9
RCIL 11
70.6
23.7
59.4
65.3
41.5
50.3
55.4
15.1
34,3
28.3
19.0
19.4
DEC
77.0
67.1
54.1
70.7
46.3
C 11
77.9
46.4
70.4
58.4
53,4
54.8
74,4
47.2
61.5
STAR 6
79.5
50.6
72.6
71.9
61.5
64.4
73.1
55.4
64.7
59.2
55.0
55.6
43.6
35.7
36.5
80.1
53.6
73.8
73.7
63.1
73.9
57.0
62.8
58.9
59.4
43.3
45.0
Joint (CNN))
82.7
75.0
80.9
81.4
80.7
80.9
82.1
79.6
80.9
76.5
81.6
80.9
93.2
79.6
80.9
MiB
35.0
43.2
36.9
55.2
48.9
50.7
11.4
18.9
15.0
41.1
29.3
31.0
40.3
10.2
13.1
SSUL 5
78.1
33.4
67.5
72.8
51.2
57.4
74.3
51.0
63.2
60.3
40.6
44.0
51.8
26.2
28.6
MicroSeg
52
80.5
40.8
71.0
77.8
60.3
65.3
73.5
53.0
63.8
64.8
43/4
46.5
70.4
35.6
38.9
Incrementer
38
79.6
59.6
75.6
77.6
60.3
70.2
NeST 47
76.8
57.2
72.2
64.3
28.3
47.2
CoinSeg
53
82.7
52.5
75.5
76.1
65.4
68.5
80.1
60.0
70.5
70.1
63.3
64.3
72.1
40.3
43.3
START
6
80.7
57.3
75.1
76.6
70.6
79.8
71.0
70.5
65.6
70.8
44.5
Ours
83.2
61.2
78.0
79.9
72.7
74.7
81.1
65.9
73.8
73.4
70.0
70.5
79.8
51.5
54.2
Joint (TranSy)
83.8
79.3
82.7
81.1
83.3
82.7
82.4
83.0
82.7
75.8
83.9
82.7
92.0
81.7
82.7
Inspired by contrastive learning, we believe that the feature centroids of new classes should be sufficiently separated from those of the most similar old classes. Specifically, at step + 1, given batch B of images from the training dataset Di+1, we compute the feature centroids of new classes C++1 by:
EiEB Sexw FL+11 x 8(ū =c})
Unc
Eien E/ (Ft+1 x 8(91.) =c})
where |1-1|2 is the 1.2 normalization. The stored GMMs have modeled the complete distribution of old classes in previous steps. Thus, we approximately estimate the centroids of old classes using the stored GMMs. For each old class c, compute geometric centroid by weighted combing the corresponding multi-attribute Gaussian means:
k= (Tok mbck)
(9)
ER
(Tekk Mak)
After computing the feature centroids for all old classes, we can achieve the GRC loss as:
1
C
Egre
(10)
C
min
ReECH+1
With LeRC. a sufficient representational distance is maintained between new classes and their most similar old counterparts. Reciprocally, the discriminative decision boundaries enable precise modeling of the distribution of new classes using GMMs, thereby facilitating the learning in subsequent incremental steps. The total loss of our method is:
L Lseg aLyd + 3.gre-
Here zee consists of multiple Binary Cross-Entropy (mBCE) loss and uncertainty loss 52 and Ltd is the refined knowledge distillation loss (6. Hyperparameters a and / balance these terms.
Experiments
4.1 Experimental Setup
Datasets and evaluation metric. We conduct comprehensive experiments on two public datasets: Pascal VOC 2012 (15) and ADE20K (57). Pascal VOC 2012 contains 10,582 training images and
7

---

Table 2: Quantitative comparison on ADE20K between our method and previous CNN-based methods (top half) and Transformer based methods (bottom half) under the overlapped setting. from our re-implementation. represents the results are from [53]
means results
ADE 100-50 steps)
ADE 50-50 (3 steps) all
ADE 100-10 steps)
ADE 100-5 (11 steps)
Method
0-100
101-150
all
0-50
51-150
0-100
101-150
all
0-100
101-150
MiB
40.5
17.2
32.8
45.6
21.0
29.3
38.2
29.2
36.0
5.7
26.0
PLOP
32.9
30.4
31.6
39.1
SDR 31
40.5
17.2
32.8
40.9
23.8
29.5
37.3
12.1
28.9
33.0
10.6
25.6
SSUL 5
41.3
18.0
33.6
48.4
20.2
29.6
40.2
18.8
33.1
39.9
17.4
32.5
IDCC 55
42.0
18.2
34.1
47.4
26.0
33.1
40.3
17.6
32.7
39.2
14.6
31.0
STAR 6
42.4
24.2
36.4
48.7
27.2
34.4
42.0
20.6
34.9
41.7
17.5
33.7
NeST 47
42.3
22.8
35.8
27.4
34.4
40.7
33.5
39.4
15.5
31.5
Ours
43.1
24.7
37,0
49.3
27.8
35.1
42.5
22.4
35.8
42.3
21.0
35.2
Joint (CNN)*
38.9
51.1
38.9
38.9
43.8
28.9
SSUL 5
41.9
20.1
34.6
49.5
21.3
30.7
40.7
19.0
33.5
41.3
16.0
32.9
Meg
35.4
32.5
34.8
41.2
34.5
STAR
37.4
28.1
35.2
36.7
42.2
24.3
NeST
47
43.5
26.5
37.9
49.7
29.3
36.2
41.8
23.8
35.9
40.5
19.9
33.7
Ours
43.9
27.3
38.4
49.9
29.8
36.6
43.7
26.5
38.0
43.6
25.8
37.7
Joint (TranS)+
44.2
30.6
39.7
50.8
34.1
39.7
44.2
30.6
39.7
44.2
30.6
39.7
1449 validation images, encompassing 20 foreground classes. ADE20K includes 20,210 training images and 2,000 validation images, distributed across 150 classes. We follow previous works to use mean Intersection-over- over-Union (mloU) as evaluation metric.
[3
Protocols. We follow the protocols in [4] to evaluate model across various incremental scenarios defined as as N₂ - Nu. where N₂ and N,, denote numbers base and novel classes, respectively. For instance, in 15-1 scenario, training begins with 15 classes, followed by the addition of one new class at each incremental step. In this paper, we focus on more challenging long-term incremental scenarios, e.g., Pascal VOC 2012 1-1, 2-2, and ADE20K 100-5. CISS has two incremental settings: disjoint and overlapped, we mainly evaluate our approach under the overlapped setup in this paper, which is more challenging and realistic. More details for protocols are provided in Appendix A.
STAR
SEAR
DED
mlalig
learning
mutheds
(a)
(b)
(c)
Figure 2: Visualization comparison for mloU. Illustration of the change of mloU with learning steps on Pascal VOC 2012 (a) 2-2 and (b) 1-1. (c) Average performance of 20 different incremental orders
Implementation details. Following previous works 5 52 6 we use Deeplab-Ve V3 S segmentation network with ResNet-101 [8] backbone. We use SGD optimizer to optimize the network. The learning rate for the initial step is set to 0.001 and 0.00025 for Pascal VOC 2012 and ADE20K, respectively, and is reduced by factor of 0.1 for the incremental steps. We train the network for 60 epochs on Pascal VOC and 100 epochs on ADE20K with 0.9 momentum and 0.0001 weight decay a, B and K are
in all steps. The batch size is set 24 for both datasets. For the set to 5, 0.05, and 3, respectively. We conduct experiments on four NVIDIA RTX 4090 GPUs using PyTorch. More implementation details can be found in Appendix A.2
4.2 Experimental Results
Experiments on Pascal VOC 2012. Besides the widely evaluated scenarios 15-1, 5-3, 10-1 in previous works, verified the performance of the proposed method two more challenging

---

Old Classes:
Acroplane
Bicycle
14 41
fM Itt
Bird
Boat
Bottle
Bus
Cur
Cat
Chair
Cow
Diningtable
Dog
Horse
Motorbike
Person
New Classes:
Potted plant
Sheep
Sofa
Train
TV
Iapu: image
SSUL
STAR
Ours
GT
Figure 3: Qualitative comparison on Pascal VOC 2012 between ours and previous methods.
incremental scenarios, e.g-, 2-2 (10 steps), 1-1 (20 steps). Such long-term incremental scenarios are more meaningful and more closely aligned with realistic applications. As shown in Tab, [4 we present quantitative comparison between our method and -fate-of-the-art CISS approaches 66 38 47 [3]. To effectively highlight the performance of our approach, we also present results from joint training, which serve as upper bounds. For ResNet-based models, our method demonstrates superior performance over existing methods across all experimental scenarios. For instance, on more challenging scenarios 2-2 and 1-1, which contain fewer classes in the initial step and learn novel classes from more steps, our method surpasses others by 3.8 and 8.5 mloU, respectively. Meanwhile, results with Swin-B show that the proposed method is suitable for Transformer-bant based models. Leveraging the superior representation capabilities of Swin-B, our method achieves an improvement of nearly mloU for all classes on the most challenging 1-1 scenario.
Furthermore, Fig. 2a and Fig. 2b show the change of average mloU for all seen classes under the scenarios of 2-2 and 1-1 at the corresponding learning steps, respectively. For instance, at 2-2 step 5, the average mloU of seen classes 0- will be calculated. We observe that our method achieves similar performance with other methods at the initial step. However, in subsequent incremental steps, other methods exhibit a significant performance degradation, while our method effectively slows down the drop. This suggests that CoGaMiD causes less forgetting of previously learned knowledge, even on more challenging long-term scenarios. We also show the average performance with 20 different incremental orders on 15-1 scenario in Fig- 2c Compared to previous works, our method demonstrates better robustness with a lower standard deviation, affirming its adaptability to diverse incremental learning settings. In Fig. 3 we visualize the qualitative results for the images from VOC 15-1. We observe that SSUL partially maintains performance in the old classes, c.g., person; however, it tends to overfit the novel classes, e.g., train and TV. Moreover, SSUL frequently produces confused predictions, reducing the performance of several classes (see chair and sefa in the second column). Although STAR alleviates this phenomenon to some extent, a similar problem still exists. Obviously, method effectively learns the new classes while preserving knowledge of learned classes with high stability (third column). More qualitative and quantitative results are reported in Appendix A..4
Experiments on ADE20K. For the more challenging ADE20K benchmarks, we present the exper- imental results across various scenarios Table In the short-term scenarios 100-50 and 50-50, method achieves improvement of at least 0.6 mloU compared to other approaches. In the

---

Table Ablation study of components in the proposed method on VOC 15-1.
VOC
15-1 (6 steps)
Baseline
GMMs
DA
GRC
0-15
16-20
all
77.8
30.9
66.6
79.1
50.5
72.3
79.9
51.8
73.2
80.1
53.6
73.8
Table 4:
Ablation study for Gaussian mixtures K coefficients of loss a, and 3 on VOC 15-1.
vOC 15-1 (6 steps)
VOC 15-1 (6 steps)
VOC 15-1 (6 steps)
K
/
0-15
16-20
all
0-15
16-20
all
0-15
16-20
all
80.1
53.6
73.8
3
79.8
53.0
73.4
0.01
79.5
51.9
72.9
5
79.6
53.1
73.3
5
80.1
53.6
73.8
0.05
80.1
53.6
73.8
79.8
53.3
73.5
7
80.0
52.7
73.5
0.1
79.9
52.8
73.3
most challenging long-term scenario, 100-5, our method surpasses the state-of-the-art approach by a substantial margin of 1.5 mloU. Using Swin-B as the backbone further enhances the performance of our method. For instance, in 100-5 scenario, our method achieves 37.7 mloU, which represents an increase of 2.5 mloU compared to ResNet-based method, and it surpasses the best method with same backbone by 1.4 mloU. Qualitative results are show Appendix A.4
4.3 Ablation Study
Ablation study on proposed components of CoGaMiD. Tab. 3 shows the contributions of three components of our approach on Pascal VOC 2012, including Gaussian mixture distribution models (GMMs), dynamic adjustment (DA) strategy, and Gaussian-based representation constraint (GRC) loss. The first row refers to the baseline with Lxeg and LEd- Modeling Gaussian mixture distribution of learned classes and generating pseudo-features increases both stability and plasticity, which are the most significant factors in incremental learning. With the DA strategy, benefiting from the adaptive alignment between updated GMMs and the anisotropic distribution, the performance is further refined. As for GRC loss, from the results in the third and last rows of the table, the application of GRC decreases the influence of confusion between new classes and similar old ones, significantly improving the performance of new classes. Combination of all components achieve the best performance.
Gaussian mixtures. Tab. leeft shows the influence of Gaussian mixtures. From the table, we observe that increasing the number of Gaussian mixtures results in slight decrease in performance. We believe that the distribution most classes can be effectively estimated using a moderate number of Gaussian mixtures, while employing larger number of mixtures may lead to overfitting the training dataset distribution. Thus, we empirically choose K for all experiments.
Coefficients. Tab. right illustrates the influence of coefficients: a and B. The results show that, in most cases, our method is not highly sensitive the coefficients. Given the constraints imposed by spatial limitations the main text, further ablation studies are included in Appendix A.3
5 Conclusions
In this paper, we propose an effective method, CoGaMiD, aiming at CISS problems by continually modeling distribution of the learned classes via GMMs. We first introduce the GMMs estimate the multivariate distribution of each old class in the corresponding steps and store them to generate pseudo-features in subsequent steps. In order adapt to the anisotropic features that arise during the model training, we design dynamic adjustment strategy to update stored GMMs using the features representation
of old classes the current data streams. Furthermore, we develop a Gaussian-basee constraint loss to maintain the discriminative distance between new classes and similar old ones. Experiments demonstrate the effectiveness of our method, especially in long-term incremental scenarios, outperforming previous fate-of-the-aa CISS methods.
10

---

6 Acknowledgment
We would like to thank the HPC Platform of Huazhong University of Science and Technology for providing computational resources.
References
[1] Donghyeon Back, Youngmin Oh, Sanghoon Lee, Junghyup Lee, and Bumsub Ham. Decom- posed knowledge distillation for class-incremental semantic segmentation. Advances in Neural Information Processing Systems, 35, 2022.
[2] Giovanni Bellitto, Federica Proietto Salanitri, Matteo Pennisi, Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Simone Calderara, Simone Palazzo, and Concetto Spampinato. Saliency-driven experience replay for continual learning. Advances in Neural Information Processing Systems, 77:103355--133333 2024.
[3] Fabio Cermelli, Matthieu Cord, and Arthur Douillard. Comformer: Continual learning in semantic and panoptic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3010-3020, 2023.
[4] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9233-9242. 2020.
[5] Sungmin Cha, YoungJoon Yoo, Taesup Moon, al. Ssul: Semantic segmentation with unknown label for exemplar-based class-incrementy learning. Advances in neural information processing systems, 34, 2021.
[6] Jinpeng Chen, Rummin Cong, Yuxuan Luo, Horace Ip, and Sam Kwong. Saving 100x storage: Prototype replay for reconstructing training sample distribution in class-incremental semantic segmentation. Advances in Neural Information Processing Systems, 36, 2023.
[7] Jinpeng Chen, Runmin Cong, Yuxuan Luo, Horace Ho Shing Ip, and Sam Kwong, Replay without saving: Prototype derivation and distribution rebalance for class-incremental semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[8] Liang- Slang-Chihh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834 848, 2017.
[9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference compuler vision and patlern recognition, pages 1290-1299, 2022.
[10] Wei Cong, Yang Cong, Jiahua Dong, Gan Sun, and Henghui Ding. Gradient- compen- sation for incremental semantic segmentation. IEEE Transactions Multimedia, 2023.
[11] Wei Cong, Yang Cong. Yuyang Liu, and Gan Sun. Cs k: Class-specific and class-shared knowledge guidance for incremental semantic segmentation. In European Conference on Computer Vision, pages 241- Springer, 2024.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[13] Arthur Douillard, Yifu Chen, Arnad Dapogny, and Matthieu Cord. Plop: Learning without forgetting for continual semantic segmentation. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4040-4050, 2021.

---

[14] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small- tasks incremental learning. In Proceedings of the European conference on computer vision (ECCV), pages 86-102. Springer, 2020.
[15] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
[16] Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost Van De Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free free continual learning. Advances in Neural Information Processing Systems, 36:6582-663 6595, 2023.
[17] Jiangpeng He. Gradient reweighting: Towards imbalanced dlass-incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16668-16677, 2024.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[19] Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach forgetting in deep generative models. Advances in Neural Information Processing Systems, 36: 16:171111,,,9, 2023.
[20] Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1888-11867,, 2023.
[21] James Kirkpatrick. Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka et al. Overcoming catastrophic forgetting in neural networks. volume 1 14, pages 3521-3526. National Acad Sciences, 2017.
[22] Lukasz Korycki and Bartosz Krawezyk. Class-incremental mixture of gaussians for deep continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4097-4106, 2024.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
[25] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class- incremental learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattem Recognition, pages 2544 2553, 2021.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[27] David Lopez-Paz Pazz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances neural information processing systems, 30, 2017.
[28] Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanuttigh. Recall: Replay- based continual learning in semantic segmentation. In Proceedings of the IEEE/CVE international conference on computer vision, pages 7026-7035, 2021.
[29] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 165. Elsevier, 1989.
[30] Umberto Michieli and Pietro Zanuttigh. Incremental learning techniques for semantic segmenta- tion. In Proceedings of the IEFF/CVF international conference on computer vision workshops, 2019.
12

---

[31] Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion- attraction of sparse and disentangled latent representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1114-1124, 2021.
[32] Gilhan Park, WonJun Moon, SuBeen Lee, Tae- You-Young Kim, and Jae-Pil Heo. Mitigating background shift in class-incremental semantic segmentation. In European Conference on Computer Vision, pages 71-88. Springer, 2024.
[33] Benedikt Pfülb and Alexander Gepperth. Overcoming catastrophic forgetting with gaussian mixture replay. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-9. IEEE, 2021.
[34] Minh Hieu Phan, Son Lam Phung, Long Tran-Thanh, Abdesselam Bouzerdoum, et al. Class similarity weighted knowledge distillation for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16866-16875, 2022.
[35] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings the IEEE conference Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[36] Grzegorz Rypešć, Sebastian Cygert, Tomasz Trzciński, and Barthomiej Twardowski. Task- recency bias strikes back: Adapting covariances in exemplar-free class incremental learning. Advances in Neural Information Processing Systems, 37:63268-6333 222289,,,,, 2024.
[37] Grzegorz Rypešć, Sebastian Cygert, Tomasz Trzciñski, and Barthomiej Twardowski. Task- recency bias strikes back: Adapting covariances in exemplar class incremental learning. Advances in Neural Information Processing Systems, (1663668-68888 2024.
[38] Chao Shang, Hongliang Li, Fanman Meng, Qingbo Wu, Heqian Qiu, and Lanxiao Wang. Incre- menter: Transformer for dlass-incremental semantic segmentation with knowledge distillation focusing on old class. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7214-7224, 2023.
[39] Wuxuan Shi and Mang Ye. Prospective representation learning for non-exemplar class- incremental learning. Advances in Neural Information Processing Systems, 37:995-1018, 2024.
[40] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[41] Gido M Van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):4069, 2020.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class incremental learning. In European conference on computer vision, pages 398 414. Springer, 2022.
[44] Qilong Wang, Yiwen Wu, Liu Yang. Wangmeng Zuo, and Qinghua Hu. Layer-specific knowl- edge distillation for class incremental semantic segmentation. IEEE Transactions on Image Processing. 2024.
[45] Haitao Wen, Lili Pan, Yu Dai, Heqian Qiu. Lanxiao Wang, Qingbo Wu, and Hongliang Li. Class incremental learning with multi-teacher distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18443-28452 2024.
[46] Jia-Wen Xiao, Chang- Bin Zhang, Jiekang Feng Xialei Liu, Joost van Weijer, and Ming-Ming Cheng. Endpoints weight fusion for class incremental semantic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7204-7213
2023.
13

---

[47] Zhengyuan Xie, Haiquan Lu, Jia-wen Xiao, Enguang Wang, Le Zhang, and Xialei Liu. Early preparation pays off: New classifier pre-tuning for class incremental semantic segmentation. In European Conference on Computer Vision, pages 183-201. Springer, 2024.
[48] Guanglei Yang, Enrico Fini, Dan Xu, Paolo Rota, Mingli Ding, Moin Nabi, Xavier Alameda- Pineda, and Elisa Ricci. Uncertainty-awee contrastive distillation for incremental semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2567- 2581, 2022.
[49] Bo Yuan, Danpei Zhao, and Zhenwei Shi. Learning at a glance: Towards interpretable data- limited continual semantic segmentation via semantic-invariance modelling. IEEE Transactions on Pattenn Analysis and Machine Intelligence, 2024.
[50] Anqi Zhang and Guangyu Gao. Background adaptation with residual modeling for exemplar- free class-ines maass-incrementll semantic segmentation. In European Conference on Computer Vision, pages 166-183. Springer, 2024.
[51] Chang-Bine Bin Zhang, IiaWen Xiao, Xialei Liu, Ying-Cong Chen, and Ming-Ming Cheng. Repre- sentation compensation networks for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7053-7064, 2022.
[52] Zekang Zhang, Guangyu Gao, Zhiyuan Fang, Jianbo Jiao, and Yunchao Wei. Mining unseen classes via regional objectness: A simple baseline for incremental segmentation. Advances in neural information processing systems, 35:24340-24433 2022.
[53] Zekang Zhang, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, and Yunchao Wei. Coinseg: Contrast inter-and intra-class representations for incremental segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 843-853, 2023.
[54] Bingchen Zhao, Xin Wen, and Kai Han. Learning semi-supervised gaussian mixture models for generalized category discovery. In Proceedings of the IEEF/CVF international conference on computer vision, pages 16623-16633, 2023.
[55] Danpei Zhao, Bo Yuan, and Zhenwei Shi. Inherit with distillation and evolve with contrast: Ex- ploring class incremental semantic segmentation without exemplar memory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 1932-11947, 2023.
[56) Hanbin Zhao, Fengyu Yang, Xinghe Fu, and Xi Li. Rbc: Rectifying the biased context in continual semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 55-72. Springer, 2022.
[57] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 633-641, 2017.
[58] Da-Wei Zhou, Fu- Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9046 001669056,440 2022.
[59] Fei Zhu, Zhen Cheng, Xu-yao Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. Advances in neural information processing systems, 34: 14306-1431888 2021.
[60] Guilin Zhu, Dongyue Wu, Changxin Gao, Rummin Wang. Weidong Yang. and Nong Sang. Adaptive prototype replay for class incremental semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10932-10940, 2025.
[61] Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. Continual semantic segmentation with automatic memory sample selection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3082-3092, 2023.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction has accurately reflected the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
Please refer to Appendix A.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer
Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will provide Github link to the code repository in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. /ttppp.cc/pblic/ggideeee http://oweesimiiiooooooooooooooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Please refer Sec.
4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer "implementation details" in Sec. 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This manuscript adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
The important components of the core method are proposed without using LLMs.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp:::///rrpppp cc/ //onoffeeencee/2225. for what should or should be described.
21

---

A Appendix
A.1 More Protocols
consider CISS for two incremental settings: disjoint and overlapped. In the disjoint setting. for each incremental step t, the training images only contain the pixels belonging to the classes that have been learned and will learned in the current step, i.e., Clit. In the overlapped setting, future classes, C*+1:7 may appear in the current step but are labeled as background classes. Thus, we primarily report the results for the overlapped setting in the main body, while the results for the disjoint setting will be presented in the subsequent appendix.
Previous works 4 1 28
A.2 More Implementation Details
Following 53 we also choose Swin-B [26 pretrained on ImageNet-IK (23] as the backbone
to implement our method. For the segmentation head, employ the dual- architecture from CoinSeg (53), which includes a dense prediction head and a proposal classification branch. We
follow [53 52 to generate 100 class- agnostic proposals using parameter-fixet Mask2Former [9] pre-trained on MS-COCO, which serves as supervision for the proposal classification branch, We use AdamW the optimizer and train the network with learning rate of 0.0001 for all steps. We train the network for 50 epochs on Pascal VOC 2012 and ADE20K with the batch size is set 16 and 8, respectively.
A.3 More Ablation Study
Proposed components. We show extra ablations of components in Tab. 5 For the second row, we train the network without pseudo-features generated by stored GMMs and do not perform the dynamic adjustment (DA) strategy. Lacking the precise boundary of the old class, the models overfit to the novel classes, and the GRC loss fails to capture the robust features of the novel classes in the current step. This confusion is further exacerbated in the subsequent steps, since the incorrect distribution of novel classes is estimated and stored, thereby leading to the performance decrease. Using GMMs enables GRC to disadvantages into advantages (see last row in Tab. and second row in Tab. 3) Based on above consideration, we believe that the effectiveness of GRC loss should be built the precise representation of each learned class. Without DA strategy, the performance of our method decreases by 0.7 mloU for all classes. This further demonstrates the effectiveness of the idea that continual adapt GMMs to the evolving models.
Table 5: Extra ablation study of components in the proposed method on Pascal VOC 2012 15-1.
VOC 15-1 (6 steps)
Baseline
GMMs
DA
GRC
0-15
16-20
all
77.8
30.9
66.6
77.0
29.8
65.8
79.7
52.1
73.1
GMMs. We show influence of different settings GMMs estimation in Tab As for the GMMs parameters, i.e., Me., we employ three type of initialization: 'random', 'k-means', 'feature means'. where 'feature means' denotes that initialize the K Gaussian mixtures for each class with the corresponding the same feature means. The results show that "k-means' initialization is preferable to other two. On the other hand, large EM step number does not necessarily yield better performance. Therefore, we choose 500 as the maximum step number for the EM algorithm. In addition, to provide more results for K>3, we have conducted the experiments for more values of K, ranging from as shown in Tab 7 The results reveal slight decline on performance when K>3. We speculate that using a large number of Gaussian components may lead to partial overfitting, particularly for classes with simple feature distributions in the training set. We believe that K=3 is relatively reasonable setting, holding performance and computational efficiency simultaneously. Moreover, we show results for different values of K on VOC 15-1, ranging from 1 to 3, for both the baseline and final model in Tab TT] The results suggest that using multiple Gaussians simultaneously improves the performance of the baseline and our final model, which demonstrates the superiority and necessity of
22

---

Table 6: Influence of GMMs settings on VOC 15-1.
VOC 15-1 (6 steps)
VOC
15-1 (6 steps)
Initialization
EM Steps
0-15
16-20
all
0-15
16-20
all
random
79.8
52.9
73,4
100
79.9
53.2
73.5
k-means
80.1
53.6
73.8
500
80.1
53.6
73.8
feature means
80.0
53.3
73.6
1000
80.2
52.6
73.6
Table 7: more values of on VOC 15-1.
VOC 15-1 (6 steps)
K
0-15
16-20
all
4
79.9
53.2
73.6
5
79.6
53.1
73.3
6
79.6
53.5
73.4
7
79.8
53.3
73.5
Table 8: Influence of different KD on VOC 15-1.
VOC 15-1 (6 steps)
Methods
0-15
16-20
all
Baseline (with standard KD from IIT 30
77.6
20.5
64.0
Baseline (with Local-POD from PI.OP [33
77.5
24.2
64.8
Baseline (with OCFM from STAR 66
77.8
30.9
66.6
Table 9: Experiments computational and memory cost with previous exemplar based methods.
Methods
mIOU
GFLOPS
#arams
storage
SSUL 55
71.4
211
60M
10M
STAR 6
72.6
272
59M
0.04M
Ours
73.8
278
59M
0.12M
our Gaussian Mixture distribution modeling approach compared to the use single prototype. We visualize the distributions of each base class in Fig. 4 From the figure, we observe that their shapes are often complex and irregular, making them difficult to approximate with a single Gaussian. We believe that the t-SNE results shown in Fig. 7 also partially reflect the multimodality of the most classes (e.g. 'person', 'sofa', "cow', 'train' etc.). These class distributions can be more accurately modeled by Gaussian Mixture Models (GMMs) rather than single Gaussian. Above qualitative and quantitative results further justify the necessity of adopting GMMs.
Knowledge distillation. trained baseline using OCFM loss in STAR the form of. Since our focus is not on improving the knowledge distillation techniques that widely used in CISS, we hope to adopt the most effective distillation loss from existing methods to establish stable baseline. We provide the results for the baseline using different distillation losses on VOC 15-1 in Tab B The results demonstrate the effectiveness of OCFM, which is why adopted it the knowledge distillation component and used it to train the model as our baseline.
Computational and memory cost. We provide comparison of overall performance (mloU) VOC 15-1, computational complexity (GFLOPs), model size (# Params), and storage cost for exemplars or prototypes across the three methods, as shown in Tab 9 The results show that our method achieves performance gains of 1.2% at similar model sizes and GFLOPS. In addition, we use 0. 1.12M memory for storaging GMMs, which is far lower than the SSUL that storage images. Even when compared with the prototype-baat method, our storage cost only increase 0.08M that almost be ignored.
Architectures. Although our current research primarily focuses Deeplab-based continual learing architectures, we attempt to implement our method on such structure used MBS and conducted the experiments VOC 15-1 compared with other methods, exhibited in Tab 10
23

---

Table 10: Experiments on Mask2Formerrbased architecture for VOC 15-1.
VOC 15-1 (6 steps)
Methods
0-15
16-20
all
MiB
4
72.6
23.1
61.7
RBC
[6
75.9
40.2
68.2
INC
38
79.6
59.6
75.6
MBS
32
82.6
72.2
80.6
Ours
82.9
73.9
81.2
LAkngglee
Aigyle
4Boa
Batte
filis
4chwv
E-Dennaak
12.0g
13Hee
15 Penoo
Figure 4: TSNE visualization for every base classes on VOC 15-1.
Table 11: Influence of K for both baseline and final model on VOC 15-1.
VOC 15-1 (6 steps)
Methods
0-15
16-20
all
Baseline
77.8
30.9
66.6
Baseline (K=1)
78.4
38.6
68.9
Baseline (K=2)
78.9
47.8
71.5
Baseline (K=3)
78.4
38.6
68.9
Ours (K=1)
79.4
51.8
72.8
Ours (K=2)
80.0
52.7
73.5
Ours (K=3)
80.1
53.6
73.8
Table 12: Influence of Core settings on VOC 15-1.
VOC 15-1 (6 steps)
Centroids
0-15
16-20
all
original
79.9
52.7
73.4
geometric
80.1
53.6
73.8
The results demonstrate the effectiveness of our method such structure, achieving competitive performance. We also believe that with further exploration, our method has the potential to perform even better on this structure.
Centroids of old classes in Lgre- We consider the two types of old centroids in Lyre: 'original' and 'gcometric'. The type of 'original' represents the K means derived from the stored GMMs
24

---

Step 0
Step 2
Cow
Cow
Step 0
Step 3
Chair
Chair
Figure 5: TSNE visualization for class 'cow" and 'chair' at different learning step on VOC 15-1.
of each learned class are used to compute the 1.2 distance between these means and new class means, while 'geometric' denotes the geometric centroids that combined by means with the corresponding weights of πT T2 the impact of the aforementioned settings, indicating that the geometric approach yields better performance. We believe that the reason of the suboptimal performance obtained by 'original' centroids may be due to treating every Gaussian centroid equally, or overemphasizing the small probability distribution with few feature samples. In this case, the learning space of the classes will be compressed, acquiring insufficient representations.
Combination with memory sampling strategy. We follow the works [S] 52
6 to employ the
memory sampling strategy to further enhance the performance of our method. Tab. 13 shows the results of our method with different memory numbers. With 20 additional raw images providing more realistic features, our method achieves performance that is nearly comparable to the previous best memory-based approach. The performance is further improved with more memory, while the required storage is half of that needed by other methods.
Table 13: Method with different memory number on VOC 15-1.
VOC 15-1 (6 steps)
Methods
0-15
16-20
all
Ours
80.1
53.6
73.8
Ours-M(2)
80.1
55.2
74.2
Ours-M(50)
80.2
57.5
74.8
SSUL-M((00)
78.4
49.0
71.4
DKD-M(100)
78.8
52.4
72.5
STAR-M(100)
79.9
56.2
74.3
Features anisotropy. The anisotropy problem has been analyzed in the [6L 37 which means the features shift in various directions during incremental learning process. We also observe the analogous phenomenon in the CISS. As depicted in Fig. 5 we find that the old class 'chair' and 'cow' showed more noticeable deviations in the subsequent incremental learning process, especially when model learned the class like "sofa" (step 3) and 'sheep' (step 2). We think that this is because the original feature space of the old class may cause confusion between similar new classes and old classes (e.g. "chair' and 'sofa', 'cow" and 'sheep'). Therefore, the model tends redistribute the features of these old classes in order to leave appropriate space for better learning of new classes.
25

---

+ 49
mlol(%%
53
learning steps
Figure 6: Illustration of the change mloU for old classes and each novel class on VOC 15-1.
Consequently, we propose a dynamic adjustment strategy that continuously updates the stored GMMs to mitigate this issue. The third row in Table iillustates the effectiveness of the proposed strategy.
Capability of preventing forgetting. To further demonstrate the effectiveness of method in preventing forgetting, we present the mloU changes for the previously learned class and each novel class during incremental learning on VOC 15-1, as shown Fig. 6 We report the mIoU of these classes after each 20 epochs training. We observe that the performance of the old classes remains nearly consistent throughout the incremental learning steps, benefiting from the proposed continual Gaussian mixture modeling method. For novel classes, our method enables the model to maintain performance in subsequent learning steps, even further improving the initial performance. We analyze that this is because when previously unknown categories are marked as new categories, discriminant distance constraint reduces the confusion between the previous categories and unknown categories in the background. These results demonstrate that the effectiveness of the proposed method in resisting forgetting.
A.4 More Experimental Results
Quantitative comparison under the disjoint setting. In Tab. 14 we present quantitative compari- son between our STAR and previous methods under the disjoint setup. For CNN-based comparisons, our method achieves a slight advantage over other methods in short-term incremental steps, i.e., 19-1 and 15-5. This advantage becomes more pronounced on VOC 15-1. Similar results can be found in the Transformer-backe backoone methods, demonstrating that our method handle various incremental settings with different backbone.
Table 14: Quantitative comparison Pascal VOC 2012 between method and previous CNN- based methods (top half) and Transformer-based methods (bottom half) under the disjoint setting.
VOC 19-1 (2 steps)
VOC
15-5 (2 steps)
VOC 15-1 steps)
Method
0-15
16-20
all
0-5
6-20
all
0-10
11-20
all
MiB
4
69.6
25.6
67.4
71.8
43.3
64.7
46.2
12.9
37.9
PLOP 13
75.4
38.9
73.6
71.0
42,8
64.3
57.9
13.7
46.5
SSUL 5
77.4
22.4
74.8
76.4
45.6
69.1
74.0
32.2
64.0
RBC [56
76.4
45.8
75.0
75.1
49.7
69.9
61.7
19.5
51.6
MicroSeg 52
80.6
16.0
77.4
77.4
43.4
69.3
73.7
24.1
61.9
CoinSeg
5
80.5
25.1
77.9
79.6
43.8
71.1
75.6
30,9
64.9
STAR
77.9
43.4
76.2
78.4
57.4
73.4
78.1
46.6
70.6
Ours
79.8
46.4
78.2
78.9
58.2
74.0
78.9
49.2
71.8
MiB 4
80.6
45.2
79.6
75.0
59.9
72.3
66.7
26.3
58.3
CoinSeg
53
82.0
34.0
80.2
82.1
55.3
75.7
82.0
46.1
73.4
Ours
82.5
67.2
81.8
82.4
61.8
77.5
82.2
57.9
76.4
26

---

PLOP
STAR
Ours
Lsipplanc
R-bisycle
3-bird
Ghott
Shatlc
6-ba.
T-car
S-cil
10-ame
1-diining ubie
12-dog
13-horse
14-motobiike
15-penann
16-poted plant
17-sheep
18-oofa
19-zais
20-monitor
Figure 7: TSNE visualization on VOC 15-1. Numbers the image represent the corresponding classes.
Image
Step
Step
Step 3
Step 4
Step 5
Step 6
GT
18
Figure 8: More qualitative results on VOC 15-1, the new classes.
Plant
Sheep
Sofa
Train
and
TV
are belong
Qualitative comparison on VOC. We present the t-SNE visualization all classes for our method and previous methods on VOC 15-1, depicted in Fig. 7 As illustrated in the figure, PLOP fails to learn discriminative representations during the incremental steps, resulting in significant forgetting. Benefiting from the feature replay strategy, STAR effectively preserves the knowledge of old classes. However, it introduces some confusion by neglecting the anisotropic distribution and imposing fragile constraint the distance between novel and old classes. The illustration demonstrates that our method achieves an excellent balance between plasticity and stability. This further supports the effectiveness of continual Gaussian mixture distribution modeling, which enables the model to exhibit strong anti-forgetting on old classes while also providing favorable adaptability to novel classes.
old classes
More qualitative results. Fig. 8 we visualize qualitative results CoGaMiD for the same images after each learning step VOC 15-1 (6 steps). For images containing only new classes, our method effectively learns these classes during the corresponding steps while maintaining consistent performance in subsequent steps. Regarding the sample set of dog and sofa, dog belongs to the old
27

---

Image
Step
Step
Step
Step 4
Step 5
Step
Figure 9: More qualitative results on ADE20K 100-10 (6 steps).
Stage
Stool
Washer
Vase
and Radiator
are belong to the new classes.
classes, while sofa is learned in step 4. Our method performs the complete segmentation of novel class sofa with the learned class dog insusceptible, and similar phenomenon shown in the fourth row. In the fifth row, a small portion of the background pixels is incorrectly classified as train due to slight overfitting during the learning of this class in step 5; however, this error promptly corrected in the subsequent step. We present the qualitative results of our method for five images at corresponding steps on ADE20K 100-10 (6 steps), as illustrated in Fig. The ADE20K dataset, with its extensive number of classes to be learned in both initial and incremental steps, poses particular challenges. As shown in these figures, our method effectively learns to predict new classes while simultaneously retaining the ability to identify old classes, demonstrating both stability and plasticity.
A.5 Limitations and Future work
Through extensive experiments two public benchmarks, our method shows excellent performance and achieves a favorable balance between stability and plasticity. However, estimating the distribution of classes using predefined number of Gaussian mixtures may not be robust for all classes, given that the complexity of each class varies. Therefore, we will extend the method to an enhanced version that adaptively models the distribution with appropriate mixtures. Furthermore, we believe that compact representation in the initial step is crucial for the learning of subsequent steps. Thus, we will discuss that how to conduct the pre-constraint on the feature space of old classes the initial step. reserving sufficient space for the learning of novel classes, in future work.
28


Reviews:
[{"review_id": "Jqf3FZIkJX", "reviewer": "Reviewer_vdnJ", "summary": "The authors present an novel framework, dubbed CoGaMiD, for class-incremental semantic segmentation. CoGaMiD utilizes a Gaussian Mixture Model to generate features for previously learned classes that may be inaccessible or scarcely available during incremental learning phases. These generated features facilitate training a segmentation model and play a crucial role in alleviating the issue of forgetting prior knowledge of old classes. Additionally, the authors propose a dynamic adjustment strategy to refine these generated features with real features collected during the incremental steps. They also introduce a Gaussian-based representation constraint loss aimed at learning discriminative features for new classes.", "questions": "Please read the questions in the weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- CoGaMiD uses GMMs to generate features for previously learned classes, assisting a segmentation model in mitigating forgetting.\n\n**Weaknesses**\n\n- The authors utilize Gaussian Mixture Models (GMMs) to generate features of previously learned classes for the purpose of updating a segmentation model. In recent work, STAR [6], features of old classes are generated from Gaussian-distributed random variables collected from earlier stages. The Gaussian-based Representation Constraint (GRC) bears similarity to the similarity-aware discriminative loss in STAR, as both aim to ensure the means of Gaussians for new classes are distinct from those of old classes. The specifics of the refined knowledge distillation loss $L_{kd}$ in Eq. 11 are unclear. Is this referring to the effectiveness of the Old-Class Features Maintaining (OCFM) loss in STAR? Given these considerations, CoGaMiD appears to be closely related to STAR, with the primary distinction being the use of a Gaussian mixture model as opposed to a single Gaussian model, which suggests that CoGaMiD might be seen as an extension of STAR. It would be beneficial to clearly delineate the differences between CoGaMiD and STAR. Additionally, to demonstrate the effectiveness of using multiple Gaussians, a key distinction from STAR, it would be valuable to present results for different values of K, ranging from 1 to 3, for both the baseline and the final models.\n- The performance observed when training with all samples simultaneously, referred to as 'joint,' represents an indirect upper boundary for the incremental model. In Tables 1 and 2, the results for the 'joint' method appear to be higher than those for DKD [1], a recent approach in CISS which utilizes binary cross-entropy loss, similar to CoGaMiD. To ensure a fair comparison and to highlight the effectiveness of the proposed methods without modifications to the architecture or minor techniques, it would be advantageous to include the reproduced results of some baseline models in Tables 1 and 2", "was_revised": true, "final_justification": "Through the discussion process, I came to agree with the authors on the empirical effectiveness of the proposed multiple Gaussian approach. Additionally, the authors mentioned plans to further support the necessity of using GMMs through additional analyses such as feature distribution visualizations.\n\nI look forward to seeing these enhancements in the final version and, with that expectation, I am raising my rating to borderline accept.", "rebuttal": "We sincerely appreciate the reviewers' thorough evaluation and constructive feedback. The major comments are answered below.\n____\n\n**1.Difference between our method and STAR.**\n\nAs you mentioned, some prior works have employed prototypes to generate the features of learned classes, with a representative being STAR [1] that you referred to. However, there were some limitations and challenges in the previous methods, which have been discussed in our submitted manuscript. Our main contribution is to address these problems in prototype-replay based CISS methods. We highlight clear differences between ours method and STAR from three points as follow:\n\n- **Gaussian Mixture Models (GMMs) vs. single prototype:** Firstly, STAR utilizes a single prototype to represent the complete distribution of each old class, which may be insufficient when the incoming data stream for a class is inherently multimodal. Secondly, the method of obtaining prototypes by directly computing the mean vectors of old class features is not easily extendable from a single prototype to multiple prototypes. Different from STAR, to address the aforementioned limitations, we are the first to ingeniously incorporate the multimodal characteristics of GMMs into the CISS framework, enabling accurate modeling of old class distributions through iterative learning with the Expectation-Maximization (EM) algorithm. The results from the table in the response '3.Results for different values of K.' also demonstrates the effectiveness of our use of multiple Gaussians.\n\n- **Dynamic GMMs adjustment vs. fixed prototype replay:** During incremental learning steps, the model needs to update its parameters to accommodate new classes. Inevitably, some old class features that are sensitive to parameter changes deviate from their original positions. This is more likely to result in a mismatch between the fixed prototypes in STAR and the changed feature distributions of the old classes. To address this issue, we propose a Dynamic Adjustment (DA) strategy that continuously updates the parameters of stored GMMs using the real-time features of old classes at each learning step, thereby adapting to the changed representations of old class distributions. The results on VOC 15-1 (Table 3 in our manuscript) below suggest that our DA strategy further improve the performance of the model on both of new and old classes.\nBaseline|GMMs|DA|0-15|16-20|all\n:-|-|-|-|-|-|\n&#10004;|&#10004;||79.1|50.5|72.3\n&#10004;|&#10004;|&#10004;|79.9|51.8|73.2\n\n- **Gaussian-based representation constraint (GRC) loss vs. similarity-aware discriminative (SAD) loss:** As you mentioned, we adopt a contrastive loss formulation similar to that of SAD. However, SAD selects a single prototype as the centroid of each old class, which ignores the non-uniform density and multimodal characteristics of feature distributions. In contrast, we refine the computation of old class centroids by adaptively combining multiple Gaussian means, weighted by the respective probability of each mixture component. This refinement enables a more faithful representation of the old class distributions. To further illustrate the impact of our GRC loss, we present additional experimental results on VOC 15-1 below, comparing the final models trained with either SAD or GRC loss. The results demonstrate that our GRC loss provides more effective guidance for learning new classes and significantly improves performance.\nMethod|0-15|16-20|all\n:-|-|-|-|\nWith $L\\_{sad}$|79.8|52.1|73.2\nWith $L_{grc}$|80.1|53.6|73.8\n\n\nIn summary, our method diverges from STAR both in features modeling (single prototype vs. multiple Gaussian mixture components) and features replaying (fixed prototype vs. Dynamic GMMs). Moreover, our modification of anchors of old class in GRC loss is line with the disign of our method, fundamentally distinguishes it from previous SAD loss in STAR. From the results of our extensive experiments and ablation study, it can be seen that our method of continual Gaussian mixture distribution modeling (CoGaMiD) is effective, particularly in challenging multi-step scenarios (e.g. VOC 1-1 (20 steps) and ADE 100-5 (11 steps)).\n\n___\n**2.The specifics of the refined knowledge distillation loss $L_{kd}$ in Eq. 11 are unclear. Is this referring to the effectiveness of the Old-Class Features Maintaining (OCFM) loss in STAR?**\n\nWe apologize for unclearly notation of $L_{kd}$ in Eq. 11. As you guessed, we trained our baseline using OCFM loss in STAR as the form of $L_{kd}$. Since our focus is not on improving the knowledge distillation techniques that widely used in CISS, we hope to adopt the most effective distillation loss from existing methods to establish a stable baseline. We provide the results for baseline using different distillation losses on VOC 15-1 as follow:\n\nMethod|0-15|16-20|all\n:-|-|-|-|\nBaseline (with standard KD from ILT [2])|77.6|20.5|64.0\nBaseline (with Local-POD from PLOP [3])|77.5|24.2|64.8\nBaseline (with OCFM from STAR [1])|77.8|30.9|66.6\n\nThe results demonstrate the effectiveness of OCFM, which is why we adopted it as the knowledge distillation component and used it to train the model as our baseline. We will clarify the notation of $L_{kd}$ in the final version.\n\n___\n**3.Results for different values of K.**\n\nFollowing your suggestion, we show results for different values of K on VOC 15-1, ranging from 1 to 3, for both the baseline and final models as follows:\n\nMethod|0-15|16-20|all\n:-|-|-|-|\nBaseline|77.8|30.9|66.6\nBaseline (K=1)|78.4|38.6|68.9\nBaseline (K=2)|78.9|47.8|71.5\nBaseline (K=3)|79.1|50.5|72.3\nOurs (K=1)|79.4|51.8|72.8\nOurs (K=2)|80.0|52.7|73.5\nOurs (K=3)|80.1|53.6|73.8\n\nThe results suggest that using multiple Gaussians simultaneously improves the performance of the baseline and our final model, which demonstrates the superiority of our Gaussian Mixture distribution modeling approach compared to the use of a single prototype.\n\n___\n**4.The results for the 'joint' method appear to be higher than those for DKD, and present the reproduced results of some baseline models.**\n\nAs you mentioned, some methods reported different joint results (e.g., 77.6 reported in DKD [4]). In our manuscript, we reported the joint results of CNN directly cited from CoinSeg [5], in order to highlight the performance gap between existing CISS methods and the best upper bound. Based on our observations, the differences in the joint results of these methods mainly stem from variations in the experimental settings (e.g., learning rate, batch size, and training epochs). Additionally, the segmentation network in CoinSeg is trained with the assistance of proposals generated by Mask2Former [5], which further enhances the performance of joint training. Following your suggestion, we conducted joint training using our method and reproduced as many results with the corresponding official code as possible on VOC 15-1. The results can be seen in the  response to **Reviewer dzLn ('1.Different joint result of CNN compared with previous methods.')**. These results suggest that our joint result is almost consistent with previous methods, ensuring a fair comparison and to highlight the effectiveness of the proposed methods without modifications to the architecture or minor techniques. We will provide more such reproduced results for other incremental scenarios in final version.\n\n___\n[1] Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation, In NIPS 2023.\n\n[2] Incremental learning techniques for semantic segmentation. In ICCVW, 2019.\n\n[3] Plop: Learning without forgetting for continual semantic segmentation, In CVPR 2021.\n\n[4] Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation, In NIPS 2022.\n\n[5] Masked-attention mask transformer for universal image segmentation, In CVPR 2022."}, {"review_id": "DAa7NlpjJp", "reviewer": "Reviewer_dzLn", "summary": "This paper introduces CoGaMiD, a novel method for Class Incremental Semantic Segmentation (CISS). The core idea is to address the limitations of recent prototype-based approaches that cannot represent the complete features of each category. The method models the feature distribution of each class with a Gaussian Mixture Model (GMM) instead of a single prototype. The additional Dynamic Adjustment overcomes the shifting feature distribution of previous classes. Gaussian-based Representation Constraint (GRC) Loss maintain discriminability between new and old classes. The experiments on Pascal VOC 2012 and ADE 20K demonstrate the effectiveness of the approach.", "questions": "- Is this method effective for Mask2Former-based  structures compared to [1][2][3]?\n- Is there any visualization result (t-SNE or segmentation result) to show anisotropy in detail?", "limitations": "The authors have provided a discussion of limitations in Appendix A.6.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to understand. \n- The experiments are solid and the performance is impressive. \n- The application of the Gaussian Mixture Model (GMM) balances the quality of features and the external space of buffer. \n\nWeaknesses: \n\n- Why is the Joint result of CNN different from previous methods? The previous work [1][2] shows that the Joint result of Deeplab-V3 with ResNet-101 is 77.4. \n- Limited intuitive experiment results showing the anisotropy problem. \n- Lack of experiments on computational and memory cost with previous exemplar-based methods. \n\n[1] Cermelli, Fabio, et al. \"Modeling the background for incremental learning in semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Park, Gilhan, et al. \"Mitigating Background Shift in Class-Incremental Semantic Segmentation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "was_revised": true, "final_justification": "The responses have addressed most of my initial concerns.", "rebuttal": "We greatly appreciate the reviewer's insightful review of our work. Our responses to each point are as follows:\n\n___\n**1.Different joint result of CNN compared with previous methods.**\n\nAs you mentioned, some methods reported different joint results (e.g., 77.4 in MiB [1] and MBS [2]). In our manuscript, we shown the joint results of CNN directly cited from CoinSeg [3] for the consideration of revealing the performance gap between the current CISS methods and the best upper bound. Based on our observations, these methods differ from each other in experimental settings (e.g., learning rate, batch size, and training epochs). Additionally, the segmentation network in Coinseg is trained with the assistance of proposals generated by Mask2Former, further enhancing the performance of joint training. Thus, we reproduce as many results on VOC 15-1 and joint results with correspond official code as possible as follow:\n\nMethod|0-15|16-20|all\n:-|-|-|-|\nMiB|35.5|13.1|30.2\nPLOP|64.1|21.7|54.0\nDKD|78.3|40.7|69.4\nCoinSeg|80.6|36.2|70.1\nSTAR|79.5|50.6|72.6\nOurs|80.1|53.6|73.8\nJoint (MiB)|79.3|73.0|77.8\nJoint (PLOP)|79.5|73.6|78.1\nJoint (DKD)|80.1|72.6|78.3\nJoint (STAR)|80.2|72.6|78.4\nJoint (Ours)|80.4|72.4|78.5\nJoint (CoinSeg)|82.7|75.0|80.9\n\nThese results show that our approach achieves the smallest performance gap (4.7%) to ours own upper bound. Moreover, our joint result is largely consistent with other methods, except for CoinSeg. Therefore, we argue that our evaluation provides a fair comparison environment, and the joint result of Coinseg serves as a reference for the best upper bound. We will provide more such results of other incremental scenarios in the final version.\n\n\n___\n\n**2.Limited intuitive experiment results showing the anisotropy problem. (W2 & Q2)**\n\nWe are grateful for the reviewer's insightful comments. We roughly treat the change in old class features during the learning of new classes as an anisotropy problem, which was analyzed in FeCAM[4]. Considering your recommendation, we conducted visualization experiments on the feature distributions of some old classes at incremental steps on VOC 15-1. We observed that the shapes of some feature distributions have undergone certain changes, indicating that part of the old class features have deviated from their initial positions during the model's updating process. We truly want to show these visualizations but there is no way to add new figures during the rebuttal process. Therefore, we will add such visualizations with more analysis in the final version so please understand it.\n\n___\n**3.Experiments on computational and memory cost with previous exemplar-based methods.**\n\nAs suggested, we provide a comparison of the overall performance (mIoU) on VOC 15-1, computational complexity (GFLOPs), model size (#Params), and storage cost for exemplars or prototypes across the three methods, as follows:\n\n**Methods**|**mIoU**|**GFLOPS**|**#Params**|**storage**\n:-|-|-|-|-|\nSSUL|71.4|211|60M|10M\nSTAR|72.6|272|59M|0.04M\nOurs|73.8|278|59M|0.12M\n\nThe results show that our method achieves performance gains of 1.2% at similar model sizes and GFLOPS. In addition, we use 0.12M memory for storaging GMMs, which is far lower than the SSUL that storage raw images. Even when compared with the prototype-based method (STAR), our storage cost only increase 0.08M that almost be ignored.\n\n___\n\n**4.Effectiveness of method for Mask2Former-based structures.**\n\nWe appreciate your thoughtful recommendations. Our current research primarily focuses on Deeplab-based continual learing architectures. The Mask2Former-based structures in MBS [2] differ from the DeepLab-based architecture used in our manuscript, requiring us to make several adjustments to adapt the method for implementation on such a framework. This might take a relatively long time. Thus, we look forward to exploring the effectiveness of our method for Mask2Former-based structures in future work, and we kindly ask for your understanding.\n\n____\n\n[1] Modeling the background for incremental learning in semantic segmentation. In CVPR 2020.\n\n[2] Mitigating Background Shift in Class-Incremental Semantic Segmentation. In ECCV 2024.\n\n[3] CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation. In ICCV 2023.\n\n[4] FeCAM:Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning. In NIPS 2023."}, {"review_id": "MX63upe6aY", "reviewer": "Reviewer_zb2d", "summary": "This paper proposes a Continual Gaussian Mixture Distribution (CoGaMiD) modeling method for class-incremental semantic segmentation task, which considers the distribution characteristics of class prototypes in Gaussian Mixture Models (GMMs) and updates the prototype distribution of old classes through the Dynamic Adjustment (DA) strategy to solve the problem of distribution shift of old-class prototypes caused by extractor updates. The author conducted extensive experiments on the Pascal VOC and ADE20K datasets to validate the effectiveness of the proposed method.", "questions": "See weaknesses. I will modify my rate based on the author's responses to the questions in the weaknesses.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The problems with the single-prototype-based approach mentioned by the author are reasonable, and the idea proposed by the author to store the distribution of class features is intuitive.\n2. The author's observation on the problem of class feature shift caused by the update of the extractor is insightful.\n3. Strong performance is achieved on several benchmarks.\n\n\n\nWeaknesses:\n1. Is the form of $ L_{grc} $ inconsistent with what the author stated in line 212? The optimization goal of $ L_{grc} $ is more like shortening the distance between new classes and their most similar old counterparts. And there might be some typos, e.g., $ \\mu_{n_o} $ in Eq. 10.\n2. Based on the calculation of the geometric centroid of the old class according to Eq. 9, how to ensure a sufficiently clear decision boundary if some classes are widely distributed in the feature space? Will this also cause overlap?\n3. Is the method of dynamically adjusting the distribution of old classes also considered a form of forgetting?\n4. It is better to clarify how the prototype is used in the continual learning process and the specific setting of the baseline in the ablation studies, which will be more friendly for readers to follow.", "was_revised": true, "final_justification": "The author's response addressed my concerns, and the method is effective.  I raise my rating to 'Accept'.", "rebuttal": "We appreciate your detailed comments and suggestions. Our responses to each point are as follows:\n\n___\n**1.Inconsistent statement and typos.**\n\nWe apologize for the misunderstanding caused by our oversights regarding the writting of Eq. 10. The goal of $L_{grc}$ is to maintain the discriminative distance between new classes and their most similar old counterparts. We will correct all identified issues in the final version:\n\n**a.** We will change $\\mu_{n_o}$ to $\\mu_{n_c}$ and modify Eq.10 to $L_{grc}=\\frac{1}{|\\mathcal{C}^{t+1}|}\\sum_{n_c\\in\\mathcal{C}^{t+1}}\\frac{1}{\\min_{o_c\\in\\mathcal{C}^{1:t}}\\||\\mu_{n_c}-\\mu_{o_c}\\||_2}$ for consistant statement.  **b.** We will change $\\pi_j$ to $\\pi\\_{cj}$ in Eq. 3 to refer to the specific class $c$.   **c.** We will modify the font style of $\\pi\\_{ck}$/$\\pi\\_{cj}$ from vector to scalar across the paper, which denotes a probability value blong to $k$-th Gaussian component for class $c$.\n\n___\n**2.How to ensure a sufficiently clear decision boundary if some classes are widely distributed in the feature space? Will this also cause overlap?**\n\nThank you for your thoughtful comment. We believe that a well-trained feature extractor enables features of the same class to be distributed in a unified and compact latent space. Consequently, the distances among the K mixture components of a given class tend to be closer to each other than to those of different classes. We argue that computing the geometric centroid provides a more intuitive and comprehensive representation of old class feature distributions than relying on the means of individual mixtures. For new class features, supervised by $L_{grc}$, it is easier to maintain an appropriate distance outside the distributions of old classes rather than being closely embedded within them. Based on the above analysis, we believe that using the geometric centroid formulation effectively avoids the issue of overlap. The t-SNE visualization results of Figure 5 in our manuscript further intuitively demonstrate that our method can ensure a sufficiently clear decision boundary. Additionally, we report the results of different $L_{grc}$ settings regarding the formulation of old class centroids on VOC 15-1 below (as shown in Appendix Table 7), to further verify the effectiveness of our design. Here, 'original' represents that using the K mean vectors derived from the stored GMMs as centriods of each learned class to compute the L2 distance between these centriods and new counterparts.\n\n\n|Centroids|0-15|16-20|all\n:-|-|-|-|\noriginal| 79.9| 52.7|73.4\n geometric|80.1| 53.6|73.8\n\n___\n**3.Wheter dynamically adjusting the distribution of old classes also be considered a form of forgetting?**\n\nThere might be a misunderstanding. In our view, the primary cause of forgetting is that the model fails to capture the complete distribution of previously learned classes due to lacking the past data during the incremental learning process. Our method of Gaussian mixture distribution modeling utilizes stored GMMs to reproduce the complete distribution of previously learned classes, which has significantly alleviated the issue of forgetting. However, the distribution of the old classes may change because the model parameters are updated to accommodate new classes. This leads to a mismatch between the fixed distribution obtained by the stored GMMs and the evolving distribution of previously learned classes reflected in the continually updated model. Thus, we propose a dynamic adjustment (DA) strategy to update the GMMs, allowing them to continuously adapt to the changes in the distribution of previously learned classes within the model. Based on the above analysis, we tend to argue that our method of dynamically adjusting the distribution of previously learned classes addresses the issue of distribution mismatch, rather than representing a form of forgetting. The results presented in the table below demonstrate the effectiveness of our DA strategy.\n\nbaseline|GMMs|DA|GRC|0-15|16-20|all\n:-|-|-|-|-|-|-|\n&#10004;|&#10004;| |&#10004;|79.7|52.1|73.1\n&#10004;|&#10004;|&#10004;|&#10004;|80.1|53.6|73.8\n\n___\n**4.Usage of prototype and specific setting of the baseline.**\n\nUsage of prototype: In continual learning process, single-prototype based methods utilize stored prototypes obtained from previous learning steps to generate old class features. This is typically done by repeatly adding random Gaussian variances to the mean vectors of old classes. The generated features are then combined with real features and fed into the classifier to facilitate the learning of new classes.\n\nSpecific setting of the baseline: In the ablation studies, we selected the segmentation network implemented with DeepLabv3 and ResNet-101 pretrained on ImageNet as our baseline model, which was surpervised by standard a multiple Binary Cross-Entropy (mBCE) loss in the initial step. During the incremental learning process, we incorporate the knowledge distillation loss from STAR [1], along with an uncertainty loss from Adapter [2], as additional supervision. \n\nWe will clarify the usage of prototype and specific setting of the baseline in the final version to help the readers to follow.\n\n___\n[1] Saving 100x storage: Prototype replay for reconstructing training sample distribution in class-incremental semantic segmentation. In NIPS 2023.\n\n[2] Adaptive prototype replay for class incremental semantic segmentation. In AAAI 2025."}, {"review_id": "zuufnGouZ2", "reviewer": "Reviewer_vPPN", "summary": "The paper integrates Gaussian mixture distribution for class-incremental segmentation, which tackles the challenge of performance degradation due to overfitting to new classes.\nThe paper proposes modeling the latent class feature distribution as a GMM. It also introduces a dynamic adjustment strategy that iteratively updates GMM parameters based on feature representations from the current model, and a Gaussian-based representation constraint loss that encourages separation between features of new and old classes.", "questions": "Please refer to the items listed in the weakness section. It would be helpful if the authors could provide an explanation to improve the clarity of the paper.", "limitations": "Limitations are discussed at the end of the paper.\nThis paper does not introduce potential negative socialtal impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "-- Strengths\n\nThe contrastive learning-inspired loss is a reasonable and effective way to preserve class separability during incremental learning, preventing overlap between similar old and new classes.\n\nThe dynamic adjustment strategy is a reasonable design to handle continual drift during incremental steps.\n\nExperiment results on Pascal VOC and ADE20K datasets are promising, especially in challenging long-term incremental settings.\n\n\n-- Weaknesses\n\nAssuming latent features follow a fixed GMM with K=3 components per class is strong. While it may work empirically, it is recommended to at least provide more discussion.\n\nSome mathematical notations (e.g., subscript/superscript usage) are not properly explained. It would improve the paper if the authors included a summary table of all symbols and notations.\n\nIt is not clearly explained how the segmentation network leverages or integrates the GMM features during classification. It is recommended to provide either diagrams or mathematical formulations specifying the corresponding inputs and outputs, and/or any special layer architecture if used.", "was_revised": true, "final_justification": "My initial rating assumed the listed issues could be resolved during the discussion phase. The authors' rebuttal has addressed most of my concerns. Considering other reviewers' comments, I intend to maintain my original (positive) rating.", "rebuttal": "We greatly appreciate the reviewer's comprehensive review and recommendation of our work. Our responses to each point are as follows:\n\n___\n\n**1. Empirically assuming fixed GMM components for per class in latent features.**\n\nConsidering the generalizability of our method, we chose to use a fixed number of components (K=3) per class for all incremental scenarios, based on the results presented in Table 4 of the paper. We show additional results on VOC 15-1 using more values of K as follow: \n\n**K**|**0-15**|**16-20**|**all**\n:-|-|-|-|\n1|79.4|51.8|72.8 \n2|80.0|52.7|73.5 \n3|80.1|53.6|73.8 \n4|79.9|53.2|73.6 \n5|79.6|53.1|73.3 \n6|79.6|53.5|73.4 \n7|79.8|53.3|73.5 \n\nThe results suggest that the overall performance of our method remains stable across most K values, indicating that it is not highly sensitive to the selection of this hyperparameter. Additionaly, the results reveal a slight decline in performance when the number of Gaussian components beyond three. We speculate that using a large number of Gaussian components may lead to partial overfitting, particularly for classes with simple feature distributions in the training set. Therefore, we choose K=3 for the final model, holding performance and computational efficiency simultaneously. Obviously, using a fixed number of GMM components is unlikely to be fully robust across all classes, because the complexity of feature space varies significantly between different classes, as discussed in Appendix  A.6 of our manuscript. We look forward to addressing this issue in future work by assigning an adaptive number of Gaussian components to each class, enabling the model to better adapt to varying levels of feature distribution complexity across different incremental scenarios. We will include these results and give more disscussion in the final version.\n\n\n____\n\n**2. Mathematical notations.**\n\nWe apologize for the unclear explaination of some mathematical notations in the manuscript. We will review these notations and provide detailed explanations in the final version. Following your valuable suggestion. Below is a summary table of all symbols with subscripts/superscripts, along with their corresponding notations, presented as comprehensively as possible. Please note that some vector symbols in the table below may be displayed unnormally due to platform limitations. We will present the standard symbols in the final version. In addition, some typos of formulas and symbols might result in misunderstanding (e.g. $\\mu_{n_o}$  should be ${\\mu}_{n_c}$ in Eq. 10.). We will modify them in the final version.\n\n**Symbols**|**Notations**|**Symbols**|**Notations**\n:-|-|-|-|\n$\\mathcal{M}^t$|segmentation model at $t$ step|$\\mathcal{D}^t$|training dataset at $t$ step\n$\\mathcal{C}^t$|class set at $t$ step|$\\mathcal{C}_b$|background class\n$f^t$|feature extractor at $t$ step|$g^t$|classifier at $t$ step\n$\\mathcal{G}^{t}$|Gaussian Mixture Models (GMM) at $t$ step|$\\boldsymbol{\\mu}_{ck}$|mean vector of $k$-th mixture component in class $c$\n$\\boldsymbol{\\Sigma}_{ck}$|covariance matrix of $k$-th mixture component in class $c$|$\\pi_{ck}$|probability of of $k$-th mixture component in class $c$\n$\\tilde{\\boldsymbol{y}}_{i}^{t}$|downsampled label of $i$-th image at $t$ step|$N_c$|number of pseudo-features to be generated for class $c$\n$\\mathcal{F}_c$|pseudo-features for class $c$|$p^t$|predictions obtained from $t$-th step segmentation model $\\mathcal{M}^t$\n$\\tilde{m}_n$|downsampled mask of $n$-th image|$F_{c}$|real features for class $c$\n$\\boldsymbol{\\mu}_{o_c}$|geometric centroid for features of old class $c$|$\\boldsymbol{\\mu}_{n_c}$|centroid for features of new class $c$\n\n___\n**3. Usage of GMM features during classification.**\n\nWe appreciate your thoughtful recommendations. At step $t+1$, we utilize the storaged GMMs to generate pseudo-features for the old classes. Specifically, for each old class $c$, we reproduce the $N_c$ pseudo-features by per training epoch: $\\mathcal{F}\\_c = \\mathcal{S}(\\phi_c^*, N_c)$, where $\\phi\\_c^{\\*}$ denotes the parameters of GMMs and $\\mathcal{S}(\\cdot)$ represents Gaussian sampling function. Then, pseudo-features are concatenated with the real features $F$ extracted from the input training images by the feature extractor $f^{t+1}$, and jointly fed into the classifier $g\\^{t+1}$. Consequently, we obtain the corresponding pseudo-prediction $p$. We compute a standard mBCE loss by: $$\\mathcal{L}\\_{mbce}=-\\frac{1}{N}\\sum_{j=1}^{N}\\sum_{c \\in   \\mathcal{C}^{t+1}}w\\cdot y_{j, c}\\cdot\\log p_{j, c}+(1-y_{j,c})\\cdot\\log(1-p_{j,c})$$ Here, $N$ is the number of reproduced pseudo-features, $j$ is the index, $\\mathcal{C}^{t+1}$ is the set of foreground classes at $t+1$ step, and $w$ is the positive weight. $y_{j,c}$ is the ground-truth of the $j$-th features sample (0 means it does not belong to class $c$, 1 means it does). In this way, the classifier $g^{t+1}$ can capture discriminative characteristics between new and learned classes in the feature space, achieving a good balance between stability and plasticity. We will include the such mathematical formulations and description in the final version."}]

Author rebuttal:

Meta-review: The paper proposes a class-incremental semantic segmentation method that models feature distributions using Gaussian Mixture Models (GMMs) to mitigate catastrophic forgetting, enhanced by a dynamic adjustment strategy for refining old-class features and a Gaussian-based loss to maintain discriminability between old and new classes.

The reviewers acknowledged several strengths of the proposed method: 1) a novel approach using Gaussian Mixture Models (GMMs) to generate features for previously learned classes in class-incremental semantic segmentation; 2) effective design of its dynamic adjustment strategy  for coping distribution shifts during incremental updates and its Gaussian-based representation constraint loss for preserving class separability between old and new classes; 3) strong empirical performances on Pascal VOC and ADE20K, particularly in long-term incremental settings. However, they also raised significant concerns initially regarding: 1) limited technical novelty w.r.t. prior work (STAR); 2) Insufficient experimental analysis on the effectiveness of using multiple Gaussians (K=3) and lack of experiments comparing computational and memory costs with previous methods; 3) unclear mathematical formulations and insufficient explanation of how the segmentation network integrates the GMM features during classification; and 4) discrepancies in baseline results compared to previous work, raising questions about fair comparison.

In their rebuttal discussion, the authors provided a detailed response including 1) further clarifications on novelty w.r.t. related work and technical design, 2) additional ablation (e.g., K values), computational cost results and experimental comparisons, 3) reproduced joint training results to show the fairness of our comparison.  The rebuttal addressed most of the reviewers' concerns, and all four reviewers responded positively with three of them raising their scores. Specifically, Reviewers dzLn and zb2d recommended acceptance, while the other two reviewers gave a positive assessment, conditioned on including the additional qualitative analysis of feature distributions.

The AC agrees with the reviewers' overall assessment: the paper presents a well-motivated and empirically effective approach to class-incremental semantic segmentation, demonstrating strong performance. While questions remain regarding the novelty/necessity of its GMM-based design, the paper’s technical contributions and experimental results justify acceptance. The authors are expected to address all reviewer concerns—including those related to novelty, GMM motivation, and technical clarity—in the final revision, incorporating insights from their rebuttal.
Decision: Accept (poster)