Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation
Authors: Jinpeng Chen
Runmin Cong
Yuxuan LUO
Horace Ip
Sam Kwong
Venue: NeurIPS 2023 poster (2023)
Keywords: Continual learning
Class-incremental semantic segmentation
Prototype replay
TL;DR: We propose STAR, a method that uses prototype replay to tackle distribution discrepancies between single-step training sets and the complete dataset in class-incremental semantic segmentation, saving storage and achieving SOTA performance.

Abstract: Existing class-incremental semantic segmentation (CISS) methods mainly tackle catastrophic forgetting and background shift, but often overlook another crucial issue. In CISS, each step focuses on different foreground classes, and the training set for a single step only includes images containing pixels of the current foreground classes, excluding images without them. This leads to an overrepresentation of these foreground classes in the single-step training set, causing the classification biased towards these classes. To address this issue, we present STAR, which preserves the main characteristics of each past class by storing a compact prototype and necessary statistical data, and aligns the class distribution of single-step training samples with the complete dataset by replaying these prototypes and repeating background pixels with appropriate frequency. Compared to the previous works that replay raw images, our method saves over 100 times the storage while achieving better performance. Moreover, STAR incorporates an old-class features maintaining (OCFM) loss, keeping old-class features unchanged while preserving sufficient plasticity for learning new classes. Furthermore, a similarity-aware discriminative (SAD) loss is employed to specifically enhance the feature diversity between similar old-new class pairs. Experiments on two public datasets, Pascal VOC 2012 and ADE20K, reveal that our model surpasses all previous state-of-the-art methods.

Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation
Jinpeng Chen', Runmin Cong?*;, Yuxuan Luo', Horace Ho Shing Ip", and Sam Kwong 'Department of Computer Science, City University Hong Kong, Hong Kong School of Control Science and Engineering, Shandong University, Jinan, Shandong, China LLingnan University, Hong Kong
j jinpechen2-c@my cityu. edu hkk rucongosdu. edu. cn, yuxuanluo4 casaanloo-ccom cemmycittyu cityu. edu. hk, horace. wew.eeii.piit.ee.e edu. hk, samkwong@ln, edu.
edu.
Abstract
Existing caass-incrementll semantic segmentation (CISS) methods mainly tackle catastrophic forgetting and background shift, but often overlook another crucial issue. CISS, each step focuses on different foreground classes, and the training set for single step only includes images containing pixels of the current foreground classes, excluding images without them. This leads to an overrepresentation these foreground classes in the single-step training set, causing the classification biased towards these classes. To address this issue, we present STAR, which preserves the main characteristics of each past class by storing a compact prototype and necessary statistical data, and aligns the class distribution of single-step training samples with the complete dataset by replaying these prototypes and repeating background pixels with appropriate frequeney. Compared to the previous works that replay raw images, our method saves over 100 times the storage while achieving better performance. Moreover, STAR incorporates an old-class features maintaining (OCFM) loss, keeping old-class features unchanged while preserving sufficient plasticity for learning new classes. Furthermore, similarity- discriminative (SAD) loss employed to specifically enhance the feature diversity between similar old-new class pairs. Experiments on two public datasets, Pascal VOC 2012 and ADE20K, reveal that our model surpasses all previous state-of-the- methods. The official code is available at https: :ttp::///iiii //github. com/ wttp:/ooooonnnno///eeee jiipennn528/STAA
Introduction
Typically, neural networks [6. 15, 16] are trained in one-time fashion.When data updates occur, retraining on the entire dataset is often necessary, as finetuning solely on new data leads to catastrophic forgetting [21] of previously learned knowledge. However, frequent retraining is time-consuming and requires the storage of all past data. Consequently. there is a growing interest in enabling networks to achieve continual learning, which aims to attain performance comparable one-time training by progressively learning from new data.
RRummin Cong is also with the Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Jinan, Shandong, China.
'Horace Ho Shing Ip is also with the Centre for Innovative Applications of Internet and Multimedia Technologies, City University of Hong Kong, Hong Kong. *Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While most continual learning research has focused on image classification, the growing expectation for computers to perform various tasks has driven the expansion of research into other domains. One such domain that has gained considerable attention is class-incremental semantic segmentation (CISS). In CISS, the training process involves multiple steps, with each step focusing on different classes. During each step, only part of the classes is considered foreground and assigned ground-truth labels, while foreground classes from past or future steps are labeled as background. Compared to other continual learning tasks, CISS must address not only catastrophic forgetting but also background shift [2]. The latter refers to the fact that the background in the current step may include pixels from past or future foreground classes, causing its semantics to change across steps. Without specific countermeasures, this can greatly confuse the network and result in poor performance.
Previous CISS methods [1. 2, 3, 9, 22, 23] effectively addressed catastrophic forgetting and back- ground shift, but they often neglect another critical issue: the discrepancies in class distributions between single-step training sets and the complete dataset. Single-step training sets include only images with pixels from corresponding foreground classes, resulting in an increased proportion of these classes compared to the complete dataset. For instance, under CISS protocol on the Pascal VOC 2012 dataset [11], overlapped 19-1, 19 foreground classes are trained in the first step, and only tymonitor is in the second step. In this protocol, fv/monitor constitutes 16.5% of pixels in the second step's 's training set but only 0.8% in the complete dataset. Hence, the network may become biased towards n/monitor, causing numerous false positives, where pixels not belonging to tv/monitor are misclassified as such. Though adjusting class weights in the loss function can mitigate this issue, the limited quantity and richness of negative samples (pixels not tv/monitor) impede full grasp of their potential characteristics.
To tackle this issue, we propose a method called STAR (STorage sAving Replay for CISS). After each training step, STAR stores key information, including a compact prototype for each current foreground class, which is derived from the features of this class, and statistical data related to the prototypes or class proportions. Using this information, STAR replays past foreground classes and repeats background pixels at appropriate frequencies during subsequent steps, enabling a thorough reconstruction of single-step training sample distributions with minimal storage cost. Compared to methods that save and replay raw images [1, 31, STAR significantly reduces storage requirements (using only - ~1% of space) while achieving better performance.
also avoids ethical concerns, as the stored data does not contain personal information. Note that although previous splay-based CISS works [1, 3, 20] consider the main function of replay to be combating forgetting, we argue that its more crucial role is in learning new classes, confirmed by [1, 3], where the versions with replay improve the performance on new classes more than old classes. Since new classes are typically trained individually or with few other new classes, there is larger overrepresentation in the single-step training set, leading to a noticeable bias in the new-class classifiers that cannot access enough negative samples. The replayed samples, however, can alleviate this problem by serving as negative samples. Extending this idea, we propose to thoroughly align the class distributions of each single-step training set with the complete dataset up to that step by controlling the replay frequency.
In addition, to coordinate with our replay strategy as well as reduce forgetting, it is important to maintain old-class features during subsequent steps. Otherwise, old-class features will deviate from the saved prototypes, rendering the replay ineffective. To address this, we propose an old-class features maintaining (OCFM) loss, which is spatially-targeted knowledge distillation that constrains the change of old-class features while allowing sufficient flexibility to learn new classes. Moreover, we introduce a similarity-awan discriminative (SAD) loss to selectively enhance discriminability between similar new and old class features, making it easier for the classifiers to distinguish them.
In summary, this paper presents the following contributions: (1) We propose STAR. method that stores necessary statistical data and compact prototypes for learned foreground classes, enabling a comprehensive reconstruction of single-step training sample distributions to align with the complete dataset, thus eliminating the bias towards specific classes. (2) We introduce an old-class features maintaining (OCFM) loss that retains learned knowledge in a spatially targeted manner, preserving old-class features while ensuring sufficient flexibility for learning new classes. We also design a similarity-awar discriminative (SAD) loss to specifically enhance the discrininability of features between similar old-new class pairs, facilitating the classification. (3) On two public datasets, Pascal VOC 2012 [11] and ADE20K [34], our STAR achieves tate-of-thhe-rr performance in CISS.
2

---

2 Related work
2.1 Continual learning
Continual learning aims to enable models to incrementally learn new knowledge while avoiding catastrophic forget old knowledge. In computer vision, most continual learning research focuses on image classification and falls into three categories: regularization-basee -based, rehearsal-basee
and structure-bard based methods. Regularization-bu based methods [17, 8, 10, 4] apply constraints to parameter updates to preserve crucial learned knowledge. Rehearsal-based methods [26, 27, 24] store a small amount of past data or generate additional old-class data for replay during subsequent steps to minimize forgetting. Structure-based methods [19, 28, 18] allocate extra parameters for new classes and keep parameters related to old classes unchanged to preserve knowledge for both.
Similar to our approach, several previous continual learning studies have also aimed at enhancing memory efficiency [14, 30, 32]. For example, [14, 30] utilize compact prototypes to retain old class information and have delved into addressing prototype ineffectiveness caused by shifts in the feature space during incremental training. [32] leverages low-fidelity exemplars rather than prototypes, with focus on narrowing the domain gap between these low-fidelity exemplars and raw images. Our method distinguishes itself in four aspects: (I) We save a single compact prototype per class for better memory efficiency, unlike some prior works [14, 32] which store multiple prototypes or exemplars. (2) We incorporates class distribution during replay, effectively rectifying the classification bias. (3) We store feature statistics beyond prototypes, offering more comprehensive cues of past classes. (4) All these studics concentrate on continual image classification, but our focus is CISS.
2.2 Class-iccremental semantic segmentation
In recent years, growing number of methods have focused on the CISS task. ILT [22] employs knowledge distillation on both output and intermediate features to reduce catastrophic forgetting. predicted probabilities to eliminate semantic contradictions between steps. SDR [23] leverages prototype matching and contrastive learning to enhance discrimination between features of different classes while applying feature sparsity techniques to reserve space for future knowledge. PLOP [9] preserves old-class knowledge using pseudo labels generated by the previous model for supervision and spatially- variant of POD [10]. SSUL [3] introduces an extra unknown class aided by salient object detection models to tackle background shift. DKD [1] adopts decomposed knowledge distillation to maintain not only the output logit but also its positive and negative components.
MiB [2] addresses background shift for the first time, using operations
RBC [33] is the only prior method that has examined the differences in class distributions between single-step training sets and the complete dataset. During incremental training. it employs pairs of input images, one normal and the other with new-class pixels erased. Additionally, it enhances the weight of old-class pixels in the loss function. These strategies do help mitigate the bias of classifiers towards new classes. However, they do not take into account the actual class proportions within the complete dataset when rectifying the bias, offering no assurance that that the classifier's bias can be approach, which replays old-class prototypes based
adjusted to appropriate level. In contrast, on the recorded class proportion in all past training samples, holds an advantage.
3 Proposed method
3.1 Problem definition and basic setup
In CISS, model incrementally learns among T steps. For each step & {1...., ,T). the training No set D' is consisted of AT images (at Nt C HxW33 and corresponding labels {ut) )= HRW where H x W indicates the image size. During step f, only subset of classes, L", is regarded as foreground classes with ground-trun labels in y All other pixels, including the actual background, past classes L 1:t-1, and future classes 1++1:7 are labeled background. Note that the sets of foreground classes for different steps are disjoint, i.e., L' n I 1:t-1 = @ and L' n I.+ 0
The model at step / is denoted as Mr, consisting of feature extractor Dg with parameters 0' and classifiers Te with parameters w², h includes the classifier for each past or current foreground class 1 L Lit represented as da.. Before training at step t, 0f and (wi (wiher initialized by
3

---

Input Imegon (
Feature Fatracio (o Nei
Festunes ( [=
Prototypes (Allcc
conpuii ccateris
Features ft
Lenhee
sad
Gie old clas repiona
Feature Estractor Va--
compate contet)
Loef
neply
old class regions
Background repetiiion
Farraes
Giber badkeroudd
Claseifiens Ob
Ourpor p
Input Images
Feature Extractor Vg*
Overview of the proposed STAR. In earlier steps, prototypes for past foreground classes have been saved. At the current step, prototypes are replayed, and background pixels are repeated for classifiers. The OCFM loss maintains similarity between old-class region features from current and previous feature extractors, while the SAD loss differentiates features for similar old-new class pairs.
Figure 1:
8--1 and (we-" and {wi JEL is randomly initialized. For an input image xf, the output of MM denoted as pf E Rr which can be computed by pl M² = a V (xf). The extracted features are denoted as S E R4xwxe which can be computed by S (af)). Here, h and represent the spatial size and the number of channels, respectively. When feeding a current image zf to previous model M or feature extractor Va the output is given by (x)), and the features are given by f( fl,t-r = Vo--r (x.))
P. L.t-
3.2 Overview of the method
The overview of the proposed STAR is shown in Fig. 1, After training at each step t the prototype of each foreground class E - and essential statistics are computed and saved. At subsequent step i, the input to the classifiers includes not only the features extracted from the input image but also proportionally replayed prototypes of all past foreground classes as well as repetitions of the features in the background region, aligning the sample distribution with the training set accumulated to this step. D1: Moreover, we employ the OCFM loss, Loefm. between the features f and f/ to preserve old-class features and apply the SAD loss, Land. to differentiate the features between similar old-new class pairs. Consequently, our total loss function is:
L Libbe aLuefm 3L
Here, Lmboe denotes multiple binary cross-entropy (mBCE) loss with a positive weight of y in 3], which is proved to be more suitable for CISS than multi-class cross-entropy loss. Loefm and Load denote the OCFM and SAD losses, respectively, balanced by hyperparameters a and 3.
3.3 Prototype preservation and replay
As previously stated, in single-step training set, the proportion of pixels from current foreground classes is excessively high, resulting in biased predictions. To address this issue, aim to com- prehensively reconstruct the class distribution the distribution in the union of all training sets up to this step D This distribution represents the achievable limit, as it is not possible to know the data from future steps in practice, and it should also be close to the distribution in the complete dataset accumulated up this step. The most straightforward solution to this problem to save and replay all past images, but this does not meet the requirements of continual learning, necessitating an alternative solution.
training samples at each step t, aligning with
We argue that the feature extractor relatively generalized in extracting features and does not contain information strongly correlated with the classes. The ability of SSUL [3] to learn new classes even

---

if the feature extractor parameters are frozen supports this claim. As result. the network's 's bias primarily stems from the classifiers. Therefore, we propose creating a prototype for each learned foreground class and replaying these prototypes in subsequent steps, allowing them to go through the classifiers together with the features extracted from input images. This process can correct the foreground class distribution accessed by the classifiers, eliminating its bias. Meanwhile, we have to ensure that each prototype contains the most representative characteristics of its corresponding foreground class with minimal storage cost, thereby efficiently maximizing the correction for the classifiers.
Specifically, after training at each step t - т, we calculate the number of pixels from each current foreground class I L in the single-step training set D
N hxm
10 = 17.
Here, j indicates the spatial location, denotes the indicator function. 56- E |gxxw represents the downsampled label aligned with the spatial size of the features ʃ that fed into the classifiers. is given by -/-- DS((('") where DS denotes a downsampling operation. Since each class treated foreground class in only one step. T/I the count of pixels from class appearing as foreground throughout the entire dataset, which determines the frequency of replaying the prototype of class in subsequent steps. Still, due to the possible presence of small number of / pixels appearing in other steps' training sets as background, we utilize the previous model to estimate the number of I pixels in the current training set fine-tune 17.. Yet, as this aspect has minimal influence on the final outcomes, we deem it optional and provide clarification in the supplementary material.
Next, in order build replay samples of class 1 in future steps, we calculate the feature center of i to capture its primary characteristies by:
Σ Ej (i.J 11015 = =}}) ENT ES U155" x 1675 = -1)) ERC
14
where & denotes the element-wise multiplication, ||-Il2 computes the L2-norm, and fi.j" refers the b4 indicates the mean direction of all feature vectors for I, serving as prototype of that embodies the typical characteristic. However, relying solely on this prototype during the replay in subsequent steps may result in inadequately robust training outcomes, as a prototype only provides a single representation of class features without encompassing any variability. Therefore, also compute the standard deviation of the feature vectors to introduce noise during the replay process. The calculation given by:
L2-normalized feature vector given by fiy 13
Nt-r Ix
( - -2)² x 1 =13] ER.
=
In addition, since we remove the length information of feature vectors through .2-normalization, we need restore during replay. Hence, we also store the mean and standard deviation of the 1.2-norms of all features for class I, denoted as pl.norm and dtnorm, which are formulated by:
- hxw
P1,norm
Σ
fts
x x = 11) ERI
(5)
t=1 1=1
Ni-T hxw
Imnorm
- M1 )2 x 1 167 = =13 ER!
(6)
With ne pt, TJ, Pt,norm and Tinorm saved, robust replay in future steps will be available. At a subsequent step i,, the classifier would originally fed only the features extracted from the input image, f.. To correct the ratio between old and new foreground classes, we also input replayed old
5

---

class samples into the classifier. For an old class 1 E L we use random variable to represent the replayed sample, which the product of two Gaussian-distribuce random variables, the feature vector direction K G(u ((") and the 1.2-norm À ~ normo U1norm 2):
X.
For each old class I, we replay 17 for 17 times per training epoch, which represents the number of pixels from | that appear outside D In each epoch, all foreground class pixels in D' also appear once, so replaying Ti times can align the ratio of old and new foreground class samples with the ratio in Dist, Assuming each epoch contains iterations, we distribute the Tj times evenly among these iterations. This implies sampling ry for Tit times in each iteration and concatenating these random samples with f to feed the classifiers for calculating the total gradient to update parameters. In this way, the bias towards any foreground classes eliminated. Simultaneously, as each sampling of r yields different features, and the distribution is controlled by stored standard deviations, the classifiers can capture sufficient potential characteristics of old classes, achieving robust discriminability.
3.4 Background repetition
Apart from the ratio between samples of old and new foreground classes, we have to also restore the proportion of background samples to fully rectify the distribution. However, due to the richer semantics contained in the background and the background shift across different steps, condensing its meaning through single prototype is impossible. Therefore, we choose to repeat background pixels in the current training set D' with a proper frequency to restore their proportion.
Similar to the strategy in Sec. 3.3, control the number of background repetitions in future steps, we calculate the number of appeared background pixels at each step / - т. As numerous background pixels exist in all steps, we not only count the number of background pixels in the single-step training set but also add the background pixel counts saved in previous steps, given by:
EEST EJ + Σj I 10 = 0} if -f
hxw Li
0}
if -
Tog
(8)
Thg
where 0 in " indicates the label background. In fact, since there may be overlapping background pixels between the training sets in different steps, we scale down - based on some prior assump- tions. However, similar to the case of foreground class counts, the impact is minimal, is optional and described in the supplementary material.
At each incremental step t, restore the proportion of the background by repeating background pixels in the current training set D by a certain time. Specifically, we consider a pixel in f as a background pixel if it meets two conditions: 1) its label is background, and 2) the model at the last step M also predicts it as background. Ideally, these requirements ensure the pixel is neither current nor past foreground pixel and can be currently categorized as background. Using these criteria, we obtain the set of background pixels B":
B' [fi vi.5 = O and Pt.j +..-- = =0,
(9)
where P1. DS(())-1) is the downsampled prediction of M During training, repeat The pixels in Bd by T times per epoch, where denotes set cardinality, to integrate nhg additional
background pixels, with Thy representing the count of background pixels outside D', This aligns the background pixel proportion with that in Di:t, By merging prototype replay and background repetition strategies, we rectify the distribution of both foreground classes and background in single- step training samples for the classifier.
3.5 Old-class features maintaining loss
The prototype replay strategy introduced in Sec. 3.3 can correct the proportion of old-class samples while conserving storage space. However, its feasibility relies on the premise that feature extraction for old-class regions does not change significantly between steps. If this condition is not met, there will be a discrepancy between the old-class features extracted by the current feature extractor and the replayed old-class prototypes, as the prototypes are created from features generated by previous

---

feature extractors. In this situation, the replay becomes ineffective because the features that the prototypes represent have become outdated. Furthermore, since there are no constraints on parameter updates, the feature extractor may suffer from uncontrolled catastrophic forgetring of old knowledge. Consequently, our goal develop a strategy that maintains old-class features, cooperating with our replay strategy and resisting catastrophic forgetting. At the same time, we do not want the network to lose plasticity for learning new classes. By balancing these objectives, we design the OCFM loss, which selectively maintains features of old-class regions in spatially targeted manner while simultaneously allowing flexibility for learning new classes.
OCFM maintains old-class features by penalizing the changes from the features extracted by the previous feature extractor, f to the features extracted by the current feature extractor, fj. in old-class regions. For a pixel, we consider it belongs to old-class regions by two requirements: 1) its current label is background, and 2) the previous model M predicts it as a foreground pixel. For these old-class pixels, the OCFM loss punishes the mean squared error between S and S f!.t-1 by:
Laofm = MSEU? x 16 = 0} x 1 = =0},
(10)
f/ 1 0} x 1P 0}),
in which MSE signifies the mean squared error.
Compared to distillations that operate on the entire spatial region [1, 1,9,31], our OCFM loss specifically targets old-class regions, preserving these features to collaborate with our replay strategy and combat catastrophic forgetting. Simultancously, the features in other regions can be updated without constraints, ensuring that the flexibility to learn new classes remains uncompromised.
3.6 Similarity-aware discriminative loss
The previous sections address the bias resulting from distribution differences between single-step training sets and the complete dataset. Despite the bias being eliminated, some similar class pairs may still be prone to confusion, as the feature extractor might not generate sufficiently discriminative features for them. When these similar classes are trained in the same step, the situation may be less severe since the mBCE loss drives their features to exhibit adequate differences. However, when they trained in different steps, the mBCE loss alone is not enough to encourage the feature extractor to generate discriminative features. This is because the old-class features are constrained by the OCFM loss, and most old-class representations that eventually flow to the mBCE loss are derived from prototypes, which, although close to the authentic features, are slightly inferior in richness.
To address this problem, straightforward solution is to introduce differentiation loss to all old-new class feature center pairs, sceking to maximize their divergence. However, this lacks targeting. as some feature pairs are already widely separated, and we should focus on those similar pairs. To this end, we refine this strategy to form our SAD loss: for each current class, we only penalize the distance between its feature center and the nearest old-class feature center, thereby directing the differentiation effort to where it is most needed. At step t, the feature center of an old class l, E L Il:t-1 is the previously saved prototype, μl,,, and the feature center of a current class E L is computed within the batch being processed, given by:
Eiev Exar (Pay
1 13)
(11)
EiEv Σ/ (ft,, Li = (3) 13)
where U represents the set of indices of images belonging to the current batch. After computing the feature centers for all current foreground classes, we can achieve the SAD loss as:
Lsad
1
Σ LEI DELLE11
min Ilrr. - -iull12-
(12)
[
With Lead, the features of each new class will be distanced from the most similar old-class prototype. When the distance between a new-class feature center and its closest old-class prototype is far enough, surpassing the distance to another old-class prototype, that old-class becomes the new closest one, and Lsad turns to penalize the distance between this new pair. Ultimately, the features of the new class will be far enough away from all old class prototypes.
7

---

Table 1: Quantitative comparison on Pascal VOC 2012 between our STAR and previous non-replay methods (top half) and replay-based based methods (bottom half) under the overlapped setting.
19-1 (2 steps)
15-5 (2 steps)
15-1 (6 steps)
10-1 (II steps)
5.3 (6 steps)
Model
old
all
old
old
old
all
old
all
MiB [2]
69.0
13.5
29.7
12.3
13.1
12.7
57.1
42.6
SDR [23]
69.1
32.6
67.4
75.4
52.6
69.9
44.7
21.8
39.2
54.6
44.0
15.5
SSUL
77.7
29.7
75.4
77.8
50.1
71.2
77.3
36.6
67.6
71.3
46.0
59.3
72.4
50.7
$6.9
RCIL [31]
78.8
52.0
72.4
70.6
23.7
59.4
55.4
1.511
34.3
RBC [35]
77.3
55.6
76.2
76.6
52.8
70.9
69.5
38.4
62.1
DKD [1]
77.8
41.5
76.0
78.8
58.2
73.9
78.1
42.7
69.7
73.1
46.5
60.4
69.6
53.5
58.1
1291
21.6
55.1
42.3
28.3
AWT [12]
78.0
50.2
71.4
77.0
37.6
67.6
73.1
47.0
60.7
71.6
51.4
57.1
79.5
50.6
72.6
73.1
RECALL [20]
68.1
55.3
68.6
67.7
54.3
65.6
67.8
50.9
64.8
65.0
53.7
60.7
SSUL-M(1001
73.0
78.4
71.4
74.0
64.1
DKD MI 100) [1]
78.0
57.7
77.0
79.1
60.6
74.7
78.8
52.4
72.5
74.0
56.7
65.8
69.8
60.2
62.9
M((0)
79.5
55.6
73.8
74.3
Table 2: Quantitative results ADE20K under the overlapped setting.
100-50 (2 steps)
100-10 (6 steps)
50-50 (3 steps)
Model
old
all
old
new
all
old
new
all
MiB [2]
37.9
27.9
34.6
31.8
14.1
25.9
35.5
22.9
27.0
PLOP [9]
41.9
14.9
32.9
40.5
13.6
31.6
48.8
21.0
30.4
39,3
17.6
32.1
RBC [33]
42.9
21.5
35.8
39.0
21.7
33.3
49.6
26.3
34.2
UCD [29]
42.1
15.8
33.3
40.8
15.2
32.3
47.1
24.1
31.8
AWT
40.9
24.7
35.6
39.1
21.3
33.2
46.6
26.9
33.5
SSUT (M300) [3]
42.8
17.5
14.4
42.9
17.7
34.5
49.1
20.1
29.8
DKD-M(300) [1]
42.4
23.0
36,0
41.7
20.1
34.6
48.8
26.3
33,9
Ours
42.4
24.2
36.4
42.0
20.6
34,9
48.7
27.2
34.4
4 Experiments
4.1 Experimental setups
Datasets and evaluation metric. We evaluate our method on two common public datasets: Pascal VOC 2012 [11] and ADE20K [34]. Pascal VOC 2012 comprises 10,582 training images and 1,449 validation images, covering 20 classes. ADE20K includes 20,210 training images and 2,000 validation mean Intersection-over-- Union (mloU) as our evaluation metric, representing the average loU across all classes
images, spanning 150 classes. Following previous works [1, 2, 3],
Experimental protocols. Following previous works, we evaluate our model with various class splits among multiple steps. Each split is denoted as N2- where N1 and N2 refer to the number of classes in the first step and in each incremental step, respectively. We have 19-1, 15-5, 15-1, 10-1, and 5-3 splits for Pascal VOC 2012, and 100-50, 100-10, and 50-50 splits for ADE20K. Furthermore, consider two CISS settings: disjoint and overlapped. The difference between them is that the disjoint excludes images with future class pixels in Dí. while the overlapped includes them (future class pixels are labeled background). Since the overlapped is closer real-world scenarios, report the results of it in this paper and put the results of the disjoint in the supplementary material.
Implementation details. Following [1, 2, 9]. we use DeepLabvv [5] with ResNet-10 101 [13] backbone pre-trained on ImageNet [7] as our segmentation network. line with 2, 9], training strategies differ for the two datasets. For Pascal VOC 2012, we train for 60 epochs with initial learning rate of 0.001 for the first step and 0.0001 for incremental steps, and empirically set y to 4. For ADE20K, we train for 100 epochs with initial learning rate of 0.00025 for the first step and 0.000025 for incremental steps, and set 7 to 30. For both datasets, we use an SGD optimizer with a momentum of 0.9. The batch size is set 24, and and 3 set to 5 and 0.05, respectively. We implement our model using Py PoTch [25] and employ two NVIDIA RTX 3090 GPUs for acceleration. We have also implemented our network using the MindSpore Lite tool!
'https:/ :/wp::///wwwww . mindspore.ccmmmmmmmmmmm cn/
8

---

old chooes
acroplanc
bicycle
bird
bost
buttle
bus
cow
diningtable
dag
motcebike
pesson
new classes:
putted plant
shoup
train
Input image
PLOP
SSUL
DKD
STAR (Ours)
Ground-truth
Figure 2: Qualitative comparison Pascal VOC 2012 between STAR and previous methods.
4.2 Comparisons
We compare STAR with previous CISS methods both quantitatively and qualitatively. Among them, MiB [2]. SDR [23], PLOP [9]. SSUL [3], RCIL [31], RBC [33], DKD [1], UCD [29], and AWT [12]
non-replay-based while RECALL [20], SSUL-M [3], and DKD-M [1] replay-based methods.
Quantitative comparison. The quantitative comparison results for Pascal VOC 2012 and ADE20K datasets are presented in Table and Table 2, respectively, where "old" indicates the mloU of classes the first step while "new" indicates the mloU of incremental classes. Compared to previous non-replay- -base models, our STAR consistently achieves leading performance across various class splits. Particularly in more challenging 15-1, 10-1, and 5-3 splits that contain more steps, our method outperforms others by least 2.9, 4.0. and 6.3 mloU, respectively. Notably, although STAR requires additional storage space for prototypes and statistics, the space needed is minimal (-41KB), causing significant unfairness compared non-replay-bare baasdd models. Meanwhile, even compared with other replay- baaed models, STAR remains highly competitive, achieving performance on par with the best-performing DKD-M while requiring less than 0.5% of its storage space (DKD-M cost -10MB to save 100 raw images from past steps). we add 50 extra raw images to provide more realistic features (Ours-M), the performance of our model can be further improved, surpassing DKD-M in four out of five class splits. Remarkably, even in this scenario, the required storage is just about half of that needed by DKD-M.
On the ADE20K dataset, our model performs even better, outperforming all previous non-replay- based and replay-basee methods without storing any extra raw images. Despite the larger class count on ADE20K necessitating the storage of more prototypes, the required storage remains a mere ~2% of that demanded by DKD-M and SSUL-M, both of which store 300 raw images.
In summary, our model achieves outstanding performance while requiring, on average, only -1% of the storage needed by other eeplay-based models, and it requires no extra image beyond the datasets. Moreover, privacy concerns tied to storing raw images are avoided. These demonstrates significance of our approach to reconstruct class distribution through prototype replay.
Qualitative comparison. Fig. 2 provides a qualitative comparison using the challenging overlapped 15-1 protocol on Pascal VOC 2012. highlighting our method's superiority over previous state-of-the- CISS methods [1, 3, 9]. The prevalent errors in other methods are false positives for new classes. For instance, in the first two examples, some pixels of old classes bus and bird are identified as new classes train, sheep, or tv/monitor. In the third example, some background pixels are misclassified as new classes sofa or nv/mnnior. Such errors stem from the underrepresentation and limited diversity of old-class or background samples in the single-step training set when training new classes, resulting in bias towards them. In contrast, our method effectively addresses these problems by rectifying the distribution of single-step training samples and providing new-class classifiers with access to old-class prototypes negative samples.

---

Table 3: Ablation study results of our method in the Pascal VOC 2012 dataset.
overlapped 15-1 steps)
Replay
BR
OCFM
SAD
D2W
all
4
V
79.3
41.8
70.3
35.5
d
37.3
12.8
31.4
79.3
48.2
71.6
v
v
79.5
50.6
72.6
4.3
Ablation study
Table 3 shows our ablation study results for STAR under the challenging overlapped 15-1 protocol, assessing its four core components: prototype replay (Replay), background repetition (BR). old-class features maintaining loss (OCFM), and similarity-aware discriminative loss (SAD). Each component is individually removed from the full model to evaluate its contribution to the overall performance.
Effectiveness of the prototype replay strategy (Replay). The first row of Table 3 presents the results without Replay. Compared to the full model's results shown in the last row, it is clear that Replay boosts the performance by 2.3 mloU. Although Replay involves replaying old-class prototypes, primarily enhances the accuracy of new classes, demonstrating an 8.8 mloU improvement. This effect arises from Replay reducing the proportion of new-class samples flowing into the classifiers and providing new-class classifiers with sufficient access to old-class features. This dual effect mitigates classifier bias and lessens the chance of misclassifying non-new-clas pixels, especially those old-class pixels, new-class pixels. Consequently, we observe significant increase in the mloU of new classes and slight improvement in the mloU of old classes.
Effectiveness of the background repetition strategy (BR). As displayed in Table 3, excluding BR (second row) leads performance drop compared to the full model (last row). Specifically, BR brings a 4.5 mloU improvement overall and notably boosts the mIoU of new classes by 15.1. Despite simply repeating background pixels from the current training set, BR effectively corrects the classifier bias due to the high proportion and diversity of background pixels. even within single-step training set. Likewise, because of the high proportion of background pixels, BR introduces more additional samples than Replay, making its removal lead to more noticeable performance degradation.
Effectiveness of the old-class features maintaining loss (OCFM). The significance of OCFM can be recognized when comparing the third and final rows of Table 3. Upon removal of OCFM, the results experience a complete breakdown, with dramatic 41.2 mloU drop compared the full model. OCFM is crucial mechanism in our model for maintaining prior knowledge. Its absence not only makes replay ineffective but also triggers uncontrolled network parameter updates, leading to intense catastrophic forgetting. Thus, the performance of both new and old classes suffers greatly.
Effectiveness of the similarity-aware discriminative loss (SAD). The final component under validation SAD, designed to differentiate the features of similar old-new class pairs. The penultimate row in Table 3 illustrates the performance without SAD, registering 1.0 mloU reduction. This attests to the efficacy of SA/) in augmenting feature differences between similar old-new class pairs, thereby making them more distinguishable to the classifiers, leading improved prediction accuracy.
5 Conclusion
In this paper, we present a novel class-incremental semantic segmentation (CISS) method named STAR that addresses the bias towards part of classes due to distribution discrepancies between single- step training sets and the complete dataset. STAR incorporates two key strategies: prototype replay and background repetition, aiming to correct the proportions foreground classes and background, respectively. By saving and replaying prototypes instead of raw images, STAR reduces storage costs -1% of other replay-bo -bsed methods, without requiring extra data. Furthermore, we propose an old- class features maintaining loss to keep old-class features unchanged while preserving the flexibility to learn new classes and a similarity-aware discriminative loss to selectively differentiate features of most similar old-new class pairs, making them distinguishable for the classifiers. Comparisons with previous state-of-the-an CISS methods and our ablation study demonstrate the superiority of our overall design and the importance of each individual component in model.
10

---

Acknowledgements
This work was supported in part by the National Key R&D Program of China under Grant 20212D01 12100, in part by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA), in part by the Hong Kong GRE-RGC General Research Fund under Grant 1 1120981 19 and Grant 12203820, in part by the National Natural Science Foundation of China under Grant 62002014, 44, in part by the Taishan Scholar Project of Shandong Province under Grant tsqn202306079, in part by Young Elite Scientist Sponsorship Program by the China Association for Science and Technology under Grant 2020QNRC001, in part by CAAI-Huawei MindSpore Open Fund.
References
[1] Donghyeon Back, Youngmin Oh, Sanghoon Lee, Junghyup Lee, and Bumsub Ham. Decomposed knowl- edge distillation for class-incremental semantic segmentation. In NewrlPS, 2022.
[2] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. CVPR, 2020.
[3] Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al. SSUL; Semantic segmentation with unknown label for exemplar-based class-incremental learning. In NewrIPS, 2021.
[4] Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
[5] Liang-Chie Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint /706.0/706.05558 2017.
[6] Rummin Cong, Yumo Zhang, Leyuan Fang. Jun Li, Yao Zhao, and Sam Kwong, RRNet: Relational reasoning network with parallel multiscale attention for salient object detection optical remote sensing images. IEEE Trans. Geosci. Remote Sens., 60:1-11, 1, 2021.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In CVPR, pages 248 244,225,,10 2009.
[8] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chune Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, 2019.
[9] Arthur Douillard, Yifu Chen, Amaud Dapogny, and Matthieu Cord. PI.OP: Learning without forgetting for continual semantic segmentation. In CVPR, 2021.
[10] Arthur Douillard, Matthicu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small- tasks incremental learning. In ECCV, 2020.
[11] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis., 88:303-338, 2010.
[12] Dipam Goswami, René Schuster, Joost van de Weijer, and Didier Stricker. Attributionawarc weight transfer: warm-start initialization for class-incrementee semantic segmentation. WACV, 2023.
[13] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 778,-78,77 2016.
[14] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In ECCV, 2020.
[15] Chongyi Li, Runmin Cong. Junhui Hou, Sanyi Zhang, Yue Qian, and Sam Kwong. Nested network with two-stream pyramid for salient object detection in optical remote sensing images. IEEE Trans. Geosei. Remote Sens., (7116691566 9166, 2019.
[16] Chongyi Li, Rummin Cong, Sam Kwong, Junhui Hou, Huazhu Fu, Guopu Zhu, Dingwen Zhang, and Qingming Huang. ASIF-Net: Attention steered interweave fusion network for RGB-D salient object detection. JEEE Trans. Cybern., 31(1):88-100 2020.
[17] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach, Intell. 40(12):2935 2221299999999220002 2017,2017
11

---

[18] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning mask weights. In ECCV, 2018.
[19] Arun Mallya and Svetlana Lazebnik. PackNet: Adding multiple tasks to single network by iterative pruning. In CVPR, 2018.
[20] Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanultigh. RECALL: Replay-based continual learning in semantic segmentation. In ICCV, 2021.
[21] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, 1989.
[22] Umberto Michieli and Pietro Zanuttigh. Incremental learning techniques for semantic segmentation. In ICCVW, 2019.
[23] Umberto Michicli and Pietro Zanuttigh. Continual semantic segmentation via repulsion-attractinno of sparse and disentangled latent representations. In CVPR, 2021.
[24] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember:
A synaptic plasticity driven framework for continual learning. In CVPR, 2019.
[25] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Veto Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[26] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Inere- mental classifier and representation learning. In CVPR, 2017.
[27] Hanul Shin, Jung Kwon Lec, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NeurlPS, 2017.
[28] Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental lcarning. In CVPR, 2021.
[29] Guanglei Yang. Enrico Fini, Dan Xu, Paolo Rota, Mingli Ding. Moin Nabi, Xavier Alameda-Pineda. and Elisa Ricci. Uncertainty-aware contrastive distillation for incremental semantic segmentation. IEFF Trans. Pattern Anal. Mach. Intell., 45(2) 2023.
[30] Yu, Bartlomicj Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van Weijer. Semantic drift compensation for class-iccremental learning. CVPR, 2020.
[31] Chang- Bin Zhang, Jia- Wen Xiao, Xialei Liu, Ying-Cong Chen, and Ming-Ming Cheng. Representation compensation networks for continual semantic segmentation. In CVPR, 2022.
[32] Hanbin Zhao, Hui Wang, Yongjian Fu, Wu, and Li. Memory-efficieen class incremental learning for image classification. IEFE Trans. Neural Netw. Learn. Syst. 33(10)55966-55 5977, 2021.
[33] Hanbin Zhao, Fengyu Yang, Xinghe Fu, and Xi Li. RBC: Rectifying the biased context in continual semantic segmentation. In ECCV, 2022.
[34] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. CVPR, 2017.
12


Reviews:
[{"review_id": "CCmp3kN6MT", "reviewer": "Reviewer_YBrR", "summary": "The paper utilizes prototypes to store information which saves over 100 times the storage while achieving better performance. It also, introduces old-class features maintaining loss and a similarity-aware discriminative loss.", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The performance of this paper is exciting as it achieves SOTA performance while saving tons of memory compared to existing replay based methods.\n\n2. The code of the paper is provided and detailed implementation details are given, which makes the paper easy to replicate.\n\n3. The paper is overall clearly written and easy to follow.", "weaknesses": "1. The use of prototypes to store past information is widely used in CSS including the paper SDR. It would be great if the authors can make a clear distinguish of the novelty of this paper and previous paper using prototypes.\n\n2. The issue of discrepancies in class distributions between single-step training sets and the complete dataset (line 42) is in fact already proposed in the paper RBC: Rectifying the Biased Context in Continual Semantic Segmentation (ECCV 2022), the authors can consider citing it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The novelty of prototype usage.**    \n\nAs you mentioned, some prior CSS works have employed prototypes to save past information, with a representative being SDR that you referred to. However, there is a clear distinction in the specific usage of prototypes between SDR and our STAR. In SDR, the primary functions of prototypes are to mitigate catastrophic forgetting and enhance the consistency of features within the same class. Specifically, in each batch, features of each old class are used to generate a \"current prototype\". Then, a specialized loss function is employed to constrain this \"current prototype\" closely match the saved prototype of the corresponding class, aiming to stabilize old-class features and alleviate catastrophic forgetting. Simultaneously, pixel-level features of each foreground class are pulled towards their corresponding prototype, promoting feature cohesion within the same class.  \n\nContrary to SDR, our STAR primarily employs prototypes for replay, aiming to rectify the classifiers' bias towards new classes. Specifically, in each training step, we save the prototypes, pixel-level feature statistics, and pixel occurrence counts for each class. For subsequent steps, we establish a random variable for every prior class using its prototype and feature statistics, which mirrors potential feature representations for that class. We then sample from this variable based on the pixel occurrence count of its class and input these random samples to the classifiers. This procedure aligns the class distribution of the single step training samples with the cumulative training set up to the current point, mitigating classifier bias. In summary, our strategy for prototype utilization diverges from SDR's both in purpose (combatting forgetting vs. bias reduction) and approach (feature constraints vs. replay). Moreover, our modeling of prototypes as statistics-driven random variables, along with our alignment of class distributions in single-step training samples, fundamentally distinguishes our approach from previous studies. From the results of our comparative experiments and ablation study, it can be seen that our prototype replay strategy is effective, contributing to the satisfactory overall performance of our STAR.  \n\n**Q2: Citing the RBC paper.**  \n\nThank you for providing the reference. Indeed, RBC also tackles the issue of discrepancies in class distributions between single-step training sets and the complete dataset. We will cite and briefly discuss this paper in the final version.  \n\nThe proposed solution of RBC is twofold. First, during incremental training, it employs pairs of input images, one normal and the other with new-class pixels erased. Second, it enhances the weight of old-class pixels in the loss function. The first strategy is equivalent to reducing half of the new-class pixels, while the second strategy enhance the weights of old classes in the loss function based on the ratio of new-to-old classes in the current image. These strategies do help mitigate the bias of classifiers towards new classes, producing improved results. However, we believe that there is still room for further refinement. This is because RBC does not take into account the actual class proportions within the complete dataset when rectifying the bias. Thus, while it reduces the bias towards new classes, it does not guarantee that the classifiers' tendency is adjusted to an appropriate level. From this perspective, our method, which replays old-class prototypes based on the recorded class proportion in the training samples accumulated up to the present, holds an advantage, because it can bring the classifiers' tendency closer to the level achieved in joint training on these samples. Meanwhile, the introduction of statistics-driven noise into replayed samples ensures a comprehensive incorporation of past classes.\n"}, {"review_id": "bEnrHzLaKL", "reviewer": "Reviewer_b1on", "summary": "The topic of this paper is about Class-Incremental Semantic Segmentation (CISS). The authors focus on another challenge of CISS: the discrepancies in class distributions between single-step training sets and the complete dataset. They propose a STAR (Storage sAving Replay) framework including a prototype replay strategy, background repetition strategy, old-class feature maintaining loss and similarity-aware discriminative loss.", "questions": "Q1: How to get the distribution of the complete dataset in practical CISS scenario?\n\nQ2: What is the key difference between the proposed prototype replay strategy and other classic memory-efficient CIL methods?\n\nQ3: For the STAR framework, how to alleviate the drift of the stored old-class prototype?\n\nQ4: The key biased distribution problem in CISS and the corresponding works should be sumarized in a related work section.\n", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Exploring the biased distribution problem in CISS is interesting.\n+ The proposed method achieves a large performance gain on several datasets.\n", "weaknesses": "1. The practicability of the proposed method seems to be limited. The authors claim that the distribution of the single-step training set should be similar as that of the complete dataset. However, the distribution of the complete dataset may be not available practical incremental learning scenarios (we can not get the information of newly arrived data).\n\n2. The key novelty of the prototype replay strategy is not clear. The authors claim that the proposed prototype replay strategy can greatly save the memory cost. In the class-incremental learning community, several works focus on the memory efficiency of rehearsal samples (e.g. [1]-[4]).  \n\n[1] Memory-Efficient Incremental Learning Through Feature Adaptation, ECCV2020\n\n[2] Semantic Drift Compensation for Class-Incremental Learning, CVPR2020\n\n[3] Memory efficient class-incremental learning for image classification, TNNLS2021\n\n[4] A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning, ICLR2023\n\n3. Concerns about the effectiveness of the saved prototypes. As described in SDC[2], the information of the stored old-class prototype may be imprecise after the model is updated on the new data.\n\n4. Some related works should be included in the paper. The new-class-biased distribution problem has been discussed in some CISS or image CIL works (e.g. [5]-[6]). These works have the similar motivation as this paper.\n\n[5] Large Scale Incremental Learning, CVPR2019\n\n[6] RBC: Rectifying the Biased Context in Continual Semantic Segmentation, ECCV2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The practicability of our work.**  \n\nAs you pointed out, we cannot know the information of newly arrived data in practical scenarios. Therefore, our experiments simulate this real-world situation, without considering the distribution of future data. Actually, when we mention the \"complete dataset\", it involves all the training samples encountered up to the present step and corresponding test samples, excluding future data. As stated in Section 2.3 of the manuscript, our goal is to align the class distribution of current single-step training samples with that of \"all training sets up to this step\". This distribution should resemble that of the \"complete dataset accumulated up to this step\" since training and test set distributions are typically comparable. With this goal achieved, the classifiers' tendency can closely resemble the outcomes in joint training on all seen training samples, which is the theoretical upper limit of continual learning.  \n\nWe admit that our description of the term \"complete dataset\" may not be clear, possibly leading to misunderstandings. Thus, we will offer a more explicit clarification in the revised version.  \n\n**Q2: The key novelty of our prototype replay strategy.**  \n\nAs you mentioned, some CIL works also focused on enhancing memory efficiency, including [1, 2, 3, 4] that you referred to. First, as for [4], the focus is memory efficiency of model parameters, which is entirely different from our study and [1, 2, 3]. For CISS, model-based methods are rarely used, so most models have the same parameter count, rendering the concern explored in [4] does not really apply. For the remaining works, [1, 2] and ours utilize compact prototypes to retain old class information. Both [1, 2] concentrated on the issue of prototype ineffectiveness caused by feature space changes during incremental training and proposed different methods to adapt previously stored prototypes to the current feature space. In contrast, our approach exploits the specific characteristics of CISS (refer to Q3 response for details) to maintain the old-class feature space, providing a different perspective to address this issue. For [1, 2], insufficient prototype adaptation could still result in prototypes losing part of effectiveness, a problem does not present in our method. On the other hand, [3] used the low-fidelity exemplars rather than prototypes. Their primary concern is narrowing the domain gap between low-fidelity exemplars and raw images, which is quite different from our work.  \n\nAdditionally, each prototype in [1] or each low-fidelity exemplar in [3] corresponds to an individual image, leading to suboptimal memory efficiency since multiple prototypes or exemplars must be stored per class for robust training. Conversely, our approach and [2] implement one prototype per class, resulting in lower memory cost. Nonetheless, our method differs from [2] in that we store both prototypes and the statistics of pixel-level features for each class. This equips us with more comprehensive cues for old classes in subsequent steps, resulting in robust training outcomes. Note that, despite storing the statistics, our total memory cost remains comparable to [2] as our prototypes are more compact. Moreover, our attention to class distribution and the strategy of replaying prototypes based on it distinguish our approach. This strategy enables us to effectively correct the network's bias towards new classes, an issue overlooked in [1]-[3].  \n\nLastly, another fundamental difference lies in the particular tasks we tackle compared to [1, 2, 3, 4]. While all fall under the umbrella of CIL, [1, 2, 3, 4] addressed classification but our focus is segmentation (CISS). Given the paucity of study on memory efficiency within the CISS domain, we believe our contribution holds innovative value.  \n\nIn summary, our method is different from prior memory-efficient CIL methods in four aspects: (1) we save a single compact prototype per class, achieving better memory efficiency. (2) Our strategy incorporates class distribution during replay, effectively rectifying classifier bias. (3) We store feature statistics beyond prototypes, offering more comprehensive cues of past classes. (4) The specific task we address is different. We will clarify the difference from these methods in the final version.  \n\n**Q3: The alleviation of drift.**  \n\nThe concept of \"drift\" presented in SDC [2] refers to the shift of the feature space during incremental learning, which impedes the effectiveness of prototypes calculated in past feature spaces. In our method, solving this problem is also crucial. Thus, we devise a solution tailored to address it in the CISS task, i.e., the Old-Class Features Maintaining (OCFM) loss. Compared to the classification task of CIL, one difference of CISS is the potential presence of old-class pixels in the current training images (although labeled as background).  Our solution takes advantage of this characteristic, maintaining the features in old-class regions unchanged while allowing other features to update. Specifically, we first use the model from the previous step to locate old-class pixels. Next, the features extracted by the current model at these old-class pixels are constrained to match those extracted by the previous model, thus maintaining the feature space of old classes and avoiding drift. Our ablation study results in Section 3.3 show that the OCFM loss can effectively alleviate drift, ensuring the efficacy of prototypes.  \n\n**Q4: Missing some related works.**  \n\nWe appreciate your suggestion. Due to the page limit of the initial submission, we briefly addressed related works in the first and third paragraphs of the \"Introduction\" section. In the final version, with one more page permitted, we will add a separate \"Related Works\" section to offer a more comprehensive summary of related works, including those also focus on the biased distribution problem you mentioned. \n"}, {"review_id": "H3VJ3JxWIi", "reviewer": "Reviewer_gzuL", "summary": "This paper proposes a class-incremental semantic segmentation (CISS) method, STAR, which incorporates four contributions.\n\n[1] Prototype preservation and replay: Aligning the proportion of old-class pixels in the single-step training set with the proportion in the complete dataset by replaying old-class prototypes.\n\n[2] Background repetition: Aligning the proportion of background in the single-step training set with the proportion in the complete dataset by repeat background pixels.\n\n[3] Old-class features maintaining loss: Maintaining old-class features to avoid the saved prototypes from ineffective.\n\n[4] Similarity-aware discriminative loss: Enhancing the feature distinction between similar old-new class pairs.\nBy combining these four components, STAR achieves good performance.\n", "questions": "See the weakness", "limitations": "This paper addresses the issue of biased classifiers observed in previous methods, and it discusses the limitations of the proposed method as well as areas for improvement properly.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The primary motivation of eliminating the distribution difference between single-step training set and the complete dataset is reasonable. The strategies of prototype replay and background repetition are proper to solve this problem.\n+ The proposed OCFM loss can mitigate catastrophic forgetting as well as keep the saved old-class prototypes effective. The proposed SAD loss can help the classifiers to discriminate similar old-new class pairs.\n+ Compared with previous CISS methods, the performance is top-tier, and the storage requirement is negligible. The experiments are complete and capable of proving the effectiveness of the proposed method.\n", "weaknesses": "[1] In some equations, the elementwise multiplication is strange. For example, in equation (3), it seems that $\\mathbb{1}\\{\\tilde{y}_{i,j}^{t-\\tau}=l\\}$ is a single element, not a vector, so how can it be element-wise multiplied with other terms? Similar situations also appear in other equations.\n\n[2] In line 120, you mention \"the complete dataset accumulated up to this step\". What does it mean? Is it the same as \"the union of all training sets up to this step\" in line 118? If they are the same, it is strange to say the distributions on them are “close”.\n\n[3] The pixel count fine-tuning part and limitation are introduced in the supplementary due to the page limitation, but authors should at least give a brief introduction in it to outline your basic principle of it.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Unclear equations.**  \n\nThank you for comments. Indeed, $\\mathbb{1}\\\\{\\tilde{y}_{i,j}^{t-\\tau}=l\\\\}$ is a single element, not a vector, so it can only multiply, other than element-wise multiply other terms. We will amend this equation to  $\\mu_l = \\frac{\\sum _{i=1}^{N^{t-\\tau}} \\sum _{j=1}^{h\\times w} (\\hat{f} _{i,j}^{t-\\tau} \\mathbb{1} \\\\{\\tilde{y} _{i,j}^{t-\\tau}=l\\\\} )}\n{\\left\\lVert \n\\sum _{i=1}^{N^{t-\\tau}} \\sum _{j=1}^{h\\times w} ( \\hat{f} _{i,j}^{t-\\tau} \\mathbb{1} \\\\{\\tilde{y} _{i,j}^{t-\\tau}=l\\\\} )\n\\right\\rVert_2}\n\\in \\mathbb{R}^c$. Since the similar situation also appears in other equations, we will also modify them accordingly.\n\n\n**Q2: Unclear descriptions.**  \n\nThe former encompasses all training samples encountered up to the current step, along with corresponding test samples. The latter, however, exclusively represents the collection of training samples encountered thus far. Essentially, the latter is a subset of the former. Since the training set and test set are usually randomly partitioned, their distributions should be close.\n\n**Q3: Briefly describe the pixel count fine-tuning and limitations in the main body.**  \n\nThank you for your suggestion. We will incorporate brief introductions about our pixel count fine-tuning strategy in Sections 2.3 and 2.4 to enhance the readability.\n"}, {"review_id": "7U5x2VAWbT", "reviewer": "Reviewer_2RZD", "summary": "A new method called STAR is introduced in this paper to address the CISS task. The fundamental idea is to remove the distribution disparity between single-step training samples and the entire dataset, which consequently eliminate the network's bias. Based on this idea, the authors designed two strategies, namely the prototype replay strategy and background repetition strategy. Both strategies help to rectify the distribution of single-step training samples while keeping the storage cost minimal. Additionally, the OCFM loss is proposed to maintain features of previous foreground classes, and the SAD loss is devised to help distinguish similar categories. Experimental results demonstrate that the network achieves state-of-the-art performance, and each module contributes to its effectiveness.", "questions": "Please address the issues mentioned in the weaknesses section and make modifications accordingly. ", "limitations": "The author appropriately discusses the limitations of the proposed method in the supplementary material. ", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "From my perspective within this field, the idea presented in this paper is innovative.\nThe proposed approach is commendable as it effectively addresses the problem of distribution differences while keeping the storage cost low, as demonstrated by the experimental results.\nMoreover, apart from CISS, the motivation and core idea have the potential to be extended to other continual learning tasks.\nThe overall writing of this paper is also clear, making it relatively easy to comprehend.", "weaknesses": "There are some grammar errors and unclear explanations in the article that may hinder readers' comprehension.\n1. Firstly, in Section 1, the sentence \"under the overlapped 19-1 protocol, ...\" is mentioned, but it does not provide a clear explanation of which dataset this protocol refers to.\n2. Secondly, in Section 2.3, the word “contain” in the sentence \"Assuming each epoch contain e iterations\" should be corrected to \"contains\" since it is in the third person singular form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Unclear protocol description.**  \n\nThe \"overlapped 19-1\" protocol is a CISS protocol on the Pascal VOC 2012 dataset, where the first training step includes 19 foreground classes, while the second training step covers only one foreground class (tv/monitor). We will provide a clearer explanation of the \"overlapped 19-1\" protocol in the revised manuscript.\n\n**Q2: Grammatical errors.**  \n\nThank you for careful reading. We will correct this sentence to “Assuming each epoch contains $e$ iterations, …”. Moreover, we will proofread our manuscript again to avoid similar grammar errors.\n"}, {"review_id": "0lZkQ3SCXJ", "reviewer": "Reviewer_DAss", "summary": "This paper proposes a memory-efficient replay-based class-incremental semantic segmentation method. Compared to the existing replay-based method, it achieves better performance while saving over 100 times the storage and can mitigate the private issue of the stored data. Furthermore, it proposes OCFM and SAD loss functions to improve plasticity and prevent catastrophic forgetting of the model. Experiments on two benchmarks (VOC 2012 and ADE 20K) verify the effectiveness and efficiency of the proposed method.", "questions": "N/A", "limitations": "The authors have discussed limitations in the supplementary material.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "### 1. The motivation of this work is well-described in the introduction section\n\n- I can easily agree with the problems: the discrepancies in class distributions, the biased classifier towards new classes, the memory storage efficiency issue of replay-based methods, and the private issue of the stored data.\n\n### 2. The proposed prototype is well-designed and efficient.\n\n- Even though the prototypes are 1-d vectors that condense the information of old classes, the Gaussian-distributed random variables can reasonably represent potential characteristics of old classes.\n\n### 3. Thorough experiments and ablation study.\n\n- The effectiveness of the proposed method is well-verified on the popular two benchmarks (VOC 2012 and ADE 20K).\n- The ablation study shows the role of each proposed component.", "weaknesses": "### 1. Recommend using an Algorithm.\n\n- Since the detailed procedure of the proposed method (Sections 2.3 and 2.4) is mostly described using manuscripts, it makes the readers to hard to understand it.\n- I think it would be better to describe the proposed method using an algorithm.\n\n\n### 2. Describing the multiple binary cross-entropy (mBCE) loss\n\n- When leveraging the prototype and background repetition, what is the ground-truth? where the predictions are come from? what is the equation of the objective function?\n- (This may overlap with question 1.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Algorithm.**  \nThank you for your valuable suggestion. Below is an algorithm describing our proposed STAR. Please note that due to the constraints of this platform, we're presenting it in plain text. In the final version, we will format it using LaTeX.  \n\n**Algorithm 1:**  Pseudo code of STAR  \n\n**Require:** feature extractor $\\Psi_\\theta$, classifiers $\\Phi_\\omega$, number of steps $T$, number of epochs $N_{epoch}$, training sets for all steps $\\\\{{D^t}\\\\}_{t=1}^T$, foreground classes for all steps $\\\\{{L^t}\\\\} _{t=1}^T$, balance weights in the total loss $\\alpha$ and $\\beta$  \n\n**for** $t$ in $1$ to $T$ **do**  \n\n&ensp;**if** $t=1$ **then**  \n\n&emsp;Random initialize $\\theta$, $\\omega$  \n\n&ensp;**else**  \n\n&emsp;Initialize $\\theta$, $\\\\{{\\omega_l}\\\\}_{l\\in L^{1:t-1}}$ with $\\theta^{t-1}$, $\\\\{{\\omega_l^{t-1}}\\\\} _{l\\in L^{1:t-1}}$  \n\n&emsp;Random initialize $\\\\{{\\omega_l}\\\\}_{l\\in L^t}$  \n\n&ensp;/* After initialization, $\\theta$ and $\\omega$ are re-denoted as $\\theta^t$ and $\\omega^t$ */  \n\n&ensp;**end if**  \n\n&ensp;**for** $n_{epoch}$ in $1$ to $N_{epoch}$ **do**  \n\n&emsp;**for** batch $U$ in $D^t$ **do**  \n\n&ensp;&emsp;$\\Psi_{\\theta^t}$ extract features, achieving $f^t$  \n\n&ensp;&emsp;**if** $t=1$ **then**  \n\n&emsp;&emsp;Feed $f^t$ to $\\Phi_{\\omega^t}$, achieving the output $p^t$  \n\n&emsp;&emsp;/* mBCE loss */  \n\n&emsp;&emsp;Compute $\\mathcal{L}_{mbce}$ as the multiple binary cross-entropy between $p^t$ and the ground-truth  \n\n&emsp;&emsp;/* parameter update */  \n\n&emsp;&emsp;Compute the gradient for $\\mathcal{L}_{mbce}$, then update $\\theta^t$ and $\\omega^t$  \n\n&emsp;&ensp;**else**  \n\n&emsp;&emsp;$\\Psi_{\\theta^{t-1}}$ extract features, producing $f^{t,t-1}$  \n\n&emsp;&emsp;Feed $f^{t,t-1}$ to $\\Phi_{\\omega^{t-1}}$, achieving the prediction of old classes, $p^{t,t-1}$  \n\n&emsp;&emsp;/* prototype replay */  \n\n&emsp;&emsp;Construct Gaussian-distributed random variables $\\\\{{r_l}\\\\}_{l\\in L^{1:t-1}}$ based on saved prototypes and statistics  \n\n&emsp;&emsp;Sample each $r_l$ a certain number of times based on the saved occurrence count of $l$ pixels and concatenate all random samples together  \n\n&emsp;&emsp;/* background repetition */  \n\n&emsp;&emsp;Identify background regions by excluding pixels currently labeled as foreground class and old-class pixels estimated in $p^{t,t-1}$  \n\n&emsp;&emsp;Duplicate background pixels a certain number of times based on the saved occurrence count of background pixels and concatenate all duplications together  \n\n&emsp;&emsp;/* mBCE loss */  \n\n&emsp;&emsp;Concatenate aforementioned random samples and background duplications with $f^t$, feed them to $\\Phi_{\\omega^t}$, achieving the output $p^t$  \n\n&emsp;&emsp;Compute $\\mathcal{L}_{mbce}$ as the multiple binary cross-entropy between $p^t$ and corresponding ground-truths // refer to Q2 response for details  \n\n&emsp;&emsp;/* old-class features maintaining loss */  \n\n&emsp;&emsp;Identify old-class regions with the help of $p^{t,t-1}$  \n\n&emsp;&emsp;Compute $\\mathcal{L}_{ocfm}$ as the mean square error between $f^t$ and $f^{t,t-1}$ in old-class regions  \n\n&emsp;&emsp;/* similarity-aware discriminative loss */  \n\n&emsp;&emsp;Compute $\\mathcal{L}_{sad}$ as the mean of the minimum distance from the feature center of each current foreground class (computed within the current batch) to all saved old-class prototypes  \n\n&emsp;&emsp;/* parameter update */  \n\n&emsp;&emsp;Compute the gradient for $\\mathcal{L}_{mbce}+\\alpha\\mathcal{L} _{ocfm}+\\beta\\mathcal{L} _{sad}$, then update $\\theta^t$ and $\\omega^t$  \n\n&emsp;&ensp;**end if**  \n\n&emsp;**end for**  \n\n&ensp;**end for**  \n\n&ensp;Compute and save the occurrence counts for all current foreground class pixels and background pixels  \n\n&ensp;Compute and save the prototypes and statistics of pixel-level features for all current foreground classes  \n\n**end for**  \n\nreturn $\\Psi_{\\theta^T}$ and $\\Phi_{\\omega^T}$\n\n**Q2: mBCE loss.**  \n\nWe use a standard mBCE loss computed as: $\\mathcal{L} _{mbce}=-\\frac{1}{H\\cdot W+N _{rp}+N _{rb}}\\sum _{j=1}^{H\\cdot W+N _{rp}+N _{rb}}\\sum _{l\\in L^t}{\\gamma\\cdot y _j^{t,l}\\cdot\\log{p _j^{t,l}}+(1-y _j^{t,l})\\cdot\\log{\\left(1-p _j^{t,l}\\right)}}$. Here, $H\\cdot W$ indicates the spatial size of the input image, $N _{rp}$ is the number of replayed prototype samples, $N _{rb}$ is the number of repeated background pixels, $j$ is the index, $L^t$ is the set of current foreground classes, $t$ is the step index, $\\gamma$ is the positive weight, $y _j^{t,l}$ is the ground-truth of the $j$-th pixel or replayed sample ($0$ means it does not belong to class $l$, $1$ means it does), and $p_j^{t,l}$ is the network's output. In the first step, $N _{rp}=N _{rb}=0$, and $p$ and $y$ only corresponds to the pixels in the input image. In later steps, $p$ and $y$ are the concatenations of the prediction/ground-truth for the input image, replayed prototypes, and repeated background pixels.\n\nQ2-1: When leveraging prototype replay and background repetition, the ground-truth for a replayed prototype sample is the one-hot encoding of the prototype's class label. Meanwhile, the ground-truth of each repeated background pixel is a all-zero vector, as the background does not belong to any class in $L_t$.\n\nQ2-2: During inference, the input image first goes through the feature extractor to produce features. Then, the classifiers output logits for each pixel, which are then passed through a sigmoid function to determine the probability of belonging to every foreground class. If none of the foreground class probabilities exceed $0.5$ for a pixel, the prediction of this pixel is \"background\". Otherwise, the prediction is the class with the highest probability.\n\nQ2-3: The overall objective function of the network is \n$\\mathcal{L}=\\mathcal{L} _{mbce}+\\alpha\\mathcal{L} _{ocfm}+\\beta\\mathcal{L} _{sad}$, where $\\mathcal{L} _{mbce}$ is the mBCE loss as mentioned before. $\\mathcal{L} _{ofcm}$ and $\\mathcal{L} _{sad}$ are described in formulas (10) and (12) in the manuscript.\n"}]

Author rebuttal:

Meta-review: This paper received all five positive scores. In the initial review procedure, some common concerns are about the motivation novelty and comparison with existing works. After rebuttal, reviewers agree that these concerns are well addressed. After checking the submission and all reviews, the AC agrees that this paper meets the requirement of the NeurIPS, and recommends acceptance.
Decision: Accept (poster)