SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\eta_ {b g} ^ {t - \tau} = \left\{ \begin{array}{l l} \sum_ {i = 1} ^ {N ^ {t - \tau}} \sum_ {j = 1} ^ {h \times w} \mathbb {1} \{\tilde {y} _ {i, j} ^ {t - \tau} = 0 \} & \text {i f} t - \tau = 1 \\ \eta_ {b g} ^ {t - \tau - 1} + \sum_ {i = 1} ^ {N ^ {t - \tau}} \sum_ {j = 1} ^ {h \times w} \mathbb {1} \{\tilde {y} _ {i, j} ^ {t - \tau} = 0 \} & \text {i f} t - \tau > 1, \end{array} \right. \tag {8}
$$
where  $0$  in  $\tilde{y}_{i,j}^{t - \tau}$  indicates the label background. In fact, since there may be overlapping background pixels between the training sets in different steps, we scale down  $\eta_{bg}^{t - \tau}$  based on some prior assumptions. However, similar to the case of foreground class counts, the impact is minimal, so it is optional and described in the supplementary material.
At each incremental step  $t$ , we restore the proportion of the background by repeating background pixels in the current training set  $D^t$  by a certain time. Specifically, we consider a pixel in  $f_i^t$  as a background pixel if it meets two conditions: 1) its label is background, and 2) the model at the last step  $M^{t-1}$  also predicts it as background. Ideally, these requirements ensure the pixel is neither a current nor past foreground pixel and can be currently categorized as background. Using these criteria, we obtain the set of background pixels  $B^t$ :
$$
B ^ {t} = \left\{f _ {i, j} ^ {t} \mid \tilde {y} _ {i, j} ^ {t} = 0 \text {a n d} \tilde {p} _ {i, j} ^ {t, t - 1} = 0 \right\}, \tag {9}
$$
where  $\tilde{p}_{i,j}^{t,t-1} = DS(p_{i,j}^{t,t-1})$  is the downsampled prediction of  $M^{t-1}$ . During training, we repeat pixels in  $B^t$  by  $\frac{\eta_{bg}^t}{|B^t|}$  times per epoch, where  $|\cdot|$  denotes set cardinality, to integrate  $\eta_{bg}^t$  additional background pixels, with  $\eta_{bg}^t$  representing the count of background pixels outside  $D^t$ . This aligns the background pixel proportion with that in  $D^{1:t}$ . By merging prototype replay and background repetition strategies, we rectify the distribution of both foreground classes and background in single-step training samples for the classifier.
3.5 Old-class features maintaining loss
The prototype replay strategy introduced in Sec. 3.3 can correct the proportion of old-class samples while conserving storage space. However, its feasibility relies on the premise that feature extraction for old-class regions does not change significantly between steps. If this condition is not met, there will be a discrepancy between the old-class features extracted by the current feature extractor and the replayed old-class prototypes, as the prototypes are created from features generated by previous
feature extractors. In this situation, the replay becomes ineffective because the features that the prototypes represent have become outdated. Furthermore, since there are no constraints on parameter updates, the feature extractor may suffer from uncontrolled catastrophic forgetting of old knowledge. Consequently, our goal is to develop a strategy that maintains old-class features, cooperating with our replay strategy and resisting catastrophic forgetting. At the same time, we do not want the network to lose plasticity for learning new classes. By balancing these objectives, we design the OCFM loss, which selectively maintains features of old-class regions in a spatially targeted manner while simultaneously allowing flexibility for learning new classes.
OCFM maintains old-class features by penalizing the changes from the features extracted by the previous feature extractor,  $f_{i}^{t,t - 1}$ , to the features extracted by the current feature extractor,  $f_{i}^{t}$ , in old-class regions. For a pixel, we consider it belongs to old-class regions by two requirements: 1) its current label is background, and 2) the previous model  $M^{t - 1}$  predicts it as a foreground pixel. For these old-class pixels, the OCFM loss punishes the mean squared error between  $f_{i}^{t}$  and  $f_{i}^{t,t - 1}$  by:
$$
\mathcal {L} _ {o c f m} = M S E \left(f _ {i} ^ {t} \times \mathbb {1} \left\{\tilde {y} _ {i, j} ^ {t} = 0 \right\} \times \mathbb {1} \left\{\tilde {p} _ {i, j} ^ {t, t - 1} = 0 \right\}, \right. \tag {10}
$$
$$
f _ {i} ^ {t, t - 1} \times \mathbb {1} \{\tilde {y} _ {i, j} ^ {t} = 0 \} \times \mathbb {1} \{\tilde {p} _ {i, j} ^ {t, t - 1} = 0 \}),
$$
in which  $MSE$  signifies the mean squared error.
Compared to distillations that operate on the entire spatial region [1, 9, 31], our OCFM loss specifically targets old-class regions, preserving these features to collaborate with our replay strategy and combat catastrophic forgetting. Simultaneously, the features in other regions can be updated without constraints, ensuring that the flexibility to learn new classes remains uncompromised.
3.6 Similarity-aware discriminative loss
The previous sections address the bias resulting from distribution differences between single-step training sets and the complete dataset. Despite the bias being eliminated, some similar class pairs may still be prone to confusion, as the feature extractor might not generate sufficiently discriminative features for them. When these similar classes are trained in the same step, the situation may be less severe since the mBCE loss drives their features to exhibit adequate differences. However, when they are trained in different steps, the mBCE loss alone is not enough to encourage the feature extractor to generate discriminative features. This is because the old-class features are constrained by the OCFM loss, and most old-class representations that eventually flow to the mBCE loss are derived from prototypes, which, although close to the authentic features, are slightly inferior in richness.
To address this problem, a straightforward solution is to introduce a differentiation loss to all old-new class feature center pairs, seeking to maximize their divergence. However, this lacks targeting, as some feature pairs are already widely separated, and we should focus on those similar pairs. To this end, we refine this strategy to form our SAD loss: for each current class, we only penalize the distance between its feature center and the nearest old-class feature center, thereby directing the differentiation effort to where it is most needed. At step  $t$ , the feature center of an old class  $l_{o} \in L^{1:t-1}$  is the previously saved prototype,  $\mu_{l_{o}}$ , and the feature center of a current class  $l_{c} \in L^{t}$  is computed within the batch being processed, given by:
$$
\dot {\mu} _ {l _ {c}} = \frac {\sum_ {i \in U} \sum_ {j = 1} ^ {h \times w} \left(\hat {f} _ {i , j} ^ {t} \times \mathbb {1} \left\{\tilde {y} _ {i , j} ^ {t} = l \right\}\right)}{\left\| \sum_ {i \in U} \sum_ {j = 1} ^ {h \times w} \left(\hat {f} _ {i , j} ^ {t} \times \mathbb {1} \left\{\tilde {y} _ {i , j} ^ {t} = l \right\}\right) \right\| _ {2}}, \tag {11}
$$
where  $U$  represents the set of indices of images belonging to the current batch. After computing the feature centers for all current foreground classes, we can achieve the SAD loss as:
$$
\mathcal {L} _ {s a d} = \frac {1}{| L ^ {t} |} \sum_ {l _ {c} \in L ^ {t}} \min  _ {l _ {o} \in L ^ {1: t - 1}} \| \mu_ {l _ {o}} - \dot {\mu} _ {l _ {c}} \| _ {2}. \tag {12}
$$
With  $\mathcal{L}_{\text{sad}}$ , the features of each new class will be distanced from the most similar old-class prototype. When the distance between a new-class feature center and its closest old-class prototype is far enough, surpassing the distance to another old-class prototype, that old-class becomes the new closest one, and  $\mathcal{L}_{\text{sad}}$  turns to penalize the distance between this new pair. Ultimately, the features of the new class will be far enough away from all old class prototypes.
Table 1: Quantitative comparison on Pascal VOC 2012 between our STAR and previous non-replay methods (top half) and replay-based methods (bottom half) under the overlapped setting.
Model	19-1 (2 steps)	15-5 (2 steps)	15-1 (6 steps)	10-1 (11 steps)	5-3 (6 steps)
old	new	all	old	new	all	old	new	all	old	new	all	old	new	all
MiB [2]	70.2	22.1	67.8	75.5	49.4	69.0	35.1	13.5	29.7	12.3	13.1	12.7	57.1	42.6	46.7
SDR [23]	69.1	32.6	67.4	75.4	52.6	69.9	44.7	21.8	39.2	-	-	-	-	-	-
PLOP [9]	75.4	37.4	73.5	75.7	51.7	70.1	65.1	21.1	54.6	44.0	15.5	30.5	17.5	19.2	18.7
SSUL [3]	77.7	29.7	75.4	77.8	50.1	71.2	77.3	36.6	67.6	71.3	46.0	59.3	72.4	50.7	56.9
RCIL [31]	-	-	-	78.8	52.0	72.4	70.6	23.7	59.4	55.4	15.1	34.3	-	-	-
RBC [33]	77.3	55.6	76.2	76.6	52.8	70.9	69.5	38.4	62.1	-	-	-	-	-	-
DKD [1]	77.8	41.5	76.0	78.8	58.2	73.9	78.1	42.7	69.7	73.1	46.5	60.4	69.6	53.5	58.1
UCD [29]	75.9	39.5	74.0	75.0	51.8	69.2	66.3	21.6	55.1	42.3	28.3	35.3	-	-	-
AWT [12]	-	-	-	78.0	50.2	71.4	77.0	37.6	67.6	73.1	47.0	60.7	71.6	51.4	57.1
Ours	78.0	47.1	76.5	79.5	58.9	74.6	79.5	50.6	72.6	73.1	55.4	64.7	71.9	61.5	64.4
RECALL [20]	68.1	55.3	68.6	67.7	54.3	65.6	67.8	50.9	64.8	65.0	53.7	60.7	-	-	-
SSUL-M(100) [3]	77.8	49.8	76.5	78.4	55.8	73.0	78.4	49.0	71.4	74.0	53.2	64.1	71.3	53.2	58.4
DKD-M(100) [1]	78.0	57.7	77.0	79.1	60.6	74.7	78.8	52.4	72.5	74.0	56.7	65.8	69.8	60.2	62.9
Ours-M(50)	77.8	56.4	76.8	79.7	61.8	75.4	79.5	55.6	73.8	74.3	57.9	66.5	71.9	62.9	65.5
Table 2: Quantitative results on ADE20K under the overlapped setting.
Model	100-50 (2 steps)	100-10 (6 steps)	50-50 (3 steps)
old	new	all	old	new	all	old	new	all
MiB [2]	37.9	27.9	34.6	31.8	14.1	25.9	35.5	22.9	27.0
PLOP [9]	41.9	14.9	32.9	40.5	13.6	31.6	48.8	21.0	30.4
RCIL [31]	42.3	18.8	34.5	39.3	17.6	32.1	48.3	25.0	32.5
RBC [33]	42.9	21.5	35.8	39.0	21.7	33.3	49.6	26.3	34.2
UCD [29]	42.1	15.8	33.3	40.8	15.2	32.3	47.1	24.1	31.8
AWT [12]	40.9	24.7	35.6	39.1	21.3	33.2	46.6	26.9	33.5
SSUL-M(300) [3]	42.8	17.5	34.4	42.9	17.7	34.5	49.1	20.1	29.8
DKD-M(300) [1]	42.4	23.0	36.0	41.7	20.1	34.6	48.8	26.3	33.9
Ours	42.4	24.2	36.4	42.0	20.6	34.9	48.7	27.2	34.4
Datasets and evaluation metric. We evaluate our method on two common public datasets: Pascal VOC 2012 [11] and ADE20K [34]. Pascal VOC 2012 comprises 10,582 training images and 1,449 validation images, covering 20 classes. ADE20K includes 20,210 training images and 2,000 validation images, spanning 150 classes. Following previous works [1, 2, 3], we use mean Intersection-over-Union (mIoU) as our evaluation metric, representing the average IoU across all classes.
Experimental protocols. Following previous works, we evaluate our model with various class splits among multiple steps. Each split is denoted as  $N_{1} - N_{2}$ , where  $N_{1}$  and  $N_{2}$  refer to the number of classes in the first step and in each incremental step, respectively. We have 19-1, 15-5, 15-1, 10-1, and 5-3 splits for Pascal VOC 2012, and 100-50, 100-10, and 50-50 splits for ADE20K. Furthermore, we consider two CISS settings: disjoint and overlapped. The difference between them is that the disjoint excludes images with future class pixels in  $D^{t}$ , while the overlapped includes them (future class pixels are labeled as background). Since the overlapped is closer to real-world scenarios, we report the results of it in this paper and put the results of the disjoint in the supplementary material.
Implementation details. Following [1, 2, 9], we use DeepLabv3 [5] with a ResNet-101 [13] backbone pre-trained on ImageNet [7] as our segmentation network. In line with [1, 2, 3, 9], training strategies differ for the two datasets. For Pascal VOC 2012, we train for 60 epochs with an initial learning rate of 0.001 for the first step and 0.0001 for incremental steps, and empirically set  $\gamma$  to 4. For ADE20K, we train for 100 epochs with an initial learning rate of 0.00025 for the first step and 0.000025 for incremental steps, and set  $\gamma$  to 30. For both datasets, we use an SGD optimizer with a momentum of 0.9. The batch size is set to 24, and  $\alpha$  and  $\beta$  are set to 5 and 0.05, respectively. We implement our model using PyTorch [25] and employ two NVIDIA RTX 3090 GPUs for acceleration. We have also implemented our network using the MindSpore Lite tool<sup>1</sup>.
<https://www.mindspore.cn/>
PLOP
Figure 2: Qualitative comparison on Pascal VOC 2012 between STAR and previous methods.
SSUL
DKD
STAR (Ours)
We compare STAR with previous CISS methods both quantitatively and qualitatively. Among them, MiB [2], SDR [23], PLOP [9], SSUL [3], RCIL [31], RBC [33], DKD [1], UCD [29], and AWT [12] are non-replay-based, while RECALL [20], SSUL-M [3], and DKD-M [1] are replay-based methods.
Quantitative comparison. The quantitative comparison results for Pascal VOC 2012 and ADE20K datasets are presented in Table 1 and Table 2, respectively, where "old" indicates the mIoU of classes in the first step while "new" indicates the mIoU of incremental classes. Compared to previous non-replay-based models, our STAR consistently achieves leading performance across various class splits. Particularly in more challenging 15-1, 10-1, and 5-3 splits that contain more steps, our method outperforms others by at least 2.9, 4.0, and  $6.3\mathrm{mIoU}$ , respectively. Notably, although STAR requires additional storage space for prototypes and statistics, the space needed is minimal ( $\sim41\mathrm{KB}$ ), causing no significant unfairness compared to non-replay-based models. Meanwhile, even compared with other replay-based models, STAR remains highly competitive, achieving performance on par with the best-performing DKD-M while requiring less than  $0.5\%$  of its storage space (DKD-M cost  $\sim10\mathrm{MB}$  to save 100 raw images from past steps). If we add 50 extra raw images to provide more realistic features (Ours-M), the performance of our model can be further improved, surpassing DKD-M in four out of five class splits. Remarkably, even in this scenario, the required storage is just about half of that needed by DKD-M.
On the ADE20K dataset, our model performs even better, outperforming all previous non-replay-based and replay-based methods without storing any extra raw images. Despite the larger class count on ADE20K necessitating the storage of more prototypes, the required storage remains a mere  $\sim 2\%$  of that demanded by DKD-M and SSUL-M, both of which store 300 raw images.
In summary, our model achieves outstanding performance while requiring, on average, only  $\sim 1\%$  of the storage needed by other replay-based models, and it requires no extra image beyond the datasets. Moreover, privacy concerns tied to storing raw images are avoided. These demonstrate the significance of our approach to reconstruct class distribution through prototype replay.
Qualitative comparison. Fig. 2 provides a qualitative comparison using the challenging overlapped 15-1 protocol on Pascal VOC 2012, highlighting our method's superiority over previous state-of-the-art CISS methods [1, 3, 9]. The prevalent errors in other methods are false positives for new classes. For instance, in the first two examples, some pixels of old classes bus and bird are identified as new classes train, sheep, or tv/monitor. In the third example, some background pixels are misclassified as new classes sofa or tv/monitor. Such errors stem from the underrepresentation and limited diversity of old-class or background samples in the single-step training set when training new classes, resulting in a bias towards them. In contrast, our method effectively addresses these problems by rectifying the distribution of single-step training samples and providing new-class classifiers with access to old-class prototypes as negative samples.
Table 3: Ablation study results of our method in the Pascal VOC 2012 dataset.
Replay	BR	OCFM	SAD	overlapped 15-1 (6 steps)
old	new	all
✓	✓	✓	79.3	41.8	70.3
✓	✓	✓	78.3	35.5	68.1
✓	✓	✓	37.3	12.8	31.4
✓	✓	✓	79.3	48.2	71.6
✓	✓	✓	✓	79.5	50.6	72.6
Table 3 shows our ablation study results for STAR under the challenging overlapped 15-1 protocol, assessing its four core components: prototype replay (Replay), background repetition (BR), old-class features maintaining loss (OCFM), and similarity-aware discriminative loss (SAD). Each component is individually removed from the full model to evaluate its contribution to the overall performance.
Effectiveness of the prototype replay strategy (Replay). The first row of Table 3 presents the results without Replay. Compared to the full model's results shown in the last row, it is clear that Replay boosts the performance by  $2.3\mathrm{mIoU}$ . Although Replay involves replaying old-class prototypes, it primarily enhances the accuracy of new classes, demonstrating an  $8.8\mathrm{mIoU}$  improvement. This effect arises from Replay reducing the proportion of new-class samples flowing into the classifiers and providing new-class classifiers with sufficient access to old-class features. This dual effect mitigates classifier bias and lessens the chance of misclassifying non-new-class pixels, especially those old-class pixels, as new-class pixels. Consequently, we observe a significant increase in the mIoU of new classes and a slight improvement in the mIoU of old classes.
Effectiveness of the background repetition strategy  $(BR)$ . As displayed in Table 3, excluding  $BR$  (second row) leads to a performance drop compared to the full model (last row). Specifically,  $BR$  brings a  $4.5\mathrm{mIoU}$  improvement overall and notably boosts the mIoU of new classes by 15.1. Despite simply repeating background pixels from the current training set,  $BR$  effectively corrects the classifier bias due to the high proportion and diversity of background pixels, even within a single-step training set. Likewise, because of the high proportion of background pixels,  $BR$  introduces more additional samples than Replay, making its removal lead to a more noticeable performance degradation.
Effectiveness of the old-class features maintaining loss (OCFM). The significance of OCFM can be recognized when comparing the third and final rows of Table 3. Upon removal of OCFM, the results experience a complete breakdown, with a dramatic  $41.2\mathrm{mIoU}$  drop compared to the full model. OCFM is a crucial mechanism in our model for maintaining prior knowledge. Its absence not only makes replay ineffective but also triggers uncontrolled network parameter updates, leading to intense catastrophic forgetting. Thus, the performance of both new and old classes suffers greatly.
Effectiveness of the similarity-aware discriminative loss (SAD). The final component under validation is  $SAD$ , designed to differentiate the features of similar old-new class pairs. The penultimate row in Table 3 illustrates the performance without  $SAD$ , registering a  $1.0\mathrm{mIoU}$  reduction. This attests to the efficacy of  $SAD$  in augmenting feature differences between similar old-new class pairs, thereby making them more distinguishable to the classifiers, leading to improved prediction accuracy.
In this paper, we present a novel class-incremental semantic segmentation (CISS) method named STAR that addresses the bias towards part of classes due to distribution discrepancies between single-step training sets and the complete dataset. STAR incorporates two key strategies: prototype replay and background repetition, aiming to correct the proportions of foreground classes and background, respectively. By saving and replaying prototypes instead of raw images, STAR reduces storage costs to  $\sim 1\%$  of other replay-based methods, without requiring extra data. Furthermore, we propose an old-class features maintaining loss to keep old-class features unchanged while preserving the flexibility to learn new classes and a similarity-aware discriminative loss to selectively differentiate features of most similar old-new class pairs, making them distinguishable for the classifiers. Comparisons with previous state-of-the-art CISS methods and our ablation study demonstrate the superiority of our overall design and the importance of each individual component in our model.
This work was supported in part by the National Key R&D Program of China under Grant 2021ZD0112100, in part by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA), in part by the Hong Kong GRF-RGC General Research Fund under Grant 11209819 and Grant 11203820, in part by the National Natural Science Foundation of China under Grant 62002014, in part by the Taishan Scholar Project of Shandong Province under Grant tsqn202306079, in part by Young Elite Scientist Sponsorship Program by the China Association for Science and Technology under Grant 2020QNRC001, in part by CAAI-Huawei MindSpore Open Fund.
[1] Donghyeon Baek, Youngmin Oh, Sanghoon Lee, Junghyup Lee, and Bumsub Ham. Decomposed knowledge distillation for class-incremental semantic segmentation. In NeurIPS, 2022.
[2] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In CVPR, 2020.
[3] Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al. SSUL: Semantic segmentation with unknown label for exemplar-based class-incremental learning. In NeurIPS, 2021.
[4] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
[5] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
[6] Runmin Cong, Yumo Zhang, Leyuan Fang, Jun Li, Yao Zhao, and Sam Kwong. RRNet: Relational reasoning network with parallel multiscale attention for salient object detection in optical remote sensing images. IEEE Trans. Geosci. Remote Sens., 60:1-11, 2021.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.
[8] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, 2019.
[9] Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord. PLOP: Learning without forgetting for continual semantic segmentation. In CVPR, 2021.
[10] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020.
[11] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis., 88:303-338, 2010.
[12] Dipam Goswami, René Schuster, Joost van de Weijer, and Didier Stricker. Attribution-aware weight transfer: A warm-start initialization for class-incremental semantic segmentation. In WACV, 2023.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[14] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In ECCV, 2020.
[15] Chongyi Li, Runmin Cong, Junhui Hou, Sanyi Zhang, Yue Qian, and Sam Kwong. Nested network with two-stream pyramid for salient object detection in optical remote sensing images. IEEE Trans. Geosci. Remote Sens., 57(11):9156-9166, 2019.
[16] Chongyi Li, Runmin Cong, Sam Kwong, Junhui Hou, Huazhu Fu, Guopu Zhu, Dingwen Zhang, and Qingming Huang. ASIF-Net: Attention steered interweave fusion network for RGB-D salient object detection. IEEE Trans. Cybern., 51(1):88–100, 2020.
[17] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935-2947, 2017.