SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\ell$  is a distance metric,  $\Delta$  is a margin between the feature vectors of different classes, and  $\hat{y}_{i,j}^{t}$  is the label of pixel  $(i,j)$ . Minimizing this loss separates the classes represented in the latent space. For step  $t > 1$ ,  $\hat{y}_{i,j}^{t}$  of an unknown-class pixel will utilize a pseudo label where its assigned label is computed based on the closest cluster. In addition, since the prototypical vectors of classes  $c \in \mathcal{C}^{1\dots t-1}$  have been well learned to represent for classes, these vectors  $\mathbf{p}_c$  (where  $c \in \mathcal{C}^{1\dots t-1}$ ) will be frozen at step  $t$  to maintain its learned knowledge of classes  $\mathcal{C}^{1\dots t-1}$ .
The set of prototypical vectors at current step  $t$ , i.e.,  $\mathbf{p}_0$  and  $\mathbf{p}_c$  where  $c \in \mathcal{C}^t$  are updated gradually with respect to the growth of feature vectors. In particular, the prototypical vector  $\mathbf{p}_c$  will be
periodically updated (after every  $M$  iterations) with momentum  $\eta$  based on the set of features  $\mathbf{f}_{i,j}$  of class  $c$ . Following common practices [20, 16], to effectively support the updating step and memory efficiency, for each class  $c$ , we only maintain a set of features  $\mathcal{S}_c$  with a fixed length of  $L$ . Algorithm 1 in the supplementary illustrates an overview of computing the prototypical contrastive clustering loss while updating the class prototypes. Fig. 2 illustrates our proposed FairCL framework.
3.3 Prototypical Constrative Clustering Loss to Catastrophic Forgetting
Knowledge Distillation is a common continual learning approach [36, 3, 12, 46] where the knowledge of the previous model will be distilled into the current model. This mechanism prevents the segmentation model from diverging knowledge learned previously and avoiding the catastrophic forgetting problem. This continual learning paradigm has been widely adopted due to its efficiency in computation. In addition, this approach also does not require data rehearsal, i.e., storing the data samples of previous tasks. In this paper, we demonstrate that our Prototypical Constrative Clustering approach is a comprehensive upper limit of the Knowledge Distillation approach. In particular, the common knowledge distillation approach can be formulated as follows:
$$
\mathcal {L} _ {\text {d i s t i l l}} \left(\mathbf {x} ^ {t}, \mathcal {F}, \theta_ {t}, \theta_ {t - 1}\right) = \mathcal {D} \left(\mathbf {f} ^ {t - 1}, \mathbf {f} ^ {t}\right) \tag {10}
$$
where  $\mathbf{f}^t$  and  $\mathbf{f}^{t - 1}$  are the features of the input  $\mathbf{x}^t$  produced by the segmentation model at step  $t$  and step  $t - 1$ , respectively; and the distance metric  $\mathcal{D}$  measure the knowledge gap between  $\mathbf{f}^t$  and  $\mathbf{f}^{t - 1}$ .
Proposition 1: The Prototypical Constrative Clustering Approach is the generalized upper bound of the Knowledge Distillation Approach.
$$
\mathcal {L} _ {\text {d i s t i l l}} \left(\mathbf {x} ^ {t}, \mathcal {F}, \theta_ {t}, \theta_ {t - 1}\right) = \mathcal {O} \left(\mathcal {L} _ {\text {c l u s t e r}} \left(\mathbf {x} ^ {t}, \mathcal {F}, \theta_ {t}\right)\right) \tag {11}
$$
Proof: Without lack of generality, we assume that  $\mathcal{D}$  is a metric that measures distance between features. Given a set of fixed prototypical vectors  $\mathbf{p}_c$ , we consider the following triangle inequality of the metric  $\mathcal{D}$  as follows:
$$
\begin{array}{l} \forall c \in \{0 \} \cup \mathcal {C} ^ {1.. t}: \quad \mathcal {D} (\mathbf {f} ^ {t}, \mathbf {f} ^ {t - 1}) \leq \mathcal {D} (\mathbf {f} ^ {t}, \mathbf {p} _ {c}) + \mathcal {D} (\mathbf {p} _ {c}, \mathbf {f} ^ {t - 1}) \\ \Leftrightarrow \mathcal {D} \left(\mathbf {f} ^ {t}, \mathbf {f} ^ {t - 1}\right) \leq \frac {1}{| \mathcal {C} |} \sum_ {c} \left[ \mathcal {D} \left(\mathbf {f} ^ {t}, \mathbf {p} _ {c}\right) + \mathcal {D} \left(\mathbf {p} _ {c}, \mathbf {f} ^ {t - 1}\right) \right] \tag {12} \\ \end{array}
$$
where  $|\mathcal{C}|$  is the number of prototypical vectors. The prototypical vectors  $\mathbf{p}_c$  and the feature vectors  $\mathbf{f}^{t - 1}$  produced by the segmentation model at step  $t - 1$  are considered as constant features as the model in the previous model  $t - 1$  has been fixed at step  $t$ . Thus, the distance  $\mathcal{D}(\mathbf{p}_c, \mathbf{f}^{t - 1})$  could be considered as constant number. Then, Eqn. (12) can be further derived as follows:
$$
\begin{array}{l} \mathcal {D} \left(\mathbf {f} ^ {t}, \mathbf {f} ^ {t - 1}\right) = \mathcal {O} \left(\frac {1}{| \mathcal {C} |} \sum_ {c} \left[ \mathcal {D} \left(\mathbf {f} ^ {t}, \mathbf {p} _ {c}\right) + \mathcal {D} \left(\mathbf {p} _ {c}, \mathbf {f} ^ {t - 1}\right) \right]\right) = \mathcal {O} \left(\sum_ {c} \mathcal {D} \left(\mathbf {f} ^ {t}, \mathbf {p} _ {c}\right)\right) \tag {13} \\ \Rightarrow \mathcal {L} _ {\text {d i s t i l l}} \left(\mathbf {x} ^ {t}, \mathcal {F}, \theta_ {t}, \theta_ {t - 1}\right) = \mathcal {O} \left(\mathcal {L} _ {\text {c l u s t e r}} \left(\mathbf {x} ^ {t}, \mathcal {F}, \theta_ {t}\right)\right) \\ \end{array}
$$
Intuitively, under this upper bound in Eqn. (13), by only optimizing our prototypical contrastive clustering loss, the knowledge distillation constraint has also been implicitly imposed. Beyond the property of generalized upper bound stated in Proposition 1, our approach offers other benefits over the knowledge distillation approach. In particular, our approach is computationally efficient, where our method only requires a single forward pass of the segmentation model. Meanwhile, the knowledge distillation demands two forward passes for both the current and previous models, which also requires additional computational memory for the previous model. Moreover, our approach provides a better representation of each class  $c$  through the prototypical vector  $\mathbf{p}_c$ . This mechanism helps to effectively maintain the knowledge of classes learned previously while allowing the model to update the new knowledge without rehearsing the old data.
3.4 Learning Conditional Structural Consistency
The conditional structural constraint plays an important role as it will ensure the consistency of the predicted segmentation map. However, modeling the conditional structural constraint  $\log \left(\frac{q(\mathbf{y}_{\setminus(i,j)}|y_{i,j})}{p(\mathbf{y}_{\setminus(i,j)}|y_{i,j})}\right)$  in Eqn. (7) is a quite challenging problem due to two factors, i.e., (1) the unknown ideal conditional
distribution  $q(\mathbf{y}_{\backslash (i,j)}|y_{i,j})$ , and the complexity of the distribution  $q(\mathbf{y}_{\backslash (i,j)}|y_{i,j})$ . To address the first limitation of unknown ideal distribution, let us consider the following tight bound as follows:
$$
\mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x})} \log \left(\frac {q \left(\mathbf {y} _ {\backslash (i , j)} \mid y _ {i , j}\right)}{p \left(\mathbf {y} _ {\backslash (i , j)} \mid y _ {i , j}\right)}\right) \leq - \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x})} \log p \left(\mathbf {y} _ {\backslash (i, j)} \mid y _ {i, j}\right) \tag {14}
$$
The inequality in Eqn. (14) always hold with respect to any form ideal distribution  $q(\cdot)$ . Thus, optimizing the negative log-likelihood of  $\log p(\mathbf{y}_{\backslash (i,j)}|y_{i,j})$  could also regularize the conditional structural constraint due to the upper bound of Eqn. (14). More importantly, the requirement of ideal data distribution during training has also been relaxed. However, up to this point, the second limitation of modeling the complex distribution  $q(\mathbf{y}_{\backslash (i,j)}|y_{i,j})$  has still not been solved.
To address this problem, we adopt the Markovian assumption [5, 39] to model conditional structural consistency. In particular, we propose a simple yet effective approach to impose the consistency of the segmentation map through the prediction at location  $(i,j)$  and predictions of its neighbor pixels. Formally, the conditional structure consistency can be formed via the Gaussian kernel as follows:
$$
- \log p \left(\mathbf {y} _ {\backslash (i, j)} \mid y _ {i, j}\right) \propto \sum_ {\left(i ^ {\prime}, j ^ {\prime}\right) \in \mathcal {N} (i, j)} \exp \left\{- \frac {\left| \left| x _ {t} ^ {i , j} - x _ {i ^ {\prime} , j ^ {\prime}} ^ {t} \right| \right| _ {2} ^ {2}}{2 \sigma_ {1} ^ {2}} - \frac {\left| \left| y _ {t} ^ {i , j} - y _ {i ^ {\prime} , j ^ {\prime}} ^ {t} \right| \right| _ {2} ^ {2}}{2 \sigma_ {2} ^ {2}} \right\} \tag {15}
$$
where  $\mathcal{N}(i,j)$  is the set of neighbor pixels of  $(i,j)$ ,  $\{\sigma_1,\sigma_2\}$  are the scale hyper-parameters of the Gaussian kernels. The conditional structural consistency loss defined in Eqn. (15) enhances the smoothness and maintains the consistency of the predicted segmentation map by imposing similar predictions of neighbor pixels with similar contextual colors.
In this section, we first describe the datasets and metrics used in our experiments. Then, we present the ablation studies to illustrate the effectiveness of our proposed method. Finally, we compare our approach with prior CSS methods to demonstrate our SOTA performance.
4.1 Datasets and Evaluation Protocols
Datasets ADE20K [49] is a semantic segmentation dataset that consists of more than 20K scene images of 150 semantic categories. Each image has been densely annotated with pix-level objects and objects parts labels. Cityscapes [10] is a real-world autonomous driving dataset collected in European. This dataset includes 3,975 urban images with high-quality, dense labels of 30 semantic categories. PASCAL VOC [13] is a common dataset that consists of more than 10K images of 20 classes.
Implementation Two segmentation network architectures are used in our experiments, i.e., (1) DeepLab-V3 [6] with the ResNet-101 backbone, and (2) SegFormer [45] with MiT-B3 backbone. Further details of our implementation will be available in our supplementary.
Evaluation Protocols: Following [12], we focus on the overlapped CSS evaluation. Our proposed method is evaluated on several settings for each dataset, i.e., ADA20K 100-50 (2 steps), ADA20K 100-10 (6 steps), ADA20K 100-5 (11 steps), Cityscapes 11-5 (3 steps), Cityscapes 11-1 (11 steps),
Backbone	Lcluster	Lclass	Lcons	0-100	100-150	all	avg
mIoU	STD	mIoU	STD	mIoU	STD	mIoU	STD
DeepLab-V3	✓	0.08	0.84	19.52	20.18	6.52	13.14	24.41	13.14
41.71	19.90	15.33	21.96	32.97	23.03	37.58	23.03
✓	✓	42.25	19.31	18.55	20.52	34.40	22.07	38.35	22.07
✓	✓	✓	43.40	19.08	24.04	19.12	36.99	21.67	40.45	21.67
SegFormer	✓	0.10	0.84	23.18	19.83	7.74	15.74	25.82	15.74
43.40	19.35	21.60	22.06	36.18	22.32	39.85	22.32
✓	✓	43.35	19.03	23.50	20.75	36.78	21.86	40.34	21.86
✓	✓	✓	43.56	18.71	25.46	19.99	37.56	21.10	40.73	21.10
Table 1: Effectiveness of our approach on the ADE20K 100-50 benchmark. Two different networks, i.e., DeepLab-V2 [5] and SegFormer [45], and three different losses, i.e., Prototypical Contrastive Clustering  $(\mathcal{L}_{cluster})$ , Fairness Loss  $(\mathcal{L}_{class})$ , and Conditional Structural Consistency  $(\mathcal{L}_{cons})$ .
Cityscapes 1-1 (21 steps), Pascal VOC 15-1 (3 steps), and Pascal VOC 10-1 (11 steps). The mean Intersection over Union (mIoU) metric is used in our experiments. The mIoU is computed after the last step for the classes learned from the first step, the later continual classes, and all classes. The mIoU for the initial classes shows the robustness of the model to catastrophic forgetting, while the metric for the later classes reflects the ability to learn new classes. To measure the fairness of the model, we also report the standard deviation (STD) of IoUs over all classes.
Our ablative experiments study the effectiveness of our proposed FairCL approach on the performance of the CSS model and fairness improvement on the ADE20K 100-50 benchmark (Table 1).
Effectiveness of the Network Backbone Table 1 illustrates the results of our approach using the DeepLab-V3 [5] with the Resnet101 backbone and the SegFormer [45] with a Transformer backbone, i.e. MiT-B3 [45]. As shown in our results, the performance of segmentation models using a more powerful backbone, i.e., Transformer, outperforms the models using the Resnet backbone. The capability of learning new classes has been improved notably, i.e., the mIoU of classes 101-150 in the full configuration has been improved from  $19.86\%$  to  $25.46\%$  while the model keeps robust to catastrophic forgetting, the mIoU has been increased from
Method	11-53 steps	11-111 steps	1-121 steps
Joint	79.30	79.30	79.30
LWF-MC [31]	58.90	56.92	31.24
ILT [25]	59.14	57.75	30.11
MiB [3]	61.51	60.02	42.15
PLOP [12]	63.51	62.05	45.24
RCIL [46]	64.30	63.00	48.90
FairCL + DeepLab-V3	66.96	66.61	49.22
FairCL + SegFormer	67.85	67.09	55.68
Table 2: Final mIoU (%) for Continual Semantic Segmentation on Cityscapes.
41.96% to 43.56% in the classes 0-100. Additionally, fairness between classes has been promoted when the standard deviation of the IoU over classes has been reduced from 21.67% to 21.10%.
Effectiveness of the Prototypical Contrautive Clustering Loss We evaluate the impact of the Prototypical Contrautive Clustering Loss  $(\mathcal{L}_{cluster})$  in improving the performance in the continual learning problem compared to the fine-tuning approach. As shown in Table 1, the clustering loss has significant improvements in the catastrophic forgetting robustness compared to using only the Softmax loss. In particular, the mIoU of classes 0-100 for both DeepLab-V3 and SegFormer backbones has been improved by  $+41.63\%$  and  $+43.30\%$  respectively that makes the overall mIoU increase by  $+26.45\%$  and  $+28.44\%$ , and the average mIoU between classes increases by  $+13.17\%$  and  $+14.03\%$ . Although the STD of IoUs has slightly increased in this setting, the major target of our  $\mathcal{L}_{cluster}$  is used to model the catastrophic forgetting and background shift problems in CSS illustrated by the significant performance improvement of mIoU.
Effectiveness of the Fairness Treatment Loss As reported in Table 1, the fairness treatment from the class distribution loss  $\mathcal{L}_{\text {class }}$  significantly improves the overall performance and the accuracy of classes. In detail, the STD of IoU from classes has been reduced by  $0.96\%$  and  $0.46\%$  for both backbones while the mIoU has been improved from  $32.97\%$  to  $34.40\%$  and from  $36.18\%$  to  $36.78\%$ , respectively. The results have shown that our approach has promoted the fairness of the model.
Effectiveness of the Conditional Structural Consistency The full configuration in Table 1 shows experimental results of our model using conditional structure constraint loss  $\mathcal{L}_{\text{cons}}$ . As illustrated in our results, the conditional structure constraint demonstrates effective improvement. Indeed, it
Method	100-50 (2 steps)	50-50 (3 steps)	100-10 (6 steps)
0-100	101-150	all	avg	0-50	51-150	all	avg	0-100	101-150	all	avg
Joint	44.30	28.20	38.90	-	51.10	32.80	38.90	-	44.30	28.20	38.90	-
ILT [25]	18.29	14.40	17.00	29.42	3.53	12.85	9.70	30.12	0.11	3.06	1.09	12.56
MiB [3]	40.52	17.17	32.79	37.31	45.57	21.01	29.31	38.98	38.21	11.12	29.24	35.12
PLOP [12]	41.87	14.89	32.94	37.39	48.83	20.99	30.40	39.42	40.48	13.61	31.59	36.64
RCIL [46]	42.30	18.80	34.50	38.48	48.30	25.00	32.50	-	39.30	17.60	32.10	-
MiB + AWT [15]	40.90	24.70	35.60	-	46.60	26.85	33.50	-	39.10	21.28	33.20	-
SSUL [4]	41.28	18.02	33.58	-	48.38	20.15	29.56	-	40.20	18.75	33.10	-
SATS [30]	-	-	-	-	-	-	-	-	41.42	19.09	34.18	-
FairCL + DeepLab-V3	43.40	24.04	36.99	40.45	49.65	26.84	34.55	41.68	41.73	20.36	34.65	39.01
FairCL + SegFormer	43.56	25.46	37.56	40.73	49.62	27.78	35.15	42.25	42.21	21.91	35.49	39.36
Table 3: Continual Semantic Segmentation results on ADE20k in Mean IoU (%).
promotes the accuracy of the initial classes and the novel classes when the mIoU has been increased from  $43.35\%$  to  $43.56\%$  and from  $23.50\%$  to  $25.46\%$  respectively with the Transformer backbone. The fairness of classes is also improved as the standard deviation of the IoU of classes 0-100 and classes 101-150 is reduced from  $19.03\%$  to  $18.71\%$  and from  $20.75\%$  to  $19.99\%$ .
Cityscapes As shown in Table 2, our FairCL outperforms previous SOTA methods evaluated on Cityscapes benchmarks. In particular, in the 11-5 task, our method using Resnet and Transformer achieves the mIoU of  $66.96\%$  and  $67.85\%$  respectively which shows better performance than prior methods. Meanwhile, the results for the 11-1 task are  $66.61\%$  and  $67.09\%$  w.r.t. the Resnet and Transformer backbones. For the 1-1 task, the mIoU of our method is  $49.22\%$  and  $55.68\%$ .
ADE20K Table 3 presents our experimental results using ResNet and Transformer backbones compared to prior SOTA approaches. Our proposed approach achieves SOTA performance and outperforms prior methods. In particular, our approach achieves the final mIoU of  $36.99\%$  for Resnet and  $37.56\%$  for Transformer in the 100-
Figure 3: Qualitative results of Our Approach and PLOP [12] on ADE20K 100-10.
50 tasks. For the 50-50 tasks, the model reaches the final mIoU of  $34.55\%$  and  $35.15\%$  for the Resnet and Transformer backbones, respectively while the result of the prior method [15] is  $33.50\%$ . Meanwhile, the overall results of our method for the 100-10 task are  $34.65\%$  and  $35.49\%$  which shows outperforming previous approaches. Fig. 3 visualizes the qualitative result of our method compared to PLOP [12]. Initially, the ground truth contains the class "car" in the first step and the class "minibike" in the third step. Then, in the fourth step, the class "bicycle" is included. As a result, PLOP [12] partly forgets the "minibike" information when learning the class "bicycle" information. Meanwhile, our method consistently maintains the information of "minibike" and predicts segmentation correctly.
Method	15-1 (6 steps)	10-1 (11 steps)
0-15	16-20	all	0-10	11-20	all
Joint	79.8	72.4	77.4	78.4	76.4	77.4
LWF [31]	6.0	3.9	5.5	8.0	2.0	4.8
ILT [25]	9.6	7.8	9.2	7.2	3.7	5.5
MiB [3]	38.0	13.5	32.2	20.0	20.1	20.1
SDR [26]	47.3	14.7	39.5	32.4	17.1	25.1
PLOP [12]	65.1	21.1	54.6	44.0	15.5	30.5
RCIL [46]	70.6	23.7	59.4	55.4	15.1	34.3
FairCL + DeepLab-V3	72.0	22.7	60.3	42.3	25.6	34.4
FairCL + SegFormer	73.5	22.8	61.5	57.1	14.2	36.6
Pascal VOC As shown in Table 4, the proposed method outperforms the prior approaches evaluated on the Pascal VOC 2012 dataset. In detail, our method achieves the overall mIoU of  $61.5\%$  in the 15-1 task while the result of the previous method [46] is  $59.4\%$ . Meanwhile, the mIoU in the 10-1 task is  $36.6\%$  which shows better performance than the prior methods.
Table 4: The mIoU (%) of CSS on Pascal VOC.
5 Conclusions and Dicussions
Conclusions: This paper introduces a novel fair
ness continual learning approach in semantic segmentation by analyzing the effect of class distribution. Furthermore, a new learning paradigm of continual learning, i.e., the prototypical Contrastive clustering loss, is proposed to sufficiently address the catastrophic forgetting and the background shift problems. The experimental results on the three CSS benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC, have shown our SOTA performance.
Limitations: Our paper has chosen specific configurations of network backbones and hyperparameters to support our hypothesis. However, the other aspects of learning have not been fully investigated, e.g., learning hyper-parameters, the selected neighbor pixels, or other forms of  $\mathcal{L}_{\text{cons}}$ .
Broader Impacts: This work studies the problem of Fairness in Continual Learning which is a step toward fairness awareness in continual semantic segmentation. Our contributions emphasize the importance of fairness in continual semantic segmentation learning and provide a solution to address the fairness concern that increases the robustness and trustworthiness of the segmentation model.
Acknowledgment This work is partly supported by NSF Data Science, Data Analytics that are Robust and Trusted (DART), and Googler Initiated Research Grant. We also thank Utsav Prabhu for invaluable discussions and suggestions and acknowledge the Arkansas High-Performance Computing Center for providing GPUs.
[42] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.
[43] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Dada: Depth-aware domain adaptation in semantic segmentation. In ICCV, 2019.
[44] J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[45] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
[46] C.-B. Zhang, J.-W. Xiao, X. Liu, Y.-C. Chen, and M.-M. Cheng. Representation compensation networks for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7053-7064, 2022.
[47] P. Zhang, B. Zhang, T. Zhang, D. Chen, Y. Wang, and F. Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. arXiv preprint arXiv:2101.10979, 2021.
[48] B. Zhao, X. Xiao, G. Gan, B. Zhang, and S.-T. Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13208-13217, 2020.
[49] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633-641, 2017.