SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thanh-Dat Truong $^{1}$ , Hoang-Quan Nguyen $^{1}$ , Bhiksha Raj $^{2,3}$ , and Khoa Luu $^{1}$
$^{1}$ CVIU Lab, University of Arkansas, Fayetteville, AR, 72701
2Carnegie Mellon University, Pittsburgh, PA, 15213
<sup>3</sup>Mohammed bin Zayed University of AI, Abu Dhabi, UAE
{tt032, hn016, khoaluu}@uark.edu, bhiksha@cs.cmu.edu
Continual semantic segmentation aims to learn new classes while maintaining the information from the previous classes. Although prior studies have shown impressive progress in recent years, the fairness concern in the continual semantic segmentation needs to be better addressed. Meanwhile, fairness is one of the most vital factors in deploying the deep learning model, especially in human-related or safety applications. In this paper, we present a novel Fairness Continual Learning approach to the semantic segmentation problem. In particular, under the fairness objective, a new fairness continual learning framework is proposed based on class distributions. Then, a novel Prototypical Contrastive Clustering loss is proposed to address the significant challenges in continual learning, i.e., catastrophic forgetting and background shift. Our proposed loss has also been proven as a novel, generalized learning paradigm of knowledge distillation commonly used in continual learning. Moreover, the proposed Conditional Structural Consistency loss further regularized the structural constraint of the predicted segmentation. Our proposed approach has achieved State-of-the-Art performance on three standard scene understanding benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC, and promoted the fairness of the segmentation model.
1 Introductions
Convolutional Neural Networks (CNNs) [5, 7] and Transformers [45, 8] have been introduced to approach semantic segmentation tasks where the models learn from the large-scale data having known classes at once. These segmentation models learned on large-scale data may perform poorly as they may counter the new objects or new environments. To bridge this gap, several approaches have been proposed to adapt the model to the new data. Domain Adaptation is one of the common approaches [39, 42, 43, 1, 18] that adaptively transfer the segmentation model into the deployed environment. However, domain adaptation cannot handle when new objects appear due to its close-set setup. Also, this approach requires access to both original and new training data. In practice, the segmentation models should be capable of learning new classes continually without
re-training from the previous data. This paradigm is defined as Continual Semantic Segmentation
The Class Distribution based on the Number of Pixels
Figure 1: The Class Distribution of ADE20K. In the ADE20K 50-50 (3 steps) benchmark, the distribution of classes in the majority group in the early step dominates the ones in the minority groups in the later steps. The distributions of classes gradually decrease over steps.
[12, 46]. The current continual semantic segmentation approaches [12, 46] concentrate on addressing two main challenges, i.e., (1) catastrophic forgetting [33, 14, 38] and (2) background shift [12, 46]. The former problem indicates the forgetting issue of the model about the previous knowledge when learning on new training data [12, 46, 4]. Meanwhile, the latter problem refers to the classes of previous or future data that have collapsed into a background class [3]. However, another critical problem that needs attention is the fairness issue in semantic segmentation.
Fairness in semantic segmentation refers to the problem of the model where it behaves unfairly between classes in the dataset due to the class distribution, i.e., the model tends to produce prediction bias toward a specific group of classes occupying a large region on an image or frequently existing in the dataset (as shown in Fig. 1). The unfair predictions could result in severe problems, especially in human-related applications that could influence human safety. Moreover, the fairness problem could even be well observed in the context of continual learning when the model encounters new classes without accessing previous training data. The prior work of continual learning in image classification [48] has also considered this problem. Several prior studies [1, 18, 19] in semantic segmentation have tried to reduce the effect of the class imbalance by introducing the weighted cross entropy [11, 44, 19], focal loss [1], over-sampling techniques [18, 47, 1]. However, the fairness problem in continual semantic segmentation has yet to be well-defined and directly addressed. There should be more attention on addressing the fairness issue in continual semantic segmentation. Therefore, this work aims to address the fairness problem in continual semantic segmentation caused by the imbalanced class distribution defined based on the number of pixels of each class in the dataset.
Contributions of this Work: This work presents a novel Fairness Continual Learning (FairCL) approach to semantic scene segmentation. Our contributions can be summarized as follows. First, under the perspective of fairness learning, the new metric is formulated to measure the fairness of the model via the error rate among classes. Then, the metric is further derived into the three main objectives, i.e., (1) the Task-specific Objective that handles the catastrophic forgetting problem, (2) the Fairness Objective that maintains the fairness of predictions produced by the model based on the class distribution, and (3) the Conditional Structural Constraint that imposes the consistency of the segmentation predictions. Second, to sufficiently model the continual learning problem, the novel Prototypical Contrastive Clustering loss is presented to address the catastrophic forgetting and the background shifting problems. Moreover, the proposed Prototypical Contrastive Clustering loss has been proven to be a generalized paradigm of knowledge distillation approaches commonly adopted in continual learning. Last, sufficient ablation studies have shown the effectiveness of our method in continual learning and promoted the fairness of the model. The empirical comparison with prior methods has shown the State-of-the-Art (SOTA) results on the standard benchmarks.
Continual Semantic Segmentation While there is enormous progress in the continual learning problem, most approaches focus on image classification. Existing continual learning methods [22, 17] have been extended for continual semantic segmentation in medical images [27, 28] and general datasets [24]. Cermelli et al. [3] addressed a background shifting problem in the continual semantic segmentation. Douillard et al. [12] introduced a multi-scale spatial distillation scheme that preserves long- and short-range spatial relationships at the feature level. Volpi et al. [41] presented a continual learning framework where the model is sequentially trained on multiple labeled data domains. Rostami et al. [34] proposed a continual learning framework under the unsupervised domain adaptation setting using data rehearsal. Saporta et al. [35] introduced a multi-head knowledge distillation framework for the continual unsupervised domain. Zhang et al. [46] presented a representation compensation module to decouple the representation learning of both old and new knowledge. Cha et al. [4] suggested finding the unknown class from the background to distinguish the representations of the potential future classes. Qiu et al. [30] proposed a self-attention transferring method to capture both within-class and between-class knowledge. Phan et al. [29] introduced a class-similarity knowledge-distillation method to revise the old classes more likely to be forgotten and better learn new classes related to the previous classes.
Class Imbalance and Fairness Approaches in Semantic Segmentation Jiawei et al. [32] presented a balanced Softmax loss that reduces the distribution shift of labels and alleviates the long-tail issue. Wang et al. [44] proposed a Seesaw loss that reweights the contributions of gradients produced by positive and negative instances of a class by using two regularizers, i.e., mitigation and compensation. Liu et al. [23] proposed an algorithm that handles imbalanced classification, few-shot learning, and
open-set recognition using dynamic meta-embedding. Chu et al. [9] proposed a stochastic training scheme for semantic segmentation, which improves the learning of debiased and disentangled representations. Szabo et al. [37] proposed tilted cross-entropy loss to reduce the performance differences, which promotes fairness among the target classes. Truong et al. [40] introduced a fairness domain adaptation approach to semantic segmentation that maintains the fairness of the predictions produced by the segmentation model on the target domain.
3 The Proposed Fairness Continual Learning Approach
Let  $\mathcal{F}$  parameterized by  $\theta$  be the deep semantic segmentation model that maps an input image  $\mathbf{x} \in \mathcal{X}$  to the segmentation map  $\mathbf{y} \in \mathcal{Y}$ ,  $\mathbf{y} = \mathcal{F}(\mathbf{x}, \theta)$ . Continual Semantic Segmentation (CSS) aims to learn a model in  $T$  steps. In each step, the segmentation model  $\mathcal{F}$  encounters a dataset  $\mathcal{D}^t = \{\mathbf{x}^t, \hat{\mathbf{y}}^t\}$  where  $\mathbf{x}^t$  is the input image and  $\hat{\mathbf{y}}^t$  is the ground-truth segmentation at time  $t \in [1..T]$ . The current ground-truth segmentation map  $\hat{\mathbf{y}}^t$  only contains the labels of the current classes  $\mathcal{C}^t$  and all the class labels of precision steps,  $\mathcal{C}^{1..t-1}$ , or the future steps,  $\mathcal{C}^{t+1..T}$  are collapsed into a background class or ignored. Formally, learning the semantic segmentation at time step  $t$  can be formulated as follows:
$$
\theta_ {t} ^ {*} = \arg \min  _ {\theta_ {t}} \mathbb {E} _ {\mathbf {x} ^ {t}, \hat {\mathbf {y}} ^ {t} \in \mathcal {D} ^ {t}} \mathcal {L} \left(\mathcal {F} \left(\mathbf {x} ^ {t}, \theta_ {t}\right), \hat {\mathbf {y}} ^ {t}\right) \tag {1}
$$
where  $\theta_t^*$  is the parameters of  $\mathcal{F}$  at time step  $t$ ,  $\mathcal{L}$  is the objective learning of the continual learning task. In CSS learning, at the current time step  $t$ , the segmentation model  $\mathcal{F}$  is expected to not only predict all the classes  $\mathcal{C}^{1..t-1}$  learned in the previous steps but also predict the current new classes  $\mathcal{C}^t$ . Three significant challenges have been identified in this learning setting and should be addressed.
- Background Shift At time step  $t$ , the labels of previous and future steps have been ignored. Thus, the pixels of these classes are ambiguous, which means these could contain either the class of previous or future steps. During learning  $\mathcal{C}^t$ , the model could consider these classes as negative samples. As a result, the model tends to learn non-discriminative features for these pixels, leading to difficulty learning new classes or forgetting the old ones.
- Catastrophic Forgetting cause the model may partially or completely forget the knowledge of classes  $\mathcal{C}^{1..t - 1}$  when learning the new classes  $\mathcal{C}^t$ . This problem could be caused by the background shift and the learning mechanism. Since classes in  $\mathcal{C}^{1..t - 1}$  are considered as the background class at time step  $t$ , the model tends to update the knowledge of the new classes while the predictions of classes incline to be suppressed.
- Fairness While the prior approaches [12, 46, 4] focus on addressing the two above challenges, the fairness issue has received less attention and has not been well addressed yet. However, fairness is one of the most important criteria as it guarantees the model behaves fairly among not only classes in  $\mathcal{C}^t$  but also classes in  $\mathcal{C}^{1..t}$  that have been learned. The fairness in CSS is typically caused by the imbalance distribution between classes as several classes occupy the larger portion or exist more frequently than other classes (Fig. 1). In addition, the appearance of training samples at the training step  $t$  also exaggerates the bias due to the new classes since the classes in the previous training steps have collapsed.
To address these challenges in CSS, we first reconsider the semantic segmentation problem from the fairness viewpoint followed by introducing a novel approach to alleviate the fairness issue based on the ideal class distribution. Then, we introduce a novel Prototypical Contrastive Clustering loss to model background classes and catastrophic forgetting.
3.1 Fairness Learning Approach to Continual Semantic Segmentation
Maintaining fairness is one of the most important factors in learning CSS as it reduces the bias of the model toward a certain group of classes. Formally, under the fairness constraint in semantic segmentation, the error rate of each class should be equally treated by the segmentation model. Given classes  $c_{a}$  and  $c_{b}$  in the set of classes  $\mathcal{C}^{1..T}$ , this constraint can be formulated as follows:
$$
\max  _ {c _ {a}, c _ {b} \in \mathcal {C} ^ {1.. T}} \left| \mathbb {E} _ {\mathbf {x}, \hat {\mathbf {y}} \in \mathcal {D}} \sum_ {i, j} \mathcal {L} \left(y _ {i, j}, \hat {y} _ {i, j} = c _ {a}\right) - \mathbb {E} _ {\mathbf {x}, \hat {\mathbf {y}}} \sum_ {i, j} \mathcal {L} \left(y _ {i, j}, \hat {y} _ {i, j} = c _ {b}\right) \right| \leq \epsilon \tag {2}
$$
where  $y_{i,j}$  and  $\hat{y}_{i,j}$  is the prediction and ground truth at the pixel location  $(i,j)$ , respectively; the loss function  $\mathcal{L}$  measures the error rates of predictions. Intuitively, Eqn. (2) aims to maintain the
differences in the error rates between classes lower than a small threshold  $\epsilon$  to guarantee fairness among classes. For simplicity, this constraint can be considered as an additional loss while optimizing Eqn. (1). However, this approach could not guarantee the model being fair due to Eqn. (3).
$$
\begin{array}{l} \max  _ {c _ {a}, c _ {b} \in C ^ {1.. T}} \left| \mathbb {E} _ {\mathbf {x}, \hat {\mathbf {y}} \in \mathcal {D}} \sum_ {i, j} \mathcal {L} \left(y _ {i, j}, \hat {y} _ {i, j} = c _ {a}\right) - \mathbb {E} _ {\mathbf {x}, \hat {\mathbf {y}}} \sum_ {i, j} \mathcal {L} \left(y _ {i, j}, \hat {y} _ {i, j} = c _ {b}\right) \right| \tag {3} \\ \leq \sum_ {c _ {a}, c _ {b} \in \mathcal {C} ^ {1.. T}} \left| \mathbb {E} _ {\mathbf {x}, \hat {\mathbf {y}} \in \mathcal {D}} \sum_ {i, j} \mathcal {L} (y _ {i, j}, \hat {y} _ {i, j} = c _ {a}) - \mathbb {E} _ {\mathbf {x}, \hat {\mathbf {y}}} \sum_ {i, j} \mathcal {L} (y _ {i, j}, \hat {y} _ {i, j} = c _ {b}) \right| \leq 2 \left| \mathcal {C} ^ {1.. T} \right| [ \mathbb {E} _ {\mathbf {x}, \hat {\mathbf {y}} \in \mathcal {D}} \mathcal {L} (\mathcal {F} (\mathbf {x}, \theta), \hat {\mathbf {y}}) ] \\ \end{array}
$$
As shown in the above inequality, the constraint of Eqn. (2) has been bounded by the loss function in Eqn. (1). Although minimizing Eqn. (1) could also impose the constraint in Eqn. (2), the fairness issue still remains unsolved due to the imbalanced class distributions indicated through Eqn. (4).
$$
\theta^ {*} = \arg \min  _ {\theta} \left[ \int \mathcal {L} (\mathbf {y}, \hat {\mathbf {y}}) p (\mathbf {y}) p (\hat {\mathbf {y}}) d \mathbf {y} d \hat {\mathbf {y}} \right] = \arg \min  _ {\theta} \left[ \int \sum_ {i, j} \mathcal {L} \left(y _ {i, j}, \hat {y} _ {i, j}\right) p \left(y _ {i, j}\right) p \left(\mathbf {y} \backslash_ {(i, j)} \mid y _ {i, j}\right) p (\hat {\mathbf {y}}) d \mathbf {y} d \hat {\mathbf {y}} \right] \tag {4}
$$
where  $p(\cdot)$  is the data distribution,  $\mathbf{y}_{\backslash (i,j)}$  is the predicted segmentation map without the pixel at position  $(i,j)$ ,  $p(y^{k})$  is the class distribution of pixels, and  $p(\mathbf{y}_{\backslash (i,j)}|y_{(i,j)})$  is the conditional structural constraint of the predicted segmentation  $\mathbf{y}_{\backslash (i,j)}$  conditioned on the prediction  $y_{i,j}$ . Practically, the class distributions of pixels  $p(y_{i,j})$  suffer imbalanced issues, where several classes in the majority group significantly dominate other classes in the minority group. Then, learning by gradient descent method, the model could potentially bias towards the class in the majority group because the produced gradients of classes in the majority group tend to prevail over the ones in the minority group. Formally, considering the two classes  $c_{a}$  and  $c_{b}$  of the dataset where their distributions are skewed, i.e.,  $p(c_{a}) < p(c_{b})$ , the gradients produced are favored towards class  $c_{b}$  as shown in Eqn. (5).
$$
\begin{array}{l} \left| \left| \frac {\partial \int \sum_ {i , j} \mathcal {L} \left(y _ {i , j} , \hat {y} _ {i , j}\right) p \left(y _ {i , j} = c _ {a}\right) p \left(\mathbf {y} _ {\backslash (i , j)} \mid y _ {i , j}\right) p (\hat {\mathbf {y}}) d \mathbf {y} d \hat {\mathbf {y}}}{\partial \mathbf {y} _ {(c _ {a})}} \right| \right| \tag {5} \\ <   \left| \left| \frac {\partial \int \sum_ {k = 1} ^ {N} \mathcal {L} \left(y _ {i , j} , \hat {y} _ {i , j}\right) p \left(y _ {i , j} = c _ {b}\right) q \left(\mathbf {y} _ {\backslash (i , j)} \mid y _ {i , j}\right) p \left(\hat {\mathbf {y}} _ {k}\right) d \mathbf {y} d \hat {\mathbf {y}}}{\partial \mathbf {y} _ {(c _ {b})}} \right| \right| \\ \end{array}
$$
where  $||\cdot ||$  is the magnitude of gradients,  $\mathbf{y}_{(c_a)}$  and  $\mathbf{y}_{(c_b)}$  are the predictions of classes  $c_{a}$  and  $c_{b}$ .
Learning Fairness from Ideally Fair Distribution To address this problem, we first assume that there exists an ideal distribution  $q(\cdot)$  where the class distributions  $q(y_{i,j})$  are equally distributed. Under this assumption, the model learned is expected to behave fairly among classes as there is no bias toward any groups of classes. It should be noted that our assumption is used to derive our learning objective and is going to be relaxed later. In other words, the ideal data is not required at training. Then, our learning objective in Eqn. (4) could be rewritten as follows:
$$
\theta^ {*} = \arg \min  _ {\theta} \left[ \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {y}), \hat {\mathbf {y}} \sim p (\hat {\mathbf {y}})} \sum_ {i, j} \mathcal {L} \left(y _ {i, j}, \hat {y} _ {i, j}\right) \frac {q \left(y _ {i , j}\right) q \left(\mathbf {y} _ {\backslash (i , j)} \mid y _ {i , j}\right)}{p \left(y _ {i , j}\right) p \left(\mathbf {y} _ {\backslash (i , j)} \mid y _ {i , j}\right)} \right] \tag {6}
$$
The fraction between the ideal distribution  $q(\cdot)$  and the actual data distribution  $p(\cdot)$  is the residual learning objective for the model to achieve the desired fairness goal. Let us further derive Eqn. (6) by taking the log as follows:
$$
\theta^ {*} = \arg \min  _ {\theta} \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x}), \hat {\mathbf {y}} \sim p (\hat {\mathbf {y}})} \left\{\mathcal {L} (\mathbf {y}, \hat {\mathbf {y}}) + \frac {1}{N} \sum_ {i, j} \left[ \log \left(\frac {q \left(y _ {i , j}\right)}{p \left(y _ {i , j}\right)}\right) + \log \left(\frac {q \left(\mathbf {y} \backslash (i , j) \mid y _ {i , j}\right)}{p \left(\mathbf {y} \backslash (i , j) \mid y _ {i , j}\right)}\right) \right] \right\} \tag {7}
$$
The detailed derivation of Eqn. (6) and Eqn. (7) will be available in the supplementary. As shown in the Eqn. (7), there are three learning objectives as follows:
- The Continual Learning Objective The first term, i.e.,  $\mathcal{L}(\mathbf{y},\hat{\mathbf{y}})$ , represents the task-specific loss which is the continual learning objective. This objective aims to address the catastrophic forgetting and background shift problems. To achieve this desired goal, we introduce a novel Prototypical Contrastive Clustering Loss that will be discussed in Sec. 3.2.
- The Fairness Objective The second term, i.e.,  $\mathcal{L}_{class} = \log \left(\frac{q(y_{i,j})}{p(y_{i,j})}\right)$ , maintains the fairness in the predictions produced by the model. This objective penalizes the prediction of
Figure 2: The Proposed Fairness Continual Learning Framework. The predicted segmentation maps are imposed the cross-entropy loss, the Prototypical Contrastive Clustering loss  $(\mathcal{L}_{cluster})$ , the Fairness Loss from Class Distribution  $(\mathcal{L}_{class})$ , and the Conditional Structural Consistency loss  $(\mathcal{L}_{cons})$
classes forcing the model to behave fairly based on the class distribution. Under the ideal distribution assumption where the model is expected to perform fairly, the  $q(y_{i,j})$  will be considered as a uniform distribution  $\mathcal{U}$ , i.e.,  $q(y_{i,j}) \sim \mathcal{U}(C)$  ( $C$  is the number of classes).
- The Conditional Structural Consistency Objective The third term, i.e.,  $\mathcal{L}_{\text{cons}} = \log \left(\frac{q(\mathbf{y}_{\setminus (i,j)}|y_{i,j})}{p(\mathbf{y}_{\setminus (i,j)}|y_{i,j})}\right)$ , regularizes the structural consistency of the prediction. This objective acts as a metric to constrain the structure of the predicted segmentation under the ideal distribution assumption. To model this conditional structure consistency, we introduce a conditional structural constraint based on the Markovian assumption discussed in Sec. 3.4.
3.2 Prototypical Contrastive Clustering Loss for Handling Unknown Classes
A successful continual learning approach should be able to model background classes without explicit supervision and confront the forgetting problem of previously learned classes when the labels of the task are provided to update the knowledge of the model [20]. A straightforward adoption of Softmax could not be enough to handle. Indeed, the unlabeled pixels will be ignored during training. Thus, it results in these unannotated pixels could be treated as negative samples. Consequently, the segmentation model tends to produce indiscriminative features for these unknown classes that limit the capability of learning new classes in future tasks or recognizing the classes learned previously.
To address this limitation, in addition to the Softmax loss, we introduce a novel Prototypical Contrastive Clustering Loss. In particular, the semantic segmentation pixel belonging to each class can be represented in latent space. Inspired by [20, 2, 21], the features representing the classes can be separated by defining it as a contrastive clustering problem [20] where features of the same class would be pulled closer while features of different classes would be pushed far away. In addition, the deep representations of unknown classes will be grouped into the same cluster of unknown classes to produce discriminative features against other classes.
Formally, for each class  $c \in \mathcal{C}^{1..t}$ , it is represented by a prototypical vector  $\mathbf{p}_c$ . In addition, the additional prototypical vector  $\mathbf{p}_0$  represents a cluster of unknown classes. Let  $f_{i,j}^t$  be a feature representation of pixel at location  $(i,j)$  of the input  $\mathbf{x}^t$ . Then, the Prototypical Contraitive Clustering Loss  $\mathcal{L}_{\text{cluster}}$  can be defined via a distance  $\mathcal{D}$  as follows:
$$
\mathcal {L} _ {\text {c l u s t e r}} \left(\mathbf {x} ^ {t}, \mathcal {F}, \theta_ {t}\right) = \sum_ {i, j} \sum_ {c} \mathcal {D} \left(\mathbf {f} _ {i, j} ^ {t}, \mathbf {p} _ {c}\right) \tag {8}
$$
$$
\mathcal {D} \left(\mathbf {f} _ {i, j} ^ {t}, \mathbf {p} _ {c}\right) = \left\{ \begin{array}{l l} \ell \left(\mathbf {f} _ {i, j} ^ {t}, \mathbf {p} _ {c}\right) & \text {I f} \hat {y} _ {i, j} ^ {t} = c \\ \max  \left\{0, \Delta - \ell \left(\mathbf {f} _ {i, j} ^ {t}, \mathbf {p} _ {c} \right. \right\} & \text {o t h e r w i s e} \end{array} \right. \tag {9}
$$