Title: Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments
Venue: NeurIPS 2023
Accepted: yes
Keywords: Fairness Continual Learning; Semantic Segmentation; Contrastive Clustering;
Abstract: Continual semantic segmentation aims to learn new classes while maintaining the information from the previous classes. Although prior studies have shown impressive progress in recent years, the fairness concern in the continual semantic segmentation needs to be better addressed. Meanwhile, fairness is one of the most vital factors in deploying the deep learning model, especially in human-related or safety applications. In this paper,  we present a novel Fairness Continual Learning approach to the semantic segmentation problem.
In particular, under the fairness objective, a new fairness continual learning framework is proposed based on class distributions.
Then, a novel Prototypical Contrastive Clustering loss is proposed to address the significant challenges in continual learning, i.e., catastrophic forgetting and background shift. Our proposed loss has also been proven as a novel, generalized learning paradigm of knowledge distillation commonly used in continual learning. Moreover, the proposed Conditional Structural Consistency loss further regularized the structural constraint of the predicted segmentation. Our proposed approach has achieved State-of-the-Art performance on three standard scene understanding benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC, and promoted the fairness of the segmentation model.

Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments

Abstract

Continual semantic segmentation aims to learn new classes while maintaining the information from the previous classes. Although prior studies have shown impressive progress in recent years, the fairness concern in the continual semantic segmentation needs to be better addressed. Meanwhile, fairness is one of the most vital factors in deploying the deep learning model, especially in human-related or safety applications. In this paper, we present a novel Fairness Continual Learning approach to the semantic segmentation problem. In particular, under the fairness objective, a new fairness continual learning framework is proposed based on class distributions. Then, a novel Prototypical Contrastive Clustering loss is proposed to address the significant challenges in continual learning, i.e., catastrophic forgetting and background shift. Our proposed loss has also been proven as a novel, generalized learning paradigm of knowledge distillation commonly used in continual learning. Moreover, the proposed Conditional Structural Consistency loss further regularized the structural constraint of the predicted segmentation. Our proposed approach has achieved State-of-the-Art performance on three standard scene understanding benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC, and promoted the fairness of the segmentation model.

1 Introductions

Convolutional Neural Networks (CNNs) [5, 7] and Transformers [45, 8] have been introduced to approach semantic segmentation tasks where the models learn from the large-scale data having known classes at once. These segmentation models learned on large-scale data may perform poorly as they may counter the new objects or new environments. To bridge this gap, several approaches have been proposed to adapt the model to the new data. Domain Adaptation is one of the common approaches [39, 42, 43, 1, 18] that adaptively transfer the segmentation model into the deployed environment. However, domain adaptation cannot handle when new objects appear due to its close-set setup. Also, this approach requires access to both original and new training data. In practice, the segmentation models should be capable of learning new classes continually without

The Class Distribution based on the Number ofPixels

0.16

0.14

0.12

Car

0.10

0.08

0.06

Person

0.04

0.02

Boat

Bag

Bottle

Clock

0.00

20

120

140Class

Index

Step1

Step3

re-training from the previous data. This paradigm is defined as Continual Semantic Segmentation

[12, 46]. The current continual semantic segmentation approaches [12, 46] concentrate on addressing two main challenges, i.e., (1) catastrophic forgetting [33, 14, 38] and (2) background shift [12, 46]. The former problem indicates the forgetting issue of the model about the previous knowledge when learning on new training data [12, 46, 4]. Meanwhile, the latter problem refers to the classes of previous or future data that have collapsed into a background class [3]. However, another critical problem that needs attention is the fairness issue in semantic segmentation.

Fairness in semantic segmentation refers to the problem of the model where it behaves unfairly between classes in the dataset due to the class distribution, i.e., the model tends to produce prediction bias toward a specific group of classes occupying a large region on an image or frequently existing in the dataset (as shown in Fig. 1). The unfair predictions could result in severe problems, especially in human-related applications that could influence human safety. Moreover, the fairness problem could even be well observed in the context of continual learning when the model encounters new classes without accessing previous training data. The prior work of continual learning in image classification [48] has also considered this problem. Several prior studies [1, 18, 19] in semantic segmentation have tried to reduce the effect of the class imbalance by introducing the weighted cross entropy [11, 44, 19], focal loss [1], over-sampling techniques [18, 47, 1]. However, the fairness problem in continual semantic segmentation has yet to be well-defined and directly addressed. There should be more attention on addressing the fairness issue in continual semantic segmentation. Therefore, this work aims to address the fairness problem in continual semantic segmentation caused by the imbalanced class distribution defined based on the number of pixels of each class in the dataset.

Contributions of this Work: This work presents a novel Fair ness C ontinual L earning ( FairCL ) approach to semantic scene segmentation. Our contributions can be summarized as follows. First, under the perspective of fairness learning, the new metric is formulated to measure the fairness of the model via the error rate among classes. Then, the metric is further derived into the three main objectives, i.e., (1) the Task-specific Objective that handles the catastrophic forgetting problem, (2) the Fairness Objective that maintains the fairness of predictions produced by the model based on the class distribution, and (3) the Conditional Structural Constraint that imposes the consistency of the segmentation predictions. Second, to sufficiently model the continual learning problem, the novel Prototypical Contrastive Clustering loss is presented to address the catastrophic forgetting and the background shifting problems. Moreover, the proposed Prototypical Contrastive Clustering loss has been proven to be a generalized paradigm of knowledge distillation approaches commonly adopted in continual learning. Last, sufficient ablation studies have shown the effectiveness of our method in continual learning and promoted the fairness of the model. The empirical comparison with prior methods has shown the State-of-the-Art (SOTA) results on the standard benchmarks.

2 Related Work

Continual Semantic Segmentation While there is enormous progress in the continual learning problem, most approaches focus on image classification. Existing continual learning methods [22, 17] have been extended for continual semantic segmentation in medical images [27, 28] and general datasets [24]. Cermelli et al. [3] addressed a background shifting problem in the continual semantic segmentation. Douillard et al. [12] introduced a multi-scale spatial distillation scheme that preserves long- and short-range spatial relationships at the feature level. V olpi et al. [41] presented a continual learning framework where the model is sequentially trained on multiple labeled data domains. Rostami et al. [34] proposed a continual learning framework under the unsupervised domain adaptation setting using data rehearsal. Saporta et al. [35] introduced a multi-head knowledge distillation framework for the continual unsupervised domain. Zhang et al. [46] presented a representation compensation module to decouple the representation learning of both old and new knowledge. Cha et al. [4] suggested finding the unknown class from the background to distinguish the representations of the potential future classes. Qiu et al. [30] proposed a self-attention transferring method to capture both within-class and between-class knowledge. Phan et al. [29] introduced a class-similarity knowledgedistillation method to revise the old classes more likely to be forgotten and better learn new classes related to the previous classes.

Class Imbalance and Fairness Approaches in Semantic Segmentation Jiawei et al. [32] presented a balanced Softmax loss that reduces the distribution shift of labels and alleviates the long-tail issue. Wang et al. [44] proposed a Seesaw loss that reweights the contributions of gradients produced by positive and negative instances of a class by using two regularizers, i.e., mitigation and compensation. Liu et al. [23] proposed an algorithm that handles imbalanced classification, few-shot learning, and

open-set recognition using dynamic meta-embedding. Chu et al. [9] proposed a stochastic training scheme for semantic segmentation, which improves the learning of debiased and disentangled representations. Szabo et al. [37] proposed tilted cross-entropy loss to reduce the performance differences, which promotes fairness among the target classes. Truong et al. [40] introduced a fairness domain adaptation approach to semantic segmentation that maintains the fairness of the predictions produced by the segmentation model on the target domain.

3 The Proposed Fairness Continual Learning Approach

Let F parameterized by θ be the deep semantic segmentation model that maps an input image x ∈ X to the segmentation map y ∈ Y , y = F ( x , θ ) . Continual Semantic Segmentation (CSS) aims to learn a model in T steps. In each step, the segmentation model F encounters a dataset D t = { x t , ˆ y t } where x t is the input image and ˆ y t is the ground-truth segmentation at time t ∈ [1 ..T ] . The current ground-truth segmentation map ˆ y t only contains the labels of the current classes C t and all the class labels of prevision steps, C 1 ..t -1 , or the future steps, C t +1 ..T are collapsed into a background class or ignored. Formally, learning the semantic segmentation at time step t can be formulated as follows:

where θ ∗ t is the parameters of F at time step t , L is the objective learning of the continual learning task. In CSS learning, at the current time step t , the segmentation model F is expected to not only predict all the classes C 1 ..t -1 learned in the previous steps but also predict the current new classes C t . Three significant challenges have been identified in this learning setting and should be addressed.

Background Shift At time step t , the labels of previous and future steps have been ignored. Thus, the pixels of these classes are ambiguous, which means these could contain either the class of previous or future steps. During learning C t , the model could consider these classes as negative samples. As a result, the model tends to learn non-discriminative features for these pixels, leading to difficulty learning new classes or forgetting the old ones.

Catastrophic Forgetting cause the model may partially or completely forget the knowledge of classes C 1 ..t -1 when learning the new classes C t . This problem could be caused by the background shift and the learning mechanism. Since classes in C 1 ..t -1 are considered as the background class at time step t , the model tends to update the knowledge of the new classes while the predictions of classes incline to be suppressed.

Fairness While the prior approaches [12, 46, 4] focus on addressing the two above challenges, the fairness issue has received less attention and has not been well addressed yet. However, fairness is one of the most important criteria as it guarantees the model behaves fairly among not only classes in C t but also classes in C 1 ..t that have been learned. The fairness in CSS is typically caused by the imbalance distribution between classes as several classes occupy the larger portion or exist more frequently than other classes (Fig. 1). In addition, the appearance of training samples at the training step t also exaggerates the bias due to the new classes since the classes in the previous training steps have collapsed.

To address these challenges in CSS, we first reconsider the semantic segmentation problem from the fairness viewpoint followed by introducing a novel approach to alleviate the fairness issue based on the ideal class distribution. Then, we introduce a novel Prototypical Contrastive Clustering loss to model background classes and catastrophic forgetting.

3.1 Fairness Learning Approach to Continual Semantic Segmentation

Maintaining fairness is one of the most important factors in learning CSS as it reduces the bias of the model toward a certain group of classes. Formally, under the fairness constraint in semantic segmentation, the error rate of each class should be equally treated by the segmentation model. Given classes c a and c b in the set of classes C 1 ..T , this constraint can be formulated as follows:

where y i,j and ˆ y i,j is the prediction and ground truth at the pixel location ( i, j ) , respectively; the loss function L measures the error rates of predictions. Intuitively, Eqn. (2) aims to maintain the

differences in the error rates between classes lower than a small threshold ϵ to guarantee fairness among classes. For simplicity, this constraint can be considered as an additional loss while optimizing Eqn. (1). However, this approach could not guarantee the model being fair due to Eqn. (3).

As shown in the above inequality, the constraint of Eqn. (2) has been bounded by the loss function in Eqn. (1). Although minimizing Eqn. (1) could also impose the constraint in Eqn. (2), the fairness issue still remains unsolved due to the imbalanced class distributions indicated through Eqn. (4).

where p ( · ) is the data distribution, y \ ( i,j ) is the predicted segmentation map without the pixel at position ( i, j ) , p ( y k ) is the class distribution of pixels, and p ( y \ ( i,j ) | y ( i,j ) ) is the conditional structural constraint of the predicted segmentation y \ ( i,j ) conditioned on the prediction y i,j . Practically, the class distributions of pixels p ( y i,j ) suffer imbalanced issues, where several classes in the majority group significantly dominate other classes in the minority group. Then, learning by gradient descent method, the model could potentially bias towards the class in the majority group because the produced gradients of classes in the majority group tend to prevail over the ones in the minority group. Formally, considering the two classes c a and c b of the dataset where their distributions are skewed, i.e., p ( c a ) < p ( c b ) , the gradients produced are favored towards class c b as shown in Eqn. (5).

where || · || is the magnitude of gradients, y ( c a ) and y ( c b ) are the predictions of classes c a and c b .

Learning Fairness from Ideally Fair Distribution To address this problem, we first assume that there exits an ideal distribution q ( · ) where the class distributions q ( y i,j ) are equally distributed. Under this assumption, the model learned is expected to behave fairly among classes as there is no bias toward any groups of classes. It should be noted that our assumption is used to derive our learning objective and is going to be relaxed later. In other words, the ideal data is not required at training. Then, our learning objective in Eqn. (4) could be rewritten as follows:

The fraction between the ideal distribution q ( · ) and the actual data distribution p ( · ) is the residual learning objective for the model to achieve the desired fairness goal. Let us further derive Eqn. (6) by taking the log as follows:

The detailed derivation of Eqn. (6) and Eqn. (7) will be available in the supplementary. As shown in the Eqn. (7), there are three learning objectives as follows:

The Continual Learning Objective The first term, i.e., L ( y , ˆ y ) , represents the task-specific loss which is the continual learning objective. This objective aims to address the catastrophic forgetting and background shift problems. To achieve this desired goal, we introduce a novel Prototypical Contrastive Clustering Loss that will be discussed in Sec. 3.2.

The Fairness Objective The second term, i.e., L class = log ( q ( y i,j ) p ( y i,j ) ) , maintains the fairness in the predictions produced by the model. This objective penalizes the prediction of

The predicted segmentation maps are imposed the cross-entropy loss, the Prototypical Contrastive Clustering loss ( L cluster ), the Fairness Loss from Class Distribution ( L class ), and the Conditional Structural Consistency loss ( L cons )

MajorityGroup

Cclass

Minority Group

Fairness Loss from Class

Distribution

ConditionalStructural

Consistency

Lcons

Prototypical

Contrastive

Cross-EntropyLoss

C

Clustering

Car

FeatureVector

Prototypical Vector!

Unknown

Tree

Building

classes forcing the model to behave fairly based on the class distribution. Under the ideal distribution assumption where the model is expected to perform fairly, the q ( y i,j ) will be considered as a uniform distribution U , i.e., q ( y i,j ) ∼ U ( C ) ( C is the number of classes).

The Conditional Structural Consistency Objective The third term, i.e., L cons = log ( q ( y \ ( i,j ) | y i,j ) p ( y \ ( i,j ) | y i,j ) ) , regularizes the structural consistency of the prediction. This objective acts as a metric to constrain the structure of the predicted segmentation under the ideal distribution assumption. To model this conditional structure consistency, we introduce a conditional structural constraint based on the Markovian assumption discussed in Sec. 3.4.

3.2 Prototypical Contrastive Clustering Loss for Handling Unknown Classes

Asuccessful continual learning approach should be able to model background classes without explicit supervision and confront the forgetting problem of previously learned classes when the labels of the task are provided to update the knowledge of the model [20]. A straightforward adoption of Softmax could not be enough to handle. Indeed, the unlabeled pixels will be ignored during training. Thus, it results in these unannotated pixels could be treated as negative samples. Consequently, the segmentation model tends to produce indiscriminative features for these unknown classes that limit the capability of learning new classes in future tasks or recognizing the classes learned previously.

To address this limitation, in addition to the Softmax loss, we introduce a novel Prototypical Contrastive Clustering Loss. In particular, the semantic segmentation pixel belonging to each class can be represented in latent space. Inspired by [20, 2, 21], the features representing the classes can be separated by defining it as a contrastive clustering problem [20] where features of the same class would be pulled closer while features of different classes would be pushed far away. In addition, the deep representations of unknown classes will be grouped into the same cluster of unknown classes to produce discriminative features against other classes.

Formally, for each class c ∈ C 1 ..t , it is represented by a prototypical vector p c . In addition, the additional prototypical vector p 0 represents a cluster of unknown classes. Let f t i,j be a feature representation of pixel at location ( i, j ) of the input x t . Then, the Prototypical Contrastive Clustering Loss L cluster can be defined via a distance D as follows:

where ℓ is a distance metric, ∆ is a margin between the feature vectors of different classes, and ˆ y t i,j is the label of pixel ( i, j ) . Minimizing this loss separates the classes represented in the latent space. For step t > 1 , ˆ y t i,j of an unknown-class pixel will utilize a pseudo label where its assigned label is computed based on the closest cluster. In addition, since the prototypical vectors of classes c ∈ C 1 ..t -1 have been well learned to represent for classes, these vectors p c (where c ∈ C 1 ..t -1 ) will be frozen at step t to maintain its learned knowledge of classes C 1 ..t -1 .

The set of prototypical vectors at current step t , i.e., p 0 and p c where c ∈ C t are updated gradually with respect to the growth of feature vectors. In particular, the prototypical vector p c will be

periodically updated (after every M iterations) with momentum η based on the set of features f i,j of class c . Following common practices [20, 16], to effectively support the updating step and memory efficiency, for each class c , we only maintain a set of features S c with a fixed length of L . Algorithm 1 in the supplementary illustrates an overview of computing the prototypical contrastive clustering loss while updating the class prototypes. Fig. 2 illustrates our proposed FairCL framework.

3.3 Prototypical Constrative Clustering Loss to Catastrophic Forgetting

Knowledge Distillation is a common continual learning approach [36, 3, 12, 46] where the knowledge of the previous model will be distilled into the current model. This mechanism prevents the segmentation model from diverging knowledge learned previously and avoiding the catastrophic forgetting problem. This continual learning paradigm has been widely adopted due to its efficiency in computation. In addition, this approach also does not require data rehearsal, i.e., storing the data samples of previous tasks. In this paper, we demonstrate that our Prototypical Constrative Clustering approach is a comprehensive upper limit of the Knowledge Distillation approach. In particular, the common knowledge distillation approach can be formulated as follows:

where f t and f t -1 are the features of the input x t produced by the segmentation model at step t and step t -1 , respectively; and the distance metric D measure the knowledge gap between f t and f t -1 .

Proposition 1: The Prototypical Constrative Clustering Approach is the generalized upper bound of the Knowledge Distillation Approach.

Proof: Without lack of generality, we assume that D is a metric that measures distance between features. Given a set of fixed prototypical vectors p c , we consider the following triangle inequality of the metric D as follows:

where |C| is the number of prototypical vectors. The prototypical vectors p c and the feature vectors f t -1 produced by the segmentation model at step t -1 are considered as constant features as the model in the previous model t -1 has been fixed at step t . Thus, the distance D ( p c , f t -1 ) could be considered as constant number. Then, Eqn. (12) can be further derived as follows:

Intuitively, under this upper bound in Eqn. (13), by only optimizing our prototypical contrastive clustering loss, the knowledge distillation constraint has also been implicitly imposed. Beyond the property of generalized upper bound stated in Proposition 1 , our approach offers other benefits over the knowledge distillation approach. In particular, our approach is computationally efficient, where our method only requires a single forward pass of the segmentation model. Meanwhile, the knowledge distillation demands two forward passes for both the current and previous models, which also requires additional computational memory for the previous model. Moreover, our approach provides a better representation of each class c through the prototypical vector p c . This mechanism helps to effectively maintain the knowledge of classes learned previously while allowing the model to update the new knowledge without rehearsing the old data.

3.4 Learning Conditional Structural Consistency

The conditional structural constraint plays an important role as it will ensure the consistency of the predicted segmentation map. However, modeling the conditional structural constraint log ( q ( y \ ( i,j ) | y i,j ) p ( y \ ( i,j ) | y i,j ) ) in Eqn. (7) is a quite challenging problem due to two factors, i.e., (1) the unknown ideal conditional

distribution q ( y \ ( i,j ) | y i,j ) , and the complexity of the distribution q ( y \ ( i,j ) | y i,j ) . To address the first limitation of unknown ideal distribution, let us consider the following tight bound as follows:

The inequality in Eqn. (14) always hold with respect to any form ideal distribution q ( · ) . Thus, optimizing the negative log-likelihood of log p ( y \ ( i,j ) | y i,j ) could also regularize the conditional structural constraint due to the upper bound of Eqn. (14). More importantly, the requirement of ideal data distribution during training has also been relaxed. However, up to this point, the second limitation of modeling the complex distribution q ( y \ ( i,j ) | y i,j ) has still not been solved.

To address this problem, we adopt the Markovian assumption [5, 39] to model conditional structural consistency. In particular, we propose a simple yet effective approach to impose the consistency of the segmentation map through the prediction at location ( i, j ) and predictions of its neighbor pixels. Formally, the conditional structure consistency can be formed via the Gaussian kernel as follows:

where N ( i, j ) is the set of neighbor pixels of ( i, j ) , { σ 1 , σ 2 } are the scale hyper-parameters of the Gaussian kernels. The conditional structural consistency loss defined in Eqn. (15) enhance the smoothness and maintain the consistency of the predicted segmentation map by imposing similar predictions of neighbor pixels with similar contextual colors.

4 Experiments

In this section, we first describe the datasets and metrics used in our experiments. Then, we present the ablation studies to illustrate the effectiveness of our proposed method. Finally, we compare our approach with prior CSS methods to demonstrate our SOTA performance.

4.1 Datasets and Evaluation Protocols

Datasets ADE20K [49] is a semantic segmentation dataset that consists of more than 20K scene images of 150 semantic categories. Each image has been densely annotated with pix-level objects and objects parts labels. Cityscapes [10] is a real-world autonomous driving dataset collected in European. This dataset includes 3 , 975 urban images with high-quality, dense labels of 30 semantic categories. PASCAL VOC [13] is a common dataset that consists of more than 10 K images of 20 classes.

Implementation Two segmentation network architectures are used in our experiments, i.e., (1) DeepLab-V3 [6] with the ResNet-101 backbone, and (2) SegFormer [45] with MiT-B3 backbone. Further details of our implementation will be available in our supplementary.

Evaluation Protocols: Following [12], we focus on the overlapped CSS evaluation. Our proposed method is evaluated on several settings for each dataset, i.e., ADA20K 100-50 (2 steps), ADA20K 100-10 (6 steps), ADA20K 100-5 (11 steps), Cityscapes 11-5 (3 steps), Cityscapes 11-1 (11 steps),

Table 1: Effectiveness of our approach on the ADE20K 100-50 benchmark. Two different networks, i.e., DeepLab-V2 [5] and SegFormer [45], and three different losses, i.e., Prototypical Contrastive Clustering ( L cluster ), Fairness Loss ( L class ), and Conditional Structural Consistency ( L cons ).

Cityscapes 1-1 (21 steps), Pascal VOC 15-1 (3 steps), and Pascal VOC 10-1 (11 steps). The mean Intersection over Union (mIoU) metric is used in our experiments. The mIoU is computed after the last step for the classes learned from the first step, the later continual classes, and all classes. The mIoU for the initial classes shows the robustness of the model to catastrophic forgetting, while the metric for the later classes reflects the ability to learn new classes. To measure the fairness of the model, we also report the standard deviation (STD) of IoUs over all classes.

4.2 Ablation Study

Our ablative experiments study the effectiveness of our proposed FairCL approach on the performance of the CSS model and fairness improvement on the ADE20K 100-50 benchmark (Table 1).

Effectiveness of the Network Backbone Table 1 illustrates the results of our approach using the DeepLab-V3 [5] with the Resnet101 backbone and the SegFormer [45] with a Transformer backbone, i.e. MiT-B3 [45]. As shown in our results, the performance of segmentation models using a more powerful backbone, i.e., Transformer, outperforms the models using the Resnet backbone. The capability of learning new classes has been improved notably, i.e., the mIoU of classes 101-150 in the full configuration has been improved from 19 . 86% to 25 . 46% while the model keeps robust to catastrophic forgetting, the mIoU has been increased from

Table 2: Final mIoU (%) for Continual Semantic Segmentation on Cityscapes.

41 . 96% to 43 . 56% in the classes 0-100. Additionally, fairness between classes has been promoted when the standard deviation of the IoU over classes has been reduced from 21 . 67% to 21 . 10% .

Effectiveness of the Prototypical Contrastive Clustering Loss We evaluate the impact of the Prototypical Contrastive Clustering Loss ( L cluster ) in improving the performance in the continual learning problem compared to the fine-tuning approach. As shown in Table 1, the clustering loss has significant improvements in the catastrophic forgetting robustness compared to using only the Softmax loss. In particular, the mIoU of classes 0-100 for both DeepLab-V3 and SegFormer backbones has been improved by +41 . 63% and +43 . 30% respectively that makes the overall mIoU increase by +26 . 45% and +28 . 44% , and the average mIoU between classes increases by +13 . 17% and +14 . 03% . Although the STD of IoUs has slightly increased in this setting, the major target of our L cluster is used to model the catastrophic forgetting and background shift problems in CSS illustrated by the significant performance improvement of mIoU.

Effectiveness of the Fairness Treatment Loss As reported in Table 1, the fairness treatment from the class distribution loss L class significantly improves the overall performance and the accuracy of classes. In detail, the STD of IoU from classes has been reduced by 0 . 96% and 0 . 46% for both backbones while the mIoU has been improved from 32 . 97% to 34 . 40% and from 36 . 18% to 36 . 78% , respectively. The results have shown that our approach has promoted the fairness of the model.

Effectiveness of the Conditional Structural Consistency The full configuration in Table 1 shows experimental results of our model using conditional structure constraint loss L cons . As illustrated in our results, the conditional structure constraint demonstrates effective improvement. Indeed, it

promotes the accuracy of the initial classes and the novel classes when the mIoU has been increased from 43 . 35% to 43 . 56% and from 23 . 50% to 25 . 46% respectively with the Transformer backbone. The fairness of classes is also improved as the standard deviation of the IoU of classes 0-100 and classes 101-150 is reduced from 19 . 03% to 18 . 71% and from 20 . 75% to 19 . 99% .

4.3 Comparison with State-of-the-Art Methods

Cityscapes As shown in Table 2, our FairCL outperforms previous SOTA methods evaluated on Cityscapes benchmarks. In particular, in the 11-5 task, our method using Resnet and Transformer achieves the mIoU of 66 . 96% and 67 . 85% respectively which shows better performance than prior methods. Meanwhile, the results for the 11-1 task are 66 . 61% and 67 . 09% w.r.t. the Resnet and Transformer backbones. For the 1-1 task, the mIoU of our method is 49 . 22% and 55 . 68% .

ADE20K Table 3 presents our experimental results using ResNet and Transformer backbones compared to prior SOTA approaches. Our proposed approach achieves SOTA performance and outperforms prior methods. In particular, our approach achieves the final mIoU of 36 . 99% for Resnet and 37 . 56% for Transformer in the 100-

Step 1

Step 3

Step 4

1-100

111-120

121-130

(car)

(bicycle)

PLOP

FairCL

50 tasks. For the 50-50 tasks, the model reaches the final mIoU of 34 . 55% and 35 . 15% for the Resnet and Transformer backbones, respectively while the result of the prior method [15] is 33 . 50% . Meanwhile, the overall results of our method for the 100-10 task are 34 . 65% and 35 . 49% which shows outperforming previous approaches. Fig. 3 visualizes the qualitative result of our method compared to PLOP [12]. Initially, the ground truth contains the class 'car' in the first step and the class 'minibike' in the third step. Then, in the fourth step, the class 'bicycle' is included. As a result, PLOP [12] partly forgets the 'minibike' information when learning the class 'bicycle' information. Meanwhile, our method consistently maintains the information of 'minibike' and predicts segmentation correctly.

Pascal VOC As shown in Table 4, the proposed method outperforms the prior approaches evaluated on the Pascal VOC 2012 dataset. In detail, our method achieves the overall mIoU of 61 . 5% in the 15-1 task while the result of the previous method [46] is 59 . 4% . Meanwhile, the mIoU in the 10-1 task is 36 . 6% which shows better performance than the prior methods.

5 Conclusions and Dicussions

Table 4: The mIoU (%) of CSS on Pascal VOC. Conclusions: This paper introduces a novel fairness continual learning approach in semantic segmentation by analyzing the effect of class distribution. Furthermore, a new learning paradigm of continual learning, i.e., the prototypical Contrastive clustering loss, is proposed to sufficiently address the catastrophic forgetting and the background shift problems. The experimental results on the three CSS benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC, have shown our SOTA performance.

Limitations: Our paper has chosen specific configurations of network backbones and hyperparameters to support our hypothesis. However, the other aspects of learning have not been fully investigated, e.g., learning hyper-parameters, the selected neighbor pixels, or other forms of L cons .

Broader Impacts: This work studies the problem of Fairness in Continual Learning which is a step toward fairness awareness in continual semantic segmentation. Our contributions emphasize the importance of fairness in continual semantic segmentation learning and provide a solution to address the fairness concern that increases the robustness and trustworthiness of the segmentation model.

Acknowledgment This work is partly supported by NSF Data Science, Data Analytics that are Robust and Trusted (DART), and Googler Initiated Research Grant. We also thank Utsav Prabhu for invaluable discussions and suggestions and acknowledge the Arkansas High-Performance Computing Center for providing GPUs.

References

N. Araslanov, , and S. Roth. Self-supervised augmentation consistency for adapting semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

J. Cen, P. Yun, J. Cai, M. Y . Wang, and M. Liu. Deep metric learning for open world semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15333-15342, 2021.

F. Cermelli, M. Mancini, S. Rota Bulò, E. Ricci, and B. Caputo. Modeling the background for incremental learning in semantic segmentation. In Proc. Conf. Comp. Vision Pattern Rec. , 2020.

S. Cha, b. kim, Y. Yoo, and T. Moon. Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 10919-10930. Curran Associates, Inc., 2021.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI , 2018.

L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV , 2018.

R. Chen, Y. Rong, S. Guo, J. Han, F. Sun, T. Xu, and W. Huang. Smoothing matters: Momentum transformer for domain adaptive semantic segmentation. CoRR , 2022.

S. Chu, D. Kim, and B. Han. Learning debiased and disentangled representations for semantic segmentation. Advances in Neural Information Processing Systems , 34:8355-8366, 2021.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR , 2016.

Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

A. Douillard, Y. Chen, A. Dapogny, and M. Cord. Plop: Learning without forgetting for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4040-4050, 2021.

M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV , 2015.

R. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences , 1999.

D. Goswami, R. Schuster, J. van de Weijer, and D. Stricker. Attribution-aware weight transfer: A warmstart initialization for class-incremental semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3195-3204, 2023.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 831-839, 2019.

L. Hoyer, D. Dai, and L. Van Gool. DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In CVPR , 2022.

T.-I. Hsieh, E. Robb, H.-T. Chen, and J.-B. Huang. Droploss for long-tail instance segmentation. In Proceedings of the Workshop on Artificial Intelligence Safety 2021 co-located with the Thirty-Fifth AAAI Conference on Artificial Intelligence , 2021.

K. Joseph, S. Khan, F. S. Khan, and V. N. Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5830-5840, 2021.

J. Li and Q. Dong. Open-set semantic segmentation for point clouds via adversarial prototype framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9425-9434, 2023.

Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.

Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in an open world, 2019.

U. Michieli and P. Zanuttigh. Incremental learning techniques for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision workshops , pages 0-0, 2019.

U. Michieli and P. Zanuttigh. Incremental learning techniques for semantic segmentation. In ICCVWS , 2019.

U. Michieli and P. Zanuttigh. Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In Proc. Conf. Comp. Vision Pattern Rec. , 2021.

F. Ozdemir, P. Fuernstahl, and O. Goksel. Learn the new, keep the old: Extending pretrained models with new anatomy and images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11 , pages 361-369. Springer, 2018.

F. Ozdemir and O. Goksel. Extending pretrained segmentation networks with additional anatomical structures. International journal of computer assisted radiology and surgery , 14:1187-1195, 2019.

M. H. Phan, T.-A. Ta, S. L. Phung, L. Tran-Thanh, and A. Bouzerdoum. Class similarity weighted knowledge distillation for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16866-16875, June 2022.

Y. Qiu, Y. Shen, Z. Sun, Y. Zheng, X. Chang, W. Zheng, and R. Wang. Sats: Self-attention transfer for continual semantic segmentation. Pattern Recognition , 138:109383, 2023.

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proc. Conf. Comp. Vision Pattern Rec. , 2017.

J. Ren, C. Yu, S. Sheng, X. Ma, H. Zhao, S. Yi, and H. Li. Balanced meta-softmax for long-tailed visual recognition, 2020.

A. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science , 1995.

M. Rostami. Lifelong domain adaptation via consolidated internal distribution. In NeurIPS , 2021.

A. Saporta, A. Douillard, T.-H. Vu, P. Pérez, and M. Cord. Multi-head distillation for continual unsupervised domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshop , 2022.

K. Shmelkov, C. Schmid, and K. Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE international conference on computer vision , pages 3400-3409, 2017.

A. Szabó, H. Jamali-Rad, and S.-D. Mannava. Tilted cross-entropy (tce): Promoting fairness in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2305-2310, 2021.

S. Thrun. Lifelong learning algorithms. In Springer Learning to Learn , 1998.

T.-D. Truong, C. N. Duong, N. Le, S. L. Phung, C. Rainwater, and K. Luu. Bimal: Bijective maximum likelihood approach to domain adaptation in semantic scene segmentation. In International Conference on Computer Vision , 2021.

T.-D. Truong, N. Le, B. Raj, J. Cothren, and K. Luu. Fredom: Fairness domain adaptation approach to semantic scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

R. Volpi, D. Larlus, and G. Rogez. Continual adaptation of visual representations via domain randomization and meta-learning. In Proc. Conf. Comp. Vision Pattern Rec. , 2021.

T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR , 2019.

T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Dada: Depth-aware domain adaptation in semantic segmentation. In ICCV , 2019.

J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2021.

E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS , 2021.

C.-B. Zhang, J.-W. Xiao, X. Liu, Y.-C. Chen, and M.-M. Cheng. Representation compensation networks for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7053-7064, 2022.

P. Zhang, B. Zhang, T. Zhang, D. Chen, Y. Wang, and F. Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. arXiv preprint arXiv:2101.10979 , 2021.

B. Zhao, X. Xiao, G. Gan, B. Zhang, and S.-T. Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13208-13217, 2020.

B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In CVPR , pages 633-641, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a fairness continual learning approach in semantic segmentation. Derived from the therotical analysis of the effect of class distribution, a new learning paradigm that is consisted of three main objectives: the task-specific objective to handle the catastrophic forgetting problem, the fairness objective that maintains the fairness predictions over classes, and the conditional structural constraint that imposes the consistency of the segmentation predictions. New SOTA results are attained on the standard benchmarks.\n\nSTRENGTHS:\n1. Deriving the learning objective through analysis of the effect of class distribution is novel. \n2. Strong performances are  achieved on several standard benchmarks.\n\nWEAKNESSES:\n- The title is misleading. In my understanding, the open-world environments means lacking of labels. While the proposed method can hardly work in unsupervised setting.\n- Some explanation through mathmatical derivation is not rigorous such as eq. 5 and eq. 12 and 13. More explanation either in the main paper or supplimentary material needed. Detail illustration can be found in the questions.\n- Some details are not clear. How the fairness objective $L_{class}$ is calculated during training is not clear and lacking a final integrated training objective. Whether there is hyperparameters between losses is not mentioned. \n- According to the ablation study, fairness improvement brought by the fairness objective is minor if measured in STD of IoU over all classes. Better measurement of the fairness is needed. For example, variation of the performance on rare classes and dominant classes with and without fairness loss can be compared in a longer learning steps.\n\nQUESTIONS:\n1. Eq. 5 is hard to understand and I think it could not be intuitively achieved.\n2. The explanation that the prototypical contrastive clustering loss is an upper bound of the knowledge distillation approach is not rigorous. In eq. (12) , the $D(p_c, f^{t-1})$ cannot directly ignored as a constant number. When $c$ is a new class from $C^t$, the $p_c$ is changing during learning and $D(p_c, f^{t-1})$ will change. Therefore, $O(D(f^t, p_c)$ is not a tight boundary. Minimizing $L_{cluster}$ is not strictly equals to the minimizing of $L_distill$. From another perspective, although prototypical constrative clustering loss don't need two forward pass of the segmentation model, it have to store the prototypical vector $p_c$ using additional memory. How to calculate the $p_c$ is also critical in achieving good performance.\n3. What is the final integrated training loss? Is there any balance weights between different losses?\n4. What is the result in the first line of DeepLab-V2 and SegFormer in Table 1?\n5. Im my understanding, the conditional structural consistency mitigate the prediction gap among neighbour pixels. However, it will also smooth the edge pixels of an instance. According to the ablation study, it also improves the fairness of classes. Could the author give more explanation on it?\n6. Minor problems: when writting the related works, it better to use the surname other than the name. For example, Ziwei et al. should be Liu et al in line 93. In eq. 5, the $y_{i,h}$ should be $y_{i,j}$ in the left term?\n\nLIMITATIONS:\nAuthors have pointed out the limitations of the proposed method and some possible improvements.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Fairness Continual Learning approach to address the continual semantic segmentation problem. Specifically, the fairness continual learning framework is proposed for fairness objective, while a Prototypical Contrastive Clustering loss for reducing catastrophic forgetting. When compared to prior work on continual semantic segmentation, FairCL is shown to improve Final mIoU significantly.\n\nSTRENGTHS:\n1. This paper analyzes several problems that need to be solved in continuous learning in detail, and clearly presents the loss function corresponding to the three objectives from the perspective of formula derivation. \n2. Extensive ablation studies are performed which provides insight into the challenges of continual semantic segmentation problem.\n\nWEAKNESSES:\n1. The organization of method is not very clear. It's difficult to follow what the final loss function is and how each part of loss is combined. \n2. About the Prototypical Contrastive Clustering Loss, I wonder whether pulling all features of unknown classes into one cluster is reasonable. Will learning similar features from categories with significant differences affect subsequent model training? \n3. How is the margin $\\triangle$ selected? \n4. In Sec.3.2, 'a pseudo label where its assigned label is computed based on the closest cluster'. How is the pseudo label computed and utilized? \n5. The compare in experiments about network backbone is unfair.  The experiment provides results on two different backbones, but neither backbone was aligned with the previous method such as Deeplab-v3 architecture with ResNet-101 in RCIL.\n\nQUESTIONS:\nI am looking forward to the authors clarifying the concerns listed in the section above: \n1. The final loss function and the combination form of each part; \n2. More detail explanation of the Prototypical Contrastive Clustering Loss; \n3. Comparison in experiment results.\n\nLIMITATIONS:\nBoth limitations and potential societal impacts are addressed in the paper. However, it might help to expand the paragraph to discuss some of the concerns listed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the continual semantic segmentation problem and highlights the lack of fairness in existing methods. It proposes a novel fairness objective based on class distributions to address the bias on the majority group classes. The paper proposes a novel prototype contrastive clustering loss to address the significant challenges of catastrophic forgetting as well as for handling unknown classes to solve the background shift problem. Furthermore, a conditional structural consistency loss is proposed to constrain the structure of the predicted segmentation. The method has achieved improvements over the existing benchmarks on ADE20K, Cityscapes and Pascal VOC datasets.\n\nSTRENGTHS:\n1. The paper is well written and the motivation is good. The problem formulation is nicely done.\n\n2. The paper points out the issue of maintaining fairness in continual segmentation systems. It is a very significant contribution to the field since all existing methods primarily focussed on reducing forgetting and addressing the background shift.  The fairness problem is well-defined in the paper along with the fairness metric which is used to obtain the error-rate among the classes.\n\n3. The novel prototype contrastive clustering loss represents the semantic segmentation pixels for different classes in the latent space and groups the features of unknown classes into the same cluster. In addition to handling unlabelled classes, it is interesting to see that the clustering loss also serves as an upper bound of the knowledge distillation approach and thus reduces the forgetting of old classes.\n\n4. The proposed method FairCL seems to be quite effective on several settings across multiple datasets with different backbones.\n\nWEAKNESSES:\n1. Although the fairness loss is improving the mIoU values for different settings on ADE20K, it is not clear if the improvement in mIoU is coming from the minority classes. While the main motivation of the paper is to ensure fairness among the classes, the ablation experiment does not completely justify this. The individual contribution of the fairness loss (L_class) should be stated since Fairness in Continual Learning is the main claim of the authors. Further analysis of the improvements in mIoU values separately for majority and minority classes on at least two datasets will be more convincing. Also, the fairness loss may be evaluated in a non-incremental setting (let's say for the first step only), and then the minority class mIoU values should be better; otherwise, the performance is also affected by forgetting, and the contribution of the fairness loss is not clear.\n\n2. The motivation for learning conditional structural consistency loss is not clear and should be discussed to give more intuition. Also, some qualitative analysis of prediction consistency will be good to show the effectiveness of the structural consistency loss.\n\n3. Results from Pascal VOC are not convincing. For the 10-1 setting, the forgetting is quite high compared to methods like RCIL. Overall, the improvement looks very insignificant. Experiments on more settings of Pascal VOC similar to previous papers like PLOP and RCIL will be good. Also, the comparison of the SegFormer backbone with the other existing methods using DeepLabV2 is not fair. If there is any specific reason for the method not being so effective for Pascal VOC (as it looks effective for ADE20K and Cityscapes), it should be analyzed.\n\nQUESTIONS:\n1. Inconsistent results: The results for the ADE20K 100-50 setting are different in table 1 and table 3. In table 1 for ablation, the overall mIoU is 34.64 for DeepLab-V2 backbone, while in table 3, the overall mIoU is 36.99. The mIoU values are the same for the SegFormer backbone though. This should be clarified since the ablation experiment is claiming the contributions of the different losses.\n\n2. While it is good to see the results with both DeepLab-V2 and SegFormer backbones, the backbone networks of the compared methods should be mentioned for more clarity and fair comparison.\n\n3. The comparison does not include a well performing baseline method - SSUL (from NeurIPS 21). If there is any specific reason for not comparing with SSUL, it should be discussed in the paper.\n\n4. See weaknesses 1, 2, 3.\n\nLIMITATIONS:\n/", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the semantic segmentation problem in the context of continual learning. In the past, the emphasis was mainly on addressing catastrophic forgetting and background shift issues. However, this paper goes beyond that and investigates the balance problem between different object categories. It introduces novel approaches for prototype contrastive learning and structure-consistency loss functions, which result in a new SOTA performance. The contributions of the paper can be summarized into three main parts:\n\ni. It introduces a novel evaluation metric to measure the model's balance across different classes.\n\nii. It proposes a new contrastive learning loss to address the issue of forgetting in continual learning.\n\niii. The paper conducts extensive experiments to demonstrate the effectiveness and rationality of the proposed methods.\n\nSTRENGTHS:\n+ The paper considers multiple perspectives to improve fairness among classes of the model.\n+ The paper designs various loss functions to enhance the segmentation performance of the model under continual learning.\n+ The supplementary materials of the paper include an excellent presentation that makes it easier for readers to understand the superiority of the model.\n\nWEAKNESSES:\n+ The paper lacks experiments specifically focused on the upper bound design for continual learning.\n+ The proposed inter-class fairness in the paper seems to be addressing the semantic segmentation problem rather than the main focus of continuous learning. However, this issue has already been well discussed in [1].\n[1] Truong, T. D., Le, N., Raj, B., Cothren, J., & Luu, K. (2023). Fredom: Fairness domain adaptation approach to semantic scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19988-19997).\n\nQUESTIONS:\nAs far as I know, the current research on semantic segmentation is primarily conducted using DeepLab v3 as the framework. Why did the paper choose to experiment with DeepLab v2 instead?\n\nLIMITATIONS:\nThe paper discusses its own limitations in the experimental section. The paper does not have any negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}