Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments
Authors: Thanh-Dat Truong
Hoang-Quan Nguyen
Bhiksha Raj
Khoa Luu
Venue: NeurIPS 2023 poster (2023)
Keywords: Fairness Continual Learning; Semantic Segmentation; Contrastive Clustering;

Abstract: Continual semantic segmentation aims to learn new classes while maintaining the information from the previous classes. Although prior studies have shown impressive progress in recent years, the fairness concern in the continual semantic segmentation needs to be better addressed. Meanwhile, fairness is one of the most vital factors in deploying the deep learning model, especially in human-related or safety applications. In this paper,  we present a novel Fairness Continual Learning approach to the semantic segmentation problem.
In particular, under the fairness objective, a new fairness continual learning framework is proposed based on class distributions.
Then, a novel Prototypical Contrastive Clustering loss is proposed to address the significant challenges in continual learning, i.e., catastrophic forgetting and background shift. Our proposed loss has also been proven as a novel, generalized learning paradigm of knowledge distillation commonly used in continual learning. Moreover, the proposed Conditional Structural Consistency loss further regularized the structural constraint of the predicted segmentation. Our proposed approach has achieved State-of-the-Art performance on three standard scene understanding benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC, and promoted the fairness of the segmentation model.

Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments
Thanh-Dat Truong', Hoang-Quan Nguyen', Bhiksha Raja.s, and Khoa Luu
CVIU Lab, University of Arkansas, Fayetteville, AR, 72701
Carnegie Mellon University, Pittsburgh, PA, 15213 Mohammed bin Zayed University of AI, Abu Dhabi, UAF {tt032, hn016, khoaluu)Quark. edu, bhiksha&cs. cmu. edu
Abstract
Continual semantic segmentation aims to learn new classes while maintaining the information from the previous classes. Although prior studies have shown impressive progress in recent years, the fairness concern in the continual semantic segmentation needs to be better addressed. Meanwhile, fairness is one of the most vital factors in deploying the deep learning model, especially in human-related safety applications. In this paper, we present a novel Fairness Continual Learning approach to the semantic segmentation problem. In particular, under the fairness objective, a new faimnsss continual learning framework is proposed based on class distributions. Then, novel Prototypical Contrastive Clustering loss is proposed address the significant challenges in continual learning, i.e., catastrophic forgetting and background shift. Our proposed loss has also been proven novel, gener- alized learning paradigm of knowledge distillation commonly used in continual learning. Moreover, the proposed Conditional Structural Consistency loss further regularized the structural constraint of the predicted segmentation. Our proposed approach has achieved State-of ttth-hrrrt performance on three standard scene under- standing benchmarks, i.c., ADE20K, Cityscapes, and Pascal VOC, and promoted the fairness of the segmentation model.
Introductions
Convolutional Neural Networks (CNNs) [5, 7] and Transformers [45, 8] have been introduced to approach semantic segmentation tasks where the models learn from the large-scale data hav- ing known classes at once. These segmentation models learned on large-scale data may perform poorly as they may counter the new objects or new environments. To bridge this gap, several ap- proaches have been proposed to adapt the model to the new data. Domain Adaptation is one of the common approaches [39, 42, 43, 18] that adap- tively transfer the segmentation model into the deployed environment. However, domain adapta- tion cannot handle when new objects appear due to its close-set setup. Also, this approach requires access to both original and new training data. In practice, the segmentation models should be ca- pable of learning classes continually without re-training from the previous data. This paradigm defined as Continual Semantic Segmentation
Figure 1: The Class Distribution of ADE20K. In the ADE20K 50-50 (3 steps) benchmark, the distribution of classes in the majority group in the early step dominates the ones in the minority groups in the later steps. The distributions of classes gradually decrease over steps.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

[12, 46). The current continual semantic segmentation approaches [12, 46] concentrate on addressing two main challenges, i.e., (1) catastrophic forgetting [33, 14. 38] and (2) background shift [12, 46]. The former problem indicates the forgetting issue of the model about the previous knowledge when learning on new training data [12, 46, 4]. Meanwhile, the latter problem refers to the classes of previous or future data that have collapsed into a background class [3]. However, another critical problem that needs attention is the fairness issue in semantic segmentation.
Fairness in semantic segmentation refers to the problem of the model where it behaves unfairly between classes the dataset due to the class distribution, i.e. the model tends to produce prediction bias toward specific group of classes occupying large region on an image or frequently existing in the dataset (as shown Fig. 1). The unfair predictions could result in severe problems, especially in human-related applications that could influence human safety. Moreover, the faimess problem could even be well observed in the context of continual learning when the model encounters new classes without accessing previous training data. The prior work of continual learning in image classification [48] has also considered this problem. Several prior studies [1, 18, 19] in semantic segmentation have tried to reduce the effect of the class imbalance by introducing the weighted cross entropy [11, 44, 19]. focal loss [1], over-sampling techniques [18, 47, 1J. However, the fairness problem in continual semantic segmentation has yet to be well-defined and directly addressed. There should be more attention on addressing the fairness issue in continual semantic segmentation. Therefore, this work aims to address the fairness problem in continual semantic segmentation caused by the imbalanced class distribution defined based on the number of pixels of each class in the dataset.
Contributions of this Work: This work presents a novel Fairness Continual Learning (FairCL) approach to semantic scene segmentation. Our contributions can be summarized as follows. First, under the perspective of fairness learning, the new metric is formulated to measure the fairness of the model via the error rate among classes. Then, the metric is further derived into the three main objectives, i.e., (1) the Task-specific Objective that handles the catastrophic forgetting problem, (2) the Fairness Objective that maintains the fairness of predictions produced by the model based on the class distribution, and (3) the Conditional Structural Constrain! that imposes the consistency of the segmentation predictions. Second, to sufficiently model the continual learning problem, the novel Prototypical Contrastive Clustering loss is presented to address the catastrophic forgetting and the background shifting problems. Moreover, the proposed Prototypical Contrastive Clustering loss has been proven to be a generalized paradigm of knowledge distillation approaches commonly adopted in continual learning. Last, sufficient ablation studies have shown the effectiveness of our method in continual learning and promoted the fairness of the model. The empirical comparison with prior methods has shown the State-of-the-Ar (SOTA) results on the standard benchmarks.
2 Related Work
Continual Semantic Segmentation While there is enormous progress in the continual learning problem, most approaches focus on image classification. Existing continual learning methods [22, 17] have been extended for continual semantic segmentation in medical images [27, 28] and general datasets [24]. Cermelli et al. [3] addressed background shifting problem in the continual semantic segmentation. Douillard et al. [12] introduced multi-scale spatial distillation scheme that preserves long- and short-range spatial relationships at the feature level. Volpi et al. [41] presented a continual learning framework where the model is sequentially trained on multiple labeled data domains. Rostami et al. [34] proposed continual learning framework under the unsupervised domain adaptation setting using data rehearsal. Saporta et al. [35] introduced a multi-head knowledge distillation framework for the continual unsupervised domain. Zhang et al. [46] presented a representation compensation module to decouple the representation learning of both old and new knowledge. Cha et al. [4] suggested finding the unknown class from the background to distinguish the representations of the potential future classes. Qiu et al. [30] proposed a self attentonn transferring method to capture both within-class and between-class knowledge. Phan et al. [29] introduced a class-similarity knowledge- distillation method to revise the old classes more likely to be forgotten and better learn new classes related to the previous classes
Class Imbalance and Fairness Approaches in Semantic Segmentation Jiawei et al. [32] presented a balanced Softmax loss that reduces the distribution shift of labels and alleviates the long-tail issue. Wang et al. [44] proposed a Seesaw loss that reweights the contributions of gradients produced by positive and negative instances of a class by using two regularizers, i.e., mitigation and compensation. Liu et al. [23] proposed algorithm that handles imbalanced classification, shot learning, and

---

open-set recognition using dynamic meta-embedding. Chu et al. [9] proposed a stochastic training scheme for semantic segmentation. which improves the learning of debiased and disentangled representations. Szabo et al. [37] proposed tilted cross-entropy loss to reduce the performance differences, which promotes fairness among the target classes. Truong et al. [40] introduced a fairness domain adaptation approach to semantic segmentation that maintains the fairness of the predictions produced by the segmentation model on the target domain.
3 The Proposed Fairness Continual Learning Approach
Let F parameterized by 0 be the deep semantic segmentation model that maps an input image x E X to the segmentation map y E y. y F(x, 0). Continual Semantic Segmentation (CSS) aims to learn a model in T steps. In each step, the segmentation model F encounters dataset DD = {x' "y} where x" is the input image and ye is the ground-truth segmentation time E [1.77]. The current ground-truth segmentation map 5 only contains the labels of the current classes C' and all the class labels of prevision steps, C2..t-1, or the future steps, Ct+1..I' are collapsed into a background class or ignored. Formally, learning the semantic segmentation at time step / can be formulated as follows:
6, 0 arg min E.,, s'Ep L (F(x'00)). 0.).
(1)
where 02 is the parameters of F at time step t. C is the objective learning of the continual learning task. In CSS learning, at the current time step i, the segmentation model F is expected to not only predict all the classes C1..t-1 learned in the previous steps but also predict the current new classes Co. Three significant challenges have been identified in this learning setting and should be addressed.
Background Shift At time step t, the labels of previous and future steps have been ignored. Thus, the pixels of these classes are ambiguous, which means these could contain either the class of previous future steps. During learning C', the model could consider these classes as negative samples. As a result, the model tends to learn non non-discriminative features for these pixels, leading difficulty learning new classes forgetting the old ones.
Catastrophic Forgetting cause the model may partially or completely forget the knowledge of classes Cl. when learning the new classes C". This problem could be caused by the background shift and the learning mechanism. Since classes in Cl.. are considered as the background class at time step f, the model tends to update the knowledge of the new classes while the predictions of classes incline to be suppressed.
Fairness While the prior approaches [12, 46, 4] focus on addressing the two above chal- lenges, the fairness issue has received less attention and has not been well addressed yet. However, fairness is one of the most important criteria as it guarantees the model behaves fairly among not only classes in C but also classes in CI..3 that have been learned. The fairness in CSS is typically caused by the imbalance distribution between classes as several classes occupy the larger portion exist more frequently than other classes (Fig. 1). In addition, the appearance of training samples at the training step / also exaggerates the bias due to the new classes since the classes in the previous training steps have collapsed.
To address these challenges in CSS, we first reconsider the semantic segmentation problem from the fairness viewpoint followed by introducing novel approach to alleviate the fairness issue based on the ideal class distribution. Then, we introduce a novel Prototypical Contrastive Clustering loss to model background classes and catastrophic forgetting.
3.1 Fairness Learning Approach to Continual Semantic Segmentation
Maintaining fairness one of the most important factors in learning CSS as it reduces the bias of the model toward a certain group of classes. Formally, under the fairness constraint in semantic segmentation, the error rate of each class should be equally treated by the segmentation model. Given classes ca and c in the set of classes Cl..T" this constraint can be formulated as follows:
max ECL..T PXSED E ca) Fxx. Ec = ca) Se
(2)
Go Cny CEL..
where Vi.j and ji.j is the prediction and ground truth the pixel location (i, j), respectively; the loss function L measures the ervor rates of predictions. Intuitively, Eqn. (2) aims to maintain the
3

---

differences in the enror rates between classes lower than a small threshold to guarantee faimess among classes. For simplicity, this constraint can be considered as an additional loss while optimizing Eqn. (1), However, this approach could not guarantee the model being fair due to Eqn. (3).
max FarepEct...
EXISED DEDDD Σ
c (36.s. Ri.s = c.) - Ex.9 Σ c (Vi.s Di.3 = a
1.3
(3)
ExSED
Σ Bl ca) Lxg Σ (v..) co) s2 [F
As shown in the above inequality, the constraint of Eqn. (2) has been bounded by the loss function in Eqn. (1), Although minimizing Eqn. (1) could also impose the constraint in Eqn. (2), the fairness issue still remains unsolved due to the imbalanced class distributions indicated through Eqn. (4).
0- arg argminn | C(x, (1plo))(5))ddd arg min
i Σ Clui,
(4)
where p(-) is the data distribution, Y((i.) is the predicted segmentation map without the pixel at position (r.j), p(y*) is the class distribution of pixels, and ((.5) (((.s) ) is the conditional structural constraint of the predicted segmentation Yyi)) conditioned on the prediction Yr.j Practically, the class distributions of pixels p(y.,) suffer imbalanced issues, where several classes in the majority group significantly dominate other classes in the minority group. Then, learning by gradient descent method, the model could potentially bias towards the class in the majority group because the produced gradients of classes in the majority group tend to prevail over the ones in the minority group. Formally, considering the two classes Ca and cs of the dataset where their distributions are skewed, i.e. p(Ca) < p(c)), the gradients produced are favored towards class O as shown in Eqn. (5).
o Ex, Clvin di doun ca)pl (a))l(yr (akk))(()))d))
(y(ca)
8 BS Shassna = (0. ddyd9
8y
where| || || is the magnitude of gradients, V(c.) and ((a) are the predictions of classes ca and c-- Learning Fairness from Ideally Fair Distribution To address this problem, we first assume that there exits ideal distribution q(-) where the class distributions ((M..)) are equally distributed. Under this assumption, the model learned is expected to behave fairly among classes as there is no bias toward any groups of classes. should be noted that our assumption is used to derive our learning objective and is going to be relaxed later. In other words, the ideal data is not required at training. Then, our learning objective in Eqn. (4) could be rewritten as follows:
0' = arg min Ex-ptr))s-p99)
Σ C(un..0..) 2.3
(6)
The fraction between the ideal distribution g(-) and the actual data distribution p(-) is the residual learning objective for the model to achieve the desired fairness goal. Let us further derive Eqn. (6) by taking the log as follows:
1 Σ log (<(Us.) + log (p(D..))
}}
8" = arg min sx
c(y,8)+
+
N
(py (t.nll))))))))))))
(7)
The detailed derivation of Eqn. (6) and Eqn. (7) will be available in the supplementary. As shown in the Eqn. (7), there are three learning objectives as follows:
The Continual Learning Objective The first term, i.e., C(y, represents the task-specific loss which is the continual learning objective. This objective aims address the catastrophic forgetting and background shift problems. To achieve this desired goal, we introduce a novel Prototypical Contrastive Clustering Loss that will be discussed in Sec. 3.2.
The Fairness Objective The second term, i.e., Leiess log (((t.))
maintains the fairness in the predictions produced by the model. This objective penalizes the prediction of
((vL..)

---

Erllan Falrery lam One Plankation
Figure 2: The Proposed Fairness Continual Learning Framework. The predicted segmentation maps are imposed the cross-entropy loss, the Prototypical Contrastive Cluster- ing loss (Leluster), the Fairness Loss from Class Distribution (Lelass), and the Conditional Structural Con- sistency loss (Ccons)
Conditoned Stratand
Ciuabderty Coens
Prodetypical
extropy
Guustniigg
cahnt
Expaster
PHRMaysRo woctor
classes forcing the model to behave fairly based on the class distribution. Under the ideal distribution assumption where the model expected to perform fairly, the ((y..) will be considered as a uniform distribution U, i..., ((v.,)) U(C) (C is the number of classes).
The Conditional Structural Consistency Objective The third term, i.e., Lcons
log 90066,9))
(( v..z) regularizes the structural consistency of the prediction. This objec-
tive acts metric to constrain the structure of the predicted segmentation under the ideal distribution assumption. To model this conditional structure consistency, we introduce conditional structural constraint based the Markovian assumption discussed in Sec. 3.4.
3.2 Prototypical Contrastive Clustering Loss for Handling Unknown Classes
A successful continual learning approach should be able to model background classes without explicit supervision and confront the forgetting problem of previously learned classes when the labels of the task are provided to update the knowledge of the model [20]. A straightforward adoption of Softmax could not be enough to handle. Indeed, the unlabeled pixels will be ignored during training. Thus, it results in these unannotated pixels could be treated as negative samples. Consequently, the segmentation model tends to produce indiscriminative features for these unknown classes that limit the capability of learning new classes in future tasks recognizing the classes learned previously.
To address this limitation, addition the Softmax loss, we introduce a novel Prototypical Con- trastive Clustering Loss. In particular, the semantic segmentation pixel belonging to each class can be represented latent space. Inspired by [20, 2, 21], the features representing the classes can be separated by defining it as contrastive clustering problem [20] where features of the same class would be pulled closer while features of different classes would be pushed far away. In addition, the deep representations of unknown classes will be grouped into the same cluster of unknown classes to produce discriminative features against other classes.
Formally, for each class E Cl..,, it is represented by prototypical vector Pe- In addition, the additional prototypical vector Po represents cluster of unknown classes. Let fig be feature representation pixel at location (i.j) of the input x'. Then, the Prototypical Contrastive Clustering Loss Coluster can be defined via distance D as follows:
Letuster (F,0,)) -> D(C())P))
(8)
is
D(F Pa) = { [E(EE Po)
If of =
[max{(, ((f)
otherwise
(9)
where f is distance metric, A is margin between the feature vectors of different classes, and oi.y is the label of pixel (i, Minimizing this loss separates the classes represented in the latent space. For step > 1, Vi., of an unknown-class pixel will utilize a pseudo label where its assigned label is computed based on the closest cluster. In addition, since the prototypical vectors of classes c E Cl..t-1 have been well learned to represent for classes, these vectors P. (where c Cl..t-l) will be frozen step / maintain its leamned knowledge of classes C1..1-1
The set of prototypical vectors at current step f, i.e., Po and P. where e C are updated gradually with respect to the growth of feature vectors. In particular, the prototypical vector Pe will be
5

---

periodically updated (after every M iterations) with momentum based on the set of features f.,; of class c. Following common practices [20. 16]. to effectively support the updating step and memory efficiency, for each class we only maintain a set of features S, with fixed length of L.. Algorithm I in the supplementary illustrates an overview of computing the prototypical contrastive clustering loss while updating the class prototypes. Fig. 2 illustrates our proposed FairCL. framework.
3.3 Prototypical Constrative Clustering Loss to Catastrophic Forgetting
Knowledge Distillation is common continual learning approach [36, 3, 12, 46] where the knowl- edge of the previous model will be distilled into the current model. This mechanism prevents the segmentation model from diverging knowledge learned previously and avoiding the catastrophic forgetting problem. This continual learning paradigm has been widely adopted due to its efficiency in computation. In addition, this approach also does not require data rehearsal, i.e., storing the data samples of previous tasks. In this paper. we demonstrate that our Prototypical Constrative Clustering approach is a comprchensive upper limit of the Knowledge Distillation approach. In particular, the common knowledge distillation approach can be formulated as follows:
Laiststt (x', F, te, 0t-1) = D(f
(10)
where f and ft-1 the features of the input x" produced by the segmentation model at step and step t - 1, respectively; and the distance metric DD measure the knowledge gap between f' and f²
Proposition 1: The Prototypical Constrative Clustering Approach is the generalized upper bound of the Knowledge Distillation Approach.
Casatau (x", F,0,,0,-1) O (Crinatee (x', ,F,O.)
(11)
Proof: Without lack of generality, we assume that 'D is metric that measures distance between features. Given a set of fixed prototypical vectors Pa, we consider the following triangle inequality of the metric DD as follows:
E (0) UC!
r(f")"~-)) S D(f", p.) + D(p.: 1' D(f', p
1 C Σ [P('', Pa) + D(p,, P1
(12)
where (C] is the number of prototypical vectors. The prototypical vectors P. and the feature vectors ft-1 produced by the segmentation model at step t are considered as constant features as the model in the previous model 1 has been fixed at step t. Thus, the distance D(p,, f') could be considered as constant number. Then, Eqn. (12) can be further derived as follows:
D(F' p(rrr~)))=
(
1 E(D(r Po) D(pe, = =o ( D(r .p))
(ED(r'
(13)
Laistalxx' F,0,, 021)) (Caiuster (x', ,F,0,)
Intuitively, under this upper bound in Eqn. (13), by only optimizing our prototypical contrastive clustering loss, the knowledge distillation constraint has also been implicitly imposed. Beyond the property of generalized upper bound stated in Proposition 1, our approach offers other benefits over the knowledge distillation approach. particular, our approach is computationally efficient, where method only requires single forward pass of the segmentation model. Mcanwhile, the knowledge distillation demands two forward passes for both the current and previous models, which also requires additional computational memory for the previous model. Moreover, our approach provides a better representation of each class c through the prototypical vector P.- This mechanism helps to effectively maintain the knowledge of classes learned previously while allowing the model to update the new knowledge without rehearsing the old data.
3.4 Learning Conditional Structural Consistency
The conditional structural constraint plays an important role as it will ensure the consistency of the pre- dicted segmentation map. However, modeling the conditional structural constraint log (oly ((م (m.)))
in Eqn. (7) is quite challenging problem due two factors, i.c., (1) the unknown ideal conditional

---

distribution g(Ma(.)) (t,y), and the complexity of the distribution ((v)).)) BR..)) To address the first limitation of unknown ideal distribution, let us consider the following tight bound follows:
( ((0G))l p(y p(y (osslbu))) - -xxxp(x Log
Expeptx) log
(14)
The incquality in Eqn. (14) always hold with respect any form ideal distribution q()). Thus, optimizing the negative log-likelihooe of p(y) could also regularize the conditional structural constraint due to the upper bound of Eqn. (14). More importantly, the requirement of ideal data distribution during training has also been relaxed. However, up to this point, the second limitation of modeling the complex distribution ((y ((-))33)))))))) has still not been solved.
To address this problem, we adopt the Markovian assumption [5, 39] to model conditional structural consistency. In particular, we propose a simple yet effective approach to impose the consistency of the segmentation map through the prediction at location (i:j) and predictions of its neighbor pixels. Formally, the conditional structure consistency can be formed via the Gaussian kernel as follows:
|
1 20
11
V 3
- Pl & E
exp
(15)
20
where ji, is the set of neighbor pixels of (i.j), f01, are the scale hyper hyper-paramenn parameters of the Gaussian kernels. The conditional structural consistency loss defined in Eqn. (15) enhance the smoothness and maintain the consistency of the predicted segmentation map by imposing similar predictions of neighbor pixels with similar contextual colors.
Experiments
In this section, we first describe the datasets and metrics used our experiments. Then, we present the ablation studies illustrate the effectiveness of our proposed method. Finally, compare approach with prior CSS methods to demonstrate our SOTA performance.
4.1 Datasets and Evaluation Protocols
Datasets ADE20K [49] is semantic segmentation dataset that consists of more than 20K scene images of 150 semantic categories. Each image has been densely annotated with pix-level objects and objects parts labels. Cityscapes [10] is real-world autonomous driving dataset collected in European. This dataset includes 3, 975 urban images with high-quality, dense labels of 30 semantic categories. PASCAL VOC [13] common dataset that consists of more than 10K images of 20 classes.
Implementation Two segmentation network architectures are used in our experiments, i.e., (1) DeepLab- V3 [6] with the ResNet- 101 backbone, and (2) SegFormer [45] with MiT-B3 backbone. Further details of our implementation will be available in our supplementary.
Evaluation Protocols: Following [12], we focus on the overlapped CSS evaluation. Our proposed method is evaluated on several settings for each dataset, i.e., ADA20K 100-50 (2 steps), ADA20K 100-10 (6 steps), ADA20K 100-5 (11 steps), Cityscapes 1 11-5 (3 steps), Cityscapes 11-1 (11 steps),
Backbone
Leluster
Lelass
Lcone
0-100
100-150
all
avg
mIoU
STD
mIoU
STD
mIoU
STD
mloU
STD
0.08
0.84
19.52
20.18
6.52
13.14
24.41
13.14
41.71
19.90
15.33
21.96
32.97
23.03
37.58
23.03
DeepLab-V3
42.25
19.31
18.55
20.52
34.40
22.07
38.35
22.07
43.40
19.08
24.04
19.12
36.99
21.67
40.45
21.67
0.10
0.84
23.18
19.83
7.74
15.74
25.82
15.74
43,40
19.35
21.60
22.06
36.18
22.32
39.85
22.32
SegFormer
43.35
19.03
23.50
20.75
36.78
21.86
40.34
21.86
43.56
18.71
25.46
19.99
37.56
21.10
40.73
21.10
Table I: Effectiveness of our approach on the ADE20K 100-50
benchmark. Two different networks, i.e., DeepLab-Ve V2 [5] and SegFormer [45], and three different losses, i.e., Prototypical Contrastive Clustering (Caluster), Fairness Loss (Lelass), and Conditional Structural Consistency (Ccons).
7

---

Cityscapes 1-1 (21 steps). Pascal VOC 15-1 (3 steps), and Pascal VOC 10-1 (11 steps). The mean Intersection over Union (mloU) metric used in our experiments. The mloU is computed after the last step for the classes learned from the first step, the later continual classes, and all classes. The mloU for the initial classes shows the robustness of the model to catastrophic forgetting, while the metric for the later classes reflects the ability to learn new classes. To measure the faireess of the model, we also report the standard deviation (STD) of IoUs over all classes.
4.2 Ablation Study
Our ablative experiments study the effectiveness of our proposed FairCI. approach on the performance of the CSS model and fairness improvement on the ADE20K 100-50 benchmark (Table 1).
Effectiveness of the Network Backbone Ta-
ble illustrates the results of our approach us-
Method
11-5
11-1
1-1
ing the DeepLab- V3 [5] with the Resnet101
3 steps
11 steps
21 steps
backbone and the SegFormer [45] with a Trans- former backbone, i.e. MiT-B3 [45]. As shown in our results, the performance of segmentation models using a more powerful backbone, i.e., Transformer, outperforms the models using the Resnet backbone. The capability of learning new classes has been improved notably, i.c., the mIoU of classes 101-150 in the full configura- tion has been improved from 19.86% to 25.46% while the model keeps robust to catastrophic forgetting, the mIoU has been increased from 41.96% to 43.56% in the classes 0-100. Additionally, faimess between classes has been promoted when the standard deviation of the IoU over classes has been reduced from 21.67% to 21.10%
Joint
79.30
79.30
79.30
LWF-MC [31]
58.90
56.92
31.24
ILT [25]
59.14
57.75
30.11
MiB [3]
61.51
60.02
42.15
PLOP [12]
63.51
62.05
45.24
RCIL [46]
64.30
63.00
48.90
FairCI. DeepLab-V3
66.96
66.61
49.22
FairCL + SegFormer
67.85
67.09
55.68
Table 2: Final mloU (%) for Continual Semantic Segmentation on Cityscapes.
Effectiveness of the Prototypical Contrastive Clustering Loss We evaluate the impact of the Prototypical Contrastive Clustering Loss (Lesuster) in improving the performance in the continual learning problem compared to the fine-tuning approach. As shown in Table 1, the clustering loss has significant improvements in the catastrophic forgetting robustness compared to using only the Softmax loss. In particular, the mIoU of classes 0-100 for both DeepLab-V3 and SegFormer backbones has been improved by +-1.63% and +4.300% respectively that makes the overall mloU increase by +26.45% and + -2284444 18., and the average mloU between classes increases by + 13.117% and +14.03%. Although the STD of IoUs has slightly increased in this setting, the major target of our Loluster is used model the catastrophic forgetting and background shift problems in CSS illustrated by the significant performance improvement of mloU.
Effectiveness of the Fairness Treatment Loss As reported in Table 1, the fairness treatment from the class distribution loss Lotass significantly improves the overall performance and the accuracy of classes. In detail, the STD of IoU from classes has been reduced by 0.96% and 0.46% for both backbones while the mIoU has been improved from 32.97% to 34.40% and from 36. 38% to 36.78% respectively. The results have shown that our approach has promoted the faimess of the model.
Effectiveness of the Conditional Structural Consistency The full configuration in Table shows experimental results of our model using conditional structure constraint loss Coons As illustrated in our results, the conditional structure constraint demonstrates effective improvement. Indeed,
100-50 (2 steps)
50-50 (3 steps) all
100-10 (6 steps)
Method
0-100
101-150
all
avg
0-50
51-150
avg
|0-100
101-150
all
avg
Joint
44.30
28.20
38.90
511.0
32.80
38.90
44.30
28.20
38.90
ILT [25]
18.29
14.40
17.00
29.42
3.53
12.85
9.70
30.12
0.11
3.06
1.09
12.56
MiB [3]
40.52
17.17
32.79
37.31
45.57
21.01
29.31
38.98
38.21
11.12
29.24
35.12
PLOP [12]
41.87
14.89
32.94
37.39
48.83
20.99
30.40
39.42
40.48
13.61
31.59
36.64
RCIL [46]
42.30
18.80
34.50
38.48
48.30
25.00
32.50
39.30
17.60
32.10
MiB AWT [15]
40,90
24.70
35.60
46.60
26.85
33.50
39.10
21.28
33,20
SSUL [4]
41.28
18.02
33.58
48.38
20.15
29.56
40,20
18.75
33.10
SATS [30]
41.42
19.09
34.18
FairCL + DeepLaab-V
43.40
24.04
36.99
40.45
49.65
26.84
34.55
41 68
41.73 42.21
20.36
34.65
39.01
FairCL + SegFormer
43.56
25,46
37.56
40.73
49.62
27.78
35.15
42.25
21.91
35.49
39.36
Table 3: Continual Semantic Segmentation results on ADE20k in Mean IoU (%).
8

---

promotes the accuracy of the initial classes and the novel classes when the mIoU has been increased from 43.35% to 43.56% and from 23.50% 25.46% respectively with the Transformer backbone. The fairness of classes is also improved as the standard deviation of the IoU of classes 0-100 and classes 101-150 is reduced from 19.03% to 18.71% 71% and from 20.75% to 19.99%.
Step1
Step 4 121-130 111-120 (minibike)
Step 3
4.3 Comparison with State-of-the-Art Methods
1-100
Cityscapes As shown in Table 2, our FairCL out- performs previous SOTA methods evaluated on Cityscapes benchmarks. In particular, in the 11-5 task, our method using Resnet and Transformer achieves the mloU of 66.96% and 67. .8.55% respec- tively which shows better performance than prior methods. Mcanwhile, the results for the 11-1 task are 66.61% and 67.09% w.r.t. the Resnet and Transformer backbones. For the 1-1 task, the mloU of our method is 49.22% and 55.68%.
(car)
(bicycle)
PLOP
ADE20K Table 3 presents our experimental re- sults using ResNet and Transformer backbones compared to prior SOTA approaches. Our pro- posed approach achieves SOTA performance and
FairCL
outperforms prior methods. In particular, our ap-
Figure 3: Qualitative results of Our Approach
proach achieves the final mloU of 36.99% for Resnet and 37.56% for Transformer in the 100- 50 tasks. For the 50-50 tasks, the model reaches the final mloU of 34.55% and 35.15% for the Resnet and Transformer backbones, respectively while the result of the prior method [15] is 33.50%. Mcan- while, the overall results of our method for the 100-10 task are 34.65% and 35.49% which shows outperforming previous approaches. Fig. 3 visualizes the qualitative result of our method compared to PLOP [12]. Initially, the ground truth contains the class "car" in the first step and the class "minibike" in the third step. Then, in the fourth step, the class "bicycle" is included. As result, PL.OP [12] partly forgets the "minibike" information when learning the class "bicycle" information. Meanwhile, method consistently maintains the information of "minibike" and predicts segmentation correctly.
and PLOP [12] on ADE20K 100-10.
15-1 (6 steps)
10-1 (11 steps)
Pascal VOC As shown in Table 4, the proposed method outperforms the prior approaches eval- uated on the Pascal VOC 2012 dataset. In detail, our method achieves the overall mloU of 61.5% in the 15-1 task while the result of the previous
Method
0-15
16-20
all
0-10
11-20
all
Joint
79.8
72.4
77.4|
78.4
76.4
77.4
LWF [31]
6.0
3.9
5.5
8.0
2.0
4.8
ILT [25]
9.6
7.8
9.2
7.2
3.7
5.5
MiB
38.0
13.5
32.2
20.0
20.1
20.1
method [46] is 59.4%. Meanwhile, the mloU in the 10-1 task is 36.6% which shows better performance than the prior methods.
SDR [26]
47.3
14.7
39.5
32.4
17.1
25.1
PLOP [12)
65.1
21.1
54.6
44.0
15.5
30.5
RCIL [46]
70.6
23.7
59.4
55.4
15.1
34.3
FairCL+ + DeepLab--V V3
72.0
22.7
60.3
42.3
25.6
34.4
5
Conclusions and Dicussions
FairCI. + SegFormer
73.5
22.8
61.5
36.6
Conclusions: This paper introduces a novel fair- ness continual learning approach in semantic segmentation by analyzing the effect of class distribution. Furthermore, a new learning paradigm of continual learning, i.c.. the prototypical Contrastive clus- tering loss. is proposed to sufficiently address the catastrophic forgetting and the background shift problems. The experimental results on the three CSS benchmarks, i.c., ADE20K, Cityscapes, and Pascal VOC, have shown our SOTA performance.
Table 4: The mloU (%) of CSS on Pascal VOC.
Limitations:
Our paper has chosen specific configurations of network backbones and hyper- parameters to support our hypothesis. However, the other aspects of learning have not been fully investigated, e.g., learning hyper-parameters, the selected neighbor pixels, or other forms of Lrons.
Broader Impacts: This work studies the problem of Fairness in Continual Learning which is step toward fairness awareness in continual semantic segmentation. Our contributions emphasize the importance of faimess in continual semantic segmentation learning and provide solution to address the fairness concern that increases the robustness and trustworthiness of the segmentation model.
Acknowledgment This work is partly supported by NSF Data Science, Data Analytics that are Robust and Trusted (DART), and Googler Initiated Research Grant, We also thank Utsav Prabhu for invaluable discussions and suggestions and acknowledge the Arkansas High- Performance Computing Center for providing GPUs.

---

References
[1] N. Araslanov, and S. Roth. Self-supervised augmentation consistency for adapting semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[2] J. Cen, P. Yun, J. Cai, M. Y. Wang, and M. Liu. Deep metric learning for open world semantic segmentation.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15333 15342, 2021.
[3] F. Cermelli, M. Mancini, S. Rota Bulò, E. Ricci, and B. Caputo. Modeling the background for incremental learning in semantic segmentation. Proc. Conf. Comp. Vision Pattern Rec., 2020.
[4] S. Cha, b. kim, Y. Yoo, and T. Moon. Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. M. Ranzato, A. Beygelzimer, Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 10919-10930. Curran Associates, Inc., 2021.
[5] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs, 7PAMI, 2018.
[6] L.-C. Chen, G. Papandreou, E. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 0558777 2017.
[7] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[8] R. Chen, Y. Rong, S. Guo, J. Han, F. Sun, T. Xu, and W. Huang. Smoothing matters: Momentum transformer for domain adaptive semantic segmentation. CoRR, 2022.
[9] S. Chu, D. Kim, and B. Han. Learning debiased and disentangled representations for semantic segmentation. Advances in Neural Information Processing Systems, 34:8355-83366 2021.
[10] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[11] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balaneed loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition (CVPR), June 2019.
[12] A. Douillard, Y. Chen, Dapogny, and Cord. Plop: Learning without forgetting for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4040-4050, 2021.
[13] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJC, 2015.
[14] R. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 1999.
[15] D. Goswami, R. Schuster, J. van de Weijer, and Stricker, Attribution-war weight transfer: warm- start initialization for class-incremental semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3195-3204, 2023.
[16] K. Hc, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[17] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEF/CVF conference Computer Vision and Pattern Recognition, pages 831-839, 2019.
[18] L. Hoyer, D. Dai, and L. Van Gool. DAFormer: Improving network architectures and training strategies for Nomain-adaptive semantic segmentation. In CVPR, 2022.
[19] T.-L. Hsieh, E. Robb, H.-T. Chen, and J.-B. Huang. Droploss long-tail instance segmentation. In Proceedings of the Workshop on Artificial Intelligence Safery 2021 co-located with the Thirty- AAAI Conference on Artificial Intelligence, 2021.
[20] K. Joseph, S. Khan, F. S. Khan, and V. N. Balasubramanian. Towards open world object detection. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 9330-5840. 2021.
10

---

[21] J. Li and Q. Dong, Open-set semantic segmentation for point clouds via adversarial prototype framework. In Proceedings of the IEEE/CVF Conferencce on Computer Vision and Pattern Recognition, pages 9425-9434, 2023.
[22] Z. and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40112)2235-2944,, 2017.
[23] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition an open world, 2019.
[24] U. Michieli and P. 7anuttigh. Incremental learning techniques for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0-0, 2019.
[25] U. Michieli and P. Zanuttigh. Incremental learning techniques for semantic segmentation. In ICCVWS. 2019.
[26] U. Michieli and P. Zanuttigh. Continual semantic segmentation via applssio-aattractin of sparse and disentangled latent representations. In Proc. Conf. Comp. Vision Pattern Rec., 2021.
[27] F. Ozdemir, P. Fuernstahl, and O. Goksel. Learn the new, keep the old: Extending pretrained models with new anatomy and images. Medical Image Computing and Computer Assisted Intervention 2018: 2 International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11, pages 361-369 Springer, 2018
[28] F. Ozdemir and O. Goksel. Extending pretrained segmentation networks with additional anatomical structures. International journal of computer assisted radiology and surgery, 14:1 1198-1999, 2019.
[29] M. H. Phan, T.-A. Ta, S. L. Phung. L. Than,, and A. Bouzerdoum. Class similarity weighted knowledge distillation for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16866-16875, June 2022.
[30] Y. Qiu, Shen, Z. Sun, Zheng, X. Chang, W. Zheng, and R. Wang. Sats: Self-attention transfer for continual semantic segmentation. Puttern Recognition, 138: 109383, 2023.
[3]] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proc. Conf. Comp. Vision Puttern Rec., 2017.
[32] J. Ren, C. Yu, S. Sheng, X. Ma, H. Zhao, S. Yi, and H. Li. Balanced meta-softmax for long-tailed visual recognition, 2020.
[33] A. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 1995.
[34] M. Rostami.
Lifelong domain adaptation via consolidated internal distribution. In NeurlPS, 2021.
[35] A. Saporta, A. Douillard, T.-H. Vu, P. Pérez, and M. Cord. Multi-head distillation for continual unsupervised domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshop, 2022.
[36] K. Shmelkov, C. Schmid, and K. Alahari. Incremental learning of object detectors without catastrophic forgetting. Proceedings of the IEEE international conference on computer vision, pages 3400-3409, 2017.
[37] A. Szabó, H. Jamali-Rad, and S.-D. Mannava. Tilted cross-entropy (Ice): Promoting fairness in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23100 2021.
[38] S. Thrun. Lifelong learning algorithms. In Springer Learning to Learn, 1998.
[39] T.-D. Truong, C. N. Duong, N. Le, S. L. Phung, C. Rainwater, and K. Luu. Bimal: Bijective maximum likelihond approach domain adaptation in semantic scene segmentation. In International Conference on Computer Vision, 2021.
[40] T.-D. Truong, N. Le, Raj, J. Cothren, and Luu. Fredom: Fairness domain adaptation approach to semantic scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[41] R. Volpi, D. Larlus, and G. Rogez. Continual adaptation of visual representations via domain randomization and meta- leaanng. In Proc. Conf. Comp. Vision Pattern Rec., 2021.
11

---

[42] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.
[43] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Dada: Depth-aware domain adaptation in semantic segmentation. In ICCV, 2019.
[44] J. Wang. W. Zhang, Y. Zang, Y. Cao, J. Pang. T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, 2021.
[45] E. Xie, W. Wang. Z. Yu. A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, 2021.
[46] C.-B. Zhang, J.-W. Xiao, X. Liu, Y.-.. Chen, and M.-M. Cheng. Representation compensation networks continual semantic segmentation. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7053-7064, 2022.
[47] P. Zhang, B. Zhang, T. Zhang, Chen, Y. Wang, and E. Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. arXiv preprint arXiv:2101.10999 2021.
[48] B. Zhao, X. Xiao, G. Gan, B. Zhang, and S.-T. Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedingy of the IEEE/CVF conference computer vision and puttern recognition, pages 12200-1317,, 2020.
[49] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633 641, 2017.
12


Reviews:
[{"review_id": "lW5FEOWUj8", "reviewer": "Reviewer_jVAT", "summary": "The paper focuses on the semantic segmentation problem in the context of continual learning. In the past, the emphasis was mainly on addressing catastrophic forgetting and background shift issues. However, this paper goes beyond that and investigates the balance problem between different object categories. It introduces novel approaches for prototype contrastive learning and structure-consistency loss functions, which result in a new SOTA performance. The contributions of the paper can be summarized into three main parts:\n\ni. It introduces a novel evaluation metric to measure the model's balance across different classes.\n\nii. It proposes a new contrastive learning loss to address the issue of forgetting in continual learning.\n\niii. The paper conducts extensive experiments to demonstrate the effectiveness and rationality of the proposed methods.", "questions": "As far as I know, the current research on semantic segmentation is primarily conducted using DeepLab v3 as the framework. Why did the paper choose to experiment with DeepLab v2 instead?", "limitations": "The paper discusses its own limitations in the experimental section. The paper does not have any negative social impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The paper considers multiple perspectives to improve fairness among classes of the model.\n+ The paper designs various loss functions to enhance the segmentation performance of the model under continual learning.\n+ The supplementary materials of the paper include an excellent presentation that makes it easier for readers to understand the superiority of the model.", "weaknesses": "+ The paper lacks experiments specifically focused on the upper bound design for continual learning.\n+ The proposed inter-class fairness in the paper seems to be addressing the semantic segmentation problem rather than the main focus of continuous learning. However, this issue has already been well discussed in [1].\n[1] Truong, T. D., Le, N., Raj, B., Cothren, J., & Luu, K. (2023). Fredom: Fairness domain adaptation approach to semantic scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19988-19997).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer jVAT\n\nDear Reviewer jVAT,\n\nWe thank the reviewer for your careful read and valuable feedback.\nWe are pleased you think of our proposed problem and approach as our strengths and significant contributions. We appreciate your constructive comments and would like to address these points as follows.\n\n[Q1] **The upper bound design for continual learning**\n\n[A1] We have illustrated the upper-bound results of our method, where the upper-bound results are the performance of the model trained on the joint data tasks. As shown in the below tables, the gap in performance of our method with the upper bound remains lower than other approaches in all three benchmarks.\n\n\n\n\nTable 1: ADE20K Dataset\n\n\n\n\n| Method | | 100-50 | | | 50-50 | | | 100-10 | |\n| :------------------ | :---: | :-----: | :---: | :---: | :----: | :---: | :---: | :-----: | :---: |\n| | 0-100 | 101-150 | all | 0-50 | 51-150 | all | 0-100 | 101-150 | all |\n| Upper Bound | 44.30 | 28.20 | 38.90 | 51.10 | 32.80 | 38.90 | 44.30 | 28.20 | 38.90 |\n| MiB + AWT | 40.90 | 24.70 | 35.60 | 46.60 | 26.85 | 33.50 | 39.10 | 21.28 | 33.20 |\n| SATS | - | - | - | - | - | - | 41.42 | 19.09 | 34.18 |\n| FairCL + DeepLab-V3 | 43.40 | 24.04 | 36.99 | 49.65 | 26.84 | 34.55 | 41.73 | 20.36 | 34.65 |\n| FairCL + SegFormer | 43.56 | 25.46 | 37.56 | 49.62 | 27.78 | 35.15 | 42.21 | 21.91 | 35.49 |\n\n\n\n\n\n\n\n\nTable 2: Pascal VOC Dataset\n\n\n\n\n| Method | | 15-1 | | | 10-1 | |\n| :------------------ | :--: | :---: | :--: | :--: | :---: | :--: |\n| | 0-15 | 16-20 | all | 0-10 | 11-20 | all |\n| Upper Bound | 79.8 | 72.4 | 77.4 | 78.4 | 76.4 | 77.4 |\n| RCIL | 70.6 | 23.7 | 59.4 | 55.4 | 15.1 | 34.3 |\n| FairCL + DeepLab-V3 | 72.0 | 22.7 | 60.3 | 42.3 | 25.6 | 34.4 |\n| FairCL + SegFormer | 73.5 | 22.8 | 61.5 | 57.1 | 14.2 | 36.6 |\n\n\n\n\n\n\n\n\nTable 3: Cityscapes Dataset\n\n\n\n\n| Method | 11-5 | 11-1 | 1-1 |\n| :------------------ | :---: | :---: | :---: |\n| | all | all | all |\n| Upper Bound | 79.30 | 79.30 | 79.30 |\n| RCIL | 64.30 | 63.00 | 48.90 |\n| FairCL + DeepLab-V3 | 66.96 | 66.61 | 49.22 |\n| FairCL + SegFormer | 67.85 | 67.09 | 55.68 |\n\n\n\n\n\n\n\n\n[Q2] **Comparison to [1]**\n\n\n\n\n[A2] In our related work section, we discussed the FREDOM paper [1] in L97-99. As the reviewer highlighted, the focus of [1] and our paper remains different since our work concentrates on the fairness problem in the context of continual learning. Meanwhile, FREDOM addresses fairness in the context of learning from the entire data at once under the domain adaptation setting.\n\n\n\n\n[Q3] **Clarification of Segmentation Network**\n\n\n\n\n[A3] DeepLab-V2 is our typo in the text. In our implementation, we use ***DeepLab-V3*** for fair comparison in our experiments (similar to PLOP [2] RCIL [3], MiB [4]). We sincerely apologize for this oversight and assure you that we are committed to correcting this typo promptly.\n\n\n\n\n### References\n\n\n\n\n[1] Truong et al. Fredom: Fairness domain adaptation approach to semantic scene understanding. CVPR, 2023.\n\n\n\n\n[2] Douillard et al. Plop: Learning without forgetting for continual semantic segmentation. CVPR, 2021.\n\n\n\n\n[3] Zhang et al. Representation compensation networks for continual semantic segmentation. CVPR, 2022.\n\n\n\n\n[4] Cermelli et al. Modeling the background for incremental learning in semantic segmentation. CVPR, 2020."}, {"review_id": "mdZ7cBnIBy", "reviewer": "Reviewer_ReY5", "summary": "The paper explores the continual semantic segmentation problem and highlights the lack of fairness in existing methods. It proposes a novel fairness objective based on class distributions to address the bias on the majority group classes. The paper proposes a novel prototype contrastive clustering loss to address the significant challenges of catastrophic forgetting as well as for handling unknown classes to solve the background shift problem. Furthermore, a conditional structural consistency loss is proposed to constrain the structure of the predicted segmentation. The method has achieved improvements over the existing benchmarks on ADE20K, Cityscapes and Pascal VOC datasets.", "questions": "1. Inconsistent results: The results for the ADE20K 100-50 setting are different in table 1 and table 3. In table 1 for ablation, the overall mIoU is 34.64 for DeepLab-V2 backbone, while in table 3, the overall mIoU is 36.99. The mIoU values are the same for the SegFormer backbone though. This should be clarified since the ablation experiment is claiming the contributions of the different losses.\n\n2. While it is good to see the results with both DeepLab-V2 and SegFormer backbones, the backbone networks of the compared methods should be mentioned for more clarity and fair comparison.\n\n3. The comparison does not include a well performing baseline method - SSUL (from NeurIPS 21). If there is any specific reason for not comparing with SSUL, it should be discussed in the paper.\n\n4. See weaknesses 1, 2, 3. ", "limitations": "/", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well written and the motivation is good. The problem formulation is nicely done.\n\n2. The paper points out the issue of maintaining fairness in continual segmentation systems. It is a very significant contribution to the field since all existing methods primarily focussed on reducing forgetting and addressing the background shift.  The fairness problem is well-defined in the paper along with the fairness metric which is used to obtain the error-rate among the classes.\n\n3. The novel prototype contrastive clustering loss represents the semantic segmentation pixels for different classes in the latent space and groups the features of unknown classes into the same cluster. In addition to handling unlabelled classes, it is interesting to see that the clustering loss also serves as an upper bound of the knowledge distillation approach and thus reduces the forgetting of old classes.\n\n4. The proposed method FairCL seems to be quite effective on several settings across multiple datasets with different backbones.", "weaknesses": "1. Although the fairness loss is improving the mIoU values for different settings on ADE20K, it is not clear if the improvement in mIoU is coming from the minority classes. While the main motivation of the paper is to ensure fairness among the classes, the ablation experiment does not completely justify this. The individual contribution of the fairness loss (L_class) should be stated since Fairness in Continual Learning is the main claim of the authors. Further analysis of the improvements in mIoU values separately for majority and minority classes on at least two datasets will be more convincing. Also, the fairness loss may be evaluated in a non-incremental setting (let's say for the first step only), and then the minority class mIoU values should be better; otherwise, the performance is also affected by forgetting, and the contribution of the fairness loss is not clear.\n\n2. The motivation for learning conditional structural consistency loss is not clear and should be discussed to give more intuition. Also, some qualitative analysis of prediction consistency will be good to show the effectiveness of the structural consistency loss.\n\n3. Results from Pascal VOC are not convincing. For the 10-1 setting, the forgetting is quite high compared to methods like RCIL. Overall, the improvement looks very insignificant. Experiments on more settings of Pascal VOC similar to previous papers like PLOP and RCIL will be good. Also, the comparison of the SegFormer backbone with the other existing methods using DeepLabV2 is not fair. If there is any specific reason for the method not being so effective for Pascal VOC (as it looks effective for ADE20K and Cityscapes), it should be analyzed. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer ReY5\n\n\n\n\nDear Reviewer ReY5,\n\n\n\n\nWe would like to thank the reviewer for your careful read and valuable feedback.\nWe appreciate that you think our paper is well-motivated and well-written, the proposed approach is well-formulated and novel, and our theoretical analysis of the upper bound of our proposed approach is significant. We appreciate your constructive comments and would like to address these points as follows:\n\n\n\n\n[Q1] **The performance improvement in minor and major classes**\n\n\n\n\n[A1] Please refer to our **Global Response** for our clarification on the performance improvement in minor and major classes in both continual learning and non-incremental settings.\n\n\n\n\n\n\n\n\n[Q2] **Motivation for learning conditional structural consistency**\n\n\n\n\n[A2] The goal of conditional structural consistency is to improve the prediction gap among neighbor pixels, thus, enhancing the smoothness of the predictions. In addition, it helps to increase fairness among classes. It is because this loss helps to clean up the spurious or ambiguous predictions produced by the major classes around minor classes. Therefore, the loss alleviates the dominance of the major groups and improves the accuracy of the minor groups, thus, resulting in fairness that has also been further improved (as illustrated in Table 1 in the paper).\n\n\n\n\n\n\n\n\n[Q3] **Results from Pascal VOC**\n\n\n\n\n[A3] In our VOC experiments, similar to [1, 2], we choose the continual learning settings of 10-1 and 15-1 since these two settings illustrate more the impact of forgetting and the effectiveness of the method to the forgetting problem. In addition, we have observed the performance improvement of our approach on Pascal VOC is less significant than the ADE20K and Cityscapes because of the minor bias in Pascal VOC. In particular, the data distributions of these datasets are visualized in Figure 1 in the rebuttal file (the PDF file of the rebuttal is attached in Global Response). We calculated the entropy value of the data distributions that illustrate the balance level of the datasets (the higher value of entropy, the more balance the dataset as the data distribution tends to be more uniform). Then, based on the data distributions and entropy values, we observe the data distributions of ADE20K and Cityscapes suffer more bias than the Pascal VOC since the entropy value of Pascal VOC (H = 0.81) is higher than ADE20K (H = 0.69) and Cityscapes (H = 0.62). Thus, ADE20K and Cityscapes suffer severe fairness issues compared to Pascal VOC. Our approach aims to improve the fairness of the model. Therefore, on the more severe bias datasets (ADE20K and Cityscapes), our approach performs more significantly.\n\n\n\n\n[Q4] **Inconsistent Results**\n\n\n\n\n[A4] Upon further investigation, we have discovered that the inconsistencies between Table 1 and Table 3 in the paper were a result of typographical errors that occurred during the typing process. We have updated the correct results in Table 1 (in the paper), consistent with our results in Table 3 (in the paper). We would like to offer our sincere apologies for this oversight.\n\n\n\n\n\n\n\n\n[Q5] **Fair Comparision**\n\n\n\n\n[A5] The comparisons in our experiments are fair with other baselines. As clarified to Reviewer jVAT, DeepLab-V2 is our typo in the text. We sincerely apologize for this oversight and are committed to promptly correcting this typo. In our implementation, we use DeepLab-V3 (with ResNet 101) for fair comparisons with PLOP [1], RCIL [2], and MiB [3] in our experiments. Similarly, we adopt SegFormer as SATS [4] for fair comparisons.\n\n\n\n[Q6] **Result of SSUL**\n\n\n\n\n[A6] We have compared our results with SSUL [5] in the table below. Our paper achieve a competitive performance compared to SSUL on the ADE20K benchmarks.\n\n\n\nTable 1: Comparisons with SSUL on ADE20 Benchmarks\n| Method | | 100-50 | | | 50-50 | | | 100-10 | |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| | 0-100 | 101-150 | all | 0-50 | 51-150 | all | 0-100 | 101-150 | all |\n| SSUL | 41.28 | 18.02 | 33.58 | 48.38 | 20.15 | 29.56 | 40.20 | 18.75 | 33.10 |\n| FairCL + DeepLab-V3 | 43.40 | 24.04 | 36.99 | 49.65 | 26.84 | 34.55 | 41.73 | 20.36 | 34.65 |\n| FairCL + SegFormer | 43.56 | 25.46 | 37.56 | 49.62 | 27.78 | 35.15 | 42.21 | 21.91 | 35.49 |\n\n\n\n\n\n\n\n\n### References\n\n\n\n\n[1] Douillard et al. Plop: Learning without forgetting for continual semantic segmentation. CVPR, 2021.\n\n\n\n\n[2] Zhang et al. Representation compensation networks for continual semantic segmentation. CVPR, 2022.\n\n\n\n\n[3] Cermelli et al. Modeling the background for incremental learning in semantic segmentation. CVPR, 2020.\n\n\n\n\n[4] Qiu et al. Sats: Self-attention transfer for continual semantic segmentation. Pattern Recognition, 2023.\n\n\n\n\n[5] Cha et al. SSUL: Semantic segmentation with unknown label for exemplar-based class-incremental learning. NeurIPS, 2021.\n"}, {"review_id": "6AQ1dVfBs6", "reviewer": "Reviewer_2Nqo", "summary": "This paper proposes a Fairness Continual Learning approach to address the continual semantic segmentation problem. Specifically, the fairness continual learning framework is proposed for fairness objective, while a Prototypical Contrastive Clustering loss for reducing catastrophic forgetting. When compared to prior work on continual semantic segmentation, FairCL is shown to improve Final mIoU significantly.", "questions": "I am looking forward to the authors clarifying the concerns listed in the section above: \n1. The final loss function and the combination form of each part; \n2. More detail explanation of the Prototypical Contrastive Clustering Loss; \n3. Comparison in experiment results.", "limitations": "Both limitations and potential societal impacts are addressed in the paper. However, it might help to expand the paragraph to discuss some of the concerns listed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper analyzes several problems that need to be solved in continuous learning in detail, and clearly presents the loss function corresponding to the three objectives from the perspective of formula derivation. \n2. Extensive ablation studies are performed which provides insight into the challenges of continual semantic segmentation problem.", "weaknesses": "1. The organization of method is not very clear. It's difficult to follow what the final loss function is and how each part of loss is combined. \n2. About the Prototypical Contrastive Clustering Loss, I wonder whether pulling all features of unknown classes into one cluster is reasonable. Will learning similar features from categories with significant differences affect subsequent model training? \n3. How is the margin $\\triangle$ selected? \n4. In Sec.3.2, 'a pseudo label where its assigned label is computed based on the closest cluster'. How is the pseudo label computed and utilized? \n5. The compare in experiments about network backbone is unfair.  The experiment provides results on two different backbones, but neither backbone was aligned with the previous method such as Deeplab-v3 architecture with ResNet-101 in RCIL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 2Nqo\n\nDear Reviewer 2Nqo,\n\n\n\n\nWe would like to thank the reviewer for your careful read and valuable feedback.\nWe are very happy you encourage that our proposed losses are well-formulated and the ablation studies have illustrated the effectiveness of our approach. We appreciate your constructive comments and would like to address these points as follows:\n\n\n\n\n\n\n\n\n[Q1] **Final loss combination**\n\n\n\n\n[A1] In our model, the final training loss is the summation of the softmax loss with labels (and pseudo labels), the contrastive clustering loss, the fairness loss, and the conditional structural consistency loss. In our work, we do not utilize the balance weights among proposed losses. Similar to other common practices [5, 8], we investigate the impact and contribution of our proposed losses and their vital roles in fairness and catastrophic forgetting (as illustrated in the ablation study). We leave the investigation of balanced weights among losses as our future work.\n\n\n\n\n[Q2] **Organization of the paper**\n\n\n\n\n[A2] We would like to clarify our paper organization as follows. Our intention was to follow a clear structured progression to present the background of the fairness continual problem, fairness metric and fairness learning learning (Secs 3.1 and 3.2), followed by our proposed methodology and its properties (Secs 3.3-3.4), results (Sec 4), and discussion of limitations (Sec 5). We aim to present our proposed approach in a logical sequence to ensure a comprehensive understanding of our method. In addition, the combination of training losses has been clarified in the previous answer (Q1-A1). Moreover, it should be highlighted that the reviewers agree our paper is well-written (Reviewer ReY5), well-motivated (Reviewer ReY5), and clear presentation of loss functions corresponding to objectives (Reviewer 2Nqo). We will improve our paper organization promptly based on the comments of the reviewer.\n\n\n\n\n\n\n\n\n[Q3] **Pulling all features of unknown classes into one cluster**\n\n\n\n\n[A3] In our proposed approach, we pull the features of unknown classes into a single cluster because we acknowledge the fact that features of novel classes or unknown classes should aggregate into a region (as proved in [5, 6, 7]) instead of scattering in the feature space where these could be misclassified into the other known classes or be produced as discriminative features (as mentioned in Sec 3.2). Later, in the future step, when the classes of unknown features are known, these will be attracted into their corresponding clusters. We acknowledge that the performing sub-clustering on the unknown features could potentially improve the performance of the model and could be explored as a future research direction of our approach.\n\n\n\n\n[Q4] **The choice of margin $\\Delta$**\n\n\n\n\n[A4] In our paper, following the common practice of choosing margin $\\Delta$ in prior work [5], we choose $\\Delta = 10$. In addition, we also perform an additional ablation study on the ADE20K (100-50) benchmark to investigate the impact of the delta. As shown in the table below, the impact of $\\Delta$ does not significantly influence the results due to the minor performance drop.\n\n\n\nTable 1: The effectiveness of choosing $\\Delta$ (ADE20K  100-50 Benchmark)\n| $\\Delta$ | 0-100 | 101-150 | all | avg |\n| :-------: | :---: | :-----: | :---: | :---: |\n| 5 | 43.04 | 23.84 | 36.68 | 40.23 |\n| **10** | **43.40** | **24.04** | **36.99** | **40.45** |\n| 15 | 42.89 | 23.92 | 36.61 | 40.26 |\n\n\n\n\n\n\n\n\n[Q5] **The use of Pseudo Labels**\n\n\n\n\n[A5] Following the common practice in [1, 2], in addition to the labels of the current task, we utilize the pseudo labels to compute the softmax loss. Moreover, the pseudo labels are used to assign the cluster in Eqns. (8-9).\n\n\n\n\n[Q6] **Fair Comparision**\n\n\n\n\n[A6] The comparisons in our experiments are fair with other baselines. As our clarification to Reviewers jVAT and ReY5, DeepLab-V2 is our typo in the text. We sincerely apologize for this oversight and we are committed to correcting this typo promptly. In our implementation, we use DeepLab-V3 for fair comparisons with PLOP [1], RCIL[2], MiB [3] in our experiments. In addition, we also adopt the SegFormer network as SATS [4].\n\n\n\n\n[Q7] **Discussion of limitations and potential societal impacts**\n\n\n\n\n[A7] Thank you very much for your suggestion. We will extend our discussion about limitations and social impacts in our paper.\n\n\n\n\n\n\n\n\n\n\n\n\n### References\n\n\n\n\n[1] Douillard et al. Plop: Learning without forgetting for continual semantic segmentation. CVPR, 2021.\n\n\n\n\n[2] Zhang et al. Representation compensation networks for continual semantic segmentation. CVPR, 2022.\n\n\n\n\n[3] Cermelli et al. Modeling the background for incremental learning in semantic segmentation. CVPR, 2020.\n\n\n\n\n[4] Qiu et al. Sats: Self-attention transfer for continual semantic segmentation. Pattern Recognition, 2023.\n\n\n\n\n[5] Joseph et al. Towards open world object detection. CVPR, 2021.\n\n\n\n\n[6] Cen et al. Deep metric learning for open world semantic segmentation. ICCV, 2021.\n\n\n\n\n[7] Li et al. Open-Set Semantic Segmentation for Point Clouds via Adversarial Prototype Framework. CVPR, 2023.\n\n\n\n\n[8] Truong et al. Fredom: Fairness domain adaptation approach to semantic scene understanding. CVPR, 2023."}, {"review_id": "z8vr77cDuf", "reviewer": "Reviewer_9fXu", "summary": "This paper proposes a fairness continual learning approach in semantic segmentation. Derived from the therotical analysis of the effect of class distribution, a new learning paradigm that is consisted of three main objectives: the task-specific objective to handle the catastrophic forgetting problem, the fairness objective that maintains the fairness predictions over classes, and the conditional structural constraint that imposes the consistency of the segmentation predictions. New SOTA results are attained on the standard benchmarks.", "questions": "1. Eq. 5 is hard to understand and I think it could not be intuitively achieved.\n2. The explanation that the prototypical contrastive clustering loss is an upper bound of the knowledge distillation approach is not rigorous. In eq. (12) , the $D(p_c, f^{t-1})$ cannot directly ignored as a constant number. When $c$ is a new class from $C^t$, the $p_c$ is changing during learning and $D(p_c, f^{t-1})$ will change. Therefore, $O(D(f^t, p_c)$ is not a tight boundary. Minimizing $L_{cluster}$ is not strictly equals to the minimizing of $L_distill$. From another perspective, although prototypical constrative clustering loss don't need two forward pass of the segmentation model, it have to store the prototypical vector $p_c$ using additional memory. How to calculate the $p_c$ is also critical in achieving good performance.\n3. What is the final integrated training loss? Is there any balance weights between different losses?\n4. What is the result in the first line of DeepLab-V2 and SegFormer in Table 1?\n5. Im my understanding, the conditional structural consistency mitigate the prediction gap among neighbour pixels. However, it will also smooth the edge pixels of an instance. According to the ablation study, it also improves the fairness of classes. Could the author give more explanation on it?\n6. Minor problems: when writting the related works, it better to use the surname other than the name. For example, Ziwei et al. should be Liu et al in line 93. In eq. 5, the $y_{i,h}$ should be $y_{i,j}$ in the left term?", "limitations": "Authors have pointed out the limitations of the proposed method and some possible improvements.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Deriving the learning objective through analysis of the effect of class distribution is novel. \n2. Strong performances are  achieved on several standard benchmarks.", "weaknesses": "- The title is misleading. In my understanding, the open-world environments means lacking of labels. While the proposed method can hardly work in unsupervised setting.\n- Some explanation through mathmatical derivation is not rigorous such as eq. 5 and eq. 12 and 13. More explanation either in the main paper or supplimentary material needed. Detail illustration can be found in the questions.\n- Some details are not clear. How the fairness objective $L_{class}$ is calculated during training is not clear and lacking a final integrated training objective. Whether there is hyperparameters between losses is not mentioned. \n- According to the ablation study, fairness improvement brought by the fairness objective is minor if measured in STD of IoU over all classes. Better measurement of the fairness is needed. For example, variation of the performance on rare classes and dominant classes with and without fairness loss can be compared in a longer learning steps.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 9fXu\n\n\n\n\nDear Reviewer 9fXu,\n\n\n\n\nWe thank the reviewer for your careful read and valuable feedback.\nWe are pleased you encourage that our theoretical analysis is novel and our approach achieves strong performance. We appreciate your constructive comments and would like to address these points as follows:\n\n\n\n\n[Q1] **Misleading Title**\n\n\n\n\n[A1] We thank you for the comment from the reviewer. However, we would like to clarify our paper title. Basically, continual learning aims to learn the AI model sequentially and address the challenges, e.g., catastrophic forgetting, background shift, and fairness, posed by the evolving and dynamic environments where the new data and classes continuously grow over time. Meanwhile, in the open-world problem setting, the data distribution can change, e.g., data distribution grows, new classes appear, or data of existing classes can evolve. Thus, there is an interference between continual learning and open-world learning in which continual learning could be considered as a part of the goals of open-world learning. Therefore, we use the open-world term in our paper title.\n\n\n\n\n[Q2] **Explanation of Eqn. (5)**\n\n\n\n\n[A2] Intuitively, Eqn (5) illustrates that the major classes of more samples will produce larger gradients to update the classifier and the network backbone. Meanwhile, the minor classes of fewer samples will produce smaller gradients. Thus, the major classes have a higher lower bound of loss value and contribute much higher importance than the minor classes having fewer samples. Therefore, this bias between major and minor classes will influence the unfair predictions produced by the model. A similar issue has been observed in [1, 2, 3].\n\n\n\n\n[Q3] **The upper proof of prototypical contrastive clustering loss**\n\n\n\n\n[A3] We clarify our proof of Eqns (12-13) in Sec. 3.3 still holds true. We acknowledge that $\\mathbf{p}_c$ is updated during the training procedure. However, at the point of the prototypical contrastive clustering loss is computed, the set of prototypical vectors $\\mathbf{p}_c$ at this point is fixed. Only after each $M$ iterations, the prototypical vectors $\\mathbf{p}_c$ will be updated. Thus, our clustering loss remains an upper bound of distillation loss due to the fixed vector $\\mathbf{p}_c$ (could be considered as constant) at the loss computation time. Therefore, without a strict argument, it remains that our derive in Eqn (12) still holds. In addition, although our approach does not directly minimize the knowledge distillation loss due to the upper bound, minimizing our clustering loss will intrinsically decrease the upper bound of the knowledge distillation loss. Thus, our approach could also implicitly minimize the knowledge distillation loss and prevent catastrophic forgetting.\n\n\n\n\n[Q4] **Memory efficiency of prototypical contrastive clustering loss**\n\n\n\n\n[A4] We would like to highlight that storing prototypical vectors requires significantly less memory than using the additional teacher model as used in distillation approaches. For example, in the ADE20K benchmark, storing DeepLab-V3 (151 classes) requires 58.664 M parameters, while storing 152 prototypical 2048-D vectors (including the unknown cluster) only uses 0.311M parameters. In addition, the computation of $\\mathcal{L}_{cluster}$ loss with $\\mathbf{p}_c$ is cheaper than a forward pass of the entire network used in distillation. Therefore, in terms of computational cost and memory, our approach remains more efficient compared to knowledge distillation approaches.\n\n\n\n\n\n\n\n\n[Q5] **First line results in Table 1**\n\n\n\n\n[A5] The first line indicates the results of softmax only (mentioned in L307).\n\n\n\n\n[Q6] **The role of conditional structural consistency in fairness**\n\n\n\n\n[A6] As our clarification to Reviewer ReY5 (Q2-A2), the goal of conditional structural consistency is to improve the prediction gap among neighbor pixels, thus, enhancing the smoothness of the predictions. In addition, it helps to increase fairness among classes. It is because this loss helps to clean up the spurious or ambiguous predictions produced by the major classes around minor classes. Therefore, the loss alleviates the dominance of the major groups and improves the accuracy of the minor groups, thus, resulting in fairness that has also been further improved (as illustrated in Table 1 in the paper).\n\n\n\n\n[Q7] **Details of $\\mathcal{L}_{class}$ and the final loss combination**\n\n\n\n\n[A7] The fairness loss $\\mathcal{L}_{class}$ is calculated based on the predictions produced by the model and the ideal uniform class distribution, as mentioned in L175-176. As our clarification to Reviewer 2Nqo (Q1-A1), the final loss is the summation of the softmax loss with (pseudo) labels, the contrastive clustering loss, the fairness loss, and the conditional structural consistency loss. Our work does not utilize the balance weights among losses. Following common practices [4, 5], we investigate the contributions of our losses and their vital roles in fairness and catastrophic forgetting. We leave the investigation of balanced weights among losses as our future work.\n\n\n\n\n[Q8] **Better measurement of the fairness**\n\n\n\n\n[A8] Please refer to our **Global Response** for our clarification on the performance improvement in major (dominant) and minor (rare) classes.\n\n\n\n\n[Q9] **Typos in Related Work and Equations**\n\n\n\n\n[A9] We have updated typos in related work and equations.\n\n\n\n\n### References\n[1] Truong et al. Fredom: Fairness domain adaptation approach to semantic scene understanding. CVPR, 2023.\n\n\n\n\n[2] Wang et al. Seesaw loss for long-tailed instance segmentation. CVPR, 2021.\n\n\n\n\n[3] Cui et al. Parametric contrastive learning. ICCV, 2021.\n\n\n\n\n[4] Joseph et al. Towards open world object detection. CVPR, 2021.\n\n\n\n\n[5] Truong et al. Fredom: Fairness domain adaptation approach to semantic scene understanding. CVPR, 2023.\n"}]

Author rebuttal:
## Global Response


We thank all the reviewers for their careful read and valuable feedback. Reviewer 9fXu and Reviewer jVAT ***weakly accept*** our paper; Reviewer ReY5 considers our work a ***borderline accept***; and Reviewer 2Nqo considers ***a borderline reject***. Our implementation will be published for research reproducibility. We are very happy the reviewers encourage that ***our work is well-motivated*** (Reviewer ReY5), ***our paper is well-written*** (Reviewers ReY5, 2Nqo), ***our method is novel*** (Reviewers jVAT, ReY5, 9fXu), and ***our approach achieves strong performance*** (Reviewers ReY5, 2Nqo, 9fXu). We have updated the typos in our paper.








On the constructive side, Reviewer ReY5 and Reviewer 9fXu noted that better fairness measurement will be beneficial to illustrate the effectiveness and the contribution of the proposed method to fairness improvement. Therefore, as suggested by the reviewers, to illustrate the performance improvement of our proposed method in major and minor classes, we include the results of the mIoU (all) and the STD among IoUs of the major group and the minor group on the ADE20K 100-50 (Table 1) and Cityscapes 11-5 (Table 2) benchmarks. As shown in the table below, our proposed approach has improved the performance of both major and minor groups. Thus, these results illustrated that the performance improvement in mIoU also comes from the minority classes. It helps to enhance the mIoU performance and reduce the STD in both major and minor classes, thus, improving the fairness of the model predictions.








Table 1: ADE20K 100-50 Benchmark
| Backbone | $\mathcal{L}_{cluster}$ | $\mathcal{L}_{class}$ | $\mathcal{L}_{cons}$ | Major | Major | Minor | Minor |
| - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
||||| mIoU | STD | mIoU | STD |
| DeepLab-V3 | yes | | | 48.78 | 18.12 | 25.13 | 21.12 |
| DeepLab-V3 | yes | yes | | 48.89 | 17.87 | 27.24 | 20.76 |
| DeepLab-V3 | yes | yes | yes | 50.11 | 17.46 | 30.52 | 20.43 |




Table 2: Cityscapes 11-5 Benchmark
| Backbone | $\mathcal{L}_{cluster}$ | $\mathcal{L}_{class}$ | $\mathcal{L}_{cons}$ | Major | Major | Minor | Minor |
| - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| | | | | mIoU | STD | mIoU | STD |
| DeepLab-V3 | yes | | | 87.44 | 9.25 | 53.37 | 16.72 |
| DeepLab-V3 | yes | yes | | 88.29 | 8.85 | 55.72 | 13.39 |
| DeepLab-V3 | yes | yes | yes | 89.20 | 8.41 | 56.70 | 11.96 |
















Similarly, to illustrate the effectiveness and robustness of our method in the non-incremental setting. We report our results after the first learning step on the ADE20K 100-50 (Table 3) and Cityscapes 11-5 (Table 4) benchmarks. Our proposed fairness approach has also contributed to the performance improvement of both major and minor classes in non-incremental settings. The comparison table of major and minor groups in the first step is illustrated below.








Table 3: ADE20K 100-50 Benchmark (Non-incremental Setting)
| Backbone | $\mathcal{L}_{cluster}$ | $\mathcal{L}_{class}$ | $\mathcal{L}_{cons}$ | Major | Major | Minor | Minor |
| - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| | | | | mIoU | STD | mIoU | STD |
| DeepLab-V3 | yes | | | 49.07 | 18.35 | 35.45 | 19.08 |
| DeepLab-V3 | yes | yes | | 49.17 | 18.31 | 35.57 | 18.24 |
| DeepLab-V3 | yes | yes | yes | 50.04 | 17.92 | 37.88 | 18.19 |




Table 4: Cityscapes 11-5 Benchmark (Non-incremental Setting)
| Backbone | $\mathcal{L}_{cluster}$ | $\mathcal{L}_{class}$ | $\mathcal{L}_{cons}$ | Major | Major | Minor | Minor |
| - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| | | | | mIoU | STD | mIoU | STD |
| DeepLab-V3 | yes | | | 87.64 | 8.96 | 53.78 | 17.16 |
| DeepLab-V3 | yes | yes | | 88.56 | 8.70 | 56.45 | 13.46 |
| DeepLab-V3 | yes | yes | yes | 89.52 | 8.08 | 57.87 | 12.08 |


Meta-review: The paper describes an approach for fairness continual learning applied to semantic segmentation. The paper provided both theoretical insights on the proposed method and convincing experimental results proving the effectiveness of the proposed approach (which outperforms competitors). The reviewers mostly asked clarifications about the method (fairness objective, hyperparameters) and a few experimental results (some missing ablation). The authors provided all the necessary clarifications and few additional results. All reviewers agree that the paper can be accepted.
Decision: Accept (poster)