Deliberated Domain Bridging for Domain Adaptive Semantic Segmentation
Authors: Lin Chen
Zhixiang Wei
Xin Jin
Huaian Chen
Miao Zheng
Kai Chen
Yi Jin
Venue: NeurIPS 2022 Accept (2022)
Keywords: domain adaptive semantic segmentation
domain bridging
data mixing

Abstract: In unsupervised domain adaptation (UDA), directly adapting from the source to the target domain usually suffers significant discrepancies and leads to insufficient alignment. Thus, many UDA works attempt to vanish the domain gap gradually and softly via various intermediate spaces, dubbed domain bridging (DB). However, for dense prediction tasks such as domain adaptive semantic segmentation (DASS), existing solutions have mostly relied on rough style transfer and how to elegantly bridge domains is still under-explored. In this work, we resort to data mixing to establish a deliberated domain bridging (DDB) for DASS, through which the joint distributions of source and target domains are aligned and interacted with each in the intermediate space. At the heart of DDB lies a dual-path domain bridging step for generating two intermediate domains using the coarse-wise and the fine-wise data mixing techniques, alongside a cross-path knowledge distillation step for taking two complementary models trained on generated intermediate samples as ‘teachers’ to develop a superior ‘student’ in a multi-teacher distillation manner. These two optimization steps work in an alternating way and reinforce each other to give rise to DDB with strong adaptation power. Extensive experiments on adaptive segmentation tasks with different settings demonstrate that our DDB significantly outperforms state-of-the-art methods.

Deliberated Domain Bridging for Domain Adaptive Semantic Segmentation
Lin Chen Zhixiang Wei Xin Jin Huaian Chen
Miao Zheng² Kai Chen?
Yi JJi
University of Science and Technology of China
2 Shanghai Al Laboratory Eastern Institute for Advanced Study
{chlin, zhixi srixianngeei@dmill wst.. edu. en, jinxindeias ac. anchen@mail www.ust...... edu.c. {zhengmiao, chenkai]@pjlab. .rg. en, jinyi080ustc. eduoeeussttuuuuu.e...o...
Abstract
In unsupervised domain adaptation (UDA), directly adapting from the source the target domain usually suffers sigmificant discrepancies and leads to insufficient alignment. Thus, many UDA works attempt to vanish the domain gap gradually and softly via various intermediate spaces, dubbed domain bridging (DB). However, for dense prediction tasks such as domain adaptive semantic segmentation (DASS), existing solutions have mostly relied rough style transfer and how to elegantly bridge domains still under-explored. establish deliberated domain bridging (DDB) for DASS, through which the joint distributions of source and target domains are aligned and interacted with each the intermediate space. At the heart of DDB lies dual-path domain bridging step for generating two intermediate domains using the coarse-wise wise and the fine-wise data mixing techniques, alongside cross-patt knowledge distillation step for taking two complementary models trained on generated intermediate samples as 'teachers' to develop superior 'student' in multi-teacher distillation manner. These two optimization steps work in an alternating way and reinforce each other give rise DDB with strong adaptation power. Extensive experiments on adaptive segmentation tasks with different settings demonstrate that our DDB significantly outperforms state-of-the-art methods. Code is available https: // //github. com/ xi wiacchenn git.
this work, we resort to data mixing to
Introduction
When training deep models domain but applying it to other unseen domains, its performance typically drops seriously due to the domain shift/discrepancy issue [45, 44, 69,55), 55]. Since annotating data in the new scenario re-train model to mitigate performance degradation too expensive and time consuming, extensive researches have resorted to unsupervised domain adaptation (UDA) [41, 15, 34, 3], which aims transfer knowledge from labeled source domain unlabeled target domain.
Generally, existing UDA methods typically reduce the domain discrepancy by leveraging information statistics metrics [10, 27, 29, 33, 35, 46, 66] adversarial training [15, 30, 34, 49, 54, 58, 3]. Both of these branches directly adapt the knowledge learned from the source domain to the target domain. However, excessive/continuous domain discrepancies tend to limit the efficiency of these methods for knowledge transfer, causing non-optimal performance, especially on dense prediction tasks such as semantic segmentation.
'Equal contribution. Correspoding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Source domaln
Intermediate Domain
Intermediate Domain
(b) Local replocc replacement
Domain Bridging
A use contentual object information
(b11 Region-level mix
Pixel-wise ambiguity.
rider car ?
use inherent object property
Target Domain
(a) Global Interpolation
Class-level
Figure Different domain bridging ways for solving the domain domain-adaptive semantic segmentation (DASS) task. (a) the global interpolation based mix may cause an unexpected pixel-wise ambiguity issue; (b) the local replacement mix techniques better preserve the semantic consistency of pixels of the same object, which is important for segmentation. The coarse region-level local replacement helps the model exploit contextual information, while the fine class-level local replacement enables the model to exploit the inherent properties of each object, making them distinguishable.
To address this issue, some recent approaches tend to mitigate the excessive domain discrepancies by gradually transferring knowledge across domains by constructing intermediate domains in the input space [41, 60, 61], feature space [11, 12], or output space (self-trainin training based) [64, 65]. Such mechanism of constructing intermediate domain is usually termed domain bridging (DB). Despite unprecedented advances achieved in UDA classification task, the most common DB approaches, such as CycleGAN [72] and Color'rransfer [47] that based style transfer, are inapplicable and cannot achieve satisfactory adaptation performance for the domain adaptive semantic segmentation (DASS). These style transfer-based DB approaches
prone to generate unexpected artifacts in the global input space and also ignore to bridge in the label space, which makes the optimization constraints insufficient for the dense prediction tasks such DASS [16, 22, 67, 68, 71J. Therefore, such dilemma drives an urgent demand to investigate a new DB method for the densely predicted DASS area.
In this paper, to consistently construct intermediate representations in the input space as well as the label space for domain bridging in DASS, we resort to the mix-based data augmentation techniques(63,
62, 18, 14, 43]. First, we study the existing data mix methods and group them into two categories according to their working way: global interpolation [63] and local replacement [62, 18, 14, 43, 71]. As shown in Fig. 1, the global interpolation based data mix may cause the pixel-wise ambiguity issue, but the local replacement based mix methods can better preserve the semantic integrity/consistency of objects for segmentation. Next, to fully exploit the local replacement based data mix for domain bridging, we further deeply explore from two complementary perspectives: the coarse region-level mix (e.g., CutMix [62], FMix [18]) and the fine class-level mix (e.g.. ClassMix [43]). We can also see from Fig. that the coarse region-level domain bridging helps the model to exploit contextual information, reducing semantics confusion (e.g., category confusion between objects such as 'truck, bus, and train'). Complementarily. the fine class-level domain bridging enables the model to fully exploit the inherent properties of each category, making each object distinguishable. However, these two groups of DB methods tend to drive the model to overly dependent on contextual information or inherent properties, causing class bias and confusion in the target domain separately.
In this work, we propose a powerful DASS method called Deliberated Domain Bridging (DDB) to carefully take advantage of data mixing techniques and gradually transfer knowledge from the source domain the target domain. As an optimization strategy, DDB consists of two alternating steps, i.e., Dual-Path Domain Bridging (DPDB) and Cross-pate Knowledge Distillation (CKD). In the first step, DPDB independently leverages the coarse region-level data mix and fine class-level data mix to construct two complementary bridging paths to train two expert teacher models, achieving granulartyy domain bridging. In the second step, CKD uses two complementary teacher models to guide one identical student model on the target domain, achieving adaptive segmentation. These two optimization steps work in an alternating way, which allows the powerful teacher and student models to reinforce each other progressively based the joint distributions of source and target domains. The main contributions of this paper are summarized as follows:
To the best of our knowledge, this is the first work that provides a comprehensive analysis w.r.t the recent domain bridging techniques when directly applied the task of domain- addativee semantic segmentation (DASS).
2

---

Based on the analysis, we propose an effective DASS method called Deliberated Domain Bridg- ing (DDB), which consists of two alternating steps Dual-path Domain Bridging (DPDB) and Cross-path Knowledge Distillation (CKD). These two optimization steps promote each other and progressively encourage two complementary teacher models and a superior student model (used for inference), achieving win-win effect.
We experimentally validate the superiority of our DDB not only in the single-source domain setting but also in the multi-source and multi-target domain settings and conclude that DDB outperforms previous methods tailored for each setting by a large margin.
2 Related Work
Domain Adaptive Semantic Segmentation (DASS). This task aims to improve the adaptation performance for the semantic segmentation model to avoid laborious pixel-wise annotation in new larget scenarios. The recent DASS works can be mainly grouped into two categories: adversarial training based methods [23, 51-53, 37] and self-training based methods [24, 73, 70, 64, 65]. For the first branch, most works tend to learn fomain-invariant representations based min-max max adversarial optimization game, where a feature extractor is trained to fool a domain discriminator and thus helps to obtain aligned feature distributions [51 [51-53,37] The second branch focuses on how to generate highly reliable pseudo labels for the target domain data for further model optimization, which drives many classic related techniques, such confidence regularized pseudo label generation [73, 70] and category-aware pseudo label rectification [64, 65]. These two branches of the DASS task both directly adapt the knowledge learned from the source domain to the target domain. However, the large continuous domain discrepancies in DASS make such direct discrepancy minimization paradigms difficult, due to the fine-grained pixel-wise gap among different domains.
Domain Bridging (DB). Instead of directly transferring knowledge from the source domain to the target domain, some UDA works in the tasks of classification and person re-identification tend to gradually transfer knowledge by building a bridge between source and target domains, iee. constructing intermediate domain on the image level [57, 41], on the feature level [11, 12), or on the output level [64, 65]. Representatively, GVB [11] designs a gradually vanishing bridge and inserts it into the task-specific classifier and the domain discriminator invariant representations, reducing the knowledge transfer difficulty. Along this road, some works [57, 60, 41, 12, 5, 50] resort to style transfer techniques [6. 22, 7, 16] and data mix techniques [63, 62, 43] for constructing various intermediate domains. However, the existing DB approaches have not yet been extensively investigated in DASS. this paper, we first perform comprehensive analysis w.r.t the recent DB techniques and find the complementarity between the coarse region-level DB and the fine class-level DB methods, then deliberauely/carefully apply these two DB methods help the task of DASS.
construct intermediate domain-
3 Deliberated Domain Bridging
3.1 Recap of Preliminary Knowledge
For domain adaptive semantic segmentation (DASS), we denote the source domain as D. ((=!5) y/ (1) ) with N samples drawn from the source domain S, where x0 X, is im- age, y/ (8) E Y2 is the corresponding pixel-wise one-hot label covering K classes. Similarly, the unlabeled target domain set is denoted as D, [=
Ji=1 with N' samples drawn from the target domain T. Note that the source and target domains share the same label space. This work aims to learn segmentation model for effectively transferring knowledge from the source domain to the target domain, finally achieving reliable pixel-wise predictions on the target data. Following previous works (64, 65], this segmentation model M consists of feature extractor that maps the image the feature space and a classifier that generates corresponding pixel-wise predictions.
3.2 Exploring Domain Bridging for DASS
Revisiting Existing DB Methods. As mentioned in Sec. 2, the previous DB methods are mainly based on style transfer [72, 47], global interpolation based mix [63], and local replacement based mix [62, 18, 14, 43, 71]. Formally, the image-level style transfer-based DB methods can be formulated
3

---

Table 1: Performance (mloU) comparison of different DB methods GTAS -ityssppsss S TT denotes that we translate the image from the source domain to the target domain. Pseudo Labeling represents the constructed self-training baseline without any DB method. + indicates that both are used, while @ indicates that these methods will be used mutually with an equal probability. The best score is indicated in underlined bold.
(A) Comparison style transter based DB methods
Comparison combined DE metheds of differ- (b) Compasieen of global bleading based and en groups.
Method
mloU
region-hased DR methods.
Methol
mloli
Sourse only
26.3#0.9
Method
mloU
Labeling
30.7.004
CycleGAN (72] -T)
37.8+-0.4
Pseudo Labeling
30.7++0.4
+ CusMix CycleGAN (S-T)
1.
Color Transfer [47] (S-TT)
38.7+1.2
+ Mixup [63]
31 0..6
ChoaMia + CycleGAN (S-T)
53.911.0
FDA (S-T)
41,3+0.6
CowMix [14]
50.7=0.4
ClassMia + FDA (S-T)
CusMix FDA (S-T)
10.6::1.1
16.811.2
Pscudo Labeling
30.7+0.4
FMix [18]
50.0:0.2
CowMix CutMix
51.740.4
CycleGAN (T-S)
28.920.5
CutMix [62]
54.9:10.2
PMix @ CurMix
50,6+0.7
Color Transfer (T--S
31.420.5
ClassMix (43]
54.3+114
PMix @ ClassMix
54.5+0.5
FDA (T-S)
42.6+0.6
CarMix ClassMix
55.2+1.0
as,
Zx-t = (xx)), Te-s (xx),
where h (-) and // (-) represent the S T translation function and - S translation function, respectively. Such style transfer-based DB approaches tend to generate unexpected artifacts at the image level and also ignore the influence of pixel-wise label correspondence.
In addition, we formalize the global interpolation mix based DB methods as,
Inew À F. + (1 11--8 A) It Vnew A y + (1 - A)- yr.
(2)
where À denotes the mixing ratio sampled from a beta distribution. Furthermore, the local replacement mix based DB methods are formulated as,
T. new = M ② x, + (1 M) O
Ynew M O y, + (1 M) O ye+
(3)
where M denotes binary mask indicating which pixel needs to be copied from the source domain and pasted to the target domain, 1 is a mask filled with ones, and O represents the element wise multiplication operation. Y/ represents the pseudo label for target domain. In particular, this local replacement DB contains two types of coarse region-level mix and fine class-level mix. As shown in Fig. 1(b), the binary mask M of the former is the cut patch [62, 18, 14] while M of the latter is obtained from the pixel-wise annotations in source domain [43].
Analyzing DB Methods with Toy Game From the above formalization, interpolation based and local replacement- basse DB methods both build bridges across the cross- domain joint distributions of input data, which can benefit the densely predicted DASS task. To verify this, we perform toy game with a simple self self-training based DASS pipeline following [50, 24] to evaluate the performance w.r.t semantic segmentation of different DB methods. As illustrated in Tab. 1 (a) and (b), although style transfer based and global interpolation based DB methods both outperform baseline (i.e., the source only scheme), they are pronouncedly inferior to their local replacement baseed counterparts. This implies that for the DASS task, (1) it not only needs to construct an intermediate domain the input space, but also the label space; (2) the local replacement based DB methods more suitable for segmentation because they can better preserve the semantic integrity/consistency for pixels belonging the same object.
that the global
In addition, Tab. (b) shows that the performance of coarse region-level CutMix [62] and fine class-level ClassMix [43] are comparable. Thus, we further conduct group of tests by combining different DB methods for a deeper study. The results are shown in Tab. (c), we can observe that (1) due the unexpected artifacts, the segmentation performance is degraded when region-level DB methods are combined with style transfer ones; (2) we surprisingly notice that the coarse region-level and fine class-level domain bridging methods can mutually reinforce/promote each other.
Analyzing the Local Replacement Based DB Methods with Visualization. For the coarse region- level data mix methods (e.g., CutMix [62]), those pixels (i.e., patch) pasted to the target domain

---

(a) Target Image/CTT
(b) Source Only
(c) Region level based
(d) Class level based
Figure 2: Visualization of the qualitative results on GTA5 - Cityscapes benchmark (boltom row), and feature space the val set utilizing UMAP [39] (top row). For clear analysis of the feature space, we select two pairs of categories suffering class bias and confusion, i.e, salmon for road, green for sidewalk, gray bus, and purple for train.
usually have a rich contextual information. For example, the pixel beneath 'person' must belong to the 'sidewalk'. However, the area beneath a 'person' is more likely to be 'road' in the target domain. Thus, such excessive hard reliance on context may introduce the issue of class bias from the source domain to the target domain, e.g., 'road' and 'sidewalk' in Fig. (c). In contrast, the fine class-level bridging method (e.g., ClassMix [43]) only pastes a set of pixels belonging to the same class to the target domain image, avoiding class bias issue, which also drives the model to discriminate different objects solely based on their inherent properties, leading to a more compact feature distribution in Fig. 2 (d). But, the scheme that only relies on the characteristics of each category may be confused by the classes that can easily be distinguished by the context, e.g., 'train' and 'bus' in Fig. 2 (d). All in all, both coarse- -grrined and enaiine DB methods have their own advantages and drawbacks, and
thus there is an urgent need to find a appropriate way to combine them to achieve win-win effect.
3.3 Progressively learning from Dual-grained Domain Bridging
Instead of directly combining individual DB methods did in the bottom of Tab. I(c), we propose an alternating optimization strategy to progressively transfer knowledge from the source domain to the target domain, which consists of two steps, i.e., Dual-Path Domain Bridging (DPDB) and Cross-paie Knowledge Distillation (CKD). These two steps are conducted iteratively, and the ending of each round will serve as the beginning for the next round (see detailed algorithm in supplemental material).
Dual-Path Domain Bridging (DPDB). To better preserve and exploit the advantages of the coarse region-level and fine class-level DB methods, create bridging paths for them independently rather than simply fusing them. Based on previous analysis and experiments, we utilize the cross-domain CutMix [62] and ClassMix [43] techniques to construct the coarse region-path (CRP) and fine class- path (FCP) domain bridging, respectively. The self-training pipeline then proceeds in the following manner along each path (here we take the coarse region-path (CRP) as example for illustration):
Following [2], to minimize the empirical risk on the unlabeled target domain, we simultaneously minimize the empirical risk the source domain and mitigate the domain discrepancy. The first item is achieved with a pixel-wise cross-entropy (CE) loss,
Hx W K y log Mc(x,)
SPC =
Σ
Σ
(4)
where Mc is the model training on the coarse region-patt path and H, W denote the sample height and width, respectively. To mitigate the domain discrepancy. we minimize the CE loss on the constructed bridging path instead of adversarial training or minimize the predefined discrepancy metric. Considering the fact that the unlabeled target domain images are involved in the bridging path construction, an additional teacher network M employed to generate denoised pixel-wine -xiee pseudo-label jt for x, through the exponential moving average (EMA) based on the weights of Mc after each training step t,
8 g+1 M
+- at
M/ + (1 11l-a) 0000 8M
(5)
where a denotes the momentum and is set to 0.99. Based on Eqn. 3, we can generate the bridging image Zerp and label Dopp the coarse region-path. In line with [50, 43], confidence-ber weight
5

---

map merp will be generated to regularize the target domain during the training process as follows,
Mary = M O + (1 M) C m,
(6)
BxW max M ((a)) denotes the ratio of pixels that exceed a threshold r on
where my the maximum softmax probability and represents the Iverson bracket. Then, we minimize the CE loss on the region-path -atth bridging.
H-W
HxW
K
Con = Σ
Σ
(iJ) glis)) log (C (6.))
m
Furthermore, the overall objective function of the self-training pipeline on the region-path bridging is summarized as,
= + f Loe (Mc (Ix)) , vs) Loo (Me (Ferp) Marp Herp)
Similarly, can obtain the overall objective function LF the fine class-path (FCP) bridging. By minimizing Lc and Lp separately for the coarse region-path and fine class- plass-path, we can obtain two complementary models. The next problem that needs to be addressed is how to appropriately integrate these two kinds of complementary knowledge in an elegant manner.
Cross-path Knowledge Distillation (CKD). Inspired by previous works [17, 21], knowledge can be transferred from teacher network to a student network by knowledge distillation the output space. Here, we reform it to extract knowledge from two complementary teachers and adaptively transfer the integrated knowledge to a student. Note that we only integrate and transfer the complementary knowledge in the unlabeled target domain. Specifically, the outputs of two teachers have been adaptively weighted and ensembled as guidance to drive the student model to learn segmentation in the unlabeled target domain. Furthermore, we experimentally choose the 'hard' distillation, i.e., ensembling the predicted softmax logits to generate a one-hot vector and utilizing the CE loss for supervising the student model Mg. The detailed distillation loss can be written as,
Caistsl = Lee (Ms (I:"0), ..e)
(9)
where =t aug represents the target images augmented by color jitter and gaussian blur, and ft is obtained by weighted ensembling the teachers' softmax logits of the target image rı. Intuitively, different samples, even different pixels in one sample, require different contributions from the two teacher models for ensembling. We adaptively generate a pixel-wise weight map w(()) in the target domain by calculating the distance pixel-by-pine between feature response f(()) before the classification layer and the centroid وy(ر) of each category. At each location, the closer the feature response is far from one centroid of certain category, the more likely it belongs to that category and thus should contribute more to the ensemble effect. Therefore, taking the coarse region-path (CRP) an example, we first (.())
utilize the trained Mc to calculate the centroid of each category in the target domain,
EnEE, Ee II(I ==
( 1C
(10)
EmEX. E, (. == 1)
Then, we define the adaptive ensemble weights w (t.)) of Mc as the softmax over feature distances to the centroids
exp(- (5) I) Σn expl- 11
(i,)) WC
(11)
(6.) Similarly, can also obtain the ensembling weight of the other path wF of MF. Furthermore, can obtain the pseudo-label Tr. following the weighted ensembling,
Tt arg max( WC (M))( + wr (Mr(s)))
(12)
2
where a denotes the softmax function. In addition, the student model is also supervised by the labeled source data to generate discriminative features
S Loe (Ms(
(13)

---

Ground Truth
Source only
region-path
Fine dlss--path
Final student
Figure 3: Comparison of qualitative results GTAS -> Cityscapes (top area), GTA5 +Synscapes Cityscapes (middle area), GTAS Cityscapes + Mapillary (bottom area) benchmarks.
Table 2: Comparison results of GTAS Citysscpes adaptation (using ResNet-101 as the backbone and DeepLabv2 as the head) in terms of mloU. distill denotes applying multi-rounds self-distillation for the student network initialized by self-supervised pre-training. The best score indicated in underlined bold.
Sddewnlk
huidding
pole
light
38m
purson
rider
mmuck
bus
train
bike
Metbod
mloL
only
16.8
CyCADA [22]
86.7
35.6
80.1
19.8
17.5
38.0
39.9
41.5
82.7
27.9
73.6
64.9
19.0
65.0
12.0
28.6
4.5
31.1
42.0
42.7
ADVENT [52]
33.1
81.0
26.8
27.2
33.5
24.7
36.7
78.8
30.5
84.8
38.5
44.5
1.7
31.6
32.4
45.5
BDL [311
91.0
44.7
84.2
34.6
27.6
30.2
36.0
36.0
85.0
43.6
$3.0
58.6
31.6
83.3
35.3
49.7
3.3
28.8
35.6
48.5
FADA [53]
91.0
50.6
86.0
43.4
29.8
36.8
43.4
25.0
86.8
38.3
87.4
38.0
85.2
46.1
6.5
25.4
37.1
50.1
CAG]66]
51.6
27.8
25.3
48.4
78.1
34.6
84.7
21.9
42.7
41.1
37.2
50.2
IAST [10]
93.8
57.8
85.1
39.5
26.7
26.2
43.1
34.7
81.9
32.9
88.0
62.6
29.0
87.3
39.2
49.6
23.2
34.7
39.6
51.5
DACS [50]
SAC[1] [1]
90.4
53.9
86.6
42.4
27.3
45.1
48.5
42.7
N7.4
40.1
86.1
67.5
29.7
88.1
49.1
54.6
9.8
26.6
45.3
53.8
CTP
58.3
86.5
27.4
28.8
38.1
42.5
85.4
38.4
91.8
37.0
87.8
52.4
44.6
$9.0
56.1
ProDA [64]
91.5
52.4
82.9
42.0
35.7
40.0
44.4
43.8
N7.0
43.8
79.5
66.5
31,4
86.7
41.1
52.5
0.0
45.4
53.8
53.7
ProDAediztll
87.8
56.0
79.7
46.3
41.8
45.6
53.5
53.5
88.6
45.2
82.1
70.7
39.2
88.8
45.5
59.1
1.0
48.9
56.1
57.5
UndoDA [33]
89.1
34.3
83.6
38.3
27.5
28.9
34.7
17.6
84.2
41.0
85.1
57.8
33.7
85.1
41.3
30.7
31.1
48.0
49.0
UndeIDA-PooIA
92.9
52.7
87.2
39.4
41.3
43.9
55.0
52.9
89.3
48.2
91.2
71.4
36.0
90.2
67.9
59.8
0.0
48.5
49.3
59.3
CPSL
37.6
59.9
48.4
55.7
CPSI Adispil
92.3
59.9
84.9
45.7
79.7
52.8
61.5
59.5
W1.9
41.5
85.0
73.0
35.5
90.4
48.7
73.9
26.3
53.8
53.9
60.8
Source only
22.4
22,1
DDB(Ours)
95.3
67.4
89.3
44A
45.7
38.7
54.7
55.7
88.1
40.7
90.7
70.7
43.1
92.2
60.8
67.6
34.2
48.7
63.7
62.7
The overall loss function of constraining the student model Mg can be written as,
Ls CS + Ldistill = Loe (ce(MM)())))))) yo) + Loe (Ms(x) Dt).
(14)
In the end, superior student model is well trained by adaptively integrating the knowledge from two complementary teacher models covering different gramularities.
Alternating Optimization Strategy. By integrating the complementary knowledge from two expert teacher models, we can obtain superior student model. In turn, this student model can be used to initialize the teacher models in the next new round, resulting in two stronger teacher models. They promote each other, achieving a win-win win effect. Ultimately, we can obtain the most powerful student model after the final round.

---

Table 3: Comparison results of GTAS (G) + Synscapes (S) +
Cityscapes (C) adaptation (using ResNet- 101 as
the backbone and Deepl.ane .av2 as the head) in terms of mloU. The best score is indicated in underlined hold.
sidewalk
building
road
Wall
polo
Hight
sky
rdeer
muck
tran
bike
Method
mloU
Source
85.1
36.9
84.1
39.0
33.3
38.7
43.1
40.2
84.8
37.1
82.4
65.2
37.8
69.4
43.
38.8
34.6
33.2
53.1
51.6
AdaptSeg [S1]
89.3
47.3
83.6
40.3
27.8
39.0
44.2
42.5
86.7
45.5
84.5
63.1
38.0
79.4
34,9
48.3
42.1
30.7
52.3
53.7
ADVENT [52]
91.8
49.0
34.6
19.4
31.5
39.9
42.9
43.5
86.3
45.1
84.6
65.3
41.0
87.1
37.9
49.2
31.0
30.3
48.8
54.2
MDAN [67]
92.4
56.1
86.8
42.7
32.9
39.3
48.0
40.3
87.2
47.2
90.5
64.1
35.9
87.8
33.8
48.6
39.0
27.6
49.2
55.2
MADAN [68]
94.1
61.0
86.4
43.3
32.1
40.6
49.0
44.4
87.3
47.7
89.4
61.7
36.3
87.5
35.5
45.8
31.0
33.5
52.1
55.7
MSCL [19]
59.6
44.9
47.6
53.1
57.7
Source only
82.5
42.4
79.0
27.2
31.7
40.8
53.0
45.6
85.3
30,9
80.6
68.7
35.7
78.3
39.0
42.7
9.6
37.3
55.9
50.9
DDB(Ours)
75.6
54.4
47.6
66.3
48.4
74.4
52.7
92.3
58.9
71.4
Table 4: Comparison results of GTA5 (G) -
Cityscapes (C) + Mapillary (M) adaptation (using ResNet- 101 as the backbone and DeepLabv2 as the head) in terms of mloU. The best score is indicated in underlined bold.
sidewale
Method
Target
wall
pole
rddrr
mack
bus
m0000
mloll
Avg:
Sounce only
M
55.7
53.3
27.1
15.2
55.3
56.6
82
9.9
20.6
26.2
22.7
21.2
33.3
ML.7
31.6
22,2
68.4
76.3
21,1
9.3
$3.3
70.6
53.5
55.3
13.5
30,9
727
72.9
21.5
32.3
11.6
4.9
0.9
5.6
20.2
36.3
14.9
7.4
35.5
30.1
32.8
C25
45.1
46.8
48.8
[26]
81.2
33.9
78.5
25.5
21.5
35.6
30.5
39.8
350
52.4
71.2
40.2
92.4
58.7
38.7
52.7
44.4
38.6
46
16.1
152
0.5
37.8
32.2
45.8
19.2
47.5
DDB(Ouns)
93.5
67.8
88.3
38.4
38.4
45.6
32.9
32.3
54.2
57.9
89.2
48.6
91.6
69.1
43.2
84.6
63.6
61.8
15.1
44.1
58.6
60.4
56.9
58.6
4 Experiments
4.1 Experimental Settings
Datasets: We use four publicly available semantic segmentation benchmarks for validation, including two synthetic scenes and two real-world scenes. Each scene has a unique structure and visual appearance. In detail, GTA5 [48] is a synthetic dataset of 24,966 labeled images obtained from video game. Synscapes [56] is also a synthetic dataset of 25,000 images created by photo-realistic rendering techniques, and its style is closer to real-world driving scenes than GTA5. Cityscapes [9] is a real-world urban dataset collected from European cities, with 2,975 images for training and 500 images for validation. Mapillary Vista [42] is a large-scale dataset collected by various imaging devices worldwide and includes 18,000 images for training and 2,000 images for validation.
Implementation details: We use the mmsegmentation [8] codebase and train models RTX 3090Ti GPUs. Following previous works [64, 65, 24, 59], we use the advanced DeepLab-v2 [4] model with ResNet101 [20] pre-trained ImageNet- 1k [13] backbone, and train the model with AdamW [36]. We set the learning rate as 6e-5 for the backbone and 6e-4 for the decoder head, use a weight decay of 0.01 and a linear learning rate warmup followed by 1.5k iterations linear decay. All experiments trained on a batch of 51 512x112 random cropped images for 40k iterations. We set the batch size to for analysis and experiments in Tab. I and Tab. and set batch size to 4 for other results. Following [50], we use the same augmentation parameters and set 7 0.968. For CutMix [62], the ratio of the selected region for cross-domain pasting is experimentally set to 0.3. For ClassMix [43], half of categories in the source domain are selected for cross-domain pasting.
4.2 Comparison with State-of-the-arts under Multiple Settings
GTA5 + Cityscapes (single-source). Tab. reports results on the validation set of Cityscapes. Note that the comparable ProDA [64], UndoDA [32], and CPSL [28] have been improved with warmup stage following existing DASS methods [51, 53]. Additionally, they also need complete numerous rounds of self-distin iif-distillation for the student model initialized by self-supervised pre-training. Compared to these methods that require redundant optimization processes, the proposed DDB method requires only two alternating optimization steps of DPDB and CKD. Despite simplicity, our method achieves a SOTA mloU score of 62.7, outperforming existing methods significantly, which also achieves the best IoU score in 7 out of 19 categories. Particularly, thanks to combining the complementary teacher

---

TTable S: Ablation studies on the key components of our proposed method the GTAS (G) - Cityscipes (C) benchmark. represents using the 'soft distillation' if the hhard distillation' column isn't ticked.
Table 6: Comparison results of ensemble from single and dis- tinct paths terms of mloU on GTAS (G) - Cityscupes (C). M and M p denote the lirst for the and fine class
region path class-path path model, sepurttly. The best scare is indi-
compencnts
mloU gain
cated underlined bold. And the cross-path ensemble indicated in italic.
source only
32.1
region path
path
mioli gain
Mc
Mg
stage DPDB
56.5
+24.4
M4
55.5
58.2
+26.1
M;
55.9
55.8
adoptive distillation ensemable
Mg
56.4
568
55.0
miloU gain
.42
55.7
51.7
stage
59.0
+26.9
CKD
59.9 +27.8
61.1 +29.0
Table 7: Performance comparison (mlot)) of the student model obtained from CKD during various rounds the whole training process on three benchmarks. The best score is indicated in underlined bold.
61.2 +29.1
region path
class
gain
path
stage
DPDB
61.4 +29.3
62.6 +30.5
hard
adaptive emsemble
G-C
32.1
61.2
62.7
62.7
stage
distillatinn
gain
G+S C
50.9
68.6
69.0
68.8
CKD
62.7 +30.6
G
32.8
57.4
58.6
58.2
models devoted to exploiting the context and inherent properties by the coarse region-patl and fine class-path, the student model performs surprisingly well in those categories that are susceptible to the class bias issue, e.g., 'sidewalk' and 'bike.' Additionally, the final student model also performs well in those categories suffering from semantic confusion, e.g., 'person, rider' and 'truck, bus, train.'
GTA5 + Synscapes - Cityscapes (multi-source). We also perform experiments under the multi- source domain setting. As shown in Tab. 3, our method obtains an impressive performance of 69.0 in mloU, outperforming the previous SOTA methods over 10.0, achieving the best performance in all classes, especially those suffering class bias issue, e.g., 'sidewalk' and 'bike.' Although the proposed DDB not tailored for multi-source DASS, our method still benefits this task by constructing intermediate domains between target and multiple source domains to facilitate knowledge transfer.
GTA5 > Cityscapes + Mapillary (multi-target). Tab. 4 displays the performance of the proposed method in a multi-target domain setting. Although the multi-target DASS is more challenging due to unknown distributions, our method can still achieve an impressive performance of 58.6 in mloU on average for multiple target domains, outperforming the existing SOTA methods by a significant margin. Moreover, the proposed DDB outperforms ADAS [26] by 31.2 in averaged mloU on the 's, by 14.1 in averaged mloU on the 'bus, train,' which indicates our method avoids the class bias and confusion issues. Such substantial performance gains comes from the DPDB-driven complementary teacher networks and the CKD-driven knowledge integration.
4.3 Ablation Study and Detailed Discussion
Complementarity Verification. To verify the complementarity of two teacher models trained on different bridging paths, we conduct an ablation where we train each path twice separately to obtain four different models. After ensembling these models pair-by-pain the segmentation results presented in Tab. 6. Unsurprisingly, the ensembled models across paths consistently perform better results than those only from a single view.
Study on Dual-path Domain Bridging, In Tab. 5, the source-only model achieves 32.1 mIoU on the target domain. Combining the self-training pipeline with the coarse region-level and fine class-level domain bridging, we can obtain two complementary teacher models, and they achieve 56.5 and 58.2 in mIoU, respectively. As shown in Fig. 3, the coarse region-pati tends to promote the model to utilize contextual information for prediction, whereas the fine class-path enables the model to focus more exploiting inherent properties. Detailed results for the two teacher models in each category are provided in the supplemental materials.
Study on Cross-path Knowledge Distillation. Since CKD is performed on the unlabeled target domain, as shown in Tab. 5, we use the more stable hard distillation, which performs 0.9 mloU higher than the soft distillation using the Kullback-Leebbe divergence. Furthermore, our proposed adaptive ensemble scheme further improves the performance by 2.1 and 1.3 in mIoU in the case of soft and hard distillation, respectively. After applying CKD equipped with the hard distillation and adaptive

---

ensemble schemes, we can consistently obtain superior student model. Fig. 3 illustrates how the student model performs after integrating the knowledge from two complementary teacher models and alleviates the class bias and confusion issues in various domain settings.
Influence of Alternating Optimization Strategy. As shown in Tab. 5, the alternation of DPDB and CKD allows the complementary teacher and student models to promote each other and gradually transfer knowledge across domains. We also test different alternating rounds on all three benchmarks, and report the performance of the student models after each round in Tab. 7 (more results are provided in the supplemental materials). The student model performs best across all three domain settings in the second round. On the other hand, the student model shows a slight performance degradation after the third round of alternate training in the multi-source and multi- target domain settings. We analyse the degradation is because the non-negligible domain conflict in these two settings.
5 Conclusions
In this paper, we study how the domain bridging techniques should be applied domain adaptive semantic segmentation. To ensure that the segmentation model takes full advantage of domain bridging while avoiding side effects, we propose an effective Deliberated Domain Bridging (DDB) method. We build dual-path domain bridging (DPDB) with the coarse region-level data mix and fine class-level data mix to construct two complementary teacher models. Then, a superior student model can be generated from cross-path distillation (CKD) based on such two teacher models. By alternating steps of DPDB and CKD, teacher models and student model would promote each other and progressively transfer knowledge from the source domain to the target domain. Extensive ablation studies demonstrate the effectiveness of our method, and the experimental results on three benchmarks in the different settings further show its versatility and robustness.
6 Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 61727809, in part by the Special Fund for Key Program of Science and Technology of Anhui Province under Grant 201903c08020002, and in part by the National Key Research and Development Program of China under Grant 2019YFC0117800.
References
[1] N. Araslanov and S. Roth. Self-supervised augmentation consistency for adapting semantic segmentation. In CVPR, pages 15384-15394, 2021.
[2] S. Ben- Ben-David, J. Blitzer, K. Crammer, and F. Percira. Analysis of representations for domain adaptation. NewrIPS, 19, 2006.
[3] L. Chen, H. Chen, Z. Wei, X. Jin, X. Tan, Y. Jin, and E. Chen. Reusing the task-specific classifier as a discriminator: Discriminator free adversarial domain adaptation. In CVPR, pages 7181-7190, 2022.
[4] L.-C. Chen, G. Papandreou. I. Kokkinos, K. Murphy, and L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4):834- 848, 2017.
[5] S. Chen, X. Jia, He, Y. Shi, and J. Liu. Semi-supervised domain adaptation based on dual-level domain mixing for semantic segmentation. In CVPR, pages 11018-1102) 2021.
Y.C. Chen, Y.-Y Lin, M.-H. Yang, and J.-.. Huang. Crdoco: Pixel-level domain transfer with cross-domain consistency. In CVPR, pages 1791-1800, 2019.
[7] J. Choi, T. Kim, and C. Kim. Self-ensembling with gan- data augmentation for domain adaptation in semantic segmentation. In /CCV, pages 6830-6840, 2019.
[8] M. Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: 2020.
// thub. wwtp://..ccocececceeeeeooo wttpp/enwwmmaaae//eeeeeeeeeeeeee
[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding, In CVPR, pages 3213-3223 2016.
10

---

[10] S. Cui, S. Wang. J. Zhuo, L. Li, Q. Huang, and Q. Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In CVPR, pages 3941-3950. 2020.
[11] S. Cui, S. Wang. J. Zhuo, C. Su, Q. Huang, and Q. Tian. Gradually vanishing bridge for adversarial domain adaptation. In CVPR, pages 12455-12464, 2020.
[12] Y. Dai, J. Liu, Y. Sun, Z. Tong, C. Zhang, and L.-Y. Duan. Idm: An intermediate domain module for domain adaptive person re-id. In ICCV, pages 11864 11874, 2021.
[13] J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In CVPR, pages 248-255, 2009.
[14] G. French, A. Oliver, and T. Salimans. Milking cowmask for semi-supervised image classification. arXiv, 2020.
[15] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. JMLR, pages 2096-2030, 2016.
[16] R. Gong, W. Li, Y. Chen, and L., V. Gool. Dlow; Domain flow for adaptation and generalization. In CVPR, pages 2477-2486 2019.
[17] J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. IJCV, 1296611789-1819, 2021.
[18] E. Harris, A. Marcu, M. Painter, M. Niranjan, A. Prügel-Bennett, and J. Hare. Fmix: Enhancing mixed sample data augmentation. In ICLR, 2021.
[19] J. He, X. Jia, S. Chen, and J. Liu. Multi-source domain adaptation with collaborative learning for semantic segmentation. In CVPR, pages 11088-10017 2021.
[20] K. He. X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[21] G. Hinton, O. Vinyals, Dean, et al. Distilling the knowledge in neural network. arXiv, 2(7), 2015.
[22] J. Hoffman, F. Tzeng, T. Park, J.-Y. Zhu, P. Tsola, K. Saenko, A. Efros, and T. Darrell. Cycada: Cycle- consistent adversarial domain adaptation. In ICMI, pages 1989-1998, 2018.
[23] J. Hoffman, D. Wang, F. Yu, and T. Darrell. Fens in the wild: Pixel-level adversarial and constraint-based adaptation. arXiv, 2016.
[24] L. Hoyer. D. Dai, and L. Van Gool. Daformer: Improving network architectures and training strategies for adapivee semantic segmentation. CVPR. 2022.
[25] T. Isobe, X. Jia. S. Chen, J. He, Y. Shi, J. Liu, H. Lu, and S. Wang. Multi-targer domain adaptation with collaborative consistency learning. CVPR, pages 1197-819, 2021.
[26] S. Lee, W. Choi, C. Kim, M. Choi, and S. Im. Adas: A direct adaptation strategy for multi- target domain adaptive semantic segmentation. In CVPR, 2022.
[27] M. Li, Y.-. Zhai, Y.-W. Luo, P.-F. Ge, and C.-X. Ren. Fihanced transport distance for unsupervised domain adaptation. In CVPR, pages 13936-13944, 2020.
[28] R. Li. S. Li, C. Hc, Y. Zhang, X. Jia. and L. Zhang. adaptive semantic segmentation. In CVPR, 2022.
pixel-level self-labeling for domain
[29] S. Li, C. H. Liu, Q Lin, Q Wen, L. Su, G. Huang, and Z. Ding. Deep residual correction network for partial domain adaptation. TPAMI, 43(7):232--2344, 2020.
[30] S. I. B. Xie, C. H. Liu, J. Liang, and C. Qin. Bi-classifier determinacy maximization for unsupervised domain adaptation. In AAAI, 2021.
[31] Y. Li, L. Yuan, and N. Vasconcelos. Bidirectional learning for domain adaptation of semantic segmentation. In CVPR, pages 6936 6945, 2019.
[32] Y. Liu, J. Deng. J. Tao, T. Chu, L. Duan, and W. Li. Undoing the damage of label shift for cross-domain semantic segmentation. In CVPR, 2022.
[33] M. Long. Y. Cao. J. Wang. and M. Jordan. Learning transferable features with deep adaptation networks. In ICML, pages 97-105, 2015.

---

[34] M. Long. Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In NearIPS. volume 31, 2018.
[35] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In ICML. pages 2208-2217. PMLR, 2017.
[36] I. Loshehilov and F. Hutter. Decoupled weight decay regularization. arXivv 2017.
[37] Y. Luo, L. Zheng, T. Guan, J. Yu, and Y. Yang. Taking closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR, pages 2507-2516, 2019.
[38] H. Ma, X. Lin, Z. Wu, and Y. Yu. Coarse-to-line domain adaptive semantic segmentation with photometric alignment and category-cente regularization. In CVPR, pages 4051-4060, 2021.
[39] L.. McInnes, Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv, 2018.
[40] K. Mci, C. Zhu, Zou, and Zhang. Instance adaptive self self-training for unsupervised domain adaptation. In ECCV, pages 415 4300, 2020.
[41] Na. H. Jung, H. J. Chang, and W. Hwang. Fixbi: Bridging domain spaces unsupervised domain adaptation. In CVPR, pages 1094-1103, 2021.
[42] G. Neuhold, T. Ollmann, S. Rota Bulo, and P. Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, pages 4990-4999, 2017.
[43] V. Olsson, W. Tranheden. Pinto, and L. Svensson. Classmix: Segmentation data augmentation for semi semi-supervised learning. In WACV, pages 1369-1378, 2021.
[44] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q Yang. Domain adaptation via transfer component analysis. TNNLS, 1992)119--100 2010.
[45] S. J. Pan and Q Yang. A survey on transfer learning. TNNLS, 22(10)) 1359, 2009.
[46] Y. Pan, T. Yao, Li Y. Wang, C.-W. Ngo, and T. Mei. Transferrable prototypical networks for unsupervised domain adaptation. In CVPR, pages 2239-2247, 2019.
[47] E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, Color transfer between images, IEEE Computer graphics and applications, 21:5):34 41, 2001.
[48] S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In ECCV, pages 102- 118, 2016.
[49] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum elassifier discrepancy for unsupervised domain adaptation. In CVPR, pages 3723-3732, 2018
[50] W. Tranheden, V. Olsson, J. Pinto, and L. Svensson. Dacs: Domain adaptation via cross-domain mixed sampling. WACV, pages 1379-1389, 2021.
[51] Y.H.. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning adapt structured output space for semantic segmentation. CVPR, pages 7472-7481, 2018.
[52] T.-H. Vu, Jain, M. Bucher, M. Cord, and Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, pages 2517-2526, 2019.
[53] H. Wang, Shen, W. Zhang. L.-Y. Duan, and T. Mei. Classes matter: fine-grained adversarial approach to cross dossin semantic segmentation. In ECCV, pages 642 659, 2020.
[54] G. Wei, C. Lan, W. Zeng, and Z. Chen. Mctaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. In CVPR, pages 16643-16653, 2021.
[55] G. Wilson and J. Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
[56] M. Wrenninge and J. Unger. Synscapes: A photorcalistic synthctic dataset for street scene parsing. arXiv, 2018.
[57] Y. Wu, D. Inkpen, and A. El-Roby. Dual mixup regularized learning for adversarial domain adaptation. In ECCV, pages 540-555, 2020.
12

---

[58] N. Xiao and L. Zhang. Dynamic weighted learning for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15242-15251, 2021.
[59] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient
design for semantic segmentation with transformers. NeurIPS, 34, 2021.
[60] M. Xu, J. Zhang, B. Ni. T. Li. C. Wang, Q. Tian, and W. Zhang. Adversarial domain adaptation with domain mixup. In AAAI, volume 34, pages 6502-6509, 2020.
[61] Y. Yang and S. Soatto. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, pages 4085 4095, 2020.
[62] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pages 6023-6032, 2019.
[63] H. Zhang, M. Cissc, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[64] P. 7hang, B. Zhang, T. Zhang. D. Chen, Y. Wang, and F. Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12414-12424. 2021.
[65] Q. Zhang, Zhang, W. Liu, and D. Tao. Category anchor-guided unsupervised domain adaptation for semantic segmentation. NeurlPS, 32, 2019.
[66] Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In ICML., pages 1404-7413. PMLR, 2019.
[67] H. Zhao, S. Zhang, G. Wu, J. M. Moura, J. P. Costeira, and G. Gordon. Adversarial multiple domain adaptation. NeurlPS, 31, 2018.
[68] S. Zhao, Li, X. Yue, Y. Gu, P. Xu, R. Hu, Chai, and K. Keutzer. Multi-source domain adaptation for semantic segmentation. NeurlPS, 32, 2019.
[69] S. Zhao, X. Yuc, S. Zhang, B. Li, H. Zhao, B. Wa, R. Krishna, J. E. Gonzalez, A. L.. Sangiovanni- Vincentelli, S. A. Seshia, al. review of single-source deep unsupervised visual domain adaptation. TNNLS, 2020.
[70] Z. Zheng and Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. IJCV, pages 1106-1122 1120, 2021.
[71] Q Zhou, Z. Feng, Q Gu, J. Pang, G. Cheng, X. Lu, Shi, and L. Ma. Context-aware mixup for domain adaptive semantic segmentation. arXiv, 2021.
[72] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pages 2223-2232, 2017.
[73] Y. Zou, Z. Yu, X. Liu, B. Kumar, and J. Wang. Confidence regularized self self-training. In ICCV, pages 5982-5991, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See supplementary materials.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zJHLKFA27ix", "reviewer": "Reviewer_XZqY", "summary": "This paper proposes an effective Deliberated Domain Bridging (DDB) for domain adaptive semantic segmentation (DASS). To this end, it takes advantage of two data mixing techniques, region-level mix and class-level mix, to train two corresponding teacher models, which eventually guide one student model on the target domain. It has been tested on several benchmarks (GTA5 to Cityscapes, GTA5 + Synscapes to Cityscapes, GTA5 to Cityscapes + Mapillary).\n", "questions": "Please answer the questions from the above weaknesses.", "limitations": "Yes, the authors addressed the limitation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n1. It is a well-written paper that addresses the limitations of previous methods (e.g., global interpolation -> pixel-wise ambiguity), toy game to show the justification of using both coarse-grained and fine-grained DB, and proposes novel learning architecture with multi-teacher and single-source distillation method.\n2. The proposed method is well proven to be effective with several benchmarks (e.g., single-source, multi-source, and multi-target settings) by widening the gap with the previous state of the arts in each benchmark.\n\n- Weaknesses:\n1. As the author illustrated in limitations, I am also a bit concerned with the training efficiency and complexity since the proposed method requires alternating optimization processes. One of the simple end-to-end optimizations is conducting EMA training on one model by combining region-level and class-level mixing techniques. It would be better to show a brief study about how the authors can extend this to end-to-end simple learning architecture.\n2. Lack of ablation study on some hyperparameters: 1) alpha in equation 5: the authors suggested updating the teacher models with EMA to avoid a denoised pixel-wise pseudo label. To prove it, an ablation study on alpha needs to be explored. 2) x_aug in equation 9: need to empirically show justification using augmentation input for a student model.    ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z4kn0K_8PW0", "reviewer": "Reviewer_PF8F", "summary": "This paper is about unsupervised domain adaptation for the task of semantic segmentation. The paper argues for the importance of gradually bridging the domain gap, instead of a direct attempt to transfer a model from the source to the target domain. Motivated by an empirical analysis about different data mixing technologies, the paper explores two region-based mixing strategies (coarse regions and finer class-wise/mask regions) as domain bridges. Specifically, two models are trained on the two different domain bridges (Dual-Path-Domain-Bridge), which act as ensembled supervision for a single student model (Cross-path knowledge distillation). This student model can then initialize the teacher models for another round of these two steps. Experimental results in three different settings confirm the effectiveness of the proposed approach with state-of-the-art results on standard benchmarks.", "questions": "Please see few questions under \"weaknesses\" above", "limitations": "Potential societal impacts are not discussed. I think a paper on domain adaptation should include a discussion about potential biases that are carried over from source domains (specifically because these are often synthetic data which often contains some hand-crafted components, like object sampling distributions, etc.).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The ablation studies in Tables 5-7 are great and showcase the impact of individual components\n- The results are impressive, with a clear improvement on the standard benchmark (GTA5->CityScapes), as well as other settings (multi-source and multi-target)\n- The experiments in Table 1 are a good motivation for the choice of data mixing strategies (CutMix and ClassMix)\n\n\n\n### Weaknesses\n\n- The writing, specifically the motivation and positioning with respect to prior work, needs improvement.\n  - There may be alternative domain bridges than data mixing, like methods that rely on self-training and choose confident target pseudo labels as intermediate source domain.\n  - The justification for exploring new domain bridges in lines 37-41 is vague and unclear: What are \"unexpected artifacts in the global input space\"? What \"optimization constraints\" are referred to?\n- I do not see why the paper only evaluates two domain bridging strategies in the ensemble. One could also include more. Relating to ensemble methods, one could expect improvements if an additional data mixing strategy is \"orthogonal\". One recent successful example for a global mixing strategy is [A] and could be easily integrated.\n- There is a related work on domain bridges for semantic segmentation that was not included: [B]\n- In line 185, shouldn't the reference go to Eq. 3?\n- I do not quite understand why the mixing weights in Eq. 11/12 help. Aren't the softmax values (i.e., scores) already an indication how far away a sample is from the decision boundary?\n- It would be good to point the reader to the supplemental material for a detailed description of the training strategy.\n- It's hard to understand and see details in Figure 1.\n\n\n**References:**\n- [A] FDA: Fourier Domain Adaptation for Semantic Segmentation. Yang and Soatto. CVPR'20\n- [B] Domain Bridge for Unpaired Image-to-Image Translation and Unsupervised Domain Adaptation. Pizzati et al. WACV'20", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XGBHrBKHI0W", "reviewer": "Reviewer_i4CE", "summary": "To address UDA in semantic segmentation, this work uses two types of data mixing strategies to artificially create intermediate bridging domains between source and target. The paper starts with a detailed analysis comparing different data mixing strategies, either done globally (mixup[61]) or locally ( CowMix [13], FMix [17], CutMix [60] and ClassMix [42]). The analysis demonstrates favorable results when using local data mixing strategies for UDA in segmentation, in particular CutMix (coarse region-wise mixing) and ClassMix (fine class-wise mixing). \n\nBased on results of the analysis, this work proposes a simple way to combine the two mixing strategies CutMix and ClassMix. \nIn the course of training, there are five models: two teacher models trained with CutMix and ClassMix, two EMA models of the two teachers, one student model trained using teachers' pseudo-labels.\n\nTraining is done in multiple rounds (fixed as 4 in the experiments). In each round:\n- The two teachers are first trained separately with CutMix and ClassMix\n- The student is then trained with pseudo-labels of two EMA models of the two teachers. Pseudo-label of a given target sample is determined as a weighted combination of softmax scores of the two EMA models (Eqn. 12). The weights have size $H \\times W \\ K$ with $K$ classes; at each spatial position, the weight vector over $K$ classes is the softmax over the feature distance to class centroids (Eqn. 11). Color jittering and gaussian blurs are used on target sample when training the student.\n- The two teachers are initialized by the student.\n\n", "questions": "- DAFormer (CVPR'22) is a recent work demonstrating great DA performance thanks to the robustness of transformer-based Segformer model. As the proposed strategy is agnostic to the model choice, I'm wondering to which extent one can push the performance by combining the proposed and DAFormer", "limitations": "Limitation on training complexity is discussed in the supplementary material.\nNo discussion on potential negative societal impacts was given.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strengths **\nOverall this is an interesting technical paper that combines multiple existing strategies, namely CutMix [60], ClassMix [42], mean teacher [42], prototypical weighting [62]  and pseudo-labelling [30]. Empirical results demonstrate better performance than previous SOTAs on comparable backbone (resnet101) and segmentation framework (deeplab-v2). Experiments are extensive. The paper is well-written and easy to follow.\n\n** Weaknesses **\n- My main concern is with the technical novelties of this work. The analysis comparing different mixing techniques, claimed as the first contribution, is somewhat interesting. However the main proposed approach is merely a mix of previous works. Actually there are no new insights that I could get from this work. \n\n- It's not clear to me how is the intermediate model selected at each stage. Is the target's validation set used to select the best model? If true, is there a risk of supervision leak from target validation set?\n\n- Missing details for the multi-source and multi-target experiments.\n\nI'm currently on the borderline, slightly leaning toward the positive side, thanks to the good results. My final decision will be adjusted based on the feedback from the authors and the discussion with other reviewers.\n\n** Typos **\n- L185: Eqn. 3 instead of Eqn. 2\n- SuppMat: Algo.1 - L7 & L12: Eqn. 3 instead of Eqn. 2\n\n========== Post-rebuttal\nI thank the authors for being active during the rebuttal and addressing all of my concerns. I'm happy to increase my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1nA60WQ20v", "reviewer": "Reviewer_u4Xd", "summary": "This paper proposes a deliberated domain bridging (DDB) method for domain adaptative semantic segmentation, where the target labels are not available during the training. In DDB, there are two parts: 1) a dual-path domain bridging step to train two teacher models with two intermediated domains using the coarse-wise and fine-wise, i.e., region-level and semantic-level, data mixing techniques. 2) a cross-path knowledge distillation step to adaptively transfer the knowledge from the two teacher models to a student model. The two steps are repeated for several rounds for a good performance. Extensive experiments on both single-source domain and multi-source multi-target domain settings are conducted to validate DDB’s superiority.", "questions": "See the weakness.", "limitations": "The authors discuss the limitations of this paper in the supplementary material. No negative social impact has been discussed. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. This paper proposes an effective method to significantly boost the UDA segmentation performance in various settings.\n2. The comprehensive ablations are done to clearly show 1) the complementarity between the two teacher models and 2) the effectiveness of the distillation step.\n\nCons:\n1. Since GTA5 to Cityscapes and GTA5 + Synscapes to Cityscapes are done, what is the performance in Synscapes to Cityscapes? This experiment shows which dataset contributes more to adapt to the real dataset.\n2. There are too many symbols, which makes the paper hard to follow. What do the numbers righter after the approach name in Tables 2, 3, and 4 mean? For example, ADVENT(19), BDL (19), FADA(20), etc.\n3. The authors claim that soft distillation and hard distillation are compared in Table 5. However, the ‘soft distillation’ choice and the explanation are missing in that table, which is a bit confusing.\n4. DDB requires two rounds for a good convergence. In each round, it needs to train three individual models and calculate two groups of category centroids by scanning the target training set for two teacher models respectively. This makes the approach cumbersome and may require more training time than others. The authors are encouraged to discuss the above issue with detailed analysis. Besides, this cumbersome training process seems in conflict with the stated ‘elegant’ method.\n5. The following paper can be included for comparison since it also studies the data mixing technique in UDA semantic segmentation. Besides, the difference between DACS which also utilizes the data mixing technique in UDA is not well stated in the paper.\n\nDsp: Dual soft-paste for unsupervised domain adaptive semantic segmentation. Proceedings of the 29th ACM International Conference on Multimedia. 2021: 2825-2833.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept