Unsupervised Domain Adaptation for Semantic Segmentation using Depth Distribution
Authors: Quanliang Wu
Huajun Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Unsupervised Domain Adaptation
Semantic Segmentation
depth density
multi-task learning
pseudo-labels refinement

Abstract: Recent years have witnessed significant advancements made in the field of unsupervised domain adaptation for semantic segmentation. Depth information has been proved to be effective in building a bridge between synthetic datasets and real-world datasets. However, the existing methods may not pay enough attention to depth distribution in different categories, which makes it possible to use them for further improvement. Besides the existing methods that only use depth regression as an auxiliary task, we propose to use depth distribution density to support semantic segmentation. Therefore, considering the relationship among depth distribution density, depth and semantic segmentation, we also put forward a branch balance loss for these three subtasks in multi-task learning schemes. In addition, we also propose a spatial aggregation priors of pixels in different categories, which is used to refine the pseudo-labels for self-training, thus further improving the performance of the prediction model. Experiments on SYNTHIA-to-Cityscapes and SYNTHIA-to-Mapillary benchmarks show the effectiveness of our proposed method.

Unsupervised Domain Adaptation for Semantic Segmentation using Depth Distribution
Quanliang Wu, Huajun Liu"
School of Computer Science
Wuhan University, Wuhan, China
fquanliangwu, huajunliu)Owhu wwt.w/wwwwu...eeeeooooeee.eeeeee cn
Abstract
Recent years have witnessed significant advancements made in the field of unsu- pervised domain adaptation for semantic segmentation. Depth information has been proved to be effective in building a bridge between synthetic datasets and real-world datasets. However, the existing methods may not pay enough attention to depth distribution in different categories, which makes it possible to use them for further improvement. Besides the existing methods that only use depth regression as an auxiliary task, we propose to use depth distribution density to further support semantic segmentation. Therefore, considering the relationship among depth distri- bution density, depth and semantic segmentation, we propose branch balance loss for these three sub-tasks in multi-task learning schemes. In addition, we also pro- pose a spatial aggregation priors of pixels in different categories, which can be used refine the pseudo- labels for self offf-rraining thus further improving the performance of the prediction model. Experiments on SYNTHIA-tor SYNTHLA-to--Ciyycappp Cityscapes and SYNTHIA- to- Ma-Malllaary benchmarks show the effectiveness of the method. The source code is available https: //t/ //gggtttbbb..eeee com/ conm/depdiee /Detth Dtwp//eepptttrrrreeeeeea Distribution
Introduction
Semantic segmentation refers to the task of assigning semantic categories to each pixel in an image, such as sky, road, car, etc. It is very important for many applications, such autonomous driving [1] and image editing [2]. In this field, approaches of recent years are mostly based on fully convolutional
network (FCN) [3] with modifications designed for pixel-wise prediction [4, 5, 6]. To tackle the
challenge of large-scale scale annotations, unsupervised domain adaptation (UDA) is broadly used. A adapt the semantic segmentation model trained on synthetic datasets [7] (source domain) to perform on real-world datasets [8, 9] (target domain). In order to obtain better adaptation, the GAN [10] structure widely used to minimize the feature distribution discrepancy, so that the model can utilize the knowledge learned from the source domain and apply it to the target domain.
typical practice of UDA
As depth information has been proved to be effective in promoting the performance of semantic segmentation [11, 12, 13, 14], it is usually used help build the semantic connection between two domains. SPIGAN [L]] makes use of the privileged information in the source data from a simulator through the privileged network, which serves as an auxiliary task and regularized segmentation network. DADA [12] is a depth-aware domain adaptation scheme to dig deeper into geometry information, which not only executes the depth regression task when training the generator, but also fuses it together with semantic information during adversarial learning. CTRL. [13] uses multi-task learning establish the relationships between different visual semantic categories and depth levels in UDA context. On the premise of obtaining or generating depth information in advance,
the main
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

CorDA [14] leverages self-supervised depth estimation to bridge the domain gap. And the correlations between semantics and depth greatly improves the performance of target semantic segmentation in the presence of a domain shift. However, the way of using depth information needs to be further explored.
[13] mentions that different semantic categories have discrete depth value ranges, but we find that different semantic classes also have their own depth distribution in images. For example, the sky is at the top of the image, the road is at the bottom of the image, cars and buses are driving on the road, people are always walking on the sidewalk, and so on. According to the above observation, different semantic categories might display relatively consistent depth distribution at their positions images. Therefore, we try make use of the structural information in more accurate way. In this paper, we propose to use the Gaussian mixture models(GMMs) to build the depth distribution for different semantic classes, which can give an estimate of the probabilities of both in-category and out-of-category pixels within the UDA context for semantic segmentation. Concretely, looking into a certain category, such as car, the depth distribution of their locations can be learned by mixture of several Gaussian models, which share common parameters across images and domains. In order to add this information to semantic segmentation, we use the multi-task learning framework to learn three sub-tasks, i.e. semantic segmentation prediction, depth regression and depth distribution density estimation. In addition, considering the internal relations among three sub-tasks, we propose to use density estimation to balance the other two sub-tasks, mainly to improve the efficiency of the main task(semantic segmentation). Moreover, we use the total proportion of class-wise pixels in the source images prior to setting different thresholds, as refine pseudo-labels and improve the efficiency of self-supervised learning on the target domain. We have demonstrated the effectiveness of our proposed method on the benchmark tasks WNNTHAA-t--Citysaal and NNNTHIA-to-Mapillaryy on which we have obtained new state-of-the-ar segmentation performance.
The main technical contributions of our work are made possible as follows:
We propose to utilize the probability density of depth distribution, which is similar in source domain and target domain, bridge the domain gap in UDA for semantic segmentation in more accurate way.
Based on the proposed density branch, an idea of branch balance training is proposed for our multi- task learning, and a branch balance strategy is designed to promote segmentation performance.
A pseudo-labels refinement algorithm is proposed, which uses the aggregation priors of pixels in different categories on the source samples.
2
Related Work
Unsupervised domain adaptation. A typical practice of UDA is to adapt semantic segmentation model trained on synthetic datasets (souree domain) to perform real-world datasets (target domain)
[15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. Hoffman and colleges are the first to leverage adversarial training conduct domain adaptation in the semantic segmentation tasks, where features aligned and the label statistics of the source domain are transferred by category specific adaptation [25]. Then, other methods are to align the distribution of source and target domains through output space [26] or feature space [18, 19], align input pixels of the source and target images [27, 28), or to refine pseudo-labels under the self- self-training framework [17, 29, 30, 31, 32]. The above methods mainly focus on improving the architecture of the segmentation networks, but may not make use of the rich structure information of the two domains.
Use of geometric information in semantic segmentation. To further boost the domain adaptation in semantic segmentation, some researches have explored the use of depth information in the source data, which is the additional information available only during training time[11, 12, 33, 13, 14]. The depth information has been increasingly used to help the domain adaptation, but the relationship between depth and semantic segmentation needs to be further explored in a more accurate way. Our method is similar to DADA[12] and CTRL[13], but we use GMMs to learn the depth distribution possibilities for different categories in the images, so as to establish the relationship between synthetic data and real data more accurately.
Multi-task learning. Our work makes use of multi-task learning (MTL.) [34], where multiple tasks predicted by different branches. SPIGAN[LI], DADA[12], GI33,da CTRL[13] all use the

---

ABP
CoNy
CONV
CONY
+
Cow
Sementics
Auy
Entroyy
irput mages
COw
CONY
V
COW
AogPool
Depth
Doma discrininator
Doman of
prediction
COW
V
CONV
CONV
V AvgPool
Density
Vidg fix
mumppyy
Figure 1: An overview of our network. The backhone is Resnet101, which follows three branches. In the main task branch, the features of the backbone CNN are fed into Deeplab-V [6] architectural de- sign with Atrous Spatial Pyramid Pooling (ASPP) for semantic segmentation prediction. In the depth regression branch, the backbone CNN features are continuously forwarded to three convolutional blocks, followed by an average pooling layer, output depth. We estimate the depth distribution density using the same network as depth regression branch. We fuse the features extracted by density estimation and the features extracted by semantic segmentation for adversarial training.
MTL for depth regression and semantic segmentation, and make use of the depth information to bridge the domain gap. However, apart from these two tasks, no other new tasks have been proposed to help semantics. Although CTRL adds a refined segmentation branch as the third task, there no close connection between these tasks. We use the standard multi-task learning, and add depth distribution density estimation branch to predict density map which closely associated with depth and semantic segmentation. Therefore, we design branch balance loss for the three branches, which can significantly improve the accuracy of segmentation. Compared to CTRL, our method has a simpler network and better prediction performance.
Pseudo-label refinement under self-training frameworks. Our work is also related self- supervised learning. By iteratively training the network with gradually improved target pseudo-labele the performance on the target domain can be further improved[17, 23, 20, 13]. Different from their approaches, we explore and add the spatial aggregation priors in different categories of source domain, as guide refine the pseudo-labels for self-training.
3 Methodology
fitted GMM can give an estimate of the probabilities of both in-sample and out-of-sample data points, which is known as density estimation. On the observation that the depth distribution of the same category between synthetic samples and real samples similar!, we utilize this information from source samples and transfer to target samples using adversarial training. We use standard multi- task learning framework to obtain three sub-tasks (as shown Fig. 1), namely semantic segmentation (main task), depth regression and depth distribution density estimation. In our framework, two auxiliary tasks work together improve the performance of the main task. Following an adversarial training scheme[25, 26). we train the network in a UDA setting. In addition, we also explore pixel aggregation priors different categories on the source domain help refine the pseudo-labels on the target domain, thus further improving the semantic segmentation performance.
3.1 Problem Formulation
Let D. and D, represent the source domain and target domain, and the samples from them represented by tuples Pa, 2.) and (Te) respectively, where 1 E IR HHXW x3 are color images, P E [l,...CI*X are semantic annotations with C' classes, and Z [Zmin Zmoal HxW are depth maps from a finite frustum. For the source domain training, we optimize the network cost using supervised loss for semantic segmentation and depth regression. Therefore, unsupervised adversarial learning scheme on the target domain within the same training process. addition, we
Pleasse the GMM parameters file of SYNTHIA and Cityscape datasets the source code website.

---

use a branch balance loss to estimate the depth distribution density on both source and target domains, which is closely related segmentation and depth results.
3.2 Supervised Learning
For semantic segmentation prediction, we adopt the standard cross-entropy loss, which is repre- sented as pixel-wise class probabilities over C. classes:
Lseg ( (P,P)
- Pj Po P.
(1)
im1
where P and P are the predicted and ground truth segmentation maps, respectively.
For depth regression, employ the berHu loss:
Ldep (2, Z) berHu (2 (2- 2))
(2)
with the reverse Huber loss defined as [35]:
lezl.
lezl S
berHu u (ex) = (e
(3)
leal > L
== 0.2max (lezl),
(4)
where 2 and Z are the predicted and ground truth depth maps, respectively. Following [33, 12]. inverse depth is adopted for the depth regression loss.
3.3 Depth Distribution Density Estimation
For density estimation, a density map of each image should be built first. Since the ground truth segmentation and depth information of source samples have been provided, this information can be used to construct Gaussian mixture models for each semantic category.
For each category E (1, C), the depth value of the pixel associated with its position (a,, y) can form tuple (x, y, 2), and the list of tuples (x, y; 2) where the associated pixel is classified as ith class can be denoted as X, For each category i, we adopt Expectation- Expectatio-Maxxiizattoo algorithm [36] learn the mixture component weights Φij. the component means pīş and variances/covariianees
Mg of GMMs from the depth distribution information X, on source samples, where j means the jth component of GMM. Therefore, can learn the GMMs for each semantic class the source domain.
The density values of each pixel can be calculated by the following formula:
p X
NieN N X piy. E.s)
(5)
where Σj = and X, N (pij, Eij). In experiment, K is defined as 5.
By calculating the density value of all the pixels by class-wise GMMs, a depth distribution density map D IR Bx of the image is constructed, which be regarded as the ground truth. For Equation 5, every pixel in describes the probability of seeing the depth value with this semantic class this pixel location under the learned GMM.
We propose branch balance loss for the depth distribution density regression, and its novelty lies in the different construction ways of the density map D. For source domain training, instead of using the pre-calculated density maps of the source samples, we use the ground truth depth, the predicted segmentation map and cre-constructed source domain GMMs to generate the density map D, for each sample. For target domain training, we use the estimated depth, the predicted segmentation maps and pre-constructed source domain GMMs generate the density map D, for each sample. We also use berHu loss (the reversed Huber criterion [35]) to estimate the density. Therefore, the branch balance loss can be expressed as:
Lrat (D,D) D) ber Hu (D-D).
(6)

---

where D is the predicted density map of the density branch, and D is the constructed density map by the other two branches.
Note that, unlike the losses used for semantic segmentation prediction and depth regression, the branch balance loss is suitable for the training of source domain and target domain. There two advantages to the proposed branch balance loss. On one hand, it can balance the training efficiency of different branches on the source domain and target domain, especially on the target domain. Because there exists no ground truth to constrain the sub-tasks training on the target domain, it may not be easy for each branch keep balance. In addition, because we use the GMMs of the source domain to calculate density maps of the target samples, we also build bridge between the two domains.
The network parameters denoted as net are learned to minimize the following objective functions the source domain and target domain:
min (Dee + Xdep + Abat Sbat) Pret D(e)
min E (Near Onne D(1)
where hyperparameters Aseg, Adep, Abat and ttar the weight values. In our experiments, use Aseg = 1.0, Adep =0.5 x 10- Abal = 10-2 Aear =5 x 10-2
3,4 Adversarial Training
We use discriminator network align the source and target domains. Unlike CTRL[13] which designs a Cross-Task Relation Layer to concatenate three entropy maps (semantic segmentation, refined semantic segmentation and depth) together, we follow the DADA fusion strategy. However, different from DADA, we use segmentation to fuse density map rather than depth map.
To be more specific, the weighted ceff-information maps [19]
(P) is calculated first by:
F
(P))
P log
Then, we fuse the C-channel F with the estimated density map Đ to produce a fused C'-chanel map F, and feed it forward to a discriminator.
More specifically,, the discriminator network parameters @D can be obtained by correctly classifying the sample domain as a source or target during training:
min
E D
log
[Fa)]=
(P=)
+ E D.
log (1- (f))]
(10)
At the same time, the prediction network parameters Onee is updated using the "fooling" objective minimization:
[lo D (
min Ones D.
(11)
The hyperparameter Andv is used to weigh the relative importance of the adversarial loss (Equation 11), and we Aude 5 x 10-2 in our experiment. In our training scheme, the model parameters of the prediction network nfnez) and the discriminator (0p) are optimized jointly. At each training iteration, we input a batch of two samples from the source domain and the target domain into the network. All loss gradients are accumulated and then propagated back to update the network.
3.5 Spatial Aggregation Priors for Pseudo-labels Refinement
After multi-task learning in the framework of adversarial training, self-training can further improve the effect of semantic segmentation. We find that pixels of large objects, such as sky and road, have a large-scale aggregation in image space, while pixels small objects, such as person and bicycle, have relatively small-scale aggregation in image space. Therefore, since most pixels in the same class are clustered together, we calculate the total proportion of class-wise pixels in the source images as prior, to set different thresholds for refining pseudo-labe labels on the target domain for self-training
5

---

N₂
Nmin
thres; NhaseD
Nbasel:
(12)
N Max
Nmi
where thres; represents the threshold of different category i, Mmin and Nmaz represent the minimum and maximum numbers of pixels of different categories from all source samples, respectively. N; represents the number of pixels of the ith category from all source samples. Nbaseo and Nbaset two default values, which are set as 50 and 4950 respectively in our experiment, so the threshold ranges from 50 to 5000. Please note that, because the aggregation scale of different categories varies greatly, we set a threshold range of 1:100, which can be further adjusted according to the actual size of classes in different datasets.
We design an algorithm which adds spatial aggregation priors for refining pseudo-labels. Algorithm is the semantic description of refining the pseudo-labed by using the spatial aggregation priors.
Algorithm 1:
Spatial prior pseudo- labels refinement algurithm
Input: A larget sample with predicted pseudo- pseudo-labels. labels.
Output: Refined pscudo-labels.
Initialize all pixels
their flags =0
2 for w=0 to W do
3 for h=0 to I/ do
if TT..= & Con fidencess 20.9 then
Search around it for pixels that satisly the following conditions:
6
Their prediction class is the same as Tus. and their confidence value 2: 0.9.
7
Iterate over taking these points as the fiducial points and search around them outward for the qualified points. Count the number of all qualified pixels, und record as Ne:
8
9
N. 2 threes then
10
Set flags of all these pixels to 1;
Pixels labeled with reserved, and their pacudo-labels used for self-supervised learning.
Our prediction model can be further improved by trffnirain schemes. Follow [17], we first train the prediction and discriminator networks for Q1 iterations. We generate semantic pseudo-labels on the target training samples using the trained prediction network, and then obtain refined pseudo-labe by Algorithm 1 to further train the prediction network on the target training samples using pseudo-labels supervision for Q2 iterations. The above pseudo-labels generation-refinemenn and self-training process is executed twice to produce higher quality semanties output on the target domain.
Experiments
4.1 UDA Benchmarks
To make fair comparison with previous methods[11, 26, 37, 38, 19, 12, 13], especially DADA and CTRL, which are the most similar to our method, we use three standard UDA evaluation protocols verify our model: SYNTHIA - Cityscapes (16 classes), SYNTHIA
Cityscapes classes), and SYNTHIA -> Mapillary (7 classes). Detailed descriptions of these settings can be found in [12] and [13]. In all settings. the SYNTHIA dataset [7] is used as the source domain. Following HEWHHHA--RAD-CIIYYYS
split consisting of 9,400 synthetic images and their corresponding pixel-wise semantic labels and depth. For target domains. we use Cityscapes [8] and Mapillary Vistas [9] datasets. Similar to [26, 19, 12, 13], we report the performance of semantic segmentation based on "mean Intersection over Union" (mloU in %) on the classes of the Cityscapes validation set, and also show the mloU (%) of the 13 classes (mloU*) excluding classes with * For depth, we use Absolute Relative Difference (Rell), ), Squared Relative Difference (Rel?), Root Mean Squared Error (RMS), its log-variant LRMS; and the accuracy metrics [39] as denoted by 81, 82, and 83. For each metric, we use 1 and 1. to denote the improvement direction.
[12, 13], we use the
4.2 Experimental Setup
All our experiments are conducted on single NVIDIA 1080Ti GPU with memory of IIGB. Our network is implemented on PyTorch [40]. Backbone is a ResNet-101 [41] initialized with ImageNet [42] weights. Like [26, 19], we apply Atrous Spatial Pyramid Pooling (ASPP) with sampling rates of (6. 12, 18, 24). In addition, use DC-GAN [43] as domain discriminator for adversarial

---

Table 1: The quantitative results of different methods for semantic segmentation performance (IoU and mloU, %) on SYNTHIA- Cityscapes(16 16 classes).
SYNTHIA Cinyccpes (16 classes)
4dewolk
building
rood
Welpe
1ight
Models
Depth
mlaUi
mloU/1
SPIGANEL]
71.1
29.8
71.4
3.7
0.3
33.2
6.4
15.6
61.2
78.9
52.7
13.1
75,9
25.5
10.0
20.5
36.8
42.4
79.2
37.2
78.8
9.9
10.5
78.2
X0.5
53.5
19.6
67.9
29.5
21.6
31,3
45.9
AdepPPachE37
82.2
19.4
79.4
6.5
10.8
77.8
82.0
54.9
21.1
67.7
90.7
17.8
32.2
46.3
CLANNSS]
81.3
37.0
80.1
16.1
13.7
78.2
81.5
53.4
21.3
73,0
$2.9
22.6
30.7
47.8
Adven|[9]
87.0
24.3
56.4
33.7
DADAFEI
38.8
CTRLL3)
44.2
524
79.7
29.4
64.3
Table The quantitative results of different methods for semantic segmentation performance (IoU and mloU, %) on SYNTHIA-> Cityscapes(7 classes) and SYNTHIA + Mapiillry (7 classes) in low-resolution and full-resolution.
SYNTHIA Citysaaps (7 classes)
(b) SYNTHIA Mapillary (7 classes)
objor
nattre
Dumun
rehiude
objorr
nature
sehicl
sky
sky
Res.
Model
Depth
mloli-
mloli|
SPIGAN[UI]
91.2
66.4
9.6
56.8
71.5
17.7
00.3
53.4
74.1
47.1
6.8
43.3
83.7
11.2
42.2
44.1
3207640
Advent [9]
86.3
72.7
12.0
70.4
81.2
29.8
62.9
59.4
82.7
51.8
18.4
67.8
79.5
22.7
54.9
54.0
DADA[12)
CTRL.US)
90.8
77.5
15.7
77.1
82.9
45.3
68.6
68.4
86.6
57.4
19.7
73.0
87.5
45.1
68.1
62.5
Ours
92.6
78.2
23.4
77.2
82.9
49.6
69.8
67.7
86.2
58.7
19.4
68.9
86.1
40.4
62.4
60.3
Advent [9]
89.6
77.8
22.1
76.3
81.4
54.7
68.7
67.2
86.9
58.8
30.5
74.1
85.1
48.3
72.5
65.2
DADAI 12]
75.9
CTRL[IS]
92.4
30.7
27.7
78.1
83.6
59.0
78.6
71.4
88.5
59.2
27.8
79.4
85.7
64.4
79.6
68.2
Ours
92.4
81.8
34.3
78.9
82.0
64.5
74.1
72.6
87.7
68.6
33.7
74.8
93.0
61.4
73.4
70.4
learning. The learning rates of the prediction and discriminator networks are set as 2.5 x 10-4 1.0 10- respectively. In self-training, the parameters are: Q1 = 54K,, Q2 = 30K.
4.3 Comparison to Other Methods
4.3.1 SYNTHIA -> Cityscappes116 classes)
Table 1 reports semantic segmentation performance of proposed model SYNTHIA
Cityscapes(16 16 classes). It shows our method achieves SOTA performance on both 16 and 13 classes, outperforming other methods by large margins. Compared the SOTA works, have improvements in 11 classes. Compared with the SOTA CTRL(13), which maps depth to a discrete probability space, our main gains come from the following classes "light" (+18.1%). "sign" (+19.6%), "person" (+3.6%), "bus" (+3.2%), "rider" (+2.7% and "pole" (+2.2%). Besides, our method shows consistent improvements on the following classes in the target domain: "wall" (+0.4%), "fence" (+0.4%), "bicycle" (+0.7%), "vegetation" (+1.2%), "car" (+1.0%) and "motorbike" (+0.7%). Fig. 2 shows the results of the qualitative comparison of our method with DADA[12] and CTRL[13]. We also show the estimated results of depth and density in the figure. From the results of density estimation, the trace of segmentation can be seen.
4.3.2 SYNTHIA - Cityscapes (7 classes) and SYNTHIA - Mapillary (7 classes)
Table 2 shows the semantic segmentation results in SYNTHIA + Mapillary benchmarks, which are trained and evaluated on their common classes. We also train and evaluate our model on the 320 x 640 resolution, so as to make fair comparison with the reference low-resolution models, In the low-resolution model in SYNTHIA - Mapillary benchmark, CTRL superior to other methods, and our model ranks second with mloU loss lower than CTRL by -2.2%. Besides, our proposed method is superior to previous works. Compared to CTRL, our mloU gains are +2.3% and +1.2% for low-resolution and full-resolution of SYNTHIA - Cityscapes, and +1.2% for full-resolution of SYNTHIA - Mapillary. Fig. 3 shows the results of the qualitative comparison of our method with 12 and CTRL[13] on these two protocols at full-resolution.
Cityscapes and SYNTHIA +
7

---

(a) Test Imsge
GT
(c) DADA
(d) CIRL
(o) Our Sogmestation
(E) Our Depth
(e) Our Dunsi by
Figure Qualitative results on SYNTHIA - Cityscapes (16 classes). (a)Test images from Cityscapes validation set; (b) ground truth segmentation; (c) DADA[12] segmentation; (d) CTRL[13] segmenta-
tion; (c) our segmentation; (f) our depth; (g) our density. Compared theirs, our method demonstrates notable improvements on "light and sign", "vegetation", "person" and "bus" classes highlighted using the yellow boxes.
4.4 Ablation Studies
In order to verify the effectiveness of our proposed components, we trained four different models for comparison, denoted MI,...4.. An ablation study is shown in Table 3. "SegPre" represents segmentation prediction branch, "DepRes" represents depth regression branch, "DenEst" represents density estimation branch, "SelfTra" means the self-training scheme in our paper without using spatial prior pseudo-labels refinement algorithm, "SpaPri" means the spatial prior pseudo-labels refinement algorithm. All the models trained and evaluated on SYNTHIA - Cityscapes (16 classes).
MI is backbone network with segmentation predict branch and depth regression branch. Unlike DADA, we didn't 't fuse features for adversarial training in M1. The result shows that segmentation result (mIoU 41.7) is lower than that of DADA (mloU 42.6). M2 adds the branch of density estimation, and this is model. The segmentation result is 44.8 (mloU), which is satisfactory improvement. Note that, for fair comparison, without self-training, CTRL result 42. (mloU), which can be found in their ablation study table[ 13]. Apart from three sub-tasks, CTRL has additional Cross- Task Relation Layer, but we choose a more concise MTL framework. Compared to DADA, M2 has mloU gain of +2.2%. Compared to CTRL without self self-training, M2 has mloU gain of +2.7%. This demonstrates the effectiveness of the depth distribution density branch proposed by us. M3 adds our self-training scheme by commonly used pseudo-labels refinement based on confidence, and M4 adopts our proposed spatial prior refinement algorithm on the basis of M3. The result of M3 demonstrates model can obtain consistent improvements by using self-supervised training. In addition, compared to M3. M4 has mloU gain of +0.6%, which shows the effectiveness of our proposed algorithm obtain higher quality pscudo-labels.
Table 3: Ablation study of different components of our method in Section 4.4
Table 4: Additional analy- sis in Section 4.5
Model
SegPre
DepRes
DenEst
SeliTra
SpaPri
mloU(%)t
Situation
mloU(%%
M1
41.7
44.1
M2
44.8
$2
43.4
MB
V
V
47.6
S3
37.8
M4
V
V
48.2
43.7
S5
44.8
8

---

(a) Test Imago
(b> (e) DADA
(d) CIRL
Our Sogmenaation
(E) Depth
(g) Our Deasity
Figure 3: Qualitative results on: SYNTHIA - Cityscapes (7 classes) (upper two rows) and SYNTHIA + Mapillary (7 classes) (lower two rows). (a) Test images from Cityscapes and Mapillary validation sets; (b) ground truth segmentation; (c) DADA[12] segmentation; (d) CTRL[13] segmentation; (e) our segmentation; (f) our depth; (g) our density. Compared to theirs, our method demonstrates notable improvements on "object", "human", "sky" and "construction" classes as highlighted using the yellow boxes.
4.5 Additional Experimental Analysis
Branch balance loss for density estimation. The novelty of this loss lies the density map constructed for regression, which is caleulated online during training. We propose the branch balance loss for multi-task learning, which is different from the SOTA methods. When the branch balance loss is removed, density regression on the source domain will the pre-constructed density map as the ground truth for supervised training, and density regression on the target domain will not exist. The mIoU result of this model is 44.1 (S1 in Table 4). Note that S5 is the result of our model, and all results in Table 4 are without the self-training scheme on SYNTHIA + Cityscapes (16 classes).
Ground truth depth for density map construction during source domain training. For compar- ison, we use the predicted depth to construct the density map the source domain training. The result of mloU is 43.4 (S2 in Table 4), which is lower than the method proposed. It is known that the density map is closely related to the segmentation map and the depth map. In our source domain training, we eliminate the influence of depth estimation by using its ground truth, but transfer the influence to segmentation by using the predicted segmentation results. This change makes the density map closely related semantic segmentation, and thus improving the segmentation prediction performance.
Features fused from density and semantic segmentation. Our feature fusion strategy is not only different from DADA, which fuses depth and segmentation results, but also different from CTRL, which concatenates the features of three branches together. For comparison, fuse all results of our three branches together. The result of mloU 37.8 (S3 in Table 4). We believe that depth density with segmentation may enhance the antagonism of segmentation, while the depth information added to the fused feature may interfere with and weaken the main task segmentation, which is why the performance of S3 drops much. In addition, the same feature fusion as DADA in our framework. The result of mloU is 43.7 (S4 in Table 4). Therefore. our fusion strategy (S5 in Table 4) can highlight the effect of the main task.
trained model to
trained another model with
t-SNE comparison features learned by two different joint spaces. Fig. 4, we use t-SNE [44] to visualize features in target domain through (a) joint space of depth and segmentation, and (b) joint space of density and segmentation. The two models are trained and evaluated following the UDA protocol SYNTHIA - Cityscapes (16 classes). For comparison, results show that our density++segmenaate joint space is better than depth+segmmntattion joint space, and it has high degree of aggregation and better class separation in the target domain.
classes circled. The t-SNE
The influence between density and depth estimation. To verify whether density can promote depth, use MI model and our M2 model in Table 3 to generate the depth results of the test images of

---

M1
M2
aidensli buliding
Rell
0.7
0.5
Rel? +
13.7
9.0
RM
20.7
18.3
LRMS4
0.9
0.7
0.21
0.26
rider
52 0.40
0.48
53t
0.56
0.66
(a) desth-sagmmenationnnom joint space
(b) dansky--eggmen joint space
Figure 4: t-SNE comparison of features learned by two different joint spaces.
Table Depth comparisons
between M1 and M2 model in Table 3.
Cityscapes dataset. The depth results are shown in Table As can be seen from the table, density estimation can improve depth results. It can be done from the following two aspects: First, the density and depth are closely related. the process of building a bridge between two domains through depth density, depth prediction is naturally promoted. Secondly, in our training, our branch balance strategy closely links depth regression with density estimation. Therefore, the density helps to improve the depth.
Sensitivity analysis of hyper parameters. Because we use multi-task learning, the parameters of each task are selected as follows: the main task segmentation is large weight, and we set it to 1.0. The two auxiliary tasks, density and depth, are small weights. Considering that density is used to fuse with segmentation, which means that density is more important than depth, we set the weight of density 10-2, and the weight of depth to 0.5 10-2, We also designed the experiments to increase the hyper parameters of two auxiliary tasks to the level of 10-1, or decrease to the level of 10-3 and the final results will worse. However, the change of parameters at the level of 10-2 has little effect on segmentation.
4.6 Limitations and Discussions
The training cost. Our method is based on multi-task learning, and will cause a relatively high cost, which is also inevitable in DADA[12] and CTRL[13] methods. DADA has two sub-tasks, while CTRL and our method have three sub-tasks. Compared to CTRL and ours, the training time of DADA is relatively short, and the memory utilization rate is low. The first stage of our training is 60K for almost 90 hours, and self-training for 10 hours. The GPU memory usage is about 10GB, However, by adding the branch of depth density estimation, our performance is improved.
The results of depth estimation. Depth estimation is auxiliary task in framework, and it is used with density to promote the main task. We use the same branch network as the previous methods 12, 13] for depth regression. Like theirs, we find that our depth regression results cannot be compared with the works focused on depth estimation. This is another limitation of our work. However, depth density is closely related to depth, so in our method, a more accurate depth and its distribution density should be able to further improve the performance of semantic segmentation. We plan to redesign the depth and density branch network in the MTL framework our further work.
Conclusion
We depth distribution to build bridge between synthetic datasets and real-world datasets for semantic segmentation in UDA context. We predict three tasks using the standard MTL framework, and the two auxiliary tasks are designed improve the performance of semantic segmentation. By using our method, a new SOTA segmentation performance is achieved.
This work is funded by the National Natural Science Foundation of China (No.41771427). We thank the anonymous reviewers for the valuable feedback and time spent.
Ackmowledgments.
10

---

References
[1] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[2] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming- Yang. Deep image harmonization. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 3789-3797, 2017.
[3] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1441-3440 2015.
[4] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions, arXiv preprint 1r..12251..0712. 2015.
[5] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jía. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattrrn recognition, pages 2881-2890, 2017.
[6] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplah: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysix and machine intelligence, 40(4):834 848, 2017.
[7] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urhan scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234-3243, 2016.
[8] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[9] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pages 4990-4999, 2017.
[10] Ian Goodfellow, Jean Touget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[11] Kuan-Hui Lee, German Ros, Jie Li, and Adrien Gaidon. Spigan: Privileged adversarial learning from simulation. arXiv preprint TeX.iv:810.037670 2018
[12] Tuan Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Dada: Depth aware domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7364-7373, 2019.
[13] Suman Saha, Anton Obukhov, Danda Pani Paudel, Menelaos Kanakis, Yuhua Chen, Stamatios Georgoulis, and Luc Van Gool, Learning to relate depth and semantics for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8197-8207, 2021.
[14] Qin Wang, Dengxin Dai, Lukas Hoyer, Lue Van Gool, and Olga Fink. Domain adaptive semantic segmentation with self-supervised depth estimation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 8515-8525, 2021.
[15] Yang Zhang, Philip David, and Boqing Gong. Curriculum domain adaptation for semantic segmentation of urban scenes. Proceedings of the IEEE international conference on computer vision, pages 2020 2030, 2017.
[16] Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa. Unsupervised domain adaptation for semantic segmentation with gans. arXiv preprint trXiv:1111.06960 2(2):2, 2017.
[17] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class- balanced self telfninaining. In Proceedings of the European conference on computer vision (ECCV), pages 289-30 305, 2018.

---

[18] Yuhua Chen, Wen Li, and Luc Van Gool. Road: Reality oriented adaptation for semantic segmentation of urban scenes. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 7892-7901, 2018.
[19] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2517-2526, 2019.
[20] Javed lqbal and Mohsen Ali. Misi: Multi-level self-supervised learning for domain adaptation with spatially independent and semantically consistent labeling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1873, 2020.
[21] Jinyu Yang, Weizhi An, Sheng Wang, Xinliang Zhu, Chaochao Yan, and Junzhou Huang. Label-driven reconstruction for domain adaptation in semantic segmentation. In European conference on computer vision, pages 480-498. Springer, 2020.
[22] Qianyu Zhou, Zhengyang Feng, Qiqi Gu, Guangliang Cheng. Xuequan Lu, Jianping Shi, and Lizhuang Ma. Uncertainty- consistency regularization for cross-domain semantic segmentation. arXiv preprint urXiv:2004. 08878, 2020.
[23] Sujoy Paul, Yi-Hsuan Tsai, Samuel Schulter, Amit K Ray-Chowdhury, and Manmohan Chandraker. Domain adaptive semantic segmentation using weak labels. In European conference on computer vision, pages 571-587. Springer, 2020.
[24] Jinyu Yang, Weizhi An, Chaochao Yan, Peilin Zhao, and Junzhou Huang. Context-aware domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2021.
[25] Judy Hoffmann, Dequan Wang, Fisher Yu, and Trevor Darrell. Fens in the wild: Pixel-level adversarial and constraint- based adaptation. arXiv preprint arXiv:1612.02645 2016.
[26] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7472-7481, 2018.
[27] Judy Hoffmann Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMLR, 2018
[28] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4085-4095, 2020.
[29] Qing Lian, Fengmao Lv, Lixin Duan, and Boqing Gong. Constructing self-motivated pyramid curriculums for cross cross-domain semantic segmentation: A aoverrveralrial approach. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6758 6767,, 2019.
[30] Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision 129(4): 1120, 2021.
[31] Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12414-12424, 2021.
[32] Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adaptation via cross-domain mixed sampling. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1379-1389, 2021.
[33] Yuhua Chen, Wen Li, Xiaoran Chen, and Lue Van Gool. Learning semantic segmentation from syn- thetic data: A geometrically guided onput-output adaptation approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1841-1850, 2019.
[34] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansheke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2021.
[35] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international conference on 3D vision (3DV), pages 239-248 IEEE, 2016.
12

---

[36] Geoffrey J McLachlan and Thriyambakam Krishnan. The EM algorithm and extensions, volume 382. John Wiley Sons, 2007.
[37] Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, and Manmohan Chandraker. Domain adaptation for structured output via discriminative patch representations. In Proceedings of the IEEF/CVP International Conference on Computer Vision, pages 1456-14655 2019.
[38] Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Taking closer look at domain shift: Category-level adversaries for semanties consistent domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2507-2516, 2019.
[39] David Figen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014.
[40] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[42] Jia Deng. A large-scale hierarchical image database. Proc. of IEEE Computer Vision and Pattern Recognition, 2009, 2009.
[43] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1.00444, 2015.
[44] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning rexearch, 9(11), 2008.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] This is similar work on semantic segmentation in unsupervised domain adaptation using depth information. The potential impact is usually the same that of semantic segmentation in general.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tCcdE7Fnck", "reviewer": "Reviewer_jvKa", "summary": "This paper proposes a novel segmentation framework that could be aware of depth distribution density and thus improve the segmentation performance using unsupervised domain adaptation. The proposed network is based on the multi-task learning architecture that enables three different tasks: segmentation, depth regression, and depth density distribution estimation. At the end of the architecture, the adversarial network is appended, and thus the segmentation performance is significantly improved. The main contributions of this paper are (1) the suggestion of the probability density of depth distribution to bridge the gap in the unsupervised domain adaptation, (2) a novel idea of branch balance training that enables improving the segmentation performance, and (3) pseudo-labels refinement algorithm that utilizes the aggregation priors of pixels in the different categories on the source images. The evaluation was performed using synthetic images as the source domain and real-world images as the target domain. The experiments demonstrate the effectiveness of the proposed method in the unsupervised domain adaptive semantic segmentation task.\n\n- The proposed architecture mainly aims at segmenting target objects in the images from the target domain, which has not been annotated from the source domain's knowledge (especially depth distribution density), which includes the annotated labels of segmentation map and depth distribution.\n\n- The architecture is based on the simple multi-task learning architecture but utilizes the additional adversarial training architecture to improve the segmentation performance. \n\n- In multi-task learning, three tasks of segmentation, depth regression, and estimating the depth distribution density are included. \n  - The segmentation task is the same as the general semantic segmentation task (supervised learning with annotated labels) using cross-entropy loss. \n  - The depth regression is also supervised learning with the annotated labels, and the regression is performed using the berHu loss function.\n  - From the segmentation map and the depth information, annotation for the density distribution can be generated. Then, the depth distribution density estimation is performed using the estimated density from the other two branches and the predicted density (branch balance training). \n\n- The proposed method was evaluated using two datasets of Cityscapes and Mapillary as the target domain. In addition, the SYNTHIA dataset is utilized as the source domain. The images in the target domain are real-world images, whereas the images in the source domains are generated synthetic images.  \n\n- The experiments report the state-of-the-art performance of the proposed depth distribution-aware unsupervised domain adaptation network compared to other state-of-the-art deep learning models.\n", "questions": "* Questions & Discussions\n0. The main questions and discussions are raised by the vague explanations or ignorance of the reviewer. Clear explanations and the discussions in the rebuttal/discussion phase would resolve the following questions. If the reviewer misunderstands the concepts or definitions, please fix the problems with clearer explanations.\n\n1. The definition of the domain adaptation should be more discussed. In this paper, the authors addressed that the proposed framework could bridge the domain gap in UDA for semantic segmentation. The domain adaptation task mainly aims at bridging the domain gap such as style differences, geometric differences, or even statistical gaps between two domains. However, this paper has not discussed the main domain gap to be resolved. The authors repeatedly advert the utilization of depth information to bridge the domain gap. The clear explanations that the utilization of depth information can bridge the “what domain gap” in this paper should be discussed. The reviewer expects that the manuscript could be improved with clear illustrations of the problem definition.\n\n2. The main technical contribution should be more discussed.\n\n- In the first contribution (Line 61-62; Page 2), the authors address that the depth distribution is similar in the source and the target domain. The reviewer agrees that this conjecture is well working while regarding the large object and obvious objects like sky, land, road, etc. However, it is worried that this conjecture could be well appliable to the small objects like bikes and pedestrians, frequently appeared objects in one domain, and many objects from the same category (e.g., many cars are in one image). Depth distribution density is based on this hypothesis, the authors should illustrate the experimental or mathematical evidence for this hypothesis. The authors mentioned this in Line 104-106; Page 3, but the statistical analysis and evidence (table or graph) should be provided.\n\n- The second contribution proposed by the authors is the branch balance training, which performs simple density regression using the existing loss function. It would be better to explain why the training method using a simple loss function can contribute to NeurIPS society. In addition, the novel idea beyond the simple equation (equation 6) should be discussed in the manuscript.\n\n- The reviewer was impressed by the novel idea of the utilization of depth distribution density to improve the segmentation performance. Despite the simple architecture based on multi-task learning, the motivation and the proposed methodology could be a novelty. Therefore, the forest could be the contribution rather than the trees.\n\n3. The training pipeline and the network architecture should be discussed. \n\n- The training procedure should be clearer. The proposed architecture can be divided into two pipelines: (1) multi-task learning architecture and (2) adversarial network architecture. In the multi-task learning architecture, the images in the source domain are used to train the network with their annotations (segmentation map, depth map, and the calculated density map). In contrast, the images in the target domain are used to train the network with a segmentation map (?), depth map (?), and the density map of the source domain. Then, it is concluded that the training process using the target domain only optimizes the parameters in the density-predictive pipeline, is it? Furthermore, the adversarial network architecture inputs the fused density map from the source and the target domain both. Therefore, it is concluded that the training process using the source and the target domains are concurrently performed on both (1) and (2) architectures. Is it truly the process to train the network? Other domain adaptation methods train the network using the following in general:\n\n  - (a) Transfer the style of the source domain to that of the target domain and then train the network (M) with the property of the target domain. Then, M predicts the segmentation output using the images in the target domain. [1]\n\n  - (b) Train the network (M) using the source domain. Then, transfer the style of the target domain to that of the source domain, and M predicts segmentation maps using the transferred images. [2, 3] \n\n  - (c) CNN architecture generates feature maps from the source and the target domains, and the CNN architecture aims at increasing the similarity between two feature maps [4].  \n\n- The reviewer is interested in whether the proposed domain adaptation method is more similar to (a), (b), (c), or other existing methods (please list the reference). Or, if the proposed methods are significantly novel, then the training pipeline should be clearly illustrated. The reviewer recommends authors illustrate the training pipeline in a visual way (figures) or mathematical formulations (or algorithm table).\n\n- Effectiveness of the adversarial training. Related to the question above, the reviewer is curious about the effectiveness of the adversarial training. As the reviewer understood, the adversarial training utilizes the combined feature maps of the density map and the predicted segmentation map. The authors should clearly describe the goal of the adversarial training. In the Generative Adversarial Network (GAN), the generator and the discriminator are simultaneously optimized by the min-max game. In the proposed network, what advantage could be realized by deceiving the discriminator here? Does the author aim at designing that the generators (here segmentation and density pipelines) aim at generating similar feature maps using the source and the target domains? In this case, the bias (biased to segmentation map or biased to density map) could be discussed. \n\n- Cost of the proposed framework. The reviewer is curious about the training time, prediction time, and memory resources. Inevitably, the multi-task learning-based network causes a heavy cost in prediction/training time and even in memory utilization. Quantitative analysis or discussion can significantly improve the quality of the manuscript. For the reviewer, since the proposed network includes multi-task learning- and adversarial learning-based architecture, the training and prediction cost can be extremely burdened compared to the conventional algorithms.\n\n[1] Lee, Seunghun, et al. \"ADAS: A Direct Adaptation Strategy for Multi-Target Domain Adaptive Semantic Segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Hou, Yunzhong, and Liang Zheng. \"Source free domain adaptation with image translation.\" arXiv preprint arXiv:2008.07514 (2020).\n\n[3] Lee, Kyungsu, Haeyun Lee, and Jae Youn Hwang. \"Self-Mutating Network for Domain Adaptive Segmentation in Aerial Images.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[4] Ganin, Yaroslav, and Victor Lempitsky. \"Unsupervised domain adaptation by backpropagation.\" International conference on machine learning. PMLR, 2015.\n", "limitations": "1. Major issues\n- Related to the question illustrated in \"Question\" the followings should be improved. (1) The definition of the domain adaptation and the domain gap should be discussed. (2) Limited contribution and novelty. (3) Vague explanations for the training pipelines. \n\n- Additional visual illustrations are required. Specifically, the predicted segmentation maps are well organized in the experiment parts. However, the outputs of the proposed network are segmentation, density, and depth maps. Therefore, the author should exhibit three maps to demonstrate that the proposed network is outstanding in the unsupervised domain adaptive semantic segmentation task. \n\n- Mathematical notations and abbreviations should be strict.\n\n  - The mathematical modeling should be invited for newly proposed terms with precise definitions. For instance, the authors proposed depth distribution density and should provide a clear definition of this term. Likewise, \"density map\" and \"depth map\" should be invited with clear mathematical definitions, including rank, dimension, size, range, and type (matrix, vector, set, scalar, etc.).\n\n  - It is better to use the familiar term: \"annotation/ ground truth/ semantic label/ ground truth segmentation map/ ground truth semantic label\" instead of \"ground truth segmentation annotation\". In this case, P_hat can be the \"predicted segmentation map\" or \"segmentation prediction\".\n\n  - In Line 135; Page 4, X_i should be re-checked. As the authors notated, X_i depends on i. However, the authors described X_i as a tuple (x, y, z) which is regardless of i. Therefore, the reviewer think X_i should be the list of tuples (x, y, z) where the associated pixel is classified as i-th class. \n\n  - In Line 136; Page 4, j is firstly invited in Pi_{i,j} without any definition. Please clarify the definition of j.\n\n  - In Line 140, Page 4, does N(~~~) indicate the flag (0 or 1) whether X_i is the gaussian distribution? In this case, if the input of the function N is only X_i, the author can omit the mean and the variance, even though N is adopted from the notation of the normal distribution \"N~(0,1)\".\n\n  - GMM first appeared in Line 84; Page 2 without any definition. Please fix it with the gaussian mixture model in Line 46; Page 2. \n\n  - Algorithm I is significantly far from the mathematical modeling and algorithm. Please simplify the table.\n\n2. Minor issues (recommendations)\n- Justify the hyper-parameter selections (Line 165-166; Page 5/ Line 179-190; Page 5/ Line 192-193; Page 6/ Line 198-203; Page 6). In addition, since the architecture is based on the multi-task learning architecture and the memory of GPU is 11GB, the training should be conducted in small batch size. The batch size and the normalization method (batch/group/instance/etc normalization) should be discussed in this case.\n\n- (if available) The experiments should be improved. The reviewer already understands that the evaluation metric of \"mean Intersection over Union (mIoU)\" can qualitatively measure the object details. However, recent studies [1-3] proposed new evaluation metrics to quantitatively measure the object detail (especially the boundaries of the target objects). Otherwise, the visualization of the feature map could illustrate the novel feature extraction when processing object details. Alternatively, the authors could refer to the activation maps [4]. More experimental or mathematical evidence should be justified to clearly report the outstanding segmentation performance in terms of the improved boundary-oriented segmentation.\n\n[1] Fernandez-Moral, Eduardo, et al. \"A new metric for evaluating semantic segmentation: leveraging global and contour accuracy.\" 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.\n\n[2] Lee, Kyungsu, et al. \"Boundary-oriented binary building segmentation model with two scheme learning for aerial images.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.\n\n[3] Cheng, Bowen, et al. \"Boundary IoU: Improving object-centric image segmentation evaluation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[4] Selvaraju, Ramprasaath R., et al. \"Grad-cam: Visual explanations from deep networks via gradient-based localization.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n\n3. Simple recommendations\n- The reviewer recommends reviewing the grammar and typo errors to improve the quality of the manuscript.\n\n- The reviewer recommends clarifying the meaning of the arrows in figure 1 (dashed line and other lines). Especially, L_bal uses the calculated density and the predicted density. What does \"entropy\" indicate? The inputs and outputs of each module are recommended to be clarified.\n\n- Motivation for utilization of the berHu loss, ResNet101, ASPP, and AvgPooling.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Strengths\n- The reviewer agrees that the significance and importance of the task addressed in this paper, especially the depth information in each category, can improve the segmentation performance. In addition, unsupervised domain adaptive semantic segmentation with depth distribution density has rarely been studied to our best knowledge, such that the idea of the proposed task and network in this paper would be a novel contribution. Furthermore, the experimental result reports the state-of-the-art performance of the proposed network, and thus the proposed network can be expected as a novel framework.\n\n- Additionally, the problem definition is clear. The authors were motivated to utilize the depth information and the depth density distribution while segmenting the target object. The idea of using depth distribution density is simple yet effective. \n\n- The manuscript is well organized and well written.\n\n\n2. Weakness\n- More detailed descriptions are illustrated in the “Limitation” section. The main limitation suggested by the reviewer is the lack of detailed description. Please see below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "b3OcmUCICUT", "reviewer": "Reviewer_pVBs", "summary": "For the UDA task, the authors propose to utilize the probability density of depth distribution to bridge the domain gap for semantic segmentation. \nThe proposed method conduct experiments on the SYNTHIA-to-Cityscapes and SYNTHIA-to-Mapillary setting.", "questions": "What is the training cost when involving the additional branches?\n\nIs the model sensitive to the weight values in Line166? How do you pick those parameters?\n\nAfter reading the whole paper, It is still very hard to understand the density of depth distribution. I highly recommend adding a visualization on the density map, and it might be helpful for readers to follow the overall idea.", "limitations": "Using additional branches for training introduced more time and resources to get a model.\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) The code is provided.\n(+) Writing and presentation are good.\n(+) The branch balance strategy and Pseudo-labels refinement are interesting.\n(-) A lot of recent UDA approaches, e.g. [R1-R4] are missing in this paper. The proposed\tmethod can not perform as well as many CVPR2021/CVPR2022/ICCV2021 works.\n(-) The most common case under the UDA setting -- GTA5->Cityscapes is missing. I suspect the performance of the proposed method in this setting.\n\n[R1] Zhang, Pan, et al. \"Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[R2] Huo, Xinyue, et al. \"Domain-Agnostic Prior for Transfer Semantic Segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[R3] Hoyer, Lukas, Dengxin Dai, and Luc Van Gool. \"Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[R4] Wang, Qin, et al. \"Domain adaptive semantic segmentation with self-supervised depth estimation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U8dAk1_mNsy", "reviewer": "Reviewer_WoDq", "summary": "This work is about unsupervised domain adaptation for semantic segmentation and how to improve the domain transfer with depth information. In contrast to prior work, the authors investigate the distribution of depth for individual categories as a better bridge between domains than plain depth prediction. Specifically, a Gaussian Mixture Model (GMM) is fit to the source training data (x-y location and depth of each pixel, separately for each class), which is then used to compute a per-pixel density map, i.e., the probability of observing a depth value at a certain pixel location and with a certain semantic class. Then, a neural network is extended to not only predict semantics and depth, but also the depth distribution. For domain adaptation, an adversarial loss is computed on features that are a combination of the semantic and depth distribution branch. Finally, a spatial-aggregation prior to compute better pseudo labels for self-training is also proposed. Experiments on standard benchmarks (that include depth ground truth in the source domain) show state-of-the-art results and an ablation study investigates the impact of the individual contributions.", "questions": "- I'm wondering if the same idea of adding depth distribution as additional task would help plain semantic segmentation on the source domain, i.e., in a non domain adaptation setting.\n- Line 142: What's a \"channel-wise GMM\"?\n- Is there any explanation why the performance drops so much for S3 in Table 4? I was actually wondering what the results look like if you did the same feature fusion as DADA.", "limitations": "One limitation has been discussed in the conclusion (sub-optimal architecture design for depth prediction branch). Potential societal impacts were not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Well written, except a few grammatical issues and typos (please get those checked)\n- The structure of the paper is well done, illustrations help understand the method\n- Ablation studies are mostly good\n- Main results show a good improvement over state-of-the-art\n- Source code is made available\n\n### Weaknesses\n\n- The motivation about class-wise depth distribution in lines 41ff seems to be based heavily on typical driving scenes. It would be better to also think about other types of scenes (e.g., indoor) and see if the same argument holds.\n- Throughout the paper, it was hard to imagine what \"spatial aggregation priors\" could mean until reading the actual section about it. Similarly in line 100, \"the spatial aggregation scale of pixels\" is unclear to me.\n- The depth distribution is advertised as domain bridge, but doesn't this modality depend on the camera settings, which may be different in source and target domains? For instance, if the focal length is different (i.e., more or less zoom), then objects further away would be represented with more pixels and thus likely change the depth distribution.\n- In nearly all equations, it would be better to end it with a comma (\",\") instead of having the comma in the beginning of the next line.\n- For equation 5, I think it would be good to add an intuitive explanation that every pixel in D describes the probability of seeing the depth value with this semantic class at this pixel location under the learned GMM.\n- Lines 155f: The argument about \"balance\" of branches is unclear to me, and also why the loss is named that way.\n- Section 3.5 needs to be re-written. First, it was not clear to me that self-training would be part of the proposed solution. Second, in line 189, it's not clear initially what the thresholds are for and how the pseudo labels are \"refined\" with them. Finally, the spatial prior refinement algorithm needs a more intuitive explanation and potentially a figure for easier understanding.\n- I do not understand the point the first paragraph in Sec. 4.5 (lines 267ff) tries to make.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "180p7kTmECn", "reviewer": "Reviewer_sfyu", "summary": "This paper addresses the task of unsupervised domain adaptation in semantic segmentation. Following the same line as previous works [11,12,13], the paper leverages depth supervision in source domain as the privileged information to improve segmentation in target domain. The paper presents an intuitive idea based on the observation that: pixels of a particular semantic class often have similar positions in the image space and have similar depth ranges across source and target. Based on which, the paper proposes a task of estimating depth distribution density 2D maps, which are then used to guide the final output-level adversarial adaptation. Reported results are better than previous SOTAs.\n\n** Technical details **\nThe network architecture is a standard MTL with three heads: one for the main task of segmentation, one for monocular depth and one for estimation the ``depth distribution density maps'' (see below for more details); the three branches are attached to the same Resnet101 backbone. The two branches segmentation and depth are supervised in source with cross-entropy and berHu losses as done in [12]. Depth distribution density supervision is gotten from segmentation prediction $\\hat{P}$ and either ground-truth depth $Z_s$ in source or estimated depth $\\hat{Z}_t$ in target.\n\nThe depth distribution of each semantic class is modeled as a Gaussian mixture model (GMM) in the image space; GMM parameters are estimated using source data. The number of components is fixed as 5. For each source image, using the depth ground-truth $Z_s$ and segmentation prediction $\\hat{P}_s$, one can produce a depth distribution density map $D_s\\in\\mathbb{R}^{H\\times W}$, i.e. at the position $(x,y)$, the value $D_s(x,y)$ is determined as the depth distribution density at $(x,y)$ of class $\\hat{P}_s(x,y)$. For each target image, depth distribution density map $D_t$ is estimated using depth prediction $\\hat{Z}_t$ and segmentation prediction  $\\hat{P}_t$.\n\nFor adaptation, this work applies adversarial training using an additional discriminator [19,12]. Input to the discriminator is a fusion of segmentation prediction $\\hat{P}$ and estimated depth density map $D$. To further boost performance with self-training, a heuristic pseudo-labelling procedure is used.", "questions": "Questions:\n- Why berhu loss is used for supervising the density branch? What is the reaso?\n\nSome analyses that could improve this work:\n  - Sensitivity analysis of hyper parameters, especially the distribution density one.\n  \n  - Although depth and density estimations are auxiliary tasks, it would be interesting to quantify (as done in [13]) or at least visualize \n  some depth and density results. Does density improve depth?\n\n  - It's not yet been a standard for UDA in segmentation due to high computational cost but it would be highly \n  appreciated if the authors can provide mean and std performance with different runs and seeds.", "limitations": "The conclusion discusses a general limitation on depth quality, same as similar works.\nNo discussion on potential negative societal impacts was given.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strengths **\n- The idea is intuitive and it seems easy to implement the density branch given the availability of existing codebase.\n- Paper presentation is quite clear and easy to understand. However writing could be improved here and there, e.g. L144-146 I don't quite get the meaning.\n\n** Weaknesses **\n- This work lacks quantitative and qualitative analysis of the new branch for density distribution estimation. It's hard for me to imagine what are produced by the density branch. What advantages brought by the joint space of segmentation and density ( as compared to depth + seg joint space)? Some qualitative density results as well as t-SNE/UMAP visualization of joint-space features are needed.\n- The self-training strategy is highly heuristic. Seemingly this strategy help recruit more rare-class pseudo-labels by relaxing the constraints for those (?) I don't understand why the values 50 and 4950 were chosen.\n\n=========== Post-rebuttal:\nI thank the authors for actively answering my questions during the rebuttal period. Most of my concerns were addressed. I'm happy to increase my score.\nIf this paper is accepted, I would like to see in the final version more discussions and qualitative results as promised by the authors, especially on the pseudo-labelling refinement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept