Learning to Adapt via Latent Domains for Adaptive Semantic Segmentation
Authors: Yunan Liu
Shanshan Zhang
Yang Li
Jian Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta-learning
domain adaptation
semantic segmentation

Abstract: Domain adaptive semantic segmentation aims to transfer knowledge learned from labeled source domain to unlabeled target domain. To narrow down the domain gap and ease adaptation difficulty, some recent methods translate source images to target-like images (latent domains), which are used as supplement or substitute to the original source data. Nevertheless, these methods neglect to explicitly model the relationship of knowledge transferring across different domains. Alternatively, in this work we break through the standard “source-target” one pair adaptation framework and construct multiple adaptation pairs (e.g. “source-latent” and “latent-target”). The purpose is to use the meta-knowledge (how to adapt) learned from one pair as guidance to assist the adaptation of another pair under a meta-learning framework. Furthermore, we extend our method to a more practical setting of open compound domain adaptation (a.k.a multiple-target domain adaptation), where the target is a compound of multiple domains without domain labels. In this setting, we embed an additional pair of “latent-latent” to reduce the domain gap between the source and different latent domains, allowing the model to adapt well on multiple target domains simultaneously. When evaluated on standard benchmarks, our method is superior to the state-of-the-art methods in both the single target and multiple-target domain adaptation settings.

Learning to Adapt via Latent Domains for A Adaptive Semantic Segmentation
Yunan Liu, Shanshan Zhang, Yang Li, Jian Yang PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensione
Information of Ministry of Education, Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China {liuyunan, shanshan bhanshan.zharp zhang, yangli1995, (aiyang)nnjuse edu. cn
Abstract
Domain adaptive semantic segmentation aims to transfer knowledge learned from labeled source domain to unlabeled target domain. To narrow down the domain gap and ease adaptation difficulty, some recent methods translate source images to target-like images (latent domains), which are used as supplement or substitute the original source data. Nevertheless, these methods neglect to explicitly model the relationship of knowledge transferring across different domains. Alternatively, in this work we break through the standard "source-target" one pair adaptation framework and construct multiple adaptation pairs (e.g. "source-latent" and "latent- target"). The purpose is to use the meta-knooweedee (how to adapt) learned from one pair guidance assist the adaptation of another pair under meta-learning framework. Furthermore, we extend our method to a more practical setting of open compound domain adaptation (a. multiple-target domain adaptation), where the target is compound of multiple domains without domain labels. In this setting, we embed an additional pair of "latent-latent" to reduce the domain gap between the source and different latent domains, allowing the model to adapt well on multiple target domains simultaneously. When evaluated on standard benchmarks, our method is superior to the state-of-the methods in both the single target and multiple- target domain adaptation settings.
Introduction
Semantic segmentation is popular task in computer vision, which assigns pixel-wise semantic labels for given images. It has been widely utilized facilitate downstream applications such as video surveillance and autonomous driving. Recent progress on image semantic segmentation has been driven by deep neural networks trained on a large amount of labeled data, which are yet expensive to obtain. An alternative way is to generate synthetic images with pixel-level ground truth readily available in an effortless way 11 2] However, the model purely trained on synthetic datasets usually performs rather poor on real data. To address this issue, domain adaptation methods are used to reduce the domain shift and learn domain-invarian representations by minimizing distribution discrepancy between source and target domains 3 4 Following the advances of
generative adversarial nets (GANs) [5], adversarial learning has been used to align representations of different domains in adversarial manner 6
g 9.. Recent studies introduce additional intermediate domain, i.e. latent domain, to narrow down the huge domain gap between source and 19
7
target domains [10 [ 12 13 [4
[5]
16 [7 18
20 21 This is achieved by image-to-img to-image
The corresponding author is Shanshan Zhang.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Adapt
Style transier
(Suurce-to-latent)
Source image
Adapt
Lubel C...............................
Meta-kaon ledge (how to ndapt)
Latent image:
Shared
image:
Lafent image
Unlabcled
Adapt (Latent-to-targee
Laheled snurce domain
targett domain
Latent imane
Label
Target images
(a)
(b)
Comparison of different domain adaptation frameworks for semantic segmentation. (a) Previous frameworks built one domain adaptation pair that directly adapts model from source domain and latent domain to target domain. In contrast, (b) we construct two domain adaptation pairs. The purpose is to use the neta-knowledge learned from one pair as guidance assist the adaptation of another pair under a meta-learning framework.
Figure I:
translation technique, which generates augmented images that share the same contents as well as ground truth labels with the source domain and similar style to the target domain.
Previous works use the latent domain as supplement or substitute to the original source domain to case the adaptation difficulty, following the traditional one-pair domain adaptation framework, as shown
in Fig. [(a) In contrast, in this paper we construct two domain adaptation pairs: source->latemt" and The purpose to use the meta-knowledge learned from one pair of "source->lateee as guidance to assist the adaptation of another pair of Vlatent->taree as shown in Fig. [Tb)] Specifically, we use a meta-learning framework to learn the meta-knowledge, which reveals that model starting from which initial condition can adapt well on the adaptation pair of "source-+laten" Meta-learning (a.k.a. learning learn) has re-surged recently due its efficacy for few-shot deep learning. In our paper. we only borrow the idea of bi-level optimization, to achieve an initialization that is good for both domain adaptation and segmentation. Please note for latent-target adaptation, since we do not have segmentation labels on target, we cannot optimize for the segmentation task. Thus, an initialization that is only good for domain adaptation is not sufficient; it is necessary to obtain an initialization that is good for both domain adaptation and segmentation. On the other hand, this initialization can be well transferred from source-latent to latent-target as it does not overfit to the domain adaptation or segmentation tasks on the latent domain thanks to the property of bi-level optimization. Recently, a couple of works have also successfully applied meta-learning to domain adaptation. The most related work to ours is (22). which requires multiple source domains or proportion of labeled target data. contrast, our proposed method makes use a generated latent domain, and thus is more flexible in practice.
Most existing domain adaptation methods only aim at adapting a model to a single target domain, without considering a more practical scenario where the target consists of multiple data distributions. To investigate a more realistic setting for domain adaptation, we further study the problem of open compound domain adaptation (a.k.a multiple-tanee target domain adaptation) (23] In this setting, the target union of multiple domains without domain labels. To address this challenge, we extend our proposed method from single-target to multi-target domain adaptation. Specifically, we embed an additional domain pair of -latentt to algin arbitrary two latent domains in the meta-training phase, which better learn the meta-knowledge of adaptation across different domains. In the meta- testing phase, the learned meta- parameters are used to initialize model parameters, promoting our model to adapt well on multiple target domains simultancously.
In summary, the contribution of our work three folds. (1) We propose meta-learning framework compatible for both single-targer and multi-targe domain adaptation settings. To the best of our knowledge, this is the first work that adopts meta-nearning framework to handle domain adaptive semantic segmentation. (2) We first generate additional images as a latent domain via style transfer from source to target domains. This latent domain is then used to construct domain pairs for meta- learning, which transfers meta-knowledge of adaptation across different domains. (3) Our approach achieves state-of-the-an performance on several challenging benchmark datasets. Also, we provide comprchensive model analyses for the proposed method.

---

Meta-training
Meta-testing
Inner loop
Outer loop
1. or T? Lais
Sor L?
Lug.
Lup.
Lads. 1
Luez
Fg
GRU
Fe
FG
Fc
GRL
Fc
Labeled image
Unlabeled image
Initialized model
Figure 2: Training procedure of meta-learingg framework for single-target domain adaptation, In the meta-training phase, we construct a domain pair of "source->laten" to learn the meta- meta-knowledge via bi-level optimization. The meta-knowled adapts well from labeled domain to a new unlabeled domain. In the meta-testing phase, we use the learned meta-knowledge guidance to assist the adaptation of "latent->target""
reveals model starting from which initial condition
2
Related Work
Since our method incorporates style transfer with meta-learning optimization for domain adaptive semantic segmentation, will review related works from the above two aspects, respectively.
2.1 Domain Adaptive Semantic Segmentation Based Style Transfer
Recent advances in domain adaptive semantic segmentation have highlighted the complementary role of pixel-level alignment, which is mainly achieved by image translation methods (e.g. style transfer) to translate images from one domain to another. Some works [TO] [1] 16 construct unified framework to learn image translation and domain adaptation in a sequential way. In contrast, BDL 13 proposes a bidirectional learning framework, consisting a closed loop optimize the adaptation and image translation networks alternatively. Some other works [15] [8] investigate cross image translation between source and target domains, enforcing the model produce consistent predictions for the original and translated images with the same contents. Instead of using adversarial learning, TGCF [4] constructs a self-ensembling learning framework, which composed teacher network and student network. To preserve the semantic information of translated image, FDA 19] and PCE 20 introduce Fourier transform to image translation. FDA [99 does not require any training to perform the domain alignment, just using simple Fourier transform and its inverse. PCE [20) uses the phase consistency of Fourier transform to provide more flexibility to align the source domain and target domain. The DHA method [2]] focuses
practical setting of multiple-target domain adaptation. It generates translated images to construct multiple latent domains, each of which contains the same "style" with specific target domain. Then couple of multiple domain classifiers are used to perform the target-to-soune alignment separately. Existing methods ignore explicitly modeling the relationship of cross-domain knowledge transfer, which is expected to promote the adaptation performance.
2.2 Meta-Learning for Deep Neural Networks
Meta-learning for neural network has a long history, but have resurged in popularity recently. This has
been largely driven by its efficacy for few-shot deep learning (24 25
reinforcement learning (26).
hyperparameter optimization (27), and neural architecture search 28
More recently, meta-learning
has also been successfully applied domain adaptation 22
30
31
22]. EAML [3] uses a meta-
adaptation framework to learn representations for adapting the model to continually evolving target domains. Meta-online [22] introduces new framework enhance performance by learning the initial conditions (i.e. model parameter) of existing domain adaptation methods, whereby the proposed framework only suitable for multi-source or semi-supervised domain adaption settings. We draw inspiration from MAML [32), and in particular from the idea of learning initial conditions [22] of neural network optimization that efficiently adapt the model to the target domain. The major difference to 22 that we construct domain pairs using the generated latent domain images, which share the same contents with the source domain and a similar style with the target domain. Such an intermediate domain makes the meta-knooleddee transfer become casier yet more effective.

---

3 Method
3.1 Preliminaries
We define the source domain as Ds and the target domain as D D D K' indicates the number of target domains. Assuming K>1, take it as the multi-target domain setting; when K=1, it degenerates to a single target domain adaptation problem.
JJ where
When assuming K>1, we first need to split the whole target data to multiple domains by discovering different styles. Here, we use the statistics of CNN features (i.e. mean and standard deviations) to represent the domain-specific style information following [21 [3 Based on this representation, we then perform k-means clustering 34 to assign domain label to each target image.
Algorithm 1
Meta- Learning for Single Target Domain Adaptation (STDA)
Input: Labeled data from Dy and D,,, and unlabelled data from Dr. Initialise: Model parameters On learning rate a, B, y, iteration number at each epoch, iteration number / and N in the meta-training and meta-testing phases, respectively.
1: for == 1,2,.... do
2: O = 1
3: Sample (Ds). (DL); and (Dr); uniformly from Ds, Dr and Dr, respectively;
4:
for j 1, ...... do
5:
Sample (Ds), and (Dr), from (Ds); and (Dz),. respectively;
6:
, = ) a V,,-, Luda (1)-1i (Ds),, (Dr),); end for
A\ Inner loop
7:
8:
Oo = ,1 - 3 Ve.., Lee (6); (DL):)
A\ Outer loop for learning initial condition
9:
for 1, ..., .N do
10:
Sample (DL), and (Dr), from (DL); and (Dr), )i respectively.
11:
O, = O n -7 Ve.-1 Luda (6, (DL),, (Dr)n) A\ Meta-testing
12: end for θ, = ON
13: end for
14:
return ,
Bases one source domain and K(K21) target domains, we use an image translation model G to generate augmented images, constructing K additional intermediate domains, i.e. latent domains. For example, the generated image on the k-th latent domain denoted as X;, shares the same content with the i-th source image X and possesses similar style information with 2 image Xix on the k-th target domain, and can be formulated as: Xix G(X X.F.), where is a random index, indicating randomly selecting an image from the target domain. Here, we use recently proposed image translation method (i.e. PCE (00 as G to generate augmented images. In order preserve the semantic information, PCE introduces the Fourier transform and designs phase consistency loss for image translation. Please refer to 20 and 21 for more details.
For the basic pairwise domain adaptation module, we use the adversarial learning strategy. Specifical- ly, the semantic segmentation network F& and domain classification network Fc are optimized via the following unsupervised domain adaptation (UDA) loss Ludaa
Ludo
Leeg + XLadds
where Leeg is the cross-entropy loss using ground truth annotations on the source domain; Lady is the adversarial loss that optimizes FG and fools Fc by maximizing the probability of misclassifying the source and target data; and À the weight used to balance the two terms. For adversarial learning, we simply insert gradient reversal layer (GRL) 440) between F& and Fc, allowing the whole model to be optimized in an end-to-end manner.
In contrast to the traditional one-pair framework (as shown in Fig. I(a),. in this work we propose
meta-learning framework that consists of multiple domain pairs (as shown in Fig. [(b). aiming at learning the meta-knowledge (how to adapt) as guidance for improving the adaption performance. In the following subsections 3.2 we will describe our meta-learning framework for single-target and aulti-target domain adaptation in detail, respectively.

---

Meta-training
Meta-testing
Inner loop
Outer loop
Sor L?
Lu or Laz?
Lkl
or
Lyn.
Lan.
Lega
Lab.
Leg
L Laie
Luz
GRI
GRI
Fc
Fe
Lych
Lany
Initialized model
Figure 3: Training procedure of our meta-learning framework for multiple-target domain adaptation. We randomly select data from two latent domains Lu1 and Lxa in the meta-training phase, and data from two target domains T& and 7k2 in the meta-testing phase. In the outer loop, we construct an additional domain adaption pair of "latent->latent", and define the supervised domain adaptation as the optimization goal. allowing to learn the meta-knowledge that reveals a model starting from which initial condition generalizes on multiple unlabeled domains. In the meta- veta-training phase, we use the learned meta-knowledde to initialize model parameters, promoting our model to adapt well on multiple target domains simultaneously.
3.2 Meta-Knowledge Learning for Single-Targer Domain Adaptation
In this section, we describe the proposed meta-learning framework for single-target domain adaptation (STDA). As shown in Fig. 2 the proposed framework consists of two pairs of domain adaptation: "source->latent" and daatent.>tageet.. In the meta-training phase, we apply the meta-optimization on the pair of "source-lltent" that a base UDA model starting from which initial parameters can adapt well from labeled data (i.e. source domain) to unlabeled data (i.e. latent domain). the meta-testing phase, the learned meta-knowledge are used to guide the optimization of "latent-ttarget" domain adaptation.
learn the meta-knowledge (i.e. initial condition
our case), revealing
(1) Meta-training phase. The optimization procedure of meta-learning for initial condition can be considered as problem of bi-level optimization as follow:
Outer loop
+ arg min Louter (Linner (O; D e D queryy
0
Inner loop
where Linner (9, D Dsupport) and (O", D Dquery) are the losses computed on support set D Dampport and query set D Queery. respectively, and the outer loop optimization starts from O'= arg min Linner once inner loop optimization has been completed. The goal of Eq. 2 is thus to learn the initial condition of base model such that it achieves minimum loss on both support set and query set. When both losses are differentiable, the meta- meta-gradient in Eq. 2 involves gradient through a gradient, which allows gradient update of outer loop to guide the inner loop optimization.
In the setting of unsupervised domain adaptation, the target data cannot be used as Dyuery for outer loop optimization due to lack of annotations. To obtain
additional labeled data serving as D Douery, we use an image translation model to generate augmented images (i.e. latent domain), which share the same contents and labels as the source images, as described in Section [.1] We describe the training procedure of the proposed meta-learning for single-target domain adaption in Algorithm [ In the inner loop, we replace Linner with the UDA loss in Eq. [ as = and assume there is no labels on the latent domain, making the model to adapt from the labeled source domain to the unlabelled latent domain. In the outer loop, we use the latent images with ground truth labels as the query set, which are used to evaluate the adaptation performance via a segmentation loss : In this way, we aim to learn an initial condition O, that enables our base domain adaptation model to adapt effectively on the "source->tan -target" domain pair:
Outer loop
Op arg min do
Σ DI.
Loaeg Tuuda Ou; Ds, Dr): D2)
Inner loop
where Dr and Dr are the identical data from latent domain, while the ground truth labels not used for Di.
5

---

Algorithm 2 Meta-Knowledge Learning for Multiple Target Domain Adaptation (MTDA)
Input: Labeled data Ds from source domain, labeled data D Di. DE from K latent
domains, and unlabeled data Dy=[D9, Di...., D -1 from K target domains.
Initialise: Model parameters Og, iteration number 1, and N, learning rate a, 3, 7.
1: for i = 1, do
2: Og 1;-
3: for j = 1,2,.... J do
4:
θ 0 - a Vōy-1 Linner (O)-1: (Ds),, (D (2) )));
\\ IInee loop
5:
end for
0-= V (Leeg (6,; (O,; (D (D2 ) (6); (DL) (DI)):)) VA Outer loop
6:
7:
for n = 1, ..., .N. do
8:
+, = 1-1-1-11 Ven-1 Inda (aada(On. (On- (D) k2 )n (D (22 )n)
A\ Meta-testing
9:
end for O, N
10: end for
return O
(2) Meta-testing phase. The parameters On learned from meta-training is used initialize the domain adaptation model for "latent->targed" The optimization can be formulated as follows:
θ = arg min Ludaa (On; Dr. Dr).
As shown in Algorithm [ we obtain the model parameters O after meta-testing optimization at the i-th iteration; at the following (i+1)-th iteration, we use O to initialize the model such that When the entire training procedure finishes, we obtain
0o=0,-1=0x (Step 2 in Algorithm an adapted semantic segmentation network FG (as shown in Fig. which is expected to predict reasonable segmentation results on the target domain.
3.3 Meta-Knowledge Learning for Multiple-Target Domain Adaptation
We propose a meta-learning framework consisting of multiple domain adaptation pairs (i.e. "source-+latent, "latent-+latent" and "latent->target") for multiple-target domain adaptation (MT- DA), promoting the adapted model to perform well on multiple target domains simultaneously. Fig. shows the training procedure.
In the setting of MTDA (K > 1). we have labeled da- the source domain Ds and generated latent domains Dz=(D), DI.... DE - and attempt to adapt the UDA model unlabeled target domain ( D D -1). In the meta- veta-training phase, we define the optimization goal is learn the meta-knowledge that a model starting from which initial condition can adapt well from a source domain to mul- tiple latent domains. Expect for applying bi-level optimization on "source->laeent" we also consider the domain adaptation pair of "latent=>laten to reduce domain shift between arbi- trary two latent domains. The motivation of such a design can be represented by toy example, as shown in Fig. 4 Let A, B, and C' indicate the center of data distribution on the source domain, and two latent domains, respectively. Suppose A, B, and C' locate at the three vertices of a triangle, our goal can be simplified to shorten the sides of AB and AC' simultancously. One simple strategy is to directly shorten the above sides, corresponding to applying domain adaptation on the pair of "hource-lattet"" Another strategy (corresponding of BC, so that AB and AC can be further shortened indirectly. Since both strategies contribute to reduce d(A, B) and d( A, C) (d for distance), we thus use two domain adaptation pairs consisting of soource->ltenti and
Align
Distance
Indirvet strafegy
Direct strategy
Figure 4: A toy example for reduc- ing the length of AB and AC.
domain adaptation the pair of "latent-*latenti is to shorten the side
in the meta- veta-training phase to learn the meta- meta-knowledge.
We describe the training procedure of the proposed meta-learning for multiple-target domain adaption in Algorithm 2 and introduce the optimization detail about the meta-training phase and the meta- testing phase as following:

---

(I) Meta- veta-training phase. At each iteration, we randomly select samples L = two latent domains, where k1, (2, I,.,, K 1) and k1 In the inner loop. Fc is used to
DL') from
discriminate the representation from FG belonging to source domain Ds or latent domain D
The loss function Linner of the inner loop can be formulated as follows:
Linner (Oo: Ds. 22 seey( (Qu; Ds) + Lade (Oo; Ds, D
(5)
where D),, indicates latent data which is assumed without labels. In the outer loop, we let Fc discriminate different latent domains. Suppose O" denotes the initial condition of outer loop optimization as @* = arg min Linner, the whole optimization goal of meta-training is defined as:
Oo
arg min Σ (O"s D/ T.ado (0"; DE D))))
(
(6)
dg
Louter (0r :D
where Op is the initial condition (i.e. model parameters) need to be optimized via meta-training optimization for the following meta-testing phase. As shown in Eq the outer loop optimization can be be recognized as solving a problem of supervised domain adaptation between two latent domains, allowing the model to generalize on different latent domains. Specifically, the loss function of outer loop Louter in Eq 6 can be divided into two terms: the former term Lisg takes the role of evaluating the segmentation performance on the query set (i.e. DA1 and D and the latter one Ladd enforces the model to learn domain-invarian representation for different latent domains by applying discrimination the domain pair of "latent=>laeent.
(2) Meta-testing phase. The initial condition Op contains the meta-knowledge of from which initial condition that a UDA model can do well in transferring domain-specific knowledge from labeled source domain to multiple unlabeled latent domains simultaneously. As shown in Fig. 3 we initialize the same UDA model with the learned O₂ as the initial condition, and let Fc discriminate latent from target domains. Then, the optimization goal of meta-testing on the domain pair of "latent->targe" is defined as follow:
θ = arg min Tuuda (9:; 1.2 D4 (2.)
Where DJ and 24 share the similar style DH and D42, respectively. When the entire training procedure finishes, we obtain an adapted semantic segmentation network FG (as shown in Fig, 3 with parameters O, which is expected to predict reasonable segmentation results on the multiple target domains simultaneously.
4 Experiments
4.1
Experimental Settings
Data preparation. In our experiments, we evaluate the proposed method on both single-target and multiple-target settings of unsupervised domain adaptation. Specifically, we take the GTAS dataset [ as the labeled source domain, while the Cityscapes (35) and C-Driving [2]] datasets are adopted as unlabeled single-target domain and multi-target domains, respectively. For the multiple-target setting on the C-Driving dataset, we use the compound data containing "cloudy", "rainy", and "snowy" images as target domain for training. During testing, we evaluate the proposed method on the compound test set and a new unseen open test set containing "overcast" images. For image preprocessing, we resize the shorter side of images to 720 and randomly crop a patch from one image at each iteration with size 600. Besides, horizontal flip is applied as data augmentation. At each iteration, we randomly select samples from each domain for training.
Implementation. We implement our proposed methods using the Py Torch v1 1.2.0 on a single NVIDIA P40 GPU (24G memory). For fair comparison, we adopt the DeepLab V2 (66 with VGG16 (37 as the segmentation network FG. For the domain classification network Fc, we use an architecture similar (6) which consists of 5 conv. layers with kernel 4 4 and stride of 2, and the numbers of channels are (64, 128, 256, 512, 1}. The paree--prraeter À in Eq. I is set to 0.01. The value of "J" and "N" (Algorithm and 2) are set to 2 in our experiments. We use the SGD optimizer with 0.9 momentum and 5x 10-5 weight decay. The learning rates 8, are empirically set to 1x10-1 5x10-5, and 1 x x00- -4 For evaluation, images from target test set are resized to 1024x512 as input and the evaluation metric (i.e. IoU) is calculated on predictions upsampled to the original size.
7

---

Table I: Comparison with state-of-the art UDA methods (VGG16) single-target domain adaptation (i.e. GTAS to Cityscapes). The performance is measured by the intersection-over-um each class and mean IoU (mloU). "A" denotes the improvement compared to source only.
(IoU) for
Method
road
sidewalk
buiding
wall
Fence
pole
light
sigo
veg.
Lerrin
sky
person
Fider
track
bus
midor.
bike
mloti
A
Source unly
61.0
18.5
66.2
18.0
19.6
19.1
22.4
15.5
79.6
28.5
58.0
44.5
1.7
66.6
14.1
11
0.0
3.2
0.7
28.3
CYCADA
85.2
37.2
76.5
21.8
13.0
23.8
22.9
21.5
80.5
31.3
60.7
50.5
9.0
76.9
17.1
28.2
4,5
98
0.0
35.4
7.1
LSE 44
86.0
26.0
76.7
33.1
13.2
21.8
16.5
78.8
25.8
74.7
50.6
18.7
8
22.5
30.5
12.3
1699
25,4
39.0
10.7
CrCDA 42
86.8
37.5
80.4
30.7
18.1
26.8
25.3
15.1
81.5
30.9
72.1
52.8
19.0
82.1
25.4
29.2
10.1
15.8
3.7
39.1
10.8
BDL 1%
89.2
40.9
81.2
29.1
19.2
14.2
29.0
19.6
83.7
35.9
80.7
54.7
23.3
82.7
25.8
28.0
2.3
25.7
199
41.3
13.0
PIT
86.2
35.0
81.9
23,2
1.2
13.5
FDA 19
86.1
35.1
80.6
30.8
20.4
27.5
30.0
26.0
82.1 30.3
73.6
52.5
21.7
81.7
24.0
30.5
29.9
14.6
24.0
42.2
13.9
LTIR 17
92.5
34.5
83.9
34.5
25.5
31.0
30.4
18.0
84,
39.6
83.9
53.6
19.3
81.7
21.1
13.6
17.7
12.3
6.5
42.3
14.0
DIST H1
88.1
35.8
83.1
25.8
23.9
29.2
28.8
28.6
85.0 36.1
82.3
22.8
82.3 26.4
38.6
0.0
19.6
17.1
42.4
14.1
TGCF 14
90.2
51.5
81.1
15.0
10.7 37.5
35.2
28.9
84.1 32.7
75.9
62.7 19.9 82.6 22.9
28.3
23.0
25,4
42.5
14.2
LDR 18
90.1
41.2
82.2
50.3
21.3 18.3
33.5
23.0
84.1 37.5
81.4
$4.2
24.3
X3.0
27.6
32.0
8.1
29.7
26,9
43.6
14.3
FADA H3
92.3
51.1
83.7
33.1
29.1
28.5
28.0
21.0
82.6
32.6
85.3
55.2
28.8 83.5
24.4
37.4
0.0
21.1
13.2
43.8
14.5
PCE (20]
90.2
44.7
40.5
75.1
28.2
30.8
16.3
Our STDA
93.1
57.8
84.1
31.4
36.1
27.5
21.2
37.5
85.5
44.5
83.8
53.8
16.4
82.2
21.7
44.9
0.0
13.2
30.2
45.5
17.2
Table 2: Comparison with state-of-the-ar methods (ResNet101) for single-target domain adaptation. (IoU) for each class and mean IoU
The performance is measured by the intersection-over-ume (mloU). "A" denotes the improvement compared to source only.
Method
road
Sitewalk
bailding
sull
Fonce
pole
light
terrain
sby
person
rider
unic
hus
traini
motor.
bike
mloU
A
Source only
34.8
14.9
29.7
35.5 18.4
13.1
70.4
34.4
34.9
35.7
J
LSE 44
90.2
40.0
83.5
31.9
26.4
32.6
38.7
37.5
81.0
34,2
84.6
61.6
33.4
82.5
32.8
45.9
6.7
29.1
30.6
47.5
13.2
PLCA 47
84.0
30.4
82.4
35.3
24.8
32.2
36.8
24.5
85.5
37.2
78.6
66.9
32.8
85.5
40.4
48.0
88
29.8
41.
47.7
13.4
BDL 13
91.0
44.7
84.2
34.6
27.6
30.2
36.0
36.0
85.0
43.6
83.0
58.6
31.6
83.3
35.3
49.7
3,3
28.8
35.6
48.5
14.2
CiCDA 42
92.4
55.3
82.3
31,2
29.1 32.5
33.2
35.6
83.5
34.8
84.2
58.9
32.2
84.7
40.6
46.
31.1
32.7
48.6
14.3
DIST 41
90.6
44.7
84.8
34.3
28.7 31.6
35.0
37.6
84.7
43.3
85.3
$7.0
31,5
83.8
42,6
48.5
1,9
30.4
39.0
49.2
14.9
LDR 18
90.8
41.4
84.7
35.1
27.5 31.2
38.0
32.8
85.6
42.1
84.9
59.6
34.4
85.0
42.8
52.7
3,4
30.9
38.1
49.5
13.2
CCM
93.5
57.6
40.6
43.2
15.6
FADA 43
91.0
50.6
86.0
43.4
29.8
36.8
43.4
25.0
86.8
38.3
87.4
64.0
38.0
85.2
31.6
46.1
6.5
23.4
37.1
50.1
13.8
LTIR 17
92.0
55.0
85.3
34.2
31.1
34.9
40.7
34.0
85.2
40.1
87.1
61.0
31.1
82.5
32.3
42.9
0.3
36.4
46.1
50.2
1599
CAG
90.4
51.6
25.3
38.2
78.1
34.6
84.7
29.3
37.2
1599
FDA 119
92.5
53.3
82.4
26.5
27.6
36.4
40.6
38.9
82.3
39.8
78.0
62.6 34.4
84.9
34.1
16.9
27.7
46.4
50.5
16.2
PCE 20
91.0
49.2
85.6
37.2
29.7
33.7
38.1
39.2 85.4
35.4
85.1
61.1 3288
84.1 45.6
46.9
0.0
34.2
44.5
50.5
16.2
PIT 38
87.
43.4
78.8
31.2
30.2
36.3
39.9
42.0
79.2
37.1
79.3
65.4
37.5
83.2
46.0
45.6
25.7
23.5
49.9
30.6
16.3
Our STDA
88.4
50.8
82.7
39.4
24.9
34.6
43.7 46.6
84.3
38.6
81.7
61.3
41.9
77.8
$0.4
39.0
5.4
40.3
53.2
51.9
17.6
4.2
Comparison with State of the Art
We evaluate our method on two unsupervised domain adaptation settings: single target and multi- target. The results are presented in Tables I and respectively.
For the single target setting, in Table T we compare our method with 12 state-of-the-ee methods. All the methods are based on VGG16 (37 backbone. It can be seen that our method for single-target domain adaptation (STDA) achieves 45.5% on mloU, and obtains 17.2% improvement compared to the source only model, both of which perform favorably against previous state-of-the-at methods. In Table 2, we compare our method with 13 state-of-the-an methods, which all use ResNer101 (39] as the backbone. It can be seen that our method achieves 51.9% on mIoU, outperforming previous state- of-the-art art methods. These results further demonstrate that our proposed method achieves consistent top results on different backbones.
For the multi- multi-target setting, we compare our method with two recently proposed multi-target meth- ods 21 23 and several of single-targe target methods [4] 6 45 46). following (2] In Table 3 we see that the proposed method for multi-targe domain adaptation (MTDA) outperforms previous methods by large margin w.r.t mIoU (4%), establishing a new state of the art. Besides, we retrain the PCE (20] model under the setting of MTDA, which achieves 31.1% w.r. mloU on the C-Driving dataset. These results indicate that our method outperforms PCE on both STDA and MTDA settings.
8

---

Cityccapes
Image
Source only
PCE
Our STDA
Ground truth
Rainy
Overcast Cloudy Snowy
Image
Source only
DHA
Our MTDA
Ground truth
Figure 5: Qualitative results of ours and previous top methods. For single-target domain adaptation, compare STDA with PCE (20] on Cityscapes (Ist row). For multi-target domain adaptation, we compare our MTDA with DHA [2 on C-Driving (from 2nd to 5th rows are "rainy", "snowy", "cloudy" and "overcast", respectively).
Table 3: Comparison with state-of-the- the-art U-
DA methods on multi-target domain adaptation (i.e.GTA5 to C-Driving). The performance is
Domain: cloudy Classe cloudy building rainy
Domnin: rainy Class: building
measured by the intersection-over-une for eachelass and mean loU (mloU). "A" de-
(IoU)
notes the improvement compared to source only.
Method
Compound (C)
Open (0)
Averuge
rainy snowy cloudy
Source only
16.2
18.0
20.9
21.2
19.1
+
Our
MIDA
AdaptSeg
23.8
22.3
CBST 445
21.3
20.6
23.9
24.7
22.6 3.5
Figure 6: The t-SNE visualization of feature space. For each category (e.g. road/building), the features are better aligned by our MTDA than STDA.
TBN-Net 46
20.6
21.9
26.1
25.5
23.5
4.4
PyCDA
21.7
22.3
25.9
25.4
23.8
OCDA
DHA
27.0
26.3
30.7
32.8
29.2
10.
source only
22.5
Our MTDA
31,5
30.2
33,0
35.0
33.2
10.7
We also provide some qualitative semantic segmentation results in Fig. where observe obvious improvements against both source only and previous top methods. These quantitative and qualitative results demonstrate the effectiveness of our method for unsupervised domain adaptation.
4.3
Model Analysis
To verify the effectiveness of our methods, we provide comparisons on different ways of using the latent domain, as shown Table 4
From the results, we have the following observations: (1) The latent domain eases adaptation via narrowing down the domain gap. Compared @ other methods, using the latent domain in different ways, all perform better. (2) The latent domain can be used to construct multiple pairs, which performs better than the standard one pair. Compared to C. @ and C obtain significant improvements (~~ 3%) by constructing two DA pairs, which distributes the adaptation difficulty on two procedures. (3) keta-knowwedde transition between DA pairs helps more than simple finetuning. In ①. our proposed STDA framework learns meta- knowledge of how to adapt from labeled data to unlabeled data, which is more effective to assist adaptation of "latent->targed" than the finetuning strategy used in @ (4) When the targel daia distributes spursely, our multi-target method further improves the performance. Our MTDA improves over STDA on the C-Driving dataset by modeling the target data multiple domains.

---

Table 4: Model analysis framework design. "A-3B" indicates adapting model from A to B. @ standard one pair UDA using Eq. D @: similar to ( but using two independent domain classifiers for S-+T and L-T; @ first trained on S-IT and then finetuned on L-xT at each iteration; ( our single-target domain adaptation (STDA) method 2 which uses the meta-knowledge learned from S-> to assist the adaptation of L->;; @ multi-target domain adaptation (MTDA) method (Fig
Method GTAS-+
Single-target (+Ciiysccaes))
Multi-target C-Driving))
Number of DA pairs
rainy
cloudy
snowy
overcast
Average
@ S ST
35.7
24.3
21.9
26.4
25.8
① L-+
1
37.2
24.7
23.6
28.3
29.6
27.5
C. (S, L)-+''
1
38.1
25.0
24.8
29.0
30.1
28.2
1 S +T and L-+T
2
40.8
27.9
26.4
29.5
31.6
29.7
(S (S-+)-
2
41.9
29.4
27.7
29.5
31.9
30.4
1. Our STDA (Fig
2
31.3
30.2
32.6
34.4
32.6
2 Our MTDA (Fig)
3
31.5
30.2
33.0
35.0
33.2
We further analyze the feature space learned with our MTDA and STDA. In Fig- 6 We provide the t-SNE visualization [50] of extracted features from F& using our STDA and MTDA, respectively. appears that our MTDA yields more generalized commain-nvvarian features. More specifically, the feature distributions different domains are better aligned in MTDA than that in STDA.
Conclusion
In this paper, propose novel meta- meta-leaming framework for single-target and multi-target domain adaptive semantic segmentation. In particular, we generate latent data via an image translation model and construct multiple domain adaptation pairs. The purpose is from some pairs as guidance to assist the adaptation of "latent-to-targer under a meta-learning framework. Experimental results show the effectiveness of method, which establishes state-of-the-art performance on both settings of single-target and multi-target domain adaptation.
the meta-knowledge learned
Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant No. 62172225), the Funds for International Cooperation and Exchange of the National Natural Science Foundation of China (Grant No. 61861 136011), 1), the Fundamental Research Funds for the Central Universities (Grant No.309200322011, 1), and the National Key Research and Development Program of China under Grant 2017YFC0820601.
References
[1] R. Richter, V. Vineet, S. Roth, and V. Koltun, Playing for data: Ground truth from computer games. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 102-118, 2016.
[2] Ros. Sellart, Materzynska, D. Vazquez, and A. Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 3234-3243, 2016.
[3] Y. Chen, W. Li, and L. V. Gool. ROAD: Reality oriented adaptation for semantic segmentation of urban scenes. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 7892-7901, 2018.
[4] Q Lian, L. Duan. F. Lv, and B. Gong. Constructing self-motivated pyramid curriculums for moss--ommain semantic segmentation: A non-adversarial approach. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pages 6757-6766, 2019.
[5] I. Goodfellow, J. P. Abadic, M. Mirza, B. Xu, D. W. Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial nets. In Proc. Conf. Neural Information Processing Systems (NeurIPS), pages 2672-2680, 2014.
[6] Y.-H. Tsai, W.C. Hung. S. Schulter, K. Sohn, M.-H. Yang. and M. Chandraker. Learning to adapt structured output space for semantic segmentation. In Proc, IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 7472-7488 2018
10

---

[7] Y. Luo, L. Zheng, T. Guan, J. Yu, and Y. Yang. Taking closer look at domain shift: Category- adversaries for semanties consistent domain adaptation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 2507-2516, 2019.
[8] Y. Luo, P. Liu, T. Guan, J. Yu, and Y. Yang. Significance-wwre information bottlneck for domain adaptive semantic segmentation. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pages 6777-6788 6776, 2019.
19] L. Du, J. Tan, H. Yang. J. Feng, X. Xue, Q. Zheng, X. Yc, and X. Zhang. SSF-DAN: Separated semantic feature based domain adaptation network for semantic segmentation. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV). pages 982-991, 2019.
[10] Z. Murez, S. Kolouri. D. J. Kriegman, R. Ramamoorhi, and K. Kim. Image to image translation for domain adaptation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 4500-4509, 2018.
[11] J. Hoffmann E. Tzeng, T. Park, J.-Y. Zhu, Isola, K. Saenko, A. Efros, and T. Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Proc. International Conference on Machine Learning (ICML.), pages 1994-2003, 2018.
[12] Y. Zhang, Z. Qiu, T. Yao, D. Liu, and T. Mei. Fully convolutional adaptation networks for semantic segmentation. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pages 6810-6818 18, 2018.
[13] Y. Li,L. L. Yuan, and N. Vasconcelos. Bidirectional learning for domain adaptation semantic segmentation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 6936-6945, 2019.
[14] J. Choi. T. Kim, and C. Kim. Self eelfmennemiiing with GAN-based data augmentation for domain adaptation in semantic segmentation. In Proc. IEEF Int. Conf. Comput. Vis. (ICCV), pages 6829-68339, 2019.
[15] Y.-C. Chen, Y.-Y. Lin, M.-H. Yang, and J.-B. Huang. CrDoCo: Pixel-level domain transfer with cross- domain consistency. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 1791-1800, 2019.
[16] Z. Wu, X. Han, Y.-., Lin, M. G. Uzunbas, T. Goldstein, S. Lim, and L. S. Davis. DCAN: Dual channel-wise alignment networks for unsupervised scene adaptation. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 535 552, 2018.
[17] M. Kim, and H. Byun. Learning texture invariant representation for domain adaptation of semantic segmentation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pages 12972-12981, 2020.
[18] Yang, W. An, Wang, X. Zhu, C. Yan, and Huang, Label-driven reconstruction for domain adaptation in semantic segmentation. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 480-498, 2020.
[19] Y. Yang, and S. Soatto. FDA: Fourier domain adaptation for semantic segmentation. In Proc. IEEE Conf.
Comput. Vis. Pattern Recognit. (CVPR), pages 4084-4094, 2020.
[20] Y. Yang, D. Lao, G. Sundaramoorthi, and S. Soatoo. Phase consistent ecological domain adaptation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 9008-9017, 2020.
[21] K. Park, S. Woo, I. Shin, and I. S. Kweon. Discover, hallucinate, and adapt: Open compound domain adaptation for semantic segmentation. In Proc. Conf. Neural Information Processing Systems (NeurIPS), 2020.
[22] D. Li, and T. M. Hospedales. Online meta-learning for multi-source and semi-superrised domain adaptation.
In Proc. Fur. Conf. Comput. Vis. (ECCV), pages 382-403, 2020.
[23] Z. Liu, Z.. Miao, X. Pan, X. Zhan, D. Lin, S. X. Yu, and B. Gong. Open compound domain adaptation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 12403-12412, 2020.
[24] S. Khodadadch, L. Boloni, and M. Shah. Unsupervised meta-leaming for few-shot image classification. In Proc. Conf. Neural Information Processing Systems (NeurIPS), pages 00132-0112,, 2019.
[25] M. Goldblum, L. Fowl, and T. Goldstein. Adversarially robust fow-shot learning: A meta-learning approach. Proc. Conf. Neural Information Processing Systems (NeurIPS), 2020.
[26] R. L. Gutierrez, and M. Leonetti. Information- thheoretcc task selection for meta-reinforreemet learning. In Proc. Conf. Neural Information Processing Systems (NeurIPS). 2020.
[27] S. Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperparameters. In Proc. Conf. Neural Information Processing Systems (NeurIPS), 2020.
[28] H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. Proc. International Conference on Learning Representations (ICLR), 2019.
[29] K. Arndt, M. Hazara, A. Ghadirzadeh, and V. Kyrki. Meta reinforcement learning for sim-to-real domain adaptation. In IEEE Int. Conf. Roboties and Automation (ICRA), pages 2725-2731, 2020.
[30] K. Qian, W. Wei, and Z. Yu. A student-teacher architccture for dialog domain adaptation under the meta-leand meta-learmig setting. AAAI Conf. Artificial Intelligence (AAAI), 2021.
[31] H. Liu, M. Long, J. Wang, and Y. Wang. Learning to adapt to evolving domains. In Proc. Conf. Neural Information Processing Systems (NewrIPS), 2020.
11

---

[32] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proc. International Conference Machine Learning (ICML), pages 1126-1135, 2017.
[33] V. Dumoulin, J. Shlens, and M. Kudlur. A learned representation for artistic style. In Proc. International Conference on Learning Representations (ICLR), 2017.
[34] T. Kanungo, D. M. Mount, N. Netanyahu, C. Piatko, R. Silverman, and Y. Wu. efficient means clustering algorithm: Analysis and implementation. IEEE Trans. Puttern Anal. Mach. Intell. (TPAMI). 24(7): 88 18992, 2002.
[35] Cordts, M. Omran, S. Pamos, T. Rehfeld, Enzweiler, R. Benenson, U. Franke, Roth, and Schiele. The cityscapes dataset for semantic urban scone understanding. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 3213-3223, 2016.
[36] L..C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and L.. Yuille. Deeplabb Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI), 40(4): 834-848, 2018.
[37] K. Simonyan, and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proe. International Conference on Learning Representations (ICLR), 2015.
[38] F. Lv, T. Liang, X. Chen, and G. Lin. Cross-domain semantic segmentation via domain-invariant interactive relation transfer. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 4333 4342, 2020.
[39] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 770-778, 2016.
[40] Y. Ganin, and V. S. Lempitsky. Unsupervised domain adaptation by backpropagation. In Proc. International Conference on Machine Learning (ICML), pages 1180 1189, 2015.
[41] Z. Wang, M. Yu, Y. Chao, R. Feris, J. Xiong, W. M. Hwu, S. Huang, and H. Shi. Differential treatment for stuff things: A simple unsupervised domain adaptation method semantic segmentation. Proc. IEEE Conf. Comput, Vis. Pottern Recognit. (CVPR), pages 12632-12641, 2020.
[42] J. Huang, S. Lu, D. Guan, X. Zhang. Contextual-relation consistent domain adaptation for semantic segmentation. Proc. Eur. Conf. Comput. Vis. (ECCV), pages 705-722. 2020.
[43] H. Wang, T. Shen, W. Zhang, I.-Y. Duan, and T. Mei. Classes matter: A fine-grained adversarial approach to cross-domain semantic segmentation. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 642-659, 2020.
[44] N. Subhani, and M. Ali. Learning from scale-invariant examples for domain adaptation in semantic segmentation. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 290-306, 2020.
[45] Y. Zhou, Z. Yu, B. V. K. Vijaya Kumar, J. Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Pric. Eur. Conf. Comput. Vis. (ECCV), pages 297-313, 2018.
[46] X. Pan, P. Luo, J. Ship, and X. Tang. Two once: Enhancing learning and generalization capacities via
IBN-Net.. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 484-500, 2018.
[47] G. Kang. Y. Wei, Y. Yang, Y. Zhuang, and A. G. Hauptmann. Pixel-level cycle association: new perspective for domain adaptive semantic segmentation. In Proc. Conf. Neural Information Processing Systems (NeurIPS), 2020.
[48] G. Li, G. Kang, W. Liu, Y. Wei, and Y. Yang. Content matching for domain adaptive semantic segmentation. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 440 456, 2020.
[49] Q Zhang, Zhang, W. Liu, and D. Tao. Category anchor-guided unsupervised domain adaptation for semantic segmentation. In Proc. Conf. Neural Information Processing Systems (NeurIPS), pages 433-443, 2019.
[50] L. v. d. Maaten, and G. E. Hinton. Visualizing data using t-sne, Journal of machine learning research, 9(No)): 2579-2605, 2008
12


Reviews:
[{"review_id": "zuk9gcLt3HW", "reviewer": "Reviewer_y8x3", "summary": "The paper works on the domain adaptation task for semantic segmentation with either one target domain or multiple target domains. In order to solve this problem, the author proposed a novel meta-learning framework to explore the better initialization for the adaptive model and achieve a clear improvement compared to other SOTA methods. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall speaking, the paper is good. It is easy to read and the proposed method is novel. The experiments were well designed and further complemented by the supplementary material. My main concerns about the submission are listed as follows.\n\n1. I wonder whether the author used any self-training methods in the paper to achieve the results shown in Table 1 of the main paper and Table 2 in the supplementary material. I think most of the baselines in both tables are achieved via self-training kind of techniques by generating pseudo-labels in various ways. If the author does not use self-training, I will be astonished by the performance and hope to see what the results will be if self-training is further added. If the author used self-training, I would like to see the comparison to the baselines without using any self-training technique  to further evaluate the achivement given by the proposed method.\n\n2. The mechasim of meta-learning used for domain adaptation is not very clear to me. In classicial meta-learning,  the meta-training often contains seveval datasets and let the network learn how to adapt to novel dataset. Thus during meta-testing phase, the model can fit to the test dataset in just a few iterations. However, in the paper the meta-training only contains a single dataset pair for the single target domain adaptataion case. So I think the model is not fully trained such that it knows how to fit to a novel dataset. This problem can be solved by training the model with multiple dataset pairs which is missing in the paper. I hope the author can further show enough explanation and it will be great if the author can provide more experimental results for the case I mentioned above.\n\n3. The adaptation performance for ResNet-101 is more ciritcal and the author might need to consider to move the results in supplementary material to the main paper.\n\n\nfinal comments after rebuttal\n\nAfter carefully reading the paper and discussing it with the authors, my concern is well addressed. Although other reviewers question the novelty of this paper about using meta-learning, meta-learning is easier to be used for domain generalization or multi-source domain adaptation problems, where there are multiple domains. I think domain adaptation is a different case and generating a latent domain to help build the meta-learning setting is still novel. Thus I raise my rating and suggest accepting this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "ZSvjbiS_ew", "reviewer": "Reviewer_fSWY", "summary": "This paper proposes a meta-learning-based UDA framework for semantic segmentation, which adopts meta-knowledge learned from source-latent adaptation to teach latent-target adaptation gradually. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well organized and the method is explained in details. But there are stills some limitations in this work:\n 1. The idea is not novel enough. On the one hand, meta-learning strategy has been applied to UDA for classification. On the other hand, meta-knowledge learning looks very similar to pre-training and fine-tuning strategy, both of whom aim to alleviate the difficulty of direct transfer from source to target. The only difference is that this method adopts an iterative strategy to update the model. \n\n2. The method is somewhat complicated, which incorporates transfer learning and adversarial learning. This will increase the complexity of model and impedes reproducibility. In addition, the author build many adaptation pairs, such as\"source to latent\", \"latent to target\", and \"latent to latent\"pairs, which makes the model too difficult to train.\n\n 3. The experimental results are not convincing. The authors claim that this paper achieves SOTA on several benchmarks, but some works are missed in this paper, e.g., Self-supervised Augmentation Consistency for Adapting Semantic Segmentation (CVPR 2021), which achieves better performance. In addition, the authors only use VGG as backbone, while recent works adopt resnet101. I think the authors would better compare with these new methods with the same setting, such as\n (1) Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (CVPR 2021).\n (2)Category anchor-guided unsupervised domain adaptation for semantic segmentation (NeurIPS2019)\n\n 4. Since this model is suitable for open compound adaptation, it would be better to compare this method to other open compound adaptation methods, such as references [21]. [23].\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have addressed the limitation and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "ZQC2wihvfYO", "reviewer": "Reviewer_Pvnj", "summary": "The paper proposes a meta-learning framework for open compound domain adaptation. The method uses k-mean clustering to estimate the domain of target samples and employs the image translation method to augment datasets. Then the augmented images are used for the meta-learning framework. The method outperforms baselines on Cityscapes and C-Driving. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1. While the paper improves performance, the novelty is somewhat limited since the paper combines the existing image translation method for DA and meta-learning framework. \n2. The paper is hard to read and explanations are not compact. Especially, notation is not very clear to understand. \n3. The paper should explore the effect of k-means clustering. If there is no k-mean clustering (i.e., all target images are treated as single target domain), how well does the method work? At the same time, it would be better to provide accuracy of SOTA single target DA baselines (e.g., [20]) in Table 2.\n4. No sensitivity analysis, no visualization of generated images, and no theoretical analysis.\n5. How many runs are averaged in the experiments? (standard deviations?)\n6. GTA5 is the only source domain in the experiments. Why don't use the Synthia dataset? In addition why there is a big gap between source-only models?\n\n\n-- POST REBUTTAL -- \n\nI appreciate the effort for the response, but my concern about the experiment still remains. The experiments are performed only on one dataset as a source domain. I encourage authors to explore other datasets to verify the effectiveness of the method. In addition, other backbones (e.g., ResNet) should be explored. I also agree with the reviewer fSWY about concerns on the novelty. Therefore, I keep my score as weak reject.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are well addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "YHeicpZiTcl", "reviewer": "Reviewer_owPT", "summary": "This work introduces a meta-learning framework, based on MAML, for single-target DA (STDA) and open compound DA (OCDA) in semantic segmentation. Similar to Li et al. [22], the idea is to learn the initial condition (model's parameters) that is more favorable for target adaptation. The proposed framework makes use of an existing image-2-image model to translate source images into, what are called here, latent images; this is to have target-like annotated data to guide meta-optimization.\n\nEach iteration of the proposed algorithm is composed of two consecutive steps: (1) meta-training operating on source and latent data to find the favorable initial condition for the next step and (2) meta-testing to perform target adaptation starting from the learned initial condition. \n\nThe meta-training optimization, similar to MAML, involves inner and outer loops. In case of STDA, the inner loop performs UDA optimization steps on labeled source and unlabeled latent samples; the outer loop, based on the optimized weights of the inner loops, meta-optimizes the network to minimize segmentation loss computed using labeled latent data. In case of OCDA with k target sub-domains, k-mean clustering is first done to assign sub-domain pseudo-labels to target images. This information is used in the image-2-image operation to translate source data into sub-domain-like data. Different to STDA, there is an additional adversarial objective in the outer loop.\n\nTo evaluate performance, this work consider two DA benchmarks: GTA5->Cityscapes (STDA) and Cityscapes->C-Driving (OCDA). Reported results show improvement of the proposed framework as compared to previous approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is the first work adopting meta-learning for domain adaptation semantic segmentation. Improvement over state-of-the-art methods demonstrates the merit of the proposed framework. The paper presentation is quite clear and easy to follow. In general this is a decent work and I'm leaning toward the positive side. There are though still a few questions and concerns that I hope to receive satisfactory answers.\n\n- What are the exact values of I,J and N (Algos 1 and 2) used in the experiments?\n\n- In the OCDA experiment, clustering is done on features coming from which model? Using domain labels, may the authors provide the clustering performance? How does bad clustering performance affect the final results?\n\n- The authors took the segmentation and discriminator models in [6] but instead of using the proposed GAN-like adversarial training in [6], the authors opted to GRL. What is the reason of this choice?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations of their work. Potential negative societal impact was not provided.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the discussion period this paper received mixed scores with one recommendation for rejection, one for acceptance, one leaning reject and one leaning accept. There was disagreement about the novelty of the proposed approach. The key novelty claimed by the authors is using a translated source image as a domain unto itself and performing meta learning to learn an adaptable model from source $\rightarrow$ latent such that further adaptation from latent $\rightarrow$ target should be more effective. All reviewers agreed that the image translation portion (i.e., generation of latent domain) was based on prior work. The debate then centered around whether the introduction of a meta learning objective using this translated image was sufficient contribution as well as whether that idea proved effective empirically. After considering all reviewer comments, discussion, author rebuttal and examining the paper, it seems that although prior work has used meta learning for domain generalization the specific use in this paper for the latent domain together with the ablation study in Table 3 showing the value of using the meta objective instead of just adapting directly from L$\rightarrow$T implies that this design decision is useful and novel. It does however seem that the study into the open compound target together with using existing clustering from [21] is less of a contribution and I would encourage the authors to revise the text to clarify their key contribution. Further, as promised please move the ResNet experiments to the main paper. 
Decision: Accept (Poster)