Learning to Adapt via Latent Domains for Adaptive Semantic Segmentation
Year: 2022

Abstract:
Domain adaptive semantic segmentation aims to transfer knowledge learned from labeled source domain to unlabeled target domain. To narrow down the domain gap and ease adaptation difficulty, some recent methods translate source images to target-like images (latent domains), which are used as supplement or substitute to the original source data. Nevertheless, these methods neglect to explicitly model the relationship of knowledge transferring across different domains. Alternatively, in this work we break through the standard “source-target” one pair adaptation framework and construct multiple adaptation pairs (e.g. “source-latent” and “latenttarget”). The purpose is to use the meta-knowledge (how to adapt) learned from one pair as guidance to assist the adaptation of another pair under a meta-learning framework. Furthermore, we extend our method to a more practical setting of open compound domain adaptation (a.k.a multiple-target domain adaptation), where the target is a compound of multiple domains without domain labels. In this setting, we embed an additional pair of “latent-latent” to reduce the domain gap between the source and different latent domains, allowing the model to adapt well on multiple target domains simultaneously. When evaluated on standard benchmarks, our method is superior to the state-of-the-art methods in both the single target and multiple-target domain adaptation settings.

heading: 1 Introduction
text: Semantic segmentation is a popular task in computer vision, which assigns pixel-wise semantic labels for given images. It has been widely utilized to facilitate downstream applications such as video surveillance and autonomous driving. Recent progress on image semantic segmentation has been driven by deep neural networks trained on a large amount of labeled data, which are yet expensive to obtain. An alternative way is to generate synthetic images with pixel-level ground truth readily available in an effortless way [1, 2]. However, the model purely trained on synthetic datasets usually performs rather poor on real data. To address this issue, domain adaptation methods are used to reduce the domain shift and learn domain-invariant representations by minimizing distribution discrepancy between source and target domains [3, 4]. Following the advances of generative adversarial nets (GANs) [5], adversarial learning has been used to align representations of different domains in an adversarial manner [6, 7, 8, 9]. Recent studies introduce an additional intermediate domain, i.e. latent domain, to narrow down the huge domain gap between source and target domains [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]. This is achieved by image-to-image
∗The corresponding author is Shanshan Zhang.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
translation technique, which generates augmented images that share the same contents as well as ground truth labels with the source domain and similar style to the target domain.
Previous works use the latent domain as supplement or substitute to the original source domain to ease the adaptation difficulty, following the traditional one-pair domain adaptation framework, as shown in Fig. 1(a). In contrast, in this paper we construct two domain adaptation pairs: “source→latent” and “latent→target”. The purpose is to use the meta-knowledge learned from one pair of “source→latent” as guidance to assist the adaptation of another pair of “latent→target”, as shown in Fig. 1(b). Specifically, we use a meta-learning framework to learn the meta-knowledge, which reveals that a model starting from which initial condition can adapt well on the adaptation pair of “source→latent”. Meta-learning (a.k.a. learning to learn) has re-surged recently due to its efficacy for few-shot deep learning. In our paper, we only borrow the idea of bi-level optimization, to achieve an initialization that is good for both domain adaptation and segmentation. Please note for latent-target adaptation, since we do not have segmentation labels on target, we cannot optimize for the segmentation task. Thus, an initialization that is only good for domain adaptation is not sufficient; it is necessary to obtain an initialization that is good for both domain adaptation and segmentation. On the other hand, this initialization can be well transferred from source-latent to latent-target as it does not overfit to the domain adaptation or segmentation tasks on the latent domain thanks to the property of bi-level optimization. Recently, a couple of works have also successfully applied meta-learning to domain adaptation. The most related work to ours is [22], which requires multiple source domains or a proportion of labeled target data. In contrast, our proposed method makes use of a generated latent domain, and thus is more flexible in practice.
Most existing domain adaptation methods only aim at adapting a model to a single target domain, without considering a more practical scenario where the target consists of multiple data distributions. To investigate a more realistic setting for domain adaptation, we further study the problem of open compound domain adaptation (a.k.a multiple-target domain adaptation) [23]. In this setting, the target is a union of multiple domains without domain labels. To address this challenge, we extend our proposed method from single-target to multi-target domain adaptation. Specifically, we embed an additional domain pair of “latent→latent” to algin arbitrary two latent domains in the meta-training phase, which better learn the meta-knowledge of adaptation across different domains. In the metatesting phase, the learned meta-parameters are used to initialize model parameters, promoting our model to adapt well on multiple target domains simultaneously.
In summary, the contribution of our work is three folds. (1) We propose a meta-learning framework compatible for both single-target and multi-target domain adaptation settings. To the best of our knowledge, this is the first work that adopts meta-learning framework to handle domain adaptive semantic segmentation. (2) We first generate additional images as a latent domain via style transfer from source to target domains. This latent domain is then used to construct domain pairs for metalearning, which transfers meta-knowledge of adaptation across different domains. (3) Our approach achieves state-of-the-art performance on several challenging benchmark datasets. Also, we provide comprehensive model analyses for the proposed method.

heading: 2 Related Work
text: Since our method incorporates style transfer with meta-learning optimization for domain adaptive semantic segmentation, we will review related works from the above two aspects, respectively.

heading: 2.1 Domain Adaptive Semantic Segmentation Based on Style Transfer
text: Recent advances in domain adaptive semantic segmentation have highlighted the complementary role of pixel-level alignment, which is mainly achieved by image translation methods (e.g. style transfer) to translate images from one domain to another. Some works [10, 11, 12, 16, 17] construct a unified framework to learn image translation and domain adaptation in a sequential way. In contrast, BDL [13] proposes a bidirectional learning framework, consisting a closed loop to optimize the adaptation and image translation networks alternatively. Some other works [15, 18] investigate cross image translation between source and target domains, enforcing the model to produce consistent predictions for the original and translated images with the same contents. Instead of using adversarial learning, TGCF [14] constructs a self-ensembling learning framework, which is composed of a teacher network and a student network. To preserve the semantic information of translated image, FDA [19] and PCE [20] introduce Fourier transform to image translation. FDA [19] does not require any training to perform the domain alignment, just using a simple Fourier transform and its inverse. PCE [20] uses the phase consistency of Fourier transform to provide more flexibility to align the source domain and target domain. The DHA method [21] focuses on a more practical setting of multiple-target domain adaptation. It generates translated images to construct multiple latent domains, each of which contains the same “style” with a specific target domain. Then a couple of multiple domain classifiers are used to perform the target-to-source alignment separately. Existing methods ignore explicitly modeling the relationship of cross-domain knowledge transfer, which is expected to promote the adaptation performance.

heading: 2.2 Meta-Learning for Deep Neural Networks
text: Meta-learning for neural network has a long history, but have resurged in popularity recently. This has been largely driven by its efficacy for few-shot deep learning [24, 25], reinforcement learning [26], hyperparameter optimization [27], and neural architecture search [28]. More recently, meta-learning has also been successfully applied to domain adaptation [29, 30, 31, 22]. EAML [31] uses a metaadaptation framework to learn representations for adapting the model to continually evolving target domains. Meta-online [22] introduces a new framework to enhance performance by learning the initial conditions (i.e. model parameter) of existing domain adaptation methods, whereby the proposed framework is only suitable for multi-source or semi-supervised domain adaption settings. We draw inspiration from MAML [32], and in particular from the idea of learning initial conditions [22] of neural network optimization that efficiently adapt the model to the target domain. The major difference to [22] is that we construct domain pairs using the generated latent domain images, which share the same contents with the source domain and a similar style with the target domain. Such an intermediate domain makes the meta-knowledge transfer become easier yet more effective.

heading: 3 Method

heading: 3.1 Preliminaries
text: We define the source domain as DS and the target domain as DT={D0T , D1T , ..., D K−1 T }, where K indicates the number of target domains. Assuming K>1, we take it as the multi-target domain setting; when K=1, it degenerates to a single target domain adaptation problem.
When assuming K>1, we first need to split the whole target data to multiple domains by discovering different styles. Here, we use the statistics of CNN features (i.e. mean and standard deviations) to represent the domain-specific style information following [21, 33]. Based on this representation, we then perform k-means clustering [34] to assign a domain label to each target image.
Algorithm 1 Meta-Knowledge Learning for Single Target Domain Adaptation (STDA) Input: Labeled data from DS and DL, and unlabelled data from DT . Initialise: Model parameters Θ0, learning rate α, β, γ, iteration number I at each epoch, iteration
number J and N in the meta-training and meta-testing phases, respectively. 1: for i = 1, 2, ..., I do 2: Θ̃0 = Θi−1; 3: Sample (DS)i, (DL)i and (DT )i uniformly from DS , DL and DT , respectively; 4: for j = 1, 2, ..., J do 5: Sample (DS)j and (DL)j from (DS)i and (DL)i, respectively; 6: Θ̃j = Θ̃j−1 − α5Θ̃j−1 Luda(Θ̃j−1; (DS)j , (DL)j); \\ Inner loop 7: end for 8: Θ0 = Θi−1 − β 5Θi−1 Lseg(Θ̃J ; (DL)i); \\ Outer loop for learning initial condition 9: for n = 1, 2, ..., N do
10: Sample (DL)n and (DT )n from (DL)i and (DT )i, respectively. 11: Θn = Θn−1 − γ 5Θn−1 Luda(Θn−1; (DL)n, (DT )n) \\ Meta-testing 12: end for Θi = ΘN 13: end for 14: return ΘI
Bases on one source domain and K(K≥1) target domains, we use an image translation model G to generate augmented images, constructing K additional intermediate domains, i.e. latent domains. For example, the generated image on the k-th latent domain denoted as XiLk shares the same content with the i-th source image XiS and possesses similar style information with z-th image X z Tk on the k-th target domain, and can be formulated as: XiLk = G(X i S , X z Tk), where z is a random index, indicating randomly selecting an image from the target domain. Here, we use a recently proposed image translation method (i.e. PCE [20]) as G to generate augmented images. In order to preserve the semantic information, PCE introduces the Fourier transform and designs a phase consistency loss for image translation. Please refer to [20] and [21] for more details.
For the basic pairwise domain adaptation module, we use the adversarial learning strategy. Specifically, the semantic segmentation network FG and domain classification network FC are optimized via the following unsupervised domain adaptation (UDA) loss Luda:
Luda = Lseg + λLadv, (1)
where Lseg is the cross-entropy loss using ground truth annotations on the source domain; Ladv is the adversarial loss that optimizes FG and fools FC by maximizing the probability of misclassifying the source and target data; and λ is the weight used to balance the two terms. For adversarial learning, we simply insert a gradient reversal layer (GRL) [40]) between FG and FC , allowing the whole model to be optimized in an end-to-end manner.
In contrast to the traditional one-pair framework (as shown in Fig. 1(a)), in this work we propose a meta-learning framework that consists of multiple domain pairs (as shown in Fig. 1(b)), aiming at learning the meta-knowledge (how to adapt) as guidance for improving the adaption performance. In the following subsections 3.2 and 3.3, we will describe our meta-learning framework for single-target and multi-target domain adaptation in detail, respectively.

heading: 3.2 Meta-Knowledge Learning for Single-Target Domain Adaptation
text: In this section, we describe the proposed meta-learning framework for single-target domain adaptation (STDA). As shown in Fig. 2, the proposed framework consists of two pairs of domain adaptation: “source→latent” and “latent→target”. In the meta-training phase, we apply the meta-optimization on the pair of “source→latent” to learn the meta-knowledge (i.e. initial condition in our case), revealing that a base UDA model starting from which initial parameters can adapt well from labeled data (i.e. source domain) to unlabeled data (i.e. latent domain). In the meta-testing phase, the learned meta-knowledge are used to guide the optimization of “latent→target” domain adaptation. (1) Meta-training phase. The optimization procedure of meta-learning for initial condition can be considered as a problem of bi-level optimization as follow:
Θ = Outer loop︷ ︸︸ ︷ arg min
θ Louter(Linner(Θ;Dsupport)︸ ︷︷ ︸
Inner loop
, Dquery) (2)
whereLinner(Θ, Dsupport) andLouter(Θ∗, Dquery) are the losses computed on support setDsupport and query set Dquery , respectively, and the outer loop optimization starts from Θ∗= arg minLinner once inner loop optimization has been completed. The goal of Eq. 2 is thus to learn the initial condition of base model such that it achieves minimum loss on both support set and query set. When both losses are differentiable, the meta-gradient in Eq. 2 involves a gradient through a gradient, which allows gradient update of outer loop to guide the inner loop optimization.
In the setting of unsupervised domain adaptation, the target data cannot be used as Dquery for outer loop optimization due to lack of annotations. To obtain some additional labeled data serving as Dquery , we use an image translation model to generate augmented images (i.e. latent domain), which share the same contents and labels as the source images, as described in Section 3.1. We describe the training procedure of the proposed meta-learning for single-target domain adaption in Algorithm 1. In the inner loop, we replace Linner with the UDA loss in Eq. 1 as Linner:=Luda, and assume there is no labels on the latent domain, making the model to adapt from the labeled source domain to the unlabelled latent domain. In the outer loop, we use the latent images with ground truth labels as the query set, which are used to evaluate the adaptation performance via a segmentation loss Louter:=Lseg. In this way, we aim to learn an initial condition Θ0 that enables our base domain adaptation model to adapt effectively on the “source→target” domain pair:
Θ0 =
Outer loop︷ ︸︸ ︷ arg min
θ0 ∑ DS ,DL,DL Lseg(Luda(Θ0;DS , DL)︸ ︷︷ ︸ Inner loop , DL) (3)
where DL and DL are the identical data from latent domain, while the ground truth labels are not used for DL.
Algorithm 2 Meta-Knowledge Learning for Multiple Target Domain Adaptation (MTDA) Input: Labeled dataDS from source domain, labeled dataDL={D0L, D1L, ..., D K−1 L } fromK latent
domains, and unlabeled data DT={D0T , D1T , ..., D K−1 T } from K target domains. Initialise: Model parameters Θ0, iteration number I , J and N , learning rate α, β, γ. 1: for i = 1, 2, ..., I do 2: Θ̃0 = Θi−1; 3: for j = 1, 2, ..., J do 4: Θ̃j = Θ̃j−1 − α5Θ̃j−1 Linner(Θ̃j−1; (DS)j , (D k1,k2 L )j); \\ Inner loop
5: end for 6: Θ0=Θi−1−β 5Θi−1 (Lseg(Θ̃J ; (D k1,k2 L )i)+Ladv(Θ̃J ; (D k1 L )i, (D k2 L )i)); \\ Outer loop 7: for n = 1, 2, ..., N do 8: Θn = Θn−1−γ 5Θn−1 Luda(Θn−1; (D k1,k2 L )n, (D k1,k2 T )n) \\ Meta-testing
9: end for Θi = ΘN 10: end for 11: return ΘI
(2) Meta-testing phase. The parameters Θ0 learned from meta-training is used to initialize the domain adaptation model for “latent→target”. The optimization can be formulated as follows:
Θ = arg min θ Luda(Θ0;DL, DT ). (4)
As shown in Algorithm 1, we obtain the model parameters ΘN after meta-testing optimization at the i-th iteration; at the following (i+1)-th iteration, we use ΘN to initialize the model such that Θ̃0=Θi−1=ΘN (Step 2 in Algorithm 1). When the entire training procedure finishes, we obtain an adapted semantic segmentation network FG (as shown in Fig. 2), which is expected to predict reasonable segmentation results on the target domain.

heading: 3.3 Meta-Knowledge Learning for Multiple-Target Domain Adaptation
text: We propose a meta-learning framework consisting of multiple domain adaptation pairs (i.e. “source→latent”, “latent→latent”, and “latent→target”) for multiple-target domain adaptation (MTDA), promoting the adapted model to perform well on multiple target domains simultaneously. Fig. 3 shows the training procedure.
In the setting of MTDA (K > 1), we have labeled data on the source domain DS and generated latent domains DL={D0L, D1L, ..., D K−1 L }, and attempt to adapt the UDA model to unlabeled target domain DT={D0T , D1T , ..., D K−1 T }. In the meta-training phase, we define the optimization goal is to learn the meta-knowledge that a model starting from which initial condition can adapt well from a source domain to multiple latent domains. Expect for applying bi-level optimization on “source→latent”, we also consider the domain adaptation pair of “latent→latent” to reduce domain shift between arbitrary two latent domains. The motivation of such a design can be represented by a toy example, as shown in Fig. 4. Let A,
B, and C indicate the center of data distribution on the source domain, and two latent domains, respectively. Suppose A, B, and C locate at the three vertices of a triangle, our goal can be simplified to shorten the sides of AB and AC simultaneously. One simple strategy is to directly shorten the above sides, corresponding to applying domain adaptation on the pair of “source→latent”. Another strategy (corresponding to domain adaptation on the pair of “latent→latent”) is to shorten the side of BC, so that AB and AC can be further shortened indirectly. Since both strategies contribute to reduce d(A,B) and d(A,C) (d for distance), we thus use two domain adaptation pairs consisting of “source→latent” and “latent→latent” in the meta-training phase to learn the meta-knowledge. We describe the training procedure of the proposed meta-learning for multiple-target domain adaption in Algorithm 2, and introduce the optimization detail about the meta-training phase and the metatesting phase as following:
(1) Meta-training phase. At each iteration, we randomly select samples Dk1,k2L ={Dk1L , Dk2L } from two latent domains, where k1, k2∈(0, 1, ...,K − 1) and k16=k2. In the inner loop, FC is used to discriminate the representation from FG belonging to source domain DS or latent domain D k1,k2 L . The loss function Linner of the inner loop can be formulated as follows:
Linner(Θ0;DS , D k1,k2 L ) = Lseg(Θ0;DS) + Ladv(Θ0;DS , D k1,k2 L )), (5)
where Dk1,k2L indicates latent data which is assumed without labels. In the outer loop, we let FC discriminate different latent domains. Suppose Θ∗ denotes the initial condition of outer loop optimization as Θ∗ = arg minLinner, the whole optimization goal of meta-training is defined as:
Θ0 = arg min θ0
∑ (Lseg(Θ ∗;Dk1,k2L )+Ladv(Θ ∗;Dk1L , D
k2 L ))︸ ︷︷ ︸
Louter(Θ∗;D k1,k2 L )
, (6)
where Θ0 is the initial condition (i.e. model parameters) need to be optimized via meta-training optimization for the following meta-testing phase. As shown in Eq.6, the outer loop optimization can be be recognized as solving a problem of supervised domain adaptation between two latent domains, allowing the model to generalize on different latent domains. Specifically, the loss function of outer loop Louter in Eq.6 can be divided into two terms: the former term Lseg takes the role of evaluating the segmentation performance on the query set (i.e. Dk1L and D k2 L ), and the latter one Ladv enforces the model to learn domain-invariant representation for different latent domains by applying discrimination on the domain pair of ’‘latent→latent”. (2) Meta-testing phase. The initial condition Θ0 contains the meta-knowledge of from which initial condition that a UDA model can do well in transferring domain-specific knowledge from a labeled source domain to multiple unlabeled latent domains simultaneously. As shown in Fig. 3, we initialize the same UDA model with the learned Θ0 as the initial condition, and let FC discriminate latent from target domains. Then, the optimization goal of meta-testing on the domain pair of “latent→target” is defined as follow:
Θ = arg min θ
(Luda(Θ0;D k1,k2 L , D k1,k2 T )) (7)
Where Dk1L and D k2 L share the similar style to D k1 T and D k2 T , respectively. When the entire training procedure finishes, we obtain an adapted semantic segmentation network FG (as shown in Fig. 3) with parameters Θ, which is expected to predict reasonable segmentation results on the multiple target domains simultaneously.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Data preparation. In our experiments, we evaluate the proposed method on both single-target and multiple-target settings of unsupervised domain adaptation. Specifically, we take the GTA5 dataset [1] as the labeled source domain, while the Cityscapes [35] and C-Driving [21] datasets are adopted as unlabeled single-target domain and multi-target domains, respectively. For the multiple-target setting on the C-Driving dataset, we use the compound data containing “cloudy”, “rainy”, and “snowy” images as target domain for training. During testing, we evaluate the proposed method on the compound test set and a new unseen open test set containing “overcast” images. For image preprocessing, we resize the shorter side of images to 720 and randomly crop a patch from one image at each iteration with size 600×600. Besides, horizontal flip is applied as data augmentation. At each iteration, we randomly select 4 samples from each domain for training.
Implementation. We implement our proposed methods using the PyTorch v1.2.0 on a single NVIDIA P40 GPU (24G memory). For fair comparison, we adopt the DeepLabV2 [36] with VGG16 [37] as the segmentation network FG. For the domain classification network FC , we use an architecture similar to [6], which consists of 5 conv. layers with kernel 4×4 and stride of 2, and the numbers of channels are {64, 128, 256, 512, 1}. The hyper-parameter λ in Eq. 1 is set to 0.01. The value of “J” and “N” (Algorithm 1 and 2) are set to 2 in our experiments. We use the SGD optimizer with 0.9 momentum and 5×10−5 weight decay. The learning rates α, β, γ are empirically set to 1×10−4, 5×10−5, and 1×10−4. For evaluation, images from target test set are resized to 1024×512 as input and the evaluation metric (i.e. IoU) is calculated on predictions upsampled to the original size.

heading: 4.2 Comparison with State of the Art
text: We evaluate our method on two unsupervised domain adaptation settings: single target and multitarget. The results are presented in Tables 1, 2 and 3, respectively.
For the single target setting, in Table 1 we compare our method with 12 state-of-the-art methods. All the methods are based on VGG16 [37] backbone. It can be seen that our method for single-target domain adaptation (STDA) achieves 45.5% on mIoU, and obtains 17.2% improvement compared to the source only model, both of which perform favorably against previous state-of-the-art methods. In Table 2, we compare our method with 13 state-of-the-art methods, which all use ResNet101 [39] as the backbone. It can be seen that our method achieves 51.9% on mIoU, outperforming previous stateof-the-art methods. These results further demonstrate that our proposed method achieves consistent top results on different backbones.
For the multi-target setting, we compare our method with two recently proposed multi-target methods [21, 23] and several state-of-the-art single-target methods [4, 6, 45, 46], following [21, 23]. In Table 3 we see that the proposed method for multi-target domain adaptation (MTDA) outperforms previous methods by a large margin w.r.t mIoU (4%), establishing a new state of the art. Besides, we retrain the PCE [20] model under the setting of MTDA, which achieves 31.1% w.r.t mIoU on the C-Driving dataset. These results indicate that our method outperforms PCE on both STDA and MTDA settings.
We also provide some qualitative semantic segmentation results in Fig. 5, where we observe obvious improvements against both source only and previous top methods. These quantitative and qualitative results demonstrate the effectiveness of our method for unsupervised domain adaptation.

heading: 4.3 Model Analysis
text: To verify the effectiveness of our methods, we provide comparisons on different ways of using the latent domain, as shown in Table 4.
From the results, we have the following observations: (1) The latent domain eases adaptation via narrowing down the domain gap. Compared to a©, other methods, using the latent domain in different ways, all perform better. (2) The latent domain can be used to construct multiple pairs, which performs better than the standard one pair. Compared to c©, d© and e© obtain significant improvements (∼ 3%) by constructing two DA pairs, which distributes the adaptation difficulty on two procedures. (3) Meta-knowledge transition between DA pairs helps more than simple finetuning. In f©, our proposed STDA framework learns meta-knowledge of how to adapt from labeled data to unlabeled data, which is more effective to assist the adaptation of “latent→target” than the finetuning strategy used in e©. (4) When the target data distributes sparsely, our multi-target method further improves the performance. Our MTDA improves over STDA on the C-Driving dataset by modeling the target data as multiple domains.
We further analyze the feature space learned with our MTDA and STDA. In Fig. 6, We provide the t-SNE visualization [50] of extracted features from FG using our STDA and MTDA, respectively. It appears that our MTDA yields more generalized domain-invariant features. More specifically, the feature distributions of different domains are better aligned in MTDA than that in STDA.

heading: 5 Conclusion
text: In this paper, we propose a novel meta-learning framework for single-target and multi-target domain adaptive semantic segmentation. In particular, we generate latent data via an image translation model and construct multiple domain adaptation pairs. The purpose is to use the meta-knowledge learned from some pairs as guidance to assist the adaptation of “latent-to-target” under a meta-learning framework. Experimental results show the effectiveness of our method, which establishes new state-of-the-art performance on both settings of single-target and multi-target domain adaptation.

heading: 6 Acknowledgments
text: This work was supported by the National Natural Science Foundation of China (Grant No. 62172225), the Funds for International Cooperation and Exchange of the National Natural Science Foundation of China (Grant No. 61861136011), the Fundamental Research Funds for the Central Universities (Grant No.30920032201), and the National Key Research and Development Program of China under Grant 2017YFC0820601.

Review summaries:
The paper proposes a meta-learning framework for open compound domain adaptation. The method uses k-mean clustering to estimate the domain of target samples and employs the image translation method to augment datasets. Then the augmented images are used for the meta-learning framework. The method outperforms baselines on Cityscapes and C-Driving. 

This work introduces a meta-learning framework, based on MAML, for single-target DA (STDA) and open compound DA (OCDA) in semantic segmentation. Similar to Li et al. [22], the idea is to learn the initial condition (model's parameters) that is more favorable for target adaptation. The proposed framework makes use of an existing image-2-image model to translate source images into, what are called here, latent images; this is to have target-like annotated data to guide meta-optimization.  Each iteration of the proposed algorithm is composed of two consecutive steps: (1) meta-training operating on source and latent data to find the favorable initial condition for the next step and (2) meta-testing to perform target adaptation starting from the learned initial condition.   The meta-training optimization, similar to MAML, involves inner and outer loops. In case of STDA, the inner loop performs UDA optimization steps on labeled source and unlabeled latent samples; the outer loop, based on the optimized weights of the inner loops, meta-optimizes the network to minimize segmentation loss computed using labeled latent data. In case of OCDA with k target sub-domains, k-mean clustering is first done to assign sub-domain pseudo-labels to target images. This information is used in the image-2-image operation to translate source data into sub-domain-like data. Different to STDA, there is an additional adversarial objective in the outer loop.  To evaluate performance, this work consider two DA benchmarks: GTA5->Cityscapes (STDA) and Cityscapes->C-Driving (OCDA). Reported results show improvement of the proposed framework as compared to previous approaches.