Learning to Adapt via Latent Domains for Adaptive Semantic Segmentation

1 Introduction . Semantic segmentation is a popular task in computer vision , which assigns pixel-wise semantic labels for given images . It has been widely utilized to facilitate downstream applications such as video surveillance and autonomous driving . Recent progress on image semantic segmentation has been driven by deep neural networks trained on a large amount of labeled data , which are yet expensive to obtain . An alternative way is to generate synthetic images with pixel-level ground truth readily available in an effortless way [ 1 , 2 ] . However , the model purely trained on synthetic datasets usually performs rather poor on real data . To address this issue , domain adaptation methods are used to reduce the domain shift and learn domain-invariant representations by minimizing distribution discrepancy between source and target domains [ 3 , 4 ] . Following the advances of generative adversarial nets ( GANs ) [ 5 ] , adversarial learning has been used to align representations of different domains in an adversarial manner [ 6 , 7 , 8 , 9 ] . Recent studies introduce an additional intermediate domain , i.e . latent domain , to narrow down the huge domain gap between source and target domains [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . This is achieved by image-to-image ∗The corresponding author is Shanshan Zhang . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . translation technique , which generates augmented images that share the same contents as well as ground truth labels with the source domain and similar style to the target domain . Previous works use the latent domain as supplement or substitute to the original source domain to ease the adaptation difficulty , following the traditional one-pair domain adaptation framework , as shown in Fig . 1 ( a ) . In contrast , in this paper we construct two domain adaptation pairs : “ source→latent ” and “ latent→target ” . The purpose is to use the meta-knowledge learned from one pair of “ source→latent ” as guidance to assist the adaptation of another pair of “ latent→target ” , as shown in Fig . 1 ( b ) . Specifically , we use a meta-learning framework to learn the meta-knowledge , which reveals that a model starting from which initial condition can adapt well on the adaptation pair of “ source→latent ” . Meta-learning ( a.k.a . learning to learn ) has re-surged recently due to its efficacy for few-shot deep learning . In our paper , we only borrow the idea of bi-level optimization , to achieve an initialization that is good for both domain adaptation and segmentation . Please note for latent-target adaptation , since we do not have segmentation labels on target , we can not optimize for the segmentation task . Thus , an initialization that is only good for domain adaptation is not sufficient ; it is necessary to obtain an initialization that is good for both domain adaptation and segmentation . On the other hand , this initialization can be well transferred from source-latent to latent-target as it does not overfit to the domain adaptation or segmentation tasks on the latent domain thanks to the property of bi-level optimization . Recently , a couple of works have also successfully applied meta-learning to domain adaptation . The most related work to ours is [ 22 ] , which requires multiple source domains or a proportion of labeled target data . In contrast , our proposed method makes use of a generated latent domain , and thus is more flexible in practice . Most existing domain adaptation methods only aim at adapting a model to a single target domain , without considering a more practical scenario where the target consists of multiple data distributions . To investigate a more realistic setting for domain adaptation , we further study the problem of open compound domain adaptation ( a.k.a multiple-target domain adaptation ) [ 23 ] . In this setting , the target is a union of multiple domains without domain labels . To address this challenge , we extend our proposed method from single-target to multi-target domain adaptation . Specifically , we embed an additional domain pair of “ latent→latent ” to algin arbitrary two latent domains in the meta-training phase , which better learn the meta-knowledge of adaptation across different domains . In the metatesting phase , the learned meta-parameters are used to initialize model parameters , promoting our model to adapt well on multiple target domains simultaneously . In summary , the contribution of our work is three folds . ( 1 ) We propose a meta-learning framework compatible for both single-target and multi-target domain adaptation settings . To the best of our knowledge , this is the first work that adopts meta-learning framework to handle domain adaptive semantic segmentation . ( 2 ) We first generate additional images as a latent domain via style transfer from source to target domains . This latent domain is then used to construct domain pairs for metalearning , which transfers meta-knowledge of adaptation across different domains . ( 3 ) Our approach achieves state-of-the-art performance on several challenging benchmark datasets . Also , we provide comprehensive model analyses for the proposed method . 2 Related Work . Since our method incorporates style transfer with meta-learning optimization for domain adaptive semantic segmentation , we will review related works from the above two aspects , respectively . 2.1 Domain Adaptive Semantic Segmentation Based on Style Transfer . Recent advances in domain adaptive semantic segmentation have highlighted the complementary role of pixel-level alignment , which is mainly achieved by image translation methods ( e.g . style transfer ) to translate images from one domain to another . Some works [ 10 , 11 , 12 , 16 , 17 ] construct a unified framework to learn image translation and domain adaptation in a sequential way . In contrast , BDL [ 13 ] proposes a bidirectional learning framework , consisting a closed loop to optimize the adaptation and image translation networks alternatively . Some other works [ 15 , 18 ] investigate cross image translation between source and target domains , enforcing the model to produce consistent predictions for the original and translated images with the same contents . Instead of using adversarial learning , TGCF [ 14 ] constructs a self-ensembling learning framework , which is composed of a teacher network and a student network . To preserve the semantic information of translated image , FDA [ 19 ] and PCE [ 20 ] introduce Fourier transform to image translation . FDA [ 19 ] does not require any training to perform the domain alignment , just using a simple Fourier transform and its inverse . PCE [ 20 ] uses the phase consistency of Fourier transform to provide more flexibility to align the source domain and target domain . The DHA method [ 21 ] focuses on a more practical setting of multiple-target domain adaptation . It generates translated images to construct multiple latent domains , each of which contains the same “ style ” with a specific target domain . Then a couple of multiple domain classifiers are used to perform the target-to-source alignment separately . Existing methods ignore explicitly modeling the relationship of cross-domain knowledge transfer , which is expected to promote the adaptation performance . 2.2 Meta-Learning for Deep Neural Networks . Meta-learning for neural network has a long history , but have resurged in popularity recently . This has been largely driven by its efficacy for few-shot deep learning [ 24 , 25 ] , reinforcement learning [ 26 ] , hyperparameter optimization [ 27 ] , and neural architecture search [ 28 ] . More recently , meta-learning has also been successfully applied to domain adaptation [ 29 , 30 , 31 , 22 ] . EAML [ 31 ] uses a metaadaptation framework to learn representations for adapting the model to continually evolving target domains . Meta-online [ 22 ] introduces a new framework to enhance performance by learning the initial conditions ( i.e . model parameter ) of existing domain adaptation methods , whereby the proposed framework is only suitable for multi-source or semi-supervised domain adaption settings . We draw inspiration from MAML [ 32 ] , and in particular from the idea of learning initial conditions [ 22 ] of neural network optimization that efficiently adapt the model to the target domain . The major difference to [ 22 ] is that we construct domain pairs using the generated latent domain images , which share the same contents with the source domain and a similar style with the target domain . Such an intermediate domain makes the meta-knowledge transfer become easier yet more effective . 3 Method . 3.1 Preliminaries . We define the source domain as DS and the target domain as DT= { D0T , D1T , ... , D K−1 T } , where K indicates the number of target domains . Assuming K > 1 , we take it as the multi-target domain setting ; when K=1 , it degenerates to a single target domain adaptation problem . When assuming K > 1 , we first need to split the whole target data to multiple domains by discovering different styles . Here , we use the statistics of CNN features ( i.e . mean and standard deviations ) to represent the domain-specific style information following [ 21 , 33 ] . Based on this representation , we then perform k-means clustering [ 34 ] to assign a domain label to each target image . Algorithm 1 Meta-Knowledge Learning for Single Target Domain Adaptation ( STDA ) Input : Labeled data from DS and DL , and unlabelled data from DT . Initialise : Model parameters Θ0 , learning rate α , β , γ , iteration number I at each epoch , iteration number J and N in the meta-training and meta-testing phases , respectively . 1 : for i = 1 , 2 , ... , I do 2 : Θ̃0 = Θi−1 ; 3 : Sample ( DS ) i , ( DL ) i and ( DT ) i uniformly from DS , DL and DT , respectively ; 4 : for j = 1 , 2 , ... , J do 5 : Sample ( DS ) j and ( DL ) j from ( DS ) i and ( DL ) i , respectively ; 6 : Θ̃j = Θ̃j−1 − α5Θ̃j−1 Luda ( Θ̃j−1 ; ( DS ) j , ( DL ) j ) ; \\ Inner loop 7 : end for 8 : Θ0 = Θi−1 − β 5Θi−1 Lseg ( Θ̃J ; ( DL ) i ) ; \\ Outer loop for learning initial condition 9 : for n = 1 , 2 , ... , N do 10 : Sample ( DL ) n and ( DT ) n from ( DL ) i and ( DT ) i , respectively . 11 : Θn = Θn−1 − γ 5Θn−1 Luda ( Θn−1 ; ( DL ) n , ( DT ) n ) \\ Meta-testing 12 : end for Θi = ΘN 13 : end for 14 : return ΘI Bases on one source domain and K ( K≥1 ) target domains , we use an image translation model G to generate augmented images , constructing K additional intermediate domains , i.e . latent domains . For example , the generated image on the k-th latent domain denoted as XiLk shares the same content with the i-th source image XiS and possesses similar style information with z-th image X z Tk on the k-th target domain , and can be formulated as : XiLk = G ( X i S , X z Tk ) , where z is a random index , indicating randomly selecting an image from the target domain . Here , we use a recently proposed image translation method ( i.e . PCE [ 20 ] ) as G to generate augmented images . In order to preserve the semantic information , PCE introduces the Fourier transform and designs a phase consistency loss for image translation . Please refer to [ 20 ] and [ 21 ] for more details . For the basic pairwise domain adaptation module , we use the adversarial learning strategy . Specifically , the semantic segmentation network FG and domain classification network FC are optimized via the following unsupervised domain adaptation ( UDA ) loss Luda : Luda = Lseg + λLadv , ( 1 ) where Lseg is the cross-entropy loss using ground truth annotations on the source domain ; Ladv is the adversarial loss that optimizes FG and fools FC by maximizing the probability of misclassifying the source and target data ; and λ is the weight used to balance the two terms . For adversarial learning , we simply insert a gradient reversal layer ( GRL ) [ 40 ] ) between FG and FC , allowing the whole model to be optimized in an end-to-end manner . In contrast to the traditional one-pair framework ( as shown in Fig . 1 ( a ) ) , in this work we propose a meta-learning framework that consists of multiple domain pairs ( as shown in Fig . 1 ( b ) ) , aiming at learning the meta-knowledge ( how to adapt ) as guidance for improving the adaption performance . In the following subsections 3.2 and 3.3 , we will describe our meta-learning framework for single-target and multi-target domain adaptation in detail , respectively .

Summary: This work introduces a meta-learning framework, based on MAML, for single-target DA (STDA) and open compound DA (OCDA) in semantic segmentation. Similar to Li et al. [22], the idea is to learn the initial condition (model's parameters) that is more favorable for target adaptation. The proposed framework makes use of an existing image-2-image model to translate source images into, what are called here, latent images; this is to have target-like annotated data to guide meta-optimization.  Each iteration of the proposed algorithm is composed of two consecutive steps: (1) meta-training operating on source and latent data to find the favorable initial condition for the next step and (2) meta-testing to perform target adaptation starting from the learned initial condition.   The meta-training optimization, similar to MAML, involves inner and outer loops. In case of STDA, the inner loop performs UDA optimization steps on labeled source and unlabeled latent samples; the outer loop, based on the optimized weights of the inner loops, meta-optimizes the network to minimize segmentation loss computed using labeled latent data. In case of OCDA with k target sub-domains, k-mean clustering is first done to assign sub-domain pseudo-labels to target images. This information is used in the image-2-image operation to translate source data into sub-domain-like data. Different to STDA, there is an additional adversarial objective in the outer loop.  To evaluate performance, this work consider two DA benchmarks: GTA5->Cityscapes (STDA) and Cityscapes->C-Driving (OCDA). Reported results show improvement of the proposed framework as compared to previous approaches.