Disentangled cyclic reconstruction for domain adaptation

The domain adaptation problem involves learning a unique classification or regression model capable of performing on both a source and a target domain . Although the labels for the source data are available during training , the labels in the target domain are unknown . An effective way to tackle this problem lies in extracting insightful features invariant to the source and target domains . In this work , we propose splitting the information for each domain into a task-related representation and its complimentary context representation . We propose an original method to disentangle these two representations in the single-domain supervised case . We then adapt this method to the unsupervised domain adaptation problem . In particular , our method allows disentanglement in the target domain , despite the absence of training labels . This enables the isolation of task-specific information from both domains and a projection into a common representation . The task-specific representation allows efficient transfer of knowledge acquired from the source domain to the target domain . We validate the proposed method on several classical domain adaptation benchmarks and illustrate the benefits of disentanglement for domain adaptation . 1 INTRODUCTION . The wide adoption of Deep Neural Networks in practical supervised learning applications is hindered by their sensitivity to the training data distribution . This problem , known as domain shift , can drastically weaken , in real-life operating conditions , the performance of a model that seemed perfectly efficient in simulation . Learning a model with the goal of making it robust to a specific domain shift is called domain adaptation ( DA ) . Often , the data available to achieve DA consist of a labeled training set from a source domain and an unlabeled sample set from a target domain . This yields the problem of unsupervised domain adaptation ( UDA ) . In this work , we take an information disentanglement perspective on UDA . We argue that a key to efficient UDA lies in separating the necessary information to complete the network ’ s task ( classification or regression ) , from a task-orthogonal information which we call context or style . Disentanglement in the target domain seems however a difficult endeavor since the available data is unlabeled . Our contribution is two-fold . We propose a formal definition of the disentanglement problem for UDA which , to the best of our knowledge , is new . Then we design a new learning method , called DiCyR ( Disentangled Cyclic Reconstruction ) , which relies on cyclic reconstruction of inputs in order to achieve efficient disentanglement , including in the target domain . We derive DiCyR both in the supervised learning and in the UDA cases . This paper is organized as follows . Section 2 presents the required background on supervised learning and UDA , and proposes a definition of disentanglement for UDA . Section 3 reviews recent work in the literature that allow for a critical look at our contribution and put it in perspective . Section 4 introduces DiCyR , first for the single-domain supervised learning case and then for the UDA problem . Finally , Section 5 empirically evaluates DiCyR against state-of-the-art methods and discusses its strengths , weaknesses and variants . Section 6 summarizes and concludes this paper . 2 PROBLEM DEFINITION . In this section , we introduce the notations and background upon which we build the contributions of Section 4 . Let X be an input space of descriptors and Y an output space of labels . A supervised learning problem is defined by a distribution ps ( x , y ) over elements of X × Y . In what follows , ps will be called the source distribution . One wishes to estimate a mapping f̂ that minimizes a loss function of the form E ( x , y ) ∼ps [ l ( f̂ ( x ) , y ) ] . The optimal estimator is denoted f and one often writes the distribution P ( y|x ) as y ∼ f ( x ) + η , where η captures the deviations between y and f ( x ) . Hence , one tries to learn f . In practice , the loss can only be approximated using a finite set of samples { ( xi , yi ) } ni=1 all independently drawn from ps and f̂ is a parametric function ( such as a deep neural network ) of the form y = f̂ ( x ; θ ) . Domain adaptation ( DA ) consists in considering a target distribution pt over X ×Y that differs from ps , and the transfer of knowledge from learning in the source domain ( ps ) to the target domain ( pt ) . Specifically , unsupervised DA exploits the knowledge of a labelled training set { ( xsi , ysi ) } ni=1 sampled according to ps , and an unlabelled data set { ( xti ) } mi=1 sampled according to pt . For instance , the source domain data could be a set of labelled photographs of faces , and the target domain data , a set of unlabelled face photographs , taken with a different camera under different exposure conditions . The problem consists in minimizing the target loss E ( x , y ) ∼pt [ l ( f̂ ( x ) , y ) ] . We suppose that a necessary condition to benefit from the knowledge available in the source domain and transfer it to the target domain is the existence of a common information manifold between domains , where an input ’ s projection is sufficient to predict the labels . We call this useful information task-specific or task-related . The complimentary information should be called task-orthogonal ; it is composed of information that is present in the input but is not relevant to the task at hand . For the sake of naming simplicity , we will call this information style . However we insist that this should not be confused with the classical notion of style . Let Πτ : X → T and Πσ : X → S denote two projection operators , where T and S denote respectively the latent task-related information space and the latent style-related information space . Let Π be the joint projection Π ( x ) = ( Πτ ( x ) , Πσ ( x ) ) . Conversely , we shall note Π̄ : T × S → X a reconstruction operator . And finally , c : T → Y will denote the labeling operator which only uses information from T . We consider that the information of the elements of X is correctly disentangled by Π = ( Πτ , Πσ ) if one can find Π̄ and c such that : C1 : c ◦Πτ minimizes the loss ( and thus fits f on the appropriate domain ) , C2 : Π̄ ◦Π fits the identity operator idX , C3 : With X , T , S the random variables in X , T , S , the mutual information I ( T , S|X ) = 0 , C4 : There is no function g : T → X such that g ◦Πτ = idX , Condition C1 imposes that the projection into T retains enough information to correctly label samples . Condition C2 imposes that all the information necessary for the reconstruction is preserved by the separation performed by Π . Condition C3 states that no information is present in both T and S. Condition C4 impose that the information contained in T alone is insufficient to reconstruct an input , and thus the information of S is necessary . Note that the symmetrical condition is unnecessary , since the combination of C1 and C3 already guarantees that S can not contain the task-related information . Overall , solving this disentanglement problem for DA implies finding a quadruplet 〈Πτ , Πσ , Π̄ , c〉 that meets the conditions above . In particular , note that conditions C3 and C4 open a perspective to a formulation of disentanglement in the general case . 3 RELATED WORK . Disentanglement between the domain-invariant , task-related information and the domain-specific , task-orthogonal , style information is a desirable property to have for DA . In the next paragraphs , we cover important work in representation disentanglement , domain adaptation and their interplay . Before deep learning became popular , Tenenbaum & Freeman ( 2000 ) presented a method using bi-linear models able to separate style from content . More recently , methods based on generative models have demonstrated the ability to disentangle factors of variations from elements of a single domain ( Rifai et al. , 2012 ; Mathieu et al. , 2016 ; Chen et al. , 2016 ; Higgins et al. , 2017 ; Sanchez et al. , 2019 ) . In a cross-domain setting , Gonzalez-Garcia et al . ( 2018 ) use pairs of images with the same labels from different domains to separate representations into a shared information common to both domains and a domain-exclusive information . We note that these approaches do not explicitly aim at respecting all conditions listed in Section 2 . Additionally , most require labeled datasets ( and in some cases even paired datasets ) and thus do not address the unsupervised DA problem . One approach to UDA consists in aligning the source and target distributions statistics , a topic closely related to batch normalization ( Ioffe & Szegedy , 2015 ) . Sun et al . ( 2017 ) minimize the distance between the covariance matrices of the features extracted from the source and target domains . Assuming the domain-specific information is contained inside the batch normalization layers , Li et al . ( 2017 ) align the batch statistics by adopting a specific normalization for each domain . Cariucci et al . ( 2017 ) aim to align source and target feature distributions to a reference one and introduce domain alignment layers to automatically learn the degree of feature alignment needed at different levels of the network . Similarly , Roy et al . ( 2019 ) replace batch normalization layers with domain alignment layers implementing a so-called feature whitening . A major asset of these methods is the possibility to be used jointly with other DA methods ( including the one we propose in Section 4 ) . These methods jointly learn a common representation for elements from both domains . Conversely , Liang et al . ( 2020 ) freeze the representations learned in the source domain before training a target-specific encoder to align the representations of the target elements by maximizing the mutual information between intermediate feature representations and outputs of the classifier . Ensemble methods have also been applied to UDA ( Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ) . French et al . ( 2018 ) combine stochastic data augmentation with self-ensembling to minimize the prediction differences between a student and a teacher network in the target domain . Another approach involves learning domain-invariant features , that do not allow to discriminate whether a sample belongs to the source or target domain , while still permitting accurate labeling in the source domain . This approach relies on the assumption that such features allow efficient labeling in the target domain . Ghifary et al . ( 2016 ) build a two-headed network sharing common layers ; one head performs classification in the source domain , while the second is a decoder that performs reconstruction for target domain elements . Ganin et al . ( 2016 ) propose the DANN method and introduce Gradient Reversal Layers to connect a domain discriminator and a feature extractor . These layers invert the gradient sign during back-propagation so that the feature extractor is trained to fool the domain discriminator . Shen et al . ( 2018 ) modify DANN and replace the domain discriminator by a network that approximates the Wasserstein distance between domains . Tzeng et al . ( 2017 ) optimize , in an adversarial setting , a generator and a discriminator with an inverted label loss . Other methods focus on explicitly disentangling an information shared between domains ( analogous to the domain-invariant features above ) from a domain-specific information . Inspired by Chen et al . ( 2016 ) , Liu et al . ( 2018b ) isolate a latent factor , representing the domain information , from the rest of an encoding , by maximizing the mutual information between generated images and this latent factor . Some domain information may still be present in the remaining part of the encoding and thus may not comply with conditions C3 and C4 . Liu et al . ( 2018a ) combine an encoder , an image generator , a domain discriminator , and a fake images discriminator to produce cross-domain images . The encoder is trained jointly with the domain discriminator to produce domain-invariant representations . Li et al . ( 2020 ) disentangle a latent representation into a global code and a local code . The global code captures category information via an encoder with a prior , and the local code is transferable across domains , which captures the style-related information via an implicit decoder . Bousmalis et al . ( 2016 ) also produce domain-invariant features by training a shared encoder to fool a domain discriminator . They train two domain-private encoders with a difference loss that encourages orthogonality between the shared and the private representations ( similarly to condition C3 ) . Cao et al . ( 2018 ) ; Cai et al . ( 2019 ) ; Peng et al . ( 2019 ) combine a domain discriminator with an adversarial classifier to separate the information shared between domains from the domain-specific information . All these methods build a shared representation that prevents discriminating between source and target domains , while retaining enough information to correctly label samples from the source domain . However , because they rely on an adversarial classifier that requires labeled data , they do not guarantee that the complimentary , domain-specific information for samples in the target domain does not contain information that overlaps with the shared representation . In other words , they only enforce C3 in the source domain . They rely on the assumption that the disentanglement will still hold when applied on target domain elements , which might not be true . Another identified weakness in methods that achieve a domain-invariant feature space is that their representations might not allow for accurate labeling in the target domain . Indeed , feature alignment does not necessarily imply a correct mapping between domains . To illustrate this point , consider a binary classification problem ( classes c1 and c2 ) and two domains ( d1 and d2 ) . Let ( c1 , d1 ) denote samples of class c1 in d1 . It is possible to construct an encoding that projects ( c1 , d1 ) and ( c2 , d2 ) to the same feature values . The same holds for ( c1 , d2 ) and ( c2 , d1 ) for different feature values . This encoding allows discriminating between classes in d1 . It also fools a domain discriminator since it does not allow predicting the original domain of a projected element . However , applying the classification function learned on d1 to the projected d2 elements leads to catastrophic predictions . Transforming a sample from one domain to the other , while retaining its label information can be accomplished by image-to-image translation methods . Hoffman et al . ( 2018 ) extend CycleGAN ’ s cycle consistency ( Zhu et al. , 2017 ) with a semantic consistency to translate from source to target domains . The translated images from the source domain to the target domain are then used to train a classifier on the target domain using the source labels . Similarly , Russo et al . ( 2018 ) train two conditional GANs ( Mirza & Osindero , 2014 ) to learn bi-directional image mappings constrained by a class consistency loss and use a source domain classifier to produce pseudo-labels on source-like transformed target samples . By relaxing CycleGAN ’ s cycle consistency constraint and integrating the discriminator in the training phase , Hosseini-Asl et al . ( 2019 ) address the DA problem in the specific setting where the number of target samples is limited . Takahashi et al . ( 2020 ) use a CycleGAN to generate cross-domain pseudo-pairs and train two domain-specific encoders to align features extracted from each pseudo-pair in the feature space . A major asset of the method is to address the class-unbalanced UDA problem by oversampling with the learned data augmentation . Yang et al . ( 2019 ) use separate encoders to produce domain-invariant and domain-specific features in both domains . They jointly train these encoders with two generators to produce cross-domain elements able to fool domain-specific discriminators . Using a cyclic loss on features , they force the information contained in the representation to be preserved during the generation of cross-domain elements . However , the cyclic loss on features does not prevent the information sharing between features expressed in C3 . More importantly it does not prevent the domain-specific features to be constant . A major drawback of these methods lies in the instability during training that might be caused by min-max optimization problem induced by the adversarial training of generators and discriminators . In the next section , we introduce a method that does not rely on a domain discriminator and an adversarial label predictor , but directly minimizes the information sharing between representations . This allows to guarantee that there is no information redundancy between the task-related and the task-orthogonal style information in both the source and the target domains . Along the way , it provides an efficient mechanism to disentangle the task-related information from the style information in the single domain case . Our method combines information disentanglement , intra-domain and cross-domain cyclic consistency to enforce a more principled mapping between each domain .

Summary: This paper studies the domain adaptation problem by addressing the challenge of splitting task-specific and task-orthogonal information in the target domain using the proposed disentangled cyclic reconstruction method. The authors further develop a variant for the unsupervised domain adaption (UDA) task. The authors argue that the existing adversarial classifier based UDA solutions do not guarantee that the domain specific information does not contain any information that overlaps with the shared information in the target domain. Another shortcoming of these baselines is the learned representation in their domain-invariant feature space might not allow for accurate labeling in the target domain. To solves these limitations, the authors directly minimize the information sharing between representation, instead of using domain adversarial classifier and adversarial label predictor.