Disentangled cyclic reconstruction for domain adaptation

The domain adaptation problem involves learning a unique classification or regression model capable of performing on both a source and a target domain . Although the labels for the source data are available during training , the labels in the target domain are unknown . An effective way to tackle this problem lies in extracting insightful features invariant to the source and target domains . In this work , we propose splitting the information for each domain into a task-related representation and its complimentary context representation . We propose an original method to disentangle these two representations in the single-domain supervised case . We then adapt this method to the unsupervised domain adaptation problem . In particular , our method allows disentanglement in the target domain , despite the absence of training labels . This enables the isolation of task-specific information from both domains and a projection into a common representation . The task-specific representation allows efficient transfer of knowledge acquired from the source domain to the target domain . We validate the proposed method on several classical domain adaptation benchmarks and illustrate the benefits of disentanglement for domain adaptation . 1 INTRODUCTION . The wide adoption of Deep Neural Networks in practical supervised learning applications is hindered by their sensitivity to the training data distribution . This problem , known as domain shift , can drastically weaken , in real-life operating conditions , the performance of a model that seemed perfectly efficient in simulation . Learning a model with the goal of making it robust to a specific domain shift is called domain adaptation ( DA ) . Often , the data available to achieve DA consist of a labeled training set from a source domain and an unlabeled sample set from a target domain . This yields the problem of unsupervised domain adaptation ( UDA ) . In this work , we take an information disentanglement perspective on UDA . We argue that a key to efficient UDA lies in separating the necessary information to complete the network ’ s task ( classification or regression ) , from a task-orthogonal information which we call context or style . Disentanglement in the target domain seems however a difficult endeavor since the available data is unlabeled . Our contribution is two-fold . We propose a formal definition of the disentanglement problem for UDA which , to the best of our knowledge , is new . Then we design a new learning method , called DiCyR ( Disentangled Cyclic Reconstruction ) , which relies on cyclic reconstruction of inputs in order to achieve efficient disentanglement , including in the target domain . We derive DiCyR both in the supervised learning and in the UDA cases . This paper is organized as follows . Section 2 presents the required background on supervised learning and UDA , and proposes a definition of disentanglement for UDA . Section 3 reviews recent work in the literature that allow for a critical look at our contribution and put it in perspective . Section 4 introduces DiCyR , first for the single-domain supervised learning case and then for the UDA problem . Finally , Section 5 empirically evaluates DiCyR against state-of-the-art methods and discusses its strengths , weaknesses and variants . Section 6 summarizes and concludes this paper . 2 PROBLEM DEFINITION . In this section , we introduce the notations and background upon which we build the contributions of Section 4 . Let X be an input space of descriptors and Y an output space of labels . A supervised learning problem is defined by a distribution ps ( x , y ) over elements of X × Y . In what follows , ps will be called the source distribution . One wishes to estimate a mapping f̂ that minimizes a loss function of the form E ( x , y ) ∼ps [ l ( f̂ ( x ) , y ) ] . The optimal estimator is denoted f and one often writes the distribution P ( y|x ) as y ∼ f ( x ) + η , where η captures the deviations between y and f ( x ) . Hence , one tries to learn f . In practice , the loss can only be approximated using a finite set of samples { ( xi , yi ) } ni=1 all independently drawn from ps and f̂ is a parametric function ( such as a deep neural network ) of the form y = f̂ ( x ; θ ) . Domain adaptation ( DA ) consists in considering a target distribution pt over X ×Y that differs from ps , and the transfer of knowledge from learning in the source domain ( ps ) to the target domain ( pt ) . Specifically , unsupervised DA exploits the knowledge of a labelled training set { ( xsi , ysi ) } ni=1 sampled according to ps , and an unlabelled data set { ( xti ) } mi=1 sampled according to pt . For instance , the source domain data could be a set of labelled photographs of faces , and the target domain data , a set of unlabelled face photographs , taken with a different camera under different exposure conditions . The problem consists in minimizing the target loss E ( x , y ) ∼pt [ l ( f̂ ( x ) , y ) ] . We suppose that a necessary condition to benefit from the knowledge available in the source domain and transfer it to the target domain is the existence of a common information manifold between domains , where an input ’ s projection is sufficient to predict the labels . We call this useful information task-specific or task-related . The complimentary information should be called task-orthogonal ; it is composed of information that is present in the input but is not relevant to the task at hand . For the sake of naming simplicity , we will call this information style . However we insist that this should not be confused with the classical notion of style . Let Πτ : X → T and Πσ : X → S denote two projection operators , where T and S denote respectively the latent task-related information space and the latent style-related information space . Let Π be the joint projection Π ( x ) = ( Πτ ( x ) , Πσ ( x ) ) . Conversely , we shall note Π̄ : T × S → X a reconstruction operator . And finally , c : T → Y will denote the labeling operator which only uses information from T . We consider that the information of the elements of X is correctly disentangled by Π = ( Πτ , Πσ ) if one can find Π̄ and c such that : C1 : c ◦Πτ minimizes the loss ( and thus fits f on the appropriate domain ) , C2 : Π̄ ◦Π fits the identity operator idX , C3 : With X , T , S the random variables in X , T , S , the mutual information I ( T , S|X ) = 0 , C4 : There is no function g : T → X such that g ◦Πτ = idX , Condition C1 imposes that the projection into T retains enough information to correctly label samples . Condition C2 imposes that all the information necessary for the reconstruction is preserved by the separation performed by Π . Condition C3 states that no information is present in both T and S. Condition C4 impose that the information contained in T alone is insufficient to reconstruct an input , and thus the information of S is necessary . Note that the symmetrical condition is unnecessary , since the combination of C1 and C3 already guarantees that S can not contain the task-related information . Overall , solving this disentanglement problem for DA implies finding a quadruplet 〈Πτ , Πσ , Π̄ , c〉 that meets the conditions above . In particular , note that conditions C3 and C4 open a perspective to a formulation of disentanglement in the general case . 3 RELATED WORK . Disentanglement between the domain-invariant , task-related information and the domain-specific , task-orthogonal , style information is a desirable property to have for DA . In the next paragraphs , we cover important work in representation disentanglement , domain adaptation and their interplay . Before deep learning became popular , Tenenbaum & Freeman ( 2000 ) presented a method using bi-linear models able to separate style from content . More recently , methods based on generative models have demonstrated the ability to disentangle factors of variations from elements of a single domain ( Rifai et al. , 2012 ; Mathieu et al. , 2016 ; Chen et al. , 2016 ; Higgins et al. , 2017 ; Sanchez et al. , 2019 ) . In a cross-domain setting , Gonzalez-Garcia et al . ( 2018 ) use pairs of images with the same labels from different domains to separate representations into a shared information common to both domains and a domain-exclusive information . We note that these approaches do not explicitly aim at respecting all conditions listed in Section 2 . Additionally , most require labeled datasets ( and in some cases even paired datasets ) and thus do not address the unsupervised DA problem . One approach to UDA consists in aligning the source and target distributions statistics , a topic closely related to batch normalization ( Ioffe & Szegedy , 2015 ) . Sun et al . ( 2017 ) minimize the distance between the covariance matrices of the features extracted from the source and target domains . Assuming the domain-specific information is contained inside the batch normalization layers , Li et al . ( 2017 ) align the batch statistics by adopting a specific normalization for each domain . Cariucci et al . ( 2017 ) aim to align source and target feature distributions to a reference one and introduce domain alignment layers to automatically learn the degree of feature alignment needed at different levels of the network . Similarly , Roy et al . ( 2019 ) replace batch normalization layers with domain alignment layers implementing a so-called feature whitening . A major asset of these methods is the possibility to be used jointly with other DA methods ( including the one we propose in Section 4 ) . These methods jointly learn a common representation for elements from both domains . Conversely , Liang et al . ( 2020 ) freeze the representations learned in the source domain before training a target-specific encoder to align the representations of the target elements by maximizing the mutual information between intermediate feature representations and outputs of the classifier . Ensemble methods have also been applied to UDA ( Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ) . French et al . ( 2018 ) combine stochastic data augmentation with self-ensembling to minimize the prediction differences between a student and a teacher network in the target domain . Another approach involves learning domain-invariant features , that do not allow to discriminate whether a sample belongs to the source or target domain , while still permitting accurate labeling in the source domain . This approach relies on the assumption that such features allow efficient labeling in the target domain . Ghifary et al . ( 2016 ) build a two-headed network sharing common layers ; one head performs classification in the source domain , while the second is a decoder that performs reconstruction for target domain elements . Ganin et al . ( 2016 ) propose the DANN method and introduce Gradient Reversal Layers to connect a domain discriminator and a feature extractor . These layers invert the gradient sign during back-propagation so that the feature extractor is trained to fool the domain discriminator . Shen et al . ( 2018 ) modify DANN and replace the domain discriminator by a network that approximates the Wasserstein distance between domains . Tzeng et al . ( 2017 ) optimize , in an adversarial setting , a generator and a discriminator with an inverted label loss . Other methods focus on explicitly disentangling an information shared between domains ( analogous to the domain-invariant features above ) from a domain-specific information . Inspired by Chen et al . ( 2016 ) , Liu et al . ( 2018b ) isolate a latent factor , representing the domain information , from the rest of an encoding , by maximizing the mutual information between generated images and this latent factor . Some domain information may still be present in the remaining part of the encoding and thus may not comply with conditions C3 and C4 . Liu et al . ( 2018a ) combine an encoder , an image generator , a domain discriminator , and a fake images discriminator to produce cross-domain images . The encoder is trained jointly with the domain discriminator to produce domain-invariant representations . Li et al . ( 2020 ) disentangle a latent representation into a global code and a local code . The global code captures category information via an encoder with a prior , and the local code is transferable across domains , which captures the style-related information via an implicit decoder . Bousmalis et al . ( 2016 ) also produce domain-invariant features by training a shared encoder to fool a domain discriminator . They train two domain-private encoders with a difference loss that encourages orthogonality between the shared and the private representations ( similarly to condition C3 ) . Cao et al . ( 2018 ) ; Cai et al . ( 2019 ) ; Peng et al . ( 2019 ) combine a domain discriminator with an adversarial classifier to separate the information shared between domains from the domain-specific information . All these methods build a shared representation that prevents discriminating between source and target domains , while retaining enough information to correctly label samples from the source domain . However , because they rely on an adversarial classifier that requires labeled data , they do not guarantee that the complimentary , domain-specific information for samples in the target domain does not contain information that overlaps with the shared representation . In other words , they only enforce C3 in the source domain . They rely on the assumption that the disentanglement will still hold when applied on target domain elements , which might not be true . Another identified weakness in methods that achieve a domain-invariant feature space is that their representations might not allow for accurate labeling in the target domain . Indeed , feature alignment does not necessarily imply a correct mapping between domains . To illustrate this point , consider a binary classification problem ( classes c1 and c2 ) and two domains ( d1 and d2 ) . Let ( c1 , d1 ) denote samples of class c1 in d1 . It is possible to construct an encoding that projects ( c1 , d1 ) and ( c2 , d2 ) to the same feature values . The same holds for ( c1 , d2 ) and ( c2 , d1 ) for different feature values . This encoding allows discriminating between classes in d1 . It also fools a domain discriminator since it does not allow predicting the original domain of a projected element . However , applying the classification function learned on d1 to the projected d2 elements leads to catastrophic predictions . Transforming a sample from one domain to the other , while retaining its label information can be accomplished by image-to-image translation methods . Hoffman et al . ( 2018 ) extend CycleGAN ’ s cycle consistency ( Zhu et al. , 2017 ) with a semantic consistency to translate from source to target domains . The translated images from the source domain to the target domain are then used to train a classifier on the target domain using the source labels . Similarly , Russo et al . ( 2018 ) train two conditional GANs ( Mirza & Osindero , 2014 ) to learn bi-directional image mappings constrained by a class consistency loss and use a source domain classifier to produce pseudo-labels on source-like transformed target samples . By relaxing CycleGAN ’ s cycle consistency constraint and integrating the discriminator in the training phase , Hosseini-Asl et al . ( 2019 ) address the DA problem in the specific setting where the number of target samples is limited . Takahashi et al . ( 2020 ) use a CycleGAN to generate cross-domain pseudo-pairs and train two domain-specific encoders to align features extracted from each pseudo-pair in the feature space . A major asset of the method is to address the class-unbalanced UDA problem by oversampling with the learned data augmentation . Yang et al . ( 2019 ) use separate encoders to produce domain-invariant and domain-specific features in both domains . They jointly train these encoders with two generators to produce cross-domain elements able to fool domain-specific discriminators . Using a cyclic loss on features , they force the information contained in the representation to be preserved during the generation of cross-domain elements . However , the cyclic loss on features does not prevent the information sharing between features expressed in C3 . More importantly it does not prevent the domain-specific features to be constant . A major drawback of these methods lies in the instability during training that might be caused by min-max optimization problem induced by the adversarial training of generators and discriminators . In the next section , we introduce a method that does not rely on a domain discriminator and an adversarial label predictor , but directly minimizes the information sharing between representations . This allows to guarantee that there is no information redundancy between the task-related and the task-orthogonal style information in both the source and the target domains . Along the way , it provides an efficient mechanism to disentangle the task-related information from the style information in the single domain case . Our method combines information disentanglement , intra-domain and cross-domain cyclic consistency to enforce a more principled mapping between each domain .

Summary: This paper proposes a new framework for unsupervised domain adaptation by applying the disentangled representations learning (DiCyR). The core idea of DiCyR is to split the raw feature into the task-related one and its complimentary context where the task-related representations are projected into a shared space for alignment. From my point of view, disentangled representations learning is the most interesting part of this work.