Reducing the Covariate Shift by Mirror Samples in Cross Domain Alignment

1 Introduction . Current deep learning models have achieved significant progress on many tasks but heavily rely on the large amount of labeled data . The generality of the model may be severely degraded when facing the same task of a different domain . So , domain adaptation ( DA ) attracts a lot of attentions in recent years . DA has several different settings , such as unsupervised [ 34 ] or semi-supervised DA [ 23 ] , open-set [ 6 ] or closed-set DA [ 38 ] , as well as single or multi-source DA [ 40 ] . In this paper , we consider the closed-set , single-source unsupervised domain adaptation ( UDA ) on the classification task . In this setting , one has the source domain data with labels and is expected to predict for the unlabeled target domain data . Both the source and target domains share the same class labels . The main challenge for DA is the domain shift . It can be further categorized into covariate shift [ 45 , 44 ] , target/label shift [ 48 ] ) , etc . Specifically , define ps ( x , y ) and pt ( x , y ) as the joint distributions of the source and target domains . The covariate shift refers to the difference of the marginal distribution of x , i.e . ps ( x ) 6= pt ( x ) , assuming the conditional probabilities cross domains are same , i.e . ps ( y|x ) = pt ( y|x ) . Most of the current methods , in terms of learning domain-invariant representation or domain alignment , are working to reduce the covariate shift explicitly or implicitly . ∗Equal Contribution †Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . They aligned the domain centers [ 34 , 8 , 4 , 47 ] or class-wise centers/prototypes [ 37 ] , regularized the margins of inter-intra class distances [ 7 , 15 , 28 ] and even aligned the two domains through sample-level mappings [ 16 , 14 , 13 ] . However , there exists a covert dilemma between the covariate shift and its assumption in practice , which bounds the performance of those methods . That is reducing the covariate shift will break its assumptions empirically . The underlying reason is that the dataset we have is a sampling result from the underlying distribution . The randomness in collecting the dataset for different domains varies a lot and introduces the sampling bias unavoidably ( i.e . the sample pattern distributes differently cross domains ) . Aligning the marginal distributions of the biased datasets will distort the internal structure of the real underlying distribution , leading the underlying conditional distribution of the dataset can not be same under the bias . Investigating the internal structure of the underlying distribution in DA has been noted by SRDC [ 49 ] using discriminative clustering [ 7 ] , by sample-level matching cross domains [ 13 , 14 ] , or by Optimal Transport Theory [ 1 ] , etc . But they all neglect the issue mentioned above . We first elaborate the dilemma mentioned above in Section 2 . Then our proposed methodology is presented in Section 3 . We introduce the “ mirror samples ” , which are generated to complement the potential bias of the sampled dataset cross domains , making each sample in the domain to have the corresponding instance in the opposite domain . We propose the mirror loss by regularizing the generated mirror sample with the existing sample to achieve more fine-grained distribution alignment . The properties of the proposed methods and the experiment results on several mainstream benchmarks are given in Section 4 and Section 5 . 2 Delimma of Covariate Shift using Samples . We first revisit the intuition behind the method of reducing the covariate shift . In cross-domain visual classification , the domain distribution can be decomposed into class-discriminative latent visual pattern distribution ( p ( x ) ) and conditional class distribution ( p ( y|x ) ) . The class-discriminative visual pattern refers to the intrinsic visual characteristics of each class , which is domain-agnostic and follows their natural distribution . The conditional distribution is the probability of those visual patterns belonging to a certain category . Reducing the covariate shift assumes the conditional distributions cross domains are same . Aligning the marginal distribution of visual patterns in certain latent space can achieve the ideal domain alignment . Biased Sampling Dataset . Note that the dataset we have is a sampling result w.r.t . the intrinsic visual property distribution . The sampling process ( i.e . dataset collection procedure ) varies for different domains and inevitably introduces sampling bias cross domains , resulting the imbalanced patterns within one category etc . One observation of this sampling bias can be seen in the Fig.1 ( a ) , where we take the category “ Bed ” and “ Calendar ” of domain “ Art ” and “ Clipart ” in Office-Home [ 51 ] as examples . For the same category , saying “ Bed ” , some of the images are Bunk bed but the others are not ; some of them have pillows while the others do not ; even some of them have people on bed while the left are empty etc . However , the ratios of those different visual patterns differ dramatically in the two domains . As shown in Fig.1 ( a ) , the ratios of “ with People ” in Bed are much lower in “ Art ” than in “ Clipart ” , while the Bunk bed ratio has higher percentage in “ Clipart ” . This difference also exists for “ Calendar ” in patterns like whether it has digit or not and whether it is daily calendar or monthly calendar . An ideal sampling process should assure the consistent ratios for these patterns following the category ’ s intrinsic distribution , but it ’ s impractical . Reducing Covariate Shift vs Assumption Violation . The biased dataset will induce a dilemma if we still follow the philosophy of reducing covariate shift . Specifically , in the biased dataset , the samples in the source domains do not have the same counterparts in the target domain in distribution . If we align the two domains by the biased data , no matter using moments/prototype alignment or minimizing the sample-based domain discrepancy , we essentially align the samples to biased positions , making the intrinsic conditional probability , which should be same , distorted . Fig.1 ( b ) gives a simplified illustration of this dilemma in one-dimension case . The underlying marginal distributions of x for source and target domains are U ( −3 , 5 ) and U ( 4 , 12 ) respectively , where U means uniform distribution . The conditional distributions of x belonging to certain class in source and target domains are N [ −3,5 ] ( 1 , 1 ) and N [ 4,12 ] ( 8 , 1 ) , where N [ a , b ] ( µ , σ ) is truncated normal distribution with support [ a , b ] , µ and σ as mean and standard deviation respectively . This is an ideal case where reducing the covariate shift can perfectly eliminate the domain gap : offsetting the target samples to left by 7 without changing the class probability ( from red solid line to green solid line in Fig.1 ( b ) ) , both the conditional distribution and marginal distribution are same . However , in practice , we only have the random samples illustrated by dots in Fig.1 ( b ) . If we still try to reduce the covariate shift , i.e . offsetting the target samples to the source samples in the same order ( any order of moments of x are same in this case ) , the resulting conditional distribution for target domain ( the red dashed line in Fig.1 ( b ) ) will be distorted , breaking the assumption of covariate shift . This dilemma is what our proposed mirror sample and mirror loss are expecting to reduce . 3 Proposed Methods . 3.1 Preliminaries . Mirror Samples . A straightforward solution to reduce the dilemma is to find the ideal counterpart sample in the other domain . Those counterpart sample pairs are in the same positions in their own distribution , i.e . having the equivalent domain-agnostic latent visual pattern . We call it Mirror Sample . If we could find those ideal mirror samples cross domains under the inevitable sampling bias , the alignment by reducing the covariate shift will not incur the conditional inconsistency anymore . The closest work to the mirror sample would be the series of pixel-level GAN based method , such as CyCADA [ 26 ] , CrDoCo [ 10 ] , etc . However , those methods are dedicated for pixel-level domain adaptation , assuming the domain gap is mainly the “ style ” difference . But domain shift is generally large and varied . What ’ s more , the adversarial losses might suffer from the mode collapse and convergence issues . The experimental comparison can be found in Appendix E. Different from those ideas , we propose a concise method to construct the Mirror Sample , which consists of Local Neighbor Approximation ( LNA ) and Equivalence Regularization ( ER ) . Optimal transport Explanation . In fact , the mirror sample can be formulated in terms of optimal transport theory [ 1 ] . Let Ts and Tt be the two transforms ( push-forwards operators ) on the two domain distributions pS and pT such that the resulting distributions are same , i.e . Ts # ps = Tt # pt . This sheds light on an elegant way to reduce the domain gap . In this context , xs ∈ DS and xt ∈ DT are the mirror for each other if Ts # ps ( xs ) = Tt # pt ( xt ) . Direct finding the push-forwards operators are almost impossible . However , if we would first find the mirror samples defined above , those operators would like be learned by those mirror constraints . The more detailed explanation and illustration can be found in Appendix A. Denotations . Similar to the existing settings of DA , denote the source samples as { ( xsi , ysi ) } n s i=1 , ysi ∈ Y and the target samples as { xtj } n t j=1 , where n s and nt are the numbers of source and target samples , Y is the label set with M classes . We also use XS = { xsi } n s i=1 , X T = { xtj } n t j=1 for brevity . The notations with “ tilde ” above refer to the mirror samples . Since the proposed method follows the framework of domain-invariant representation , all the above notations as well as the mirror samples are in the latent space after certain transformation . 3.2 Local Neighbor Approximation . The Local Neighbor Approximation ( LNA ) is expected to generate mirror samples using the local existing data in the same domain . Considering that if the source and target domains are aligned ideally , the source and target domains are two different views of the same distribution . The mirror pairs , although in different domains , are exactly the same instance in the aligned space . Inspired by d-SNE in [ 57 ] , we use the nearest neighbors in the opposite domain to estimate the mirror of a sample . Fig.2 gives an illustration of the local neighbor approximation cross domains . Formally , denote d as the distance measure of two samples . To construct the mirror of target sample xtj , we first find the nearest neighbor set in the source domain as X̃S ( xtj ) , called mirror sets as follows : X̃S ( xtj ) = arg > kx∈XSd ( x , x t j ) ( 1 ) where > kΩ is a “ top-k ” operation that selects the top k-smallest elements in set Ω with respect to the distance measure d. Then we estimate the mirror sample of xtj by weighted combination of the samples in X̃S ( xtj ) . This is following the conclusion that the learned features after the feature extractor lie in a manifold [ 3 , 57 ] : x̃s ( xtj ) = ∑ x∈X̃S ( xtj ) ω ( x , xtj ) x ( 2 ) where ω ( x , xtj ) is the weight of the element x in the mirror set X̃ S ( xtj ) . The weight can be inversely proportional to t d ( x , xtj ) or simply 1/k . Symmetrically , we can also have the corresponding mirror of source sample xsi in target domain as x̃ t ( xsi ) analogous to Eq.1 and 2 . 3.3 Equivalence Regularization Although LNA provides a way to estimate the mirror sample , it can not guarantee the equivalence of the mirror pairs . We propose an anchor-based Equivalence Regularization ( ER ) method to enhance the equivalence cross domains . In detail , define the centers for class c in source and target domains as µsc and µ t c respectively . If the distributions are aligned , those class-wise centers for the same class should be same . This means they could be the anchors cross domains . Inspired by the probability vector used in SRDC [ 49 ] , we introduce the relative position of sample xtj to an anchor µ t c of its domain as : qtc ( x t j ) = exp { −d ( xtj , µtc ) } ∑M c=1 exp { −d ( xtj , µtc ) } ( 3 ) where d is a distance measure . Then the relative position vector w.r.t . all the anchors is qt ( xtj ) = [ qt1 ( x t j ) , q t 2 ( x t j ) , · · · , qtM ( xtj ) ] ( 4 ) where M is the class number . For xtj ’ s mirror sample x̃ s ( xtj ) estimated by LNA , its relative position to its anchors µsc is symmetrically written as : qsc ( x̃ s ( xtj ) ) = exp { −d ( x̃s ( xtj ) , µsc ) } ∑M c=1 exp { −d ( x̃s ( xtj ) , µsc ) } ( 5 ) Its relative position vector w.r.t . to all source anchors is analogously written as : qs ( x̃s ( xtj ) ) = [ qs1 ( x̃ s ( xtj ) ) , q s 2 ( x̃ s ( xtj ) ) , · · · , qsM ( x̃s ( xtj ) ) ] ( 6 ) Note that Eq.3 , 4 , 5 and 6 can be used to get the corresponding qsc ( x s i ) and q s ( xsi ) , q t c ( x̃ t ( xsi ) ) and qt ( x̃t ( xsi ) ) by switching the source and target domains . To regularize the equivalence of the mirror pairs , we minimize the Kullback-Leibler divergence of the relative position vectors of them , forming the mirror loss : Lmr , x = 1 nt ∑nt j=1 KL ( qt ( x̃s ( xtj ) ) ‖qt ( xtj ) ) + 1 ns ∑ns i=1 KL ( qs ( x̃t ( xsi ) ) ‖qs ( xsi ) ) ( 7 ) where the first term of right side of the equation is aligning the relative position vector of target sample xtj and its virtual mirror x̃ s ( xtj ) in source domain , while the second term is aligning the source sample xsi to its virtual mirror x̃ t ( xsi ) in target domain . One should note that although some existing works use KL divergence losses such as TPN [ 37 ] or SRDC [ 49 ] etc. , mirror loss is quite different . The mirror loss minimizes the divergence of the relative position vectors of mirror pairs to the anchors , ensuring the constructed sample by LNA to be the equivalent . The other methods do not involve constructed samples . They generally minimized the KL divergence of different distributions for the same sample .

Summary: To reduce domain shift (covariate shift due to different domain distributions), the authors propose a loss based on Equivalence Regularization and Mirrored Samples.  The main idea is to: 1) find class prototype on both source and target domain and 2) encourage each sample's relative embeddings in the source and target domains to be similar.  Experiments are carried out on 4 datasets and obtained competitive accuracy against SOTA methods.