Reducing the Covariate Shift by Mirror Samples in Cross Domain Alignment

1 Introduction . Current deep learning models have achieved significant progress on many tasks but heavily rely on the large amount of labeled data . The generality of the model may be severely degraded when facing the same task of a different domain . So , domain adaptation ( DA ) attracts a lot of attentions in recent years . DA has several different settings , such as unsupervised [ 34 ] or semi-supervised DA [ 23 ] , open-set [ 6 ] or closed-set DA [ 38 ] , as well as single or multi-source DA [ 40 ] . In this paper , we consider the closed-set , single-source unsupervised domain adaptation ( UDA ) on the classification task . In this setting , one has the source domain data with labels and is expected to predict for the unlabeled target domain data . Both the source and target domains share the same class labels . The main challenge for DA is the domain shift . It can be further categorized into covariate shift [ 45 , 44 ] , target/label shift [ 48 ] ) , etc . Specifically , define ps ( x , y ) and pt ( x , y ) as the joint distributions of the source and target domains . The covariate shift refers to the difference of the marginal distribution of x , i.e . ps ( x ) 6= pt ( x ) , assuming the conditional probabilities cross domains are same , i.e . ps ( y|x ) = pt ( y|x ) . Most of the current methods , in terms of learning domain-invariant representation or domain alignment , are working to reduce the covariate shift explicitly or implicitly . ∗Equal Contribution †Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . They aligned the domain centers [ 34 , 8 , 4 , 47 ] or class-wise centers/prototypes [ 37 ] , regularized the margins of inter-intra class distances [ 7 , 15 , 28 ] and even aligned the two domains through sample-level mappings [ 16 , 14 , 13 ] . However , there exists a covert dilemma between the covariate shift and its assumption in practice , which bounds the performance of those methods . That is reducing the covariate shift will break its assumptions empirically . The underlying reason is that the dataset we have is a sampling result from the underlying distribution . The randomness in collecting the dataset for different domains varies a lot and introduces the sampling bias unavoidably ( i.e . the sample pattern distributes differently cross domains ) . Aligning the marginal distributions of the biased datasets will distort the internal structure of the real underlying distribution , leading the underlying conditional distribution of the dataset can not be same under the bias . Investigating the internal structure of the underlying distribution in DA has been noted by SRDC [ 49 ] using discriminative clustering [ 7 ] , by sample-level matching cross domains [ 13 , 14 ] , or by Optimal Transport Theory [ 1 ] , etc . But they all neglect the issue mentioned above . We first elaborate the dilemma mentioned above in Section 2 . Then our proposed methodology is presented in Section 3 . We introduce the “ mirror samples ” , which are generated to complement the potential bias of the sampled dataset cross domains , making each sample in the domain to have the corresponding instance in the opposite domain . We propose the mirror loss by regularizing the generated mirror sample with the existing sample to achieve more fine-grained distribution alignment . The properties of the proposed methods and the experiment results on several mainstream benchmarks are given in Section 4 and Section 5 . 2 Delimma of Covariate Shift using Samples . We first revisit the intuition behind the method of reducing the covariate shift . In cross-domain visual classification , the domain distribution can be decomposed into class-discriminative latent visual pattern distribution ( p ( x ) ) and conditional class distribution ( p ( y|x ) ) . The class-discriminative visual pattern refers to the intrinsic visual characteristics of each class , which is domain-agnostic and follows their natural distribution . The conditional distribution is the probability of those visual patterns belonging to a certain category . Reducing the covariate shift assumes the conditional distributions cross domains are same . Aligning the marginal distribution of visual patterns in certain latent space can achieve the ideal domain alignment . Biased Sampling Dataset . Note that the dataset we have is a sampling result w.r.t . the intrinsic visual property distribution . The sampling process ( i.e . dataset collection procedure ) varies for different domains and inevitably introduces sampling bias cross domains , resulting the imbalanced patterns within one category etc . One observation of this sampling bias can be seen in the Fig.1 ( a ) , where we take the category “ Bed ” and “ Calendar ” of domain “ Art ” and “ Clipart ” in Office-Home [ 51 ] as examples . For the same category , saying “ Bed ” , some of the images are Bunk bed but the others are not ; some of them have pillows while the others do not ; even some of them have people on bed while the left are empty etc . However , the ratios of those different visual patterns differ dramatically in the two domains . As shown in Fig.1 ( a ) , the ratios of “ with People ” in Bed are much lower in “ Art ” than in “ Clipart ” , while the Bunk bed ratio has higher percentage in “ Clipart ” . This difference also exists for “ Calendar ” in patterns like whether it has digit or not and whether it is daily calendar or monthly calendar . An ideal sampling process should assure the consistent ratios for these patterns following the category ’ s intrinsic distribution , but it ’ s impractical . Reducing Covariate Shift vs Assumption Violation . The biased dataset will induce a dilemma if we still follow the philosophy of reducing covariate shift . Specifically , in the biased dataset , the samples in the source domains do not have the same counterparts in the target domain in distribution . If we align the two domains by the biased data , no matter using moments/prototype alignment or minimizing the sample-based domain discrepancy , we essentially align the samples to biased positions , making the intrinsic conditional probability , which should be same , distorted . Fig.1 ( b ) gives a simplified illustration of this dilemma in one-dimension case . The underlying marginal distributions of x for source and target domains are U ( −3 , 5 ) and U ( 4 , 12 ) respectively , where U means uniform distribution . The conditional distributions of x belonging to certain class in source and target domains are N [ −3,5 ] ( 1 , 1 ) and N [ 4,12 ] ( 8 , 1 ) , where N [ a , b ] ( µ , σ ) is truncated normal distribution with support [ a , b ] , µ and σ as mean and standard deviation respectively . This is an ideal case where reducing the covariate shift can perfectly eliminate the domain gap : offsetting the target samples to left by 7 without changing the class probability ( from red solid line to green solid line in Fig.1 ( b ) ) , both the conditional distribution and marginal distribution are same . However , in practice , we only have the random samples illustrated by dots in Fig.1 ( b ) . If we still try to reduce the covariate shift , i.e . offsetting the target samples to the source samples in the same order ( any order of moments of x are same in this case ) , the resulting conditional distribution for target domain ( the red dashed line in Fig.1 ( b ) ) will be distorted , breaking the assumption of covariate shift . This dilemma is what our proposed mirror sample and mirror loss are expecting to reduce . 3 Proposed Methods . 3.1 Preliminaries . Mirror Samples . A straightforward solution to reduce the dilemma is to find the ideal counterpart sample in the other domain . Those counterpart sample pairs are in the same positions in their own distribution , i.e . having the equivalent domain-agnostic latent visual pattern . We call it Mirror Sample . If we could find those ideal mirror samples cross domains under the inevitable sampling bias , the alignment by reducing the covariate shift will not incur the conditional inconsistency anymore . The closest work to the mirror sample would be the series of pixel-level GAN based method , such as CyCADA [ 26 ] , CrDoCo [ 10 ] , etc . However , those methods are dedicated for pixel-level domain adaptation , assuming the domain gap is mainly the “ style ” difference . But domain shift is generally large and varied . What ’ s more , the adversarial losses might suffer from the mode collapse and convergence issues . The experimental comparison can be found in Appendix E. Different from those ideas , we propose a concise method to construct the Mirror Sample , which consists of Local Neighbor Approximation ( LNA ) and Equivalence Regularization ( ER ) . Optimal transport Explanation . In fact , the mirror sample can be formulated in terms of optimal transport theory [ 1 ] . Let Ts and Tt be the two transforms ( push-forwards operators ) on the two domain distributions pS and pT such that the resulting distributions are same , i.e . Ts # ps = Tt # pt . This sheds light on an elegant way to reduce the domain gap . In this context , xs ∈ DS and xt ∈ DT are the mirror for each other if Ts # ps ( xs ) = Tt # pt ( xt ) . Direct finding the push-forwards operators are almost impossible . However , if we would first find the mirror samples defined above , those operators would like be learned by those mirror constraints . The more detailed explanation and illustration can be found in Appendix A. Denotations . Similar to the existing settings of DA , denote the source samples as { ( xsi , ysi ) } n s i=1 , ysi ∈ Y and the target samples as { xtj } n t j=1 , where n s and nt are the numbers of source and target samples , Y is the label set with M classes . We also use XS = { xsi } n s i=1 , X T = { xtj } n t j=1 for brevity . The notations with “ tilde ” above refer to the mirror samples . Since the proposed method follows the framework of domain-invariant representation , all the above notations as well as the mirror samples are in the latent space after certain transformation . 3.2 Local Neighbor Approximation . The Local Neighbor Approximation ( LNA ) is expected to generate mirror samples using the local existing data in the same domain . Considering that if the source and target domains are aligned ideally , the source and target domains are two different views of the same distribution . The mirror pairs , although in different domains , are exactly the same instance in the aligned space . Inspired by d-SNE in [ 57 ] , we use the nearest neighbors in the opposite domain to estimate the mirror of a sample . Fig.2 gives an illustration of the local neighbor approximation cross domains . Formally , denote d as the distance measure of two samples . To construct the mirror of target sample xtj , we first find the nearest neighbor set in the source domain as X̃S ( xtj ) , called mirror sets as follows : X̃S ( xtj ) = arg > kx∈XSd ( x , x t j ) ( 1 ) where > kΩ is a “ top-k ” operation that selects the top k-smallest elements in set Ω with respect to the distance measure d. Then we estimate the mirror sample of xtj by weighted combination of the samples in X̃S ( xtj ) . This is following the conclusion that the learned features after the feature extractor lie in a manifold [ 3 , 57 ] : x̃s ( xtj ) = ∑ x∈X̃S ( xtj ) ω ( x , xtj ) x ( 2 ) where ω ( x , xtj ) is the weight of the element x in the mirror set X̃ S ( xtj ) . The weight can be inversely proportional to t d ( x , xtj ) or simply 1/k . Symmetrically , we can also have the corresponding mirror of source sample xsi in target domain as x̃ t ( xsi ) analogous to Eq.1 and 2 . 3.3 Equivalence Regularization Although LNA provides a way to estimate the mirror sample , it can not guarantee the equivalence of the mirror pairs . We propose an anchor-based Equivalence Regularization ( ER ) method to enhance the equivalence cross domains . In detail , define the centers for class c in source and target domains as µsc and µ t c respectively . If the distributions are aligned , those class-wise centers for the same class should be same . This means they could be the anchors cross domains . Inspired by the probability vector used in SRDC [ 49 ] , we introduce the relative position of sample xtj to an anchor µ t c of its domain as : qtc ( x t j ) = exp { −d ( xtj , µtc ) } ∑M c=1 exp { −d ( xtj , µtc ) } ( 3 ) where d is a distance measure . Then the relative position vector w.r.t . all the anchors is qt ( xtj ) = [ qt1 ( x t j ) , q t 2 ( x t j ) , · · · , qtM ( xtj ) ] ( 4 ) where M is the class number . For xtj ’ s mirror sample x̃ s ( xtj ) estimated by LNA , its relative position to its anchors µsc is symmetrically written as : qsc ( x̃ s ( xtj ) ) = exp { −d ( x̃s ( xtj ) , µsc ) } ∑M c=1 exp { −d ( x̃s ( xtj ) , µsc ) } ( 5 ) Its relative position vector w.r.t . to all source anchors is analogously written as : qs ( x̃s ( xtj ) ) = [ qs1 ( x̃ s ( xtj ) ) , q s 2 ( x̃ s ( xtj ) ) , · · · , qsM ( x̃s ( xtj ) ) ] ( 6 ) Note that Eq.3 , 4 , 5 and 6 can be used to get the corresponding qsc ( x s i ) and q s ( xsi ) , q t c ( x̃ t ( xsi ) ) and qt ( x̃t ( xsi ) ) by switching the source and target domains . To regularize the equivalence of the mirror pairs , we minimize the Kullback-Leibler divergence of the relative position vectors of them , forming the mirror loss : Lmr , x = 1 nt ∑nt j=1 KL ( qt ( x̃s ( xtj ) ) ‖qt ( xtj ) ) + 1 ns ∑ns i=1 KL ( qs ( x̃t ( xsi ) ) ‖qs ( xsi ) ) ( 7 ) where the first term of right side of the equation is aligning the relative position vector of target sample xtj and its virtual mirror x̃ s ( xtj ) in source domain , while the second term is aligning the source sample xsi to its virtual mirror x̃ t ( xsi ) in target domain . One should note that although some existing works use KL divergence losses such as TPN [ 37 ] or SRDC [ 49 ] etc. , mirror loss is quite different . The mirror loss minimizes the divergence of the relative position vectors of mirror pairs to the anchors , ensuring the constructed sample by LNA to be the equivalent . The other methods do not involve constructed samples . They generally minimized the KL divergence of different distributions for the same sample .

Summary: In this paper, the author proposed a sample-level alignment method for solving the covariate shift problem in unsupervised domain adaptation. They noticed that the existing methods do not work well since they neglect the bias in the datasets (e.g. different visual patterns in the same category). To fix this problem, for each sample in the source/target domain, they construct a mirror sample in the other domain by a weighted average of the top K nearest neighbors. They also derived new losses to preserve the geometric structures of latent representations in the subspace. The overall idea is straightforward but interesting. The experimental results achieve SOTA performance on public benchmarks.