Reducing the Covariate Shift by Mirror Samples in Cross Domain Alignment
Year: 2021

Abstract:
Eliminating the covariate shift cross domains is one of the common methods to deal with the issue of domain shift in visual unsupervised domain adaptation. However, current alignment methods, especially the prototype based or sample-level based methods neglect the structural properties of the underlying distribution and even break the condition of covariate shift. To relieve the limitations and conflicts, we introduce a novel concept named (virtual) mirror, which represents the equivalent sample in another domain. The equivalent sample pairs, named mirror pairs reflect the natural correspondence of the empirical distributions. Then a mirror loss, which aligns the mirror pairs cross domains, is constructed to enhance the alignment of the domains. The proposed method does not distort the internal structure of the underlying distribution. We also provide theoretical proof that the mirror samples and mirror loss have better asymptotic properties in reducing the domain shift. By applying the virtual mirror and mirror loss to the generic unsupervised domain adaptation model, we achieved consistently superior performance on several mainstream benchmarks.

heading: 1 Introduction
text: Current deep learning models have achieved significant progress on many tasks but heavily rely on the large amount of labeled data. The generality of the model may be severely degraded when facing the same task of a different domain. So, domain adaptation (DA) attracts a lot of attentions in recent years. DA has several different settings, such as unsupervised [34] or semi-supervised DA [23], open-set [6] or closed-set DA [38], as well as single or multi-source DA [40]. In this paper, we consider the closed-set, single-source unsupervised domain adaptation (UDA) on the classification task. In this setting, one has the source domain data with labels and is expected to predict for the unlabeled target domain data. Both the source and target domains share the same class labels.
The main challenge for DA is the domain shift. It can be further categorized into covariate shift [45, 44], target/label shift [48]),etc. Specifically, define ps(x, y) and pt(x, y) as the joint distributions of the source and target domains. The covariate shift refers to the difference of the marginal distribution of x, i.e. ps(x) 6= pt(x), assuming the conditional probabilities cross domains are same, i.e. ps(y|x) = pt(y|x). Most of the current methods, in terms of learning domain-invariant representation or domain alignment, are working to reduce the covariate shift explicitly or implicitly. ∗Equal Contribution †Corresponding Author
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
They aligned the domain centers [34, 8, 4, 47] or class-wise centers/prototypes [37], regularized the margins of inter-intra class distances [7, 15, 28] and even aligned the two domains through sample-level mappings[16, 14, 13].
However, there exists a covert dilemma between the covariate shift and its assumption in practice, which bounds the performance of those methods. That is reducing the covariate shift will break its assumptions empirically. The underlying reason is that the dataset we have is a sampling result from the underlying distribution. The randomness in collecting the dataset for different domains varies a lot and introduces the sampling bias unavoidably (i.e. the sample pattern distributes differently cross domains). Aligning the marginal distributions of the biased datasets will distort the internal structure of the real underlying distribution, leading the underlying conditional distribution of the dataset cannot be same under the bias. Investigating the internal structure of the underlying distribution in DA has been noted by SRDC [49] using discriminative clustering [7], by sample-level matching cross domains [13, 14], or by Optimal Transport Theory[1],etc. But they all neglect the issue mentioned above.
We first elaborate the dilemma mentioned above in Section 2. Then our proposed methodology is presented in Section 3. We introduce the “mirror samples”, which are generated to complement the potential bias of the sampled dataset cross domains, making each sample in the domain to have the corresponding instance in the opposite domain. We propose the mirror loss by regularizing the generated mirror sample with the existing sample to achieve more fine-grained distribution alignment. The properties of the proposed methods and the experiment results on several mainstream benchmarks are given in Section 4 and Section 5.

heading: 2 Delimma of Covariate Shift using Samples
text: We first revisit the intuition behind the method of reducing the covariate shift. In cross-domain visual classification, the domain distribution can be decomposed into class-discriminative latent visual pattern distribution (p(x)) and conditional class distribution (p(y|x)). The class-discriminative visual pattern refers to the intrinsic visual characteristics of each class, which is domain-agnostic and follows their natural distribution. The conditional distribution is the probability of those visual patterns belonging to a certain category. Reducing the covariate shift assumes the conditional distributions cross domains are same. Aligning the marginal distribution of visual patterns in certain latent space can achieve the ideal domain alignment.
Biased Sampling Dataset. Note that the dataset we have is a sampling result w.r.t. the intrinsic visual property distribution. The sampling process (i.e. dataset collection procedure) varies for different domains and inevitably introduces sampling bias cross domains, resulting the imbalanced patterns within one category etc. One observation of this sampling bias can be seen in the Fig.1(a), where we take the category “Bed” and “Calendar” of domain “Art” and “Clipart” in Office-Home[51] as examples. For the same category, saying “Bed”, some of the images are Bunk bed but the others are
not; some of them have pillows while the others do not; even some of them have people on bed while the left are empty etc. However, the ratios of those different visual patterns differ dramatically in the two domains. As shown in Fig.1(a), the ratios of “with People” in Bed are much lower in “Art” than in “Clipart”, while the Bunk bed ratio has higher percentage in “Clipart”. This difference also exists for “Calendar” in patterns like whether it has digit or not and whether it is daily calendar or monthly calendar. An ideal sampling process should assure the consistent ratios for these patterns following the category’s intrinsic distribution, but it’s impractical.
Reducing Covariate Shift vs Assumption Violation. The biased dataset will induce a dilemma if we still follow the philosophy of reducing covariate shift. Specifically, in the biased dataset, the samples in the source domains do not have the same counterparts in the target domain in distribution. If we align the two domains by the biased data, no matter using moments/prototype alignment or minimizing the sample-based domain discrepancy, we essentially align the samples to biased positions, making the intrinsic conditional probability, which should be same, distorted. Fig.1(b) gives a simplified illustration of this dilemma in one-dimension case. The underlying marginal distributions of x for source and target domains are U(−3, 5) and U(4, 12) respectively, where U means uniform distribution. The conditional distributions of x belonging to certain class in source and target domains are N[−3,5](1, 1) and N[4,12](8, 1), where N[a,b](µ, σ) is truncated normal distribution with support [a, b], µ and σ as mean and standard deviation respectively. This is an ideal case where reducing the covariate shift can perfectly eliminate the domain gap: offsetting the target samples to left by 7 without changing the class probability (from red solid line to green solid line in Fig.1(b)), both the conditional distribution and marginal distribution are same. However, in practice, we only have the random samples illustrated by dots in Fig.1(b). If we still try to reduce the covariate shift, i.e. offsetting the target samples to the source samples in the same order (any order of moments of x are same in this case), the resulting conditional distribution for target domain (the red dashed line in Fig.1(b)) will be distorted, breaking the assumption of covariate shift. This dilemma is what our proposed mirror sample and mirror loss are expecting to reduce.

heading: 3 Proposed Methods

heading: 3.1 Preliminaries
text: Mirror Samples. A straightforward solution to reduce the dilemma is to find the ideal counterpart sample in the other domain. Those counterpart sample pairs are in the same positions in their own distribution, i.e. having the equivalent domain-agnostic latent visual pattern. We call it Mirror Sample. If we could find those ideal mirror samples cross domains under the inevitable sampling bias, the alignment by reducing the covariate shift will not incur the conditional inconsistency anymore. The closest work to the mirror sample would be the series of pixel-level GAN based method, such as CyCADA [26] ,CrDoCo[10], etc. However, those methods are dedicated for pixel-level domain adaptation, assuming the domain gap is mainly the “style” difference. But domain shift is generally large and varied. What’s more, the adversarial losses might suffer from the mode collapse and convergence issues. The experimental comparison can be found in Appendix E. Different from those ideas, we propose a concise method to construct the Mirror Sample, which consists of Local Neighbor Approximation (LNA) and Equivalence Regularization (ER).
Optimal transport Explanation. In fact, the mirror sample can be formulated in terms of optimal transport theory [1]. Let Ts and Tt be the two transforms (push-forwards operators) on the two domain distributions pS and pT such that the resulting distributions are same, i.e. Ts#ps = Tt#pt. This sheds light on an elegant way to reduce the domain gap. In this context, xs ∈ DS and xt ∈ DT are the mirror for each other if Ts#ps(xs) = Tt#pt(xt). Direct finding the push-forwards operators are almost impossible. However, if we would first find the mirror samples defined above, those operators would like be learned by those mirror constraints. The more detailed explanation and illustration can be found in Appendix A.
Denotations. Similar to the existing settings of DA, denote the source samples as {(xsi , ysi )}n s
i=1, ysi ∈ Y and the target samples as {xtj}n t j=1, where n s and nt are the numbers of source and target samples, Y is the label set with M classes. We also use XS = {xsi}n s i=1, X T = {xtj}n t
j=1 for brevity. The notations with “tilde” above refer to the mirror samples. Since the proposed method follows the
framework of domain-invariant representation, all the above notations as well as the mirror samples are in the latent space after certain transformation.

heading: 3.2 Local Neighbor Approximation
text: The Local Neighbor Approximation (LNA) is expected to generate mirror samples using the local existing data in the same domain. Considering that if the source and target domains are aligned ideally, the source and target domains are two different views of the same distribution. The mirror pairs, although in different domains, are exactly the same instance in the aligned space. Inspired by d-SNE in [57], we use the nearest neighbors in the opposite domain to estimate the mirror of a sample. Fig.2 gives an illustration of the local neighbor approximation cross domains. Formally, denote d as the distance measure of two samples. To construct the mirror of target sample xtj , we first find the nearest neighbor set in the source domain as X̃S(xtj), called mirror sets as follows:
X̃S(xtj) = arg>kx∈XSd(x, x t j) (1)
where >kΩ is a “top-k” operation that selects the top k-smallest elements in set Ω with respect to the distance measure d. Then we estimate the mirror sample of xtj by weighted combination of the samples in X̃S(xtj). This is following the conclusion that the learned features after the feature extractor lie in a manifold [3, 57]:
x̃s(xtj) = ∑
x∈X̃S(xtj) ω(x, xtj)x (2)
where ω(x, xtj) is the weight of the element x in the mirror set X̃ S(xtj). The weight can be inversely proportional to t d(x, xtj) or simply 1/k. Symmetrically, we can also have the corresponding mirror of source sample xsi in target domain as x̃ t(xsi ) analogous to Eq.1 and 2.
3.3 Equivalence Regularization
Although LNA provides a way to estimate the mirror sample, it cannot guarantee the equivalence of the mirror pairs. We propose an anchor-based Equivalence Regularization (ER) method to enhance the equivalence cross domains.
In detail, define the centers for class c in source and target domains as µsc and µ t c respectively. If the distributions are aligned, those class-wise centers for the same class should be same. This means they could be the anchors cross domains. Inspired by the probability vector used in SRDC [49], we introduce the relative position of sample xtj to an anchor µ t c of its domain as:
qtc(x t j) = exp{−d(xtj , µtc)}∑M c=1 exp{−d(xtj , µtc)}
(3)
where d is a distance measure. Then the relative position vector w.r.t. all the anchors is qt(xtj) = [ qt1(x t j), q t 2(x t j), · · · , qtM (xtj) ] (4)
where M is the class number. For xtj’s mirror sample x̃ s(xtj) estimated by LNA, its relative position to its anchors µsc is symmetrically written as:
qsc(x̃ s(xtj)) = exp{−d(x̃s(xtj), µsc)}∑M c=1 exp{−d(x̃s(xtj), µsc)}
(5)
Its relative position vector w.r.t. to all source anchors is analogously written as: qs(x̃s(xtj)) = [ qs1(x̃ s(xtj)), q s 2(x̃ s(xtj)), · · · , qsM (x̃s(xtj)) ]
(6)
Note that Eq.3, 4, 5 and 6 can be used to get the corresponding qsc(x s i ) and q s(xsi ), q t c(x̃ t(xsi )) and qt(x̃t(xsi )) by switching the source and target domains.
To regularize the equivalence of the mirror pairs, we minimize the Kullback-Leibler divergence of the relative position vectors of them, forming the mirror loss:
Lmr,x = 1
nt ∑nt j=1 KL ( qt(x̃s(xtj))‖qt(xtj) ) + 1 ns ∑ns i=1 KL ( qs(x̃t(xsi ))‖qs(xsi ) ) (7)
where the first term of right side of the equation is aligning the relative position vector of target sample xtj and its virtual mirror x̃
s(xtj) in source domain, while the second term is aligning the source sample xsi to its virtual mirror x̃ t(xsi ) in target domain.
One should note that although some existing works use KL divergence losses such as TPN [37] or SRDC [49] etc., mirror loss is quite different. The mirror loss minimizes the divergence of the relative position vectors of mirror pairs to the anchors, ensuring the constructed sample by LNA to be the equivalent. The other methods do not involve constructed samples. They generally minimized the KL divergence of different distributions for the same sample.

heading: 3.4 Model with Mirror Loss
text: The mirror samples actually augment the existing datasets to { xsi }ns i=1 ∪ {x̃s(xtj)}n t
j=1 for source and {xtj}n t j=1 ∪ {x̃t(xsi )}n s
i=1 for target. The mirror loss setups the constraints cross domains between {x̃s(xtj)}n t j=1 and {xtj}n t j=1, {xsi}n s i=1 and {x̃t(xsi )}n s
i=1, rather than the existing dataset, expecting to relieve the sampling bias and the mentioned dilemma.
Fig.3 illustrates the model structure we use here. After the backbone we have the feature f ∈ Rdf , then we use additional one full-connected layer to have feature representation g ∈ Rdg , which is finally fed into the final classifier. We incorporate the mirror loss by applying LNA and ER to the source and target features f and g, resulting the mirror loss Lmr,f and Lmr,g . For the labeled source data {(xsi , ysi )}n s i=1, we use cross entropy loss, i.e.Ls = − 1ns ∑ns i=1 ∑M c=1 I(ysi = c) log psi,c , where psi,c is the predicted probability for class c and I is the indicator function. For the unlabeled target data {xtj}n t j=1, we follow the unsupervised discriminative clustering method [27] to introduce an auxiliary
distribution as soft pseudo label. Then we have the target-related loss as Lt = − 1nt ∑nt i=1 z t i · log pti, where pti ∈ RM is the predicted probability using current learned network and zti ∈ RM is the auxiliary distribution with each entity as zti,c ∝ pti,c ( ∑nt
i=1 p t i,c)
1/2 . Note that zti will be updated once for
each epoch rather than iterating until convergence in [27]. To sum up, the total loss of the model is:
L = Ls + Lt + γ(Lmr,f + Lmr,g) (8)

heading: 4 Asymptotic Properties of Mirrors
text: We present the theoretical analysis for the mirror sample-based alignment in terms of target error RT (h) and source error RS(h) following the theoretical framework of [2]. If the mirror-based
alignment empirically aligns the underlying distribution cross domains (Proposition 1), we could have lower target error asymptotically (Proposition 2). The proofs are given in Appendix C. Proposition 1. Denote ΦS(x),ΦT (x) as the density function for domain S and T , with supports as DT andDS respectively. H as the hypothesis class from features to label space. If ΦS(x) a.s. = ΦT (x),
then dH∆H(S, T )→ 0, where dH∆H(S, T ) = 2 suph,h′∈H ∣∣Prx∼DS [h(x) 6= h′(x)]−Prx∼DT [h(x) 6=
h′(x)] ∣∣.
Proposition 1 states that the distribution alignment for certain learnable space will reduce the domain discrepancy in terms of functional differences dH∆H. The above distribution alignment can be achieved empirically by minimizing the Lmr,x in Eq.(7). From the definition, we can see that Lmr,x is minimized if and only if qs(x̃s(xtj)) = qt(xtj) and qt(x̃t(xsi )) = qt(xsi ) for every mirror pairs for xsi and x t j . It means: 1) the class centers (anchors) for both source and target domain are same, 2) the mirror pairs cross domains have the same position relative to the common centers µc, c = 1, 2, · · · ,M . Thus the empirical density function Φ̂S(x) and Φ̂T (x) over XS and XT are same, i.e.Φ̂S(x) = Φ̂T (x). Since XS and XT are sampled from DS and DT w.r.t. the underlying density function ΦS and ΦT , Glivenko–Cantelli theorem [41] could assure that when nt, ns →∞, we have
ΦS(x) a.s. = Φ̂S(x) = Φ̂T (x) a.s. = ΦT (x) (9)
where a.s. means almost surely. So based on Proposition 1, minimizing Lmr,x will reduce dH∆H(S, T ) to zero empirically when the number of samples is large. Proposition 2. Define λ = minh∈H{RS(h, hS) +RT (h, hT )} same to [2], where hS and hT are the labeling functions in each domain. Denote λm + 12d m H∆H as the term of λ+ 1 2dH∆H when Lmr,x is minimized. If minimizing Lmr,x aligns the distribution in the learned space, we have
λm + 1
2 dmH∆H ≤ λ+
1 2 dH∆H (10)
Note that λ+ 12dH∆H is the main gap between source and target error stated in [2]. Proposition 2 indicates that when the mirror loss is minimized, we would get a lower gap. The key insight behind proposition 2 is that if the discrepancy of the underlying distribution is empirically approaching to 0, we can have a more relaxed effective hypothesis ofH, leading to a lower value of λ. This advantage can be obtained by mirror loss.

heading: 5 Experiments

heading: 5.1 Datasets and Implementations
text: Datasets. We use Office-31 [42], Office-Home[51], ImageCLEF and VisDA2017[39] to validate our proposed method. Office-31 has three domains: Amazon(A), Webcam(W) and Dslr(D) with 4,110 images belonging to 31 classes. Office-Home contains 15,500 images of 65 classes with four domains: Art(Ar), Clipart(Cl), Product(Pr) and RealWorld(Rw). ImageCLEF contains 600 images of 12 classes in three domains: Caltech-256(C), ILSVRC 2012(I) and Pascal VOC 2012(P). VisDA2017 contains ∼280K images belonging to 12 classes. We use “train” as source domain and “validation” as target domain.
Implementations. We implement our model in PyTorch. We use ResNet50 [24] or ResNet101 pre-trained on the ImageNet as backbone shown in Fig.3. The learning rate is adjusted by ηp = η0(1 + αp)
−β like [17], where p is the epoch which is normalized in [0, 1], η0 = 0.001, α = 10 and β = 0.75. The learning rate of fully connected layers is 10 times of the backbone layers. When calculating the centers µtf,c and µ t g,c, we use the class-wise centers of source domain µ s f,c and µ s g,c as the initial centers in k-Means clustering. To enhance the alignment effect, we use the centers µf,c = 0.5µ s f,c + 0.5µ t f,c and µg,c = 0.5µ s g,c + 0.5µ t g,c in calculating the mirror loss in Eq.(7). To estimate the virtual mirror using LNA, the additional operation “Top-k” can be implemented using priority queue of length k. It only brings additional O(ns) and O(nt) computation costs during training. The virtual mirror weight, i.e. ω(w, xtj) is 1/k. All the experiments are carried out on one Tesla V100 GPU.

heading: 5.2 Comparison with SOTA Results
text: We compare our methods with several types of SOTA methods under the same settings on the four datasets. They include the methods using maximum mean discrepancy and its variants: e.g. DAN [34], JDDA [7], SAFN [56], MDD [56] etc; class-wise center/centroid based methods: e.g. CAN [28]; adversarial learning related methods, e.g. DANN [17], ADDA [50] etc; as well as the most recent methods such as MCSD [59], SRDC [49] etc.
Table 1 shows the average results of all the tasks. Detailed results for each task can be found in supplementary material. We can find that our method has made a consistent and significant improvement over the existing SOTA methods. For the relatively simple datasets Office-31, our method improves by 0.3% comparing with SOTA. For the more challenging dataset Office-Home and the large-scale dataset VisDA2017, our method improves about 0.7% on average.

heading: 5.3 Model Analysis
text: We take Office-Home and Office-31 as examples to investigate the different components of the proposed model. Average results for all tasks are presented. Detailed results are given in the appendix.
Ablation Study. To investigate the efficacy of the proposed mirror loss, we experiment several model variations. The “Baseline” model only uses the backbone and the source and target losses, i.e. Ls +Lt in Eq.(8). Then the mirror loss is added gradually. Specifically, we applied the mirror loss to the last pooling layer of the backbone, i.e. feature layer f in Session 3.4 (“BK Mirror”), the output of first full-connected layer after the backbone, i.e. the feature layer g in Session 3.4 (“FC Mirror”) and finally both. Table 2 gives the results when k = 3 and λ = 1.0. From the results, we can see that by adding the mirror loss, the performance will boost at least 5.2% on Office-31 and 4.2% on Office-Home. The difference between the “FC Mirror” and “BK Mirror” is small. When we apply the mirror loss to both layers f and g, the final results can achieve SOTA on the datasets. In fact, only using losses on the source and target domains neglects the connection between the underlying distributions cross domains, which is what the mirror loss focuses on.
Sensitivity Analysis. The parameter k in Eq.(1) controls how we construct the virtual mirrors. A larger k means choosing a larger mirror set and it may lead to more indistinguishable mirrors. A smaller k may get an unreliable or even wrong mirror. In Table 3, we investigate the accuracies with different ks by setting k = 1, 3, 5, 7, 9. In terms of average accuracy, we can see that k = 3 is the best choice. The parameter γ in Eq.(8) controls the weight of the mirror loss. We tried different values from 0.0 to 3.0 to investigate the best choice for different tasks. Note the γ = 0.0 means we do not use mirror loss. Table 3 also gives the average results on both Office-31 and Office-home datasets. We can see that different tasks have different optimal γs (refer to the supplementary material). In most of the tasks, the optimal γ is in range 1.0 ∼ 2.0 for the two datasets. Besides above, we also evaluate the sensitivity of the key components to construct the mirror samples in Eq.2, i.e. the weight of w and the selection of d. We use Euclidean and Gaussian-kernel distance, combined with the weight of 1/k and inverse proportion of the distance. The final results varied not much on Office-Home, from 72.4 to 73.4, indicating that these components are robust to the variations. Detailed results are in Appendix E.
Visualization of Mirror Set To further illustrate the virtual mirror in real datasets, we visualize the mirror set defined in Eq.(1) in Fig.4 and 5. We use the embeddings in layer f to find the top-3 similar samples in the source w.r.t. to certain sample in target domain. As a comparison, the same top-3 similar samples by the model without mirror loss are also given. Overall, we can see that the top-3 similar samples with mirror loss are more similar compared to the results without mirror loss. Specially, the “clock” class in Fig.4 might consist of quite dissimilar samples although they belong to the same class for the model without mirror loss. In Fig.5, the “Bottle” has mirror sets belonging to the same class, but our proposed method gives results much more similar. For the “Helmet”, the results without mirror loss consist of even different class samples, such as “mouse”.
Impacts of the Biased Dataset. As pointed out in Section 2, the visual patterns are biased among the source and target domains, which will incur the dilemma of domain alignment. To elaborate how our proposed methods solve this issue, we compare the error rates of each visual pattern using the mirror samples or not. Table 4 gives the results for “Bed”, “Calendar” and “Bucket” in the task of “Ar → Cl” in Office-Home. For the category “Bed”, there is no “Bunk bed” for domain Ar while 7.1% for Clipart. Among the error-predicted samples, the “Bunk bed” consists of 2.0% for our proposed
method, which is much lower than 8.5%, the results of the baseline method without using mirror loss. Similar observation can be found for other patterns. These results directly validate that the proposed methods alleviate the bias of the performance resulting from the biased dataset.
Visualization of embeddings. We visualize the alignment procedure by t-SNE [25] of the feature embeddings. We record both source and target features in layer f during different training epochs. Then t-SNE is carried out for all the features at different epochs once to assure the same transformation is applied to them. Fig.6 shows the detailed evolvement of the different classes and distributions for the task of Ar to Rw in Office-Home. As a comparison, the evolvement for the model without mirror loss is also given. At beginning, the samples of source and target are messy. As the training progressing, such as epoch 100 in Fig.6(b) and Fig.6(e), the cluster discriminality and the cluster shape are clearer and more consistent. At the final stage, i.e. epoch 200, our proposed method has much more “shape” similarity between source and target domains. This reflects the proposed mirror and mirror loss have achieved higher consistency between the underlying distributions.

heading: 6 Related Works
text: Researchers have designed substantial approaches to eliminate the domain gap, from model side or data side. From the model side, existing methods try to align the two domains using different terms of discrepancy metrics, like maximum mean discrepancy (MMD) [21, 34], differences of the first- or high-order statistics of distributions [47, 58, 8], inter or intra class distance [7, 57], and even Geodesic Flow Kernel [52, 18] in Grassman manifold. Furthermore, to enhance alignment effects, adversarial learning between domain discriminator and classifier is also widely investigated, like gradient reversal layer in DANN [17], ADDA [50], conditional adversarial learning CDNN[35], 3CATN [30], multiple classifiers discrepancy [43] and batch spectral penalization [9] etc. There are also the meta-learning[53], disentangling learning [29] and dynamic weighting [54] following this line. The manipulation of data in cross domain alignment provides a strong complement to the model design. They include techniques like generating virtual or intermediate domain that boosts the performance by reducing the difficulty the model is facing [55, 12, 36, 37], data augmentation [31] that mimics the target domain. Inspired by GAN [19], generating the sample directly is also an effective way for domain adaptation to enhance the model capability cross domains. Typical works are PixelDA [5], CoGAN[33], CyCADA[26] and CrDoCo[10] etc. They generally work on the pixel-level domain gap like style or depth differences. In sum, almost all the methods are working to reduce covariate shift implicitly or explicitly.
As far as we know, there is few researches discussing the sample-level domain alignment, not alone considering the biased dataset. One type of the most related works is the methods using sample-level domain alignment. SRDC [49] used samples to form structure regularization for each domain. [16, 11, 13] took advantage of a simplified discrete version of optimal transport theory. [37] regularized the distance distribution of each sample for different prototypes. However, they do not consider the biased issue discussed in Section 2. The other type is the generative method like CyCADA [26] etc. Although generating virtual sample in target domain is like mirror sample in our methods, those work only apply to the pixel-level task or the classification tasks with strict domain gap assumption. Those methods heavily rely on GAN-based method, which is complicated comparing to our method and suffers from mode collapse and divergence. In fact, aligning the distribution by sample data can trace back to the work of [46, 20], but there is no further discussion in the UDA field.

heading: 7 Conclusion
text: In this paper, we uncover the dilemma when using the sampled data to reduce the covariate shift in unsupervised domain adaptation and further propose the mirror sample and mirror loss to solve this issue. The mirror sample is constructed using the local neighbor approximation and equivalence regularization, expecting to approximate equivalent samples in the marginal distributions. The ablation analysis as well as the visualizations demonstrate the efficacy of the proposed mirror and the mirror loss. Current construction method of mirror samples is a start point, which would have brittle improvements when the dataset is extremely biased or sparse. It is also limited to the visual classification task. More sophisticated and generic methods are expected. What’s more, we believe that both the dilemma and the idea of using equivalent sample for distribution alignment should have more discussion in both domain adaptation and transfer learning.

Review summaries:
In this paper, the author proposed a sample-level alignment method for solving the covariate shift problem in unsupervised domain adaptation. They noticed that the existing methods do not work well since they neglect the bias in the datasets (e.g. different visual patterns in the same category). To fix this problem, for each sample in the source/target domain, they construct a mirror sample in the other domain by a weighted average of the top K nearest neighbors. They also derived new losses to preserve the geometric structures of latent representations in the subspace. The overall idea is straightforward but interesting. The experimental results achieve SOTA performance on public benchmarks.

To reduce domain shift (covariate shift due to different domain distributions), the authors propose a loss based on Equivalence Regularization and Mirrored Samples.  The main idea is to: 1) find class prototype on both source and target domain and 2) encourage each sample's relative embeddings in the source and target domains to be similar.  Experiments are carried out on 4 datasets and obtained competitive accuracy against SOTA methods. 

This paper is addressing the domain adaptation problem as previous methods neglect the structural properties of the underlying distribution. This paper proposes a virtual mirror concept with Local Neighbor Approximation (LNA) and Equivalence Regularization (ER). The proposed method achieves state-of-the-art performance on several benchmarks.