Reducing the Covariate Shift by Mirror Samples in Cross Domain Alignment
Authors: Yin Zhao
Minquan Wang
Longjun Cai
Venue: NeurIPS 2021 Poster (2021)
Keywords: covariate shift
mirror samples
mirror loss
unsupervised domain adaptation

Abstract: Eliminating the covariate shift cross domains is one of the common methods to deal with the issue of domain shift in visual unsupervised domain adaptation. However, current alignment methods, especially the prototype based or sample-level based methods neglect the structural properties of the underlying distribution and even break the condition of covariate shift. To relieve the limitations and conflicts, we introduce a novel concept named (virtual) mirror, which represents the equivalent sample in another domain. The equivalent sample pairs, named mirror pairs reflect the natural correspondence of the empirical distributions. Then a mirror loss, which aligns the mirror pairs cross domains, is constructed to enhance the alignment of the domains. The proposed method does not distort the internal structure of the underlying distribution. We also provide theoretical proof that the mirror samples and mirror loss have better asymptotic properties in reducing the domain shift. By applying the virtual mirror and mirror loss to the generic unsupervised domain adaptation model, we achieved consistently superior performance on several mainstream benchmarks.

Reducing the Covariate Shift by Mirror Samples in Cross Domain Alignment
Yin Zhao
Minquan Wang" Alibaba Group
Alibaba Group
yinzhao Maizhao.2@@alll zydaliiabaaaabaaa inc. com
minquan www.:amnawwglllbbbaiiiiioo com
Longjun Cai Alibaba Group
longjun. clweallbaaaa com
Abstract
Eliminating the covariate shift cross domains one of the common methods to deal with the issue of domain shift in visual unsupervised domain adaptation. However, current alignment methods, especially the prototype based or sample-level based methods neglect the structural properties of the underlying distribution and even break the condition of covariate shift. To relieve the limitations and conflicts, we introduce novel concept named (virtual) mirror, which represents the equivalent sample in another domain. The equivalent sample pairs, named mirror pairs reflect the natural correspondence of the empirical distributions. Then mirror loss, which aligns the mirror pairs cross domains, is constructed to enhance the alignment of the domains. The proposed method does not distort the internal structure of the underlying distribution. We also provide theoretical proof that the mirror samples and mirror loss have better asymptotic properties in reducing the domain shift. By applying the virtual mirror and mirror loss to the generic unsupervised do- main adaptation model, we achieved consistently superior performance several mainstream benchmarks.
Introduction
Current deep learning models have achieved significant progress many tasks but heavily rely on the large amount of labeled data. The generality of the model may be severely degraded when facing the same task of different domain. So, domain adaptation (DA) attracts lot of attentions in recent years. DA has several different settings, such as unsupervised (34) or semi-supervised DA 23]. open-set [6] or closed-set DA (38], as well single multi-source DA
40 In this paper,
consider the closed-set, single-source unsupervised domain adaptation (UDA) on the classification task. In this setting, one has the source domain data with labels and is expected predict for the unlabeled target domain data. Both the source and target domains share the same class labels.
The main challenge for DA is the domain shift. It be further categorized into covariate shift 45 44, target/label shift (48) ettc. Specifically, define p (x. y) and p (a. y) as the joint distributions of the source and target domains. The covariate shift refers to the difference of the marginal distribution of x, i.e. pa (x)) + pz()), assuming the conditional probabilities cross domains are same, i.e. n² (v]a) = p² (1)z). Most of the current methods, in terms of learning domain-invariant representation or domain alignment, are working to reduce the covariate shift explicitly or implicitly.
"Equal Contribution Corresponding Author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

0.5
Bed
50.04
Calendar
0.4
P'Cy(F) + (((()
Bigeried Unsterying
Source Domain Samp is
Target Domain Samples
20.08
0.3
Undastying
15.0%
30.04
Redacieg Coveriste Shift
10.0H
20.0%
Art
0.2
Clipart
10.0%
0.1
0.0H
0.0%
W. People
W. Pillow
Bunk Bod
Single
Digit
0.0
7009X
Aligndd pain
-2.5
0.0
2.5
5.0
7.5
10.0 12.5
(a) Example visual patterns and ratios.
(b) One-dimension illustration of the dilemma.
Figure (a) The ratios of samples having certain visual patterns for the same class (Bed and Calendar) from different domains (Art v.s. Clipart) in Office-Home, (b) In one-dimension case, aligning the biased datasets of different domains (green dots vs red cross) will distort the underlying distribution (red dashed line), making the conditional probability in the alingned space (2) not same, i.e.p' (y it) # (y)ā)
They aligned the domain centers
34
8
4
47
or class-wise centers/prototypes 37 regularized and even aligned the two domains through
the margins of inter-intra class distances sample-level mappings 16 14 13
7
15 28
However, there exists covert dilemma between the covariate shift and its assumption in practice, which bounds the performance of those methods. That is reducing the covariate shift will break its assumptions empirically. The underlying reason is that the dataset we have is sampling result from the underlying distribution. The randomness in collecting the dataset for different domains varies a lot and introduces the sampling bias unavoidably (i.e. the sample pattern distributes differently cross domains). Aligning the marginal distributions of the biased datasets will distort the internal structure of the real underlying distribution, leading the underlying conditional distribution of the dataset cannot be same under the bias. Investigating the internal structure of the underlying distribution in DA has been noted by SRDC [49] using discriminative clustering (7]. by sample-level matching cross domains [13 [4], or by Optimal Transport Theory e.c. But they all neglect the issue mentioned above.
We first elaborate the dilemma mentioned above in Section Then our proposed methodology is presented in Section 3 We introduce the "mirror samples", which are generated to complement the potential bias of the sampled dataset cross domains, making each sample in the domain to have the corresponding instance in the opposite domain. We propose the mirror loss by regularizing the generated mirror sample with the existing sample to achieve more fine-grained distribution alignment. The properties of the proposed methods and the experiment results on several mainstream benchmarks are given in Section and Section
5
Delimma of Covariate Shift using Samples
We first revisit the intuition behind the method of reducing the covariate shift. In cross-domain visual classification, the domain distribution can be decomposed into class-discriminative latent visual pattern distribution (p(x)) and conditional class distribution (p(y[=)). The dissimiisee visual pattern refers to the intrinsic visual characteristics of each class, which is domain-agnostic and follows their natural distribution. The conditional distribution is the probability of those visual patterns belonging to a certain category. Reducing the covariate shift assumes the conditional distributions cross domains are same. Aligning the marginal distribution of visual patterns in certain latent space can achieve the ideal domain alignment.
Biased Sampling Dataset. Note that the dataset we have a sampling result w.r.t the intrinsic visual property distribution. The sampling process (i.e. dataset collection procedure) varies for different domains and inevitably introduces sampling bias cross domains, resulting the imbalanced patterns within one category etc. One observation of this sampling bias can be seen in the Fig ((a) where we take the category "Bed" and "Calendar" of domain "Art" and "Clipart" in Office- [5 as examples. For the same category, saying "Bed", some of the images are Bunk bed but the others

---

not; some of them have pillows while the others do not; even some of them have people on bed while the left are empty etc. However. the ratios of those different visual patterns differ dramatically in the two domains. As shown in Fig [(a]] the ratios of "with People" in Bed are much lower in "Art" than in "Clipart", while the Bunk bed ratio has higher percentage "Clipart". This difference also exists for "Calendar" in patterns like whether it has digit or not and whether it is daily calendar or monthly calendar. An ideal sampling process should assure the consistent ratios for these patterns following the category's s intrinsic distribution, but it's impractical.
Reducing Covariate Shift vs Assumption Violation. The biased dataset will induce dilemma if we still follow the philosophy of reducing covariate shift. Specifically, in the biased dataset, the samples in the source domains do not have the same counterparts in the target domain in distribution. If we align the two domains by the biased data, no matter using moments/prototype alignment or minimizing the sample-based domain discrepancy, we essentially align the samples to biased positions, making the intrinsic conditional probability, which should be same, distorted. Fig[T(h) gives a simplified illustration of this dilemma in one-dimension case. The underlying marginal distributions of x for source and target domains are U (-3,3 5) and (4, 12) respectively, where U means uniform distribution. The conditional distributions of belonging to certain class in source and target domains are N[-3,s (1, 1) and Nia,12 (8, 1), where Niu.s) (1, σr) is truncated normal distribution with support [a, b). f1 and mean and standard deviation respectively. This is an ideal case where reducing the covariate shift can perfectly eliminate the domain gap: offsetting the target samples to left by 7 without changing the class probability (from red solid line to green solid line in Fig (bb). both the conditional distribution and marginal distribution are same. However, in practice, we only have the random samples illustrated by dots in Fig [(b] If we still try to reduce the covariate shift, Le. offsetting the target samples to the source samples in the same order (any order of moments of are same in this case), the resulting conditional distribution for target domain (the red dashed line in ((b) will be distorted, breaking the assumption of covariate shift. This dilemma is what our proposed mirror sample and mirror loss are expecting reduce.
Proposed Methods
3.1 Preliminaries
Mirror Samples. A straightforward solution to reduce the dilemma is to find the ideal counterpart sample in the other domain. Those counterpart sample pairs are in the same positions in their own distribution, i.e. having the equivalent domain-agnostic latent visual pattern. We call it Mirror Sample. we could find those ideal mirror samples cross domains under the inevitable sampling bias, the alignment by reducing the covariate shift will not incur the conditional inconsistency anymore. The closest work to the mirror sample would be the series of pixel-level GAN based method, such as CyCADA (26) [CrDoCo[[0], etc. However, those methods are dedicated for pixel-level domain adaptation, assuming the domain gap is mainly the "style" difference. But domain shift is generally large and varied. What's more, the adversarial losses might suffer from the mode collapse and convergence issues. The experimental comparison can be found in Appendix E. Different from those ideas, we propose concise method to construct the Mirror Sample, which consists of Local Neighbor Approximation (LNA) and Equivalence Regularization (ER).
Optimal transport Explanation. In fact, the mirror sample can be formulated in terms of optimal transport theory (. Let T and T be the two transforms (push-forwards operators) on the two domain distributions ps and pT such that the resulting distributions
same, i.c. T T T This sheds light on an elegant way to reduce the domain gap. In this context, x² E D and x' DT are the mirror for each other if Tap" (x*) T'pp' Direct finding the push-forwards operators are almost impossible. However, if we would first find the mirror samples defined above, those operators would like be learned by those mirror constraints. The more detailed explanation and illustration can be found in Appendix A.
Similar to the existing settings of DA, denote the source samples as ((r), vf " of E y and the target samples as {x))z' where n² and n' are the numbers of source and target samples, y is the label set with M classes We also use X = {xr) XT = (x/))'' for brevity. The notations with "tilde" above refer to the mirror samples. Since the proposed method follows the
Denotations.
3

---

framework of domain- invariant representation, all the above notations as well as the mirror samples are in the latent space after certain transformation.
3.2 Local Neighbor Approximation
The Local Neighbor Approximation (LNA) is expected to generate mirror samples using the local existing data in the same domain. Considering that if the source and target domains are aligned ideally, the source and target domains two different views of the same distribution. The mirror pairs, although in different domains, are exactly the same instance in the aligned space. Inspired by d-SNE in [57]. we use the nearest neighbors in the opposite domain to estimate the mirror of a sample. Fig 2 gives an illustration of the local neighbor approximation cross domains. Formally, denote the distance measure of two samples. To construct the mirror of target sample zY, we first
find the nearest neighbor set in the source domain X (x²)), called mirror sets as follows:
(af) arg T =ex=d(2x
where Th is "top-k" operation that selects the top k-smallest elements in set Q with respect to the distance measure d. Then we estimate the mirror sample of æ by weighted combination of the samples in X (zf). This is following the conclusion that the learned features after the feature extractor lie in manifold 57):
(a ExExs w(a, 2z)x
where wxr, xr)) is the weight of the element x the mirror set X (x)). The weight can be inversely proportional to t d((z, 2²)) or simply 1/k. Symmetrically, we can also have the corresponding mirror of source sample of in target domain as म6 (x:²) analogous to Eq and
3.3 Equivalence Regularization
Although LNA provides way estimate the mir- ror sample, it cannot guarantee the equivalence of the mirror pairs. We propose an anchor-base based Equiv- alence Regularization (ER) method to enhance the equivalence cross domains.
Virtual Minros
Real Samples
x+)+ = T3
xx)
(xf)
In detail, define the centers for class e in source and target domains as p. and μ² respectively. If the distri- butions are aligned, those class-wise centers for the same class should be same. This means they could be the anchors cross domains. Inspired by the prob- ability vector used in SRDC [49], we introduce the relative position of sample z to an anchor p4: of its domain as:
Figure Local Neighbor Approximation: the mirror sets X and X are calculated by E using virtual mirrors. We show the case when we set 3.
exp d d(( m3)}
(aa)))
EM exp( d(r)), m&))
where d a distance measure. Then the relative position vector w.r.t all the anchors is
g' d((x)) = [ot (zf) gb(a)) (afa))
where M is the class number. For az/ 's mirror sample j (xt)) estimated by LNA, its relative position its anchors S symmetrically written as:
expl-d(5* m2)}
( (af))
(5)
ΣM exp -d- (z (2)
Its relative position vector w.r.t. to all source anchors is analogously written as:
(6)

---

Source Domain
Mirror Sample
Mirror Loss
Clssifier
LNA
ER
miirror
Backbone
Cluster
Louiraa
Target Domain
Source Data
Run per step Run per epoch
Source Feature
Target Data
Target Feature
Figure 3: Overall structure of the model: the LNA and ER applied to the output of the backbone and the first FC layer to calculate the mirror and mirror loss. The detailed algorithm can be found in Appendix.
Note that Eq (af (xx)) by switching the source and target domains.
and can be used to get the corresponding (x;)) and (=())) (1 (x())) and
To regularize the equivalence of the mirror pairs, we minimize the Kullback-Leible divergence of the relative position vectors of them, forming the mirror loss:
1 n2 Σ (L (# (x))) (x)))
TE E
L
where the first term of right side of the equation is aligning the relative position vector of target sample 2 and its virtual mirror J² (xx)) in source domain, while the second term is aligning the source sample x to its virtual mirror z² (xz) in target domain.
One should note that although some existing works KL divergence losses such TPN 37 SRDC (49 etc., mirror loss is quite different. The mirror loss minimizes the divergence the relative position vectors of mirror pairs to the anchors, ensuring the constructed sample by LNA to be the equivalent. The other methods do not involve constructed samples. They generally minimized the KL divergence of different distributions for the same sample.
3.4 Model with Mirror Loss
The mirror samples actually augment the existing datasets to {=;],- (²)(())) for source and {at {5 (xt)) for target. The mirror loss setups the constraints cross domains between (=)))) and {at 1)= {rfrr"
)= and {#* (=t)) rather than the existing dataset, expecting to relieve the sampling bias and the mentioned dilemma.
here. After the backbone have the feature f Rdy, then we use additional one full-connected layer to have feature representation g E RRd, which is finally fed into the final classifier. We incorporate the mirror loss by applying LNA and ER the source and target features f and g. resulting the mirror loss Lmr,f and Lmn.g- For the labeled source data ((nf, of ) we cross entropy loss, i.e.f* f Er Ees T(y c) log pf,c. where Pf. is the predicted probability for class and is the indicator function. For the unlabeled target data (x²) )
Fig illustrates the model structure
we follow the unsupervised discrininative clustering method [27] to introduce an auxiliary
distribution as soft pseudo label. Then we have the target-related loss as L nズ == of log pf, where p E R is the predicted probability using current learned network and 2 RM is the auxiliary distribution with each entity as 2/,0 X
1 n²
Pl.e
(Eri )2/2" Note that 2f will be updated once for
each epoch rather than iterating until convergence in (7). To up, the total loss of the model is:
-= L. + + (Lmrn + Emr.g)
4 Asymptotic Properties of Mirrors
We present the theoretical analysis for the mirror sample-based alignment in terms target error
R (h) and source error (s) following the theoretical framework of [2]. If the masror-based
5

---

aligmment empirically aligns the underlying distribution cross domains (Proposition I.. we could have lower target error asymptotically (Proposition 2 The proofs are given in Appendix C.
Proposition 1. Denote (,(x),, TT(x) as the density function for domain S and T, with supports as DJ and DS respectively. H as the hypothesis class from features to label space. If s()) = (r))), then dHau(S + O, where duau(S, = =suu PT (h(x) + ht' hx(x)) P (h(x) # (x)))
Proposition D states that the distribution alignment for certain learnable space will reduce the domain discrepancy terms of functional differences dHAH. The above distribution alignment can be achieved empirically by minimizing the Lurr, in Eq. From the definition, we can see that f minimized and only if q" (a])) = q' (a.)) and (3' (*())x))) = 4 dx())) for every mirror pairs for x. and azj- It means: 1) the class centers (anchors) for both source and target domain same, 2) the mirror pairs cross domains have the same position relative to the common centers Me: C 1,2,... ... Thus the empirical density function (s(x)) and (1:) over X and X are same, i.c. & (x) (ar). Since X and X are sampled from D$ and DJ w.r.t. the underlying density function 4 and rr, Cirvenko-Canteer theorem [41] could assure that when n', n² - oc,
we have
$2(())
s (z) = Tz)) (2)
where a.s. means almost surely. So based on Proposition I minimizing Lmr, will reduce dHAH(S, T) to zero empirically when the number of samples is large.
Proposition 2. Define À minkeH (Rs(h, hs) + RT ht)) same to 12), where hy and hy are the labeling functions in each domain, Denote Am + 4 2dHHA as the term of X + HHAS when Lmr, minimized. minimizing fmr, aligns the distribution the learned space, we have
Am +
ma 2 HAN S SA+ dHAH
(10)
2
Note that + EdZaH is the main gap between source and target error stated in [2]. Proposition indicates that when the mirror loss is minimized, we would get a lower gap. The key insight behind proposition is that if the discrepancy of the underlying distribution is empirically approaching to 0, we can have more relaxed effective hypothesis of H. leading to a lower value of X. This advantage can be obtained by mirror loss.
Experiments
5.1 Datasets and Implementations
Datasets. We Office-31 [42, Office-Home (1], ImageCLEF and VisDA2017[39 to validate our proposed method. Office-31 has three domains: Amazon(A), Webcam W) and Dsir(D) with 4.1 10 images belonging 31 classes. Office-Home contains 15,500 images of 65 classes with four domains:
Art(Ar), Clipart(CI), Product(Pr) and RealWorld(Rw)) ImageCLEF contains 600 images of 12 classes in three domains: Caltech-256(C), ILSVRC 201 20) and Pascal VOC 2012(P). VisDA2017 contains 280K images belonging to 12 classes. We use "train" as source domain and "validation" as target domain.
Implementations. We implement our model in Py Torch. We use ResNet50 24 or ResNet101 pre-trained the ImageNet backbone shown in Fig The learning rate is adjusted by Tp- 7o(1 + -$ like [17), where p is the epoch which is normalized in [0, 1], n7u 0.001, and 8 = 0.75. The learning rate of fully connected layers is 10 times of the backbone layers. When calculating the centers 1 and 1g,e. we use the class-wise centers of source domain pr.. and nÇ.c as the initial centers in k-Means clustering. To enhance the alignment effect, we use the centers Mf.c = 0.5 + 0.5 and #g.c = 0.5p g.c + 0.5y in calculating the mirror loss in Eq. To estimate the virtual mirror using LNA, the additional operation "Top-k" can be implemented using priority queue of length k. It only brings additional Onn²) and O(n') computation costs during training. The virtual mirror weight, i.e. w(w, x²)) is 1 1/k: All the experiments are carried out on one Tesla V100 GPU.
10

---

Table I: Comparsion with SOTA methods(%). All the results are based on ResNet50 except those with mark 1, which are based on ResNet101. Red indicates the best result while Blue means the second best. The mark means the result is reproduced by the offically released code.
Method
Office-31
Office-Home
CLEF
VisDA2017
Source Model 24
76.1
46.1
80.7
52.4
6D
88.9
68.1
88.5"
74.6
JDDA
80.2
58.5
83.3*
62.5*t
MCSD 59
90.7
69.6
90.0
71.3
SAFN 56
87.1
67.3
88.9
76.1
CAN 28
90.6
68.8
89.6"
87.2
RSDA-MSTN 22
91.1
70.9
90.5
75,8
SHOT
88.7
71.6
87.2"
79.6
SRDC 49
90.8
71.3
90.9
BSP-TSA
90.6
71.2
88.9"
82.0
FixBi 36
91.4
72.7
86.0"
87.2
Ours
91.7
73.4
91.6
87.9"
Table 2: Ablation studies using Office-Home and Office-31 datasets based on ResNet50.(K
Table 3: Parameter Sensitivity Analysis for k
Parameters
Office-Home
Office-31
Baseline
FC Mimor
Bk Mirror
OfficeHHome
Office-31
1
71.7
90.2
73.4
91.7
85.5
71.8
90.3
71.7
89.7
7
71.3
89.8
71.8
90.0
9
71.4
89.8
73.4
91.7
5.2
Comparison with SOTA Results
We compare our methods with several types of SOTA methods under the same settings on the four
datasets. They include the methods using maximum mean discrepancy and its variants: e.g. DAN
34
JDDA 77), SAFN [56]] MDD [56] etc; class-wise center/centroid based methods: e.g. CAN
adversarial learning related methods, e.g. DANN [1]. ADDA [50] etc; as well as the most
recent methods such as MCSD [59, SRDC [49] etc.
Table shows the average results of all the tasks. Detailed results for each task can be found in supplementary material. We can find that our method has made a consistent and significant improvement the existing SOTA methods. For the relatively simple datasets Office-31, method improves by 0.3% comparing with SOTA. For the more challenging dataset Office-Home and the large-scale dataset VisDA2017, our method improves about 0.7% on average.
5.3 Model Analysis
We take Office-Hoon Home and Office-31 as examples investigate the different components of the proposed model. Average results for all tasks are presented. Detailed results are given in the appendix.
Ablation Study. To investigate the efficacy of the proposed mirror loss, we experiment several model variations. The "Baseline" model only uses the backbone and the source and target losses, i.c. L + C in Eq.(8) Then the mirror loss is added gradually. Specifically, we applied the mirror loss to the last pooling layer of the backbone, i.c. feature layer f in Session 3.4 (*"B Mirror"), the output of first full-connected layer after the backbone, i.e. the feature layer in Session 3.4 ("FC Mirror") and finally both. Table 2 gives the results when k = 3 and À 1.0. From the results, we can see that by adding the mirror loss, the performance will boost at least 5.2% on Office-31 and 4.2% on Office- Home. The difference between the "FC Mirror" and "BK Mirror" is small. When we apply the mirror loss to both layers I and y. the final results can achieve SOTA the datasets. In fact, only using losses on the source and target domains neglects the connection between the underlying distributions cross domains, which is what the mirror loss focuses on.

---

9-10
Fays
WI
XDO
Visualization of mirror sets for Office Home: the source domain "Product" and the target domain is "Art".
Figure 4:
Figure 5: Visualization of mirror sets for Office 31: the source domain "Webcam" and the target domain is "Amazon".
Tarqet
Tarpet
W/O Mirror, Ar-RW, epoch
(b) W/O Mirror, Ar-RW, epoch 100
(c) W/O Mirror, Ar-RW, epoch
(d) Mirror, Ar-RW, epoch (c) Mirror, Ar-RW, epoch 100 Figure 6: Visualization of cluster evolvement for task Ar-Rw in Office-Home. Different classes are distinguished by color.
(f) Mirror, Ar-RW, epoch 200
Sensitivity Analysis. The parameter k in [q. controls how we construct the virtual mirrors. A larger & means choosing larger mirror set and it may lead to more indistinguishable mirrors. smaller k may different ks by setting choice. The parameter in Eq. (8) controls the weight of the mirror loss. We tried different values from 0.0 to 3.0 to investigate the best choice for different tasks, Note the y 0.0 means we do not use mirror loss. Table gives the average results on both Office-31 and Office-home datasets. We can see that different tasks have different optimal ys (refer to the supplementary material). most of the tasks, the optimal 7 is in range 1.0 2.0 for the two datasets. Besides above, we also evaluate the sensitivity of the key components to construct and the selection of d. We use Euclidean and coussiaa-keende distance, combined with the weight of 1/k and inverse proportion of the distance. The final results varied not much on Office-Home, from 72.4 to 73.4, indicating that these components are robust to the variations. Detailed results are in Appendix E.
an unreliable or even wrong mirror. In Table 3 we investigate the accuracies with is the best
1, 3, 5, 7, 9. terms of average accuracy, we can see that
mirror samples in 2 i.e. the weight of w
Visualization of Mírror Set To further illustrate the virtual mirror in real datasets, we visualize the mirror set defined Eq.(I] in Fig [ We use the embeddings in layer find the top-3 similar samples in the source w.r.t. to certain sample in target domain. As comparison, the same top-3 similar samples by the model without mirror loss are also given. Overall, we can see that top-3 similar samples with mirror loss are more similar compared to the results without mirror loss. Specially, the "clock" class in Fig might consist of quite dissimilar samples although they belong to the same class for the model without mirror loss. In 5 the "Bottle" has mirror sets belonging to the same class, but our proposed method gives results much more similar. For the "Helmet", the results without mirror loss consist of even different class samples, such as "mouse".
Impacts of the Biased Dataset. As pointed out in Section 2] the visual patterns are biased among the source and target domains, which will incur the dilemma of domain alignment. To elaborate how our proposed methods solve this issue, we compare the error rates of each visual pattern using the mirror samples not. Table gives the results for "Bed", "Calendar" and "Bucket" in the task of "Ar -+ CI" in Office-Home. For the category "Bed", there is no "Bunk bed" for domain while 7. % for Clipart. Among the preoicered samples, the "Bunk bed" consists of 2.0% for proposed
8

---

Table 4: Error Rates for visual patterns of task Ar - CI in Office-Home. "Source/Targe" refers to the ratios (%) of the visual pattern in source and target domains. "W/O Mirror" gives the error rates (%) using the baseline model without mirror loss.
Category
Bed
Calendar
Bucket
Bunk
With
With Pillow
Daily
W/O.
With
W/O.
Visual Pattern
Bed
People
Calendar
Digit
Brush
People
Source/Targe Target
0.0/7.1
2,5/20.4
12,5/14.3
0.0/20.5
15.0/42.6
0.0/37.0
82,5/100
W/O Mirror
8.5
28.8
59.0
42.0
31.0
42.0
57.0
Mirror
2.0
14.9
51.0
28.5
24.0
25.8
38.0
method, which is much lower than 8.5%, the results of the baseline method without using mirror loss. Similar observation can be found for other patterns. These results directly validate that the proposed methods alleviate the bias of the performance resulting from the biased dataset.
Visualization of embeddings. We visualize the alignment procedure by t-SNE 25 of the feature embeddings. We record both source and target features in layer f during different training epochs. Then I-SNE is carried out for all the features at different epochs once to assure the same transformation is applied to them. 6 shows the detailed evolvement of the different classes and distributions for the task of Ar to Rw in Office-Home. As comparison, the evolvement for the model without mirror loss is also given. At beginning, the samples of source and target are messy. As the training progressing, such as epoch 100 in Fig (6(b] and Fig [ the cluster discriminality and the cluster shape are clearer and more consistent. At the final stage, i.c. epoch 200, our proposed method has much more "shape" similarity between source and target domains. This reflects the proposed mirror and mirror loss have achieved higher consistency between the underlying distributions.
6 Related Works
Researchers have designed substantial approaches to climinate the domain gap, from model side or data side. From the model side, existing methods try to align the two domains using different terms of discrepancy metries, like maximum mean discrepancy (MMD) [2 34), differences of the first- or high-order statistics of distributions [47 [58 8. inter or intra class distance 57). and even Geodesic Flow Kernel [52] 18 in Grassman manifold. Furthermore, to enhance alignment effects, adversarial learning between domain discriminator and classifier is also widely investigated, like gradient reversal layer in DANN (17). ADDA [50), conditional adversarial learning CDNN(35), 3CATN B0). multiple classifiers discrepancy
43 and batch spectral penalization [9] etc. There
are also the meta -eattallaani disentangling learning (29 and dynamic weighting [54] following
this line. The manipulation of data in cross domain alignment provides a strong complement to the model design. They include techniques like generating virtual or intermediate domain that boosts the performance by reducing the difficulty the model is facing [55] [2] 36 37], data augmentation 31 that mimics the target domain. Inspired by GAN [1,, generating the sample directly is also an effective way for domain adaptation to enhance the model capability cross domains. Typical works are PixelDA (3), CoGAN CyCADA (6) and CrDoCo I0 etc. They generally work on the pixel-level domain gap like style or depth differences. In sum, almost all the methods are working to reduce covariate shift implicitly or explicitly.
As far as we know there is few researches discussing the sample-level domain alignment, not alone considering the biased dataset. One type of the most related works is the methods using sample-level domain alignment. SRDC [49] used samples to form structure regularization for each domain. [16 IT [3] took advantage of a simplified discrete version of optimal transport theory. 37 regularized the distance distribution of each sample for different prototypes. However, they do not consider the biased issue discussed in Section 2 The other type the generative method like CyCADA 26) etc. Although generating virtual sample in target domain is like mirror sample in our methods, those work only apply to the pixel-level task or the classification tasks with strict domain gap assumption. Those methods heavily rely on GAN-based method, which is complicated comparing to our method and suffers from mode collapse and divergence. fact, aligning the distribution by sample data can trace back to the work of [46 20), but there is no further discussion in the UDA field.

---

7 Conclusion
In this paper, we uncover the dilemma when using the sampled data to reduce the covariate shift in unsupervised domain adaptation and further propose the mirror sample and mirror loss to solve this issue. The mirror sample constructed using the local neighbor approximation and equivalence regularization, expecting to approximate equivalent samples in the marginal distributions. The ablation analysis as well the visualizations demonstrate the efficacy of the proposed mirror and the mirror loss. Current construction method of mirror samples is a start point, which would have brittle improvements when the dataset is extremely biased or sparse. classification task. More sophisticated and generic methods are expected. What's 's more, we believe that both the dilemma and the idea of using equivalent sample for distribution alignment should have more discussion in both domain adaptation and transfer learning.
also limited to the visual
References
[1] Computational optimal transport. Foundations and Trends in Machine Learning. 1 11(5-6)335--077, 2019.
[2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2):151-175, 2010.
[3] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1798)11798-1828 2013.
[4] Karsten M Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Berhhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22 (14):449-a) c57, 2006.
[5] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsu- pervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference compuder vision and pattern recognition, pages 3722-3731, 2017. 2017.
[6] Zhangji Cao, Mingsheng Long, Jianmin Wang, and Michael Jordan, Partial transfer learning with selective adversarial networks. In Proceedings of the IEFE Conference on Computer Vision and Patter Recognition, pages 2724-2732, 2018
[7] Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3296-3303, 2019.
[8] Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, and Xian-Sheng Hua. Homm: Higher-order moment matching for unsupervised domain adaptation. order, 1(10):20, 2020.
[9] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learning, pages 1081-1090. PMLR, 2019.
[10] Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, and Jia-Bin Huang. Crdoco: Pixel-level domain transfer with cross-domain consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1791-1800, 2019.
[11] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, pages 7739, 2017.
[12] Shuhao Cui, Shuhui Wang, Junbao Zhuo Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adversarial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12455-12464, 2020.
[13] Bharath Bhushan Damodaran, Benjamin Kellenberger, Remi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[14] Debasmit Das and CS George Lec. Sample-to-sample correspondence for unsupervised domain adaptation. Engineering Applications of Artificial Intelligence, 73:80-91, 2018.
[15] Zhijie Deng, Yucen Luo, and Jun Zhu. Cluster alignment with a teacher for unsupervised domain adaptation.
In Proceedings of the IEEE International Conference Computer Vision, pages 9944-9953 2019.
10

---

[16] R Flamary, N Courty, D Tuia, and A Rakotomamonjy. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell, 2016.
[17] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200 2030, 2016.
[18] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman, Geodesic flow kemee for unsupervised domain adaptation, In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2066-2073 IEEE, 2012.
[19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems. pages 6672-2680, 2014.
[20] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 344):5, 2009.
[21] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL http
j/mmr. .wg. http//paper/vv31.1 faapere//13/rreettnn122 grettonn2tt hhmml html
[22] Xiang Gu, Jian Sun, and Zonghen Xu. Spherical space domain adaptation with robust pscudo-label loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9101-9110, 2020.
[23] Gewen He, Xiaofeng Liu, Fangfang Fan, and Jane You. Classification-wware semi-supervised domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 964-965, 2020.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[25] Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. In Advances in neural information processing systems, pages 857-864, 2003.
[26] Judy Hoffman, Eric Tzeng, Tacsung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexci Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMLR, 2018.
[27] Mohammed Jabi, Marco Pedersoli, Amar Mitiche, and Ismail Ben Ayed. Deep clustering: On the link between discriminative models and k-means. IEEE Transactions Pattern Analysis and Machine Intelligence, 2019.
[28] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4893-4902, 2019.
[29] Seunghun Lee, Sunghyun Cho, and Sunghoon Im. Dranet: Disentangling representation and adaptation networks for unsupervised cross-domain adaptation. CoRR, abs/2103. 13447, 2021. URL Tttthhtta:///ara org/ htr/aba/21033 13447
[30] Jingjing Li, Erpeng Chen, Zhengming Ding, Lei Zhu, Ke Lu, and Zi Huang. Cycle-consistent conditional adversarial transfer networks. In Proceedings of the 27th ACM International Conference on Multimedia, pages 747-755, 2019.
[31] Shuang Li, Mixue Xie, Kaixiong Gong, Chi Harold Liu, Yulin Wang, and Wei Li. Transferable semantic augmentation for domain adaptation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1 1516-11525, 2021.
[32] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[33] Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. Advances in neural information processing systems, 29:469 477, 2016.

---

[34] Mingsheng Long, Yue Cao. Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105, 2015.
[35] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. arXiv preprint Stti::1705.100601 2017.
[36] Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1094-1103, 2021.
[37] Yingwei Pan, Ting Yao, Yehao Li, Yu Wang. Chong- Wah Ngo, and Tao Mei. Transferrable prototypical networks for unsupervised domain adaptation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 22.99 2247,2247, 2019.
[38] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pages 754-763, 2017.
[39] Xingchao Peng. Ben Usman, Neela Kaushik, Judy Hoffmmnn Dequan Wang. and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.
[40] Anastasia Pentina, ETH SDSC, and Christoph H Lampert. Multi-source domain adaptation with guarantees. In NeurlPS 2019 Workshop on Machine Learning with Guarantees, 2019.
[41] Svetlozar Rachev, Lev B. Klehanov, Stoyan Stoyanov, and Frank Fabozzi. Glivenko-cantelli theorem and maanttiinnkantrrovice invariance principle. 10. 1000/978-114614486666 (Chapteer 12):283-296, 2013.
[42] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. permission. transferring visual category models to new domains. 2010.
[43] Kuniaki Saito, Kohei Watamabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2018
[44] Yuan Shi and Fei Sha. Information theoretical learning of discriminative clusters for unsupervised domain adaptation. arXiv preprint arXiv: 6433. 2012.
[45] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likeldhood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[46] Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in neural information processing systems, pages 1433- 1440, 2008
[47] Baochen Sun and Kate Sacnko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pages 443-450. Springer, 2016.
[48] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33. 2020.
[49] Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8725 8735, 2020.
[50] Eric Treng, Judy Hoffman, Kate Sacnko, and Trevor Darrell, Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 7167-7176. 2017.
[51] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. 2017.
[52] Jindong Wang, Wenjic Feng, Yiqiang Chen, Han Yu, Mciyu Huang, and Philip S Yu. Visual domain adaptation with manifold embedded distribution alignment. In Proceedings of the 26th ACM international conference on Multimedia, pages 402-410, 2018.
[53] Guoqiang Wei, Cuiling Lan, Wenjun Zeng. and Zhibo Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. CoRR, 10t.22003.1355.. 2021. URL https //arxiv org/ abs///as///00 13575
12

---

[54] Ni Xiao and Lei Zhang. Dynamic weighted learning for unsupervised domain adaptation. CoRR. abs/2103.13814 2021. hitttas/://arzi
http:///wwwwwww..a/aaaaaaaaa org/aba/2103 13814
[55] Minghao Xu, Jian Zhang, Binghing Ni, Teng Li, Chengjie Wang. Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup, arXiv preprint arXiv:1912.01888 2019.
[56] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature approach for unsupervised domain adaptation. In Proceedings the IEEE International Conference on Computer Vision, pages 1426 1435, 2019.
[57] Xiang Xu. Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d-sne: Domain adaptation using stochastic neighborhood embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2497-2506, 2019.
[58] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger- Platz. Central moment discrepancy (cmd) for comain-invariann representation learning. arXiv preprint (it.:0700000811. 2017.
[59] Yabin Zhang, Bin Deng, Hui Tang, Lei Zhang, and Kui Jia. Unsupervised multi-class domain adaptation: Theory, algorithms, and practice. arXiv preprint vrXiv:2002.086%6 2020.
[60] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. arXiv preprint 19Xiv:1904.0580/ 2019.
13


Reviews:
[{"review_id": "tnkw1vIH1YX", "reviewer": "Reviewer_ZzRU", "summary": "In this paper, the author proposed a sample-level alignment method for solving the covariate shift problem in unsupervised domain adaptation. They noticed that the existing methods do not work well since they neglect the bias in the datasets (e.g. different visual patterns in the same category). To fix this problem, for each sample in the source/target domain, they construct a mirror sample in the other domain by a weighted average of the top K nearest neighbors. They also derived new losses to preserve the geometric structures of latent representations in the subspace. The overall idea is straightforward but interesting. The experimental results achieve SOTA performance on public benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "technical concerns:\n- In fig.1(b) one-dimension case, this example is somehow misleading. It is trying to convey the message that distribution alignment is equivalent to sample alignment. One may fail to align two distributions due to the different number of samples drawn from each distribution. This example is not convincing. \n- In fig.2, by the definition of mirror set in line 121 eq(1) (let's consider Euclidean distance), for each sample in source, its corresponding mirror set will always consist of samples close to the left boundary, for instance, blue triangles in the dashed circle. Is this true for the 2D example in fig.2?\n- According to line 223-225 and Algorithm 1 in the appendix, the centers of the target are computed based on the pseudo labels that generated from the classifier. Since the classifier may produce wrong labels for target examples, these target centers can be inaccurate. The mirror losses are highly dependent on these inaccurate centers to promote the latent representations preserve geometric structures in the subspace. Does the mirror sample still a semantic counterpart? Will their labels change from one category to another? Will this introduces extra noise for supervised learning (classification task)? Is there any strategy to handle these problems? \n\n*Originality*: The novelty is limited. As the author claimed, the main contribution of this paper is the idea of mirror samples and mirror losses. CyCADA uses GAN to generate a 'mirror sample' in the target domain for each sample from the source domain, while in this paper mirror samples were constructed by inquiring neighbors. I am quite interested in the comparison between these two approaches. However there is no comparison experiments for these two methods.\n\n *Quality*: The proposed method is well motivated. Their claims have theoretical analysis and are supported with good experimental results. But most ideas rely on heuristic knowledge. Considering the aforementioned technical concerns, the proposed method for constructing mirror samples is not principled. \n\n*Clarity*: The paper is easy to follow. But there are typos and grammar errors, e.g., line 146 eq(3),  $\\mu_c^s$ should be $\\mu_c^t$ .\n\n*Significance*: The significance of this submission is moderate.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "s-geOx0kx41", "reviewer": "Reviewer_2iWh", "summary": "This paper is addressing the domain adaptation problem as previous methods neglect the structural properties of the underlying distribution. This paper proposes a virtual mirror concept with Local Neighbor Approximation (LNA) and Equivalence Regularization (ER). The proposed method achieves state-of-the-art performance on several benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. claim concerns:\na) For Equivalence Regularization (ER), this paper adopts clustering to obtain the class centers for target domains. However, it is critical to have reliable clusters as the anchors. How to make sure this would happen? Moreover, if the target clusters are not reliable, this will also distort the internal structure of the intrinsic underlying distribution, which is one of the main claims in the paper. The details about this concern are not clear in this paper.\n\n2. technical detail concerns:\na) Equation (7) consists of two parts: the relation between target samples and source virtual mirrors (first loss term), and the relation between source samples and target virtual mirrors (second loss term). It would be great to discuss more these two terms. For example, which term is more critical in this problem? Why?\n\n3. experiment concerns:\na) There are some other state-of-the-art papers missing in this paper. For example, RSDA (CVPR’ 20) also achieves competitive performance on several benchmarks (e.g., 91.1 on Office-31, 70.9 on Office-Home, and 90.5 on ImageCLEF). FixBi (CVPR’ 21) adopts similar ideas that generate virtual samples and calculate the consistency between source and target domains. Moreover, FixBi also achieves state-of-the-art performance (e.g., 91.4 on Office-31, 72.7 on Office-Home, and 87.2 on VisDA-2017). It would be better to include them in the experiment comparison (also compare the difference of approaches).\n\nReferences:\n1. (RSDA) Xiang Gu, et al. \"Spherical Space Domain Adaptation with Robust Pseudo-label Loss\", CVPR, 2020.\n2. (FixBi) Jaemin Na, et al. \"FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation\", CVPR, 2021.\n\nb) The proposed method contains two parts: Local Neighbor Approximation (LNA) and Equivalence Regularization (ER). The ablation studies only investigate the details about ER (in Table 2). The contribution of LNA is not very clear. It would be better to investigate LNA as well. For example, designing experiments to adopt LNA but with other types of loss functions (e.g., those in SRDC [41]).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential negative societal impact is not discussed in this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "F5Yp7PCQEo", "reviewer": "Reviewer_SJwM", "summary": "To reduce domain shift (covariate shift due to different domain distributions), the authors propose a loss based on Equivalence Regularization and Mirrored Samples.\n\nThe main idea is to: 1) find class prototype on both source and target domain and 2) encourage each sample's relative embeddings in the source and target domains to be similar.\n\nExperiments are carried out on 4 datasets and obtained competitive accuracy against SOTA methods.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The idea of using mirrored samples to align domain embedding is quite interesting. \n- The author proposes to ER (Equivalence Regularization) to compute domain anchors as class prototypes for each domain.\n- Relative vector coordinates for each sample can then be computed relative to domain anchors.\n- Evaluation and ablations are conducted on 4 datasets, showing competitive accuracy against SOTA methods.\n\nWeaknesses:\n- The LNA and ER are run per-epoch, I am not convinced how they can effectively alter the embedding, if the datasets are large.\n- The author seems to assume both source and target have groundtruth labels. Is this a common practice in the domain adaptation usecases? Usually the target domain label is not accessible.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for unsupervised domain adaptation (UDA) by constructing "virtual mirror" samples of source domain in target domain (and vice versa) and then "aligning" the corresponding mirror pairs across domains for matching feature distributions. The method is novel (reviewer zzRU raised a concern on limited novelty by pointing out the connection with CyCADA but I think the proposed idea is sufficiently different) and shows strong empirical performance. The method has some heuristic components as pointed out by reviewers (eg, reliance on k-means to get clusters, the method to construct mirrors) however the authors have pointed out connections to optimal transport theory in their response. I suggest the authors make this more prominent in the revised version of the paper. Overall, the paper is above the acceptance threshold in my view.
Decision: Accept (Poster)