Domain Adaptation with Invariant Representation Learning: What Transformations to Learn?
Authors: Petar Stojanov
Zijian Li
Mingming Gong
Ruichu Cai
Jaime G. Carbonell
Kun Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Domain adaptation
transfer learning
deep learning
adversarial training
autencoders

Abstract: Unsupervised domain adaptation, as a prevalent transfer learning setting, spans many real-world applications. With the increasing representational power and applicability of neural networks, state-of-the-art domain adaptation methods make use of deep architectures to map the input features $X$ to a latent representation $Z$ that has the same marginal  distribution across domains. This has been shown to be insufficient for generating optimal representation for classification, and to find conditionally invariant representations, usually strong assumptions are needed. We provide reasoning why when the supports of the source and target data from overlap, any map of $X$ that is fixed across domains may not be suitable for domain adaptation via invariant features. Furthermore, we develop an efficient technique in which  the optimal map from $X$ to $Z$ also takes domain-specific information as input, in addition to the features $X$. By using the property of minimal changes of causal mechanisms across domains, our model also takes into account the domain-specific information to ensure that the latent representation $Z$ does not discard valuable information about $Y$. We demonstrate the efficacy of our method via synthetic and real-world data experiments. The code is available at: \texttt{https://github.com/DMIRLAB-Group/DSAN}.

Domain Adaptation with Invariant Representation Learning: What Transformations to Learn?
Petar Stojanov Zijian Li Mingming Gong",
Ruichu Cai Jaime G. Carbonell', Kun Zhang'
Carnegie Mellon University
School of Computer Science, Guangdong University of Technology
School of Mathematics and Statistics, University of Melboume
4 Broad Institute of MIT and Harvard
Abstract
Unsupervised domain adaptation, prevalent transfer learning setting, spans many real-world applications. With the increasing representational power and applicability of neural networks, state-of-the-arr domain adaptation methods make use of deep architectures to map the input features X to a latent representation Z that has the same marginal distribution across domains. This has been shown to be insufficient for generating optimal representation for classification, and to find conditionally invariant representations, usually strong assumptions are needed. We provide reasoning why when the supports of the source and target data from overlap, any map of X that is fixed across domains may not suitable for domain adaptation via invariant features. Furthermore, we develop an efficient technique in which the optimal map from X to Z also takes domain-specific information as input, in addition to the features X. By using the property of minimal changes of causal mechanisms across domains, our model also takes into account the domain-spednt does not discard valuable information about Y. We demonstrate the efficacy our method via synthetic and real-world data experiments. The code is available https: //gi thub. Com/DMMRLAB-Groupp.
information to ensure that the latent representation Z
Introduction
Unsupervised domain adaptation (UDA) is common setting for supervised learning, in which the labeled training and unlabeled test data come from different distributions. More formally, given features X e IRd and labels Y R. we observe labeled source and unlabeled target domain instances, represented by (xS, Sy
) my and x (xT) yur respectively, where ps (X,Y) * YY,,,Y), and ms and my are the number of observations in the source and target domains, respectively. The main challenge of domain adaptation is to use the given source domain observations for learning a predictor that will perform well in the target domain. To do so, procedure needs to make use of some similarities between the two domains. One way to formalize this is by making assumptions about how the joint distribution joint P(X, Y) changes across domains. For example, in the well-studied setting of covariate shift [3] 44 20
(x y
35 2 8). the marginal distribution P(X)
changes while the conditional distribution P(Y Y XX (i.c. the optimal predictor) is shared across domains.
However, in many real- world applications, P(Y can also change, and this requires making use of further assumptions. One such assumption is that the factorization P(X, Y) = allows for addressing the changes in P(Y) and P independently in situation where their
"These authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

respective changes are simple and easier to capture [48 30] In this particular setting, the problem is generally broken down into: (1) Target shift: P(Y) changes across domains while P(X stay
48 (2) Conditional shift: P(X changes across domains but P(Y) stays (3) Conditional-targer shift: Both P(X and P(Y) change independently across domains the most general setting under this generating process assumption 4 6 Then, assumptions about the changes of the factors of the joint distrubution can be made so that the problem is solvable, such as location-scale transformation [48 [6]. that the changing parameters lie low-dimensional manifold [33], and algorithms can be designed to enforce these constraints and make use of them for prediction in the target domain.
the same 34
21
29
the same 48
26
24
Another fruitful view of the problem is through the lens of representation learning, due to wide-spread applicability of neural architectures for many real-world problems. In particular, state of the art deep learning techniques harness the high representational capacity of neural networks to transform the input data into a latent feature representation which is predictive of the target variable Y in the source domain, and has the same distribution across domains. Formally, this means learning function (encoder) Φ : X + Z from the input space to latent space Z, such that (2) (Z). At the same time, function h Z Y can be learnt minimize the risk in the labeled source domain M. The hope is then, that the overall function := h o & will have low prediction risk in the target domain. However, the deseriedd theoretical and methodological framework does not guarantee that the learnt representation Z will have any relevant information for predicting Y in the target domain. Namely, one can easily have situation in which the learnt representation Z is marginally invariant (PS (2)= pr (Z)), but not conditionally invariant (P$ (Z(Y) * p" (Z]Y)), as discussed in [53]. This means that the learnt function g := h & can have very good prediction performance in the source domain, but generalize very poorly to the target domain.
In many methods, the same encoding function 6(X) across domains is used to learn invariant latent representations. This enjoys computational benefits and makes the learning proce- dure relatively simple, and the vast majority of approaches among many) employ this technique. However, in certain situations, the same encoding function across domains cannot learn a marginally invariant representation that is optimal for classification in the target domain. There certain studies which implement
x
(I2 22 23 [7]
Z
X
The underlying data- generating process under condi- tional shift, of the observed vari- ables Y and X, and the latent vari- able Z. 8x represents the chang- ing parameters of P( across domains.
Figure I:
(37 38 Unfortunately, such methods suffer from the following drawbacks: (1) the exact motivation behind having separate encoders for each domain is not clear; (2) including two separate encoders may be inefficient because it greatly increases the number of parameters that we need learn; (3) in the field of UDA based on invariant representation learning, there is still no principled way to guarantee that the learned marginally invariant representation Z has sufficient structural (semantic) information, and has the potential to be conditionally invariant.
encoders &y and ΦT.
In this paper we assume the setting of conditional shift, and we make use of the data-generating process to: (i) justify the use of two separate encoding functions in order to infer the latent representation, (ii) implement the two encoding functions more efficiently, and (iii) constrain the latent representation Z to have meaningful structure which useful for prediction domain. In Section 2, we first motivate the use of two separate encoders to infer Z. via rigorous treatment and an illustrative example. Subsequently, in Chapter 3, we introduce an efficient way to implement two separate functions for inferring Z. In Chapter 3 we shall also introduce principled way to ensure that the latent representation Z contains useful information for prediction, and finally, in Chapter 4 we provide empirical evaluation.
the target
1.1 Related Work
As previously mentioned, there a vast body of work exploring approaches to make use of latent invariant representations for the purposes of unsupervised domain adaptation, using both deep learning and more traditional techniques. Namely, invariant representations were considered as a linear projection in (26), where theoretical guarantees were established regarding the conditional

---

32"
AC
ACA a st. 1 (A) (P
8.5.
[s(a] fr(a), Va e
Target 0
Souree
Sornnc Y
Source
Target Y
Tirget -1
Turget Y
(a)
(b)
Figure 2: Two different scenarios in which a single encoder ((X) is not sufficient to learn Z.
invariance of the latent representation and the identifiability of the joint distribution in the target domain. Furthermore, this framework was generalized to nonlinear maps invariant latent representations, parameterized by neural network termed DANN 12]. In this framework, the constraint that p$ (Z) (T( can be enforced in various ways, including adversarial training, Maximum Mean Discrepancy (22), Wassertstein Distance [ and Margin Disparity Discrepancy [51). However, none of these studies have considered using two separate encoders for the two domains.
Subsequently, this reasoning has been considered in combination with image translation (CYCADA) [19]. Another assumption frequently made in recent studies is that the input features X have clustering structure in which each cluster has different label Y. Therefore, further directions were pursued by incorporating additional information contained in the unlabeled target domain data, such as pseudo-labels provided by the classifier which initially trained on the labeled source domain data [23 41 (termed CDAN and SLPP, respectively). In addition, one can also make use of this assumption to enforce that the prediction function does not pass through high-density regions in input space, performed by the DIRT-T algorithm (3).. The method by [37 makes use of two separate transformation functions, but does not regularize their output to preserve semantic information in any way. In addition to these, approaches, the DSN algorithm 4 takes into models the changing parameters of P(X, Y) in the latent representation and enforces that they are different from the invariant representation Z. The only method that domain- -speiffc information into account is the study by [38. in which domain information given and continuous, and therefore contains lot more information than the discrete domain index that is typically considered. In 17], a spherical architecture is used for Φ, however there is only one encoder for both domains. The study by (77 performs bridging between the source and the target domains using data-augmentation, but this is a technique specific to image data.
aware of which takes
In addition to domain adaptation methodologies in the setting where the is only one la- beled source domain, there have been several efforts tackle the problem of multiple-souree unsupervised domain adaptation. Namely, the method by 3 tackles the problem by using modified kernel SVMs to operate on both input features X and marginal distributions Px. Furthermore, the method by [46] assumes that the target domain is a linear mixture of the source domains, and infers the linear coefficients and makes use of them for prediction in the target domain. Furthermore, in [7]. data-generating process information is discovered from the multiple-domain data, and made use for recovering the changing parameters in the target domain order construct a predictor for
While the above-mentioned methods use various approaches to improve the quality of the learnt invariant latent representation and show competitive performance, there is no guarantee that these approaches are general enough to encompass all scenarios. Namely, the challenge that existing methods face is two-fold: (1) there is no principled way to ensure that marginal invariance (P*(Z) = (Z)) preserves conditional invariance (P$ (Z|Y) = (T)))Y)), (2) there are cases (especially in information, in addition to the features X. The aim of this paper is to address these two issues. In this paper, introduce the first invariant representation learning method that makes of the data-generating process to justify and efficiently use domain-specific information in the encoder ø and to properly constrain ensure that Z contains relevant label-specific information.
datasets) where the encoding function é needs domain-spectte
3

---

2 What Does it Take to Learn a Useful Invariant Representation?
To gain insights into what information is necessary for learning an intermediate representation Z via ø. let us consider the data-generating process depicted on Figure [ In this graphical model, the label Y is generated first from its prior distribution P(Y). Then, the invariant representation Z is generated from Y via the generating mechanism P(Z and X is subsequently generated from P(X]Z;0x) where 0x are the changing parameters of P(X) YY) across domains. We consider 0x as parameter vector with a constant value for each point within domain, and we generally assume that 0x contains little information, i.e. the change across domains in P(X is minimal. This generating process, in part or in whole, can be exploited in many classification applications. For example, in object classification in images, Y is the label (the object the picture was taken of), X is the observed pixels, and Z can correspond to latent features with semantic relevance, such as some function of the edges and contours on the image. In this case, 0x can correspond to different resolution, illumination conditions, or other environment-specific changes that are not relevant for predicting the class Y. In this graphical representation. we see that generally speaking, Z is conditionally dependent from 0x given X, although they may be marginally independent. This implies that in order to recover Z. given that X is considered, the information of x should also be considered in the transformation, because it is in the Markov Blanket of Z (the Markov Blanket of a variable of interest consists of all other variables that are either its parents, its children, or the parents of its children in the probabilistic graphical model. The Markov Blanket represents all of the information needed to predict the variable of interest).
To gain intuition why considering the Markov Blanket is important, consider the following scenario, depicted in Figure 2a In this setting, we are given the source domain with Uniform distribution, and the target domain with Gaussian distribution. Here, we have a region A C X' (highlighted in red), in which both of the domains' distributions have support, so there support overlap. However, each domain has a different density in this region. The following proposition claims that in this situation, we cannot infer an invariant representation Z from X with a single encoder Φ:
Proposition 1: Let A C X be region in inpul space such that pS(X E PT E A) > 0, and pS(X E EA) A) I PT A). Furthermore, let & X an encoder s.I. A = {a Ma)) E B} for some B C Z. Then, there is no function & s.t. ps (x(X) E B) = (o(X) B).
A) 0 and
+ 2 be
All omitted proofs can be found in the supplementary materials. This proposition implics that if the supports of the marginal distributions of the source and target domains overlap, and if the distributions are different in the specific region of the support overlap (set A), then we cannot use a fixed transformation to transform the data in that region an invariant representation in Z space.
There is additional case in which a single encoder & is not sufficient to learn optimal latent representation Z. The scenario is illustrated on Figure 2b] In this case, even though both domains have the same distribution P(X) in the region of support overlap between the source and the target domain (labeled in red), the value of Y is opposite between the two domains. The following proposition demonstrates that in this setting, a single encoding function I inherently sub-optimal:
Let the true labeling functions in the source and target domain be fss fr X + Y respectively. Let A C X be region s.t. fs(a) + fr()), Va E A. Let X2 + y be a composition of representation learner o X + Z and a classifier h : Z + y. If o is the same function across domains, then for a 0-1 loss, the risk over the region A: e e^(g) = (g) + +4 (g) 2 1, where eg (g)
Proposition 2:
and (g) are the source and target domain risks respectively.
In this scenario, no matter what criterion one uses to find the representation for downstream classification using h. as long as the two domains use the same transformation & for the representation, the points in A cannot be correctly classified in the target domain, if they are correctly classfied in the source domain. This because the final classifer of the data, y = h Φb fixed across domains. As a consequence, any point E A will be mapped to the same class label, no matter which domain is in. These two scenarios generally occur low-dimensional datasets, such as scientific and healthcare applications. In very high-dimensional structured data such images and text, there will

---

be some domain comain--peccfic dimensions which will have very different values in the source and the target domain (such as brightness in images in daytime vs. nighttime). This will result in no overlap between the supports of the source and target domain distributions, and thus the domain-speciffe dimensions will act as domain specific information. In this case using á which only a function of X is sufficient. However, even in this scenario, we need to find principled way to constrain the encoder such that Z contains relevant label-specific information for the prediction in the target domain.
The observations from the perspective of the causal mechanism and 8x provide hints how to tackle the problem of UDA using invariant latent representations in general way. First of all, we observe that 0x needs to be an input to the encoder ø. in addition to X. Secondly, in this causal mechanism, between and Z (and therefore the relationship between X and Y) is minimal. This allows us to: (1) model latent variable xx as a function of the domain index and provide it is input 6(X,0x 0x), instead of learning two separate encoders Øs and ΦT: (2) mimic the causal mechanism of generating X, given by X = o( 2 0x), ). via a decoder Ø - x Q + X, in which we constrain the influence of Ox be minimal. The hope is then that such a decoder would act as a regularizer on the encoder Φ. forcing it to preserve important semantic information when inferring Z. In this paper, we design a method based on this reasoning in order to address the problem of UDA in a principled manner.
generating X from Z
can safely assume that the influence of 0x the relationship
3 Proposed Method: Domain-Specifie Adversarial Training
Motivated by the above discussion, we aim to design use of the domain-specific parameters 0x. Let 0x E 10x 0x). which are its possible values in the source and target domains. The model for the proposed method DSAN (Domain-Specific Adversarial Network) is depicted in Figure
network which can make
We first consider the domain index of the data: j E {S, (which indicates whether data-point comes from the source or target domain), and parameterize the estimated latent domain-specific parameters 0%²²) as function of the domain index of the i-the data-point ji (0 for source, 1 for target domain). We make use of (زاق and data encoder & to obtain 2,55 = <(x (
X
where x(() and 2!") represent the the i-th input point and latent representation point respectively, from the j-th domain. We can then use the latent representation z!" to obtain a softmax probability
for the predicted label label g(5t) h())))). and this predictor can be trained in the source domain. To enforce the condition μs (Z) PT (Z), we also have an adversarial predictor which predicts the domain index, given by j = (z! In order to constrain the influence of domain-specift changes 0x. our model also needs to make use of (or mimic) the data- getaretennninggggg process depicted on Figure I in particular the process of generating X from Z and 0x. This can be achieved via decoder which reconstructs the features X in both the source and the target domain, from Z and 0x, given by
x(0) = H((())) (.2)). As described so far, our model consists of the following loss functions:
my meT
my
Lreconst.
||x: x11, Lelassif. ms
Voclog (ūie),
ms + m&
ms+mr
my
Line.
fj + (1 je) log(1 30), Coent Σ
h(z? )d log(h(()
Here, Creconst. Ledassif.. and Linv. are the reconstruction, classification and adversarial loss respec- tively. Leent. is the conditional cross-entropy of the predictions in the target domain, and intuitively, it serves to enforce that the decision boundary does not cross data-dense regions [32]. The current loss functions do not involve a constraint on the encoding transformation e which would prevent it from discarding valuable information about Y. We now describe how we can make use of the data-generating process and the assumption of minimal change across domains to enforce such a constraint.
3.1 Enforcing Minimal Change by Mutual Information Minimization
Before we delve into the details of how to enforce minimality of the influence of 0x on the generating process of X from Y, we need to find way to measure it. One way to formally do so is the joint
5

---

mutual information I(0x; (X,Y), Conveniently, this term is related to distributional divergence measure that can be minimized, as stated by the following lemma:
Lemma 1: Let X,Y ~ P(X,Y; 0x and let 0x be a discrete random variable with possible values
0% and 6 in the source and targer domains respectively. with uniform prior P(Ox 0xx )=== = 0.5, P(Ox = 8 0.5. Then, I[0x: (X, = JSD( (SD(Px ,x 0 x =xx) where JSD is the Jensen-Shannon Divergence, and Px xx,x:x=00 and Px xxxx:x==0g are the source and largel domain joint distributions respectively.
Therefore, to measure the influence of Ox on the generating process as mimicked by our decoder, we need to measure the ISD between the joint distributions as implied by the reconstructed data, given by JSD(P (S xxx=0xxx0000 xx (note that here
using inferred domain-specific
variables 0% and Ox). In theory, given enough data and expressiveness of the decoder, we can assume that we can learn to reconstruct the data perfectly. However, due to lack of labeled data in the target domain, we can only reconstruct the unlabeled features in the target domain, and have no access to P X
To cope with this, we can use the inferred domain-specing variable 0x to translate the source domain data to the target domain via the decoder: X trans. 0(z xx This translated data from the source to the target domain is labeled and can serve as an approximation the joint distribution in the target domain, which we can use constrain the JSD. A very intuitive way to minimize the JSD of the joint distributions of the source domain reconstructed and translated data, is to first define new random variables V == [X's YSS" and Virans := [X trane. y as concatenations of the features and the label. Then, the JSD can be minimized by ensuring that the distributions of this joint feature-label vector is the same between the source reconstructed and translated data: P(V)= = P(Virens.). This can be achieved in several different ways, on of which is training an adversarial classifier on V and Virans.. However, since Y is usually a scalar variable with a specific semantic meaning and often has simple parametrie form, concatenating to X would discard this prior knowledge, rendering this approach statistically inefficient.
Fortunately, the following theorem, adopted from 15,. gives upper bound of the JSD which is easy to minimize, and which makes use of the specific parametric form of Y the case of classification:
Theorem 151: Let Pyx|0x=0s and Pyx|0x=0T denote the source and target domain distributions respectively. Let Qvix denote the conditional distribution of Y given X specified by the auxiliary classifier he. We have:
00 |Pxx o S 201 ISD x x oz | Px
+ o2 KL KL(PY|Xxxx 11 + c2 KL = 11
where Cj and (2 are upper bounds of 1/ P (yl=x))(())))))) and 1 Qx (z) (x)) respec- tively (μ is a o-finite measure). Here, JSD( Xx|0x= ag l (x) is equal to the ISD between the source and target domain marginal distributions of and is fixed, we can ignore this term when we minimize the upper bound using our algorithm. Therefore, to minimize the two KI. divergences on the right-hand side:
(L ( 11Q;xx) and KI. KI. (Pxxxee==TT1
it suffices to train joint auxiliary classifier he on the reconstructed source domain data SS and the translated data TTans.. whose task is to learn to predict the source domain label. Therefore, we introduce cross cross-entropy loss L.rane. for the auxiliary classifier, whose two terms can be used to minimize the two KL divergences respectively:
C ms
C mn
Ctrans ΣΣ Vig log((f,)
ΣΣ
Viy log(ūi, where of = (o())))
i=
i=1

---

and trans. = he (2 itrons. are the softmax predictions of the reconstruction and the translation of the source domain respectively, using predictor h.. Therefore, the total loss function can be formulated in the following manner:
Ltotal Aa Frcoost. + Xa AeCclassif. x Ayninv. + X inn. + >eheent.
and can be minimized by alternating optimization:
min Liotal &.0.h.e
min b,y fine." ha
0%
Linv. (x8,27
5s H625,000 ,
j
0
S
S
Auxiliary fiar
trans.
he
oood 10000
Data Encoder
Dota Decoder
= =0128,00
T
E irans.
TT = -0 x
Label Classifer
0 X
y
Figure 3: A diagram of the proposed autoencoder framework. Here, the domain index is mapped to 0x. Subsequently, the input data and 0x are mapped via é to a latent representation z, which in turn is reconstructed & using the decoder é. and 0x as additional input. In addition, the penalty Line. is enforced ensure invariance between 2 and ZT. The hidden representation also used predict y the source domain. Finally, Ltrans-9 enforces the minimal change of P(X across domains. The pink and orange lines depict the flow of the source and target data and domain-specific information respectively.
This alternating minimization problem can be solved using Gradient-Reversa Layers (GRL), as shown in [3). Line. also be expressed in terms of Maximum Mean Discrepancy (MMD) [16]. Our model is now equipped with way to ensure that when reconstructing the data, 0 cannot play a major role in the decoder Φ. Since 0x correspond to minimal changes of the conditional distribution P(X[Y), this forces Z to retain Y-specific structure, and in implicitly regularizes the encoder é, resulting in a conditionally invariant representation. In the following section, we will empirically demonstrate this property of the constraint.
Empirical Evaluation
In order to provide intuition about the proposed method's advantages in the challenging scenarios described above, we first designed a simple but informative 2D simulated example for domain adaptation using mixtures of Gaussian distributions. This simulated dataset is presented on Figure (), and representative of the example discussed in Figure 2 since there a significant region of overlap between the two domains, in which the points have opposite labels across domains (details about the dataset can be found in the supplementary materials). In this experiment, we compared our method DSAN, with the following baselines: (1) DSAN-unreg, which is our proposed method but without regularizing the transformation é using the mutual information minimization in our loss; (2) DSN, the method proposed in (4) which makes use of domain-specific private encodings, but which are only used for reconstruction the data, and not for inferring the shared invariant representation
Z; (3) DANN, the classical domain- aadiinaddeen advesaaiia method proposed by [13]. We used 2D hidden
7

---

representation Z (more details about implementation and tuning can be found in the supplement).
The results are presented in Figure 4b demonstrating that our approach greatly outperforms the baselines. To visualize how our algorithm able to solve this challenging example with much greater success, we also present the 2D invariant representations in the source and target domains on Figure At the top, we have the hidden representations Z of DSAN-unreg. and on the bottom the ones yielded by our method DSAN (run on separate random initialization). On the left side we present Z in the source domain, labeled by the true values, in the middle we show this representation in the target domain labeled with the prediction of the algorithm, and on the right-hand side present Z in the target domain labeled with the true labels. From this figure, one can appreciate that the unregularized version of our approach yields a representation Z which discards lot of information about Y, as can be seen in parts 5b and 5c] On the other hand, parts 5e and [5] show that our algorithm with the mutual information minimization constraint can successfully preserve label-specific information in the invariant representation.
4.1 Real Data Experiments
To evaluate our method on real datasets, we consider three datasets and respective tasks from various domains of applications: cross-domain Wi-Fi localization, Amazon product reviews and image classification. For detailed descriptions of the experimental design, hyperparameter tuning and neural network architectures, well ablation studies, we refer the interested reader to the supplementary materials. We performed evaluation on the following datasets: Wi-Fi localization: introduced in grid. In
[4], is a dataset in which wireless signal data collected in hallway area discretized each grid, data was recorded from 67 access points (dimensions). The task is to predict the location from the signals (which has been converted a classification problem with 19 classes). There are three domains in this dataset (with 1 140 points each), collected in three different time points. This is low-dimensional dataset, in which we verified via manual inspection that both scenarios of support overlap described in Section 2 hold; Amazon Review: Amazon Review is another benchmark for multi-domain sentiment analysis. It contains positive and negative reviews of four kinds of products: Kitchen appliance (K), DVDs (D), Electronics (E), and Books (B). ImageCLEF: A standard UDA benchmark dataset for image classification, consisting of three domains: Caltech- 256(C,,, ImageNer ILSVRC(()), and Pascal VOC20 (PP)), consisting of 12 classes. Baselines: We compare DSAN with classical approaches like TCA (2 and GFK 14, as well as with some deep transfer learning models. Three recently proposed methods, DSR (. DIRT-T (22, MDD [51] BSP (7). MSTN 42) and RSDA [17] are included. For both the real-world and simulated experiments, we ran and tuned the baseline methods ourselves (all baselines for Wi-Fi and simulated, and MDD and DIRT-T for Amazon dataset).
The classification accuracies the Amazon-Revier Dataset for unsupervised domain adaptation are shown in Table which shows that our algorithm model outperforms the baselines in the vast majority of source-target directions. We also perform Wilcoxon signed-rank test on the experiment result of different random seed, and the p-value is 0.0216. The performance of our method on the Wi-Fi localization dataset is shown on Table [ In this experiment, domain to create 10 replicate experiments, and we present the accuracies and standard deviations across all replicates. From the results, one appreciate that our method outperforms all baselines, for the majority of the pairs. We calculated a Wilcoxon p value p 0.002 across all replicate experiments combined. The performance of our method on the ImageCLEF dataset is presented on Table 3 in which one can appreciate that we achieve state-of-the-as performance on most transfer directions. For this experiment, certain baselines such as MDD and RSDA use pre-trained ResNet neural network architecture as the transformation function o. For the proposed method, we instead used features obtained from a pre- pre-trained ResNet architecture, and we incorporated them into the autoencoder neural network described in the previous section (more details on the architecture and hyperparameters can be found in the supplementary information). While we currently obtain sfae-oo-the--n performance on this dataset, we believe that adapting our method to make use of ResNet fine-tuning will further increase the performance.
sub-sampled 950 points in each
Conclusion
In this paper, we have demonstrated that the existing paradigm of representation learning for domain adaptation using fixed encoder & of the data, can be sub-optimal when the same points can
8

---

Model DANN DSN DSAN-unreg. DSAN
Accuracy
58.2+t4.3
58.8+5.2
79.442.2
87:1.0
Source dem, dots
Source dom, class
(h) Results on the simulated data.
Target dom.
(a) Scatter plot simulated data
Figure 4
Table Accuracy (%) on Wi-Fi localization dataset for unsupervised domain adaptation
Models
tl + tl - t3
12 + tl
12 - t3
- tl
-> 12
Average
DANN
12
11.30
33.15
38.034...0
32.41-2.3
28.54-008
31.12::1.6
32.59
DSN
31.25::1.5
34.23+2.4
34.32+2.8
29.47=0.7
27.0::1.6
31.8
31,35
DSR
5
34.20::1.3
34.62-44.8
30.6143.55
34.72+1111
33.13=0.5
31.61+4099
31.13
MDD [11
30.92:42.1
34.42+1..8
30.01
27.58=18
33.25=1.0
27.3412.3
30.59
DIRT-T
332
36.38
33.6612.7
37.1243.4
31.85
33.98
28.4842.68
33.58
BSP
35.511..
32.1612.1
31.15+14
31.06-199
32.91=1.1
33,8312.2
32.72
DSAN-U 31.16:11.1
29.01
32.76+0.9
32.18=1.1
31.02=1.7
33.62.11.8
31.45
DSAN 40.45:1..6
37.35.12.3
38.22::2.4
39.94
36.47=3.5
34.18
37.77
have different labels across domains. We have introduced method which takes domain-speciie information in order provide the data encoder with the necessary flexibility and appropriate constraints, in order to retain valuable information about Y in the latent representation Z. We note that in many real-world applications with high-dimensional datasets, the supports of the source and target domains may not overlap. In this case, domain-specctf information contained in the features X and can be automatically used by the encoder (XX). However, o(X) would then have the same problem with excessive flexibility and can still suffer from finding trivial representations of that discard information about Y. Therefore, our principle for constraining Nomain-specific encoders would still be beneficial in this scenario. For future work, promising direction is combining this framework motivated by the data-generating process with more powerful encoders, such the spherical neural network proposed by [17].
6 Acknowledgements
are very grateful to the anonymous reviewers their help improving the paper. This work was supported in part by the National Institutes of Health (NIH) under Contract RO1HL159805, by the NSF-Convergence Accelerator Track-D award #2134901, by the United States Air Force under Contract No. Australian Research Council Project DE2 10101624. Ruichu and Zijian were supported in part by National Science Fund for Excellent Young Scholars (62122022), Natural Science Foundation of China (61876043, 61976052).
16660017-C7715 and by grant from Apple Inc. GM supported by

---

Table 2: Accuracy (%)
Amazon Review dataset for unsupervised domain adaptation
Methods
B-D
B-E
B-K
D--B
D-=E
D-*K
E-D
E-+B
E-K
K-B
K-D
K-+E
Avg
NN
49.6
49.8
50.3
53.3
51.0
53.1
50.9
51.2
52.2
52.3
TCA
63.6
64.2
63.3
64.2
74.8
65.4
74.5
65.5
GPK 14
66,4
65.5
69.2
66.3
63.7
67.7
62.4
63.4
73.8
65.5
65.0
73.0
66.8
SA
67.0
70.8
72.2
67.5
67.1
69.4
61.4
64.9
70.4
64.4
64,6
68.2
67.3
64.2
65.4
62.4
66.3
68.9
616
74.7
62.7
74.0
CORAL
36
71.6
65.1
67.3
70.1
65.6
67.1
67.1
66.2
77.6
68.2
68.9
75,4
JGSA 45
66.6
75.0
72.1
55.5
67.3
65.6
51.6
50.8
55.0
58.3
56.4
51.7
60.5
DANN
78.4
73.3
77.9
72.3
75.4
78.3
71.3
73.8
85.4
70.9
74.0
84.3
64.9
MDD 51
77.1
77.0
74.7
74.1
76.3
70.2
83.3
82.8
75.4
DIRT-T 32
78.6
76.1
75.5
76.8
75.2
79.1
69.6
71.0
84,2
69.2
73.3
79.5
75.7
EasTLL
40
79.8
79.7
80.9
79.9
80.8
82.0
75.0
75.3
84.9
76.5
76.3
82.5
79.5
BSP
79.33
73.86
75.86
75.67
74.77
77.2
72.81
71.3
84.02
70,92
73.59
84.29
76.14
RSDA 17
80.1
77.8
84.4
76.6
79.4
82.3
73.6
71.1
86.6
71.8
74.7
84.4
77.9
DSAN-U
81.0
78.6
78,9
78.4
79.4
83,3
76.2
74.5
87.4
73.1
77.0
83.0
78.9
DSAN
82.7
80.8
82.6
79.5
81.4
85.3
76.7
75.1
88.0
73.8
77.3
85.0
80.7
Table 3: Accuracy (%) on ImageCLEF dataset for unsupervised domain adaptation (* reproduced by
Models
I T-
P-1
I-C
C-1
C-
P C
Average
DResNet-50 18
74.81 0.3
83.91
91.5 1 0.3
78.0 0.2
65.5 10.3
91.2 1 0.3
80.7
iCAN 50
79.5
89.7
94.7
89.9
78.5
92.0
87.4
CDAN 2.1
7.3
90.7+0.2
97.+0.3
91.3+003
74.2+0.2
94.3+0.3
87.7
SymNets 52
80.2+0.3
93.6+0.2
97.010.3
93.420.3
78.740.3
96.410.1
89.9
SAFN+ENT
79.3+0.1
93.3+0.4
96.3+0.4
91.7=#00
77.6+0.1
95.3++0.1
88.9
CAT 10
77.210.2
91.610.3
95.510.3
91.310.3
75.31006
93.6.10.5
87.3
DANN 12
75.0++0.6
86.0+0.3
96.2+0.4
87.0+0.5
74.3+0055
91.5+0.6
85.0
DANN+S 7
78.3+:0.5
91.0+:.4
96.8+0.2
91.8+0..6
77.7+00.5
95.2-
88.5
RSDA-DANN 17
79.2=00.4
93.0+0.2
98.3=0.4
93.6+#0.4
78.5+0.33
98.2+0.22
90.1
RSDA-MSTY MSSTN 17
79.8-10.2
94.510.5
98.0.10.4
94.2110.4
79.2.10.3
97.310.3
90.5
MSTN' 42
77.3-0.3
91.310.4
96.8-0.2
91.2110.5
77.7-0.22
95.010.55
88.2
DSAN-U
79.2-0.2
94.120.11
96.210.2
93.5210.5
17.4:0.88
90.010.3
88.3
DSAN
0..
95.2.10.3
97.510.2
94.7.10.5
79.6
96.710..2
90.7
(a)
(b)
(c)
(d)
(e)
(
Figure
Z in the simulations. First three peateer-plos. show it without regularization: (a) Z of the source domain labeled with true labels, (b) Z the target domain labeled with the predictions (c) Z of the target domain with the true labels. (d), (e), and (f) show the same, but with regularization.
10

---

References
[1] S. Ben-Davvd, J. Blitzer, K. Crammer, and Percira. Analysis of representations for domain adaptation. In Proceedings of NIPS 2006, 2007.
[2] Ben-David, S. Shalev-Shwarte and R. Umer. Domain adaptation-can quantity compensate for quality? In ISAIM 2012, 2012.
[3] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to new unlabeled sample. In NIPS 2011, pages 2178-21880 2011.
[4] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pages 343-351, 2016.
[5] R. Cai, Li, P. Wei, J. Qiao, K. Zhang, and Z. Hao. Learning disentangled semantic represen- tation for domain adaptation. In IJCAI: proceedings of the conference, volume 2019, page 2060. NIH Public Access, 2019.
[6] Q. Chen, Y. Liu, Z. Wang, Wassell, and K. Chetty, Re-weighted adversarial adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7976-7985, 2018.
[7] X. Chen, S. Wang, M. Long, and J. Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. International conference on machine learning, pages 1081-1090. PMLR, 2019.
[8] C. Cortes, Y. Mansour, and M. Mohri. Learning bounds for importance weighting. In NIPS 23, 2010.
[9] N. Courty, R. Flamary, A. Habrard, and A. Rakotomamonjy. Joint distribution optimal trans- portation for domain adaptation. NIPS, 2017.
[10] Z. Deng, Y. Luo, and J. Zhu. Cluster alignment with a teacher for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9954,9953,22 2019.
[11] B. Fermando, A. Habrard, M. Sebban, and T. Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE international conference on computer vision, pages 2960-2967, 2013.
[12] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv: 7495, 2014.
[13] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Nomain-adversarte training of neural networks. Journal of Machine Learning Research, 17(1):20962200 -0300, 2016.
[14] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow kernel for unsupervised domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2066-2073. IEEE, 2012.
[15] M. Gong, Y. Xu, C. Li, K. Zhang, and K. Batmanghelich. Twin auxilary classifiers gan. In Advances in Neural Information Processing Systems, pages 1328-1337, 2019.
[16] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. JMLR, 13:723-773, 2012.
[17] X. Gu, J. Sun, and 7. Xu. Spherical space domain adaptation with robust pseudo-label loss. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9101-9110, 2020.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
11

---

[19] J. Hoffman, E. Tzeng, T. Park, J. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMIR, 2018.
[20] J. Huang, A. Smola, A. Gretton, K. Borgwardt, and B. Schölkopf. Correcting sample selection bias by unlabeled data. NIPS 19, pages 601-608, 2007.
[21] Z. C. Lipton, Y. Wang. and A. Smola. Detecting and correcting for label shift with black box predictors. arXiv preprint arXiv: 2018.
[22] M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. International conference on machine learning, pages 97-105. PMLR, 2015.
[23] M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 1640-1650, 2018.
[24] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision, pages 2200-2207, 2013.
[25] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning-Vei Volume 70, pages 2208-2217. JMLR. org, 2017.
[26] T. D. I. C. G. M. Gong. K. Zhang and B. Schölkopf. Domain adaptation with conditional transferable components. In ICML 2016, 2016.
[27] J. Na, H. Jung, H. Chang, and W. Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. arXiv preprint arXiv:201 1.09230.020... 2020.
[28] S. J. Pan, W. Tsang, Kwok, and Q. Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2): 220, 2010.
[29] M. D. Plessis and M. Sugiyama. Semi-supervised learning of class balance under class-prior change by distribution matching. In ICML-12, pages 823-830, 2012.
[30] B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In ICML-12, Edinburgh, Scotland, 2012.
[31] H. Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference, 90:227-244, 2000.
[32] R. Shu, H. H. Bui, H. Narui, and S. Ermon. A approch to unsupervised domain adaptation. In Proc. 6th International Conference Learning Representations, 2018.
[33] P. Stojanov, M. Gong, J. G. Carbonell, and K. Zhang. Data-driven approach multiple-source domain adaptation. Proceedings of machine learning research, 89:3487, 2019.
[34] A. Storkey. When training and test sets are different: Characterizing learning transfer. In J. Candela, M. Sugiyama, A. Schwaighofer, and N. Lawrence, editors, Datasel Shift in Machine Learning, pages 3-28. MIT Press, 2009.
[35] M. Sugiyama, T. Suzuki, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60:699-746, 2008.
[36] B. Sun, J. Feng, and K. Saenko. Return of frustratingly easy domain adaptation. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
[37] E. Tzeng, Hoffman, K. Saenko, and Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[38] H. Wang, H. He, and D. Katabi, Continuously indexed domain adaptation. arXiv preprint 0mXiv:2007.01800 2020.
12

---

[39] J. Wang, Y. Chen, S. Hao. W. Feng, and Z. Shen. Balanced distribution adaptation for transfer learning. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1129-11344 IEFF, 2017.
[40] J. Wang, Y. Chen, H. Yu, M. Huang, and Q. Yang. Easy transfer learning by exploiting intra- domain structures. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1210-1211 1215.122...... IEEE, 2019.
[41] Q. Wang and T. P. Breckon. Unsupervised domain adaptation via structured prediction based selective pseudo-labeling. arXiv preprini arXiv: 0eee19911.7798. 2019.
[42] S. Xie, Z. Zheng, L. Chen, and C. Chen. Learning semantic representations for unsupervised domain adaptation. In International conference on machine learning, pages 5423-5432. PMLR, 2018.
[43] R. Xu, G. J. Yang, and I.. Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 1426-1435, 2019.
[44] B. Zadrozny. Learning and evaluating classifiers under sample selection bias. In ICML-04, pages 114-121, Banff, Canada, 2004.
[45] J. Zhang. W. Li, and P. Ogunbona. Joint geometrical and statistical alignment for visual domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 1859-1867, 2017...6,.0077.
2017.
[46] K. Zhang, M. Gong, and B. Schölkopf. Multi-source domain adaptation: A causal view. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[47] K. Zhang, M. Gong, Stojanov, B. Huang, Q. LIU, and C. Glymour. Domain adaptation as problem of inference on graphical models. Advances in Neural Information Processing Systems, 33, 2020.
[48] K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang. Domain adaptation under target and conditional shift. In International Conference Machine Learning, pages $19-827, 2013.
[49] K. Zhang, V. Zheng, Q. Wang, J. Kwok, Q. Yang, and I. Marsic. Covariate shift in hilbert space: A solution via sorrogate kernels. In International Conference on Machine Learning, pages 388-395, 2013.
[50] W. Zhang, W. Ouyang, L. and Xu. Collaborative and adversarial network for unsupervised domain adaptation. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 3801-3809, 2018
[51] Y. Zhang, Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. volume 97 of Proceedings of Machine Learning Research, pages 7404-7413, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
[52] Y. Zhang, H. Tang, K. Jia, and M. Tan. Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 5031-5040, 2019.
[53] H. Zhao, R. T. d. Combes, K. Zhang, and G. J. Gordon. On learning invariant representation for domain adaptation. arXiv preprint arXiv: /901. 09453, 2019.
13


Reviews:
[{"review_id": "zneRShUmUtO", "reviewer": "Reviewer_DkiW", "summary": "This paper focuses on making use of latent invariant representations for unsupervised domain adaptation. The goal is to address two State-Of-The-Art existing methods issues. The first is there is no principled way to ensure that marginal invariance (P^S(Z) = P^T (Z)) preserves conditional invariance (P^S (Z|Y) = P^T (Z|Y)). The second is there are cases (especially in lower-dimensional datasets) where the encoding function Phi needs domain-specific information, in addition to the features X. \nThis paper introduces an invariant representation learning method that uses the data-generating process to justify and efficiently use domain-specific information in the encoder Phi and constraint it to ensure that Z contains relevant label-specific information.\nResults on three real datasets and one synthetic dataset show the proposed Domain-Specific Adversarial Network (DSAN) model’s performances (accuracy and Wilcoxon p-value) compared with existing methods and an unregularized DSAN (DSAN-U) version.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#Originality:\n\nUsing the data-generating process under conditional shift based on the causal mechanism of generating X from Z with Theta_X represents the changing parameters of P(X|Y) across domains is a brilliant idea. I appreciated how the authors motivated the use of two separate encoders to infer Z in chapter 2. Indeed, model Theta_X as a latent variable and provide it as an input to Phi in addition to X and, then, mimic the causal mechanism generating X via a decoder constraining the influence of Theta_X to be minimal (acting as a regularizer on the first encoder and forcing it to preserve semantic information when inferring Z) is very interesting and well intuitively explained.\n\n#Quality:\n\nThe submission technically sounds well, but there are mathematical notation problems: in line 21, k and m_S, m_T are not defined. Line 152, in proposition 1, the output space of Phi should be corrected. The notation of A elements must be homogenized (we meet a x in proposition 1 and an a in proposition 2). In proposition 2 it could be more appropriated to put the indices risks such as \\epsilon_S^A(g) instead of \\epsilon ^A(g) _S. Line 217 the second “reconstruct” loss has to be modified by “classif” loss. In Theorem 1 some comas are missed in distribution formulations. After line 256, there are some mistakes in the equations: the index j seems to be corrected by c. Line 259 denoted \\lambda_T could be confusing notation because T is yet used for the target domain. After line 260, in the equations, \\rho and h_a are not defined. \nIt could be appreciated that authors give more intuitions of some concepts such as “Markov Blanket” (line 141). Also, the authors have to improve the end of chapter 3 because it is not intuitively and difficult to follow. \nThen, in line 210, authors must cancel one “label” word. Line 253, “RHS” is not defined. \nTo conclude, this work seems to be in progress and should be improved because of all these points.\n\n#Clarity: \n\nPlease, read the previous remarks about “Quality” to understand why the submission is not clearly written.  \n\n#Significance:\n\nThe experimental results sound well bust MSTN approach is announced line 312 but missing in tables. Also, in line 326, the authors have to rectify the Figure 4(b) reference instead of “Table 4(b) because there is no table 4 in the paper. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors write no elements about limitations and societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical issue in this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Around 5 hours. "}, "extra_text": {}}, {"review_id": "oQ_SGkIkxn", "reviewer": "Reviewer_d96Z", "summary": "**Summary and contributions:**\nThe paper considers representation learning for unsupervised domain adaptation (UDA). They show that, in some cases, it is necessary to learn encoders that vary across domains, something which is rarely done in practice. They thus propose a new method for incorporating domain-dependent features in the encoding process.\n\nThe end model/objective contains many different components: it is essentially a DANN with a decoder that aims to reconstruct the inputs by using domain-dependent and independent features. Furthermore, the decoder is forced to rely as little as possible on the domain-dependent features. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Overall thoughts**: I think that the realization (and reasons) for using domain-dependent encoders are interesting and valuable. But I find the proposed objective too complicated (with 5 losses/components !)  to really be used by the community.\n\n**Strengths**:\n- **Valuable realization: domain-dependent encoders can be necessary**: The authors provide simple theoretical results that characterize some settings where domain-dependent encoders are necessary (e.g. conditional shift with shared support). Although very intuitive, I do not know of any previous work that pointed this out and I think that it is a valuable contribution to the UDA + representation learning subfield.\n- **Useful/thorough appendices**: the appendices are useful and of good quality: extended related work, ablation, experimental settings… I especially appreciate the ablation study that investigates whether all the components of the final objective are necessary.\n- **Decent performance**: On the presented datasets, the proposed method significantly outperforms the baselines. Similar results on standard UDA benchmarks would nevertheless be much stronger (see weaknesses below).\n\n**Weaknesses**:\n- **Very complicated objective**: I find the final objective to be a combination of too many terms (5 different components with their hyperparameters) to be practical. Ablation study shows that each component help, but there must be a simpler way of reaching the desired result.\n- **Missing important benchmark**: the standard UDA benchmark for image classification is OfficeHome/Office31. These results are crucially missing. Even more so given that (1) the results on ImageCLEF seem strong; (2) you seem to provide code for OfficeHome (in `mdd.py`). \n\n\n**Questions:**\n- Important: how many seeds were used to generate the results from Fig. 4.b? Were the results averaged over all seeds or was the best selected? These are important experimental details that are missing. I found that information for other datasets but not ImageCLEF. Given that in the original table standard deviations were provided, please add standard deviations.\n- Why do you bold the results from DSAN in table 4.b. when they are not the best (P->C or I->C)? I initially thought that the statistical difference might not have been significant but that wouldn’t explain why CDAN I->C is not bolded.\n- I’m having difficulties understanding why the results are much lower for every model in the ablation study and somehow make a huge jump when everything is together. Did you use the same hyperparameter strategy for the ablation study? If so do you have an idea as to why that happens? Providing standard deviations would really help to make sense of those results.\n\n**Suggestions**:\n- explain somewhere that for ImageCLEF you are working directly from features from a Resnet50 pre-trained on ImageNet (if that is the case).\n- use “section” instead of “chapter”\n- define what you mean by “minimality”, currently, it’s simply thrown to you in line 277 without saying why this is what you need/meant.\n\n**Minor suggestions**:\n- line 21: $y^{\\mathcal{S}}$ -> $y_k^{\\mathcal{S}}$\n- line 21: $x^{\\mathcal{S}}$ -> $x_k^{\\mathcal{S}}$\n- line 49: domain -> domain\n- line 50: .However -> . However\n- line 83: $Z$ to contain -> $Z$ contains\n- line 104: makes, -> makes <???>,\n- line 108: of which -> of, which\n- line 132: i.e. -> i.e., \n- line 132: low-dimensionality -> low information (? dimensionality has nothing fo with minimality)\n- Proposition 1: I think you are pissing an assumption in the proposition that states that in the support the density is different (you say it in the text but it should appear in the proposition).\n- Theorem 1: define $c_1,c_2$ (constants that upper bound [...])\n- Figure 3: it’s currently hard to parse. Although it might have more to do with the complexity of the objective than the illustration. - I would also use a darker green to improve readability. \n- Figure 4.b.: remove newline before “)”.\n- Table 1: DANN t1-t2 ((1.3) -> (1.3)\n- Table 1: what does parenthesis mean? if standard deviation then please use $\\pm$\n- Figure 5: withour -> without\n- References: many duplicated entries e.g. 21-22 or 23-24 or 33-34. Please clean that up.\n- Checklist: remove instructions (see line 486-487)\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are briefly mentioned, e.g., that the method might not be as useful in high dimensions (line 177-181) \"high dimension [...] no overlap [...] $\\phi$only a function of X is sufficient\". The discussion of limitations could be extended, e.g., the fact that the objective is very complicated and what are the downsides (e.g. hyperparameter tuning for balancing each component).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "Cv3BgLBlui2", "reviewer": "Reviewer_219a", "summary": "The paper proposes a unsupervised domain adaptation method through the lens of learning invariant representations. The paper validates theoretically shows some limitations of current methods when learning from distinct source and target distributions. The paper validates experimentally on a large scale image dataset and 2 other datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the paper achieves promising results and the paper provides some theoretical justifications for their method, overall the paper is not motivated well and can be challenging to understand. For example, figure 2 is not explained very well in the text or caption, and figure 3 does not detail their proposed architecture. The architecture bears resemblance to Cycle-GAN and other related work. The paper cites CYCADA as related work but does not compare against it. The paper also does not cite very relevant work such as [1,2,3] and many others. The notation used can also be confusing in the text. The main novelty over CYCADA seems to be the L_{inv}^{trans}  which is used to minimized the KL divergence between marginals of the source and target distributions, however such ideas have existed in the community for some time. \n\n\n[1] https://arxiv.org/abs/2004.07780\n[2] https://openreview.net/forum?id=lQdXeXDoWtI\n[3] https://arxiv.org/abs/2003.00688", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not discuss any societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "AzXYfzz4ii0", "reviewer": "Reviewer_8KhW", "summary": "This paper analyzes the mechanism of domain adaptation and proposes a new strategy to reduce the gap between the two domains. The valuable information to achieve accurate classification can be well preserved. Extensive experiments are conducted on multiple dataset, such as Wi-Fi localization and Amazon Review.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n\n- It is interesting to understand the mechanism behind domain adaptation and this paper analyzes it from both empirical and theoretical perspectives. These discussions may bring some insights to the community.\n\n- The experimental results show that the proposed method achieve higher performance compared with the existing methods, which verifies the effectiveness of the proposed method.\n\n- This paper really make a contribution to the community.  Beyond the invariant features, the authors discuss that the domain-specifc information is also valuable. It has been empirically demonstrated by extensive experiments.\n\nConcerns: \n- The paper proposes a question about domain adaptation, i.e., 'What Transformations to Learn'. However, this question seems to not be answered explicitly. Could the authors conclude it with two or three sentences?\n- The presentation of the paper can be refined.  For example, the fonts of captions in Figure 3 is very different from that in the main paper. The titles of Figure 4 can be improved.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7.5h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study unsupervised domain adaptation. They point out that if the data supports overlap, the encoder might need domain-specific information to recover an invariant representation which elicits a high-accuracy classifier. They propose a method which models this domain-specific information as a latent variable while encouraging it to have minimal influence on a decoder.

Reviewers agreed that the motivating analysis (necessity of domain-dependent encoders) was informative. Most reviewers (DkiW, 8KhW, 219a) thought the experiments were promising (reviewer d96Z remarked that OfficeHome results were missing, but the authors address this in the rebuttal). Reviewers also tended to agree that the method itself was overcomplicated and this might hurt its usefulness in practice (319a, d96Z), but I feel this shouldn't by itself prevent acceptance.
Decision: Accept (Poster)