Domain Aggregation Networks for Multi-Source Domain Adaptation
Authors: Junfeng Wen
Russell Greiner
Dale Schuurmans
Venue:  (2020)
Keywords: Domain Adaptation
Transfer Learning
Deep Learning

Abstract: In many real-world applications, we want to exploit multiple source datasets of similar tasks to learn a model for a different but related target dataset -- e.g.,  recognizing characters of a new font using a set of different fonts. While most recent research has considered ad-hoc combination rules to address this problem, we extend previous work on domain discrepancy minimization to develop a finite-sample generalization bound, and accordingly propose a theoretically justified optimization procedure. The algorithm we develop, Domain AggRegation Network (DARN), is able to effectively adjust the weight of each source domain during training to ensure relevant domains are given more importance for adaptation. We evaluate the proposed method on real-world sentiment analysis and digit recognition datasets and show that DARN can significantly outperform the state-of-the-art alternatives.

Under review as a conference paper at ICLR 2020
DOMAIN AGGREGATION NETWORKS FOR MULTI-SOURCE DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
In many real-world applications, we want to exploit multiple source datasets of similar tasks to learn model for a different but related target dataset e.g.. recognizing characters of a new font using a set of different fonts. While most recent research has considered ad-hoc combination rules to address this prob- lem, extend previous work on domain discrepancy minimization to develop a finite-sample generalization bound, and accordingly propose a theoretically jus- tified optimization procedure. The algorithm we develop, Domain AggRegation Network (DARN), is able to effectively adjust the weight of each source domain during training to ensure relevant domains given more importance for adap- tation. We evaluate the proposed method on real-world sentiment analysis. digit recognition and object recognition datasets and show that DARN can significantly outperform the fifte-of-theaaae art alternatives.
INTRODUCTION
Many machine learning algorithms assume the learned predictor will be tested on the data from the same distribution as the training data. This assumption, although reasonable, is not necessarily true for many real-world applications. For example, patients from one hospital may have different distribution of gender, height and weight from another hospital. Consequently, diagnostic model constructed at one location may not be directly applicable to another location without proper adjust- ment. The situation becomes even more challenging when we want use data from multiple source domains to build model for a larget domain, as this requires deciding, e.g., how to rank the source domains and how to effectively aggregate these domains when training complex models like deep neural networks. Including irrelevant or worse, adversarial, data from certain source domains can severely reduce the performance on the target domain, leading to undesired consequences.
To address this problem, researchers have been exploring methods of transfer learning (Pan & Yang. 2009) or domain adaptation (Mansour et al., 2009a Cortes et al., 2019), where a model is trained based labelled source data and unlabelled target data. Most existing works have focused on single-source to single-targs ("one-to-one" domain adaptation, using different assumptions such covariate shift (Shimodaira 2000 Gretton et al., 2009 Sugiyama & Kawanabe 2012) concept drift @Jian & [007| Gama et al. al.] 2014). When dealing with multiple source domains, one may attempt to directly use these approaches by combining all source data into large joint dataset and then apply one-to-one adaptation. This naïve aggregation method will often fail as not all source domains are equally important when transferring specific target domain. There are some works on multi-source to single-target adaptation. Although many of them are theoretically motivated with cross-domain generalization bounds, they either use ad-hoc aggregation rules when developing actual algorithms ÇZhao et al. 2018 Li et al. (018) or lack finite-sample analysis (Mansour et al.) 2009b c Hoffman et al. 2011 8a). This leaves gap between the theory for multi-source adaptation and theoretically sound algorithm for domain aggregation.
This research has three contributions: First, we extend prior work on one-to-one adaptation using discrepancy (Cortes et al. 2019) to develop a finite-sample cross-domain generalization bound for multi-source adaptation. We show that in order to improve performance on the specific target do- main, there is trade-off between utilizing all source domains to increase effective sample size, versus removing source domains that are underperforming or not similar to the target domain. Sec- ond, motivated by theory and domain adversarial method (Ganin & Lempitsky 2015 Ganin

---

Under review as a conference paper at ICLR 2020
et al. 2016), propose Domain AggRegation Network (DARN), that can effectively aggregate multiple source domains dynamically during the course of training. Unlike previous works, our ag- gregation scheme (Eq. (6). which itself is of independent interest in some other contexts, is direct optimization of our generalization upper bound (Eq. (3) without resorting to heuristics. Third, our experiments sentiment analysis, digit recognition and object recognition show that DARN significantly outperform state-of-the-ar methods.
Section 2 introduces necessary background one-to-one adaptation based discrepancy. Then Section elaborates our theoretical analysis and the corresponding algorithm deployment. Section discusses about related approaches more detail, highlighting the key differences the approach developed here. Section empirically compares the performance of the proposed method to other alternatives. Finally, Section concludes this work.
BACKGROUND ON CROSS-DOMAIN GENERALIZATION
This section provides necessary background from previous work on one-to-one domain adapta- tion (Cortes al., 2019). Let A' be the input space, ) C R be output space and H (h X' - )) be a hypothesis set. loss function I. y ) + TR+ is j-admissible"Iit
// EV
[L v) Mly",v) S pa lv/ y/|l
The discrepancy Mansour et al. 2009a) between two distributions P, Q over X' is defined as
disc P) max b.
[Cr(h, h²) Lo(h, R') where Cp(h, h') Eg- (L())(
This quantity can be computed or approximated empirically given samples from both distributions. For classification problems with 0-1 loss, it reduces to the well- dd-distance (Kifer et al. 2004] Blitzer et al.) 2008 Ben-David et al., al., 2010) and can be approximated using a domain domain-classifeer loss w.r.t. H (Zhao et al., 2018 Ben-Daavid et al.) 2007 Sec.4), while for regression problems with L2 loss, it reduces to a maximum eigenvalue (Cortes & Mohri 2011 Sec.5); see Section [32 for more details. For domains fP), (Q, fQ) where fP. X + are the corresponding
labeling functions, we have the following cross-domain generalization bound:
8, Cortes al. (019)) Let Ry (H) be the Rademacher complexity of H given sample size m. Ho = (x I. (()), fo(x) h H) be the set of functions mapping x to its loss w.r.t fo and H.
Theorem (Proposition
TH 1 min NEW
max #Esupp(P)
(ff(z) -(xx]++
max (f((z)-((z)) @Ésupp(()
be a constant measuring how well H. can fit the true models where supp(P) means the support of the empirical distribution P (using the from Eq. D and MQ = SUPzEX IA h(x). fo(x)
be the upper bound on loss for Q. Given Q with m points sampled iid from Q labelled according to fQ. for 8 E (0,1), vh E H. w.p. least 8.
(og(1/5)
fp(h,fp) s fo) disc (P,Q) 2mm (HQ) + TH + Mo
2m
This thcorem provides way generalize across domains when have sample Q labelled accord- ing to fo and an unlabelled sample P. The first term is the usual loss function for the sample Q, while the second term disc P. Q) can be estimated based on the unlabelled data Q, P. 7M measures how well the model family H. can fit the example from the datasets, and it is not controllable once H is given. The final term, as a function of sample size m, determines the convergence speed.
For example, the common T.p losses are j-addmisiible Cortes et al. 2019 Lemma 23).
2

---

Under review as a conference paper at ICLR 2020
3 DOMAIN AGGREGATION NETWORKS
3.1 THEORY
def (1,2,...,k3} and a target domain Suppose we are given source domains {(S,, fs.) : E [A] (T, fr) where S I are distributions over X' and fs: fr X + y are their respective labelling functions. For simplicity, assume that each sample S, has m points, drawn iid from S; and labelled according to fs.- We are also given m unlabelled points T drawn iid from T. We want to leverage all source domains" information to learn model h E H. minimizing C fT).
One naive approach could be to combine all the source domains into large joint dataset and con- duct one-to-one adaptation to the target domain using Theorem [ However, including data from irrelevant or even adversarial domains is likely to jeopardize the performance on the target domain, which is sometimes referred to negative transfer (Pan & Yang 2009). Moreover, certain source domains may be more similar or relevant to the target domain than the others, it makes more sense adjust their importance according to their utilities. We propose find domain weight o; 2 such that Σ 0g = 1 to achieve this. Our main theorem below sheds some light on how we should combine the source domains (the proof is provided in Appendix A
Theorem 2 Given i source domains datasets ((r))
(1) : [b],j E [m]]] with m iid examples .Uj
each where S, {x and (0) fs. (z)) ). ). for any a A {o : a 2 0. O.E, 1).8 (0, 1), and vh H. w.p. at least 1 -
log(1 (1(()))
(h, S Σ at (Cg, (h. + disc(T, St) + 2mm (Hs.) + mv.s) |aallaMs
2m
(3)
where Hs. fx + [(h(z).fs,zz) h E H. is the set of functions mapping x to the corresponding loss, THIs is a constant similar to Eq. 2 with Q = S, P T and Ms = SuPxe(4),exxAe f(,(z), (x)) is the upper bound on loss on the source domains.
There are several observations. (I) In the last term of the bound, m/|la serves as the effective sample size. If a is uniform (i.e., [1/k,.... 1/k] hot, then we effectively only have points from exactly one domain. (2) R.,. (Hs,) determines how expressive the model family H is w.r.t. the source data S, It can be estimated from samples (Bartlet) & Mendelson 2002 Theorem 11), but the computation is non-trivial for model family like deep neural networks. The 77H is uncontrollable once the hypothesis class 7t (e.g., neural network archi- tecture) is given. Using a richer H. is not always beneficial. Richer H. can reduce the THZ and also help us find a better function l with smaller source losses L3, but it will increase the R,, (Hs,) ). As removing these two terms does not seem to be empirically significant, we ignore them below for simplicity. (3) Let Gh.t def L3, (h. fs.) ) + disc(T, S.). Small go indicates that we can achieve similar to the target domain (i.e., small disc(T, St),, estimated
), then the effective sample size is km; if a is one-
small loss on domain St, and from T, T.). We may want to emphasize on S; by setting a; close to 1, but this will reduce the effective sample size. Therefore, we have trade-off between the terms in this bound by choosing a proper a. (4) When S; and 7' are only partially overlapped, it might be difficult to find a suitable ag, In such cases, we can artificially split the given source domains into smaller datasets (e.g., by using clustering) then apply our method on the finer scale. This strategy requires that the learning algorithm has low computational complexities w.r.t. k, the number of source domains. As we will see later in Section 3.3 this is indeed the case for our algorithm.
Before we proceed to develop an algorithm based on the theorem, let us compare Eq. (3) to existing finite-sample bounds. The bound of Blitzer et al.] (2008) Theorem 3) is informative when we have access to a small set of labelled target examples. using this small labelled target set as an additional source domain Sk+41. We can also perform better model selection using such labelled set. The bound of Zhao et al. (2018, Theorem 2) is based on the dy-distance, which is special case of disc in our bound. Moreover, our bound (Eq. (3) use sample- based Rademacher complexity, which is generally tighter than other complexity measures such as VC-dimension (Bartlett& & Mendelson 2002 Koltchinski et al. 2002 Bousquet et al. 2003).
such cases, we can improve our bound by
3

---

Under review as a conference paper at ICLR 2020
3.2 ALGORITHM
In this section, we illustrate how to develop a practical algorithm based on Theorem 2 Ignoring the constants, we would like minimize the upper bound of Eq. 3
min min KE aE
(a(a)) = (Bh, x) + +
(4)
where gs. - I........ gh.kk and 7 is hyper-paramettr. tion exactly given h, then we can treat the optimal o (4) as a function of h and solve the outer minimization over h effectively. In the following, we show how achieve this.
can solve the inner minimiza-
Given gh. the inner minimization can be reformulated as second-order cone programming prob- lem, but has no closed-form solution due the 7||0||2 term. Consider the following problem (z T recovers the inner minimization):
min aE
(2, + ||0//22
(5)
The Lagrangian for its dual problem is
M(a V. = -z a + ||a||2 - X a VERAZO
Taking the derivative w.r.t. a and setting to zero gives
0A
- z a/llal2 aallall - X =1= = a²/1aa" = v1+1 A
da
Notice that a * so we have the constraint following dual problem
||z v1 + A/I2 = 1. Using this ax in gives the
max A
V
s.t. || - vl + A|I2 = and x = 0
We would like to decrease as much as pos- sible, and the best we can attain is the v² sat- isfying ||[z // 111]+112
1 where [v]+ max(0, v). In this case, the optimal A can
be attained as X; 0 if 2, V² > O, other- wise X; = 1/ 2, see Fig. T Although we do not have closed-form expression for the op- timal v*, we can use binary search to find it, starting from the interval [žmin- 1, 2max] where mina Amax are the minimum and maximum of z respectively. Then we can recover the primal solution as
Figure 1: Optimal solution to Eq. 5 (best viewed in color). X compensates what below the v", while 2 is chosen such that the vector of the green bars has L2 norm of I.
a' 12=12--'1)++/l0 -=1]++1++ (6)
Eq. (6) gives rise to new way to project any vector z to the probability simplex, which is of independent interest and may be used in some other contexts It resembles the standard projection onto the simplex based on squared Euclidean distance (Duchi et al. 2008 Eq.(3)). One subtle but crucial difference is that our Eq. 5 uses |||||2 instead a [ Recall that z -ga/T. Here can be interpreted as temperature parameter. On one hand, if >> 0, all z will have similar values and thus the optimal D² will be close to 2max and will be close to uniform. On the other hand, as 0, mmax will stand out from the rest z; and eventually μ/ Emax 1. This means the gh.i corresponding to the zmax is small enough so we focus solely on this domain and ignore all other domains (even though this will reduce effective sample size discussed in Section 3.1.
From a different perspective, if we define Fī(a)) = ||a]|2. then Eq. (5) equals -FA where FA, FA convex conjugates of each other, where we use A to emphasize their dependency on the simplex domain. Then our objective Eq. (4) can be expressed as
min KEW
-Fa(z) - -
For example, if we consider z to be the logits of the final classification layer of neural network, this projection provides another way produce class prohahilities similar to the softmax transformation.
4

---

Under review as a conference paper at ICLR 2020
Layer
x
Forward pass of As;
Usla**
X
disc(T,.p)
Forward pass of As;
Forward pass Xy
Xy
Aggregation using of
disc(T, S,)
Gradient reversal (rs(h))
hiaa
of xs, and øT are fed to the network. Is, will go through the classificaionn/eggrssiin path on the upper by box, while all will go through to the corresponding discrepancies (in the lower ha box). The gradients from the discrepancies will be reverted during backpropagation.
Figure 2: Model architecture with two source domains (best viewed in color).
We can optimize objective and train neural network h using gradient-based optimizer. The optimal a= (h) is merely a function h and we can backprop through it. To facilitate the gradient efficiently backprop through the projection of Eq. (6), derive the Jacobian J = 000000 in Appendix This ensures effective end-to-end training.
computation and show that
Now elaborate more on how to compute Gh,i- The Lg, (ss,) ) is the task loss. The diso(T; S;) depends whether the task is classification regression. For classification, disc coincides with the dis-iitane,, so we use the domain classification loss (Zhao et al. [0188] Ben-David et al. 2007):
2m
disc(f,ãj) S.) = 2111mmm)
1-mm min Cr.s. his
ēr.s. (ha) = 2m Σ
Iha(x) 5xe
where čT,s, (ha) is the sample domain classification loss of a domain classifier ha X + {0, 1}. This minimization over fa will become maximization once move it outside of the disc due to the minus sign. Then our objective consists of min, and max,,- which resembles adversarial train- ing (Goodfellow et al.) 2014): learning task classifier h to minimize loss and domain classifier ha to maximize domain confusion. More specifically, if we decompose the neural network h into feature extractor hisa and label predictor hy (i.e., h(x) hy (2))) classifier ha top of heea to classify heea hz between S, and T as binary classification problem, where the domain itself the label (see Fig. (2. To achieve this, we use the logistic loss approxi- mate ēT.s, (ha) and apply the gradient reversal layer (Ganin & Lempitsky 2015 Ganin et al. 2016) when optimizing disc through backpropagation.
learn a domain-
we are solving regression problems with L2 loss, then (T,, = 1| f - Ms. lla la is the largest eigenvalue (in magnitude) of the difference of two matrices My E, heea (x (T) TTa (z) (TT) and
Ms. 1 E, h1ea(1)) (z. M (a)))) (Mansour et al. 2009a Cortes & Mohri 2011 Sec.5), which can be conveniently approximated using mini-batches and few steps of power iteration.
3.3 COMPLEXITY
Here we analyze the time and space complexities of the algorithm in each iteration. Similar to MDAN (Zhao al. 2018), in each gradient step, need compute the task loss Cg, (h, Js.) and the disc(T, (7, .S)) (or d_ disttannc using mini-batches from each source domain
[kt]. The question
is whether one can maintain the ((k) complexity given that we need to compute the weights using Eq. (6) and backprop through it. For the forward computation of xx, in order to compute the threshold y² to the > 0 relative precision, the binary search will cost O log(1/e)). As for the backward pass of gradient computation, according to our calculation in Appendix B the Jacobian = da/ōz has concise form, meaning that it is possible to compute the matrix-vector product Jv for a given vector v in O(k) time and space. Therefore, our space complexity is the same as MDAN but our time complexity is slightly slower by factor of log(1/e). In comparison, the time
5

---

Under review as a conference paper at ICLR 2020
complexity for MDMN et al. 2018 is O(k²) because it requires computing the pairwise weights within the source domains. When we have lot of source domains, MDMN will be noticeably slower than MDAN and our DARN.
4 RELATED WORK
The idea of utillizing data from the source domain fs) to train model for different but related target domain (T, fr) has been explored extensively for the last decade using different assump- tions Pann & Yang 2009; /Zhan al., 2015). For instance, the covariate shift scenario (Shimodaira)
2000
Gretton et al. 2009 Sugiyama & Kawanabe, 2012 Wen et al. 2014) assumes 5 * but Is J',, while concept drift (Jiang & Zhai 2007 Gama al. 2014) assumes S = 7 but fs # fr- More specifically. Theorem [ (Cortes et al. (2019) shows that both covariate shift (as measured by the discrepancy disc) and model misspecification (as controlled by 77)) contribute to the adaptation
performance (Wen et al. 2014).
Finding a Nomminn-nvvariann feature space by minimizing a distance measure is common practice in domain adaptation, especially for training neural networks. (Tzeng et al. (2017) provided a com- prehensive framework that subsumes several prior efforts on learning shared representations across domains TTzng et al.) 2015 Ganin et al., 2016). DARN uses adversarial domain classifier and the gradient reversal trick from Ganin et al. (2016). Instead of proposing a new loss for each pair of the source and target domains, our main contribution is the aggregation technique of computing the mixing coefficients a, which is derived from theoretical guarantees. When dealing with multiple source domains, our aggregation method can certainly be applied to other forms of discrepancies such as MMD (Gretton et al.] 2012: Long et al., 2013) 2016). and other model architectures such as Domain Separation Network (Bousmalis al. 2016), cycle-consistent model (Hoffman al.) 2018b), class-dependent adversarial domain classifier (Pei et al. 2018) and Known Unknown Dis- crimination (Schoenauer- et al. 2019).
Our work focuses on multi-source to single-target adaptation, which has been investigated in the literature. Sun al. (2011) developed generalization bound but resorted to heuristic algorithms to adjust distribution shifts. Zhao et al. (2018) proposed a certain ad-hoc scheme for the combina- tion coefficients a, which, unlike ours, are not theoretically justified. Multiple Domain Matching Network (MDMN) (Li et al.) 2018)) computes domain similarities not only between the source and target domains but also within the source domain themselves based on Wasserstein-like measure. Calculating such pairwise weights can be computationally demanding when we have a lot of source domains. Their bound requires additional smooth assumptions on the labelling functions fs, fr, and is not finite-sample sample bound, as opposed to ours. As for the actual algorithm, they also use ad- hoe coefficients a without theoretical justification. Mansour et al. (2009b]c) consider multi-source adaptation where T E, S,S, is convex mixture of source distributions with some weights Bi- Our analysis does not require this assumption. Hoffman et al. (2018a) provides similar guarantees with different assumptions, but unlike ours, their bounds are not finite-sample bounds.
5 EXPERIMENTS
In this section, we compare DARN with several other baselines and state-of-the-ale methods for the popular tasks: sentiment analysis, digit recognition and object recognition. The following methods compared. (1) The SRC (for source) method uses only labelled source data train the model. It merges all available source examples to form a large dataset to perform training without adaptation. (2) TAR (for target) is another bascline that uses only m labelled target data instances. serves as upper bound of the best we can achieve if we had access to the true label of the target data. (3) Domain Adversarial Neural Network (DANN) (Ganin et al., (016) similar to our method in that we both use adversarial training objectives. It is not obvious how to adapt DANN to the multi- source setting. Here we follow previous protocol (Zhao et al. 2018) and merge all source data to form large joint source dataset of km instances for DANN to perform adaptation. (4) Moment Matching for Multi-Source Domain Adaptation (M3SDA) Peng et al. 2019) is a recent state-of- the-art method that combines moment matching and maximizing classifier diserepancy (Saito al. 2018). We use their public code with a few necessary adjustments (change classification head based on the number of classes etc.) (5) Multisource Domain Adversarial Network (MDAN) (Zhao al.)

---

Under review as a conference paper at ICLR 2020
2018) resembles method in that we both dynamically assign each source domain importance weight during training. However, unlike ours, their weights are not theoretically justified. We use the soft version of MDAN since they reported that it performs better than the hard version, and we use their code. (6) Multiple Domain Matching Network (MDMN) (Li et al. 2018) computes weights not only between source and target domains but also within source domain themselves. We use their code of computing weights in our implementation. All the methods are applied to the same neural network structure to ensure fair comparison. Our PyTorch implementation will be available online after acceptance.
5.1 SENTIMENT ANALYSIS
Setup. We use the Amazon review dataset (Blitzer et al. 2007: Chen et al.) 2012) that consists of positive and negative product reviews from four domains (Books, DVD, Electronics and Kitchen). Each of them is used in turn as the target domain and the other three are used as source domains. Their sample sizes are 6465, 5586, 7681, 7945 respectively. We follow the common protocol (Chen] et al.] [012]] Zhao et al.) (018)) of using the top-5000 frequent unigrams/bigrams of all reviews as bag-of-words features and train a fully connected model (MLP) with [1000, 500, 100] hidden units for classifying positive versus negative reviews. The dropout drop rate is 0.7 for the input and hidden layers. In each run, we randomly sample 2000 reviews from each domain as labelled source or unlabelled target training examples, while the remaining instances are used as test examples for evaluation. The pyper-paramedte epochs and the mini-batch size is 20 per domain. The optimizer is Adadelta with a learning rate of 1.0. The soft version of MDAN has an additional parameter ? = 1/T which is the inverse of our temperature т. The chosen parameters are 7 10.0 for MDAN and 7 0.9 for our DARN, which are selected from a wide range of candidate values.
are chosen based on cross-validation. The model is trained for 50
Results and Analysis. Table summarizes the classification accuracies. The last column is the average accuracy of four domains, and the standard errors are calculated based on 20 runs (1) Most of the time, DANN joint source data performs worse than SRC which has no adaptation. This may be because it does not adjust for the each source domain, and as a result, it fails to ignore irrelevant data to avoid negative transfer. (2) Some domains are harder to adapt to than the others. For example, the accuracies of SRC and TAR on the Electronics domain are very close to each other, indicating that this requires little to no adaptation. Yet, our DARN the closest to the TAR performance here. The Books domain is more challenging as the improvement over the SRC method is small, even though there exists a large gap between SRC and TAR. (3) DARN is always within the best performing methods and significantly outperforms others in the Electronics domain. Note that MDMN additionally computes similarities within source domains in each iteration, which can be computationally expensive (O(k?) per iteration) if the number of source domains is large. Instead, our method focuses on the discrepancy between source and target domains (O(k) per iteration) and can achieve similar performance this problem.
Table 1: Classification accuracy (%) of the target sentiment datasets. Mean and standard error over 20 runs. The best method(s) (excluding TAR) based on one-sided Wilcoxon signed-rank test at the 5% significance level is(are) shown in bold for each domain.
Method
Books
DVD
Electronics
Kitchen
Avg.
SRC
79.15
0.39
80.38 + 0.30
85.48 4 0.10
85.46 + 0.3
82.62 0.20
DANN
79.13
0.29
80.60
1 029
85.27 4 0.14
85,56 4 028
82.64
:: 0.14
M3SDA
79.42
= 0.17
80.82
+ 0.3
85.52 + 0.39
86.45
+ 0.4
83.05
0.14
MDAN
79.99
0.20
81.66
+ 0.19
84.76 + 0.17
86,82
+ 0.13
83.31
0.08
MDMN
80.13
0.20
+ 0z1 81.57
81.58
85.61
4 0.13
87.13
0.11
83.61
# 0.07
DARN
79.93
0.19
0
85.75 + 0.16
87.15 0.14
83.60
0.08
TAR
84.10
0.13
83.68
0.12
86.11
4 0.30
88.72
1 0.14
85.65 = 0.09
5.2 DIGIT RECOGNITION
Setup. Following the setting from previous work (Ganin et al. 2016 Zhao et al. 2018), we use the four digit recognition datasets in this experiment (MNIST, MNIST-M, SVHN and Synth). MNIST
7

---

Under review as a conference paper at ICLR 2020
is a well-known gray-scale images for digit recognition, and MINST-M (Ganin & Lempitsky 2015) is a variant where the black and white pixels are masked with color patches. Street View House Number (SVHN) (Netzer et al. (01T) is standard digit dataset taken from house numbers in Google Street View images. Synthetic Digits (Synth) (Ganin & Lempitsky 2015) is a synthetic dataset that mimic SVHN using various transformations. One of the four datasets is chosen as unlahelled target domain in turn and the other three are used as labelled source domains.
MNIST images are resized to 32 32 and represented as 3-channel color images in order match the shape of the other three datasets. Each domain has its own given training and test sets when downloaded. Their respective training sample sizes are 60000, 59001, 73257, 479400, and the re- spective test sample sizes are 10000, 9001, 26032, 9553, In each run, 20000 images are randomly sampled from each domain's 's training set as actual labelled source or unlabelled target training ex- amples, and 9000 images are randomly sampled from each domain's 's test set actual test examples for evaluation. The model structure shown in Appendix [ There is no dropout and the hyper- parameters are chosen based on cross-validation. It is trained for 50 epochs and the mini-batch size is 128 per domain. The optimizer is Adadelta with a learning rate 1.0. Validation selected y = 0.5 for MDAN and y 0.1 for DARN.
Results and Analysis. Table 2 shows the classification accuracy of each target dataset over 20 runs. (1) All methods except DANN can consistently outperform SRC. Again, without proper ad- justment for each source domain, DANN with joint source data can perform worse than SRC. This suggests the importance of ignoring irrelevant data to avoid negative transfer. (2) DARN signifi- cantly outperforms MDAN and MDMN across all four domains, especially on the MNIST-M and SVHN domains. Notice that even though MDAN and MDMN have generalization guarantees, they both resort to ad-hoc aggregation rules to combine the source domains during training. Instead, our aggregation (Eq. (6) is direct optimization of the upper bound (Theorem 2 thus is theoretically justified and empirically superior for this problem.
Table 2: Classification accuracy (%) of the target digit datasets. Mean and standard error over 20 runs. The best method (excluding TAR) based on one-sided Wilcoxon signed-rank test at the 5% significance level is shown in bold for each domain.
Method
MNIST
MNIST-M
SVHN
Synth
Avg.
SRC
96.78
0:0
60.80
0.21
68.99 + 0.69
84.09
0.27
77.66 + 0.14
DANN
96.41 4.113
60.10
0.27
70.19 + 1.30
83.83
# 0.25
77.63 0.5
M3SDA
96.95
0:006
65.03
71.66 + 1.16
80.12
# 0.56
78.44 0:026
MDAN
97.10 #0.0
64.09
= 0.31
77.72 + 0.00
85.52
# 0.19
81.11 +
MDMN
97.15 + 0.09
64.34
+ 0.27
76.43 + 0.48
85.80
+ 0.21
80.93
a.16
DARN
98.09 0.00
67.06
=
81.58 0.14
86.79
0.09
83.38 0:006
TAR
99.02 0.00
94.66 = 0.10
87.40 0.17
96.90
= 0.09
94.49 + 0.07
5.3 OBJECT RECOGNITION: OFFICE-HOM HomE
To showcase the applicability of our method to more complicated real-world tasks, we use the chal- lenging Office-Home dataset /Venkateswara et al. 2017). It contains images of 65 everyday objects such as spoon, sink, mug and pen from four different domains: Art, Clipart, Product and Real-World. One of the four datasets is chosen as unlabelled target domain in turn and the other three are used as labelled source domains.
The respective sample sizes are 2427, 4365, 4439, 4357, In each run, 2000 images are randomly sampled from each domain as labelled source or unlabelled target training examples, and the rest images are used as test images for evaluation. We use the ResNet50 He et al. (2016) pretrained from the ImageNet in PyTorch as the base network for feature learning and put MLP with [1000, 500, 100, 65) units on top for classification. It is trained for 50 epochs and the mini- size is 32 per domain. The optimizer is Adadelta with learning rate of 1.0. MDAN uses 7 1.0 while DARN uses = 0.5.
Table 3 shows the classification accuracy of each target dataset over 20 runs. Most existing works in the literature focused on single-source to single-target adaptation on this problem (c.g., see (Long et al. 2018)). Compared them, using multi-source methods can significantly boost performance,
8

---

Under review as a conference paper at ICLR 2020
revealing the importance of using multiple source domains when possible. Even though this is significantly more challenging problem with more classes and much fewer images compared to the digits datasets, our DARN achieves state-of- sfate-of-the-art the-art performance in this setting, excelling existing methods by a noticeable margin.
Table 3: Classification accuracy (%) of the Office- datasets. Mean and standard error over 20 runs. The best method (excluding TAR) based on one-sided Wilcoxon signed-rank test at the 5%
significance level shown in bold for each domain.
Method
Art
Clipart
Product
Real- World
Avg.
SRC
58.02 = 0.47
57.29
0:30
74.26
2022
77.98
0.25
66.89 = 0.16
DANN
57.39
57.35
0:0
73.78
78.12
66.66 0.19
M3SDA
64.05
0.61
62.79
1:037
76,21
0.30
78.63
0.2.
70.42 +: 0.18
MDAN
68.14
= 0.58
67.04
1:021
81.03
0:22
82.79
0.15
74.75 =: 0.18
MDMN
68.67
67.75
0.200
81.37
83.32
+ 0.14
75.28
+ 0.15
DARN
70.00
0.38
68.42
0.3.4
82.75
0.21
83.88
0.16
76.26
0.13
TAR
71.19 0.38
79.16
0.16
90.66
0.15
85.60
+ 0.14
81,65 + 0.12
5.4 VISUALIZING DOMAIN IMPORTANCE
To demonstrate how DARN can effectively aggregate multi- ple source domains, visualize the source domain weights (i.e., a in DARN) for the Amazon dataset. We also com- pared the weights produced by MDMN, using the original authors' code.
ahdben Hemories Dw0
Fig. 3 and Fig. 4 compare the evolution of source domain weights during training. In each subfigure, every row corre- sponds the weights of the source domains when learning for one target domain. The white stripe indicates there is no target domain weight and darker stripe means less weight. They are evaluated at the end of each epoch over 50 epochs. To avoid noisy values due to small mini- size, the val- ues are exponential moving averages with a decay rate of 0.95. (1) For DARN, as Electronics and Kitchen are more related to each other than Books and DVD, their respec- tive weights remain higher during training. This is reason- able since we have overlapping products (e.g.. blenders) in both domains. The a is changing dynamically during train-
Figure 3: Domain weights of DARN for the Amazon data.
indden Aleeoates DND
ing, showing the flexibility of our method to adjust domain importance when needed. (2) In comparison, the domain weights produced by MDMN are not very stable. We take a closer look at the MDMN weights and notice that they change drastically, especially towards the end of the train- ing. It can produce alternating one-hot vectors a, changing from one domain to a different domain and ignoring the rest. This instability makes their domain weights hard to interpret.
Figure 4: Domain weights of MDMN for the Amazon data.
6 CONCLUSION AND FUTURE WORK
Our work uses the discrepancy (Mansour al., 2009a Cortes et al. 2019) derive a finite-sample generalization bound for multi-source to single-target adaptation. Our theorem shows that, in order to achieve the best possible generalization upper bound for a target domain, we need to trade-off be- tween including all source domains to increase effective sample size and removing source domains that are underperforming or not similar to the target domain. Based on this observation, we derive an algorithm, Domain AggRegation Network (DARN), that can dynamically adjust the weight of each source domain during end-to-end training. Experiments on sentiment analysis, digits recog- nition and object recognition show that DARN outperforms oftt-oof-hheart alternatives. Recent

---

Under review as a conference paper at ICLR 2020
analysis (Zhao et al. 2019: Johansson et al. 2019) show that solely focusing learning domain invariant features can be problematic when the marginal label distributions on y between source and target domains are significantly different. Thus it makes sense to take 7H into consideration when small amount of labelled data is available for the target domain, which we will explore in the future.
REFERENCES
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pp. 137-144, 2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning. 19(1-2):151-175, 2010.
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 440-447, 2007.
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, pp. 129-
136, 2008.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pp. 343- 351, 2016.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning the- ory. In Summer School on Machine Learning, pp. 169-207. Springer, 2003.
Minmin Chen, Zhixiang Xu, Kilian Q Weinberger, and Fei Sha. Marginalized denoising autoen- coders for domain adaptation. In Proceedings of the 29th International Coference on Interna- tional Conference on Machine Learning, pp. 1627-1634. Omnipress, 2012.
Corinna Cortes and Mehryar Mohri. Domain adaptation in regression. International Conference Algorithmic Learning Theory, pp. 308 323. Springer, 2011.
Corinna Cortes, Mehryar Mohri, and Andrés Munoz Medina. Adaptation based on generalized discrepancy. The Journal of Machine Learning Research, 20(1):1-30, 2019.
John Duchi, Shai Shalev-Shwarze Yoram Singer, and Tushar Chandra. Efficient projections onto the I 1-ball for learning in high dimensions. In Proceedings of the 25th international conference Machine learning, pp. 272-279. ACM, 2008
João Gama, Indre Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey concept drift adaptation. ACM computing surveys (CSUR), 46(4):44, 2014.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pp. 1180-1189, 2015.
Yaroslav Ganin, Eygeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversann training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-22) 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 6672-2680, 2014.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning. 3(4)55, 2009.
10

---

Under review as a conference paper at ICLR 2020
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 23(Mar):723-775 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorithmss and theory for multiple-sourcs adaptation. In Advances in Neural Information Processing Systems, pp. 8246-8256, 201 80.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pp. 1994-2000 2018b.
Jing Jiang and ChengXiang Zhai. Instance weighting for domain adaptation in NLP. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 264-271, 2007.
Fredrik Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 527-536, 2019.
Daniel Kifer, Shai Den-D, and Johannes Gehrke. Detecting change in data streams. In Proceed- ings of the Thirtieth international conference on Very large data Mooss-VVluue 30, pp. 180-191. VLDB Endowment, 2004.
Vladimir Koltchinskii, Dmitry Panchenko, et al. Empirical margin distributions and bounding the generalization error of combined classifiers. The Annals of Statistics, 30(1):1-50, 2002.
Yitong Li, David E Carlson, al. Extracting relationships by multi-domain matching. In Advances in Neural Information Processing Systems, pp. 6798-6809, 2018.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, pp. 97-105, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, pp. 136- 144, 2016.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In 22nd Conference on Learning Theory, COLT 2009, 2009a.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. In Advances in neural information processing systems, pp. 1041-1048, 2009b.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadch. Multiple source adaptation and the Rényi divergence. In UAI, pp. 367-374, 2009c.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 13455:334-1355 2009.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adap- tation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision pp. 1406-1415, 2019.
11

---

Under review as a conference paper at ICLR 2020
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 3723-3732, 2018.
Alice Schoenauer-Sebag, Louise Heinrich, Mare Schoenauer, Michele Sebag, Lani Wu, and Steve Altschuler. Multi-domain adversarial learning. In International Conference on Learning Repre- sentations, 2019.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning non-stationary environments: Introduction to covariate shift adaptation. MIT press, 2012.
Qian Sun, Rita Chattopadhyay, Sethuraman Panchanathan, and Jieping Ye. A two-stage weighting framework for multi-source domain adaptation. In Advances neural information processing systems, pp. 505-513, 2011.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 14440004777 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In (IEEE) Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions:
Relating covariate shift to model misspecification. In ICML, pp. 631 1113,, 2014.
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In Twenty-ninhh AAAI conference on artificial intelligence, 2015.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In Advances Neural Information Processing Systems, pp. 3559- 2018.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. International Conference on Machine Learning, pp- 2019.
12

---

Under review as a conference paper at ICLR 2020
A PROOF OF THEOREM 2
Theorem Given source domains datasets ((0)) -b})) : [k],j E [m]] with m iid examples
each where S. = {x and yj ;) = Ss. (1)g) ), for any a E A = fox: at 2 0. 0.E. a, = 1),6 E (0, 1), and vh E H. w.p. at least 1 -
(0)
Cr(h, JT) 1 Ees ai (Lg (h,, (s.) + (1s + 29 (Hs:) + mH.s) + Ms log(1/0)
2m
2m
where Hs. to + (s,( h E H. is the set of functions mapping a to the corresponding loss. THIs is a constant similar to Eq. (2) with Q =St, P= T and Ms EEE [(h(z), (x)) is the upper bound on loss on the source domains.
Proof: The proof is similar to Cortes et al. (211 Zhao et al. 2018). Given E A, the mixture S = ay can be considered as the joint source data with km points, where point (x()) from S, has weight a,/m. Define & SuPRER (h, E, o,L3, (h, fs.). Changing point ((() from
Si will change Tb most Mga; Using the McDiarmid's inequality, we have Pr(Ø B[ > S
exp (-
2m²m
m As result, for (0, 1), w.p. at least - the following holds for any h E H
M3lall
log log(1/0)
LT(h, .T) S E anC (h, [s,) + [ + ||a||2
2m
Now we bound E[D)] Let H.s. = fo L(h(z)), : he H).
E [[ = E3
sup Lr Cr (h, fr) - Σ aiLz (h. NEW
< Eg
sup NEW
Σ
Cr fs.) - (h, (h, (h.sss))
+ sup hEH
(rh - E Ch
< Eg
Oj sup hEH
(h, - Lg. (h, fs.))
+
Σ a sup (Cr - Ls. (h. fs.)) hEH
sup LIEH
[s,, L3, (1, Ss:)) (1,
+ Σ
as sup (Lr(h, ST) - Ls. (h,, fs.)) heH
Σ ar Rm (Hs,) Σ
aj sup (Cr(h, (r) - Lss, (h,
kCH
Σ a4 (2 (Hs,) + disc S nH,)).
where first and second inequalities are using the subadditivity of sup, followed by the equality using the independence between the domains {S.), the second last inequality is due the standard "ghost sample" argument in terms of the Rademacher complexity and the last inequality is due to Cortes et al. 2019 Proposition 8) for each individual S:-
13

---

Under review as a conference paper at ICLR 2020
B JACOBIAN
Here we calculate the Jacobian Jız = das/02z for Eq. (6) in the main text:
a" = 2 11 /11z v 1|+
In the following, we write a a² v² v² simplify notations. Let {i 2 - 03 be the support of the probability vector a. Jig = 0 if i & S or j & S since as = 0 in the former case while 2, does not contribute to the in the latter case. Now consider the case i, S. Let
K == 1| =1]++11 Ejes(zy v). Then 0g (2 v) # and
do: = ر:=} Øzj
dv 1
aK K² dzj
1 K
( 6i=;
dv 0K dzj
(z v)
(7)
dzj
K
dzj
By the definition of v, where 6; the indicator or delta function. Now we compute Øzy ōzj we know that
and
22 (e3- - =1 -Sp222v
Σ 2j
+
Σ
2j
=
JES
jES
jES
2
ZjEs2j
VA
V
|||
[S]
where
A
Σ
2j
-
|
Σ jES
2
-
jES
dv
B₁
where
Bj =
Σ
2z'
|S|2
(8)
dzj
[S]
ISIVA
J'ES
The first right-arrow is due to the quadratic formula and realizing that EjeS 25/|S| is the mean of the supported 2j so y must be smaller than it (i.e., we take in the + of the quadratic formula, otherwise some of the 2, will not be in the support anymore). And
aK 0zj
Ov
Bj VA
1-1
Oz,
Plugging Eq. 8 and Eq.
(9) in Eq.
7 gives
do, dzj
1
1
B
1
8
K
S
V
[S]
Note that
Eves 2j' -15/2
Evesley v) + |S|(v -2g) 22)
-1s1a
K Eves(2y v)
Epes(ej - v)
Then
das 1
1
|S|
Jij 02j K
di=j
S
v A
S
03
|S|
aj
In matrix form,
1 Diag(s) J - K
S VA
T
a oss
a o s
S
S
S
where ......., sk]] is vector indicating the support = 85es and is element-wiss multi- plication. More often, we need to compute its multiplication with a vector
v S
S
s
(
JV
+
aos
K
VA
S
Note that all quantities except A have been computed during the forward pass of calculating Eq. (6) can computed O(SS]) time so the overall computation is still O(k) since |S| < k.
14

---

Under review as a conference paper at ICLR 2020
31
38
-33
-15
TatB
TAX
(a) Training data
(b) Test data learned models
(c) a values
Figure 5: Regression experiment (best viewed in color).
C REGRESSION ON SYNTHETIC DATA
Due to the lack of publicly available multi-souree regression datasets, we demonstrate our method on synthetic regression problem here.
Setup. We use eight source domains, where the ith (i {0, 1, ...... 7}) domain data is generated by
~ ~ 7 & 0.22) and the output is y sin(ar) where ~ N 0.052) is random noise.
These eight domains evenly cover the sin function on [- π (see Fig. Sa). Next we construct four
target domains, where the jth (j {0, 1, 2, 2}) domain is generated by 2j N
3x 0.42).
Similar to the source domains, these four target domains evenly cover n, π (see Fig. 5bb. Each source/target domain has 100 data points.
We use labelled source data and unlabelled target data for learning. take on one target domain at time, and learn a linear model from all eight source domains with MSE loss. The goal is to see whether our method can focus on the relevant source domains and learn a linear model that can perform well the target domain.
Results and Analysis. First, Fig. 5b shows the learned models. The learned linear models can fit the target data very well. This shows that DARN learn meaningful model for specific target domain, using only labelled source data and unlabelled target data. Second, Fig. (egssss the source domain weights (the o:) for each target domain after training. The weight colors correspond to the colors in Fig. 5a It is noticeable that DARN can focus well on the respective relevant source domains for each target domain and ignore the rest. Note that these values are automatically learned during the training of the model.
15

---

Under review as a conference paper at ICLR 2020
D
MODEL ARCHITECTURE FOR DIGIT RECOGNITION
4
Conv 3x3x64
MaxPool 2x2
Conv 3x3x1 28
MaxPool 2x2
Conv 3x3x256
MaxPool 2x2
MNIST
FC 2048
FC
Label prediction
1024
10
MNIST-M
FC 2048
FC 2048
FC
251
Domain prediction
FC 2048
FC
SVHN
Domain prediction
2048
88
FC 2048
FC 2048
FC
Domain prediction
Synth
Figure 6: Model architecture for the digit recognition.
16


Reviews:
[{"review_id": "r1lvRFUliS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an aggregation algorithm (DARN) for the multi-source domain adaptation problem which is highly useful in real-world applications.\nThe proposed method is based on the theoretical extension of the single-source domain discrepancy measure proposed by Mansour et al. 2009 to the multi-source setting.\nThis paper also showed the effectiveness of the proposed method on some real-world datasets.\n\nStrengths\nThe paper introduces new technical insights to understand their bound, e.g. effective sample size.\nThe paper proposed the way to estimate coefficient, optimal \\alpha, with theoretical justification, and I think this is the biggest contribution of this paper and is interesting.\nThe proposed method is also able to be used in the regression task since it is based on the disc which can be estimated in the regression task.\n\nWeakness\nThe main theorem of this paper is an extension of existing methods, so the novelty of theoretical analysis is somewhat limited.\nA naive approach to estimate coefficient with single-source domain discrepancy measures such as [1]Mansour (2009), [2,3] Ben-David(2007, 2010), [4] Kuroki et al (2019), and W1-distance is not considered.\nExperimental results itself are fine but not complete.\n  - Although disc can be easily estimated in the regression task (differently from d_A distance which is a special case of disc), there are no experimental results of the regression task even in the synthetic data.\n  - It would be also better to show the coefficient of existing methods that have no theoretical justification.\n  - It would be better to compare with a naive approach that uses domain discrepancy between each source and target as (fixed) coefficient since this approach such as Mansour (2009), Ben-David(2007, 2010) and Kuroki et al (2019) which explicitly consider the hypothesis class has theoretical justification in the form of generalization error bound in the target domain.\n\nOverall, I like the approach of the proposed method, especially tuning coefficient during training procedure although novelty in the theoretical analysis is somewhat limited.\nSo this work has to be supported with more detailed experimental results to express the potential of this approach fully.\nFor this reason, I think it is okay but not good enough at this time.\n\n****After the authors' response****\nIncrease rating.\n\n[1] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and\nalgorithms. In COLT, 2009.\n[2] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for\ndomain adaptation. In NeurIPS, 2007.\n[3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman\nVaughan. A theory of learning from different domains. Machine Learning, 2010.\n[4] Seiichi Kuroki, Nontawat Charoenphakdee, Han Bao, Junya Honda, Issei Sato, and Masashi Sugiyama.\nUnsupervised domain adaptation based on source-guided discrepancy. In AAAI, 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyxp9P_RuH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n###Summary###\nThis paper tackles the multi-source domain adaptation by aggregate multiple source domains dynamically during the training phase. The observation is that in many real-world applications, we want to exploit multiple source datasets of similar tasks to learn a model for a different but related target datasets.\n\nFirstly, the paper derives a multiple-source domain adaptation upper-bound from single-to-single domain adaptation generalization bound, based on the theoretical work from Cortes et al (2019). The idea is similar to Zhao et al (2019), which introduces a weighted parameter \\alpha to combine the source domains together. \n\nSecondly, based on the theoretical result, the paper proposes an algorithm to minimize the upper bound of the theoretical result. The upper bound can be simplified as the quartic form (Eq. 4) and can be optimized with the Lagrangian form. Since no closed-form expression for the optimal v can be derived, the authors propose to use binary search to find it. \n\nBased on the theoretical results and the algorithm, the paper introduces Domain AggRegation Network (DARN), which contains a base network for feature extraction, h_y to minimize the task loss and h_d to evaluate the discrepancy between each source domain and target domain.  The loss is aggregation with the parameter \\alpha.\n\nFinally, the paper conduct experiments on sentimental analysis benchmark, Amazon Review and digit datasets. The paper selects MDAN, DANN, MDMN as the baselines. On the amazon review dataset, the performance of the proposed DARN model is comparable with the MDMN baseline. On the digit dataset, the model can outperform the baselines. \n\n\n### Novelty ###\n\nThe theoretical results in this paper are extended from Cortes et al (2019) and Zhao et al (2018).  Thus, the theoretical contribution of this paper is limited. \n\nThe algorithm proposed in this paper is interesting. However, the motivation of the proposed method is to minimize the upper bound, not the loss itself, i.e. L_T(h, f_T). Intuitively, when the upper bound of the loss is minimized, it will be beneficial to minimize the loss itself. But it's not guaranteed as the upper bound contains other variables, such as the number of training samples and model complexity. If the training samples and model complexity (think about the parameters in the deep models) are significantly large, the upper bound of the loss might be also very large. \n\nAs for the experimental results, the paper only provides results on the sentimental analysis results and digit datasets, which are small benchmarks. The selected baselines are not sufficient. The improvement from the baselines is also limited. \n\n\n\n###Clarity###\n\nOverall, the paper is well organized and logically clear. The images are well-presented and well-explained by the captions and the text. \n\nThe derivation of the algorithm in Sec 3.2 is logically clear and easy to follow. \n\n###Pros###\n\n1) The paper proposes a new theoretical upper-bound based on the prior works, the upper-bound and its derivation are interesting and heuristic to the domain adaptation research community. \n2) The paper is applicable to many practical scenarios since the data from the real-world application is typically collected from multiple sources.\n3) The paper is overall well-organized and well-written. The claims of the paper are verified by the experimental results.\n\n###Cons###\n\n1) The critical issue of this paper is that the algorithm is designed to minimize the upper bound. The idea is intuitive when the upper bound is small. However, the proposed upper bound in the paper involves other parameters, such as the model complexity and the number of training samples. \nIt's an intuitive idea to weight different source domains in multi-source domain adaptation. The paper derives the weight by the Lagrangian form to minimize the upper bound. While another trivial trick is to evaluate \\alpha by the domain closeness between each source domain with the target domain. \n2) The experimental results provided in this paper are weak. In the abstract and introduction,  the paper motivates the multi-source domain adaptation (MSDA) problem by arguing that the MSDA has a lot of real applications.  But the paper only provides empirical results on sentimental analysis and digit recognition.  Besides, the results on the sentimental analysis are comparable with the compared baselines. \n\nIt will be also interesting to see how does the proposed method perform on large-scale datasets such as DomainNet and Office-Home dataset:\nDomainNet: Moment Matching for Multi-Source Domain Adaptation, ICCV 2019. http://ai.bu.edu/DomainNet/\nOffice-Home: Deep Hashing Network for Unsupervised Domain Adaptation, CVPR 2017. http://hemanthdv.org/OfficeHome-Dataset/\n\n3) The novelty of this paper is incremental as the theoretical results are extended from Cortes et al (2019) and Zhao et al (2018). \n\nBased on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs.\nTo improve the rating, the author should explain the following questions:\n1). Why minimizing the upper bound in this scenario would help to minimize the loss on the target domain, when the upper bound can be substantially huge? How about just evaluate \\alpha by the closeness of the source domain with the target domain?\n2). In the introduction, the paper motivates the multi-source domain adaptation (MSDA) problem by arguing that the MSDA has a lot of real applications. While the experiments are only performed on sentimental analysis and digit recognition. How about evaluating the proposed methods on real image recognition such as DomainNet or Office-Home? \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1l3mdg7qS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies multi-source domain adaptation problem. First this paper proposes a new theory for this domain that extends generalized discrepancy theory to multi-source setting. After derive a new generalization bound, this paper also proposes a new method based on the theory. Evaluation on real world datasets are proposed to show the efficiency of the proposed method.\n\n+ The theory in this paper improve bounds for multi-source DA in previous paper. The new bound provides new insight and helps the design of algorithm.\n+ This paper proposes elegant method to tackle new terms in the loss function and gives its complexity analysis. \n+ The evaluation results show that the algorithm is efficient.\n\nComments:\n- The main contribution of the proposed theory is the alpha term. Is \\eta_{\\mathcal{H}} is a better estimation that previous adaptability term in multi-source DA? What is the role of \\eta_{\\mathcal{H}} in the algorithm design? How to control it in empirical algorithm?\n- The evaluation of the proposed method is not complete. Some baseline DA methods [A, B] and datasets [C, D] are not considered. \n\n[A] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. \n[B] Saito, Kuniaki, et al. \"Maximum classifier discrepancy for unsupervised domain adaptation.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n[C] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), 2010. \n[D]H.Venkateswara, J.Eusebio, S.Chakraborty, and S.Panchanathan. Deep hashing network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for the detailed replies to the reviewers.
Their score was slightly improved, this paper is still below the bar given high competition of ICLR2020.
For this reason, we decided not to accept this paper.
Decision: Reject