Multi-EPL: Accurate Multi-source Domain Adaptation
Authors: Seongmin Lee
Hyunsik Jeon
U Kang
Venue:  (2021)
Keywords: Multi-Source Domain Adaptation
Label-wise Moment Matching
Pseudolabel
Ensemble of Feature Representation

Abstract: Given multiple source datasets with labels, how can we train a target model with no labeled data? Multi-source domain adaptation (MSDA) aims to train a model using multiple source datasets different from a target dataset in the absence of target data labels. MSDA is a crucial problem applicable to many practical cases where labels for the target data are unavailable due to privacy issues. Existing MSDA frameworks are limited since they align data without considering conditional distributions p(x|y) of each domain. They also do not fully utilize the target data without labels, and rely on limited feature extraction with a single extractor. In this paper, we propose Multi-EPL, a novel method for multi-source domain adaptation. Multi-EPL exploits label-wise moment matching to align conditional distributions p(x|y), uses pseudolabels for the unavailable target labels, and introduces an ensemble of multiple feature extractors for accurate domain adaptation. Extensive experiments show that Multi-EPL provides the state-of-the-art performance for multi-source domain adaptation tasks in both of image domains and text domains.

Under review as a conference paper at ICLR 2021
MULTI-EPL: ACCURATE MULTI-SOURCE DOMAIN ADAPTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Given multiple source datasets with labels, how can we train target model with no labeled data? Multi-source domain adaptation (MSDA) aims to train a model using multiple source datasets different from target dataset in the absence of target data labels. MSDA is crucial problem applicable to many practical cases where labels for the target data are unavailable due to privacy issues. Existing MSDA frameworks are limited since they align data without considering condi- tional distributions p(x by) of each domain. They also do not fully utilize the target data without labels, and rely limited feature extraction with single extractor. In this paper, we propose MULTI-EPL, a novel method for multi-source domain adaptation. MULTI-EPL. exploits label-wise moment matching align condi- tional distributions p(x (y). uses pseudolabels for the unavailable target labels, and introduces an ensemble of multiple feature extractors for accurate domain adapta- tion. Extensive experiments show that MULTI-EPL provides the Mate-of-the-ar performance for multi-source domain adaptation tasks in both of image domains and text domains.
INTRODUCTION
Given multiple source datasets with labels, how can we train larget model with labeled data? A large training data are essential for training deep neural networks. Collecting abundant data is unfortunately an obstacle in practice; even if enough data are obtained, manually labeling those data is prohibitively expensive. Using other available or much cheaper datasets would be a solution for these limitations; however, indiscriminate usage of other datasets often brings severe generalization error due to the presence of dataset shifts (Torralba & Efros (2011)) 1)). Unsupervised domain adap- tation (UDA) tackles these problems where no labeled data from the target domain are available, but labeled data from other source domains are provided. Finding out domain-invariant features has been the focus of UDA since it allows knowledge transfer from the labeled source dataset to the unlabeled target dataset. There have been many efforts to transfer knowledge from single source domain to a target one. Most recent frameworks minimize the distance between two domains by deep neural networks and distance-based techniques such as discrepancy regularizers (Long et al. (2015; 2016; 2017)), adversarial networks (Ganin et al. (2016); Tzeng et al. (2017)), and generative networks (Liu al. (2017); Zhu et al. (2017); Hoffman al. (2018b)).
While the above-mentioned approaches consider one single source, we address multi-source domain adaptation (MSDA), which is very crucial and more practical in real-world applications as well as more challenging. MSDA is able to bring significant performance enhancement by virtue of ac- cessibility to multiple datasets as long as multiple domain shift problems are resolved. Previous works have extensively presented both thcoretical analysis (Ben-Davie et al. (2010); Mansour et al. (2008); Crammer et al. (2008); Hoffman al. (2018a), Zhao et al. (2018): Zellinger al. (2020)) and models (Zhao et al. (2018): Xu et al. (2018); Peng et al. (2019)) for MSDA. MDAN (Zhao et al. (2018)) and DCTN (Xu et al. (2018)) build adversarial networks for each source domain to generate features domain-invariant enough to confound domain classifiers. However, these approaches do not encompass the shifts among source domains, counting only shifts between source and target domain. SDA (Peng et al. (2019)) adopts moment matching strategy but makes the unrealistic assumption that matching the marginal probability p(x) would guarantee the alignment of the con- ditional probability p( o(ly). Most of these methods also do not fully exploit the knowledge of target

---

Under review as a conference paper at ICLR 2021
domain, imputing to the inaccessibility the labels. Furthermore, all these methods leverage one single feature extractor, which possibly misses important information regarding label classification.
In this paper, propose MULTI-EPL (Multi-souree domain adaptation with Ensemble of feature extractors, Pseudolabels, and Lasel-wise moment matching), a novel MSDA framework which mit- igates the limitations of these methods of not explicitly considering conditional probability p(xly), and relying on only one feature extractor. The model architecture is illustrated in Figure 1. MULTI- EPL aligns the conditional probability p(xly) by utilizing label-wise moment matching. We employ pseudolabels for the inaccessible target labels to maximize the usage of the target data. Moreover, generating an ensemble of features from multiple feature extractors gives abundant information about labels to the extracted features. Extensive experiments show the superiority of our methods.
Our contributions are summarized follows:
Method. We propose MULTI-EPL, a novel approach for MSDA that effectively ob- tains domain-invarian features from multiple domains by matching conditional probability p(x(y), utilizing pseudolabels for inaccessible target labels to fully deploy target data, and using an ensemble of multiple feature extractors. It allows invvriann features to be extracted, capturing the intrinsic differences of different labels.
Analysis. We theoretically prove that minimizing the label-wise moment matching loss relevant to bounding the target error.
Experiments. We conduct extensive experiments on image and text datasets. We show that 1) MULTI-EPL provides the state-of- oftth--artt-ttttte accuracy, and 2) each of our main ideas significantly contributes to the superior performance.
2 RELATED WORK
Single-source Domain Adaptation. Given a labeled source dataset and an unlabeled target dataset, single-source domain adaptation aims to train a model that performs well the target domain. The challenge of single-source domain adaptation is to reduce the discrepancy between the two domains and to obtain appropriate domain- domain-invarian features. Various discrepancy measures such as Maximum Mean Discrepancy (MMD) (Tzeng et al. (2014); Long et al. (2015; 2016; 2017); Ghifary al. (2016)) and KL divergence (Zhuang et al. (2015)) have been used as regularizers. Inspired from the insight that the domain-invarinn
features should exclude the clues about its domain, constructing adversarial networks against domain classifiers has shown superior performance. Liu al. (2017) and Hoffman et al. (2018b) deploy GAN to transform data across the source and target domain, while Ganin al. (2016) and Tzeng et al. (2017) leverage the adversarial networks to extract common features of the two domains. Unlike these works, we focus on multiple source domains.
Multi-source Domain Adaptation. Single-soure domain adaptation should not be naively em- ployed for multiple source domains due to the shifts between source domains. Many previous works have tackled MSDA problems theoretically. Mansour et al. (2008) establish distribution weighted combining rule that the weighted combination of source hypotheses is good approximation for the target hypothesis. The rule is further extended to a stochastic case with joint distribution over the input and the output space in Hoffman al. (201 8a). Crammer et al. (2008) propose the general theory of how to sift appropriate samples out of multi-source data using expected loss. Efforts to find out transferable knowledge from multiple sources from the causal viewpoint et al. (2015). There have been salient studies on the learning bounds for MSDA. Ben-David et al. (2010) found the generalization bounds based on H-v--divvrgeence, which are further tightened by Zhao et al. (2018). Frameworks for MSDA have been presented as well. Zhao et al. (2018) 8) pro- pose learning algorithms based on the generalization bounds for MSDA. DCTN (Xu al. (2018)) resolves domain and category shifts between source and target domains via adversarial networks. M$SDA (Peng al. (2019)) associates all the domains into a common distribution by aligning the moments of the feature distributions of multiple domains. Lin et al. (2020) focus on the visual sen- timent classification tasks and attempts to find out the common latent space source and target domains. Wang et al. (2020) consider the interactions among multiple domains and reflect the infor- mation by constructing knowledge graph. However, all these methods do not consider multimode structures (Pei et al. (2018)) that differently labeled data follow distinct distributions, even if they are drawn from the same domain. Also, the nomain-invarian features in these methods contain the label information for only one label classifier which lead these methods to miss a large amount of
made in Zhang
2

---

Under review as a conference paper at ICLR 2021
Label Classificr fica
Feature Extractor
Duta Distributions
Label Classifier
Extractor Classifier fec
Concatenate
fiofona
Feature
Extractor fe2
Label Classifier
Target Data
Class Label Class Label
Sourcel
Sourec2 Data
Class Label 3
fie.2
MULTI-EPU for n=2. MULTI-EPL consists of pairs of feature extractor and label classifier, one extractor classifier, and one final label classifier. Colors and symbols of the markers indicate domains and class labels of the data, respectively.
Figure 1:
label information. Different from these methods, frameworks fully count the multimodal struc- tures handling the data distributions a label-wise manner and minimize the label information loss considering multiple label classifiers.
Moment Matching, Domain adaptation has deployed the moment matching strategy to minimize the discrepancy between source and target domains. MMD regularizer (Tzeng et al. (2014); Long et al. (2015; 2016; 2017); Ghifary et al. (2016)) can be interpreted as the first-order moment while Sun al. (2016) address second-order moments of source and target distributions. Zellinger al. (2017) investigate the effect of higher-order moment matching. M'SDA (Peng et al. (2019)) demon- strates that moment matching yields remarkable performance also with multiple sources. While previous works have focused matching the moments of marginal distributions for single-source adaptation, we handle conditional distributions in multi-source scenarios.
PROPOSED METHOD
In this section, we describe our proposed method, MULTI-EPL. We first formulate the problem
definition in Section 3.1. Then, describe our main ideas Section 3.2. Section 3.3 elaborates how to match label-wise wise moment with pseudolabels and Section 3.4 extends the approach by adding the concept of ensemble learning. Figure shows the overview of MUL.IIEPPI.
3.1 PROBLEM DEFINITION
Given set of labeled datasets from N source domains Si...., SN and unlabeled dataset from target domain T, we aim to construct model that minimizes test error on T. We formulate source domain S, as a tuple of the data distribution jss, on data space X' and the labeling function (s, S,
ns. (As.:ls.). Source dataset drawn with the distribution Mss. is denoted as Xs. = {(x)) 3, ) Likewise, the target domain and the target dataset are denoted as T = (#T, and X {x] j=1' respectively. narrow our focus down to homogeneous settings in classification tasks: all domains share the same data space X' and label set C.
1=1
3.2 OVERVIEW
We propose MULTI-EPL based on the following observations: 1) existing methods focus aligning the marginal distributions p(x) not the conditional ones p(xly), 2) knowledge of the target data not fully employed target label is given, and 3) there exists large amount of label information loss since domain invariant features are extracted for only one label classifier. Thus, we design MULTI- EPI. aiming solve the limitations. Designing such method entails the following challenges:
3

---

Under review as a conference paper at ICLR 2021
1. Matching conditional distributions. How align the conditional distribution,
p(x(y), of multiple domains not the marginal one, p(x)?
2. Exploitation of the target data. How can fully exploit the knowledge of the target data despite the absence of the target labels?
3. Maximally utilizing feature information. How maximally utilize the information that the domain-inwe -nvariinn features contain?
We propose the following main ideas address the challenges:
1. Label-wise moment matching (Section 3.3). We match the label-wise moments of the domain-invariant features so that the features with the same labels have similar distributions regardless of their original domains.
2. Pseudolabels (Section 3.3). We use pseudolabels as alternatives to the target labels.
3. Ensemble of feature representations (Section 3.4). We learn to extract ensemble of fea- tures from multiple feature extractors, each of which involves distinct ionain-invariant features for its own label classifier.
3.3 LABEL-WISE MOMENT MATCHING WITH PSEUDOLABELS
We describe how MUL.II-PLL matches conditional distributions p(xly) of the features from multi- ple distinct domains. In MULTI-EPL, a feature extractor Se and a label classifier fie lead the features to be domain-invariant and libboriiffrmttive at the same time. The feature extractor fe extracts fea- tures from data, and the label classifier fie receives the features and predicts the labels for the data. We train fe and fic. according to the losses for label-wise moment matching and label classification, which make the features
and label-informative respectively.
Label-wise Moment Matching, To achieve the alignment of label-wise moment matching loss as follows:
features, we define
( +1 Limm, |C| (x2
K ΣΣΣ
Σ fe(P)) f=y
Σ fe(xp') (1) Jivp"=
TDD.C
TDD'
k=I CEC
where K is hyperparameter indicating the maximum order of moments considered by the loss, 'D and D' are two distinct domains amongst the N + domains, and nDpe is the number of data labeled as c Xp. We introduce psendolabels for the target data, which are determined by the outputs of the model currently being trained, to manage the absence of the ground truths for the target data. In other words, we leverage fie(fe(x" (xT)) give the pseudolabel to the target data x". Drawing the pseudolabels using the incomplete model, however, brings mis-labeling issue which impedes further training. To alleviate this problem, we set threshold т and assign the pseudolabels to the target data only when the prediction confidence is greater than the threshold. The target examples with low confidence are not pseudolabeled and not counted in label-wise moment matching.
By minimizing Limm, K. the feature extractor fe aligns data from multiple domains by bringing consistency distributions the features with the same labels. The data with distinct labels are aligned independently, taking account of the multimode structures that differently labeled data fol- low different distributions.
Label Classification. The label classifier fie gets the features projected by fe inputs and makes the label predictions. The label classification loss is defined as follows:
Lie
Lee (fco(fefe(x)))
T&S
where Lee is the softmax cross-entropy loss. Minimizing Lie separates the features with different labels so that each of them gets distinguiinn......l..
3.4 ENSEMBLE OF FEATURE REPRESENTATIONS
In this section, we introduce ensemble learning for further enhancement. Features extracted with the strategies elaborated in the previous section contain the label information for single label classifier. However, each label classifier leverages only limited label characteristics, and thus the conventional scheme adopt only pair of feature extractor and label classifier captures only small part of
4

---

Under review as a conference paper at ICLR 2021
the label information. Our idea is to leverage ensemble of multiple pairs of feature extractor and label classifier in order to make the features to be more label-informative.
We train multiple pairs of feature extractor and label classifier in parallel following the label-wise moment matching approach explained in Section 3.3. Let n denote the num- ber of the feature extractors in the overall model. We denote the (feature extractor, la- bel classifier) pairs as fic,)) (Je,2. fte.2)...., (fe.n fio.a) and the n resultant features as feat1, fealg...., f'eat, f'eat,, where feat; is the output of the feature extractor fe,i- After obtaining n different feature mapping modules, we concatenate the n features into one vector fealfina concat( (fetttffaa. fontt,fece featg,... ..... feat,.). The final label classifier fine takes the concatenated feature as input, and predicts the label of the feature.
Naively exploiting multiple feature extractors, however, does not guarantee the diversity of the fea- tures since it resorts to the randomness. Thus, we introduce a new model component, extractor classifter, which separates the features from different extractors. The extractor classifier fee gets the features generated by feature extractor as inputs and predicts which feature extractor has generated the features. For example, if = 2, the extractor classifier fee attempts to predict whether the input feature is extracted by the extractor fe,1 or Se.2. By training the extractor classifier and multiple feature extractors at once, we explicitly diversify the features obtained from different extractors. We train the extractor classifier utilizing the feature diversifying loss, Lja:
f = N+1
1
Σ
n1;
n
Leo (x)
nT T
Loe (aaJ(x()))) (3)
n.s.
TVT
j=1k=1 k=1
where n is the number of feature extractors.
3.5 MULTI-EPL: ACCURATE MULTI-SOURCE DOMAIN ADAPTATION
Our final model MULTI-EPL consists of pairs of feature extractor and label classifier, (fe,1, fie,). (fe.2 fle.2).... (Se. fic,n), one extractor classifier fee. and one final label classifier finafinal. We first train the entire model except the final label classifier with the loss L:
C=
J=Ecum Lin.k + Limm,K.k K. + 3L
(4)
k=1
where Lie,k is the label classification loss of the classifier fic,k- Limm, K, is the label-wise moment matching loss of the feature extractor Je,k, and a and ,3 are the hyperparameters. Then, the final label classifier is trained with respect to the label classification loss Lic, using the concatenated features from multiple feature extractors.
ANALYSIS
We present theoretical insight regarding the validity of the label-wise moment matching loss. For simplicity, we tackle only binary classification tasks. The error rate of a hypothesis on a domain D is denoted (h) Pr(h(x) + (x()) where (p the labeling function on the domain D. We first introduce k-th order label-wise moment divergence.
Definition 1. Let D and D' be two domains over input space X C. I where is the dimension of the inputs. Let C be the set of the labels, and pe(x) and (x( be the data distribution given u(x|y = c) and (x(xx)
y'(xly c) for the data distribution and pt on the domains 'D and D', respectively. Then, the k-th order label-wise moment divergence (LM, D') of the two domains DD and D' over X is defined
that the label is i.e. Ha(x)
p(c) ( (x(x)) II dx 5=1
72
=
(1((x) II (x
(5)
CEC iEA
where = -- (fI....,
ECIE
k} is the set of the tuples of the nonnegative
integers, which add up to k. p(c) and p'(c) are the probability that arbitrary data fromfrom 'D and 'D' to be labeled as e respectively, and the data x & X is expressed as ......... kn).
The ultimate goal of MSDA is to find a hypothesis h with the minimum target error. We nevertheless train the model with respect the source data since ground truths for the target are unavailable. Let
5

---

Under review as a conference paper at ICLR 2021
N datasets be drawn from N labeled source domains S SN respectively. We denote i-th source dataset Xs, {(x)" 0j }=ر The empirical enror of hypothesis in i-th source domain S; estimated with Xs, is formulated as Es (h)
از ),)))) Given a weight vector a 001,02,.... . aN) such that Σ O; = the weighted empirical source error is formulated as (h) = Ei a,és, (h). We extend the theorems in Ben-David et al. (2010); Peng et al. (2019) and derive bound for the target error (T(h,, for h trained with source data, in terms of k-th order label-wise moment divergence.
Theorem 1. Let H. be hypothesis space of VC dimension d. nts be the number of sam- ples from source domain S., m - ns, be the total number of samples from N source domains SN. and B3 (B1...,,xx) with B n8, Let define hypothesis arg fainne that minimizes the weighted empirical source error, and hypothesis hi arg min (h) that minimizes the true target error. Then, for any ō E (0,1) and O. there exist N integers n me n and N constants On2 ..... On such that
d dEM, 7))
(h(h) s (T(T) + 8m,8,m.55 +
Σ
02i
2) + ane
(6)
1=1
k=
with probability at least - 6, where na =4 4v
/
of)
2 Za(loo(( 3m )+1 +1)+2 log
and
X minnexter(b)) + es, (h)).
Proof. See the Appendix A.I.
Speculating that all datasets are balanced against the amnotations, i.e., p(c) p'()) = c for any
E C. Limm, K is expressed as the of the estimates of dLM.k with k = l,...,. K. The theorem provides an insight that label-wise moment matching allows the model trained with source data to have performance comparable the optimal one the target domain.
EXPERIMENTS
We conduct experiments to answer the following questions of MULTI-EPL.
Q1 Accuracy (Section 5.2). How well does MULTI-EPI perform in classification tasks? Ablation Study (Section 5.3). How much does each component of MULTI-EPL contribute to performance improvement?
Q3 Effects of Degree of Ensemble (Section 5.4). How does the performance change as the number of the pairs of the feature extractor and the label classifier increases?
5.1 EXPERIMENTAL SETTINGS
Datasets. We use three kinds of datasets, Digits-Five, Office-Caltech10- and Amazon Reviews Digits-Five consists five datasets for digit recognition: MNIST (LeCun et al. (1998)), MNIST- M (Ganin & Lempitsky (2015)), SVHNS (Netzer et al. (2011)), SynthDigits (Ganin & Lempitsky (2015)), and USPS? (Hastie et al. (2001)). We set one of them source domains. Following the conventions in prior works (Xu et al. (2018): Peng et al. (2019)), we randomly sample 25000 instances from the source training set and 9000 instances from target training set to train the model except for USPS for which the whole training set is used.
target domain and the rest as
/nttps::/pecplee.esss http:///pwaa.......eeeeeee/ee//a htt.://///..ooooaeae////maaaa//a
"ntpp://yaroslaa.ganii...
httt:://ufldl..ttanooo www.rfordeedd//cccuuuumreee...
"nttp://yaroslaav.gam"
"https://ww.kaggh" com/bi stwumanga wwwpumaann/ussssdattee

---

Under review as a conference paper at ICLR 2021
Table I: Summary of datasets.
Datasets
Features
Labels
Training set
Test set
Properties
MNIST
Ix28x28
10
60000
10000
Grayscale images
MNIST-M
3x32x32
10
59001
9001
RGB images
Digits-Five
SVHN
3x32x32
10
73257
26032
RGB images
SynthDigits
3x32x32
10
479400
9553
RGB images
USPS
1x16x16
10
7291
2007
Grayscale images
Amazon
3x300x300
10
958
958
RGB images
Office-
Caltech
Variable
10
1123
1123
RGB images
Caltech10
DSLR
3x1000
10
157
157
RGB images
Webcam
Variable
10
295
295
RGB images
Books
5000
2 2000
4465
5000-dim vector
Amazon
DVDs
5000
2
2000
3586
5000-dim vector
Reviews
Electronics
5000
2000
5681
5000-dim vector
Kitchen appliances 5000
2
2000
5945
5000-dim vector
Office-Caltechi is the dataset for image classification with 10 categories that Office31 dataset and Caltech dataset have in common. It involves four different domains: Amazon, Caltech, DSLR, and Webcam. We double the number of data by data augmentation and exploit all the original data and augmented data as training data and test data respectively.Amazou Reviews dataset contains customers' reviews 4 product categories: Books, DVDs, Electronics, and Kitchen appliances. The instances are encoded into 5000-dimensional vectors and are labeled as being either positive or target and the rest
negative depending on their sentiments. We each of the four categories as sources. For all the domains, 2000 instances are sampled for training, and the rest of the data are used for the test. Details about the datasets are summarized in Table 1.
Competitors. We use 3 MSDA algorithms, DCTN (Xu et al. (2018)), M'SDA (Peng et al. (2019)), and MDSDA-3 (Peng et al. (2019)), with state-of-the-art performances as baselines. All the frame- works share the same architecture for the feature extractor, the domain classifier, and the label classi- fier for consistency. For Digits- Rigit we use convolutional neural networks based on LeNet5 (LeCun et al. (1998)). For Office-Caltech10, ResNet50 (He et al. (2016)) pretrained on ImageNet is used as the backbone architecture. For Amazon Reviews, the feature extractor is composed of three fully- connected layers each with 1000, 500, and 100 output units, and single fully-connected layer with 100 input units and 2 output units is adopted for both of the extractor and label classifiers. With Digits- Five, LeNet5 (LeCun et al. (1998)) and ResNet14 (He et al. (2016)) without any adaptation are additionally investigated in two different manners: Source Combined and Single Best. Source Combined, multiple source datasets train the model with each source dataset independently, and report the result of the best performing one. Likewise, ResNet50 and MLP consisting of 4 fully-connected layers with 1000, 500, 100, and 2 units are investigated without adaptation for Office- Caltech 10 and Amazon Reviews, respectively.
simply combined and fed into a model. In Single Best,
Training Details. We train our models for Digits-Five -Fve with Adam optimizer (Kingma Ba (2015)) with 31 = 0.9, 8/2 0.999, and the learning rate of 0.0004 for 100 epochs. All images are scaled 32 32 and the mini batch size and K = 2. For the experiments with Office-Caltechl 10, all the modules comprising our model are trained following SGD with the learning rate 0.001, except that the optimizers for feature extractors have the learning rate 0.0001. We scale all the images to 224 224 and set the mini batch size to 48. All the hyperparameters are kept the same as in the experiments with Digits-Fire Five. For Amazon Reviews, we train the models for 50 epochs using Adam optimizer with 31 0.9, 82 0.999, and the learning rate of 0.0001. We set a = 3 = 1, K = 2, and the mini batch size to 100. For every experiment, the confidence threshold is set to 0.9.
128. We set the hyperparameters
0.0005, 8 1,
5.2 PERFORMANCE EVALUATION
We evaluate the performance of MULTI-EPL with n = against the competitors. We repeat ex- periments for each setting five times and report the mean and the standard deviation. The results are summarized in Table 2. Note that MULTI-EPL provides the best accuracy in all the datasets, showing its consistent superiority in both image datasets (Digits-Five, Office-Caltechl dataset (Amazon Reviews). The enhancement is remarkable especially when MNIST-M is the target domain in Digits- Five, improving the accuracy by 11.48% compared to the of-te-of-the-art methods.
and text

---

Under review as a conference paper at ICLR 2021
Table 2: Classification accuracy on Digits-Five, Office-Caltech10, and Amazon Reviews with and without domain adaptation. The letters before and after the slash represent source domains and target domain respectively. In Digits-Five, T. M, S. D, and U stands for MNIST, MNIST-M, SVHN, and Amazon Reviews, indicate each
SynthDigits, and USPS respectively. In domain using the first letter its name. SC and SB indicate Source Combined and Single Best, respectively. Note that MUL.TI-EPL. shows the best performance.
(6) Digits-Five
Method
M+S+D+U/T
T4S+D+UM
T+M+D+US
T+M+S+UD
T+M+S+DUU
Average
(SC)
97.58++0.18
61.7241..38
75.15+0.76
80,29+0.66
81,58+11511
79.27+0.90
ResNetld (SC)
98.22:10.26
63.53:0.84
7..08=1.63
92.85:0048
94.5140.31
85.64:1.70
LeNet5 (SB)
97.09--0.14
51,10+1.88
76.75+0,57
79.92-00.50
83.28++0.92
77.63-00.8
ResNet14 (SB)
97.0741.00
49.48:41.30
81,40=0.70
91.79=0.533
91.5442.68
$2.33+1.225
DCTN
99.28:10.06
71.99:11.58
1.0
91.5510.65
98.43:10.23
87.92:1072
MSSDA
98.75:1:0.05
67.77-10.71
81.75=0.61
88.51110.29
97.17:1.222
86.79=10388
M'SDA 8
98.99-10.00
72.47-40.19
81,40=0,28
-9.51+00.37
97.40+0.11
87.95++0.2
MULTI-EPL (n=2)
99.31-10.04
83.955+0.00
0..93=0.39
93.15-+0.17
98.49:10.08
92.37=00.11
(b) Office Caleehh10
Method
C+D+W/
A+D+WIC
A+C+W/D
A+C+D/W
Average
ResNet50
95.47:1..25
91.5910.51
99.36.10.78
99.26.10.77
96.42.1.0.88
ResNet50 (SB)
95.03=0.448
89.0511.88
99.87-10.28
98.24=0.61
95.55.00.56
DCTN
95.05=0.224
90.60:20.1
100,0=0.00
99.46=0.62
96.28:10.39
SDD
95.14:10.11
93.59110.40
99.49-10.53
99.86-0.119
97.02
M SDA-3
94.36=0.26
91.70=0.71
09.75:10.35
99. 09.39=0.15
96.30:20.37
MULTI-EPL (a=2)
95,74:0.29
93.91
99.87-10.28
9,18
97.35
(c) Amazon Reviows
Method
D+E+K/B
B+E+K/D
B+D+K/E
B+D+E/K
Average
MLP (SC)
79.76=:0.70
82.18:0.59
84.2220277
87.23=0.51
83.40=0.52
MI.P (SB)
79.00-+0.92
80.38+0.61
84.76+0.45
87.46-0.36
82.90-0.55
DCTN
78.92-0.56
81.22- 1.01
83.56 1.52
0.71
82.54--D.95
M'SDA
78.97=0.79
80.51+0.99
83.63-0.688
85.99-0.85
82.27--083
M
80.26-10.43
81.80.10.72
85.02.1.334
86.99-0.56
83.52:00.51
MULTI-EPL (n=2)
81.14-0.29
83.13-+0.45
86.47 0.335
88.53 0.33
84.82--1 0.33
5.3 ABLATION STUDY
We perform an ablation study on Digits-Five to identify what exactly enhances the performance of MULTI-EPL. We compare MULTI-EPL with 3 of its variants: MULTI-O. MULTI-PL, and MULTI- EPL-R. MULTI-O aligns moments regardless of the labels of the data. MULTI-PL trains the model without ensemble learning. MULTI-EPL-R exploits ensemble learning strategy but relies ran- domness without the extractor classifier and the feature diversifying loss.
The results are shown in Table 3. By comparing MULTI-O with MUL.TIPII, observe that sidering labels in moment matching plays a significant role in extracting domain-invariant features. The remarkable performance gap between MULTI-PE PL and MULTI-EPL with n 2 verifies the ef- fectiveness of ensemble learning. Comparing MULTI-EPL and MULTI-EPL-R, MULTI-EPL shows a better performance than MULTI-EPL-R in half of the cases; this means that explicitly diversifying loss often helps further improve the accuracy, while resorting to randomness for feature diversifica- tion also works in general. Hence, we conclude that we are able to apply ensemble learning approach without concern about the redundancy in features.
5.4 EFFECTS OF ENSEMBLE
We vary the number of pairs of feature extractor and label classifier, and repeat the performance evaluation on Digits-Five. The results are summarized in Table While an ensemble of two pairs gives much better performance than the model with single pair, using more than two pairs rarely brings further improvement. This result demonstrates that two pairs of feature extractor and label classifier are able cover most information without losing important label information in Digits- Five. It is notable that increasing n sometimes brings small performance degradation. As more feature extractors adopted to obtain final features, the complexity of final features increases.
8

---

Under review as a conference paper at ICLR 2021
Table 3: Experiments with MULTI-EPL and variants.
Method
T+S+D+U/M M+S+D+U/T
T+M+D+US
T+M+S+U/
T+M+S+D/U
Average
MULTI-O
98.75+0.055
67.77=0.71
81.75=0.61
88.51=0.29
97.17=0.22
86.79=0.38
MULTI-PI.
99.14+0.06
79.32-0.73
84.77+0.39
91.91+0.05
98.49+0.16
90.73-0.28
MULTI-EPL-R (n=2)
99.34:10.5
83.24-0811
86.96.10.4
92.88-00155
98.56.10.17
92.20=
MULTI-EPL (n=2)
99.31110.4
83.95-0.90
86.93.1.39
93.15=0017
98.49.10.08
92.37-10.31
MULTI-EPL (n=3)
99.31
82.78=0.77
87.1020.29
92.85=0.24
98.48200009
92.10=0.27
MULTI-EPL (n=4)
99.3020.07
82.74=0.55
86.65=0.41
92.86=0.15
98.50=0.08
92.01=0.25
harder for the final label classifier to manage the features with high complexity compared to the simple ones. This deteriorates the performance when we exploit more than two feature extractors.
CONCLUSION
We propose MULTI-EPL, a novel framework for the multi-source domain adaptation problem. MULTI-EPL overcomes the problems in existing methods of not directly addressing conditional distributions data p(xly). not fully exploiting knowledge target data, and missing large amount of label information. MULTI-EPL aligns data from multiple source domains and the target domain considering the data labels, and exploits pseudolabels for exploiting unlabeled target data. MULTI- EPL further enhances the performance by generating an ensemble of multiple feature extractors. Our framework exhibits superior performance on both image and text classification tasks. Consid- ering labels in moment matching and adding ensemble learning idea shown to bring remarkable performance enhancement through ablation study. Future works include extending our approach to other tasks such as regression, which may require modification the pseudolabeling method.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Mach. Learn., 79(1-2)1511175, 2010.
Koby Crammer, Michael J. Kearns, and Jennifer Wortman. Learning from multiple sources. JMLR, 2008.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICMI,, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversann training of neural net- works. JMLR, 2016.
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep ceannssrucctioncaassffaaatt networks for unsupervised domain adaptation. In ECCV, 2016.
Trevor Hastie, Jerome H. Friedman, and Robert Tibshirani. The Elements Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statisties. Springer, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. adaptation. In NIPS, 2018a.
Algorithms and theory for multiple-source
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 86(11))2278-2324, 1998.
Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua. Multi-source domain adaptation for visual sentiment classification. In AAAI, 2020.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to- to-image translation networks. In NIPS, 2017.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan, Unsupervised domain adaptation with residual transfer networks. In NIPS, 2016.
Mingsheng Long. Han Zhu, Jianmin Wang. and Michael Jordan. Deep transfer learning with joint adaptation networks. In ICML, 2017.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. NIPS, 2008
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, 2011.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adap- tation. In AAAI, 2018
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019.
Marshall Harvey Stone. Applications of the theory of boolean rings to general topology. Transac- fions of the American Mathematical Society, 1937.
10

---

Under review as a conference paper at ICLR 2021
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR, 2011.
E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. CoRR, abs/1412.3477, 2014.
Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning combine: Knowledge aggregation for multi-source domain adaptation. In ECCV, 2020.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi- source unsupervised domain adaptation with category shift. In CVPR, 2018.
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. Central moment discrepancy (CMD) for domain-invariant representation learn- ing. CoRR, abs/1702.08811, 2017.
Werner Zellinger, Bernhard Alois Moser, and Susanne Saminger-Platz. Learning bounds for moment-based domain adaptation. CoRR, abs/2002.08260, 2020.
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: causal view. In AAAI, 2015.
Han Zhao, Shanghang Zhang, Guanhang Wu, José M. F. Moura, João Paulo Costeira, and Geoffrey J. Gordon, Adversarial multiple source domain adaptation. In NIPS, 2018.
Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired toage-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
Fuzhen Zhuang, Xiaohu Cheng. Ping Luo, Sinno Jialin Pan, and Qing He. Supervised representation learning: Transfer learning with deep autoencoders. In IJCAI, 2015.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 PROOF FOR THEOREM 1
In this section, we prove Theorem 1 in the paper. We first define k-th order label-wise moment divergence dLM,Ks and disagreement ratio ep(h,, h2) of the two hypotheses h1, h2 H on the domain D.
Definition 1. Let D and be two domains over an input space X' C R" where n the dimension of the inputs. Let C. be the set of the labels, and pe(x) and p¹ (x) be the data distributions given that the label is G, i.e. [x(x) = μ(x|y = c) and px((x) = p'(xly = c) for the data distribution and pl the domains DD and D', respectively. Then, the k-th order label-wise moment divergence (LM D') of the two domains D and D' over X is defined as
p(c) [ Ho(x)
72
(D
=
(x,) dx (x(()) II (2,) 1=1
(7)
GECiEA,
where A = (iI....,, E N E و k) integers, which add up to k; p(e) and (c) are the probability that arbitrary data from DD and 'D' to be labeled as respectively, and the data x E X is expressed as (Z1,.... Znn)
3 is the set of the tuples of the nonnegative
Definition 2. Let 'D be domain over input space X C Rn with the data distribution u(x).
Then, we define the disagreement ratio ep ( h2)) of the two hypotheses h1, h2 E H on the domain D as
(2(
Pr (x((x) * h2(x))-
(8)
x~v((x)
Theorem 2. (Stone-Werd Weierstrass Theorem (Stone (1937))) Let K be compact subset of R and f: K + IR be continuous function. Then, for every > 0, there exists polynomial, P: K + R. such that
sup lf(x)) P(x) <c. x
(9)
Theorem 2 indicates that continuous functions on compact subset of K are approximated with polynomials. We next formulate the discrepancy of the two domains using the disagreement ratio and bound with the label-wiee -wse moment divergence.
Lemma 1. Let D and D' be two domains over an input space X R", where is the dimension of the inputs. Then, for any hypotheses h1 h2 E H and any e >> O, there exist ne E N and a constant an, such that
1 an
The (D, +e.
(eD(h,, (h1,
(10)
Proof. Let the domains D and 'D' have the data distribution of μx() and p'(x), respectively, over an input space X, which a compact subset of R.,, where the dimension of the inputs. For brevity, denote JED (h,, h2) eD (h1,
AD,D' Then,
A jeD (11,))2) eD (hi, (eD(hi, h2) - eD (h1,
< sup
sup
Pr x><((x)
[ (x) * h2(x)) xxxp(())
Pr
(he (x) * hz(x))
(11)
(xxx))inn(x))/))))d f (1
sup b.H x For any hypotheses h2 the indicator function (x)x(((() is Lebesgue integrable on X, i.e. 1 (x0 is I function. Since set of continuous functions is dense in (X),, for every
12

---

Under review as a conference paper at ICLR 2021
0, there exists continuous L' function f defined on X' such that
[h, (x)+(((( s(x) S 4
(12)
for every X, and the fixed hn and h2 that drive equation 11 to the supremum. Accordingly,
f(x) S hxx()))/2(xx) f(xx + By integrating every term the inequality over X, the inequality,
(13)
J [(x)f(xxdxx = [((x]1n (x) S $ 1xx) +
(14)
4'
follows. Likewise, the same inequality on the domain DD' with p instead of μ holds. By subtracting the two inequalities and reformulating it, the inequality,
1
p(x)
(x))) (x)
2
(15)
1 ((xx - f pl (x))
S 2
induced. By substituting the inequality in equation 15 the equation 11,
AD.D' $ Jf. f(x)
f (xx) + 2
(16)
By the Theorem 2, there exists polynomial P(x) such that
e sup [f(x) P(x) 4'
(17)
xEX
and the polynomial P(x) is expressed
P(x) Σ
II (z))²,
(18)
1EAp
j=1
E = k}} is the set of the tuples of the nonnegative integers, which add up to k. aj is the coefficient of each term of the polynomial, and x (æ1, Ø2,..... ..), By applying equation 17 to the equation 16 and substituting the expression in equation 18,
where is the order of the polynomial, A₂ = {i= E
[ n (xx))
ADDD P(x))x
+c
x
( ((xx)
II ( dx 1 (1(xx)
ai
II
(xy)
te
LEAN
k=11EAt
j=1
Σ ai x
m(x) II
( dx //m/// ['(x))
'x +e
k=1 i
f=1
| m(x) II
(zz) (x - 1 ((()) II
lail
bx +c
k= 1==1es IEAx
1=1
==1
II (1 - Sr
Joil
Σ p(c)Ma(x)
(x
+
GEC
CEC
(19)
where p(c) and pc(c are the probability that an arbitrary data is labeled as class e in domain 2 and D'. respectively, and (x(x) = p(xly = c) and vx((x) p' d'xxly c) are the data distribution given
13

---

Under review as a conference paper at ICLR 2021
that the data is labeled class domain D and D', respectively. For as. maxiEa. lail.
D
Σ p()))((())
II
x dx
En'(c)a((x)
II (xz) +e
X
k=1
iEAi
j=1
CEC
j=1
ΣV
Σ
p(c) Mc(x) II (ma) p2(c) / (6(x)) II (xy)² +
n
aA
x
k=1
i DEC
1=1
On.
dLM,((D,D)) te,
2
(20)
for an, = 2 maxiskkn, AD.-
Let N datasets be drawn from N labeled source domains S1, S2.... Sy respectively. We denote i-ith source dataset {(x)) 29 }} Xs,. The empirical error of hypothesis h in i-th source domain S; estimated with Xs, is formulated és, (h) ns; En 1 ((x))))yf e.- Given positive and o 2 O, the weighted empirical
weight vector a = 02, AN) such that Ei=1 source error is formulated as (h) = Eie (h).
Lemma 2. For N source domains S1, S2,S,,..., SN. let n.s be the number of samples from source domain Si.
Ei= nsg be the total number of samples from N source domains, and f (ii, B2..... By) with B. ns, Let t (h)) be the weighted true source error which is the weighted sum of (h) = Pix~p(x) [h(x) # y). Then,
m
2mm?
Pr ([( - Ea (h)) 2 c] 5
(21)
a
i= Br
Proof. It has been proven in Ben-David et al. (2010).
We now turn our focus back to the Theorem in the paper and complete the proof. Theorem Let H be hypothesis space of VC dimension d, ns, be the number of sam- ples from source domain S, m = E nss be the total number of samples from N source domains SI,...,Sy. and 3 = (81,...,3x) with Bi = "s, Let us define hypothesis h arg min that minimizes the weighted empirical source error, and hypothesis h = arg min (h) that minimizes the true target error. Then, for any 8 E (0.1) and > O, there exist N integers n....., nn and N constants an):
an such that
eT(h) s ( + 3mm +
Σ
aj 1=1
21 + and Σ deM.C(S,,)))
(22)
h=1
with probability at least 8, where na.amm = 41/
a?
2d[(oo(( 2g +1)++)+22
lop (
and
3.
m
X; minnew H (1)(h) + + (h))
n
Proof.
lea (1) - - (1)) = Σ auss, - - 5
Σ as les, (h) eT (h))-
(23)
1=1
We define Ag = arg min es, (h) + (h) for every = 1, 1,....., N for the following equations. MEH
We also note that the 1-triangular inequality (Crammer al. (2008)) holds for binary classification
14

---

Under review as a conference paper at ICLR 2021
tasks, i.e, ep(h1, h2) I eD cD ha) cp h2) for any hypothesis h1, h2, hy H. and domain D. Then,
cp(, Ecllo, H²) = c (h)(h) (
(24)
for the ground truth labeling function lp on the domain DD and two hypotheses h, h E H. Applying the definition and the inequality to equation 23,
(h) cT(h) S
(lcs, (h) -
==1
O4 (es, (hi) + ks, h,) cT (0)
(25)
By the definition of h, es, (h;) cT(h;) = X for A; minNeH (er(h) + cs, (h)). Additionally, according to Lemma 1, for any > 0, there exists an integer ne and a constant anz such that
css (h, - cT (h,, 12) & Ont
A 2
(26)
1=1
By applying these relations,
N
(Ea(h) ET(h))
Xi + ang
(L,
i=1
k=
(27)
N
A
+ Ong
(S, 7)
2
i=1
k=1
By Lemma 2 and the standard uniform convergence bound for hypothesis classes of finite VC di- mension (Ben-David et al. (2010)),
N
(T(h) S ca (h) +
+
Σ
Oi
Ai
+
an!
dIMM,(SST k=1
2
N
S ( + B.
+
Σ
ai
Ai
+
- Ong
1
Σ
dI..
2
iml
k=
N
S (la( + タ Am
+
aii
Ai +
Ont
dEM.A(S;.))
(28)
2
2
1=1
k=1
N
< (((itt + n...... +
+
\; +
ant
(S.,
2
2
1=1
k=1
N
n
(T( + 3m, +
Σ
a6
2 + a
dLM.A(S,,T)
1=1
k=
The last inequality holds by equation 27 with h h4.
15


Reviews:
[{"review_id": "zDtV_sSUpq1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary and contributions\n    - In this work, the authors proposed an algorithm for multi-source domain adaptation. While the results seem promising, the technical contribution is incremental and limited. Meanwhile, more empirical results are needed to validate the effectiveness of the framework. \n- Strengths:\n    - The paper is well written and easy to follow.\n    - The problem investigated in this paper, i.e., multi-source domain adaptation, is of significance.\n- Weaknesses:\n    - The technical contribution of this work is limited. Label-wise moment matching or adversarial training (e.g., [1]) has been a common practice in single-source domain adaptation. The authors simply applies this idea of multi-mode aware domain adaptation to multi-source domain adaptation. Moreover, comparing the first line, i.e., MULTI-0, in Table 2 with M3SDA in Table 1(a), we find that this label-wise moment-matching makes almost no contribution.\n    - The empirical results, especially the ablation studies, do not hold. Pseudo-labeling the unlabeled data in the target domain and using multiple feature extractors can also be easily used by M3SDA and DCTN. Expecting a definite performance boost, I expect the authors to provide such results. Moreover, pseudo-labeling and ensemble learning, however, are not novel; they are widely adopted techniques and can be easily incorporated into any algorithm, e.g., M3SDA, for performance improvement. \n    - Even on the dataset Office-Caltech and Amazon Reviews, the performance improvement of the proposed algorithm is minor. \n\t\t\n\t\t\n[1] Pei, Z., Cao, Z., Long, M., & Wang, J. (2018). Multi-adversarial domain adaptation. AAAI 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dahCP3_Z75s", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--Paper summary--\n\nThe authors propose a novel method for multi-source domain adaptation (MSDA). For effective adaptation, the proposed method adopts three techniques: (1) label-wise moment matching, (2) pseudo-labeling target data, and (3) ensembling multiple feature extractors. Experimental results show that the proposed method outperforms several state-of-the-art methods in both image and text domains.\n\n--Review summary--\n\nAlthough the design of the proposed method seems reasonable, its novelty is marginal. Additionally, the evaluation in the experiments is somewhat unfair. I vote for rejection.\n\n--Details--\n\nStrength\n\n- This paper is well-organized and is easy to follow. I believe that the proposed method can be easily implemented without any obstacles.\n- Good performance in both image and text domains is appealing. Such results should be highly appreciated especially in machine learning community.\n\nWeakness and concerns\n\n- Marginal novelty. The three techniques that the proposed method adopted are all similar to those already proposed in the literature. I could not find any novel and specific design or strategy to combine them specialized for MSDA.\n\t- Class-wise distributional alignment is a common technique in recent domain adaptation methods, e.g. [R1] and [R2].   \n[R1] \"A DIRT-T Approach to Unsupervised Domain Adaptation,\" ICLR 2018.  \n[R2] \"Unsupervised Domain Adaptation via Regularized Conditional Alignment,\" ICCV 2019.\n\t- Pseudo-labeling is also a common technique in recent domain adaptation methods, e.g. [R1] and [R3].  \n[R3] \"Asymmetric Tri-training for Unsupervised Domain Adaptation,\" ICML 2017.\n\t- Using multiple feature representations is not so common but is presented in [R4] and [R5].  \n[R4] \"Domain Adaptation with Ensemble of Feature Groups,\" IJCAI 2011.  \n[R5] \"Domain Separation Networks,\" NeurIPS 2016.\n- The design of the feature diversifying loss is not reasonable. It can be minimized by just making feature representations to be easy to discrminate their extractors, which does not necessarily increase the diversity of the representations. For example, given two extractors that share the same parameters, adding a large offset to outputs from one extractor leads to high performance of the extractor classifier but does not increase diversity of the feature representations. \n- The exprimental setting is somewhat unfair. Since the proposed method utilizes n feature extractors, the model complexity in the proposed method should be n times larger than that in existing methods. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qxc6YB9xX0r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the multi-source domain adaptation (MSDA) problem. The authors argue that the existing MSDA solutions (1) do not explicitly consider distribution conditioned on labels of each domain, (2) rely on limited feature extraction based on one extractor, (3) do not well explore target data due to the absence of label. Correspondingly, Multi-EPL is proposed based on moment matching.  \n\nAlthough the design of the proposed method seems reasonable, its novelty is marginal. Additionally, the evaluation in the experiments is somewhat unfair. I vote for rejection.\n\nThe paper is well organized. The technical details are clearly presented. A few comments are summarized below.\n-In Eq.1, the authors minimize the discrepancy between every two distinct domains. It is unclear to me why not minimize the pairs of each source domain and target domain, (N-pairs). The goal is to align the distributions between source and target. Please clarify the motivation of aligning two source domains. \n\nBesides, it would be good to have one baseline only considering label-wise moment matching losses for only between N source and 1 target pairs.\n\n-In page 5, the motivation of diversifying features from different extractors is unclear to me. Please clarify the benefit of classifying feature according to extractor ID labels. Moreover, the ablation study presented in section 5.3, does not show a clear improvement by introducing the diversifying loss. I would encourage the authors to design another analytical experiment to show its effectiveness.\n\n-The performance improvement compared to the state of the arts is limited. Specifically, for Digit-Five dataset, a missing recent work [ref-1] reports an average performance of 91.8. To show the consistent performance improvement over this strong baseline, I’d encourage the authors cite and compare it under the same setting.\n[ref-1] Hang Wang et al., Learning to Combine: Knowledge Aggregation for Multi-Source Domain Adaptation, ECCV 2020\n\n-Besides [ref-1], there are several other recent MSDA papers are missing, including but not limited to\n[ref-2] Chuang Lin et al., Multi-source Domain Adaptation for Visual Sentiment Classification, https://arxiv.org/abs/2001.03886 \n[ref-3] Haotian Wang et al., Tmda: Task-specific multi-source domain adaptation via clustering embedded adversarial training. ICDM 2019.\n\n-Other minor point. A typo in page 7, “threshoold” -> “threshold”\n\nUpdates: Thanks for the authors' response. Some of my queries (1st and 3rd) were clarified. However, unfortunately, I still think more needs to be done to show the superiority of the results. I retain my original decision.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-v-k12R4v0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a MULTI-EPL for multi-source domain adaptation. The key idea includes two folds: (1) to align label-wise moment, and (2) to ensemble multiple feature extractor. Experimental studies on 3 datasets are done to verify the proposed MUTL-EPL.\n\nOverall, the paper is well-written. The technical approach is simple and sound. My major concern is on the technical significance of the method. Here are the detailed comments:\n\n(1)\tOne motivation of the paper is that current methods fail to consider the shifts among sources. However, there are some multiple source transfer methods explicitly modelling the inter-domain similarities, e.g., [ref1].  The paper also misses some important multiple-source references. Please refer to the survey [ref2] for different types of multiple source transfer methods. It would be better to have a comprehensive discussion on the related works.\n\n(2)\tThe proposed label-wise moment matching is not new in transfer. Early subspace based work, e.g., JDA [ref3], and latest semantic deep learning based transfer methods, e.g., [ref4] share the similar idea. \n\n(3)\tThe threshold \\taw is used to obtain the good target labels.  On the one hand, it is unclear how this parameter should be set for different transfer tasks. On the other hand, high confidence score does not imply correct target label prediction. Error reinforcement may happen even with a well-tuned \\taw.\n\n(4)\tThe usage of ensemble feature extractor is actually using high complexity to enhance prediction accuracy. The scalability could be an issue of the proposed method, especially considering that multiple source may have extremely large data size. \n\n(5)\tSensitivity analyses on the balancing parameters \\alpha and \\beta should be done. \n\n(6)\tWhy data augmentation is done on office-caltech10 datasets? There are many datasets containing multiple domains with sufficient data, e.g., office-home, Please use these datasets instead of constructing `artificial’ real-world dataset. \n\n(7)\tThe baseline methods can be further improved (please consider [ref2] for more baselines). Regarding the current results, MUTLT-EPL performs larger improvements over M3SDA on the 2nd, 3rd, and 4th tasks in Digit –five dataset, while only achieves marginal improvements in other tasks, e.g.. tasks in office-caltech10 dataset. More analyses on the performance difference of different tasks should be discussed. \n\n(8)\tBased on the ablation study, Mutil-epl-r achieves comparable results with multi-epl, which indicates that the extractor classifier and feature diversifying loss have less importance in the overall objective. \n\n[ref1] Source-target similarity modelings for multi-source transfer gaussian process regression\n\n[ref2] Multi-source Domain Adaptation in the Deep Learning Era: A Systematic Survey\n\n[ref3] Transfer Feature Learning with Joint Distribution Adaptation\n\n[ref4] Deep Transfer Learning with Joint Adaptation Networks\n\nUpdate: Thanks for the authors' response. However, I am not convinced on several points, e.g., (3) - (7). Considering the other reviewers' comments, I think the paper needs to be further improved. Thus, I will keep my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed a solution to the problem of multi-source domain adaptation. All the reviewers have two concerns: 1) the technical contribution/novelty is limited, and 2) the experimental results are not convincing. Therefore, this paper does not meet the standard of being published in ICLR. The authors are encouraged to improve this work by addressing the issues raised by the reviewers. 
Decision: Reject