Unified Principles For Multi-Source Transfer Learning Under Label Shifts
Authors: changjian shui
Zijian Li
jiaqi li
Christian Gagné
Charles Ling
Boyu Wang
Venue:  (2021)

Abstract: We study the label shift problem in multi-source transfer learning and derive new generic principles. Our proposed framework unifies the principles of conditional feature alignment, label distribution ratio estimation, and domain relation weights estimation. Based on inspired practical principles, we provide a unified practical framework for three multi-source label shift transfer scenarios: learning with limited target data, unsupervised domain adaptation, and label partial unsupervised domain adaptation. We evaluate the proposed method on these scenarios by extensive experiments and show that our proposed algorithm can significantly outperform the baselines.   

Under review as a conference paper at ICLR 2021
UNIFIED PRINCIPLES FOR MULTI-SOURCE TRANSFER LEARNING UNDER LABEL SHIFTS
Anonymous authors Paper under double-blind review
ABSTRACT
We study the label shift problem in multi-source transfer learning and derive new generie principles. Our proposed framework unifies the principles of conditional feature alignment, label distribution ratio estimation and domain relation weights estimation. Based on inspired practical principles, we provide unified practical framework for three multi-source label shift transfer scenarios: learning with lim- ited target data, unsupervised domain adaptation and label partial unsupervised domain adaptation. We evaluate the proposed method on these scenarios by ex- tensive experiments and show that our proposed algorithm can significantly out- perform the baselines.
INTRODUCTION
Transfer learning (Pan & Yang 2009) is based on the motivation that learning a new task is easier after having learned several similar tasks. By learning the inductive bias from of related source domains (S).... Sr) and then leveraging the shared knowledge upon learning the target domain T, the prediction performance can be significantly improved. Based on this, transfer learning arises in deep learning applications such as computer vision (Zhang et al. 2019 Tann et al. 2018 Hoffman jet al., 2018b), natural language processing (Ruder al. 2019 Hou sby et al., 2019) and biomedical engineering (Raghu et al. 2019 Lundervold & Lundervold 2019) Zhang & An 2017).
To ensure reliable transfer, it is critical understand the theoretical assumptions between the domains. One implicit assumption in most transfer learning algorithms is that the label proportions remain unchanged across different domains (Du Plessis Sugiyama, 2014) (i.e., S(y) T(y)-- However, in many real-world applications, the label distributions can vary markedly (i.c. label
shift) (Wen et al. 2014: Lipton et al., 2018 Li et al. 2019b), in which existing approaches cannot guarantee small target generalization error, which recently proved by Combes al. (2020).
Moreover, transfer learning becomes more challenging when transferring knowledge from multiple sources to build model for the target domain, this requires an effective selection and leveraging the most useful source domains when label shift occurs. This is not only theoretically interesting but also commonly encountered in real-world applications. For example, in medical diagnostics, the disease distribution changes over countries (Liu et al.) 22044 Geiss et al. [2014). Considering the task of diagnosing a discase country without sufficient data, how can we leverage the information from different countries with abundant data to help the diagnosing? Obviously, naïvely combining all the sources and applying one-to-one single source transfer learning algorithm can lead to unde- sired results, as it can include low quality or even untrusted data from certain sources, which can severely influence the performance.
In this paper, study the label shift problem in multi-source transfer learning where Sz(y) * T(y). We propose unified principles that are applicable for three common transfer scenarios: unsupervised Domain Adaptation (DA) (Ben-Davie partial unsupervised DA with supp (y)) C (v)) (Cao et al,, 2018), where prior works generally treated them separate scenario. It should be noted that this work deals with target shift without assuming that semantic conditional distributions are identical (i.e., Se(aly) + T(xly)) which is more realistic for worl-world problems. Our contributions in this paper two-folds:
al. 2010), limited target labels (Mansour al. (0200) and
(I) We propose to use Wasserstein distance (Arjovsky et al.) 2017) to develop a new target gen- eralization risk upper bound (Theorem T). which reveals the importance of label distribution ratio

---

Under review as a conference paper at ICLR 2021
estimation and provides a principled guideline to learn the domain relation coefficients. Moreover, we provide a theoretical analysis in the context of representation learning (Theorem22) 2 which guides to learn a feature function that minimizes the conditional Wasserstein distance well controls the weighted source risk. We further reveal the relations in the aforementioned three scenarios lie in the different assumptions for estimating label distribution ratio.
(II) Inspired by the theoretical results, we propose Wasserstein Aggregation Domain Network (WADN) for handling label-shift in multi-source transfer learning. We evaluate our algorithm on three benchmark datasets, and the results show that our algorithm can significantly outperform state- of-the-art principled approaches.
RELATED WORK
Multi-Source Transfer Learning Theories have been investigated in the previous literature with
different principles to aggregate source domains. the popular unsupervised DA, (Zhao et al. 2018
Peng et al., [219)) Wen et al. 2020 Li et al. 2018b) adopted H-divergence (Ben- David al. 2007),
discrepancy (Mansour et al. 2009) and Wasserstein distance (Arjovsky et al. 20117) of marginal dis-
tribution di (z), (
to estimate domain relations and dynamically leveraged different domains.
These algorithms generally consists source risk, domain discrepancy and an un-observable term 1), the optimal risk all the domains, which are ignored in these approaches. However, [ombees iet al. (2020) pointed out, ignoring the influence of 1) will be problematic when label distributions between source and target domains are significantly different. Therefore it is necessary to take v) into consideration by using a small amount of labelled data is available for the target domain (Wen et al., 2020). Following this line, very recent works (Konstantinov & Lampert 2019; Wang et al., 20199 Mansour et al.] (0200) started to consider measure the divergence between two domains given label information for the target domain by using J'-discrepancy (Mohri & Medinal 2012). However, we empirically showed these methods are still unable to handle label shift.
Label-Shift Label-Shift (Zhang et al., 2013 Gong et al., 2016) is a common phenomena in the transfer learning with S(y) + 7 (v) and generally ignored by the previous multi-source transfer learning practice. Several thcoretical principled approaches have been proposed such as (Azizzade- nesheli al. 2019; Garg et al.) 2020). In addition, (Combes et al. 20200 Wu et al. 2019) analyzed the generalized label shift problem in the one-to-one single unsupervised DA problem but did not provide guidelines of levering different sources to ensure reliable transfer, which is more chal- lenging. (Redko et al., (2119) proposed optimal transport strategy for the multiple unsupervised DA under label shift by assuming identical semantic conditional distribution. However they did not con- sider representation learning in conjunction with their framework and did not design neural network based approaches. Different from these. we analyzed our problem in the context of representation learning and propose an efficient and principled strategies. Moreover, our theoretical results high- lights the importance of label shift problem in a variety of multi-source transfer problem. While the aforementioned work generally focus on the unsupervised DA problem, without considering unified rules for different scenarios (e.g. partial multi-source DA).
THEORETICAL INSIGHTS: TRANSFER RISK UPPER BOUND
We assume scoring hypothesis defined on the input space X' and output space y with h: X x
R. is K-Lipschitz w.r.t. the feature æ (given the same label), i.e. for Vy, [1(21, y) - h( l2 S
K|| 22 and the loss function É: R x R R.,, is positive, L-Lipschitz and upper bounded by Lmax. We denote the expected risk w.r.t distribution D: RD = ((ew)> v)) and its
empirical counterpart (w.r.. 2) (h) = ((ew)e ((((z, w)). We adopted Wasserstein-1 distance (Arjovsky et al. (017)) as a metric to measure the similarity of the domains. Compared with other divergences, Wasserstein distance has been theoretically proved tighter than TV distance (Gong et al. 2016) or Jensen- Shaeennnn divergence (Combes et al. 2020).
Based on previous work, the label shift is generally handled by label-distribution ratio weighted loss: (h) =
y)) with atv) = T(y) We also denote âe as its empirical counterpart, estimated from samples. Besides, to measure the task relations, we define simplex A with A|1] 2 0, Er A(] 1 as the task relation coefficient vector by assigning high weight to
2

---

Under review as a conference paper at ICLR 2021
the similar task. Then we first present Theorem 1, which proposed theoretical insights about how to combine source domains through properly estimating À.
Theorem 1. Let {S. = ((x, ve) Na yr target i..dd. samples. For vh H with H the hypothesis family and vX, with high probability 2 1 - 45, the target risk can be upper bounded by:
and i = b respectively be T source and
(y(h) = Σ 211 LK Σ E (T(T( WIISS 8) + Lumad deup
X
log(1/5)
2N
Lemax sup ||a, âe Comp(Ns: ...... No.
where N = Ei Ns. and Bt = /Ns/NN and das = max a+(y) the maximum true label distribution ratio value. (-ll)) is the Wasserstein-1 I distance with Ly-distance as cost func- tion. Comp( Ns....., ,,,, NT. is a function that decreases with larger N NN, given fixed 6 and hypothesis family H. (See Appendix Eforr details)
Remarks (1) In the first two terms, the relation coefficient À is controlled by a--weighted loss R3g (h) and conditional Wasserstein distance By-r(r) W1 (T(T = -)]l = v)). To min- imize the upper bound, need to assign higher A([] to the source & with smaller weighted prediction loss and a smaller weighted semantic conditional Wasserstein distance. Intuitively, we tend leverage the source task which is semantic similar to the target and easier to learn. (2) If each source have equal observations with 3 = 1, then the third term will become |||||,, L2 norm regularization, which can be viewed as an encouragement of uniformly leveraging all the sources. Combing these three terms, we need to consider the trade-off between assigning higher X[[] to the source that has a smaller weighted prediction loss and conditional Wasserstein distance, and assigning balanced X[[] for avoiding concentrating the gap between ground-trutt touut and empirical label ratio. Therefore if we can estimate a good å these terms can be small. In the practice, If target labels are available, âg can be computed from the observed data and & - ag-- If target labels are absent (unsupervised DA), we need to design methods and to properly estimate âg (Sec. 4p. (4) Comp (No, ... Nsy. Ny, 5) is function that reflects the convergence behavior, which decreases with larger observation numbers. If we fix H, 6, N and Nr, this term can be viewed constant.
only one source. (3) |å, ag lll indicates
Insights in Representation Learning Apart from Theorem D we propose novel theoretical analysis in the context of representation learning, which motivates practical guidelines in the deep learning regime. We define a stochastic feature function g and we denote its conditional distribution y)dz. Then we have:
w.r.t. latent variable Z (induced by g) as S(z|Y y) =S2 g(2l=)
Theorem 2.
We assume the settings of loss. the hypothesis are the same with Theorem [ We further denote the stochastic feature learning function g + Z, and the hypothesis h x y + R. Then V\, the target risk is upper bounded by:
gh, S (1) LK Σ
- =)),
where RT (h,g) = B(z,o))T(aa)) Ba (11(2, v))-
Theorem 2 reveal that to control the upper bound, we need to learn g that minimizes the weighted conditional Wasserstein distance and learn (g, h) that minimizes the weighted source risk.
Comparison with previous Theorems. Our theory proposed an alternative prospective to under- stand transfer learning. The first term is w-weigtted loss. And it will recover the typical source loss minimization there is no label shift with av(y)) = (Li et al. 2019a Peng et al. 2019 Zhao et al.,
2018 Wen al., 2020). Beside, minimizing the conditional Wasserstein distances has been shown to be advantageous, compared with W1(S (=) (Long et al. 2018 Moreover, Theorem ex- plicitly proposed the theoretical insights about the representation learning function g. which remains elusive for previous multi sourc- transfer theories such as (Wang et al. 2019a Mansour et al. 2020
Konstantinov & Lamperi 2019
Li et al. 2019a Peng et al.) 2019).
3

---

Under review as a conference paper at ICLR 2021
Table 1: Practical principles under different scenarios
Transfer Scenarios
Learn (g. h)
Estimate de
Estimate
Multi-Source Transfer with Limited Target Data
(v))/S(v)
Sec. 4.1
Sec. 4.3
Unsupervised Multi-Source DA
Sec. 4.2
Partial Unsupervised Multi-Source DA
UNIFIED PRACTICAL FRAMEWORK IN DEEP LEARNING
The theoretical results in Section motivate general principles follow when designing multi- source transfer learning algorithms. We summarize those principles in the following rules.
(I) Learn a 9 that minimizes the weighted conditional Wasserstein distance as well as learn (g. h) that minimizes the we-whigtted source risk (Sec. 4.1).
(II) Properly estimate the label distribution ratio Oe (Sec. 4.2).
(III) Balance the trade-off between assigning a higher A(t] to the source that has a smaller weighted prediction loss and conditional Wasserstein distance, and assigning balanced A(t). (Sec. 4.3).
We instantiate these rules with a unified practical framework for solving multi-source transfer learn- ing problems, as shown in Tab [ We would like point out that our original theoretical result is based on setting with the available target labels. The proposed algorithm can be applied to unsuper- vised scenarios under additional assumptions.
4.1 GUIDELINES IN THE REPRESENTATION LEARNING
Motivated by Theorem 2, given fixed label ratio estimation á, and fixed X, we should find representation function g : X + Z and a hypothesis function h: Z y - R such that:
min 9.
(h, g) Co Σ> y W = W
(1)
Explicit Conditional Loss When target label information is available, one can explicitly solve the conditional optimal transport problem with g and for a given Y y. However, due to the high computational complexity in solving T x [ optimal transport problems, the original form is practically intractable. To address this issue, we propose to approximate the conditional distribution on latent space Z as Gaussian distribution with identical Covariance matrix such that Sx(z = v) R (C Σ) and Ý(=lY = y) & AC(,,,)) >). Then we have W W1 (S = x) = = < ICy Ca 2l (see Appendix G for details). Intuitively, the approximation term is equivalent the well known feature mean matching (Sugiyama & Kawanabe 2012), which computes the feature centroid of each class (on latent space Z) and aligns them by minimizing their L.2 distance.
Implicit Conditional Loss When target label information is not available (e.g. unsupervised DA and partial DA), the explicit matching approach can adopt pseudo-label predicted by the hy- pothesis h as surrogate of the true target label. However, in the early stage of the learn- ing process, the pseudo-labels can be unreliable, which can lead to an inaccurate estimate of (S( = (> = y)). To address this, the following Lemma indicates that estimat- ing the conditional Wasserstein distance is equivalent estimating the Wasserstein adversarial loss weighted by the label-distribution ratio.
Lemma 1. The weighted conditional Wasserstein distance can be implicitly expressed as:
C = =>>]<<<>>>>>>>>>>>>>>>>>>>>>>>>> = v)) max di dr
where (=) (eaal~ (Y = y), and d1, dr 2 - R4 are the 1-Lipschitz domain discriminators Ganin et al. 2016 Arjovsky al. 2017).
Lemma 1 reveals that instead of using pseudo-labels to estimate the weighted conditional Wasser- stein distance, one can train 7' domain discriminators with weighted Wasserstein adversarial loss,
4

---

Under review as a conference paper at ICLR 2021
which does not require the pseudo-label of each target sample during the matching. On the other hand, B2 can be obtained from &G, which will be elaborated in Sec. 3.2.
In practice, adopt a hybrid approach by linearly combining the explicit and implicit matching strategies for all the scenarios, which empirical results show its effectiveness.
4.2 ESTIMATE LABEL DISTRIBUTION RATIO Ot
Multi-Source Transfer with target labels When the target labels are available, å can be directly estimated from the data without any assumption and ãz - ar can be proved from asymptotic statistics.
Unsupervised Multi-Source DA In this scenario, it is impossible to estimate a good å with- out imposing any additional assumptions. Following (Zhang al.] 2013] Lipton et al. 2018 Azizzadenesheli et al.] 2019; Combes et al.) 2020), we assume that the conditional distributions aligned between the target and source domains (i.e., Sr(=ly) T(=ly)). Then, we denote Syy)) Tv) as the predicted t-sourcc/target label distribution through the hypothesis h, and also define Cs,, [y, A|] S, Jargmax, ((2,),))
y, Y k] is the t-source prediction confusion ma- trix. We can demonstrate that if the conditional distribution aligned, we have (vy) Ta. (v), with Ta, (Y y) Σx Css ly. (k)x(() the constructed target prediction distribution from the I-source information. (See Appendix forr the proof). Then we can estimate â through matching these two distributions by minimizing (T (T) Ta. (v)), which is equivalent to:
(T(())
D
D
1
Σ (()) log( Σ Cs.la. s.t. Vy V, 2 O,
-a(()))))) (2)
min 6,
y=1
y=1
In the aforementioned part, we have assumed the conditional distribution aligned, which a feasible requirement in algorithm, since the goal of g exactly aims gradually achieving this. In the experiments, we iteratively estimate å₂ and learn
Unsupervised Multi-Source Partial DA When supp(T(y) S (u)p(S,(y)) ar is sparse due to the non-overlapped classes. Accordingly, addition to the assumption Sx(2ly) = T(=ly) as in unsupervised DA, also impose such prior knowledge by adding a regularizer ||ā:]l to the objective of Eq. (2) to induce sparsity in âg (See Appendix | for more details).
In training the neural network, since the non-overlapped classes will be automatically assigned with a small or zero de, (g, h) will be less affected by the classes with small á, Our empirical results effectively validate its capability in detecting non-overlapping classes and show significant improve- ments over other baselines.
4.3 ESTIMATE TASK RELATION CORFFICIENT À
Inspired by Theorem given fixed â and (g, h), we estimate through optimizing the derived upper bound.
T
S>) min (11 + Co Σ Ai = = =)) + Ci
B.
(3)
T
VL, All 0. Σ A
In practice, iēs; (h,9) 9) is the weighted empirical prediction and E Byttty W (f(El
$= y)) approximated by the dynamic feature centroid distance Σ, T(y) C* (See Appendix for details). Thus, solving À is standard convex optimization problem.
4.4 ALGORITHM DESCRIPTION
Based on the aforementioned components, we present the description of WADN (Algorithm T in the unsupervised scenarios (UDA and Partial DA), which iteratively updates (g, h), åt, and X. When
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Wasserstein Aggregation Domain Network (unsupervised scenarios, one iteration)
Require: Labeled source samples Si...., Sy. Target samples T
Ensure:
Label distribution ratio Oe, task relation simplex A. Feature Function Classifier h, Do-
main critic function d1,...,dry dr, class centroid for source C² and target CV (vi [1,7], E )).
P D DNN Parameter Training Stage (fixed ar and A) <44
2: for mini-bated of samples (Xs, . .s,) ~ St, ....... (XS,. ys,) ~ Sr, (XT) f do
3: Predict target pseudo-label yr = argmx,, argmax,Algxx (gg()))
4: Compute source confusion matrix for each batch (un-normalized)
B. Cg.- = # [argmax,y/ y (12,, v²) y, Y k] (t= (I=1,...,,7
5: Compute the batched class centroid for source CØ and target C?
6:
Moving Average for update source/target class centroid: (e) 0.7)
7:
Update Source class centroid
C² =
C + (1 e1) CH
8: Update Target class centroid
C + (1 61 x C
9:
Updating g, 1,, d1, dr (SGD and Gradient Reversal), by solving:
E, X1) (h, +cCo E, A C*I12 min max 9.h dr
Classification Loss
Explicit Conditiona Loss
+ 6)6 E, Ses (z))d(22) - (es) d(+))
Implicit Conditional Loss
10: end for
11: D Estimation á; and À 444
Compute the global(normalized) source confusion matrix
Cs, S Jargmax,, y') = y. Y k] (t I.....,))
13: Solve (denoted as {a()T=)) by (Sec. 4.2 Unsupervised DA or Partial UDA). Update aę by moving average: a₁ Eι a + (1 (1-e1) a
15: Compute the weighted loss and weighted centroid distance, then solve A (denoted as X)) from And updating À by moving average: X a X + (1 G) x X'
Sec. 4.3
updating À and og,, we used package CVXPY to optimize the two standard convex losses after each training epoch, then we updating them by using the moving average. As for WADN under target label information, we did not require pseudo-label and directly compute & shown in Appendix L
EXPERIMENTS
In this section, compare proposed approaches with several baselines for the popular tasks. For all the scenarios, the following baselines are evaluated: (I) Source method applied only labelled source data to train the model. (II) DANN (Ganin et al. 2016). We follow the protocol Wen al. (2020) to merge all the source dataset as a global source domain. (III) MDAN (Zhao et al.) 2018); (IV) MDMN (i al. 2018b); (V) MASDA (Peng et al.
2019) adopted maximizing classifier dis- crepancy (Satoo et al.] 2018) and (VI) DARN (Wen et al. 2020 For the conventional multi-source transfer and partial unsupervised multi-source DA, we additionally compare specific baselines. All the baselines are re- re-implemented in the same network structure for fair comparisons. The detailed network structures,
settings, training details are put in Appendix M
We evaluate the performance on three different datasets: (I) Amazon Review. (Blitzer et al. 2007) It
contains four domains (Books, DVD, Electronics, and Kitchen) with positive and negative product reviews. We follow the common data pre-processing strategies as (Chen et al. (2012) to form 5000-dimensional bag-of-words feature. Note that the label distribution in the original dataset is uniform. To enhance the benefits of the proposed approach, we create label distribution drifted task by randomly dropping 50% negative reviews of all the sources while keeping the target identical. (show in Fig (a)). (II) Digits. It consists four digits recognition datasets including MNIST, USPS (Hull 1994), SVHN (Netzer et al. 2011)) and Synth (Ganin et al. 2016). We also create a slight label distribution drifi for the sources by randomly dropping 50% samples digits 5-9 and keep

---

Under review as a conference paper at ICLR 2021
target identical. (showed in Fig. (b)) (III) Office-Home Dataset (Venkateswara al. 2017 contains 65 classes for four different domains: Art, Clipart, Product and Seal-World, We used the ResNet50 (He et al. 2016) pretrained from the ImageNet in PyTorch as the base network for feature learning and put a MLP for the classification. The label distributions these four domains different and did not manually create label drift (showed in (c).
5.1 UNSUPERVISED MULTI-SOURCE DA
In the unsupervised multi-source DA, we evaluate the proposed approach on all the three datasets. We use similar typer-parameter selection strategy in DANN (Ganin et al. 2016). All reported results are averaged from five runs. The detailed experimental settings are illustrated Appendix
M The empirical results are illustrated in Tab. 7 and 3] Since did not change the target label distribution throughout the whole experiments. then we still use the target accuracy as the metric. We report the means and standard deviations for each approach. The best approaches based
two-sided Wilcoxon signed-rank test (significance level 0.05) are shown in bold.
Table 2: Unsupervised DA: Accuracy (%) on the
Digits.
Target
MNIST
SVHN
SYNTH
USPS
Average
Source
44.50
67.14 1.40
11.40
78.11 11.31
1..12
79.05
DANN
86.9941.53
69.5642.26
78.7341.30
86.71
80.52
MDAN
87.8642.24
69.13
79.77+1.00
86.50
80.81
MDMN
+1.88
69. 69.84+1.59
80.27
+0.88
86.61
81.00
MOSDA
87.22 11.70
68.89
1..33
80.01
11.77
86.39 :: 1.66
80.87
DARN
86.98 11.29
68.59 1.79
80.68
10.61
86.85
1..8
80.78
WADN
89.07
71.66=0.77
82.06
+..89
90.07=1.10
83.22
Full TAR
98. 08.70.0.15
85
20.00000.0
95.10+0.14
96,64
93.91
Table 3: Unsupervised DA: Accuracy (%) on Office-Home
Target
Art
Clipart
Product
Real- Geal-World
Average
Source
49.2540..0
46.89 10.61
66.54.1.72
73.64
59.08
DANN
50.32
11.16
11.16
68.1811.27
73.71 1.63
60.58
MDAN
67.93+0.36 66.61 1.33
79.24
81.82+0.65
73.90
MDMN
68.3840.58
67.42+0.58
82.49+0.56
83.32
75.28
MA
63.77+1.07 62.30
75.85
79.92
70.46
DARN
69.89 68.61
10.50
83.37
84.29
76.54
WADN
73.781n.48
70.18
10.54
86.32
10.38
87.28
10.87
79.39
Full TAR
76.1740.16
79.37
90.60
87.65
83.45
The empirical results reveal significantly better performance (Ne 3%) different datasets. For understanding the working principles WADN, we evaluate the performance under different levels of source label shift in Amazon Review dataset ([)) The results show strong practical benefits for WADN during a gradual larger label shift. In addition, we visualize the task relations in digits (Fig and demonstrate non-uniform X, which highlights the importance of properly choosing the most related source rather than simply merging all the data. E.g. when the target domain is SVHN, WADN mainly leverages the information from SYNTH, since they more semantically similar and MNIST does not help too much for SVHN (observed by Ganin et al. (2016)). The additional analysis and results can be found in Appendix 0
5.2 MULTI-SOURCE TRANSFER LEARNING WITH LIMITED TARGET SAMPLES
We adopt Amazon Review and Digits in the multi-source transfer learning with limited target sam- ples, which have been widely used. In the experiments, still use shifted sources. We randomly sample only 10% labeled samples (w.r.t target dataset in unsupervised DA) as training set and the rest 90% samples the unseen target test set. (See Appendix M for details). We adopt the same
7

---

Under review as a conference paper at ICLR 2021
02-
MNST
0.47
0.32
0.21
SUMIN
1
70
UOPP SYNTH
74
AX3h
0.36
0.31
0.33
10%
2016
Drosped
10%
60%
50%
MMST
SV/1N
SYNTH
USPS
1% 3% 5%
on
10%
(a)
(h)
(c)
Figure 1: (a) Unsupervised DA with Amazon Review dataset. Accuracy under different levels of shifted sources (higher dropping rate means larger label shift). The results are averaged on all target domains. See the results for each task in Fig. (12). (b) Visualization of À in unsupervised DA, each row corresponds to one target domain. (c) Transfer learning with limited target labels in USPS. The performance of WADN consistently better under different target samples (smaller portion indicates fewer target samples).
prper-parameters and training strategies with unsupervised DA. We specifically add recent base- line RLUS (Konstantinov & Lampert| (2019) and MME (Saito et al. 2019), which also considered transfer learning with the labeled target.
Table 4: Multi-source Transfer: Accuracy (%) on Source-Shifted Amazon Review
Target
Books
DVD
Electronics
Kitchen
Average
Source Tar
72.59
73.02
81.59 41.58
77.03
76.06
DANN
67.35
66.33+2.42
78.03
74.31
71.50
MIAN
68. 68.70+2.99
69.30+2.21
78.78 21
74.07+1.89
72.71
M'SDA
69.28
67.40+0046
76.28
76.50+1.19
72.36
DARN
68.57
68.7 68.7741.81
80.1941.00
77.51
73.76
RLUS
71.83+11.7
49.64+22.9
81.98 + 1.01
78.69
75.54
МME
69.66+0.58
71.36
78,88
76.64+1.73
74.14
WADN
74.83 10.84
75.05 10.62
84.23
10.58
81.53
78.91
Full TAR
84. 84.10+0.13
83.68 +0.12
86. 1 +0.32
88.72+0.14
86.65
Table 5: Multi-source Transfer: Accuracy (%) on the Source-Shifted Digits
Target
MNIST
SVHN
SYNTH
USPS
Average
Source + Tar
79.63
56.48 1.90
69.6441.38
86.29 1..5
73.01
DANN
86.77+1.30
69.13+1.09
78.82
86.54+1.08
80.32
MDAN
86.93+1.05
68.25
79.80
86.23+1.41
80.30
M'D
85,88 1.206
68.84 1.05
76.29 10.95
87.15 1.10
79.54
DARN
86,58+1.46
68. 1S.8641..30
80.47+0.07
86,80+0.50
80.68
RLUS
1..61
70.50 1..50.0..94
79.52. 1.30
86.
16.70+1.13
81.08
МME
87.24
65. 65.20
80.31
+0.60
87.88 20.76
80.16
WADN
88.32+1.17
70.64
81.53
90.53
82.75
Full TAR
98.70.1o.18
85.20 10.09
95.10.10.14
96.64 10.13
93.91
The results are reported Tabs. 5 which also indicates strong empirical benefits. To show the
effectiveness of WADN, select various portions of labelled samples (1% 10%) on the target. The results in [i) on USPS dataset shows a consistently better than the baselines, even in the few target samples.
8

---

Under review as a conference paper at ICLR 2021
5.3 PARTIAL UNSUPERVISED MULTI- -OURCE DA
In this scenario, we adopt the Office-Home dataset to evaluate our approach, as it contains large (65) classes. We do not change the source domains and we randomly choose 35 classes from the target. We evaluate all the baselines on the same selected classes and repeat times. All reported results are averaged from 3 different sub-class selections (15 runs total), showing in T (See Appendix M for details.) We additionally compare PADA (Cao et al., 2018) approach by merging all the sources and use one-to-one partial DA algorithm. We adopt the same hyper-parametee and training strategies with unsupervised DA scenario.
Table 6: Unsupervised Partial DA: Accuracy (%) on Office-Home (#Source: 65, #Target: 35)
Target
Art
Clipart
Product
Real- World
Average
Source
50.56.1.42
49.79 1.14
68.10
78. 18.24.0.78
61.67
DANN
53.8642.28
52.71
71.25++2.44
1..92
63.69
MDAN
67.56.1.39
65.38_1.1 1.30
81.49+1.99 1.92
83, 83.44
74,47
MDMN
68.1341.08
65.27 + 1.98
81.33
84.00
74.68
S'SDA
65.10+1.97
61.80 1.99
76. 76.19+2.41
79. 79.14
70.56
DARN
71.53
69.31 :
82.87 1:.56
84.76
77.12
PADA
74.37 74.37.0.84
69.64
00.80
83.45
85.64
78.28
WADN
80.06
75.90
1.06
89.55 10.72
90.40 0.339
83.98
The reported results are also significantly better than the current multi-source DA or one-to-one partial DA approach, which verifies the benefits of WADN: properly estimating â; and assigning proper À for each source.
85-
DHAMI
PACA
90
85
80
AAPAPO
45 dis
MM
XX XXXXXXXX XXXXXXXXXXXX
XXXXXXXXX XXXXXX XXXX
(b)
Figure 2: Analysis on Partial DA of target Product. (a) Performance (mean + std) of different selected classes on the target; (b) We select 15 classes and visualize estimated & (the bar plot). The "X" along the x-axis represents the index of dropped 50 classes. The red curves are the true label distribution ratio. See Appendix P for additional results and analysis.
Besides, we change the number of selected classes (Fig aa)), the proposed WADN still indicates consistent better results by large margin, which indicates the importance of considering â and A. In contrast, DANN shows unstable results in less selected classes. (See Appendix PP for details) Beside, WADN shows a good estimation of the label distribution ratio (Fig (b)) and has correctly detected the non-overlapping classes, which indicates its good explainability.
6 CONCLUSION
In this paper, we proposed a new theoretical principled algorithm WADN (Wasserstein Aggregation Domain Network) solve the multi-source transfer learning problem under target shift. WADN provides unified solution for various deep multi-source transfer scenarios: learning with limited target data, unsupervised DA, and partial unsupervised DA. We evaluate the proposed method by extensive experiments and show its strong empirical results.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Adversarial invariant feature learning with ac- curacy constraint for domain generalization. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 315-331. Springer, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875 2017.
Wasserstein gan. arXiv preprint
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar, Regularized learn- ing for domain adaptation under label shifts. In International Conference Learning Represen- tations, 2019. httttp:ss//ppereeeeeee
net/ forum? (tae:ummmdddr100r3333333
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regullarization, In Advances in Neural Information Processing Systems, pp- 998-1008, 2018.
Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein distance for mixture distributions with applications in adversarial learning and domain adaptation. arXiv preprint arXiv:1902.00415. 2019.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pp. 137-144, 2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1-2):151-77,, 2010.
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 440-447, 2007.
Silvia Bucci, Antonio D' D'Innocente, and Tatiana Tommasi. Tackling partial domain adaptation with self-supervision. In International Conference Image Analysis and Processing, pp. 70-81. Springer, 2019.
Zhangjic Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135-150, 2018.
Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning transfer examples for partial domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2985-2994, 2019.
Minmin Chen, Zhixiang Xu, Kilian Q Weinberger, and Fei Sha. Marginalized denoising autoen- coders for domain adaptation. In Proceedings of the 29th International Coference on Interna- tional Conference on Machine Learning, pp. 1627-1634, 2012.
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. Multi-source cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhihong Chen, Chao Chen, Zhaowei Cheng. Ke Fang, and Xinyu Jin. Selective transfer with rein- forced transfer network for partial domain adaptation. In AAAI Conference on Artificial Intelli- gence, 2020.
Stergios Christodoulidis, Marios Anthimopoulos, Lukas Ebner, Andreas Christe, and Stavroula Mougiakakou. Multisource transfer learning with convolutional neural networks for lung pat- tern analysis. IEEE journal of biomedical and health informatics, 21(1):76-84, 2016.
Remi Tachet des Combes, Han Zhao, Yu-Xian Xiang Wang, and Geoff Gordon. Domain adaptation with conditional distribution matching and generalized label shift. arXiv preprint arXiv:2003.044/ 2020.
10

---

Under review as a conference paper at ICLR 2021
Marthinus Christoffel Du Plessis and Masashi Sugiyama. Semi-supervised learning of class balance under class-prior change by distribution matching. Neural Networks, 50:110-11: 2014.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversariall training of neural net- works. The Journal of Machine Learning Research, (11)):009-20300 2016.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary C Lipton. A unified view of label shift estimation. arXiv preprint arXiv:2003.07554 2020.
Linda S Geiss, Jing Wang. Yiling J Cheng. Theodore J Thompson, Lawrence Barker, Yanfeng Li, Ann L Albright, and Edward W Gregg. Prevalence and incidence trends for diagnosed diabetes among adults aged 20 to 79 years, united states, 1980-2012. Jama, 3 312(12))1218-12)) 2014.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International con- ference on machine learning. pp. 2839-2848, 2016.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Judy Hoffman, Brian Kulis, Trevor Darrell, and Kate Saenko. Discovering latent domains for multi- source domain adaptation. In European Conference on Computer Vision, pp. 702-715. Springer, 2012.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang, Algorithms and theory for multiple-source adaptation. Advances in Neural Information Processing Systems, pp. 8246-8255 2018a.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. PMLR, 2018b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. arXiv preprint arXiv: 1et.//9922.0.,,.. 2019.
Jonathan J. Hull. database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550- 55,444,, 1994.
Maximilian Isse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. arXiv preprint arXiv:1905.10427, 2019.
Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. In Interna- tional Conference on Machine Learning, pp. 3488-3498, 2019.
Joshua Lee, Prasanna Sattigeri, and Gregory Wornell. Learning new tricks from old dogs: Multi- source transfer learning from pre-trained networks. Advances in Neural Information Processing Systems, pp. 4370-4380 2019.
Jingmei Li, Weifei Wu, Di Xue, and Peng Gao. Multi-source deep transfer neural network algorithm. Sensors, 19(18):3992, 2019a.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018a.
11

---

Under review as a conference paper at ICLR 2021
Yitong Li, David E Carlson, et al. Extracting relationships by multi-domain matching. In Advances in Neural Information Processing Systems, pp. 6798 680886609.200 2018b.
Yitong Li, Michael Murias, Samantha Major, Geraldine Dawson, and David Carlson. On target shift in adversarial domain adaptation. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 616-625, 2019b.
Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua. Multi-source domain adaptation for visual sentiment classification. arXiv preprint arXiv:2001 003886, 2020.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International Conference on Machine Learning, pp. 3122-3130, 2018.
Jing Liu, Yuling Hong. Ralph B D' Agostino Sr, Zhaosu Wu, Wei Wang, Jiayi Sun, Peter WF Wilson, William B Kannel, and Dong Zhao. Predictive value for the chinese population of the framingham chd risk assessment tool compared with the chinese multi multi-provincial cohort study. Jama, 291()1); 2591-2599, 2004.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Alexander Selvikvág Lundervold and Arvid Lundervold. An overview of deep learning in medical imaging focusing on mri. Zeitschrifi für Mediiinischee Physik, 29(2):102-127, 2019.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arxiv:0902.34400, 2009.
Yishay Mansour, Mehryar Mohri, Ananda Theertha Suresh, and Ke Wu. A theory of multiple-source adaptation with limited target labeled data. arXiv preprint arXiv:2007.09762. 2020.
Mehryar Mohri and Andres Munoz Medina. New analysis and algorithm for learning with drift- ing distributions. International Conference on Algorithmic Learning Theory, pp. 124-138. Springer, 2012.
Saeid Motiaan, Marco Piccirilli, Donald A. Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2.
XuanLong Nguyen, Martin J Wainwright, Michael I Jordan, et al. On surrogate loss functions and f-divergences. The Annals of Statistics, 37(2):876-94) 600666904222 2009.
Sinno Jialin Pan and Qiang Yang. and data engineering, 22(10))1345-1335 2009.
survey on transfer learning. IEEE Transactions on knowledge
Zhongyi Pei, Zhangjic Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adap- tation. arXiv preprint 1809:v1800.021111 2018
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in neural information processing systems, pp- 3347-3357, 2019.
levgen Redko, Nicolas Courty, Rémi Flamary, and Devis Tuia. Optimal transport for multi-source domain adaptation under target shift. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Re- search, pp. 849-8888898 858, PMLR, 16-18 Apr 2019. URL w89/redke. 9a. html
mlr., press//
12

---

Under review as a conference paper at ICLR 2021
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Tutorials, pp. 15-18, 2019.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference Com- puter Vision and Pattern Recognition, pp. 3723-3732, 2018.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8050-8058, 2019.
Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8503-8512, 2018.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp- 4077-4087, 2017.
Petar Stojanov, Mingming Gong, Jaime G Carbonell, and Kun Zhang. Data-driven approach to multiple-source domain adaptation. Proceedings of machine learning research, 89:3487, 2019.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary envirouments: Introduction to covariate shift adaptation. MIT press, 2012.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018
Ben Tan, Erheng Zhong, Evan Wei Xiang, and Qiang Yang. multi-transfer: Transfer learning with multiple views and multiple sources. In Proceedings of the 2013 SIAM International Conference on Data Mining, pp. 243-251. SIAM, 2013.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018-5021 00270, 2017.
Boyu Wang, Jorge Mendez, Mingbo Cai, and Eric Eaton. Transfer learning via minimizing the performance gap between domains. Advances in Neural Information Processing Systems, pp- 10645-10655, 2019a.
Haotian Wang, Wenjing Yang. Zhipeng Lin, and Yue Yu. Tmda: Task-specific multi-source domain adaptation via clustering embedded adversarial training. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 1372 1377.1377. IEEE, 2019b.
Jonathan Weed, Francis Bach, al. Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance. Bernoulli, 25(44):2620-266 2019.
Pengfei Wei, Ramon Sagana, Yiping Ke, Yew-Soon Ong, and Chi- Chi-Keong Goh. Source-target sim- ilarity modelings for multi-source transfer gaussian process regression. International Confer- on Machine Learning, pp. 3722-3731, 2017.
13

---

Under review as a conference paper at ICLR 2021
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distribu- tions: Relating covariate shift to model misspecification. In International Conference on Machine Learning. pp. 631-639, 2014.
Junfeng Wen, Russell Greiner, and Dale Schuurmans. Domain aggregation networks for multi- source domain adaptation. Proceedings of the 37th International Conference on Machine Learn- ing, 2020.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetrically-rllxxd distribution alignment. In International Conference on Machine Learn- ing, pp. 6872-6881, 2019.
Yi Yao and Gianfranco Doretto. Boosting for transfer learning with multiple sources. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 1855- 1862. IEEE, 2010.
Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Jing Zhang, Wanging Li, Philip Ogunbona, and Dong Xu. Recent advances in transfer learning for cross-dataset visual recognition: A problem-oriented perspective. ACM Computing Surveys (CSUR), 52(1):1 2(11:11-8,, 2019.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819-827, 2013.
YiNan Zhang and MingQiang An. Deep learning-and transfer learning-based super resolution re- construction from single medical image. Journal of healthcare engineering, 2017.
Yu Zhang and Qiang Yang. A survey multi-task learning. arXiv preprint arXiv: 2017.
Yu Zhang and Dit- Yan Yeung. A convex formulation for learning task relationships in multi-task learning, arXiv preprint arXiv: 120.33566, 2012.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In Advances in neural information processing systems, pp. 8559-8570, 2018
Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. arXiv preprint 1iv:::9911 199991111510 11544, 2019.
Sicheng Zhao, Bo Li, Pengfei Xu, and Kurt Keutzer. Multi-source domain adaptation in the deep learning era: A systematic survey. arXiv preprint arXiv:2002.12169, 2020.
Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and clas- sifier for cross-domain classification from multiple sources. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 5989-5996, 2019.
14

---

Under review as a conference paper at ICLR 2021
A
ADDITIONAL EMPIRICAL RESULTS
Table 7: Unsupervised DA: Accuracy (%) on Source-Shifted Amazon Review.
Target
Books
DVD
Electronics
Kitchen
Average
Source
18.15
69.51
82.0940.88
73.30
73.81
DANN
4..59 67.23
40.1
+4.7
72.00
MDAN
68.77+2.31 67.81
80.96+0.77
75.67+1.96
73.30
MDMN
70.56+1.05 69.64+0.73
82.7 82.71 +0.71
77.05+0.78
74.99
M'SDA
19.09+1.26
68.67+1.37
81.3440.66
76.10+1.47
73.79
DARN
68.68
81.51 +0.81
7
74.78
WADN
73.72 10.63
79.64 0.33
84.64
10.48
83.73
+0.50
+0.50
80.43
Full TAR
84.10+0.13
83.6840.12
86. +
88.72+0.14
86.65
B ADDITIONAL RELATED WORK
Multi-source transfer learning Practice has been proposed from various prospective. The key idea is to estimate the importance of different sources and then select the most related ones, to mit- igate the influence of negative transfer. In the multi-source unsupervised DA, (Sankaranarayanan
Balaji et al., 2019 Pei al. 2018 Zhho et al.] 2019 Zhu al.] 2019; Zhao et al, 2019 Stojanov et al. 2019: Li et al. 2019b; Wang et al. 2019b; Lin et al. 2020) proposed different practical strategies in the classification, regression and semantic segmentation problems. In the presence of target labels, |Hofmman et al. (2012); Tan al. (2013); ; Wei et al. (0077); Yao & Doretto (2010); Konstantinov
al. 2018
2020
Lampert (2019) used generalized linear model to learn the tar- get. [Christoouuliiis et al. (2016); Li al. (209a);; Chen et al. (2019) focused on deep learning approaches and Lee et al. (2019) proposed an ad-hoc strategy to combine to sources in the few-shot target domains. These ideas generally data-driven approaches and do not analyze the why the proposed practice can control the generalization error.
Label-Partial Transfer Learning Label-Partial can viewed as a special case of the label-shift. I Most existing works focus on one-to-one partial transfer learning (Zhang et al. [0088] Chen al. 2020 Bucci et al,, 20199] Cao et al., 2019) by adopting the re-weighting training approach without formal understanding. In our paper, we first rigorously analyzed this common practice and adopt the label distribution ratio as its weights, which provides a principled approach in this scenario.
B. OTHER SCENARIOS RELATED TO MULTI-SOURCE TRANSFER LEARNING
Domain Generalization The domain generalization (DG) resembles multi-source transfer but aims at different goals. A common setting in DG is to learn multiple source but directly predict on the unseen target domain. The conventional DG approaches generally learn a distribution in-
variant features (Balaji al. 2018 Saenko et al.. 2010 Motiian al., 2017 llse et al.) 2019)
conditional distribution invariant features (Li et al. 2018a 3a: Akuzawa et al. 2019). However, our the-
oretical results reveal that in the presence of label shift (i.e (ve / 1) and outlier tasks then learning conditional or marginal invariant features can not guarantee a small target risk. Our theoretical result enables formal understanding about the inherent difficulty in DG problems.
Few-Shot Learning The few-shot learning (Finn et al. 2017 Snell et al. 2017 Sung et al. 2018) can be viewed as a very specific scenario of multi-source transfer learning. We would like to point out the differences between the few-shot learning and our paper. (1) Few-shot learning generally involves a very large set of source domains T > 1 and each domain consists a modest number of observations Ns.- In our paper, we interested in the modest number of source domains T but each source domain including a sufficient large number of observations (Ns, > 1). (2) In the target domain, the seo- setting generally used K-samples (K is very small) for each class for the fine-tuning. We would like to point out this setting generally violates our theoretical assumption. In
Since supp(T(v) C supp(Se(y)) then naturally have T(y) # Se(v).
15

---

Under review as a conference paper at ICLR 2021
paper, assume the target data is ii.d. sampled from D(z, g). is equivalently viewed that first i..d. sample ~ Dyy), then i..d. sample I ~ D(rly). Generally the D(z) is non-uniform, thus few-shot setting are generally not applicable for our theoretical assumptions.
The goal of multi-task learning (Zhang Yang 2017) aims to improve the prediction performance of all the tasks. In paper, aim at controlingg the prediction risk of specified target domain. We also notice some practical techniques are common such as the shared parameter (Zhang & Yeung, 2012), shared representation (Ruder 2017). etc.
Multi-Tash Learning
C ADDITIONAL FIGURES RELATED TO THE MAIN PAPER
3EE LIWe Marewernn.nee Rrdre
Munelll
Did
0.00
Kiklin
0.05
(a) Amazon
(b) Digits
(c) Office-Home
Figure 3: Label distribution visualization. (a) One example in Amazon Review dataset with sources: Book, Dvd, Electronic and target: Kitchen. We randomly drop 50% of the negative reviews in all the sources while keeping target label distribution unchanged. (b) One example in Digits dataset with Sources:
MNIST, USPS, SVHN and Target Synth. We randomly drop 50% data on digits 5-9 in all sources while keeping target label distribution unchanged. (c) Office- Home dataset. The original
label distribution is non-uniform. See Appendix M for details.
D TABLE OF NOTATION
Table 8: Table of Notations
RD(h) E Elng)DDlh(z, 8))
Expected Risk distribution D w.r.t. hypothesis
Rooh) N Ei=l ((((x v1)
Empirical Risk on observed data (1)), that are i...d. sampled from D.
a and &
True and empirical label distribution ratio a(y) = T(y)/S(y)
Rg(h) = 1. En a ( 9:))
Empirical Weighted Risk on observed data {(z;, H Conditional distribution w.r.t latent variable Z that induced by feature learning function g.
S(z/y)= -[,g( yddr
Conditional Wasserstein distance on the latent space Z
E PROOF OF THEOREM 1
Proof idea Theorem 1 consists three steps in the proof:
Lemma 2. If the prediction loss assumed as L-Lipschitz and the hypothesis is K K-Lipschitz w.n.
the feature (given the same label), i.e. for VY [11((11))) - -h(x2, w))l2 K|| 22l12. Then the target risk can be upper bounded by:
(T( < Σ A1) (l) + LK Σ WI
v)) (4)
Proof. The target risk can be expressed as:
RT(M(2,)) v)) (((x,))((((x,)))) BynT((y) Een
16

---

Under review as a conference paper at ICLR 2021
By denoting (yy) then have:
T(y) Sly)*
By u)) = (ty) ((h),,
Then we aim to upper bound E Ex (h((, v)). For any fixed y,
( v)) Stal) (h(r,, S
S(xlv))
DEX
Then according to the Sesttroviic-Rubbisteee duality, for any distribution coupling 7 E (Tl()), y)), then we have:
inf f
E(h(zp: u)) E(h(zg, (zp: Rgg)
XxX
S inf
1e(hlapp u)) - e(h(2gg 4)) (Tpp
Jxxx
{
S L inf
[h(aps)) v)) - h(zg v) (x mng)
Xxxx
[ 1| - x0lladh(0pp: xxx
S LK inf
= LKW, (x
= y))
The first inequality obvious; and the second inequality comes from the assumption that is L- Lipschitz; the third inequality comes from the hypothesis is K K-Lipschitz w.r.t. the feature (given the same label), i.e. for vY = y, [[((11,)) h(22, I K||m1 22l12-
Then have:
(h) s By a (M
-Lu,))-sa(y)(((( M S(r = v))
Rg(h) LKE L (x -=) =))
W
Supposing each source S; we assign the weight A[t] and label distribution ratio az(y) T(y) by combining this T source target pair, we have:
then
(h) S Σ Rg (h) + LK Σ Wi - v))
Then will prove Theorem 1 from this result, we will derive the non-asymptotic bound, estimated from the finite sample observations. Supposing the empirical label ratio value is âg then for any simplex we can prove the high-probability bound.
E. BOUNDING THE EMPIRICAL AND EXPECTED PREDICTION RISK
Proof.
We first bound the first term, which can be upper bounded as:
X(t]Ro (h) Σ A Ghe (h)) < sup Σ Alt RS: (h) Σ Alt (h)) + sup Σ A(t Roy (h)
Σ (()
sup
Bounding term (I) According to the McDiarmid inequality, each item changes at most
22 2 Then have:
Ns.
2t
((I) (ET) 2 1) S exp(
T X2 (2(1)ar(y)222 )= =8 8
Ei=1 B.N
17

---

Under review as a conference paper at ICLR 2021
By substituting 6, at high probability - 8 we have:
AL1 Be
log(1/0)
(I) S R(I) + I Tf doup
2N
Where Timax = SUPлEH ((h) and N = Ei= Ns, the total source observations and ft = No. the frequency ratio of each source. And d max d (T)()())))
=...7 maxye(1.>] (y)), the maximum true label shift value (constant).
Bounding Esup(!), the expectation term can be upper bounded as the form of Rademacher Com- plexity:
E(I S 2E, ST sup
Aldk
Σ
TN (auly)e(h(ze ut))
(we (E
A B sup 2 Σ
Σ
TN (c(())((h(zx)))))
h
(w..BeJES.
< 2 ssup Eg Bss sup Σ
TN (a((v)((((z).)))
h
(ES
sup (R, (e, H) 22|(, H)
Where R(C, 71) sup, (e, H) sup, suPh~H Es,,r E(min)s TN [o)()))(((z,, 9r)),, repre- sents the Rademacher complexity w.r.t. the prediction loss E, hypothesis and true label distribution ratio at-
Therefore with high probability 1 8, we have:
T
log(1/8)
Σ Rg (h)
Σ All (h)) S R(e,h) + L L Soup
sup h
B
2N
1=1
Bounding Term (II)
For all the hypothesis h, we have:
Ns.
Σ Agg (h)
Σ A1) (1)l
Σ Ald
1
Σ (a(v(i)) a(v(i)))((M)
No.
DJ
Σ
A
(a(Y = - -ay = -= y)){(( = v)1
Ns.
Where 8(Y = y) =A>
No. E(h(z; yi = y)), represents the cumulative error, conditioned on given label Y = According to the Holder inequality, we have:
E(Y==
D
Σ
A|t Ns.
1
Σ
(a =)) -Y =Y))((Y =v)
Σ Ait
1 1 la, - âellell((Y = 2)
Ns.
Lmax Σ A[l]|lar âe
Lmax sup lla, āell2
{
Therefore, Ah E H. with high probability - 8 we have:
Alt|
log (1/5)
M (N) Σ ( (h)+2R(E,M))
I Imax cup
+amax sup 1 ôell2
B1
2.V
18

---

Under review as a conference paper at ICLR 2021
E.2 BOUNDING EMPIRICAL WASSERSTHIN DISTANCE
Then we need to derive the sample complexity of the empirical and true distributions, which can be decomposed as the following two parts. For any 4, we have:
By (T y)) = y)) E ~~ ( = y))
=
(T y)] = y)) ([) =))((\\)) = 9))
E Bi
= -)) -V)
B Borri - = v))
Bounding (I) We have:
E yyTTTi (1) Wi(T(Z/V = v)lls((=/ = v)) - WiT = -)) = v)
Σ T(v) (Wi(T(x)Y w)lss v)) W1(T((/Y =1)l = =))
< Σ T()) sup (Wi(T(=)X = -W|STIaY= = 5\) - Wi = v)ll - v))
V
supp (Wi(T(xlY S) = 5) Wi(T(Z1 - = v))
S sup [Wi(S(<+\ = 1)) = -v)) - W1(8/(2/Y = /x = v))
"
Wi = (= = v) (i () g))]
= =sup
= v) = v)) + - = y))
=
"
The first inequality holds because of the Holder inequality. As for the second inequality, we use the triangle inequality of Wasserstein distance. (Pll))Q S (P + (P()) + (PPlll)).
According to the convergence behavior of Wasserstein distance (Weed et al. 2019). with high prob- ability 2 26 we have:
W3 (5, (x)) = U)) + - = 4)) S k(6, N N3, NY)
Where N6,,N3, N2) (N,) +G / log (2)
where N
Ng.
is the number of Y = in source / and NY is the number of Y = in target distribution. Cr.y C 8t,y 2, Sy >> 2 are positive constant in the concentration inequality. This indicates the convergence behavior between empirical and true Wasserstein distance.
If we adopt the union bound (over all the labels) by setting d 8/12%, then with high probability > 1 - 28, we have:
sup WT (8(x U = v)) - - y)) S (06, Ng. N
where x(6, No N = Cr
CN +
1 log( 20
No.
Again by adopting the union bound (over all the tasks) by setting +- 5/T, with high probability 28, we have:
w)) E WI = w>) v)) s sup N6,,NN Ng, Σ E(JEETTT Wi (T = w)|lS(E\Y
Where x(O, . N2)
Cr.y (N - + CC,
V
1 log
2T )
No.
NY
19

---

Under review as a conference paper at ICLR 2021
Bounding III)
We can bound the second term:
()]l y)) E but Wi B (T By-T(y) Wi (T Wy
vx1l v))
SupWi(F(x)'' = = =y) T(y) - Fvy)
$ C
- -
"
Where C sup, Wi (T = y)|l =)) a positive and bounded constant. Then need to bound E ( by adopting MicDiarmid's inequality, we have at high probability 1 - 6:
log(1/5)
Σ T()) - -((i s Ex - -
2NT
y
log(1/8)
= -2E, Σ of( +
2NT
Then bound R.g E, (T)) We use the properties of Rademacher complexity [Lemma 26. 11, Shalev-Shwartz & Ben-David 2014 and notice that ((v) is a probability simplex, then we have:
2 log(2(D/)
E. Cot(y) S
V
N₂
Then we have T(v) fv²)
2 21og(2
log 1/8) 2NT
NT
Then using the union bound and denoting +- 8/T, with high probability 2 and for any simplex A, we have:
Σ Wi v)]l b)) Σ y(m (T((T()Y==))1333 v)
21og(2(3"))
log(T/5)
Ciax (
Ng
V
2N
where Cmax sup, C Cla
Combining together, can derive the PAC-Leaming bound, which is estimated from the finite samples (with high probability 1 - 46);
X2
(gg(
RT(h) S Σ Ae Role (h) I.H N. Bin W WiTTE ,))|s(=\ vi) Temax poup
3
2.V
2log(2|)))
log(T/8)
+ 22 + Limax sup/lo: della + sup Ng + Cmaxl
Nr
2NV
Then we denote Comp (Ns.... Ny. 6) = 2R h) + sup, k(8, N N + Ciaxx (V
2log(2(D'1) NT
+
as the convergence rate function that decreases with larger Ns....., Ny. Bedsides, R(l,h) sup, Re(e, H) is the we-ghittted Rademacher complexity. Given fixed hypothesis with finite VC dimension it can be proved R(E, h) = min NS.. O( No. ie (Shalev-Shwartz &
(T8) 2NT
Ben- Ben-David 2014).
II the hypothesis the neural network, the Rademacher complexity still he bounded analogously.
20

---

Under review as a conference paper at ICLR 2021
F PROOF OF THEOREM 2
We first recall the stochastic feature representation g such that g: X' + and scoring hypothesis h: 2 y R and the prediction loss E with f R R.
Proof. The marginal distribution and conditional distribution w.r.t. latent variable Z that are induced by 9. which can be reformulated as:
S(z) = (22)=)((z)dz S(zly) = g( y)dr
In the multi-clas classification problem. we additionally define the following distributions: (* = S(Y = k. 2) S(Y = k))(=|Y = k)
(*(2 T(Y T( A)T =)
Based on (gguyen al. conditioned fixed point [x, y) w.r.t h and is (E2=)(=t))((((2,,))) v)). Then taking the expectation over the S(z, v) we have:
2009) and (z(2x) is stochastic representation learning function, the loss
Rs(
B (z,,)) (
y
Σ
S(y = k) S(x|8 = [(<l=)e(M(2,)) k))dzdr
k=1
y
Σ S(y = k)) S11. S(a)V= 1) 13)dzz
VI
Σ S(y = k)) /. (((El = =k)((((Z, y k))dz
S(2, - k))((((z,
k))dz
Σ I (=)E(M(E))= =k))dz
Intuitively, the expected loss w.r.t the joint distribution S can decomposed as the expected loss on the label distribution S(y) (weighted by the labels) and conditional distribution S(-ly) (real valued conditional loss).
Then the expected risk on the S and T can be expressed as:
Rs(h,g)
Σ
<(h(z,y=k)) = (z)dz
k=1
|
Ry (h = Σ [ (o(2,,==)))rr (=)dz k=1
'Note this definition different from the conventional binary classification with binary output, and it is more suitable in the mulli-classification scenario cross entropy loss (Hofiman el al. 2018a). For example, we define = log(-) and h(2,, 2) E (0, 1) as scalar score output. Then E(h(,, 8)) can viewed as the cross-entropy loss for the netrok-network.
$ 'An alternative understanding is based on Markov chain. this case it is DAG with S(slz)
X 2 7, X S(yla), Y A S A 7 2 X. (S is the output of the scoring function). Then the pected loss over the all random variable can be equivalently written I [P(,, 2,,)) f(s) d(z, y, 2, (()z)))(
f P(e, () x) y)((s)d(z, y)d(z)d(s). Since the scoring S determined by h(z, y). then P(sly. 2) 1. According to the definition we have P(z|z) g(=)z),
y){(s)
P(z, y) S(z, v). then the loss be finally expressed Estnng Eocs ((h(2,))
21

---

Under review as a conference paper at ICLR 2021
By denoting aty) T(y) S(y)
have the a-weighted loss:
Rg(h, ( =T(Y = 1) (1((z 1)) = 1) +TY)+++-00 =2) ([l((,,-=)))((2) =2)
+ T(Y = =b) (( = k)dz
Then we have:
(h, R3 (h, ΣT( ( - R)]
Under the same assumption, we have the loss function E(h(z, Y k)) is KL--ipschitz w.r.t. the
cost ||2 (given a fixed k). Therefore by adopting the same proof strategy Caattroviic-RRubinsteri duality) Lemma 2, we have
KLT(Y ()()(5(=))* (=|Y 1)) + KLT(Y k)W1(S(2\Y +)]T(( k))
k (i(((= = (=|Y = =y))
Therefore, have:
Rr S (g. g) + LKEY~T()) (i( T) =
Based on the aforementioned result, we have 11 1, .,... T and denote S = Se and orly) on(y) Syy):
Ry (h, (h, & A[t (h, 9) +
= v))
Summing over I,..., T. we have:
T
T
Rh - Σ
R|t| (h, + LK
Wi =))
1=1
G APPROXIMATION W, DISTANCE
According to Jensen inequality, we have
(i(S) = v) = v)) s (IW2(S((E)Y = (xI
b))
Supposing S;(=\Y= =) ~ NCC Σ) and 7(= v) R NN(",, Σ). then we have:
() i C²I Trace(2E Trace(2E-2(233))
We would like point out that assuming the identical covariance matrix is more computationally efficient during the matching. This is advantageous and reasonable in the deep learning regime: we adopted the mini-batch (ranging from 20-128) for the neural network parameter optimization, in each mini-batch the samples of each class are small, then we compute the empirical covari- ance/variance matrix will be surely biased to the ground truth variance and induce much higher complexity to optimize. By the contrary, the empirical mean is unbiased and computationally ef- ficient, we can simply use the moving the moving average to efficiently update the estimated mean value (with unbiased estimator). The empirical results verify the effectiveness of this idea.
H PROOF OF LEMMA I
For each source S,, by introducing the duality of Wasserstein-1 distance, for E V, we have: ((l((
= sup E (x)s((H Er (en((=l)d( ||
= sup Σ S((2)))d(2) Σ ((2lu)d(2)
1
T(y)
[s((z,))((2)) - (zx,)))d(2)
sup
Tw))
S,())
22

---

Under review as a conference paper at ICLR 2021
Then by defining (=(2) -1 ((a.))~)S)) 8( ((x))))))) an y), we can see for each pair
T(==)
observation (z, sampled from the same distribution, then ār(Z = =z) = =ae(Y = -y). Then we have:
Σ (T))((((HWIITTE))))
Σ sup E (a Σ ((,)))d(2))
[d]
- sup (((())((())(()) - ET())(())
|
sup (z) - Ex-T(=)
1d
We propose a simple example to understand ā,: supposing three samples in St { 1), (z2, Y 1), (za, Y = =)) then āς(z1) (x2) (1) and (x(23) = a. (0). Therefore, the conditional term is equivalent to the label-weighted Wasserstein adversarial learning. We plug in each source domain weight j|] and domain discriminator d,, we finally have Lemma 1.
1 DERIVE THE LABEL RATIO LOSS
We suppose the representation learning aims at matching the conditional distribution such that T(lyy) s (zly).tt, then we suppose the predicted target distribution as Tvy). By simplifying
the notation, we define f()) ulgmax,hez.) y) the most possible prediction label output, then have:
T()) Σ T(f(z) y/1 = k)T(Y =k) = Σ S(((()= -|Y k) = k)
k=1
1=1
y =Esute S(((z)) = y, y, y= =k)o((k)= Tar (1)
The first equality comes from the definition of target label prediction distribution, T(y) = [[()) = v) T(((2) y) = Ei = Y = k) Ek T(f(z) = y/Y
kY k).
The second equality T(((2) = y/8 k) Sx([2)) y k) holds since 11, T(=ly) s S,(y)), then for the shared hypothesis f. we have [(f(z) = y/Y = =) = Sx((())))> = y/Y = k).
The term Sz(f(z) y, Y k) the (expected) source prediction confusion matrix, and we denote its empirical (observed) version as ((()(z) = y, = k).
Based on this idea, in practice we want to find å to match the two predicted distribution T and Ta If we adopt the Kl.-divergence as the metric, we have:
T(g) Ta, (3)
min DKL (T = min E log de
min -Ey- (Tg(
min Σ (v)) log(
Sc(f(2) =k)a(k))
We should notice the nature constraints of label ratio: (ár(z) > 0, Ey de (3)) = 1}. Based on this principle, we proposed the optimization problem to estimate each label ratio. We adopt its empirical counterpart, the empirical confusion matrix Cs, [v. kj = S.(f(2) Y = k). then the optimization loss can be expressed as:
D
min
Σ T log > C,,
S.L. Vy y. av)) 2 0,
Σ (ar()) =1
23

---

Under review as a conference paper at ICLR 2021
Classifier
S1
Lalf
g
centroid
Feature Extractor
d1
adv
dr
CT
adv
Critic
Figure 4: Network Structure Proposed Approach. consists three losses: the weighted Classifi- cation losses; the centroid matching for explicit conditional matching; the weighted adversarial loss for implicit conditional matching, showed Eq. 6
J LABEL PARTIAL MULTI-SOURCE UNSUPERVISED DA
The key difference between multi-conventional and partial unsupervised DA is the estimation step of o₁. In fact, we only add a sparse constraint for estimating each á;;
D
min
Σ
(()) log (csl C2llárl1
y=1
(5)
s.t. Vy E V. ), 2 O, Σar(v) =1
Where C2 is the hyper-parameter to control the level of target label sparsity, to estimate the target label distribution. the paper, we denote C2 = 0.1.
K EXPLICIT AND IMPLICIT CONDITIONAL LEARNING
Inspired by Theorem 2, we need to learn the function X' + Z and h y + R to minimize:
min 9.h
Σ
A g Co Σ Alt gM W1 ->) y))
This can be equivalently expressed as:
min
Σ
A14 (lg (h, + cCo Σ A B = w))
9.h
+ (1 (1-
A WI WiS = (+|( v))
Due to the explicit and implicit approximation of conditional distance, we then optimize an altern-- tive form:
min max
Att (h. eCo Σ
C Explicit Conditional Less
g.h di wo..d
Classifiaaion Loss
(6)
+ (1-c)Co Σ Attl (( (2) B B2- d(z)]
Implicit Conditional Loss
Where
24

---

Under review as a conference paper at ICLR 2021
C Σ ( 1 (ve mn) 2 the centroid of label Y = y in source St.
CV Σ ( Tu [² the centroid of pseudo-label Y Up in target St. (If it the unsupervised DA scenarios).
2t
dx(2) = 1 [(x.))s.)jal aY y). namely if each pair observation (z, y2 from the distribu- tion, then āz (Z = 2) = (Y
d1.-.. .dr are domain discriminator (or critic function) restricted within 1-Lipschitz func- tion.
E [0, 1] is the adjustment parameter in the trade-off of explicit and implicit learning. Based on the equivalence form, our approach proposed a theoretical principled way to tuning its weights. In the paper. we assume e = 0.5.
(v) empirical target label distribution. (In the unsupervised DA scenarios, approxi- mate it by predicted target label distribution Tv))))
Gradient Penalty In order enforce the Lipschitz property of the statistic critic function, we adopt the gradient penalty term (Gulrajani et al. 2017). More concretely, given two samples ~ (.()) and 2, (2)) we generate an interpolated sample Zint Éz, + (1 - {)², with & ~ Unif(0, I 1]. Then we add gradient penalty [ 12 as regularization term to control the Lipschitz property w.r.t. the discriminator d,,-.. dr.
L ALGORITHM DESCRIPTIONS
We propose detailed pipeline of the proposed algorithm in the following, shown in Algorithm As for updating À and Otg, we iteratively solve the convex optimization problem after each training epoch and updating them by using the moving average technique.
and 3
For solving the À and ar, notice that frequently updating these two parameters in the mini-batch level will lead to an instability result during the training. [ As a consequence, we compute the accumulated confusion matrix, weighted prediction risk, and conditional Wasserstein distance for the whole training epoch and then solve the optimization problem. We use CVXPY to optimize the two standard convex losses.
Comparison with different time and memory complexity. We discuss the time and memory complexity of our approach.
Time complexity: In computing each batch we need to compute T re-weighted loss, T domain adversarial loss and T explicit conditional loss. Then our computational complexity is still (T)(T) during the mini-batch training, which is comparable with recent SOTA such MDAN and DARN. addition, after each training epoch we need to estimate o and X, which can have time complexity (TT))) with each epoch. (If we adopt SGD to solve these two convex problems). Therefore, the our proposed algorithm time complexity (TT)))). The extra y term in time complexity due to the approach of label shift in the designed algorithm.
Memory Complexity: Our proposed approach requires ((T) domain discriminator and OT])) class-feature centroids. By the contrary, MDAN and DARN require O(T) domain discriminator and M3SDA and MDMN require O domain discriminators. Since our class-feature centroids defined in the latent space (z), then the memory complexity of the class-feature centroids be much smaller than domain discriminators.
'In the label distribution shift scenarios, the mini-batch datasets highly laheled imhalanced. If evaluate ae over the mini-hatch, it can he computationally expensive and unstable. 'The optimization problem w.r.t. Or and X is not large scale, then using the standard convex solver is fast and accurate.
25

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Wasserstein Aggregation Domain Network (unsupervised scenarios, one iteration)
Require: Labeled source samples Si...., Sr, Target samples T
Ensure:
Label distribution ratio dn and task relation simplex A. Feature Learner g. Classifier h,
Statistic critic function d1, d1,..., dr, class centroid for source Cj and target CV (Vi = [, )).
1: DDD DNN Parameter Training Stage (fixed at and A) <44
2: for mini- batch of samples (xs, . .ss) ~ S1, Si..... (XS,. ys,) ~ Sr, (XT) ~ f do
3: Predict target pseudo-label yr = argaax, aggmax,Algxx (gg()))
4: Compute source confusion matrix for each batch (un-normalized)
Cg.- = # (argmax y //2,, v²) y, Y k] (I=1,...,,7
5: Compute the batched class centroid for source CØ and target C?
6: Moving Average for update source/target class centroid: (We set Ej = 0.7)
7:
Source class centroid update
C = El C² + (1 -1) x Cy
8:
Target class centroid update C = x C* + 61) x C
9: Updating g, h, di..... ... .d (SGD and Gradient Reversal), based on (q.
10: end for
11: D > Estimation â; and 444
12: Compute the global(normalized) source confusion matrix
Cs. = (aggmax,, v(= y, Y ki] (1 ,,....7) .7)
13: Solve ar (denoted as {a(TT=)) by Equation 2 (Or 5g in the partial scenario).
14: Update by moving average: a = 9 x a; (1 (1) az
15: Compute the weighted loss and weighted centroid distance, then solve (denoted as X) from
Sec. 2.3.
16: Updating À by moving average: À 0.8 x A + X
Algorithm Wasserstein Aggregation Domain Network (Limited Target Data, one iteration)
Require: Labeled source samples Si,..., Sr, Target samples I, Label shift ratio ot
Ensure: Task relation simplex X. Feature Learner g. Classifier h, Statistic critic function 41,...,dy .......d, class centroid for source C and target C (vi = 1,, 1,T T.yyEM) y).
DDD > DNN Parameter Training Stage (fixed A) 444
2: for mini-batch of samples (Xs, .s,). ~ Si...., (xSy. ys,) Sr, (xT) 7 do
3: Compute the batched class centroid for source C² and target C
4: Moving Average for update source/target class centroid: (We set 61 = 0.7)
5:
Source class centroid update
C = 6 C + (1 -1) x CH
6:
Target class centroid update
C = E x C + +1 61) x C
7: Updating g, f, d1, di..... dr (SGD and Gradient Reversal), based Eq.
8: end for
9: DDD Estimation X 444
10: Solve À by Sec. 2.3. (denoted as X')
11: Updating À by moving average: X é1 +
M DATASET DESCRIPTION AND EXPERIMENTAL DETAILS
M. AMAZON REVIEW DATASET
We used the amazon review dataset (Blitzer et al., 2007). It contains four domains (Books, DVD, Flectronies, and Kitchen) with positive (label "1") and negative product reviews (label "0"). The data size is 6465 (Books), 5586 (DVD), 7681 (Electronics), and 7945 (Kitchen). We follow the common data pre-processing strategies Chen et al. (2012): use the bag-of-words (BOW) features then extract the top-5000 frequent unigram and bigrams of all the reviews.
We also noticed the original data-set are label balanced D(y = 0) = D(y = 1). To enhance the benefits of the proposed approach. we create a new dataset with label distribution drift. Specifically, in the experimental settings, we randomly drop 50% data with label "0" (negative reviews) for all the source data while keeping the target identical, showing in Fig (5)
26

---

Under review as a conference paper at ICLR 2021
We choose the MLP model with
feature representation function [5000, 1000) units
Task prediction and domain discriminator function [1000, 500, 100] units,
We choose the dropout rate as 0.7 in the hidden and input layers. The hyper-parann syper-parameters are chosen based on vols-valiiation. The neural network is trained for epochs and the mini-batch size 20 per domain. The optimizer is Adadelta with learning rate of 0.5.
Experimental Setting We use the amazon Review dataset for two transfer learning scenarios (lim- ited target labels and unsupervised DA). We first randomly select 2K samples for each domain. Then create drifted distribution of each source, making each source 2 1500 and target sample still 2K.
In the unsupervised DA, use these labeled source tasks and unlabelled target task, which aims to predict the labels on the target domain.
In the conventional transfer learning, random sample only 10% dataset (s= 200 samples) the target training set and the rest 90% samples as the target test set.
We select Co 0.01 and C₁ 1 for these two transfer scenarios. In both practical settings, we set the maximum training epoch as 50.
Original Label distribttion Amazon Reviow
Shifted Label distribution Amazon Review
0.8
0.5
Feequncy
0.9
0.5
Book
Book
0.1
Dud
Dvdl
Electrontc
Electronic
0.0
0.0
Nogotive
Postiro
Negative
Postive
(a)
(b)
Figure 5: Amazon Review dataset (a) Original Label Training Distribution; (b) Label-Shifted dis- tribution with sources tasks: Book, Dvd, Electronic, and target task Kitchen. We randomly drop 50% the negative reviews for all the source distribution while keeping the target label distribution unchanged.
M.2 DIGIT RECOGNITION
We follow the same settings of Ganin et al. (2016) and we use four-digit recognition datasets in the experiments MNIST, USPS, SVHN, and Synth. MNIST and USPS are the standard digits recogni- tion task. Street View House Number (SVHN) Ganin et al. 2016)) is the digit recognition dataset from house numbers in Google Street View Images. Synthetic Digits (Synth) Ganin et (2016) is synthetic dataset that by various transforming SVHN dataset.
We also visualize the label distribution these four datasets. The original datasets show an almost uniform label distribution the MNIST well as Synth, (showing in Fig. 7 (a)). In our paper, we generate label distribution drift on the source datasets for each multi-source transfer learning- Concretely, we drop 50% of the data digits 5-9 of all the sources while keep the target label distribution unchanged. (Fig. (b) illustrated one example with sources: Mnist, USPS, SVHN, and Target Synth. We drop the labels only the sources.)
MNIST and USPS images are resized to 32 X 32 and represented as 3-channel color images to match the shape of the other three datasets. Each domain has its given training and test sets when downloaded. Their respective training sample sizes are 60000, 7219, 73257, 479400, and the respective test sample sizes 10000, 2017, 26032, 9553.
27

---

Under review as a conference paper at ICLR 2021
The model structure is shown in Fig. 6 There is dropout and the hyperparameters chosen based on cross-vale vooss-validttinn. It trained for 60 epochs and the mini-batch size is 128 per domain. The optimizer is Adadelta with learning rate of 1.0. We adopted y 0.5 for MDAN and 7 0.1 for DARN in the baseline (Wen et al.) 2020).
Experimental Setting We use the Digits dataset for transfer learning scenarios (limited target labels and unsupervised DA). Notice the USPS data has only 7219 samples and the digits dataset relatively simple. We first randomly select 7K samples for each domain. We create a drifted distribution of each source, making each source ss 5300, and the target sample still 7K.
In the unsupervised DA, we use these labeled source tasks and unlabelled target task, which aims to predict the labels on the target domain.
In the transfer learning with limited data, we random sample only 10% dataset (=s 700 samples) as the target training set and the rest 90% samples as the target test set.
We select Co 0.01 and C1 as the maximum prediction loss C1 = max, R (h) as the hyper- parameters across these two scenarios. The maximum training epoch is 60.
1. Feature extractor: with 3 convolution layers.
'layerl': 'conv': [3, 3, 64], 'relu': []. "maxpool': [2, 0],
'layer2": 'conv': [3, 3, 128), 'relu': []. 'maxpool': [2, 2, 0],
'layer3'; 'conv': [3, 3, 256], 'relu': []. 'maxpool': [2, 2, 0},
2. Task prediction: with 3 fully connected layers.
'layerl': 'fe'; [*, 512], 'act.fn': 'relu',
'layer2'; 'fc': [512, 100), 'act_fn': 'relu',
'layer3': 'fc': [100, 10],
3. Domain Discriminator: with 2 fully connected layers.
reverse reverse_gracee gradient))
'layerl': 'fe': [ 256), 'act fn': 'relu',
'layer2': 'fe': [256, 1].
Figure 6: Neural Network Structure in the digits recognition Ganin et al. 2016
0.2
Label distribution: Mrist
0.2
Label distribution Mnist
0.1
0.1
Labe distrbitioon USps
Label distribution Usps
Proqucter Frequency
92
0.2
WHN
Label XVEN
02
0.2
Trequency
0.2
Synt
0.2
Labe oI Dielttttooo Synth
2
(a)
(b)
Figure 7: One example in Digits dataset with Sources: MNIST, USPS, SVHN and Target Synth. We randomly drop 50% data on digits 5-9 in all sources while keeping target label distribution unchanged.
M.3 OFFICE-HOME DATASET
To show the dataset in the complex scenarios, we use the challenging Office-Home dataset /Venkateswara et al.) 2017). It contains images of 65 objects such as spoon, sink, mug, and
28

---

Under review as a conference paper at ICLR 2021
pen from four different domains: Art (paintings, sketches, and/or artistic depictions), Clipart (cli- part images), Product (images without background), and Real- World (regular images captured with camera). One of the four datasets is chosen unlabelled target domain and the other three datasets are used as labeled source domains.
The dataset size is 2427 (Art), 4365 (Clipart), 4439 (Product), 4357 (Real-World). We follow the same training/test procedure as (Wen et al., (020). We additionally visualize the label distribution D(y) in four domains in Fig 9 which illustrated the inherent different label distributions. We did not re-sample the source label distribution to uniform distribution in the data pre-processing step. All the baselines are evaluated under the same setting.
We use the ResNet50 (He et al. 2 pretrained from the ImageNet in PyTorch as the base network for feature learning and put an MLP with the network structure shown in Fig. 10
Experimental Settings We use the original Office-Home dataset for two transfer learning scenar- ios (unsupervised DA and label-partial unsupervised DA). We use SGD optimizer with learning rate 0,005, momentum 0.9 and weight size is 32 per domain. As for the baselines. MDAN use y 1.0 while DARN use 7 y= 0.5. We select Co = 0.01 and C1 as the maximum prediction loss C1 = max, R (h) as the hyper-parameters across these two scenarios.
value Ic-3. It is trained for 100 epochs and the mani-batch
In the multi-source unsupervised partial DA, we randomly select 35 classes from the target (by repeating samplings), then at each sampling we run 5 times. The final result is based on these 3 x 5 = 15 repetitions.
Real Woeld Product Clipart
Spane
Sial
Mikg
Ikke
Kante
Keytaard
Glasa
Alam--3scct Dede-Larp
Laamer
Uhbor
Fan
Figure 8: Samples Images From Home dataset four domains with non-uniform label distribution.
Venkateswara et al. 2017). which consists
Label distribution: Art
Frequency
0.025
Danl In Ionnlllnn ml
0.000
Label distribution: Clipart
Frequency Frequency Frequency
0.025
0.000
Label distribution: Product
0.025
willlll ont Ju
0.000
Label distribution: RealWorld
0.025
0.000
10
20
30
40
60
Figure 9: Label distribution of Office- Dataset
N ANALYSIS ON THE PSEUDO-LABELS
29

---

Under review as a conference paper at ICLR 2021
1. Feature extractor: ResNet50 (He et al. 2016),
2. Task prediction: with 3 fully connected layers.
'layerl': 'fc": [ 256], 'batch normalization', "act fn': 'Leaky relu',
'layer2'; 'fc': [256, 256], bbatch normalizatiinn', 'act_nn': 'Leaky-relu',
'layer3'; 'fe': [256, 65].
3. Domain Discriminator: with 3 fully connected layers.
reverse gradient)
'layerl": 'fe': [ 256), 'batch saaichhorrmlizationi 'act_fn': 'Leaky_relut
'layer2': 'fe': [256, 256], natcchnormaliiztionn "act_fn': "Leaky_relut
'layer3': 'fe': [256, 1], 'Sigmoid'.
Figure 10: Neural Network Structure in the Office-Home
m
Ammm
Aotiracy ot Plaudo-aa
0.75
0.70
Accurny of PPouute-aaee
0.85
0.7
0,70
0.60
0.85
0.00
0.5
0.58
04
0:45
0.50
0.40
40
60
Epoch
Epoch
(a) Amazon
Review:
target
(b) Digits: target SVHIN
(c) Digits: target Synth
DVDs
Figure 11: Evolution of accuracy w.r.t. the predicted target pseudo-labels in different tasks in unsu- pervised DA.
O ANALYSIS IN UNSUPERVISED DA
O.1 ABLATION STUDY: DIFFERENT DROPPING RATE
To show the effectiveness of our proposed approach, we change the drop rate of the source domains, showing Fig. 12). We observe that task Book, DVD, Electronic, and Kitchen, the results are significantly better under large label-shift. In the initialization with almost no label shift, the state-of-the-art DARN illustrates a slightly better (< 1%) result.
O.2 ADDITIONAL ANALYSIS AMAZON DATASET
We present two additional results to illustrate the working principles of WADN, showing in Fig-
13 and (14).
We visualize the evolution approach to estimate A. We observe that in the source shifted data, DARN shows inconsistent estimator Fig. (3). This is different from the observation of Wen et al. 2020). We think it may in the conditional and label distribution shift problem, using fts (h(z)) + Discrepancy (S(r)) T(2)) to update À is unstable. In contrast, WADN illustrates relative consistent estimator of À under the source shifted data.
between DARN and WADN, which both used theoretical principled
In addition, WARN gradually and correctly estimates the unbalanced source data and assign higher wights a for label y = 0 (first row of Fig. These principles WADN jointly promote significantly better results.
30

---

Under review as a conference paper at ICLR 2021
80
DACK DAVR
82
DACK
78
80
DAVI
76-
74
74
72
70
68
06
10%
20%
30%
40%
50%
10%
20%
30%
40%
60%
Dropped
Dropped
(a)
Target: Book
(b) Target: DVD
Targe:: Electranis
Target: Kitchen
87
86
BANK
85
84
83
80
78
81
74
20%
40%
50%
10%
20%
Dropped
Dropped %
(c) Target: Electronics
(d) Target: Kitchen
Figure 12: Different label drift levels on Amazon Dataset. Larger dropping rate means higher label shift.
10
30
40
30
(a) DARN
(b) WADN
Figure 13: Source Shifted Amazon Dataset. Evolution of during the training. B=Books, D=DVD, E=Ellctronics, K=Kitchen.
O.3 ADDITIONAL ANALYSIS ON DIGITS DATASET
We show the evolution of â WADN, which verifies the correctness of our proposed principle. Since we drop digits 5-9 the source domains, the results Fig. (15) illustrate higher å on these digits.
PARTIAL MULTI-SOURCE UNSUPERVISED DA
From Fig. (66). WADN is consistently better than other baselines, given different selected classes.
31

---

Under review as a conference paper at ICLR 2021
target:
larget UVU
alpha D
alpht LJ
alpha E
alpha
aipha K
alpha K
(a) Target: Book
(b) Target: DVD
apha 5
aipha
alpha D3
appha
sipha K.
alphat E:
(c) Target:
Electronics
Target: Kitchen
Figure 14: Amazon Dataset. WADN approach: evolution of & during the training. Darker indicates
higher Value. Since we drop y the sources, then the true at (0) will be assigned with higher value.
32

---

Under review as a conference paper at ICLR 2021
Target
Target
alpha SVHN
alpha SYN
alphe USPS
alphe USPS
Target: MNIST
(b) Target: SVHN
USSS
alphe MNIST
alpha MNISTT
aipma SVHN
alpha SVHN
alpha USPS
aiph Synth
10
40
(c) Target: Synth
(d) Target: USPS
Figure 15: Digits Dataset. WADN approach: evolution of å, during the training. Darker indicates higher value. Since drop digits 5 on source domain, therefore, ozly), y [5.9] will be assigned with relative higher value.
Besides, when fewer classes are selected, the accuracy in DANN, PADA, and DARN is not drasti- cally dropping but maintaining relatively stable result. We think the following possible reasons:
The reported performances are based on the average of different selected sub-classes rather than one sub-class selection. From the statistical perspective, if take a close look at the variance, the results in DANN are much more unstable (higher std) inducing by the different samplings. Therefore, the conventional domain adversarial training is im- proper for handling the partial transfer since it is not reliable and negative transfer still occurs.
In multi-source DA, it is equally important to detect the non-overlapping classes and find the most similar sources. Comparing the basclines that only focus on one or two principles shows the importance of unified principles in multi-source partial DA.
We also observe that in the Real-World dataset, the DANN improves the performance by a relatively large value. This is due to the inherent difficultly of the learning task itself. In fact, the Real-World domain illustrates much higher performance compared with other domains. According to the Fano lower bound, a task with smaller classes is generally easy learn. possible the vanilla approach showed improvement but still with much higher variance.
Fig (77
showed the estimated å, with different selected classes. The results validate the 18
correctness of WADN in estimating the label distribution ratio.
33

---

Under review as a conference paper at ICLR 2021
ACC
201
15
25
25
Selected Classes (N]
* Seledted Classes (3)
SSlected Classes (R)
(a) Target: Art
(b) Target: Clipart
Target: Real-Woris
Figure 16: Multi-source Label Partial DA: Performance with different target selected classes.
34

---

Under review as a conference paper at ICLR 2021
MMM
M
MMA
MI
XX XXXXXXXXX X0000000000000
XXXX00XXXX XXXXXX
(a) Target: Art
MMh A.... mn.
M
M.M. M, M
NotwP!
M
WMw M X X XX XXXXXXXX x XXXXXXXXXXXXX X
XXXXXXXXX XXXXXX X XXXX
(b) Target: Clipart
M
M
MI
MMA
XX XXXXXXXX XXXXXXXXXXX
XXXXXXXXX XXXXXX XXXX
(c) Target: Product
Mmma
M
MMW
M
AM
MI
XX X0XX00XXX XXX0XXXXXXXXXXX
XXXXX0XXXX
(d) Target: Real-World
Figure 17: We select 15 classes and visualize estimated å (the bar plot). The "X" along the x-axis represents the index of dropped 50 classes. The red curves are the ground-truth label distribution ratio.
35

---

Under review as a conference paper at ICLR 2021
M
M
Amma Mr MMMM Ar Al
MM XXXX XXXX
X
(a) Target: Art
Am w MMM Ana
M All AAMIIA
IMA. Andhm
AnD AAL
AMM. AAMIlE
MM
M
IM
X XX XX XX
XXXX XXXX X X
XX X XXXXX
XXX
XX
(b) Target: Clipart
M AM
A AM
ATMM M AM
MiMMMmm Min.
X XX
XX
X XX
XXXX XXXX X X
XX X x
XXX XX
(c) Target: Product
M
Mm
M mal AM A MA
MM All M MMDA AMMil M XOXX
(d) Target: Rcal-Worle
Figure 18: We select 35 classes and visualize estimated á (the bar plot). The "X" along the x-axis represents the index of dropped 30 classes. The red curves are the ground-truth label distribution ratio.
36


Reviews:
[{"review_id": "qtneB3gwSc4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper is concerned with label shift in multi-source transfer learning setup. In particular authors look into target shift without assuming conditional distributions to be the same in source and target. They propose a unified frameworkthat can be used for learning with no target labels and limited target labels.\nThey show that the performance on target depends on how well we are able to estimate the ratio of label distrib between source and target, and the gap between real and estimated ratios,. It also depends on the weights assigned to each source tasks. Authors then proceed to show how to learn a model in various settings using the knowledge about the bounds.\n\nOverall, this paper CAN be really impactful, but as of now, it is really hard to follow and understand how it compares to other methods (not just empirically). \n1) The structure of the paper is strange, it is impossible to just read the paper and get enough information on any of the subsections,  everything is in the appendix. Related work is missing (see appendix), the theorem with bounds (that is used through the paper) appears without at least a sketch/idea of how authors got to it and where wasserstein distance comes from. Since it is not intuitive, and since it is not clear how these bounds compare to DANN bounds (https://arxiv.org/pdf/1505.07818.pdf or Ben-David bounds for that matter), it is really hard to believe in the algorithm authors derive, even though it seems to give good empirical improvements\nAlso K section in appendix is essential for understanding and should be moved to the main paper.\n2) Related work absolutely needs to be in the main paper It is impossible to understand what connection your work has with respect to established models like DANN and CDANN. Additionally Wassertein distance was used in a number of papers already,  https://arxiv.org/pdf/2009.02831.pdf, https://arxiv.org/pdf/1707.01217.pdf, https://arxiv.org/pdf/1909.08675.pdf. Again not clear on connection if any and comparison\n3) Experiments:\n- For DANN, did i understand correctly that you take all source tasks as one \"source\" domain and all the target as another domain. Why not to consider all the source tasks as separate domains and target as an additional domain, DANN allows for it easily (CE instead of log loss in adv head)\n- You should be comparing to CDAN https://arxiv.org/pdf/1705.10667.pdf which has been shown to better than DANN\n-  Also didn't understand your setup for experiments. For example on digits, what are your sources and what is your target task? A table shows only target, does it mean that source is all other digits datasets?\n\nMinor:\n- in Into, you mention \"without assuming that conditional distributions are the same\". I think it is most common to assume p(y|x) is the same, not p(x|y) as you show.\n- what exactly is \"label partial unsupervised domain adaptation\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vz7YRUC6haw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors focus on the label shift problem in multi-source transfer learning and derive new generic principles to control the target generalization risk. They propose a framework that unifies the principles of conditional feature alignment, label distribution ratio estimation, and domain relation weights estimation. A WADN algorithm is proposed for 3 multi-source label shift transfer scenarios:  learning with limited target data, unsupervised DA, and label partial unsupervised DA. The proposed WADN algorithm is validated on different scenarios on common benchmark datasets (Digits, HomeOffice, Amazon Review), and results indicate that it can outperform related SOTA methods for these scenarios. \n\n- Although the paper is well written, and all the key concepts are described in some detail, I found some parts of Section 2 difficult to follow.  The supplementary material in appendices provides much additional information for the reader: proofs, results, etc. In terms of the organization, the authors present their review and analysis of the SOTA literature in Appendix A (no in the main paper). Therefore, it is not immediately clear in the main paper how their framework and algorithm are motivated by challenges in literature.  \n\n- The experimental validation is limited in some respects. The authors do present averaged results over independent replications, using some cross-validation process.  There should be further analysis of the impact on the performance of growing: the number and size of the sources, the degree of shift, and diversity among source and target domains? The experimental section should be expanded to compare results on different backbone networks. Their model should be also compared with SOA methods in terms of time and/or memory complexity.\n\n- Tables 2-7 shows the lower-bound (source or source + tar), but does not show upper-bound results, like when training a DL model on the source and target data that are labeled (in the UDA scenario).  These tables also show DA results with an average value (last column). This is common in DA papers, but I still fail to see the point of averaging across different problems.\n\n- It seems like their code is not made available, so there is a concern that the results in this paper would be very difficult for a reader to reproduce.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MXvvgc92uP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper has made a good attempt to provide a unified approach for unsupervised domain adaptation. The proposed approach is applicable to three scenarios which have been traditionally  treated as three separate problems. The three problems that are treated in a unified way are Unsupervised Domain Adaptation (UDA), limited target labels and partially unsupervised domain adaptation. Another feature of the proposed approach is that it deals with target shift without assuming that conditional distributions are identical, a more realistic assumption for real-world problems.  Results on three different benchmark datasets are provided. Results show uniform improvements in the range of 2-6% over methods compared in various tables. The paper can be improved by providing comparisons with recent UDA and domain generalization methods from Balaji, Sankaranarayanan (CVPR 2018, NIPS 2018) and Balaji and Feizi (ICCV 2019). The ICCV paper also uses the Wasserstein distance for unsupervised domain adaptation. Given that one of the problems that is considered is UDA, I am not sure why the authors have not compared their approach on the Office dataset that is used in UDA papers. A recent paper from Saenko and Trevor Darrell [Saito, K., Kim, D., Sclaroff, S., Darrell, T., & Saenko, K. (2019). Semi-supervised Domain Adaptation via Minimax Entropy. arXiv preprint arXiv:1904.06487] has considered the problem of small source. small target and large unlabeled target data as a domain adaptation problem. Since the authors consider the limited target labels problem as a one of the cases, comparisons with this paper should also be provided.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DRYoj1Z2gyb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper aims to provide a unified principle for multi-source transfer learning under label shifts. Based on this principle, this paper claims that a unified algorithm is proposed for various multi-source label shift transfer scenarios: learning with limited target data, unsupervised domain adaptation and label partial unsupervised domain adaptation. The proposed algorithm is validated on three benchmark datasets. The proof seems correct via combining existing single-domain DA theory and the theory regarding Wasserstein distance. The main theorem (Theorem 1) assumes that we can get the label information in the target domain, which is not realistic in many DA problem settings (e.g., UDA or multi-source UDA in this paper). In many DA problem settings, we have to use pseudo labels to replace with true labels in the target domain, which should be analysed in the proposed theorem. However, this paper does not make any efforts to theoretically analyse the effect of pseudo labels, which results in that this paper has very limited impacts on the DA field. Besides, there are some misleading conclusions in this paper. \n\nPros:\n1.\tThe proposed theorems seem correct.\n2.\tThe proposed algorithm has good results after using the label-distribution ratio. The proposed method can address various DA problems and some machine learning problems. \n\nCons:\n1.\tThe main theorem (Theorem 1) assumes that we can get the label information in the target domain, which is not realistic in many DA problem settings (e.g., UDA or multi-source UDA in this paper).\n2.\tIn many DA problem settings, we have to use pseudo labels to replace with true labels in the target domain. If the quality of pseudo labels is low, can we still obtain good adaptation results in the target domain using the proposed method? If not, how does the quality of pseudo labels affects the performance of DA methods?\n3.\tIn Theorem 1, this paper claims that Comp() decreases with larger observation numbers. However, it is still unknown if the Rademacher complexity (regarding deep networks) will converge to zero when we have infinite observations. This means that the claim made in the main content is not true, which will mislead broader ICLR readers. This claim is true under a fixed hypothesis with finite VC dimension (as said at the top of Page 20), which is not linked with your deep-network-based algorithm.\n4.\tThe motivation of this paper is unclear. Why should we use theory based on Wasserstein distance? How about directly using pseudo labels (like SHOT in ICML20)? There are many questions regarding the motivation of this paper.\n5.\tThe proposed method can only be used to address three transfer scenarios (presented in this paper)? Are there difficulties to apply the proposed method to address other DA problems? The motivation (testing the proposed method in the presented three problem settings) is unclear. \n6.\tThe format of this paper is poor, which should be revised before submission.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The question the authors address is relevant and interesting mostly in the UDA setting. However, there exists several recent works that have 
highlighted the importance of label distribution ratio in DA (Wu et al., Combes et al. etc.), hence the main contribution of the
paper is to propose a novel analysis and results in the multi-source setting. That said, the paper has mixed reviews and
after going through the paper, the reviews and the discussion, I tend to agree with some of the reviewers that while 
the idea is interesting, the paper lacks in several points that makes it unsuitable to publication, for now.

Here are the main points leading to the decision.


A) UDA is usually the most frequent situation that occurs in domain adaptation and the most difficult to handle. 
The theoretical novelty of the bound comes only from the multi-source aspect that seems to be original

B) there is a strong contradiction in the paper. In the intro, they state that the paper addresses situations where conditional distributions differ. However in 4.2 they assume that they are finally equal.

In section 4.1, the authors show that for optimizing their problem, they need to have labels, mostly for estimating the class-conditional distributions. When these labels are available in the target domain, the problem is pretty simple and there exists many simple baselines that can handle this problem. However, in a UDA setting, they do not have label and proposes a method for estimation label proportion by assuming S_t(z|y) = T(z|y), which is in contradiction with their initial hypotheses S_t(z|y) != T(z|y). Hence under their assumption, the left hand side of Lemma 1 is zero and the equality is useless.
Hence, I would suggest the authors to avoid such a contradiction.

Under equality of S_t(z|y) = T(z|y), the approach proposed by the authors bears strong similarity with the work of Redko et al 2019 (cited in their paper). So I would highly to recommend them to compare with that algorithm. .


C) the authors use a lot a trick related to filtering, moving average.... I guess those parts is important for making the approach works and they are not properly analyzed.

D) The paper is  confusing in its writing and somehow this confusion makes the theoretical details hard to understand.
For instance, in section 3 the loss function is defined as having two variables but used one line after with only 1. In the theorem, it is not clear whether the true labelling function intervenes or how the y in h(x,y) is related to the true
labels. I guess a clarification is needed here for making the soundness of the theoretical results.
Decision: Reject