A Unified Domain Adaptation Framework with Distinctive Divergence Analysis
Authors: Zhiri YUAN
Xixu HU
Qi WU
Shumin MA
Cheuk Hang LEUNG
Xin Shen
Yiyan HUANG
Venue: Accepted by TMLR ()

Abstract: Unsupervised domain adaptation enables knowledge transfer from a labeled source domain to an unlabeled target domain by aligning the learnt features of both domains. The idea is theoretically supported by the generalization bound analysis in Ben-David et al. (2007), which specifies the applicable task (binary classification) and designates a specific distribution divergence measure. Although most distribution-aligning domain adaptation models seek theoretical grounds from this particular bound analysis, they do not actually fit into the stringent conditions. In this paper, we bridge the long-standing theoretical gap in literature by providing a unified generalization bound. Our analysis can well accommodate the classification/regression tasks and most commonly-used divergence measures, and more importantly, it can theoretically recover a large amount of previous models. In addition, we identify the key difference in the distribution divergence measures underlying the diverse models and commit a comprehensive in-depth comparison of the commonly-used divergence measures. Based on the unified generalization bound, we propose new domain adaptation models that achieve transferability through domain-invariant representations and conduct experiments on real-world datasets that corroborate our theoretical findings. We believe these insights are helpful in guiding the future design of distribution-aligning domain adaptation algorithms.

Published in Transactions on Machine Learning Research (12/2022)
A Unified Domain Adaptation Framework with Distinctive Divergence Analysis
Zhiri Yuan
ywanakiri201220maal.co com
School of Data Science, City University of Hong Kong
Xixu Hu
School of Data Science, City University of Hong Kong
Qi Wu
gi. aww@eityy.edu..h
School of Data Science, City University of Hong Kong
Shumin Ma
shemmnnm edu.cn Guangdong Provincial Key Laboratory of Interdisciplinary Research and Application for Data Science, BNU-HKBU
United International College
Cheuk Hang Leung
chleung877@cittn.eee
School of Data Science, City University of Hong Kong
Xin Shen
#shen&sse. culk. edu.hk Department of System Engineering and Engineering Management, The Chinese University of Hong Kong
Yiyan Huang
Jiyhuang3- cityu. edu.hk
School of Date Science, City University of Hong Kong
Reviewed on OpenReview: https: // openreview. forum? id=
Abstract
Unsupervised domain adaptation enables knowledge transfer from a labeled source domain to an unlabeled target domain by aligming the learnt features of both domains. The idea is theoretically supported by the generalization bound analysis in Ben-David al. (2007), which specifies the applicable task (binary classification) and designates specific distribu- tion divergence measure. Although most distribution-aligning domain adaptation models seek theoretical grounds from this particular bound analysis, they do not actually fit into the stringent conditions. In this paper, we bridge the long-standing theoretical gap in lit- erature by providing a unified generalization bound. Our analysis can well accommodate the coasiifiaation/rgreeesss
tasks and most commonly-used divergence measures, and more importantly, it can theoretically recover a large amount of previous models. In addition, we identify the key difference in the distribution divergence measures underlying the diverse models and commit comprehensive in-depth comparison of the commonly-used divergence measures. Based on the unified gemeralization bound, we propose new domain adaptation models that achieve transferability through domain-invariant representations and conduct experiments on real-world datasets that corroborate our theoretical findings. We believe these insights are helpful guiding the future design of distributionnaaigningg domain adap- tation algorithms.
1 Introduction
Domain adaptation (DA) originates from real-world situations where the source and target samples are drawn from different. distributions. In many applications, collecting full labels for the target domain is

---

Published in Transactions on Machine Learning Research (12/2022)
prohibitively labor-expensive and time-consuming. To correctly label the target domain, DA transfers the knowledge learned from the labeled source domain. Given that distributions of the two domains differ, a remedy to address the distributional difference is to do transfer learning by learning a Nomain-invariann and
task-spocific feature space BBen-David et al. 2010 Tzeng et al. 2014 Ganin et al. 2016 Saito et al. 2018
Xu et al. 2019 Zhang el. al. 2019).
To do so, most distributiin-allgning DA models base themselves on the theoretical analysis in Benn-avvd et al. (2007) and use the source error regularized by domain discrepancy the training objective function. However, the generalization bound given by [en--avvdd et al. (0077) only applies to the specific binary classification task and the H-divergence that measures the domain similarity. It theorctically restricts the development of DA models for more general multi-class classification and regression tasks (Tzeng et al. 2014 Ganin et al. 2016: Saito et al.) 2018 Xu et al. 2019 Sun & Saenko 2016). Moreover, to measure the domain discrepancy, various distribution divergence measures other than the H-divergence have been applied to DA models (Tzeng al.] 2014 Ganin et al., 2016 Saito et al. 2018 Xu et al. 2019 Long al. 2017). Thus, a gap exists between the proliferating DA models and the needed theoretical support. In this work, we bridge the theoretical gap by providing unified generalization bound, regardless of the task type and the divergence measures. Over the various divergence measurcs, we commit a comprchensive comparison and deliver helpful insights for guiding the future design of distribution-aligning DA algorithms. With our analysis framework, we recover the classical DA models in literature and derive new DA model variants. We empirically show that these model variants can help improve the transfer performance with experiments on classical DA datasets.
We summarize our contributions of this work follows. (1) We propose unified generalization bound that can easily adapt to both regression and classification tasks. Meanwhile, it can well accommodate the commonly-used divergence measures, including f-divergence, Wasserstein distance, MMD, etc. Thus, it greatly relaxes the stringent constraints imposed by the theoretical foundation in Ben-David et al. (2007). (2) Our analysis underpins and can help recover most classical distributiinnalignnnng DA models in literature. More importantly, new DA models can be derived based on our framework. With experimental studies, we show that these new models greatly enrich the model choices and can even improve the transfer performance. (3) We make comprehensive comparison of common discrepancics from the perspectives of norm topology, convergence rate and computation cost, which heuristically provides solid theoretical contributions to the DA community.
2 Related Work
It is a common belief in DA that minimizing the divergence between source and target domains and the prediction error on the source domain simultancously could improve the model performance on the target domain. Thus, a large number of previous works focus on constructing suitable regularizing terms to measure the divergence between domains (Redko et al.] 2020). Total variation commonly used to measure the difference between two probability functions, and to our best knowledge it is the first divergence function used in DA (Ben-David et al. 2010). MMD is also popular {Long et al. 2015) in that it can characterize more complicated relations wetween domains by mapping the input data. to a latent feature space. Wasserstein distance is another famous divergence candidate for its neat phisical meaning related to the optimal transport theory (Shen et al. 2018). Recently, many rescarchers come up with fancier divergences, which may be not natural at first glance but work well in terms of the model performance. For example, [Saioo et al. (2018) use the task-specific classifiers to construct a regularizer, and Xu al. (2019) try to align the classifiers of both domains to fixed scalar. In order to maintain the scale of learned features for regression tasks, Chen et al.) (2021) align the domains in terms of principal angles and the representation subspace distance. In general, there are numerous divergence functions for DA, but further discussion on why they could perform well in DA remains untouched in literature.
The theoretical gaps between the target error analysis and vast numbers of DA models in literature have been partially investigated. Ben--avvid et al.) (2007 2010) firstly prove that the target error could be bounded by source error and H-divergence or the total variation between domains. Later, Long et al. (2055) prove similar results for MMD. However, the theoretical results in Ben-David et al. (2007) 2010) and Long et al.
2

---

Published in Transactions on Machine Learning Research (12/2022)
2015) only hold for binary classification tasks. Acuna et al (2021) fix this gap for f-divergences and classification lasks, and Shen et al. (2018) generalize the result. for Wasserstein-1 distance and regression tasks. In general, it remains a question whether a specific divergence measure theoretically works in that roducing the corresponding divergence between domains could indeed enhance the model performance on the larget domain. In this work, we propose a unified generalization bound analysis framework to make up for the needed theoretical support in previous works.
The comparisons between divergence measures have been rarely investigated. Only few rescarchers explain the intuition why discrepancy measure would outperform another, especially for f-divergences (Sason & Verdú 2016). Nowozin et al. (2016) and Arjovsky et al. (2017) argue that MMD and Wasscrstein distance could outperform the total variation, but they only compare them from one single aspect. It is quite natural that one model could outperform or underperform others on different tasks, as we will see in the experiments. In this work, we make a more comprehensive comparison over which divergence measure would be beneficial.
3 Preliminaries
Let X and Y represent the input and output space, respectively. In unsupervised DA, we are given source domain D. = {(x], v.)) ki C x with N₂ labeled samples and target domain Dj = fx", J.m C X with N₂ unlabeled samples. It is assumed that the two domains share the same input space, but are characterized by different distributions on the input space. In this work, we use P to denote the distribution on the input space of the source domain and Q to denote that of the target domain. Let h1 he : X + y be the true mapping functions on the source domain D, and the target domain Dr respectively. A hypothesis is function h X - Y; and we denote the hypothesis set H C (h X x 3)- Tet L. y' x y IR+, be the loss function. Then the source error of a hypothesis h w.r.t. he under distribution P is defined as: [h h.) == ((p(L( (x)). Similarly, we can define the target error of a hypothesis h, namely, Co ha) := (x=x)(((xx), (x)). A DA task aims to learn suitable h E H. from the labeled source samples as well as the unlabeled target samples such that the target error Lo (h, he) is small.
Can a hysis that achieves the minimal source error generalize well on the larget domain? At first glance it sounds impossible in that the distributions characterizing the inputs differ and the mapping functions (h, and he) may also differ. With Theorem 1 Ben-David et al. (2007) verify that a hypothesis that can work well on the source domain can indeed generalize well on the target domain given that the two domains are similar. It thus theoretically underpins most unsupervised DA models that are later developed.
Theorem 1. BBn-David et al., 2007) Let H be a hypothesis set such that H {h : X X {0. 1}} and the v|. Let h" be the optimal hypothesis that achieves the minimum joint error on both the source and target domains: h²- argmin [Lr ha) + LQ hh)),, and let X denote the joint
loss function be
error of the optimal hypothesis h²: x := Lp(h", h.)) + Lo(ti, ha). Then, for any h E H, we have
Lo(h, ha) S Cp(h, h1) + (P. Q) +
(1)
Here, (P. is the H-divergence between distributions P and Q, and it measures the similarity of the two domains. For binary classification tasks where 7 = (h: X + {0,1]) it is defined as dx(P, Q) & 2 sup [hex))() - Exx@h(x)).
ReH
Theorem confirms the generalization bound in terms of the source error Lp(h, and the discrepancy between domains (P. Q). But two elements have been made fixed: the task (binary classification ion) and the measure of distributional difference (H-divergence). Real-world DA applications seldom fit strictly into the binary classification framework. To extend to more general settings like multi-class classification or even regression tasks, Mansour et al. (2009) take one step further to establish the generalization bound by relaxing the constraints over the hypothesis set H with the following Theorem 2
Theorem 2. MMnnsour et al.) 2009) Assume that the loss function L, is symmetric and obeys the triangle inequality. Define h2 == argmin Lp ho), and hī == argmin (2 ht). For any h E H. the following inequality holds,
NEW
hER
Lo(h, ha) & (r(h,h)) (P. + Cr(hi h;) L (r( hz).
(2)
3

---

Published in Transactions on Machine Learning Research (12/2022)
In Eq. (2. the divergence measure used, namely distz (P, Q), is the discrepancy distance. The discrepancy distance between two distributions P.Q over A is defined as
dist 2 (P,Q) Q) sup [Cp(h,h') Lo(h, A.VEH
Compared to the H-divergence, discrepancy distance is more general. The discrepaney distance is pretty flexible because its power measuring the distribution distance be controlled by the richness of the hypothesis set H. and loss functions L.
Although Mansour et al. 2009) remove the constraints over the hypothesis set H. to extend the generalization bound analysis to more realistic applications, we identify by looking into Eq. (2) that a hypothesis h can only generalize well on the target domain when the objective function Cp (g) ( )) is minimized. On one hand, compared to minimizing the true souree error Cp(h,,, in Eq. 1 minimizing Cp ht) results in suboptimal hypothesis because hī only an approximate to the true labeling function h, and is usually implicit in real-world applications. On the other hand, in practice, the discrepancy distance dist H (P, Q) is hard to implement in the deep neural networks. In all of the DA applications, practitioners mainly choose a specific divergence such as the -hanno- divergence, Wasserstein distance, MMD distance. etc. There exists a gap between the proliferating DA models that takes various divergence measures and the needed theoretical support to verify the effectiveness of the models. What's more, considering there are multiple divergence measure choices, the conundrum arises: what difference does make if a practitioner chooses MMD distance rather than Wasserstein distance? How to choose the best divergence measure? In this work, bridge the afforcmentioned theoretical gap by providing unified generalization bound, regardless of the task type and the divergence measures. In addition, provide the detailed comparison of various divergence measures in DA scenario and deliver helpful insights for guiding future design of DA algorithms.
Table 1: Popular f-divergences and corresponding f functions.
f-divergence
f(t) tlogt
f f-divergence
f(t)
Mallback-Leibler (KL)
Reverse KL
logt
Neyman x2
9-1
Pearson x²
8² -
Jensen-Shannow (JS)
flog t
(1 + t)
log
Ltt 2
Squared Hellinger (SH)
(Vt - 1)²
Total Variation (TV)
7 -
3.1 Divergence Measures between Distributions
Before proceeding to the detailed analysis of the generalization bound, we briefly introduce the divergence measures that have been commonly used in literature.
f-divergence (Sason Verdú 2016). Let f RR + R be a convex, semi-contimuous function satisfying f(1) = Let P be two distributions, satisfying that dP absolutely continuous w.r.t. dQ, and both are absolutely continuous w.r.t. a base measure dr. The f-divergence D, from P to is given by
dP Dy( f
Different. choices of f have been proposed in literature. For good review, see Pardo (018). the popular choices of the f f-divergence and their corresponding f functions.
Table
lists
Wasserstein distance (Villani 2009). The Wasserstein distance between distributions P and Q can be interpreted as the optimal (minimal) transportation cost of moving the mass from P into the mass of Q- Let p E toc) and d(.,-) be distance function of X. Let I(P, Q) denote the set of all possible joint distributions whose marginals are P, Q respectively. The Wasserstein-p distance between P and Q given by
Q₂(P.Q) A inf Id(x,, x d')
(EI((22))
In real applications, the case p 1 is of particular interest. Thus, we mainly consider the Wasserstein-1 distance (W1) in the following context.
4

---

Published in Transactions on Machine Learning Research (12/2022)
MMD (Sriperumbudur et al., 2009b). Given kernel function k X x X + R, let & be the feature map such that k(x, x²) = (b(r) o(a'), and He be the reproducing kernel Hilbert space (RKHS) associated with the kernel k. The MMD distance between distributions P and Q is:
MMDx - [[z=>(((a) - Ex-cz((x)
(3)
4 A Unified Generalization Bound
In this section, we propose unified generalization bound that can both accommodate the general DA tasks (classification/ /regressorreerssiioo) and the commonly-used distribution divergence measures. Based this unified bound, we provide the theoretical support for the DA models in the literature that lack the necessary theoretical analysis. In addition, with this generalization bound in hand, we are able to propose new DA models which replenish the current DA literature and greatly enrich the model choices. We will empirically verify the efficiency of these now models on benchmark DA datasets in Section G In this section, we begin with Theorem 3 where we prove that the larget error can actually be upper bounded by the true source error together with the domain difference plus a constant for any hypothesis set and any symmetric loss function that satisfies triangular inequality. The proof of theorems and propositions in the following context is deferred to Appendix A
Theorem 3. Assume the loss function L is symmetric and satisfies triangular incquality. For any hypothesis set H. il always holds that
Co(h, S Cp(h, h.) + dist (P.Q)+
(4)
At first glance, it seems that Eq. 4 is nothing different compared with Eq. 1 except the divergence term. There are actually improvements in our result. The unified bound in Theorem 3 does not only apply to the binary classification task. We relax the constraints on the loss function and impose no constraints on the hypothesis set, which improve the versatility of such an upper bound for any practical algorithm. Compared to Eq. (2) we move one step further to directly minimize the true source error (h, ht) rather than Lp hg), where estimating h first can amplify the source error and minimizing the corresponding Cr,h,, h3) will finally produce a suboptimal hypothesis. Hence our proposed Eq. (4) can produce better hypothesis to generalize well on the target domain.
Although we take the discrepancy distance as the divergence measure as in Eq. 2 we will not end here. We will show the connections between the hypothetical ? Q) and the commonly-used divergence measures, including Wasserstein distance, MMD and f-divergence. It's already known that Wasserstein-1 distance (W1) is special case of discrepancy distance (Shen et. al. 2018 Villani 2009 Sriperumbudur et al. 2009bb. In the following analysis, focus the other two divergence families and show that the discrepancy distance can actually recover the MMD distance (in Proposition 4 and partially recover the
vast f-divergence family (in Proposition (
Proposition 4. Let L(u, v) = ul and k be a kernel function on X. Let H consist of functions with norm no greater than 1 in RKHS associated with kernel ki then for any two distribations P, Q, dist 24 (P,Q) = 2MMD, (P, Q).
Lemma 5. [BenDavidd et al. 2010 Let L(u, v) = =lu-ol and H consists of functions on X with values in [0, 1], then for any two distributions P. Q, dist " (P.Q)= - (DQ)).P)vPPPO)
Proposition 6. For any f-divergence Dj satisfying that f is continuous and strictly conver, there exists a constant C > 0, such that for any two distributions P, Q, Drv(P,Q) S C,D,(P.Q).
Proposition 4 verifies that MMD distance can be recovered by the discrepancy distance. With Proposition 6 one can check that the f functions for commonly-used f-divergence (including KL, JS, SH, etc.) indeed fit into the continuous and strictly convex conditions. Thus, combining Lemma Proposition 6 with Theorem 3 we see that the inequality in Eq. 4 still holds for if the divergence measure is taken to be
these fddivergence. It thus confirms that the generalization bound in Theorem 3 can well accommodate the commonly-used f-divergence.
Compared with Theorem and 2 Theorem more general in that it provides theoretical support to most DA algorithms in literature. For example, if we take M(,, v) |u ol and let. H consist of all classifiers
5

---

Published in Transactions on Machine Learning Research (12/2022)
takings values in 10 (namely, binary classification task), then the divergence measure dist (P,Q) is equal to the H-divergence and we can recover the classical result in Theorem | With our approach, we can also theoretically recover the model proposed in Shen et al. (2018), where the authors prove the efficiency of Wasserstein distance measuring the distributional difference for classification tasks. We defer the detailed discussion over the connections with the classical models in the next subsection.
Theorem is also novel in that the underlying analysis framework can overcome the limitations of the generalization bound analysis in previous literature. For example, in Long et al. (2015), the authors prove that using multippe kernel MMD is efficient in classification tasks with the help of Parzen window classifier, which limits the analysis to only classification tasks thus regression tasks do not apply. With our analysis methodology, we can generalize the efficiency analysis to any universal kernel MMD and to both classification and regression tasks. Acuna et al. (2021) prove that f-divergence could be used to derive generalization bound for classifications under a pretty strong assumption on the hypothesis set, while we do not need additional assumptions.
The target error bound analysis that we propose actually still holds irrespective of the problem setting (covariate contt/condiiioall shift). The difference in solving these two kinds of problems lie in how to guide the algorithm design with the same theoretical target error bound. In the covariate shift setting where it's assumed that the condit ional distributions of Y|X between the source and target domains are the same, the third term X in the target error bound in Eq. (4) vanishes. Thus, it suffices to just learn the invariant representations while minimizing the source risk, as we have done in the experiments. In general settings where there can be conditional shift, it is not sufficient to simply align the marginal distributions while achieving small error on the source domain, since the third term X can not be neglected (as supported by Theorem 3 of [1] and the lower bound on the joint error of source and target domains in Theorem 4.3 of [2]). But we should also note that some researchers point out that the third term can be reduced to rather small value if the hypothesis space is rich enough ([4)), making it still possible for the model to adapat well to the target domain by only minimizing the domain discropancy and the source crror even if there is conditional distribution shift.
4.1 Connections with Classical Models
In Table 2 we list and summarize some classical DA models in literature. From the last column, see that most models are proposed without the necessary theoretical explanation on why it can generalize well on the target domain. In this section, we take three models (MCD (Saito et al. 2018), AFN (Xu et al., 2019) and DAN [Long et al., 2015)) ) as examples to illustrate how our theoretical analysis can help explain the underlying mechanism. Our analysis can also recover other benchmark models, which we will omit here.
4.1.1 Model MCD (Saito et al., 2018)
Saito et al. 2018) propose to align source and target features with the task-specific classifiers discrimina- for for multi-class classification task. Specifically, to incorporate the relationship between class boundaries and target samples, the authors train two discriminators hy and h2 to maximize the target features' discrep- ancy. Denote the number of classes by m, then the output space y = {0, 10m.. Let L(u, = -m Em lus - Uyl where u, v.V& E IR, then the training objective for the model MCD is
Lp(hi,h,) ) + Lr(h2, - (ha h2(r)),
(5)
Next, we show that minimizing the objective function in Eq. (5) is actually equivalent to minimizing the source error together with measuring the total variation distance between the two domains. Typically, we will prove that min -Ez-o (hz hz)))) returns the domain divergence 2 * Drv Q), given that h2 and ha can simultaneously perform well the source domain. Expressing the total variation distance in the form of discrepancy distance, we have
1 Drv(P.Q)
sup (en Es (hy
2 MTT
6

---

Published in Transactions on Machine Learning Research (12/2022)
Table 2: Selected models in the literature.
Literature
Model
Hypothesis set
Divergence measure
Theoretical analysis
et al. (2007
11: - {0,13}
H-divergence
Ben-David et al.
(2010)
h: + {0,1)}
Total variation
Tzeng et al. (2014)
DDC
h: + +(0, 1)
MMD
Long et al.
(2015)
DAN
h: + 10,1
MK-MMD
Sun Sacn ko
(2016)
CORAL
hh: - -0011
CORAL Loss
Ganin ct al.
(2016)
DANN
h4:2- + +00.17*
Proxy A-distance
Courty et al.
(2017)
JDOT
hh:X - > {0,1)
Wasserstein distance
Long ct al.
(017))
JAN
{h:X-1 -> > +0.1)
JMMD
Izeng et al.
(2017)
ADDA
{h:X- + +100+10110))
Long et al.
2018)
CDAN
11,0>10,100
Saito et al.
(0018)
MCD
1h:0~<0.1 (0.1))
-norm
Sankaranarayanai et al. (0018)
GTA
Shen et al.
(2018)
WDGRL
1 (0.1)
Wasserstein distance
Xu et al. (0199)
AFN
{h:X + {0.1)"}
MMFND
Zhang et al.
[2019)
MDD
(h:X-) + {0,1)"}
Disparity Discrepancy
Kang et al.
2020
CAN
[h:X {0.1)"3
CDD
Acuna et al.
(2021
f-DAL
(h:X' {0,1)*)
Df, discrepancy
By assuming that the classifiers h, h2 will work well on the source domain, the authors suggest that the term (h) (2())) is small enough to omit. Thus the following equivalence holds:
min Lp(h1, h.) + Cp(22, h.) - (L) h2(x))
.2
+ min (p(1, h.)) + Cr(12, 112) + sup h,,
ER (1),
+ min Lr h.)) + Lr(h2, h.)) + sup b1.
L
L
h1
+ min (p())))))) + Lp(h2, hs) + (P.Q).
bi.
4.1.2
Model 2: AFN (uu et al. 2019)
Xu et al. (019 point out that sometimes the hypothesis set 7 is so rich that the upper bound will greatly deviate from zero. To restrict H, the authors place a restrictive scalar to match the correspouding mean feature norms. With hyperparameter À E,, the objective function train the model AFN is
Cp h.) + XCC R) (h(h
(6)
Now we use our analysis methodology to theorctically prove that Eq. is actually an upper bound of the target error. Notice that
LQ h4) SCQ(R, + Lo(h, R)
LQ h4) + (h,, R) Cr, R) Lo(h, R) SCr(h, (.), &r (h, R) + (h R) LR he) Lr(R,h,).
Given that CQ(R, h2) and Lp(R, h2) are constants, the above calculations verify that Eq. does work generalization bound.
4.1.3 Model 3: DAN (Long et al. 2015)
Compared with Tzeng al. (2014) who firstly use the Gaussian kernel MMD between features of the last layer as the regularizer, the DAN model is novel in that the authors the Gausspln-Gaussian kernel function

---

Published in Transactions on Machine Learning Research (12/2022)
and consider the sum of MMD for features in miltiple layers. Now show that our analysis framework can incorporate the DAN model.
First, & Gausiinn kernel function is still a kernel function (Berlinet & Thomas Agnan 201 thus guarantee that using the malsiple the target error. By abusing the notations Long et al. (2015), we denote the feature space after I-th layer of the source domain (resp. target domain) by D. (resp. Dj). The objective function of DAN is
Theorem g and Proposition |
Gaussian kernel MMD can efficiently bound
Cp h2) + + 8
(MD D)),
1=
where the sum is taken through the squared MMDs between features of the 6th, 7th and 8th layer. Let ① D/, where ① is the direct sum, then it can be verified that MMD (Daum, Dium) 1=6
Daum ① D Drum
M D.). Thus by viewing D and D as the source and target domains, it is reasonable to take MM D to construct the thcoretical upper bound of the target domain, which mcans that the objective function in Long et al. (2055) can upper-bound the larget error.
5 Comparison of Divergence Measures
We have shown that the generalization bound in Eq. 4 can admit most of the distribution divergence measures, and such conclusion verified by the abundant literature in Table 2 natural question is, is there any difference on choosing another divergence measure? If so, will there be any guidance on how to choose the suitable divergence measure? Through this section, illustrate in detail the difference of the various divergence measures. In Section
5.3 we conclude that there is a tradeoff in the selection of a specific divergence measure, and thus we leave the second question for open discussion.
5.1 Topologies Induced by Metrics
When mcasuring the difference between distributions P and Q, we hope that the discrepancy mcasure could truly reflect the extent of the difference. Otherwise, it is unclear whether minimizing the domain discrepancy can indeed align the feature spaces. In this respect, total variation distance is unsatisfactory. Take an extreme example. Let X' = R, and Po = 50,6 E R. namely, the support of Po contains only the point (0). Then the total variation distance between Po and Po is given by
0, 0 0=
Drv Pa)
1, 0 #
The corresponding neural network model will fail to minimize Drv Po) w.r.t. initialized non- zero by backpropagation, making it impossible to align the two domains. However, Wasserstein distance and MMD are known to be able to get rid of this problem (Arjovsky et al. 2017 Li et al. 2017).
To illustrate the underlying mechanism, we look into the norm topologies induced by the metries. For space X' with a metric function d, the norm lopology т of (X, d) consists of the sets that could be written as the union of open balls B(x,r) = {x' E d(a,x') < r} with center x E X' and radius r E R4. If X' is known and fixed, we simply omit and say that 5 is induced by d. We say a topology Ti is courser than T2. denoted by тI C T2, any open ball in Ti also open in Tg. If T1 C To and T2 C 71, then we say тΙ and T2 are equivalent. Roughly speaking, the metric with coarser the following lemma, we could establish the link between the norm topologies and the corresponding metric functions which will facilitate the later analysis.
topology should perform better. With
Lemma 7. Given two metrics di, d2 on X, if there exist a, b E R4, such that di (z, x') aaxd2 x')h V X, then the norm topology Tı of (X, d1) is coarser than the norm topology T2 of (X, d2)..
It. worth noting that some members (although not all) of the f-divergence are metric functions, such total variation and the SH distance. Thus, the analysis over norm topology still applies these f-divergence.
8

---

Published in Transactions on Machine Learning Research (12/2022)
are able to compare the norm topologies induced by the distance measures in Section 3.1 Assume that the underlying space X' is compact with diameter M < cc. By Theorem 6.15 of Villani 22099, it holds that
With Lemma 7
Wi(P. Q) S M (Pv(..Q).
implying the topology induced by the Wasserstein- distance is coarser than the topology induced by the total variation. As for the MMD distance, for a kernel function :(::.) satisfying that C == supx,x ((z, x) < oo (e.g., a Gaussian kernel, or Laplacian kernel), by Theorem Sriperumbudur et al. (2009b), we have
&M S VC DTv(P.Q).
(8)
It also suggests coarser topology induced by the MMD distance than by the total variation. Thus, the conclusions above help explain why in many DA applications, Wasserstein and MMD) distance perform the best. But as we will see in the following analysis and experiments, it is not the norm topology that has the final say over the model performance. The implementation details of the domain divergence also significantly affect the model performance.
5.2 Convergence Rate
In this part, we compare the various divergence measures from the perspective of the convergence rates w.r.t. the sample size. In practice, the domain discrepancy between distributions P and Q is estimated by the finite samples from D, and D,. Thus, how the empirical distribution divergence dist (D., D,) converges to the true domain divergence ((,
H (P, Q) is vital to the success of the algorithm. Notice that for arbitrary
loss function L and hypothesis class H. it holds that
Idist ( D.) dist 7. (P. Q11 S dist (D., P) dist (Dr, Q)-
Thus, it is enough to analyze the convergence properties of the discrepancy between a typical distribution and its empirical distribution, namely, dist (D,, (or dD, Q)). The faster the empirical distribution converges to its undorlying distribution, the faster will the empirical domain difference converge to the truc domain divergence.
We introduce the convergence rate (CR) measure the convergence performance. Fix the underlying space X. Given a distribution and a set of realizations D {Ii,..., _N}, we denote by PN = Σm /2./N the empirical distribution of P. The convergence rate of probability distance metric D(..)) is said a, if for any distribution P on X and 8 e (0. there exists a constant C such that the following incquality holds with probability at least 1- 8,
D(PN, P) S CN-".
For notation simplicity, we denote distance metric D(--)) that has convergence rate a as CR(D) a. Obviously, for a probability metric D, the higher the convergence rate a, the faster D P) converges w.r.t. the sample size N. And thus, the corresponding empirical domain difference will converge to the true domain divergence.
In Proposition & we list the convergence rates total variation distance, Wasserstein-1 distance and MMD. We leave the discussion of Proposition to the next subsection (Section 5.3)
Proposition 8. Assume X' is of dimension d.
(1). For any f-divergence Dy that is indeed metric function (e.g., total variation, SH dislance, Pearson x² distance, etc.), CR(Dy) & }. In particular, CR(Drv) =
(2). Tf X is compact, then CR(W1) 3
(3). (Theorem and Corollary 10 in Sriperumbudur et al. 2009al) If the kernel function k charac- teristic and supaex x) < co, then CR(MMD&)

---

Published in Transactions on Machine Learning Research (12/2022)
Table 3: Comparison the divergence measures commonly used in literature.
f-divergences
Wasserstein distance
MMD
Computation cost Convergence rate Topology
Low S 2
Low y
High Coarse
Fine
Coarse
5.3 A Summary
In this part, we summarize the properties of commonly-used divergences and make a brief comparison between them.
[-divergence is large divergence lamily that covers various divergence measures. Though many researchers focus on only a few f-divergence, such as the total variation and JS divergence, there can be more divergence choices in the applications. The topology determined by f-divergence is rather fine. However, estimating the f-divergence with neural networks can be Lime-consuming and unstable, as the implementation needs
the adversarial networks (Zhang et al. 2019 Acuna et al. 2021
Wasscrstein distance enjoys a very concrete intuition and a direct connection to the optimal transport theory (Villani 2009). Compared with f-divergence, it has a weaker topology. Given that directly computing the Wasserstein-p distance for arbitrary p 2 challenging, researchers usually adopt the Wasserstein-1 distance. The Wasserstein-l distance in some cases limited in that it only measures the difference between the first-order moments of two distributions (Arjovsky et al. 2017).
MMD uses the kernel function to measure the domain discrepancy. Similar to the Wasserstein-1 distance, MMD has a weaker topology than f-divergence, thus minimizing the domain discrepancy with MMD can indeed align the feature space. However, unlike Wasserstein-1 distance, MMD (e.g. MMD with Gaussian kernel) could measure higher-order moments of two distributions (Li et al. 20170. More importantly, is easy implement in neural networks. In applications, instcad of fixing single kernel MMD, rescarchers have the freedom combine Gaussian MMDs with multiple bandwidths (Long et. al.] 2017 2015 2018 Li et al. 2017). However, evaluating the MMD distance encounters the computational cost that grows quadratically with sample size (Arjovsky et al. 2017). which limits the scalability of MMD distance.
We summarize the pros and cons of common divergences in Table 3 We observe that there is no consensus on which divergence measure can uniformly outperform its competitors. That may explain why works taking various divergence measures proliferate in the DA literature.
6 Experiments
In this section, we enrich the DA models by proposing new model variants that take divergence measures which are not covered in literature before, including total variation (TV), Neyman x², Pearson x² KL, Reverse KL, SH, JS and MMD with Laplacian kernel functions.
Let's first give a brief description over the proposed models. In the unsupervised DA setting, we are given N₂ labeled source samples {(x), N together with N, unlabeled samples (xf, N We'll first. train a feature extractor F(::0y) parameterized by 0, extract the features. We denote the features extracted Js (for source features) and Je (for largel. features). Namely, J = (F(x;;; { F = (F N The feature extractor followed by discriminator network D(-; 0s) parameterized by 0& to minimize the source error [((x), 0 0a). Here, L takes cross-entropy loss for classification problem and mean squared error for regression problem. Meanwhile, good transferability requires the alignment of domains. Thus, the feature extractor is required to extract the domain-invariant features, guaranteed by minimizing the distance between the features dist (F= (F) So the general objective for the proposed models is:
min (C((x, vi.:0,,00) + (F",) 8,,8a
10

---

Published in Transactions on Machine Learning Research (12/2022)
source
Discriminator
(x sh):
{xc In
Feature extractor
D(:;0a)
min { 0, 0a) + Adist (Fe, }1)} 0g
8()
{xc
NoVe
(F*
targel
Figure 1: The network flow of the proposed models. All samples, matter from the source domain (xx J (blue) or the target domain {x (green), are fed into a feature extractor Ff) to extract features that are both discriminative and donain-invariant. The discriminative features are achieved by training a discriminator D(:; 0s) to minimize the loss function L(:; f 0a), while the domain invariance is measured by dist L (F,, F().
H
where Ả is a positive hyperparameter to be tuned. For illustration purposes, we sketch the general network flow for the proposed models in Figure For the domain divergence measure dist in the proposed models, we take specific divergence measures mentioned in the last paragraph.
6.1 Image Classification
We test. the efficacy of the new DA model variants and compare with a variety of baselines on three datasets: Office-Home, Digits and Office-31. All experiments this section are run on Dell Precision 7920 with Intel Xcon@ Gold 6256 CPU at 3.6GHz, and a set of NVIDIA Quadro GV100 GPU cards.
OfficeHHome consists of 15,500 images in 65 object classes in office and home settings, forming four tremely dissimilar domains: Artistic images (Ar), Clipart images (CA), Product images (Pr), and Real- World images (Rw). For the Digits dataset, we investigate two digits dalasets MNIST and USPS with two transfer tasks (M- U and U--)). We train and evaluate all the methods following the splits and evaluation protocol from Long et al.) (2018), which divides MNIST into 60,000 training images and 10,000 lest images, and divides USPS into 7,291 training images and 2,007 test images, respectively. Office-31 is the most widely used dataset in DA literature. Tt consists of 4,652 images in 31 categories collected from three sources: Amazon (A), Webcam (W) and DSLR (D). We evaluate all methods on six transfer tasks A-W, D-W,, W-DD ADD, D W-A
Baseline methods. We compare with the three models discussed in Section 4.1 and the following classical and state-of-the-art models. JAN (Long et al.) 2017) and CAN (Kang et al. 2020 measure the domain discrepancy with variants of MMD. MDD (Zhang et al. 2019 adopts 7-JSD (a member of f-divergence), to align the two domains. f-DAL (Acuna et al. (022]) adopts a f-divergence variants of MDD to align the two domains. WDGRL (Shen et al. 2018) estimates the domain discrepancy by the Wasscrstein distance. DANN Gannin et al.) 2016) analyzes the domain divergence by adversarial neural networks.
The models we propose. We propose to multiple f-divergences as regularizers and test the cor- responding DA models' effects. Specifically, we use Squared Hellinger distance, Pearson x² distance, KL divergence, lotal variation, etc. We also test MMD with Laplacian kernel and compare with the Gaussian kernel. For the Office- and Office-31 experiments, ResNet-50 (He et al. 2016) pre-trained on the Ima- geNet (Deng et al. 2009: is adopted as the backbone feature extractor. For the Digits experiments, LeNet [LeCun al. 1998) serves as the backbone feature extractor. The classifiers are 2-layer neural networks with width 1024, 2048 and 500 correspondingyy. We use separate neural networks to approximate the divergences.
11

---

Published in Transactions on Machine Learning Research (12/2022)
For the Office datasets, adopt 3-layer neural networks with the same width as the classifiers. For the Digits dataset, use 2-layer neural network with width 500.
We follow the standard experiment protocol for unsupervised DA from Ganin et al. (2016) and report the average accuracy for each experiment as in Acuna et al. (2021). All algorithms are implemented in PyTorch. For models using f-divergences, we introduce domain critic inspired by (hhen et al. (2018) and Nowozin et al. (2016) estimate the f-divergence in a min-max manner. The domain critic is a 3-layer neural network with ReLU activation functions. We solve the min-max problem using gradient reversal layers motivated by Ganin et al. (2016). We take the mini-batch SGD optimizer with the Nestorov momentum 0.9 and the learning rates of the classifier are set to be 10 times to that of the feature extractor.
Table 4: Accuracy (%) on the Office-Home datasets in two subtables. The performance with the baseline models are recorded above the middle line and the models we propose below the line in each subtable.
Method
Ar-CI
Ar-Pr
Arw
CI-AAr
CI+Prr
Cl-RRw
Avg
ResNet-50
31.9
50.0
58.0
37.4
41.9
46.2
11.7
DAN(Gaussian)
46.3
66.8
74.2
57.5
63.1
66.8
62.5
DANN
45.6
59.3
70.1
47.0
58.5
60.9
56.9
JAN
45.9
61.2
68.9
50.4
59.7
61.0
57.9
MCD
53.2
73.8
77.7
62.8
68.8
70.7
67.8
AFN
52.0
71.7
76.3
64.2
69.9
71.9
67.7
MDD
54.9
73.7
77.8
60.0
71.4
71.8
68.3
WDGRL
48.2
61.5
73.0
54.6
64.0
63.9
60.9
CAN
50.3
71.7
76.0
69.9
70.3
66.5
f-DAL
53.7
71.2
76.3
60.2
68.4
69.0
66.6
DAN OAa(aallaa
45.7
66.5
74.5
57.0
62.4
66.5
62.1
TV
46.4
65.1
73.6
55.6
64.6
66.5
62.0
Neyman x²
50.0
66.8
75.6
56.5
66.5
67.9
63.9
Reverse KL
55.2
74.6
78.4
57.2
69.3
71.4
67.7
SH
55.0
73.3
77.1
60.4
69.1
70.7
67.6
JS
53.5
72.9
76.5
58.8
67.3
68.6
66.3
KL
55.7
73.4
77.9
60.0
70.8
72.5
68.4
Pearson x²
55.5
69.4
78.7
61.1
73.5
72.9
68.5
Method
Pr +Ar
Pr-+CI
Pr-
RAr
Rw-+C1
Rw PPr
Avg
ResNet-50
38.5
31.2
60.4
53.9
41.2
59.9
47.5
DAN(Gaussian)
53.6
74.2
67.2
78.3
60.6
DANN
46.1
43.7
68.5
63.2
51.8
76.8
58.4
JAN
45.8
43.4
70.3
63.9
52.4
76.8
58.8
MCD
59.5
50.5
77.9
74.3
59.1
82.5
67.3
AFN
63.7
51.4
77.1
70.9
57.1
81.5
67.0
MDD
61.2
53.6
78.1
72.5
60.2
82.3
68.0
WDGRL
54.3
51.2
72.9
67.9
58.9
78.1
63.9
CAN
59.0
49.7
76.2
68.5
58.5
81.8
65.6
J DAL
60.2
52.6
76.9
71.4
59.0
80.7
67.0
DaN(Taplacian)
53.4
40.7
73.3
66.7
48.5
78.1
60.1
TV
53.3
73.0
67.5
79.5
63.4
Neyman x2
56.0
53.0
75.3
70.4
57.8
80.4
65.5
Reverse KL
60.7
47.0
79.7
72.2
58.6
82.2
66.7
SH
62.5
52.9
77.7
72.3
58.0
81.4
67.5
JS
61.7
52.1
76.3
71.5
57.2
80.8
66.6
KL
62.7
53.4
78.6
72.7
59.2
82.7
68.2
Pearson X2
63.0
54.3
80.0
73.9
60.6
83.6
69.2
We record the accuracy results on the Office-Home tasks Table 4 the results on the Digits tasks in Table
and the results the Office-31 transfer tasks in Table 0 We divide each table with horizontal line to
12

---

Published in Transactions on Machine Learning Research (12/2022)
Table 5; Accuracy (%)
on the Digits datasets.
Method
M-+U
U-+M
Avg
DAN (Gaussian)
88.9
87.6
88.3
DANN
91.6
95.9
93.8
JAN
84.8
90.7
87.7
MDD
93.6
97.3
95.5
MCD
94.7
96.7
95.7
AFN
89.9
96.3
93.1
WDGRL
97.4
94.1
95.8
CAN
90.9
94.1
92.5
D-DAL
91.7
95.9
93.8
DAN (Laplacian)
90.1
90.1
90.1
TV
91.3
96.6
94.0
Neyman x2
92.4
95.3
93.9
Pearson X²
92.0
95.4
93.7
KL
93.0
95.8
94.4
Reverse KT.
93.3
96.5
94.9
SH
94.8
97.3
96.1
JS
93.5
96.6
95.1
Table 6: Accuracy in (%) on the Office-31 datasets.
Method
A W
+ W
W +
A A-D D
D - A
W - A
ResNet-50
68.4 0.2
96.7 + 0.1
99.3 + 0.1
68.9 + 0.2
62.5 0.3
60.7 + 0.3
(Gaasssi))
84.3 0.8
98.5 1 0.4
100.0 0.0
86.1 0.6
65.6 1 0.5
64.4 0.4
DANN
82.0 0.4
96.9 0.2
99.1 0.1
79.7 0.4
68.2 0.4
67.4 0.5
JAN
85.4 0.3
97.4 0.2
99.8 0.2
84.7 0.3
68.6 + 0.3
70.0 + 0.4
MCD
88.6 + 0.2
98.5 0.1
1.0
92.2 0.2
69.5 0.1
69.7 0.3
AFN
92.5 = 0.7
98.9 + 0.2
100.0
94.8 0.3
72.7 0.4
0
MDD
94.5 = 0.3
98.4 0.1
100.0
9 0.2
74.6 + 0.3
0.1
WDGRL
92.8 = 0.3
95.6 + 0.5
99.3 + 0.3
88.2 0.2
69.9 + 1.4
CAN
93.2 0.2
98.4 0.2
99.8 + 0.2
92.9 + 0.2
76.5 + 0.3
76.0 0.3
f-DAL
95.4 + 0.7
98.8 + 0.1
100.0 0.0
93.8 + 0.4
74.9 1.5
74.2 + 0.5
DAN(Laplacian)
83.0 + 0.8
98.6 + 0.3
100.0 + .0
85.4 + 0.6
65.6 + 0.7
64.0 + 0.3
TV
91.2 + 1.8
98.3 + 0.1
100.0 + .0
89.3 + 1.0
68.144 0.9
69.4 + 1.3
Neyman
88.9 0.9
98.0 0.1
100.0 .0
89.8 + 0.8
72.7 1.3
Pearson X
92.5 + 0.5
98.3 0.2
100.0 + .
87.7 0.4
72.3 0.6
74.4 : 0.2
KL
91.1 0.7
98.4 0.1
100.0 + 0
88.1 0.7
75.0
0.00
74.3 + 0.1
Reverse KL
91.6 0.3
98.6 0.1
100.0 - 0
88.6 0.8
75.1 0.1
74.0 - 0.4
SH
91.6 = 0.7
98.6 0.1
100.0 + 0
89.9 - 0.8
75.0 + 0.1
74.1 + 0.4
JS
91.8 + 1.1
98.6 + 0.1
100.0 = .0
90.1 + 0.4
76.1 + 0.5
74.0 + 0.4
separately record the baseline models' performance (above the line with the model name) and our proposed models" performance (below the line with the name of the divergence measure used). We highlight the corresponding highest accuracy among the bascline models and among the models we proposc.
From the 3 tables, we make the following observations:
(1) The framework we propose greatly enriches the transfer model choices. Furthermore, some models can even beat the SOTA models. For example, in Table 4 the model with Pearson x2 distance achieves the highest average accuracy score among all the DA models. Tn Table the model using SH distance performs
13

---

Published in Transactions on Machine Learning Research (12/2022)
the best average. In Table c the model using JS divergence achieves the highest accuracy in 4 out of 6 tasks among all the models we propose. Such observations explicitly reveal the great potential of the f-divergence family on transfer learning tasks, which fill in the blanks in literature.
(2) But we should also notice that in terms of each specific task in Table 4 there does not exist a typical distribution divergence measure that can universally outperform. Besides, for some specific tasks (such as Ar + CI, Ar + Pr in Table 4 other models rather than the model using Pearson x² perform better. In addition, there are also tasks in Table [ where we fail to beat the bascline models. That reminds us to be prudent in selecting suitable DA model.
(3) Furthermore, DAN with Laplacian kernel MMD performs similarly to DAN with Gaussian kernel MMD, and outperforms the latter one on the Digits dataset in Table 5 It suggests that a wider choice of the kernel functions may be helpful, given that many previous works adopting MMD (Long et al. 2017 only consider the Gaussian kernel MMD.
2015
2018
(4) We visualize the features learnt by the models using JS, SH and the other 7 baselines on Office-31 in Figure 2 in the Appendix. It shows that the model with domain similarity measured by JS separates inputs of different classes the best among all. That explains why the model with JS performs pretty well in Table
6.2 Image Regression
In addition to the classification lasks, we also test the efficacy of the new DA model variants on two image regression datasets: MPI3D and dSprites. MPI3D a simulation-to-real dataset of 3D objects, consisting of three domains: Toy (T), RealistiC (RC) and ReaL (RL). Each domain contains 1,036,800 images, with every image having two continuous factors of variations to be predicted: rotation about a vertical axis at the base and second rotation about horizontal axis. We evaluate all methods on six transfer tasks: RL-RC, RL-T,, RC-T, RC-RL, T-RL, and TCRC. For each task, we separately predict the two rotation angles and record the sum of mean absolute error (MAE) over the two predictions as the transfer performance of this task. dSprites is a 2D synthetic dataset composed of three domains: Color (C), Noisy (N) and Scream (S). Each domain consists of 737,280 images with three variants of continuous factors to be predicted: scale, position X, and position Y. We evaluate all methods on transfer tasks: C-N, C-+, N-*C, N-S, S-CC, S-N. Similar to the experiment on the previous dataset, we record the sum of MAE over the three predictions in each transfer task as the transfer performance. All experiments in this section are run with the same device as in the previous section.
We compare with the following DA models that can be applied to regression problems: DAN, DANN, MCD, AFN and JDOT (Courty et al. 2017). For the MPI3D and dSprites experiments, ResNet-18 He et al. 2016) pretrained on ImageNet (Deng et al. 2009) is adopted as the backbone feature extractor. The regressors are a 2-layer convolutional neural network with BatchNorm and ReLU as the activation function, followed by an average pooling layer and a l-layer neural network with width 1024. For models using f fidivergenees, we introduce domain critic inspired by (Nowozin et al., 2016 to estimate the f divivergence in min-max manner. The domain critic is a 2-layer convolutional neural network followed by a 1-layer neural network with ReLU as activation functions. We solve the min-max problem using gradient reversal layers motivated by (Ganin et al. 2016). We take the mini-batch SGD optimizer with the Nestorov momentum 0.95 and the learning rates of the regressors are set to be 10 times that of the feature extractor.
We follow the standard experimental protocol for unsupervised DA and report the MAE on the two datascts in Table 7 and 8 As in the previous lables, the baseline models' performances are recorded above the line with the model name and our proposed models' performances are below the middle line with the name of the divergence measure used. We highlight the corresponding lowest MAE among the baselines and among the models we propose.
From Table 7 andd [ we can see that the framework we propose enables us to design more domain adaptation regression models with better performance. For example, in Table all the proposed models reduce the MAE by a large margin with Total Variation divergence performing the best. In Table S our proposed models also greatly improve the regression performance by lowering the MAF nearly by half. These all demonstrate
14

---

Published in Transactions on Machine Learning Research (12/2022)
the powerful transferability of the proposed model guaranteed by unified theoretical framework. We should also notice that the performance of different. /-divergences varies across datasets. For example, the Total Variation divergence performs poorly in the previous two classification tasks, compared with other f f-divergences like Pearson x2. However, the Total Variation divergence achieves the best results in the MPI3D regression lasks. But for the second regression dataset dSprites, the KL. divergence and the MMD method with Laplacian kernel achieve the best. These inspire us to try as many as possible divergences the pilot study period determine which is the best for our specific task. These could bring significant improvement to the performance. And luckily, theoretical framework provides generalization guarantee for large variety of divergences, which greatly enriches people's choices.
Table MAE the MPI3D dataset.
Method
RI-+RC
RL-TT
RC-RL
RC-T
T-
T-RRC
Avg
ResNet-18
0.17 + 0.02
0.44 0.04
0.19 + 0.02
0.45 + 0.03
0.51 0.01
0.50 + 0.03
0.377
DAN DAN(Caussian)
0.12 + 0.03
0.35 + 0.02
0.12 + 0.02
0.27 1 0.02
0.40 + 0.02
0.41 0.04
0.278
DANN
0.09 0.01
0.24 = 0.04
0.11 0.03
0.41 + 0.03 0.03
0.48 0.02
0.37 = 0.04
0.283
MCD
0.13 = 0.02
0.40 + 0.04
0.15 0.02
0.45 0.01
0.52 0.02
0.50 =50=00 0.03
0.358
AFN
0.18 = 0.03
0.45 + 0.02
0.20 + 0.03
0.46 + 0.03
0.53 + 0.02
0.52 0.04
0.390
JDOT
0.16 = 0.02
0.41 0.01
0.16 + 0.02
0.41 0.02
0.47 + 0.02
0.47 0.02
0.353
(AN(Taplacian)
0.11 + 0.04
0.21 0.01
0.09 + 0.01
0.19 0.01
0.26 0.01
0.23 + 0.02
0.182
TV
0.06 + 0.00
0.18 = 0.01
0.07 = 0.02
0.13 + 0.02
0.18 + 0.01
0.16 + 0.01
0.132
Neyman X²
0.09 = 0.02
0.19 0.00
0.00
0.09 0.01
0.18 0.01
0.19 = 0.00
0.19 0.00
0.157
Reverse KL.
0.10 + 0.00
0.20 + 0.00
0.09 + 0.00
0.18
0.20
0.161
SH
0.11 + 0.00
0.21 0.01
0.10 1 0.01
0.21 0.00
0.21
0.176
JS
0.09 = 0.00
0.20 0.01
0.01
0.09 0.01
0.17
0.20
0.19
0.156
KL
0.10 = 0.01
0.19 0.00
0.10 + 0.01
0.18 0.01
0.00
0.20
0.161
Pearson x²
0.09 = 0.02
0.19 + 0.00
0.08 + 0.00
0.17 + 0.01
0.19 0.00
0.18 = 0.00
0.149
Table 8: MAE on the dSprites dataset.
Method
C-N
C-S
N-C
N-+
S+C
S-N
Avg
ResNet-18
0.94 + 0.06
0.90 + 0.08
0.16 + 0.02
0.65 + 0.02
0.08 + 0.01
0.26 + 0.03
0.498
DAN(Ganssian)
0.70 + 0.05
0.77 0.09
0.12 0.03
0.50 + 0.05
0.06 + 0.02
0.11 + 0.04
0.377
DANN
0.47 + 0.07
0.46 = 0.07
0.16 + 0.02
0.65 + 0.05
0.05 0.00
0.10 + 0.01
0.315
MCD
0.81 = 0.09
0.81 0.12
0.17 : 0.12
0.65 =
0.07 0.02
0.19
0.450
AFN
1.00 + 0.04
0.96 +: 0.05
0.16 1: 0.03
0.62 0.04
0.08 1 0.01
1.00
0.523
JDOT
0.86 + 0.03
0.79 0.02
0.19 1 0.02
0.64
0.10 0.02
0.23 0.04
0.468
DAN (Aa(llpla)ian)
0.18 0.01
0.28 + 0.00
0.08 + 0.00
0.23 + 0.01
0.05 + 0.00
0.07 + 0.01
0.151
TV
0.22 + 0.00
0.25 0.01
0.08 + 0.01
0.25 0.00
0.06 + 0.01
0.18 0.00
0.171
Neyman x²
0.22 + 0.01
0.25 + 0.00
0.09 + 0.02
0.24 0.01
0.08 + 0.01
0.10 1 0.02
0.164
Reverse KT.
0.22 + 0.01
0.25 0.01
0.10 0.02
0.22 + 0.04
0.10 + 0.04
0.08 0.02
0.161
SH
0.22 + 0.00
0.21 + 0.02
0.10 + 0.02
+ 0.02
0.06 + 0.00
0.11 0.01
0.160
JS
0.22 = 0.00
0.24 0.01
0.08 + 0.00
0.23 0.02
0.05 0.00
0.11
0.155
KL
0.22 = 0.00
0.23 = 0.03
0.03
0.07 + 0.01
0.07 + 0.01
0.09 1 0.02
0.152
Pearson X"
0.23 = 0.03
0.25 0.00
0.08 - 0.01
0.23 0.03
0.11 = 0.03
0.11 0.00
0.168
7 Conclusion
In this work, we focus on the gemeralization bound for unsupervised domain adaplation (DA) problems. We provide a unified generalization bound to bridge the theoretical gap between the stringent generalization bound analysis in the pioncering work in Ben-David et al. (2007) and the diverse distribution-alignnng DA models in literature. This unified bound can well accommodate most commonly-used divergence measures for both classification and regression tasks, and fully recover large amount of previous models. We further
15

---

Published in Transactions on Machine Learning Research (12/2022)
provide a comprehensive comparison of the commonly-used divergence measures in terms of the norm topol- ogy; convergence rate and computation cost. In addition, we propose new DA models that are theoretically guaranteed by the unified generalization bound. We empirically show that a wider range of divergence mea- sures could greatly enrich the model choices and improve the model performance. Our work thus provides theoretical support to many previous models that adopt specific divergence measures and sheds lights on future studies of DA.
Acknowledgements
We thank the anonymous reviewers from TMLR and our Action Editor for their constructive feedback and thorough suggestions for improvements.
Qi Wu acknowledges the support from the Hong Kong Research Grants Council [Genoral Research Fund 142061 17, 11219420, and 11200219], CityU SRG-Fd fund 7005300, and the support from the CityU-JD Digits Laboratory in Financial Technology and Engineering, HK Institute of Data Science. The work described in this paper was partially supported by the InnoHK initiative, The Government of the HKSAR, and the Laboratory for Al-Powered Financial Technologies. Shumin Ma acknowledges the support from: Guangdong Provincial Key Taboratory of Interdisciplinary Research and Application for Data Science, BNU-HKBU United International College (2022B12120100)
Guangdong Higher Education Upgrading Plan (2021-2025) of 'Rushing to the Top, Making Up Shortcomings and Strengthening Special Features' with UIC research grant (R0400001-22)) and UTC ()
References
David Acuna, Guojun Zhang, Marc Teva Law, and Sanja Fidler. f-domain-adversarial learning: Theory and algorithms. In JCML, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Botiou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223, 2017.
Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
Shai Ben-Davice John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151- 175, 2010.
Alain Berlinet and Christine Thomas-Agnan Reproducing kernel Hilbert spaces in probability and statistics. Springer Science Business Media, 2011.
Xinyang Chen, Sinan Wang, Jianmin Wang, and Mingsheng Long. Representation subspace distance for domain adaptation regression. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Interna- tional Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1749 1759, 18 24 Jul 2021.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Proceedings of the 31st International Conference on Neural In- formation Processing Systems, NIPS'17, pp. 3733-333 3742, Red Hook, NY, USA, 2017. ISBN 9781510860964.
Imre Csiszár. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica, 2(1-4):191-25) 193,13,, 1972.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability Theory and Related Fields, 162 2015.
16

---

Published in Transactions on Machine Learning Research (12/2022)
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aamain-adversarian training of neural networks. The journal of machine learning research, 17(1):2096 2030, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 770 778, 2016. doi: (0.110/CCVPR.20
Guoliang Kang, Lu Jiang, Yunchao Wei, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation net- work for single-and multi-source domain adaptation. 1EEE transactions on pattern analysis and machine intelligence, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabés Póczos. Mid gan: Towards deeper understanding of moment matching network. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS117, pp. 2200 2210, 2017. ISBN 9781510860964.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, pp. 97-105, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adap- tation networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 2208-2217, 2017.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, volume 31, 2018.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT, 2009.
Sebastian Nowazin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers us- ing variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 271-279, 2016.
Leandro Pardo. Statistical inference based on divergence measures. 2018.
levgen Rodko, Emilic Morvant, Amaury Habrard, Marc Sebban, and Younès Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees. arXie preprinl arXiv:2004.1183 2020.
Paul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and Ilya O Tolstikhin. Practical and consistent estimation of f-divergences. Advances in Neural Information Processing Systems, 32:4070-4080. 2019.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognilion, pp. 3723 37332, 2018.
Swami Sankaranarayanan,, Yogesh Balaji, Carlos Domingo Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8503 8512, 2018.
Igal Sason and Sergio Verdú. f-divergence incqualities. IEEE Transactions on Information Theory, 62(11): 5973 6006, 2016.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAT, 2018.
17

---

Published in Transactions on Machine Learning Research (12/2022)
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Gert RG Lanckrict, and Bernhard Schölkopf. Kernel choice and classifiability for rkhs embeddings of probability distributions. In NTPS, volume 22, pp. 1750 1758, 2009a.
Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert R. G. Lanckriet. On integral probability metrics, d-divergences and binary classification, 2009b.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. European conference on computer vision, pp. 443 450, 2016.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Sacnko, and Trevor Darrell. Deep domain confusion: Maxi- mizing for domain invariance. arXiv preprint arXiv:1412.3477, 2014.
Eric Tzeng, Judy Hoffman, Kate Sacnko, and Trevor Darrell. Adversarial discriminative domain adaptation.
In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 7167-7176, 2017.
Cédric Villani. Optimal transport: old and new, volume 338. 2009.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1426-14- 1435, 2019.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413, 2019.
A Appendix
A.1 Proof
Proof. (Theorem
ho(h, < Co(h, Lo(h",
s Lp(h, 11) Lp(h, ht) - Lo(h, h1") +er
Lp(h, h2) Lp(h., h1) dist (P. Q) ter
= Lp(h, ho) + dist (P. + er.
7
Proof. (Proposition Given the kernel function k, by the description of maximum mean discrepancy in Sriperumbudur et al. 2009b) MMD; could be equivalently defined
MMD (P,D)(P.)) Q) = sup [m~r(h(z)) (h(r))
hcH
Note that wh, / E H. h N is function in RKHS. Further, have
|| Il ,z I || + ||''l 2,
namely that h - h' is an element in RKHS with norm no greater than 2. Since there is an explicit bi-jection between H and the H. == (h E RKHS |/ S 2), given by + 2h, we have
dist (P.Q)
sup 6.
())p)/(()) 1 n((x)) E. (2()))
=2 sup [z-Ph(x) -(argh)) k
= 2MMD
Thus could conclude that in this dist / is two times the MMD
18

---

Published in Transactions on Machine Learning Research (12/2022)
Proof. (Proposition 6 We will refer to the proof of Theorem 3.1 in Csiszár (9722). Let 1, W6m wy
1, II (Q), then in this case, Q" = Q. Let 2, K 2, and let f(t) tf()). which is also continuous for >> 0 and almost everywhere differentiable. Let 7(() (1) at each point t where f(() is differentiable. Then,
as(uo) = min + 6) f(vo) (w)) f(vo - 6) 50(u))) > 0
for ug E (6, K] is continuous, and og is positive and bounded in (ő, K]. Thus
1 min as(uo) > 65ugSk
6&
is a positive finite constant depending on f. Here, we emphasize that the above constants wo, 6, 6, are independent Q. Then for any distribution P, by Equation (3.25) of Csiszár (1922.
Drv(P.Q) S 66, 0lw0 Q))..Q).
Proof. (Lemma
7 Given any open ball
B = B(oo, r), 20 E Xr R++
in topology Ti, in order to show that B also open in T2, we need to show that for any 21 B, there an open ball in T2 that inside and contains 21. Indeed, for any 21 E B, denote 8 d1 (zo, T1), then Consider the open ball B' {a: E X d2(x1, x) < ( It is direct that for any E B', we have
di (1 2) s d1 (zo. 21) (2
S + ad2 (x) z)
<5+ax a
Thus B' C B, hence B is open in 72. This shows that T2.
Proof. (Proposition Fix an integer N. Note that when dist takes the form of the total variation Wasserstein-1 distance, dist is bounded. Thus dist (PN, (PN, P) function of 21,..., EN satisfics the bounded difference property. Indeed, for any (#1, IN), (#f,..., E X that only differs in the i-th coordinate (namely, zy x for all j + i), we denote their empirical distributions by PN. Py respectively, there exists such that
Idist diist((Nv (PN, P) dist 21 P\ s (PN, (PN, M
N
By McDiarmid's inequality, with probability at least 1 8, we shall have
dist (PN. (PN. P) Ep, dist # (PN P) + M
log 8'
2N
Thus we only need to figure out that how large E. dist H (PN: P) is. If there exists a constant, and a constant C such that
Ep, dist (PN, CN
then the convergence rate will be min B). For the Wasserstein-1 distance, by Theorem in Fournier & Guillin 2015), there exists a constant such that
Epp Wi &)
For the total variation distance, by Theorem of Rubenstein al. 2019 there exists a constant such that
Ep, Dry (PN. P) $ !
19

---

Published in Transactions on Machine Learning Research (12/2022)
Furthermore, by Proposition 6 for any f-divergence metric Dy satisfying the assumptions, shall have
Drv(P.Q) scDD((PQQ.
Thus Epy (y(P P) must converge to 0 with rate O(N-a) with a 1.2 Otherwise, C2D,(PN, P) would be smaller than Drv (Px, P) when N is sufficiently large, which contradicts Proposition 6 Hence for such metric Dy, its convergence rate is no greater than 2'
A.2 Supplementary figures
20

---

Published in Transactions on Machine Learning Research (12/2022)
(n) JS
(b) SH
(c) WDGRL
(d) DAN
(e) JAN
(f) DANN
(g) MDD
(h) MCD
(i) APN
Figure 2: t-SNE visualizations of features learnt from the 9 models on Office-31. The red (bluc) points represent the source (larget) domain features and one clustering represents one class.
21


Reviews:
[{"review_id": "hFUMKId0fu", "reviewer": "Reviewer_PMEH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.The paper proposes a new unified bound that can theoretically explaining alignment-based adaptation method.\n\n2.Based on the proposed bound, the paper further proposes new methods based on different distance estimator. The experimental results demonstrate the effectiveness of the proposed bound.\n\nWeaknesses:\n\n1. There are so few details in the proposed methods in this paper.  For example, how can we optimize these methods? I think that the authors should mention more details about the optimizatio process.\n\n2. The paper claims that the bound can be generalized for both classification and regression tasks, but there are regression experimental comparisons in this paper.\n\n3. The paper only includes a part of baseslines in the Table 4-6, but neglect more recent baselines, such as CAN and f-DAL mentioned in Table 2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To address the unsupervised domain adaptation problem, this paper investigates the theory of feature-level based domain alignment methods. This paper first points out that there is a theoretical gap between the widely used generalisation bound analysis and practical methods, and then further proposes a unified generalisation bound to theoretically explain the success of a variety of traditional alignment-based adaptation techniques. In addition, the authors also provide a conprehensive experimental comparisons new adaptation method and baselines. The results clearly demonstrate the proposed methods are competitive with classical models.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please refer to the \"Weaknesses\" in the last section. I think all weakness should be addressed.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ibJCNOR6Uo", "reviewer": "Reviewer_vrkj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. This paper is well-motivated by uncovering the long-standing theoretical gap, comparing the previous generalization bound analysis, and proposing a new unified bound. \n2. The proposed unified bound is universal for the alignment-based adaptation method, which is supported by theoretically explaining classical models.\n3. Experiments empirically verify the effectiveness of the proposed bound by promising results of the derived new methods. \n4. The extensive comparison of discrepancies is insightful for new alignment-based methods development in the future.\n\n### Weaknesses\n1. It seems this paper only focuses on feature alignment-based domain adaptation methods, supported by compared methods in Table 4 -Table 6. However, there is another mainstream line of methods based on self-training or pseudo-labeling. Is the proposed framework able to provide theoretical support for such methods? If not, for me, it is an overclaim to call the framework as a unified domain adaptation framework. It would be better to add feature-level domain alignment or similar words as the description.\n\n2. I appreciate the theoretical contribution of this paper. But I have some questions as to the practical/empirical value of the proposed bound. As is observed by the authors, \"there does not exist a typical distribution divergence measure that can universally outperform\". I do not expect the authors to tackle this problem in this paper, but it would be better to provide an in-depth analysis of why this problem happens. In that way, follow-up works can benefit from this paper to develop more practical and universally effective adaptation methods.\n\n3. Missing information on the newly derived methods in Table 4 - Table 6. What is the specific optimization objective for these methods? Do they share the same hyper-parameters including the loss coefficient?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the theory of using feature-level domain alignment to tackle the unsupervised domain adaptation problem. This paper first uncovers the long-standing theoretical gap between the widely used generalization bound analysis and various alignment-based domain adaptation methods. Then, the authors propose a unified generalization bound and use it to theoretically explain why several classical alignment-based adaptation methods perform well. To show the universality of the bound analysis, the authors provide a comprehensive comparison of common discrepancies and further derive new adaptation methods by using different divergences. Experimental results on three popular benchmarks show that the derived methods are competitive compared to classical models.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I would request the authors make the following important changes:\n\n1. Add missing details of the newly derived methods and try to find why some promising divergences occasionally perform badly. Is this because different adaptation tasks require different hyperparameters or something else?\n\n2. As for the presentation, I would first suggest not using \"a unified domain adaptation framework\" alone if the proposed bound is not universal for all domain adaptation methods. I would also suggest the authors give a detailed introduction to domain adaptation via feature alignment by a vivid figure and specific optimization process. Otherwise, readers who know little about domain adaptation may be curious about why domain alignments benefit target adaptation performance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d7L24PZM4B", "reviewer": "Reviewer_7NW6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Well written;\n2. A detailed summary. Few researchs compare the difference among different discrepancy. \n\nWeakness:\n1. I still think this paper is a small survey with respect to domain adaptation theory. But it seems that the aurthors have not claimed that.\n\n2.In your abstract, you claim that \"we propose new domain adaptation models and conduct experiments on real-world datasets that corroborate our theoretical findings\". But I cannot find the new models you proposed (All models have been proposed before). So do you have over-claimed? Please explain the sentence  \"we propose new domain adaptation models and conduct experiments on real-world datasets that corroborate our theoretical findings\". \n\n3. The theoreties discussed in your paper can only address the shift between marginal distributions, so when the conditional distributions are different, the thoreties cannot handle. The reason is that you omit the influence of feature transformation.\n\nPlease see the following papers to better understand:\n\n1) How does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches? AAAI 2021.\n\n2) On Learning Invariant Representation for Domain Adaptatio. ICML 2019.\n\nAbove two papers both consider UDA with more deeply understanding. \n\n4. Missing Reference:\nA survey on domain adaptation theory.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "It seems that this paper is a small survey related to the learning bounds and related theoretical applications of domain adaptation.\n\nThis paper compares different learning bounds with different distribution discrepancies.\n\nThe authors claim that they propose new domain adaptation models and conduct experiments on real-world datasets that corroborate our theoretical findings.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The paper should discuss  UDA theory when the conditional distributions are different. See weakness 3).\n\n\n\n", "broader_impact_concerns": "No ethical implications", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper rethinks the current theory and methods for domain adaptation and identify a gap between theory and practice. To fill this gap, this paper proposes a unified generalization bound which can explain the success of various domain adaptation methods. The proposed theory is also validated via comprehensive results.

After several revisions, all the reviewers think the paper makes a good contribution and the presentation is of good quality now. Based on the review comments and suggestions, I would like to recommend acceptance of this paper.
Decision: Accept (poster)