On $f$-Divergence Principled Domain Adaptation: An Improved Framework
Authors: Ziqiao Wang
Yongyi Mao
Venue: NeurIPS 2024 poster (2024)
Keywords: learning theory
unsupervised domain adaptation
f-divergence
generalization
Primary area: learning_theory
TL;DR: We give an improved theoretical framework for f-divergence-based domain learning

Abstract: Unsupervised domain adaptation (UDA) plays a crucial role in addressing distribution shifts in machine learning. In this work, we improve the theoretical foundations of UDA proposed in Acuna et al. (2021) by refining their $f$-divergence-based discrepancy and additionally introducing a new measure, $f$-domain discrepancy ($f$-DD). By removing the absolute value function and incorporating a scaling parameter, $f$-DD obtains novel target error and sample complexity bounds, allowing us to recover previous KL-based results and bridging the gap between algorithms and theory presented in Acuna et al. (2021). Using a localization technique, we also develop a fast-rate generalization bound. Empirical results demonstrate the superior performance of $f$-DD-based learning algorithms over previous works in popular UDA benchmarks.

On f-Divergence Principled Domain Adaptation: An Improved Framework
Ziqiao Wang Tongji University Shanghai, China zi zigiaowang@tonggj.eeem
Yongyi Mao University of Ottawa Ottawa, Canada ymao@uottawa.co
Abstract
Unsupervised domain adaptation (UDA) plays crucial role in addressing distribu- tion shifts in machine learning. In this work, we improve the theoretical founda- tions of UDA proposed in Acuna et al. (2021) by refining their f-divergence- discrepancy and additionally introducing new measure, f-domain discrepancy (f-DD). By removing the absolute value function and incorporating scaling pa- rameter, f-DD obtains novel target error and sample complexity bounds, allowing to recover previous KL-based results and bridging the gap between algorithms and theory presented in Acuna al. (2021). Using a localization technique, we also develop a fast-rate generalization bound. Empirical results demonstrate the superior performance of f-DD-based learning algorithms over previous works popular UDA benchmarks.
Introduction
Machine learning often faces the daunting challenge of domain shift, where the distribution of data in the testing environment differs from that used in training. Unsupervised Domain Adaptation (UDA) arises solution to this problem. In UDA, models are allowed to access to labelled source domain data and unlabelled target domain data, while the ultimate goal is to find a model that performs well on the target domain. The mainstream theoretical foundations of UDA, and more broadly, domain adaptation [2], primarily rely on the seminal works of discrepancy- based theory [3, 4]. In particular, [3, 4] characterize the етrоr gap between two domains using hypothesis class-specified discrepancy measure e.g., H -iiveergeeee While these works initially focus on binary classification tasks and zero-one loss, [5] extend the theory
more general setting. Subsequently, this kind of theoretical framework was extended by various works [6-10, 1, 11], all sharing some common properties such the ability estimate the proposed discrepancy from finite unlabeled samples. Importantly, these theoretical results often inspire the design of new algorithms, such as domain-adversarial training of neural networks (DANN) [12] and Margin Disparity Discrepancy (MDD) [9], directly motivated by -iv--divegeence.
Recently, [1] proposes an f-divergence- based domain learning framework, which generalizes various previous frameworks (e.g.. those based on HAH -ivveerver))) and have demonstrated great empirical successes. However, argue that this framework has potential limitations, at least in three aspects.
First, their discrepancy measure based on the variational representation of f-divergence in [13] (cf. Lemma 2. Although this variational formula is commonly adopted, its weakness has been pointed out in several works [14, 15]. For example, from this formula, one cannot recover the well-known Donsker and Varadhan's (DV) representation of KL divergence [16]. This reveals a second limitation: some existing domain adaptation or transfer learning theories are based on the DV representation of KL, such [17-20], and the framework by [1], although including KL as a special case of f-divergence, fail to unify those theories. Furthermore and more critically,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the discrepancy measure proposed by [1] contains an absolute value function, while the original variational representation does not. Notably, this absolute value function is necessary in their derivation of target error upper bound, and the fundamental reason behind this is still due to the weak version of the variational representation relied upon. Specifically, a variational representation of an f-divergence is lower bound of the divergence, but using weak lower bound creates technical difficulties in proving an upper bound for target domain error. [1] chooses to add an absolute value function accordingly, potentially leading to an overestimation of the corresponding f-divergence. In fact, this absolute value function is removed in their proposed algorithm, termed f- F-Domain Adversarial Learning (f-DAL). While f-DAL outperforms both DANN and MDD in standard benchmarks, this choice exhibits a clear gap between their theory and algorithm.
In this work, to overcome these limitations and explore the full potential of f-divergence, we present an improved framework for f-divergence-basee domain learning theory. Specifically, we apply a more advanced variational representation of f-divergence (ef. Lemma 2.2), independently developed by [21, 22] and [23]. After introducing some preliminaries, the organization of the remainder of the paper and our main contributions are summarized below.
In Section 3, we revisit the theoretical analysis in [1], where we refine their f-divergence- based discrepancy by Lemma 2.2 while retaining the absolute value function in the definition. The resulting target error bound (cf. Lemma 3.1) and the KL.-based generalization bound (cf. Theorem 3.1) complement the theoretical framework of [1].
In Section 4, we design a novel f-divergence- based domain discrepancy measure, dubbed f- DD. Specifically, we eliminate the absolute value function from the definition used in Section 3, incorporating a scaling parameter instead. We then derive an upper bound for the target error (cf. Theorem 4.1) and the sample complexity bound for our f-DD. The generalization bound based on empirical f-DD naturally follows from these results. Notably, the obtained target error bound allows us to recover the previous KL-based result in [19] (ef. Corollary 4.1).
In Section 5, to improve the convergence rate of our f-DD-based bound, we sharpen the bound using a localization technique [24, 10]. The localized S-DD allows for a crisp application of the local Rademacher complexity-
concentration results [24]. while also enabling us achieve a fast- rate target error bound (cf. Theorem 5.1). As concrete example, we present generalization bound based on localized KL-DD (cf. Theorem 5.2), where our proof techniques are directly connected to fast-rate PAC- Bayeesan bounds [25, 26] and fast-rate information information-theoreticc generalization bounds [27-29].
In Section 6, we conduct empirical studies based on our f-DD framework. We show f-DD outperforms the original f-DAL. in three popular UDA benchmarks, with the best performance achieved by Jeffereys-DD. Additionally, we note that the training objective in f-DAL. aligns more closely with our theory than with [1] (cf. Proposition 1). We also show that adding the absolute value function leads to overestimation and that optimizing the scaling parameter in f-DD may not be necessary in practical algorithms.
2 Preliminaries
Notations and UDA Setup Let X' and y be the input space and the label space. Let H be the hypothesis space, where each E H. is hypothesis mapping X' to Y. Consider a single-source domain adaptation setting, where μ and are two unknown distributions on X x V, characterizing respectively the source domain and the target domain. Let S = {X,, Y.)"" ~ be a labeled source-domain sample and T = [ @m be an unlabelled target-domain sample, where V is the marginal distribution of X in the target domain. We use f and & to denote the empirical distributions on X' corresponding to S and T, respectively. The objective of UDA is to find a hypothesis h E H based on S and T that "works well" the target domain.
Let é: y x y + RJ be a symmetric loss (e.g., E(y.y) = 0 for all y E 3). The population risk for each h E H in the target domain (i.c. target error) is defined as (h) & Ex,,y)~, [(()(X), Y)), and the population risk in the source domain, R2, (h), is defined in the same way. Since μ and v are unknown to the learner, one often uses recourse to the empirical risk in the source domain, which, for
a given S, is defined as (h) A E(X,Y)= [(()(X), Y)] E
((),(,,), Y.). Furthermore, let Su and fy be the ground truth labeling functions for the target domain and source domain, respectively, i.c. fx( = arg max u(Y = y y/ and fy (x) arg max u(Y y/z)

---

With little abuse of the notation, we will simply use &(h, to represent f(h(z). (x))) when the same z evaluated, serving as the "disagreement" for h and h' on æ. Additionally, following the conventional literature on DA theory [5], we assume that the loss function satisfies the triangle property'. For the readers' convenience, summary of all notations provided in Table 5 in the Appendix.
Background on f-divergence The family of /-divergence defined as follows.
Definition 2.1 (f-divergence [30]). Let P and Q be two distributions on O. Let á R4 + IR be convex function with ((1) P < Q', then f-divergence defined as (P(()) & where dP is Radon-Nikodym derivative.
④b Ee
dP
aQ
The f-divergence family contains many popular divergences. For example, letting ((z) log (or x log r + c(x - 1) for any constant c) recovers the definition KL divergence.
The f-divergence discrepancy measure by [1] is motivated by the variational formula of f-divergence that utilizes the Legendre transformation (L.T).
Lemma 2.1 ([13)). Let o3 be the convex conjugate? of Φ, and G {g: O dom(Ø))) Then
Do(PIIQ) sup Be~ [g(0]] Es~Q ( (g))))
gEG
However, it is well-known that the variational formula in Lemma 2.1 does not recover the the Donsker and Varadhan's 's (DV) representation KL divergence (cf. Lemma I). We will elaborate on this later. Recently, [23] and [21, 22] concurrently introduce novel variational representation for f-divergence, which is also implicitly stated in [31, Theorem 4.2], as given below.
Lemma 2.2 ([21, Corollary 3.5]). The variational formula of f-divergence
Do(P))Q)= sup Ea-r [o()] inf (TE--Q 10 (g(() +a)] -a)- aER
This variational representation is "shift transformation" of Lemma 2.1 (i.e. 9 - 9 o)) Note that this representation shares the same optimal solution as Lemma 2.1 (clearly identified as the corresponding f f-divergence), but Lemma 2.2 considered tighter in the sense that the representation in Lemma 2.2 is flatter around the optimal solution. [23] provides a comprehensive study to justify this, and they also show that Lemma 2.2 can accelerate numerical estimation of f-divergences.
Here, to illustrate the advantage of Lemma 2.2, we use KL divergence as an example. Specifically, let ((z) log x x + then its conjugate function is o" p1( e 1. Substituting d" into Lemma 2.1, we have
DKL (PIIQ) sup Ep [o( Eq (E0 gGg
1]
(1)
This representation is usually called based KL. On the other hand, with the optimal a" = - log EeQ [co(0)] Lemma 2.2 will give us
EQ [eel0)+a] -1-a3 sup Ep [o(0)] log EQ [coto]] (2) 9Eg
DKL (PIIQ)= = sup Ep [o))]] inf OER
9Eg
Notice that Eq. (2) immediately recovers the DV representation of KL. Since log(x) S x 1 for I > 0, it is evident that, as a lower bound of KL divergence, Eq. (2) is pointwise tighter than Eq. (1). In Appendix C, also show the variational representations of some other divergences obtained from Lemma 2.2.
In the context of UDA, it may be tempting to think that using point-wise smaller quantity (in Lemma 2) as the key component of an upper bound for target error, is essentially desired. However,
'The triangle property of loss function indicates that (() < f(yn ys) Eyys, ye) for any 3/3 E y. We say that P is absolutely continuous with respect Q, written P & Q. if ((A) = P(A) for all measurable sets C O. For a function f X Ru (-o0, +oc). its convex conjugate is f"( A sup (x. - f(x).
3

---

neither Lemma 2.1 nor Lemma 2.2 is able to directly give such an upper bound. To elaborate,
as (x) 2 x when (1)) 0 (cf. Lemma B.2), both Lemma 2.1 and Lemma 2.2 imply that D& I sup, Ep [a(0)] Ro [o()]]. Bearing this in mind, UDA typically requires an upper bound for the quantity supn E, [e o h(x)] - Ey [e h(X)), and simply restricting G in Lemma 2.1 and Lemma 2.2 to composition of H.. and & can only provide lower bound for such a quantity.
Before we propose an improved discrepancy measure, we first revisit the absolute discrepancy in [1] by using Lemma 2.2 instead. This also serves as review of the common developments in the DA theory.
3 Warm-Up: Refined Absolute f fDDivergence Domain Discrepancy
Based on Lemma 2.2, we refine the discrepancy measure of [1, Definition 3] as follows.
Definition 3.1. For given h H. the D discrepancy from y is defined as
1 (ullv) & sup
Ex [e(h, h')] (e (2)h)))
MEH
where 12 (e h') inf (E, 16 hl) + a)] aa.
Remark 3.1. Removing the absolute value function D (ulli) does not alter its non-negativity, To see this, consider h' h. By the definitions of o and 6*, inf 6= (a) a 6(1) we have E [e(h, W²]] = (f o 11)=
= O. Consequently, since h exists in H. & without the absolute value function. In addition, due to the absolute value function, the relation between h. (Mall/) with the one in [1] is no longer clear.
Dh (nlpp) 2 holds even
While the absolute value function is not required for ensuring non-negativity, it is crucial for deriving the subsequent error bound for the target domain.
Lemma 3.1. Let X" min eH (h) )+ (h",)), then for any h H. we have
(h) S (2)
Remark 3.2 (Regarding X*). This error bound shares similarities with the previous works [3, 4, 9, 10. 1]. For example, the third term X is the ideal joint risk for the DA problem. As widely discussed in prior studies, this term captures the inherent challenge the DA task and might be inevitable [4, 32). However, it has also been pointed out that this X* term can be significantly pessimistic, particularly in the case of conditional shift [33]. In fact, similar to the X --free bound [33, Theorem 4.1], it is simple matter to replace X* with the cross-domain error term min R2 (fw). Ry (u(f))) in Lemma 3.
(and all the other target error bounds in this paper). See Appendix D.3 for details. Given that [1] also uses X", we use X² in the bounds for consistency in the remainder of this paper.
In the sequel, we will give a Rademacher complexity based generalization bound for the target error. Let (F) denote the empirical Rademacher complexity of function class F = {f : X' - R} for some sample S [34]. Notice that a shift transformation of a function class will not change its Rademacher complexity, so the generalization bound based on D closely resembles the one presented in [1, Theorem 31, which contains a Lipschitz constant of d², Here, we give a generalization bound specialized for A... wherein the Lipschitz constant of 6* can be explicitly determined. we have
Theorem 3.1. Let E(-,-) :) E [0. 1]. Then, for any h E H. with probability at least
(h) S (h) + D.. (AllD) + (xH)s(H()))))))))))))))))()))) + 49 (7H)) + +0+ (v [og(1/0)
log (L/0)
+
n
m
where He = (x>a))))))) (((M))MMM))) (()))l,, h E H).
[1] also applics Rademacher complexity-based bound to further upper bound X by its empirical version, namely X- = ming-ew (h"h") ) + Ro(h"), However, since there is no target label available, Ro (h²) is still not computable, invoking X* here has clear advantage.

---

New f-Divergence-Based DA Theory
While D serves as valid domain discrepancy measure in DA theory, it exaggerates the do- main difference without appropriate control. It's noteworthy that [1] attempts to demonstrate their discrepancy with the absolute value function is upper bounded by D [1, Lemma 1]. but this is problematic; sup U > O does not imply sup U sup [U] when U is not a positive function. Note that the error in [1, Lemma I] also identified [35]. Furthermore, when designing their f-DAL algorithm, they drop the absolute value function in their hypothesis-
f-divergence (see Eq. (5)). Consequently, the remarkable performance of f-DAL reveals significant gap from their theoretical foundation.
To bridge this gap, we introduce new hypothesii-sppecifie w-divergence-baeee DA framework. Our new discrepancy measure is dubbed f-domain discrepancy, or fDD,, defined without the absolute value function and with an affine transformation.
Definition 4.1 (f-DD). For given h E H. the f-DD measure h.H is defined as
D H D (w|lm) A sup
EN lt E(h, h()] (tl
MEH,EER
where p (tl (tl h') infa EEn (t- E(h, aa] aa}
Remark 4.1. H G. then D² is an affine transformation of Lemma 2.1 (i.e. g lg + a)) and scaling transformation of Lemma 2.2. Imporiantly, unlike h.H. it's easy to see that D (v(lp) I DDD(lll)).
à Our D (w(lu) retains some common properties of the discrepancies defined in the DA theory literature. First, as / 0 leads to E,, [t. e(h, h')] = ب., (I o h') O, the non-negativity of (vllp) is immediately justified. In addition, its asymmetric property is also preferred in DA, as discussed in the previous works [9, 10]. Moreover, when v, by the definition of d", we have D (v)lp)) = 0.
To present an error bound, the routine development, as in Lemma 3.1, is insufficient; we require a general version of the "change of measure" inequality, given below.
Lemma 4.1. Let (z)) & o(x + 1), and a/* is its convex conjugate. For any h h E H. and t R. define KR (t) inf, E, [wb (t e(h, 1²) + a)). Let K₁((t) supweH KR (t), then for any h, h E H.
K (E (ED [e(h, - En (e(h,N)) S D h. (vllu),
where K the corvex conjugate of Ky-
It is worth reminding that K2 and Ký both depend on h, although we ignore h in the notations to avoid cluttering. We are now ready give target error bound based on our f-DD.
Theorem 4.1. For any h H. we have
D (wllm) + K2(()
(h) S Rh) inf 120
X
(3)
Furthermore, let é e [0, 1), if o is twice differentiable and &" monotone, then
2 h. (vllR) D* x
(h) S (h) +
(4)
(0(1)
The following is an application of Eq. (4), where we consider the case of KL, namely D. (v]l
Corollary 4.1. Let é E [0, 1), then for any h H. we have (h) (h) + 2D
h. (и]lи) + X
As DAA (vllm) < DRI.(ull)), the bound in [19, Theorem 4.2] can be recovered by Corollary 4.1 for the same bounded loss function. We also remark that the boundedness assumption in Corollary 4.1 can be further relaxed by applying the same aub-Gaussian assumption in [19, Theorem 4.2].
5

---

To give generalization bound, the next step involves obtaining a concentration result for f-DD, as given below.
Lemma 4.2. Let E [0, 1] and let to be the optimal achieving the superum in D (vllu)) Assume o is I.-Lipschitz, then for any given h, with probability at least 1 5, we have
(v 1og(1/0)
log(1/6)
D (v|lm) s (OllA) + (t (H + 2L |tol (t
m
Remark 4.2. The function o² needs not to be globally Lipschitz continuous; it can be locally Lipschitz for bounded domain. For example, in the case of KL, o' is e-Lipschitz continuous when E E [0. 1]. Moreover, although the distribution-depenade
quantity to might not always have a closed-form
expression, in the case of DDD, we know that to 2(E, [EE,(((0,, -E, corresponding optimal hypothesis.
where ht is the
Var n. (R(h,N"))
also straightforward to obtain concentration results for (t) (t) and (,) - - (h). Substituting these results into Eq. (3) obtains the final generalization bound based on f-DD. Or alternatively, one can directly substitute Lemma 4.2 into Eq. (4) (See Appendix D.7). However, in this case, the empirical f-DD and other terms in Lemma 4.2 will feature square root, slowing down the convergence rate. We address this limitation in the following section.
5 Sharper Bounds via Localization
A localization technique in DA theory is recently studied in [10]. We now incorporate it into our framework with some novel applications. First, we define a localized hypothesis space, formally referred to as the (true) Rashomon set [36-38].
Definition 5.1 (Rashomon set). Given data distribution μs, hypothesis space and loss function f, for a Rashomon parameter r 0, the Rashomon set H. is an r-level subset of H. defined as: H, {h H. (,) (h) < r..
Notice that the Rashomon set H,, implicitly depends the data distribution. In this paper, specifically define H. by the source domain distribution μ. Then, we define our localized f-DD: Definition 5.2 (Localized f-DD). For given h Hr,, the localized f-DD from j to v is defined as
H.H. (vilm) A sup MEH.,120
Ey [tltth, h²)] - 12 h²).
Remark 5.1. Compared with J-DD, localized J-DD restricts ha to Hr, and h Hrr where ri and r may or may not be equal. In addition, the scaling parameter t is now restricted to R instead of R.
Clearly, Dh.H. (ulli) non-decreasing when increases, and it upper bounded by D. (H|[D).
As also mentioned at the end of the previous section, Eq. (4) of Theorem 4.1 (and Corollary 4.1), involves square root function for f-DD, potentially indicative of slow-rate bound (e.g.. if h (1/n)), then the bound decays with O(1/ VTi).. We now show that how the localized f-DD achieves fast-rate eπrоr bound.
Theorem 5.1. For any E Hr, and constants C₁, C2 E +00) satisfying (Ci) S
C C.C.E [e(h, for any h' Hrr the following holds:
(h) S (a)
+
C.
D
1 A.H. (vlle) + C2 C2R + X;
where X; mininEH. R2(hh)) + (h") and (2)((h) = supWEH, By [E(h, 1()).
Remark 5.2. By the triangle property, R (h) ST+ In this case, a small Tı will reduce both the first term and the third term in the bound. However. determining the optimal value for intricate. On the one hand, we hope r is small so that h.H. (v|lи) and R,, (h) are both small. On the other hand, if is too small, specifically if X², then it's possible that N; > X2 becuuse the optimal hypothesis minimizing the joint risk may not exist in Hr- Additionally, if both rı and are too small, the effective space for optimizing Cn and C2 may also be limited. Therefore, the value of v involves complex trade-off among the three terms.
6

---

Overall, we expect to be as small as possible, aligning with the principle of empirical risk minimization for the source domain in practice. We may let r > X so that the optimal hypothesis is guaranteed to exist in the Rashomon set Hr Furthermore, if
is unavoidably large, we prefer small C2 so that C2 C2 (h) is small. If X* itself is negligible, we use a vanishing r + rı. In this case, one can focus on minimizing 1/C1 while allowing C2 to be large.
Combining Theorem 5.1 with Lemma 4.2 and following routine steps will obtain the generalization bound based on J-DD, where the local Rademacher complexity [24] will be involved. However, one may feel unsatisfied without an explicit clue for the condition (C,
C2Cy [e(h, h()
in Theorem 5.1. In fact, exploring concentration results under this condition is a central theme in obtaining fast-rate PAC-Bayesian generalization bounds [25, 19-41, 26] and the cemmatioonthhoreit generalization bounds [27-29, 42]. Building upon similar ideas from these works, we now establish sharper generalization result for our localized KL-DD measure, where the fast-rate condition is more explicit. One key ingredient is the following result.
Lemma 5.1. Let 8 E [0, 1), and let the constants C1 > 0 and C2 E (0, 1) satisfy the condition (e - - 1) (1 minfr1 + r. C2 min(r1 + +r,1)) S C.C2. Then, for any h E Hr, and h E Hy, we have
Dh (vlle) (1 C2) ((),, C1
E, [(hh,))]
inf Cr.Cs
Remark 5.3. As an extreme case, if + 7 O (implying E,, [e(h, h')] = 0), then let C2 - the condition in the lemma indicates that C₁ 1.26. Hence, the optimal bound becomes E, [e(h, h')l S O. 0.79D%. H.r KI. (vllu). This bound remains valid even without r + Tı + O. It holds when the Rashomon set H, is "consistent" with a given h, meaning all hypotheses in H, have similar predictions on the source domain data. As an another case, if f Ti 2 and supw Ep [6(h, h()l is also large, we may prefer small Cz. such setting C2 0.1. In this case, the condition becomes (c - C1 1) C2 S C₁, suggesting that C1 < 3.74. This results in the optimal bound E. [e(h, h')] E. [e(h, h')] 0.27D h.H (vllm) + 0.1 [e(h, h)). The condition in Lemma 5.1 is common in many fast-rate bound literature, such as [28, Theorem 3].
We are now in position to give fast-rate generalization bound for localized KL-DD. Theorem 5.2. Under the conditions in Lemma 5.1. For any h E Hry with probability at least 1 - 8. have
DR. (DllA) (h) ( (H,) + Ris (Himes(rrr))
(h) (Ra(h) +
(ii
C1
[og((10) 108(1/0)) + +0 (
(ri r log(1/)))) (1g(1)))))1)) + (logg(/))) m
r
+ +
X;
n
Remark 5.4. Due to the non-negativity of f-DD, a similar generalization bound also applies to the Jeffereys divergence (or symmetrized KL divergence) [43] counterpart, which is simply the sum of KL divergence and reverse KL divergence (i.e. Dzy (ùll0) + DEL (Bll))) Furthermore, considering the fact that (xl(v < log(1 + (x(())) < x (v(lp)) [30]. one might anticipate similar bound for x2-DD, which we defer to Appendix D.11.
This generalization bound suggests that when + is small, not only are the first four terms (including the local Rademacher complexities) reduced, but it also causes O (1 + 4 to dominate the convergence rate of the bound. In practice, when empirical risk of the source domain is always minimized to zero (i.c. the realizable case), then r1 itself may have fast vanishing rate (c.g.. O(L/n)). In Appendix D.12, we provide concrete example to further illustrate the superiority of localized f-DD.
Algorithms and Experimental Results
6.1 Domain Adversarial Learning Algorithm
In a practical algorithm, the hypothesis space consists of two components: the representation part, denoted as Hrep = {hrep X + 2), where Z is the representation space (c.g., the hidden output
7

---

of a neural network), and the classification part, denoted as Hels fhols 2 - V). Therefore, the entire hypothesis space is given by {hels Peep Hrops hets Hels}- The training objective in f-DAL [1]
min max (R) + nd, REH NEW
Here, (h, Ep [E(h,)t) Ba Oo- (eth, where r) is trade-off parameter and é is the surrogate loss used to evaluate the disagreement between h and h', which may may not be the same f. Note that, to better align with our framework, we change the order of ji and á in dp.c in the original f-DAL. This modification is minor, as in either case, its optimal value belongs to f-divergence (such as KL and reverse KL, x² and reverse x²).
Table 1: Accuracy (%) on the Office-31 benchmark.
Method
A
D
W-D
A
Avg
ResNet-50 [44]
68.4+0.2
96.7+0.1
99.3-0.11
68.9+0.2
62.5-003
60.7+0.3
76.1
DANN [12]
82.0+0.4
96.9=0.2
99.1=0011
79.7+0.4
68.2=0.4
67.4+0.5
82.2
MDD [9]
94.5+0..3
98.4=0.1
100.01.0
93.54:0.2
74.6=0.3
72.240.11
88.9
KL [45]
87.9+0.4
99.0=0.2
100.0
85.640.6
70.1=1.1
69.3+0.77
85.3
J-DAL [1]
95.4110.7
98.8=0.11
100.01.0
93.8:0..4
7..9=1.5
74.210.5
89.5
Ours (KL-DD)
94,910.7
98.720.1
100.01.0
95.910.6
74.6-0.9
74.6100.7
89.8
Ours (x2 (DDD
95.3+002
98.7+0..1
100.0+.0
95.0+0.4
73.7-0.5
75.6+0.2
89.7
Ours (Jeffreys-DD)
94.910.7
99.1 40.2
100.01.0
95.910.6
7..0-0.5
74.610.77
90.1
Table Accuracy (%) on the Office- benchmark.
Mothod
Ar-4CI
Ar-PPr
Ar-sRw
Cl-AAr
Cl-Pt
Cl-Rw
Pr-sAr
Pr-sCI
Pr-sRw
Rw-1Ar
Rw-1C1
Rw--Pr
ResNer-51
37.4
46.2
60.4
53.9
59.9
MDD [9]
$4.9
73.7
77.8
60.0
71.4
71.8
61.2
78.1
72.5
60.2
82.3
68.
[-DAL
$4.7
71.7
77.8
61.0
72.6
72.2
60.8
53.4
80.0
73.3
60.6
83.8
68.5
Ours (KL-DD)
55.3
70.8
78.6
62.6
73.8
73.6
62.7
53.4
80.9
75.2
61.3
84.2
69.4
(y (D)D
73.4
74.8
84.1
(efiereysDD)
73.9
81,3
75.2
Eq. (5) results in adversarial training strategy. Specifically, the outer optimization spans the entire hypothesis space. Meanwhile, within the inner optimization, given a = hrep o hats- the representation component hrep is shared for h' In other words, the optimization is carried out for h in H' Hela hrep & {hels heee E Hele). The overall training framework of our f-DD is illustrated in Figure 2 in Appendix.
Clearly, as also discussed previously, maxy (h, S max (h, h²),, which presents clear gap between the theory and algorithms in [1]. In contrast, this training objective aligns more closely with our D R. Formally, we have the following result.
Let (h, h²) Ep ((h,h)] 1 M). Assume H. is sufficiently large s.t. L: X' R. we have max dp, h/) max dz (h, h²) Dh (QllA) (o(lll.).
Proposition 1.
In our algorithm, we use da.e R²) to replace dp. R²) in Eq. (5). Proposition implies that either the optimal dz,; (h, h²) or the optimal (h, h') coincides with f-DD. Moreover, as is typically unbounded in practice (e.g., cross- cross-entropy loss), considering the unbounded nature of t, Proposition suggests an equivalence between optimizing (hh, h²) through Áí and optimizing t((h, h') through both and h'. In this sense, f-DAL has already considered the scaling transformation. Later will empirically investigate whether explicitly optimizing is necessary.
(h,
Furthermore, we highlight that the training objective in our algorithm, namely Eq. (5) with dh, h!) replaced by da (h, h'), is de facto motivated by the insights from Section 5. In that section, we demonstrate that when the source domain error small, the f-DD term without square-root function provides more accurate reflection of generalization (see, for instance, Remark 5.3). Given that the empirical risk of the source domain is always minimized during training, we expect the final hypothesis to fall within the subset of H. (i.e. Rashomon set with r1).

---

6.2 Experiments
The goals of our experiments unfold in three aspects: 1) demonstrating that utilizing the f-DD measure (i.e., using dg h²) as the training objective) leads to superior performance on the benchmarks; 2) confirming that the absolute discrepancy (i.e. (dp,0(h,))) h() leads to degradation in performance; 3) showing that optimizing over t may be unnecessary in practical scenarios.
Dataset We use three benchmark datasets: 1) the Office31 dataset [46], which comprises 4, 652 images across 31 categories. This dataset includes three domains: Amazon (A), Web- cam (W) and DSLR (D); 2) the Office-Home dataset [47], consisting of 15, 500 images dis- tributed among four domains: Artistic images
Table 3: Accuracy (%) on the Digits datasets
Method
M-+U
U-M
Avg
DANN [12]
91.8
94.7
93.3
f-DAL [1]
95.3
97.3
96.3
(Ar), Clip Art (CI), Product images (Pr), and Real-world images (Rw); and 3) two Digits datasets, MNIST and USPS [48] with the as- sociated domain adaptation tasks MNIST USPS (M-+) and USPS - MNIST (U-M) We follow the splits and evaluation protocol established by [49], where MNIST and USPS have 60, 000 and 7, 291 training images, as well as 10, 000 and 2, 007 test images, respectively.
Ours (KL-DD)
95.7
98.1
96.9
Ours (x2-DD)
95.4
97.3
96.4
Ours (Jeffereys-DD)
95.9
98.3
97.1
Discrepancy Measures In algorithms, mainly focus three specific discrepancy mea- sures: KL-DD, x² DDD and the weighted Jeffereys- DDD. Specifically, the weighted Jeffereys-DD is m DxL. (+ll)) (D) + ?2 (xll..)l()) where " and 7/2 are tunable hyper-parameters. We note that Jeffereys divergence is not explored in [1] while also an f-divergence with Φ(x) = (x 1) log x, and advantages of Jeffereys divergence studied in [18, 45, 19].
Baselines and Implementation Details The primary baseline for our study is the preceding f-DAL [1]. Note that, with the exception of Digits, the results reported by [1] for f-DAL rely on maximum values obtained from their divvieegee and weighted Jensen-Shannee divergence across individual sub-tasks. In our comparison, as special cases of f-DAL. Furthermore, we compare the results reported by [45] on the Office-31 dataset, although denoted "KL", their method incorporates Jeffereys divergence in their algorithms.
also include DANN [12] and MDD [9] as they may be interpreted
Our implementation closely follows [1]. For Office-31 and Office-Home, trained ResNet-50 [44] as the backbone network, while both the primary classifier in h and the auxiliary classifier in h' consist of two-layer Leaky-Rel networks. In the case of Digits, use LeNet [50] as the backbone network and a two-layer ReL.U network with Dropout (0.5) for the classifiers. Other hyper-parameter settings and the evaluation protocol remain consistent with [1], and the reported results represent aver- age accuracies over different random seeds.
utilize pre-
Figure 1: Failure of absolute discrepancy. The y- axis is the estimated f-divergence.
Boosted Benchmark Performance by f-DD Tables 1-3 collectively demonstrate the superior performance of our weighted Jeffereys-DD com- pared to other methods across the three bench- marks. Notably, the most significant improve- ment over f-DAL observed on rifice-Home (70.2% vs. 68.5%). Remarkably, this perfor- mance even surpasses the combination of f- DAL with a sampling-based implicit alignment approach [51] (See Table in Appendix), specifically designed handle the label shift issues. In addition, unlike findings in [1], where x² outperforms other methods nearly all tasks, use of
Table 4: Comparison between KL-DD and OptKL-
DD
Method
Office-31
Office-Home
Digits
KL-DD
89.8
69.4
96.9
OptKL-DD
89.6
69.2
96.5

---

tighter variational representation-base discrepancy reveals that x² is no longer superior to KL. In fact, our KL-DD slightly outperforms x²DDD in all three benchmarks.
Failure of Absolute Discrepancy We also perform experiments on A >D and Ar->CI using the absolute discrepancy (i.e., max (1, ht) Specifically, we compare the x2-based discrepancy with (w/) and without (w/o) the absolute value function. Figure illustrates that such a discrepancy can easily explode during training, demonstrating its tendency to overestimate /-divergence. Additional results for KL are given in Figure 3 in Appendix.
Optimizing over In the paragraph following Proposition 1, we discuss the observation that optimizing over may not be necessary. Empirical evidence indicates that setting t = with hyper-paraman tuning (e.g.. through 17) obtains satisfactory performance. Now, let's investigate the selection of t for KL-DD. Instead of using a stochastic gradient- based optimizer for updating t. we invoke quadratic approximation for the optimal t. as studied in [23]. Specifically, we define Gibbs
E. [((_hhh)))) (.., [e(h.h"] In-
measure du' = Fy 3]] then the optimal t" = 1 + At", where At" = (eth,,)
terested readers can find detailed derivation of this approximation in [23, Appendix BJ. Substituting t= + At", we have the training objective for approximately optimal KL-DD (OptKL-DD), Table presents an empirical comparison between OptKL-DD and the original KL-DD, where is simply set to I. The results indicate that OptKL-DD does not provide any improvement on these benchmarks. Similar observations also hold for x2. in which case optimal t has an analytic form (see Appendix E), suggesting that using& t I, at least for KL and x². might be sufficient in practice.
Additional experimental results, including an ablation study on 17. t-SNE [52] visualizations and other comparisons, be found in Appendix E.
7 Other Related Works
Apart from those mentioned the introduction, various other discrepancy measures are explored in DA theories and algorithms. These include the Wasserstein distance [53-55], Maximum Mean Discrepancy [56, 57], second-order statistics alignment [58, 59], transfer exponents [60, 61]. Integral Probability Metrics [62] and so on. Notably, [62] defines a general Integral Probability Metrics (aPMs)-based discrepancy measure. It's noteworthy that the intersection of the IPMs family and the f-divergence family results in the total variation. Consequently, both corresponding discrepancy measures can consider HA H-divergence as special case. Additionally, one of our baseline models [45] diverges from the adversarial training strategy. Instead, they directly minimize the KL divergence between two isotropic Gaussian distributions (source domain Gaussian and target domain Gaussian) in the representation space. Here, the Gaussian means and variances correspond to the hidden outputs of the representation network. For further literature on DA theory, readers are directed to recent survey by [63].
Domain Adaptation
f-divergence Moreover, the combination of f f-divergence and adversarial training schemes has been extensively studied in generative models, including f-GAN [64, 65]. x²-GAN [66] and others. In the DA context, [67] introduce a f-divergence--art based discrepancy measure while still relying on case. Additionally, [68] investigates a- R-Rényi divergence for multi-source DA, and [69] provides some intriguing interpretations x² -dvivergeecce- based generalization bound for covariate shifts.
Lemma 2.1 and focusing solely on the
8 Conclusion and Future Work
In this work, we present an improved approach for integrating f-divergence into DA theory. Theoreti- cal contributions include novel DA generalization bounds, including fast-rate bounds via localization. On the practical front, the revised f-divergence-bar based discrepancy improves the benchmark perfor- mance. Several promising future directions emerge from our work. Firstly, beyond its usefulness for local Rademacher complexity, the Rashomon set H, also relates to another generalization measure, Rashomon ratio [37], which may give an alternative perspective
generalization in DA. Additionally, exploring transfer component-based analysis [60] for tight minimax rates in DA, invoking a power transformation instead of the affine transformation in f-DD, holds promise.
10

---

Acknowledgments and Disclosure of Funding
This work supported partly by an NSERC Discovery grant. The authors would like thank Loic Simon for bringing reference [35] to their attention and for the insightful feedback provided on this work. The authors also thank Fanshuang Kong for the extensive experimental guidance provided throughout this project. Furthermore, the authors are thankful to the anonymous AC and reviewers for their careful reading and valuable suggestions.
References
[1] David Acuna, Guojun Zhang, Mare Law, and Sanja Fidler. f-domain adversarial learning: Theory and algorithms. In International Conference on Machine Learning, pages 66-75. PMLR, 2021.
[2] Joaquin Quionero-Camne Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009.
[3] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning. 79 (1-2):151-177) 2010.
[5] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In The 22nd Conference on Learning Theory, 2009.
[6] Corinna Cortes, Mehryar Mohri, and Andrés Muñoz Medina. Adaptation algorithm and theory based generalized discrepancy. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 169-178, 2015.
[7] Pascal Germain, Amaury Habrard, François Laviolette, and Emilic Morvant. A pac-bayesian approach for domain adaptation with specialization linear classifiers. In International conference on machine learning, pages 738-746. PMLR, 2013.
[8] Pascal Germain, Amaury Habrard, François Laviolette, and Emilic Morvant. A new pac- bayesian perspective on domain adaptation. In International conference on machine learning, pages 359. PMLR, 2016.
[9] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 7404-7413. PMLR, 2019.
[10] Yuchen Zhang, Mingsheng Long, Jianmin Wang, and Michael Jordan. On localized discrepancy for domain adaptation. arXiv preprint arXiv:2008.066 06244, 2020.
[11] Changjian Shui, Qi Chen, Jun Wen, Fan Zhou, Christian Gagné, and Boyu Wang. A novel domain adaptation theory with jensen- divergence. Knowledge-Ber Based Systems, 257: 109808, 2022.
[12] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. networks. The journal of machine learning research, 17(1):2096-2330, 2016.
training of neural
[13] XuanLong Nguyen. Martin J Wainwright, and Michael Jordan. Estimating divergence func- tionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(1) 1):5847-5861 2010.
[14] Avraham Ruderman, Mark Reid, Darío García-Gareía, and James Petterson. Tighter varia- tional representations of f-divergences via restriction to probability measures. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages
1 1155 1 162, 2012.

---

[15] Jiantao Jiao, Yanjun Han, and Tsachy Weissman. Dependence measures bounding the explo- ration bias for general measurements. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 1475-1479. IEEE, 2017.
[16] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on pure and applied mathematics, 36 (2):183-212, 1983.
[17] Xuetong Wu, Jonathan H Manton, Uwe Aickelin, and Jingge Zhu. Clormation-theoeete analysis for transfer learning. In 2020 IEEE International Symposium Information Theory (ISTT), pages 2819-2824. IEEE, 2020.
[18) Yuheng Bu, Gholamali Aminian, Laura Toni, Gregory W Wornell, and Miguel Rodrigues. Characterizing and understanding the generalization enoor of transfer learning with gibbs algorithm. In International Conference Artificial Intelligence and Statistics, pages 8673- 8699. PMLR, 2022.
[19] Ziqiao Wang and Yongyi Mao. Information-iin -heorrtti analysis of unsupervised domain adapta- tion. In International Conference on Learning Representations, 2023.
[20] Fanshuang Kong, Richong Zhang, Ziqiao Wang, and Yongyi Mao. On unsupervised domain adaptation: Pseudo label guided mixup for adversarial prompt tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18399-18407, 2024.
[21] Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral prob- ability metrics. In International Conference on Machine Learning, pages 115-124. PMLR, 2020.
[22] Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral proba- bility metrics. The Journal of Machine Learning Reseurch,
2021.
[23] Jeremiah Birrell, Markos Katsoulakis, and Yannis Pantazis. Optimizing variational repre- sentations of divergences and accelerating their statistical estimation. IEEE Transactions on Information Theory, 68(7):4553 4572, 2022.
[24] Peter Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. Annals of Statistics, 33(4):1497-1533 2005.
[25] Catoni. Pac-bayesian supervised classification: the thermodynamics of statistical learning.
Vol. 56. Lecture Notes Monograph Series. Institute of Mathematical Statistics, 2007.
[26] Pierre Alquier. User-friendly introduction pac-bayes bounds.
arXiv preprint
arXiv:2110.112166 2021.
[27] Fredrik Hellström and Giuseppe Durisi. Fast-rate loss bounds via conditional information measures with applications to neural networks. In 2021 IEEE International Symposium on Information Theory (ISIT), pages 952-957, IEEE, 2021.
[28] Fredrik Hellström and Giuseppe Durisi. A new family of generalization bounds using sample- wise evaluated CMI. In Advances in Neural Information Processing Systems, 2022.
[29] Ziqiao Wang and Yongyi Mao. Tighter ixformation-theorette generalization bounds from supersamples. In International Conference on Machine Learning. PMLR, 2023.
[30] Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge university press, 2023.
[31] Aharon Ben-Tal and Mare Teboulle. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance, 17(3):449-477 2007.
[32] Shai Ben-David and Ruth Urner. On the hardness of domain adaptation and the utility of unlabeled target samples. Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23, pages 139-153. Springer, 2012.
12

---

[33] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-7532. PMLR, 2019.
[34] Peter I. Bartlett and Shahar Mendelson, Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
[35] Rodrigue Siry, Ryan Webster, Loic Simon, and Julien Rabin. On the theoretical equivalence of several trade-off curves assessing statistical proximity. Journal of Machine Learning Research, 24:185, 2024.
[36] Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 16(3):199-231, 2001.
[37] Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the existence of simpler machine learning models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1827-1858, 2022.
[38] Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning variable's importance by studying an entire class of prediction models simultancously. J. Mach. Learn. Res., 20(177):1-81, 2019.
[39] Yevgeny Seldin, François Laviolette, Nicolo Cesa-Bianchi, John Shawe-Taylor, and Peter Auer. Pac-bayesian inequalities for martingales. IEEE Transactions on Information Theory, 58(12): 7086-7093, 2012.
[40] Ilya O Tolstikhin and Yevgeny Seldin. Pac-bayes-empirical--emmsss inequality. Advances in Neural Information Processing Systems, 26, 2013.
[41] Jun Yang. Shengyang Sun, and Daniel M Roy. Fast-rate pac-bayes generalization bounds via shifted rademacher processes. Advances Neural Information Processing Systems, 32, 2019.
[42] Ziqiao Wang and Yongyi Mao. Sample-conditioned hypothesis stability sharpens information- theoretic generalization bounds. In Advances in Neural Information Processing Systems, 2023.
[43] Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007): 453 461, 1946.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[45] A. Tuan Nguyen, Toan Tran, Yarin Gal, Philip Torr, and Atilim Gunes Baydin. KL guided domain adaptation. In International Conference on Learning Representations, 2022.
[46] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the 1 11th European conference on Computer vision: Part IV, pages 213-226, 2010.
[47] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
[48] Jonathan Hull. database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-5544 1994.
[49] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 2018.
[50] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232 1998.
13

---

[51] Xiang Jiang, Qicheng Lao, Stan Matwin, and Mohammad Havaei. Implicit class- cass-conditioned domain alignment for unsupervised domain adaptation. In International Conference on Machine Learning, pages 4816-4827 PMLR, 2020.
[52] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11) 2008.
[53] Rémi Flamary, Nicholas Courty, Davis Tuia, and Alain Rakotomamoniy. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell, 1(1-40):2, 2016.
[54] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30, 2017.
[55] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Thirty-second AAAI conference on artificial intelligence, 2018.
[56] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
[57] Boqing Gong, Kristen Grauman, and Fei Sha. Connecting the dots with landmarks: Discrimina- tively learning domain-invarian features for unsupervised domain adaptation. In International conference machine learning, pages 222-230. PMLR, 2013.
[58] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pages 443-450. Springer. 2016.
[59] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, 2016.
[60] Steve Hanneke and Samory Kpotufe. On the value of target data in transfer learning. Advances in Neural Information Processing Systems, 32, 2019.
[61] Steve Hanneke, Samory Kpotufe, and Yasaman Mahdaviyeh. Limits of model selection under transfer learning. arXiv preprint arXiv:2305.00/52, 2023.
[62] Sofien Dhouib and Setareh Maghsudi. Connecting sufficient conditions for domain adaptation: source-guided uncertainty, relaxed divergences and discrepancy localization. arXiv preprint arXiv:2203.0507 05076, 2022.
[63] levgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebhan, and Younes Bennani. A survey on domain adaptation theory. arXiv preprint arXiv:2004. /829, 2020.
[64] Sebastian Nowozin, Botond Cseke and Ryota Tomioka. f-gan: Training generative neural
samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
[65] Jiaming Song and Stefano Ermon. Bridging the gap between f-gans and wasserstein gans, In International Conference on Machine Learning, pages 9078-9087 PMLR, 2020.
[66] Chenyang Tao, Liqun Chen, Ricardo Henao, Jianfeng Feng, and Lawrence Carin Duke. x² generative adversarial network. In International conference on machine learning, pages 4887- 4896. PMLR, 2018.
[67] Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetricallly-reaaxe distribution alignment. In International conference machine learning, pages 6872-6889 6881. PMLR, 2019.
[68] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Multiple source adaptation and the rényi divergence. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pages 367-374, 2009.
14

---

[69] David moun-Smmth,, Alexander D' Ammour Avi Feller, and Steve Yadlowsky. Tailored overlap for learning under distribution shift. NeurlPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
[70] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[71] Colin McDiarmid. Concentration, pages 195-248. Springer Berlin Heidelberg, 1998.
[72] Stéphane Boucheron, Olivier Bousquet, and Gábor Lugosi. Theory of classification: A survey of some recent advances. ESAIM: probability and statistics, 9:323-375, 2005.
[73] Gavin Crooks. Inequalities between the jenson-shannon and jeffreys divergences. In Tech. Note 004, 2008.
[74] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248 255, 2009.
[75] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[76] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[77] Tao Sun, Cheng Lu, Tianshuo Zhang, and Haibin Ling, Safe self-refinement for transformer- based domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7191-7200, 2022.
[78] Jinjing Zhu, Haotian Bai, and Lin Wang. Patch-mix transformer for unsupervised domain adaptation: A game perspective. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3561-3571, 2023.
15

---

Appendices
The structure of Appendix is outlined as follows: Section A provides table summarizing the notations used throughout the paper. In Section B. we present a collection of technical lemmas crucial to our analysis. Additional variational representations for certain f-divergences explored in Section C, where we also further discuss on why using tighter variational representation of f-divergence important. Section D restates our theoretical results, provides detailed proofs, and introduces supplementary theoretical findings. For more details about experiments and additional empirical results, refer Section
A Summary of Notations
For easy reference, Table summarizes the key notations used in this paper.
Table 5: Summary of notations.
Notation
Definition
X, y. H
input, label and hypothesis space
source domain distribution and target domain distribution
S,
source sample {(X,,, Y and target sample T { pm empirical source distribution and empirical target distribution loss between the predictions returned by h and h'
P. ((h(z), (x)) or h(h, (h), (h)
(h) A E. [((h(X). ()),, (h) & E. [e(h(X), Y)] Sml (((()),
Ra(h)
(P(I))
EQ Φ dP f d-dvergeence between P Paddd Q
D (ulle)
E [e(h, - 1 (L))))
SUPWEH
min EH (h²) + Rh"h")
Rs(F)
(Z.)] (empirical) Rademacher complexity E Er... [supJeF 1= ( ((h(x), h (x)) h E H}
He (VIlA) (teo (xx)
SUPMEHHTE Eg [t- E(h, h')l - 1 (te otlo f-DD
infa {E (t (t- E(h, h1') + a)] a
d(r +1)
(t)
inf Ep IV (t -(h, h!) + +
kt(t) to Hr tae (vllk)) x (h) Hrop. Hets (h. N²) 8 (h.((1.1))
supwee ((t)
optimal t &achieving the superum in D h. (ullu) {n E (h) S r}
1 SUPWeEH,,.120 ED [te((, h 1 (tl
min EP. R,,h") + (h*))) SupWEE, Ey h(()
(Trop XX' + 2 and (han Z + Y} E₁ [(h, M) E. Ø- (e(h, w))l
surrogate loss used in practical algorithms
Ep [(h))n)] 1 (l h²)
B Some Technical Lemmas
The well-known Donsker-Varadhan Varadhan representation of KI. divergence is given below. Lemma B.1 (Donsker and Varadhan's 's variational formula). Let Q, P be probability measures on O for any bounded measurable function θ - R. we have DKL (QIIP) = sup, Ba-Q [0))
log Eg- f(0).
16

---

The following lemma is largely used. Lemma B.2. Let o' be the Fenchel conjugate of 6 and (er = 0(x+1), then 1 (z) 6² 6'(x) (z)-
Furthermore, if & satisfies ((1) = O, we have v (x) 2 0, equivalently 6 (x) 2
Proof. By definition, ()())) = sup; xt (t + 1). Let L 1, then ( = supy x(t' 1)
((t') = x0 (x) - Z. If (1) = 0, then 6 (x: = sup; xt (tt) 2 ((1) I. This completes the proof.
Definition B.1 (Empirical Rademacher Complexity [34]). For any function class F= [s: - R). the empirical Rademacher complexity defined as Rs(F) _ Re,,, [sup]EF -i=1 ci where S ( and E1mm sequence of i.i.d. Rademacher variables.
The Rademacher complexity-based generalization bound is given below.
Lemma B.3 ([70, Theorem 3.3]). Let F be family of functions mapping from to |0, 1] and let i.i.d. sample S {Z+)==> we have Es [sup)er (Ez [f(Z)] Ei f(z))) - 2Es Ris (Fs(F))
Then for any J E F, with probability at least 1 - over the draw of S, we have
[(Z,) (F) +0 (V log(1/0)
B(S(Z)) <
The following result is from [21, Proposition 4.1.], and the corresponding detailed proof is given in [22, Corollary 6.3.11].
Lemma B.4. Assume that o is twice differentiable on its domain and ø" is monotone. Let v(z) 6(x + 1) and let f X - -aa,b,, b), then for each E R, we have inf, Ex [3'" (tf(X) + +a)l - (b-a)² (1"(1)
.t.
The following two lemmas are used for deriving fast-rate bound.
Lemma B.5 ([71, Lemma 2.8]). Let (xx) e'-z-1 be the Bernstein functioon, If a random variable X satisfies E [ and S b, then [ex B(b)[(x))
Proof. It's easy verify that B(x) is increasing function for O. Thus, B(x) S B(b) for x S Then,
e ++ r + B(Bb).
For the bounded random variable with zero mean, we have
E E S [X S [())[(}]]
(6)
The last incquality is by e" 2 r + This completes the proof.
Lemma B.6 (Talagrand's inequality [72, Theorem 5.4]). Let and let F be class of functions
from X + IR. Assume that sup E [f()] f(x) < for any x. Then, with probability at least 1-8, 8,
a
2Var(f(X)) log(1/5)
4b log(1/5)
sup [[(X)] Σ ((x)) < 2E fer
sup E [f(]]] - Σ f(Xi) [EF
+V
n
3n
C
Variational Representations Beyond KL Divergence
C.1 --Dveergeecee
For 2 diverrgence, let Φ(x) = (x 1)2 for x > 0, then g"
y² 4 + y-
Simply plugging ø' into Lemma 2.1 will give
EQ (o())))
(P( = sup Er [g(0)] - E. [a(
(7)
4
17

---

Similarly, simply plugging 6- into Lemma 2.2 will give
(PII)) = sup Ep la(0)] Eeq [b(0)) Varq (g(0))
where the optimal a = Be [g())] in Lemma 2.2.
Notice that Vare (o(0)) Ee [o(0))²] we can see that, lower bound of x², Eq. (8) is tighter than Eq. (7),
If further consider the affine transformation of Lemma 2.1 and the scaling transformation of saateeessey-Chhhmaan Robbins lower bound and the Cramér-Rao and van Trees lower bounds [30].
Lemma 2.2, we can recover
More precisely, let g(-) ag(-) ) + b be substituted to Eq. (7), where a, E IR, it easy to see that
(ag(0) + b)
4 + (0) + = sup (Ep [g(0)] - Eg (o)))
(P)()) = sup Ep (0) + bj EQQ g.a.b
Varq (o(0))
where the optimal a" 2(Er Enl and = -a Eq [o())
Varo(
Let g(-) = (.) be substituted to Eq. (8), where E R. then we have
x² (PI1Q) = sup Ep [tg(()] - EEQ [tt( Varq (o(0) = sup (Ep [g( Eq [g()))) g.²
4
Varq (g(0))
where the optimal 1" a" -Bol Therefore, Lemma 2.2 recovers Eq. (9). Varg
C.2 Reverse KL Divergence
( The reverse KL. divergence DKL. (QI[P) can be simply obtained by exchanging the orders of and Q in the KI. divergence DRL. (Pl In this case, the DV representation reverse KI. is
(kl((
DKL (QIIP) sup Eq [9()) log Ep [0(9)
(10)
gEع
To obtain this from Lemma 2.2, let ((x) = log(z). then plugging (²(y) = -1 1 -looo- into Lemma 2.2, we have
DKL. (QIIP) sup Eq (g(0) + aa + 1 (llo((9((0) ai].
g.a
Now reparameterizng 9 + a e0+a
we have
DKL (QI = ssup- 100-RR EQ [0(0)+a) + + Ep (g( + a] sup E.g [g(0)] log Ep [c000]]
9.0
This recovers Eq. (10).
C.3 Jeffereys Divergence
Jeffreys divergence, member of the t /-divergence family with d(x) (x - log is the of KI. divergence and reverse KI. divergence. our algorithm implementation, we obtain the variational formula for Jeffreys divergence by simply combining the variational representations of KL and reverse KL, namely
sup Ep (n( - log B2Q [0000] sEg
+ sup Eg [o( - log Ep [ca(e)]
gEg
Moreover, there is tight relationship between Jensen-Shannen (JS) divergence and Jeffreys diver- gence [73]: Das (|[)) < Dieetterryss (PIIQ), log use JS divergence our algorithms, minimizing Jeffreys divergence simultaneously minimizes JS divergence.
I+t (P .Although we don't directly
2
18

---

C.4 Understanding the Importance of a Tighter Variational Representation
In simplified terms, Lemma 2.1 gives a variational representation expressed as:
(P(l)) sup Vi (g), gE
while Lemma 2.2 provides variational representation as:
D&
sup (g) gEg
where we have expressed the quantities to be maximized in Lemma 2.1 and Lemma 2.2 respectively as V (g) and (g). It is clear that for each specific E G, we have (2(g) 2 (g) (since Lemma 2. special case of Lemma 2.2 where a 0). This is when we say V22 is a (point-wise) tighter variational representation than Vi.
In the two optimization problems above, the optimal solution in both cases gives rise to same divergence. That is. when the two optimization problems can be solved perfectly, there is no advantage of one representation over the other. However, when the global optimal of the optimization problems is not attainable, any g obtained in an optimization effort gives rise to an estimate of the divergence, namely
(g) the first case and as (2(g) in the second. It is then clear that due to V/2 being tighter variational representation, V22 consistently provides a closer approximation to (PIl)) than V1 for the same g.
This virtue carries over to the min-max training strategy of UDA. In that case, y is reparameterized by Il h'. We desire the inner maximization, whether of V22 of Vi, give rise a good estimate of a similar divergence while the outer minimization aims to reduce this divergence. If the inner maximization is perfect, then the choice between V1 and V2 makes difference. However, if the inner maximization isn't perfect, 1/2 gives a point-wise superior result compared to Vi, leading to flatter optimization region around the optimal solution. In other words, using V2 as the maximization objective gives a better estimate of the corresponding f-divergence than V/1
D Omitted Proofs and Additional Results
D.1 Proof of Lemma 3.1
Lemma 3.1. Let X* ming-ex EH (h²) + (h²), then for any E H, we have (h) S (h) + "
(m|lv) + X
Proof. For a given h E H. let a E R,
(h) (h) SE, [e(h, h h²]] + E, [e(h [o]] - E, [e( fu]
(11)
iinf E. [e(h, h1) a - a E, [e(h" (()( - E2 [e(h, f.Jl
< inf Ey 10* (E(h, (4(1,10)+
+ a:] - a + E, [f)),, B,
B, (((h,,2)
(12)
< inf Ex 10" (E(h, + + - a E, [e(), + EE (e(h", + E2 [e(h",
fu))
(13)
< E, 16t (L(h, (²) + (()) a - E, (((h, +
<D (wllp) +
where Eq. (11) and (13) are by the triangle property of loss function, Eq. (12) is by Lemma B.2, and the last inequality is by the definition of f-DD. This completes the proof.
D
D.2 Proof of Theorem 3.1
Theorem 3.1. Let ((.,)) :) E [0, 1]. Then, for any h E H. with probability at least 1 8, we have
(AllE) + 2cH)s(H)) + 48h (HC") + x (V (Voox(110) /log(1/))
+ log(1 1/5)
(h) S (a)h)) + D h.H KL
where H. {x + k(h(()), (()))),, H}.
19

---

Proof. We first prove concentration result for the DA.33 ( measure.
Let B be constant such that B min Ey elchh)) Bo [csthNi)} for any 0, h', then for any given h, with probability at
Lemma
D.1. Assume
[0, 1].
least 1 8, we have
D (ullP) DRE(Ol))) & 28s(H)) + RT(exxoHl) +0 [log(100)
(
(1/6)
1
m
Proof of Lemma D.I. For a fixed A E H. we have
(pll)) [)) - B. (Alle)
KL
En ((hh,1)) log Ey [c9th.)]]
TE (e(h, R() - log lEg [ceth:]]]]
sup
sup NEW
WeW
Ry [((h, h()] - log Ey [c2(h.N]]
(Rp (((h,))) - -log BB [ccchN)))
s sup MEW
(14)
log E,, [cllAAAs]] log Bo [e(chN3]]
sup NeW
E [e(h, h')l - EE [e(h, h')ll
sup N'E
(og((/6)
<29Rs(H) + +
sup NEH
log Eo [e(().0)) log Be [cc(hNN)]
(15)
where Eq. (14) by sup |U| sup IV s sup sup - |V| S sup s -IVl I sup UU V| and Eq. (15) is by Lemma B.3.
Let 3 be constant such that < min { LE (e(hh)))
.Bo [ccrnmo]} for any D, h, h. For given
W, W.L.O.G.. assume that E, [e (h,N'))
2 E. (((
then
Eg ((h,h'')
Ep
A log E, ee(h)))))
=log
1
Ev ee((id0)] -1
E5
E, [f((h.)'))
[E (aand)) - E. [c
R2, [e((n.b']]
< Es [e(thN))]
- Ev [epronn0]]
(16)
where the first inequality is by log(æ 1) for >> O, and the last inequality is by the definition of B.
Plugging the inequality above into Eq. (15), have
KL (ull)) - BA.H -DAM (All2)
(VDE [og((10))
<2 +
Ep [(N))]]] - E, [e [0
sup WEH 3
log(1/6)
log(1/o)
<29rs(H)) + Ri (exp 7
+
(17)
where the last inequality is again by using Lemma B.3. This completes the proof.
20

---

Recall Lemma 3.1, we have
(h) (,) (Mlll) + x*
(V
1og(1//0)
(bp(h) + (H) +0
+ Dha (1|122 +
(18)
+ Rkp(1) + 18s)HH) + Texp Dh (A|l()) + +0 (v
[10((10)
log(1/0) (
+
11
m
(19)
2e <Rp(h) 49ts (HH)
3 RT (allo) + (11)+)))++++++ (1 log(L/0)
log (1/6)
71
m
(20)
where Eq. (18) and Eq. (19) are by Lemma B.3 and Lemma D.I, respectively. For the last inequality, notice that the loss bounded between [0, 1] so the exponential function is e-Lipschitz in this domain,
then we apply the Talagrand's 's lemma [70, Lemma 5.7], i.c. (expo... s (H-).
Finally, due to the boundedness of the loss, there always exists constant 3 [1, such that B < min By [e((h,))) Eg [e((h.9)] }. and simple choice is 3 = 1. Plugging 8 = 1 into Eq. (20)
will complete the proof.
D.3 Joint Error-Free Target Error Bound
[33] presents a X "-frree target error bound by using the cross-domain error min ( (f.), (f,))) We now illustrate that it is possible incorporate this term into all of target error bounds.
In particular, the X term appears due to the following triangle inequalities:
Ro (h) - (h) SE, [E(h, ((h,""") + E, [e(h", E, [8(h, fw)l
IE, [e(h, E, (((h", F, [e(h, h²)] + B, [e(h",
E. [e(h, - RE [e(h, h()) +\*
To avoid X", we use fu and fu instead as the middle hypothesis for the application of triangle inequalities:
(h) (h) Ey 1e(h, E2 (6(h, fuJ] EE (e(fr.
(21)
Au (h( - (h) < E. [E(h, fa) - E,, [e(h, fu]] +E, [e(Sun fu)].
(22)
As
If [Ag] and |As] are upper bounded by some hypothesis class-specific divergence du v), then combining Eq (21) and Eq. (22) will give us:
(h) s (h) + due dn, v) + min(Rr (fx)), R,((t)
where (f,f)) E, [e(ff. fu)l and (fx) E, fw)] Clearly, if 7 is large enough such that fo and fy are all achievable, |A1,, and |As] are upper bounded by our f-DD defined in this paper. In fact, as long as fu and fo are close enough to H. our f-DD can upper bound [Ag] and [Ag], with similar adaptations used in [33, Lemma 4.1].
D.4 Proof of Lemma 4.1
Lemma 4.1. Let (x(x:) & o(x + 1), and s/ is its convex conjugate. For any W., h E H and t E R. define Kw,,00 A infa E,, 1wb 1.-,8h, h²) + a)). Let Ky(() = -supwen (1), then for any (E, (L,, [((h, En (e(h, (/)) S D (x((()))))
hi H.
where KJ is the convex conjugate of K,.
21

---

Proof. By Lemma B B.2, we know that
(t) inf p 16s (te(h, R1) + aa]]
= inf Eu 1 (Le(h,, + a)] -a a B [e(h, = r u. (1l (1lo))) 11) BB (e(h,
According to the definition of Ky for each 1, we have
Ke (1) 10 h²) 1E [e(h, Kn(1).
This indicates that
tE [e(h, h') tBy [e(h, h')l + (tll))))))))))) th (e 1(h, h')l S (t)...
Since this inequality holds for any t, by rearranging terms, we have
sup & (E,, [e(h, k')] En [e(h, W)I) (t) I suptE, [e(h, ((),,')] (tl h') s D3 (v(lk). (t
(23)
Notice that the most left hand side the definition of Kj thus have
K₁ (E (e(h, B, (e(h, s D h. ((l(())))))))) (wlla).
This completes the proof.
D.5 Proof of Theorem 4.1
Theorem 4.1. For any h H. we have
h.H (vllm) (vll))) + Ky(() X²
(h) S Ra(h)+ + inf 120
(3)
Furthermore, let l [0, if o is twice differentiable and Φ" is monotone, then
(h) S (h)
v (0
2
D h. (v(l() X
Proof. We first follow the similar developments in Lemma 3.1.
(h) (h) SEE [e(h, + E, [e(h", fz)] Ex, [f(h, fu)l
SE, [E(h, E. [ Ep, [E(h, h + E, [e(h", fw)
= Ev (((h,,")] E, [e(h, h()] +X"
By Lemma 4.1, we know that K2 (A) *D. (u]lι). This indicates that
( 2 suptA (t) sup tA (t).t).
tER
120
We hint that with more careful handling of the aforementioned step, obtain bound for the absolute mean deviation (i.e. (Al) in the end. However, the current development suffices for our immediate objectives.
Notice that when O, this holds trivially. When t 0, the above inequality is equivalent to
A < inf 1>0 f
(t)(t) + th (w/lz)).
Hence, we have
A < inf
(t)(t) + DD ****((
10
/
22

---

Plugging the bound for A into the inequality at the beginning of the proof. we obtain the first desired result
(h) R2 S inf (vx(1) D D (vlle)) + X
120
For the second part, we apply Lemma B.4 here, then it is easy that kx(t) S
Therefore,
D.. (vlla)
2D h. (ullp)
(h) - (h) inf (1) + 1
+
+ +
(1
where the last equality is obtained by lettingt = 2220 h.H (v|lp). This completes the proof.
D.6 Proof of Corollary 4.1
Corollary 4.1. Let l E [0, 1). thenfor any h E H. we have (h) S (1) + v 22 (u|li) + X
Proof. In the case of KL, ((x) x log z + 1, and 1/0" (x) Hence, this corollary can be directly obtained from Eq. (4) in Theorem 4.1 by substituting ("(1)
D.7 Proof of Lemma 4.2 and Generalization Bounds for f-DD
Lemma 4.2. Let& e [o, 1] and let to be the optimal achieving the superum in Dh (vllp). Assume o² is L-Lipschitz then for any given with probability at least 1 5, we have
(V 1og(1/0)
log log(1/8)
Th (vllm) & th (0llA) + 2 Ita ( + 2L [ta| (H)L)
171
Proof of Lemma 4. For fixed h.
h (vllp) - DD.
= dhus sup E. ( M3) + Eap (ar - (loE((,, W/ + +a]] supsup Ry [te(h, h')] + R2 la (I((((, +
I dus sup E, (he h/) + En Ja - (tull(h, h²) + a)] (E, [toe( h')] + Ep la ('tol(( h') a)))
sup
En ( 11) Ep [loe(h, + supsup E 10 (loe(h, h²) + a:) - R. 16 (lol(h, h²) + a)]l
w
( 1108(1/0)
(0g(1/0)
<2 (6 R4 (HL + o
2is Home + 0
m
n
(o8((/0)
log(1/8)
<2 to Bt (HL) 2L [tl Rs(())))))))))))))))))))))) +o
m
where the last two inequalities are by the scaling property of Rademacher complexity and Talagrand's lemma [70]. This completes the proof.
The generalization bound for F-DD is then given as follows, which clearly shows slow convergence rate.
Theorem D.1. Under the conditions in Lemma 4.2. Let I be twice differentiable and d" is monotone, for any h H. with probability at least 1 8. we have
(h) s (h) +
DD
h. (0)() + V ( + Rts(H()) -0 (
1og(1/0)
+ 1og(1/0)
+0
+ +0
m
n
23

---

Proof of Theorem D.I. The concentration result for (h) - Ra(h) is exactly the same as Eq. (18)
in the proof of Theorem 4.1. Then putting everything together and by v VEE= Ty E J will complete the proof.
D.8 Proof of Theorem 5.1
Theorem 5.1. For any E Hr. and constants C1 C2 E (0, +00) satisfying (C1) S
Cr ((), any h/ Hy, the following holds:
(h) I (a) + th. (vllA) + C2 Xr,
C1
where X, min (h") + (h") and (h) = SupWEH, Ep [&(h, h')l
Proof. For a given h E Hr,, let h; = arg mins EH, (,(h") (h")), then following similar steps
in Lemma 3.1 and Theorem 4.1, first have
(h) - (h) S [e(h, [+] 1 [e(h, [,) + X
Then, we can apply Lemma 4.1 for bounding E, [e(h, h, - E2, [e(h, hT)] here, which gives us
(h) (h) S inf (t)(t) D (v(le())) X5. where (I(() suPWER, (w),,()).
(24)
The condition for C₁ and C2 exist indicates that (]((C)) = sup (w,((Ci) S (102T1 MEH,
Replacing / by Ci and plugging the inequality above into Eq. (24) give us 1 _ D Hr (vlle) + C2R (h) + +
(h) (h)
inf C.C2s C2
which completes the proof.
D.9 Proof of Lemma 5.1
Lemma 5.1. Let l E [0, 1), and let the constants C₁ > 0 and C2 E 1) satisfy the condition
(eCr - -1 -1-1] ---11 (1 min + r, 11 + C3 min(ri + r, +)) S C,C2. Then, for any h E Hr, and
h E H,, we have
D (ullK) (1 C2)LE [((h,
E. [((),)))) < inf C₁
Ci
Proof. Let C' = 1+ where C E (0, 1) and let g(-) = E(h(-), h(()), then we aim to bound the weighted gap: E, [g( C. Ix(X) By definition,
D (μ|1и² = sup tE, [g log E, [cto(x]]
g.²
ssu IE, [g( [X) C²IE,, [g(X)] log E,, [e(ooxx)
(25)
9.8
Then, recall that l < 1, and we use Lemma B.5 obtain that
log E,, ((((x))
<t(1 E lo( B(t){ [o(x)) C'E, 1(xx))²
=,()) ( -20)
B())( 20") (E, lo(x))) -(CC" 1)E, [x()]
=,())&E, [((())) B() 1)) (E, (x(x))) 1CEx,. [g()),,
(26)
'Note that, in this context, the random variable has a non-zero mean. Therefore, the first expectation term in Eq. (6) within the proof of Lemma B.5 should be retained.
24

---

where the function B-) is the Bernstein function defined in Lemma B.5.
Since 0 S y [x) & min(r +r, 1} and g(-) E [0, 1]. have
B [(g(X)) B(t) 1) (En (xx)))) - CEE (g()) BB(t) [x(x)] + B(t)+(C² - minffr +r, 1 [g(x)] CBn [g(X)
Our theme here is to obtain log E, [e(lo(x) [cfl0cx)-CCR.ioxxx < 0, so having the following incquality hold is sufficient
B(t)L B() min(ri + r, 1 [a(X) - Clu [o(X)] S 0.
Therefore, log E, [ - 0 will hold if the following satisfies for and C,
B(t) B()) min +r, C S 0.
Equivalently, substituting the expression the Bernstein function B(-) gives us
(c' min(mr +r, + C2 min(r +r, - sC,
(27)
Now let Ci and C C2 satisfy Eq. (27), then for any h hi, by re-arranging terms in Eq. (25),
we have
1
(D (ullm) + log [[010tX)--0+03),)xxx)) + C2 (0(h, h/)
Ev ((hh,h') Ep [8(h, 1()]
C1
n.Hl (u|lm) + C2E [e(h,
C1
where the last inequality holds because log E, (40166XX--(4CC)))bLXxXX < 0 when Eq. (27) is satisfied. This completes the proof.
D.10 Proof of Theorem 5.2
Theorem 5.2. Under the conditions in Lemma 5.1. For any h E Hy. with probability at least 1 8, we have
h.H. DLL
(DllA)
(h) Rh)
C2R
Ri (H..) His (H (Hmnxxtrmi))
C1
+ +0 (v
log log(1/8)
log(1/0)
(11 + log(1/8)
r
(log(1/0)
+ 0
+ X;
m
Proof. We first apply both Lemma B.6 and Lemma B.3 for bounding (h) with local Rademacher Complexity Rs(Hr) = Es,.. SuPNEH., E
e, (m)
(V() r log(1/5)
+ log(1/8)
sup (h) Rh) I2ES on sup Ru RM) - Ra(()) +0
n
<20s((H))+ (1 ( n
log(1/8)
(28)
n
where we use the fact that Var, (8(h, fu)) S (h) S r for l [0, 1].
Similar to Lemma D.I, let = to achieve the supreme in 1.H. (v(l))) Let 3 <
min Ey (ean)) Ep [e(thN]] for any à and h W E (e.g. 3 = 1), then the concentration
25

---

result of localized KL-DD is derived below,
Dit.. (vllm) Dkt. (ōllă) Ssup E [Boe(h, [()] Es [toe(h, 16 16]] + sup h
log 1 [evee(h.0)) log L2 [ct0e(nnAt)]
W/
Eu [ctue(h.A']] -Ea [ctrechuA3]]
suup Ey ( h(] Ep [tue(h, Will + sup h
(29)
( (
+ 2 e [tol Rs(H()) + log(1/6)
log(1 1
$2 to Ry + O
m +
m
n
(30)
where Eq. (29) follows from the similar developments in Eq. (16).
Putting Eq. (30), Eq. (28), Lemma 5.1 and Theorem 5.1 all together, we have
P
(h) & (h)= + inf
C1 +C
O (is (H2,) + Rs (H..) + Rr (H())
C1,C2
+ 0 1og(1/0)
log(1/6)
o (V 11 log(1/6)
+ rlog(1/0)
r rlog(1/0)
+
+ N.
m
Finally, using vā + vb S 2/a and O Ri (Hfr) Rs (Hh)) -0 0 (H (Hnsaxmri))) will conclude the proof.
D.11 Generalization Bounds based on 2DD
Theorem D.2. Let E(.-)) [0, For any Hr C1 0 and C2 E (0, satisfying C Var ((h,h)) < C2E (f(h, for any it E Hr. with probability at least 5, we have
D... (illA)
(h) S (h)
(2) O RR (H1.) + Rs (Himas(toos))) X;
C2
log (1g( (gg(1/8)
(11 + (1/) +
r (1g((/))
+
+0
Proof. We first give similar result Lemma 5.1 based on 2-DDD. Again, let g(.) = E(h, M.). By Eq. (8), we know that our localized 2DD
H. (ullm) = sup (Ew [g(X)] - E, (x())) ta Var (g(X))
D H.
Wit
4
Let C² 1 C' where C E 1). Then, for any 1 h',
E. - C. [x(X) < inf n. H. (ulla) 120 t x²
1
(Var, (g(X))
CTp [x(X)]
4
When and C satisfy (o(x)) CE2 [g(X)] < h.H, infe, d (vile) ClE,, [g(X)].
we have E, [g(X)] E, [g(X)) s
26

---

For the concentration result,
Di (ullm) D. x² (0llž)
X²
<supt tupt(e (EE, [g(x)] - En [x(x)] Vary (g(x)) t (Bo [o 1x [x(x)]) + Vara (a(X))
N,
4
12
=sup/ (Eev (a( - Ee (x)) + 1 (Ep [x(x) By (x(x))) + 4 (Vara (o(x))
Var, (g( (x()))
N,t
<2 [to Rs (H,) + 2 ltol Ri (H,) +0 (1
11 (1 log(1/6)
71 log(1/6)
log (1/8)
B (Varp (o(X)) Var (g(X)))
+ sup h 4
Notice that
t
sup h
(Varg (x))
Var (x)))
4
t
[B [x)]
Rp [x)] + (T (x))) - (Rpp (a (x)))))
sup
4
=sup 4
(Ep [o²(x))
Ep [9² (x) + (E, [x)
Ep (x))) (E, (o( + Ep (x()))
t Ep (x)] - -E [g [X)]
sup 4
+ sup M
(En [o(x)) - En (x()))
2
s + (V
log(1/5)
n
where we use the fact that g² 2 2-Lischhitz in g E 10, 1].
Notice that mentioned in Remark 4.2, 4o (E [e(h,]"')] -E, [e(),))))) Putting everything together Var, (E(,,"(()
and following the last several steps as in Theorem 5.2 will complete the proof.
D.12 Threshold Learning Example
Consider popular threshold learning example.
Example (Threshold Learning). Let X' = R. A threshold function, denoted he, is defined s.t. ho(z) = 0 if x < c and hz(z) = 1 2 c. Let the source domain j be an uniform distribution on
[0, and let the target domain be another uniform distribution on [0, 2]. Let H (hele E [0, }1} and assume the ground-truth hypothesis h for the both the source and target domains. Let E be
the zero-one loss.
b.H Recall that KL (иllи) supwit tE, (E,]((h,, - log tt(hh,, Hence,
hy H DRL. (ullm) = sup (-r : 2 5.1d0 fidx-logg (1² e'dax + S dddx sup (
1 el +
t 4 log 2
0.11.
2
Now let = 1/4, so H. thole E [ 1... by a similar calculation we have
hy 2 (vllp) = sup { 8 4 4
+ e² + = 0.048 and R. 4'
log(
Based on Remark 5.3, we can set C22 = 0.1 and C = 3.74. Hence, we have
h1 H1 (иllи) + 0.1 - R, = 0.038 < 1
hy
0.27 D
DRi
(и|lи) = 0.36.
27

---

Since h E Hr C. H. we have = X = in this case. This example justifies that the localization technique can significantly tighten the bound. Note that C2 0.1 and C1 3.74 are not the optimal choice so localized bound can be even tighter.
In addition, as more extreme case, let - so H, ( Then by similar calculation, we have
1 Ho (vllm)= = R O, while Dki. (v|lm) remains unchanged. Notice that Rx(hy) - R2(h2) = Dkl. 0 so localized f-DD gives tightest target emror bound in this case.
D.13 Proof of Proposition
Proposition 1.
Let da (h, W/) Eg [e(h, k)]
Th ( W.. Assume H is sufficiently large s.1.
L: X R. we have max (p, 1() = max di.0 (h, 16) D
h. (ØllA) S Do(PllA).
Proof. Since : X + IR, le : X' + lR and ul + a x + IR, then is straightforward to have max; d h/) max di (h, h²) D (AllD). Additionally, as te H! C 9. they upper bounded by (o(ll)))
E More Experiment Details and Additional Results
We adopt the experimental setup from [1] and build upon their publicly available code, accessible age some settings from the implementations of [9] and [49], which can be found at higgghhhhhcoo/uuu///////eeeeeeee and https://gttuu..com/// wow/thumm//DAN, respectively. Our code is available at http:///htbbb.oom//ggggaaaaaaaae
Additionally, we lever-
hets
Minimizing elassification error labeled data
h(x,)
ERM: min R (h)
x X
h
h'(x+)
hrep
f DDD: max d (h,ht)
h'(x,)
h'ts
Approsimating -divergence- based domaii discrepancy
Figure 2: Illustration of the adversarial training framework for f-DD-based UDA. The framework includes the representation network (hyep), the main classifier (hels), and the auxiliary classification network (Maz). It jointly minimizes the empirical risk on the source domain and the approximated f-DD between the source and target domains.
Specifically, for experiments on Office-31 I and Office- we utilize a pretrained ResNet-50 backbone on ImageNet [74]. Our f-DD is trained for 40 epochs using SGD with Nesterov Momentum, setting the momentum to 0.9, the learning rate to 0.004, and the batch size to 32. Hyperparameter settings and training protocols closely align with [9] and [49]. Particularly, on Office-31, we vary trade-off parameter for our KL-DD within [3, 4.5, 5.75). and for our "DD within [1, 1.75, 2]. For Jeffreys-DD, we choose 17,, 777227222 from 10. 1.2, 3, 3. 3.75. 4.5, 5, 5.75). On Office-Home, for KL-DD is chosen from [3, 3.75, 4.5), n for x 2DD from [3, (3.3.75) and 1771 1/22 for Jeffreys-DD from [0, 1, 1.2, 3.75, 4.5]. For the Digits datasets, we train f-DD for 30 epochs with SGD and
28

---

Nesterov Momentum, setting the momentum to 0.9 and the learning rate to 0.01. The batch size is set to 128 for M-+U and 64 for U- Other hyperparameters closely follow those used by [1] and [49]. The trade-off parameter v for both KL-DD and x is selected from [0.75, 1.. and 1771 1772 for Jeffreys-DD from (0, ,.010 0.55). All experiments are conducted on NVIDIA V100 (32GB) GPUs.
Comparison with f-DAL + Implicit Alignment [1] also investigate the empirical performance of combining f-DAL with a sampling based implicit alignment technique from [51], enhancing their original f-DAL. However, our findings reveal that our f-DD consistently outperforms f-DAL with implicit alignment, as detailed in Table 6. This observation suggests that the performance gain can be achieved by simply adopting more tightly defined variational representation. Furthermore, in Table 6, the entry labeled f-DD (Best) corresponds to the average of the maximum accuracy across KL-DD, DDD, and Jeffereys-DD for each subtask. Notably, this aggregated result shows slight improvement on Office-31 when compared to the individual performance of Jeffereys-DD.
Table 6: Comparison between f-DD and f-DAL
Method
Office-31
Office-Home
f-DAL
89.5
68.5
f -DAL+Imp. Align.
89.2
70.0
Jeffereys-DD
90.1
70.2
f-DD (Best)
90.3
70.2
Ablation Study We adjust the trade-off hyper- apper-parametery v) in KL-DD and present the outcomes for Office-31 and Office-Home, As depicted in Table and Table 8, we can see that the performance exhibits relatively low sensitivity changes in 17. Hence, the tuning process need not be overly meticulous.
Table 7: Ablation Study on Office-31 for KL-DD
3.75
4.5
5.5
5.75
A
+D
95.5=0.7
95.4+00.4
95.4++0.7
95.9++0.6
+A
74.5=0.1
74.6+0.77
74.6+10.4
74.5+10.5
Table Ablation Study on Office-Home for KL-DD
17
3
3.75
4.5
Ar-+CI
55.3=0.4
54.9=
55.3+0.1
Pr->Rw
80.7+0.1
80.8+0.2
80.9+0.1
Additional Results for Absolute Divergence In addition to Figure 1, we present the performance of the absolute KL discrepancy Figure 3(a-b). The consistent observations persist, indicating that the absolute version of the discrepancy tends to overestimate the f-divergence, leading to breakdown in the training process.
Table 9: Comparison between 2DD and Optx?-DD
Method
A-+D
Ar-+CI
x2DDD
95.0++0.4
55.2++0.3
DDDD
93.110.3
53.940.3
29

---

(a) KL (Office-31)
(b) KL (Office-Home)
(c) x² (Office-31)
(d) x2 (Office-Home)
Figure Comparison between D and Da. The y-axis is the estimated corresponding divergence and the æ-axis is the number of iterations.
Details and Additional Results for Optimal f-DD As outlined in Section 6, instead of invoking stochastic gradient-based optimizer (e.g., SGD) to update i, we utilize a quadratic approximation for the optimal as presented [23]. The approximately optimal KL-DD (OptKL-DD) is expressed as follows:
DA. (all() sup (1 + A(") Ey [e(h, - log Ey [<([+A))RONN]]
Here, a Gibbs measure defined as dj' a cenna') Ey (c and At is determined by the formula
At" E. [e(1, M() Ey [((),
Var (L(h, h"))
This approximation is obtained by [23] through Taylor expansion around with additional details provided in their Appendix B. In our implementation of OptKL-DD, similar to KL-DD, we use to replace l.
(a) x² in f-DAL
(b) x2 in f-DD
(c) KL-DD
(d) Jeffereys-DD
Figure 4: Visualization results of representations obtained by using I-SNE. The source domain (blue
points) is U and the target domain (orange points) is M.
For the optimal ²DD (Optx?- (-Dt)?-DD), have its analytic form, as shown in Appendix C:
(Ev [e(h, h')] E2 (((h,h'))² Var, (((h,h')
( (vllp) sup h
By replacing é by i, we utilize the above the training objective in our implementation. Table illustrates that the optimal form does not improve performance two sub-tasks, namely A-DD and Ar- and in fact, may degrade performance. Consequently. when the surrogate loss itself is unbounded, optimizing over t may not be necessary, at least for x" x DDD and KL-DD.
To visualize model representations (output of hrep) trained with f-DAL, X²DDD, KL-DD, and Jeffereys-DD, we leverage t-SNE [52]. In Figure 4, we present visualization results using USPS (U) as the source domain and MNIST (M) as the target domain. Notably, while the original /DAL already provides satisfactory results, our f-DD achieves further improvements in representation alignment.
Visualization Results
30

---

More Advance Network Architecture While our empirical study primarily aims to compare with [1] and improve their algorithms, in this section, we also evaluate our f-DD on more complicated models, which can serve as a valuable reference for practitioners.
In particular, update original backbone, ResNet-50, with pretrained transformer-based models. Specifically, we use the pretrained Vision transformer (ViT) base model (vit-base-pater sasse-pptch16-2222 [75] and pretrained Swin-based transformer hata--bas--pacch---nnneeee other settings in our algorithms unchanged. The results on Office-31 are presented in Table 10.
[76], while keeping all
Table 10: Accuracy (%) the Office-31 benchmark.
Method
W
D W
W -
A A
D
W A
Avg
ViT-hased [75]
91.2
99.2
100.0
90.4
81.1
80.6
91.1
SITT-ViT [77]
97.7
99.2
100.0
98.6
83.5
82.2
93.5
PMTrans-Ve VIT [78]
99.1
99.6
100.0
99.4
85.7
86.3
95.0
Ours-ViT
98.0 0.2
99.2 +
100 - 0.0
98.6 0.2
83.5 1
83.9 0.4
93.9
Swin-based [76]
97.0
99.2
100.0
95.8
82.4
81.8
92.7
IMTrans-Swiid (78)
99.5
99.4
100.0
99.8
86.7
86.5
95.3
Ours-Swim
98.7 +: 0.2
99.2 + 0.0
100 + 0.0 99.2 0.2
86.1 + 0.1 85.6 +: 0.3
94.8
Results on models other than ours taken directly from [78, Table 2]. Here ViT-based and Swin-based refer to source-only training using ViT and Swin transformers, respectively, and "Ours" particularly refers current SOTA, P.Trrns, (especially for ViT), but close to it, this should demonstrate the practical utility of the proposed approach. Notably, our approach not only significantly improves the respective backbones but also outperforms the strongest baseline, SSRT-ViT [77], as compared in [78].
weighted-Jeffereyy discrepancy. While
method does not outperform the
The fact the our approach does not outperform PMTrans should come at surprise. In particular, PMTrans and all other compared models all include while our approach not only neglected these ingredients but also simply use hyperparameter based further tuning.
earlier ResNet-50 settings without
One notable difference between our method and other SOTA approaches is the absence of any pseudo- labeling strategy our framework. Specifically, our theoretical framework is constructed under the assumption of zero knowledge of target labels. In a strategy that successfully exploits pseudo- labelling, usually explicit or implicit prior knowledge are available (e.g.. for the purpose of selecting hyperparameters relating to pseudo labeling). For example, such knowledge may present itself as labeled validation set in the target domain. Such knowledge, if available, may change our problem setup and require refinement of our theory. We anticipate that combining our method with advanced pscudo- lecudo-lbbeligg strategies, data augmentation techniques like Mixup (which itself inherently relies pscudo-labels), and label smoothing will improve results, possibly approaching the current SOTA performance.
31

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We highlight the motivations and contributions of this work the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the last section of the Appendix, we discuss how the lack of pseudo-labeling techniques makes our framework slightly weaker than the current state-of-the-ee algorithm in certain tasks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer:
[Yes]
Justification: All assumptions are given in the theorem statements or in the preliminary section, and all the technical details provided in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our implementation closely follows the previous works, pointed out in the experiments section, with our source code repository referenced in Appendix E. Additionally, we explained all the additional details that needed for reproducing our results in main text and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
33

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We only some well-known public DA datasets, which be easily found, and we provide our source code in the supplementary materials and point out the source code repository in Appendix E.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
httaa//gguiiee/odduuumiissss sosioooooisi for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (httpa: // httppsscccppplcc//pcccce
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the standard deviations of our experimental results in the tables, except for Office-Home. Although we obtained the standard deviations for Office-Home, we excluded them from the table because previous baselines did not report them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: / hwwrrps.....///
inne??
Answer: [Yes]
Justification:
Our work conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not see any socictal impacts with our paper, as it is mainly theoretical in nature.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
35

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There is no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the datasets and models properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "mIxOtDRrHS", "reviewer": "Reviewer_zRKu", "summary": "In this paper, new expected risk analysis based on f-divergence is provided for the unsupervised domain adaptation problem. Although there are prior researches on expected risk analysis based on f-divergence, several issues have been pointed out, such as the fact that the variational representation of f-divergence used in these studies does not recover the Donsker-Varadhan representation of KL-divergence, and the use of the absolute value of the variational representation as a measure of domain discrepancy. \nIn this paper, to address these issues, the authors adopt an alternative variational representation of f-divergence and, based on this, provide an upper bound evaluation of the expected risk in the target domain, namely ``target risk $\\le$ source risk + marginal domain discrepancy + joint error''. Additionally, a sample approximation version of the derived upper bound is also provided, allowing it to be estimated from the data  (excluding the joint error part, as in conventional bounds).", "questions": "- Are no assumptions specifically placed on the hypothetical set $\\mathcal{H}$?\n\n- In Lemma 4.2, is there a way to estimate the value of t_0 when t_0 cannot be written in closed form (e.g. for KL or Jeffereys employed in the Experiments?)? \n\n- The Rashomon set used in Section 5 appears to be in fact the set to be estimated (as the true expected risk of the source domain is unknown). How exactly is the Rashomon set constructed in this paper?\n\n- In the experiments, three types of discrepancies are evaluated for the proposed method, namely KL-DD, $\\chi^2$-DD and Jeffereys-DD. Then, do you have any insights on which discrepancy measure should be used for which type of problem? I think the question of which measure to use to evaluate domain discrepancy is critical not only for theorists but also for practitioners.\n\n- Is the f-DD proposed in Definition 4.1 always 'better' than the existing f-divergence-based discrepancy (Definition 3.1)? I am wondering whether there are cases where using absolute values to define the discrepancy is an advantage.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly discusses what are difficulties  with the conventional DA theory using f-divergence and explains how it is solved by the proposed approach. Especially, this paper provides a solid theoretical foundation, with detailed assumptions and rigorous proofs that are well-documented in the appendix. \n\n- Previous expected risk bounds in UDA have often been given by relatively simple inequality evaluations, following the formulation given by Ben-David et al. In contrast, a similar upper bound evaluation using the f-DD proposed in this paper requires an inequality evaluation for ``change of measure\" (as given in Lemma 4.1), and it can be seen that this is not an incremental extension of the conventional DA theory.", "weaknesses": "- I don't think there is enough information needed when trying to calculate the derived upper boundary from the sample. For example, $t_0$ in Lemma 4.2 and the construction of the Rashomon set in Sec 5 should be discussed in more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you sincerely for your positive evaluation and constructive comments. Our responses follow.\n\n>- Are no assumptions ...?\n\n**Response.** Our theoretical results, except for Proposition 1 which requires $\\mathcal{H}$ to be sufficiently large, do not rely on any specific assumptions about $\\mathcal{H}$. The hypothesis class can be either finite or infinite. Additionally, in our localized $f$-DD, we restrict the entire hypothesis class to a smaller Rashomon set. While we do not require any assumptions on $\\mathcal{H}$ here, we focus our analysis on the relevant hypotheses, as those that perform poorly on the source domain are unlikely to be found by any effective UDA algorithm.\n\n>- In Lemma 4.2, is there a way to estimate ...?\n\n**Response.** We have discussed the method for approximating $t_0$ (or the optimal $t$ equivalently) in Lines 339-354 (and Line 1183-1195), which is based on a quadratic approximation, and compared the results with simply setting $t_0=1$ in Table 4. Moreover, there are other methods to estimate $t_0$ as well. For example, assigning $t_0$ an initial value and considering it a trainable parameter, then applying SGD to update it during training.\n\nWe will highlight the estimation of $t_0$ in a more noticeable way in our next revision.\n\n>- The Rashomon set used ... How exactly is the Rashomon set constructed in this paper?\n\n**Response.**  Please notice that in our experiments, we do not need to explicitly construct the Rashomon set or determine its threshold value. The hypothesis found by UDA algorithms is always within some Rashomon set of our interest. In practical scenarios, where the true risk of the source domain is unknown, its empirical risk serves as a proxy for the true risk. As long as the UDA algorithm minimizes the classification error of the source domain, the obtained hypothesis falls within a restricted hypothesis class, i.e., the Rashomon set with some relatively low threshold value $r$, although the exact value of $r$ may be unknown as it relates to the generalization error of the source domain.\n\nWe believe the confusion might stem from the motivation behind our localized $f$-DD. We apply localization techniques to show that the convergence rate based on $f$-DD could be faster if we restrict the entire hypothesis class to the Rashomon set. This approach aligns with practical applications because, for any UDA algorithm, regardless of how the source and target domains are aligned, a good classifier must first be trained on the source domain through empirical risk minimization. Given that the hypothesis should first perform well on the source domain, many hypotheses are not of interest in theoretical analysis. By removing such redundancy, the UDA algorithm enjoys a fast-rate generalization guarantee (cf. Theorem 5.2). Consequently, the training objective in our algorithm, namely Eq. (5), does not need to include a square-root function for $f$-DD, in contrast to Eq. (4), which considers the entire hypothesis class.\n\nAdditionally, to demonstrate the advantage of localized $f$-DD, we provide a threshold learning example in Appendix D.12, where the Rashomon set can be explicitly constructed since the ground truth hypothesis is known.\n\n>- In the experiments, three types of discrepancies are ...\n\n**Response.** When comparing KL-DD and our weighted Jeffreys-DD, we always recommend using the weighted Jeffreys-DD. This measure is a weighted combination of KL and reverse KL (i.e. $\\gamma_1 \\mathrm{D}\\_{\\rm KL}(\\hat{\\mu} \\|\\ \\hat{\\nu}) + \\gamma_2 \\mathrm{D}\\_{\\rm KL}(\\hat{\\nu} \\|\\ \\hat{\\mu})$). In most UDA problems, the difficulty of transferring knowledge from the source domain to the target domain may not be the same as transferring from the target domain to the source domain. From this perspective, the weighted Jeffreys-DD provides more flexibility to reflect this difference during training.\n\nWhen comparing KL-DD and $\\chi^2$-DD, while KL is theoretically smaller than $\\chi^2$, their empirical performance is quite similar. In fact, $\\chi^2$-DD may even give better results on some sub-tasks, such as A$\\rightarrow$W in Office-31. We believe there may be some practical optimization benefits of $\\chi^2$ on certain sub-tasks (e.g., large quantity being more informative), which might not be easily explained by generalization bounds.\n\nOverall, we believe that weighted Jeffreys-DD is a good option in practice, especially when no further prior knowledge of the dataset is available.\n\n>- Is the f-DD proposed in Definition 4.1 always 'better' ...\n\n**Response.** In the context of approximating the true $f$-divergence between two distributions, our $f$-DD is indeed always better than the existing $f$-divergence-based discrepancy (Definition 3.1). This is because the additional absolute value function transforms all negative elements to positive, leading to a larger and more pessimistic approximation. Beyond the scope of approximating true $f$-divergence, while we are not aware of any hypothesis classes where Definition 3.1 is smaller than $f$-DD, from our current understanding, such cases are unlikely to be of practical interest. We are, however, open to discussing this further. Additionally, even without the absolute value function, the previous $f$-DAL defined in Acuna et al. (2021) is based on a weaker variational representation of $f$-divergence. As a lower bound of $f$-divergence, our $f$-DD is always better to theirs, as detailed in Appendix C.4.\n\nFurthermore, it is worth noting that while the $f$-DAL paper proposes an absolute value $f$-divergence-based discrepancy, it removes the absolute value function in their algorithm. Thus, the practical impact of the absolute value function has not been demonstrated in any existing works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KlP0tWCkto", "reviewer": "Reviewer_HkGQ", "summary": "This paper improves the theoretical foundations of UDA proposed by previous work, named f-DD. By removing the absolute value function and incorporating a scaling parameter, f-DD yields novel target error and sample complexity bounds, allowing us to recover previous KL-based results and bridging the gap between algorithms and theory presented in Acuna et al. Leveraging a localization technique, this paper also develops a fast-rate generalization bound. Empirical results demonstrate the superior performance of f-DD-based domain learning algorithms over previous works in popular UDA benchmarks.", "questions": "How to implement the training objective (5)?", "limitations": "Please refer to the weakness.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1) This paper holds significant theoretical significance in the field of UDA (Unsupervised Domain Adaptation);\n2) The proof of the theorem is very solid; \n3) The experiments are also sufficient.", "weaknesses": "1) The readability of the paper is poor. It is almost entirely composed of definitions, remarks, lemmas and theorems, lacking a figure to introduce the motivation of this paper and explain why the improved framework is effective. 2) It is difficult to reproduce the results, as the training objective (5) is very abstract and unclear how to implement it experimentally. 3) This paper requires a substantial foundation of reading other papers in order to be understood.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you sincerely for your valuable feedback on our paper. Our responses follow.\n\n\n>- The readability of the paper is poor. It is almost entirely composed of definitions, remarks, lemmas and theorems, lacking a figure to introduce the motivation of this paper and explain why the improved framework is effective. \n\n**Response.** Thank you for highlighting this concern. We understand that the paper may seem heavy on definitions, remarks, lemmas, and theorems, as it falls under the \"Learning Theory\" category, where theoretical analysis is often central.\n\nWe appreciate your suggestion to include a figure to improve readability. We have added a figure in the updated PDF that provides an overview of our improved framework (please see our general response). Additionally, we note that our improved framework is more effective because it provides a better variational representation of $f$-divergence compared to previous approaches. This, being one of our key motivations, is inevitably supported by a theoretical perspective.\n\n>- It is difficult to reproduce the results, as the training objective (5) is very abstract and unclear how to implement it experimentally. \n\n>- How to implement the training objective (5)?\n\n**Response.** We hope the uploaded figure in the general response could help to clarify the implementation of training objective. Specifically, it is implemented similarly to the adversarial training strategy proposed in DANN [R1]. To restate, our training objective is:\n$$ \\min_{h\\in\\mathcal{H}}\\max_{h'\\in\\mathcal{H'}}R_{\\hat{\\mu}}(h)+ \\eta {d}_{\\hat{\\mu},\\hat{\\nu}}(h,h').\n$$\n\nOur model is a deep neural network (e.g., ResNet-50) consisting of a representation network (i.e., $h_{\\text{rep}}$) and two classification networks. The main classification network (i.e., $h\\_{\\text{cls}}$) is used for predictions, while the auxiliary classification network (i.e., $h'_{\\text{cls}}$) is used to calculate the domain disagreement (e.g., $\\ell(h\\_{\\text{rep}} \\circ h\\_{\\text{cls}}, h\\_{\\text{rep}} \\circ h'\\_{\\text{cls}})$).\n\nFor the outer minimization, $\\min\\_{h} R\\_{\\hat{\\mu}}(h) + \\eta \\max\\_{h'} {d}\\_{\\hat{\\mu},\\hat{\\nu}}(h, h')$, we fix the parameters in the auxiliary classification network and jointly minimize the classification loss for the source domain data (i.e., $R\\_{\\hat{\\mu}}(h)$) and the approximated $f$-DD between the source and target domains (i.e., $\\max\\_{h'} {d}\\_{\\hat{\\mu},\\hat{\\nu}}(h, h')$). Note that $\\max\\_{h'} {d}\\_{\\hat{\\mu},\\hat{\\nu}}(h, h')$ represents the empirical version of our $f$-DD. Then,\nfor the inner maximization, $\\max\\_{h'} {d}\\_{\\hat{\\mu},\\hat{\\nu}}(h, h')$, we fix the main classification network $h\\_{\\rm cls}$ and then maximize ${d}\\_{\\hat{\\mu},\\hat{\\nu}}(h, h')$ by training the auxiliary classification network. \n\nThis adversarial training strategy effectively minimizes both the source domain risk and the domain discrepancy between the source and target domains. We will provide additional details on the implementation to ensure completeness.\n\n[R1] Yaroslav Ganin, et al. \"Domain-adversarial training of neural networks.\" Journal of Machine Learning Research 17.59 (2016): 1-35.\n\n>- This paper requires a substantial foundation of reading other papers in order to be understood.\n\n**Response.**  We acknowledge that understanding this paper may require some background knowledge. However, we have made efforts to self-contain all theoretical background within the main text or the Appendix to make it more friendly to a broader audience. We would appreciate it if  the reviewer could point out more specific areas that may be unclear or less accessible to general readers. We are more than willing to improve the presentation of our paper and welcome any constructive feedback that can help enhance its clarity. We sincerely hope that requiring comprehensive background knowledge is not taken as a negative point against the acceptance of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYVWGCB8VB", "reviewer": "Reviewer_CqSQ", "summary": "This paper aims to develop an improved version of f-divergence-based unsupervised domain adaptation (UDA) learning theory. In particular, the authors introduce a novel f-divergence-based domain discrepancy measure (f-DD) by combining the two existing concepts, which are f-divergence and domain discrepancy. Based on that f-DD measure, the paper next provides a generalization bound on the target domain, which is shown to be sharper than the existing related bound. The experimental results consistently demonstrate that f-DD outperforms the original f-DAL in three popular UDA benchmarks, with the best performance achieved by Jeffereys-DD.", "questions": "Please refer to my comments about the weaknesses of the paper.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. The idea of introducing f-divergence-based UDA, targeting a better risk-bound on the target domain is novel and interesting. All the main statements of the paper are theoretically supported, though I did not have enough time to verify all of those propositions/theorems carefully. \n\nThe experimental results consistently demonstrate that f-DD outperforms the original f-DAL in three popular UDA benchmarks, with the best performance achieved by Jeffereys-DD.", "weaknesses": "The novelty of the paper is quite limited since the f-divergence-based domain discrepancy measure (f-DD) is proposed by combining the two existing concepts, which are f-divergence and domain discrepancy. \n\nIn Theorem 5.2, the authors claim that the application of the localization technique gives a fast-rate generalization, they do not provide a concrete evidence. Could the author give some explanations/clarifications for that. \n\nMoreover, the experimental part of the paper seems not be very convincing since it only provides experiments with quite small datasets (Office31, Office-Home, MNIST & USPS) and simple model (e.g., Lenet). It raises the concern about capability of f-DD in more complicated settings with large datasets and backbone network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you sincerely for your constructive comments. Our responses follow.\n\n>- The novelty of the paper is quite limited since the f-divergence-based domain discrepancy measure (f-DD) is proposed by combining the two existing concepts, which are f-divergence and domain discrepancy.\n\n**Response.** We respectfully disagree with the assessment that the novelty of our paper is limited due to the combination of $f$-divergence and domain discrepancy. Our paper aims to advance the existing framework rather than merely combining these concepts. The title of our paper highlights that we present an \"improved framework\" compared to the existing work (proposed by Acuna et al. (2021)), reflecting our focus on improving the previous results. While the variational representation of $f$-divergence is indeed discussed in earlier literature, the novel aspect of our paper lies in its application within a new $f$-divergence-principled theory for UDA.\n\nWe now summarize our novel theoretical contributions here: 1) Acuna et al. (2021) introduced the absolute value function into their discrepancy definition, leading to a disconnection from the original $f$-divergence, and potentially allowing it to go to infinity (as illustrated in Figure 1 in our paper). Their Lemma 1 claims that their notion is upper-bounded by $f$-divergence; but unfortunately their Lemma 1 is flawed. Specifically, their Eq. (B.5) in the appendix is incorrect as they use $\\sup A = \\sup |A|$ whereas essentially $\\sup A \\leq \\sup |A|$, and hence the equality is not justified. This disconnection suggests that, strictly speaking, a rigorous $f$-divergence-based theoretical guarantee for the target error in UDA has not been established prior to our work; 2) In addition, we apply the localization technique in the analysis. The localization technique has been studied extensively in localized Rademacher complexity since 2005, or even earlier in the context of empirical processes theory. In UDA, before Zhang et al. (2020), the localization technique was mentioned in Example 4 of Hanneke & Kpotufe (2019). The novelty in our Section 5 lies in our novel application of localization: previous works like Zhang et al. (2020) utilize localization to achieve better sample complexity results  (e.g., $\\mathcal{O}(\\frac{1}{n}+\\frac{1}{m})$), while we directly incorporate the localization technique prior to proving the sample complexity results. In Lemma 5.1, we use it to remove the square-root function in Eq. (4) from Section 4, and as expected, it also helps to improve sample complexity in Theorem 5.2. Our proof techniques are in fact quite novel.\n\n\nWe sincerely hope the reviewer could reconsider the novelty and impact of our contributions in light of these clarifications.\n\n>- In Theorem 5.2, the authors claim that the application of the localization technique gives a fast-rate generalization, they do not provide a concrete evidence. Could the author give some explanations/clarifications for that.\n\n**Response.** We would like to direct the reviewer to Appendix D.12, as also referenced in Line 259 of the main text, where we provide a concrete example demonstrating this. Specifically, in the threshold learning example discussed in Appendix D.12, we show through straightforward calculations that the quantity \n$5*\\mathrm{D}^{h\\_{\\frac{1}{2}},\\mathcal{H}\\_{\\frac{1}{4}}}\\_{\\rm KL}(\\nu||\\mu)+0.1*R^r_\\mu=0.265$ is indeed smaller than $\\sqrt{\\mathrm{D}^{h\\_{\\frac{1}{2}},\\mathcal{H}}\\_{\\rm KL}(\\nu||\\mu)}=0.36$. This illustrates that the localization technique results in a tighter bound.\n\n>- Moreover, the experimental part of the paper seems not be very convincing since it only provides experiments with quite small datasets (Office31, Office-Home, MNIST & USPS) and simple model (e.g., Lenet). It raises the concern about capability of f-DD in more complicated settings with large datasets and backbone network.\n\n**Response.** For additional experimental results of more complicated networks, please refer to Lines 1201-1227 in Appendix E, where we present results using the pretrained Vision Transformer (ViT) base model and the pretrained Swin Transformer. Additionally, ResNet-50 was used for the Office31 and Office-Home datasets, while LeNet was used only for the Digits dataset, where it has already demonstrated strong performance.\n\nFurthermore, we have included experimental results on the VisDA-2017 dataset in the uploaded PDF file (please see our general response).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eruwfhwwEu", "reviewer": "Reviewer_EpWo", "summary": "This paper studies the learning theory aspect of the domain adaptation problem, where the key is to bound the estimation errors between expectations over shifting distributions. Specifically, this work improves the recently developed $f$-divergence-based generalization analysis, where the main results ensure a tighter generalization upper bound and the consistency between theory and method. For finite sample setting, a sharp bound is provided to accelerate the asymptotic rate. Numerical simulation is conducted to demonstrate the superiority of the theory-guided method over the existing discrepancy-based framework.", "questions": "There seem no major faults in this submission, and I only have the following minor concerns. \n\nQ1. Theory and methodology. The major result for the target error bound is provided in Eq. (4) in Thm. 4.1 and the specific bound w.r.t. KL-divergence is presented in line 162, where the induced learning objective consists of source risk and the square root of cross-domain KL-divergence. However, it seems that the optimization objective Eq. (5) considers the divergence without the square root directly. I understand the optimal solutions are the same for these two objectives (if the optimal solutions ensure 0 cross-domain discrepancies). But considering Eq. (4) is closely related to the major merit of this work, i.e., the tight bound, the consistency between  Eq. (4)  and  Eq. (5) seems to be important. Some justifications are highly expected. \n\nQ2. Method application. As far as I understand this work, the derived $f$-DD measure can be applied to existing works whose primary goal is discrepancy minimization. Thus, it could serve as a plug-and-play module for existing SOTA DA methods. Thus, some detailed discussions on the capability of $f$-DD w.r.t. existing methods are highly expected.\n\nQ3. Following Q2, apart from the experiments in the current version, some comparisons between SOTA DA methods and their combination with $f$-DD objective are highly expected.\n\nQ4. The clarity w.r.t. definitions could be improved, e.g., $K_{h',\\mu}(t)$ depends on the hypothesis $h$ while the justification (i.e., line 178) is provided after the definition (i.e., line 176). A thorough check for these issues could improve the readability.\n\nQ5. Some notations seem to be inconsistent. For example, the notation $I_{\\nu}^{\\phi}(h,h')$ in line 132 is inconsistent with $I$ in line 129; the notations $\\mathbb{E}_{\\nu}$ in line 132 seems to be incorrect (probably should be expectation over $\\mu$?).", "limitations": "The limitations are discussed in the checklist, and there seems no potential negative societal impact.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The motivation is clear, i.e., improving the $f$-divergence-based bound and bridging the gap between method and theory, and the presentation is easy to follow.\n+ The technical part is generally sound and the justifications are sufficient.\n+ The experiment results are superior compared with recently developed generalization bounds.", "weaknesses": "+ Some notations are inconsistent in theoretical analysis.\n+ The proposed algorithm needs further justifications.\n+ The experiment comparison could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you sincerely for your careful reading and valuable feedback on our paper. Our responses follow.\n\n\n\n>- The proposed algorithm needs further justifications.\n\n>- Q1. Theory and methodology. The major result ... the consistency between Eq. (4) and Eq. (5) seems to be important. Some justifications are highly expected.\n\n**Response.** The objective in the algorithm is in fact motivated by the findings from Section 5, where we demonstrate that when the error of the source domain is small, the $f$-DD term without the square-root function should more accurately reflect generalization (e.g., see Remark 5.3). Given that the empirical risk of the source domain is always minimized during training, we believe the final hypothesis lies within the subset of $\\mathcal{H}$ (i.e., Rashomon set with $r_1$). Consequently, using Eq. (4) in this case is weak (as it has slow convergence). Note that $R^r_\\mu(h)$ in Section 5 is upper-bounded by $r+r_1$, which we believe should not tend to infinity at the end of training. \n\nWe will provide additional justification to clarify the proposed objective function in Eq. (5).\n\n\n>- The experiment comparison could be improved.\n\n>- Q2. Method application. As far as I understand this work, the derived $f$-DD measure can be applied to existing works whose primary goal is discrepancy minimization. Thus, it could serve as a plug-and-play module for existing SOTA DA methods. Thus, some detailed discussions on the capability of $f$-DD w.r.t. existing methods are highly expected.\n\n>- Q3. Following Q2, apart from the experiments in the current version, some comparisons between SOTA DA methods and their combination with $f$-DD objective are highly expected.\n\n**Response.** Thank you for your insightful suggestion regarding our $f$-DD being a plug-and-play module for existing SOTA methods. In fact, we do compare our $f$-DD with SOTA results on the Office-31 dataset, as shown in Table 10 of Appendix E. In particular, for a fair comparison, we replaced ResNet-50 with pretrained Vision Transformer (ViT) and Swin Transformer backbones. The results are close to the SOTA, with $93.9$ (ours) vs. $95$ (SOTA) for ViT and $94.8$ (ours) vs. $95.3$ (SOTA) for Swin-based transformer. In Lines 1214-1227, we provide a detailed explanation for why our $f$-DD currently does not outperform SOTA. Specifically, our $f$-DD lacks additional ingredients such as advanced pseudo-labeling techniques, Mixup, and label smoothing, which are commonly invoked in UDA algorithms.\n\nWe acknowledge that integrating $f$-DD with SOTA methods could be beneficial, though it may require certain efforts for adjustments due to the complicated training objectives in these methods. Thus, rather than directly combining $f$-DD with SOTA methods, which could be complex, we have explored combining domain Mixup techniques with $f$-DD. Additionally, in response to other reviewers' requests for experiments on a larger dataset, we have included results on the VisDA-2017 dataset in the uploaded PDF (please find it in our general response). These results demonstrate the potential of $f$-DD to achieve SOTA performance.\n\nWe will include these new results in the next revision.\n\n\n>- Some notations are inconsistent in theoretical analysis.\n\n>- Q4. The clarity w.r.t. definitions could be improved, e.g., $K_{h',\\mu}(t)$ depends on the hypothesis $h$ while the justification (i.e., line 178) is provided after the definition (i.e., line 176). A thorough check for these issues could improve the readability.\n\n**Response.** Thanks for pointing out this. We will revise the definition of $K_{h',\\mu}(t)$, and thoroughly proofread the paper to identify any similar issues.\n\n>- Q5. Some notations seem to be inconsistent. For example, the notation $I^\\phi_{\\nu}(h,h')$ in line 132 is inconsistent with $I$ in line 129; the notations $\\mathbb{E}_{\\nu}$ in line 132 seems to be incorrect (probably should be expectation over $\\mu$?).\n\n**Response.** Thank you for catching that. You are correct—these are typos. Specifically, the $I$ in line 129 should be $I^\\phi_{\\nu}(h,h')$, and the expectation should be with respect to $\\mu$. We appreciate your careful reading.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wtx0Bpmx8l", "reviewer": "Reviewer_vWvG", "summary": "This study addresses the gap in the theory and algorithms of unsupervised domain adaptation based on f-divergence proposed by Acuna et al. 2021. Specifically, while the theory uses absolute values, the algorithms do not, and this issue is resolved by introducing a single scaling factor. The newly proposed f-DD generalization bound is derived based on Rademacher complexity, and tighter bounds are obtained using the localization technique.\n\nAs a specific domain adaptation algorithm, an adversarial type algorithm is proposed, yielding favorable results in benchmarks.", "questions": "Please provide the source of the technique that resolves the absolute value with a scaling parameter.\nAdditionally, clarify whether the claim in this paper—that there is no need to adjust the scaling parameter—holds generally.", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This study bridges the gap between theory and practice in existing UDA methods based on f-divergence, advancing the foundational research in domain adaptation. Furthermore, the derivation of sharper bounds using the recently introduced localization technique for DA is highly commendable as a contribution to the theoretical framework of DA.\nThe authors validate their theoretical contributions with empirical results, showing superior performance on popular benchmarks.", "weaknesses": "While the empirical validation is strong, it is limited to specific benchmarks. Broader validation across diverse datasets and tasks would strengthen the findings. It is nice to present some insight into what kind of dataset the proposed f-DD works well (and why), and also into what kind of dataset it does not work well (and why).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you sincerely for the valuable feedback on our paper. Our responses follow.\n\n\n>- While the empirical validation is strong, it is limited to specific benchmarks. Broader validation across diverse datasets and tasks would strengthen the findings. It is nice to present some insight into what kind of dataset the proposed f-DD works well (and why), and also into what kind of dataset it does not work well (and why).\n\n**Response.** Thank you for your suggestion. We have included additional experimental results on the VisDA-2017 dataset in the updated PDF (please see our general response).\n\nRegarding when $f$-DD performs well or poorly: we believe $f$-DD tends to work effectively when the marginal distributions $P_X$ differ, but the conditional distribution $P_{Y|X}$ is similar or close across the source and target domains (e.g., in cases of covariate shift). This is because the calculation of prediction disagreement between domains relies on the marginal distributions. Conversely, if there is a significant difference in the conditional distributions $P_{Y|X}$ between the two domains, $f$-DD's effectiveness may be limited. However, such issue may exist in most domain discrepancy guided UDA algorithms.\nIn such cases, utilizing additional knowledge about target domain labels, potentially through advanced pseudo-labeling techniques, might improve performance. We note that our theoretical framework assumes zero knowledge of target labels, so having such knowledge could require adjustments to our problem setup and refinement of our theory. \n\n\n>- Please provide the source of the technique that resolves the absolute value with a scaling parameter. \n\n**Response.** We believe that incorporating an absolute value function into the variational representation of $f$-divergence, as done by Acuna et al. (2021), is an unusual approach for upper bounding some objective quantities. This method changes the definition of $f$-divergence, as discussed at the beginning of our Section 4. In contrast, introducing a scaling parameter into the variational formula to achieve a better upper bound is a more conventional approach with a well-established history in generalization theory, such as PAC-Bayesian generalization bounds. For a detailed exploration of the importance of scaling parameters in upper-bounding techniques using $f$-divergence, we refer to [R1].\n\n\n[R1] Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral probability metrics. In ICML 2020.\n\n\n\n>- Additionally, clarify whether the claim in this paper—that there is no need to adjust the scaling parameter—holds generally.\n\n**Response.**  We note that we do theoretically justifies this in our Proposition 1 (simply because the surrogate loss $\\hat{\\ell}$ and $t$ are both unbounded). Specifically, in our Proposition 1, when the hypothesis space $\\mathcal{H}$ is sufficiently large and the outputs of $\\hat{\\ell}\\circ h'$ and $t\\ell\\circ h'$ span the entire space of $\\mathbb{R}$ (as the crudest choice), then one expects that any optimal $\\hat{\\ell}\\circ h'$ that maximizes $\\tilde{d}$ can be matched by a corresponding $t{\\ell}\\circ h'$ to achieve the same value of $d$, and vice versa. This suggests that optimizing over $t{\\ell}\\circ h'$ is equivalent to optimizing over $\\hat{\\ell}\\circ h'$. In practice, we do not know if $\\mathcal{H}$ is large enough so that this argument holds, hence, we also empirically validate that optimization over $t$ is unnecessary in practice. See the Table 10 on Page 9 for details about optimizing over $t$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their constructive comments and valuable feedback. In addition to addressing your individual comments separately, we have also uploaded a PDF file that contains a figure and a table. Specifically:

1. Figure: In response to Reviewer HkGQ's comment that the paper lacks a figure to illustrate the proposed method and that the implementation of the objective function is not straightforward, we have added an overview figure, which introduces the adversarial training framework of our $f$-DD-based UDA algorithm.


2. Table: Reflecting suggestions for additional experimental results: Reviewer CqSQ recommended conducting experiments on a larger dataset and with a more complex network structure, Reviewer EpWo suggested combining our $f$-DD with SOTA UDA methods and Reviewer vWvG also suggested testing $f$-DD on additional datasets and tasks.

To accommodate these suggestions, we have included additional results on the VisDA-2017 dataset in the table, which is more challenging than those used in our original paper. In particular, we replaced ResNet-50 with the Swin Transformer as the backbone, and we also attempted to apply domain mixup, a crucial component in many existing SOTA methods, into our $f$-DD algorithm. Specifically, we mixed source and target domains at the pixel level to create a third mixed domain, then jointly minimized the $f$-DD between the source and mixed domain and the $f$-DD between the target and mixed domain. This method improved performance, bringing it close to the SOTA method, e.g., PMTrans.

Due to time constraints during the rebuttal period, we trained $f$-DD for only 20 epochs, which is relatively few compared to SOTA methods. We expect that with additional effort in tuning hyperparameters, the performance of $f$-DD (with and without MixUp) can be further boosted. 

We will include the figure and the new experimental results in the revision.

Meta-review: This paper proposes to improve the existing framework for unsupervised domain adaptation based on the notion of f-divergence proposed by Acuna et al.(2021) by defining new f-domain discrepancy leading to novel and improved bounds on the target error. Using a localization technique, some fast rate generalization bound is also provided. 

During rebuttal authors have answered to numerous issues raised by the reviewers, including adding some clarification on the contribution and its scope, adding a figure to summarize the principle of the approach and discussing the improvement of the empirical results for comparing SOTA results with a new experiment on the VisDA dataset.  
Overall, reviewers have appreciated the contribution notably the fact to offer tighter generalization bounds with a new change of measure inequality via f-divergence, a novel localization approach with original proof, and the potentiality to reach SOTA results while providing a strong theoretical framework.  

I propose then to accept the paper.  
I strongly recommend that authors take into account the feedback of the reviewers in the final version, in particular for the correction of the typos, and the integration of the different explanations and clarifications provided in the rebuttal (including the proposed figure) as well as the additional experiments.
Decision: Accept (poster)