Gradual Domain Adaptation via Gradient Flow
Authors: Zhan Zhuang
Yu Zhang
Ying Wei
Venue: ICLR 2024 spotlight (2024)
Keywords: Domain adaptation
gradual domain adaptation
gradient flow
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A domain adaptation method that dynamically generates intermediate domains and gradually updates the classifier.

Abstract: Domain shift degrades classification models on new data distributions. Conventional unsupervised domain adaptation (UDA) aims to learn features that bridge labeled source and unlabeled target domains. In contrast to feature learning, gradual domain adaptation (GDA) leverages extra continuous intermediate domains with pseudo-labels to boost the source classifier. However, real intermediate domains are sometimes unavailable or ineffective. In this paper, we propose $\textbf{G}$radual Domain Adaptation via $\textbf{G}$radient $\textbf{F}$low (GGF) to generate intermediate domains with preserving labels, thereby enabling us a fine-tuning method for GDA. We employ the Wasserstein gradient flow in Kullback–Leibler divergence to transport samples from the source to the target domain. To simulate the dynamics, we utilize the Langevin algorithm. Since the Langevin algorithm disregards label information and introduces diffusion noise, we introduce classifier-based and sample-based potentials to avoid label switching and dramatic deviations in the sampling process. For the proposed GGF model, we analyze its generalization bound. Experiments on several benchmark datasets demonstrate the superiority of the proposed GGF method compared to state-of-the-art baselines.

Published as a conference paper at ICLR 2024
GRADUAL DOMAIN ADAPTATION VIA GRADIENT FLOW
Zhan Zhuang Yu Zhang" :1, Ying Wei
1 Soouthern University of Science and Technology, Ciity University of Hong Kong NNayyang Technological University 1225006 60000063mma....mm sustech. mersstechedducrr.. cn, yu. zhang. uut. com, y ying. we i @ntu.tuuue.. iennt.uduussgg
ABSTRACT
Domain shift degrades classification models on new data distributions. Conven- tional unsupervised domain adaptation (UDA) aims to learn features that bridge labeled source and unlabeled target domains. In contrast to feature learning, gradual domain adaptation (GDA) leverages extra continuous intermediate domains with pseudo-labels to boost the source classifier. However, real intermediate domains are sometimes unavailable or ineffective. In this paper, we propose Gradual Do- main Adaptation via Gradient Flow (GGF) to generate intermediate domains with preserving labels, thereby enabling fine-tuning method for GDA. We employ the Wasserstein gradient flow in Kullback-Leible divergence to transport samples from the source to the target domain. To simulate the dynamics, we utilize the Langevin algorithm. Since the Langevin algorithm disregards label information and introduces diffusion noise, we introduce classifier-based and sample-based potentials to avoid label switching and dramatic deviations in the sampling process. For the proposed GGF model, we analyze its generalization bound. Experiments on several benchmark datasets demonstrate the superiority of the proposed GGF method compared to state-of-the-as baselines.
INTRODUCTION
Unsupervised Domain Adaptation (UDA) stands fundamental and classical problem in machine learning (Pan & Yang, 2010). Its primary objective revolves around the transfer of the knowledge from well-trained source domain related yet unlabeled target domain, thereby reducing the need for time-consuming manual labeling and data preprocessing.
Early discrepancy- UDA methods (Borgwardt al., 2006; Pan et al., 2010; Tzeng al., 2014; Courty et al., 2014; Long et al., 2015; Ganin & Lempitsky, 2015; Sun & Saenko, 2016; Ganin et al., 2016: Courty et al., 2017: Tzeng et al., 2017) have shown promise in learning domain-invariant feature representations. However, recent studies (Zhao et al., 2019; Chen et al., 2019; Tang & Jia, 2020; Yang al., 2020) reveal that simply aligning source and target domains likely reduces discriminability. To address this issue, transport-bb based (Kirchmeyer et al., 2022; Gao et al., 2023; Xiao et al., 2023) and synthetic sample-based (Cui et al., 2020; Xu et al., 2020; Wu et al., 2020; Choi et al., 2020; Dai et al., 2021; Na et al., 2021; Jing et al., 2022; Na et al., 2022) approaches have been developed. Apart from them, Kumar et al. (2020) proposed a learning paradigm called Gradual Domain Adaptation (GDA), which resorts to an extra sequence of continuous unlabeled samples as intermediate domains to adapt the source classifier to the target domain via self self-training instead of feature alignment.
However, in many real-world scenarios where only unlabeled data from the target domain are available, constructing appropriate intermediate domains remains an open question ( (Wang et al., 2022), which transforms the UDA setting into the GDA setting. Prior studies have demonstrated that generating additional intermediate domains through generative models or interpolation algorithms offers viable approaches and improves the learning performance of the target domain. For instance, Sagawa & Hino (2022) utilized Continuous Normalizing Flow (CNF) (Chen et al., 2018) to interpolate between the source and target domains. GIFT (Abnar et al., 2021) and GOAT (He et al., 2023) use optimal transport and linear interpolation within mini-batch to form geodesic paths. Those methods share
Corresponding authors.

---

Published as a conference paper at ICLR 2024
Labeled Source:
1900s
Energy E (1:)
Unlobeled Torget: 1970s
clessferddsee energy
sampe-dpee
Encoder
(a)
eurry
mengy
((0g 12 Mill
(0% (o%6 "
(Cey (1
high energy
20 = b4 E ((e)
low energy
(b)
yo Yo yo Yn
Yo
π
Figure Illustration of the proposed GGF method. (a) The three designed energies collectively define gradient flow in latent space. (b) The training strategy of GGF involves gradual fine-tuning
of the classifier by utilizing the intermediate domains constructed in the GGF method.
similar spirit to transport-based and synthetic sample-based UDA methods. Unfortunately, they suffer from two limitations. Firstly, one-step interpolation and CNFs with minimal constraints can result in ambiguous labels for synthetic samples. Secondly, self self-training with multiple pseudo-labeling iterations may lead to training instability (Chen al., 2022).
To address the limitations above, we propose Gradual domain adaptation via Gradient Flow (GGF), a novel method construct intermediate domains when intermediate domains absent. Gradient flow refers to the continuous process gradient descent, which follows the steepest descent direction to minimize a given energy function. As shown Figure 1, in the proposed GGF method, the designed gradient flow transports feature representations from the source domain to the target domain along curve to minimize the following three designed energies: (1) distribution based energy that is to shift the features from the source domain to the target domain. (2) classifier-bat based energy that is to preserve label information, and (3) sample-bused energy that is avoid large noise generated samples. To the best of our knowledge, we are the first to propose the construction of intermediate domains, while meantime allowing the source classifier to be gradually fine-tuned to align with the target distribution. We provide theoretical analysis on the generalization bound of the proposed GGF method, showing that the gradient flow with smaller continuous discriminability and transport loss contributes lower target error. We conduct comprehensive experiments to evaluate the proposed GGF on various domain adaptation scenarios, where GGF outperforms state-of of-the-art methods.
2
PRELIMINARIES AND BACKGROUND
2.1 SETTING AND NOTATION
In the context of GDA, consider the data of the labeled source domain, T unlabeled intermediate domains, and the unlabeled target domain to be sampled from the distribution 14o {pr) and pr+1 over X', respectively. We denote the target distribution by æ. Consider the hypothesis class H. where for any classifier h E H. h X' y maps inputs to predictions. We assume that there exists labeling function in each domain: fu.. = ft E H. Given a loss function É(, .). the generalization error defined (h) (h, fn) (Ez,,,(h((x) f,))) The source classifier ho can be learned with minimal error éja (ho) using supervised learning, and the objective of GDA is to evolve this classifier ho hr over the intermediate domains so as minimize the target error e (hr)). The UDA problem shares the objective and can be converted into a GDA problem by generating intermediate domains.
Self-training Self-training (ST) (French et al., 2017; Zou et al., 2019; Prabhu al., 2021; Liu et al., 2021) (a.ka. pseudo-labeling) is a domain adaptation method that does not rely feature learning. We use the Wasserstein-2 distance W2(n, v) (see Appendix. A. to quantify the domain shift. Under the assumption that the shift between adjacent domains is small, previous studies (Kumar et al., 2020; Wang et al., 2022) have shown that ST can effectively update the classifier h to adapt to the next domain with samples S,, expressed mathematically below:
1
he ST((h-11, arg min
Σ xeS.
e(h(x), (x)1(x))),
(I)
hER |St
2

---

Published as a conference paper at ICLR 2024
where (x) denotes pseudo labels of x by hy-1- Kumar et al. (2020) proposed Gradual Self- Training (GST) as the baseline algorithm for GDA, which updates the classifier using ST with pscudo-labels on the intermediate domains in sequence. In this paper, we generate intermediate domains gradually while preserving labels, enabling us to update the classifier in a distinct manner by fine-tuning it directly on the transformed samples.
2.2
WASSERSTEIN GRADIENT FL.w (WGF)
A flow describes denendennddnt diffeomorphic map of particles in system. Let (x( [0, 1] x I" Rn denote a flow, so that a vector field of position and time ux(x) defines flow via an ordinary differential equation (ODE), i.e.,
d (x) = (x)), Px x.
(2)
dt
Let P2 (IB") denote the space of probability measures on IR" with finite second moments, For a metric space (P2",, W2), Otto (2001) first proposed WGF to solve the porous medium equation. The vector field of WGF is defined as u, = - ww,& (w,&(m)) where a general energy 8 {(4)) consisting of the following three terms associates with distribution μr or group of particles (Santambrogio, 2017),
E(pe)= H(po(x))dxx / Vxx)ddx(x)
// W(X (xdd(x))))()))
(3)
The three terms represent the internal, (external) potential, and interaction energy, respectively. Simply put, internal energy (c.g. entropy H(u)= = f log μ)) is related to distribution density, potential energy is related to the potential field V in Euclidean space, and interaction energy captures the interactions between particles. Correspondingly, the vector field can be calculated as
Up W.,
-E (Pt) V (11' (M)) VV - (VW) * Mas
(4)
where * denotes the convolution operator. We can conceptualize the Dullback-Leibler (KL) diver- gence, maximum mean discrepancy (MMD) (Arbel et al., 2019), or other related metrics (Mroueh & Rigotti, 2020; Korba et al., 2021; Glaser et al., 2021) as the energy functional. Leveraging the gradient flow's descent property, we can establish progressive flow that reduces the difference between distributions over time. This approach proves valuable for dataset transformation and data augmentation (Alvarez-Melis & Fusi, 2021; Hua et al., 2023). In the next section, we will delve into our proposed method in the context of domain adaptation.
3 GRADUAL DOMAIN ADAPTATION VIA GRADIENT FLOW (GGF)
In this section, we present the novel GGF method, designed to construct intermediate domains along the gradient flow of three energy functions and gradually adapt a source classifier to the target domain. We commence by providing comprehensive exposition of the three energies and their corresponding sampling techniques. Subsequently, we unify these components into a cohesive framework. To allow the versatility and extensive applicability of our approach in both data and latent spaces, utilize the symbol "x" to represent both samples and features in the subsequent sections.
3.1 DISTRIBUTION BASSDD ENERGY FOR SHIFTING FEATURES FROM SOURCE TO TARGET
Considering the high computation cost of MMD (Arbel et al., 2019) and KSD (Mroueh & Rigotti, 2020), we opt (L( (T) as the distribution- energy with the resulting vector field given as,
= V w. (L( = Vw. log
dup - log + V log π.
(5)
This vector field consisting of the above two log-density components drives the distribution μe towards the target distribution π under the influence of both diffusion and drift (Jordan et al., 1998). We simulate the associated flow following the common practice of Langevin Monte Carlo (LMC) (Roberts & Tweedie, 1996) as a discretied sampling method. The LMC guarantees convergence in the Wasserstein distance, as described in Eq. (32) (Dalalyan & Karagulyan, 2019). Concretely, given set of samples To from the source distribution po, the LMC iteratively updates the samples as
X X, ?hh Vx, log क (x.) + 22m
3

---

Published as a conference paper at ICLR 2024
where Ç. and 7/1 represent the distribution of the target domain, noise sampled from standard Gaussian distribution, and the step size, respectively. The log-density Vx log (xx), also known as the (Stein) score function sx (x), can be estimated via various methods (Hyvärinen & Dayan, 2005; Vincent, 2011; Song et al., 2020), even when the probability density function is not available. Here, we resort to Denoise Score Matching (DSM) (Vincent, 201 1) with neural network s(x; Φ) model the score function. The core idea of DSM is to minimize the discrepancy between the score and the model's predicted score after injecting noise clean x, whose objective follows,
JDSM Egg (xx.x)
2 [|s(xx Φ) V2 log
(xxx)] (x
Simple augmentation methods such as Gaussian noise e ~ 8(0,0²1 can be employed to compute the latter term (i.e., Va log go (x|x) = -, where a denotes the noise variance. By using efficiently estimate the scores of perturbed data X in those low-
qo (x, x) qa (x))))(x), density areas of the target distribution. Once the score network has been trained, we can readily construct intermediate domains following the sampling process in Eq. (6).
Challenges There still remain two main challenges with the sampled intermediate domains. Firstly, the sampling disregards the label information of the samples, which likely introduces either condition shift Puo (y) x) + P. (y/ x) prior shift P (v) j (x((v) between the source and target domains and thereby puts the decision boundary in danger of collapse. Secondly, The noise (a.k.a. diffusion) term introduced by LMC may cause some samples to deviate drastically from the true data distribution, given that the score network cannot accurately estimate the shifted points. To address both issues, we propose the following classifier-bast -basieer--aedddde and sample-based energy.
3.2 CLASSIFIER-BASTE ENERGY FOR PRESERVING LABEL INFORMATION
We use the cross-entropy loss and the entropy of the logits as the classifier-basse potential energy:
Coelu, h, = - u(x) log h(x)du(xx) Cu(u, h)) - -- l((x) log (x))dx(x),
(8)
where y(x) denotes the corresponding label of input x, and h(x) denotes its prediction. For the cross-entropy loss, lower energy ensures that the predictions of the shifted samples remain consistent with the original labels. Meanwhile, lower entropy of logits yields higher prediction confidence, avoiding excessively smooth predictions. By introducing À we balance between the two potentials to accommodate different distribution characteristics of datasets. The flow that implements this classifier- based energy is exactly the gradient with respect to inputs via backpropagation. i.c.
X = x, - (2 - X) (xE(X,, h4, Vt) + XV Lu(x,, h4)).
(9)
Observation Our experiments show that the LMC monotonically reduces the Wasserstein distance between the constructed intermediate domains the target domain. Including the classifier-based energy, however, results in an initial decrease in the distance to minimum but followed by an increase, indicating balance between feature shift and label preserving. We resolve this issue in Section 3.4 by setting a proper number of intermediate domains using this stationary point.
3.3 SAMPLE-BASED ENERGY FOR REDUCING NOISE AND RECTIFYING FLOW
To mitigate the noise of generated samples, we borrow the idea from stochastic interpolant (Liu al., 2023; Liu, 2022; Albergo & Vanden-Eijnden, 2023; Lipman al., 2023; Albergo al., 2023) where flows connecting two distributions do not deviate too much. Those methods use interpolation techniques create interpolants x,, based on which they directly estimate the vector field of the samples through a neural network uy different from WGF that derives a flow from a prior energy. Specifically, we adopt the rectified flow approach (Liu al., 2023: Liu, 2022) to generate interpolants the following flow matching objective:
at time r E [o, 1] as x, = (1 - 7 7)x0 + TX1, and subsequently estimates the vector field vy (x,) with
JFM = E Exoopo XINTT [/'' I/vo (x,) - v(x x,, )1
(10)
4

---

Published as a conference paper at ICLR 2024
where the conditional vector field v(x x,) = X; =1 Xo- The estimated vector field (x+) defines a flow, which can be discretized using the Euler method as X1+1 = X, + T780 (x²). Even without explicit prior energy like WGF, this flow guarantees high energy for source samples and low energy for target samples for each interpolation pair.
3.4 COMPLETE ALGORITHM
We summarize the three energies above into a unified Wasserstein gradient flow. To balance the three terms, we use step sizes 1,, 12, f/,, and halt the iterated sampling process once the minimum Wasserstein distance to the target domain is reached. We construct each intermediate domain with a iterations, resulting in a total of a'' iterations across T intermediate domains. Each iteration follows
X++1
111 (x(, 2) + 2 1/2 Vx, (x+; h4s Ve Vo (xt)
(11)
Distribution based
Classifier-based
Sample-based
comprehensive algorithm and complexity analysis are described in Appendix C.
THEORETICAL ANALYSIS
This section provides a theoretical analysis of the proposed GGF method. The primary distinction between our analysis and prior theoretical works (Kumar et al., 2020; Wang et al., 2022) is that we do not use any given intermediate domains. Instead, we design Wasserstein gradient flows to generate the intermediate domains, which incrementally shift the source distribution while preserving labels.
We set the labeling function fr of the last intermediate and target domains be the same, which is reasonable considering the synthetic nature of the intermediate domain. Our analysis, with detailed proof in Appendix B, has two main steps: first, we offer an upper-bound of the target error in Lemma 1, using the risk on the last intermediate domain Epr (hr) and the final Wasserstein distance W2 (NT, T). Then, we derive the two terms and provide a generalization bound in Theorem 1, To facilitate our analysis, adopt the assumptions consistently with prior works (Dalalyan & Karagulyan, 2019; Kumar et al., 2020; Wang et al., 2022):
Assumption Each predictor function h H and labeling function f E H is R-Lipschitz in l norm, i.e. Vx, x' E x [h(x)= - h((²)l S R|i - x'|l
Assumption The loss function e p-Lipschitz, i.e., Vy.y' E y: 1. e (g/, (116,.)1- 11y y and [e(-,v) - [ ((,83)) < y
Assumption The Rademacher complexity (Bartlett & Mendelson, 2002) of hypothesis class H denoted by R2 (H) is bounded by B vi i.e., RR(H)) < v
B
Assumption The potential V m-strongly convex, and M-Lipschitz smooth, iee., Vx, x' E X:
1. V(x) S (x)) (VV()), x' - 1) - 3 1| x11
(m--strongly convex)
2. V(x) V(x') + (v) 2) + || - 2'12 (M-Lipschitz smooth)
Assumption 5 Consider the vector field u, is bounded, i.e, ||u.|| < U
With Assumptions and as well as the shared labeling function fr, we present Lemma Despite the seemingly similarity with Lemma in (Wang et al., 2022), note that this paper considers the Wasserstein distance over X.
Lemma For any classifier h H. the generalization error the target domain is bounded by the error on the last generated intermediate domain and the Wasserstein distance:
ex (h) S cpr (h) + 2pRWi (JuT,T) T)
(12)
Our method gradually refines the source classifier ho from the source (4o to the last intermediate domain fr-11 enabling iteration analysis. Therefore. we introduce Proposition 1, which states the bounded variance of generalization error after updating classifier domain.
5

---

Published as a conference paper at ICLR 2024
(The Stability of GGF) Consider (21 Y4) (a)==1 are i..d. samples from domain with distribution μ.. and h2 is classifier. GGF provides a map T2 that transports 2; to the next domain and updates the classifier h, by empirical risk minimization (ERM) the shifted samples as he = arg min -i=1 E(T (TT) 9:). Denote the labeling functions of the two domains are Ip and fw, then, for any 8 E (0, 1), with probability at least 1 - 6, the following bound holds true:
Proposition 1
pB + (log(1/0)
1 (hy) - cv (h+)) S p(Enr (f) - fu(x)) + RA,)+ +0
(13)
VTi
where A - Ei= |
(x:) means the average shift distance for I, on Tp-
In comparing the stability of the GST algorithm (Wang et al., 2022) with ours, we note that the primary difference lies in the first term. Specifically, the GST algorithm bounds the error difference by the Wasserstein distance Wp(,, 2) over the joint distribution between adjacent domains. For our method, the error bounded using two parts: an expected difference in labeling functions Ez (x) (x)),, bounded by the Continuous Discriminability defined in B.2 and 2) transport cost Aus bounded by the vector field limit.
Theorem (Generalization Bound for GGF) Under Assumptions 1-5, if hv is the classifier on the last intermediate domain updated by GGF, then for any ō (0, 1), with probability least 1 the generalization error of hor on target domain is upper bounded as:
naTU + +211 1pm)aT W2 (Ho, π) + 3.3M /TP
(<hrr)== S Epo (ho) + Euo (14g)) + pR
(14)
pB + log /oggo/11))))) 1/5) +
T
Vn
Based on the above results, we derive the generalization bound for GGF in Theorem 1. The first two terms Eua (ho) and tμо (ht) represent the source generalization errors of the source classifier and the target labeling function. Besides, the eμ (ht) term can also be replaced with lower term pK, representing the continuous discriminability above. The naTU term corresponds to the transport cost of samples across intermediate domains through aT iterations with step size of 171 and abbreviated as 1y. Meanwhile, W2 (Jo; π²) denotes the Wasserstein distance between the source and target over X.
Remark:
Prior theoretical works (Kumar et al., 2020; Wang et al., 2022; Dong et al., 2022) link the generalization bound to the Wasserstein distance over X' y across given intermediate domains. For the GST algorithm, disregarding the ( ((//vnT) term introduced in the online learning framework for improved sample complexity, the bound proposed in (Wang et al., 2022) simplifies to
Ex (hT) S END (ho) + pV R +I Wp(m--1, +
(
(pB /og(1//) /n
T) (15)
which suggests that the optimal intermediate domains should be along the Wasserstein geodesic path from the source to target domains, which is unfortunately non-trivial to pinpoint on unlabeled domains. Instead, the novel upper bound we establish in Eq. (14) provides promising and practical error bound for analyzing GDA when intermediate domains are generated along the Wasserstein gradient flow. Moreover, optimization of the hyperparameters a, T, and y. By fixing two of them, the optimal value for the remaining is deterministic aligning with the lowest upper bound.
analysis offers insights into minimizing the upper bound through the
EXPERIMENTS
In Section 5.2, we evaluate the effectiveness of the proposed GGF on three GDA benchmarks and pre-trained features of the UDA task, Office-Home. The experimental results demonstrate that GGF constructs practical intermediate domains and gradually updates the initial classifier more accurately. In Section 5.3, conduct the ablation studies for the three kinds of energy, pseudo-labeling multiple iterations, and the number of intermediate domains.

---

Published as a conference paper at ICLR 2024
Portrailts
Category: Pen, Accuracy: 36% 53%
Source Durrenn Ant
Accursty
get Domain Cápart
0.80
Adapted Source Domain
0.75
GOAT
Wesderzinn Destncee
4.00
2.00
20AT
0.00
10 12
Insormeddo edaatmediitteeooo Domain Index
Figure 3: Accuracy (%) and 2₂ distance to target domain over intermediate domains Portraits.
Figure 4: The adaptation example of the "pen" category samples from Art to Clipart domain.
To aid understanding, we construct two toy datasets: mixture of Gaussian blobs and two- moon (details in Appendix D.I). Figure 2 illus- trates how GGF transports source samples to the target and updates the classifier.
Source
Adapted Source Domain
Target
5.1 EXPERIMENTAL SETTINGS
Datasets We mainly evaluate our method on five datasets. Portraits is a gender (binary) clas- sification dataset with 37,921 facing portraits from 1905 2013. We follow the chronolog- ical split from Kumar et al. (2020), creating source domain (first 2000 images), intermedi- ate domains (14000 images not used here), and a target domain (next 2000 images). Rotated MNIST is variant of the MNIST dataset (LeCun, 1998) consisting of 4000 source images and 4000 target images rotated by 45 to 60% degrees, as per He et al. (2023) and Kumar et al. (2020). Office-Home (Venkateswara al., 2017) is well-known UDA dataset with 65 categories across four domains: Artistic (Ar), Clipart (CI), Product (Pr), and World (Rw). VisDA-2017 (Peng et al., 2017) is large-scale dataset including a simulation-to- UDA task with 152,397 synthetic training images and 72,372 real-world test images across 12 categories.
Two toy examples. The dashed lines in the first and other columns correspond the initial and updated classifiers, respectively.
Figure 2:
Implementation Following the setting in (Sagawa & Hino, 2022), we use semi- supervised UMAP (MeInnes al., 2018) as the feature extractor to reduce the dimen- sionality of input data while preserving the class discriminability. For the UDA datasets, we use the extracted features as input for ex- periments. Further implementation details are deferred to Appendix E.
Table 1: Accuracy (%) on GDA tasks.
Portraits
MNIST 45"
MNIST 60"
Source Self Train
76.50
57.06
37.47
76.99
60,22
39.81
GST (4) (Kumar et al., 2020)
81.45
66.45
56.87
GOAT (He et al., 2023)
83.17
61.45
46.29
CNF (Sagawa & Hino, 2022)
84.57
62.55
42.18
GGF (Ours)
86.16
67.72
54.21
5.2 MAIN RESULTS
Results in Table reveal that all three methods can generate efficient intermediate domains for GDA. and the proposed GGF outperforms the others, especially for large domain shifts. Notably, our method is competitive with the baseline GST algorithm with about four real intermediate domains. In Figure 3, observe the accuracy evolution with the number of intermediate domains on Portraits for each method. GOAT creates samples along the Wasserstein geodesic but has smaller improvements in datasets with small shifts. CNF's fluctuating accuracy indicates some ineffective domains, while our gradient flow-basd based method consistently reduces domain differences and improves accuracy.
We evaluate GGF on Office- Home tasks and notice that it further improves the accuracy by up to 0.5%, as shown in Table 2. Additionally, we provide visual evidence source sample adaptation in Figure demonstrating that GGF proficiently transfers source features to the target domain, achieving
7

---

Published as a conference paper at ICLR 2024
Table 2: Classification accuracy (%) on Office- dataset with ResNet-50. The best accuracy is indicated in bold, and the second best is underlined.
Mithod
Ar-+CI
Ar-PP
Ar-Ru
Cl-AAr
Cl-Pt
CI-Rw
Pr-sAr
Pr-+Cl
Pr-Rt
Ru-+Ar
Ro-CI
Re-+Py
Ang.
DANN (Clanin Lengisky, 2015)
45.6
99.3
70.1
47.0
58.5
60.9
46.1
43.7
68.5
63.2
51.5
76.8
57.6
(Xie sl, 2018)
49.5
70.3
76.3
60.4
68.5
69.6
61.4
48.9
15.7
70.9
15.0
X1.1
65.7
GVB GD (Cui st 2020y
57.0
14,7
79.8
64.6
74.1
74,6
65.2
53.1
$1.0
74,5
19.7
$4.3
70.4
RSDA (Cu 2020)
53.2
77.3
81.3
66.4
74.01
76.5
67.9
53.0
$2.9
15.8
57.5
85.4
70,9
LAMDA (Lo 2012)
57.2
78.4
$2.6
66.1
80.2
81.2
05.6
55.1
$2.5
71.6
59.2
83.9
72.0
SENTRY (Prallu at. 2012)
61.
77.4
S0.1
66.3
71.
74.7
66.8
63,0
$0.9
74.0
66.3
84.1
72.2
FaBi (Na al. 2023)
58.1
77.3
$0.4
67.7
79.5
78.1
65.8
57.9
$1.7
76.4
62.9
M6.7
72.7
(Liu (Li
2021)
19.0
79.6
83.4
68.4
77.1
76.7
68.9
56.4
$3.0
15.3
62.2
85.1
73.0
CoVi (Na et 2002)
38.5
78.1
80.0
68.1
BOU
77.0
66.4
60.2
82.3
76,6
63.6
85.5
23.1
KSDA + OGF
40.1
77.9
82.3
68.4
78.3
77.2
67.8
80.5
$2.5
75.8
61.0
85.2
23.1
Covi + GGF
59.3
29.0
80.4
69.3
80.1
78.1
66.8
61.2
83.1
76.2
62.8
85.5
73.6
superior feature alignment. In Appendix D, we provide GGF's performance on large-scale datasets (D.2), along with comparative analysis of different feature extractors (D.3) and visualizations (D.5).
5.3 ABLATION STUDY
Different Energy Functions
Table 3: Ablation study on different energies.
We conduct an ablation study on the three types of energy in GGF, as outlined in Table 3. Our findings highlight the im- portance of both the classifier- based and sample-based ener-
Distribution-hssedd
Claseffer-bbsed
Sample-hased
Accuracy
MNIST
(+0.99)
84.02 (e) 42)
66.88 (+0.36)
53.15 (50.75)
84.60 (=0 21)
67.45 (40.60)
(10.33)
86.16 (20.19)
67.72 (20.34)
$4.21 (20.86)
gies in achieving optimal performance with GGF. Notably, the classifier-based energy, functioning as a regularization term, significantly enhances the performance the MNIST 60' task by preserving category information. This is crucial given the substantial domain shift between the source and target domains, as well as the unsatisfactory predictions of the initial classifier in the target domain.
Sensitivity Analysis on Step Sizes 7) We perform sensitivity analysis by rescaling each step size while keeping the other two fixed on Portraits datasets. The results, shown in Table 4, demonstrate the numeri- cal proximity and robustness of our hyperpa- rameters. Notably, we observe significant decrease in performance only when reduc-
Table 4: default setting of rescale ratio is 100%.
Sensitivity analysis on step sizes
The
25%
50%
75%
100%
150%
200%
400%
79.00
79.15
86.45
86.35
84.85
84.75
84.10
0.08
84.75
85.80
86.15
86.35
74.45
78.
68.00
da
86.05
86.50
86.35
86.40
86.15
ing 77, or increasing 1/2, indicating that the classifier-based energy dominates the distribution- based energy. This suggests that the former induces larger velocity components, pushing samples away from the decision boundary. Fortunately, our hyperparameter optimization method avoids converging solutions where the classifier-bare based energy dominates, these solutions do not lead a reduction in the Wasserstein distance from samples to the target domain.
Comparison of Self-Training (ST) and Fine-Tuning (FT) Figure compares the two updating methods of GGF: ST with pseudo-labels and FT with preserving labels. In the Portraits task with smaller o, ST with a smaller confidence threshold outperforms FT, but as a increases, FT is stable and better. In the MNIST 60" task, ST performs poorly with small due cumulative instability. These results demonstrate that ST is less effective with larger domain gaps and more pscudo- processes, highlighting the advantages our preserving labels approach. Additionally, sufficiently large represents the one-step adaptation. As shown in the figures, gradually updating the classifier as in the GDA setting works better than updating the classifier only in the last intermediate domain.
RELATED WORK
Unsupervised Domain Adaptation (UDA) UDA aims to facilitate knowledge transfer from a source domain to target with only unlabeled instances. Early UDA methods, divided into the two major categories of discrepancy- and adversarial-bawe center around learning domain-invariam representations. The discrepancy being minimized in discrepancy- based ways includes the Maximum
8

---

Published as a conference paper at ICLR 2024
Portraits
MNIST
0.55
Presernngi is
0.86
STatheo01
0.53
S2
ST WT
Accuracy
0.85
0.50
0.47
0.84
0.45
0.83
Presaniing
wit cm0.3
0.42
wan cH0.2
win CHO.4
0.40
2
8
16
32
64
128
8
16
32
64
128 Alpha
256
512
1024
Alpha
Figure 5: Accuracy comparison of two updating methods with varying hyperparameter a on Portraits and MNIST 60" tasks, using different confidence thresholds e for self-training. With fixed sampling iterations a?,, the number of intermediate domains T decreases as increases.
Mean Discrepancy (MMD) (Borgwardt et al., 2006; Long et al., 2015), second-order statistics (Sun & Saenko, 2016), and optimal transport distances (Courty et al., 2014; 2017; Shen et al., 2018; Li et al., 2020; Kerdoncuff et al., 2021). Instead, adversarial- based methods pursue domain-invariin that are indistinguishable by domain discriminator (Ganin & Lempitsky, 2015; Tzeng et al., 2017; Long al., 2018; Liu et al., 2019). Recent research (Chen al., 2019; Yang al., 2020; Tang & Jia, 2020) suggests a negative impact on discriminability when dealing with two wildly dissimilar domains, which motivates the transport- and synthetic sample-basek methods.
features
Transport-based methods preserve domain-specific features by training feature extractor without transfer loss and then transferring these features from the target the source domain using optimal transport or sampling techniques. For instance, Kirchmeyer et al. (2022) and Fan & Alvarez-Meih (2023) trained an optimal transport map between conditional distributions in feature space and performs reweighting for label matching and robust pre-training. Recently, Gao et al. (2023) and Xiao et al. (2023) leveraged diffusion and energy-based models to shift target data for test-time input adaptation and preserve the image structure and latent variables related to class information. Syntheric sample-bon based methods create intermediate samples at the input or feature levels by employing techniques such as geodesic flow kernels (Gong al., 2012), adaptors (Choi et al., 2020), generative models (Hoffman et al., 2018; Gong et al., 2019; Hsu et al., 2020; Cui et al., 2020) data augmentation (Xu et al., 2020; Wu al., 2020; Fatras et al., 2022; Dai al., 2021; Na et al., 2021; Jing et al., 2022; Na et al., 2022; Sahoo et al., 2023). The generated samples are utilized for improved feature learning, consistency regularization, robust discriminator training.
Gradual Domain Adaptation (GDA) Unlike UDA methods prioritizing feature learning, GDA updates the model using unlabeled sequential intermediate domains. allowing more lne-grained adaptation. Kumar et al. (2020) first introduced the setting with a straightforward gradual self-training (GST) algorithm. Meanwhile, Wang et al. (2020) proposed an adversarial-baset method for similar setting with continuous indexed domains. Based on the assumption of gradually shifting distributions, prior works (Kumar et al., 2020; Dong et al., 2022; Wang et al., 2022) provide and improve the upper generalization bound. Researchers have explored various algorithms for challenging scenarios to obtain the ideal intermediate domains. For instance, when the sequence of extra unlabeled data is unavailable, Chen & Chao (2021) proposed an Intermediate Domain Labeler (IDOL) module to index them. In cases where intermediate domains insufficient, prior studies have shown that generating additional intermediate domains through continuous normalizing flow (Sagawa & Hino, 2022) or optimal transport and linear interpolation (Abnar et al., 2021: He et al., 2023) can reduce the distance between adjacent domains and improve the performance of GST.
CONCLUSION
We introduce gradual domain adaptation via gradient flow (GGF), novel approach to create con- tinuous intermediate domains and incrementally finetune the classifier. Theoretically, We offer an upper- bound analysis of target error. Empirically, demonstrate that GGF outperforms prior synthetic-based GDA methods and enhances performance compared to sfate-of-the-arn UDA baselines. Our results underscore GGF's ability to generalize effectively across various pre-trained features.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is supported by NSFC key grant under grant no. 62136005, NSFC general grant under grant no. 62076118, Hong Kong RMGS 9229111, and Shenzhen fundamental research program JCY1200100324105000003
REFERENCES
Samira Abnar, Rianne den Berg. Golnaz Ghiasi, Mostafa Dehghani, Nal Kalchbrenner, and Hanie Sedghi. Gradual domain adaptation in the wild: When intermediate distributions are absent. arXiv preprint arXiv:2 066.06080, 2021.
Michael Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. 2023.
Michael Albergo, Nicholas M Boffi, and Eric Vanden Vanden-Eijnden. Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint arXiv:2333.08879, 2023.
David Alvarez-Melis and Nicold Fusi. Dataset dynamics via gradient flows in probability space. In International Conference on Machine Learning. PMLR, 2021.
Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton. Maximum mean discrepancy gradient flow. Advances in Neural Information Processing Systems, 2019.
Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, 2006.
Karsten M Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22( 22(14):449-e57 2006.
Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning. In Advances in Neural Information Processing Systems, 2022.
Hong-You You Chen and Wei Chao. Gradual domain adaptation without indexed intermediate domains. In Advances in Neural Information Processing Systems, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, 2018.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability discriminability: Batch spectral penalization for adversarial domain adaptation. In International Conference on Machine Learning. PMLR, 2019.
Xiang Cheng and Peter Bartlett. Convergence of langevin meme in kl-divergence. Algorithmic Learning Theory. PMLR, 2018.
Jongwon Choi, Youngjoon Choi, Jihoon Kim, Jinyeop Chang, Ilhwan Kwon, Youngjune Gwon, and Seungjai Min. Visual domain adaptation by consensus- based transfer to intermediate domain, In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Nicolas Courty, Rémi Flamary, and Devis Tuia. Domain adaptation with regularized optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part 14. Springer, 2014.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, 2017.
10

---

Published as a conference paper at ICLR 2024
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adversarial domain adaptation. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattemn Recognition, 2020.
Yongxing Dai, Jun Liu, Yifan Sun, Zekun Tong, Chi Zhang, and Ling- Yu Duan. Idm: An intermediate domain module for domain adaptive person re-id. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
Arnak Dalalyan. Further and stronger analogy between sampling and optimization: Langevin monte carlo and gradient descent. In Conference on Learning Theory. PMLR, 2017.
Arnak Dalalyan and Avetik Karagulyan. User-friendly guarantees for the langevin monte carlo with inaccurate gradient. Stochastic Processes and their Applications, 129(12):5278-51) 2019.
Jing Dong. Shiji Zhou, Baoxiang Wang, and Han Zhao. Algorithms and theory for supervised gradual domain adaptation. Transactions on Machine Learning Research, 2022.
Alain Durmus and Eric Moulines. High-dimensional bayesian inference via the unadjusted langevin algorithm. 2019.
Alain Durmus, Szymon Majewski, and Blažej Miasojedow. Analysis of langevin monte carlo via convex optimization. Journal of Machine Learning Research, 20(1):2666-27) 21 2019.
Jiaojiao Fan and David Alvarez-Melis. Generating synthetic datasets by interpolating along gen- eralized geodesics. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, 2023.
Kilian Fatras, Hiroki Naganuma, and Ioannis Mitliagkas. Optimal transport meets noisy label robust loss and mixup regularization for domain adaptation. In Conference on Lifelong Learning Agents. PMLR, 2022.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurelic Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al. Pot: Python optimal transport. Journal of Machine Learning Research, 22(1)3351133555 2021.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. arXiv preprint arXiv: /7000700600202 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarre training of neural networks. Journal of Machine Learning Research, (7(1):2096 2030, 2016.
Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-driven adaptation test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Pierre Glaser, Michael Arbel, and Arthur Gretton. Kale flow: relaxed kl gradient flow for probabilities with disjoint support. In Advances in Neural Information Processing Systems, 2021.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Rui Gong, Wen Li, Yuhua Chen, and Lue Van Gool. Dlow: Domain flow for adaptation and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2019.
Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-labos loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Yifei He, Haoxiang Wang, Bo Li, and Han Zhao. Gradual domain adaptation: Theory and algorithms. arXiv preprint 2...:330...8.522 2023.
11

---

Published as a conference paper at ICLR 2024
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference Machine Learning. PMLR, 2018.
Han-Kat Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung- Yu Tseng, Maneesh Singh, and Ming-Hsuan Hsuan Yang. Progressive domain adaptation for object detection. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, 2020.
Xinru Hua, Truyen Nguyen, Tam Le, Jose Blanchet, and Viet Anh Nguyen. Dynamic flows on curved space generated by labeled data. In International Joint Conference on Artificial Intelligence, 2023.
Aapo Hyvärinen and Peter Dayan. Estimation of normalzzdl statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Mengmeng Jing. Lichao Meng, Jingjing Li, Lei Zhu, and Heng Tao Shen. Adversarial mixup ratio confusion for unsupervised domain adaptation. IEEE Transactions on Multimedia, 2022.
Richard Jordan, David Kinderlehrer. and Felix Otto. The variational formulation of the fokker-planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
Tanguy Kerdoncuff, Rémi Emonet, and Mare Sebban. Metric learning in optimal transport for domain adaptation. In Proceedings of the Twenty-Ninth International Conference International Joint Conferences on Artificial Intelligence, 2021.
Matthicu Kirchmeyer, Alain Rakotomamonjy, Emmanuel de Bezenac, and patrick gallinari. Mapping conditional distributions for domain adaptation under generalized target shift. In International Conference on Learning Representations, 2022.
Anna Korba, Pierre-Cyre discrepancy descent. In International Conference on Machine Learning. PMLR, 2021.
Frankosraankoski.. Szymon Majewski, and Pierre Ablin. Kernel stein
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning. PMLR, 2020.
Trung Le, Tuan Nguyen, Nhat Ho, Hung Bui, and Dinh Phung. Lamda: Label matching deep domain adaptation. In International Conference Machine Learning. PMLR, 2021.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
Michel Ledoux and Michel Talagrand. Probubility Banach Spuces: isoperimetry and processes, volume 23. Springer Science & Business Media, 1991.
Mengxue Li, Yi-Ming Zhai, You-Wei Luo, Peng-Fei Ge, and Chuan-Xian Ren. Enhanced transport distance for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2020.
Yaron Lipman, Ricky TQ Chen, Heli Ben Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. 2023.
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning. PMLR, 2019.
Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In Advances in Neural Information Processing Systems, 2021.
Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. 2023.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning. PMLR, 2015.
12

---

Published as a conference paper at ICLR 2024
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, 2018.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint wrXiv:1802.0342) 2018.
Youssef Mrouch and Mattia Rigotti. Unbalanced sobolev descent. In Advances in Neural Information Processing Systems, 2020.
Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Jaemin Na, Dongyoon Han, Hyung Jin Chang, and Wonjun Hwang. Contrastive vicinal space for unsupervised domain adaptation. In European Conference Computer Vision, 2022.
Felix Otto. The geometry of dissipative evolution equations: the porous medium equation. 2001.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-135) 2010.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE transactions on neural networks, 12(2):199-210, 2010.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv: 2017.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
Gareth O Roberts and Richard I. Tweedie. Exponential convergence of langevin distributions and their discrete approximations, Bernoulli, pp. 341-363, 1996.
Shogo Sagawa and Hideitsu Hino. Gradual domain adaptation via normalizing flows, arXiv preprint arXiv:2206. / 1492, 2022.
Aadarsh Sahoo, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Select, label, and mix: Learning discriminative invariant feature representations for partial domain adaptation. In Proceedingg of the IEEF/CVF Winter Conference on Applications of Computer Vision, 2023.
Adil Salim, Anna Korba, and Giulia Luise. The wasserstein proximal gradient algorithm. In Advances in Neural Information Processing Systems, 2020.
Filippo Santambrogio. (Euclidean, metric, and Wasserstein] gradient flows: an overview. Bulletin of Mathematical Sciences, 7:87-154, 2017.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence. PMLR, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation aligmment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16. 16, 2016, Proceedings, Part III 14. Springer, 2016.
Hui Tang and Kui Jia. Discriminative adversarial domain adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
13

---

Published as a conference paper at ICLR 2024
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. In Advances Neural Information Processing Systems, 2019.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 23(7):1661-1674 2011.
Hao Wang. Hao He, and Dina Katabi. Continuously indexed domain adaptation. In International Conference on Machine Learning. PMLR, 2020.
Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In International Conference on Machine Learning. PMLR, 2022.
Yuan Wu, Diana Inkpen, and Ahmed El-Roby. Dual mixup regularized learning for adversarial domain adaptation. In European Conference on Computer Vision, 2020.
Zehao Xiao, Xiantong Zhen, Shengeai Liao, and Cees G. M. Snoek. Energy-based test sample adaptation for domain generalization. In International Conference on Learning Representations, 2023.
Shaoan Xie, Zibin Zheng. Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In International Conference Machine Learning. PMLR, 2018.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Jianfei Yang, Han Zou, Yuxun Zhou, Zhaoyang Zeng, and Lihua Xie. Mind the discriminability: Asymmetric adversarial domain adaptation. In European Conference on Computer Vision, 2020.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning. PMLR, 2019.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning. PMLR, 2019.
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
14

---

Published as a conference paper at ICLR 2024
A MATHEMATICAL BACKGROUND
A.1 WASSERSTEIN METRIC
(2(R") denote the space of probability measures on R" with finite second moments, i.c. P2 (R") {u E P (IR"), [|x]]] [d/])) < oo). Given μr, E P2(RR"), the Wasserstein-p distance between
them is defined
inf / W)(,, (xET(()))
|| dll, v)
(16)
R xR"""
where T(u, v) is the set of all possible couplings between μ and v. For all joint distribution и E [(, v), have jux:) Ja-- TT. y)dy and v(y) fR-- (I, y)zz. The integral the right-hand side is also considered as the cost of the optimal transport (OT) problem (Kantorovitch's formulation), and the is the optimal transport plan. Besides, the monotonicity of the Wasserstein-p distance can be easily shown using Jensen's 's inequality, which implies that for S S Wp(,, v) S Wq(U, v).
In the measurable space (P2(I*"), W2) (a.k.a the Wasserstein space), the inner product is defined as:
(p,")pp= =see (u(x), (x)) (x)(x).
(17)
A.2 DESCENT PROPERTY OF WASSERSTHIN GRADIENT FLOW
For measurable space (P2(R") W2), the vector field of a Wasserstein gradient flow is defined as u, Vw, (we).. The descent property is fundamental property of Wasserstein gradient flow, demonstrating that the rate of change of an energy functional E(pz) over time is always non-positive along the gradient flow. This property can be mathematically derived using the following equation:
d({)) (((p))~(vvnl (Vw.e(mo) dt
dise" dt
= (VV,E(()), - VV, = V VV,, S 0. (18)
The first equality is derived using the Chain Rule, while the second equality obtained by applying the definition of Wasserstein gradient flow.
A.3 DISCRETIZATION SCHEMES OF WASSERSTEIN GRADIENT FLOW
There are two main discretization methods for Wasserstein Gradient Flow. The forward scheme utilizes gradient descent in the Wasserstein space to determine the steepest movement direction. Given an energy functional E(u)) and a step size y, the forward scheme updates the distribution as follows:
M4+1 (T 7 V W (14)) P4-
(19)
The backward scheme, also known as the Jordan-Kiing Kinderlehrer-Otte (JKO) scheme Jordan et al. (1998), is a well-known discretization method for Wasserstein gradient flow. involves solving an optimization problem to obtain the updated distribution p4+1, and is represented as follows:
1 2y
Me+1 argmin (EP)(R() E(H) + W (c,
(20)
B PROOFS
B. PROOF OF LEMMA 1 (ERROR DIFFERENCE OVER LAST INTERMEDIATE AND TARGET DOMAIN)
Lemma For any classifier h E H. the generalization error on the target domain is bounded by the error on the last generated intermediate domain and the Wasserstein distance:
<x (h) S Epr (h) + 2pRWW, (HTT
(12)
15

---

Published as a conference paper at ICLR 2024
Note that the generalization error is defined (h) (h, Ezzz fa(()), where h represents the predictor and fu represents the labeling function defined on the data distribution p.
Proof. We start by using the definition of the generalization enror to get
1 (h) cxr (h)) = Rezur [((h(a) Bx'n (1(h (T(
f (((()), / ( (r(())) dr
(21)
f ((h()), - ((h (T())
where y is any coupling (joint distribution) of for and v. Then we have
1pr (h) -x(h)) = (((h(z) - e ((()
/ (I(M()) ((()))))))))))(()))))))))))))))) ((h())
+ (())))(())))(()))))))))))))))
(22)
/ ((11(())~(()))) (2)) -
/ 2pR 11'111
In the above inequalities, have used the triangle inequality and the Lipschitz continuity property of the loss function and the predictor f (label function f) to derive the desired result. After considering the arbitrary coupling 7, can use the definition of the Wasserstein distance get
G (h) S Spr (h) inf / 2pR ||2
x' dy
(23)
cuT (h) + (HT, S 6 (h) + 2pRW2((T;
B.2 DEFINITION OF CONTINUOUS DISCRIMINABILITY
Definition 1 (Continuous Discriminability) Let fo hū and fr ht be the Bayes optimal predictors on source and target domain. For R- R-Lipschitz labelers {ft)T=1 on the intermediate domains, we can define Ss to satisfy the following minimization objective and denote the minimum as K. In particular, we let Ko be the value of K when all labeling functions within the intermediate domains are set to hi,, and there exists S Ko:
T-1
K= inf Σ EoNa. (f((x) f(t1(1)] S Eampo (x)(2) h+((2) Ko
(24)
In this study, only fo and JT are determined based on the true distributions of the source and target domains. For theoretical analysis, we need to appropriate labeling functions for intermediate domains. As stated in Definition 1, we minimize the bound above to specify the labeling function and obtain the minimum value K. It worth noting that depends on the distribution of the generated intermediate domains. The discriminability of the feature representations can be measured by the term À = (h²) + (x()), ), as described the classic UDA theory (Ben-David et al., 2006). When only one intermediate domain exists, K equals X. For more intermediate domains, we can define K to represent the Continuous Discriminabiliit of the model across domains. an ideal scenario, minimizing the value K to zero is possible. Figure 1(b) provides such example, where Ezwne (f) (x)) = 0 for any two adjacent domains, indicating that while two adjacent labeling functions may differ, they perform equally well on the distribution of the former domain. However, since is con-deterministic, we utilize the deterministic upper bound Ko for further analysis, which corresponds to the source generalization error of the target labeling function Euo (Jr).
16

---

Published as a conference paper at ICLR 2024
B.3 PROOF OF PROPOSITION (THE STABILITY OF GGF)
(The Stability of GGF) Consider { (1i Ys Wi)i=1) are i..d. samples from a domain with distribution and hy is classifter. GGF provides a map TH that transports æ; to the next domain y and updates the classifier hy by empirical risk minimization (ERM) on the shifted samples as hy = arg min H Ei=1 ((T, (24), W.). Denote the labeling functions of the two domains are fy and fw, then, for any ô E (0, 1), with probability at least 1 - 6, the following bound holds true:
Proposition 1
[DB+ log (lo((1/)
len (by) tv (hw)) plBaru (2(1)) - [u(x)) RA,)+ +0
(13)
Vn
where Au = ||: TH" (z:) means the average shift distance for a; Tp
Proof. We first introduce Cp (Nys fu) and use the triangle inequality split the left-hand side (LHS)
into two terms as
1x, (hw) cu (ho)) = cen (hys fp) cu (bvr fu)!
(25)
I Cp (Mpr Cu (Reps Su)l + 1ep (b) cu (hhv Sw)l
For the first term, we can express it as the difference in the labeling function:
(hus - cu (1ys full (x), - (o()
Ex (hp (2'),
S Bznp ll (1)( - -e
(26)
pKa 1 (2)) [u(z)]-
For the second term, we introduce the empirical error éy (h, Ju) E fu(=))) where Iy represents the i-th sample drawn from the distribution μs. Then, using the triangle inequality, we can then obtain the following:
1ep (hig Su) ev (lv: S2)l S le (hus Ex (1ius + 1ē (v) éu (hur Sv)l + 1e (hwv - c (hw, fu)l-
(27)
To bound the difference between the generalization and empirical errors, we utilize the Rademacher complexity (Bartlett Mendelson, 2002). This bound also described in Lemma A. of the previous work by Kumar et al. (2020). Then, we can get
H) v
log(1/8)
1e fv) cu (o): 12 (1pps)) &n (hv. fw)l + +
n
(28)
S lē (1) &y (hvv fu)l O pB + Vlog(1/5)
Vn
We can obtain the second inequality using Talagrand's Contraction Lemma (Ledoux & Talagrand, 1991), which bounds the Rademacher complexity of the composition of hypothesis set H and Lipschitz function l by (H)), where p is the Lipschitz constant of and R. (H) is the Rademacher complexity of the hypothesis set 7L concerning the given dataset.
We rely on the fine-tune tuning classifier updating method to obtain the first term the inequality in Eq. (28). This approach involves updating the classifier using the transported samples with preserving
17

---

Published as a conference paper at ICLR 2024
labels. Specifically, we have hu (2%) hy (xx) Then, get
E(h ( - Σ (hv
1e Su) & (N fo)
(Triangle inequality)
16 (24) fx(x)) (h, To (2)) fo [T
(Triangle inequality)
1e (1:) (a(ss) e (lt (2:) (m)).
o Σ e o (T)))))) (h, o T (z.), (x:))
(29)
(Using hy (x.)) = h. o (x.))
(A (m.) e (hr (T.)). T(x)))
(Using p-Lipschitz) Σ p).((())) -o Tx
(Using R-Lipschitz) pfl 12, (aail
Σ
i=1
(Definition of A2) pRAp (Limitation of u.) pRojaU
By plugging Eqs. (26)-(29) into Eq. (25), we reach the conclusion.
B.4 PROOF OF THEOREM 1
Theorem (Generalization Bound for GGF) Under Assumptions 1-5, if fvr is the classifier on the last intermediate domain updated by GGF, then for any 5 (0, 1), with probability at least 1 8, the generalization error of hr on target domain is upper bounded as:
3.3M /TP
G (hrr I Euo (ha) CHo pR
naTU + 2(1 mpm)a" W2 (No, T) +
m
(14)
pB (1))
+o
T
nn
Proof. In Lemma 1, we have the inequality (hr) S Epr (hr) + 2pRW2 (Ho,, (). To prove this inequality further, we will bound the first term Gpr (hr) using the accumulation of Proposition 1, and bound the second term W2 (Hr, т) based on the nature of the gradient flow.
We apply Proposition 1 recursively to get
T=
1E (ho) - Epr (hr)) <
Σ
6pp-1 (ha- - cn (hr)
PRqaTU (1.- pEito +0 [B + Vlog(1/0)
(30)
Only the label functions fo and fr are determined based the true distributions of the source and target domains. To select the optimal label functions f, for the intermediate domains, we minimize the first term of the right-hand side of Eq. (30), similar to the definition of K and Kg. Therefore, for
18

---

Published as a conference paper at ICLR 2024
simple strategy for assigning all labeling functions fo the intermediate domains be fr, have
l=
E Barpe (1)-1 St- (14-n (0 (2),
Bunup 1e (ho(tz)
(ho(2)) fr
(31)
E Ego l (ho()), -Ezuag (ho (2),
r1 e (1o( (x)) + En (fo(2)), fr(()) Bx (o))
E E fr = =Cro (fr) (h+))
To bound W2 (μT, we use the convergence properties of the LMC in the Wasserstein metric, which have been investigated extensively (Durmus & Moulines, 2019; Dalalyan, 2017; Cheng & Bartlett, 2018; Dalalyan & Karagulyan, 2019; Durmus et al., 2019; Vempala & Wibisono, 2019). Suppose the potential is m-strongly convex and M M-Lisschitz smooth. Then by the Theorem I from Dalalyan & Karagulyan (2019), we have
W2 (wr, n) S (1 mn)oT
oT W2 (No, T) + 1.65(5555mmm)))
(32)
where 7) S m- is the constant step-size, and p is the dimension of samples. This theorem shows the LMC implies exponential convergence in the Wasserstein distance. Moreover, replacing the LMC with the JKO scheme (Jordan et al., 1998) would further eliminate the second term (Salim et al., 2020).
Finally, by combining Eq. (30)-(32), we have 6 (hr) S Eur (br) + 2p
S 4u (ho) + (hir) + PRRaTU +0 pB + Vlog(1/5) T
+ 2pRRW2(TT, m)
Vn
3.3M TP
(33)
$ Eno (ha) + (h+p + pR
naTU + - mmm ) W₂ (Ma, T) a)+
m
pB + log (1/5) T +0
Vn
where we conclude.
C COMPLETE ALGORITHM AND COMPLEXITY ANALYSIS
The process construct intermediate domains is outlined in Algorithm 1. Assuming that N represents the size of the source domain dataset, the space and time complexities are O(N) and ((.N), respectively. In practice, we divide the dataset into multiple batches, keeping consistent complexities.
Algorithm 1: Construct Next Intermediate Domain (X4; Mrs &x,do M4y D8 Ot, (1)
Input: Samples (X,, yr), Classifier h, Score network (,, d). Rectified flow (x()).
Hyperparameters or, 7, X
Output: Adapted samples (x++1, Mt++) # Simplify the classifier-based potential
C(x,, he, y) = (), h4, 8)
(Xx,X,, (4)
# Update samples using three energy functions and construct each domain after iterations repeat a times
X4 f- X;
11 (x:: + / 2m1 1/2 Vx. L(xi, he, Wc) + 13 (xi) Classifier-ben
inaribattoonbaeed
sample-based
end
(X++1. M141) + Ut)
T- Exwpe ff(() - ft+1(2)]. K is able to 'We can use pK to bound this, where A is denoted as inf ΣT represent the Continuous Discriminabiliry of the intermediate domains. However, since K is non-deterministicc analyze the deterministic upper hound Eng (hz)-
19

---

Published as a conference paper at ICLR 2024
Algorithm 2: Complete Algorithm with Fixed Hyperparameters
Input:
Source samples (Xeres Merc), target samples Xigt. expected number of intermediate domains T, fixed hyperparameters 77 and a, learning rate k. evaluation frequency K
Output: Target classifier hr
entropy loss LCE source samples; Train score network , for distribution-base energy according to JDSMi
Train initial classifier ho according
Train rectified flow Ug for sample-based energy according to JFM;
t <- O, Xo + Xare, dis +- W2(Xu, Xtgt)))
OR operator
short-circuit operator
while I. mod K of 0 OR (2(x,, Xigt) & dis do mod K then
dis t- (x Xtgt);
end
(X(+1, V++1) < Construct Next Intermediate Domain (Xt, M/s 5x,dr hes Vg. at, he+1 f arg a (x 3641); f t 1;
end
Algorithm 3: Complete Algorithm with Bilevel Optimization
Input:
Source samples (Xerer Merc), target samples Xtyt, expected number of intermediate domains T' initialized hyperparameters and a, learning rate k
Output: Target classifier hr
Train initial classifier hgg score network Sn. and rectified flow ug.
for epoch t- MaxEpoch do
1 t 0, Xo f- Xare, dís +- (x Xtgt);
Fix n, update a
while L mod K +f 0 OR W2(X1, S dis do
mod K then
dis + W2(xe, Xtg:);
end
(X++1, 1t+1) Construct Next Intermediate Domain (X,, ye, 8x,0 he, uo a. 17);
h41 + arg minnex((xx+)) Ut41); t 1;
end
Update a as a + L7
# Fix update !) fors+ 0 to T do
(X++1, U1+11 Construct Next Intermediate Domain (X4, Me, 8x,ds de hes co a. 1));
hx+1 f- arg min ( (xx 11+41:;;
end
Update as t kV,, W2(x,, Xigt); end
Algorithm 2 provides a comprehensive overview of our complete algorithm. This approach dy- namically determines the optimal number of iterations based on the Wasserstein distance within mini-batch. To compute this distance, we employ the Sinkhorn solver from the toolbox (Flamary et al., 202 1), which has demonstrated nearly Onn²) space and time complexities, where n signifies the batch size. To further alleviate the computational burden associated with distance calculation, introduce an evaluation frequency parameter K. Consequently, the space and time complexities be- come O(N n²2) and O(TTN Tn² / K), respectively. If fix the values for the hyper-parameter T, the space and time complexities reduce to O(N) and OaTNN), respectively.
Due to the monotonicity property, our analysis in Eq. (14) suggests that Hyper-parametee with the lowest upper bound remains deterministic when others are held constant. We introduce a bi-level optimization approach, detailed in Algorithm 3, find appropriate hyper-parameter settings. In our experiments, apply this optimization exclusively to the GDA dataset. We do not perform hyperparameter optimization for the Office-Home and VisDA-2017 datasets due to gradient vanishing issues caused by numerous iterations. Instead, we use grid search to determine the prperpparameter.
20

---

Published as a conference paper at ICLR 2024
D ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
D.1 TOY DATASET
Figure 6 illustrates the toy datasets. The first dataset is mixture of three Gaussian blobs (3 classes), while the second dataset is the two-moon dataset (2 classes) implemented using scikit- learn Pedregosa et al. (2011). Both datasets consist of 1000 samples per class in the source domain. To construct the target domain, we sample an additional 1000 samples from the same distribution and then shift the samples. We change the mean standard variance for the Gaussian blob dataset to create shifted distribution. We rotated samples around the origin 45 degrees for the two-moon dataset.
Donaan
(a) Mixture Gaussian Blobs
100
(b) Two Moons
Figure The left figures show the source domain data and initial classifier, the middle ones show the target domain data and estimated probability density by denoise score matching, and the right ones show an intuitive comparison between the source and target domains.
D.2 EVALUATION ON THE LARGE-SCALE DATASET
Table 5: Accuracy (%) VisDA- 2017 (ResNet-50).
To further validate the effectiveness of algorithm, we ducted an additional experiment on this dataset. We utilized the state-of-the-an RSDA method for feature extraction and applied GGF in the feature space. As shown in Tables and 6, the results demonstrate notable improvements, with an average accuracy in- crease of 2.3% across classes and a 1.8% enhancement in overall accuracy.
Mathod
Synthetic Real
CDAN (Long al., 2018)
70.0
MDD (Zhang 2019)
74.6
RSDA (Gu al., 2020)
75.8
RSDA GGF
77.6
Table 6: The detailed accuracy (%)
VisDA campared with RSDA.
aero
bicycle
car
motor
person
plant
skate
train
tnick
Avg.
Acc.
RSDA
87.3
88.9
74.0
94.2
0.06
44.8
93.4
84.5
42.4
73.72
75.75
RSDA GGF
92.4
85.2
87.1
73.3
94,1
0.07
89.7
70.0
94.0
90.6
84,5
43,9
75.98
77.56
21

---

Published as a conference paper at ICLR 2024
D.3 COMPARATIVE EVALUATION OF VARIOUS FEATURE EXTRACTORS
Table 7: Accuracy (%) on Office-Hooreeee dataset with ResNet-50 (Red for increased accuracy).
Method
Ar-CCI
Ar-Pr
Ar-Rw
Cl+Ar
Cl-Pt
Cl-Rw
Pr-AA
Pr-+C1
Pr-RR
Ru-+Ar
Ru-+CI
Rw-Pr
Avg.
ResNes- 50 (hnn- (une))
46.2
71.5
34.0
58.4
68.1
69.7
55.8
42.3
72.9
66.9
47.3
76.0
62.4
ResNet-50 GGF
48.5
74.0
76.7
61.2
70.0
60.3
44.2
75.4
69.1
49.5
64,9
RosNet-50
+2.3
+2.5
+2.5
70.9
71.0
MSTN CGF
58.1
75.9
79.7
66.5
75.7
75.1
63.6
58.6
81.5
74.3
60.0
81.2
71.3
A for MSIN
+0.5
+5.0
+27
+6.1
+4.7
+3.9
+4.2
+2.3
+1.9
+3.4
45.6
44.2
+3.9
RSDA
53.2
77.7
$1.3
66-4
74.0
76.5
67.9
53.0
82.0
75.8
57.8
85.4
70,9
RSDA GUP
60.1
71.9
82.2
08.4
78.3
71.2
67.8
00.3
82.5
75.8
61.0
85.2
73.1
for RSDA
+6.9
+0.2
+2.0
+4.3
-0.1
+6.7
+0.5
+3.2
+2.2
CaVi
58.5
80.0
60.2
82.1
63.6
73.1
CuVi+ à CoVi
59.2
79.0
80.1
83.1
76.2
73.6
+0.7
+0.9
+0.4
+1.2
-0.1
+0.9
+0.4
+1.4
+1.0
-0.4
-0.9
+0.5
In Table 7, we reiterate the performance of our method on the Office-Home dataset with three different feature extractors. Our primary objective and contribution revolve around the generalization of pre-trained features by any feature extractor, which is readily accessible nowadays, to target domains. Through the comprehensive empirical evaluation, we provide compelling evidence to demonstrate the consistent superiority of GGF even when considering pre-trained features generated by CoVi, which has already shown the SOTA performance in reducing the distribution shift.
D.4 COMPUTATIONAL. OVERHEAD ANALYSIS
In this section, we detail the computational overhead associated with our approach. The training process of GGF can be broadly divided into two stages. The first stage involves training the score network and rectified flow, with the time required contingent upon the num- ber of training epochs. The second stage entails the generation of intermediate domains and the subsequent gradual fine-tuning of the classifier. GGF discretizes the Wasserstein gradient flow using forward scheme, which leads to a time complexity that is linearly dependent on the number of intermediate domains and the scale of the dataset. During this stage, our time consumption for generating intermediate domains and updating the classifier is commensurate with those of the baseline methods, GOAT and CNF. Table 8 presents the running time for each method on the two GDA datasets.
Table 8: Running time (s) on GDA tasks.
Datasets
GOAT
CNF
GGF (Ours)
Portraits MNIST 45"
2.84
5.25
5.76
7.95
14.23
29.53
D.5 VISUALIZZATION
To demonstrate the process of generating intermediate domains for the proposed GGF method, we visualize the evolution of source samples using t-SNE on the Portraits, and MNIST 45 in Figures 7-8, respectively. We equally generated small number of intermediate domains and selected them for visualization. These figures show that the source samples evolve over the intermediate domains, indicating that the gradual domain adaptation algorithm effectively adapt the source distribution to the target distribution.
In the UDA task Office-Home, which involves 65 categories, we conduct detailed analysis of the t-SNE visualizations for specific classes. As shown in Figures 9 and 10, we visualize the categories where the performance can be improved (success cases) and degraded (failure cases).
In the success cases, we observe that same-class samples in the source domain tend to cluster together with slight variance, while their corresponding target domains more widely distributed and do not group as a single cluster. Additionally, the intermediate domains generated by GGF can better cover the target distribution. In the failure cases, we observe that the performance degradation mainly due to the possible diffusion of source domain samples to other similar categories. For instance, in Figure 10(c), the source samples in the "Bottle" category shift in the opposite direction of the target domain distribution, resulting in 6% decrease in accuracy for this category. However, all those samples are classified into a similar category "Soda".
22

---

Published as a conference paper at ICLR 2024
domain = Source Donain
domain = P20 intermediate Domain
domain = #40 Intermedion ate Domain
dombln 860 Interned dlte Domahh
domaain elO trarmedatn Domsh
domenn A Intemmddatt Doraa
domann #120 Inieemeckute Domain
dtmmain - Targel Domain
40
Figure 7: The t-SNE of features from the source, intermediate, and target domains Portraits.
Domahn
Internediee
domaln intomeiane Domaln
domalnn #11800 Intermodiate
100
75
Intarmedina
Intarmadiate
intemedaane Domain
A4200 Interned
100
100
-100
100
1100
Figure The t-SNE of features from the source, intermediate, and target domains on MNIST 45
23

---

Published as a conference paper at ICLR 2024
Categery: Lamp, Accurncy: 22% 80%
Category Totte, Acsumay: 59%
Category: Ciphoards
Accunay:
63%
Source Doman
Trrpel Denai: Real
Adupted Bouice Domain
Damarr: Art
Sourra Domann Cipart
anger Corran:: Phocuct
Tarpal Damar:: Product
Mapiet Source Domait
Adapted Bourte Doman n
(a)
(b)
(c)
Catagory: Accuracy: 57%
Catagory: Flyfopa, Accuracy: 78%
Category: Laptop. Acturacy: 56%
Cliprt
geg Comain: Hual,
larget Demain: Adaplis
Adapped
(d)
(e)
(f)
Figure 9: Visualization of success cases, with source samples (in yellow), the last shifted intermediate samples (in red), and target samples (in blue). Each subfigure corresponds category in single task and displays the corresponding accuracy change for that category.
Domain
Soure Doman:
Donair: Product
(a)
(b)
(c)
Figure 10: Visualization of failure cases, and the settings are the same as Figure 9.
24

---

Published as a conference paper at ICLR 2024
To further validate the effectiveness of our generated intermediate domains, we present visual representations depicting the relationship between the transported examples and the pre-existing examples in the intermediate domains two GDA datasets. These visualizations Portraits and rotated MNIST are depicted in Figure 11 and Fig- ure 12, respectively. In these figures, ellipses represent the variance of samples, while transparency indicates the domain index (with the source domain being the most transparent and the target domain being the least transparent). Each intermediate domain is represented solely by its mean and variance within these visualizations.
20-
-10
-00
Figure 11: T-SNE visualiza- tion of generated (grey) and existing intermediate domains (colorful) on Portraits.
For Portraits, due to the imperfect alignment facial features over the years, previous work (Chen & Chao, 2021) has shown limited overall improvement when using existing intermediate domains. shown, while our transported examples lie along straight trajectory latent space from source to target, they do not precisely match the real intermediate examples. By generating smoother transport, surpasses the accuracy of using the existing examples. For rotated MNIST, many categories perfectly align our transported examples and the existing examples in the I-SNE visualizations, This indicates our model successfully learned the "rotation" features. However, for some categories, the alignment is still poorer, leading to lower performance than gradual self-training on the real intermediate samples.
method achieves more stable accuracy improvements and even
00
cleB S 1567
70
Clss 166F)
-40
60-
-30
-40
70-
dles a
-80
20
-10
(a) Class
(b) Class
(c) Class 5
(d) Class
daal (GGT
ClaB 106F1
-10
5-
-30
-15
-35
-60
-25
-15
<lea (173m))
-20
-40
-50
-60
-00
-20
-00
(e) Class
(f) Class 3
(g) Class
(h) Class 9
Figure T-SNE visualization of generated (grey) and existing domains (colorful) on rotated MNIST. First row: categories well-aligned with real domains. Second row: categories with fewer alignments.
In summary, our generated examples are comparable or even superior to the existing intermediate samples, effectively bridging the gap between the source and target domains.
25

---

Published as a conference paper at ICLR 2024
Table 9: Hyperparameters of GGF on different datasets.
Dataset
Confidence Threshold
A
T
1/2
f/2
Portraits
0.05
0
10
20
0.03
0.08
0.01
MNIST 45
0.2
100
60
0.01
0.005
0.002
MNIST
n/a
0.1
0.02
Ar-CI
n/a
0.1
0.05
0.001
Ar Pr
n/a
0.03
0.01
Ar RRw
n/a
20
0.05
0,02
0.0001
Cl-Ar
n/a
10
20
0.01
0.01
0.001
CI-Pr
n/a
10
20
0.01
0.001
0.001
CI RR
n/a
10
20
0.001
0.001
0.001
Pr-tAr
n/a
0.03
0.06
0.001
Pr-CI
0.2
0.05
0.001
Pr-Rw
n/a
100
20
0.005
0.005
0.0001
Rw-AA
n/a
10
50
0.2
0.2
0.001
Rw-
n/a
1
10
10
0.1
0.1
0.001
Rw-Pr
n/a
0.01
0.01
0.001
E IMPLEMENTATION
E.I TRAINING SETTINGS
We use the official implementations 23 for the GOAT (He et al., 2023) and CNF (Sagawa & Hino, 2022) methods and use UMAP (MeInnes et al., 2018) to reduce the dimensions of three GDA datasets to 8. We conduct experiments on single NVIDIA 2080Ti GPU. In the context of the two UDA datasets, namely Office-Home and 2217 we re-implement the RSDA (Gu al., 2020) and CoVi (Na et al., 2022) as feature extractors on single NVIDIA V100 GPU and then conduct our methods on the latent space similarly. For the Rectified Flow (Liu et al., 2023; Liu, 2022), we adopt the official implementation' Our code is publicly available at https//giihbb.com//wwwwozz
E.2 HYPERPARAMETERS
In our experiments, we train three neural networks (i.e., score network, rectified flow, and classifier), each consisting of three fully connected layers. For the classifier, we train it in two steps: training the ini- classifier and then updating using the intermediate domains. We apply SGD optimizer with a learning rate 10 for training all modules and updating the clas- sifier. The batch size for each domain is set to 1024. To improve the training of the score network and rectified flow, we weigh the target domain samples using the classi- fication confidence of the initial classifier.
Portraits
0.86
Accuracy
0.84
658 0.001
0.82
0.01
eta 0.02
eta 0.04
32
64
128
The value of alpha
Figure 13: Accuracy comparison with varying hyperpa- rameters on Portraits.
During the gradual adaptation phase, we update the classifier in each intermediate domain using self-training or fine-tuning
with five epochs. For self-training, we set confidence threshold to filter out the least confident examples, which means that 100% of the samples will removed when updating the classifier. For fine-tuning, we use all generated samples with preserving labels and note that the confidence threshold e not applicable this context.
We have discussed how to optimize hyperparameters in Appendix C. Here, we provide the hyperpa- rameters in Table 9, where À is the balancing weight between the cross-entropy and entropy in the class-based energy function, and T. and 77 used in the sampling process. As shown in Figure 13, our method is not particularly sensitive to those hyperparameters when setting a as a small integer.
26


Reviews:
[{"review_id": "uThx457qv8", "reviewer": "Reviewer_MZDz", "summary": "The work proposes a gradual domain adaptation method that improves classification results. The method applies Wasserstein gradient flow to minimize a novel energy function. The samples from the source distribution flow gradually to the target distribution by multiple intermediate domains. The experiments show that the classification accuracies are comparable to current classification methods.", "questions": "1. Do you make assumptions about how the distributions look like and if the source and target distributions are close?\n2. The accuracies grow as the energy is minimized. Is there a way to measure the quality of the flowed images, both qualitatively or using some metrics like FID?\n3. Is it possible to use the Wasserstein distance in the distance-based energy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is novel in terms of minimizing an energy function and using that to help a classification task.\n2. The writing is clear and the paper is easy to follow.\n3. The paper conducts several experiments and numerical comparisons to show that the classification results are comparable with other methods.", "weaknesses": "1. In Table 2, it looks like the results are only the best accuracy for three tasks.\n2. In Figure 1, it is a bit unclear what the source distribution and target distribution are. Are they both portraits but only the source distribution has labels? It would be more clear if you defne what are $\\mu_t, \\pi$, such as writing them as metrics based on $x,y$.\n3. We recommend the authors cite the following two recent works on MMD and gradient flow:\n\nFan, J. and Alvarez-Melis, D., 2023. Generating synthetic datasets by interpolating along generalized geodesics. arXiv preprint arXiv:2306.06866.\n\nHua, X., Nguyen, T., Le, T., Blanchet, J. and Nguyen, V.A., 2023. Dynamic Flows on Curved Space Generated by Labeled Data. arXiv preprint arXiv:2302.00061.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10DYfgjDWY", "reviewer": "Reviewer_KhtB", "summary": "This paper studies the gradual domain adaptation problem. It leverages gradient flow analysis for generating intermediate domains between the source and target domain to facilitate the adaptation. The proposed approach consists of three main components: (i) A distribution based loss function that transports source features to target ones. (ii) A classifier based regularizer to preserve the label information while transporting the features. (iii) sample based noise reduction regularizer. The proposed approach is supported both theoretically and experimentally where performance gains are consistent.", "questions": "In addition to the points raised in the weaknesses, I have the following question:\n\n1) From the methodology part, can you explain how is the combined approach in section 3.1 and 3.2 related to the Classifier-Free Diffusion Guidance [A]? \n\n[A] Classifier-Free Diffusion Guidance, NeurIPSW 2021.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of this work are:\n\n(1) The paper is fairly well-written. \n\n(2) The proposed approach is novel and theoretically supported.\n\n(3) The experiments support the effectiveness of the proposed approach.", "weaknesses": "While I am not an expert in this field, I would recommend/suggest the following for improving the paper:\n\n1) The methodology section needs some elaboration for better readability. How are the score networks trained? The same question goes to the rectified flow $\\nu_\\theta$.\n\n2) The theoretical analysis, while supporting the proposed approach, seem not related to the experiments conducted in section 5. I would recommend investing some space in the main paper for implementation details and computational burdens.\n\n3) Regarding the experiments: The following experiments are missing from the main work:\n\n(3a) Hyperparameter ablations: While Table 3 shows the effectiveness of each component of the proposed approach, sensitivity analysis with regard to $\\eta_1, \\eta_2, \\eta_3, \\lambda$ are missing.\n\n(3b) While GGF showed enhanced performance when combined with two approaches (Cove and RSDA), it is important to compare different methods in terms of runtime and computational overhead.\n\n(3c) Experiments on more challenging datasets such as DomainNet and ImageNet-C where domain shifts are more severe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sQ5pQUeIbO", "reviewer": "Reviewer_vJxQ", "summary": "This paper introduces Gradual Domain Adaptation via Gradient Flow (GGF) to generate intermediate domains (when they are not given), which are then used for the fine-tuning process of a gradual domain adaptation (GDA) setting. In the proposed GGF method, feature representations from the source domain are gradually transported to the target domain by using a Wasserstein gradient flow (WGF) that aims to minimize the following three designed energies: distribution-based energy, classifier-based energy, and sample-based energy. In addition, the authors provide a theoretical analysis of its generalization bound. Experimental results on several benchmark datasets demonstrate the advantage of the proposed GGF method compared to other current baselines.", "questions": "Please address my concerns/questions in the Weaknesses part above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and relatively easy to follow. The idea of generating (synthetic) intermediate domains with preserving labels is novel and interesting.  I appreciate the authors' effort to provide a theoretical guarantee of a target error-bound of the proposed GGF.", "weaknesses": "1. Regarding the proposed method, can the authors provide an estimation of the sufficient number of the intermediate domains $T$ to obtain a pre-defined target performance? What happens when $T \\to \\infty$?\n\n2. Current experimental results of the paper are supportive and promising, but not very convincing. In particular,\n\n- Lack the empirical comparison with the very related baseline [A] in both Tables 1 & 2. \n\n- Currently, the experiment results can only show the benefit of GGF when applying for two baselines (including RSDA and Covi). I am curious how GGF can be combined with others mentioned in the paper.\n\n- GGF seems to be a time-consuming. Comparison regarding the running time with other mentioned baselines? \n\n[A]. @inproceedings{wang2022understanding,\n  title={Understanding gradual domain adaptation: Improved analysis, optimal path and beyond},\n  author={Wang, Haoxiang and Li, Bo and Zhao, Han},\n  booktitle={International Conference on Machine Learning},\n  pages={22784--22801},\n  year={2022},\n  organization={PMLR}\n}\n\nClarification: \n1. In Fig. 1, the transformation from the source to the target is performed in the latent space, while everything in the paper is about the original data $x$. Could you please clarify that?\n\n2. The proposed method relies on the labeling function $f$ for each domain, while we do not have access to the labels of target samples in a general UDA framework. Could you please comment on this?\n\n3. Could you please double-check the last equality in Eq. (5). It seems the integral over $\\mu_t$ term was missed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cvptQQWGzc", "reviewer": "Reviewer_T69j", "summary": "The manuscript introduces a novel approach for generative gradual domain adaptation, termed Gradual Domain Adaptation via Gradient Flow (GGF). This method produces intermediate domain data and iteratively adapts classifiers through a sequence of these intermediate domains. The core concept involves creating intermediate domains between the source and target domains by optimizing a Wasserstein gradient flow to reduce distribution discrepancies. The gradient flow specifically aims to minimize three energies (which the Wasserstein gradient flow is decomposed into): \ni) a distribution-related energy for aligning source features with the target\nii) a classifier-related energy to maintain label integrity\niii) a sample-related energy to attenuate noise. \n\nThis facilitates the incremental adaptation of the source classifier to the target domain via fine-tuning processes. The authors offer a theoretical framework to constrain the target error based on the properties of the gradient flow. The efficacy of GGF is supported through extensive experiments on standard Domain Adaptation benchmarks.", "questions": "+ [**UDA Experiment Lacks Explanation**] The experiment on Office-Home (shown in Table 2) shows \"RSDA+GGF\" and \"CoVi+GGF\" without explanation. I know CoVi also (implicitly) generates intermediate domains. How do you apply GGF on top of these methods? The paper lacks a detailed explanation of your experimental protocols here.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ [**Novelty**] The idea of connecting gradual domain adaptation (GDA) and Wasserstein gradient flow is natural, given that previous works have already developed theories of GDA with Wasserstein distance. However, the decomposition of Wasserstein gradient flow into three energy terms by [Santambrogio, 2017] is less known in the machine learning community. The authors managed to apply this decomposition to the GDA problem in a non-trivial way: they relate each energy term with some machine learning notion, and find proper ML loss terms as proxies for these energy terms.\n\n+ [**Theoretical Guarantee**] The theoretical analysis grounds the approach well by providing insight into properties like transport cost and label preservation that are useful for analyzing GDA. Bounding the target error connects the gradient flow construction to actual adaptation performance. This helps justify design choices made in GGF.\n\n+ [**Extensive Experiments**] The method is general and can work with different base classifiers and DA techniques that provide feature representations. Comprehensive experiments on benchmark datasets including rotated MNIST, Portraits and Office-Home demonstrate clear improvements over prior arts. The consistent gains across tasks highlight the effectiveness of GGF.\n\n+ [**Visualization**] I quite appreciate the visualizations in Fig. 1, 2, 4, which illustrate the proposed method clearly.", "weaknesses": "+ [**No Explanation of the Original Energy Decomposition**] The algorithm is motivated by [Santambrogio, 2017]'s decomposition of Wasserstein gradient flow into three energy terms. However, the authors only describe them as \"internal, (external) potential, and interaction energy\" without demonstrating their specific definitions. It would be more convincing if the authors can list the original definitions of the energy terms and compare the deviation of their designed loss terms from the original terms.\n\n+ [**Too Many Hyperparameters** ] The loss has three terms, each with a learning rate; there is also an `alpha` term and a `T` term as hyper-parameters. Overall, I feel the number of hyper-parameters is a little high, which may makes the algorithm harder to apply in practice.\n\n+ [**Number of Intermediate Domains should be varied for fair comparison**] In Sec. 5.2, the authors show the improvement of their proposed GGF over GOAT & CNF on the Portraits dataset in Fig. 3. However, the number of intermediate domains in Fig. 3 is fixed to 19. For the GOAT algorithm, its authors demonstrate that the optimal number of intermediate domains for GOAT is below 5, and more intermediate domains may harm its performance. The current choice of 19 intermediate domains might be optimal for GGF, but not for GOAT. So I urge the authors to compare GGF vs. GOAT/CNF across various numbers of intermediate domains (e.g., 2, 4, 8, 16).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)