Principled Federated Domain Adaptation: Gradient Projection and Auto-Weighting
Authors: Enyi Jiang
Yibo Jacky Zhang
Sanmi Koyejo
Venue: ICLR 2024 poster (2024)
Keywords: federated domain adaptation
federated learning
domain adaptation
machine learning
Primary area: general machine learning (i.e., none of the above)
TL;DR: We address the problem of Federated Domain Adaptation in a principled way through investigating server aggregation rules, yielding both theoretical and practical contributions.

Abstract: Federated Domain Adaptation (FDA) describes the federated learning (FL) setting where source clients and a server work collaboratively to improve the performance of a target client where limited data is available. The domain shift between the source and target domains, coupled with limited data of the target client, makes FDA a challenging problem, e.g., common techniques such as federated averaging and fine-tuning fail due to domain shift and data scarcity. 
To theoretically understand the problem, we introduce new metrics that characterize the FDA setting and a theoretical framework with novel theorems for analyzing the performance of server aggregation rules. Further, we propose a novel lightweight aggregation rule, Federated Gradient Projection ($\texttt{FedGP}$), which significantly improves the target performance with domain shift and data scarcity. Moreover, our theory suggests an $\textit{auto-weighting scheme}$ that finds the optimal combinations of the source and target gradients. This scheme improves both $\texttt{FedGP}$ and a simpler heuristic aggregation rule. Extensive experiments verify the theoretical insights and illustrate the effectiveness of the proposed methods in practice.

Published as a conference paper at ICLR 2024
PRINCIPLED FEDERATED DOMAIN ADAPTATION: GRADIENT PROJECTION AND AUTO- WEIGHTING
Enyi Jiang" enyi jiiiiiiii j2@illinois.es edu UTUC
Yibo Jacky Zhang Stan
Sanmi Koyejo sanmiêcs stanford. edu Stanford
edu
ABSTRACT
Federated Domain Adaptation (FDA) describes the federated learning (FL) setting where source clients and server work collaboratively to improve the performance of target client where limited data is available. The domain shift between the source and target domains, coupled with limited data of the target client, makes FDA a challenging problem, e.g., common techniques such as federated averaging and fine-tuning fail due to domain shift and data scarcity. To theoretically understand the problem, we introduce new metrics that characterize the FDA setting and theoretical framework with novel theorems for analyzing the performance of server aggregation rules. Further, we propose novel lightweight aggregation rule, Federated Gradient Projection (FedGP), which significantly improves the target performance with domain shift and data scarcity. Moreover, our theory suggests an auto-weighting scheme that finds the optimal combinations of the source and target gradients. This scheme improves both FedGP and simpler heuristic aggregation rule. Extensive experiments verify the theoretical insights and illustrate the effectiveness of the proposed methods in practice.
INTRODUCTION
Federated learning (FL) is a distributed machine learning paradigm that aggregates clients' models on the server while maintaining data privacy (McMahan et al. 2017). FL is particularly interesting in real-world applications where data heterogeneity and insufficiency are common issues, such as healthcare settings. For instance, small local hospital may struggle to train generalizable model independently due to insufficient data, and the domain divergence from other hospitals further complicates the application of FL. A promising framework for addressing this problem is Federated Domain Adaptation (FDA), where source clients collaborate with the server to enhance the model performance of target client (Peng et al. 2020). FDA presents a considerable hurdle due two primary factors: (i) the domain shift existing between source and target domains, and (ii) the scarcity of data in the target domain.
Recent works have studied these challenges of domain shift and limited data in federated settings. Some of these works aim to minimize the impacts of distribution shifts between clients (Wang et al.)
2019; Karimireddy et al.] 2020 Xie et al. 2020b), e.g., via personalized federated learning (Deng et al. commonly presume that all clients possess ample data, an assumption that may not hold for small hospitals in cross-silo FL setting. Data scarcity challenges can be a crucial bottleneck in worldworld scenarios, e.g., small hospitals lack data whether labeled or unlabeled. In the special (and arguably less common) case where the target client has access to abundant unlabeled data, Unsupervised Federated Domain Adaptation (UFDA) (Peng al.) 2020 Feng et al.) 2021 Wu & Gong. 2021) may be useful. Despite existing work, there remains an under-explored gap in the literature addressing both challenges, namely domain shift and data scarcity, coexist.
2020 Li et al., 2021, Collins et al., 2021 Marfoq et al., 2022). However, these studies
To fill the gap, this work directly approaches the two principal challenges associated with FDA, focusing carefully designing server aggregation rules, i.e., mechanisms used by the server to combine updates across source and target clients within each global optimization loop. We focus on
"Equal contribution

---

Published as a conference paper at ICLR 2024
aggregation rules as they easy to implement requiring only the server to change its operations (e.g., variations of federated averaging (McMahan et al., (017)), and thus have become the primary target of innovation in the federated learning literature. In briff, our work is motivated by the question:
How does one define "good" FDA aggregation rule?
To our best understanding, there are no theoretical foundations that systematically examine the behaviors of various federated aggrega- tion rules within the context of the FDA. Therefore, we introduce metrics characterize
Ey
theoretical framework that establishes the FDA settings and employ them to analyze the performance of FDA aggregation rules. The proposed metrics characterize (i) the divergence between source and target domains and (ii) the level of training data scarcity in the target domain. Leveraging the proposed theoretical framework, we propose and analyze two aggregation approaches. The first is a simple heuristic FedDA, a simple convex combination of source and target gradients. Perhaps surprisingly, we discover that even noisy gradients, computed using the limited data of the target client, still deliver valuable signal. The second is a novel filtering-based gradient projection method, FedGP. This method is designed extract and aggregate beneficial components of the source gradients with the assistance of the target gradient, as depicted in Figure [ FedGP calculates a convex combination of the target gradient and its positive projection along the direction of source gradients. Intriguingly, using generalization analysis on the target domain, our theoretical framework unravels why FedGP may outperform FedDA- specifically, we find that performing the projection operation before the convex combination is crucial.
Bas
8,s
Figure 1: FedGP filters out the negative source gradients (colored in red) and convexly combines gr and its projec- tions to direction of the remaining source gradients (green ones).
Importantly, our theoretical framework suggests the optimal weights for combining source and target gradients, leading auto-weighted versions of both FedGP and FedDA. In particular, we find that the under-perforning FedDA is significantly improved by using auto-weighting enough to be competitive with FedGP, demonstrating the value of our theory. Across extensive datasets, demonstrate that FedGP, as well as the auto-weighted FecDA and FedGP, outperforms personalized FL and UFDA baselines. Our code is at futhtpss:: / wwg...h........... com jacky///ac/yyyy//// wtt.eekyyyyy//tttedeeeee AutoFedGP
Summary of Contributions, Our contributions are both theoretical and practical, addressing the FDA problem through federated aggregation in principled way.
We introduce theoretical framework understanding and analyzing the performance of FDA aggregation rules, inspired by two challenges existing in FDA. Our theories provide a principled response to the question: How do we define a " "good" FDA aggregation rule?
We propose FedGP as effective solution the FDA challenges of substantial domain shifts and limited target data.
Our theory determines the optimal weight parameter for aggregation rules, FedDA and FedGP.
This auitowweggtinng scheme leads further performance improvements.
Extensive experiments illustrate that our theory is predictive of practice. The proposed meth- ods outperform personalized FL and UFDA baselines on real-world ColoredMNIST, VLCS, Terralncognita, and DomainNet datasets.
THE PROBLEM OF FEDERATED DOMAIN ADAPTATION
We begin with a general definition of the problem of Federated Domain Adaptation and, subsequently a review of related literature in the field.
Notation. Let D be data domain on ground set Z. In supervised setting, data point z E Z is the tuple of input and output data We denote the loss function as l 6 x Z + R, where the parameter space is θ RT, an mi-dimensional Euclidean space. The population loss is (0) = (e 2), where E=-D the expectation w.r.t. D. Let D be finite sample dataset
In this paper, the terms distribution and domain are used interchangeably. For example, let a be the inputs and y be the targets, then (z. y)
2

---

Published as a conference paper at ICLR 2024
drawn from D, then (0(0) == 1 EzeB e(o, 2). where |D] is the size of the dataset. We use [a]
[N] := .1. NJ. By default, (,,), and || denote the Euclidean inner product and Euclidean norm, respectively.
In FDA, there are N source clients with their respective source domains (Ds, JEEN and target client with the target domain Dr. For Vi E [N]. Ds, denotes the ndhh source client dataset, and Dr denotes the target client dataset. We focus on the setting where 'Drl is relatively small. In standard federated learning. all clients collaborate to learn a global model orchestrated by the server, i.e, clients cannot communicate directly, and all information is shared from/to the server. In contrast, FDA uses the same system architecture to improve a single client's 's performance.
Definition 2.1 (Aggregation for Federated Domain Adaptation (FDA)). The FDA problem is federated learning problem where all clients collaborate to improve the global model for a target domain. The global model is trained by iteratively updating the global model parameter
8 - (4.Agg(((VV (f))EIM (e),())),
where V is the gradient, is the step size. We seek an aggregation strategy Aggr(.) such that after training, the global model parameter 0 minimizes the target domain population loss function (0). Note that we allow the aggregation function Aggr(.) to depend on the iteration index.
There are a number of challenges that make FDA difficult to solve. First, the amount of labeled data in the target domain is typically limited, which makes it difficult to learn a generalizable model. Second, the source and target domains have different data distributions, which can lead to mismatch between the features learned by the source and target models. Moreover, the model must be trained in privacy-preserving manner where local data cannot be shared.
2.1 RELATED WORK
Data heterogeneity, personalization and label deficiency in FL. Distribution shifts between clients remain crucial challenge in FL. Current work often focuses improving the aggregation rules:
Karimireddy et al. (2020) use control variates and Xie et al.] (2020b) cluster the client weights to
correct the drifts among clients. More recently, there works (Deng et al. 2020 Li al., 202T
Collins et al.] | Marfoq et al. (2022) concentrating on personalized federated learning by finding better mixture of local/global models and exploring shared representation. Further, recent works have addressed the label deficiency problem with self-supervising or semi-supervision for personalized models (Jeong et al.) 2020 He et al. 2021 Yang et al.. 202T). To our knowledge, all existing work assumes sufficient data for all clients nevertheless, the performance of client with data deficiency and large shifts may become unsatisfying (Table D Compared to related work on personalized FL, our method is more robust to data scarcity on the target client.
Unsupervised federated domain adaptation. There is a considerable amount recent work on unsupervised federated domain adaptation (UFDA), with recent highlights in utilizing adversarial networks (Saito et al.] 2018 Zhao et al. 2018), knowledge distillation (Nguyen et al.) 202I), I), and source-free methods (I (Liang et al.] 2020). Peng al. (000)); [i al. (2020) is the first extend MSDA into an Fl. setting; they apply adversarial adaptation techniques to align the representations of nodes. More recently, in KD3A (Feng et al. 2021 and COPA (Wu & Gong 202I). the server with unlabeled target samples aggregates the local models by learning the importance of each source domain via knowledge distillation and collaborative optimization. Their work assumes abundant data without labels in the target domain, while small hospitals usually do not have enough data. Also, training with unlabeled data every round computationally expensive. Compared to their work, we study more important challenge where the data (not just the labels) are scarce. We show our approaches achieve superior performance using substantially less target data on various benchmarks.
Using additional gradient information in FL. Model updates in each communication round may provide valuable insights into client convergence directions. This idea has been explored for robustness in FL, particularly with untrusted clients. For example, Zeno++ (Xie et al.) 2020a) and FITrust (Cao et al. 2021) leverage the additional gradient computed from a small clean training dataset on the server compute the scores of candidate gradients for detecting the malicious adversaries. Differently, our work focuses on a different task of improving the performance of the target domain with auto-weighted aggregation rules that utilize the gradient signals from all clients.
3

---

Published as a conference paper at ICLR 2024
3 A THEORETICAL FRAMEWORK FOR ANALYZING AGGREGATION RULES FOR FDA
This section introduces general framework and a theoretical analysis of aggregation rules for federated domain adaptation.
Additional Notation and Setting. We use additional notation to motivate functional view of FDA. Let gD θ + with (D( == (0). Given a distribution π on the parameter space θ we define an inner product (9D: (DD:9D'> > = Ee~- (0 (0), DD. (0)) We interchangeably denote л as both the distribution and the probability measure. The inner product induces the L -norm on ID as ||gz] :=
v V (0)]2 With the I -oor,,,m we define the I& space as {g : θ + θ l1gl < x.). Given an aggregation rule Aggr(-), we denote Izgg= (0) = Aggr((Ups, (0) 95, (0)). Note that we do not care about the generalization on the source domains, and therefore, for the theoretical analysis, we can view Ds = Ds without loss of generality. Throughout our theoretical analysis, we make the following standard assumption about Dr, and we use the hat symbol, 7, to emphasize that a random variable is associated with the sampled target domain dataset Dr.
Assumption 3.1. We assume the target domain's local dataset Dr {Zitiej)) consists of n. i.i.d. samples from its underlying target domain distribution Dr. Note that this implies BB, 195,1 yDr
Observing Definition] 2.1 we can see intuitively that a good aggregation rule should have Taggr be "close" the ground-truth target domain gradient gDr From functional view, we need to measure the distance between the two functions. We choose the L -noom, formally stated in the following. Definition 3.2 (Delta Error of an aggregation rule Aggr(-)). We define the following squared error term to measure the closeness between Jagg= and gD i.e.,
2gger == B., lll - Il
The distribution π characterizes where to measure the gradient difference in the parameter space.
The Delta error 2 is crucial in theoretical analysis and algorithm design due to its two main bene- fits. First, indicates the performance of an aggregation rule. Second, reflects fundamental domain properties that are irrelevant to the aggregation rules applied. Thus, the Delta error disentangles these two elements, allowing in-depth analysis and algorithm design.
Concretely, for the first benefit: one expects an aggregation rule with small Delta error to converge better as measured by the population target domain loss function gradient Vlp,
Theorem 3.3 (Convergence and Generalization). For any probability measure π over the parameter space, and an aggregation rule Aggr(-) with step size p. > O. Given target domain sampled dataset Dr, update the parameter for T steps by gr+1 == 6¹ pg Assume the gradient Vi(0,=) and (0) are 3-Lipschitz in 0 such that 0f + 0 Agggr- Then, given step size S 2 and small enough > O, with probability at least 1 we have
1
(VC. Aagg- + +OO)) + +o +
VV (8 01
where C. = 11,,1/T( (B,,(1//((B(000gr))) (0gggr))) and B, (Aggg))) C R.m is the ball with radius e centered at Oagge- The C. measures how well the probability measure π. covers where the optimization goes. i.e. Oaggz-
Interpretation. The left-hand side reveals the convergence quality of the optimization with respect the true target domain loss. As we can see, a smaller Delta error indicates better convergence and generalization. In addition, we provide an analysis of single gradient step in Theorem A. showing similar properties of the Delta error. the above theorem, я arbitrary, allowing for its appropriate choice to minimize the C.. Idcally, would accurately cover where the model parameters are after optimization. We take this insight in the design of an auto-weighting algorithm, to be discussed later.
The behavior of an aggregation rule should vary with the degree and nature of source-target domain shift and the data sample quality in the target domain. This suggests the necessity of their formal characterizations for further in-depth analysis. Given source domain Ds, can measure its distance to the target domain D, as the L norm distance between fD3 and the target domain model ground-truth gradient gD,, hence the following definition.
4

---

Published as a conference paper at ICLR 2024
Definition 3.4 (L* Source- Target Domain Distance). Given source domain Ds, its distance the target domain Dr is defined as
d, Dr) = lgDr 9Ds 10
This proposed metric d, has some properties inherited from the norm, including: (i. symme- try) dr (Ds: DT) (DT, Dz); (ii, triangle inequality) For any data distribution D we have (Ds, Dr) & da (Ds. D) + d,, (Dr,D), D): (iti. zero property) For any D we have (D, 0.
To formalize the target domain sample quality, we again measure the distance between Dr and Dr. Thus, its mean squared error characterizes how the sample size affects the target domain variance.
Definition 3.5 (L* Target Domain Variance). Given the target domain Dr and dataset Dr (2hhien)) where 2, Dr is sampled i.i.d., the target domain variance is defined
of (DT) l9p, ID, E.p lIpp, Te((,2))i =: - n where (2) the variance of single sampled gradient function Vt(;;)).
02)
nt
Taken together, our exposition shows the second benefit of the Delta error: it decomposes into a mix of the target-source domain shift d,, (Ds, Dr) and the target domain variance (Dr) for at least wide range of aggregation rules (including our FedDA and FedGP).
Theorem 3.6 (A3 Aggr Decomposition Theorem). Consider any aggregation rule Aggr(-) in the
form of Aagg= = N EiEiN Faggsl9p. 9Ds, i.e. the aggregation rule defined by a mapping Fagge: I x T,x - T.a If Faggr r is affine w.r.. to its first argument (i.e., the target gradient function), and Vg E L -Faoollg [9 9. and the linear mapping associated with Fzogr has its eigenvalue bounded (Amin: \mar), then for any source and target distributions (Ds.)ieNN) Dr, Dr we have 2g I -/2 Eie(N] A Dog where
(2)2
A Aggr.Ds. I m mazz X2 Amin
max Amax) ) Amin) dn (Ds, DT)
n
The implications of this theorem significant: first, it predicts how effective aggregation rule would be, which we use to compare FedGP vs. FecDA. Second, given an estimate of the domain distance dz (Ds,, Dr) and the target domain variance a² (Dr), we can optimally select hyper-parameterd for the aggregation operation, process we name the auto- seig-weggttng scheme.
Interpretation.
With the relevant quantities defined, we can describe an alternative definition of FDA, useful for our analysis, which answers the pivotal question of how we define "good" aggregation rule.
Definition 3.7 (An Error-Analysis Definition of FDA Aggregation). Given the target domain variance a (Dr) and source-target domain distances Vi E [N] : dz (Ds,, Dr), the problem of FDA is find a good strategy Aggr(-) such that its Delta error A is minimized.
These definitions give a powerful framework for analyzing and designing aggregation rules:
given aggregation rule, can derive its Delta error and see how it would perform given an FDA setting (as characterized by the target domain variance and the cource-targel distances);
given an FDA setting, we can design aggregation rules minimize the Delta error.
METHODS: GRADIENT PROJECTION AND THE AUTO- WETOOWEIGHTING SCHEME
To start, may try two simple methods, i.e., only using the target gradient and only using source gradient (e.g.. the jth source domain). The Delta error of these baseline aggregation rules is straightforward. By definition, have that
A Dr only (Dr). and Ab., only (Ds. Dr).
This immediate result demonstrates the usefulness of the proposed framework: if 9aggr only uses the target gradient then the emor is the target domain variance; if Jagge only source gradient then
5

---

Published as a conference paper at ICLR 2024
the error is the corresponding source-target domain bias. Therefore, good aggregation method must strike a balance between the bias and variance, i.e. a bias-vat variance trade-off, and this is precisely what we will design our auto wetihheig mechanism to do. Next, propose two aggregation methods and then show how their auto-weighting can be derived.
4.1 THE AGGREGATION RULES: FEDDA AND FEDGP
straightforward way to combine the source and target gradients is to convexly combine them, as defined in the following.
Definition 4.1 (FedDA). For each source domains i E [N], let f3 E [0, 1] be the weight that balances between the 2th source domain and the target domain. The FedDA aggregation operation is
FedDA((oDs, (0)) (0)
1 N
Σ (1 (1-3,)95, (0) B:9Ds; (8))
Let us examine the Delta error of FedDA. Theorem 4.2. Consider FedDA. Given the target domain Dr and source domains Ds...... DSN we have PedDA S N Ei=1 Leduass where
A FeaDa,s, -
Bl Dr).
Therefore, we can see the benefits of combining the source and target domains. For example, with dd (Ds,, Dr). We note that A Fednz,5, attains the upper
3 = 3. we have A S. - bound Theorem 3.6 with Amas = Amin = 1 3,. This hints that there may be other aggregation rules that do better,
shown in the following.
Intuitively, due to domain shift, signals from source domains may not always be relevant. Inspired by the filtering technique in Byzantine robustmess of FL (Xie et al.) 2020a), propose Federated Gradient Projection (FedGP). This method refines and combines beneficial components of the source gradients, aided by the target gradient, by gradient projection and filtering unfavorable ones. Definition 4.3 (FedGP). For each source domains E [N], let 3, E [0, 1] be the weight that balances between th source domain and the target domain. The FedGP aggregation operation is
FedGP FedGP((gDs, (0)) 9D, (0)) ( (1- Bd95, (A) (05, (0)lgDs (0))
N Σ 1=1
where Proj+ (95, (9) 9Ds, (0)) = max (95, gDs, (0)), 0g (0) (0) is the opera- tion that projects 96, (0) to the positive direction of 9Ds; (A).
We first derive the Delta error of FedGP, and compare that FedDA. Theorem 4.4 (Informal Version). Consider FedGP. Given the target domain Dr and N source
domains Dsy..... DSN, we have 2 < Lim1 A FadGr,S. where
23, 82 0 (Dr) + 8772 (Ds., Dr):
A 2escs Rs (11-
m
In the above equation, is the model dimension and 72 E, [T() E [0,1] where r(0) is the sin(-) value of the angle between 9Ds (0) and gn, (0) gDg (0).
We note the above theorem is the approximated version of Theorem A.5 where the derivation is non-trivial, The approximations are mostly done in analog to a mean- field analysis, which are detailed in Appendix
Interpretation. Comparing the Delta error of FedGP (equation 2 and that of FedDA (equation D]. that FedGP is more robust large source-target domain shift d, (Ds,, Dr) given <1. This aligns with our motivation FedCP which filters out biased signals from the source domain. Moreover, our theory reveals a surprising benefit of FedGP as follows. Note that
2 - 2 S; R 8 22 (Ds,, Dr) 23:-87 0 (Dr).
In practice, the model dimension m > 1 while the 72 < 1, thus we can expect FedGP to be mostly better than FedDA with the same weight B, With that said,
the auto-weighting scheme.

---

Published as a conference paper at ICLR 2024
4.2 THE AUTO-WEIGHTING FEDGP AND FEDDA
Naturally, the above analysis implies a good choice of weighting parameters for either of the methods. For each source domains St, we can solve for the optimal 33 that minimize the corresponding Delta errors, i.e., A S, (equation I for FedDA and A Ce (equation 2 for FedGP. Note that for
A FedGP.S. we can safely view 28, -
given the high dimensionality of our models. Since either of the Delta errors is quadratic in m 3,, they enjoy closed-form solutions:
(Dr)
(D1)
(3)
dD DT) (D,)
(Ds, Dr) + (D
The exact values of o2 ( Dr), ground-trute -ruth target domain gradient 9Dr, Fortunately, using only the available training data, we can efficiently obtain unbiased estimators for those values, and accordingly obtain estimators for the best B4 The construction of the estimators is non-trivial and is detailed in Appendix A.5
are unknown, since they would require knowing the
The proposed methods are summarized in Algorithm [ and detailed in Appendix C During one round of computation, the target domain client does B local model updates with B batches of data. In practice, use these intermediate local updates {0), 1 estimate the optimal 84, where S stands for the local model update using the jth batch of data. In other words, we choose to be the empirical distribution of the model parameters encountered along the optimization path, aligning suggestion for an ideal
with Theorem 3.3
We observe that FedGP, quite remarkably, is robust to the choice of 3: simply choosing 3 = 0.5 is good enough for most of the cases observed in our experiments. On the other hand, although F FedDA is sensitive to the choice of 3. the auighweighted procedure significantly improves the performance for FedDA, demonstrating the usefulness of our theoretical framework.
Algorithm 1 FDA: Gradient Projection and the Auto Weighting Scheme
Input: N source domains Ds {Ds, J target domain Dr; N source clients {Cs, target client Cr. server S; number of rounds R; aggregation rule Aggr; whether to use auto_ wttht. Initialize global model h global Default [ + [0.5)
for r = 1,2,..., R do for source domain client Cs, in [ J.V do Initialize local model S, + "global optimize n on Ds., send hg? to server S. end for
(1-1))
Target domain client Cr initialize h477) h (r-1)
global" optimizes h44 Dr, send h to server S.
Server S computes model updates gr t h())) (r-1) and gs, + n!? h (1-1)) global for i [N].
global
if auto "'auto_wwigiaa o__eightt then
Cr sends intermediate local model updates (5/
LB Dy J=z S.
S estimates of {dz (Ds,, Pr) 72 and ox using (gs |. and Lo b
S updates N according to 3 end if
S updates the global model (r) h (1-1)) end for
global + Aggr((gs))) IN
18)1
globbl
EXPERIMENTS
In this section, we present and discuss the results of real dataset experiments with controlled domain shifts (Section 5.1) and real-world domain shifts (Section
5.2)
Ablation studies on target data
scarcity and visualizations are available in Appendix C.5
&
C.8 Synthetic data experiments B In Appendix
verifying our theoretical insights are presented in Appendix
C,3 we show our
methods surpass UFDA and Domain Generalization (DG) methods on PACS (Li et al. 2017), Office-
Home /Venkateswara et al., 2017). and DomainNet (Peng et al. 2019). Implementation details and extended experiments be found in the appendix.
7

---

Published as a conference paper at ICLR 2024
5.1
SEMI-SYNTHETIC DATASET EXPERIMENTS WITH VARING SHIFTS
Tantion-MMIST No Features
GIAR-1NNORY Textures
Methinn-NSS Label StiRs
Neias Lovel
No.0 Lovd
(a) FasHion-MNIST noisy features
(b) CIFAR-I0 10 noisy features
(c) Mashion-MNIST label shifts
Figure 2: The impact of changing domain shifts with noisy features or label shifts.
Datasets, models, and methods. We create controlled distribution shifts by adding different levels of feature noise and label shifts to Fashion-MNIST (Xiao et al. (017) and CIFAR-IO (Krizhessky et al. 2009) datasets, adapting from the Non-IID benchmark (Li et al. 2022) with the following two settings: I) Noisy features: We add Gaussian noise levels of std = 0.4, 0.8) to input images of the target client of two datasets, create various degrees of shifts between source and target domains. 2) Label shifts: We split the Fashion-MNIST into two sets with 3 and 7 classes, respectively, denoted D1 and D2- Dg = 1/ portion from D2 and (1 - 17) portion from D2. Dr = =1 ([]-)) 17) portion from D and 1) portion from D2 with 1) = [0.45, 0.30, 0.15, 0.10, 0.05, 0.00). In addition, we use a CNN model architecture. We set the communication round R = 50 and the local update epoch to 1, with 10 clients (1 target, 9 source clients) in the system. We compare the following methods: Source Only (only averaging the source gradients), Fineture__ei sauue_offlli offffffiinn (fine- tuning locally after source-only training), FedDA (3 = 0.5), FedGP(3 0.5), and their auto-weighted versions FedDA_Asa Auto and FedGP_ Auto, the Oracle (supervised training with all target data (Dr))
and Target Only (only use target gradient )Dr)). )). More details can be found in Appendix C.7
wuih-wed methods and FedGP keep a better trade-off between bias and variance. As shown in Figure when the source-target shifts grow bigger, FedDA, Finetune__r Peeuunnofflll offffflinee and Source Only degrade more severely compared with auto-weighted methods, FedGP and Target Only. We find that auto-weighted methods and FedGP outperform other baselines in most cases, being less sensitive to changing shifts. In addition, auto-weighted FedDA manages to achieve significant improvement compared with the fixed weight F FedDA, with a competitive performance compared with FedGP _AAto, while FedGP_ Auto generally has the best accuracy compared with other methods. which coincides with the theoretical findings. Full experiment results can be found in Appendix C.7
5.2 REAL DATASET EXPERIMENTS WITH REAL-WORLD SHIFTS
Datasets, models, baselines, and implementations. We use the Domainbed (Gulrajani & Lopez- Paz
2020ab benchmark with multiple domains, with realistic shifts between source and target clients. We conduct experiments on three datasets: ColoredMNIST (Arjovsky et al.) 2019), VLCS (Fang
ct al. 2013). TerraIncognita (Beery et al. 2018)) datasets. We randomly sampled O. 0.1% % samples of ColoredMNIST, and 5% samples of VLCS and TerraIncognita for their respective target domains. The task is classifying the target domain. We use a CNN model for ColoredMNIST, ResNet- 18 He et al. (2016) for VLCS and Terralncognita. For baselines in particular, in addition to the methods in Section 5.1 we compare (1) personalizatton baselines: FedAvg, Ditto (Li et al., (02T).
FedRep (Collins et al.) 2021), APFI (Deng et al. 2020). and KNN-Per (Marfoq et 2002); (2) UFDA methods: KD3A (Feng et al., (202T) (current SOTA): note that our proposed methods use few percentages of target data, while UFDA here uses 100% unlabeled target data; (3) DG method: report the best DG performance in DomainBed (Gulrajani & Lopez-Paz] 2020b). For each dataset, we test the target accuracy of each domain using the left-out domain the target and the rest as source domains. More details and full results are in Appendix C.2
Our methods consistently deliver superior performance. Table reveals that our auto- weighte methods outperform others in all cases, and some of their accuracies approach/outperfom the corresponding upper bound (Oracle). The auto-weighted scheme improves FedDA significantly.
8

---

Published as a conference paper at ICLR 2024
Interestingly, we observe that FedGP, even with default fixed betas (3 0.5), achieves competitive
results. Our methods surpass personalized FL, UFDA, and DG baselines by significant margins.
ColOredMNIST (0.1%)
VLCS (5%
Terre (5%)
Domains
+90%
+80%
-90%
Ave
L
V
Avg
Avg
Source Only
56.8(0.8)
62.4(1.8)
27.8(0.8)
49.0
90.5(5.3)
60.7(1.8)
70.2(2.0)
69. 62.1(2.0)
72.6
37.5
FodDA
601,((2.5)
65.1(1.3)
33.0(3.2)
52.9
97.7(0.5)
68.2(1.4)
75.3(1.5)
76.7(0.9)
79.5
64.7
FedGP
83.7(9.9)
74.4(4.4)
89.8(0.5)
82.4
99.410.3)
71.1(1.)
73.63.1)
78.7(1.3)
80.7
71.2
AetD Auto
85.3(5.7)
73.1 88.9(11))
82.7
99.8(0.2)
73.1(1.3)
78.4(1.5
83.7(2.3)
83.8
74,6
FedGP Auto
86.2(4.5)
76.5(7.3)
89.6(0.4)
84.1 99.910.2)
73.2(1.8)
78.6(1.5)
83.47(2.5)
83.8
74.4
Target Only
85.6(4.8)
73,5(3.0)
87.1(3.4)
82.1
97.8(1.5)
68.9(1.)
72.3(L.7)
76.0(1.9)
78.7
67.2
FedAvg
63.2
10.9
48.7
96.3
68.0
69.8
68.7
75.7
30.0
Ditto
62.2
71.3
19.3
50.9
95.9
67.5
70.5
66.1
75.0
28.8
FedRep
65.4
36.4
31.7
44.5
60.4
70.3
70.1
73.0
20.9
APEL
43.8
61.6
30.2
45.2
68.6
61.0
65.4
49.9
61.2
52.7
KNN-per
67.5
67.6
12.4
49.1
97.8
65.9
75.7
74.4
78.4
43.1
KD3A (100% data)
65.2
73.1
9.7
49.3
99.6
63.3
78.1
80.5
79.0
39.0
Best DG
49.9
62.1
65.7
73.3
78.7
78.7
48.7
Oracle
90.000.4)
80,3(0.4)
90.000.5)
86.8
100.000.0)
72.7(2.5)
78.7(14) 82.7(1.)
83.5
93.1
Table I: Target domain test accuracy (%) on ColoredMNIST, VLCS, and DomainNet.
5.3 ABLATION STUDY AND DISCUSSION
The effect of source-target balancing weight 3. We run FedDA and FedGP with varying 3 on Fashion-MNIST, CIFAR1O, and ColOred-MNIST, shown in Figure In most cases, observe FedGP outperforming Fe,DA, and FedDA being more sensitive to the varying 3 values, suggesting that FedDA_AAiuu Auto can choose optimal enough 3. Complete results are in Appendix C.6
Effectiveness of projection and filtering. We show the effectiveness of gradient projection and filter- ing Fashion-NN -NNIST and CIFAR- 10 noisy feature experiments Table [ Compared with FedDA, which does not perform projection and filtering, projection achieves a large (15%) performance gain, especially when the shifts are larger. Further, we generally get a 1% 2% gain via filtering.
Fachias- NIST RIST/CUIE CIFARTI
Nuise level
0.4
0.6
0,8
64 48
proj
09.51/64.04
65 46/62
68: 6082911
FeaGP
71.0065.
28
68.01/63.29
62.22%1.59
(a) Fashion-MNIST (b) CIFAR-10
(c) ColoredMNIST (target: +90%)
Figure 4: Ablation study on projection and filtering.
(0.4 noise level) (0.4 noise level)
Figure 3: The effect of 3 FedDA and FedGP.
Discussion. We analyze the computational and communication cost for our proposed methods in Appendix C.10 and C.4. where we show how the proposed aggregation rules, especially the auto-weighted operation, can be implemented efficiently. Moreover, we observe our experiments that Fineeunee_offe lawuuneeeffiieeee in is sensitive its pre-trained model (obtained via FedAvg), highlighting the necessity of a deeper study of the relation between personalization and adaptation. Lastly, although different from UFDA and semi-supervised domain adaptation (SSDA) settings, which use unlabeled samples (Saito et al. 2019 Kim & Kim 2020), we conduct experiments comparing them in Appendix C.3 on DomainNet and C.12 SSDA. Our auto-weighted methods have better or comparable performance across domains, especially for large shift cases.
6 CONCLUSION
We provide a theoretical framework that first formally defines the metrics to connect FDA settings with aggregation rules. We propose FedGP, filtering-bsed aggregation rule via gradient projection, and develop the auto-weighted scheme that dynamically finds the best weights both significantly improve the target performances and outperform various baselines. In the future, plan to extend the current framework to perform FDA simultaneously on several source/target clients, explore the relationship between personalization and adaptation, well as devise stronger aggregation rules.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is partially supported by NSF III 2046795, IIS 1909577, CCF 1934986, NIH IROIMH111226-01A, NIFA award 2020-67021-32799, the Alfred Sloan Foundation, and Google Inc.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 00289, 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 356-473,44 2018.
Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Gong. Fltrust: Byzantine-robus. federated learning via trust bootstrapping. In Proceedings of NDSS, 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In International Conference on Machine Learning, pp. 1089-20999 PMLR, 2021.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint MxXix:2003.13461 2020.
Jean Ogier Du Terrail,
Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for cross-silo federated learning realistic healthcare settings. In NewrIPS, Datasets and Benchmarks Track, 2022.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference Computer Vision, pp. 1657-1664, 2013.
Haozhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, and Wei Chen. Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3274-3283. PMLR, 18-24 Jul 2021.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2020a.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020b.
Chaoyang Hc, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssil: Tackling label deficiency in federated learning via personalized self-supervision. arXiv preprint arXiv:2 10.00400, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2024
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5132-5143. PMLR, 13-18 Jul 2020.
Taekyung Kim and Changick Kim. Attract, perturb, and explore: Learning feature alignment network for semi-supervised domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 591-607. Springer, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEFE international conference on computer vision, pp. 5542-5550, 2017.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In IEEE International Conference on Data Engineering, 2022.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6368. PMLR, 2021.
Xiaoxiao Li, Yufeng Gu, Nicha Dvornek, Lawrence H Staib, Pamela Ventola, and James S Duncan. Multi-site fmri analysis using privacy-preserving federated learning and domain adaptation: Abide results. Medical Image Analysis, 65:101,,5, 2020.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6028-6039. PMLR, 13-18 Jul 2020.
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In International Conference Machine Learning, pp. 15070-15092. PMLR, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Ciamuunication-Effiiieeee Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp. 1273-1282 PMLR. 20-22 Apr 2017.
Tuan Nguyen, Trung Le, He Zhao, Quan Hung Tran, Truyen Nguyen, and Dinh Phung. Most: Multi-source domain adaptation via optimal transport for student- learning. In Uncertainty in Artificial Intelligence, pp. 225-235 PMLR, 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406- 1415, 2019.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. In International Conference on Learning Representations, 2020.
Daniel Rothchild, Ashwinee Panda, Enayat Ullah. Nikita Ivkin, Ion Stoica. Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-cftiicn federated learning with sketching. In International Conference Machine Learning, pp. 8253-8265 PMLR, 2020.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3723-3732, 2018
11

---

Published as a conference paper at ICLR 2024
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. ICCV, 2019.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- erated learning with matched averaging. International Conference on Learning Representations, 2019.
Guile Wu and Shaogang Gong. Collaborative optimization and aggregation for decentralized domain generalization and adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6484 6493, October 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Zeno++: Robust fully asynchronous SGD. In Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10495-10503. PMLR, 13-18 Jul 2020a.
Ming Xie, Guodong Long, Tao Shen, Tianyi Zhou, Xianzhi Wang, Jing Jiang, and Chengqi Zhang. Multi-center federated learning, 2020b.
Dong Yang, Ziyue Xu, Wenqi Li, Andriy Myronenko, Holger R Roth, Stephanie Harmon, Sheng Xu, Baris Turkbey, Evrim Turkbey, Xiaosong Wang, al. Federated sunii-upereieed learning for covid region segmentation in chest ct using mulli-nntionna data from china, italy, japan. Medical image analysis, 70:101992, 2021.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation, Advances in neural information processing systems, 31, 2018.
12

---

Published as a conference paper at ICLR 2024
Appendix Contents
Section
A
Supplementary Theoretical Results
Proof of Theorem
Proof of Theorem
Proof of Theorem
A.4
Proof of Theorem
A.5
Additional Discussion the Auto-weighting Method
Section
B
Synthetic Data Experiments
Section
Supplementary Experiment Information
Algorithm Outlines for Federated Domain Adaptation
Real-World Experiment Implementation Details and Results
Additional Results on DomainBed Datasets
Auto- seit-Weighted Methods: Implementation Details, Time and
Space
Complexity
Visualization of Auto-Weighted Betas Values Real-World
Distribution Shifts
C.6
Additional Experiment Results on Varying Static Weights (3)
C.7
Semi-Synthetic Experiment Settings, Implementation, and Results
Additional Ablation Study Results
C.9
Implementation Details of FedGP
C.10
Gradient Projection Method's Time and Space Complexity
C.11
Additional Experiment Results on Fed-Heart
Comparison with the Semi-Supervisee Domain Adaptation
(SSDA) Method
A SUPPLEMENTARY THEORETICAL RESULTS
In this section, provide theoretical results that omitted in the main paper due space limitations. Specifically, in the first four subsections, we provide proofs of our theorems. Further, in subsection A.5 we present omitted discussion for auto-weighting method, including how the estimators are constructed.
A.1 PROOF OF THEOREM 3.3
We first prove the following theorem where we study what happens when do one step of optimiza- tion. While requiring minimal assumptions, this theorem shares the same intuition as Theorem 3.3 regarding the importance of the Delta error.
Theorem A.1. Consider model parameter 0 ~ and an aggregation rule Aggr(-) with step size μ >> Define the updated parameter as
0- 0 pragg=(0).
Assuming the gradient VE(0,2) 2) is Lipischhttz in @ for any 2, and let the step size S 1 7. we have
Es, lld,, (8+) - (0) < (1lor,, 2g
Proof. Given any distribution data D, first prove that Vln is also --Lipschitz below. For v0,, 0g E O:
([v6=(01) - (1)( = EELD VE(0,,
Ve(02, 2)]1
Ey~D Ve(01,2)
VE(02,2))
(Jensen's inequality) (VE(; 2) is >-Lischhitz)
< Ez~Dy 1 - 0.ll
0 0z
13

---

Published as a conference paper at ICLR 2024
Therefore, know that ED, is y-smooth. Conditioned on a 0 and a Dr, and apply the definition of smoothness we have
(D (0+) LD,, (0) S (0),0 - -0) + 10 011
-VV (0) 3r (0)) + 2 14nggg (0)1
2
-,((VED, (0), Trgge (0) V (0) + VeD, (0))
7H²²
+
2 [ VlD, (0) + Vl (0)1²
=-<( (u((Ved, (0), Juggr (0) VlD (8)) // (0)113)
7112
(110ggg (0) VLD, VED, (0)11 2ggggg Vo),
2
(1 - 7u ((VeD, (0), Vlz (0) Jaggr (0))
712
w)] 10) + 7422
+ 2
lTngor (0) - ViD, (0)lI
2
S (M 712)
IV (0)1 |1 (0) ((D,(())
(Cauchy-Schwarz inequuulll incquality)
712
(e) + 7/²2
18aggr (0) VED, (0)]I²
2
2
#1 71
(IVED, (0)11= |lāagrr (0) VlDr (0)
(AM-GM inequality)
2
212
74²²
+ 2 w)ll ve, (0) +
lmngge (0) - ((),(0)11
2
-- (I (0)11 11agg: VlD (0)1
(4)
Additionally, note that the above two inequalities stand because: the step size p S 1 y and thus 2/2 = -2²( 7) 2 0. Taking the expectation Dr on both sides gives
EB,,e o5,, lD (0+) (D. < / (BB, (IIV (0)1 Es, Illgngg (0) - VlDr (0)])
Note that we denote 9DT = V.D,,- Thus, with the 1 norm notation we have
E ED (LD, (0+) - lD, (0) (190r -E, liAgggr - 9D, 11?)
11
Finally, by Definition 3.2
A E.,, llāggg= 9D, which concludes the proof.
Theorem A.2 (Theorem 3.3 Restated). For any probability measure over the parameter space, and an aggregation rule Aggz(-) with step size μ > Given target dataset Dr, update the parameter for T steps as
g++1 = 0¹ -I@ggge
Assume the gradient Ve(0, 2) and @auge (0) is 3/2 --iiscchitz in such that 0' + @Aggg- Then, given step size μ S and a small enough e > 0, with probability at least 8 we have
IIVE, (0T)
1 (Vc. -Gggr + +ole) + +0 8z
+ O(E))
where C. ED, (B. and (. C 1m is the ball with radius centered at
① Aaggr- The C. measures how well the probability measure т covers where the optimization goes, i.e.,
i Aaggr
Proof. We prove this theorem by starting from a seemingly mysterious place. However, its meaning assured to be clear proceed.
14

---

Published as a conference paper at ICLR 2024
Denote random function f: R RR
f(0) 11āager (0) VEDr (0)1l,
(5)
where the randomness comes from Dr. Note that is y-L_ipschitz by assumption. Now we consider Be (fkgg)) C I"", i.e., the ball with radius e centered at 0aggz. Then, by 7-Lipschitzness we have
/ (())d(())
B f(0)
dr((0)
z f (f(Oager) 7e) dT(
((
(F(O2ogz) - (B))
Therefore,
[(
Ea [(0)
T. (Be (Bagg))))
Taking expectation w.r.t. Dr on both sides, we have EB 5(0guaz))) s Es,
Ba B(0) +(E))
π (eBe (Be(0gge))
2
BBr
1
E [(]]] O(c)
(Cauchy-Sehwarzz)
TT TBB.(Oagar)
C. Es,
EAnx i(0)
+(e))
(by definition of C.))
C. EB, Bows [(0]] 2 + +(c)
(Jensen's 's inequality)
C. -2.ggr + O(e)
Therefore, by Markov's inequality, with probability at least - we have a sampled dataset Dr such that
1 ((angrr) S S F S VVC. Agggr (e//8)
1
(6)
Conditioned on such event, we proceed to the optimization part.
Note that Theorem A. characterizes how the optimization works for one gradient update. Therefore, for any time step O,.... T 1, we can apply (4) which only requires the Lipschitz assumption:
lD. (0(+1) ED (0) S (I (lV (12) l1āaggr (8') (lDx(")))3)
On both sides, summing over tvert= = T gives
T-1 Σ
(xr (0, 1
Σ |V (05)
@g (00) VlD. (8))1)
1=0
Dividing both sides by T, and with regular algebraic manipulation we derive
T
/e (0)) 2 (Cor - Co (0T)) +
Σ
Iāgge (0') - VlD, (0))
t=
==
Note that we assume the loss function (: 2 + of Section Thus, we have
R4 is non-negative (described at the beginning
(00) 1
(0))1
Ggagg (f') VlD, 11
T
pTT
T
15

---

Published as a conference paper at ICLR 2024
Note that assume given Dr we have 0f Baggz- Therefore, for any 0 there exist T. such that
Vt > T. 110' Dagez <e.
(8)
This implies that 14 Te:
1 (Ilūngg=((''1 ||0 0 0ugg= + Oagg= - 1f1 - 11 -Oaggell + 110 80 2c.
(9)
Moreover,
{ also implies Vt1, t2 > Te:
VlDD((0))) -Ven, (0%) || - r/10 1
(7-Lipschitzness) (10)
2e.
The above inequality means that {VLD, (0')), is a Cauchy sequence. Now, let's get back to 77.. For vT > T. we have
T
20
T.
T
VlD, (0')12
@gnge (0') - VED, (0')11
Baggr (0') - VeD, (8'))
T
T
T
t=
T-1
=
1
+ T
Σ
llāgger (0') (l')(1'))
t=
T-1
+ T
Σ
llāaggr (0') Ingge (Ongg=) + Jaggr (Paggr) - (
1
Σ
+
(1āaggr (0') Tnggr (Onggz )1 + 1gāgge (Oaggz) - VED,
(triangle inequality)
=
+ T
Σ
(O(E) + 10āoog (Oaggz) VED,
(by
1
+ 0()) +
[Iarqu (Daggz) VeDr (Oaggz) + VlDr (Oagg:) vlDr(
T
T
<
1
+ O()) + T
Σ
(1@nggr (Oaggr) - VlDr (e (Bagg=) + O(e)) 2
T
(by (10)
= = 1
((())
+ O(E) + [[ggg-- (Onggr) VlD (Oaggr))
(11)
Then, continue with what have done the beginning of the proof of this theorem:
1
T
+ +O(E) + f(Ongg=))
(by (5)
so +
+ + C.. A + 0(e/0)
2
(by (6)
Therefore, combining the above we finally have: for VT T. with probability at least 1 1 8,
Vel, (0))
O())+ 8² (VC+ (CCe-ARog
Abgge + 0(s))
(12)
T 10
16

---

Published as a conference paper at ICLR 2024
To complete the proof, let us investigate the left hand side.
T-1
T--1
T VVD,
(0))
Σ
VlD, (0')l Σ
+ T T=
T 1=
= =o (7)
|V (0')
T
1
20
Σ
(IIVE,, VeD, - (f=)
T
T
(triangle inequality)
1
T-1
0
Σ
(0(c) + [V
(by 10
T
T
T=I
0
(E)) + Vl (1
(13)
Combining and 13 we finally have
IV (0T) 1 + (ee) + 5z (vC. CV.ggr
A2ger + 420ggr+((0)) +0(e)²
which completes the proof.
A.2 PROOF OF THBOREM 3.6
Theorem A.3 (Theorem 3.6 Restated). Consider any aggregation rule Aggr(-) in the form of
Jagur = N Eig(n]] F I :Ds, i.e., the aggregation rule is defined by a mapping Fnger: L x L + L" 1f Faggz is affine w.r.t to its first argument (i.e., the target gradient function), and Vgg E L Fagg=19.90 = 9. and the linear mapping associated with Faggz has its eigenvalue bounded [Amin; \mar), then for any source and target distributions (Ds,)eelN) Dr, Dr we have A Aggrr I N EiENN] A A Aggn.Ds, where
(2) + max ((1 Amarr) ) )3 Amin) cd Dr)
A2 As
max moar X A
n
Proof. Let's begin from the definition of the Delta error.
A Apor = E,, 11gD, Iager11? 11
2
E3,
gDT N
Σ
Fagg= 10, 9Ds,
VE[N]
B
JDr
Fagg= 10 9Ds,
N
iE[N]
< sE[N]
E ES
10,, |DDr F 9Ds,
(Jensen's 's Inequality)
N
1
2 Agg
(14)
N
SE[N]
where we denote
A
Aggr, -Ez |pr Pager Paggrl98,, 9Ds,
17

---

Published as a conference paper at ICLR 2024
We continue upper bounding this term. Noting that Figgzlt.-] is affine in its first argument, can denote
9: Ep, Paggr 196, 9Ds, Pager [B |0 DDs, Fager (yDr: DDs,
(15)
Therefore,
||
A Aggr.Ds
=B 9Dg - g Fagg: ID, 9Ds,
110D, -lll + E,,
0 Fagg: I0. 9Ds,
2E Br (0Dr 9.9 Fager 19., 9Ds, 1
ll gll Dr
g Fagg= I95 9Ds.
2(grr -9.9 -,,, Fnggr 19. 9Ds, 1)
lllrr Øll Pr
Fagg-
D Pngg-190rr.
JDs,
2 - 0.9 g)=
(by 15)
1lp,, g11 + + Br
0 Fagg= 9Ds,
(16)
The above derivation is based on properties of inner product. Next, we deal with the two terms in (16). Denote G: L L the linear mapping associated with the affine mapping !'p,.l
By definition, V91.92 E L*:
Faggr [Ds,] Faggr [92, SDs,
Glgi] - G[ga] G[g - -92]-
(17)
For the first term in 16 we can do similar trick as the following.
1gpr - gl =
|gDr 9Ds. + gDs. - gll
1|DD 9Ds. + 9Ds, Faggr lgD, IDs, I1
(by (15)
1900r 9Ds, Pager Lyps, 9Ds,
Pagg- [gDr DDs, 11
(18)
1|9DT 9Ds, Glggs, -1
(by 11
[|( G)L9Ds, _9Dr
(I stands for identity map)
S max((11 Amarx )2, 1,,(1 Amin) ) Ilgss -DD l1
max((1 Amaz) )2, Amin)² }d,, (Ds. Dr)
where (18) is by the identity assumption of Preger This assumption is valid in then sense that: if all of the inputs an federated aggregation rule are the same thing, the aggregation rule should output the same thing.
This upper bounds the first term in [66,, and lets move on the second
T6 as the following.
E Er g
Fagge 195, IDs, = Es, Fages [oDr, 9Ds,
Faggs 196, 9Ds
(by 15
Es, Glgpr - -95/11²
(by (17
S max 2 PE
9DD - g l
max()2 to
(by Definition 3.5
Combining the above, we finally have
A Aggr.Ds.
16 < max 2 mexx 22 )o max((1 Amaa) )2. Amin 2 (Ds, D
Putting 14 and the above inequality together, and noting that o o2( x2(2) proof is complete.
we can see that the
18

---

Published as a conference paper at ICLR 2024
A.3 PROOF OF THEOREM 4.2
Next, we prove the following theorem.
Theorem A.4 (Theorem 4.2 Restated), Consider FedDA, Given the target domain Dr and N source domains D .www
N
1
2 PednA
A Sad where = -(-
N
the Delta error when only considering Ds; as the source domain.
Proof. Recall
N (1-3)))5. SigDs.)
Σ N
((L
IF FFODDA
Thus,
A FedDA
EB, lØDr
TredDall -B, 190r
Σ N
1
(1 B:)gs, + BLgDD.) 112
N Σ
=B
(DDr -(1-3i)933 + +.igDs.)
1
N
i=1
N
E D |or, - (1 - (1)9,, + B.gDss 11
(19)
where the inequality is derived by Jensen's inequality. Next, prove for each i [N]:
E 190, (1 - 1- + R.gDs, 1 E., [1 - (.)( -5,) + 3:(0pr 9Ds, Il
²,)) 11 - 96, 11 + 5. 1gvr -9Ds, 1l
+ + - -1,1,,8,8,,,p (gDr 9B, 9Dr -9Ds )m
(1 (D) (Ds, Dr) + +(11 (,)) 1oor - 18,1.00r -9Ds)
(1
DT)
2 2 FadDA.S.
Plugging the above equation into
19 gives the theorem.
A.4 PROOF OF THEOREM 4.4
Theorem A.5 (Theorem 4.4 Rigorously). Consider FedGF. Given the target domain Di and N
source domains Ds...... Dsy
N
2 Fedip
N
where
A Sedce,
= -1- + (,(0) (0) 9Ds, (0)]l²] 8,
(0), uDs (0)
23, 10,
(0)) (95, (0) gDr (0), uDs, (0))
+ -1-
In the above equation, 8,(0) = 1(g (0),90)) IDS, (0)) > 0) is the indicator function and it is 1 if
the condition is satisfied. T; (0) : yy sin (())
where pc(0) is the angle between gDs, (0) and
Tops, (0) o. [
gDr (f). Moreover, UDs (0) gDs, (0) /lg0 0)|
19

---

Published as a conference paper at ICLR 2024
Proof. Recall that
N
N
FFedGP (0)
= N (1 (1133))03 (0) + 3,Proi (05, (0)lDss, (0))
Orodse Ds, (0),
N
1=1
where we denote
geddG De. (0) (11 (1)),, (0) + (1r00 (0))
Then, we can derive:
A Deaee = EB 13511
TFeOGP 11 - BB o olly,, (0) - Freac (0)1
N
= Fr (0)
Fredor Ds. (0)12
N
= -Ep
(0Dz (0) Freder. De, (0))1
N
D Drr oll.plll00000 (0) Trease, Ds, (0)]
(20)
where the inequality derived by Jensen's inequality.
Next, we show EB,, a [191, (0) Dreder, (0)ll² A S, First, we simplify the notation.
Recall the definition of Gzedce, Ds, is that
Greaer Ds. (0) (1 B,)g,, (0) + 9, m (0). ),0gps, (0) 190s, (0) (1 (1-3))93) B.)D,, (0) +
(0), UDs (0) uD, (0).
Let us fix 0 and then further simplify the notation by denoting
= 9D, (0)
= 9D, (0)
UDo, (0)
= (0)
Therefore, have Grease Ds, (0) = (1 8 + (0,
Therefore, with the simplified notation,
EB, 119 (0) - Trodsr, (0) = F, |1(1 -,)0++ + B:8 u/u o|12 Es,
w/12
(1 -i w)ll
SR u, 10 u/u v) + (1 (1- (0)::)(--- w)l1
1B,00 u, (
u/u u1|21 + (1 -3,) -,) 11| 2|
+ + - 1,(0,,,6(0- 23, - 1000111-EE,,,, 18( v. u) (u, i - wi}
2,37118(11 u,
(expanding the squared norm)
+(1 - 2 Il| = (23; BB) 10(0- v)] + 33 BB, 1110 w/u +
u
+ 28, - B,) b(v,, w)(u, 0- w)) 1 283 BTō(1 - 5) - u, u)) (v, 4)}
where the last equality is by merging the similar terms. Next, we deal with the terms A and B.
Let us start from the term B. Noting that is either 0 or 1, we can see that 8(1 - 8) = 0. Therefore,
B 0.
20

---

Published as a conference paper at ICLR 2024
As for the term A, noting that 8 8, expanding the squared term we have:
A = 15 [1l u/u- 21(²] = 15, (1,, u/u - u) - (1 83011
EB, (v. u/u w1 + ED (1)) -| 12E,,,188 18
() u)u v, e\
_, (v, u)u 21l3| + B [11 [1--))11111 2)
Therefore, combining the above, we have
p, -1p (0) Bredce,Ds, (0)1 (1 -B,) 11| u1l3] + S
[0 u.) 0113]
+ +233 - E,) 18(0 v, u}
+ 23:(1 - 3.) b,, 1ō(u, u)(u, i - w\} + E 111 - 8)
Let us give an alternative form for [u, u)u - vl12 would become the domain-shift. Note that [1 u)u u - el| is the distance between o and its projection to u. We can see that [1 u/u -l = ||u vll Ik(e,a)a-oll in, have
aim to connect this term to ||u wll which
||u (0) Therefore, plugging this
B,, 119 (0) TrodGP.Do Dz, (0)1² = (1 -1- B, [|| u1 + S.E 1ô1lu 01 (0)
+ (28 E, 18 -0,-vuu 10,00,0,0, u
+ 2 - B,) [6uu, u)(u, d wi}l + S.E [(1 5)]1 u/|
Writing the abbreviations u. é, into their original forms and taking expectation over 0 on the both side we can derive:
Es, , 1l.., (0) Trease. Ds. (0)] = = (1- + , L 10, (0) (0))),)))())))))_))))))) (0) 9Ds, (0)
+ (2,3, - 3, (0) gD, (0), WDs; (0))
+ 213 - -3,)
(0)) (05, (0) gDD (0), ups, (0))
+ g, (11 (0))
2 SS.
Combining the above equation with 20 concludes the proof.
Approximations. As we can see, the Delta error of FedGF rather complicated at its precise form. However, reasonable approximation can be done to extract the useful components from it which would help us to derive its weighted version. the following, we show how we derive the approximated Delta error for FedGP, leading to what we present in (
First, we consider an approximation which is analogous to a mean- mean-field approximation, i.e., ignoring the cross-terms in the expectation of a product. Fixing &,(0) = 5 and (0) = F, i.e., their expectations. This approximation is equivalent to assuming 8(0), ((0) can be viewed as independent random variables. This results in the following.
2 Fedge, Ds, ~ (1 - o.) + 33,87 dz (Ds,, Dr))
+ (23, - 5E
g. D. (9D (8) - 9D, uDs, (0)) + 8² 8)1l,,, 11:
The term E,, W0, (0) gDD (0), uDs, (0)) 2 is the variance of 95, when projected to direction UDs, (0).
We consider further approximation based on the following intuition: consider a zero-mean random
vector E Rm with i.idd entries ún for E [m]. After projecting the random vector to a fixed m unit vector R the projected variance is B(ū, n)² E{E Σn= B(
21

---

Published as a conference paper at ICLR 2024
Σ d0 u (2( i.e., the variance becomes much smaller. Therefore, knowing that the m parameter space θ is m--dimensional, combined with the approximation that Iö, (0) - 9Dz (0) is element-wise i.i.d., we derive a simpler (approximate) result.
Fedee Ds. = (1 - 3.) (28, 85
(Dr) + dz
Dr)² + 8 1 8)
m
In fact, this approximated form can already be used for deriving the auto-weighted version for FedGP, as it is quadratic in 8 and all of the terms can be estimated, However, we find that in practice = 1, and thus simply setting = 1 makes little impact on A Fease,Dss' Therefore, for simplicity, we choose ă = 1 as approximation, which results in
(11 (1 (1)3
(23: (29:-9)) 8
+((T))+B77 Dr)2
Fedce,Dss Ds.
m
This gives the result shown in
Although many approximations are made, we observe in our experiments that the auto-weighted scheme derived upon this is good enough improve FedGP.
A.5 ADDITIONAL DISCUSSION OF THE AUTO- WEIGHTIIG METHOD AND FEDGP
In this sub-section, show how estimate the optimal 8 for both FedDA and FedGP. Moreover, we discuss the intuition behind why FedGP with a fixed 3 = 0.5 is fairly good in many cases.
In order to compute the 3 for each methods, shown in Section 4.2 we need to estimate the following three quantities: a (Dr),, da (Ds,. Dr), and d2 Dr). In the following, we derive unbiased estimators for each of the quantities, followed by a discussion of the choice of π
The essential technique is to divide the target domain dataset Dr into many pieces, serving as samples. Concretely, say we randomly divide Dr into B parts of equal size, denoting Dr Uf=z1 Di (without
loss of generality we may assume |Drl can be divided by B). This means
B
95, B
1
Σ 96
(21)
Note that each 9Bz is sample of dataset formed by |Drl/D data points. We denote D n as the corresponding random variable (i.e., a dataset of [Drl/B B sample points sampled i..dd from Dr). Since we assume each data points in Dr is sampled i.i.d. from Dr, we have
E B = E[gs;] = =E = 9Dr
(22)
Therefore, we may view Dj as i.i.d. samples of DT. B, which we can used for our estimation.
Estimator of (Dr).
First, for (Dr), derive that
0 (DT) = Elggrr 95, 1 -Elggrr B
9
l1
j=l
E
(gDr 95+ 11
Ba
1
Bl|gD, 98.
(by (2
Ba j=1
(DT,B)
(by 22
B
22

---

Published as a conference paper at ICLR 2024
Since we have B i..d. samples of Dr.B, use their sample variance, denoted as 0 (Dr, as an unbiased estimator for its variance o (Dr,B). Concretely, the sample variance is
B
1
(DT.B)
Σ j=1
199;
-5,11
B
Note that, as shown in 2 9B, is the sample mean. It is a known statistical fact that sample variance is an unbiased estimator of the variance, i.c.,
(Drr)))
Combining the above, our estimator for (f is
B 9
1 (Dr.B) (Dr) B n
96, 11
(B - 1]B
Estimator of (Ds. Dr).
For (Ds,, Dr), we adopt similar approach. We first apply the following trick:
de (Ds,, Dr) = 1lpps - 9Dr a = 119Ds, 9Dr l + Bligr, 9Dr, l1 - Ellgrr - 9D, 11
Elgg - yDr + 9Dr - 9 1 Bllgor - 9 l1
(23)
Elg,, - 95r 11 - (Drr))
(24)
where is due to that Elgo, I -DD and therefore the inner product term
(oDD, 9D,, 9Dr 95r mm] (gDs, -Dr, BloDr 95 (5r,,l)= 0.
This means
E Bligos !Dr + DDr - 9 11 llyps ppr l1 + Blyypr PDr.al12
We can see that
24 has an unbiased estimator the following
B
(r).
19ps,
95; 11 - (Dr.D)
B
Estimator of (Ds,
Finally, left to find an estimator for (D Dr). Note that we estimate 72 DT)
directly, but not 72 and d (Ds, (D) separately. is because our aim in finding an unbiased estimator.
Concretely, from Theorem 4.4 we can see its original form (with 8; = 1) is
Eg ( (7) (0) gDs. [1)
Es 19p (0) (gD, (0), UDS. (0)) (0) (25)
where WDs, (0) = DDs, (0)/ll9ps, (0)ll-
Denote grs.9rs Frs θ as the following:
grs(O) gD, (0) (gD, (f), uDs, (0) (O)) (0)
Ørs (0) 9D, (0) (gp (0) DDs (0) (D) (0)
(1) 9D (0) (o (0), uDe uDs (0).
Therefore, we have the following two equations:
25 |!9Ts|12 1
E [rrs] ElA/s] grs-
23

---

Published as a conference paper at ICLR 2024
This means use samples Dj- to compute samples of Trs- and then estimate 25 ingly.
accord-
Applying the same trick before, have
25
119rs I| = slrsll + Ellrr - 91 El ors l
E + DTS 99s 1 Ellors gTs
s Elirs grsl
Note that ll can be estimated unbiasedly by B Σラ=1 10rll Moreover, El - grsll is the variance of frs and thus can be estimated unbiasedly by the sample variance.
Putting all together, the estimator of 72d (Ds., Dr) (rigorously speaking, the unbiased estimator of is
B
B Σ stsl
1
Trslll j=1
ITS
B
B
B
1=1
k=1
Therefore, we have the unbiased estimators of o² (Dr). d2 (Ds,, Dr). and FDs, Dr). Then, we can computed estimated optimal (8F #eeDA and geede according to section 4.2
The choice of IT.
The distribution ı characterizes where in the parameter space @ we want to measure of (Drr. de (Ds.. Dr), and the Delta errors.
In practice, we have different ways to choose т. For example, in our synthetic experiment, we simply choose to the point mass of the initialization model parameter. It turns out the Delta errors computed at initialization are pretty accurate in predicting the final test results, as shown in Figure 5 For the more realistic cases, we choose ı to be the empirical distribution of parameters along optimization path. This means that we can simply take the local updates, computed by batches of data, Dj_ and estimate accordingly. Detailed implementation shown in Section C.4
Intuition of why FedGP is more robust to the choice of (3.
To have intuition about why FedGP is more robust the choice of B compared FedDA, examine how varying 3 affects their Delta errors.
Recall that
2 Sd = (1 -11 (D))
Dr)
2 adde S. 23 - 3,) + 8 Dr).
Now suppose we change Bg + B/. and since 0 < 72 < 1, we can see that the change in A Aeda,, is less than the change A? In other words, FedGP should be more robust varying 3 than FedDA.
B SYNTHETIC DATA EXPERIMENTS IN DETAILS
The synthetic data experiment aims to bridge the gap between theory and practice by verifying our theoretical insights. Specifically, we generate various source and target datasets and compute the corresponding source-target domain distance d= Dr) and target domain variance o, (Dr)) We aim to verify if our theory is predictive of practice.
this experiment, we use one-hidden-layer neural networks with sigmoid activation. We generate datasets DI.... ...... Dg each consisting of 5000 data points as the following. We first generate 5000 samples x E 50 from a mixture of 10 Gaussians. The ground truth target set to be the sum of 100 radial basis functions, the target has 10 samples. We control the randomness and deviation of the basis function to generate datasets with domain shift. As result, D2 to D, have an increasing domain shift compared D,. We take Dj as the target domain. We subsample (uniformly) 9 datasets
24

---

Published as a conference paper at ICLR 2024
Method with the Smallest Delta Error
Method with the Best Test Result
Method with the Best Test Result
Target Domain Variance
0.08
FodCP
TedDA
Target Deraiin Veriance
0.08
TedDA
FodGP
Target Dornain Veriance
0.08
TedDA (aciol
FodOP (auco)
0.06
Target Only
Source Only
0.06
Source Only
Targel Only
0.06
Source Only
Jerget Only
0.04
0.04
0.02
0.02
0.00
0.00
0.00
0.00
0.02
0.04 0.06 0,08
0.00
0.02
0,04 0.06 0.08
0.00
0.02 0.04 0.06 0.08
Source-Target Domain Distance
Source Tariee Domain Distance
Source-Targel Domain Distance
(a)
(b)
(c)
Figure 5: Given specific source-target domain distance and target domain variance: (a) shows which aggregation method has the smallest Delta error; (b)&(c) present which aggregation method actually achieves the best test result. In (b), FedDA and FedGP use fixed 0.5. In (c), FedDA and FedGP adopt the auto-weighted scheme. Observations: Comparing (a) and (b), we can see that the prediction from the Delta errors, computed at initialization, mostly track the actual test performance after training. Comparing (b) and (c), we can see that FedDA is greatly improved with the auto-weighte weighted scheme. Moreover, we can see that FedGP with a fixed 3 = 0.5 is good enough for most of the cases. These observations demonstrate the practical utility of our theoretical framework.
DI..... Dj from D1 with decreasing number subsamples. As a result, DJ has the smallest target domain variance, and Di has the largest.
Dataset. Denote radial basis function (x)x) e pl1 and we set the target ground truth be the sum M 100 basis functions as y = E 6i, where each entry of the parameters are sampled once from (-00,5, 0.5). We set the dimension of æ to be 50, and the dimension of the target to be 10. We generate N = 5000 samples of æ from Gaussian mixture formed by 10 Gaussians with different centers but the same covariance matrix Σ = I. The centers are sampled randomly from 0-0.5, 0.5) We use the ground truth target function y(-) to derive the corresponding data y for each æ. That is, we want our neural networks to approximate y(-) on the Gaussian mixture.
Methods. For each pair of (D,, DJ) where i.j 1,...,.... from the 81 pairs of datasets. We compute the source-target domain distance dD; D') and target domain variance (D() with æ being the point mass on only the initialization parameter. We then train the 2-layer neural network with different aggregation strategies (D,, Df). Given the pair of datasets, identify which strategies have the smallest Delta error and the best test performance on the target domain. We report the average results of three random trials.
As interpreted in Figure 5 the results verify that (i) the Delta error indicates the actual test result, and (ii) the auto weighte strategy, which minimizes the estimated Delta error, is effective in practice.
C SUPPLEMENTARY EXPERIMENT INFORMATION
In this section, we provide the algorithms, computational and communication cost analysis, additional experiment details, and additional results semi-synthetic and real-world datasets.
C.I DETAILED ALGORITHM OUTLINES FOR FEDERATED DOMAIN ADAPTATION
As illustrated in Algorithm for one communication round, each source client Cs, performs supervised training on its data distribution Ds, and uploads the weights to the server. Then the server computes and shuffles the source gradients, sending them to the target client Cr. On Cy. it updates its parameter using the available target data. After that, the target client updates the global model using aggregation rules (c.g. FedDA, PedGP, and their auto-weighted versions) and sends the model to the server. The server then broadcasts the weight to all source clients, which completes round.
25

---

Published as a conference paper at ICLR 2024
C.2 WORLD EXPERIMENT IMPLEMENTATION DETAILS AND RESULTS
Implementation details We conduct experiments on three datasets: Colored-MNIST (Arjovsky al., 2019) (a dataset derived from MNIST (Deng 2012) but with spurious features of colors), VLCS (Fang et al., 2013) (four datasets with five categories of bird, car, chair, dog, and person), and Terralncognita (Beery et al.) 2018) (consists of 57,868 images across 20 locations, each labeled with one of 15 classes) datasets. The source learning rate is le-3 for Colored-MNIST and 5e-5 for VLCS and Terralncognita datasets. The target learning rate is set to of the source learning rate. For source domains, we split the training/testing data with 20% and 80% split. For the target domain, we use a fraction (0.1%) % for Colored-MNIST, 5% for VLCS, 5% for Terralncognita) of the 80% split of training data to compute the target gradient. We report the average of the last 5 epochs of the target accuracy on the test split of the data across trials. For Colored-MNIST, we use a CNN model with four convolutional and batch-norm layers. For the other two datasets, we use pre-trained ResNet-18 (He et al. 2016) models for training. Apart from that, we use the cross-entropy loss as the criterion and apply the Adam (Kingma & Ba, 2014) optimizer. For initialization, we train 2 epochs for the ColSTed-MNIST, VLCS datasets, as well as 5 epochs for the TerraIncognita dataset. We run the experiment with random seeds and report the average accuracies over five trials with variances. Also, we set the total round R = 50 and the local update epoch to 1.
Personalized FL benchmark We adapt the code from Marfoq et al. 2022 to test the performances of different personalization baselines. We report the personalization performance on the target domain with limited data. We train for 150 epochs for each personalization method with the same learning rate as our proposed methods. For APFL, the mixing parameter is set to 0.5. Ditto's penalization parameter is set to 1. For knnper, the number of neighbors is set to 10. We reported the highest accuracy across grids of weights and capacities by evaluating pre-tra pre-trained FedAvg for knnper. Besides, we found out the full training of DomainNet is too coneccong for personalized baselines.
Full results of all real-world datasets with error bars As shown in Table 2 Colored-MNISTT Table (VLCS), and Table 4 (Terralncognita), auto-weighted methods mostly achieve the best performance compared with personalized benchmarks and other FDA baselines, across various target domains and on average. Also, FedGP with a fixed weight 3 0.5 has a comparable performance compared with the auto-weighted versions. FedDA with auto weights greatly improves the performance of the fixed-weight version.
Colored-MNIST
Domains
+90%
+80%
-90%
Avg
Source Only
56.82 (0.80)
62.37 (1.75)
27.77 (0.82)
48.99
Finetune OiftineeOfttlnee
66.58 (4.93)
69.09 (2.62)
53.86 (6.89)
63.18
FedDA
60.49 (2.54)
65.07 (1.26)
33.04 (3.15)
52.87
FedGF
83.68 (9,94)
74.41 (4.40)
89.76 (0.48)
82.42
FedDA__Arr Auto
85.29 (5.71)
73.13 (7.32)
88.83(1.06)
82.72
PedGP Leuo Auto
86.18 (4.54)
76.49 (7.29)
89.62(0.43)
84.10
Target Only
85.60 (4.78)
73.54 (2.98)
87.05 (3.41)
82,06
Oracle
89.94 (0.38)
80.32 (0.44)
89.99 (0.54)
86.75
Table 2: Target domain test accuracy (%) on Colored-MNIST with varying target domains.
C.3 ADDITIONAL RESULTS ON DOMAINBED DATASETS
In this sub-section, we show (1) the performances of our methods compared with the SOTA unsuper- vised FDA (UFDA) methods DomainNet and (2) additional results on PACS (Li et al. 2017) and Office-Home (Venkateswara et al. 2017) datasets.
Implementation We randomly sampled 15% target domain samples of PACS, Office-Home, and DomainNet for our methods, while FADA and KD3A use 100% unlabeled data the target domain. To have a fair comparison, we use the same model architecture ResNet-50 for all methods across three datasets. Additionally, for NetainNet, we train FedDA, FedGP and auto-weighted methods
26

---

Published as a conference paper at ICLR 2024
VLCS
Domains
C
L
V
S
Avg
Source Only
90.49 (5.34)
60.65 (1.83)
70.24 (1.97)
(1.99)
72.62
Finetune_Ofie Offline
96.65 (3.68)
68.22 (2.26)
74.34 (0.83)
74.66 (2.58)
78.47
FedDA
97.72 (0.46)
08.17.(1.4))
75.27 (1.45)
76.68 (0.91)
79.46
FedGP
99.43 (0.30)
71.09 (1.24)
73.65 (3.10)
78.70 (1.31)
80.72
FecDA_ Auto
99.78 (0.19)
73.08 (1.31)
78.41 (1.51)
83.73 (2.27)
83.75
FedGP _uto
99,93 (0.16)
73.22 (1.81)
78.62 (1.54)
83,47 (2.46)
83.81
Target Only
97.77 (1.45)
68.88 (1.86)
72.29 (1.73)
76.00 (1.89)
78.74
Oracle
100.00 (0.00)
72.72 (2.53)
78.65 (1.38)
82.71 (1.07)
83.52
Table 3: Target domain test accuracy (%) on VLCS with varying target domains.
Terra Incogiiaa
L100
L38
L43
L46
Avg
Source Only
54.62 (4.45)
31.39 (3.13)
36.85 (2.80)
27.15 (1.21)
37.50
Finetune Offline
77.45 (3.88)
75.22 (5.46)
61.16 (4.07)
58.89 (7.95)
68.18
FedDA
77.24 (2.22)
69.21 (1.83)
58.55 (3.37)
53.78 (1.74)
64.70
FedGP
81.46 (1.28)
77.75 (1.55)
64.18 (2.86)
61.56 (1.94)
71.24
FedIDA_Auto
82.35 (1.91)
80.77 (1.43)
68.42 (2.66)
66.87 (2.03)
74.60
FedGe _uto
81.85 (2.30)
80,50 (1.60)
68.43 (2.08)
66.64 (1.52)
74.36
Target Only
78.85 (1.86)
74.25 (2.52)
58.96 (3,50)
56.90 (3.09)
67.24
FodAvg
38,46
21.24
39.76
20.54
30.00
Ditto (L ct al.
2021
44,30
12.57
40.16
17.98
28.75
FedRep (Colins ct a.
2021
45.27
6.15
21.13
10.89
20.86
APFL (Deng et al. 2020
65.16
64.42
38.97
42.33
$2.72
KNN-per (Marlog
2022
42.20
39.86
50.00
40.21
43.07
Oracle
96.41
(0.18)
95.01 (0.28)
91.98 (1.17)
89.04 (0.93)
93.11
Table 4: Target domain test accuracy (%) on Terralncognita with varying target domains.
with 20 global epochs and local epochs; for the other two datasets, we train our methods for 50 epochs. For the auto-weighted methods, notice that it converges faster than their fixed-weight counterpart. Therefore, we set the learning rate ratio to be 0.25 to prevent overfitting for DomainNet and PACS. For Office-Home, we set the rate ratio be 0.1 on A domain and 0.5 on other domains. For DomainNet, the target batch size is set to 64 for auto-weighted methods and 16 for FedDA and FedGP; the source batch size is set to 128. For PACS, the target batch size for auig-weighted methods and 16 for FedDA and FedGE; the source batch size is 32. For Office- batch size is 64 for autg-weighted methods and 16 F FedDA and FedGP: the source batch size 32. Also, we initialize 2 epochs for DemainNet, PACS, and domain A of Office- for other domains Office-Homee we perform the initialization for 5 epochs. Also, we use a learning rate of 5e for source clients. The learning rate of the target client is set to of the source learning rate.
the target
Comparison with UFDA methods We note that our methods work differently than UFDA methods. Le.. we consider the case where the target client possesses only limited labeled data, while the UFDA case assumes the existence of abundant unlabeled data on the target domain. As shown in Table 5 FedGP outperforms UFDA baselines using ResNet-101 in most domains and on average with significant margin. Especially for quick domain, when the source-targed tarree domain shift is large (Source Only has much lower performance compared with Oracle), our methods significantly improve the accuracy, which suggests our methods when the shift is large. However, we observe that the estimated betas from the auto-weighted scheme seem to become less accurate and more expensive compute with larger dataset, which potentially leads to its slightly worsened performance compared with FedGP. In future work, we will keep exploring how to speed up and better estimate the bias and variances terms for larger datasets.
achieve better bias-variance trade-off under the condition
compare methods with the SOTA Domain Generalization (DG) methods on the PACS dataset. The results show that our FedDA, FedGE and their nuig-weighted versions able to outperform the DG method with significant margins.
PACS results
As shown in Table 6
27

---

Published as a conference paper at ICLR 2024
Domains
clip
info
paint
quick
real
sketch
Avg
Source Only
52.1
23.1
47.7
13.3
60.7
46.5
40.6
FADA (100%)
59.1
21.7
47.9
8.8
60.8
50.4
41.5
KD3A (100%) (ResNct-50)
63.7
15,4
53.5
11,5
65.4
53.4
43.8
FedDA (ResNet-50)
67.1
26.7
56.1
33.8
67.1
55.7
51.1
FedGP (ResNet-50)
64.0
26.6
56.8
51.1
71.3
52.3
53.7
FedDA_aus auto (ResNet-50)
62.0
27.8
56.7
50.9
68.1
53.3
53.1
FedGP auto (ResNet-50)
62.2
27.7
56.8
50.7
68.4
53.5
53.2
Best DG
59.2
19.9
47.4
14.0
59.8
50.4
41.8
Oracle
69.3
34.5
66.3
66.8
80.1
60.7
63.0
Table 5: Target domain test accuracy (%) on DomainNet. FedGP and auto-weighted methods generally outperform UFDA methods with significant margins by using 15% of the data.
Domains
A
P
Avg
FedDA
92.6
89.1
97.4
89.2
92.0
FedGP
94.4
92.2
97.6
88.9
93.3
FedDA_ Auto
94.2
90.9
96.6
89.6
92.8
FedGP_ Auto
94.2
93.7
97.3
88.3
93.4
Best DG
87.8
81.8
97.4
82.1
87.2
Table 6: Target domain test accuracy (%) on PACS by using 15% data samples. We see FedGP and auto-weighted methods outperformed DG methods with significant margins.
Office-Home results As shown Table 7 we compare our methods with the SOTA DG methods on the Office-Home dataset. The results show that FedDA, FedGF and their auto- outo-weighted versions are able to outperform the DG method with significant margins. We found that the auto-weighting versions of FedDA and FedG? generally outperform their fixed-weight counterparts. We notice that FedDA where the fixed weight choice of B = 0.5 is surprisingly good on Office-Home. We observe that on Office-Homee the source-only baseline sometimes surpasses the Oracle performance on the target domain. Thus, we conjecture that the fixed weight / = 0.5 happens to be good choice for FedDA on Office-Home, while the noisy target domain data interferes with the weighting mechanism. Nevertheless, the auto weighted F FedGP still shows improvement over its fixed-weight version.
Domains
A
C
P
R
Avg
Source Only
50.9
66.1
74.3
76.2
66.9
FedDA
67.9
68.2
82.6
78.6
74.3
FedGP
63.8
65.7
81.0
74.4
71.2
FedDA Auto
67.4
65.6
82.2
75.8
72.7
FedGP Auto
66.1
64.5
82.1
74.9
71.9
Oracle
70.9
58.5
87.4
75.0
73.0
Best DG
64.5
54.8
76.6
78.1
68.5
Table 7: Target domain test accuracy (%) Office-Home by using 15% data samples. We when
source-target shifts are small, FedDA works surprisingly well; FedCP _uto manages to improve FedGP.
C.4 AUTO- WEIGHTED METHODS: IMPLEMENTATION DETAILS, TIME AND SPACE COMPLEXITY
Implementation As outlined in Algorithm we compute the source gradients and target gradients locally when performing local updates. For the target client, during local training, it optimizes its model with B batches of data, and hence it computes B gradients (effectively local updates) during this round. After this round of local optimization, the target client receives the source gradients (source local updates) from the server. Note that the source clients do not need to remember nor send its updates for every batch (as illustrated in Algorithm T. but simply model update per source
28

---

Published as a conference paper at ICLR 2024
client and can compute the average model update divided by the number of batches. Also, because of the different learning rates for the source and target domains, we align the magnitudes of the gradients using the learning rate ratio. Then, on the target client, it computes [d,(ss pr) and ox (Dr) using fgs. 1 and (o } j=1 Our theory suggests we can find the best weights ( N values for all source domains per round, as shown in Section 4.2 which we use for aggregation.
We discovered the auto-wer weighwe versions (FedDA_ Auto and FedGP (ottG_ usually have quicker convergence rate compared with static weights, we decide to use smaller learning rate to train, in order to prevent overfitting easily. In practice, we generally decrease the learning rate by some factors after the initialization stage. For Colored-MNISTT, we use factors of (0.1, 0.5, 0.25) for domains (+90%; +80%, -90%). For domains of the other two datasets, we use the same factor of 0.25. Apart from that, set the target domain batch size to 2, 4, 8 for Colored-MNIST, VLCS, and Terralncognita, respectively.
The following paragraphs discuss the extra time, space, and communication costs needed for running the weighte methods.
Extra time cost To compute more accurate estimation of target variance, need to use smaller batch size for the training on the target domain (more batches lead to more accurate estimation), which will increase the training time for the target client during each round. Moreover, need to compute the auto weights each time when we perform the aggregation. The extra time cost in this computation is linear the number of batches.
Extra space cost In each round, the target client needs to store the batches of gradients on the target client. Lee., the target client needs store extra B model updates.
Extra communication cost Since the aggregation is performed on the server, the extra communica- tion would be sending the B model updates from the target client the central server.
Discussion We choose to designate the target client to estimate the optimal 3 value as we assume the cross-silo setting where the number of source elients is relatively small. In other scenarios where the number of source clients is more than the number of batches used on the target client, one may choose to let the global server compute the auto weights as well as do the aggregation, i.e., the target client sends its batches of local updates to the global server, which reduces the communication cost and shifts the computation task to the global server. one intends to avoid the extra cost induced by the auto-weighting method, we note that the FedGP with fixed 3 = 0.5 can be good enough, especially when the cource-targed domain shift is big. To reduce the extra communication cost of the target client sending extra model updates to the server, we leave it to be an interesting future work (Rothchild et al. 2020).
C.5 VISUALIZATION OF AUTO- WEIGHTED BETAS VALUES ON WORLDWORLD DISTRIBUTION SHIFTS
Figure 6 and Figure 7 show the curves of auto weights 3 for each source domain with varying target domains on the CoNOT dataset, using FedDA and FedGP aggregation rules respectively. Similarly, Figure g and Figure 9 are on the VLCS dataset; Figure 10 and Figure [I are on the Terralncognita dataset. From the results, we observe FedDA_ usually has smaller 3 values compared with F TedCP_Auto, pending on the specific target domain (from le to 1c-1). while FedGP _AAto usually has weights around 1e-1, Also, FedDA_Auto has different weights for each source domain while interestingly, FedGP _uto has the almost same weights for each source domain for various experiments. Ad- ditionally, the patterns of weight change are unclear- in most cases, they have increasing or increasing-thee beessinntteen-deee then-decreasing pattern.
FedDA_Auto has drastically different ranges of weight choice de-
C.6 ADDITIONAL EXPERIMENT RESULTS ON VARYING STATIC WEIGHTS (3)
FedGP is less sensitive the choice of the weight parameter B. enjoying a wider choice range of values, compared with F FedDA. Additionally, under fixed weight conditions, FedGP generally outperforms FedDA in most cases.
From the Table 8
10 and II
29

---

Published as a conference paper at ICLR 2024
100
100
(a) +90%
(b) +80%
(c) -90%
Figure 6: FedDA_Aur ColOTed-MNIST
1000
(a) +90%
(b) +80%
(c) -90%
Figure 7: FedGP Auto ColOTed-MNIST
"spcch
Ipach"
"pach"
Touch
(a) V
(b) L
(c) C
(d) S
Figure 8: FedDA_Aute VLCS
g-
g
Spach
1psch
Spach
1occh
(a) V
(b) L
(c) C
(d) S
Figure 9: FedGP _uto VLCS
Spch
1poth
1pach
Sooch
(a) L.100
(b) L38
(c) L43
(d) I.46
Figure 10: FedDA_ Auto Terralncognita
C.7 SEMI-SYNTHETIC EXPERIMENT SETTINGS, IMPLEMENTATION, AND RESULTS
In this sub-section, we empirically explore the impact of different extents of domain shifts on FedDA, FedGP, and their auto-weighted version. To achieve this, conduct semi-synthetic experiment,
30

---

Published as a conference paper at ICLR 2024
Spach
1pach
1path
Toath
(a) L100
(b) L38
(c) L43
(d) L46
Figure 11: FedGP Auto Terralncognita
0.4
0.8
FedDA
61.21
61.10
59.15
49.90
29.80
17.65
FedGP
61.21
63.82
64.30
65.31
64.80
38.48
Table 8: The effect of 3 FedDA and FedG2 on CIFAR- 10 dataset with 0.4 noise level.
-90%
0.2
0.4
0.8
FedDA
84.41
73.55
54.19
35.16
31.16
27.59
FedGP
84.41
88.96
89.50
89.95
90.03
9.85
Table 9: The effect of B on FedDA and FedGP Colored-MNIST -90% domain.
+90%
0.2
0.4
0.8
FedDA
88.96
82.37
71.32
62.03
58.77
55.23
FedGP
88.96
89.98
89.76
89.84
90.18
69.47
Table 10: The effect of 3 on FedDA and FedG2 on Colored-MNIST +90% domain.
+80%
0.4
0.8
FedDA 73.66
75.12
70.21
65.14
61.84
61.32
FedGP 73.66
73.61
74.39
79.32
80.11
76.67
Table 11: The effect of 33 on FedDA and FedGP on Colored-MNIST +80% domain.
where manipulate the extent of domain shifts by adding different levels of Gaussian noise (noisy features) and degrees of class imbalance (label shifts). We show that the main impact comes from the shifts between target and source domains instead of the shifts between source domains themselves.
Datasets and models We create the semi-synthetic distribution shifts by adding different levels of feature noise and label shifts to Fashion-MNIST (Xiao et al. (017) and CIFAR-10 (Krizhessky al. 2009) datasets, adapting from the Non-IID benchmark (Li et al. 2022). For the model, we use CNN model architecture consisting of two convolutional layers and three fully-connected layers. set the communication round source clients) in the system.
and the local update epoch to 1, with 10 clients (I target,
Baselines We compare the following methods: Source Only: we only use the source gradients by averaging. Finetune OititueeeOOtlinee:e we perform the same number of 50 epochs of fine-tuning after FedAvg. FedDA (3 0.5): convex combination with a middle trade-off point of source and target gradients. FedGP (3 = 0.5): middle trade-off point between source and target gradients with gradient projection. Target Only: we only use the target gradient (8 0). Oracle: a fully supervised training on the labeled target domain serving as the upper bound.
Implementation For the experiments the Fashion-MNIST dataset, set the source learning rate to be 0.01 and the target learning rate to 0.05. For CIFAR- 10, we use 0.005 source learning rate and 0.0025 learning rate. The source batch size is set 64 and the target batch size is 16. We
31

---

Published as a conference paper at ICLR 2024
partition the data to clients using the same procedure described in the benchmark (Li et al. 2022). We use the cross-entropy loss as the criterion and apply the Adam (Kingma & Ba 2014) optimizer.
Setting 1:
Noisy features We add different levels of Gaussian noise to the target domain control the source-targe domain differences. For the Mashion-MNIST dataset, we add Gaussian noise levels of std = (0.2, 0.4, 0.6, 0.8) to input images of the target client, create various degrees of shifts between source and target domains. The task is to predict 10 classes on both source and target clients, For the CIFAR-I0 dataset, we use the same noise levels, and the task is to predict 4 classes on both source and target clients. We use 100 labeled target samples for Fashion-MNIST and 10% of the labeled target data for the CIFAR-I0 dataset.
Setting 2: Label shifts We split the Fashion-MNISTT into two sets with 3 and 7 classes, respectively, denoted as D2 and D2. A variable E [0, 0.5] is used to control the difference between source and target clients by defining Ds 17 portion from D1 and (1 77) portion from D2, Dr= = (1 - -77 portion from D and 1) portion from D2. When r = 0.5, there is no distribution shift, and when - 0, the shifts caused by label shifts become more severe. We use 15% labeled target samples for the target client. We test on cases with 17 = [0.45, 0.30, 0.15, 0010, 0.05, 0.00).
wuto-wedghted methods and FedGP maintain a better trade-off between bias and variance Table 12 and Table 13 display the performance trends of compared methods versus the change of source-target domain shifts. In general, when the source-target domain difference grows bigger, FedDA, Finetune Offline, and FedAvg degrade more severely compared with auto-weighted methods, FedGP and Target Only. We find that auto-weighted methods and F FedGP outperform other baselines in most cases, showing good ability to balance bias and variances under various conditions and being less sensitive to changing shifts. For the label shift cases, the target variance decreases as the domain shift grows bigger (easier to predict with fewer classes). Therefore, auto-weighted methods, FedGP as well as Target Only surprisingly achieve higher performance with significant shift cases. In addition, auto- weighted FedDA manages to achieve a significant improvement compared with the fixed weight FecIA, with competitive performance compared with FedGP _Auto, while F FedGP AAtt generally has the best accuracy compared with other methods, which coincides with the synthetic experiment results.
Connection with our theoretical insights Interestingly, we see that when the shift is relatively small (17 = 0.45 and noise level for Rashion-MNISTT, FedAvg and FecDA both outperform FedGP. Compared with what we have observed from our theory (Figure 5) adding increasing levels of noise can be regarded as going from left to right on the x-axis and when the shifts are small, we probably will get into an area where FedDA is better. When increasing the label shifts, we are increasing the shifts and decreasing the variances simultaneously, we go diagonally from the top-left to the lower-right in Figure 5 where expect FedAvg is the best when start from small domain difference.
Tables of noisy features and label shifts experiments Table 12 and Table 13 contain the full results. We see that FedGP, FedDA_Aus and FedCP Auto methods obtain the best accuracy under various conditions; FedGP FedGP_Auto __Atto outperforms the other two in most cases, which confirms the effectiveness of our weight selection methods suggested by the theory.
Impact of extent of shifts between source clients In addition to the source-target domain dif- ferences, we also experiment with different degrees of shifts within source clients. To control the extent of shifts between source clients, we use a target noise = 0.4 with [3, 5, 7, 9] labels available. From the results, we discover the shifts between source clients themselves have less impact on the target domain performance. For example, the 3-label case (a bigger shift) generally outperforms the 5-label one with a smaller shift. Therefore, we argue that the source-target shift serves as the main influencing factor for the FDA problem.
C.8 ADDITIONAL ABLATION STUDY RESULTS
The effecc of target gradient variances We conduct experiments with increasing numbers of target samples (decreased target variances) with varying noise levels [0.2, 0.4, 0.6) on Fashion-MNISY and
32

---

Published as a conference paper at ICLR 2024
Doshion-MNISS
CIFAR-10
Target noise
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Source Only
83.94
25.49
18.55
16.71
14.99
20.48
17.61
16.44
16.27
Finetune Offline
81.39
48.26
40.15
36.64
33.71
66.31
56.80
52.10
49.37
FedDA_ 0.5
86.41
69.73
58.6
50.13
45.51
62.25
54.67
49.77
47.08
FedGP 0.5
76.33
75.09
71.09
68.01
62.22
66.40
65.28
63.29
61.59
FedGP _
79.40
77.03
71.67
63.71
54.18
21.46
20.75
19.31
18.26
LedDA_Auto
78.25
77.03
72.68
67.69
62.85
65.83
64.09
62.29
60.39
FedGP _Atto
76.19
75.09
71.46
67.53
62.93
67.02
65.26
63.12
61.30
Target Only
74.00
70.59
66.03
61.26
57.82
60.69
60.25
59.38
59.03
Oracle
82.00
82.53
81.20
75.60
72.60
73.61
70.12
69.22
68.50
Table 12: Target domain test accuracy (%) by adding feature noise Fashion-MNIST and CIFAR-
10 datasets using different aggregation rules.
17 Source Only
0.45
0.3
0.15
0.1
0.05
0
83.97
79.71
69.15
59.90
52.51
0.00
Finetune Offline
79.84
80.21
83.13
85.43
89.63
33.25
FedDA 0.5
82.44
80.85
77.50
76.51
68.26
59.56
FedGP 0.5
82.97
83.24
85.97
88.72
91.89
98.71
FedGP_1
77.41
73.12
62.54
53.56
27.62
0.00
FedDA_Auto
83.94
83.91
86.50
89.45
91.87
98.51
FedG?_ _utto
84.68
84.14
86.72
89.58
92.03
98.53
Target only
81.05
82,44
84.00
88.02
89.80
98.32
Oracle
87.68
88.06
90.56
91.9
93.46
98.73
Table 13: Target domain test accuracy (%) by adding feature noise to the Fashion-MNIST dataset using different aggregation rules.
CIFAR- 10 datasets with 10 clients in the system. We compare two aggregation rules FedGP and FedDA, as well as their auto-weighted versions. The results are shown in Table 15 (fixed FedDA and FedGP) and Table T6 (auto-weiggtee FedDA and FedGP). When the number of available target samples increases, the target performance also improves. For the static weights, we discover that FedGP can predict quite well even with small number of target samples, especially when the target variance is comparatively small. For auto-weighted FedDA and FedGP, we find they usually have higher accuracy compared with FedGP, which further confirms our auto-weighted scheme is effective in practice. Also, we observe that sometimes F FedDA_Aute performs better than FedGP_Aue (e.g. on the Fashion-MNIST dataset) and sometimes vice versa (e.g. on the CIFAR-10 dataset). We hypothesize that since the estimation of variances for FedGP is approximation instead of the equal sign, it is possible that FedDA_Ar Auto can outperform FedGP __Atto in some cases because of more accurate estimations of the auto weights 3. Also, we notice auto-weighted scheme seems to improve the performance more when the target variance is smaller with more available samples and the source-target shifts are relatively small. In addition, we compare different levels of data scarcity. We show our methods consistently outperform FedAvg across all cases, which further confirms the effectiveness of our proposed methods.
methods with FedAvg, using
Number of labels
9
7
5
Source Only
11.27
18.81
25.54
34.1 11
Finetune_ Offline
37.79
64.75
68.69
65.41
FedDA_ 0.5
55.01
57.9
54.16
61.62
FedGP 0.5
68.50
68.64
64.43
66.59
FedGP
65.27
59.07
26.37
39.86
Target Only
63.06
65.76
61.8
63.65
Oracle
81.2
81.2
81.2
81.2
Table 14: Target domain test accuracy (%) on label shifts with [3,5,7,9] labels available on Fashion- MNIST dataset with target noise 0.4 and 100 target labeled samples.
33

---

Published as a conference paper at ICLR 2024
Noise level
0.2
0,4
0.6
FedAvg
FodDA
FodGP
FedAvg
FodDA
PedGP
FedAvg
FodDA
FedGP
100
75.98
69.73
75.09
59.36
58.60
71.09
49.94
50.13
68.01
200
76,50
72.07
74.21
60.20
70.93
48,56
52.67
70.31
Mashion-MNIST
75.55
76.59
58.74
47,90
54.97
1000
76.12
77.92
78.68
62.33
68.26
75.17
50.81
59.16
71.63
5%
24.50
62.24
64.21
21.25
46.89
63.57
19.42
47.56
61.39
CIFAR-10
10%
22.86
62.25
65.92
22.35
54.67
65.39
18.60
49.77
63.67
15%
23.20
59.16
65.97
23.25
56.93
65.11
17.88
51.83
63.73
Table 15: Target domain test accuracy (%) by adding feature noise=0.2, 0.4, 0.6 on the Fashion- MNIST and CIFAR-I0 datasets with different numbers of available target samples using fixed weights, in comparison with FedAvg. We methods generally are more robust than FedAvg with significant improvements.
Noise level
0.2
0.4
FedDA_As
FedGP
FedDA
FedSF Auto
FedDA, Auto
FedGP Auto
100
79.04
75.45
72.21
71.93
66.16
67.47
Fashion-MNIST
200
79.74
76.74
74.30
72.96
69.27
69.04
79.48
78.65
75.21
74.55
71.40
70.72
1000
80,23
79.91
76.75
76.35
73.16
73,16
5%
63.04
65.62
60.79
62.84
60.02
60.47
CIFAR-10
10%
65.72
67.41
64.43
65.17
62.25
62,94
65.92
63.36
63.14
Table 16: Target domain test accuracy (%) by adding feature noise=0.2, 0.4, 0.6 the Fashion-
MNIST and CIFAR-10 datasets with different numbers available target samples using auto weights.
C.9 IMPLEMENTATION DETAILS OF FEDGP
To implement fine- lineggraneed projection for the real model architecture, we compute the cosine similarity between one source client gradient and the target gradient for each layer of the model with threshold of O. In addition, we align the magnitude of the gradients according the number of tar- get/source samples, batch sizes, local updates, and learning rates. In this way, we implement FedGP by projecting the target gradient towards source directions. We show the details of implementing static and auto-weighted versions of FedGP in the following two paragraphs.
Stitic-weighted FedGP implementation Specifically, we compute the model updates from source globbll and G.,²' = =hl²) - 11(r-11 respectively. In our real training process, because we use different learning rates, and training samples for source and target clients, need to align the magnitude of model updates. We first align the model updates from source clients to the target client and combine the projection results with the target updates. We use lrT and Irs to denote the target and source learning rates; batchsizer and balchsizes for target and source domains, respectively; ny is the labeled sample size on target client and n, is the sample size for source client Cs,; 7s is the rounds of local updates on source clients. The total
and target clients as (+)) ~ G (1(
the batch sizes
gradient projection Pep from all source clients {Glr] J.M projected on the target direction Gr could be computed as follows. We use L to denote all layers of current model updates. n, denotes the number of samples trained on source client Cs., which is adapted from FedAvg (McMahan et al. 2017) to redeem data imbalance issue. Hence, we normalize the gradient projections according the number of samples. Also, USeC concatenates the projected gradients of all layers.
Par 3 CP ((15) - 4 ( (1477 l.ff b 182
TutchotocT
(r-1)
globel
Lastly, a hyper--aramettr /3 is used to incorporate target update Gy into P&P to have a more stable performance, The final target model weight 1.(r) at round is thus expressed as:
(r)
1 (r-1) 1) + (1 B) Pap + 3 GT
34

---

Published as a conference paper at ICLR 2024
weih-ted FedGP Implementation For auto-weighted scheme for FedGP, compute dynamic weight 3g for each source domain Ds, per communication round. With a set of computed {3, /=1 weight values, the weighted projected gradients for certain epoch can be expressed as follows:
(G Por Ulce EL-1 GP (( "Siooll) (MT) nlp ")))
(
Mulubat)
Similarly, we need to incorporate target update GT into Per. The final target model weight h4y at round is thus expressed as:
(r-1) h (r) h1- PGP
n4 Bi
Gr
Σ N
C.10 GRADIENT PROJECTION METHIOD'S TIME AND SPACE COMPLEXITY
Time complexity: Assume the total parameter m and we have layers. To make it simpler, assume each layer has average of parameters. Computing cosine similarity for all layers of one source client is (((T)) Om2 //). We have N source clients the total time cost for GP is O(N-m m/²
Space complexity: The extra memory cost for GP (computing cosine similarity) is O(1) per client for storing the current cosine similarity value. In real implementation, the whole process of projection is fast, with around 0.023 seconds per call needed for N = 10 clients of Fashion-MNIST experiments on the NVIDIA TITAN Xp hardware with GPU available.
C.I1 ADDITIONAL. EXPERIMENT RESULTS ON HEA-H
As showcase of realistic healthcare setting, show the performances of our methods compared with personalized baselines on the Fed-Heart dataset from FLamby (Du Terrail et al.) 2022). We randomly sample 20% data for the 0, 1,3 centers and 100% data for the center since there only 30 samples on the target domain. As shown Table 17 our methods generally outperform
other baselines with large margins. KNN-per (Marfoq et al. 2022) may not fit this scenario since the neural network we used only consists of one layer.
center
0 (20%)
(20%)
(100%)
(20%)
Avg
FedDA
79.81
78.65
67.50
62.22
72.05
FedGP
78.85
80.45
68.75
65.33
73.35
FedDA_Auto
80.77
80.90
68.75
71.11
75.38
FedGP_ Auto
80.77
79.78
68.75
69.78
74.77
Source only
76.92
75.96
62.50
55.56
67.74
FedAvg
75.96
76.40
62.50
55.56
67.61
Ditto
76.92
73.03
62.50
55.56
67.00
FedRep
78.84
65.17
75.00
60.00
69.75
APFL
51.92
57.30
31.25
42.22
45.67
KNN-Per 56.00
56.00
56.00
56.00
56.00
Table 17: Target domain test accuracy (%) on Fed-Heart. FedGP and auto-weighted methods generally outperform personalized FL methods with significant margins.
C.12 COMPARISON WITH THE SEMI-SUPERVISED DOMAIN ADAPTATION (SSDA) METHOD
In this sub-section, we show the performances of our methods compared with SSDA methods. However, we note that the suggested SSDA methods cannot be directly adapted to the federated setting without major modification. Kim & Kim (2020) uses feature alignments and requires access to the source and target data at the same time, which is usually difficult to achieve in federated learning. As for Saito al. (009)) the overall adversarial learning objective functions consist of
35

---

Published as a conference paper at ICLR 2024
loss objective on both source and target labeled data and the entropy coming from the unlabeled target data, which also cannot be directly adapted to federated learning. On the contrary, auto-weighted methods and FedGP have the flexibility to do the single source-tareet domain adaptation, which can be compared with the SSDA method, though we notice that our setting is different from SSDA since do not have unlabeled data the target domain and do not leverage the information coming from this set of data. Here, we perform experiments on real-world datasets: our results suggest that auto-weighted methods and FedGP outperform MME (Saito al. 2019) when the shifts are large even without using unlabeled data (overall our proposed methods have comparable performance with MME). Also, we observe single cource-targe domain adaptation setting, auto-weighted FedGP usually has better performance than auto-weighted FedDA and FedGP.
0->1/1 +>0
0 0->2/2-> ->0
1->2/2-> ->1
MME Saito et al. 2019
79.55 / 89.28
25.98 12.99
14.66 24.43
FedGF
79.34 88.34
90.23 64.47
90.23 78.70
FedDA_Auto
79.31 88.38
87.00/56.07
89.13 66.59
FedGP Auto
79.34 71.59
90.23 00.23/80.20 80.95
90.23 79.34
Table 18: Colored-MNIST (0: +90%, 1: +80%, 2: -90%)
0->1
1->2
2->3
MME (Saito et al. 2019)
68.68
72.86
81.90
FedGP
67.92
75.30
77.45
FedDA_Auto
67.92
75.73
76.97
FedCP Auto
68.45
74.21
77.69
Table 19: VLCS (0: C, 1: L, 3. S)
0->1
1->2
2->3
MME (Saito et al. 2019)
74.51
54.91
58.50
FedGP
72.35
58.19
60.03
FedDA_Auto
72.31
54.26
61.33
TedGP_Aut
72.42
56.27
61.90
Table 20: Terralncognita (0: L100, L38, L43, 3: L46)
36


Reviews:
[{"review_id": "NPBZ3F3ko3", "reviewer": "Reviewer_nVHX", "summary": "This paper considers two important issues in federated learning: small data at client sites and domain shift across clients. Simple, intuitive strategies such as gradient projection and auto-weighting for mitigating these issues are proposed. Several interesting theorems are proved regarding federated aggregation, gradient projectin. Results on three datasets are provided.", "questions": "Try on DomainNet, Office Home and PACS datasets. Although these datasets are from 2019 and 2017, atleast they are more challenging datasets.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Nice treatment of federated learning in the presence of domain shift and small data. A good mixture of theoretical and experimental work.", "weaknesses": "I loved the paper till I came to the experiments section. In this day and age, should we still be doing experiments with ColoredMNIST, VLCS, CIFAR-10 and TerraIncognita? ColoredMNIST, VLCS and TerraIngocnita are from 2019, 2013 and 2018 respectively! This raises the questions whether the proposed solutions will scale to larger and difficult datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zjrYF3HCeK", "reviewer": "Reviewer_28v7", "summary": "This paper studies the optimal design of an aggregator in federated domain generalization (generalizing to a target client/domain in cross-silo). The FedDA method works by aggregating interpolations (i.e. weighted averages) of the target domain gradient and the source domain gradients. The FedGP method further projects the source gradient onto the \"positive direction\" of the domain gradient before interpolation (i.e. zeroing out conflicting directions). The interpolation factor for each domain is chosen such that the $L^\\pi$ distance between the source and target gradients is minimized w.r.t. some prior $\\pi$ on the parameters. The authors show that this error can be decomposed as a noise term of the target domain and a distance term between the source and the target domains. The optimal interpolation factor would then balance the source and target gradients based on those terms, which can be estimated during training in a scheme called auto_weight. Extensive experiments show the benefits of this approach.", "questions": "Can you train your algorithms and compare them on federated datasets that follows the setting of interest in the paper (cross-silo with data scarcity)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's analysis and experiments are well-detailed.\n- The analysis is interesting and covers many aspects of the design of an optimal aggregator in the federated domain generalization setting.\n- The method is intuitive and is easy to implement (save for the auto-weighting scheme).\n- The improvement seems to be significant in terms of generalizing to the target domain with respect to personalized federated learning algorithms.", "weaknesses": "- It seems like it would be better to compare the algorithms presented in this paper to domain generalization algorithms, such as the ones shown in DomainBed's GitHub repo.\n- The methods presented make sense mostly in the cross-silo setting, as mentioned in the paper, which limits its applicability to general federated learning problems with a relatively larger number of clients that can benefit a lot from methods for generalizing to new clients.\n- It is mentioned multiple times that data scarcity is the setting of interest, in which FedDA and FedGP are supposed to perform more favorably. However, we do not see experiments showing the effect of data scarcity on the robustness of the performance of FedGP vs. FedAvg, for example.\n- Personalized federated learning algorithms are relevant for comparison, but I think that direct comparison of such algorithms with FedGP might put them at a disadvantage since they are not specifically designed for domain generalization. ColoredMNIST, VLCS, and TerraIncognita datasets are more concerned with shifts in p(x) or p(x|y), whereas personalized FL is more concerned with shifts in p(y) and p(y|x), i.e. personalizing the prediction rather than adapting to spurious correlations or invariant attributes. You should either choose federated datasets for comparison, or you should compare your algorithm to domain generalization algorithms, e.g. IRM and others. Or why not use a hospital dataset that fits the setting you described in the paper? For example, you can consider FLamby [2].\n- One work [1] from federated continual learning might be of interest (which even shares the same name FedGP). It is motivated from a similar intuition, which is to remove from the gradient its projection onto the negative direction of the reference gradient.\n- In algorithm 1, the auto_weight scheme requires intermediate gradients for each domain, which might require a lot of memory.\n\n[1] FedGP: Buffer-based Gradient Projection for Continual Federated Learning. Dai et al. 2023.\n[2] FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. Terrail et al. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nUEhld5sD", "reviewer": "Reviewer_tJ7A", "summary": "This paper proposes two algorithms to solve federated domain adaptation, a case in which there exists a distributional/domain shift between clients in federated learning. The authors tackle the problems of domain shift and data scarcity in their work. To solve these problems, they propose to design algorithms concerning the __server aggregation rule__, i.e., how the server merges the different gradients of the same model coming from the clients. The two proposed methods are called __FedDA__ which does a convex combination of clients' gradients (including the target), and __FedGP__, which extracts information from source clients' gradients based on the target client gradients.", "questions": "__Q1.__ Concerning __FedDA__ and __FedGP__ aggregation schemes, in order to have a convex combination, shouldn't $\\sum_{i=1}^{n}\\beta_{i}=1$? Is this constraint enforced? For instance, in Figure 9 (appendix), the sum of betas exceeds 1.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "__Originality.__ The authors provide a novel theoretical framework for the analysis of Federated Learning under heterogeneity.\n\n__Quality.__ The paper is globally well-written and clear. Parts of the experimental section could be improved\n\n__Clarity.__ The novel theoretical framework is easy to follow. Assumptions, notation and definitions are clearly stated and the proposed algorithms are intuitive.\n\n__Significance.__ This paper tackles an extremly important problem in federated learning, i.e., how to deal with __client heterogeneity__. In this sense, besides being important for the niche of federated DA, it can also impact federated learning in general.", "weaknesses": "__Major__\n\n__W1.__ The description of the real-world experiments in the main paper is not sufficient. While the authors do provide enough information in the appendix, the main paper does not fully describes the methodology the authors employed in consolidating the results of Table 1. For instance, how are the labeled data points chosen for the experiments? How does the performance change w.r.t. to the choice of data points (i.e. standard deviation of the accuracy on target domain)? These questions are answered in the appendix, but they should be clear in the main paper.\n\n__Minor.__ (note, this point __did not__ impacted negatively in my review).\n\n__W2.__ I would like to raise that, while this is a Federated DA paper, the authors assume access to a (small) set of labeled data in the target domain. This somewhat breaks the rules of _Unsupervised_ DA, and may bias performance towards methods that use target labeled data when comparing with UDA algorithms such as KD3A. This remains somewhat true even when supposing a small amount of labeled target data, depending on the degree of distributional shift. Nonetheless, __I do think the authors use labeled data in a clearly motivated and justified way__.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HbKiipNiug", "reviewer": "Reviewer_d97G", "summary": "The submission studied domain adaptation under the federated setting. Two methods (FedDA (1) and FedGP(2)) to aggregate gradients were proposed based on the analysis of the delta error of an aggregation rule (Theorem 3.6). An auto-weighting rule (3) was proposed (FedDA_Auto and FedGP_Auto), too. Experiments showing the robustness (Figure 2) and target accuracies (Table 1) justified the proposed methods.", "questions": "(b) Given the federated learning nature, multiple target clients seem more practical. How would the proposed method scale with the number of target clients?\n\n(c) The current source clients are given and assumed to be well-trained. What are the potential and challenges to extending the proposed method to a scenario where every client learns and transfers simultaneously?\n\n(d) What if one negates the direction of the projections of negative source gradients (e.g., the projections of g_s3 and g_s4 in Figure 1)?\n\n(e) The behaviors of FedDA in Figure 2(a) and Figure 2(c) seem contracdict to each other. Is it trivial? Or may I have a clarification?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The submission identifies the factors that affect the performance of an aggregation rule and then proposes solutions based on the findings.\nThe target domain enjoys the robustness and performance increases brought by the solutions.", "weaknesses": "Despite a comfortable reading experience and leading performance results, I would like to raise a concern about the problem formulation.\n\n(a) From the federated learning perspective, the server and the clients are learning together to achieve a better performance measured by the sum of ALL clients. Therefore, federatively speaking, paying the whole attention to ONE target client might not align with the original intention of studying federated learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)