FedDAR: Federated Domain-Aware Representation Learning
Authors: Aoxiao Zhong
Hao He
Zhaolin Ren
Na Li
Quanzheng Li
Venue: ICLR 2023 poster (2023)
Keywords: federated learning
healthcare
fairness
personalization

Abstract: Cross-silo Federated learning (FL) has become a promising tool in machine learning applications for healthcare. It allows hospitals/institutions to train models with sufficient data while the data is kept private. To make sure the FL model is robust when facing heterogeneous data among FL clients, most efforts focus on personalizing models for clients. However, the latent relationships between clients' data are ignored. In this work, we focus on a special non-iid FL problem, called Domain-mixed FL, where each client's data distribution is assumed to be a mixture of several predefined domains.  Recognizing the diversity of domains and the similarity within domains, we propose a novel method, FedDAR, which learns a domain shared representation and domain-wise personalized prediction heads in a decoupled manner. For simplified linear regression settings, we have theoretically proved that FedDAR enjoys a linear convergence rate.  For general settings, we have performed intensive empirical studies on both synthetic and real-world medical datasets which demonstrate its superiority over prior FL methods. Our code is available at https://github.com/zlz0414/FedDAR.     

Published as a conference paper at ICLR 2023
FEDDAR: FEDERATED DOMAIN REPRESENTA- TION LEARNING
Aoxiao Zhong' Hao Zhalln Ren', Na Li', and Quanzheng Li* Harvard University MMassachusetts Institute of Technology
3 Massachusetts General Hospital/Haryard Medical School
aox aoxiaozhonggg. harvard. edu, hache@mi t. edu, zhaol paaolineeng..o www.iinggaaaaar.d.cooo harvard.edu
harvard.edu, 1i quanzhenggmghh hww.://mg..hhh.....aamaam harvard. edu
ABSTRACT
Cross-silo Federated learning (FL) has become a promising tool in machine learning applications for healthcare. It allows hospitals/istiiutions to train models with sufficient data while the data is kept private. To make sure the FL. model is robust when facing heterogeneous data among FL clients, most efforts focus on personalizing models for clients. However, the latent relationships between elients' data are ignored. In this work, we focus on a special non-iid FL problem, called nomain-mixed FL, where each client's data distribution is assumed to mixture of several predefined domains. Recognizing the diversity of domains and the similarity within domains, we propose novel method, FedDAR, which learns a domain shared representation and domain-wise personalized prediction heads in a decoupled manner. For simplified linear regression settings, have theoretically proved that FedDAR enjoys a linear convergence rate. For general settings, we have performed intensive empirical studies on both synthetic and real-world medical datasets which demonstrate its superiority over prior FL methods. Our code is available https::///// http//ggtttbbbb com/ 2120414 41/44//////..R..
INTRODUCTION
Federated learning (FL. (McMahan et al., 2017a) is a machine learning approach that allows many clients(e.g. mobile devices organizations) collaboratively train model without sharing the data. has great potential to resolve the dilemma in real-world machine learning applications, especially in the domain of healthcare. A robust and generalizable model in medical application usually requires a large amount diverse data to train. However, collecting a large-scale centralized dataset could be expensive or even impractical due to the constraints from regulatory, ethical and legal challenges, data privacy and protection (Rieke et al., 2020).
While promising, applying FL to world problems has many technical challenges. One eminent challenge is data heterogeneity. Data across the clients are assumed to be independently and identically distributed (iid) by many FL algorithms. But this assumption rarely holds in the real world. It has been shown that non-iid data distributions will cause the failure of standard FL strategies such as FedAvg (Jiang et al., 2019; Sattler et al., 2020; Kairouz et al., 2019; Li et al., 2020). As an ideal model that can perform well on all clients may not exist, it requires FL algorithms to personalize the model for different data distributions.
Prior theoretical work (Marfoq et al., 2021) shows that is impossible improve performances on all clients without making assumptions about the client's 's data distributions. Past works on personalized FL methods (Marfoq et al., 2021; Sattler et al., 2020; Ghosh et al., 2020; Mansour et al., 2020; Deng et al., 2020) make their own assumptions and tailor their methods to those assumptions. we propose a new and more realistic assumption where each client's data distribution is mixture of several predefined domains. We call our problem setting Domain-mixed FL.. is inspired by the fact that the diversity of the medical data can be attributed to some known concept of domains, e.g.. different demographic/ethnic groups of patients (Szezepura, 2005; Ranganathan & Bhopal, 2006; NHS, 2004), different manufacturers or protocols/workflows of image scanners (Märtensson et al.,
this paper,
"Equal contribution

---

Published as a conference paper at ICLR 2023
2020; Ciompi et al., 2017) and on. It necessary to address the ubiquitous issue of domain shifts among ethic groups (Szezepura, 2005; Ranganathan & Bhopal, 2006; NHS, 2004) or vendors (Yan et al., 2019; Garrucho et al., 2022; Guan & Liu, 2021) in healthcare data. Despite of the domain shifts, same domain at different clients are usually considered have the same distribution. The data heterogeneity between FL. clients actually comes from the distinct mixtures of diverse domains at clients. These factors motivate us to personalize model for each domain instead of client.
Although our method is inspired by healthcare applications where the domain shifts issue is well- known and domain labels are very basic and accessible, we believe that can be generally applied to other domains like finance or recommendation systems where users/humans with different demogra- phy are involved (Ding et al., 2021; Asuncion & Newman, 2007). However would require a deep understanding of the data and background knowledge to verify the data distribution assumption as well as the accessibility of the domain label.
Marro et al., 2021) and FcdMMinMax(Pppaakii et al., 2021) makes similar assumption on data distribution as ours. However. FedEM assumes the domains unknown and tries to learn linear combination of several shared component models with personalized mixture weights through an EM-like algorithm. FedMinMax doesn't It acknowledge the domain shift between domains and still aims to learn one shared model across domains by adapting minmax optimization to FL. setting.
Our Contributions. We formulate the proposed problem setting, Domain-mixed FL. Through our analysis, we find prior FL methods, both generic FL methods like FedAvg (McMahan al., 2017a), and personalized FL methods like FedRep (Collins et al., 2021), are sub-optimal under our setting. To address this issue, we propose new algorithm, Federated Domain-Aware Representation Learning ( (FedDAR). FedDAR learns a shared model for all the clients but embedded with domain- wise personalized modules. The model contains two parts: an shared encoder across all domains and a multi-headed predictor whose heads are associated with domains. For an input from one specific domain, the model extracts representation via the shared encoder and then use the corresponding head to make the prediction. FedDAR decouples the learning of the encoder and heads by alternating between the updates of the encoder and the heads. It allows the clients to run many local updates on the heads without overfitting on domains with limited data samples. This also leads to faster convergence and better performed model. FedDAR also adapts different aggregation strategies for the two parts. We use a weighted average operation to aggregate the local updates for the encoder. With additional sample re- weightttnng, the overall training objective is equally weighted for each domain to encourage the fairness among domains. While for the heads, we propose a novel second-order aggregation algorithm to improve the optimality of aggregated heads.
We theoretically show our method enjoys nice properties like linear convergence and small sample complexity in a linear case. Through extensive experiments on both synthetic and real-world datasets, we demonstrate that FedDAR significantly improves performance over the state-of-the-ar personalized FL. methods. To the best of our knowledge, our paper is among the first efforts in domain-wise personalized federated learning that achieve such superior performance.
2 RELATED WORK
Besides the literature we have discussed above, other works personalization and faimess in federated learning are also closely related to our work.
Personalized Federated Learning. Personalized federated learning has been studied from a variety of perspectives: i) local fine-tuning (Wang et al., 2019; Yu et al., 2020) ii) meta-lcarning (Chen et al., 2018; Fallah et al., 2020; Jiang et al., 2019; Khodak et al., 2019) iii) local/global model interpolation (Deng et al., 2020; Corinzia et al., 2019; Mansour et al., 2020). iv) clustered FL that partition clients into clusters and learn optimal model for each cluster (Sattler et al., 2020; Mansour (Vanhaesebrouck et al., 2020; Ghosh et al., 2020) et al., 2021). v) Multi-Tarn Task Learning
et al., 2017; Smith et al., 2017; Zantedeschi et al., 2020) (Hanzely & Richtárik, 2020; Hanzely et al., 2020; T Dinh et al., 2020; Huang et al., 2021; Li et al., 2021a) vi) local representations or heads for clients (Arivazhagan et al., 2019; Liang et al., 2020; Collins et al., 2021)(Luu) (Luo et al., 2022). vii) personalized model through hypernetwork or super model (Shamsian et al., 2021; Chen & Chao, 2021; Xu al., 2022). The personalization module in our approach is similar vi) and (Zhu et al., 2021) with a multi-branch network. However, the targets we are personalizing the model for are the domains instead of clients.
2

---

Published as a conference paper at ICLR 2023
Fairness in Federated Learning. There are two commonly used definitions of fairness in existing FL works. One is client fairness, usually formulated as client parity (CP), which requires clients to have similar performance. A few works (Li et al., 2021a; a; 2019; Mohri et al., 2019; Yue et al., 2021; Zhang et al., 2020) have studied on this. Another group fairness. fundamental tradeoff between group fairness and accuracy has been studied (Menon & Williamson, 2018; Wick et al., 2019; Zhao & Gordon, 2019), and various fair training algorithms have been proposed(Roh et al., 2020; Jiang & Nachum, 2020; Zafar et al., 2017; Zemel et al., 2013; Hardt al., 2016). Since the notions of group fairness is the same in FL setting, most of existing FL works adapt methods from centralized setting (Zeng et al., 2021; Du et al., 2021; Gálvez et al., 2021; Chu al., 2021; Cui et al., 2021). In this work, our method not designed specifically for certain group fairness notions like demographic parity. Instead, we aim to achieve the best possible performance for each domain through personalization, admitting the difference between data domains. Moreover, our concept of data domains is not limited as demographic groups It can also be applied to any other mixture of domain data, long our assumptions hold.
the centralized setting, the
3 PROBLEM: DOMAIN-MIXED FEDERATED LEARNING
Notations. Federated learning involves multiple clients. We denote number of clients as We use i e [n] é (1, .... } to index each client. Client i has a local data distribution D. which induces local learning objective, i.c., the expected risk (f) = E. [ [L(f(x)) (..).. v.]], where f X + y the model mapping the input x E X' to the predicted label f(z) y and y x y + IR is generic loss function. In real practice, client i E [n]] has finite number, say Li, of data samples, i.e.,
S, {xx) va //) L Σn= L; denotes the total number of data samples.
Problem Formulation of Domain-mixed Federated Learning. We introduce new formulation of FL problem by assuming each clients' local data distribution a weighted mixture of M domain specific distributions. Specifically, we use {Pm)) m=1 to denote data distributions from M predefined domains. For client i, its local data distribution D.- Em Tt.m D,, where the mixing coefficients Tim stand for the probabilities of client i's data sample coming from domain m. Take medical application as an example, different hospitals are clients and different ethnic groups are domains. Each ethnic group have different health data while each hospital's 's data is a mix of ethnic group data.
Further, the domains of the data samples assumed to be known. We (xc, y, 2) to represent the input features, label and domain. The goal of our problem is to learn a model f 2) that can perform well in every domain, as shown by the following learning objective,
triplet of variables
M
min R(f) = M
Σ Rm (f(.)) m=1
(1)
where Rm m)) (=..m))D.... [((, m)) w)]. Our problem focuses on the setting that each domain have a different conditioned label distribution, i..,, Pm (y]zz) is different in each domain m.
3.1 COMPARISON WITH PRIOR ROMAIN-UNAWARE FL. PROBLEM FORMULATIONS
Our FL problem introduces the concept of the domain and focuses the model's performance in each domain. Many prior FL. formulations does not recognize the existence of the domains. For example, the original federated learning algorithms like FedAvg (McMahan et al., 2017a), FedProx (Li et al., 2020) learn a globally shared model that via minimizing the averaged risk, i.e., miny + Σ, (f).. Some variants consider the fairness across the clients. To do so they optimize the worst client's performance, instead the averaged performance, i.e., miny max; (f). Further, personalized FL algorithms, such as FedRep (Collins et al., 2021), customize the model's prediction for each client whose objective is min 1/2 Σx= (..)...
All the FL algorithms mentioned above will lead sub-optimal solutions to our problem since they do not make domain specific predictions. We illustrate this point by the following toy example of linear regression: We assume the data in m 'th domain is generated via the following procedure: æ e Rd i.i.d sampled from a distribution p(z) with mean zero and covariance Ig. The label y E E obeys y x B" B'ww, w where B" dx is ground truth linear embedding shared by all domains, and w E R is the linear head specific to domain m. Under this setting, D.,, stands for data (ar, y)
3

---

Published as a conference paper at ICLR 2023
Algorithm FEDDAR
Input: Data Sini number of local updates Th for the heads, F for representation; number of communication rounds T; learning rate 7). Initialize representation and heads dº, .......... h M
for 1, . T do
Server sends 6'-1 h
1-1 y.... MMM to the clients; for client = 2, .... n2 in parallel do
Client initializes h < hf-1 vm E [M].
for
Th do h h f GRD(Ri, ( o M n). for all m [M].
end for
Client sends updated heads h/m and Hessians H HR.. (h)mm) ) to the server. end for
Server aggregate the heads for each domain:
for m [M] do
Nmm +- (HEADAGGG((1,,)) 2. HR.... (1 (1))))))) via Equation
end for
Server sends ..., . h'M to the n clients; for client
2, ..., in parallel do for = 1 to Tú do
o. +- GRD (R, -
JM 0i s,:51 .7). (
end for
Client sends updated representation of 0; end for
1.7o
to server.
Server computes the new representation via averaging of + E La x Φi. end for
where ~ p(x:) and y = x: B' wm. For each client, the local data D, is mix of data from different domains with mixed coefficients, i.e., D, Em Ti,m im
FedAvg: learns a single model and across the all clients via the following objective,
min Σ B( Bw)² B.
Σ
Timm
(. (y Bw))
2n
2n
VE(m)
vE(N],me(|M]
FedRep: learns shared representation B and separated heads w; for each clients rather than for each domain m.
min B.w W........
E (u- Bwx
Σ
TI
(m. (.. -D (yr x Bw,,) (3)
2n 3E(m)
2n
iE(m)
ME(M]
FedDAR: contrast, in the linear case, our proposed method, FedDAR, which will be introduced next, learns shared representation B and separate heads w.. for each domain
Tim Σp
min 2 M.
E Te.m
(mma))) (y x. Bwn)²
(4)
2M
iE[n] mc(M]
From the above formulations, we can see that FedAvd and FedRep are not able to achieve the zero FL problem.
error in our
4 PROPOSED METHOD: FEDDAR
To solve the Domain-mixed FL problem, we propose method called, Federated Domain-Aware Representation Learning (FedDAR). In the following, we first introduce the model, learning objective and the details of the federated optimization algorithm.
4

---

Published as a conference paper at ICLR 2023
4.1 ALGORITHM OVERVIEW
Our model is made of a shared encoder o(:; 0) and M domain specific heads hm (:; wm) whose are parameterized by neural networks with the weights & and wm. Vm [M]. According to our problem formation in Equation 1, our algorithm aims to solve the following optimization,
1 R(o, hi, har) = M Σ
Rm (fem o o6)
(5)
We decouple the training between encoder and heads. Specifically, we alternates the learning between the encoder and the heads. The learning is done federatedly and has two conventional steps: (1) local updates; (2) aggregation at the server. Algorithm shows the pseudocode code.
Empirical Objectives with Re- weightit,, Empirically, the objectives are estimated via the finite data samples each client. We use S. to denote the set of samples from domain m client i,
with Lim := S..ml denoting the sample size. Further, L, : Em=1 Lim is the number of samples in client i while Lm := Ei=1 Lim is the total number of samples belonging to domain m across all the clients. We denote the empirical risk at client specific to domain as R. (hm 6) Tram E(eyyESS. f(hm ((x)).)) y). The empirical risk at client designed as R2 hi,... hx) = Em Lu.
L I; um Rs,m (hm o の). where thm = I... M re-weights the risk for each domain. Combining commonly used weighted average FL objective R(Ø, hi,..,,,,. = E.= L Ri (i,..,, har), the overall empirical risk is derived as the following,
M
R( hi. hMS) = Σ
L 1=1
Li R, hi,... hM) M
Σ m=1
Rm (he o p),
(6)
where Rm (hmm x) == Eis Lisl Iam Rim o)) This is consistent with Equation 5.
Lm
4.2 LOCAL UPDATES AT CLIENTS
In each communication round, clients use gradient descent methods to optimize representation o)::0) and local heads hm Wm) for E [M] alternately. We use i denote the current round. For a module f, ft-1 denotes its optimized version after L - 1 rounds. Each round has multiple gradient descent iterations. We use ft. to denote the module in round after iterations. Since the updates are made locally, clients maintain their own copies of both modules, we use subscripts to index local copy at client i, e.g., f. We use GRD to denote a generic gradient-base optimization step which takes three inputs: objective function, variables, learning rate and maps them into a new module with updated variables. For example, the vanilla gradient descent has the form GRD(L( Sw. 1) =
For the heads, client performs Th local gradient- based updates to obtain optimal head given the current shared encoder di-1, For s E [Tr], client i updates via hi.. + GRD(R,, (hi 1r-1), ( , -)) For the shared encoder, the clients executes 7 local updates. Specifically, for [To]. client i updates the local copy of the encoder via of + CRD(R, { ) 5/.0-1,0 n). The te-eiihhting mentioned in last section is implemented by re-weighting each sample with um when calculating the loss functionn.
4.3 AGGREGATION AT SERVER
We introduce two strategies: (1) weighted average (WA); (2) second-order aggregation (SA).
Weighted average means the aggregated model parameters are the average of the local model's parameters weighted by the number of data samples. Specifically, for the shared encoder, we have L..
w m. 0 Di= L Similarly for each head, we have w', = = Lm
Second-order aggregation is generates the globally optimal model given a set of locally optimal model, as shown in the following,
complex strategy. Idcally, we want the head aggregation
"" E arg min J(w)= e Σ as.T(w) given w = arg min R,(w) Vi E [n].
(7)
5

---

Published as a conference paper at ICLR 2023
where Ji is i'th client's virtual objective, a; a; L,/L is the importance of the client, L5 is the number of data samples. We call J, the virtual objective to distinguish it from the real learning objective R4. The virtual objective is defined as objective that the local updates give the optimal solution w.r.t it. It is introduced since the local updates during two aggregated are not guaranteed to optimize the head to optimal w.r.t the real objective. For example, if each local update is single step gradient descent with a learning rate i.e., wott = w² v w (ww).. Then the virtual objective becomes Iiw) = R((w")+ + (w) ((("))++ 1 I w I13 which satisfies w."" g++ arg min Ti (()). Such virtual objective leads the solution of problem averaging strategy.
2n
w" n E. wx which is the simple
However, in real practice, the local updates are usually more complicated which makes the virtual objective closer to the true objective. We consider the case that the virtual objective is the second- order Taylor expansion of the true objective, i.e., I(w) = R(w,) + (w w²) V. R R(w²) 2 {ww w²) H& (w")
w') where HR is the Hessian matrix. Then each round of local update equivalent to a Newton-like step, w w HR, (2n) (V"'",) While w un V ( is the desired globally optima. Leveraging the fact that, V R(w) Eien a; V R, and HR(w) Eieny R,HR, (w), can get wi+1 from w. via the following equation, which we call second-order aggregation,
w 1+
-R(ww))+
a, (w")) Σ kE[n]
(8)
Specifically, to implement second-order aggregation, each round, the local clients first optimize the model locally for several epochs. Then we compute the Hessian matrices for each local model and send them to the server for aggregation. Note that sending the Hessian takes a communication cost being quadratic to the size of the weight. In real practice, the predictive head is usually small, e.g., a linear layer with hundreds of neurons. Thus it is acceptable to aggregate the Hessian matrix of the head's parameters.
In the following, we provide two instances of our second-order aggregation with linear head. 1. Linear Regression where (,(w) = ビ. E/ (w) x! - )²² is quadratic itself. Thus the second order Taylor expansion of the objective itself, i.e., J(ww) = (w).. ). In this case, (w) X! X;
where X; [ar .z.") "T["]T is the data matrix of client i.
2. Binary Classification where (,ww) = 1 Σ/n of log o ( +1)+ log(1 o (w x()). L,
is the sigmoid function. Let ue! a ow T x²)) denote model's output. The gradient and the Hessian
are, R. = E, u ) = 1 diag(ı; y:) and HR,(w) = 2, x SX; where S A diag(u) -.),.... 1.. classification. Please refer to the text book (Murphy, 2022) for the exact equations.
(1(( (1 -1- n ))) Similar formulas can be derived for the multiclass
Remark. In practice, when the dimension of is larger than the number of samples of a certain domain, the Hessian may have small singular values which cause numerical instability. To mitigate this issue, one can either directly the representation dimension ie to some smaller number add (fulll-connected) projection layer on top of a pretrained encoder to compress the representations to a lower dimensional space.
4.4 THEORETICAL RESULT OF FEDDAR
For a simplified linear regression setting as discussed in domain-mixed FL (4) (cf. details in Appendix A). we give below the sample complexity required for an adapted version of our algorithm (Algorithm 2 in the appendix) to enjoy lincar convergence. Due to the space limit, we only provide an informal statement highlight the result. The formal statement and the proof are deferred in the appendix.
Theorem 4.1 (Sample complexity of FedDAR convergence in linear case (informal)). Consider the linear setting for domain-mixed FL in (4). At each iteration, suppose that the number of samples used by each of n clients to update the encoder, is 52 di2 ), and that the aggregate number of samples nn used in the update for the domain- domain-specifer specific heads, is (i(k). Then, for suitably chosen step size, the distance between the encoder B, Algorithm 2 outputs and the true encoder B converges at a linear rate.

---

Published as a conference paper at ICLR 2023
Remark. As our algorithm converges linearly to the true encoder, the per-iteration sample com- plexity of our algorithm gives a good estimate of the overall sample complexity. Since we expect the output of the encoder to be significantly lower-dimensional than the input (i.e. k < d), our result indicates that Algorithm 2's sample complexity is dominated by (2 implying that the complexity reduces significantly as the number of clients n increases. Moreover, a key implication of our result is the capacity for our algorithm to accommodate data imbalance across domains. We note that our approach requires (ddk²) samples per iteration for the update of the shared representation B IRdxk whilst needing only (?(k) samples per iteration for the update of each domain head. In particular, domains with more data can contribute disproportionately
the ((dk²) samples required to learn the common representation, whilst domains with fewer data need only provide (2():) samples to update its domain head during the course of the algorithm. Whenever k < d, which we believe is a reasonable assumption for many practical applications (e.g. medical imaging), the requirement of ((k) samples per domain is relatively mild. Conversely, forgoing the shared representation structure would require each domain to learn a separate dddimennsionl classifier, requiring ((d) samples per domain, which can pose a challenge in problems with domain data imbalance.
EXPERIMENTS
We validate our method's effectiveness on both synthetic and real datasets. We first experiment on the exact synthetic dataset described in our theoretical analysis verify our theory. We then conduct experiments on a real dataset, FairFace (Kärkkäinen & Joo, 2019), with controlled domain distributions to investigate the robustness of our algorithm under different levels of heterogeneity. Finally, we compare our method with various baselines on real federated learning benchmark, EXAM (Dayan et al., 2021) with real-world domain distributions. We also conduct extensive ablation studies on to discern the contribution of each component of our method. Full details of experimental settings can be found in Appendix B.
5.1 SYNTHETIC DATA
We first run experiments on the linear regression problem analyzed in Appendix A. We generate (do- main, data, label) samples as the following, 2; ~ M(T;). æ; ~ (N Ia). ys ~ NNw,, B x; o) where o = 10-3 controls label observation errors, M(TT;) is a multinomial domain distribution with parameter π. = [Tr, Mt.M] E A The hyper-parameters of domain distributions π,; are drawn from Dirichlet distribution, i.e., म. ~ Dir(up). where p E AM is prior domain distribution over M domains, and a > 0 is a concentration parameter controlling the heterogeneity of domain distributions among clients. The largest domain distribution heterogeneity is achieved as a + where each client contains data only from a single randomly selected domain. On the other hand, when a + oc, all clients have identical domain distributions that are equal to the prior p. We generate ground-truth representation B E d and domain specific heads w Vm E [M] by sampling and normalizing Gaussian matrices.
Figure 5.2 shows result of our experiments where we set n = 100 clients, M = 5 domains, feature dimension & 2. We vary the number of training samples per client from 5 to 20. The result shows that FedDAR-SA. achieves four orders of magnitude smaller errors than all the baselines: (1) Local-Only where each client train a model using its own data; (2) FedAvg which learns a single shared model: (3) FedRep which learns shared representation and client-specific heads. (4)Separate FedAvg which trains separate models for each domain using FedAvg. The results demonstrate that our method overcomes the heterogeneity of domain distributions across clients. FedDAR- WA fails to converge under this setting, confirming the effectiveness of the proposed second-order aggregation.
5.2 REAL DATA WITH CONTROLLED DISTRIBUTION
Dataset and Model. We use FairFace (Kärkkäinen & Joo, 2019), a public face image dataset containing 7 race groups which are considered as the domains. Each image is labeled with one of 9 age groups and gender. We use the age label as the target to build multi-class age classifier. We created an FL setting by dividing training data into n clients without duplication. Each client has domain distribution म. Dir(op) sampled from Dirichlet distribution. The total number of samples at each client L = 500 is set to be the same in all experiments. We control the heterogeneity of domain distributions by altering a. The label distributions uniform for all the clients.
7

---

Published as a conference paper at ICLR 2023
18
LOCK Only Sep, Fustin PECDARIGA
Average domain test accuran
FedDAR-SA
FedDAR-WA
12
4.3
18"
42
25
16
32
Marber makes campeesasee
Figure I: Performance under different number of training samples per client, the error bars show the standard error from three independent runs.
Figure 2: Age classification accuracy as a function of representation dimension
Table 1: Min, max and average test accuracy of age classification across 7 domains (race groups) on FairFace with number of clients n 5, number of samples at each client L
500
Method
Max
Min
Avg
Max
Min
0.5
Avg
Max
Min
Avg
Max
Min
100
Avg
Seporate reweighting)
41.8-2.2
4,1+1.3
19.340.5
42.4-06
9,7+0.6
23.5-1.2
4144-0.7
933+0.
23.4+0.8
41,440.4
8.5+16
220-1.0
37.111.1
45.7.0..
18.8102
41.6.0.
45.21088
08.6.007
408_0.7
FedAvglw/ reweighiing)
42.8.006
4588008
19.21033
416.0.3
46.000.
38.310.5
40910.3
43.311
38.411.5
1..5...3
FedAAs Multi-head
46.8=099
22,4428
49.1-0.
249914
400=1.0
51.1=04
34,7402
403200
36.4209
398=0.7
FoIDAR--A
46.5-1.9
34.0+19
405+03
49.9-12
400+00
439-0.2
49.3+0.3
40.0+0.6
427+0.
486+00
40.2+0.5
42.5-0.4
FelDAR-S. SA
48.2.1.0
39,31088
42.410.44
50.0...3
40.6.044
434....
39.0...5
41.210.3
43.7.0.
49,0.0.3
41.210.4
436..0.5
Implementation and Evaluation. We use Imagenet(Deng et al., 2009) pre-trained ResNet-34 (He
al., 2016) for all experiments on this dataset. All the methods are trained for T 100 communi- cation rounds. We use Adam optimizer with learning rate of x 10 for the first 60 rounds and 1 x 10-5 for the last 40 rounds.
Metries and Results. Our evaluation metrics are the classification accuracy on the whole validation set of FairFace for each race group. We don't 't have extra local validation set for each client since we assume the data distribution within each domain is consistent across the clients. In Table 5.2, we report the accuracy averaged over the final 10 rounds of communication following the common practice (Collins et al., 2021). The result shows our FedDAR achieved the best performance compared with the baselines. Note that FedAvg + Multi-head also uses Equation 5 objective for fair comparison.
Effect of k. The limitation of using FedDAR-S SA instead of FedDAR- RA is the need of tuning the dimension of representation k. Figure 5.2 shows results of the average domain test accuracy with different k. We can that FedDAR-SA can achieve better accuracy with properly chosen k. We use = for all results with FedDAR-Sa. SSA in Table 5.2.
Robustness Varying Levels of Heterogeneity. From the result with various a, we can observe that the performance of FedDAR-SA is very stable no matter how heterogeneous the domain mixtures are. However, the baselines' accuracy decrease when becomes smaller.
5.3 REAL DATA WITH REAL-WORLD DATA DISTRIBUTION
Dataset and Model. We use the EXAM dataset (Dayan et al., 2021), a large-scale, real-world healthcare FL study. We use part of the dataset including 6 clients with a total of 7,681 cases. We use race groups domains. The dataset is collected from suspected COVID-19 patients at the visit of the emergency department (ED), including both Chest X-rays (CXR) and electronic medical records (EMR). We adopt the same data preprocessing procedure and the model (Dayan al., 2021). Our task is to predict whether the patient received oxygen therapy higher than high-flow oxygen in 72 hours which indicates severe symptoms.
Baselines. (1) methods that learn one global model, FedAAg(McMaahn et al., 2017a), FedProx(Li et al., 2020), FedMin (aapaaaak et al., 2021) along with their local fine-tuned variants; (2) train M separate models with FedAvg; (3) train one global model with FedAvg first, then fine-tune on M domains separately with FedAvg; (4) client-wise personalized FL approaches, FedRep(Collins et al., 2021), FedPer(Ariin Arrivazhaaaaa et al., 2019), AG-Fedavgg et al., 2020), FedBN(Li et al., 2021b).
Implementation and Evaluation. We apply 5-fold cross-validation. All the models are trained for T = 20 communication rounds with Adam optimizer and learning rate of 10-1 The models evaluated by aggregating predictions on the local validation sets and then calculating the area under curve (AUC) for each domain. We also report the AUCs averaged clients' local validation set.
8

---

Published as a conference paper at ICLR 2023
Table 3: AUCS result on EXAM dataset with the domain being race group. Numbers are the means and standard deviations of metrics from 5-fold cross-validation.
Methods
White
Black
Asian
Latino
Other
Min
Avg
Client Avg
Local
.761#.023
8155.005
.83820099
.8892.076
8404.003
.759=0026
.8294.032
.7954.023
separate FodAvg
.796+.022
.94++005
.788+-047
.649-113
8266-004
.606-0080
.751+.026
.759+.027
FedAvg
.83++0027
.854+.04
.887+0022
.844.102
-900-.038
.773-049
.811+.019
856+.000
FedAvg + FT
7884:004
835+.025
.92+.0055
.817..136
.82++.048
-73
.844.0024
8855.0016
FedAvg separate FT
832+0032
846+.043
.03+-025
.69+-0099
.91++0026
.784-055 084
.82++0017
863+-024
FedProx
8344.017
8644.006
.903.0055
.8802.085
.9124.030
-08=.030
88994.023
868#.012
FodProx FT
806+.003
8424.009
.910=.025
.95550085
.88++0031
-29
.866+000
.888.0144
FedMinMax
8391.027
8674.0054
8944..099
.9162.053
.9033.034
822
8844.000
.8721.016
FedBN
787+.027
840+
883+.039
867=.090
852++0043
.766=0013
846+.020
.856..010
FedRep
837+.020
869+-000
-888-042
.91++0083
.10++0028
8112-028
884+-025
867+-013
FedPer
835=.025
8654.073
.999.0077
_96420036
911#:031
813=0047
8874.021
873+.011
LG-FedAvg
.330=.029
.858.0052
.906..032
.902+0050
.903+.033
.814-034 034
.80++0019
8677.0177
FedDAR-W.
.884+.007
1896
902++.034
.952+.041
.28-H.022
.872 .015
912:t:004
898:10006
FodDAR-SA
.888+0044
.895+.038
-28=-002
.939+.046
.918+.016
.868
020
.919t.0114
912+t.001
Average Performance Across Domains and Clients. Table shows the average of AUCs across domains and clients. We can see that our methods, both FedDAR- WA and FedDAR -SA, achieve significantly better performance than all the baselines under both domain-wiss wise and client-wise metrics. The gap between our domain-wise personalized approach and other client-wise personalized baselines shows the validity of learning domain wise personalized models facing diversity across domains. The reason that fine-tuning methods induce worse results is mainly because of the imbalanced label distribution. Each local training dataset doesn't 't have enough positive cases to do proper fine-tuning.
Fairness Across Domains. The AUCs of each specific domain in Table 3, show that our proposed FedDAR method uniformly increases the AUC for each domain. The column of the minimum AUC among domains also verifies that our method indeed improves the fairness across the domains.
Ablation Studies. i) re-weighting (RW): First two rows in Table 2 shows adding sample re-weighting sigmifi- cantly improves the fairness across the domains. The minimum AUC among domains is improved by a large margin (> 0.05); ii) multi-head (MH), domain as imput feautre (DI) and alternating update (Alter): Comparing three blocks in Table 2, we see that adding multi-head alone does not improve results. We conjecture that alter- nating update prevents the overfitting of the heads with limited samples. This is also shown by the result in Table 5.2, where Fe tends to perform badly on certain underrepresented domains especially when domain distri- butions are highly heterogeneous (a is small). Mcanwhile, using domain labels directly as feature input is not as good as multi-heade and not compatible with alternating update: iii) projection (Proj) and aggregation method (AGG): Results in Table 2 shows that using second-order aggregation with the projection of the features gives the best result.
Table 2: Ablation results of different components' contribution in FedDAR.
RW MH DI Alter Proj AGG
Domain Avg A Min
Client Avg
N/A
861/
/.773
.856
N/A
881 /824
.873
N/A
880
7825
.866
WA
885
834
.80
SA
878
826
.871
N/A 8677 / 806
.852
WA
912
.872
WA
.918/
.863
.904
SA
.199 .868
.912
CONCLUSIONS
We propose a novel personalized federated learning framework that assumes the mixture of domain data distribution. Our approach, FedDAR, achieves a balanced performance across domains by learning a global representation and -omaiin-pecific heads, despite the heterogeneity of domain distributions across elients. Our method is effective, as supported by both theoretical and empirical justifications. It has been tested on face recognition and medical imaging FL datasets and can be casily extended to other complicated tasks. However, our method has some limitations: i) it requires the domain information for all samples; ii) it does not consider heterogeneity of label distributions; iii) it has a potentially expensive communication cost caused by sending Hessian matrices, especially when the output dimension is big. We plan to address these limitations in future work, along with other research directions such as improving fairness across domains and exploring the setting where domains are structured, hierarchical, continuously indexed (Wang et al., 2020; Nasery et al., 2021) or multi-dimensio (characterized by multiple factors).

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work has been supported by NIH IR01HL159183,
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint arXiv:1912.0081111 2019.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Fei Chen, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta- weta-learning for recommenda- tion. arXiv preprint arXiv: /802.0787, 2018.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. International Conference on Learning Representations, 2021.
Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic eneryption for arithmetic of approximate numbers. In International conference on the theory and application of cryptology and information security, pp. 409-437, Springer, 2017.
Lingyang Chu, Lanjun Wang, Yanjie Dong, Jian Pei, Zirui Zhou, and Yong Zhang. Fedfair: Training fair models in cross-silo federated learning, arXiv preprint arXiv:2109.05662, 2021.
Francesco Ciompi, Oscar Geessink, Babak Ehteshami Bejnordi, Gabriel Silva De Souza, Alexi Baidoshvili, Geert Litjens, Bram Van Ginneken, Iris Nagtegaal, and Jeroen Van Der Laak. The importance stain normalization colorectal tissue classification with convolutional networks. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp. 160-163. IEEE, 2017.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. International Conference on Machine Learning, pp. 2089-2099. PMLR, 2021.
Luca Corinzia, Ami Beuret, and Joachim M Buhmann. Variational federated multi-task learning. arXiv preprint 19066.6666 2019.
Sen Cui, Weishen Pan, Jian Liang, Changshui Zhang, and Fei Wang. Addressing algorithmic disparity and performance inconsistency in federated learning. Advances in Neural Information Processing Systems, 34, 2021.
Ittai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amilcare Gentili, Anas Z Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford J Wood, Chien-Sung Tsai, et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine. 27(10): 1735-1743 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13361,, 2020.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34:6478-6490, 2021.
Wei Du, Depeng Xu, Xintao Wu, and Hanghang Tong. Fairness-aware agnostic federated learning. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pp. 181-189. SIAM, 2021.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint arXiv:2002.09948, 2020.
10

---

Published as a conference paper at ICLR 2023
Borja Rodriguez Gålvez, Filip Granqvist, Rogier Dalen, and Matt Seigel. Enforcing faimess in private federated learning via the modified method differential multipliers. NeurIPS 2021 Workshop Privacy in Machine Leurning, 2021.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015.
Lidia Garrucho, Kaisar Kushibar, Socayna Jouide, Oliver Diaz, Laura Igual, and Karim Lekadir. Domain generalization in deep learning-base based mass detection in mammography: A large-scale multi-center study. arXiv preprint arXiv:2201. 11620, 2022.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586- 19597, 2020.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: survey. IEEE Transactions on Biomedical Engineering,
2021.
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.05516 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33:2304-2315 2020.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangehuan Liu, Jian Pei, and Yong Zhang Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7865-7873, 2021.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on paltern analysis and machine intelligence, 16(5):550 554, 1994.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 702-712. PMLR, 2020.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv: 10999//220 /2488, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint arXiv: 1912.04977, 2019.
Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling shuffling. In International Conference on Machine Learning, pp. 5213-5225. PMLR, 2021.
Kimmo Kärkkäinen and Jungscock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv: 1908 04913, 2019.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient-based meta-learning methods. arXiv preprint arXiv:1906.0277 027017 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-baas based learning applied to document recognition. Proceedings of the IEEE, 26(11):227-22224 1998.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint arXiv: 10497, 2019.

---

Published as a conference paper at ICLR 2023
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6368. PMLR, 2021a.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fed(bbn: Federated learning on non-(iid) features via local batch normalization. International Conference on Learning Representations, 2021b. URL https:tt///c..... /tpp//pprrreeee ew. net. forum? d=6YEQ0 0Qg0gnnn@TTO CG.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis- Phillppp Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
Zhengquan Luo, Yunlong Wang. Zilei Wang, Zhenan Sun, and Tieniu Tan. Disentangled federated learning for tackling attributes skew via invariant aggregation and diversity transferring. arXiv preprint arXiv:2206.068/ 2022.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.
Othmane Marfoq, Giovanni Neglia, Laetitia Kameni, and Richard Vidal. Personalized federated learning through local memorization. arXiv preprint arXiv:2 1tt:2/111.99 11.09300. 2021.
Gustav Mårtensson, Daniel Ferreira, Tobias Granberg, Lena Cavallin, Ketil Oppedal, Alessandro Padovani, Irena Rektorova, Laura Bonanni, Matteo Pardini, Milica G Kramberger, et al. The reliability of deep learning model in clinical out-of-distribution mri data: multicohort study. Medical Image Analysis, 66:101714, (01114,277 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communiiation-efficieet learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017a.
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963. 2017b.
Aditya Krishna Menon and Robert C Williamson. The cost of fairness in binary classification. In Conference Fairness, Accountability and Transparency, pp. 107-1 18. PMLR, 2018.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- lional Conference on Machine Learning, pp. 4615-4625. PMLR, 2019.
Kevin P Murphy, Probabilistic machine learning: an introduction. MIT press, 2022.
Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time. Advances in Neural Information Processing Systems, 34, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
NHS. 2004.
survey for england 2004, health of ethnic minorities, URL https: bttps://digitae nhs uk ha/ddattaaap aarraa-nnndiiffffoo information/ heal pubicattionsssaatiiiiaaalaltlall wttp:///eaaat-ssseeeoeoerooe-oee for-engaan///
Health
Afroditi Papadaki, Natalia Martinez, Martin Bertran, Guillermo Sapiro, and Miguel Rodrigues. Federating for learning group fair models. arXiv preprint arXiv:2110.01999, 2021.
Meghna Ranganathan and Raj Bhopal. Exclusion and inclusion of nonwhite ethnic minority groups in 72 north american and european cardiovascular cohort studies. PLoS medicine, 3(3)e44, 2006.
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):177, 2020.
12

---

Published as a conference paper at ICLR 2023
Yuji Roh, Kangwook Lee, Steven Euijong Whang. and Changho Suh. Fairbatch: Batch selection for model fairness. arXiv preprint arXiv:2012.01696 2020.
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 3273)3370--372, 2020.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworss. In International Conference on Machine Learning, pp. 9489-9502. PMLR, 2021.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Ala Szezepura. Access to health care for ethnic minority populations. Postgraduate medical journal, 81(953):141-1447222 2005.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394 -1400,, 2020.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International Conference on Machine Learning, pp. 10434-10443, PMLR, 2021.
Paul Vanhaesebrouck, Aurélien Bellet, and Mare Tommasi. Decentralized collaborative learning of personalized models over networks. In Artificial Intelligence and Statistics, pp. 509-517. PMLR, 2017.
Roman Vershynin. High-dimenssooall probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.01807, 2020.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Fichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv: 1eXivv:9100 1910. 10252, 2019.
Michael Wick, Jean-Baptise Tristan, et al. Unlocking fairness: trade-off revisited. Advances in neural information processing systems, 32, 2019.
An Xu, Wenqi Li, Pengfei Guo, Dong Yang, Holger Roth, Ali Hatamizadeh, Can Zhao, Daguang Xu, Heng Huang, and Ziyue Xu. Closing the generalization gap of cross-silo federated medical image segmentation. arXiv preprint arXiv:2203. 10144,014444 2022.
Wenjun Yan, Yuanyuan Wang, Shengjia Gu, Lu Huang, Fuhua Yan, Liming Xia, and Qian Tao. The domain shift problem of medical image segmentation and vendor-adaptation by unet-gan. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pp. 623-631. Springer, 2019.
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation. arXiv preprint 2XXx:20022047588 2020.
Xubo Yue, Maher Nouiehed, and Raed Al Kontar. Gifair-fl: An approach for group and individual fairness in federated learning. arXiv preprint arXiv:2108.0274: 2021.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics, pp. 962-970. PMLR, 2017.
Valentina Zantedeschi, Aurélien Bellet, and Mare Tommasi. Fully decentralized joint learning of personalized models and collaboration graphs. In International Conference Artificial Intelligence and Statistics, pp. 864-874. PMLR, 2020.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
13

---

Published as a conference paper at ICLR 2023
Yuchen Zeng, Hongxu Chen, and Kangwook Lee. Improving fairesss via federated learning. arXiv preprint arXiv:21 10.221100110001 10.1100.55 15545, 2021.
Daniel Yue Zhang. Ziyi Kou, and Dong Wang. Fairfl: A fair federated learning approach to reducing demographic bias in privacy-sensitive classification models. In 2020 IEEE International Conference on Big Data (Big Data), pp. 1051-1060. IEEE, 2020.
Han Zhao and Geoff Gordon. Inherent tradeoffs in learning fair representations. Advances in neural information processing systems, 32, 2019.
Chen Zhu, Zheng Xu, Mingqing Chen, Jakub Konečný, Andrew Hard, and Tom Goldstein. Diurnal or nocturnal? federated learning of multi-branch networks from periodically shifting distributions. In International Conference on Learning Representations, 2021.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2023
A FEDDAR FOR LINEAR REPRESENTATION
A.1
SETUP
We retain the setup for linear regression considered at the start of Section 3.1. We additionally define W [wi,... , -wyy"" E R RM as the concatenation of domain specific heads. For notational convenience, we let m Vi,m) denote an (input, output) sample coming from client and the m-th domain. To measure the distance between any two matrices A, B with the same dimensions, we use the principal angle distance (Golub & Van Loan, 2013), given by distí A, B) _ || || B Bll2. where A_ denotes matrix whose columns form basis for the orthogonal complement of the range of A. To simplify the analysis, we further make the following assumptions.
Assumption A.1 (Sub-Gaussianltty) For each E [M] and E [n]], the samples Zi,m. Rd are independent, mean zero, have covariance La. and has subgaussian norm i.e. for every E Rd. Elexp(v mmm) )] I exp([|=|]2/2
Assumption A.2 (Domain diversity). Let miin, A Omin
W*), i.e., Anin, the minimum VM
singular value of the head matrix. Then min, > 0.
Assumption A.3 (Ground truth normalization). The true domain parameters satisfy 1 Vi < l for each [M], and B" has orthonormal columns.
S
All the above assumptions aim to simplify the theoretical analysis whilst only imposing mild con- straints on the data distribution and the parameters of the target functions. Similar assumptions have also been adapted in prior work (Collins et al., 2021).
A.2 FEDDAR ADAPTED TO LINEAR REGRESSION
We analyze an adapted version of our FedDAR algorithm. Since the linear regression problem has an analytic solution, ease analysis, update the heads {w, JM at the server in closed form using local gradient information. Meanwhile, we update the representation B by taking a step using the averaged local gradients. Algorithm 2 shows the procedure of this adapted version.
The local objective for i-th client in m-th domain at 1-th iteration, fi. (Wm: B¹) is defined as the following,
L (wi... w. TT x m
fi (Wm: B³)
A
where L is the number of samples from domain at client i. We assume in each iteration the data points {m vi.m hee(r, are all newly sampled from the distribution. We denote I = =mm L Note that since the objective function has quadratic form, thus its gradient w.r.t either w.n or B has a linear form of A,m Wm. a;,n or Ct.m Ci.m which we write down explicitly in Appendix B. After every global update of the representation B, we apply an additional QR decomposition to normalize it to be column-wise orthogonal.
A.3 CONVERGENCE ANALYSIS
We first provide brief proof sketch. Overall, our approach largely follows that in Collins et al. (2021), with few differences needed handle the spreading of a domain's 's data across different clients. We note that we also tightened the analysis compared with Collins et al. (2021), such that each domain only needs (Ak) samples as opposed to O(k²) samples as in Collins et al. (2021) (where the requirement for each client to have Okk²) samples since they considered the case where each client has a separate head). This can yield significan improvement when k is moderately large and there is data imbalance.
1. First, in Lemma A.5, we show that our estimated weight matrix W++1 e R M (which is our estimation at time t + 1 of the true domain weights matrix W*) satisfies the relationship
w W (B*B") B F,,
15

---

Published as a conference paper at ICLR 2023
Algorithm 2 FEDDAR for linear regression
Input: Step size 7i number of rounds T each agent i E [n]] collects L samples, and sends Z =
Client initialization:
Eist 1y ) x; ( to the server.
Server initialization: finds UDU rank-k SVD( nT. >, Z.); sets B t U.
for := 1.....,, do
Server sends current B' to clients. Client computation for W++1: for client E [n] do
Selects L new samples {(0] y.)}.
Computes V J.m (Wm, Bd = A A Wm ai for each domain [M].
Sends (A al Dt mm back server.
end for
Server update for W Server chooses w { (wm E RR. V um E.
-. fi (Wm, B³)) =0]. vm
(M].ie.. w that satisfies
(E; A (im)w = =E,
Sends W = [w],... WMM
T E Mx to clients.
Client computation for B
for client E [n] do
Selects L new samples f2/ v/
Computes VnJf. (wo B²) C B' - ci.m for each m E [M].
i,
Sends (VBfi.m (1 BB), m back to server.
end for
Server update for BC+1;
Server computes B +- B // Em E, D. E V VBfmmwwt B).
Server performs QR decomposition B R++1 =QR(BLT B(+1).
Server updates B <- B end for
where B' our estimation of the true (high to low dimensional) representation embedding show be bounded (sufficiently small)
B' time t, and F, is error term that in terms of dist(B) B B*), ), with the scale of the error depending on the (random) number of samples Lm. seen at time for each domain m [M]] Bounding Fell in our setting requires some care since the samples for each domain are spread over many clients. We
refer the reader to Section A.4.2 for the details.
2. Second, we show in Section A.4.3 that the update for B satisfies the relationship (see equation 21)
B+ = B " (1Q') W M HQ:
M
where Q" : W T (B") T and Hq denotes an error term which can be shown to be bounded (sufficiently small) in terms of dist(B', B²). Further simplifying. we have
d BB) ( B -B1) B'
(B'(WA+I)T B3 (W")) w
17
H1a)
M
M
dist B BB)
" (B1) B ( Tw++__ (B1) HQ
M
M
dist B", B*) dist(B', Jo (W++1)
"1
"
(B") ) HQ
M
M
By upper bounding | || in terms of dist(B' B²) and providing an appropriate lower bound on amin (W W.+1), can then show by picking r sufficiently small and under linear rate (see Equa- tion 29). Again, a difference from the analysis in Collins et al. (2021) is our handling of bounding |Hll since the samples for each sample are spread over different clients.
other suitable assumptions, the quantity dist(B', B*) decays
16

---

Published as a conference paper at ICLR 2023
3. An issue created by the spreading of samples for domain across different clients is how to pick an appropriate sample size such that the [| and || terms can suitably bounded. In Lemma A. 10, we prescribe a suitable sample size for each client such that each domain gets sufficient samples (with high probability) for the purposes of our analysis.
We now proceed with detailed analysis.
We first present a theorem that states our adapted FedDAR(Algorihm 2) enjoys linear convergence. The theorem is followed by multiple remarks which highlight key detailed points of our convergence result.
Theorem
A.4 (Algorithm 2 convergence), Define Eo
dist (B BB). maxx,
Omax
JM W
mmin, == minn
VM
W*)
Let Gmax, Jmin,-
Suppose
dk2 k4
12
))
L L Ω
max
(9)
n.
ES
mi (2 Ti,m)
Then, for any and any & 1/(44 with probability at least Te-80
dist(B", BB) (1 1oo min, //2 B*).
(10)
Linear convergence speed: The convergence of BT to B is linear, assuming that (1) miin 1 W*) and that (2) - nEg min E (0,
7M
Initialization of B": For our convergence result to be meaningful, we need dist( B²) to be close to O. We show in Appendix A that algorithm's 's choice of initial 8 ensures that dist B B*) close enough to whilst preserving privacy. When the number of samples is uniform across the domains, this comes only at the cost of a logarithmic increase in sample complexity.
Sample complexity: The per-iteration sample complexity per client is L. We note that in the requirement for (9), need that L 2 (idk2 x /²/n)); this comes from the updates for B E Rdxk While we expect that d could be large, a large number of clients helps to mitigate the increase in sample complexity arising from d. We also need L 2(k2 x2 2i=1 Tt,,) for every domain [M]; this requirement comes from the updates for w for each of the M domains.
A.4 PROOF OF THEOREM A.4
A.4.1 ANALYSIS FOR UPDATE OF W4++1
Since we are analyzing the update step for any iteration t, unless necessary we drop all superscripts. Let Lm Ei= Lim denote the number of samples from domain [M]] across the clients.
Then, we can express V um E Si.m (Wm, B)
Lum
fimm (am(Wmm/ B)
Σ
w B x
B x.mm
i=1
j=l
Since
p (BB')
x
it follows that following Algorithm 2,
12 Leam
n
Laa
B
J (B)
w
[+1
B T a ( (mr TB² win- (11)
T
I
Gm
Reexpressing, assuming Gm is invertible, have
n
Li.
w6-1 =B BB
BBBB'
G-1
B
T B²
w
B B w
Lm
f,m
jml
(12)
17

---

Published as a conference paper at ICLR 2023
Intuitively, assuming Lm is large enough,
xd 6,m
(m),m) La.
Lm
1=1
Hence,
Lim
Gm
iim (I)) 1,m1
B"
w
B B'
Lm
i=1
j=1
This then implies that
W = "B''"" F...
(13)
where the m-th row of
n
F
G
B xi.m (a),m)
BB") wm
B B",n
Lim
Note the similarity of equation 18 to (17) in (Collins et al., 2021). Following a similar analysis (Collins et al., 2021), we should also be able bound the Frobenius norm of F in terms of dist B, B.).
Below, formalize the argument. First, we have the following lemma.
Lemma A.5 (Update for W++1), For each time t, let L == Σ Lim denote the number of samples from domain m E [M] across the n clients at time t. For convenience, we drop the time index unless absolutely necessary. We define the terms
Limm
Lim
Xm
(T,
G=
B a (T) i.m
Lim. i=1
Lm 1=1
j=1
Then, assuming that Gm is invertible, the update for W takes the form
W W TB B+F
(14)
where the m-th row of F is
FT = (Gml B Xm BB B" win)
(15)
Proof. We can express V Vw... E fim (w B) as
n
Lim
( (wm, B) Σ
(w,, B x i,m
Vm B x
i=1
Since
Vr.n (wi)) ( s.m
it follows that following Algorithm 2,
T Li
n
Li m
xx f.
(xr
(m)
(B)
w
41 m
a ao/_maa? (T[ win- (16)
L
Gm
Reexpressing, assuming Gm is invertible, we have
n
Lu
wL+1 BBBBu " +
G-1
B
1
(xd (m)
B B'w
Lm
s,m
(17)
18

---

Published as a conference paper at ICLR 2023
This then implies that where the m-th row of F is
W = - B B+F.
(18)
Lim
F
G
B T. s,m
1.m
B² win
B B'w w
Lm
i=1 j=l
Xm G.
Gm B w'm)
(G X Bm B' G B X BB B )"))
(G (B Xm BB BB) BB) w
A.4.2 BOUNDING ||F]|
We will proceed to bound the Frobenius norm of F. We begin by showing that Gm exists and (both lower and upper) bounding its spectral norm.
Lemma A.6. Let Luiin [M] Lm- Let 8k 10C Moo(oo for some absolute constant
Vermin 880 80kk (n)(M) Gml exists for each
C. Suppose that 0 S 8% 1. Then, with probability least 1 e m E [M), and
1
[G=l122 S
Vm E [M]. 1-
Proof. Note that
12 Limm
Gm =
(
B BJ x
Lm
B T m. Since BT B 7, it follows that each vi is i.i.d I-subgaussian. Then, applying the same argument in Theorem 1 of Vershynin 2018, we have (cf. equation (4.22) in Vershynin 2018)
Let pm
v
Miin (Gm) -
(19)
VT.
VTem
8u.m
with probability at least 1= for 2 and some absolute constant C, assuming that 0 S 6k.m S Consider the choice = =9VK -9Vkk/logMM)) log(M). Then,
8k.m = C ( Vlm
VI 9Vhlog(M))
< 100 Vlllo M < 10C
Vlylog M
+
VLm
vLm
VLmin
Suppose choose Teuin such that 5k.m < 1. Then, taking union bound, with probability least 1 Me = = =1- .M exp(-81k) log(M))=
logg
10C / M Vimin > O Vm E [M].
Tmiu (Gm) 2 8k.m 2
(20)
Therefore, with probability at least - -xxp(- 801 log(M)), G..! exists for every m E [M], and in addition,
1
||G ||2 5
Vm [M].
1 1-
19

---

Published as a conference paper at ICLR 2023
We next bound the operator norm of term B' X -B BB"
Lemma A.7. Let Lmin = mi Lm. Let òj: == JOCvkylog for some absolute constant C.
V.mmi
Suppose Lmin such that S 8x < 1. Then, with probability at least -50k log
|B Xm BB BBB)BBll I dist(B",
Proof. We will use an e-net argument, similar to the proof Theorem 4.6.1 in (Vershynin, 2018). First, by Corollary 4.2.13 in (Vershynin, 2018), there exists 1/4-net A of the unit sphere S with cardinality N - 99. Using Lemma 4.4.1 in (Vershynin, 2018), we have that
B Xm BB* B
max ( Xm (I BB B*) =,=)|-
zEN
To prove our result, by applying union bound over m E [M], it suffices show that with the probability at least 1 c 1004 log M
6km
max (( X BB BB)BB 2, 2,2)| - =EN
Vm E [M],
2
where we recall that
kk.m C V 9Vklog(M)
S8k.
Tim
VLm
We will assume that minm Lm := Liinn 2 1 is chosen large enough such that 6R.m S 1.
For a fixed & E 5 observe that
1 n Lm
Liam
(B Xm BB BB) 2,
BT (o) T BB BB"BBB) 2,2)
j=1
Lim
(2 Tu. ( ((1)))))) T2),
Lm
1,m
j=1
where we defined "n.m= == B TT am and vi (B*) BB )ark
Since each a!_m I-subgaussian, |B = 1, and I1(1 BB B = dist(B', B), it follows that IT "e um is subgaussian with norm most and (v/ T is subgaussian with norm at most dist B", B). Thus, the random variable ai... = (= T= w/ ((v? (m T2) (for fixed unit 2) is sub- exponential with sub-exponential norm at most dist B" B). Moreover, note that af is mean-zero, since
E T (m 11 = =
x 1.m (x.))) ) (LB-B"" B
B -1-110 -B B
as m is assumed to have identity covariance. Thus, the ag 's are i.i.d mean-zero subexponential variables each with subexponential norm most dist(B B", B). Hence, by Bernstein's 's inequality (cf. Corollary 2.8.3 in (Vershynin, 2018)),
IP (( XI -B B 2,
6R dist(B' B"
2
2
Li.
Sa dist
=
a.m
2
1=1
Skm dist BB,B),
(os BB, (
< 2exp
-emin(
)m
dist Ba, B)
dist(B", B)
= 2expl
-co2 Lm)
2eepp(-cC' (k: 81k log M) log log(M))).
20

---

Published as a conference paper at ICLR 2023
Above used the assumption that 0k,m S 1 to simplify the minimum operator in the exponent. Taking a union bound over each z E N, follows that
PP
[IIB*
Xm (I BB B B 8k,m dist , B))
$ PP 2 max =EN
(B X (I BBB)BB") 2, 2,2)|
2 on B)
$ 2 9
(k: + 81k log(A))
S exp(-51kk log M).
where the last inequality follows by picking C large enough (but still is absolute constant). (Here the choice of 51 in the exponent is somewhat arbitrary; any choice smaller than 81 should work). By applying union bound over the domains m [M], this then completes our proof.
We are now finally ready to bound ||
Lemma A.8. Let Lmuin = minme[M] Lmn- Let 8 == 10Cv/k/log(M) for some absolute constant C.
VImin
Suppose that S 8g < 1. Then, with probability at least
2c 200 ())
50k
|F I
(B',,)
1
Proof. By Lemma A.6 and Lemma A.7, we have that with probability at least 2e- 50k log
(Bndist(B" (B" B)). || Xm (T BB B²) s |
B xm BB B*]
1
The proof then follows by recalling that the m-th row, F.T takes the form
F =Gm
B Xm
BB BB) w
A.4.3 ANALYSIS UPDATE FOR B++1
Similarly (Collins et al.. 2021), define
Q' w -B" (W")
Below, we drop the time index and use B, Q, W to denote B¹, Q', and W respectively. Based on algorithm 2, we have that
n
B++1
ww m
D w
M
M
n
J.j.n
(AA WBT) - (Am (A) Tw, Afm. = em (2(2))
M
Lmn
m=1
i=1
f=1
M
Lum
(Al.mm Tw
M
m=
i=1
M
Lum
B
17
x s.m (xr)
am Wm T
M
L
m==1
1=1
f=1
M
Lim
=B
1
2. (2) T
9m WT
17
Q W
(21)
M
M
Lm
M
iml
He
21

---

Published as a conference paper at ICLR 2023
Above, define Om E R to denote the m-th row of Q (viewed column vector). Note again that since
Li (x)
6,00 (2)))) ) La
L
the term Ho in equation 21 can be appropriately bounded. Note the resemblance of equation 21 to (53) in (Collins et al., 2021); ); the crucial difference that we will need to lower bound / o (W*), instead of amin (W*) in (Collins et al., 2021). Thus we should be able to carry out the rest of the analysis similar way to the outline in (Collins et al., 2021) and derive analogous result Theorem in (Collins et al., 2021).
We first bound the error term HQ-
Lemma A.9. Let
M
Limm x
HQS
Tn In (on(w"t)) (m)
- T
M
im
M
m=1
j=1
Let 7kk 20kvd for some absolute constant Suppose that 0 S 7k k. Then, for any with evTL.
probability least exp(-99d) -5e-50k M
|1 "rddisi
Proof. As before, we may omit the time superscript t in cases where it is clear for notational convenience. The proof based on the argument in Lemma 5 in (Collins al., 2021). Again, the main tool is an e-net argument. We first bound || lla and ||20m 12-
Bounding (m With probability at least - -2e -50 log M for each m E [M]. we have that
1m 2
((B' B w" + Fm) B "nn ll2
(BBB(BB) I)B" w + +I Fm
dist(B* B
BB w llz + | |mil22
S Widist(BB' (B', B²) 1 8
dist(B',
$ 2//diist(BB BB).
Above, we utilized the assumption that || ll VT. the orthonormality of B' (which was derived as the orthogonal matrix from a Gram-Schmidt procedure), the assumption that < 5g < 1/2, as well Lemma A.8 which bounds Fm ll2 (for all m) with probability at least 1 2e 50k log
Bounding w,.: Note that for notational convenience, we let wm denote w For each f and every m [M], we have that
20 ++1
1|2 = (BB) B''n", + Fm l2
< lwwm +
1 F
amllle +
dist(B', B
5x
< 3vK,
with probability at least 1 - 2e 26-50k log M, where again we used Lemma A.8 to handle F 12. the assumption that dg < 1/2, and the fact that dist B B²) 2
For the rest of the proof, we condition on the event
E = |1 _ 2Vkdist(BB', B²) and || Wm ll2 - 3vk Vm e [M]]
which holds with probability least -e -5-50k log
22

---

Published as a conference paper at ICLR 2023
c-net argument to bound Ho: Again, note that there exists 1/4-net As. of the unit sphere sk-1 and an 1/4-net Na of the unit sphere S with cardinalities less than equal to 9 and gd respectively.
Note now that by Equation 4.13 in (Vershynin, 2018), we have
12
Li.m
[Hlll2
a 4,m
(ar),))
T 4m w
1 TTw M
M
Li
M
Iaam
< 2m
max
Σ
qo w u, v) 4m w Tm
vEN M
Li
m=1
i=1 j=l
M
27 max WENG.DEN,
Σ m=1
1
Σ
( xk.)) (lor am (m (9m u, 0²)
M
Lm
(22)
Fix now a E Na and E Nr. Note now that (u x. ) (m)) 9m (m
subexponential with norm less than or equal to 4l (mll//ll222 S 6kdist (B, B²), since the product of two subgaussian variables u a and (ar,,) qm WmuW with subgaussian norms bounded by and ||gm haalll///w//lll respectively. Note also that
E
[
ar (1or Im t
E [( Wm u, w>}
Thus, by Bernstein's 's inequality, carrying on from equation 22, we have that
1
[" (uTx(..) ((0))))) on wn)) (am w 2) 20)
P
M
Lm
m=1
i=1
< exp
-enLmin
6kdist B B-)
(dist( B*) B*)
S exp
-enL
/kd (,,)))
where we will choose p such that B S to simplify the exponent in the way we did, and e is an absolute constant that may change from line to line. Above, we also used the fact that ΣM Lm nL (recall that L is the total number of samples per agent and there are n agents).
Consider the choice
0 = 10 Avdddis((',, B"),
cVnL
Then,
Lr.
ar a ((ar Om wm)
{Ai w. u. 20)
IP
M
Lm
m=
(=1
<
exp
-onL
kdist B¹, B*)
expp eex 1opp-11000
100d).
Taking union bound over all N₂ and Nr. it follows then that
Hollz 2 2p)
P
< gd+h exp(-100d) I exp((-9d))
17
where above used the fact that k.
23

---

Published as a conference paper at ICLR 2023
A.4.4 COMBINING EARLIER ARGUMENT: CONVERGENCE OF FEDDAR
As seen in Lemma A.8, require that miiu [i] Im to be lower bounded. However, since Lm is stochastic variable, we are unable to directly lower bound Below, we provide a result that converts lower bound each client's 's sample size L (a deterministic quantity high-pun bigh-probability lower bound on Liin-
control) to
Lemma A.10. Let Limnn minme[M] L.n- For any a > 0, suppose that for each m [M]
182 log M
16
2a
L L2 max
n Di=1 Ti,
-i= Ti.m
-i=1
7imm
Then, with probability at least exp(-90)
Lmiu 2 a
Proof. Note that
n ((domain(z)) = m).
Lm
jm
which is a sum of nL independent random variables bounded between and 1. Moreover,
E[Lm]
Tim
where a. is the probability that datapoint comes from domain m for client i. Note finally that
E E[ (1dommain(z?)= m)) Tim.
Hence, by Bernstein's 's inequality, it follows that for any s > O,
82
Limm I Σ Timm L -
Sexp
S exp
Ei=
Σj Tim + 8/3
Since we wish to perform union bound over the M domains, we seek to choose s and such that
$2/2 Di=1 Ej= Tim 8/3
exp
S exp log M),
so that
82/2
M exp
< M (-91 M) S exp (-90 log M). Σm E/m1 Tim + 8/3
To this end, note that we need
$212
2 91 log M
Ei=1 E/=1 Tim 8/3
2 2 2. 2.911 M
Tim + s/3
j=1
182 1822og log M
182 log M
182 log M
Σ Timm +
3
3
Suppose we pick L such that
Tim L 2 182 log M,
24

---

Published as a conference paper at ICLR 2023
so that
182 log M
182 log M
n
182 log M Σ Tim L
+
S
Σ
Trm
3
3
Then, by picking s 2/E Tim L, it follows that
$2/2
exp
$ exp exp((191 (-91 M),
Ei Li=1 2==11 Tim + 8/3
such that for each E [M],
12
P
Lim S
Ts L Σ TimmL
S Eexp(-91 log M).
i=1
1=1
By choosing such that
n
Tim 4,
it follows that
Eis Ti,mL
Lim S
S exp(-991 log M).
The result now follows by choosing such that also satisfies
D Kim
2 2 a
for each m.
Lemma A.11 (Descent lemma). Define Eg == 1 dist? B²) and Tmax, = T and min,, == Omin
VM W'
) Let Fmax. Jeinn
Consider any iteration
VM
Suppose that
400dk2
L
(23)
TCC
(min { /... (256/22222 5x2)))²²
where O is absolute constant. Suppose also that
2 (100Ck log M)
182 log M
16
[min 1. /.880/(25555
L 2 max
Σ Ttm
Σ Js.
Σ
Ti.
which by Lemma A.10, ensures that with probability least -90
L 2 (100Ck log M)
(24)
(min { 8 (25 5x2)))²²
where L =minmEiM] Dt denotes the minimum number of samples from any domain ieration I, and C > is an absolute constant.
Then, for any 17 S 1/4422 (.xx,)))) we have
( BB) < - 1) Egg m2)) /2 (iistt(B (B', B"),
with probability at least -80
25

---

Published as a conference paper at ICLR 2023
Proof. We begin with the observation that
W+1 -W(("" (B") ( F
B++ = B² 2 (Q') (Q'))@++!____ W -Ho-
where
Q' = - (W*)(B")'
and
M
Lim x
11
Σ m=1
1
(x)) ( ) fmw
(Q') W++1 M
7 (Q0')TTt1
= M
1=1
Above B++1 denotes the estimate of B before we perform the QR decomposition. We note that the updates for W and B are exactly analogous the updates for W and B as seen in the proof of Lemma 6 in (Collins et al., 2021). The only two differences are
1. The definitions of F in our paper and (Collins al., 2021) are slightly different. However, in both cases,
8k 8%
| -
dist dist (B" B)])
for some term 6₁. 1/2 with high probabilities. In our case, this event holds with probability
at least 2e -50 log M whilst in (Collins et al., 2021), the event holds with probability
least exp(-11012 log n).
2. The update for B in (Collins et al., 2021) takes the form
T " (1 --A11(Q') - -0¹) w4+1 rn
( B++1 = -
- (Q') 1---('-"" Tn
where < r S 1 is a ratio term used in (Collins et al., 2021), and m above represents the number of samples used by each learner in (Collins et al., 2021) (which is different from our use of m as an index over the domains). However, we note that with high probabilities,
H2ll2 S (y
7 1 A'A(')) -Q Ø¹)
W I mydist((" (B', B*),
rn
where the definition of 7% in both papers differ but both satisfy the assumption that S k.
Due to these similarities the updates for W and B++1 with the update (Collins et al., 202 1), the proof of this lemma follows naturally from the proof of Lemma 6 in (Collins et al., 2021), by plugging in y (Q') W in the update for B in place rg 2021). In particular. following the same analysis as in (Collins et al., 2021), we see that on the events in Lemma A.8 and Lemma A.9, following the equation immediately after Equation (84) in (Collins et al., 2021), we have
2 IQ*) W++1 in (Collins et al.,
(1- (1-18 11-no m/n Ey + 21)
8
dist BB)BB) S
max,+
dist B' B*),
V1 41 (1- max
0z)2
where in our case 5g = 6x + 7o.
M6. Then, by choosing
de 16Eb/(25 5x?),
(25)
it follows that 0g < 1/5, and so
0z 0z)
5k
1 - n min, Fo + 2n
max, < - 4n
5 -
min, e/2,
maxx,
26

---

Published as a conference paper at ICLR 2023
in equation (85) in (Collins et al., 2021), such that
dist(B (B*+) BB) S (1 Eig /2st B²).
remains for us understand what the constraint on 5g spelt out in equation 25, and the constraints on 6% and ?/ (in Lemmas A.8 and A.9 respectively) mean in our choice of the sample size I. for each agent, and the domain size Ly at each iteration. Observe that we need
6% 10C LOCVlylogg KV/loo M
(26)
Vimin
2
20kvd
76 nV
S 2
(27)
10CVly M
20kvd
5 6% + Y
I (25
(28)
VLmin
eV'L
where c, C are absolute constants. By choosing
Lini 2 (1g0Cklog M)
(min (2. SE, (25 5)t
400dk
L
(min ( 82/ 5n
we ensure that the requirements in equation 26, equation 27 and equation 28 are all satisfied.
The final result then follows by applying Lemma A.10.
This then yields the following convergence result, which is more complete statement of A.4. Theorem A.12 (Convergence result for Algorithm 2). Define Eo == 1 dist d ( and maxx = Faa
VM (*)
and Tmin, == mmin (/SW'') )r) Let K. == mmaz,
Suppose that
(400dk2 L
1
NCC
min ( (25 (25-5k2)}
where is absolute constant. Suppose also that
log M
16
2 (100Ck log M) min (1/2,8/
L 2 max
Ei= Tim Σ 7m
Tim
Then, for any 1) S 1/(40 max,+) we have
(1 /2) distt (B'sttB"")) BB)) dist B²)
with probability al least -80 Then for any T and any 1) S 1/ 1/440 we have
dist (B', B²) < - (B"', B*).
(29)
with probability at least Te
By assuming that min. the bound in Theorem decays exponentially. We note that the total number of samples required per client scales with L Log(1/e). In addition, order for the result to be meaningful, we implicitly assume that Ep is close to such that
0 1 Ego
min <
To do so, we note it is possible choose 8 such that dist B²) is close enough to 0, with only a logarithmic increase in sample complexity when the number of samples is uniform across the domains. The argument follows the proof of Theorem in (Tripuraneni al., 2021).
27

---

Published as a conference paper at ICLR 2023
Theorem A.13. Suppose Assumptions A.I, A.2, A.3 all hold. Suppose also that 2,
0.d (1 Ia)
independently for all E [n]. Suppose each client sends the server Z; = "== (v; ) (x²))
0.
as well as the integer value of Li such that the server can compute == n.. Ei=1 Ze Then, the server computes U UDU + rank-k SVD (2), and sets B = U. Lel
1.
A
1
ΣΣ
wmtt. (mtt,)((Wmt..) (mm())))) ==
nLo
where j) denotes the sample of the j-th sample from the i-th client. Let mmin, mnin (A), and let o max,- == T max((A) Suppose that L 2 epolylog(d, d, nL Jo dk? (no Then, with probability least (nI..)-100) have that
max, m k2dd
o min, nL
In particular, when the number of samples is uniform across the domains, we have that
K kk22d
dist Bist Sō
nT.0
where recall that == Fmax, / min,,, and
1 W+ M
m
O max
Ouin,
Omin
W VM
Proof. We omit the proof since it is slight variant of Theorem 3 in (Tripuraneni et al., 2021). For completeness, note that in the case when the number of samples is uniform across the domains, some algebra shows that
k2
dist( B²)² sō
2 min, TT.
However, since k/4M || 13 < kM max,+: we have that
S 4x²
02
min,
max,*
which proves the last statement in the theorem.
28

---

Published as a conference paper at ICLR 2023
B ADDITIONAL EXPERIMENTAL RESULTS
B. EXPERIMENTS FAIRFACE DATASET FOR GENDER CLASSIFICATION
Table 4: Min, max and average test accuracy of gender classification across 7 domains (race groups) on FairFace with number of clients n 5, number of samples at each client L; 500.
Task
Method
a = 0.1
0.5
a =
a
100
Max
Min
Avg
Max
Min
Avg
Max
Min
Avg
Max
Min
Avg
FedAvg
92.0
71.7
83.9
89.8
77.6
84.5
91.0
77.4
84.2
90.5
77.1
84.7
FedAvg + Multi-hean
90.2
48.7
78.9
89.2
77.8
84.1
91.6
76.8
83.9
91.1
77.5
84.5
Gender
FedDAR-W-
89.8
53.4
80.9
91.5
76.7
84.3
91.2
76.1
84.3
90.0
76.8
84.1
FedDAR-SA
92.2
73.4
85.1
91.3
78.1
85.2
91.4
78.2
85.1
92.2
78.1
85.6
We also conduct experiments for gender classification FairFace with the same settings. The best representation dimension is k = for this task, probably due to the smaller diversity across the domains. We can see that the results shown in Table 4 have similar trend as the results in Table 5.2.
B.2 EXPERIMENTS ON DIGITS DATASET
Table 5: Min, max and average test accuracy of digits classification across 5 domains with number of clients 5, number of samples each client L,
500.
Method
=
0.1
a = 0.5
=
1
=
100
Max
Min
Avg
Max
Min
Avg
Max
Min
Avg
Max
Min
Avg
FedAvg
97.1
60.7
80.6
97.2
64.3
81.7
96.1
74.8
85.2
96.8
71.0
85.1
FedAvg + Multi-head
94.3
26.5
55.9
94.3
44.8
68.3
94.1
56.7
74.6
95.0
52.3
74.5
FedDAR-WA
97.3
52.3
79.8
97.3
64.7
83.1
96.6
74.5
86.3
97.1
70.6
86.3
We perform additional experiments on the digits dataset with five data domains with feature shift (Li al., 2021b). Details are described in the following paragraphs. From Table 5, we can that FedDAR- WA outperforms FedAvg consistently except in the case where domain distributions are extremely heterogeneous (a 0.1). In this case, each client tends to have data from only one domain. It is difficult for the proposed method to learn a good domain-specifie head for the domain with the most different data (more obvious feature shift) under this circumstance. For other levels of heterogeneity, although the min and max domain accuracies are similar between FedAvg and FedDAR-WA, the average accuracies are improved as result of the domain personalized model. On the other hand, without an alternative update of the head and representation, FedAvg + Multi-head will overfit quickly. We don't 't include the results of FedDAR-S: here because using representation dimension A 64 causes numerical instability during head aggregation and failure to converge. While using representation dimension k < 32 leads to lower accuracy.
Datasets. We use the same digits dataset containing five different data domains as (Li et al., 2021b). Specifically, we use SVHN (Netzer et al., 2011), USPS (Hull, 1994), SynthDigits (Ganin & Lempitsky, 2015), MNIST-M (Ganin & Lempitsky, 2015) and MNIST (LeCun et al., 1998) as five data domains. Similarity to the experiments on FairFace datraset, the training data is divided into clients without duplication. Each client has a domain distribution n. Dir(ap) sampled from Dirichlet distribution.
Implementation Details. We adapt the codebase from (Li et al., 2021b). A 6-layer CNN with convolutional layers and 3 fully- cully-connected layers is used, with the last layer as domain We use SGD optimizer with learning rate 10 and cross-entropy loss. The batch size is set to 32, and the total communication rounds is set to 100. For each method, we first train the model for 10 rounds with 1 local epoch using FedAvg as warmup. The accuracy shown is the average over the last ten communication rounds. We repeat experiment for each setting three times with different random seeds and report the averages.
head.
29

---

Published as a conference paper at ICLR 2023
B.3 FURTHER EXPERIMENTAL DETAILS
B.3.1 SYNTHETIC DATA
For the synthetic data experiments, we adapt the code from (Collins et al., 2021) and follow a similar protocol. The ground -ruth matrices W E R and B* E R are generated following the same way as (Collins et al., 2021) by sampling each element from i.i.d. standard normal distribution and taking the QR factorization. The same L samples are used for each client during the whole training process. Test samples are generated in the same way as the traning samples but without noise. For all the methods, models are initalized with ramdom Gaussian samples. We a 0.4 for experiments in Figure 5.2.
B.3.2 REAL DATA WITH CONTROLLED DISTRIBUTION
Implementation details. We use Imagenet(Deng et al., 2009) pre-trained ResNet-34 (He et al., 2016) for all experiments on this dataset. All the methods are trained for T 100 communication rounds, with 20 rounds of FedAvg as warmup. For FedDAR-W WA and FedDAR-SA, 5 epochs of local updates are executed for both heads and representation each round. For the baselines, 5 epochs of local updates are executed at each round for fair comparison. We use Adam optimizer with a learning rate of x 10-4 for the first 60 rounds and 10 for the last 40 rounds. The images to 224 x 224 with only random horizontal flip for augmentation. The learning rate and the number of local epochs is tuned by grid search with fixed batch size of 64, We tuned the projection dimension for FedDAR-SA among (4,8,16,32,64) with a 1.0 and used
resized
for all other a.
Our evaluation metrics are the classification accuracy on the whole validation set of FairFace for each race group. We don't have extra local validation set to each client since we assume the data distribution within each domain is consistent across the clients. The numbers reported are the average over the final 10 rounds of communication following the standard practice in (Collins al., 2021), and the average of three independent runs with different random seeds.
B.3.3 REAL DATA WITH REAL- WORLD DATA DISTRIBUTION
Dataset details. The detailed statistics of the partial EXAM dataset is summarized in Table 6. The "Other" category includes American Indian or Alaska native, native Hawaiian or other Pacific islander and patients with more than one race or unknown race. >HFO % means the percentage of cases with positive labels (receiving oxygen therapy higher equal to high-flow oxygen with 72 hours).
Table 6: Data summary of the partial EXAM dataset used in our study.
Site
White
Black
Asian
Latino
Other
> zFO %
Site-1 59.6%
10.0%
3,4%
2.0%
24.9%
12.4%
Site-2 75.0%
11.1%
2.8%
0.6%
10.5%
9.1%
Sitc-3
46.5%
26.3%
4.2%
7.0%
16.0%
9.6%
Sitc-4
71.4%
6.3%
4.2%
0.8%
17.2%
11.4%
Sitc-5
44.0%
28.4%
1.6%
6.3%
19.8%
9.9%
6 0.0%
0.0%
100.0%
0.0%
0.0%
18.8%
Implementation details. We apply 5-fold cross validation. The input of the model is chest x-ray image resized to 224x224 paired with a 22-dimensional electronic health rEcord data, the representation dimension is 278 if it is not projected. All the models are trained for T = 20 communication rounds with Adam optimizer and a learning rate of x 10-4. For each round we do 1 local epoch for all the methods. For all the methods, the models are initialized with the same pretrained model as in (Dayan al., 2021) without any warmup. For FedDAR-SA and FedDAR- WA, we execute 5 epochs of update for heads on each round, and set representation dimension 16 for FedDAR SA. Hyperparameters including learning rate, number of epochs for head update and representation dimension are tuned through grid search with fixed batch size of 36. For FedRep,FedDAA Rand FedPer. For LG-FedAvg, parameters and all other layers as local representation. For FedMinMax, multiple local iterations executed during each round instead of one step of GD for reasonable comparison. For FedProx
treated the last fully-connected layer the global
30

---

Published as a conference paper at ICLR 2023
tuned jl among {0.05, 0.1, 0.25. 0.5) and used 0.1. For the fine-tuning methods, only fine-tune the global trained model locally with Adam optimizer and learning rate of 5e for epoch since more epochs of fine-tuning leads to worse results.
The models are evaluated by aggregating predictions on the local validation sets then calculating the area under curve (AUC) for each domain. The average AUCs on local validation set of clients are also reported. The AUC shown is first averaged over the last five communication rounds, and then averaged over five runs of 5-fold cross validation.
C DISCUSSION ON COMMUNICATION AND PRIVACY
Communication For FedDAR-WA. the only communication overhead comes from the extra parameters of multiple heads for different domains, which only slightly increase the communication cost. For FedDAR-SA, we need to send Hessian with ka N2 parameters from each client to the server each round. This might be costly when both representation dimension k and output dimension N are large. However, compared to sending millions of parameters of neural network, the extra communication cost is acceptable.
Privacy For the FedDAR- WA, there no extra parameters shared compared to FedAvg. So there is no additional privacy risk introduced. Privacy techniques like homomorphic encryption (Cheon et al., 2017) or differential privacy (McMahan et al., 7b; Kairouz et al., 2021) that apply to FedAvg also works for FedDAR-Wa A-WA.. In fact, the multi-head design of different domains makes it harder to perform gradient based attack(Zhu al., 2019) targeting our method. Because the attacker need to first figure out which domain the sample comes from. For FedDAR -A,,,, the only extra parameters shared is the Hessian matrices, which are aggregated results from all the local data. Recovering the information for specific sample from Hessian is extremely difficult. Under the worst circumstance, what the attacker can recover from the Hessian the label and the features at last layer, which hardly ease the difficulty of recovering original input.
31


Reviews:
[{"review_id": "kvp6DALRoi", "reviewer": "Reviewer_GcPS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a domain-aware representation learning method (FedDAR) for the non-iid FL problem. The FedDAR assumes data on clients are from multiple domains and learns a classifier head for each domain. A representation module is shared for all classifier heads and updated by the vanilla FedAvg. The authors also proposed a second-order aggregation method to update domain-aware classifier heads, whose effectiveness is validated by ablation studies. Experiments on both synthetic data and real-world datasets validate the effectiveness of the proposed method.\n \n \n \n \n\n", "summary_of_the_review": "This paper proposes a domain-aware representation learning method (FedDAR) for the non-iid FL problem. The authors provide solid convergence analysis and validate FedDAR’s performance on both synthetic and real-world datasets. However, assumptions on domain labels and simple classifier heads limit the application of FedDAR. Besides, as there are limited comparisons with existing methods sharing similar ideas or settings, the paper’s contributions could be limited, and the results are less persuasive.", "strength_and_weaknesses": "Strength\n1. solid theoretical analysis on convergence.\n\n2. extensive experiments on both synthetic data and real-world datasets validate the effectiveness of the proposed method.\n\nWeaknesses\n1. The domain shift problem in FL has been extensively discussed in various works in recent years, such as [1-3]. Specially, [4] proposed a similar architecture where a representation module is shared and multiple classifiers heads are applied for target domains. As these methods are not mentioned or compared, it is uncertain if the contributions of the proposed FedDAR hold and the experiments are not convincing.\n\n2. This work assumes that there is labeled information about the domain a sample belongs to, which limits the proposed FedDAR’s application as it is usually not the case in FL. Besides, as there are no comparisons with those unsupervised methods[1,3] handling the domain shift problem in FL, it is unclear how many improvements the labeled information brings.\n\n3. As discussed in Sec.1 and Sec 2., personalized models have been recognized to have superior performance to vanilla FL models. Then comparisons in Sec.3.1, Sec.5.1, and Sec.5.2 would be less persuasive as they compare the FedDAR with vanilla FedAvg rather than other personalized methods. For example, will FedDAR outperform a vanilla model fine-tuned on a client’s local data?\n\n4. The second-order aggregation shares similar ideas as those personalization methods with a meta-learner [5]. Besides, as sharing and updating a Hessian matrix would be communication expensive and unstable, it requires a client to have sufficient training data and limits FedDAR to have simple classifier heads. Experiments in Appendix B.2. also indicates this problem. \n\n[1] Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferring (https://arxiv.org/abs/2206.06818)\n\n[2] FedBN: Federated Learning on Non-IID Features via Local Batch Normalization (https://arxiv.org/abs/2102.07623)\n\n[3] Federated Adversarial Domain Adaptation (https://arxiv.org/abs/1911.02054)\n\n[4] Diurnal or Nocturnal? Federated Learning of Multi-branch Networks from Periodically Shifting Distributions (https://openreview.net/forum?id=E4EE_ohFGz)\n\n[5] Personalized Federated Learning: A Meta-Learning Approach (https://arxiv.org/abs/2002.07948)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aZxXt1Idod", "reviewer": "Reviewer_XSkK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of personalized (to a client) federated learning in which the distribution from each client is assumed to be drawn from a mixture of predefined domains. The proposed model learns a shared data representation with independent prediction modules (heads) for each client. Moreover, the authors show that in the linear case, the proposed approach enjoyed linear convergence and empirically, that on both artificial and real-world data, the proposed approach outperforms existing federated learning methods.", "summary_of_the_review": "The authors presented a domain-aware approach for federated learning together with a theoretical analysis of the (per iteration) sample complexity of the methodology and empirical results using both synthetic (sections 5.1 and 5.2) and convincing real-world data (Section 5.3). The contributions of each of the components of the model are justified with an ablation study in Section 5.3. Though the theoretical results are interesting, they may not be applicable to many real-world scenarios; and though the weights u_m seem reasonable, they are not justified or motivated.", "strength_and_weaknesses": "The theoretical results in Theorem 5.1 are interesting and highlight the sample requirements scaling with k^2, which is convenient, specially, if k << d as the authors argue. However, it is also important to recognize that modern applications involving (medical) images and text leverage encoders with representations whose dimensionality is in the high hundreds or lower thousands, which may be prohibitive for data-poor clients and in applications where sample size is limited, e.g., rare diseases and hard to obtain data such as pathology. Moreover, the large number of clients scenario described by the authors in which the cost decreases as n^-1 is not that realistic in healthcare domains.\n\nThe toy example in Section 3.1 is welcome, however, it is not necessarily clear or discussed why in the FedRep scenario, where heads are specified, one for each client, the representation is not flexible enough to account for the differences in domain. One can certainly devise a toy example where it is possible.\n\nThe need and motivation for u_m (inverse probability weighting) is not provided. Moreover, no intuition is provided about the difference or relationship between \\hat{\\cal R}_m and \\hat{\\cal R}_i, which only differ by u_m.\n\nThe justification for only using second order aggregation for the heads (with low enough dimensionality) is reasonable (accounting for communication costs), however, its relative benefit compared to only considering simpler aggregation for the encoder (which is understandable), is not provided, though explored in the experiments.\n\nThough the use of the artificial data in Section 5.1 is justified, the considered generative process seems too simplistic to resemble any realistic scenario. Further, the results in Table 1 may be more impactful if the experiment is repeated multiple times to better understand the variation around the performance statistics (max, min and average accuracy). It will be also important to note the value of k used for the results in Table 1.\n\nIt is not necessarily clear whether AUC is a proper metric to assess fairness across domains. Perhaps the authors can elaborate on the reasoning behind their fairness claim.\n\nThe results in Table 3 are impressive, however, it is not clear why the authors used a cross-validation scheme considering that one of the claims is that the proposed approach is not data hungry. Further, the dimensionality of the EXAM data is not specified.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V9ZL16d-Ml_", "reviewer": "Reviewer_eZTL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a federated learning problem with multiple heterogeneous clients, where each client’s data distribution is a distinct mixture of predefined domain distributions. It is assumed that the domains of the data samples are known. The goal is to federatively learn a model that can perform well in every domain. For this purpose, the paper proposes FedDAR algorithm, which learns a shared representation across all domains and a separate predictor head for each domain. Training of the encoder and the head predictors are decoupled. Learning a shared representation with a small number of outputs allows domain heads to be accurately learned even with a small set of per-domain samples. Experiments show that FedDAR achieves significant performance gains over methods that do not take domain labels into account.", "summary_of_the_review": "This work tackles the important problem of domain-aware federated learning. The proposed algorithm is intuitive, and its applicability is fortified by convergence analysis in a simplified setting and experimental results in a complex setting. While there are some issues related to privacy and computational costs, my first impression of the paper is positive.", "strength_and_weaknesses": "Strengths: \n\n-Shared representation structure enables the algorithm to learn low-dimensional classifiers for each domain, which benefits domains with a small number of data instances. \n\n-Experiments demonstrate that the proposed algorithm comes with significant performance gains compared with client-centric or domain-agnostic federated learning methods. \n\nWeaknesses: \n\n-How FedDAR optimizes the learning objective in Equation 1 is unclear. Minimizing the expected risk averaged over all domains may not lead to a model which can perform well in every domain. This paper does not address fairness issues, nor does it achieve minimization of risks on a per-client basis, as each client has a data distribution that is a different mixture of predefined domains. While it is mentioned that the algorithm uses sample re-weighting, it is not clear how this is achieved with the current objective function. \n\n-The paper does not study the privacy implications of sharing model parameters.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PtUB1LJorCf", "reviewer": "Reviewer_p1BK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the problem of data heterogeneity in cross-silo federated learning. It assumes that datasets in each client are composed of a mixture of known domains, and domain metadata is available for each sample.\n\nBuilding on a fair learning formulation across domains (where each domain plays an equal role), the authors propose to train a neural network composed of two parts: a representation shared across domains and linear heads specific for each domain, called, FedDAR. Throughout each round, each part is trained alternatively, with the other part freezed. For the local heads, the authors consider two variants, FedDAR-WA (equivalent to FedAvg) and FedDAR-SA (second-order, requires to transmit hessians).\n\nThe paper proves on a toy linear model that the shared trunk converges to the true one linearly (Theorem 1).\n\nNumerical experiments on synthetic data, real data with controlled distribution (fairface), and real data without controlled distribution (EXAM healthcare dataset) demonstrate that the method improves over previous personalization methods. An ablation study helps to understand the role of the different components.", "summary_of_the_review": "Although I have some concerns regarding the choice of baselines, I think the paper makes an overall compelling case for the proposed approach. I am willing to upgrade my score if the authors address my questions.", "strength_and_weaknesses": "# Strengths\n\n- The method proposed by the authors reaches the best performance on all datasets\n- Well written overall\n\n# Weaknesses\n\n- The main weakness of the paper stems from the way baselines are handled\n  - Harmonized baselines across examples (for synthetic and fairface, separate FedAvg is not proposed although it's a natural baseline)\n  - Unless mistaken, FedAvg with per-domain reweighting is not considered as a baseline, except in the top row of Table 2 (ablation study). I think it would be fairer to compare all methods with the same optimization problem, to really demonstrate the benefit of sharing the trunk in the representation (given the results of Table 2, FedDAR would still probably be the best method).\n- Writing could be improved in some sections (cf detailed comments below)\n- The approach is limited to known domain mixtures (acknowledged by the authors)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster