Benchmarking Algorithms for Federated Domain Generalization
Authors: Ruqi Bai
Saurabh Bagchi
David I. Inouye
Venue: ICLR 2024 spotlight (2024)
Keywords: federated learning
distributed learning
domain generalization
out-of-distribution generalization
benchmarking
data paritioning.
Primary area: datasets and benchmarks

Abstract: While prior federated learning (FL) methods mainly consider client heterogeneity, we focus on the *Federated Domain Generalization (DG)* task, which introduces train-test heterogeneity in the FL context. Existing evaluations in this field are limited in terms of the scale of the clients and dataset diversity. Thus, we propose a Federated DG benchmark that aim to test the limits of current methods with high client heterogeneity, large numbers of clients, and diverse datasets. Towards this objective, we introduce a novel data partition method that allows us to distribute any domain dataset among few or many clients while controlling client heterogeneity. We then introduce and apply our methodology to evaluate 14 DG methods, which include centralized DG methods adapted to the FL context, FL methods that handle client heterogeneity, and methods designed specifically for Federated DG on 7 datasets. Our results suggest that, despite some progress, significant performance gaps remain in Federated DG, especially when evaluating with a large number of clients, high client heterogeneity, or more realistic datasets. Furthermore, our extendable benchmark code will be publicly released to aid in benchmarking future Federated DG approaches.

Published as a conference paper at ICLR 2024
BENCHMARKING ALGORITHMS S FOR FEDERATED Do- MAIN GENERALIZATION
Ruqi Bai, Saurabh Bagchi & David I. Inouye
Elmore Family School of Electrical and Computer Engineering Purdue University
West Lafayette, IN 47907, USA (bail16 sbagchi, di noue @purdde. edu
ABSTRACT
While prior federated learning (FL) methods mainly consider client heterogeneity, we focus on the Federated Domain Generalization (DG) task, which introduces train-test heterogeneity in the FL context. Existing evaluations in this field are limited in terms of the scale of the clients and dataset diversity. Thus, we propose a Federated DG benchmark that aim to test the limits of current methods with high client heterogeneity, large numbers of clients, and diverse datasets. Towards this objective, we introduce novel data partition method that allows to distribute any domain dataset among few or many clients while controlling client heterogeneity. We then introduce and apply our methodology evaluate 14 DG methods, which include centralized DG methods adapted to the FL context, FL methods that handle client heterogeneity. and methods designed specifically for Federated DG on 7 datasets. Our results suggest that, despite some progress, significant performance gaps remain in Federated DG, especially when evaluating with large number of clients, high client heterogeneity, or more realistic datasets. Furthermore, our extendable benchmark code will be publicly released aid in benchmarking future Federated DG approaches.
INTRODUCTION
Domain generalization (DG) (Blanchard et al., 2011) formalizes special case of train-test het- erogeneity which the training algorithm has access to data from multiple source domains but the ultimate goal is to perform well on test data coming from different distribution from training distribution -i.... a type of -fistr-diittibutiin generalization instead of the standard in in-distribution generalization. While most prior DG work focuses on centralized algorithms, another natural context federated learning (FL) Konečný et al. 2016), which is distributed machine learning context that assumes each client or device
local dataset. These local datasets could exhibit heterogeneity, which we call client heterogeneity (c.g., class imbalance between clients). Although train-test hetero- geneity (in DG) and client heterogeneity (in FL.) are independent concepts, both could be naturally defined in terms of domain datasets. For example, suppose network of hospitals aimed to use FL to train a model to predict disease from medical images. Because the equipment is different across hospitals, it is natural to assume that each hospital contains data from different domains or environ- ments (or possibly a mixture of domains if is large hospital)-thi this a case of domain-based client heterogeneity. Yet, the trained model should be robust to changes in equipment within a hospital or to deployment in a new hospital that joins the network- bottokkbothhhoo are cases of domain based train-test heterogeneity. The interaction between these types of heterogeneity produces new algorithmic and theoretic challenges yet may also produce new insights and capabilities. Solutions to Federated DG with both types of heterogeneity could increase the robustness and usefulness of FL approaches because the assumptions more naturally align with real-world scenarios rather than assuming the datasets are i.i.d. This could enable training partial datasets, increase robustness of models to benign spatial and temporal shifts, and reduce the need for retraining.
In the centralized regime, various approaches have been proposed for DG, including feature se- lection, feature augmentation, etc. Most of these methods are not applicable in the FL regime which poses unique challenges. In the FL regime, client heterogeneity has long been considered

---

Published as a conference paper at ICLR 2024
statistical challenge since FedAvg (MMMaha al., 2017). where it experimentally shows that FedAvg effectively mitigate some client heterogeneity. There are many other extensions based on the FedAvg framework tackling the heterogeneity among clients in FL, for example using variance reduction method (Karimireddy et al., 2020). An alternative setup in FI., known as the personalized setting, aims to learn personalized models for different clients to tackle heterogeneity, for example Hanzely et al. (2020). There are also unsupervised FL methods tackling domain translation instead of classification (Zhou et al. 2022] Wang al. 2023). However, none of these works consider model robustness under domain shift between training and testing data. Recently, a few works in the FL regime tackling DG (Liu et al. 202T Zhang et al. 2021] Nguyee et al. 2022) Tenison et al.) 2022b) have been proposed, however their evaluations are limited in the following senses: 1) The evaluation datasets are limited the number and diversity of domains. 2) The evaluations are restricted to the case when the number of clients is equal to the number of domains, which may be an unrealistic assumption (c.g., a hospital that has multiple imaging centers or a device that is used in multiple locations). The case when clients number might be massive are of both theoretical and application interests. 3) None of the works consider the influence of the effect of the number of communication rounds. We provide an overview of the tasks in [Talle I] considering both the heterogeneity between training and testing datasets (standard vs. domain generalization) and among clients (domain client heterogeneity). While some studies have addressed the standard supervised learning task, there is need for fair evaluation to understand the behavior of domain generalization algorithms in the FL context under those new challenges.
There are several benchmark datasets available for evaluating domain generalization (DG) methods in the centralized setting. These benchmarks, such as DomainBed (Gulrajani and Lopez-Paz 2020) and WILDS (Koh et al. 2021 provide multiple datasets that are suitable for assessing the performance of DG algorithms. However, they did not explicitly consider the unique challenges that arise in the federated learning (FL) setting. On the other hand, there are also benchmarks specifically designed for FL. For instance, the LEAF (Caldas et al. 2018)) and FLamby (Terrail et al., 2022) benchmark provides standardized framework for evaluating FL algorithms. They include several datasets from various domains and allows researchers to assess the performance of their algorithms in a realistic FL scenario. Another benchmark for FL is PFLBBench (Chen et al. 2022), which focuses on evaluating personalized FL. methods. PFLBennch provides 12 datasets containing various applications. Though these FL-bbsed benchmarks consider statistical heterogeneity, they fail to consider the DG task adequately. Moreover, the level of statistical heterogeneity present in these datasets is insufficient for proper DG evaluation. In summary, DG benchmarks do not consider FL challenges, and FL benchmarks do not consider DG challenges.
Major contributions: We develop benchmark methodology for evaluating Federated DG with various client heterogeneity contexts and diverse datasets, and we evaluate representative Federated DG approaches with this methodology. 1) We develop a novel method to partition dataset across any number of clients that is able to control the heterogeneity among clients. (see [Section 3). 2) We propose the first Federated DG benchmark methodology including four important dimensions of the experimental setting (see (Section 4). 3) We compare three broad approaches to Federated DG: centralized DG methods naively adapted to FL setting, FL methods developed for client heterogeneity (e.g., class imbalance), and recent methods specifically designed for Federated DG on 7 diverse datasets. Our results indicate that there still exist significant gaps and open research directions in Federated DG. 4) We release an extendable open-source library for evaluating Federated DG methods.
Notation: Let [A] == 21,2,.... AA denote the set of integers from to A. Let d E [D] denote the d-th domain out of D total training domains and let E [C] denote the e-th client out of C total clients. Let DD C [D] denote a subset of domain indices. Let L(0;p) denote generic objective with parameters 0 given a distribution p, which is approximated via samples from p. Let pa and pe denote the distribution of the d-th domain and the c-th client, respectively. Let S denote set of samples.
2 BACKGROUND: FEDERATED DOMAIN GENERALIZATION METHODS
In this section, we briefly introduce the problem backup and setup. Furthermore, evaluation on FL regimes often requires data partition methods to distribute training data to each client. Thus we also introduce the existing partition methods in this section.
2

---

Published as a conference paper at ICLR 2024
2.1 PROBLEM BACKGROUND AND SETUP
Domain generalization: Unlike standard ML which assumes the training and test data are inde- pendent and identically distributed (i.i.d.,, the ultimate goal of DG is to minimize the average or worst-case loss of test domain distributions using only samples from the training domain distributions. Formally, given a set of training domain distributions {ps d E ) minimize the average or worst case loss over test domain distributions, i.c.,
min Pie EAED
L(B: Pa) or min max ((0:pa): O deD
(I)
where pa) = () [1(1,, y; 0) where E is per-sample loss function such as squared or cross-entropy loss. Those two objectives are the ultimate goal for domain generalization. In the real- world setting, never have access to Past. There are works establishing objectives to approximate those two objectives. For example, Fish (Shi et al. 2021) is constructed using average training loss with penalty, and IRM (Arjovsky et al., 2019), GroupDRO (Sagawa al. 2019) are inspired from the worst-case loss over a set of data distributions constructed from the training domains.
There are mainly two kinds of DG test scenarios. The first scenario is the general DG where the test dataset form a new domain. Another scenario is called subpopulation shift where the test set is subpopulation of the training distributions with the goal of better performance on the worst-case domain DG is challenging where it breaks the i.i.d. assumptions in traditional machine learning.
Federated DG: Federated DG represents an intuitive progression from centralized DG. Distributed learning is pivotal in numerous real-world applications, and its inherent architecture often results in multi-domain data. The quest for improved generalization is instinctive in this context. However, Federated DG introduces add two layers of complexity. First, the heterogeneous client distribution are harder to converge even without considering the train-test heterogeneity (Zhao et al. 2018. McMahan et al. 2017 Karimireddy et al., 20200 Yu et al. [019 [asu et al., 2019; Wang et al. 2019 Li ct al., 2019). Second, privacy considerations and communication constraints typically prohibit the direct comparison of data across domains, thereby limiting the applicability of many conventional centralized DG methods.
Formally, the FL problem can be abstracted as follows:
Vc E [C], 0g = (pt Pc), Oii = 0 Oglobal) and glabal Agg((1, 02, ... ,00).
Locally optimize given local distribution
A Aggregate client model parameters server
where the client distributions may be homogeneous (i.e., V (c.c)).p. = p2) or heterogeneous (i.e., 3c * c', Pe * P. Opt minimizes an objective ((0: initialized at 0iii. The most common objective in Federated learning empirical risk minimization (ERM), which minimizing the average loss over the given dataset. Agg aggregates the client model parameters, where the most common aggregator simply a (weighted) average of the client parameters, which corresponds to FedAvg (McMahan al. 2017).
Domain- based client heterogeneity: While previous client heterogeneity (i.e., Bc c', D # Pe ) is often expressed as label imbalance, i.e., Pv(v) # (v)) we make a domain-based client heterogeneity assumption that each client distribution a (different) mixture of train domain distributions, i.e., P=(2, v) Edep We,d Pa(x, y) where We.d is the weight of the d-th domain for the c-th client. At one extreme, FL. with i.i.d. data would be equivalent to the mixture proportions being the same across all clients, i.e., Ve, c' We,d We',d, which we call the homogeneous setting. On the other extreme where the heterogeneity is maximum given the client number, we call complete heterogeneity (in |Section Table summarizes the train-test heterogeneity in DG and the domain-basee client heterogeneity from the FL context, where we focus on Federated DG.
2.2 CURRENT DATA PARTITION METHODS
In the FL context, the method for partitioning training data is crucial. As of now, there are three primary techniques for dataset partitioning. These methods are all fall in short in some properties we need. See Sectionn2 3 for a comprehensive analysis. In this section, we primarily present these methods as background. Shards partitioning (McMahan et al. 2017) is deterministic partition
In Wilds (Koh et al. 2021), , they treat subpopulation shift as another kind of train-test heterogeneity.
3

---

Published as a conference paper at ICLR 2024
Table 1: Domain- Federated DG considers the domain heterogeneity both among the clients' local datasets (rows) and between the training and test datasets (columns). This paper focuses on the DG setting (right column).
Between train and test datasets (train-test heterogeneity)
Standard supervised learning
Domain generalization (our focus)
Among clients
Homoggeneus (A 1)
FL with homogeneous clients
Federated DG with homogeneous clients Federated DG with heterogencous clients Federated DG with complete heterogeneity
Heterogencous (0 < I)
FL with heterogencous clients
Complete Heterogeneity (A 0)
FL with complete heterogeneity
method. The datasets are initially ordered according to their labels. Then the datasets are partitioned into 2 x C' shards, and each client E [C] is allocated 2 shards. Dirichlet Partitioning (Yurochkin cct al. 2019; Hsu et al. 2019) is stochastic partition method where each client hold the same number of samples but the proportion among labels stochastic following probability vector sampled from the Dirichlet distribution Dir(ap), where p represents the prior label distribution on the whole training dataset, and control the heterogeneity level. As - 0, each client predominantly holds samples of a single label. Conversely, as a - 0o, the distribution of labels for each client becomes identical. Semantic Partitioning (uuan et al. 2022) tries create client heterogeneity on the features of data samples. Firstly, the data is processed through pretrained model. The outputs from this model are then used to fit a Gaussian Mixture Model that features K clusters for each class Y. These KY clusters are then merged iteratively using an optimal bipartite for random label pairs.
3 HETEROGENEOUS PARTITIONING METHOD
In this section, to thoroughly assess current Federated DG methods, we formulate optimization problem that an ideal partition method should aim to solve. We then demonstrate that previous partition methods (subsection 2.2) are not feasible solutions. Subsequently, we introduce new partition method, termed Heterogeneous Partitioning, which is an approximate optimal solution. Generally, partition method effectively handles partitioning D types of diteger-numbered objects into C groups. It's broadly applicable, suitable for domain adaptation, ensuring fairness in Federated Learning (FL), and managing non-iid FL regimes.
Assume we have a set of all samples S = {Ir; d and let S. denote the set of samples assigned the e-th client. For each client E [C], define its domains as the set of domains from which e has least one sample., i.e., D. {d E [D] 3( d,) Se, d, = d}.
Table 2: Comparison of differ- ent partition methods.
C1
C2
Denote P as a partition method.
Shards
No
Yes
For Federated DG, it is common that clients have datasets collected from disjoint domains Koh et al. (20TT). Therefore, it's crucial that P can generate the strongest possible heterogeneity given the datasets and clients to reflect those common scenarios. This requirement can be encoded as either a single domain per client or no intersection of client domains, depending on the numbers of clients and domains (see constraint Ci). Furthermore, in distributed setting, to ensure a reasonable comparison with respect to domain generalization in the centralized setting, the total datasets received over the distributed system should match that of the centralized setting; and due to the privacy protocol, clients should not share the datasets. Thus, we need to preclude the partitions P where some data from S is used by any client clients either have duplicated data, which translate to constraint C2. Thus, the feasible region of partition P is defined as P E Ci C2-
Dirichlet
Yes
No
Semantic
No
Yes
Ours
Yes
Yes
Complete Heterogeneity (constraint C1): P can generate the possibly strongest domain heterogeneity given the datasets and clients, i.e., P E C where C₁ is defined as
G a pp
if D, then [De = 1, Vc; if C < D, then (D. Del = 0, Vc o. c)
(2)
4

---

Published as a conference paper at ICLR 2024
True Partition (constraint C2): P must produce true set partitions {Se}, E [C/. A true set partition a non-empty, exhaustive, and pairwise disjoint family of sets, i.e., P E C2 where C2 is defined as
C Us. S. -. S, and S. Se 0, Ve vc.
C2 P S. +
cml
The primary goal of this benchmark is to investigate the impact of client domain heterogeneity on the performance of various Federated DG methods. When considering client heterogeneity in a federated setting, a trivial example would be one client holding 99% of the total data. This scenario essentially mirrors centralized DG, which is already well-understood. However, the genuine uncertainty arises when there's heightened client heterogeneity introduced into domain generalization, especially when data sizes across different clients are nearly equal. In such a situation, the influence of client heterogeneity is at its peak. Our aim is to reduce the sample size imbalance between clients, which we encode as the empirical variance among them. Given this, we define an ideal partition? as the solution for the following problem:
C
P E arg min PECiNC, C Σ IIc nl12
where n. = | Ee Re. Notice that in many scenarios, it is not always able to achieve 0 loss, because n. may not be feasible for constraints and 2. For example, consider the case of complete heterogeneity with C = D but imbalance of domains, it is impossible to achieve balance ne = ñ and complete hetereogeneity [Dx] 1. Most previous partitions forces n. = thus violating the constraints. In particular, Dirichlet Partitioning violates Cz. Shard and semantic violate C1. We compare different partition methods in Table 2 See (ubsection A.2 for detailed discussion on them.
We construct Heterogeneous Partitioning {P*}, A E [0, 1], to Dolve[aqn. 4 It is parameterized by X, where it can generate Complete Hetereogeneity (A 0) and always satisfy True Partition (for all X E [0, 1)) by construction; further it aims to achieve the best possible balance given these constraints. Heterogeneous Partitioning allows samples allocation from multiple domains across an arbitrary number of clients C' while controlling the amount of domain- based client heterogeneity via parameter X, ranging from homogeneity to complete heterogeneity. In addition, when À = O, Heterogeneous Partitioning is optimal for Eqn. when there's more clients than domains C > D, Further, when C' < D, Eqn. is NP-hard problem proposed by Graham (1969) and Heterogeneous Partitioning is linear time greedy approximation. The proof is deferred to subsection A. Heterogeneous Partitioning contains the following two steps, see Algorithm for pseudo code.
Step 1: Constructing complete heterogeneity. We explicitly construct Po to satisfy C1 while greedily trying to balance the variance. The key idea to create the complete heterogeneity is to partition one domain among the fewest possible clients.
We adopt the following approaches depending on the size of the network C' and total domains D.
Case If C S D. the domains are sorted descending order according number of sample size in each domain, and are iteratively assigned to the client e* which currently has the smallest number of training samples, that
e² 2 arg min
Σ n''
<E|C]
d'EDD
where na denotes the total sample size of domain d. In this case, Po ensures that no client shares domains with the others, i.e., DD D' 0, but attempts balance the number of training samples between clients.
Case II: If C >>> D, first assign all the domains one by to the first D clients; then, starting from client e = D + 1, we divide the currently largest average domain d' between the new client and the original clients associated with it. That is, for all E (D D + 2,..., C), we assign d' to c, and reassign the same sample sizes for those d' E (l,...,cc which share the domain d' as
Y1d
d' E arg max dE(D) IO Dz)
D ED2)
where rounding to integers is carefully handled when not perfectly divisible. Notice that in this case, some clients may share domain, but no client holds two domains simultaneously, that is
5

---

Published as a conference paper at ICLR 2024
[De for E [C]. In this case, Po attempts to balance the number of samples across clients as much as possible while partitioning one domain among the fewest possible clients.
Step 2:
Computing domain sample counts for each domain with interpolation. We define Pa through the sample counts for each client e E [C] per domain d E [D], denoted as nd,e(A) based on the balancing parameter À E [0, 1]:
Rd
1id E Del Ed 1|d E Del
Px: (ddc(A)= =
+(1-
homogeneous
complete heterogeneous
where Dr. [C] constructed by Po Step 1. This is simply convex combination between
homogeneous clients (A = 1) and extreme heterogeneous (A = 0). A smaller value of À corresponds to more heterogeneous case. Given the number of samples per client (C') per domain d [D), we simply sample Rdd,e without replacement from the corresponding domain datasets and build up the client datasets. Step 2 ensures our partition Px satisfy the constraint C22, because all samples are selected (as proven by Ee Ddd, na) and there is no overlap (sample without replacement), and no client has zero samples given Del > the 0, cases.
BENCHMARK METHODOLOGY AND EVALUATIONS
In this section, we aim conduct comprehensive evaluation of the Federated DG task by considering four distinct dimensions of the problem setup equipped by Heterogeneous Partitioning. The four dimensions are 1) dataset type: 2) data difficulty; domain- client heterogeneity; and 4) the number of clients. This section is organized as follows: we introduce our benchmark datasets subsection 4.1 covering the first two dimensions, next we introduce 14 methods we included in our evaluation from three different approaches (subsection fubseciii 4.2) then we introduce our benchmark setting and the evaluation results on all selected methods in insubsecttion 4.3 brought the domain-based client heterogeneity and number of clients into consideration.
4.1 DATASET TYPE AND DATASET DIFFICULTY METRICS
While most Federated DG work focuses on standard image-based datasets, we evaluate methods over diverse selection of datasets. These datasets encompass multi-domain datasets from simpler, pseudo- realistic ones to the considerably more challenging realistic ones. Specifically, our study includes five image datasets and two text datasets. Additionally, within these 7 datasets, we include one subpopulation shift within image datasets (CelebA) and another within text datasets (Civilcommentss) Furthermore, our dataset selections span diverse range of subjects including general objects, wild camera traps, medical images, human faces, online comments, and programming codes. We also introduce dataset difficulty metrics to measure the empirical challenges of each dataset in the Federated DG task.
Dataset Difficulty Metries. To ensure a comprehensive evaluation of current methods across different difficulty levels, we have curated range of datasets with varying complexities. We define two dataset metrics to measure the dataset difficulty with respect to the DG task and with respect to the FL context using the bascline objective ERM. For DG difficulty. performance without and with samples from the test domain (i.c., the later is able to "cheat" by seeing part of test domain samples during training). For FL difficulty, we attempt to isolate the FL. effect by computing (A), the ratio of ERM-based FedAvg À client heterogeneity over centralized ERM on in-domain samples. These dataset difficulty metrics can be formalized as follows, for all E [0, 1]
compute RpG, the ratio of the ERM
RoG ERM ERM_ Per 1eG.es))
REy FedAvg_Per Seriavg_eer([SS P STN X)
ff U SSGG-ess Spo-es))
ERM_Perf (SpG-trinn SIN
where ERM__eMMrrerr_ is the performance of ERM using the first argument training and the second for test, FedAvq__ePee ferf similar but with the client heterogeneity parameter X, SpU-tsia denotes samples from the training domains Dusin SIG-test denotes samples from the test domains Destt and Sio-es and S DG-est are 20%, 80% partition respectively of Spe For (A), we use SIN (test samples from the training domains) instead of Spa.es isolate the FL. effect from the DG effect. We

---

Published as a conference paper at ICLR 2024
apply these metrics in our 7 selected datasets and include the summary table in Table 6 Smaller RpG and (A) indicate a more challenging dataset. For instance, FEMNIST has RpG = indicates the lack of domain heterogeneity. To contrast, IwildCam and CivilComments have small RçI. showing the challenges from the FL. side.
4.2 BENCHMARK METHODS
In this benchmark study, we explore three categories of Federated DG methods. The first category is centralized DG methods adapted into FL regimes. The second category is FL methods tackling client heterogeneity and the third category Federated DG methods. Please see/Appendix for a detailed discussion current available methods to solve Federated DG. To provide a comprehensive evaluation, we assess the performance of several representative methods from each of these categories.
specifically, we choose IRM (Arjovsky et al.) 2019). Fish (Shi et al. 2021), Mixup (Zhang al.) 2017). MMD (Gretton et al. 2006), Coral (Sun and Saenko 2016). GroupDRO (Sagawa et al. (019)
and adapted them into FL. regime by applying them in every local clients and keeping the aggregation identical to FedAvg, i.e.. weighted averaging aggregation. All those methods collapse to FedAdan
if locally there is only one domain available. We consider methods. FedDG (Liu et al. 202T)
FedADG (Zhang et al., (202)), FedSR (Nguyen et al. 2022) and FedGMA (Tenison et al. 2022a).
which are naturally designed to solve Federated DG. We also consider FedProx (Li et a.. 2020)
and Scaffold (Karimireddy et al. (2020) from the Federated methods tackling client heterogeneity
approach. All those methods will compared to two baselines. ERM objective with Adam optimizer (Kingma and Ba 2014) and its FL counterpart FedAdam (Reddi et al. 2020), which is variant of FedAvg (McMahan et al. 2017).
4.3 MAIN RESULTS
In this section, we present the performance results of 14 representative methods, derived from distinct research areas, on 7 diverse datasets. For each dataset, we fix the total computation and communication rounds for different methods for a fair comparison.
Client number Equipped with our Heterogeneous Partitioning, we can explore various levels of client heterogeneity and relax the assumption that C' D so that we can leverage both rseudo-rellistic and real-worke datasets and evaluate methods on larger scale of FL clients. In particular, we set the number clients to 100.
Validation domain In DG task, we cannot access the test domain data. However, are particularly concerned about the model performance outside the training domains, thus we preserve small access as held-out validation domains, and the held-out validation
portion of the domains domains are used for model selection and carly stopping. Please see ree[Appenix for more detail.
After training, we choose the model according to the early-stopping the communication round which achieves the best deld-out-domain performance, and finally we evaluate the performance on the test-domain in Table 3 Table 4 and Table 5| See Seelsubsett D.4 detailed hyperparameters choices. We make the following remarks on the main results from Table 3, Table a/ [Table Because FEMNIST has low DG difficulty, we defer the results on FEMNIST in the Appendix (Table 11).
Remark 4.1. FedAvg with an ERM objective a strong baseline, especially on the general DG datasets. We observe FedAvg-EER ERR outperforms other methods multiple times. It still servers strong baseline that is challenging to beat across general DG datasets, similar to the centralized case stated in DomainBed (Gulrajani and Lopez-Paz) 2020) and WILDS (Koh et al. 2021). We recommend always including FedAvg as a baseline in all future evaluations.
Remark 4.2. Most centralized DG methods degrade in the FL setting. For image datasets, the DG methods adapted to the FL setting (except for GroupDRO) show significant degradation in performance compared to their centralized counterparts as can be seen when comparing the C = 1 column to the C > 1 columns in Table 3 Further, degradation can be seen in PACS and CelebA when moving from the homogeneous client setting (A = 1) to the heterogeneous client setting (A 0.1).
Remark 4.3. FL methods tackling client heterogeneity help convergence. Notably, IWildCam and CivilComments datasets bring greater challenges in the model convergence Table 6 In this scenario. FL methods tackling the client heterogeneity are able to better tackle this challenge while other methods all failed in this context, see Table and Table 5) Given the fact that this is
7

---

Published as a conference paper at ICLR 2024
Table 3: Test accuracy on PACS and IWildCam dataset with held-out-domaii validation where FedAvg-ERM is the simple baseline (B). "-" means the method is not applicable in that context. Bold for best and italies is for second best in each column. We report the standard deviation among runs. Please see the Table 8 in the appendix for higher precision report.
PACS (D 2)
IWildCam (D 243)
C= 100 (FL)
C (centralized)
C (0L)
(centralized)
À = 1
À 0.1
A= 0
A
A 0.1
A= 0
22
FedAvg-ERM ERM
0.94 à: 0.01
0.95 1 0.02
0.96 1 0.02
0.95 0.02
6.94 0.00
0.30
a 0.00
0.25 4: 0:00
0.20 0.01
IRM
0.92 1:2::0000 0.01
0.92 4:2::000 0.04
0.86 4: 0.04
0.32 1: 0.00
0.20 1: 0.01
0.18 0.01
DG Adapted
Fish
0.94 + 0.08
0.63 +.13 0.13
0.35 0.35++0..11 0.11
0.35 0.00
0.18 + 0.00
0.15 0.01
Mixup
0.00
0.05
0.83
0:33 0.00
0.0
0.07 0:01
MMD
0.93 2: 0.02
0.92 2: 0.04
0.56 0.00
0.32 1: 0.00
0.19 1: 0.01
0.16 0.01
Dorp
0.90 3:03=:000 0.01
0.92 +: 0.04
0,56 0.04
0:33 +: 0.01
0.19 + 0.01
0.16 0:01
GroupDRO
0.93
0:21 0.00
0.13
0:01
FedProx
0.90 0.03
0.89 1: 0.03
0.90 =: 0.02
0.25 4: 0.00
0.20 1: 0.00
0.17 0.00
Scaffold
0.50 0.03
0.89 + 0.02
0.90 + 0.02
0.28 + 0.00
0.26 0.00
0.18 0.07
APL
0.91
0:26
0.16 0:01
0.03
FedDG
0.93 0.02
0. 0.95 = 0. 02
0.94 : 0.02
0:27 + 0.00
0.24 0:00
0.17 = 0.00
FedADO
0.91 0.01
0.94 +: 0.00
0.91 =: 0.01
0.28 1: 0.01
0.25 0.01
0.16 4 0.00
FedSR
0.64
0.55
0.54
0.18
0.14 0.01
0.09 0.00
FedGMA
0.75 +: 0.06
0.73 +: 0:09
0.72 0.01
0.22 + 0.00
0.15 0:00
0.09 + 0.00
common challenge in Federated DG, there's a need for future research to simultaneously enhance the performance with both client heterogeneity and train-test heterogeneity.
Remark 4.4. Federated DG methods still in need. Upon evaluating current Federated DG methods
on larger network scales, they all fail to outperform ERM. Among the four methods we evaluating, FedDG performs the best and could achieve higher accuracy on PACS with À = 0. However, FedDG requires sharing all local datasets amplitude information to the aggregation server, thus brings privacy concerns. FedADG methods contains tens of hyperparameters, and due to the nature of GAN (Goodfellow et al.) (020). it is challenging to optimize. FedSR and FedGMA also did not outperform ERM objective. It shows a clear demand for further improvement.
Remark 4.5. Addressing the subpopulation shift could be an initial step. In the centralized setting, all evaluated methods surpassed the performance of ERM on two subpopulation shift datasets: CelebA and CivilComments. However, in a Federated context, only FedDG managed to outpace ERM on CelebA, and yet it couldn't match the performance of centralized approaches. Further exploration in federated DG is in demand to bridge this gap.
Remark 4.6. The performance of real-world data significantly degrades as À decreases. This can be seen from IWildCam and P5050 dataset at Table 3) and Table 5] While it is challenging and expensive to run models for IWildCam and Py150, they show the largest differences between methods and demonstrates the woal-world challenge of Federated DG. We suggest including IWildCam and Py150 150 in most future DG evaluations given their unique nature across datasets.
Table 4: Test accuracy on CelebA and Camelyon17 dataset with geld-out-domain validation where FedAvg-ERM is the simple baseline (B). "_" means the method is not applicable in that context. Bold is for best and italics is for second best in each column. We report the standard deviation among
runs. Please see the Table 9 in the appendix for higher precision report.
CalehA (D) = 2)
Camelyon17
C ==1
C= 100 (FL)
C (centralized)
C' = (FL)
(centralized)
A
A=
A=1
A 0.1
A=0
20
FedAvg-RRM
0.77+00.44
0.61 + 0.02
0.52 + 0.04
0.45 + 0.02
0:90 + 0.01
0.35 + 0.01
0.95 + 0.00
0.95 0.00
DO Adapted
IRM
0.89 1: 0.01
0.71 4: 0.05
0.16
0.07
0.74 1: 0.01
0.91 1: 0.06
0.92
0.95 1: 0.00
0.88
0.95 1: 0:00)
0.88
Mixup
0:20 0.458
0.13 +: 0.22
0.13 0:22
0.34 +: 0.01
0.96 1:55+:00 0.00
0.95 -: 0:01
MMD
0.81 0.05
0.81 0.00
0.77 0103
0.92 1: 0.03
0.95 1: 0.90
0.95 :: 0:01
DeepCoral
0.85 0.04
0.81
0.78
0.01
0.95
0.95
GrouDROO
0.87 0.03
0.81 - 0.09
0.76 0.06
0:92 + 0.03
0.95 + 0.01
0.95 0.00
FedProx
0.13 0.24
0.12 0.25
0.00
0.94 + 0.00
0.94 + 0:00
0.9g 0.07
Scaffold
0.73 0.01
0.78 = 0.
0.80 = 0.02
0.91 + 0.00
0.93 : 0.00
0.93 - 0.00
AFL
0.81 4: 0.02
0.53 s: 0.011
0.53 =: 0.01
0.94 1 0.01
0.95 - 0.01
0.93 1 0.01
FedixG
0.36
0.53
0.46
0.86 0.01
0.85 + 0:00
0.87 +
FoIADG
0.67 0.00
0.67 0.01
0.60 1: 0. 0.01
0.91 0.00
0.93 : 0.00
0.93 - 0.00
FedSR
0.38 0.02
0.36 J: 0.04
0.38 2: 0.02
0.93 1: 0.01
0.92 - 0.01
0.93 + 0.00
FodGiMA
0.62
0.62
0.40
0.90 0.01
0.85 + 0:01
0.82 0.03
8

---

Published as a conference paper at ICLR 2024
Table 5: Test accuracy on CivilComments and Py150 datasets with held-out-domain validation where FedAvg-ERM is the simple baseline (B). "-" means the method is not applicable in that context. Bold for best and italics for second best in each column. FedDG and FedADG are designed for image dataset and thus not applicable for CivilComments and Py We report the standard deviation among 3 runs. Please see the Table 10 in the appendix for higher precision report.
CivilComments
16)
Py150 (D 0477)
C= 100 (FL)
C== 100 (FL)
(oentralized)
==
A
A=
(centralized)
À
FedAvg-ERM ERM
0.54 = 0.00
0.36 =: 0.03
0.35 2: 0.02
0.33 = 0.01
0.68 =: 0.00
0.68 0.00
0.65 = 0.00
0.54 =: 0.00
DO Adlapeed
IRM
0.00
0.59
0.53
0.68
0.60
0.65
0.63
Fishh
0.67 1: 0.00
0.42 1: 0.16
0.34 +: 0.17
0.66 0.00
0.85 = 0.00
0.65 :: 0.00
0.64 0.00
MMD
0.63 0.00
0.63 0.01
0.61 + 0.01
0.06 + 0:00
0.63 = 0.00
0.63 + 0.00
0.61 + 0.00
DeepCoral
0.58 0.00
0.52
0.66 0.00
0.85 0.00
0.65
GroupDRO
0.64 =: 0.00
0.48 + 0.01
0.47 = 0. 0.00
0.51 + 0:00
0.59 = 0.00
0.60 = 0.00
0.61 = 0.00
FedProx Scaffold
0.18 0.03
0.04
0.64 0.00
0.63
0.61
0.39 = 0.02
0.38 # 0.02
0.35 :: 0.01
0.64 = 0.00
0.61 0.00
0.62 = 0.00
AFL
0.55 d: 0.02
0.47 1: 0.01
0.44 1: 0.02
0.49 =: 0.00
0.49 1: 0.00
0.47 = 0.00
FedSR
0.36
0.34
0.52
0.53 =:
0.53
0.45
FedGMA
(1.21 = 0.03
0.20 = 0.02
0.20 2: 0.02
0.62 = 0.00
0.61 = 0.00
0 = 0.00
Additional DG challenges from FL. For further understanding, we explore some additional questions on several smaller datasets because it is computationally feasible to train many different models on these datasets. Specifically, we explore how the number of clients, amount of communication (i.e., the number of server aggregations) in the federated setup, and client heterogeneity affects the performance of various methods. The figures and detailed analysis are provided in the Appendix but we highlight two remarks here.
Remark 4.7. The number of clients C' strongly influences performance. Performance in DG plunges from 90% to as low as 10% as C shifts from 1 to 200 as suggested in Fig. 2, We explore this on representative methods on three datasets. We advocate for future Federated DG assessments to consider larger number of clients, like 50 to 100 rather than only considering small numbers of clients. This demonstrates pressing, unaddressed challenges in Federated DG when grappling with many clients. Notably, FedADG and FedSR degrade faster with increasing number of clients.
Remark 4.8. The number of communications does not monotonically affect the DG performance. In the FL context, we note unique implicit regularization phenomenon: optimal performance achieved in relatively few communication rounds Fig. 3) For instance, with PACS, the ideal communication round is just 10 (while maintaining constant computation). Contrarily, for in-domain tasks, FL theory indicates that increased communication enhances performance, as shown by (Stich) 2018. Theorem 5. Investigating how DG accuracy relates to communication rounds, and potential implicit regularization through early stopping, offers a compelling direction for future research.
5 CONCLUSION AND DISCUSSION
We'v ve established a benchmark methodology for evaluating DG tasks in the FL regime, assessing 14 prominent methods on 7 versatile datasets. Our findings indicate that Federated DG remains an unresolved challenge, with detailed gaps outlined in appendix Table 12) Recommendations for future evaluations of Federated DG. 1) Evaluation there will be no real train-test heterogeneity. 2) The FL regime introduces two main challenges for domain generalization (DG) methods. Firstly, the complete heterogeneity scenario limits the exchange of information between domains. Secondly, when (AR. < 1), it poses convergence challenges for the methods. One way to produce lower RyL increase the number of clients. In future evaluations, it would be valuable to assess the capabilities and limitations of proposed methods in handling these challenges. 3) Always include FedAvg as the baseline comparison. 4) Evaluation on both Subpopulation shift datasets and general DG datasets since the methods might perform differently. Suggestions for future work in Federated DG. 1) The federated domain generalization (DG) task remains unsolved, and further work is needed to address the challenges. 2) While our primary focus remains on domain generalization (DG), it is important for future methods to also address the issue of improving convergence when the FL learning rate (RHL) is small. This consideration holds particular significance in real-world applications where efficient convergence is crucial. We hope this work provide better foundation for future work in Federated DG and spur research progress.
the dataset where RDG < 1. If RDG x 1,

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
Code for reproduce the results is available at the following link:
https//giihbbb.omm/ooooyyattftdt Beee_BBenchhkkk
We include detailed documentation in using our code to reproduce the results throughout the paper. We also provide documentation in adding new algorithm's DG evaluation in the FL context. The code contains all the experiments included in section 4. To help reproducibility, we have
1. Include the requirements.kg txt file generated by conda to help environment setup.
2. The code contains a detailed Readme file to help code reading, experiment reproducing and future new method implementing.
3. We include the config file identical to the paper including the seed to help reproducibility.
4. We include the command needed run the experiments.
5. The model structure and tokenizers are available.
6. We include the hyperparameter grid search configuration to reproduce Table 7
7. The dataset preprocessing scripts, transform functions are also included.
We hope our reproducible and extendable codebase could help both new methods implementations and better evaluation in the Federated DG fields.
ACKNOWLEDGEMENT
This work was supported by Army Research Lab under Contract No. W911NF-2020-2211 R.B. and D.I. also acknowledge support from NSF (ISS-2212097) and ONR (N00014-33--10166) Any opinions, findings, and conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect the views of the sponsor(s).
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02880 2019.
Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-See Distributed sgd with quantization, sparsification and local computations. Advances in Neural Information Processing Systems. 32, 2019.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to new unlabeled sample. In J. Shylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 24. Curran As- sociates, Inc., 2.11. URL http::::p////ocettt/ttt/t wwwpa/aa22---a.e...eee pdf
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf': A benchmark for federated settings. arXiv preprint arXiv:1812.0109) 2018
Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding, pFL-bench: A comprehensive benchmark for personalized federated learning. In Thitty-sitth Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2022. URL
net/ nett/formm2ddd2gtlbyyuu
Wei Du, Depeng Xu, Xintao Wu, and Hanghang Tong. Faimess- agnostic federated learning. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 181-189. SIAM, 2021.
10

---

Published as a conference paper at ICLR 2024
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Ronald Graham. Bounds on multiprocessing timing anomalies. SIAM journal on Applied Mathematics, 17(2):46-429, 1969.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sampe-probbemm Advances in neural information processing systems, 19, 2006.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33:2304 2015, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint
2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143, PMLR, 2020.
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. International Conference on Artificial Intelligence and Statistics, pages 4519 4529. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint /412.6980, 2014.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard I Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning, pages 5637-5664. PMLR, 2021.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05490 2016.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5400-5409, 2018
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of 2019.
fedavg on non-iid data. arXiv preprint
Xianfeng Liang, Shuheng Shen, Jingchang Liu, Zhen Pan, Enhong Chen, and Yifei Cheng. Variance reduced local sgd with lower communication complexity. arXiv preprint arXiv:1912.12844 2019.
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Amn Heng. Feddg: Federated domain generalization medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1013-1023, 2021.
11

---

Published as a conference paper at ICLR 2024
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
Brendan McMMaan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-efficcenn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pages 1273-1282. PMLR, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional Conference on Machine Learning, pages 4615 4625. PMLR, 2019.
Tuan Nguyen, Ser-Nam Lim, and Philip HS Torr. Fedsr: Simple and effective domain generalization method for federated learning, International Conference on Learning Representations, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning, pages 18347-18377. PMLR, 2022.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295. 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 191.:::9910087111 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter, 2020.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun. Nicolas Usumier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937, 2021.
Sebastian Stich. Local sgd converges fast and communicates little. arXiv preprint arXiv: 1805.09777, 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443-450. Springer, 2016.
Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan, Edouard Oyallon, Eugene Belilovsky, and Irina Rish. Gradient masked averaging for federated learning, 2022a. BtRLhttss://
org/ hbs//bb//i 2201. 11986
Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan, Edouard Oyallon, Eugene Belilovsky, and Irina Rish. Gradient masked averaging for federated learning. arXiv preprint arXiv:2201. 2022b.
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang Hc, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint 22.1:22200006621 0406200 2022.
Jinbao Wang, Guoyang Xie, Yawen Huang, Jiayi Lyu, Yefeng Zheng, Feng Zheng, and Yaochu Jin. Fedmed-gan -ann: Federated domain translation on unsupervised cros-ttoyality brain image synthesis, 2023.
Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning resource constrained edge computing systems. IEEE journal selected areas in communications, 12(6):1205-11) 1221, 2019.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI conference on artificial intelligence, pages 6502-6509, 2020.
12

---

Published as a conference paper at ICLR 2024
Hao Yu, Sen Yang. and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 5693-5700, 2019.
Honglin Yuan, Warren Richard Morningstar, Lin Ning, and Karan Singhal. What do mean by generalization in federated learning? In International Conference on Learning Representations, 2022. [RL/Atttss:/oppezeeee htwp://wwwwww..eeeeeeeeeeeeeeeee
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang. and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
Liling Zhang, Xinyu Lei, Yichun Shi, Hongyu Huang. and Chao Chen. Federated learning with domain generalization. arXiv preprint arXiv:2111.10887, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:/806.000 2018.
Zeyu Zhou, Sheikh Shams Azam, Christopher Brinton, and David Inouye. Efficient federated domain translation, In The Eleventh International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Data Partition in Federated DG
A. Heterogeneous Partitioning Algorithm and its guarantee
14
2 Other Partition Methods
15
B Current Methods in solving DG
17
Datasets and Difficulty Metric
18
Dataset Introduction
Dataset partition Setup
D Benchmark Experimental Setting
19
D Model Structure
19
D.2 Model Selection
19
D.3 Early Stopping
19
D.4 Hyperparameters)
20
Additional F--specffic challenges for domain generalization
F
Supplementary Results
22
G Gap Table
22
Training Time, Communication Rounds and local computation
24
A DATA PARTITION IN FEDERATED DG
A.1 HETEROGENEOUS PARTITIONING ALGORITHM AND ITS GUARANTEE
Denote Po the complete heterogeneous case corresponding Heterogeneous Partitioning with = O. We have the following guarantees on the optimality of Po.
Proposition A.1. When C 2 D, Po is optimal for Eon. 4 When C < D. Eqn. 4 is NP hard, and Po fast greed approximation.
Proof. Case I. When C 2 D, Po generates [Sa), E [C] such that for all clients which share same domain, their sample size is the same, that is, for all e, C" share some domain d D, there holds
ne = Td.
(8)
Further, for sample variance, there holds
Σ
|nc n2l22
C -
=
C
Σ
||:: ń - ny + ml
2C(C-1) -
i= j=1
c
Σ
Σ
||m: - n1 112-
-
i=1 j=1
14

---

Published as a conference paper at ICLR 2024
Therefore, Egn. 4 boils down to
C c
P
arg min
2
[, myl12
(10)
PeC 2C - 1)
1=1 j=1
Chaining with Eqn. 8 we have
C Im ny
D D Σ xm
2C ΣΣ
nylle
(11)
2C(C
where î. is the same sample size of clients which share the domain i. The equality comes from the second equation in [gg. 6 where reassign the same sample sizes for those c' (1,....c) which share the domain d as
Rd
(12)
Ee
1/ DD]
In addition, notice that always divide the currently largest average domain d' between the new client and the original clients associated with That is,
d' E arg max dE(D]
(13)
Therefore, the discrepancy between ñ. and ny, i.j E [D] is minimized. It is straightforward to verify such partition Po also satisfy the true partition (constraint). Thus, Po is optimal for Eqn. 4
Case II. When < D, the problem corresponds to the classic NP-Hard problem known as Multiway Number Partitioning. We employ well-known linear-time greedy algorithm to address this (Grhaam 1969).
train-test beterogeneity
homogeneity (A=1)
heterogeneity (2=0.1)
complete heterogeneity (R=0)
D
client
client
client
Deest
client 2
client
client 2
domain spilt when C ad D,
client
client
client
[b]
(
Figure 1: Distinct color refers to distinct domain data, and À the domain balancing parameter. (a):
train-test domain heterogeneity. (b): domain partitioning when C < D and domain partitioning when
C' > D. (c): domain partitioning illustration when C' < D; homogeneous (A 1), heterogeneous
(A 0.1), and extreme heterogeneous (A 0).
We provide the pseudo code for Heterogeneous Partitioning in Alg. I and an illustration in Fig. In Fig. represents multi-domain dataset with 4 training domains. Each color represents domain and the height of the bar represents the number of samples in this domain. (b) shows the complete heterogeneity with scenarios C < D and C D. (c) is detailed partition result with different interpolating with the C < D case. can be seen that in the complete heterogeneity case, the partition satisfies constraint 2, further, the sample size between clients are roughly even. For other cases, True Partition (Constraint) is satisfied and also the sample size between clients are roughly even.
A.2 OTHER PARTITION METHODS
15

---

Published as a conference paper at ICLR 2024
Shards. By adjusting the constant from 2 to other value, it's possible to regulate the het- erogeneity certain extent. However, if the dataset contains imbalance number of samples per domain, shards method could not achieve the complete heterogeneity. For instance, if dataset contains 2 domains with 20, 30 data sam- ples respectively. Shard partitioning will not be able to assign each client with data from only
Algorithm 1 Domain Partition Algorithm
Input Number of clients C, heterogeneity param- eter À E |0. 1], and samples from each domain (Si. SS), where 71...., RD denote the num- ber of samples per domain and w.l.o.. are as- sumed to be in descending order, i.e., n1 2 n2 2
n.D
1/ Divide domain indices across clients if C S D) then
one domain.
Vc, D. < 0 for d = 1,2,... do // Find client with fewest samples
Dirichlet Partitioning. draws q Dir(op), where p characterizes a prior domain distribu- tion over D domains for each client E [C]. 0 is a concentration parameter controlling the heterogeneity among clients. As a - 0, each client predominantly holds samples of single label. Conversely, as a + co, the distri- bution of labels for each client becomes identi- cal. This method is not a desirable partitioning in the following sense. 1) If sampling with re- placement, then the datasets across clients may not be disjoint, and some data samples may not be allocated to any client, therefore breaking properties and 4. 2) If sampling without re- placement, the method is inapplicable when the available data from one domain doesn't 't meet the collective demand for this domain's 's data from all clients, breaking property 2.
c¹ E arg min EdeDd nd
<E[C] De. D. Udd
end for
else > then
Ve E -1,,2....
D), D. 5 {c)
for c +1 C do
1/ F/Find on average largest domain parti- tion
d' E arg max
Σ 2
dE(D]
D. + {dd
end for end if
Partition samples across the clients
Ve, S. <- 0 for e E [C), d E [D] do
Semantic Partitioning. This method aims to partition dataset based on the feature similarity without domain labels. It first processes the data using a powerful pretrained model. The outputs before the fully connected layer later used as features. Then, thses features are fit a Gaus- sian Mixture Model form K for each clusters for each class Y. These KY clusters are then merged iteratively using an optimal bipartite for random label pairs across different class. This partition method is more heuristic and highly rely on the quality the pretrained model. Furthermore, since there is no hard code label, and the partition is based on KL divergence, this method could not give a complete heterogeneity which violate property 1. Despite that this method could not guarantee property 1, it is a good partition method to create "domain" Deamainn-heterggeeee -heterogeneity without having domain labels.
IdeDell-s (a,c(A) = X xa + (1 - A) ES lddEDel
S' + Samp www.eelaaeeeeeeeeeeeeeeeeeeeeeee (Sa, nd,e)
+ Sa
// Remove from domain // Append client dataset
S. S'
end for Output (8), 82, ... ,S))
Example 1. We provide a toy example of partition a dataset containing 5 training domains with 10, 20, 30, 40, 100 data samples respectively. We illustrate how shards and Dirichlet methods work and highlight that disjointedness flexibility and controllability are violated when our goal is to partition the data into 2 clients. Shards partitioning might allocation [10, 20, 20, 0, 50) samples the first client, and [0, 0, 10, 40, 50] samples to the second client. This lead to the share of domain 3 and 5 between the two clients showing the violation Complete Heterogeneity (Constraint). The Dirichlet partitioning will sample distribution from Dirichlet distribution with RJ O. However, no matter what distribution are sampled, at least 3 domains will be neglected, leading to the violation of True Partition (Constraint). In contrast,
Heterogeneous Partitioning will allocate [10, 20, 30, 40, 0] to the first client and [0, O, O, 0, 100] to the second client, effectively maintaining Complete Heterogeneity (Constraint), True Partition (Constraint), in addition, the sample size difference is O in this case.
16

---

Published as a conference paper at ICLR 2024
B CURRENT METHODS IN SOLVING DG
Centralized DG methods Most work solving in DG lies in the centralize regime. A predominant and effective centralized DG approach is through representation learning. [Arjovsky et al. (2019) tries to learn domain invvariat feature by aligning the conditional distribution of [X) among different domains. [Sun and Saenko (0166), and [i et al. (2018) tries to explicitly align the first order and second order momentum in the feature space. There are also methods trying to promote the out-o--distribution generalization by posting constraint
the gradient information among different domains, where Shi et al. (202T) tries to align the gradient among different domains, and Rame et al. (2022) enforces domain invariance in the space of the gradients of the loss. Other approaches include distributionally robust optimization (Sagawa et al. 2019), where this method learns the worst-case distribution scenario of training domains. Furthermore, Zhang et al. (2017) is not specifically design for solving domain generalization, but found to be effective as well when we extrapolate the data from different domains in the training dataset. [ et al. (2020) introduces domain- maxip which extrapolates examples and features across training domains.
Federated DG methods Limited research has focused explicitly on solving the Federated DG by design. FedDG Liu et al. (2021) introduced specific FL paradigm for medical image classification, which involves sharing the amplitude spectrum of images among local clients, violating the privacy protocol. Another approach, FedADG (Zhang et al., 2021), utilizes a generative adversarial network (GAN) framework in FL., where each client contains four models: a featurizer, a classifier, a generator, and discriminator. FedADG first trains the featurizer and classifier using empirical loss and then trains the generator and discriminator using the GAN approach. However, this method requires training four models simultaneously and tuning numerous hyperparameters, making convergence challenging. A novel aggregation method called Federated Gradient Masking Averaging (FedGMA) (Tenison al. 2022b) aims to enhance generalization across clients and the global model. FedGMA prioritizes gradient components aligned with the dominant direction across clients while assigning less importance to inconsistent components. FedSR (Nguyen et al.) (202)) proposes simple algorithm that uses two locally-computable regularizers for domain generalization. Given the limited literature on solving domain generalization (DG) the federated learning (FL) setting, we selected all the aforementioned algorithms.
FL methods tackling client heterogeneity Another line of research in FL. aims guar- antee convergence even under client heterogene- ity, but these FL-based methods still assume the train and test datasets do not shift (i.e., they do not explicitly tackle train-test heterogeneity of the domain generalization task). We include this line of work because methods considering converging over a heterogeneous dataset might bring better DG ability implicitly. The empirical observation of the statistical challenge in fed- erated learning when local data is non-IID was first made by /Zho et al. (2018). Several sub- sequent works have analyzed client heterogene- ity by assuming bounded gradients (Yu al.
Algorithm 2 Modified AFL
Initialize 0, 3
Transmit 0 all clients [C]. for t = 1, 1,2....,T ".....Ton. T'do do // Update 0
Transmit / to all clients for = 1,2,..., do
Update 0g using SGD with multiple itera- tion.
Submit 0g to the server.
end for
10c. 0 < Σ ne I1 Update 8
I/ Weighted average.
Transmit 0 to all clients c for c 1,2,..., do Calculate ba,e- = Ca Submit updated Cd end for for 1,2, D, do 3 +- Proj 78 (ec) end for end for Output 0
2019 Basu et al. 2019 Wang et al., 2019; Li
2019) or bounded gradient dissimilar- ity (Li et al. 2020). and additionally assum-
ing bounded Hessian dissimilarity (Karimireddy et al., 2020; Khaled et al.) 2020) Liang et a
2019). From this category, we selected FedProx (Li et al. 2020), which addresses statistical het- erogeneity by adding a proximal term to the local subproblem, constraining local updates to be closer to the initial global model. Scaffold (Karimireddy et al. 2020) utilizes variance reduction account for client heterogeneity.
17

---

Published as a conference paper at ICLR 2024
FL methods tackling fairness Agnostic Federated Learning (AFL) (Mohri et al. 2019 Du et al. 2021) shares similarities with Domain Generalization in a Federated context. This is evident as both approaches address scenarios where the test distribution diverges from the training distribution. AFL constructs framework that naturally yields a notion of fairness, where the centralized model is optimized for any target distribution formed by mixture of the client distributions. Thus, AFL. is
a good method to evaluate especially when tackling subpopulation shift tasks. AFL introduces minimax objective which is identical to the GroupDRO (Sagawa al. 2019).
min max [(, = Σ Bald(O):
(14)
0 DEAD
where & denotes the model parameter and / is weight vector taking values in the simplex Ap of dimension D. AFL introduces an algorithm applying projected gradient ascent on 3 and gradient descent on 0. It is designed for C' = D, where they assumes each client domain. Further, it requires communication per iteration. We made the following two modifications to accommodate general C' o D cases and expensive communication cost:
To allow C + D, we construct new objective as the following:
C
min max L B) = E (0. = ΣΣ (o). c1d=l
(15)
# BEAp
2. To reduce communication cost, we allow to be updated multiple iterations locally per communication. Further, we maintain the update of on the central server, in AFL, given that it requires projection of the global updates onto the simplex. This projection is not equivalent averaging the locally projected updates of (3.
Modified AFI. avoids the frequent communication compared to AFL
C DATASETS AND DIFFICULTY METRIC
C.1 DATASET INTRODUCTION
In this section, we introduce the datasets we used in our experiments, and the partition method we used to build heterogencous datasets in the training and testing phase as well as the heterogencous local training datasets among clients in the FL.
FEMNIST is an FL prototyping image dataset of handwritten digits and characters each users created as a natural domains, widely used for evaluation for client heterogeneity in FL. Though it contain many training domains, it lacks significant distribution shifts across domains ( (RoG = 1), and considered easy compared to other datasets.
PACS It is an image dataset for domain generalization. It consists of four domains, namely Photo (1,670 images), Art Painting (2,048 images), Cartoon (2,344 images), and Sketch (3,929 images). This task requires learning the classification task on a set of objects by learning on totally different renditions. Rpo = 0.960 makes it easy dataset as domain generalization in our setting. Notice that choosing different domain as test domain might give us different Rpg.
IWildCam It real-world image classification dataset based on wild animal camera traps around the world, where each camera represents a domain. It contains 243 training domains, 32 validation and test domains. Usually people cares about rare speicies, thus we utilize the macro score as the evaluation metric instead of standard accuracy, as recommended in the original dataset's reference Koh et al. 2021). The RpG = 0.449 makes it a very challenging dataset for domain generalization.
CelebA CelebA (Celebrity Attribute) (Liu al. 2015) is large-scale face attributes dataset. It's one of the most popular datasets used in the facial recognition and facial attribute recognition domains. We use variant of the original CelebA dataset from (Sagawa et al.) 2019) using hair color as the classification target and gender as domain labels. This forms a subpopulation shift task which is a special kind of domain generalization.
18

---

Published as a conference paper at ICLR 2024
Camelyon17 It is medical dataset consisting of set of whole-slide images from multiple institutions (as domain label) to detect breast cancer. consist training domains, validation domain and 1 test domain. We partition the data following wilds benchmark (Koh et al. 2021
CivilComments is real-world binary classification text-based dataset formed from the comments from different demographic groups people, containing & demographic group. The goal is judge whether the text is malicious or not, which is also subpopulation shift dataset.
Py150 is a real-world code-completion dataset which is challenging given massive domains 8421, where the domain is formed according to the repository author. The goal is predict the next token given the context previous tokens. We evaluate models by the accuracy on the class and method tokens.
Table 6: Summary of selected datasets as well as their difficulty metric, where n is the number of samples, D is the number of domains, C is the number of clients used.
Dataset
Statistics
Difficulty Metric
D
C
Rix
(A)
À 0.1
À
FEMNIST
737036
2586
100
1.000
0.980
0.981
0.981
PACS
9991
100
0.960
1.000
1.000
1.000
IWildCam
203029
243
100
0.449
0.869
0.714
0.571
CelebA
162770
4
100
0.661
0.760
0.805
0.797
Camelyon17
410359
100
0.969
1.000
1.000
1.000
CivilComments
448000
16
100
0.984
0.618
0.533
0.532
Py150
150000
8421
100
0.969
0.999
0.998
0.998
C.2 DATASET PARTITION SETUP
For each dataset, we first partition the dataset into categories, namely training dataset, in-domain validation dataset, in-domain test dataset, held-out validation dataset and test domain dataset. For FEMNIST and PACS dataset, we use Cartoon and Sketch as training domain, Art-painting as held- out domain, Painting test domain. For training domain, training domain datasets as in-domain validation dataset and in-domain test dataset respectively. For IWildCam, CivilComments and Py150, directly apply the Wilds official partitions.
partition 10%, 10% of the total
D BENCHMARK EXPERIMENTAL SETTING
D.1 MODEL STRUCTURE
In this benchmark, for image-based datasets, we use ResNet-50 He et al. (2016) dataset. For CivilComments and Py150 dataset, we choose DistilBERT Sanh et al. (2020) and CodeGPTRaddord Radford et al. (019)) respectively recommended by Wilds (Koh al. 202
D.2 MODEL SELECTION
We conduct held-out domain model selection with 4 runs for each methods. The oracle model selection evaluates the model based the performance the held-out validation domain. The results are reported based on the best run.
D.3 EARLY STOPPING
We conduct early stopping using the held-out validation dataset in our evaluation. For each dataset and method, We first run certain communication rounds, then select the model parameters which achieves the best performance on the validation set. We report the held-out validation dataset the main paper, and report the results using the in-domain validation set in Appendix F
19

---

Published as a conference paper at ICLR 2024
D.4 HYPERPARAMETERS
In this section, we present the hyperparameters selected for the evaluation. We grid search times of starting with learning same ERM and other hyperparameters from the methods' original parameters. We than select the hyperparameter based on the best performance on the held-out domain validation set. Please refer the Table 7 to review all hyperparameters.
Table 7: Hyperparameters used in the evaluation.
Dataset Name
PACS
CelebA
Camelyon17
FEMNIST
IWildCam
CivilComments
Py150
Total Communication
80
20
20
40
50
10
10
ERM
10
x 10-3
x 10-a
1x 10
3 10-5
1x 1x-5
8 10
10-5
10-3
x 10-3
1x 10-3
3x 10-5
1x 10-6
x 10-6
IRM
penalty weight
1000
1000
1000
100
100
10
1000
Ir
3 10
10
x 10-4
1 10
1 10-3
1 10
1 10-6
meta-lr
0.1
0.1
0.1
0.1
0.1
0.1
Ir
3 10-5
3
10-4
1 10-3
1x 10-3
3 10-5
Mixup
0.2
0.2
0.1
0.2
0.2
3 x 10-5
0.0001
x 10-3
1 10
3 10-3
1x100 10-0
1 10m5
MMD
penalty weight
Coral
Ir
3 x 10
1 10
1 10-3
1x 10-5
3 10-5
1 10-6
5 10
penalty weight
3 x 10-5
3 x 10-4
x 10-3
10-3
x 10-5
1 10-5
x 10-3
GroupDRO
group
0.01
1x
1 10
x 10-5
x 10-3
10-4
10-5
3 10-5
10-5
10-6
FedProx
10-3
0.1
10-3
1x 10-3
10
Scaffold
10
10
1x
x 10
3 10
10
AFL
3 10-5
x 10
x 10-3
10-3
x 10-5
8 x 10-
8 10-6
Ir
3 x 10
x 10
1 10
1
x 10
FedDG
U(0,1)
(0)
U(0,1)
U(0,1)
Classifier
2 10-1
2 10-1
2
10-1
x 10-1
Gen Disc
7 10-4
10-4
7 10-4
10-3
10-5
FedADG
7 10-10-
10-4
10
10-3
1 10-5 0.2
0.15
0.25
0.1
0.05
Ir
x 1
10-4
1 10
10-4
3 10-3
5 10
3 10-6 0.01
FedSR
12 regularizer
0.01
0.01
0.01
0.01
0.01
0.01
emi regularizer
0.001
0.001
0.001
0.001
0.001
0.001
0.001
x 10
1 10
x 10
x
x 10
5 10
1 10
FedGMA
mask-threshie threshold
0.1
0.4
0.4
E ADDITIONAL FL-SPECIFIC CHALLENGES FOR DOMAIN GENERALIZATION
also include some deeper exploration over the effect of number of clients and communication frequency, which are unique to the FL. regime.
As mentioned in subsection 4.3
i) Massive number of clients: In this experiment, explore the performance of different algorithms when the number of clients C' increases on PACS. We fix the communication rounds 50 and the local number of epoch is 1 (synchronizing the models every epoch). Fig. 2 plots the held-out DG test accuracy versus number of clients for different levels of data heterogeneity. The following comments are order: given communication budget, 1) current domain generalization methods all degrade a lot after C' > 10, while the performance ERM and FedDG maintain relatively unchanged as the clients number increases given communication budget. FedADG and FedSR are are sensitive the clients number, and they both fail after C 2 20. 2) Even in the simplest homogeneous setting X 1, where each local client has i.i.d training data, current domain generalization methods IRM, FISH, Mixup, MMD, Coral, GroupDRO work poorly in the existence of large clients number, this means new methods are needed for DG in FL context when data are stored among massive number of clients. ii) Communication constraint: To show the effect of communication rounds convergence, plot the test accuracy versus communication rounds in Appendix Fig. 3] We fix the number of clients C = 100 PACS and decreases rounds of communication (together with increasing local epochs).
20

---

Published as a conference paper at ICLR 2024
PACS
ColebA
Camelyon17
0.8
0.95
Accuracy
0.6
0.4
0.4
0.2
0.93
50
100
150
200
50
100
150
200
50
100
150
200
number of clients
Scaffold
Fedsr
Fedgma
Figure 2: Performance based on accuracy versus the number of clients across the PACS, CelebA, and Camelyon17 datasets.
=
0.8
0.8
0.8
Accuracy
0.6
0.4
0.2
0.2
10
30
40
10
30
20
30
number of epoch per iteration
ERM
Scaffold
FedSR
FedGMA
Figure PACS: Held-out DG test accuracy vs. varying communications (resp. varying echoes
1, homogeneous
A= 0.1, heterogeneous
0, domain seperation
1.0
0.8
0.6
0.4
20
60
80
20
60
80
20
60
80
ERM
Fish
MMD
GroupDRO
Scaffold
FedDG
FedSR
Mixup
Figure 4: Convergence curve on PACS; total clients and training domains (C, D) (100, 2);
increasing domain heterogeneity from left to right: = 01,
= 1, homogeneous
= 0.1, heterogeneous
A 0, domain seperation
0.3
0.2
0.1
0.0
25
ERM IRM
Fish
MMD
GroupDRO
Scaffold
FedDG FedADG
FedSR
Mixup
Coral
Fed
AFL
FedGMA
Figure 5: Accuracy versus communication rounds for IWildCam: Total clients number C' =433; increasing heterogencity from left to right panel: À = (1, 0.1,
21

---

Published as a conference paper at ICLR 2024
That is, if the regime restricts the communication budget, then increase its local computation E to have the same the total computations. Therefore, the influence of communication on the performance is fair between algorithms because the total data pass is fixed. We observe that the total number of communications does not monotonically affect the DG performance. With decreasing number of total communication, most methods' performance first increase then decrease. This might be interesting implicit regularization phenomena in the FL context. Without discussing DG task, usually frequent communications lead to faster convergence. The relationship between DG performance and communications requires further exploration.
F SUPPLEMENTARY RESULTS
In the main paper, we provide experiments results using held-out validation early stopping. Here we report the results using in-domain validation set for reference. We also report the convergence curve for each methods on each dataset for reference. We observe that under most of cases, the held-out validation gives better model. Thus, recommend using held-out validation stopping when we can access multiple training domains. However, if the training dataset contains only 2-3 domains, should consider using in-domain validation set.
perform early
More results on PACS and IWildCam dataset.
Results on FEMNIST dataset. As mentioned in the main paper, we include the FEMNIST dataset here for reference. It is evident from Table 11 that À does not significantly influence the final domain generalization (DG) accuracy, and whether using in-domain validation or held-out-domain validation does not impact the final DG accuracy well. This suggests a lack of statistical heterogeneity across different domains. Furthermore, we observe that changing does not significantly affect the convergence. Most of the results do not converge the performance level of the centralized counterpart. This discrepancy arises from the challenge posed by the large number clients, where RA. 0.980 1.
homogeneous
A 0.1, heterogeneous
A domain seperation
0.8
0.4
0.0
0
20
40
60
80
0
20
40
60
80
0
20
40
60
80
MMD
GroupDRO
Scaffold
FedSR
Coral
Frox
FedADG
FedGMA
Figure 6: Accuracy versus communication rounds for FEMNIST; total clients and training domains
(K, M) (100, 2586); increasing domain heterogeneity from left right panel: 01,
G GAP TABLE
We list the gap table in Table 12 for summarizing the current DG algorithms performance gap w.r. FedAvg-ER.M in the FL context, in particular, positive means it outperforms FedAAv-ERM. negative means worse than FedAvg-ERM. can seen that the on the simple dataset, the best DG migrated from centralized setting is better than BedAvg-ERM. In the complete heterogeneity case, no centralized DG algorithms can be adapted to and FDG methods performs comparably good in this setting. However, they fail in harder datasets. In the hardest setting, currently the Federated methods dealing with data heterogeneity performs the best. It is worth noting that while federated learning methods that address client heterogeneity perform better than other methods. they still fall short of achieving centralized empirical risk minimization (ERM). This highlights the need for future research and development of DG methods in the FL regime.
22

---

Published as a conference paper at ICLR 2024
A= 1, homogeneous
A= 0.1, heterogeneous
A= 0, domain seperation
0.9
0.8
0.7
0.6
0.5
10
10
10
ERM
Fish
MMD
GroupDRO
Scaffold
FedDG
FedSR
IRM
Mixup
Coral
Frox
AFL
FedADG
FedGMA
Figure 7: Accuracy versus communication rounds for Camelyon17. total clients and training domains (K, M) (100, 4); increasing domain heterogeneity from left to right panel: À (1, 0).
= 1, homogeneous
0.1.1 heterogeneous
A= 0, domain seperation
0.8
0.6
0.2
0.0
20
30
30
ERM
Fish
MMD
GroupDRO
Scaffold
FedDG
Fed5R
IRM
Mixup
Coral
FedProx
AFL
FedADG
FedGMA
Figure 8: Accuracy versus communication rounds for CelebA; total clients and training domains K, M) (100, 2); increasing domain heterogeneity from left to right panel: X = (1, 0.1, 0).
23

---

Published as a conference paper at ICLR 2024
Table 8: Test accuracy on PACS and IWildCam dataset with held-out-domain validation where FedAvg-EEe AaMMA-Mg-MMM is the simple baseline (B). "-" means the method is not applicable in that context. Bold for best and italics is for second best in each column. We report the standard deviation among runs.
PACS (1) 2)
WwildCan (2) - 243)
C-1
C-100 (PL)
C-1
C (FL)
feentralized)
A
A-0.1 0.1
A-0
(cemmalized)
A 0.1
2
FodAvg-ESR EEM
0.943 +: 0.012
0.948 0.020
0.963 = 0.016
0.964 + 0.018
0.343 + 0.00V
0.298 = 0.002
0.245 4 0.0022
0.196 +: 0.010
DG Aduyeee
IRM
Fish
0 01007
0. 0.856
+ 0.00 021
0 + 127
0:920 =: 0.041
0.866 # 0.014
0.345 + 0.107
0.321 = 11.034
0.354 + +00.00
0.196 #: 0.006
0.185 0.044
0.154 +.0.0
0.179 0.005
Micup
0 0.000
0.666 + 0.062
0.825 + +-0.50
0.331 0.007
0.086 +0.000
0.072 + 0.007
MMD
0.928
0.002
0.921 1 0.041
0.855 1 0.044
0.524 0.002
0.185 0.005
0.163 0.007
DecyCooal
0.528
11007
0.9201 0.041
0.806 + 0.043
0.329 0.008
0. 0.380 0.000
0.163 0.007
GroupDRD
0.528
11007
0.944 0,023
0.940 0.015
0.211 010022
0. 0.000
0.198 + 0.005
=
FolProx
0,586 0.025
0.891 + 0.025
11806 + 0.114
0:261 + 0.008
0.204 + 0.000
0.167 + 0.I0E
Scaffeld
0.506 0.025
0.892 + 0.024
0.806 + 0.024
6. 0:881 = 0. 0.008
0 0.255 :: 0.004
0.176
608
AFL
0.026
0,915 0.000
0,911 0.041
0.256 0.010
0,054 0.001
FedLKG
0.020
0.947 0.018
0.915
6.687
0.274 0.061
0.235 0.002
0.104
FedADG
0.011
0.942 1 0.001
0,305 0.001
0.259 0.012
0.852 0.010
0,154 0.001
FoISR
0 0.640
0.086
0.000
0.177 0.005
+.005
0.003
KaIGMA
0.760 0.050
0.790 0.087
0.721
0.000
Table 9: Test accuracy on CelebA and Camelyon17 dataset with held-out-domain validation where
FedAvg-ERM is the simple baseline (B). "_" means the method is not applicable in that context. Bold is for best and italies is for second best in each column. We report the standard deviation among 3
runs.
CelebA
Conelyon17 3)
C- (FI.
(FL.
(oontalizal)
A-O
(semtoliral)
FRlAAg-FRM
0,760
0:606:t
0.517
0.446
0,9,9
0.950 0.004
0.946 0.004
DNG Adigtes
Fish
IRM
0.144
0.206::0:00
0.912
0.002
0.881
0.948
Mixup
0.288
0,498
0:27
0.125 0:218
0.340 0.002
0.949
0.003
MMD
0.866
0.01
0.027
0.706 + 0.090
0,022 0.027
0.946
0.877 0.000
DocyCooall
11822
0000
0,813 0.017
0.782 0.022
0.940 + 0.007
0.948
0.0014
0380
GreupDRO
0.888
0.004
0.511 0.088
0.761
+ 0.056
0.910 0.027
0.947 :: 0.006
0.953 0.001
FedProx
0:126 0:237
0.121 0:265
0.000 0.000
0.500 0.002
0.035 0:000
0.907 007
Scaflold
0.728 ::: 0.010
0.776 =: 0.016
0.800
1: 0.077
0.942 :: 0.0011
0.929 1 0.005
0,332
=: 0.001
AFL
0.811 1: 0.024
0.894 1 0.000
0,825
1: 0.000
0.942 1: 0.011
0,947 1 0.006
0.932 1 0.008
FedLGG
0.361 1: 184
0.581 4 0.112
0,464 =: 0.200
0.363 2: 0.012
0.866 à 0.005
0,822 :: 0.015
P0G
FolADG
0.674 + 000
0.669 + 0.005
0.600 600
0.906 0.001
0.933 + 0.004
0.925 + 0.002
RaISR
0.381 + +0.018
0.365 + 0.041
0.3K3
+ 0.015
0.934 + 0.010
0.17 +.000
0.925 + 0.003
FedGMA
0.620.1.0000 0.0112
0.615 1 0.018
0,657 0,008
0.901 1: 0.006
0.854 1 0.006
0,815 1 0.025
Table 10: Test accuracy on CivilComments and Py150 datasets with held-out-domain validation where FedAvg-ERM is the simple baseline (B). "_" means the method is not applicable in that context. Bold is for best and italics is for second best each column. FedDG and FedADG are designed for image dataset and thus not applicable for CivilComments and Py150. We report the standard deviation among runs.
CivitCommenss (D) 26)
PL150 G877)
C- (HL)
C- 100 (FL)
(contmlizal)
A-1
x-0.1
A-0
(ccntralized)
A
A-0
FodAAg-FRM
0.541 0.003
0.359 0.028
0.317-0.015 0.018
0.325 0.009
0.883 + 0.003
0.683
0.000
0.650 0.000
0.601 + 0.000
DG Adaped
TRM
0:611 0.001
18587 + 01.
0.531-000 0.053
0.677 + 0.000
0.655 0.001
0.653 +
0.627 0.001
Fish
0.671
0.000
0.424 1. 0.158
0.340 = 0.173
0.063 0.000
0,050 1: 0.000
0.648
0.001
0.642 1 0.001
MMD
0.638
0.000
0.634 = 0.012
0.618 0.014
0.606 0.001
0.627 = 0.000
0.62)=0.000 0.000
0.5:0 + 0.000
DecpCCoal
0.586= = 0,000
0.515 0.061
0.661 = 11,150
0.600 = 00003
0.651
2: 0.000
0.601 (1001
011530
= 0.000
GRo
0.638 = 0.003
0.475 +: 0.014
0.474 +: 0. 063
0.515 05155+00000
0.580 + 0.000
0.603 0.003
0.907 : 0.002
FodProx
0.181 0,502
0.374 = 110333
I1177 =: 0.0066
0UG88 d: 0.001
0,63
10000
0.622 à: 0,100
FDXI F.
Scathold
0.389 9.015
0,376 = 11011
01302 0.014
0,642 2: 0.0011
0.688
10000
0.617 0.100
AFI.
0.552 + 0.021
0.174 - 0.014
0.485
0.019
0.496 + 0.000
0.003 0.465
0.467 + 0.000
FoiSR
0.360 0:003
0.338 0.003
0.319 + 0.005
0.533 + 0.002
0.526 0.001
0.645 + 0.005
FeGGMA
0.209 0.028
0.024
0.200 0.022
0.020 0.000
0.613
0.000
0.000
H TRAINING TIME, COMMUNICATION ROUNDS AND LOCAL COMPUTATION
In this section, provide training time per communication in terms of the wall clock training time. Notice that for a fixed dataset, most of algorithms have similar training time comparing to FedAvg-FRRM, where FedDG and FedADG are significantly more expensive.
24

---

Published as a conference paper at ICLR 2024
Table 11: Test accuracy on FEMNIST dataset with held-out validation where FedAvg-ERM is the simple baseline (B). "L" means the method is not applicable in that context. Bold for best and italics is for second best in each column.
FEMNIST (Held-Out-Domain)
C'
C 100 (FL)
(centralized)
==1
X 0.1
== 0
FedAvg-ERO
0.854 + 0.001
0.837 = 0.003
0.838 + 0.002
0.834 = 0.001
IRM
0.844 + 0.001
0.832 = 0.007
0.822 + 0.005
0.821 = 0.004
DG Adapted
Fish
0.849 = 0.002
0.833 0.004
0.829 + 0.003
0.826 0.008
Mixup
0.834 + 0.002
0.828 = 0.009
0.821
0.816 = 0.005
MMD
0.844 = 0.003
0.843 0.006
0.829 0.006
DeepCoral
0.846 = 0.002
0.840 0.005
0.836 = 0.002
0.851 = 0.003
GroupDRO
0.841 + 0.004
0.835 0.005
0.814 + 0.002
0.805 0.003
FedProx
0.865 0.001
0.864 = 0.001
0.863 0.001
Scaffold
0.860 0.002
0.859 + 0.002
0.858 0.002
AFL
0.629 = 0.010
0.593 + 0.037
0.584 = 0.020
FedDG
0.859 + 0.002
0.857 + 0.002
0.856 + 0.002
FDG
FedADG
0.817 0.026
0.800 = 0.061
0.785 0.025
FedSR
0.832 0.015
0.832 0.020
0.831 0.014
FedGMA
0.849 = 0.010
0.842 = 0.004
0.837 0.003
Table 12: Gap Table: Current Progress in solving DG in FL context
CivilComments
Domain Seperation
Free of Data Leakage
IWildCam
1amelyon17
PACS
CelebA
PyLSO
Centralized adopted methods
-0.010
-0.047
+0.265
+0.003
+0,266
+0.003
Federuted methods
-0.010
0.010
+0.319
-0,033
+0,266
-0.012
FedDG
X
-0.008
-0.010
+0.014
00094
FedADG
-0.013
0.060
+0.152
0.017
FedSR
-0.407
-0.104
-0.154
-0033
-0.008
-0.124
FedGMA
-0.225
0.094
+0.098
-0.096
-0.145
-0.037
Table 13: Wall-clock Training time per communication (unit: s). There might be variance due the machine's status and resource availability.
PACS
IWildCam
CelebA
Camelyon17 C= 100
CivilComments
Py150
FEMNIST
C 100
C 243
C= 100
C 100
C= 100
C= 100
FedAvg-ERM
143
6301
298
845
3958
6566
262
IRM
147
6454
309
1163
4085
7089
297
DG Adapted
Fish
148
7072
312
1003
5483
7770
324
Mixup
144
6294
302
933
2641
MMD
144
6663
312
963
4024
7603
287
DeepCoral
144
6597
313
879
3901
7212
287
GroupDRO
145
9311
310
750
4690
8121
307
FedProx
169
6921
1219
4310
4502
6513
288
Scaffold
167
6876
1344
4623
4421
6389
281
FedDG
352
32172
9232
13784
989
FedADG
181
11094
6502
9945
907
FedSR
151
7136
2267
6567
4403
8020
280
FedGMA
143
6795
558
2036
4525
65455
261
25

---

Published as a conference paper at ICLR 2024
26


Reviews:
[{"review_id": "RYl98WV4Jy", "reviewer": "Reviewer_PJ3y", "summary": "This paper introduces a benchmark for federated domain generalization, which is a challenging problem that requires learning a model that can generalize to heterogeneous data in a federated setting. The paper presents a novel data partitioning method that can create heterogeneous clients from any domain dataset, and a benchmark methodology that considers four factors: number of clients, client heterogeneity, dataset diversity, and out-of-distribution generalization. The paper also evaluates 13 Federated DG methods on 7 datasets and provides insights into their strengths and limitations.", "questions": "No.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper proposes a comprehensive and rigorous benchmark for Federated DG that covers various aspects of the problem and can be easily extended to new datasets and methods.\n* The paper provides a clear and detailed description of the data partitioning method and the benchmark methodology, as well as the implementation details of the methods and datasets.\n* The paper conducts extensive experiments and analyzes the results from different perspectives.", "weaknesses": "The paper mainly summarizes the experimental observations, but does not offer much theoretical analysis or explanation for why some methods perform better than others in certain scenarios, which would be helpful to gain more insights into the FDG problem and the design of effective methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5GdOYhI2b", "reviewer": "Reviewer_AnGM", "summary": "This paper proposes a benchmark for domain generalization (DG) in federated learning. Specifically, they (1) develop a novel method to partition a DG dataset to any number of clients, (2) propose a benchmark methodology including four important factors, (3) experiment with a line of baselines and datasets.", "questions": "1. In Eq. (1), two kinds of DG is mentioned. Which kind of DG is mainly used in your benchmark? \n2. The performance for some algorithm, for example, FedSR, is very low, and consistently lower than FedAvg. Could you explain the reason behind?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper contributes to an important topic. I believe federated DG is an important problem in FL, especially cross-device FL when the trained FL global model need to generalize to a large amount of clients that do not participate in FL training. \n2. The paper is well-written and easy to follow. \n3. The proposed benchmark includes a variety of datasets and algorithms. \n4. The experiments regarding the number of clients is highly related to cross-device FL, and indicate an important drawback in the current federated DG experiments.", "weaknesses": "1. In the context of domain generalization, we are particularly concerned with whether models trained on limited source domains can generalize to new target domains. This paper also uses held-out domains (in Appendix C.2). I believe that this aspect should be more explicitly explained in the main body of the text; otherwise, readers might easily misconstrue that all domains were used to construct training clients, which is misleading. \n2. Error bars are not provided for experiment results, the conclusion may be influence by random fluctuation. \n\nMinor: \n1. Page5 line 12: homogeneous -> homogeneity\n2. When considering the second kind of DG, it is relevant to “performance fairness” in FL, which encourage a uniform distribution of accuracy across clients. Although works in this direction might emphasize more on participating clients, I believe at least the AFL algorithm [1] can be a good supplement to the benchmark. \n\n[1] Mehryar Mohri, Gary Sivek, Ananda Theertha Suresh. Agnostic Federated Learning. ICML 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJqZdGYbIQ", "reviewer": "Reviewer_EtKR", "summary": "This paper explores the intersection between Federated Learning and Domain Generalization, namely, Federated Domain Generalization (FDG), in which different domains are separated among clients that will collaboratively learn a model that generalizes to unseen domains. This paper pushes on an important direction: on the methodology behind evaluating FDG algorithms. In this sense, the authors: (i) Present an interesting review of existing practice in FDG; (ii) Propose a novel way of partitioning clients in FDG; (iii) Propose new metrics for evaluating the hardness of benchmarks; (iv) provide extensive evaluation of FDG methods.", "questions": "__Q1.__ Since Domain Adaptation is a related problem to Domain Generalization, is it possible to apply the proposed partitioning scheme to Federated Domain Adaptation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper plays the same role of [Gulrajani and Lopez-Paz, 2021] for FDG, i.e., an important paper that provides an in-depth discussion about how to evaluate existing methods. In this angle, the paper provides extensive experimentation and interesting insights. In this sense, the paper is quite important for the field.\n\nFurthermore, as the authors discuss in the paper, the federated setting poses __new challenges__ to DG. This is especially related to the new partition method that the authors propose, and the main novelty of the paper. This contribution helps merging the fields of federated learning and domain generalization, making the evaluation of FDG algorithms more realistic. As a consequence, this direction is quite impactful and helpful for the field of FDG.", "weaknesses": "Overall, I have no major concerns with this paper. My only critique is a (minor) lack of clarity. In section 4., the term ERM is never defined, and the empirical risk minimization method is never properly described. While, for knowledgeable audiences, this does not hurt the understanding of the paper in general, this makes the paper harder to read for beginners. I suggest authors add a description of the ERM in page 3, on the Federated DG paragraph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)