Benchmarking Algorithms for Domain Generalization in Federated Learning
Authors: Ruqi Bai
Saurabh Bagchi
David I. Inouye
Venue: Submitted to ICLR 2023 (2023)
Keywords: Domain Generalization
Federated Learning
Benchmark.

Abstract: In this paper, we present a unified platform to study domain generalization in the federated learning (FL) context and conduct extensive empirical evaluations of the current state-of-the-art domain generalization algorithms adapted to FL. In particular, we perform a fair comparison of nine existing algorithms in solving domain generalization {either centralized domain generalization algorithms adapted to the FL context or existing FL domain generalization algorithms } to comprehensively explore the challenges introduced by FL. These challenges include statistical heterogeneity among clients, the number of clients, the number of communication rounds, etc. The evaluations are conducted on three diverse datasets including PACS (image dataset covering photo, sketch, cartoon, and painting domains), iWildCam (image dataset with 323 domains), and Py150 (natural language processing dataset with 8421 domains). The experiments show that the challenges brought by federated learning stay unsolved in the realistic experiment setting. Furthermore, the code base supports fair and reproducible new algorithm evaluation with easy implementation.

Under review as a conference paper at ICLR 2023
BENCHMARKING ALGORITHMS FOR FEDERATED Do- MAIN GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we present unified platform to study domain generalization in the federated learning (FL) context and conduct extensive empirical evaluations of the current state-of-the-arr domain generalization algorithms adapted FL. particular, we perform fair comparison of 11 existing algorithms in solving do- main generalization either centralized domain generalization algorithms adapted to the FL context or existing FL. domain generalization algorithms to compre- hensively explore the challenges introduced by FL. These challenges include sta- tistical heterogeneity among clients, the number of clients, the number of com- munication rounds, etc. The evaluations are conducted on five diverse datasets including PACS (image dataset covering photo, sketch, cartoon, and painting do- mains), FEMNIST (image dataset containing writing digits and characters written by more than 3500 users), iWildCam (image dataset with 323 domains), Py150 (natural language processing dataset with 8421 domains) and CivilComments (natural language processing dataset with 16 domains). The experiments show that the challenges brought by federated learning stay unsolved in realistic ex- perimental settings. Furthermore, the code base supports fair and reproducible evaluation of new algorithms with little implementation overhead.
INTRODUCTION
Federated learning (FL) Konečný et al. (2016) is a distributed machine learning approach that as- sumes each client or device owns local dataset and this local dataset cannot be exchanged or centrally collected because of privacy or communication constraints. Given this context, a natural paradigm for FL (e.g., FedAvg McMMhan et al. (207)) is to alternate between two stages: clients locally update the model based on its local dataset and a central server aggregates client models. Because the clients may be phones, network sensors, hospitals, or alternative local information sources, the local datasets are naturally heterogeneous between clients. Specifically, there are at least two types of realistic statistical data heterogeneity in the FL context. Client heterogeneity is the data heterogeneity between clients involved in training c.g. hospitals may different stain- ing procedures or imaging equipment. Train-test heterogeneity is the data heterogeneity between the new client that not involved in training or
training and testing data-e.g. the performance a natural shift in real-world test data due to changes over time, location, or context.
Client heterogeneity has long been considered statistical challenge since federated learning was introduced. FedAvg McMahan et al.] (2017) has experimentally shown that their methods effec- tively mitigate some client heterogeneity. There are many other extensions based the FedAvg framework tackling the heterogeneity among clients in FL Hsieh et al. (2020); Li et al. (2020); Karimireddy et al. (2020). There is an alternative setup FL, known as the personalized setting, which aims to learn personalized models for different clients to tackle heterogeneity. Numerous recent papers have proposed FL. models and algorithms to accommodate personalization Smith et al. (2017): Chen et al. (2018); [Hanzely et al.] (2020): [ Dinh et al. (2020): Deng et al.) (2020); Acar ct al. (2021). However, these prior works only train the model on simple data sets such as MNIST, EMNIST, and CIFARIO and the client heterogeneity is constructed mainly through class imbalance, which assumes the ratio of data from each class is different for each client but the class conditional distributions homogeneous across clients. Class imbalance is special kind of heterogeneity called prior probability shift. In practice, due to the difference between the location of the local data collector (cameras, sensors, etc), real data heterogeneity is more complex than simple class

---

Under review as a conference paper at ICLR 2023
Split
Homogeneous (A=1)
Heterogenous (A=0.1)
Domain Separation (A=D)
Train test Heterogeneity Traiiing
Client
Client
Client 1
Vatdation,
Domatins
Domain Split K
Client 2
Client 2
Client
Domain Split when K
Client
Client
Client 3
Our benchmark simultaneously evaluates frain-test heterogeneity (i.e., domain general- ization) in the top left and client heterogeneily across domains by splitting the domain data amongst clients. The client heterogeneity can be homogeneous (left), heterogencous (center), or domain separated (right), where M the number of domains. K is the number of clients, color denotes domain data, and À is the domain balance parameter. The right three panels demonstrate the domain split for homogeneous, heterogeneous, and domain separation when K S M.
Figure 1:
imbalance as in many prior works. Furthermore, most prior FL works do not consider train-test heterogeneity but rather assume the train and test data are i.idd
While most FL works have usually ignored train-test heterogeneity, the task of domain generaliza- tion (DG) Blanchard et al. (2011) formalizes special case of train-test heterogeneity training algorithm has access to data from multiple source domains and the goal is to perform well on data from wnseen test domain. There is an active line of research
which the
domain generalization in the centralized setting Muandet et al. (2013); (aito et al. (2018); ; Ganin Lempitsky (2015);
Long al. 2015); Arjovsky et al. (2019), Sagawa al. (2019) Shi al. 2021 Li et 2018))
Current benchmark papers Koh et al. (2021) Gulrajani & Lopez-Paz (2200) provide thorough com-
parisons between different algorithms for DG. However, these centralized DG benchmarks do not consider the unique constraints of the FL context. In particular, they fail to provide insights on how the client dataset heterogeneity, the number of clients, and the communication budget will influence the generalization ability. From the FL. side, to the best of our knowledge, there are currently only four published works, FedADG FedADG/Zhang (202)) and FedGMA Tenison et al. (2022) that tackle DG in the FL context but their evaluations the number and diversity of domains. FedDG Liu et al. 2 only evaluates on retinal fundus images with 4 domains and prostate T2-weighted MRI images from 6 domains, and FedADG only focuses on PACS and VLCS, which share similarities in terms of number of domains (each with only 4 domains) and sample size. FedGMA Tenison et al. (2022) only focuses on ColorMNIST with few domains. However, consider more realistic datasets with thousands of domains. See Sec. 4.2 for details. 2) Their evaluations are restricted to the case when the number of clients is equal to the number of domains, which may be an unrealistic assumption (e.g., hospital that has multiple imaging centers vice that is used in multiple locations). For example, FedSR Nguyen et al. (2022) and FedADG Zhang et al. 202T) only evaluates on the case when the number clients equals to the number of domains. However, we show in this paper, FedSR Nguyen et al. (2022) and FedADG ZZang al. 2222T) are sensitive to clients number, and they fail even on the simple dataset PACS when the clients number large than 20. See Sec. 4.3 massive number of clients for detail. 3) They fail to show the network effect of FL; in particular, neither work considers the influence of the number of clients the performance, and FedADG Zhang al. (2021) does consider the effect of the number of communication rounds. In summary, the current DG benchmarks fail to consider challenges unique to FL, and the few FL methods for DG have limited evaluations.
al. (202T). FedDG Liu et al. (20TT). [guSenn et al.)
limited in the follwing senses: 1) The evaluation datasets are limited
de-
Therefore, more systematic evaluation is needed both to aid in systematic progress at the intersection of FL and DG but also aid in answering many open questions. For example, how does client
2

---

Under review as a conference paper at ICLR 2023
domain heterogeneity influence the performance of current algorithms? What is the performance of a direct translation of centralized DG algorithms (if applicable) to the FL context? How does the performance of current algorithms scale with the number of clients and communications rounds on complicated real-world datasets?
This work addresses the above questions, and our contributions can be sum- marized as follows. 1) We propose a standardized definition of client domain heterogeneity that unique to the FL context and interpolates between domain homogeneity and domain separa- tion (see Figure Tp while limiting the class imbalance. In particular, we develop an experimental setup method to split dataset domain samples among any number of clients (see (subsection 3.1). 2) We provide a fair comparison over multiple representative centralized DG methods adapted to the Fl. context as well as four prior works on federated domain generalization on five different benchmark datasets. 3) We also explore the impact on the generalization ability of client domain heterogeneity, the total client number, communication rounds, which are unique to the FL. con- text. 4) From these results, we identify significant generalization gaps between centralized do- main generalization and domain generalization in the federated Learning setting. 5) We release an extensible open-source code library for studying domain generalization in the FL context (see https: com/ anonymous
Major contributions:
m1 http///bnnaakiingeeeeeeer
BACKGROUND AND RELATED WORK
Domain generalization. Domain generalization (DG) task Blanchard et al. (201)) tackles the clas- sification problem where the domains of test and training dataset are heterogeneous, i.e., the test data comes from an unseen domain distribution. Existing DG algorithms design alternative objec- tives seeking for a good approximation usually by utilizing training data from multiple domains. Schölkopf et al. (202T) claims that current machine learning model only captures superficial statis- tical correlation rather than underlying causal relationship between data and their labels. Inspired by causality Lopez-Paz Paz et al. (2017): Peters et al. ) Seeinze-Demll al. (201 Arjovsky et al. (201)) proposes IRM seeking invariant predictor on top of the representation. Li et al.]((20))) in- stead, explicitly tries to align the feature representation distribution by minimizing the maximum mean discrepancy (MMD) Gretton et al. (2006). Similarly, [Sun & Saenko (2016) proposes CORAL which utilizes the second-order information. Besides invariant representation, Shi et al. (2021)
solves the DG problem using regularization, the proposed FISH algorithm is a first-order method aligning the gradient directions generated by data from different domains. [Sagawa et al. (2019) introduces the objective that tackles the worst-performed domain. The proposed algorithm did not directly optimize the objective by solving the minimax objective, but to solve ERM while increas- ing the importance of domains with larger loss during training. ZZhang et al. (2017) proposes straightforward data augmentation principle called Mixup that trains neural network on convex combinations of data from two distinct domains.
Federated Domain Generalization: Limited works seek to solve the DG task in the FL context. Liu et al. (202T) recently proposed FedDG, a federated learning paradigm specifically designed for medical image classification. The proposed method requires sharing the amplitude spectrum of images among local clients. Zhang al. (20II) applies generative adversarial network (GAN) Goodfellow et al. (220) in the FL. context, where each client contains four models: a featurizer Φ a classifier w. generator G and discriminator F. The featurizer &b and classifier w solves the classi- fication task, while the generator G learns the feature distribution Z. During local training. FedADG Zhang al. 2222 ffirst trains the featurizer & and classifier w by minimizing the empirical loss, then trains the generator G and the discriminator F using the GAN Goodfellow et al. (2020) approach. After the local training, the central server aggregates the local featurizer. classifier and generator. However, this method requires training 4 models together along with tens of hyperparameters to tune, which makes it hard to converge empirically. [Tenison et al.] (2022) proposes a new aggrega- tion method called Federated Gradient Masking Averaging (FedGMA) with the goal of improving generalization across clients and of the global model. Their gradient masking prioritizes gradient components that are aligned with the overall dominant direction across clients while the inconsistent components of the gradient are given less importance. Yuan et al. (222T) introduced the concept par- ticipation gap to identify dataset heterogeneity. They train models using a set of participating clients and examine their performance on held-out data from these clients as well as an additional set of non-participating clients. Therefore it is closely related to domain adaptation in FL context, which
3

---

Under review as a conference paper at ICLR 2023
is different than the domain generalization setting considered in this benchmark, i.e., the training and test domains do not overlap. Nguyen et al. (2022) proposed FedSR where they enable domain generalization while still respecting the decentralized and privacy-preserving natures of FL context by enforcing f2-norm and a conditional mutual information regularizer on the representation. In TTall I] we give overview of domain heterogeneity across two dimensions: 1) between training and testing datasets (i.c., standard vs domain generalization task) and 2) among clients (i.e., domain imbalance between clients). While some work has considered the standard supervised learning task (left column), new fair evaluation is needed to understand the behaviour of domain generalization algorithms in the federated context including the influence of data heterogeneity, communication budget, and the number of clients.
Table 1: Different Tasks with Domain Heterogeneity in the FL Context
Between training and testing datasets
Among clients
Standard Supervised Learning
Domain Generalization (our focus)
Homogencous (A 1)
Standard FL with Domain Homogeneity
Federated DG with Domain Homogeneity
Heterogeneous (0 < À < 1)
FL with Domain Heterogeneity
Federated DG with Domain Heterogeneity
Domain separation (A- 0)
FL with Domain Separation
Federated DG with Domain Separation
3 PROBLEM SETUP
3.1 DOMAIN HETEROGENEITY.
Implementation of domain heterogeneity We now provide a concrete procedure for implement- ing domain heterogeneity for the benchmark (see Figure I for an illustration). Given the number of training samples for all M domains, denoted by LJ m=1 where n'm is the number of samples for domain m, we first assign "primary" domains Ps S (1,2...,,M] M} to each client via the domain split KK function defined in Algorithm [ i.e. [P SpmainSplit(( K, 1 Algorithm| | carefully LM
Algorithm DomainSplit function where w.L.ogg. the domains assumed to be in de- scending order, i.e., n1 2 n22 2 ... 2 n.M-
Input K, if K s M then
Vk, P& < 0 for m 1....... M do
ks E arg mill& Emern, Vmm Pa- t Pa- U {mm}
handles two cases: fewer clients than domains (K < M) and more clients than domains (K > M). In the first case, the domains are sorted in descend- ing order and are iteratively assigned to the client k. which has the smallest number of training sam- ples En' Py- Ram currently. In this way, the al- gorithm outputs {P&) such that no client shares domains with the others but attempts to balance the total number of training samples between clients. In the case K > M, we first assign the domains one by one to the first M clients. Then, starting from client k = .M + 1, we iteratively split the largest domain m², where the samples are evenly split among all clients where m² Ps. In this way, some clients may share one domain, but no client holds two domains simultaneously. Again, this also attempts to balance the number of samples across clients as much as possible.
end for else if K > M then
Yk: E {1,2, 2, ..... M). P t [k}
for k = M + 1,..., K do
m E arg max, m
Σ _
Pa + {m*} end for end if Output Ps
After selecting the primary domains Pte for each client, we define the training sample counts, denoted nm (A), for domain m, client k. and domain balance parameter À E 10,
T (A) x T'm K
1 1/m E Px] -7mm. ΣK
+1 A)
[Im E Px]
where rounding to integers is carefully handled when not perfectly divisible and where 1[:] is an indicator function. This is simply a convex combination between a uniform splitting of domains among clients (i.e., the am term) and splitting where each client has disjoint set of domains (i.e.,
me., 1i tulms- K > -M and then we try to split domains evenly based on number the ΣØ_, [M]
of samples as defined in Algorithm
After defining 7m, (A), can denote the total training samples related client l with domain balance parameter À n*(A) EmE 1mm,k (A).
4

---

Under review as a conference paper at ICLR 2023
3.2 FEDERATED DOMAIN GENERALIZATION
In the federated domain generalization problem, are interested in collaboratively training model across the clients to perform well on unseen test domains D E Dies, which is different from training domains 1,.... M. Therefore, constructed from each client (0) = E(z,)))D, [e(((z, y); 0)) = (1/ng) Σn f(rt Viz 0), but on the unseen test domains, in either average (or worst-case sense) as defined below
focus minimizing not the average loss the source domains
min DD ((((((
or min sup ((z.g)> (ll((l,, 0)]] D-Diie
(2)
To solve equation Z in the federated learning context, we need consider client domain heterogene- ity, as it can introduce unique challenges for domain generalization. In particular, local client may not have access to all the domains, and their domains may only overlap partially; in the extreme case, local data included in one client may naturally form a unique domain. Therefore, whether cen- tralized generalization ability is achievable when local client has heterogeneous domains remains unknown. Further, if some distributed algorithms as if the server has information of all the domains, the communication cost remains unclear. In addition, the impact the generalization ability of the total clients number remains unknown.
achieve the centralized generalization ability
Adapting Centralized DG Methods to FL Setting. To adapt centralized methods, we simply run the centralized DG method at each client locally with their own local dataset (see (subsection 3.1 for how the local datasets are created), and then compute an average of model parameters at each communication round (see next paragraph). This approach is straightforward for the homogeneous (A 1) and heterogencous (A = 0.1) settings where each client has data from all domains- albeit quite imbalanced for A 0.1. This can be seen biased updates at each client based on biased local data. Similarly, this approach can be implemented in the domain separation case if at least one client holds multiple non-overlapping domains (i.e., 3k, Pisl 2 2, which would happen if K < M). However, if all clients only have one primary domain, i.e., Vk, [Ps] 1, which will happen 2 M, this simple approach cannot be extended to the domain separation setting (A = 0) because centralized DG methods require data from at least two domains. In fact, these centralized DG methods degenerate to ERM there only one domain per client. Extending these methods to the case where all clients only have one domain is an interesting direction for future work.
Synchronization Schedule and Batch Creation For all algorithms, we run E epochs locally on each client and then the server computes weighted average of the resulting models, i.e., gt-1 Ek The
n 05+1, Because each epoch runs through the whole dataset, the k-th client runs Em Rlm.///BBBBBB.B. B batches, where B denotes the batch size. For FedAvg (ERM) and FedGMA, we uni- form randomly sample a batch from the local dataset without considering the domain labels. For the FL. adaptations of centralized algorithms, we use the sampling method from the WILDS benchmark Koh et al. (02T), namely each client uniform randomly samples two domains from its local dataset and then uniform randomly samples B/2 examples from each domain without replacement. Be- yond simple model averaging, FedGMA Tenison et al. (2022) additionally adopts a global masking operation at the server for update changes. We refer the readers to Appendix B.2 for the detailed hyperparameters including learning rate, batch size, and model selection.
4
MAIN RESULTS
We adapt six representative centralized DG methods into FL context, include IRM Arjovsky et al. (2019). Fish Shi et al. (202T), MMD
Li et al. (2018). Coral [unn & Saenko (2016). GroupDRO
Sagawa al. (2019). Mixup Zhang al. (2117) and compare them with FedDG Liu et al. 202
FedADGiZhang et al. (2021). FedSR Nguyen et al. (2022) and FedGMA Tenison et al. (2022 which
naturally designed for solving domain generalization tasks in federated learning.
4.1 BASELINE SETTING (PACS AND FEMNIST-62)
In the baseline setting, we consider three domain heterogeneity regimes image classification tasks PACS Li et al. (2017) widely used in domain generalization, and on FEMNIST-62 (digits and characters) Cohen et al. (201)). prototype dataset in the FL context.
5

---

Under review as a conference paper at ICLR 2023
PACS. We evaluate PACS with K 100 clients, M = 2 training domains (cartoon, sketch), validation domain (art painting) and 1 test domain (photo). The maximum communication rounds is set to 50 with 1 local epoch per communication. The domain balance parameter was varied E 11.011.00 0). For the domain separation case À = O, each client locally only has one training domain (i.e., Vk, [P&| 1); in this case, centralized methods suitable (see subsection 3.2 so we only compare the four federated domain generalization methods to ERM.
Figure 2 plots the held-out test accuracy versus communication rounds on PACS with increasing domain heterogeneity. As seen from Figure 2 1) most algorithms perform reasonably well in the homogeneous case (λ 1) except FedADG and FedSR. In fact those two fail in all three cases. They are sensitive to the client number and work favorably when K is small, e.g. K = 2, see [subsec- fion 4.2 for discussion on the effect of number of clients. 2) As domain heterogeneity increases, i.e., À from to 0.1, the algorithms consistently converge slower and have worse test accuracy, which demonstrates that domain heterogeneity among clients
unique extra challenge introduced by FL. In particular, the centralized DG methods FISH, CORAL, IRM, and MMD extended to the FL setting have poor performance compared with that of ERM in the heterogencous case, while Group- DRO outperforms ERM both in homogeneous and heterogeneous case. 3) In the domain separation case, because K > M for PACS, each client locally only holds one training domain, and thus no centralized methods are suitable to use. FedDG requires sharing the amplitude spectrum of images among local clients, which causes privacy concerns. Therefore, even for this dataset containing only four domains total, prior works struggle to compete with ERM.
= homogeneous
A= 0.1, heterogeneous
A 0, domain separation
0.8
DS Tost Accurny
0.6
0.4
0.2
20
50
30
10
40
Communications
Communications
Communications
Fish
Coral
GroupDRO
FedADG
--- FedGMA
IRM
MMD
Mixup
FedDG
Fod5R
Figure 2: Accuracy versus communication rounds for PACS; total clients and training domains (K, M) (100, 2); increasing domain heterogeneity from left to right: À 01,
We also report the final test accuracy using held-out-domain validation in Table 2 similar results with in-domain validation is deferred to Table. 3 in See SecAAppendix C It shows as the domain heterogeneity increases (A from 1 to O), the test accuracy of current centralized meth- ods (except GroupDRO) degrades dramatically using either validation principle. For the cur- rent four natural FL algorithms, only FedDG works comparably to ERM, FedGMA performs worse than the above two on PACS, and be- cause FedADG and FedSR are sensitive to the number of clients, they both fail to converge in this case as K 100.
Table 2: Test accuracy on PACS dataset with held- out-domain validation; total client number K 100; total training domain number M = 100. N.A. refers to not applicable.
Centralized A ==1
x= ==1.1
A= 0
ERM IRM
0.8389
0.8766
0.9144
0.9377
0.9180
0.8156
0.5449
N.A.
Fish
0.9030
0.8497
0.6311
N.A.
Mixup
0.8635
0.7653
0.5551
N.A.
MMD
0.9186
0,8150
0.6341
N.A.
Coral
0.9216
0.8150
0.5515
N.A.
GroupDRO
0,9060
0.9395
0,9437
N.A.
FedDG
0.8922
0.9234
0.9275
0,9521
FedADG
0.8922
0.1892
0.0592
0.0598
FedSR
0,8754
0.1246
0.1263
0.1257
FEMNIST-62 (digits and characters). We consider the FL prototyping dataset FEMNIST-
FedGMA
N.A.
0.882
0.8467
0.8446
62 with K = 100 clients and treat handwritten digits and characters each users created as natural domains. We split into M = 2586 training domains, 320 validation domains, and 331 test domains. The maximum communication rounds is set to 50 with local epoch per communication. The do- main balance parameter is varied x E {1, 0.1, 0}. For the homogeneous and heterogencous cases, each client locally holds M 2586 training domains. The data are divided according to Eqquation 1

---

Under review as a conference paper at ICLR 2023
For the domain separation case, given that K < M, we use Algorithm[] I split the domains each client. Therefore, each client holds multiple non-overlapping domains. All the methods are ap- plicable this dataset. We observe that the domain balance parameter À decreases from 1 to 0, FedGMA, FedSR are consistently comparable to ERM while the others fail. The in-domain and Sec Dee/Appediix C.
held-out domain accuracy reported in Table 7
homogeneous
A 0.1. heterogeneous
O. domain separation
0.8
DG Test Accuray
0.6
0.1
0.2
20
30
40
20
30
10
20
30
10
Communications
Communications
Communications
0-
ERM
IRM
Fish
-+-
MMD
-A- Coral
Mixup
GroupDRO
FedSR
FedGMA
Figure 3: Accuracy versus communication rounds for FEMNIST; total clients and training domains MK, =
(100, 2586); increasing domain heterogeneity from left to right panel: À = 01)..1.0)
4.2 MORE REALISTIC DATASETS (PY150, CIVILCOMMENTS, (WILDCAM)
In this subsection, we tackle more realistic domain generalization tasks in the FL con- text. We include three domain heterogeneity regimes on natural language processing dataset Py150 (with 8421 domains), CivilComments (with 16 domains), and image dataset iWild- Cam (with 323 domains). Note Mixup Zhang
Table 3: Test accuracy on IWildCam dataset with held-out-domain validation; total client number K = 243, training domain M 243.
Centralized
X
0=
x=
ERM
0.2727
0.1762
0.1241
0.0707
IRM
0.2629
0.0718
0.0900
N.A.
al.] (2017), FedDG Liu et al.] (2021). and
Fish
0.3103
0.0781
0.0977
N.A.
Mixup
0.2923
0.0264
0.0370
N.A.
FedADG Zhang et al. (021) ) are only suitable for image datasets, thus are excluded in the ex- periments on Py150 and CivilComments.
MMD
0.3134
0.0718
0.0924
N.A.
Coral
0.3270
0.0718
0.0926
N.A.
GroupDRO
0.2531
0.0530
0.0908
N.A.
FedDG
0.2771
0.1445
0.1191
0.0503
In the following experiments on these three datasets, we choose maximum communication round C = 50 for IWildCam, and C = 10 for Py150 and CivilComments, 1 local epoch per communication. For the homogeneous and heterogeneous case, respectively, each client locally holds all the training domains. For the domain separation case, domains to each client. When K < M, client locally holds non-overlapping domains; all of the centralized methods are applicable in this case. When K M, each client locally only holds one training domain; thus, no centralized method is applicable, only four natural federated domain generalization methods are applicable, we compare them with ERM.
FedADG
0.0049
0.0049
0.0049
0.0049
FedSR
0.0056
0.0056
0.0056
0.0056
FedGMA
N.A.
0.0106
0.0957
0.0106
Algorithm to split the
Py150: We evaluate P5150 with 100 clients, 5477 training domains, 261 validation and 2471 test domains. Given that K < M, we compare all the methods except Mixup, FedDG and FedADG in all three different domain heterogeneity regimes. IWildCam: We evaluate IWildCam with 323 clients, 243 training domains, 32 validation and 48 test domains. We compare all the methods in homogeneous and heterogeneous regimes. Given that K = M, we can only compare FedDG, FedADG, FedSR, FedGMA with ERM in the domain separation regime. CivilComments: Civil- Comments is special kind of DG where the test domain is subpopulation of the training domain, and our goal is to perform well on the worst-case domain. CivilComments contains 100 clients and 16 domains. We compare all the methods except Mixup. FedDG and FedADG in homogeneous and heterogeneous regimes. Given that K < M, we can only compare FedSR, FedGMA with ERM in the domain separation regime.
7

---

Under review as a conference paper at ICLR 2023
Table 4: Test accuracy with held-out-domain validation; total client number K 1000.
(a) Py150 Dataset with M
5477
(b) CivilComments Dataset with M = 16
Centralized
A=1
A A-10 0.1
A A=0
Centralized
A
A A= 0.1
A 0
ERM
0.6833
0.6827
0,3874
0.3343
ERM
0,5413
0.3929
0.3390
0.3331
IRM
0.6771
0.6722
0.374
0,3734
IRM
0.6408
0.6153
0.6093
N.A.
Fish
0.6634
0.6731
0.3734
0,2487
Fish
0.6713
0.6435
0.6086
N.A.
MMD
0.6558
0.6731
0.3745
0,3039
MMD
0.6520
0.6121
0.6093
N.A.
Coral
0.6558
0.6723
0.3755
0.3325
Coral
0.5847
0.6121
0.6093
N.A.
GroupDRO
0.5127
0.6824
0.3358
0.3356
GroupDRO
0.6383
0.6028
0,4954
N.A.
FedSR
FedSR
0.6442
0.3603
0.3389
0.3188
FedGMA
N.A.
0.6628
0.3737
0,3567
FedGMA
N.A.
0.3341
0.3157
0.3195
1, homogeneous
= heterogeneous
A 0, domain separation
1.0
0.6
Test
0.4
0.2
100
150
200
100
150
200
50
100
150
200
number client
number client K
number of client
-
BRM
IRM
Fah
Mouu
-
MMD
---
Cora
GroupDRO
-8- FedDG
FedADG
- FedSR
Figure 4: PACS: Held-out DG test accuracy versus number of clients.
We compare the final test accuracy using held-out validation in Table 4a for Py150, Table 4b for CivilComments, and Table 3] for IWildCam. (Using two validation criterion are summarized in Table 8. Table and Table 10 in Appendix Cj. The results show that the performance of ERM dominates the other algorithms on these three datasets. No algorithm achieves its centralized domain generalization ability, where centralized corresponds training on the centralized dataset gathered from all the training clients. We also plot accuracy versus communication figures in Figure 7 for
P5150, Figure for CivilComments and Figure for IWildCam in Sec. C
4.3 ADDITIONAL FL-SPECIFIC CHALLENGES FOR DOMAIN GENERALIZATION
Besides domain heterogeneity, also investigate the challenges brought by FL, including massive number clients number and communication constraints, which are unique to the FL setting.
i) Massive number of clients: In this experiment, explore the performance of different algo- rithms when the number of clients increases on PACS. We fix the communication rounds = 50 and the local number of epoch is (synchronizing the models every epoch). Figure 4 plots the held-out DG test accuracy versus number of clients for different levels of data heterogeneity. The following comments are in order: given communication budget, 1) current domain generalization methods all degrade lot in particular after K 2 50, while the performance ERM and FedDG main- tain relatively unchanged and FedSR are are sensitive to the clients number, and they both fail after K 2 20. 2) Even in the simplest homogeneous setting À where each local client has i.i.d training data, current domain generalization methods IRM, FISH, Mixup, MMD, Coral, GroupDRO work poorly in the existence of large clients number, this means new methods are needed for DG FL context when data are stored among massive number of clients.
the clients number increases given communication budget. FedADG
ii) Communication constraint:
To show the effect of communication rounds on convergence, we plot the test accuracy versus communication rounds in Figure We fix the number of clients K = 100 on PACS and decreases rounds of communication (together with increasing local epochs), that is. C (50, 10, 5) (with E (1, 5. 10)). That is, if the regime restricts the communication budget, then we increase its local computation E to have the same the total computations. Therefore, the influence of communication the performance is fair between algorithms because the total
8

---

Under review as a conference paper at ICLR 2023
data pass is fixed. We observe that 1) the curves are relatively "flat" for most of algorithms, this is predictable because we vary the number of pass of the data E per round for a changing C, and locally the aggregation rules are the same @++1 = Σ R= Th 0,+1, where ni is the training sample size at client k. 2) In particular, ERM, GroupDRO, and FedDG can achieve comparable good performance when communication budget is low (C 10) comparable to when there communication budget is high (C- 50), showing their communication efficiency the FL context.
- 1. homogeneous
A - 0.1, heterogeneous
A - O, domain separation
0.8-
DG Test DeTert.ccuras Accuracy
0.6
0.4
0.2
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
number communication
number communication
number communication
ERM
IRM
Fish
Mixup
MMD
Coral
GroupDRO
FodDG
FedADG
Figure 5: PACS: Held-out DG test accuracy vs. varying communications (resp. varying echoes )
CONCLUSION AND DISCUSSION
This work evaluates multiple algorithms to solve domain generalization task in the federated learning context. We evaluate the influence of client domain heterogeneity, the total number clients, and communication rounds the domain generalization ability. We show that DG in FL context is an unsolved problem, and it brings abundant new challenges. Specifically, the following aspect might be future directions of interest: 1) domain heterogeneity across the clients dramatically impacts the generalization ability on image dataset PACS and IWildCAM as well as NLP datasets Py150 and CivilComments; designing new FL algorithm to recover centralized domain generalization ability remains open; 2) previous works (eg, FedSR, FedADG) evaluate the generalization ability built upon small number of clients K is not enough, massive clients setting needs to be take into consideration; 3) more realistic datasets need to be considered in the domain generalization in FL. context; 4) For the domain separation case, few prior works are applicable to the case where each client only holds one domain new DG algorithms for the FL setting are required for this case. We list the gap table below for summarizing the current DG algorithms performance gap w.r. ERM in the FL context, in particular, positive means it outperforms ERM, negative means it is worse than ERM. It can be that in the on the simple dataset, the best DG migrated from centralized setting is better than ERM. In the domain separation case, centralized DG algorithms can be adapted to and FedDG and FedADG performs comparably good in this setting. However, they fail in harder datasets. Federated DG algorithms that outperforming ERM, supporting NLP dataset, and free of data sharing are still in need.
Table 5: Gap Table: Current Progress in solving DG in FL context
Comparing to FedAvg-ERM
DG migrated from centralized setting FedDG
FedADG
FedSR
FeaGMA
Domain Sepuration
X
Baseline Dataset: PACS
+2.93%
+1.1%%
87.99%
-1.200
-9.32%
Baseline Dataset: FEMNIST
+5.
0..3%%
-0.28%
Realistic Dataset: Py150
+3.91%
+2.24%
Realistic Dataset: CivilComments
+27.03%
-1.43%
-1.36%
Realistic Dataset: IWildCam
-10.44%
-3.17%
-2.01%
-6.51%
-6.26%
Free Data Sharing
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ruizhao Zhu, Ramon Matas, Matthew Mattina, Paul What- mough, and Venkatesh Saligrama. Debiasing model updates for improving personalized federated training. In International Conference on Machine Learning, pp. 21-31. PMLR, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprini arXiv: /900 02293, 2019.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Shawe- Taylor, R. Zemel, P. Bartlett, Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran As- sociates, Inc., 2011. URL htt http::/ss///.roreennggg.g. merr http:///rrr...aoooroso....o////p b57lecea 6a 98249824422 af 2a1. Papere.ppppppppppp pdf
Fei Chen, Mi Luo, Zhenhua Dong. Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint
2018.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of
mnist to handwritten letters. corr abs/1702.05373 (2017). arXiv preprint arXiv: arXiv: 2017.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13611, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pp. 1180-1189, PMLR, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the co--sample-problert Advances in neural information processing systems, 2006.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and opti- mal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33:2304 2315, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2), 2018
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference Machine Learning, pp. 4387- 4398. PMLR, 2020.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:16/0.05492 2016.
10

---

Under review as a conference paper at ICLR 2023
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017. 2017.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429 450, 2020.
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain gen- eralization on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1013-1023, 2021.
Mingsheng Long. Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105, PMLR, 2015.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Dis- covering causal signals in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6979-6987, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiienn learning of deep networks from decentralized data. In Artificial intelli- gence and statisties, pp. 1273-1282. PMLR, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Tuan Nguyen, Ser-Nam Lim, and Philip HS Torr. Fedsr: Simple and effective domain gener- alization method for federated learning. International Conference on Learning Representations, 2022.
Jonas Peters, Peter Bühhlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947-1012, 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv:191..0877 2019.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2018
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning, 2021. URL
Yuge Shi, Jeffrey Seely, Philip HS Torr, Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.099377 2021.
Virginia Smith, Chao-Kat Kai Chiang. Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443-450. Springer, 2016.

---

Under review as a conference paper at ICLR 2023
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau velopes. Advances in Neural Information Processing Systems, 33:21394 399-21944,,,05, 2020.
Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan, Edouard Oyallon, Eugene Belilovsky, and Irina Rish. Gradient masked averaging for federated learning. arXiv preprint 0rXiv:2201.71980 2022.
Honglin Yuan, Warren Morningstar, Lin Ning, and Karan Singhal. What do we mean by generaliza- tion in federated learning? arXiv preprint arXiv:21 14216,, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
Liling Zhang, Xinyu Lei, Yichun Shi, Hongyu Huang, and Chao Chen. Federated learning with domain generalization. arXiv preprint arX4v 2021.
12

---

Under review as a conference paper at ICLR 2023
A REPRODUCIBILITY STATEMENT
Code for reproduce the result is available the anonymous link We include detailed documentation in using our code to reproduce the results throughout the paper. We also provide documentation in adding algorithm's 's DG evaluation in the FL. context.
B EXPERIMENTS SETTING SUPPLEMENTARY
B.1 DATASETS AND MODELS
In this section, we introduce the datasets we used in our experiments, and the split method we used to build heterogeneous datasets in the training and testing phase well as the heterogeneous local training datasets among clients in the FL. Datasets. We contain three datasets as the benchmark: PACS al. (0117). IWildCam Koh et al. (021) and Py Koh Koh et al. (2021). These three datasets cover different levels of difficulty as well as different types of tasks. PACS and IWildCam are both image classification datasets and 150 is natural language processing (NLP) dataset. PACS is an image dataset for domain generalization. It consists of four domains, namely Photo (1,670 images), Art Painting (2,048 images), Cartoon (2,344 images), and Sketch (3,929 images). This task requires learning the classification task on a set of objects by learning on totally different renditions. The Py150 a natural language processing dataset containing 150, 000 python source code dataset from 8, 421 repository. The goal is to predict the next token given the context of previous tokens. This is real- real-world NLP dataset that contains multiple repositories which naturally form multiple domains. The Wi contains wild animals captured by multiple heats or motion- activated static cameras. Due to the variation in camera model, position, color, background, and relative animal frequencies, the samples form multiple domains. It contains 203, 029 images from 323 different camera traps, the images contain 182 different animal species. For 15150 and IWildeam datasets, we follow the same split method as the Wilds Koh et al. (202I). For PACS, domain, and photo as the test domain, and
cartoon and sketch as the
training domains, art- aut-ppinting as the 90% of the data from cartoon and sketch domains be the training domains, and about 5% to be in-domain validation, and other 5% to be the in-domain test set. During the sampling, we keep the class distribution the same among the training dataset, the in-domain validation dataset, and the in-domain test dataset. Models. For image classification datasets PACS and IWildCam, we use ResNet50 model He al. (2016). and Py Py150 natural language processing (NLP) dataset where we use OpenAl Radford
Radford et al. (2019) to train.
B.2 HYPERPARAMETERS AND MODEL SELECTION
Hyperparameters To make fair comparisons, we allocate the same budget during training for each algorithm each dataset. The budget includes the times of allowed hyperparameter search, model architecture, local computation resources, and communication rounds. For each dataset, we fix the model architecture and initialization to be the same. We conduct eight times hyperparameter searching for each algorithm, and choose the set of hyperparameters that achieves the best perfor- mance. For all the datasets and algorithms, we set 100% for clients' participation in the training during each communication.
For PACS, we fix the number of clients nication happens after one epoch of local training. the batch size is 64. We use Adam optimizer and the learning rate is x 10 -3 except for FedSR algorithm, where we choose SGD with Ir= 0.002, momentum= 0.9 and weight decay= 5 x 10-4.
100, 50 communications total where each commu-
For Py150, we set K = 100, and the total number of communication is 3 in the centralized case and 10 in the distributed case. The batch size is 96. We use AdamW optimizer, Ir 8 10-5 and c 1 x 10-8
For IWildcam, the number of clients K 243, 12 communications in the centralized case, and 50 communications in the federated learning case, and the batch size is 16. We use Adam optimizer where the learning rate is 3 10-5
13

---

Under review as a conference paper at ICLR 2023
For FEMNIST, the number of clients K 100, 20 communications in the centralized case, and 40 communications in the federated learning case, and the batch size is 64. We use Adam optimizer where the learning rate is 10
For CivilCommennts, the number of clients K = 100, 5 communications in the centralized case, and 10 communications in the federated learning case, and the batch size is 16. We Adam optimizer where the learning rate is x 10-6.
For each dataset, choose the hyperparameters starting from their default value consistent with the choice in previous benchmarks DomainBed Gulrajani & Lopez-Paz (020)) Wilds Koh et al. (021)) their value proposed in the original paper.
Model Selection In DG, model selection could significantly affect the model performance. During training. we evaluate the aggregated model by using the validation dataset after each communica- tion round. After performing communication rounds, we select the model that achieves the best performance on the validation dataset. This work adopts two model selection methods for the DG task: in-domain and -ut--mminn model selection Gulrajani & Lopez-Paz (2020). In-domain model selection method uses validation dataset which is independent and identically sampled from the training dataset. The held-out-domain model selection method uses validation dataset that only contains examples from set of domains that do not overlap with the training and testing domains.
C OTHER EVALUATION
We put our extra evaluations here for reference. The experiments are summarized in terms of dataset.
C.1 PACS and FEMNIST
Table 6: Test accuracy on PACS with two model selection criterion: in-domain held-out domain validation; total clients number K 2. First column "centralized" corresponding the centralized domain generalization accuracy, i.e., client number K = 1. Increasing domain heterogeneity from left panel right panel: À (1, 0,1,1 0). The number in boldface highlights the highest test accuracy in that column. N.A. refers to not applicable.
nn-Domain Validation
Weld-OuttDDmaal Validation
Centralized
À 1
À 0.1
À 0
Centralized
X ==1
>= 0.1
À
0
ERM
0.8389
0.8982
0.8234
0.8958
0.8593
0.9084
0.9144
0.9377
IRM
0.918
0.9174
0.8814
N.A.
0.9347
0.9467
0.9389
N.A.
Fish
0.8916
0.9359
0.9204
N.A.
0.9449
0.9497
0.9251
N.A.
Mixup
0.9060
0.9371
0,9072
N.A.
0,9012
0.9371
0.9072
N.A.
MMD
0.9072
0.9299
0.8928
N.A.
0.9210
0.9263
0.9335
N.A.
Coral
0.8886
0.9138
0.8988
N.A.
0.9204
0.9341
0.9311
N.A.
GroupDRO
0.8952
0.9018
0.8964
N.A.
0,9150
0.9491
0.9269
N.A.
FedDG
0.9024
0.8701
0.8683
0.6856
0,8922
0.8772
0.9036
0,9383
FedADG
0.9024
0.7958
0.9569
0.9473
0.8922
0.9126
0.9329
0.9341
FedSR
0,8246
0,8725
0.9060
0,8928
0,8617
0.8898
0,9060
0.8952
FedGMA 0.8485
0.8850
0.7922
0.8114
0.8602
0.8850
0.9425
0.9251
We report the final test accuracy using two validation criterion in Table It shows that the homo- geneity case (A = 1 column) may even slightly outperform the counterpart centralized domain generalization accuracy in the simple case (small client and domain numbers). This could come from the natural regularization brought by the FL. In the domain separation case (A = 0 column), although FedADG is not communication efficient as shown in Fig 2, it seems to be more robust across validation strategies whereas FedDG performs poorly using in-domain validation.
Overall, expected, existing DG algorithms with enough communication rounds able to per- form reasonably well this simple setting where the dataset simple and the number clients and domains are small.
14

---

Under review as a conference paper at ICLR 2023
Table Test accuracy on FEMNIST dataset with two model selection criterion: in-domain held-
out-domain validation; total client number K = 100; total training domain number M = 3500.
In-Domain Validation
Held-Out-Domain Validation À =
Centralized
= 1
À 0.1
À = 0
Centralized
À = 0.1
A= 0
ERM
0.8544
0.8368
0.8398
0.8399
0.8544
0.8368
0.8398
0.8399
IRM
0.8442
0.8303
0,8389
0,8328
0.8442
0,8303
0,8389
0.8328
Fish
0.8487
0.8331
0.8290
0.8264
0.8487
0.8331
0.8290
0.8264.
Mixup
0.8337
0.8275
0.8168
0.8161
0.8337
0.8275
0.8168
0,8161
MMD
0,8442
0.8430
0,8323
0,8293
0.8442
0.8430
0,8323
0.8293
Coral
0.8457
0.8403
0.8328
0.8513
0.8457
0.8403
0.8328
0,8513
GroupDRO
0.8413
0.8447
0.8139
0.8052
0.8413
0.8447
0.8139
0.8052
FedDG
FedADG
FedSR
0,8470
0.8320
0.8316
0,8306
0.8470
0.8300
0.8316
0.8306
FedGMA
N.A.
0.8486
0,8395
0.8371
N.A.
0.8486
0.8395
0.8371
C.2 MORE REALISTIC DATASETS (PY150, CIVILCOMMENTS, (WILDCAM)
To observe the convergence of each algorithm, we plot Figure for Py150 (resp. Figure for Civil- Comments and Figure for IWildCam). It shows that with realistic dataset as well as with non-trivial number of clients, all of the algorithms tend be more sensitive domain heterogeneity. Even in the heterogeneous case, where each client locally holds all the training domains, their generalization abilities on the unseen domains are much worse than its centralized counterpart; let alone to solve the even harder domain separation case.
We also reported accuracy using the in-domain and held-out domain validation for Py150 in Table CivilCommnets in Table 9 and [WildCAM in Table 10
Table 8: Test accuracy 15150 dataset with two model selection criterion: domain validation; total client number K = 100.
in-domain / held-out-
In- Domain Validation
Held-Out Dumann Validation
Centralized
==
À = 0.1
== 0
Centralized
==1
X x= 0.1
==
ERM
0.6743
0.6827
0.3874
0.3343
0.6833
0.6827
0.3874
0.3343
IRM
0.6771
0.6722
0.374
0.3734
0.6771
0.6722
0.374
0.3734
Fish
0.6634
0.6726
0.3734
0.2487
0.6634
0.6731
0.3734
0.2487
MMD
0.6495
0.6731
0.3745
0.3039
0.6558
0.6731
0.3745
0.3039
Coral
0.6495
0.6723
0.3755
0.3103
0.6558
0.6723
0.3755
0.3325
GroupDRO
0.5127
0.6824
0.3358
0.3356
0.5127
0.6824
0.3358
0.3356
FedSR
FedGMA
N.A.
0.6624
0.3737
0.3567
N.A.
0.6628
0.3737
0.3567
Table 9: Test accuracy CivilComments dataset with two model selection criterion: in-domain / held-out-domain validation; total client number K
100.
In-Domain Validation
Held-Out-Domall Validation
Centralized
X = 1
À 0.1
X = 0
Centralized
X x=
X = 0.1
A = 0
ERM
0.5413
0.3929
0.3390
0.3331
0.5413
0.3929
0.3390
0.3331
IRM
0.6213
0.6153
0.6093
N.A.
0.6408
0.6153
0.6093
N.A.
Fish
0.6038
0.6435
0.6086
N.A.
0.6713
0.6435
0,6086
N.A.
MMD
0.5981
0.6121
0.6093
N.A.
0.6520
0.6121
0.6093
N.A.
Coral
0,5827
0.6121
0.6093
N.A.
0.5847
0.6121
0.6093
N.A.
GroupDRO
0.6383
0.6028
0.4804
N.A.
0,6383
0.6028
0.4954
N.A.
FedSK
0.6442
0.3603
0.3389
0,3188
0.6442
0,3603
0.3389
0,3188
FedGMA
N.A.
0.3341
0.3157
0.3195
N.A.
0.3341
0.3157
0.3195
15

---

Under review as a conference paper at ICLR 2023
Table 10: Test accuracy on IWildCam dataset with two model selection criterion: in-domain held-
out-domain validation; total client number K = 243.
In- Domain Validation
Held-Out-Domain Validation
Centralized
A = 1
À 0.1
== 0
Centralized
==
== 0.1
A=
ERM
0.2727
0.1762
0.1241
0.0707
0.2727
0.1762
0.1241
0.0707
IRM
0.2845
0,0710
0.0903
N.A.
0.2629
0.0718
0.0903
N.A.
Fish
0.3103
0.0781
0.0977
N.A.
0.3103
0.0781
0.0977
N.A.
Mixup
0.2620
0.0264
0.0371
N.A.
0.2923
0.0264
0.0370
N.A.
MMD
0.3225
0.0711
0.0924
N.A.
0.3134
0.0718
0.0924
N.A.
Coral
0.2857
0.0718
0.0926
N.A.
0.3270
0.0718
0.0926
N.A.
GroupDRO
0.2531
0.0524
0.0908
N.A.
0,2531
0.0530
0.0908
N.A.
FedDG
0.2866
0.1445
0.1023
0.0503
0.2771
0.1445
0.1191
0.0503
FedADG
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
FedSR
0.0056
0.0056
0.0056
0.0056
0.0056
0.0056
0.0056
0.0056
FedGMA
N.A.
0.0081
N.A.
0.0081
- homogeneous
A 0.1, heterogeneaus
À O, domain separation
DG Test Accuraty
0:1-
0.2
0.0
10
10
Communications
Communkations
Commurication:
ERM Fish
Coral
GroupDRO
FedADG
---
FedGMA
IRM
MMD
Mixup
FedDG
Fod5R
Figure 6: Accuracy versus communication rounds for CivilComments: total clients and training domains = 100 increasing domain heterogeneity from left to right: À = 01,0.1,0).
A 1, homogeneous
A 0.1, heterogeneaus
À O, domain separation
0.7
##
DG Test Accuracy
0.6
0.5
2
4
10
10
Communications
Communkctiions
ERM
Fish
MMD
Coral
GroupDRO
FedGMA
Figure 7: Accuracy versus communication rounds for Py150 Total clients number K 100;
increasing domain heterogeneity from left to right panel: (1,0.1,0).
16

---

Under review as a conference paper at ICLR 2023
A homogeneous
= 0.1, heterogencous
A O, domain soparation
DG T Acarnty
0.10
0.05
10
10
20 Communications
Communications
Communications
-0- ERM
Fish -
MMD
Coral
Moxu
GroupDRO
-B-
FedADG
-
Fee5R
Figure 8: Accuracy versus communication rounds for IWildCam; Total clients number K =243 increasing heterogeneity from left to right panel: À 01)
C.3 ADDITIONAL FL-SPECIFIC CHALLENGES FOR DOMAIN GENERALIZATION
In this subsection, we reported the additional results related to Sec 4.3. Including Table II Table 12 Table Table 14 for test accuracy using the held-out and in-domain validation for PACS with K 20, 50, 100, 200, respectively.
Table 11:
Test accuracy PACS dataset with two model selection criterion: in-domain / held-out- domain validation; total client number K 20; total training domain number M 2.
In- Domain Validation
Seld-Out-Domaan Validation À
Centralized
X
= 0.1
À =
Centralized
X 0.1
À
ERM
0.8389
0.9293
0.9455
0.9569
0.8389
0.9305
0.9539
0.9485
IRM
0.9180
0.9461
0.9275
N.A.
0,9180
0.9479
0.9431
N.A.
Fish
0,9030
0.9449
0.9443
N.A.
0.9030
0.9515
0.9467
N.A.
Mixup
0.8635
0.9581
0.9431
N.A.
0.8635
0.9581
0.9461
N.A.
MMD
0.9186
0.9461
0,9389
N.A.
0,9186
0,9491
0.9431
N.A.
Coral
0.9216
0.9449
0.9461
N.A.
0.9216
0.9491
0.9455
N.A.
GroupDRO
0,9060
0,9401
0.9521
N.A.
0.9060
0.9461
0.9539
N.A.
FedDG
0.9024
0.9275
0.9275
0.8952
0,8922
0.9365
0.9425
0.9521
FedADG
0,9024
0.1347
0.1269
0.1257
0.8922
0.1455
0.1269
0.1257
FedSR
0.8470
0.1850
0.1886
0.1497
0.8470
0.1784
0.2090
0.1222
FedGMA
Table 12: Test accuracy PACS dataset with model selection criterion: in-domain / held-out- domain validation; total client number K = 50; total training domain number M =
I- Domain Validation
Held-Out- Sudd-Out-Dmmin Validation )= 1
Centralized
A 1
X = 0.1
== 0
Centralized
X x= 0.1
== 0
ERM
0.8389
0.9371
0.9437
0.9533
0.8389
0.9479
0,9539
0.9509
IRM
0.9180
0.9371
0.9401
N.A.
0.9180
0.9455
0,9389
N.A.
Fish
0.9030
0.9407
0.9030
N.A.
0.9030
0.9401
0,9162
N.A.
Mixup
0.8635
0.9503
0.9407
N.A.
0.8635
0.9335
0.9419
N.A.
MMD
0.9186
0,9389
0.9329
N.A.
0.9186
0.9413
0,9389
N.A.
Coral
0.9216
0,9389
0.9228
N.A.
0.9216
0.9419
0.9413
N.A.
GroupDRO
0.9060
0.9443
0,9515
N.A.
0,9060
0.9419
0.9515
N.A.
FedDG
0.9024
0.9329
0.9287
0.9515
0.8922
0.9377
0.9371
0.9515
FedADG
0,9024
0.1862
0.1862
0.1862
0.8922
0.1814
0.1820
0.1832
FedSR
0.8246
0. 0.1269
0.1401
0.1509
0.8754
0.1305
0.1521
0.1275
FedGMA
17

---

Under review as a conference paper at ICLR 2023
Table 13: Test accuracy on PACS dataset with two model selection criterion: in-domain held-out- domain validation; total client number K 100; total training domain number M = 2.
In- Domain Validation
Sald-OutDoomald Validation
Centralized
A =
0.1
À
Centralized
==
A= 0.1
À
ERM
0.8389
0.8766
0.9144
0.8958
0.8389
0.8766
0.9144
0.9377
IRM
0.9180
0.8156
0.5449
N.A.
0.9180
0.8156
0.5449
N.A.
Fish
0.9030
0.8497
0.6311
N.A.
0,9030
0.8497
0.6311
N.A.
Mixup
0,8635
0.7653
0.5551
N.A.
0,8635
0.7653
0,5551
N.A.
MMD
0.9186
0.8150
0.6341
N.A.
0.9186
0.8150
0.6341
N.A.
Coral
0.9216
0.8150
0.5515
N.A.
0.9216
0.8150
0,5515
N.A.
GroupDRO
0.9060
0.9395
0.9437
N.A.
0,9060
0.9395
0.9437
N.A.
FedDG
0,9024
0.8868
0.8880
0.8952
0.8922
0.9234
0.9275
0.952I
FedADG
0.9024
0.0915
0.0732
0.0592
0,8922
0.1892
0.0592
0.0598
FedSR
0.8246
0.118
0.1341
0.1138
0.8754
0.1246
0.1263
0.1257
FedGMA
Table 14: Test accuracy PACS dataset with two model selection criterion: in-domain / held-out- domain validation; total client number K 200: total training domain number =
In Domain Valdation
Huld-Out- Domain Validation
Centralized
0.1
À 0
Centralized
À
== 0.1
== 0
ERM
0.8389
0.8635
0.8287
0.8539
0,8389
0.8287
0,9210
0.8659
IRM
0.9180
0.6790
0,5000
N.A.
0,9180
0.7802
0.4934
N.A.
Fish
0.9030
0.7802
0.4850
N.A.
0.9030
0.7802
0.4892
N.A.
Mixup
0.8635
0.6701
0.5551
N.A.
0.8635
0.6671
0.5551
N.A.
MMD
0.9186
0.8150
0.6341
N.A.
0,9186
0,8150
0.6341
N.A.
Coral
0.9216
0.8150
0.5018
N.A.
0.9216
0.8150
0.4916
N.A.
GroupDRO
0.9060
0.8144
0.6090
N.A.
0.9060
0.8162
0.6844
N.A.
FedDG
0.9024
0.7916
0.8898
0.8868
0.9204
0.9204
0.8898
0.8934
FedADG
0.9024
0.0915
0.0732
0.0592
0.8922
0.1892
0.0592
0.0598
FedSR
FedGMA
Table Test accuracy on PACS dataset with two model selection criterion: in-domain held-out- domain validation; total client number K = 200; total training domain number M 2: Communica- tion Rounds C 5.
In nn-oomiin Validation
Out-Domain Validation
Centralized
A =
0.1
A == 0
Centralized
A=
X == 0.1
== 0
ERM
0.8389
0.9006
0.9132
0.9257
0.8389
0.9006
0.9132
0.9257
IRM
0.9180
0.8599
0.7431
N.A.
0,9180
0.8599
0.7431
N.A.
Fish
0.9030
0.8275
0.6479
N.A.
0.9030
0.8275
0.6479
N.A.
Mixup
0.8635
0.8222
0.5760
N.A.
0.8635
0.8222
0,5760
N.A.
MMD
0.9186
0.8605
0.7413
N.A.
0,9186
0.8605
0.7413
N.A.
Coral
0.9216
0.8593
0.7413
N.A.
0.9216
0.8593
0.7413
N.A.
GroupDRO
0,9060
0.9048
0.8916
N.A.
0,9060
0.9048
0.8916
N.A.
FedDG
0.9024
0.8940
0.9108
0.8952
0,8922
0.8940
0,9108
0.9521
FedADG
0.9024
0.0915
0.0732
0.0592
0.8922
0.1892
0.0592
0.0598
FedSR
FedGMA
18

---

Under review as a conference paper at ICLR 2023
Table 16: Test accuracy on PACS dataset with two model selection criterion: in-domain held-out- domain validation: total client number K = 200; total training domain number M 2: Communica- tion Rounds C 10.
In-Domain Validation
Held-Out-Domain Validation
Centralized
0.1
À =
Centralized
= 0.1
À
ERM
0.8389
0.9186
0.8587
0.8958
0.8389
0.9078
0.8970
0.9377
IRM
0.9180
0.8347
0.7707
N.A.
0,9180
0.8473
0.7707
N.A.
Fish
0,9030
0,8473
0.6737
N.A.
0,9030
0.8473
0.6737
N.A.
Mixup
0.8635
0.8359
0.5796
N.A.
0.8635
0.8359
0.5796
N.A.
MMD
0.9186
0.8341
0.7707
N.A.
0,9186
0.8473
0.7707
N.A.
Coral
0.9216
0,8473
0.7707
N.A.
0.9216
0.8473
0.7707
N.A.
GroupDRO
0.9060
0,9222
0.9186
N.A.
0.9060
0.9222
0.9186
N.A.
FedDG
0.9024
0,9084
0.8880
0.8952
0.8922
0.9084
0.9275
0.9521
FedADG
0.9024
0.0915
0.0732
0.0592
0.8922
0.1892
0.0592
0.0598
FedSR
FedGMA
C.4 TRAINING TIME, COMMUNICATION ROUNDS AND LOCAL COMPUTATION
In this section, provide training time per communication in terms of the wall clock training time. Notice that for fixed dataset, most of algorithms have similar training time comparing to ERM, where FedDG and FedADG are significantly more expensive.
Table 17: Wall-clock Training time per communication (unit: s).
Wall Clock
PACS
FEMNIST
Py150
CivilComments
IWildCam
ERM
143
262
6566
3958
6301
IRM
147
297
7089
4085
6454
Fish
148
324
7770
5483
7072
Mixup
144
264
N.A.
N.A.
6294
MMD
144
287
7603
4024
6663
Coral
144
287
7212
3901
6597
GroupDRO
145
307
8121
4690
9311
FedDG
352
N.A.
N.A.
32172
FedADG
181
N.A.
N.A.
11094
FedSR
151
280
4403
7136
FedGMA
143
261
6545
4525
6795
19


Reviews:
[{"review_id": "xlJ95tWZOg", "reviewer": "Reviewer_fDMj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a benchmark for domain generalization (DG) algorithms in federated learning (FL). The authors considered more realistic settings with larger client numbers and different heterogeneity levels compared to prior works. The benchmarks included two new datasets for DG in FL and multiple existing DG algorithms. \n", "summary_of_the_review": "I am leaning towards weak reject as I think the benchmark needs to be more comprehensive as discussed in the weaknesses above.\n", "strength_and_weaknesses": "Strengths\n\nThe first paper to benchmark DG in FL settings and considered different setups and difficulties in the benchmark. The list of algorithms included in the benchmark is also comprehensive.\nThe paper is written clearly and easy to follow.\n\nWeaknesses\n\nOnly image datasets were considered in the benchmarks while there are many existing works for DG in the text domain [1,2,3]. Also, only two datasets were considered more realistic which seems to be small as a benchmark. \nSome parts of the results are not clear, e.g. why does $\\lambda=0.1$ give better performance than $\\lambda=1$ for some algorithms in Table 4?\n\nReferences:\n\n[1] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL 2020.\n\n[2] Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., & Zettlemoyer, L. Demix layers: Disentangling domains for modular language modeling. arXiv 2021.\n\n[3] Chronopoulou, A., Peters, M. E., & Dodge, J.  Efficient hierarchical domain adaptation for pretrained language models. arXiv 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T-4Z8a1Lr6f", "reviewer": "Reviewer_UVoT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a benchmark for Federated Learning in the context of domain generalization. The authors thoroughly evaluate a number of popular DG algorithms on this benchmark. ", "summary_of_the_review": "The paper presents a useful benchmark and highlights the challenges of existing DG methods extended to FL in the context of this benchmark. The evaluation of some existing algorithms is of interest.  Some related work are not fully discussed. The reviewer also currently has some concerns about what data information is transmitted across clients and servers in the methods evaluated.\n\nUpdate: The authors made  substantial changes to the paper text that have greatly clarified the setting under study, thus have largely addressed my concerns. I am thus increasing my score ", "strength_and_weaknesses": "Strengths\n- The benchmark considers a relevant setting\n- The evaluations are detailed covering a variety of methods\n- The work highlights the difficulties in this setting\n\n*Weakness*\n- The intersection of DG and FL has been considered by several authors recently.. The  following related references and methods are not discussed or compared:\n   - Tension et al “Gradient Masked Averaging for Federated Learning” \n   - Yuan et al “What do we mean by generalization in Federated Learning” \n- The wide array of approaches are appreciated but there are limited details on how these are adapted to the federated setting. Some of these algorithms e.g. MMD, FISH,  seem like they would require access to the different domains at each iteration thus it’s not clear to the reviewer how they are implemented here since clients should not share data. \n- Although the performance is evaluated it is not discussed if these approaches come with additional communication or training time overhead. FL is often concerned with these constraints. Related to the previous point are there additional information besides the models transmitted across clients in the various methods?\n- The datasets are interesting however in addition to these it might be good to include at least one popular dataset from the FL literature such as FEMNIST (which has been studied in OOD settings in other work) or CIFAR \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MfcNDQ7ggx", "reviewer": "Reviewer_qSGW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new benchmark for federated domain generalization (FedDG) which explores more realistic challenges under the federated learning scenario. The benchmark designs a fair and diverse comparison on the data heterogeneity and communication costs. The experimental results show the challenges on the FedDG stay unsolved in the realistic experiment setting.", "summary_of_the_review": "The advantages of this work are that the overall benchmark design is innovative and much needed by the academic community, but the disadvantage is that it does not use the recent SOTA method as a comparison, which is insufficient to support the seriousness and necessity of the DG problem under the FL scenario. The lack of comparison of methods from non-iid heterogeneous perspective in the FL field makes the conclusions of this paper less convincing. And the design of the experiment needs to be presented in a clearer and more structured way.", "strength_and_weaknesses": "Strengths:\n1. The DG problem under FL is important and lack of discussion, the proposed benchmark can better promote the research of new algorithms.\n2. The experimental settings for statistical heterogeneity challenge under FL is novel and practical.\n3. Sufficient experimental results and reproducible code.\n\nWeaknesses:\n1. The compared domain generalization methods are old and not representative, which leads to an unconvincing conclusion of this paper.\n2. There are lots of FL methods which also focus on the statistical heterogeneity problem and need to be included in the benchmark.\n3. The description of proposed experimental setting is not intuitive, making it hard to understand and does not give a clear framework for comparison.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4mTg1bvgwt_", "reviewer": "Reviewer_oUmB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is focused on benchmarking algorithms for domain generalization (DG) in the federated learning (FL). They evaluate 9 algorithms on three datasets (PACS, iWildCam and Py150). In particular, authors compare centralized DG algorithms and two FL algorithms for DG (FedADG and FedDG). The paper studies the ability of these methods to handle different challenges of DG-FL such as statistical heterogeneity among clients, number of clients, number of communication rounds. The results show more work is needed to design algorithms in the large client setting even for simple datasets as well as to handle local data statistical heterogeneity.", "summary_of_the_review": "Overall, the analysis presented in the paper is of high importance to the community and has the potential to bring new advances in the DG-FL by outlying unsolved challenges of the current methods. However, more work and effort is needed to make this paper strong and comprehensive which is necessary for these types of benchmarking papers.", "strength_and_weaknesses": "Strengths:\n-  Domain generalization (DG) in federated learning (FL) is an important problem and has not been a systematic comparison of the methods. This paper presents the first such study.\n- The paper presents major challenges that need to be overcome to address DG-FL problems and shows the inability of the existing methods to do so. This is important work needed to advance the field.\n- Using homogeneous and heterogeneous splits (as measured by lambda) presents a systematic way to measure how well the algorithm can adapt to heterogeneous  environments.\n\nWeaknesses:\n- For benchmark papers, it is extremely important that the paper is well written, well structured and reads well. However, this is not currently the case. There is a lot of information, but the paper does not organize information in a succinct and clear way making it hard to follow. For example, problem setup is never fully defined; part of experiments with varying lambda are in one paragraph, the other in another; it is not clear how centralized DG are extended to FL setting; many tables are not referred to in the paper; it is not explained why centralized DG methods can not be extended to be applicable to lambda=0 setting and so on. These are just a few examples, but the overall impression of the paper is that much more effort is needed to have a well organized and structured paper. \n- The methods are tested only on three benchmark dataset. More analysis is needed to make the comparison truly comprehensive\n- Similarly, more recent FL algorithms should be included for the comprehensive analysis.\n\nMinor: \n- In the paragraph Challenges from more realistic datasets, it should be Fig. 2 (c),\n(d) not (b), (c)\n- In paragraph 4.1 the authors write that they analyze two values of the lambda parameters, but the figure shows three.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject