Client-agnostic Learning and Zero-shot Adaptation for Federated Domain Generalization
Authors: Seunghan Yang
Seokeon Choi
Hyunsin Park
Sungha Choi
Simyung Chang
Sungrack Yun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated learning
Domain generalization
Zero-shot adaptation

Abstract: Federated domain generalization (federated DG) aims to learn a client-agnostic global model from various distributed source domains and generalize the model to new clients in completely unseen domains. The main challenges of federated DG are the difficulty of building the global model with local client models from different domains while keeping data private and low generalizability to test clients, where data distribution deviates from those of training clients. To solve these challenges, we present two strategies: (1) client-agnostic learning with mixed instance-global statistics and (2) zero-shot adaptation with estimated statistics. In client-agnostic learning, we first augment local features by using data distribution of other clients via global statistics in the global model's batch normalization layers. This approach allows the generation of diverse domains by mixing local and global feature statistics while keeping data private. Local models then learn client-invariant representations by applying our client-agnostic objectives with the augmented data. Next, we propose a zero-shot adapter to help the learned global model to directly bridge a large domain gap between seen and unseen clients. At inference time, the adapter mixes instance statistics of a test input with global statistics that are vulnerable to distribution shift. With the aid of the adapter, the global model improves generalizability further by reflecting test distribution. We comprehensively evaluate our methods on several benchmarks in federated DG.

Under review as a conference paper at ICLR 2023
CLIENT-AGNOSTIC LEARNING AND ZERO-SHOT ADAP- TATION FOR FEDERATED DOMAIN GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Federated domain generalization (federated DG) aims to learn a client-agnosti global model from various distributed source domains and generalize the model to new clients in completely unseen domains. The main challenges of federated DG are the difficulty of building the global model with local client models from dif- ferent domains while keeping data private and low generalizability to clients, where data distribution deviates from those of training clients. To solve these challenges, we present two strategies: (I) client-agnostic learning with mixed instance-globud statistics and (2) zero-shot adaptation with estimated statistics. In client-agnostic learning, we first augment local features by using data distribu- tion of other clients via global statistics in the global model's batch normalization layers. This approach allows the generation of diverse domains by mixing local and global feature statistics while keeping data private. Local models then learn client-invariant representations by applying our client-agnostic objectives with the augmented data. Next, propose zero-shot adapter to help the learned global model to directly bridge a large domain gap between seen and unseen clients. At inference time, the adapter mixes instance statistics of test input with global statistics that are vulnerable to distribution shift. With the aid of the adapter, the global model improves generalizability further by reflecting test distribution. We comprehensively evaluate our methods on several benchmarks in federated DG.
INTRODUCTION
A huge amount of data is being collected every second from wide range of 1oT devices, and the data have been utilized for building robust deep learning models. Federated learning (FL) has emerged as a promising paradigm to train the model indirectly accessing the distributed data such that it reduces privacy leakage. Pioneering studies such as FedAvg (McMahan et al.) (017)) and FedProx (Li et al. 2020) train each local model on its own data while keeping data private and transmit model parameters to the server for obtaining generalized global model. The parameters from local clients are aggregated in the server, and the server parameters are broadcasted to clients. This process is iteratively performed until the global model converges to stationary point, and user privacy is ensured by sharing aggregated parameters not data itself with other clients.
In real-world scenarios, local data are collected from various domains across clients coming from different characteristics of sensors and surrounding environments. For example, in autonomous driving tasks, each vehicle captures street views and infrastructures differently from others due to variances in camera sensors, region, and other factors. These local data deviates in terms of the distribution in feature space, inducing non-iid data across clients, denoted domain shift (Li al. 20216 Jiang et al., 2021). Currently, most studies have tried to solve the issues of FL on non-iid iid data, especially heterogeneous label distribution (Li et al.) 2020 Karimireddy et al. 2020 Wang et al. 2020). but domain shift has not been fully explored in the literature yet. Domain shift also exists between training and test clients. After federated learning, the learned FL model is deployed to new customers outside the federation, e.g., new vehicles or medical centers, where data distri- bution is shifted from those of clients inside the federation. However, most works only focus on improving model performance of the clients participated in FL., while neglecting generalization on unseen clients. In this paper, we treat federated domain generalization (federated DG), which aims to collaboratively learn a alient-agnostic federated model from various distributed source domains and generalize the learned model to new clients unseen domains, as illustrated Fig.

---

Under review as a conference paper at ICLR 2023
Previous FL works (Li et al. 2021b Andreux at 2020) also tried to solve and analyze the do- main shift problem in federated learning, but they only focused on training personalized models in- side the federation not building the generalized global model for new clients. To improve the gen- eralizabilityy of the FL model, federated DG has
Server
Deployyng inllayyin to new client
been explored by (Jiang et al. [002]] Wu & Gong 2021 Liu al. 2021] Yuan al.] 2021). How- ever, there are some limitations such as threat
Client
Client
Client K
New client
of privacy leakage (Wu Gong 2021 Liu et al.
2021) by sharing private information with other clients, performance limitation (Yaan et al. 2021) focusing aggregation rather than local train- ing, and need of multiple target data for test-time adaptation (Jiang al. 2021). In the literature of domain generalization of centralized learning, multi-source domain generalization (Gulrajani &
Training
Training
domais skift
Federated domain generalization: a client has data belonging to single domain that is different from other clients' domains, and the learned global model is deployed to new client. Lopez-Pai Paz 2021) has been widely used to build the generalized model using multiple source do- mains. In federated learning, sharing data with other clients is strictly restricted preventing from serious privacy leakage, so these generalization methods are not be applicable to federated DG.
Figure 1:
This paper presents two approaches: (1) client-agnostic learning with mixed instance-glose global statis- ties for local training and (2) zero-shot adaptation with estimated statistics for inference. Our pro- posed methods, named FedIG-A, allow local models learn client-invarian representations from other clients' data distribution while preserving privacy and let the learned global model directly generalize to unseen domains. this end, we adopt FedBN (Li et al.) 2021b) that is designed to mit- igate domain shift across clients. In FedBN, local clients use local batch normalization (BN) layers and keep them local to learn -pent-sp representations while the remaining parts are aggregated in the server learn client-invariant representations. However, is difficult explicitly train local models to learn client-invariant representations only using single domain local data. To solve the issue, propose novel client-agnostic learning with mixed statistics. In client-agnostic learning, we augment local features using data distribution of other clients via aggregated BN statistics from the global model, i.e., global statisties. Our proposed augmentation randomly mixes instance-level and global feature statistics to produce diverse domain features. We then apply client-agnostic loss functions learn
representations. Note that our method exploits global statistics that do not pose additional privacy leakage, the same amount with FedAvg. At inference time, we intro- duce a zero-shot adapter for helping the learned global model to directly bridge large domain gap between seen and unseen clients. We mix instance statistics of a test input with global statistics that are vulnerable to distribution shift. The optimal interpolation values are different across test samples instance-
in each BN layer, thus we design the adapter for estimating the interpolation value wise manner. With the aid of the adapter, the global model improves generalizability further by reflecting test distribution. We conduct extensive experiments domain including PACS (Li al. (017)), VLCS (Fang et al. 2013), and OfficeHome /Venkateswara et al. 2017) in the federated setting and show the effectiveness of our components.
several DG benchmarks in image
RELATED WORKS
2.1 FEDERATED LEARNING
Federated learning (FL) has been extensively studied to train a global model using distributed datasets, while ensuring user privacy and reducing communication overhead issues. Most recent FL. approaches focus on solving the issues of non-iid data distribution over clients, especially heteroge-
neous label distribution (Li et al. 2020 Karimireddy et al., 2020 Wang et al. 2020). Especially, FedProx (Li et al. 2020) incorporates proximal term into local loss functions to regularize the local model reducing ing the gap with the global model. Only a few works (Li et al. 2021b Andreux et al. 2020 Jiang et al. 22221) point out domain shift across different clients in FL. FedBN (Li et al.) 2021b) and SiloBN (Andreux et al. 2020) keep BN statistics locally without aggregating them in the server for mitigating domain shift, but both methods only focus on boosting performance of clients
2

---

Under review as a conference paper at ICLR 2023
inside the federation. TsmoBN (Jiag al. 2021) tackles the generalization ability of previous works. They present updating test batch normalization to adapt the global model to target clients, but it requires lots of data in target domain. Moreover, three methods do not deal with building the client-agnostic global model, ie.. training local clients without domain generalization algorithms. We go beyond these approaches building the client-agnoste global model while mitigating domain shift and directly generalizing the model to new target clients indicating zero-shot adaptation.
2.2 DOMAIN GENERALIZATION
Domain generalization aims to train a model from multi-source or single-source domain data such that it can generalize to unseen target domains. Multi-source domain generalization (Li et al. 2019;
Xu al. 2021a Seo et al. 2020 Zhao et al. 202T Pandey et al. 2021 Nam & Kim. 2018 Chen et al. 2022 2021| Lv et al. (022) has been extensively explored to learn domain-nvvan invariant repre-
sentations by minimizing domain discrepancy over multiple source domains. These methods can be applied to federated DG to help building the client-agnostt model. However, client data should be shared across clients for multi-source domain generalization. and it leads to serious privacy issues
in federated learning. Single-source domain generalization (Zhou et al. 2021 Li et al. 2021a| Xu
et al. 2021b Wang et al.] 2021 Carlucci et al. 2019 Huang et al. 20200] Kim et al. 2021) has tried to learn a generalized model with single source data. These algorithms can be applied to federated
DG without privacy leakage. They can improve the generalization ability through domain expan- sion or regularization, but the performance improvement is limited since they are designed to use individual source domain data. It cannot fully exploit the advantage of federated learning. Recently, federated DG has been studied to treat the distributed multi-source domain setting. COPA (Wu & Gong 202 D and FedDG (Liu et al. 202T) apply multi-source domain generalization methods (Li jt al., 2019 Xu et al., 2021a) to the distributed setting without sharing raw data across clients, but they share classifiers and style distribution with other clients, respectively, which contain private information. Another work, CSAC (Yuan et al. (02T)) proposes an aggregation method solving do- main shift, but it requires pre-training stage and achieves marginal performance improvement due to naive local training with single domain data. Compared to these works, we propose utilizing data distribution of other clients without privacy leakage for learning client-invarian invariantattttta representations.
3 CLIENT-AGNOSTIC LEARNING AND ZERO-SHOT ADAPTATION
3.1 PRELIMINARIES
Notation and Problem Formulation: Let X' and y denote the input space and the label space, respectively. k-th client has single domain data D2 = {[ of distributed source domain data of K clients inside the federation. In federated domain general- ization (federated DG), there exists domain shift across clients, where each client data D3 sampled from Domain-sspecif distribution (2) different with other clients. D, indicates the target test domain data from a new client outside the federation, which distribution (At, 3)) is shifted from that of training data. F& is the feature extractor parameterized by 0, and C. is the classifier parameterized by c. Federated DG aims to learn a generalized global model Coo o Fo X - y by aggregating K distributed clients' models {Fe Co. JK trained on source data {Dx} 1K such that the global model generalizes to unseen domain Dt-
31" 3= and (Di...., } is the set
Domain shift over clients hinders obtaining the generalized global model since the local models are easily over-fitted their domains, indicating large model divergence across clients (Li et al. (0021b). Even though data from various domains and a large amount of data are used through federated learning, domain shifts in the distributed setting negatively affect the generalization ability both inside and outside the federation. Local models should learn domain-invariant representations such that the generalized global model is collaboratively obtained from the local models.
Challenges:
FedBN: We adopt FedBN (Li et al. 2021b) for dealing with domain shift across clients. FedBN e''pph + 8/. where n4, and of are BN statisties of 1-th BN layer in k-th client, which are caleulated as rumning means and standard deviation. % and 8% are learnable affine parameters, and a' indicates an input tensor. BN statistics remain local client-specific part while the remaining parts uploaded and broadcasted by the
keeps all BN layers local, and these local BN layers operate as %% o
3

---

Under review as a conference paper at ICLR 2023
oip Me
pla uu
of
Conv
Local BN
ReLU
fu
LeF
Eq. (2)
Rix
pl
Leact
Eq. (5)
d12
V2
fi
Intermediate feature point
LeAFt Eq, (4)
Posssibl ble fenture points
MixIG standardization
wh
-Stndandized
statistics
Forward to obbaii local feature
Standardized with Global statisties
U--Ufform(..1
Forward obtain augmented festure
Standardized with MixtG
Figure 2: The local feature fi, k is extracted by batch statistics (Local BN), and the augmented feature fi, is extracted by mixed instance and global statistics (MixIG). MixIG is constructed by randomly interpolating instance and global statistics, and an intermediate feature á is standardized by MixIG (right) to possibly various features. This operation repeated on every BN layer. With both local and augmented features, local models learn client-invariant representations by Eq. and 5
server, denoted as a client-agnostic part. We can expect that local BN statistics learn client-specific characteristics and the client-agnostic part learns client-invariant representations. In this paper, use = (0", 0*} indicating client-agnostic and client-specific parts, respectively.
In the centralized setting that accesses to multiple source datasets together, using multiple BN layers forces the model to learn domain-specific and Nomain-invarian characteristics separately, finding the common knowledge in multiple domains with ERM (Gulrajani & Lopez- 202T) as follows:
1
ne Σ
LCenteralized CE
CE(Coa (F(0g. (F (fr,,)), Ut,k),
D
1=
where CB(:)) is the cross-entropy loss, and (0g, ΦG) are shared across all domains, thus it can learn characteristics while k-th coomain-spccifi information learned by BN statistics gr. In federated learning, local client only has single domain data different with centralized learning, thus the model cannot explicitly learn domain- inviriinnn representations. To solve the issue, we pro- pose mixed instance-glone globall statistics in BN layers to learn
representations.
3.2 CLIENT-AGNOSTE LEARNING WITH MIXED INSTANCF-GLOBAA STATISTICS
In local training, k-th local model is trained with the cross-entropy loss on k-th dataset as follows:
1
LCE
CE(Cos (F(pg. (2t,k)) w.,s)
DD
Before starting local training, the client receives the global model parameters 10g ΦG) and initialize the local model {0g, 3 with the global parameters while 0f remains. Then, F'og.02) and Co. are trained on local data for long epochs. Although {0%, Φx is initialized with the generalized global model, there is way learn client-invariant representations only using single domain data with the cross-entropy loss because the direct use of other clients' data leads to privacy issue. To mitigate the issue, we propose generate diverse domain using statistics in BN layers from the global model.
Mixed Instance and Global Statistics (MixIG): Global statistics that are aggregated with local BN statistics in the server reflect training data distribution over clients. We exploit this property for data augmentation in local training as illustrated in Fig. 2 In naive local training, an input tensor af normalized with statistics of batch samples in 1-th BN layer, and a local feature fi, is always calculated with statistics from single domain local data. It makes the local model learn representations only in single domain. Here, we propose to normalize inputs with mixed statistics exploiting global statistics beyond local statistics. We mix mean and standard deviation of each sample with global statistics {HG, Jc i.e., running mean and standard deviation, as follows:
= + (1 1---l u ) and of = u'o + (1 11-wld ('''a
(3)
where n4 and of indicate instance mean and standard deviation along the channel axis of the in- termediate feature of the i-th input to the 1-th BN layer, respectively. u' E RC is an interpolation weight vector, where each element independently sampled from uniform distribution U U(, 1) each
4

---

Under review as a conference paper at ICLR 2023
iteration, and C' is the feature dimension in I-th BN layer. The feature normalized by instance statis- tics contains local representative characteristics, and the normalized feature with global statistics is composed of global representations. By randomly interpolating these two statistics in all BN layers, we obtain more diverse data fully utilizing the characteristics of local and global domains. In Fig. 2, denote the intermediate feature by di.x normalized with MixIG, and the augmented feature fi,A
is obtained with {ua. d M We train the local model using both fi.k and fi.o in a client-agnoss way, which is described in the next section. While previous works (Zhou et al.) 202T: Li et al. 2021a) augment features with random noise values or styles of batch samples, our method gets access to aggregated data distribution for safe and diverse augmentations as multi-source domain generalization. It is noting that our method uses global statistics in BN layers for data augmentation, which can reduce threat of privacy leakage different with Wu & Gong (2021); Liu et al. (2021).
Client-agnostic Learning Objectives: We propose a client-agnostic feature loss as follows:
Ak
CCAFL
Σ
1 12
(4)
D2
With this loss function, the local model can explicitly extract the client-agnostic features by min- imizing the distance between the original and augmented features. Here, we perturb the features on the client-specific part, i.e., BN statistics, thus the client-agnostic part explicitly learns client- invariant characteristics helping mitigate domain shift. In addition, train the local classifier to classify the features from other domains forcing the classifier to be client-agnostic. To achieve it, the local classifier Co., is trained with client-agnostic classification loss follows:
LCACL
CE(Cos W1.))
(5)
DE
Our client-agnostic learning can be considered regularization method that forces local models not to deviate largely from the global model. Different with the previous work (Li et al., (020) that directly regularizes local weight parameters with the global model, our proposed learning considers the importance of weight parameters for fiient-invarian representations with diverse domain data.
The overall loss for local optimization as follows:
Liotal = LcE + λ1 -CACL + X2 LCAFL.
(6)
where A and A2 are balancing parameters. After local training for long epochs, model parameters
are aggregated by FedAvg (McMahan et al. (217)) in the server, i.e., 0g Ex= me 0g and dG
n
ΣK Be dh. We denote FedIG as our federated model trained by Eq. 66.
3.3 ZERO-SHOT ADAPTATION
At inference time, the globle model, F(og.92) and Coa. is deployed to unseen clients. It cannot generalize well to completely unseen domains, where data distribution is shifted from training dis- tribution. the literature of domain generalization using multiple BN layers, Seo et al. (2020) uses ensemble predictions from multiple BN layers, and Chen et al. 2022): Zhou et al. (2022) get the prediction from the selected BN that is most related to test input. However, they cannot be appli- cable to the federated setting since local client model cannot access local BN layers of other clients, i.e, only global BN layers are allowed to access due to user privacy. In addition, recent test-time adaptation works (Gong al., 2022 You al., T02T Hu et al. 2021) use the interpolated statistics between instance and learned statistics to reflect test distribution, but their interpolation parameters manually fixed suitable to the target domain or generated from the rule-based function containing sensitive hyper- parameters. Here, we propose to dynamically generate instance-wiss interpolation parameters for mixing instance and global statisties with learning- network.
Interpolated BN Statistics: We utilize statistics of the test input with global statistics as follows:
(7) where of and of indicate instance mean and standard deviation of the input in 1-th BN layer. of and of are used for normalizing the test input tensor. a' is an interpolation parameter that adjusts the contribution of instance statistics of the test sample. Ideally, optimal a is selected to test domains or test inputs, but we cannot access test domain. We propose a zero-shot adapter that is carefully designed to dynamically generate a for each input in both seen and unseen domains.
1² = a' a'u4 + (1 - (1-aa))d "u'a and of =d==a'o a'a? + (1 (1-a1)a a')o'c.
5

---

Under review as a conference paper at ICLR 2023
Design of Zero-shot Adapter: We design the zero-shot adapter G2 parameterized by y, which aims to generate proper a for the test sample. The zero-shot adapter is separately added each BN layer in the feature extractor. We set an input of the adapter in (-th BN layer as the channel-wise distance between instance and global statistics {ui Pa:of - d'} E R2c and an output is a'. With this design, the adapter estimates the statistics based on the distance between input and global statistics as an instance-wist manner at the test time.
Zaro-shot aduptation dia vi.
xLa
de
Y
P
E4
Eq. (9)
Freere
Zero-shot adupter Go!
Updaae
Figure 3: Zero-shot adapter takes the difference between instance and global statisties, and it gen- erates estimated statistics by Eq. and Eq. (
Training Strategy for Zero-shot Adapter: In local training, we freeze the main model, i.e., Fex and Co.. and train the adapter with the cross- ertsopy loss to classify the inputs as follows:
CA
CCE(Cor (F10g,02) (Ta,))) v1.))
(8)
D&
where Of indicates the interpolated statisties described in Eq. (7). and a' is generated from Gipl, as shown in Fig. [ To prevent the zero-shot adapter from over fittinggting to each local training data, apply reparameterization trick (Kingma & Welling 2013). We generate a' sampled from the gaussian distribution reparameterized by the zero-shot adapter as follows:
(1u) and 2t is sampled from (0, 1). T(-) is a clamp function to ensure a' within the range of [0. 1]. Here, the main network and the adapter are alternately trained with the loss functions Eq. (6) and Eq. (8) respectively, freezing the other model. The training procedure for the adapter does not affect the performance on the main network since the purpose of the adapter is only to learn how to interpolate instance and global statistics for minimizing the cross-entropy loss on the trained network, which simulates the test scenario in new clients. Local zero-shot adapters are also aggregated by FedAvg. Zero-shot Adapter Inference Time: We set a' as e', which the mean of 8' e The test input is normalized by interpolated statistics using a' at I-th BN layer, and this process operates via one forward propagation same with the naive inference process. We analyze the inference cost in the experimental section. We denote our overall framework by FedIG-A
a' = T(6'2 4+++)) +e'), where
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and Settings: We conduct extensive experiments on three DG benchmarks: PACS (Li et al. 2017). VL.CS (Fang et al., 2013). and OfficeHome (Venkateswara et al. 2017). PACS contains seven categories from Photo (P), Art painting (A), Cartoon (C), and Sketch (S) domains, where domain shift is large across four domains (Dou et al. 2019:; Chen et al. 2022) compared to VLCS and OfficeHome (see analysis in A.4). VLCS consists of five categories which are collected from
VOC2007 (V) (Everingham al. 2010 LabelMe (L) (Russell et al. 2008). Caltech- 101 (C) (Fei-
Fei et al. 2004), and SUN (S) (Xiao et al. 2010). where domain shift stems from the type of camera. Ofme @ennateewwaa et al. 2017) contains 65 categories collected from four domains. Artistic (A), Clipart (C), Product (P), and Real world (R), and the domain shift problem is not as severe as other two datasets (Du et al. 2022 Wu & Gong 202T). In federated DG, the client has single domain data, and there are total four clients in each DG benchmark. The global model is collaboratively learned with three clients, and the learned model is evaluated on the remaining client.
Implementation Details: In the federated learning process, all clients use the same architecture and syyer--arameen settings. We use R8sNet-18 (He et al. (016)) pretrained on ImageNet as the backbone network. For FedIG, all batch normalization (BN) layers are replaced with two BN layers, i.e., local and global BN layers, respectively. We added two fully connected layers in front of each BN layer for the zero-shot adapter. We train the network using SGD with momentum of 0.5, the fixed learning rate of 0.01, and the batch size of 64 for 200 iterations at each round. Total 40 rounds conducted following the federated DG setting Yuaa et al. (021). We set d and X2 to 0.1 and 4.0 in Eq. (6). respectively. Following Li et al. (2019), we only update the classifier by Eq. (5). We conduct ablation studies about balaneing parameters in A.

---

Under review as a conference paper at ICLR 2023
Table 1: Variants of MixIG on PACS.
Table Ablation studies of client-agnostic learn-
ing objectives on PACS.
Mixing methods
Acc.
(FedBN)
80.79
Objectives
Acc.
Global: 0.0
81.75
X (FedBN)
80.79
Instance: ta = 1.0
82.52
81.17 CAPL
Global & instance: e
83.43
82:72
Global & instanee: u (0, 1)
84.07
CAFL&CACI
84,07
Table 3: Comparison studies to show the effectiveness our components on PACS.
Aug.
Loss
Inference
Acc.
Avg.
(a)
CE FedProx
Global statistics
92.74 (0.97)
92.82 (0.90)
77.08 (0.59)
76.43 (0.93)
75.26 (0.99)
75.08 (0.85)
78.28 (1.14)
78.95 (1.64)
80.86
80.79
(b)
Mixstyle MixIG
CAL
Global statisties
92.04 (0.63)
80.19 (0.28)
77.14 (0.48)
82.53 (0.94)
82.98
92.99 (0.61)
82.17 (1.15)
77.71 (1.13)
83,40 (0.19)
84,07
BIN; learned alpha
93.86 (0.53)
78.81 (0.45)
79.54 (0.40)
81.94 (0.88)
83.54
Ensemble BNs
93.70 (0.39)
83.17 (1.09)
76.08 (1.30)
83.14 (1.02)
84,02
JABN
93.29 (0.13)
80.18 (1.16)
79.38 (1.33)
83.58 (0.91)
84.11
(c)
MixIG
CAL
MixNorm
93.60 (0.52)
82.77 (0.41)
78.30 (1.13)
83.12 (0.74)
84.45
Random alpha
90.67 (0.59)
74.82 (2.97)
74,96 (0.76)
79.03 (1.64)
79.87
Fixed alpha (0.5)
93.59 (0.25)
79.27 (1.40)
79.64 (1.00)
82.43 (1.12)
83.73
Zero shot adapter
94.24 (0.33)
84.30 (0.44)
(1.16)
83.79 (0.49)
85.53
Evaluation Protocols: We follow the standard DG evaluation protocols in DomainBed (Gulrajani & Lopez-Paz 202T) including dataset splits, image augmentations, and the model selection strategy. For federated DG, we set the model selection strategy as single-source DG validation on local clients and multi-source DG validation on the server. We give the details in A.2 In this protocol, we reproduce all competitive methods for a fair comparison in the federated setting. All experiments are reported the average accuracy and standard deviation over four runs with different random seeds.
4.2 ABLATION STUDIES OF OUR COMPONENTS
Variants of MixIG: We conduct experiments with variants to deeply analyze MixIG in Table D Note that we use FedBN as the baseline and apply two client-agnostic learning objectives with data from variants of MixIG, but the zero-shot adapter is not used to separately analyze MixIG in this experiment. When we only use global statistics for MixIG, i.e., u = 0.0 in Eq. (3). the input is nor- malized with statistics of the global model. Global statisties reflect data distribution of whole train- ing clients, thus our objectives with normalized features allow the model to learn domain-invarian representations with global information. In the case of using instance statistics only, the style re- moved features (Huang & Belongie| (017)) become closer to the original features, which indicates invarintn information to be learned within local domain. Features normalized with global and instance statistics improve 0.96% and 1. 1.73% of the average performance on PACS, respectively. By mixing global and instance statistics, the model can take advantage of both characteristics, where nomain-invariant information is learned from global and local domains. Moreover, using randomly mixed statistics generates more diverse domain data such that we achieve the performance improve- ment further. Since global and instance statistics are randomly used for each BN layer, the model can effectively learn representations within local client, across clients, and within other clients. We also analyze the effect of distribution range in A.3
Effectiveness of Client-agnostic Learning: We apply two loss functions, client-agnostic classifi- cation loss (CACL) and client-agnostic feature loss (CAFL), on augmented features. CACL builds a client-agnostic classifier, and CAFL forces the feature extractor to learn domain-invariant infor- mation by strictly minimizing the distance between two features. In Table we show that each loss function is effective, and CACL and CAFL complementary operate to each other.
Client-agnostic Learning beyond Regularization: In Table (a). we experiment the weight reg- ularization method, FedProx (Li et al., 2020). Client-agnostic learning with the features only using global statistics can be considered as regularization not to deviate from the global model, but it even outperforms FedProx (81.75% vs 80.86%), here set μ to 0. Our approach regularizes the model considering the importance of model parameters suitable for client-invariant representations on local data. Beyond regularization, FedIG trains local clients with diverse domain data via MixIG.
Effectiveness of MixIG: MixIG is one type of feature augmentation for DG. We compare with Mixstyle (hhou et al.) 2021), which randomly mixes instance statistics within batch samples to synthesize styles and learn style-agnostic features. In (b), we replace MixIG with Mixstyle for fair comparison, and MixIG consistently outperforms Mixstyle on all domains. Mixstyle only
7

---

Under review as a conference paper at ICLR 2023
accesses the single-domain data in local training while MixIG exploits global statisties, and it shows that feature augmentation with global statistics makes the model more robust to domains.
Comparison Studies of Inference Methods: In Table (c), we compare with various inference algorithms. First, we compare with BIN (Nam & Kim 2018). denoted by learned alpha, which replaces all BN layers to the weighted summation of BN and IN layers in the backbone network. In training, the learnable interpolation parameters for BN and IN are optimized, and learned parameters are used for test samples. Since the interpolation parameters are fitted to training datasets, it cannot generalize well on unseen domains. Next, we adapt various inference algorithms on the trained model using FedIG for a fair comparison. We obtain ensemble predictions from multiple local BN layers. Learned multiple BN statistics can partially reflect the distribution of unseen domains, but the performance on some domains that are severely deviated from training domains is downgraded. Furthermore, this method leads to privacy issue due to sharing multiple local statistics. IABN (Gong et al., 2022) calibrates learned statistics with instance statistics by the rule-based function when difference between learned and instance statisties is large. LABN has sensitive hyper-pant parameters, and these parameters should be properly selected for each domain to consistently improve the performance performs IABN on all domains without any sensitive hyperparameters. MixNorm (Hu et al.] (22TT) augments test input data with various spatial augmentations, here we use the original data and four augmented data, estimating test distribution more accurately. It slightly boosts the performance, but the inference time and the memory usage increase. We also compare the zero-shot adapter with naive approaches, random and fixed alpha (You et al. [022]) Compared to two results, our zero-shot adapter can generate more proper alpha values for each BN layer as an instance-wise manner. shows that layer-wise and instance-wise generation is more effective than using the fixed alpha.
zero-shot adapter
all domains. Our learning based zero-shot adapter out-
Computational Overhead Analysis: Table shows the computational cost of each method. Training and inference times are measured as the average time per iteration with batch size 64 6342 and a single NVIDIA RTX A5000. FedAvg, FedBN, and FedIG do not require additional process- ing to forward the test input at inference, while BIN consumes more time for calculating instance nor- malization. We also add several inference methods (IABN, MixNorm) to FedIG to confirm the computational efficiency and performance of our FedIG- A. MixNorm forwards multiple data by batch; here, we use five for each test sample. It requires five times more memory and computations than other methods. IABN calculates instance statistics in each BN layer and gets alpha values from the rule-based function. It shows better efficiency than MixNorm, but with less performance gain. FedIG-A achieves the best accuracy by increasing marginal parameters and computations. Our FedIG and FedIG-A have a trade-off between perfor- mance and overhead, but both are superior to other baselines terms of efficiency and performance. Users can choose the method according the requirements of the target devices.
Table 4: Comparison of computational cost.
Acc. denotes the average accuracy PACS.
Mothods
AParms.
Training
Inference
Aoc.
Intel Xeon Gold
FedAvg
11.3M
1.43ss
1.34ms
77.35
FedBN
11.31M
1.33s
80.79
FedIG
11.31M
2.25ms
1.34ms
84.07
BIN w/ MixlGi
11.31M
3 3,24ms
1.35ms
83.54
FediG MixNomm
11.31M
2.25ms
6.70ms
84.45
HellG w/ IABN
11.31M
2.25m
1.57ms
84.11
FedIG-A 11.55M
3.50ms
1.37ms
85.53
Effectiveness of Zero-shot Adapter: Using the zero-shot adapter, we plot alpha values on each layer from all test samples in unseen A domain in Fig. 4 Alpha values are different for both instance- wise and layer-wise. Since the degree of distribution shift of samples is different even in the same domain, it effective to use different alphas for samples and layers. More results and analysis are described in A.4 In t-SNE plots, FedIG-A has fewer points in the middle, which indicate non- meaningful features, and extracts more class-discriiinative features than FedIG.
(b)
Figure 4: (a) alpha values each layer from all test samples, and (b) t- SNE of FedIG and FedIG-A in unseen A domain, where features are normalized by global (left) and estimated statistics (right).
8

---

Under review as a conference paper at ICLR 2023
Table 5: Classification accuracy comparison results on PACS, VLCS, and OfficeHome. Gray color indicates methods posing privacy issues.
PACS C
VLCS
OfficeHome
Paradigm
Method
5
Avg.
Avg.
Avg.
Decentralized w/o DG
FedAvg
90.40 (0.97)
72.52 (2.28)
72.59 (0.37)
73.90 (1.74)
77.35
74.86
63.47
FedProx
90.49 (0.69)
72.41 (1.06)
73.09 (0.91)
72.93 (1.48)
77.23
74.42
63.02
FedBN
92.74 (0.97)
77.08 (0.59)
75.08 (0.85)
78.28 (1.14)
80.79
75.33
64,08
RandAug
93.57 (0.60)
77.15 (1.39)
71 (2. 14)
66 64.64 (2.34)
77.26
74.32
64.59
Mixstyle
92.75 (0.45)
80.21 (0.33)
76.77 (0.72)
79,98 (2.22)
82.43
75.54
63.24
SBA
87.22 (1.76)
72.01 (3.22)
73.12 (1.03)
79.52 (1.63)
77.9%
72.77
59.47
Decentralized w/ DXG
RandConv
91.78 (0.40)
(1.55)
70.56 (3.28)
78.78 (1.46)
79.57
73.68
63.25
1,2D
93.59 (9.55)
(1.12)
75.81 (1.37)
82.76 (9.55)
82.96
75.37
63.29
JiGeen
92.14 (0,44)
72.51 (2.00)
72.76 (1.24)
73.34 (1.96)
77.69
75.13
64.09
RSC
92.53 (0.86)
78.10 (0.56)
75.95 (1.08)
79.82 (1.31)
81.60
75.90
63.01
SelfReg
93.41 (0.76)
78.30 (1.16)
74.94 (0.43)
77.13 (2.09)
80,94
72.79
64.85
COPA
(1.07
83.75 (0.29)
78.58 (0.96)
84.45 (1.33)
85,37
74.51
62,42
Decentralized w/ federated DG
FedDG
94.45 (0.44)
83.83 (0.28)
73.41 (1.33)
78.40 (0.73)
82,52
75.28
64,90
CSAC
94.83 (0.42)
80.48 (1.25)
75.46 (1.58)
79.56 (1.35)
82.58
75.21
64.35
FedIG
92.99 (0.61)
82.17 (1.15)
77.71 (1.13)
83.40 (0.19)
84.07
76.62
64.01
FedIG-A
94,24 (0.33)
84.30 (0.44)
79.80 (1.16)
83.79 (0.49)
85.53
76.68
64.71
4.3 COMPARISON WITH COMPETITIVE METHODS
Competitive Methods: Table we compare our FedIG and FedIG-A with representative meth-
ods of FL and DG. (1) Federated learning: FedAvg (McMahan et al.) 2017). FedProx (Li et al.,
2020 and FedBN (Li et al. 2021b); (2) Data augmentation method: RandAug (Cubuk al 2020
(3) Augmentation-based DG methods: Mixstyle (Zhou et al. 2021). SFA (Li et al.) 2021 a). Rand-
Conv (Xu et al. 2021b), and L2D (Wang et al. 2021); (4) Regularization based DG methods: Ji-
Gen (Carlucci al. 2019), RSC (Huang al. 2020). and SelfReg (Kim et al. 2021 I); (5) Federated
DG methods: COPA (Wu & Gong 2021). FedDG (Liu et al. 202T). and CSAC Yuan et al. (02T).
Performance Analysis: We first analyze the results PACS with severe distribution shift be- tween domains. Decentralized without DG methods achieve low performance compared to other paradigms. FedProx regularizes local models not to deviate from the global model, but it cannot solve domain shift. FedBN mitigates domain shift with local BN layers, and it can improve the performance on unseen clients. However, the performance is very limited since they do not con- sider learning ciient-invarian representations, explicitly. Decentralized with DG methods show per- formance improvement on several domains. Especially augmentation- based DG methods achieve significant accuracy improvement, but performance on few domains is downgraded a lot. Since augmentation-basse DG methods learn domain-invariant representations only within single domain, source domain does not cover test distribution. JiGen,
it is not effective when augmentation RSC, and SelfReg consistently improve or maintain the performance on four domains compared to FedAvg, but the improvement is very marginal. COPA and FedDG allow the model to learn client- invariant representations by utilizing data information from other clients, and they obtain the highest performance several domains among competitive methods. However, they cause serious privacy issue. Our methods, FedIG and FedIG-A, achieve the state-of-the-art performance without sharing private information across clients. FedIG outperforms all competitive methods with large margin on most domains except for COPA. With zero-shot adaptation, we boost the performance further without significantly increasing the computational cost. On VLCS and OfficeHome, domain shift is relatively small compared to PACS. FedIG and FedIG-A consistently improve the performance almost all domains compared to other methods, indicating that our methods can be safely applied to any domains (see more results in A.5). We show the results on cross-silo FL in A.6 Moreover, we analyze the performance on clients inside the federation in A.7
CONCLUSION
We presented alient-agnostic learning with mixed instance-global statistics for local training and zero-shot adaptation with estimated statistics for inference. Our mixed instance-global statistics generate diverse domain features helping local clients to learn client-invariant representations while ensuring user privacy. In addition, gap between training and test clients at inference time. Extensive experiments on federated DG benchmarks showed the effectiveness of our methods.
proposed zero-shot adapter directly bridges large domain

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our experimental evaluation is conducted with publicly available DomainBed (Gulrajani & Lopez- Paz 202T) and CSAC (Yuan et al. 202I). We provide the data preprrcessing and hyper-parameter settings in Section 4.1 and pseudo-codes in A.8 Together with the references of related works and publicly available codes, our paper contains sufficient details to ensure reproducibility.
REFERENCES
Mathicu Andreux, Jean Ogier du Terrail, Constance Beguier, and Eric W Tramel. Siloed federated learning for multi-centric histopathology datasets. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, pp. 129-139, Springer, 2020.
Fabio M Carlucci, Antonio D D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoqing Liu, and Yizhou Yu. Compound domain generalization via meta-knowledde encoding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1119-7129, 2022.
Yang Chen, Yu Wang, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei. A style and semantic memory mechanism for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9164 9173, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generaliza- tion via agnos-tg learning of semantic features. featrres Advances in Neural Information Processing Systems, 32, 2019.
Dapeng Du, Jiawei Chen, Yuexiang Li, Kai Ma, Gangshan Wu, Yefeng Zheng, and Limin Wang. Cross-domain gated learning for domain generalization. International Journal of Computer Vi- sion, pp. 1-16, 2022.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference Computer Vision, pp. 1657-1664, 2013.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178 IEEE, 2004.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lec. Ro- bust continual test-time adaptation: Instance-aware bn and prediction-balancedd memory. arXiv preprint ar arXiv:2208.00 11 2022.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URL/tttss://ppeneeiiii net forum?
1 XXdXeXDoWtI
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser-Nam Lim. Mixnorm: Test-time adaptation through online normalization estimation. arXiv preprint arXiv:2110.11478, 2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In European Conference on Computer Vision, pp. 124-140. Springer, 2020.
Meirui Jiang, Xiaofei Zhang, Michael Kamp, Xiaoxiao Li, and Qi Dou. Tsmobn: Interventional generalization for unseen clients in federated learning. arXiv preprint arXiv:21 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning. PMIR, 2020.
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lee. Selfreg: Self- supervised contrastive regularization for domain generalization. In Proceedings the IEEE/CVE International Conference on Computer Vision, pp. 6629,9628, 2021.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes. arXiv preprint
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
Da Li, Jianshu Zhang, Yongxin Yang. Cong Liu, Yi-Zhe Song, and Timothy Hospedales. Episodic training for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1455-1455, 2019.
Pan Li. Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 1895, 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedingy of Machine Learning and Sys- tems,
2020.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. FedBN: Federated learn- ing non-lID features via local batch normalization. In International Conference on Learning Representations, 2021b.
net
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain gen- eralization on medical image segmentation via episodic learning continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1013-1023, 2021.
Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8046 80568 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-fffiienn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017.
Hyconseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style-invariant neural networks. Advances in Neural Information Processing Systems, 31, 2018
Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh Ap. Generalization unseen domains via inference-tine time label preee-nres target projections. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12924-192111 1933, 2021.

---

Under review as a conference paper at ICLR 2023
Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: database and web-based tool for image annotation. International journal of computer vision, 77(1):157- 2008.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In European Conference on Computer Vision, pp. 83. Springer, 2020.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5018-5027, 2017.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 1-7623,, 2020.
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 834 2021.
Guile Wu and Shaogang Gong. Collaborative optimization and aggregation for decentralized domain generalization and adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6484-6493, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14382-14392, 2021a.
Zhenlin Xu, Deyi Liu, Junlin Yang. Colin Raffel, and Mare Niethammer. Robust and generalizable visual representation learning via random convolutions. In International Conference Learning Representations, 2021b. URI
eww hwt.:/ee//eeeeee stte:fforummiddBBsMM00
Fuming You, Jingjing Li, and Zhou Zhao Test-time batch statistics calibration for covariate shift. 2021.
arXiv preprint
Junkun Yuan, Xu Ma. Defang Chen, Kun Kuang, Fei Wu, and Lanfen Lin. Collaborative semantic aggregation and calibration for separated domain generalization. arXiv e-prints, pp. arXiv-2110. 2021.
Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhiming Luo, Yaojin Lin, Shaozi Li, and Nicu Sebe. Learning generalize unseen domains via memory-based multi-source meta-learning for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6277-6286, 2021.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference Learning Representations, 2021.
7MVxxp
Ziqi Zhou, Lei Qi, Xin Yang, Dong Ni, and Yinghuan Shi. Generalizable cross-modality medi- cal image segmentation via style augmentation and dual normalization. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20856-20865, 2022.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PERFORMANCE DEPENDENCY OF HYPER-PAREE PARAMETERS IN OBJECTIVES
In Eq. (6) there are two pyper-parameters. A and Ag. and we analyze performance dependency of these parameters. It is worth noting that it is an important ablation study in DG, where the DG model
should generalize well on several domains, not sensitively depending on these parameters. In Fig. 5
6 we conduct experiments changing each parameter while fixing another parameter, Xι = 0.1
and X2 = 4.0. Although the optimal hypperparammttes are different with each domain, choosing A1 in [0.1,0.5] and Ag in [2.0,5.. 5.0] shows consistent results on all domains. In this range, FedIG-A consistently achieves high performance on all PACS domains compared to competitive methods (see
Table 5). We train and evaluate models by setting syper-parameter: to optimal values for each domain, but we set A and À to 0.1 and 4.0 for all datasets in our experiments.
94.5
$0.5
98.6
82
P domain
A domain
C domain
S domain
Figure 5: Ablation studies of client-agnosti classification loss on PACS. We conduct experiments using various A (X-axis) and get the performance (y-axis).
86.5
80
25
09
73.5
04
64
37.5
82
P dommman
A domala
C domaiin
S domaia
Figure 6: Ablation studies of client-agnostic feature loss on PACS. We conduct experiments using various X2 (x-axis) and get the performance (y-axis).
A.2 EVALUATION PROTOCOLS FOR FEDERATED DG
We use training-domain validation set for the model selection in Gulrajani Lopez- (202I). We adopt different selection strategy for the client side and the server side to obtain the best global model in the federated DG setting. In local training, the client model uploaded to the server when the validation accuracy on its single domain is best within 200 iterations (validation per 20 iterations). In the server, the best round model is selected when the average of the validation performance on seen clients is maximized among 40 rounds (validation per 1 round). The server obtain the validation performance from clients using the aggregated server parameters, thus it can be possible use the average of the validation performance seen clients. It is practical and effective to use single-source DG validation on local training and multi-source DG validation on the server.
A.3 MORE STUDIES ON MIXIG
Table we conduct experiments using different range of uniform distribution for mixing instance and global statistics in Eq. (3) The optimal range of augmentation is different for each test domain. In other words, the desired strength or type augmentation is different for each domain, as shown in Table. 5 Different with ingmentation-basee DG methods, method interpolates instance and global information. The performance is consistently improved in all domains due to safe and diverse augmentation using features normalized within instance and global statistics. With extrapolation
13

---

Under review as a conference paper at ICLR 2023
U(-0.1,.1), the performance on S domain, which is severely deviated with P. A, and C domains, is largely improved. Using extrapolated statistics generates more diverse samples compared to using interpolated statisties, but it induces the performance drop not severely different domains.
Table 6: Accuracy PACS using various range of distribution for MixIG.
PACS
VLCS
Distribution
P
A
S
Avg.
Avg.
U(0 1)
94.24 (0.33)
84.30 (0.44)
79.80 (1.16)
83.79 (0.49)
85.53
76.68
0.0.0,0.5)
94.46 (0.21)
84.50 (0.17)
80.97 (1.15)
84.46 (0.09)
86.10
76.42
U
94.58 (0.21)
82.89 (0.59)
78.14 (1.48)
83.18 (0.04)
84.69
76.36
1.-0.1,1.1)
94.46 (0.64)
83.64 (0.83)
80.12 (0.36)
84.59 (0.05)
85.70
76.46
A.4 ANALYSIS OF ALPHA VALUES FROM SHRO ADAPTER
At the test time, we use alpha values in Eq. from the trained zero-shot adapter for inference. We plot alpha values on each layer from all test samples in PACS, VLCC, and OfficeHome in Fig. 7 8) and 9 In PACS, alpha values in P. A, and C domains have similar distribution on each layer. In the low-level layers, different alpha values are used for test samples between 0.1 and 0.6, i.e. using different amount of instance statistics, and in the high-level layers, alpha values are within 0.0 and 0.6. In S domain, alpha values at the low-level layers lower than other domains. Since there is a large domain gap between and other domains, distribution of alpha values are different. Interestingly, alpha values in the middle-level layers are almost same across test samples, e.g., almost all test samples get 0.2 on the 9-th BN layer in domain. Alpha values in VLCCS are all between 0.0 and 0.2, and it shows that global statisties work well without instance statistics because domain shift existing in VLCS is not large compared to PACS. Similarly, alpha values in OfficeHome are smaller than those PACS. The distribution of alpha values is similar across domains because domain shift is not large. The degree of shift between domains can be inferred from the distribution of alpha from the zero-shot adapter.
05-
0.
0.0
0.0
04-
0.5
011
Figure 7: Alpha values on each layer from test samples in (upper left), A (upper right), C (lower left), and S (lower right).
14

---

Under review as a conference paper at ICLR 2023
3.00
3.00
0.75
0.755
0.56
0.50
0.25
0.55
86 1.00
1.00
0.7%
4.01
0.500
0.550
0.825
0.22
0.000
0.000
0.200
9.000
0.75
0.773
0.550
0.15
0.122
0.125
Figure 8: Alpha values on each layer from test samples in V (upper left), I. (upper right), C (lower left), and S (lower right).
06
64
64-
12-
03
06'
05-
000
0.
20
06
LOX rumber
Figure 9: Alpha values each layer from test samples in A (upper left), C (upper right), P (lower left), and R (lower right).
15

---

Under review as a conference paper at ICLR 2023
A.5 EXPERIMENTAL RESULTS ON VLCS AND OFFICEHOME
In T our FedIG and FedIG-A achieve the state-of-the-art performance on VLCS. Several meth- ods downgrade the performance on V domains compared with FedAvg, egg. CSAC achieves low performance on V and domains, but we can get consistent better performance on all domains. In Table 8 both decentralized with DG and federated DG paradigms cannot get large improvement on OfficeHome because domain shift across clients is very small. FedIG-A gets comparable results with the state-of-the-ee tart--rr previous methods.
Table Comparison results on VLCS. Gray color indicates methods posing privacy issues.
VLCCS C
Paradigm
Method
Avg.
FedAvg
73.53 (0.87)
57.92 (1.08)
96.22 (0.93)
71.77 (1.15)
74.86
Decentralized w/o DG
FedProx
73.04 (0.53)
57.85 (1.27)
95.97 (0.75)
70.83 (3.91)
74.42
FedBN
71.78 (0.98)
58.71 (1.14)
96.67 (0.53)
74.15 (1.59)
75.33
RandAug
72.42 (0.78)
58.32 (0.74)
95.79 (0.86)
70.76 (2.27)
74.32
Mixstyle
72.61 (0.66)
58.52 (0.66)
97.69 (0.51)
73.33 (1.37)
75.54
SFA
65.08 (0.81)
61.55 (1.22)
96.29 (1.19)
68.15 (1.07)
72.77
Decentralized w/ DG
RandConv
70.83 (0.90)
57.16 (1.50)
95.64 (0.23)
71.08 (2.24)
73.68
L2D
72.84 (1.59)
59.78 (0.69)
97.97 (0.54)
70.89 (1.83)
75.37
JiGen
73.65 (0.57)
58.09 (0.69)
98.06 (0.35)
70.72 (1.21)
75.13
RSC
75.27 (1.00)
59.79 (1.22)
97.01 (1.01)
71.51 (1.00)
75.90
SelfReg
68.13 (0.76)
60.37 (0.66)
88.64 (3.09)
74.03 (0.19)
72.79
COPA
71,50 (1.05)
61.00 (0.89)
93.83 (0.41)
(0.74))
74.51
FedDG
71.05 (0.62)
59.46 (1.08)
96.64 (0.86)
73.96 (0.74)
75.28
Decentralized w/ federated DG
CSAC
72.96 (0.91)
59.78 (0.84)
96.52 (0.38)
71.60 (1.20)
75.21
FedIG
73.69 (0.90)
59.14 (0.74)
98.00 (0.29)
75.65 (1.00)
76.62
FedIG-A
73.93 (1.14)
59.29 (0.28)
97.84 (0.37)
75.64 (0.86)
76.68
Table 8: Comparison results on OfficeHome. Gray color indicates methods posing privacy issues.
OfficeHome P
Paradigm
Method
Avg.
FedAvg
56.82 (0.31)
50.53 (0.60)
72.38 (0.24)
74.16 (0.38)
63.47
Decentralized w/o DG
FedProx
56.58 (0.63)
49.44 (0.39)
72.15 (0.30)
73.90 (0.46)
63.02
FedBN
58.29 (0.70)
51.16 (0.48)
72.80 (0.59)
74.06 (0.51)
64.08
RandAug
58.50 (0.34)
52.18 (0.77)
73.17 (0.36)
74.52 (0.52)
64.59
Mixstyle
55.57 (1.24)
53.31 (0.65)
70.90 (0.81)
73.18 (0.27)
63.24
SFA
50,99 (1.20)
50.97 (0.35)
66.84 (0.65)
69.08 (0.23)
59.47
Decentralized DG
RandConv
56.19 (0.80)
53.20 (0.47)
71.66 (0.57)
71.94 (0.46)
63.25
L2D
54.70 (1.43)
56.36 (0.28)
69.96 (0.85)
72.13 (0.35)
63,29
JiGen
58.20 (0.67)
50,00 (0.02)
73.99 (0.56)
74.18 (0.00)
64.09
RSC
56.67 (0.59)
49.59 (1.05)
71.61 (0.47)
74.16 (0.49)
63.01
SelfReg
59.26 (0.26)
51.84 (0.86)
73.46 (0.18)
74.85 (0.62)
64.85
COPA
53.39 (0.16)
57.46 (0.47)
68.61 (0.15)
70.24 (0.38)
62,42
FedDG
59.87 (0.06)
53.51 (0.31)
72.81 (0.89)
73.41 (0.41)
64.90
Decentralized w/ federated DG
CSAC
58.97 (1.13)
51.61 (0.26)
72.57 (0.18)
74.25 (0.49)
64.35
FedIG
57.68 (0.64)
52.90 (0.13)
71.79 (0.47)
73.68 (0.39)
64.01
FedIG-A
58.62 (0.51)
53.47 (0.21)
72.34 (0.45)
74.39 (0.17)
64.71
16

---

Under review as a conference paper at ICLR 2023
A.6 EXPERIMENTS CROSS-SILO FL
Experimental Setup: We expand 3 training clients to 30 clients by distributing each dataset to 10 clients PACS. It makes the client have the small amount of data. We consider both iid and non-iid label distribution across clients. We equally distribute the data to 10 clients for iid label distribution while we use Dirichlet distribution (a 0.5 and a 0.1) for non-iid label distribution. We illustrated class distribution of clients in Fig. Tо In this setup, we randomly select 10 clients for local training with iterations on each round, and total 120 rounds are conducted.
Experimental Results: In Table 10 and we compare our FedIG-A with FedProx Li et al. (2020) and FedBN (Li et al. 2021b) Our method works well when the number of clients is large. The performance is slightly downgraded in non-iid label distribution compared to the performance iid label distribution, but we can expect that this negative effect alleviated if the method for non-iid label distribution is added together. In the significantly improves the performance compared to the baselines as shown in Table [[] demon- strates that method effectively solves the domain shift problem even severe label shift exists.
of severe non-iid label distribution, FedIG-A
domain
C domain
domain
(a)
Coste ID
ID
Clint ID
ClentiD
A domain
domain (b)
domain
CD
Client ID
Clent D
domain
domain
domain
(c)
Figure 10: We plot the data distribution of clients using (a) iid data partition, (b) non-iid data par- tition following Dirichlet distribution of 0.5, and (c) non-iid data partition following Dirichlet distribution a = 0.1 when A, C, and S domains are source and is target. The color bar denotes the number of data samples, and x-axis indicates client IDD and y-axis indicates class ID. Each rect- angle represents the number of data samples of specific class in a client. In this setup, 30 clients participated in FL from and S domains, and we the FL model P domain.
17

---

Under review as a conference paper at ICLR 2023
Table Performance iid label distribution with 30 training clients.
PACS
Method
C
S
Avg.
FedProx
94.95 (0,42)
73.97 (0.48)
70.72 (0.28)
73.76 (0.24)
78.35
FedBN
94.53 (0.85)
75.95 (0.17)
70.53 (2.03)
78.45 (2.16)
79.86
FedIG-A
94.76 (1.14)
84.20 (0.31)
79.20 (2.02)
82.87 (1.76)
85.26
Table 10: Performance non-iid label distribution (a 0.5)
with 30 training clients.
PACS
Method
P
C
S
Avg.
FedProx
92.25 (0.25)
71.75 (3.56)
74.87 (0.61)
69.30 (6.14)
77.04
FedBN
92.82 (0.13)
74.78 (1.55)
75,50 (0.95)
71.71 (2.66)
78.70
FedIG-A
93.62 (0.30)
79.54 (0.83)
82.02 (0.15)
77.49 (1.17)
83.17
Table 11: Performance on non-iid label distribution (a = 0.1) with 30 training clients.
PACS
Method
P
C
S
Avg.
FedProx
83.17 (0.42)
62.65 (4.42)
66.95 (0,95)
55.34 (4.33)
67.03
FedBN
84.41 (1.40)
61.62 (0.48)
69.05 (3,81)
59.85 (4.00)
68.73
FedIG-A
92.13 (0.89)
73.29 (2.90)
73.19 (1.12)
71.96 (0.34)
77.64
A.7 PERFORMANCE ON CLIENTS INSIDE THE FEDERATION
We measure the performance clients inside the federation, i.e., personalized performance, in Ta- ble 12 PACS with a large domain shift across training clients, the model with local statistics, i.e. the client-specific part, achieves the best performance compared to the model with global statistics, i.e. the global model. Global statistics reflect data distribution all clients, but it causes the per- formance degradation
each test domain, where data distribution is shifted from other clients. Our proposed zero-shot adapter can reduce the performance gap between using local and global statis- ties with the aid of instance statisties of the test input. In the case of VLCS and OfficeHome, global statistics help the model to generalize well to each test domain. Since the distribution shift across clients is not large, global statistics represent more general distribution compared local statisties. It improves the model performance a lot. The zero-shot adapter also helps the model to generalize well seen domains VLCS and OfficeHome.
Table 12: Personalized performance on PACS, VLCS, and OfficeHome.
PACS
Method
P
A
C
S
Avg.
FedIG (local statistics)
97.85 (0.89)
93.69 (2.06)
94.92 (0.91)
95.51 (0.37)
95.49
FedIG (global statistics)
96.86 (0.95)
92.46 (1.79)
93.81 (1.04)
95.02 (0.40)
94.53
FedIG-A (global statistics)
97.40 (0.79)
93.53 (1.77)
94.53 (1.08)
95.33 (0.44)
95.20
VLCS
Method
L
S
Avg.
FedIG (local statistics)
83.79 (1.86)
71.72 (0.84)
99.44 (0.41)
80.76 (1.15)
83.93
FedIG (global statistics)
84.80 (1.74)
70.12 (1.24)
99.80
(0.23)
81.90 (1.62)
84.15
FedIG-A (global statistics)
84.92 (1.80)
70.29 (1.37)
99.85 (0.18)
82.09 (1.54)
84,29
OlficeHome
Method
Avg.
FedIG (local statistics)
66.57 (1.68)
76.11 (1.21)
87.81 (1.32)
79.78 (1.80)
77.57
FedIG (global statistics)
68.05 (2.03)
76.77 (1.20)
88.19 (1.14)
80.53 (1.91)
78.38
FedIG-A (global statistics)
68.50 (1.82)
76.63 (1.16)
88.08 (1.25)
80.70 (1.99)
78.48
18

---

Under review as a conference paper at ICLR 2023
A.8 PSEUDO-CODE FOR REPRODUCIBILITY
We describe pseudo-codes of client-agnostic learning and zero-shot adaptation in Table T3 and 14 Note that federated DG benchmarks only contain four clients, i.e., three for training and for test, thus all three clients participate in the federation at every round.
Table 13: Pseudo-code for FedIG-A training.
Global weights (02. 0g), Φa. pac, Local clients' weights (f)00)) (or D (
Total round Teaxs Total local iteration Emazi
Server executes:
initialize {0g. ). &G. PG: for each round 1, ...g Toer do
St + (random set of m clients);
for each client & E St in parallel do
Load (0%, 0f] @r. Pp f LocalUpdate(k; (0% (8,00)) &G. PG):
Update global weights (02, (, Φa. PC by FedAvg (McMahan et al. 2017 Output: {0% 0g ØG. PG.
function LocalUpdate(k; {0%, Ggt. ΦG, PG): // Run client k Load 0g. Ox. (Px 08. &G: PP: for each local iteration
1,.., Emaxx do
Shuffle training set Ds:
Fetch mini batch (Ii,ks 31 from D&:
Train the main network Obtain loss LC by Eq. 2
Obtain augmented features {f. h with 0% by Eq. 3
Obtain loss LCACL ACAFL L Eq. 44 and (5)
Update local weights {0g, 02) Φb by minimizing Eq.
// Train the zero-shot adapter Obtain features with estimated statistics by Eq.
7 and
Update local weights fpu by minimizing Eq. 8
Output: {02 0 Φk.
Table 14: Pseudo-code for FedIG-A inference.
Global weights {0g, 0g), &G. Pa. Test client's weights {0r, 02}, dr. PPG
Deploy to test client:
Load (0], 07), di, P2: (6g, (g). de. Pa:
Obtain test set D;;
for each test forward = 1,..., do
Fetch x,, from D;;
Obtain fic by Eq. 7 and
Obtain the prediction with Co.;
19


Reviews:
[{"review_id": "z9Jjywi6I0", "reviewer": "Reviewer_Rg7n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article tackles the problem of federated domain generalization. The method includes two parts: the first part mixes the BN parameters of each local model with the global model, and performs feature augmentation to improve model robustness; the second part introduces a zero-shot adaptor for unseen client generation. The method is verified over three datasets and results look promising. ", "summary_of_the_review": "The paper needs a significant revision to improve the presentation and clarify the rational of the model design. The current version is far from reaching the acceptance bar of ICLR. ", "strength_and_weaknesses": "Strength:\n\n* The experiments are comprehensive and results are good. \n* The method is simple and lightweight. \n\nWeaknesses:\n\nFirst of all, I would say that the paper is not well written. Though the method is simple, it is not easy to understand all the components as a unified model. Below I give my comments as well as the parts that are not clear to me.\n\n* My major concern involves with the core model design. It is not reasonable for me that the method aims to keep two groups of BN parameters at each BN layer, at the same time to let the features generated by the two parameter groups be same by Eq.4. Would this make the BN parameters same? I am wondering how will a simplified model performs, e.g., by dropping all local BN, $\\mathcal{L}_{CE}$ in Eq.6 and the loss in Eq.4. \n* It is not clear to me how the  output features of the two BN layers are fused? Of if they are not fused, the model will cause the increase of computational cost in local clients and some analysis should be provided.\n* From a unified perspective, I am wondering is $\\alpha$ in Eq.7 equivalent to $\\mu$ in Eq.3, only that $\\alpha$ are dynamically generated rather than sampling $\\mu$ from a gaussian distribution. If it is this case, I would suggest to use the same symbol to ease the understanding.\n* The paper presents that \"a zero-shot adapter that is carefully designed to dynamically generate $\\alpha$ for each input in both seen and unseen domains\". Then I have two questions:\n    * The $L_A$ in Eq.8 requires label $y_{i,k}$ for loss computation. However, it is not clear where the labels are obtained for unseen domains since in Sec. 4.1, it mentions that the unseen domain is only used for evaluation, which means that the data are not labeled. \n    * Regarding the specific implementation in Eq.9, a MLP is used for generating $\\delta$ and $\\epsilon$. I am concerning whether the network design is promising. Is it necessary to constraint the range of the two values so as to avoid generating unexpected values (e.g., 10000).\n* The statement \"we use diverse augmented domain from other clients beyond regularization.\" is not clear. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WYmuhn8DqA", "reviewer": "Reviewer_RmwN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to learn a global model in FL from multiple distributed source domains and generalize the model to new clients in unseen domains at inference time. The authors propose client-agnostic learning with mixed instance-global statistics\nfor local training along with  zero-shot adaptation with estimated statistics for inference, which can be viewed as a privacy-preserving multi-source domain generlaization method. To obtain client-invariant representations, the authors augment local features using data distribution of other clients via aggregated BN statistics from the global model.\n\n", "summary_of_the_review": "While the paper has merits mainly w.r.t. experimental results, there are major concerns regarding the problem itself, the proposed method, and some claims in the paper", "strength_and_weaknesses": "Strength: \n\nThe main strength of the paper is the experimental results, which are provided in an organized manner covering a broad range or DG benchmarks and comparing with a sufficiently number of baselines. However, the reviewer has major concerns regarding the problem itself, the proposed method, and some claims in the paper: \n\n\nWeaknesses: \n\nThis paper addresses adaptation to a new completely unseen test domain. The main goal of FL is to help clients who participate in training in terms of obtaining a global model, which addresses their own test error on their own test distributions. Otherwise, why should a client participate in training, which is aimed to optimize a model for a completely different test domain? \n\nThe introduction of zero-shot adapter for example increases computational costs for local clients while it does not directly benefit their own test error. I am not convinced that a client is willing to participate in such collaborative learning. For example, consider cross-silo FL where multiple hospitals plan to learn a model collaboratively. Their main motivation is to obtain a model that helps their own patients at the inference time. \n\n-----\n\nAnother major issue is that what if the model used to train does not have batch normalization layers. It may be a simple CNN. In particular, batch nomlization has been shown to suffer from 1) increasing  computational primitive, which incurs memory overhead 2) introducing hidden hyper-parameters that have to be tuned 3) breaking the independence between training examples in the minibatch (Brock ICLR21 and ICML21). \n\nAndy Brock, Soham De, Samuel L. Smith, and Karen Simonyan. \"High-performance large-scale image recognition without normalization.\" In International Conference on Machine Learning (ICML) 2021.\n\nAndy Brock, Soham De, and Samuel L. Smith. \"Characterizing signal propagation to close the performance gap in unnormalized ResNets.\" In International Conference on Learning Representations (ICLR) 2021.\n\n-----\n\nThe authors assumed that client $k$ has a single domain data. What if client $k$ has different domains across training/test time, which is a natural scenario.  \n\n-----\n\n\"local models are easily over-fitted to their domains\"\n\nThis happens only when a model is trained locally for several iterations. If the model is aggregated frequently, this will not happen in standard FL training. \n\nThe authors proposed to locally train $\\{\\theta_k^a,\\theta_k^s,\\phi_k\\}$ for long epochs. There are several work in distributed learning, which show that local updates are not quite useful both experimentally and theoretically (Karimireddy ICML20). It will be nice if the authors explain why local training for long epochs is useful in this problem? \n\nSai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. \"Scaffold: Stochastic controlled averaging for federated learning.\" In International Conference on Machine Learning (ICML) 2020.\n\n-----\n\nFor MixIG: clients still need to share BN statistics to be averaged so the claim that this method has the same privacy guarantees as FedAvg is not accurate. Indeed, in addition to sharing $\\theta_k,\\phi_k$, sharing those statistics leads to additional privacy leakage.  \n\n-----\n\nThe interpolation mean vector is drawn uniformly at random  from $(0,1)$. Why did the authors choose this specific distribution? It seems to me that these weights should be considered as additional hyperparameters. Another major issue is that whether those weights are client-specific or shared among clients? Based on current notation, it seems that all clients use the same weights for BN layer $l$. \n\nThis paper has two additional hyperparameters $\\lambda_1$ and $\\lambda_2$., which makes the solution less practical given the challenges of tuning these parameters. \n\n-----\n\nMinor comments: \n\nTypo in Section I: \"Af inference time\"\n\nThe first phrase in the second sentence of paragraph \"Challenges:\" is incomplete. \n\nTypo in the first sentence of paragraph: \"Mixing Instance and Global Statistics (MixIG):\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SsZJqFP4wwA", "reviewer": "Reviewer_p4rT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work targets federated domain generalization.  The solution is based on FedBN, which disentangles local model parameters into two groups: domain-specific and domain-invariant. Specifically, the authors designed two strategies: (1) client-agnostic learning with mixed instance-global statistics on BN parameters, as well as designed learning loss terms; (2) a zero-shot adaptor to estimate the interpolation parameter. ", "summary_of_the_review": "The ideas and designed solutions are interesting and technically correct. The experiments are extensive. The writing is clear.\nWhat concerns me the most is the technical contribution of this work as it is heavily dependent on FedBN. Please provide more discussion and comparison to highlight your contribution in the rebuttal. I am willing to change my score if I can receive convincing feedback.", "strength_and_weaknesses": "Strength:\n\n-- The designed two strategies are interesting and technically correct (especially the zero-shot adaptor). Most prior works conduct linear interpolation based on a pre-defined and fixed parameter, which is believed sub-optimal. While in this work, the authors designed an adaptive parameter estimation method to estimate the parameter.\n-- The writing is clear and easy to follow. I feel no difficulty understanding the technical details of this work. The experimental results are extensive.\n\nWeakness:\n\n-- The work is heavily dependent on FedBN. The main difference is that author of this work designed an adaptive interpolation parameter estimation method. This jeopardizes the novelty and technical contribution of the whole work.\n-- I am a little conservative about Eq. 4. If Eq. 4 stands, does that mean the u^l in Eq.3 tends to be 1?\n-- The improvement of the designed solutions in Table 5, is not significant on some datasets. For example, on OfficeHome, the CSAC achieves 64.35, and the proposed solution achieves 64.71, which is a marginal improvement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I5-3mxIgH5", "reviewer": "Reviewer_LXCP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Federated domain generalization aims to learn a global model from various distributed source domains and generalize the learned model in completely unseen domains, while keeping the privacy of the source domains. This paper proposes client-agnostic learning with mixed instance-global statistics and zero-shot adaptation with estimated statistics to implement federated domain generalization. This paper augments local features via the global statistics in the global model's batch normalization layer. And local models learn client-invariant representations based on proposed agnostic objectives with the augmented data. A zero-shot adapter is utilized to help the learned global model bridge a domain gap between seen and unseen clients.", "summary_of_the_review": "Overall, if authors can address my concerns of the generalization ability of client-invariant features and the experiment settings, I recommend to accept this paper.  The studied problem is important, and few current works have studied it. And the proposed method shows some novelty.", "strength_and_weaknesses": "\nStrengths\n1. The problem that this paper aims to solve is important for the real-world FL.\n2. The related work section well reviews the federated domain generalization and the problems of directly using conventional domain generalization methods into federated domain generalization. Especially the differences between the multi-source and single-source methods can well motivate the necessity of developing multi-source domain generalization methods with privacy protection.\n3. This paper has considered the privacy problem and the proposed technique does not introduce extra privacy leakage than FedAvg. The motivations of the proposed methods are clear. And the method shows some novelty.\n\nWeaknesses\n1. It seems that the client-invariant features are learned based on mixing local and global statistics. However, the global statistics do not include the un-seen clients. How to ensure the claimed client-invariant features can generalize well on un-seen clients.\n2. The experiment setting is limited. Only four clients are considered. The cross-silo FL may have much more clients.\n3. There is no theoretical analysis of the generalization performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject