FedWon: Triumphing Multi-domain Federated Learning Without Normalization
Authors: Weiming Zhuang
Lingjuan Lyu
Venue: ICLR 2024 poster (2024)
Keywords: federated learning
multi-domain federated learning
Primary area: general machine learning (i.e., none of the above)
TL;DR: Our new approach removes batch normalization layers to address multi-domain federated learning problem and is versatile for both cross-silo and cross-device FL, even with a batch size of 1.

Abstract: Federated learning (FL) enhances data privacy with collaborative in-situ training on decentralized clients. Nevertheless, FL encounters challenges due to non-independent and identically distributed (non-i.i.d) data, leading to potential performance degradation and hindered convergence. While prior studies predominantly addressed the issue of skewed label distribution, our research addresses a crucial yet frequently overlooked problem known as multi-domain FL. In this scenario, clients' data originate from diverse domains with distinct feature distributions, instead of label distributions. To address the multi-domain problem in FL, we propose a novel method called Federated Learning Without Normalizations (FedWon). FedWon draws inspiration from the observation that batch normalization (BN) faces challenges in effectively modeling the statistics of multiple domains, while existing normalization techniques possess their own limitations. In order to address these issues, FedWon eliminates the normalization layers in FL and reparameterizes convolution layers with scaled weight standardization. Through extensive experimentation on five datasets and five models, our comprehensive experimental results demonstrate that FedWon surpasses both FedAvg and the current state-of-the-art method (FedBN) across all experimental setups, achieving notable accuracy improvements of more than 10% in certain domains. Furthermore, FedWon is versatile for both cross-silo and cross-device FL, exhibiting robust domain generalization capability, showcasing strong performance even with a batch size as small as 1, thereby catering to resource-constrained devices. Additionally, FedWon can also effectively tackle the challenge of skewed label distribution.

Published as a conference paper at ICLR 2024
FEDWON: TRIUMPHING MULTI-DOMAIN FEDERATED LEARNING WITHOUT NORMALIZATION
Weiming Zhuang
Lingjuan Lyu
Sony AI
Sony Al 1 lingjuan 1v 1v@sony.com
we weiming. zhuang@ssny.com
ABSTRACT
Federated learning (FL) enhances data privacy with collaborative in-situ train- ing on decentralized clients. Nevertheless, FL encounters challenges due to non- independent and identically distributed (non-i.i.d) data, leading to potential perfor- mance degradation and hindered convergence. While prior studies predominantly addressed the issue of skewed label distribution, our research addresses crucial yet frequently overlooked prohlem known as multi-domain FI.. In this scenario, clients' data originate from diverse domains with distinet feature distributions, instead of label distributions. To address the multi-domain problem in FL, we propose a novel method called Federated learning Without normalizations (Fed- Won). WedWon draws inspiration from the observation that batch normalization (BN) faces challenges in effectively modeling the statistics of multiple domains, while existing normalization techniques possess their own limitations. In order to address these issues, FedWon eliminates the normalization layers in FL and reparameterizes convolution layers with scaled weight standardization. Through extensive experimentation on five datasets and five models, our comprehensive ex- perimental results demonstrate that FedWon surpasses both FedAvg and the cur- rent state-of of-the-art method (FedBN) across all experimental setups, achieving no- table accuracy improvements of more than 10% certain domains. Furthermore, FedWon is versatile for both cross-silo and cross-device FL, exhibiting robust do- main generalization capability, showcasing strong performance even with a batch size small 1, thereby catering to resource-constrained devices. Additionally, FedWon can also effectively tackle the challenge of skewed label distribution.
INTRODUCTION
Federated learning (FL) has emerged as promising method for distributed machine learning, en- abling in-situ model training on decentralized client data. It has been widely adopted in diverse ap- plications, including healthcare (Li et al. 2019 Bernecker al., 2022). mobile devices (Hard al.] 2018 Paulik et al., (021), and autonomous vehicles (Zhang et al. 2021 Nguyen et al., [2021] Posner al. 2021). However, FI. commonly suffers from statistical heterogeneity, where the data distri- butions across clients are non-independent and identically distributed (non-i.i.d) (Li et al. 2020a). This is due the fact that data generated from different clients highly likely have different data distributions, which can cause performance degradation (Zhao et al. 2018 Hsieh et al. 2020 Tan
ct al. 2023) even divergence in training (Zhuang et al. 2020 2022b Wang et al. 2023).
The majority of studies that address the problem of non-i.i.d data focus on the issue of skewed label distribution. where clients have different label distributions (Li et al., [0206]] Hsich et al., 2020 Wang et al. 2020: Chen et al. 2022). However, multi-domain FL, where clients' data are from different domains, has received less attention, despite its practicality in reality. TigureTTaleepinn practical examples of multi-domain FL.. For example, multiple autonomous cars may collaborate on model training, but their data could originate from different weather conditions or times of day, leading to domain gaps in collected images (Cordts et al. 2016 Yu et al. 2020). Similarly, multiple healthcare institutions collaborating on medical imaging analysis may face significant domain gaps due to variations in imaging machines and protocols (Bernecker et al. (022). Developing effective solutions for multi-domain FL is a critical research problem with broad implications.
two
"Corresponding author

---

Published as a conference paper at ICLR 2024
4 BN
Muneing Varaace
Aggregation
Server
Rusring 5ith BN Layer
0
Duta
Training
Training
Dass
Training
Martnn g Vorance
Clert
Clere
Autonomaus Driving
Client Sunny
Client 2: Night
Client K: Snowy
Rusning Mean
(a) Illustration of Multi-domain FL.
(b) Statistics of BN Layers.
Figure/ 1: (a) We consider multi-domain federated learning, where each client contains data of one domain. This setting is highly practical and applicable in real-world scenarios. For example, au- tonomous cars in distinct locations capture images in varying weather conditions. (b) Visualization of batch normalization (BN) channel-wise statistics from two clients, each with data from a single domain. The upper and lower figures are results from the 4-th and 5-th BN layers of 6-layer CNN, respectively. It highlights different feature statistics of BN layers trained on different domains.
However, the existing solutions are unable to adequately address the problem of multi-domain FL. FedBN (Li al. 202I) attempts to solve this problem by kceping batch normalization (BN) param- eters and statistics (loffe & Szegedy/ (015) locally in the client, but is only suitable for cross-silo FL (Kairouz et al., 2021). where clients are organizations like healthcare institutions, because it requires clients to be stateful (Karimireddy et al. 2020)) (keeping states of BN information) and participate training in every round. As result, FedBN is not suitable for cross-device FL, where the clients are stateless and only fraction of clients participate in training. Besides, BN relies on the assumption that training data are from the same distribution, ensuring the mean and variance of each mini-batch are representative of the entire data distribution (Toffe & Szegedy 2015). Figure Tb shows that the running means and variances of BNs differ significantly between two FL. clients from different domains, well as between the server and clients (statistics of all BN layers are in
Figure [2] in Appendix). Alternative normalizations like Layer Norm (Ba et al. 2016 and Group Norm (W & He. 2018) have not been studied for multi-domain FL, but they have limitations like requiring extra computation in inference.
This paper explores fundamentally different approach to address multi-domain FL. Given that BN struggles to capture multi-domain data and other normalizations come with their own limitations, further ask the question: is normalization indispensable to learning general global model for multi-domain FL.? In recent studies, normalization-fre ResNets (Brock et al., (2021a) demonstrates comparable performance standard ResNets et al. (0166). Inspired by these findings, we build upon this methodology and explore its untapped potential within the realm of multi-domain FL.
We introduce Federated learning Without normalizations (FedWon) to address the domain dis- crepancies among clients in multi-domain FL. FedWon follows FedAvg (McMahan et al. 2017) protocols for server aggregation and client training. Unlike existing methods, FedWon removes normalization layers (c.g., BN layers), and reparameterizes convolution layers with Scaled Weight Standardization (Brock et al. 2021a). We conduct extensive experiments on five datasets using five models. The experimental results indicate that FedWon outperforms state-of-the-arr methods on all datasets and models. The general global model trained by FedWon can achieve more than 10% im- provement certain domains compared to the personalized models from FedBN (Li et al. 2021) Moreover, our empirical evaluation demonstrated three key benefits of FedWon: 1) FedWon is ver- satile to support both cross-silo and cross device FL; 2) FedWon achieves competitive performance on small batch sizes (even on a batch size of 1), which is particularly useful for resource-construingg devices; 3) FedWon can also be applied address the skewed label distribution problem.
2

---

Published as a conference paper at ICLR 2024
Conv,
Conv
Conv:
Conv
BN,
BN
BN,
BN,
Conv
Conv
WSGonw;
WSGonv
Client 1
Client 1
Client
Client
Client
Client
Conv
BN
Conv
Conva
Conv
WSConv
BN
BN
BN;
BN2
WSConv
WSConv
Server Aggregation
Server Aggregation
Server Aggregation
Client 2
Client 2
Client 2
Client
(a) FedAvg
(b) FedBN
(c) Fed (Ours)
Figure 2: Illustration of three algorithms: (a) FedAvg aggregates both convolution (Conv) layers and batch normalization (BN) layers in the server, (b) FedBN keeps BN layers clients and only aggregates Conv layers; (c) Our proposed Federated learning Without normalizations (FedWon) re- moves all BN layers and reparameterizes Conv layers with scaled weight standardization (WSConv).
In summary, our contributions are as follows:
We introduce FedWon, simple yet effective method for multi-domain FL. By removing all normalization layers and using scaled weight standardization, FedWon able learn general global model from clients with significant domain discrepancies.
To the best of our knowledge, FedWon is the first method that enables both cross-silo and cross-device FL without relying on any form of normalization. Our study also reveals the unexplored benefits of this method, particularly in the context of multi-doman FL.
Extensive experiments demonstrate that FedWon outperforms trate-ofthhe-ee methods on all the evaluated datasets and models, and suitable for training with small batch sizes, which is especially beneficial for cross-device FL in practice.
PRELIMINARY
Before diving into the benefits brought by removing normalizations, we first introduce FL with batch normalization. Then, we review alternative normalization methods and normalization-fii networks.
2. FEDERATED LEARNING WITH BATCH NORMALIZATION
Batch normalization (BN) Ioffe & Szegedy 2015), commonly used as a normalization layer, has been a fundamental component in deep neural networks (DNN). The BN operation is defined as:
BN(c) = 2
where and variance computed from mini-batch of data, and y and 9 learnable parameters. The term is small positive value that is added for numerical stability.
BN offers several benefits, including reducing internal covariate shift, stabilizing training, and celerating convergence (Ioffe & Szegedy) 2015). Moreover, it is more robust to hyperparameters (Bjorck al. 2018) and has smoother optimization landscapes (Santurkar al. 2018). However, its effectiveness is based on the assumption that the training data is from the same domain, such that the mean μ and variance computed from mini-batch are representative & Szegedy 2015). In centralized training, BN has been found to struggle with modeling the statis- ties from multiple domains, leading to the development of domain-specific BN techniques (Li al.) 2016 Chang et al.) 2019). Similarly, in multi-domain FL. DNN with BN can encounter difficulties in capturing the statistics of multiple domains while training a single global model.
the training data (Toffe)
Federated learning (FL) trains machine learning models collaboratively from decentralized clients, coordinated by a central server (Kairouz et al. (021)) It enhances data privacy by keeping the data locally clients. FedAvg (McMahan et al. 2017) is the most popular FL algorithm. A common issue in FL non-i.i.d data across clients, which could lead to performance degradation
and difficulties in convergence (Hsieh al. 2020 Zhuang al. 2021: 2022c Wang et al. 2023
3

---

Published as a conference paper at ICLR 2024
Skewed label distribution, where clients have different label distributions, is widely discussed non- i..dd problem with numerous proposed solutions (Li et al. 2020b; Zhang et al. 2023 Chen et al,
2022
To address this problem, multiple works provide solutions that introduce special operations
on BN to personalize a model for each client (Lu et al. 2022). For example, SiloBN (Andreux et al.)
2020 keeps BN statistics locally in clients. FixBN (Zhong et al. 2023) only trains BN statistics in
the first stage of training and freezes them thereafter. FedTAN (Wang et al., 2023) tailors for BN by performing iterative layer-wise aggregations, introducing numerous extra communication rounds.
In contrast, multi-domain FL, where the data domains differ across clients, has received less atten- tion (Chen et al., 2018 Shen et al., 2022). FedBN (Li et al., 2021) and FedNorm (Bemnekkr al. 2022) addresses this issue by locally keeping the BN layers in clients and aggregating only the other model parameters. PartialFed (Sun et al., 202T) keeps model initialization strategies in clients and use them to load models in new training rounds. While these methods excel in cross-silo FI., where clients are stable and can retain statefulness, they are unsuitable for cross-device FL by design. In the latter scenario, clients are stateless, and only a fraction of clients participate in each round of training (Kirrouz et al. 202T). Besides, FMTDA (Yao et al. 2022) adapts source domain data in the server target domains in clients, whereas we do not assume availablility of data in the server.
2.2 ALTERNATIVE NORMALIZATION METHODS
BN has shown to be effective in modern DNNs (Toffe & Szegedy 2015), but it also has limitations in various scenarios. For example, BN struggles to model statisties of training data from multiple domains (Li et al. 2016 Chang et al. 2019). and it may not be suitable for small batch sizes (Toffe)
2017 Wu & He| (018). Researchers have proposed alternative normalizations such as Group Norm (uu & He 2018)) and Layer Norm (Ba et al. 2016). Although these methods remove some of the constraints of BN, they come with their own limitations. For example, they require additional computation during inference, making them less practical for edge deployment.
Recent studies have shown that BN may not work well in FL. under non-i.id data (Hsieh et al, 2020), due to external covariate shift (Du et al.] 2022) and mismatch between local and global statistics (Wang et al. 2023). Instead, researchers have adopted alternative normalizations such as GN (Hsieh et al., 2020 Casella et al. 2023) or LN (Du et al. 2022 Casella al. (023) to mitigate the problem. However, these methods inherit the limitations of GN and LN in centralized training. and the recent study by Zhong et al. (2023) shows that BN and GN have no consistent winner in FL.
2.3 NORMALIZATION-FREE NETWORKS
Several attempts have been made to remove normalization from DNNs in centralized training us- ing weight initialization methods (Hanin & Rolnick 2018 Zhang al. 2019 De & Smith 2020). Recently, Brock et al. (2021a) proposed a normalization- free network by analyzing the signal propa- gation through the forward pass of the network. Normalization-fre network stabilizes training with scaled weight standardization that reparameterizes the convolution layer to prevent the mean shift in the hidden activations (Brock et al., 2021ap. This approach achieves competitive performance compared to networks with BN on ResNet (He et al. (016)) and EfficientNet (Tan & Le 2019). Building on this work, Brock et al. further introduced an adaptive gradient clipping (AGC) method that enables training normalization-fee networks with large batch sizes (Brock et al. 2021b).
3 FEDERATED LEARNING WITHOUT NORMAIIZATION
In this section, we present the problem setup of multi-domain FL and propose FL without normal- ization to address the problem of multi-domain FL.
3.1 PROBLEM SETUP
The standard federated learning aims to train a model with parameters 0 collaboratively from total N N decentralized clients. The goal is to optimize the following problem:
K
K
min f(0) = Σ (1) = Σ Pr DD (f BeR!
(2)
k=1
4

---

Published as a conference paper at ICLR 2024
where K E N is the number of participated clients (K 3 N), (0) is the loss function of client k, Pk: is the weight for model aggregation in the server, and & is the data sampled from distribution Dr of client k. FedAvg (McMahan et al. 2017) sets p. be proportional to the data size of client k. Each client trains for E E N local epochs before communicating with the server.
Assume there are N clients in FL and each client & contains n4g N data samples {(r) vt ))) Skewed label distribution refers to the scenario where data in clients have different label distribu- tions, i.e. the marginal distributions PM(y) may differ across clients (Pu(y) de (v) for differ- ent clients & and k'), In contrast, this work focuses on multi-domain FL, where clients possess data from various domains, and data samples within client belong the same domain (Kairouz et al. 2021: Li et al.) 2021). Specifically, the marginal distribution (Px(2) ~ (ar) for different clients and k'). Within each client, the data samples, represented as and Tj., drawn from the same marginal distribution Pa(ar)) holds that PA (x:) ~ 'P (xz) for all i.j E 1,2,.... 72.- Figure Ta illustrates practical examples of multi-domain FL.. For example, autonomous cars in different locations could capture images under different weather conditions.
(x(x) may vary across clients
3.2 NORMALIZATION-FREEE FEDERATED LEARNING
Figure lb demonstrates that the BN statistics of clients with data from distinct domains siderably dissimilar in multi-domain FL. Although various existing approaches have attempted to address this challenge by manipulating
replacing the BN layers with other normalization layers (Li et al. 2021] Du et al. 2022 Zhong et al. 2023), they come with their own set of limitations, such as additional computation cost during inference. To bridge this gap, we propose a novel approach called Federated learning Without normalizations (FedWon)that removes all normalization layers in FL.
However, simply removing all normalization layers would lead to deteri- orated performance FL. Figure3]comper the performance of training in single dataset (SingleSet) and FedAvg without normalization on four domains of the Sefie-Caltech-111 dataset (Further details on the experi- mental setup are provided in Section 4 FedAvg without (w/o) BN yields inferior results compared to SingleSet w/o BN. The domain gaps among clients could amplify the challenges in FL when training without BNs.
SingltSct
Testing Accuracy (9)
Festfing w/o IN
50
Compared with FedAvg (McMahan et al. 2017), ,, our proposed FedWon completely removes the normalization layers in DNNs and further repa- rameterizes the convolutions layer. We employ the Scaled Weight Stan- dardization technique proposed by Brock et al. (202 to reparameterize the convolution layers after removing BN. The reparameterization formula can be expressed as follows:
Arsonn Calech WebCary O51R Offie-Caltech-10 Datasets
Figure 3: FedAvg with- out (w/o) BN yields in- ferior results.
Wi. 14
W.J
(3)
where Wi, is the weight matrix of convolution layer with i as the output channel and j as the input channel, 7 is a constant number, N the fan-in convolution layer, #4 = (1/N) Σ, Wi.; and of = (1/N) E, - p:) are the mean and variance of the i-th row of Wi,. respectively. By removing normalization layers, FedWon eliminates batch dependency, resolves discrepancies between training and inference, and does not require computation for normalization statistics in inference. We term this parameterized convolution as WSConv.
Figure 2 highlights the algorithmic differences between our proposed FedWon and the other two FL algorithms: FedAvg (McMahan et al.) 2017) and FedBN (Li et al. 2021). FedAvg aggregates both convolution and BN layers on the server; FedBN only aggregates the convolution layers and keeps BN layers locally in clients. Unlike these two methods, FedWon removes BN layers, replaces convolution layers with WSConv, and only aggregates these reparameterized convolution layers. Prior work theoretically shows that BN slows down and biases the FL convergence (Wang et al.) 2023). FedWon circumvents these issues by removing BN while preserving the convergence speed that BN typically facilitates. Furthermore, FedWon offers unexplored benefits multi-domain FL, including versatility for both cross-silo and cross-device FL, enhanced domain generalization, and compelling performance small batch sizes, including a batch size small 1.
5

---

Published as a conference paper at ICLR 2024
Table I: Testing accuracy (%) comparison of different methods on three datasets. Our proposed
FedWon outperforms existing methods in most of the domains. FedWon achieves the best average testing accuracy in all datasets.
Domains
SingleSet
FedAvg
FedProx
+GN
+LN
SiloBN
FixBN
FedBN
Ours
MNIST
94.4
96.2
96.4
96.4
96,4
96.2
96.3
96.5
96.8
Digit-Five
SVHN
67.1
71.6
71.0
76.9
75.2
71.3
71.3
77.3
77.4
USPS
95.4
96.3
96.1
96.6
96,4
96.0
96.1
96.9
97.0
SinthDigits
80.3
86.0
85.9
86.6
85.6
86.0
85.8
86.8
87.6
MNIST-M
77.0
82.5
83.1
83.7
82.2
83.1
83.0
84.6
84.0
Average
83.1
86.5
86.5
88.0
87.1
86.5
86.5
88.4
88.5
Amazon
54.5
61.8
59.9
60.8
55.0
60.8
59.2
67.2
67.0
Caltech-10
Caltech
40.2
44.9
44.0
50.8
41.3
44.4
44.0
45.3
50.4
DSLR
81.3
77.1
76.0
88.5
79.2
76.0
79.2
85.4
95.3
Webcam
89.3
81.4
80.8
83.6
71.8
81.9
79.6
87.5
90.7
Average
66.3
66.3
65.2
70.9
61.8
65.8
65.5
71.4
75.6
Clipart
42.7
48.9
51.1
45.4
42.7
51.8
49.2
49.9
57.2
Infograph
24.0
26.5
24.1
21.1
23.6
25.0
24.5
28.1
28.1
DomainNer
Painting
34.2
37.7
37.3
35.4
35.3
36.4
38.2
40.4
43.7
Quickdraw
71.6
44.5
46.1
57.2
46.0
45.9
46.3
69.0
69.2
Real
51.2
46.8
45.5
50.7
43.9
47.7
46.2
55.2
56.5
Sketch
33.5
35.7
37.5
36.5
28.9
38.0
37.4
38.2
51.9
Average
42.9
40.0
40.2
41.1
36.7
40.8
40.3
46.8
51.1
"+GN means FedAvg+GN, "+LN means FedAvg+LN
4 EXPERIMENTS ON MULTI-DOMAIN FL
In this section, we start by introducing the experimental setup for multi domlinn FL. We then validate that FedWon outperforms existing methods in both cross-silo and cross-device and achieves comparable performance even with batch size of 1. We end by providing ablation studies.
4.1 EXPERIMENT SETUP
conduct experiments for multi-domain FI. using three datasets: Digits-Five (Li al., 2021). Office-Caltech110 (Gong et al., 2012). and DomainNet (Peng et al., 2019). Digits-Five con- sists of five sets of 28x28 digit images, including MNIST (LeCun et al. 1998), SVHN (Netzer et al.,
Datasets.
2011, USPS (Hull, 1994), SynthDigits (Ganin Lempitsky 2013), MNIST-M (Ganin & Lempit- sky 2015); each digit dataset represents a domain. Offic-CClteche Caltech-10 consists of real-world object
images from four domains: three domains (WebCam, DSLR, and Amazon) from Office-31 dataset
(Seenko et al., 2010) and one domain (Caltech) from Caltech-256 dataset (Griffin et al. (007). Do- mainNet (Peng et al., 2019) contains large-sized 244x244 object images in six domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. To mimic the realistic scenarios where clients may not collect a large volume of data, we use subset of standard digits datasets (7,438 training samples for each dataset instead of tens of thousands) adopted in Li al. (202T). We evenly split samples of each dataset into 20 clients for cross-device FL with total 100 clients. Similarly, tailor the DomainNet dataset to include only 10 classes of 2,000-5,000 images. To simulate multi-domain FL, we construct a client to contain images from a single domain.
Implementation Details. We implement FedWon using PyTorch (Paszke et al., 2017) and run experiments cluster of eight NVIDIA T4 GPUs. We evaluate the algorithms using three archi- tectures: 6-layer convolution neural network (CNN) (Li al. 2021) I for Digits-Five dataset, AlexNet Krizhevsky et al. 2017) and Re (He et al. 2016) for Oflice-Caltech-- 10 dataset, and AlexNet Krizhevsky et al. 2017) for DomainNet dataset. We use cross-entropy loss and stochastic gradient optimization (SGD) the optimizer with learning rates tuned over the range of [0.001, 0.1] for all methods. Based on SGD, we adopt adaptive gradient clipping (AGC) that is specially designed for normalization-free networks (Brock et al. 2021b). More details provided in the supplementary.

---

Published as a conference paper at ICLR 2024
FedAvg (B-2)
FedWon (B=2)
FedWon (B=1)
100
Testing Accuracy (%)
80-
Texting Accuracy (%)
05-
60-
90
MNIST
SVHN
85
USPS
FedWon
SyncDigits
20
FedAvg
MNISTM
0
80 90 100 Training Rounds
30 40 50
70
MNIST
SVHN
USPS
Synchetic
MNIST-M
Digits-Five Datasets
Figure 4: Testing accuracy comparison of FedWon and FedAvg Digits-Five dataset. Left: com- parison of performance using small batch sizes =1,, 2}, where 10 out 100 clients are randomly selected to train in each round. Right: comparison of testing accuracy the course of training with randomly selected 10 out of total of 100 clients and batch size B
4.2 PERFORMANCE EVALUATION
We compare the performance of our proposed FedWon with the following three types of methods: (1) state-of-the-art methods that employ customized approaches BN, including SiloBN (Andreux et al., 2020), FedBN (Li et al. 2021), and FixBN (Zhong et al. (023); (2) baseline algorithms, including FedProx (Li al. 2020b), FedAvg (McMahan et al. 2017), and SingleSet (i.e. training a model independently in each client with a single dataset); (3) alternative normalization methods,
including Fe and FedAvg+L.N that replace BN layers with GN and LN layers, respectively.
Table presents a comprehensive comparison of the aforementioned methods under cross-silo FL. 10, and DomainNet datasets. Our proposed FedWon outperforms the
on Digits-Five, state-of-the-art methods on most of the domains across all datasets. Specifically, FedProx, which adds proximal term based on FedAvg, performs similarly to FedAvg. These two methods better than SingleSet in Digits-Five Five dataset, but they may exhibit inferior performance compared to SingleSet in certain domains on the other two more challenging datasets. SiloBN and FixBN perform similarly to FedAvg, in terms of average accuracy; they are not primarily designed for multi-domain FL. and are only capable of achieving the baseline results. In contrast, FedBN specifically designed to excel in multi-domain FI. and outperforms these methods.
Besides, we discover that simply replacing BN with GN (FedAvg+GN) can boost the perfor- mance of FedAvg as GN does not depend on the batch statistics specific to domains; FedAvg+GN achieves comparable results as FedBN on Digits- Five and proposed FedWon surpasses both FedAvg+GN and FedBN in terms of the average accuracy on all datasets. Although FedWon falls slightly behind FedBN less than 1% two domains across these datasets, it outperforms FedBN by more than 17% on certain domains. These results demon- strate the effectiveness of FedWon under the cross- silo FI. scenario. We report the mean of three runs of experiments here and results with standard de- viation in Table 22 in the Appendix.
Table 2: Performance comparison using small batch sizes B = {1,2] on Office-Caltech- 10 dataset. The abbreviations A, C, D, and W spectively represent domains: Amazon, Cal- tech, DSLR, and WebCam. Our proposed Fed- Won achieves outstanding performance com- pared to existing methods.
datasets. Notably,
Methods
A
C
D
W
FedAvg+GN
60.4
52.0
87.5
84.8
FedAvg+LN
55.7
43.1
84.4
88.1
FedWon
66.7
55.1
96.9
89.8
FedAvg
64.1
49.3
87.5
89.8
Ge
63.5
52.0
81.3
84.8
FedAvg+LN
58.3
44.9
87.5
86.4
2
FixBN
66.2
50.7
87.5
88.1
Effectiveness on Small Batch Size. T pares the performance of our proposed FedWon with ofte--of-the-ar methods using small batch sizes = {1,2] on Effice-Caltech- dataset.
SiloBN
61.5
47.1
87.5
86.4
FedBN
59.4
48.0
96.9
86.4
Fed
66.2
54.7
93.8
89.8
FedWon achieves outstanding performance, with competitive results batch size of 1. While

---

Published as a conference paper at ICLR 2024
Table Testing accuracy comparison on randomly selecting fraction C {0.1, 0.2) out of total of 100 clients for training each round with batch size B 4 on Digits-Five dataset. FedWon consis- tentl outperforms FedAvg. We report the mean (standard deviation) of three runs of experiments.
Method
MNIST
SVHN
USPS
SinthDigits
MNIST-M
Average
FedAvg
98.2 (0.4)
81.0 (0.7) 97.2 (0.5)
91.6 (1.6)
89.3 (0.5)
91.5 (0.8)
0.1
FedWon (Ours)
98.6 (0.1)
85.4 (0.3)
98.3 (0.2)
93.6 (0.2)
90.5 (0.3)
93.3 (0.1)
FedAvg
97.9 (0.1)
80.2 (0.0) 97.0 (0.1)
91.2 (0.0)
89.3 (0.0)
91.1 (0.0)
0.2
Wen (Ours)
98.7 (0.1)
86.0 (0.3) 98.2 (0.2)
94.1 (0.2)
90.8 (0.1)
93.6 (0.1)
FedAvg+GN and FedAvg+LN also achieve comparable results batch size B they require additional computational cost during inference to calculate the running mean and variance, whereas our method does have such constraints and achieves even better performance. The capability of our method to perform well under small batch sizes is particularly important for cross cross-device FL, as some edge devices may only be capable of training with small batch sizes under constrained resources. We have fine-tuned the learning rates for all methods and reported the best ones.
Cross-device FL with Small Batch Size and Client Se- lection. We assess the impact of randomly selecting a fraction clients to participate in training in each round, which is common in cross-device FL where not all clients join in training. We conduct experiments with fraction C' = (0.1, 0.2) out of 100 clients Digits-Five Five dataset, ie., K = {10, 20} clients are selected to participate in training in each round. Table 3 shows that the perfor- mance of our FedWon is better than FedAvg under all client fractions. FedBN is not compared as not ap- plicable in cross-device FL. We also evaluate small batch sizes in cross-device FL. with K 10 clients selected per round. Figure E (left) shows that the performance of FedAvg degrades with batch size B 2, while our pro- posed FedWon with batch sizes B (1, achieves con- sistently comparable results to running with larger batch sizes. Besides, Figure 4 (right) shows the changes in test- ing accuracy over the course of training. It indicates that FedWon achieves better convergence speed without BN.
Clirnt
Client
Server
Image
Local Model Local Model Global Model
FedAug
BodAvg wio BN
FedAug
FedWon FedBN
Methods
C+C
S++C
FedAvg w/o BN FedAvg
0.811
0.873
0.865
0.869
FedBN
0.901
0.929
Visualization and Analysis of Feature Maps. We aim to further study the reason behind the superior performance of FedWon. Figure 5 (top) visualizes feature maps of the last convolution layer of client local models and one server global model on the Office-Caltech-111 10 dataset. The feature maps of FedAvg without (w/o) BN have a limited focus on the object of interest. While FedAvg and FedBN perform better, their feature maps display notice- able disparities between client local models.
FedWon (Ours)
0.993
0.997
Figure 5: Analysis of feature maps with the Caltech- 10 dataset. Top: visualiza- tion of feature maps of the last convolu- tion layer. Bottom: comparison erage cosine similarity of feature maps between client (C+C), and between client and server (S++C).
In contrast, FedWon showcases superior feature map visualizations, with subtle differences observed among feature maps from different models. To provide further insight, we present the average cosine similarity of all feature maps between client local models (C++C) and between the server global model and client local model (S++C) in Figure (bottom). These results demonstrate the effectiveness of FedWon, which achieves high similarity scores, approaching the maximum value of 1. This finding suggests that FedWon excels at effectively mitigating domain shifts across different domains. Building upon these insights, we extend our analysis to demonstrate that FedWon exhibits superior domain adaptation and generalization capabilities empirically in Table 12 in the Appendix.
We also demonstrate that Fed Won achieves significantly superior performance on medical diagnosis in Appendix B.I which is encouraging and shows the potential of FedWon in the healthcare field.
8

---

Published as a conference paper at ICLR 2024
4.3 ABLATION STUDIES
We conduct ablation studies to further analyze the impact of WSConv batch sizes B 32 dataset.
Table 4: Ablation studies on the impact of WS- Conv Caltech- 10 dataset. It significantly boosts
and B 2 on the Office-Caltech: Table compares the performance with and without WSCCnv after removing all normal- ization layers. It demonstrates that replacing convolution layers with WSConv significantly enhances performance. These experiments use a learning rate of 0.01 for = 2. We provide more exper- iment details in the Appendix
performance on both batch sizes 32 and 2.
WSConv
C
W
63.7
51.0
96.3
91.2
32
46.4
37.3
68.8
71.2
0.08 for B 32 and
67.2
55.6
96.9
93.2
54.7
44.0
84.4
78.0
EXPERIMENTS ON SKEWED LABEL DISTRIBUTION
This section extends evaluation from multi-domain FL to skewed label distribution. We demonstrate that our proposed FedWon also effective addressing this problem.
Dataset and Implementation. We simulate skewed label distribution using CIFAR-10 dataset (Krizhevsky et al., 2009). which comprises 50,000 training samples and 10,000 testing samples. We split training samples into 100 clients and construct i.i.d data and three different levels of label skewness using Dirichlet process Dir(a) with
{0.1, 0.5, 1}, where Dir(0.1) 1) is the most het- erogeneous setting. We run experiments using MobileNetV2 (Sandler et al., 2018) with fraction C' = 0.1 randomly selected clients (i.e., K = 10) out of a total of 100 clients in each round.
Methods i..d
Dir (1)
Dir (0.5)
Dir (0.1)
Testing Accurayy (6)
FedAvg
75.0
64.5
61.1
36.0
50
FedAvg+GN
65.3
58.8
51.8
21.5
40-
FedAvg+LN
69.2
61.8
57.9
23.3
FixBN
75.4
64.1
61.2
34.7
30
Fodfing
FedWon (Ours)
75.7
72.8
70.7
41.9
20
FesAug FeeNN
FedAigg UN
10
50
100
150
2000
250
300
Training Rounds
Figure 6: Testing accuracy comparison using MobileNetV2 as backbone on CIFAR-10 dataset. Left: performance on different levels of label skewness, where Dir (0.1) represents the most skewed label distribution setting. Right: changes in testing accuracy over the course of training on Dir (0.5).
Performance Comparison. Figure 6 (left) compares FedWon with FedAvg, FedAvg+GN, Fe- dAvg+LN, and FixBN. FedWon achieves similar performance FedAvg and FixBN the i.i.d setting, but outperforms methods across different degrees of label skewness. We do not compare with FedBN and SiloBN they are suitable for cross-device FL and provide the comparison of cross-silo FL scenario in Table 15 in the Appendix. Figure (right) shows changes in testing accu- racy over the course of training under the Dir (0.5) setting. FedWon converges a better position than the other methods. These experiments indicate the possibility of employing our proposed FL without normalization to solve the skewed label distribution problem.
CONCLUSION
In conclusion, we propose FedWon, a new method for multi-domain FL by removing BN layers from DNNs and reparameterizing convolution layers with weight scaled convolution. Extensive experiments across four datasets and models demonstrate that this simple yet effective method out- performs slate-of-the-art methods in wide range of settings, Notably, FedWon is versatile for both cross-silo and cross-device FL. Its ability to train on small batch sizes is particularly useful for resource-cones constreecnd devices. Future work conduct evaluations of this method under a broader range of datasets and backbones for skewed label distribution. Extending this paradigm from super- vised semi-supervised and unsupervised scenarios is also of interest.

---

Published as a conference paper at ICLR 2024
REFERENCES
Mathieu Andreux, Jean Ogier du Terrail, Constance Beguier, and Eric W. Tramel. Siloed federated learning for multi-centric histopathology datasets, In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, pp. 129-139. Springer International Pub- lishing, 2020. ISBN 978-3-030-60548-3.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Tobias Bernecker, Annette Peters, Christopher L Schlett, Fabian Bamberg, Fabian Theis, Daniel Rueckert, Jakob Weiß, and Shadi Albarqouni. Fednorm: Modality- normalization in feder- ated learning for multi-moda liver segmentation. arXiv preprint arXiv:2205.110%6 2022.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. Advances in neural information processing systems, 31, 2018.
Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. International Conference on Learning Representations, 2021a.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. International Conference on Machine Learning, pp- 1059-10711 PMLR, 2021b.
Bruno Casella, Roberto Esposito, Antonio Sciarappa, Carlo Cavazzoni, and Marco Aldinucci. Ex- perimenting with normalization layers in federated learning on non-iid scenarios. arXiv preprint 1rXiv:2303310630, 2023.
Woong-Gi Chang, Tackgeun You, Sconguk Sco. Suha Kwak, and Bohyung Han. Domain-specine batch normalization for unsupervised domain adaptation. Proceedings of the IEEE/CVF con- ference Computer Vision and Pattern Recognition, pp. 7352-7362, 2019.
Chen Chen, Yuchen Liu, Xingjun Ma, and Lingjuan Lyu. Calfat: Calibrated federated adversarial training with label skewness. Advances in Neural Information Processing Systems, 2022.
Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods training deep neural networks. arXiv preprint 10X,::/8006.6763, 2018.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion anal- ysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp. 168-172. IEEE, 2018
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Bcn20000: Dermo- scopic lesions in the wild. arXiv preprint arXiv: 02288, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 13:19964-19975. 2020.
Zhixu Du, Jingwei Sun, Ang Li, Pin-Yu Chen, Jianyi Zhang, Hai" Helen" Li, and Yiran Chen. Rethinking normalization methods in federated learning. In Proceedings of the 3rd International Workshop on Distributed Machine Learning, pp. 16-22, 2022.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pp. 1180-11888 PMLR, 2015.
10

---

Published as a conference paper at ICLR 2024
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kemel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp- 2066-2073. IEEE, 2012.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile 2018.
keyboard prediction. arXiv preprint
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387- 4398. PMLR, 2020.
Jonathan J. Hull. A database for handwritten text recognition research. JEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in Batch-normalized models. Advances in neural information processing systems, 30, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456, pmlr, 2015.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Ad- vances and open problems in federated learning, Foundations and Trends in Machine Learning, 14(1-2):1 210, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sehastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143, PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever. and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84 90, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE,
1998.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37:50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 4500, 2020b.
Wenqi Li, Fausto Milletari, Daguang Xu, Nicola Rieke, Jonny Hancox, Wentao Zhu, Maximilian Baust, Yan Cheng, Sébastien Ourselin, M Jorge Cardoso, et. Privacy-preserving federated brain tumour segmentation. In International Workshop on Machine Learning in Medical Imaging, pp. 133-141. Springer, 2019.
Xiaoxiao Li, Mcirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. arXiv preprint 102iv:2/02.076233 2021.
11

---

Published as a conference paper at ICLR 2024
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normaliza- tion for practical domain adaptation. arXiv preprint TrXiv:1603.04777 2016.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Wang Lu, Jindong Wang, Yiqiang Chen, Xin Qin, Renjun Xu, Dimitrios Dimitriadis, and Tao Qin. Personalized federated learning with adaptive batchnorm for healtheare. IEEE Transactions on Big Data, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficiee. learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Anh Nguyen, Tuong Do, Minh Tran, Binh Nguyen, Chien Duong, Tu Phan, Erman Tjipu- tra, and Quang D Tran. Deep federated learning for autonomous driving. arXiv preprint 1rXiv:2110.05754, 2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier van Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde, et al. Federated evalua- tion and tuning for on-device personalization: System design & applications. arXiv preprint arXiv:2102.08503 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415 1415, 2019.
Jason Posner, Lewis Tseng, Moayad Aloqaily, and Yaser Jararweh. Federated learning in vehicular networks: opportunities and solutions. IEEE Network, 35(2):152-159, 2021.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Eision-ECCT 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp. 213-226. Springer, 2010.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in neural information processing systems, 31, 2018.
Yiqing Shen, Yuyin Zhou, and Lequan Yu. Cd2-pfed: Cyclic distillation-guidee channel decoupling for model personalization in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10041-10050, 2022.
Benyuan Sun, Hongxing Huo, Yi Yang, and Bai. Partialfed: Cross-domain personalized feder- ated learning via partial initialization. Advances in Neural Information Processing Systems, 34: 23309-23320, 2021.
Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference machine learning, pp. 1105-6114. PMLR, 2019.
Yue Tan, Chen Chen, Weiming Zhuang, Xin Dong, Lingjuan Lyu, and Guodong Long. Is het- crogeneity notorious? taming heterogeneity to handle test-time shift in federated learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2024
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for silo federated learning in realistic healthcare settings. arXiv preprint arXiv:2210. 10. 0.02600, 2022.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham I 10000 dataset, a large collection of multi-souree dermatoscopic images common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Represen- tations, 2020. /ttttppp::///eeeeiieee net/ wtt.e/ooommnmdddaaaaaa
Yanmeng Wang, Qingjiang Shi, and Tsung-Hui Chang. Why batch normalization damage federated learning on non-iid data? arXiv preprint arXiv:2301.02980 2023.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Chun-Han Yao, Boqing Gong, Hang Qi, Yin Cui, Yukun Zhu, and Ming-Hsuan Yang. Federated multi-target domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pp. 1424-1433, 2022.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. 0dd100k: A diverse driving dataset for heterogeneous multitask learn- ing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636-2645, 2020.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint
2019.
Hongyi Zhang, Jan Bosch, and Helena Holmström Olsson. End-to-end federated learning for tonomous driving vehicles. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lv. Addressing catastrophic forgetting in federated class-continus learning. arXiv preprint arXiv:2303.069377 2023.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. CoRR, abs/ /a//8806.00882, 2018. URL
1806.00588
Jike Zhong, Hong- You Chen, and Wei-Lun Chao. Making batch normalization great in federated deep learning. arXiv preprint arXiv:2303.06538 2023.
Weiming Zhuang. Yonggang Wen, Xuesen Zhang, Xin Gan, Daiying Yin, Dongzhan Zhou, Shuai Zhang, and Shuai Yi. Performance optimization of federated person re-identification via bench- mark analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pp-
955-963 2020.
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsuper- vised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4912-49211 2021.
Weiming Zhuang, Xin Gan, Yonggang Wen, and Shuai Zhang. Fasyfl: A low-code federated learn- ing platform for dummies. IEEE Internet of Things Journal, 1(15):1374-11375, 2022a.
Weiming Zhuang, Xin Gan, Yonggang Wen, and Shuai Zhang. Optimizing performance of fed- erated person re-identification: Benchmarking and analysis. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2022b.
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. International Conference on Learning Representations, 2022c.
13

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETUP
In this section, we provide more details of experimental setups, including datasets, model architec- tures, and implementation details.
A.1 DATAETS
Figure 8 and 9 visualize three multi-domain datasets used in this work; these three datasets are Digits- Five Li et al. 2021). Office-Caltcch-Il (Gong et al. 2012), and DomainNet (Peng et al.) 2019), respectively. It shows that images under each dataset have significant domain gaps. We construct multi- FL by constraining each FL client to contain samples of the same domain. Each image sample from one client. Each FL client contains images of a dataset (domain). We follow FedBN
(ii et al. 202I to preprocess and transform these datasets.
A.2 MODEL ARCHITECTURES
Table illustrates the model architectures for experiments on the Digits-Five dataset and Table illustrates the model architectures for experiments on Office-Caltech-e and DomainNet datasets. For the convolution layer (Conv2D), the hyperparameters are in the sequence of input dimension, output dimension, kernel size, stride, and padding. For the max pooling layer (MaxPool2D)) the hyperparameters are kernel and stride. For the fully connected layer (FC), the hyperparameters input and output dimensions. For the batch normalization (BN) layer, the hyperparameter is the number of channels. For group normalization, the hyperparameters are the number of groups and the number of channels. FedAvg+LN shares similar model architecture sets the number groups to 1. The methods with BN are Standalone, FedAvg (McMahan et al. 2017), FedProx (Li al. 2020b), SiloBN (Andreux
Ge but
al.) (020)), FedBN (Li al., (021), and FixBN (Zhong et al. (0233) These methods share the same model architecture. Note that the model architecture not exactly the same as the ones used in FedBN (Li al. 2021). where they use one- dimension BN layer as regularizer between FC layers but we use Dropout such that the comparisons are fair terms of model architectures.
Besides, we use the default implementation of ResNet-18 (He et al. 2016) and MobileNetV2 (aszke al., al.] (017)) in PyTorch (Paszke et al. (2077) for methods with BN on the OfficeCCaltech- 10 dataset and CIFAR-10 dataset, respectively. FedWon replaces the convolution layers in ResNet-18 and MobileNetV2 with WSConv and removes batch normalization layers. FedAvg+GN and Fe- dAvg+LN replace BN layers with GN layers. Specifically, FedAvg+GN sets the number of groups to 32 by default, but sets output dimension is 144 (to ensure divisibility); FedAvg+LN sets the number of groups in GN to 1. The source code will be released.
when the output dimension smaller than 32, and to 24 when the
A.3 IMPLEMENTATION AND TRAINING DETAILS
Listing provides the implementation of WSConv in PyTorch. We employ the architectures de- scribed in Section A.2 to implement FedWon, adhering to the client training and server aggrega- tion protocols of FedAvg (McMahan al., 2017). We implement FedWon based on both EasyFL. (Zhuang et al.) 2022a) for skewed label distribution experiments and FedBN original implementation for multi-domain FL experiments. For the implementation FedBN, we reference the open-source To implement SiloBN (Andreux et al. 2020), we modify the FedBN implementation to aggregate only the BN parameters while keeping the BN statistics local. Unfor- tunately, the souree code for FixBN (Zhong al. 2023 is not publicly available, we implement based on the description provided the paper.
code available in Github [
Besides, we summarize the compared algorithms in Table
14

---

Published as a conference paper at ICLR 2024
98
X
g
(a) MNIST
(b) SVHN
(c) USPS
(d) SynthDigits
(c) MNIST-M
Figure 7: Visualization of samples from Digits-Five dataset.
(a) Amazon
(b) Caltech
(c) DSLR
(d) Webcam
Figure 8: Visualization of samples from Sffice-Clttec--11 dataset.
(a) Clipart
(b) Infograph
(c) Painting
(d) Quickdraw
(e) Real
(l) Sketch
Figure 9: Visualization samples from DomainNet dataset.
class WSConv (nn.
def init__nit___llll_____ (self, htwpchhannnlocut_chhheeeeeee
padding=0, http::ppp....cooo....oopapaaaaaa pade padding_mode- zeros'):
super (NSConv, ii______ii____hchh_____a_ out out_channels, kernei_size,
nn. wmw.intt.erviee__omaa..e.eeeee.. = oe ottp:rcccccccceeeee..etttotttoo ort.aaaaeeeerccccc....eeeeaaaaar 1, 1, 1))
5
_pps to- ret.ires.rrddaaauuu.... sf
(), requires_grad
celT.rregistee__bfffe((eess _pp,, _es,, persistent=False)
oelf.reeiister_bfffeet fan _an_in, persistent-False)
def standardizze___iihhtttte
12
mean te axis= (1,2,), keepdins=True)
13
var = forch.var (orch.var(self.weighh miis=11,2,3 2,,,, keepdins-Irue)
14
scale = torch. rsqrt wwwrcc..a..mumm///. maximum(var + self. fan_in, self.eps))
return
mean) scale self.gain
16
17
def (self, x):
18
return F.conv2d
input=x,
htt.tttseel..sstaaadiiiaaaa weightttt
21
bias=self.bias,
22
stride-seel..ttidd,
23
padding=seeff.aaddii
dilatioo=seeff.dllattoo
groups=self.grooup
26
27
Listing 1: WSConv implementation in PyTorch.
15

---

Published as a conference paper at ICLR 2024
Table 5: Model architectures of Six-layer CNN for experiments on Digits-Five dataset.
Layer
Methods with BN
FedWon
FedAvg+GN
Conv2D((3, 64, 5, 1, BN(64), ReLU
WSConv2D(3, 1,
Conv2D(3, 64, 5, 1, GN(32, 64), ReLU MaxPoo12D(2,,) 2)
1
ReI.U
MaxPool2D(22, 2)
MaxPool2D((2,2)
Conv2D(64, 64, 5, 2,
WSComv2D(64, 64, 1,
Conv2D(64, 64, 5, 2)
2
BN(66), ReLU
ReLU
GN(32, 64), ReLU MaxPool2D(2,, 2)
MaxPool2D(2,, 2)
MaxPool2D(2,
Conv2D(64, 128, 5, 2)
WSComv2D(64,, 128, 5, 2,
Conv2D(64, 128, 5, I, 2)
3
BN((18), ReLU
ReLU
GN(64, 128), ReLU
4 Dropout, FC(6272, 2048)
Dropout, FC(6222, 2048)
Dropout, FC(6272, 2048)
ReLU
ReLU
ReLU
Dropout, FC(2048, 512)
Dropout, FC(2048, 512)
Dropout, FC(2048, 512)
5
ReLU
ReLU
ReLU
6
FC(512, 10)
FC(512, 10)
FC(512, 10)
Table 6: Model architectures of AlexNet for experiments on Office-Caltech-10 10 and DomainNet datasets.
Layer Methods with BN
FedWon
FedAvg+GN
Conv2D(3, 64, 11,4,2 4, 2) BN(64), ReLU
WSConv2D(3, 64, 11, 4, 2)
Conv2D(33 64, 11,
1
ReLU
GN(32, 64), ReLU
MaxPool2D(33, 2)
MaxPool2D(3..)
MaxPool2D(3, 2)
C6nx2D(64, 192, 5, 1,
WSConv2D(64, 192, 5, 2)
Conv2D(64, 192, 5, 2)
2
BN((12),, ReLU
ReLU
GN(32, 192), ReLU MaxPool2ID(3, 2)
MaxPool2D(3, 2)
MaxPool2DD3..)
Conv2D(192, 384, 3, 1, 1)
Conv2D(192, 384, 3, 1, 1)
Conv2D(192, 384, 3, 1, 1) GN(64,384), ReLU
3
BN(384), ReLU
ReLU
Conv2D(384, 256, 3, 1, 1) BN((56), ReLU
WSConv2D(384, 256, 3, 1, I) ReLU
Cony2D(384, 256, 3, 1, 1) GN(64, 256), ReLU
4
Conv2D(256, 256, 3, 1, 1) BN((56),, ReLU
WSConv2D(256,, 256, 3, 1, 1) ReLU
Conv2D(256. 256, 3, 1, 1) GN(64, 256), ReLU MaxPool2D(3.
5
MaxPool2D(3. 2)
MaxPooo22D(3,2)
6
AdapiveevygPoll2D66. 6)
AdapiiveAvgPool2D6,. 6)
AdaptiveAvgPool2266. 6)
Dropout, FC(9216, 4096)
Dropout, FC(9116,C 4096)
Dropout, FC(9216,
7
ReLU
ReLU
ReLU
8 Dropout, FC(4096, 4096)
Dropout, FC(4096, 4096)
Dropout, FC(4096, 4096)
ReLU
ReLU
ReLUU FC(4096. 10)
9
FC(4096, 10)
FC(4096, 10)
By default, conduct experiments with local epochs E 1 and batch size B = 32 across all datasets. Stochastic gradient optimization (SGD) used as the optimizer, with learning rates tuned in the range of [0.001, 0.1] for all methods. Specifically, for FedWon experiments with batch size of B = 32, we incorporate adaptive gradient clipping (AGC) (Brock et al.) 2021b), which is specifically designed for normalization-free networks. AGC applies gradient clipping the weight matrix wi E R xM of the th layer, where the gradient G' R X is clipped with a threshold X before updating the model. The clipping operation for each row of G¹ can be expressed as follows:
A | | LE
G
X
G
|
(4)
G
otherwise,
16

---

Published as a conference paper at ICLR 2024
Table Summary of compared methods on different aspects. and x means the method supports and does not support the attribute, respectively. O means that no prior studies are conducted to analyze whether the method supports the attribute.
Method
Has no BN Multi-domain FL. Skewed Labeld Distribution Cross-silo FI. Cross-device FI.
FedAvg
FedAvg+GN
FedAVG+LN
V
FixBN
SiloBN
FedBN
X
FedWon
where || || is the the Frobenius norm, i.e. | =
(Wa5)?), |
max max p, with default 1 3. We only AGC for FedWon with batch size B 32 and bypass AGC on small batch sizes such as B = (1,2,4] The impact of AGC and the clipping threshold is further analyzed in Section
We tune the learning rates for the methods compared in the main manuscript and provide their specific learning rates below. Table 8 illustrates the learning rates of different methods on three datasets, corresponding to the experiments of Table 1 in the main manuscript. We use a clipping threshold of 0.64 Digits-Five, 1.28 for Lffice-Caltech-ae Additionally, Table 9 presents the learning rates used for experiments with small batch sizes B {1,2, 4) on Office-Caltech- 10 and Digits-Five datasets. Table 10 displays the learning rates used for experiments using ResNet-18 18 as the backbone. All experiments on Digits- Five are trained for 100 rounds and experiments on Office-Caltech- 10 and DomainNet are trained for 300 rounds.
and 1.28 for the DomainNet dataset.
For evaluation of skewed label distribution, all experiments are with local epoch E 5 for 300 rounds. We use SGD as the optimizer and tune the learning in the range of [0.001, 0.1] I] for different algorithms.
Table 8: Learning rates of different methods in the experiments of Table in the manuscript.
Datasets
Standalone
FedAvg
FedProx
o+GN
+LNN
SiloBN
FixBN
FedBN
Ours
Digits-Five 0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.05
Caltech-10
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.1
DomainNet
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.05
0.05
*+GN means FedAvg+GN, +LLN means FedAvg+IN
Table 9: Learning rates of experiments on small batch sizes. Left: learning rates of experiments of
small batch sizes B {1, 2, 4] on Office-Caltech- dataset. Right: learning rates of experiments of small batch sizes of randonly selecting fraction C {0.1, 0.2, 0.4) out of total clients on Digits-Five dataset.
FedAvg Ours
B FedAvg SiloBN FixBN FedBN FedAvg+GN FedAvg+LN Ours
0.1
0.01
1
0.001
0.001
0.005
0.1
2
0.005
0.01
2
0.001
0.001
0.001
0.001
0.001
0.001
0.01
0.1
0.01
0.04
4
0.001
0.01
0.01
0.01
0.001
0.001
0.03
0.2
4
0.01
0.04
0.4
4
0.01
0.04
B
EXPERIMENTS
This section provides more experiment results that provide further insights into the behavior of FedWon and shed light the effects of different parameters.
17

---

Published as a conference paper at ICLR 2024
Table 10: Learning rates 77 of different methods in the experiments of using ResNet-20 as backbone.
FedAvg
Fe
FedAvg+I.N
SiloBN
FixBN
FedBN
Ours
7
0.1
0.03
0.01
0.05
0.03
0.03
0.1
Table 11: Evaluation Fed-ISIC2019 dataset with medical images from six different centers. Fed- Won outperforms FedAvg and FedBN by a signifcant margin in all domains.
Methods
Center
Center
Center
Center 4
Center 5
Center 6
FedAvg
0.40
0.21
0.37
0,42
0.39
0.43
FedBN
0.31
0.38
0.43
0.39
0.30
0.36
Won (Ours)
0.46
0.43
0.52
0.56
0.40
0.59
Table 12:
Comparison methods on domain generalization capability using the Office-Caltech- 10 dataset, where we employ Amazon, Caltech, and DSLR as the seen domains during training and WebCam as the unseen domain for evaluation.
Seen Domains
Unseen Domain
Methods
Amazon
Caltech
DSLR
WebCam
FedAvg w/o BN
38.0
33.7
40.6
28,8
FedAvg
58.9
42.7
59.4
52.5
FedBN
65.6
48.0
78.1
61.0
Fed (Ours)
66.1
51.6
90.6
67.8
B.1 EXPERIMENTS ON MEDICAL IMAGES
To further study how our proposed FedWon benefits multi-domain FL in real-world scenarios, we extend evaluation diagnosis of skin lesions using datasets from ISIC2019 Challenge (Codella et al. 2018 Combalia et al. 2019) and the HAM10000 (Tschandl et al. 2018) dataset. The dataset contains images collected from four hospitals, where one hospital with different imaging tech- nologies. We follow Flamby (Terrail et al. 2022) to construct them as six different centers: BCN, Vidir-molemax, Vidir-modern, Rosendahl, MSK, and Vienna-dias, with each center's 's images repre- senting unique domain. In total, the dataset encompasses 23,247 images of skin lesions, including 9930 training samples and 2483 testing samples from BCN; 3163 training samples and 791 testing samples from Vidir-molemax, 2691 training samples and 672 testing samples from Vidir-moderm, 1807 training samples and 452 testing samples from Rosendahl, 655 training samples and 164 testing samples from MSK, and 351 training samples and 88 samples from Vienna-dias. In the experimental setup, we simulate the scenarios where multiple healthcare centers collaborate to train a skin lesion diagnosis model, with each client representing healthcare center. The task to conduct image classification for different melanoma classes.
2016) (without any pre-training) with local epoch E' 1 and batch size B = 64 for 50 rounds. We use SGD optimizer with learning rate ? 0.005 for FedAvg and FedWon and y 0.001 for FedBN. The learning rate tuned among (0.001, 0.005, 0.01, 0.05). We follow the implementation in Flamby to use a weighted focal loss (Ln et 2017) and data augmentations.
We run the experiments using ResNet- 18 (He
Table II shows the testing accuracy of FedAvg, FedBN, and our proposed FedWon across the six healthcare center domains. In this challenging setting, FedBN only achieves similar performance to FedAvg. In contrast, FedWon outperforms both FedAvg and FedBN in all domains by significant margin. The results are inspiring and demonstrates the potential deploying FedWon to healthcare application scenarios, where data is often scarce, isolated, and spans multiple domains.
18

---

Published as a conference paper at ICLR 2024
Table 13: Testing accuracy (%) comparison using ResNet-20 on Office-Caltech-10 Dataset.
Methods
Amazon
Caltech
DSLR
WebCam
Avg
FedAvg
45.3
36.4
68.8
76.3
56.7
FedAvg+GN
44.3
31.1
71.9
74.6
55.5
LL
34.4
26.2
59.4
44.1
41.0
FixBN
34.9
33.8
62.5
78.0
52.3
SiloBN
40.6
29.3
59.4
81.4
52.7
FedBN
57.3
37.3
90.6
89.8
68.8
FedWon
63.0
46.7
90.6
86.4
71.7
100
100
Testing Accuracy (%)
80
Testing Accuracy (%)
90-
60
MNIST
SVHN
40
USPS
80
FedWon FedBN
SyncDigits MNISTM
Average
USPS
20
Sights
FedBN
SVHN
MNIST-M T-M
10
20 30 40 50 60 70 80 90 100 Training Rounds
70
5
10
15
20
25
30
35
40
45
50
Number of Clients
(a) Cross-silo FL Training
(b) Domain Heterogeneity
Testing accuracy (%) comparison FedBN and FedWon on Digits-Five dataset: (a) compares testing accuracy throughout training cross-silo FL with total 5 clients (one client per domain) and batch size B = 32; (b) comparison of different degrees of domain heterogeneity.
Figure 10:
B.2 ADDITIONAL ANALYSIS
Domain Generalization Capability. We expand our analysis to investigate the domain adaptation and generalization capabilities of FedWon. Our experiments are conducted on the Cffce-CClltech- 10 dataset, where we employ Amazon (A), Caltech (C), and DSLR (D) as the seen domains during training. while WebCam (W) is exclusively reserved ically for zero-shot evaluation. We use the client local models to evaluate the seen domains and use the server global model to test on the unseen domain; while to be fair for FedBN, model with averaged BN layer parameters from the seen domains. [2 presents compelling ev- idence that FedWon only excels in performance robust generalization capabilities on the unseen domains. These results demonstrate an additional advantage of FedWon, highlighting its capability for domain generalization.
unseen domain only for evaluation, specif-
employ a global
the seen domains but also exhibits the most
Evaluation on Alternative Backbones. In addition to evaluating the effectiveness of FedWon using AlexNet (Krizhevsky et al. 2017) the Aiffic-Calltch 10 dataset, Table 13 also compares testing
accuracy on common backbone, ResNet-20 (He et al. 2016). Interestingly, replacing BN with GN or LN is not effective on ResNet-20 as on AlexNet. FedAvg+GN and FedAvg+LN only achieve similar or even worse performance than FedAvg. FedBN (Li et al. 202T), instead, achieves better performance than the other existing methods. Nevertheless, our proposed FedWon consistently outperforms the state-of-the-e methods even with ResNet-20 the backbone.
Analysis on Different Degrees of Domain Heterogeneity. We evaluate the performance of the proposed FedWon under different degrees of domain heterogeneity. To simulate varying degrees of domain heterogeneity, we follow the approach taken by FedBN (Li et al, 202I) and create different numbers of clients with the same domain the Digits-Five dataset. We start with 5 clients, each containing data from one domain, and then add clients at a time, with each new client containing one of the Digits-Five datasets, respectively. We evaluate the performance of the algorithms for
19

---

Published as a conference paper at ICLR 2024
Table 14: Performance comparison of FedBN and FedWon under different local epochs E'
{1, 4, 8) on offic-Calltchh- 10 dataset. FedWon maintains performance and consistently outper- forms FedBN under different numbers of local epochs.
Methods
Amazon
Caltech
DSLR
Webcam
Average
FedBN
67.2
45.3
85.4
87.5
71.4
FedWon
67.0
50.4
95.3
90.7
75.6
FedBN
66.7
43.6
84.4
89.8
71.1
FedWon
68.8
51.1
93.8
84.8
74.6
FedBN
64.6
45.8
87.5
89.8
71.9
Fen
64.6
49.3
96.9
91.5
75.6
Table 15: Evaluation cross-silo skewed label distribution using MobileNetV2 with 10 clients constructed by splitting the CIFAR-10 dataset with Dir (0.1).
Methods
FedAvg
FedAvg+GN
FedAvg+LN
SiloBN
FixBN
FedBN
FedWon
Accuracy
66.95
68.11
71.65
70.8
66.22
69.07
76.45
Table 16: Comparison of FedWon and FedAvg on total of 1000 clients on Digits-Five dataset, with selection of only 0.1 clients per round.
Methods
MNIST
SVHN
USPS
SynthDigits
MNIST-M
Average
FedAvg
96.0
71.2
94.7
82.9
82.8
85.5
FedWon
96.4
73.6
95.5
83.7
81.9
86.2
Table 17: Comparison of FedWon and PartialFed on Office-Caltech-10 10 dataset.
Methods
Amazon
Caltech
DSLR
Webcam
Average
PartialFed-Fix
58.3
44.9
88.1
91.2
70.6
PartialFed-Adapiin
63.4
45.4
85.6
90.5
71.3
FedWon (Ours)
67.0
50.4
95.3
90.7
75.6
Table 18: Ablation studies on the impact of WSConv and AGC
Batch Size
WSConv
AGC
Amazon
Caltech
DSLR
Webcam
Average
63.7
51.0
96.3
91.2
75.6
/
65.
52.0
90.6
89.8
74.4
32
V
46.4
37.3
68.8
71.2
55.9
27.1
19.6
37.5
28.8
28.2
V
65.1
51.1
93.8
86.4
74.1
67.2
55.6
96.9
93.2
78.2
2
53.1
38.7
87.5
78.0
64.3
54.7
44.0
84.4
78.0
65.3
different numbers clients from N = (5, 10, 15, ..,, More clients represent less heterogeneity as more clients have overlapping domains of data. Figure comparess the performance of FedWon and FedBN under these settings. The results show that the performances of both FedWon and FedBN increase the degree of heterogeneity decreases. FedBN outperforms FedAvg in all the settings as evidenced in Li et al. (202I). However, our proposed FedWon achieves even better performance than FedBN all domains and all levels of heterogeneity.
20

---

Published as a conference paper at ICLR 2024
Average Test Acc. (%)
80
Average Test Acc. (%)
80
75
75-
70
70-
65
60
55
55
60
0.01
0.02
0.04
1666
0.64
1.28
2.50
0.01
0.02
0.04
0.32
0.64
1.28
2,56
Clipping Threshold A
Clipping Threshold A
(a) Batch Size
32
(b) Batch Size
Figure 11: Impact of clipping threshold À of adaptive gradient clipping (AGC) OfficeCaattch-110
dataset, using different batch sizes B.
Testing Accuracy Changes Throughout Training. Figure T0 illustrates the changes in testing accuracy throughout the training process on the Digits-Five dataset. Specifically, Figure T0a com- cross-silo FL involving a total clients (one
pares the performance of FedWon and FedBN client per domain) and a batch size of B 32. FedWon outperforms FedBN in certain domains or demonstrates similar performance in others. Notably, FedWon achieves better performance in the early stage of training FedWon exhibits faster convergence, achieving satisfactory level of accuracy more quickly than FedBN, These results complement the results in Figure 4 (right) in the main manuscript that compares Fed and FedAvg in cross-device FL scenario.
Impact of Local Epochs. Table compares the performance our proposed FedWon and FedBN (Li et al. )021) under different local epochs E {1, 4, 8) #ffice-alttcc-10 dataset. Fed- Won maintains performance and consistently outperforms FedBN under different numbers of local epochs. We these experiments with batch size B 32 and the learning rate the in Table on the Office-Caltech. 10 dataset.
the
Evaluation on Cross-silo FL for Skewed Label Distribution. Table 15 compares different algo- rithms in cross-silo FL for skewed label distribution on the CIFAR-10 dataset, which complements cross-device FL experiments in Figure O FedWon also consistently outperforms all other methods in cross-silo FL. We run experiments with 10 clients under Dir(0.1) of non-i.i.d data, batch size for 200 rounds.
B 64, and local epoch
Evaluation on Cross-device FL of 1000 clients. Table 16 compares FedWon and FedAvg on total of 1000 clients on Digits-Five dataset, with selection of only 0.1 clients per round. FedWon also generally outperforms FedAvg under this setting. These experiments are run with batch size B = 2 and learning rate of 0.02.
Addtional Comparison with PartialFed. Table 17 further compares FedWon with two variant implementations of PartialFed (Sun et al.] (202T) on Office-Caltech- 10 dataset. FedWon generally achieves superior performance to FertialFed, especially on the average testing accuracy.
B.3 ADDITIONAL ABLATION STUDIES
Impact of WSConv and AGC. We analyze the impact of WSConv and AGC, which supplements the ablation study presented in the main manuscript. Table 18 shows the impact of these two com- ponents with batch size B 32 and small batch size removing the normalizations, using WSConv significantly improves the performance sizes. AGC, however, shows positive impact only with batch size = 32, as specifically designed for larger batch sizes. Consequently, do not adopt AGC in the experiments with small batch sizes (B = 21, 43). We run these experiments with learning rate n = 0.08 for B 32 and 7 0.01 for B 2.
on the Office- Cahtee-altech---ee 10 dataset. After
both batch
Impact of Clipping Threshold À for AGC. We further extend to evaluate the impact of clip- ping threshold À under batch sizes B 2 and B 32. Figure II shows the aver- age testing accuracy on the OfficeCCaltech-11 dataset using different clipping thresholds A = (0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56). When the batch size B = 32, the performance
21

---

Published as a conference paper at ICLR 2024
Table 19:
Evaluation on the impact of using AGC optimizer on different algorithms. AGC also benefits other methods, while our proposed FedWon achieves the best overall performance.
Methods
AGC
Amazon
Caltech
DSLR
Webcam
Average
FedAvg
61.8
44.9
77.1
81.4
66.3
FedAvg
62.5
45.3
75.0
84.8
66.9
FedAvg+GN
60.8
50.8
88.5
83.6
70.9
FedAvg+GN
V
64.1
48.0
90.6
88.1
72.7
FedAvg+LN
55.0
41.3
79.2
71.8
61.8
FedAvg+LN
V
59.4
42.2
84.4
79.7
66.4
FixBN
59.2
44.0
79.2
79.6
65.5
FixBN
58.9
43.1
75.0
88.1
66.3
SiloBN
60.8
44.4
76.0
81.9
65.8
SiloBN
59.4
44.4
78.1
83.0
66.2
FedBN
67.2
45.3
85.4
87.5
71.4
FedBN
70.3
45.3
87.5
88.1
72.8
FedWon (Ours)
67.0
50.4
95.3
90.7
75.6
rather insensitive to different values of À when it is not too small (larger than 0.08). When the batch size B 2, the best clipping threshold is À 1.28 and the performance is sensitive to differ- ent values. Consistent with the finding in Table [8] we recommend avoiding using AGC when the batch size small. These results provide insights into selecting an appropriate clipping threshold for multi- multi-domain FL.
Impact AGC on Other Algorithms. Table compares the results with and without AGC with the same learning rate for different methods. AGC also benefits other methods, while our proposed FedWon achieves the best overall performance.
B.4 COMPLEMENTARY EXPERIMENTS
Visualization of BN Statistics. Figure 12 visualizes the running mean and variance of BN layers of the 6-layer CNN. complements Figure 1b in the main manuscript and shows the discrepancies of BN statistics between clients and between a client and the server all BN layers.
Visualization of Feature Maps. Figure presents the visualization feature maps obtained through three methods: FedAvg. FedBN, and proposed FedWon. These feature maps the output of each convolution layer in AlexNet on the Office-Caltech-II dataset, which encompasses data from four distinct domains, namely Amazon, Caltech, DSLR, and WebCam. FedWon exhibits significantly enhanced feature maps on the object of interest compared to those produced by the FedAvg and FedBN.
Effectiveness on Small Size. Table 20 compares performance of our proposed FedWon with ex- isting methods on Office-Caltech-I1 10 dataset with batch size performance also in this setting, complementing the experiments of batch size B = {1,2) in Table 2 in the main manuscript. Additionally, Figure 14 compares the testing accuracy over the course of training of FedWon with batch sizes B (1, 2, 4 Digits-Five dataset. Different batch sizes tend
FedWon achieves the best
have similar trend of convergence.
Effectiveness on Selection of Clients. T compares the performance of FedAvg and FedWon on cross-device FL on the Digits-Five dataset with a fraction C = 0.4 of clients out of a total of 100 clients to participate in training each round. FedWon achieves superior performance also in this setting, complementing the experiments of
0.2) in Table in the main manuscript.
Comparison of Methods with Variances. Table 22 presents testing accuracy comparison of dif- ferent methods on three datasets with mean (standard deviation) of three runs of experiments. complements the results in Table T in the main manuscript.
22

---

Published as a conference paper at ICLR 2024
Layer
Layer
Layer
Layer
Layer
Client Cliett
Roning Versnee
Rurning Meur
Running Mean
Rurning Mean
Running Mexr
Rarning Meun
(a) BN statistics of models in two clients.
Layer
Layer
Layer
Layer
Layer
Client
Sorver
Ruming Uniteen
Running Moan
Burning Mean
Running Mean
Running Mear
Bamning Mean
(b) BN statisties of models in client (Client 1) and the server. Layer
Layer
Layer
Layer
Layer
Client 2
Ruming Vntteen
Running Mean
Punning Mean
Running Mean
Running Mear
Raming Meon
(c) BN statistics of models in client (Client 2) and the server.
Figure 12: BN statistics of all layers in 6-layer CNN.
Table 20: Performance comparison using small batch sizes B 4 ffiice-alttec-100 dataset.
Our proposed FedWon achieves outstanding performance compared to existing methods.
B Methods
Amazon
Caltech
DSLR
WebCam
FedAvg
65.6
46.7
78.1
88.1 1
FedAvg+GN
60.9
52.0
84.4
89.8
FedAvg+LN
54.2
44.9
78.1
72.9
FixBN
66.2
50.2
78.1
91.5
SiloBN
63.5
48.9
78.1
88.1
FedBN
67.2
50.7
90.6
91.5
Fed
68.8
54.2
96.9
91.5
Table 21: Testing accuracy comparison on randomly selecting a fraction C 0.4 out total of 100 clients for training each round with batch size B = 4. FedWon consistently outperforms FedAvg on Digits-Five dataset. We report the mean (standard deviation) of three runs of experiments.
Method
MNIST
SVHN
USPS
SynthDigits
MNIST-M
Average
FedAvg
98.1 (0.0)
80.5 (0.1)
97.0 (0.2)
91.4 (0.2)
89.4 (0.1)
91.3 (0.0)
0.4
FedWon (Ours)
98.8 (0.0)
86.4 (0.2) 98.4 (0.2)
94.2 (0.2)
91.0 (0.3)
93.7 (0.0)
23

---

Published as a conference paper at ICLR 2024
Image
convl
conv2
conv3
conv4
conv5
Image
conv1
conv2
conv3
conv4
conv5
(a) Amazon FedAvg
(g) DSLR -FedAvg
Image
conv1
conv2
conv3
conv4
conv5
Image
conv1
conv2
conv3
conv4
conv5
(b) Amazon
FedBN
(h) DSLR
FedBN
Image
convl
conv2
conv3
conv4
conv5
Image
convl
conv2
conv3
conv4
conv5
(c) Amazon
FedWon
(i) DSLR FedWon
Image
convl
conv2
conv3
conv4
conv5
Image
conv1
conv2
conv3
conv4
conv5
(d) Caltech
FedAvg
(i) WebCam FedAvg
Image
convl
conv2
conv3
conv4
conv5
Image
convl
conv2
conv3
conv4
conv5
(c) Caltech
FedBN
(k) WebCam
FedBN
Image
conv1
conv2
conv3
conv4
conv5
Image
conv1
conv2
conv3
conv4
conv5
(l) Caltech FedWon
(I) WebCam FedWon
Figure 13: Visualization of feature maps of FedAvg, FedBN, and proposed FedWon the Office-Caltech-10 dataset, which contain four domains: Amazon, Caltech, DSLR, and WebCam.
100
Testing Accuracy (%)
80
60
MNIST
40
SVHN
B B=1
USPS
B
SyncDigits
20
B
MNISTM
10 20 30 40 50 60 70 80 90 100 0
Training Rounds
Figure 14: Testing accuracy (%) comparison of different batch sizes B {1,2,4) using FedWon on Digits-Five dataset with 10 randomly selected clients out of 100 clients.
24

---

Published as a conference paper at ICLR 2024
Table 22: Testing accuracy (%) comparison of different methods on three datasets. Our proposed Wed outperforms existing methods in most of the domains, FedWon achieves the best average testing accuracy in all datasets. We report the mean (standard deviation) three runs of experiments.
Domains
Standalone
FedAvg
FedProx
+GN
+LN
SiloBN
FixBN
FedBN
Ours
94.4
96.2
96.4
96.4
96.4
96.2
96.3
96.5
96.8
MNIST
(0.2)
(0.2)
(0.0)
(0.1)
(0.1)
(0.0)
(0.1)
(0.1)
(0.2)
67.1
71.6
71.0
76.9
75.2
71.3
71.3
77.3
77.4
SVHN
(0.7)
(0.5)
(0.8)
(0.1)
(0.4)
(1.0)
(0.9)
(0.4)
(0.1)
Digit-Five
95.4
96.3
96.1
96.6
96.4
96.0
96.1
96.9
97.0
USPS
(0.1)
(0.3)
(0.1)
(0.2)
(0.4)
(0.2)
(0.2)
(0.2)
(0.1)
80.3
86.0
85.9
86.6
85.6
86.0
85.8
86.8
87.6
SynthDigits
(0.8)
(0.3)
(0.2)
(0.1)
(0.3)
(0.3)
(0.1)
(0.3)
(0.2)
77.0
82.5
83.1
83.7
82.2
83.1
83.0
84.6
84.0
MNIST-M
(0.9)
(0.1)
(0.2)
(0.5)
(0.3)
(0.4)
(0.8)
(0.2)
(0.2)
83.1
86.5
86.5
88.0
87.1
86.5
86.5
88.4
88.5
Average
(0.4)
(0.1)
(0.1)
(0.1)
(0.0)
(0.3)
(0.0)
(0.1)
(0.1)
54.5
61.8
59.9
60.8
55.0
60.8
59.2
67.2
67.0
Amazon
(1.8)
(1.2)
(0.5)
(1.8)
(0.3)
(1.3)
(1.8)
(0.9)
(0.7)
Office- Cal.Callehhhhh 10
40.2
44.9
44.0
50.8
41.3
44.4
44.0
45.3
50.4
Caltech
(0.7)
(1.2)
(1.9)
(3.3)
(1.2)
(1.2)
(0.8)
(1.3)
(2.8)
81.3
77.1
76.0
88.5
79.2
76.0
79.2
85.4
95.3
DSLR
(0.0)
(1.8)
(1.8)
(1.8)
(1.8)
(1.8)
(1.8)
(1.8)
(2.2)
89.3
81.4
80.8
83.6
71.8
81.9
79.6
87.5
90.7
Webcam
(1.0)
(1.7)
(2.6)
(5.2)
(2.0)
(2.0)
(2.9)
(1.0)
(1.2)
66.3
66.3
65.2
70.9
61.8
65.8
65.5
71.4
75.6
Average
(0.4)
(0.7)
(1.0)
(2.5)
(0.7)
(0.2)
(0.8)
(1.0)
(1.4)
42.7
48.9
51.1
45.4
42.7
51.8
49.2
49.9
57.2
Clipart
(2.7)
(2.0)
(0.8)
(0.5)
(0.7)
(1.0)
(1.8)
(0.5)
(0.5)
24.0
26.5
24.1
21.1
23.6
25.0
24.5
28.1
28.1
Infograph
(1.6)
(2.5)
(1.6)
(1.1)
(1.2)
(2.1)
(0.9)
(0.8)
(0.2)
34.2
37.7
37.3
35.4
35.3
36.4
38.2
40.4
43.7
Painting
DomainNet
(1.6)
(3.3)
(2.0)
(2.0)
(0.6)
(1.9)
(0.7)
(0.7)
(1.2)
71.6
44.5
46.1
57.2
46.0
45.9
46.3
69.0
69.2
Quickdraw
(0.9)
(3.4)
(3.8)
(1.0)
(1.2)
(2.8)
(3.9)
(0.8)
(0.2)
51.2
46.8
45.5
50.7
43.9
47.7
46.2
55.2
56.5
Real
(1.0)
(2.3)
(0.6)
(0.3)
(0.7)
(0.9)
(2.8)
(2.6)
(0.4)
33.5
35.7
37.5
36.5
28.9
38.0
37.4
38.2
51.9
Sketch
(1.1)
(0.9)
(2.3)
(1.8)
(1.3)
(1.9)
(2.0)
(6.7)
(1.9)
42.9
40.0
40.2
41.1
36.7
40.8
40.3
46.8
51.1
Average
(0.5)
(1.5)
(0.5)
(0.0)
(0.3)
(0.4)
(0.3)
(1.5)
(0.2)
"+GN means Ge,Avg+GN, 4LNN means FedAvg+LN
25


Reviews:
[{"review_id": "4KYl8joGQ5", "reviewer": "Reviewer_C95h", "summary": "This paper introduces a scaled weight standardization method (FedWon) for federated learning. The proposed method eliminates the normalization layers in FL and reparameterizes convolution layers with scaled weight standardization to counter the drawbacks of common normalization layers in the FL model. Extensive experiments on real-world datasets validate the effectiveness of FedWon and demonstrate its robust generalization capability for both cross-silo and cross-device FL.", "questions": "1. Please refer to weakness.\n\n2. Sec.4 claims that the FedWon will achieve competitive results even at a batch size of 1 while [https://arxiv.org/abs/1602.05629] shows the FedAvg will degenerate to the FedSGD and be less effective. It is an interesting topic, and do you mind to report more details about the learning curves (communication round v.s accuracy) at different batch sizes or on different datasets?\n\n3. What is the impact of local epochs on the proposed method?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper studied the multi-domain FL problem and proposed a novel FL method FedWon which employs the Scaled Weight Standardization technique as an alternative to the batch-normalization module. \n\n1. The FedWon can achieve competitive performance to the SOTA methods without additional computation cost during inference.\n\n2. Experiments on multi-domain datasets show the FedWon overperforms the conventional FL methods even if the batch size of the training process is small (1 or 2), and the visualization of feature maps demonstrates that the FedWon can effectively mitigate domain shifts across different domains.\n\n3. The paper is well-written and organized. Extensive experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The paper brought the Scaled Weight Standardization (SWS) technique to handle the multi-domain FL problem. However, there is less analysis about the SWS’s impacts to the FL process, e.g., will it lead to a better convergence bound?\n\n2. Many methods are compared in the paper, some of them have BN, some of them are suitable for cross-silo FL only, and some of them are suitable for cross-device FL, it would be clearer to have a structured summarization to help understand the scenarios where these methods are suitable for.\n\n3. Only one dataset is evaluated for the skewed label distribution problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rMYCAeV3ND", "reviewer": "Reviewer_hdps", "summary": "This paper considers the problem of multi-domain federated learning. This paper proposes to remove batch normalization and reparameterize the convolution layer weights for another kind of normalization. Finally, experiments show that this simple technique yield considerable improvement for the performance of multi-domain FL over many baselines, e.g., FedAVG and FedBN.", "questions": "* Why the proposed method is only applied to the convolution layer? From Eq. 1, it seems that it can be applied to any kinds of layers represented by a weight matrix $W$. \n* In Eq. 1, how is $\\gamma$ chosen? In particular, how robust is the training result to the choice of $\\gamma$?\n* Since the proposed method does not depend of batch statistics, I'm curious why FedWon B=1 better than B=2 (e.g., Figure 4)?  \n* Also, in figure 4, I'm curious to see how would FedAvg (B=1) perform.\n* It is claimed in this paper that FedBN can't do cross-device FL. However, there is not enough evidence, as far I can tell, from the paper that supporting this argument. Can the authors elaborate more on why FedBN can't do cross-device FL?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method is easy to implement, and can be potentially plug into many existing methods. The experiments are extensive and the paper is well-writing in general.", "weaknesses": "* I have many questions that I wish could be solved. Some of them are from questionable arguments from the paper, some of them are from the abnormal experimental results, and some of them are from the my curiosity in why the proposed method would work. Please see the Questions section for details. \n* The novelty is quite limited, where the proposed method is to use an existing reparametrization trick (Brock et al. (2021a)) in the FL setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0gkX3g5uTd", "reviewer": "Reviewer_Vk8b", "summary": "This paper aims to address the challenge of multi-domain federated learning (FL) where clients have data from diverse domains with distinct feature distributions. The proposed method, Federated Learning Without Normalizations (FedWon), eliminates normalization layers and reparameterizes convolution layers. Extensive experiments on various datasets and models demonstrate that FedWon outperforms existing methods.", "questions": "- In the upper image in Figure 1b, it seems that the green points representing the server are entirely invisible, while in the lower image, it appears that there is no information at all.\n- The presentation of feature maps in Figure 5 doesn't seem very informative. What information can we extract from the feature maps, and how are they related to the model's performance? Additionally, it appears that there is not much difference between the feature maps of FedAvg and FedBN.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to read, and the comparison figure (Figure 2) effectively illustrate the differences with previous methods.\n- The ablation experiments for cross-silo federated learning cover various factors affecting model performance, such as batch size and client sampling rate.", "weaknesses": "- The proposed method lacks innovation; it essentially directly applies the weight standardization and gradient clipping from the NF-Net series [1, 2] to the federated learning setting. It does not offer targeted improvements to address the unique challenges of the federated learning setting.\n- The experiments for cross-device FL in the paper are not sufficient for the proposed method's effectiveness. The cross-device FL experiments only include a single dataset and 100 clients.\n\n[1] Brock, Andrew, Soham De, and Samuel L. Smith. \"Characterizing signal propagation to close the performance gap in unnormalized ResNets.\" International Conference on Learning Representations. 2020.  \n[2] Brock, Andy, et al. \"High-performance large-scale image recognition without normalization.\" International Conference on Machine Learning. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfeDgljk15", "reviewer": "Reviewer_M4Q8", "summary": "This paper introduces a novel method called Federated learning Without normalizations (FedWon) to address the problem of multi-domain federated learning (FL). FedWon eliminates normalization layers and reparameterizes convolution layers with scaled weight standardization to effectively model the statistics of multiple domains. Experimental results on five datasets demonstrate that FedWon outperforms the current state-of-the-art methods. FedWon is versatile for both cross-silo and cross-device FL, exhibits robust domain generalization capability, and performs well even with a small batch size of 1, making it suitable for resource-constrained devices.", "questions": "This paper mainly discusses the challenges of normalization in federated multi-domain learning and proposes to eliminate normalization. However, other directions exist to solve the federated multi-domain problem, such as PartialFed and FMTDA, which do not follow the normalization idea. What are the advantages and characteristics of FedWon compared with these methods?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality:\nThe FedWon method is a new approach to reparameterizes convolution layers with scaled weight standardization to effectively model the statistics of multiple domains. This is a creative combination of existing ideas that addresses the limitations of traditional batch normalization federated methods. \n\nQuality:\nThis work provides a thorough experimental results that demonstrate the effectiveness of FedWon compared to other customized approaches on batch normalization. The experiments are well-designed, and the results are statistically significant.\n\nClarity:\nThe paper is well-written and easy to understand. The authors provide clear explanations of the FedWon method and its implementation. The experimental results are presented in a clear and concise manner.\n\nSignificance:\nThe research direction is crucial as federated multi-domain learning is essential in real-world applications where data may originate from multiple sources with distinct characteristics.", "weaknesses": "1. Lack of Technical Novelty: The paper's technical innovation is limited to reparameterizing the convolution layers using the Scaled Weight Standardization technique. While this approach may have some benefits, it lacks novelty as the Scaled Weight Standardization technique has been proposed and utilized in previous studies. \n\n2. Insufficient Theoretical Analysis: The paper lacks a theoretical analysis of the effectiveness of the proposed Scaled Weight convolution (WSConv) layer. The authors should provide a more rigorous theoretical foundation to explain why the WSConv layer is suitable for addressing the challenges in federated multi-domain learning.\n\n3. Limited Comparative Analysis: The paper does not sufficiently compare with other federated multi-domain methods, such as PartialFed and FMTDA.\n\n4. Limited Applicability to Convolutional Neural Network (CNN) Models: The proposed method is limited to CNN-based deep learning models, which restricts its applicability to a specific class of models, such as recurrent neural networks (RNNs), graph neural networks (GNNs) or transformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)