Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition
Authors: Sara Pieri
Jose Renato Restom
Samuel Horváth
Hisham Cholakkal
Venue: NeurIPS 2023 poster (2023)
Keywords: Computer Vision
Federated Learning
Image Classification
Neural Network Architectures
Transformer
CNN
Data Hetereogenity
non-IID
TL;DR: An experimental study on architectural design choices to increase performance under non-IID federated settings for computer vision models

Abstract: Federated Learning (FL) is a promising research paradigm that enables the collaborative training of machine learning models among various parties without the need for sensitive information exchange. Nonetheless, retaining data in individual clients introduces fundamental challenges to achieving performance on par with centrally trained models. Our study provides an extensive review of federated learning applied to visual recognition. It underscores the critical role of thoughtful architectural design choices in achieving optimal performance, a factor often neglected in the FL literature. Many existing FL solutions are tested on shallow or simple networks, which may not accurately reflect real-world applications. This practice restricts the transferability of research findings to large-scale visual recognition models. Through an in-depth analysis of diverse cutting-edge architectures such as convolutional neural networks, transformers, and MLP-mixers,  we experimentally demonstrate that architectural choices can substantially enhance FL systems' performance, particularly when handling heterogeneous data.  We study visual recognition models from five different architectural families on four challenging FL datasets. We also re-investigate the inferior performance convolution-based architectures in the FL setting and analyze the influence of normalization layers on the FL performance. Our findings emphasize the importance of architectural design for computer vision tasks in practical scenarios, effectively narrowing the performance gap between federated and centralized learning.

Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition
Sara Pieri" Jose Renato Restom"
Samuel Horvath
Hisham Cholakkal
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)
Abstract
Federated Learning (FL) is a promising research paradigm that enables the collabo- rative training of machine learning models among various parties without the need for sensitive information exchange. Nonetheless, retaining data in individual clients introduces fundamental challenges achieving performance on par with centrally trained models. Our study provides an extensive review of federated learning applied visual recognition. underscores the critical role of thoughtful archi- tectural design choices in achieving optimal performance, factor often neglected in the FL literature. Many existing FL solutions are tested on shallow or simple networks, which may not accurately reflect real-world applications. This practice restricts the transferability of research findings large-scale visual recognition models. Through an in-depth analysis of diverse cutting-edge architectures such as convolutional neural networks, transformers, and MLP-mixers, we experimen- tally demonstrate that architectural choices can substantially enhance FL systems' performance, particularly when handling heterogeneous data. We study 19 visual recognition models from five different architectural families on four challenging FL datasets. We also re-investigate the inferior performance of convolution- bassd architectures in the FL setting and analyze the influence of normalization layers on the FL performance. Our findings emphasize the importance of architectural design for computer vision tasks in practical scenarios, effectively narrowing the performance gap between federated and centralized learning. Our source code available https:://// //g hgtgggtt..b..../ com/ sarapieri/fed_eeeee git
Introduction
The growing focus on data privacy and protection [37 61] has sparked significant research interest in federated learning (FI.) as provides an opportunity for collaborative machine learning in many
domains, such as healthcare [16] 14
42 555 mobile devices [9), internet of things (IoTss) [53 25
and autonomous driving 440, [6]. In FL., data is kept separate by individual clients learn global model on the server side in decentralized way.
Despite the potential benefits of federated learning, the FL environments are highly non-trivial, the local datasets of the individual clients may not accurately represent the global data distribution. As each client is unique end user, the heterogeneity caused by uneven distributions of features, labels, or unequal amount of data across clients poses a serious obstacle [27). Therefore, the setting is inherently challenging since FL methods must accommodate both statistical diverseness and exploit similarities in client data. Typically, the non-independent and identically distributed (non-III) nature of the clients drastically affects the model's performance, resulting in lower accuracy and slower convergence than the counterparts trained with all the data collected in single location.
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

CIFAR-B
CelebA
201-
GLD-23K
Clapues i 10
Closses
Closses - 203
Clent
10k
of magaa/tentt Class Distribation
16
ofimagui torn
00g 1,000
Class Distribution
Images Samples
Images Samples
Class Distribttoon
Emages Samples
Class Distribution
Imagee Samples
RESNET EPICIENTNET CONV
SWIN VII SWINV2 DEIT
PMIXER RESMLIP GMLP
CONVMIXER COATNET MAXVIT
RANDFORMERR POOLPORMER
RIFORNER CAFORMER CONVEORMER
DENTITY TORMER
CNN
Transformers
MLP Mixers
Hybrids
Metal Formers
Figure I: Our study incorporates wide range of datasets, each presenting varying data heterogeneity levels, including Fed-ISIC2019 and GLD-23K, which are inherently federated datasets due their image acquisition and sensor mechanisms. Differently from prior literature, we extensively evaluate models from each visual architectural family. Additionally, we assess the effectiveness of four distinct optimization methods orthogonally to the architectural choice. Our findings empirically underscore the significant influence of architectural design choices in narrowing the disparity with centralized configurations and effectively handling data heterogeneity.
To this end, the problem of client heterogeneity received significant attention from the optimization
community 38 58 29 36
75 50 15).
In parallel to these developments in FL., visual recognition tasks have witnessed remarkable progress in recent years, primarily due to the advancements in deep learning models. These models have achieved state-of-the- art performance when trained on centralized datasets. However, when deployed an FL setting, these image recognition architectures often exhibit performance degradation. Recent
works have investigated the utilization of pretrained models to address the non-iid issue 4 (44] Qu et
al. (57) examined the robustness of four neural architectures across heterogeneous data, introducing the idea of tackling the problem from an architectural standpoint and encouraging the development of "foderated- "aaderated-frienlyy"" architcctures. The study suggests that self-attention- based architectures are robust to distribution shifts compared to convolutional architectures, making them more adept in FL tasks. In this work, we perform a comprehensive study across 19 visual recognition models from five different architectural families four challenging FL datasets. We also re-in investiggte the inferior performance of convolution- based architectures and analyze the influence of normalization layers on the model performance in non-III settings.
Contributions: This work strives to offer architectural design insights that enable immediate performance improvements without the need for additional data, complex training strategies, or extensive hyperparameter tuning. The key contributions of this work are the following:
We perform exhaustive experimental analysis comparing 19 different -tate-of-the-art (SOTA) models from the five leading computer vision architectural families, including Con- volutional Neural Networks (CNNs), Transformers, MLP-Mixers, Hybrids, and Metaformer- like architectures. Some of the architectural families we studied were never previously introduced in FL, demonstrating the feasibility of addressing data heterogeneity FL from an architectural perspective. These architectures were tested across four prevalent CV feder- ated datasets under the most common con-identicalness sources, such as feature distribution skew, label distribution skew, and domain shift in highly heterogeneous data. Our study, comprehensively illustrated in Figure I embodies step toward more efficient, practical, and robust FL. systems.
Our experiments in highly heterogeneous settings reveal that convolution operations are not inherently sub-optimal in FL applications and Metaformer-like architectures generally exhibit greater robustness in non-lIlD settings.
We illustrate that Batch Normalization (BN) adversely impacts performance in the heteroge- neous federated learning framework. To counteract this, we empirically demonstrate that

---

replacing BN with Layer Normalization is an effective solution to mitigate the performance drop. This study case underlines how architectural design choices can significantly influence FL. performance.
We conduct a study on complex networks with four different optimizers, establishing that the application of optimization methods does not yield substantial performance improvements in the context of complex architectures. We argue that altering the architecture, in practical scenarios, offers more effective and simpler-to-implement choice.
2 Related Works
Federated learning (FL) aims to train models in a decentralized fashion, harnessing edge-device computational capabilities to safeguard the privacy of client data. In this setup, clients retain their data on the device, coordinating with server to develop a global model. A distinct challenge in FL
is data heterogeneity, as client data often non-IID. The issue of addressing this heterogeneity has garnered significant focus from the optimization domain. To enhance the established aggregation method, FedAvg [49]. various strategies have emerged. These encompass optimization improvements for federated averaging such as FedAVGM [23], FedProx (38), SCAFFOLD (29]. and FedExP [26]
and methods manage data heterogeneity and imbalance like FedShuffle 21 FedLC 34 and
MooN 36
A complementary angle to explore is how the choice of models can elevate FL training outcomes. It has been observed that pretrained models can significantly counteract non-IID challenges 4 [54] Some previous studies have highlighted the impact of Batch Normalization (BN) on performance drops under heterogeneous scttings [39], and few works (22). [24] suggested replacing BN with Group Normalization. Our work is related to Qu et al.'s examination [57] of neural architecture in visual recognition robustness across heterogeneous data splits. Our experiments compare 19 cutting- edge models from diverse computer vision architectural families against demanding heterogencous datasets and different optimizers.
3 Optimization Methods for Federated Learning
In this section, we present brief overview of the optimization techniques utilized in our study to evaluate their effectiveness in handling data heterogeneity when applied in conjunction with complex computer vision architectures.
Federated Averaging (FedAVG) [49]: FedAVG was proposed by McMahan al. (2017) as the standard aggregation method for FL.. It offers a straightforward and effective approach where local models are trained individual devices, and their parameters averaged to update the global model. However, FedAVG faces challenges in scenarios involving non-IID and unbalanced data, which are prevalent FL settings.
FedAVG with Momentum (FedAVGM) 23 To address some of the limitations of FedAVG, the addition of momentum term has been explored. This technique enhances the standard FedAVG method by updating the global model at the server using a gradient- operates the mean of the client model updates. The inclusion of momentum facilitates efficient navigation through the complex optimization landscape and accelerates learning, particularly in the presence of sparse or noisy gradients.
server optimizer that
To handle the heterogeneity in data and device availability in FL, Li et al. proposed FedProx (Federated Proximal). This method introduces proximal term into the local loss function, preventing local models from deviating significantly from the global model. The proximal term acts regularizer, penalizing large deviations from the current global model. This regularization approach improves overall performance and enhances robustness in FL scenarios.
FedProx 38]
SCAFFOLD [29]: SCAFFOLD is stochastic algorithm designed to enhance the performance of Federated Averaging (FedAVG) by introducing a controlled stochastic averaging step. This step reduces the variance of gradient estimates by correcting the local updates. SCAFFOLD employs control variates (variance reduction) to address the "client-drift" issue local updates. Notably, SCAFFOLD requires fewer communication rounds, making it less susceptible to data heterogeneity
3

---

The TOD ArC Arcani TMM
THE Tigh
THe
Sebt
Candiion-
Pelformn
DuT6 SwIK3T
Move15
Confore-53 CAfamerdia
Sixi
Spid
Gaad
Splan
Spert
Toon C3PR
DaRS Pestiiooonnn CINNR
Data Dww.
(A) CNNS
(b) Trunsformes
(c) Hybrid
G6 M..--Misee
(e) MetaFormers
Figure 2: Model Accuracy Analysis across CIFAR-IO splits grouped by the architecture type. The presented results depict the accuracy performance of various models on distinct partitions of the CIFAR-10 10 dataset. Each split represents unique partition, progressing from an IID scenario (Split-1) to a non-IID scenario (Split-3). The models under comparison exhibit similar parameters, FLOOSS, and approximately matching ImageNet and centralized accuracies. Notably, Metaformer- like architectures, including Metaformers (e), Transformers (b), and MLP-Mixers (d), exhibit a comparatively smaller performance degradation on Split-2 and Split-3. The pest-performing models are CA-Former and ConyFormer, which are Metaformers with advanced token mixers.
or client sampling. Furthermore, the algorithm leverages data similarity among clients, leading to faster convergence.
4 Architectures for Computer Vision
In this section, we provide an overview of the architecture families studied in this work.
Convolutional-based models: We include Resnet [9] and EfficientNet [62 as the study of Qu al. [57). and extend it by adding a recently introduced architecture, ConvNeXt [45]. EfficientNet and Resnet consist of stack of convolutional layers, pooling layers, and batch normalization layers. ConvNext was conceived by modernizing ResNet-50 to pair with the performance of vision transformers. The key changes introduced at the component-level are: replacing the typical batch normalization with layer normalization. using inverted residual blocks and larger kernel sizes, and substituting the more common ReLU [ with the GeLU 20 nonlinearity.
Vision Transformers: ViTs were introduced to the computer vision community by Dosovitskiy et al. 13 after becoming the de-facto standard for natural language processing tasks. The Transformer architecture reconsiders the need for convolutions and replaces them with a self-attention mechanism.
ViTs work by splitting the image into non-overlapping patches treated tokens and re-weighting them based on a similarity measure between each token pair. The main differences with CNNs are the lack of inductive bias of the vanilla self-attention and the global receptive field capable of capturing long-distance dependencies as opposed to the local receptive field of convolutions. As representatives of this family of architectures, we include ViT (60), Swin (44] Swin V2 (43) and DeiT [66].
first proposed by Tolstikhin al. 664 It was the first architecture to show that although convolutions and self-attention are sufficient to achieve top performance, they not necessary. The MLP-Mixen take inspiration from ViT's of patches for input representation but fully use multi-layer perceptrons instead of the typical self-attention mechanism of Transformers. For our experiments, we make of the original MLP-Mixer (64), the ResMLP variation (65) and which uses a special gating unit to enable cross-token interactions.
MIP Mix-Mrxers: MLP-Mixen.
the gMLP 41
Hybrids: Many efforts have been made incorporate features from CNNs into Transformers and vice versa. We group this set of mixed models under the same 'Hybrids' category. Namely ConvMixer 677. CoATNet [8], and Max VIT [69]. ConvMixer combines transformer-lis patch design with convolution-basse layers to mix patch embeddings' spatial and channel locations. CoATNet vertically stacks convolution layers and attention layers. Finally, Max hierarchically stacks repeated blocks composed of multi-axia self-attention and convolutions.
MetaFormers: Although the success of transformer-basee architectures was longly attributed to self-attention, recent works 73 74 have started to identify the overall structure of the transformer block as the critical component in achieving competitive performances. Based on this hypothesis,

---

Metaformer is constructed from Transformers without specifying the design of the token mixers. If the token mixer is self-attention or spatial MLP, the architecture becomes a Transformer or an MI.P-Mixer.
Although most of the Transformers and MI.P-Mixers formally fall into the 'Metaformers' structure, we group in this family only the works published by the authors under this name. Within the networks selected, we have some working with simple operators in the token mixers, published with the explicit intent of pushing the limit of the Metaformer structure. Respectively, RandFormer, IdentityFormer (74), and PoolFormer 773 have a random matrix, an identity mapping, and a pooling operator as the token mixer. Moreover, ConvFormer [74] applies depthwise separable convolutions the token mixer, and CAFormer (74) further combines it with vanilla self-attention in the deep stages. We also add to this category RIFormer (70). architecture that optimizes the latency/accuracyy trade-off by using re-pan p--araametrization 12 to remove the token mixer without significantly compromising the performance.
5 Experimental Setup
This study aims to enrich the FL landscape by introducing a variety of cutting-edge architectures for computer vision. Within the FL framework, we conduct a comparative analysis of models that have similar parameters and FLOPS and roughly equivalent accuracy in centralized settings to the benchmark model, ResNet-50. Ensuring fairness, all the models under considcration fall within similar parameter range (21-31M) and floating point operations per second (FLOPS) range (4-6G) Each model exhibits a comparable centralized test accuracy on ImageNet-IK M...
Models:
Unless explicitly stated, all models are pretrained on ImageNet- IK and subsequently finetuned on the specified dataset.
Federated learning: In an effort to ensure that the insights derived from this study remain indepen- dent of the training heuristics, have standardized and simplified the training strategy across all architectures. Our approach involves performing FedAVG for a specific number of communication rounds reach convergence: 100 rounds for both the CIFAR-10 and Fed-ISIC2019 datasets, 30 rounds for the CelebA dataset, and 200 rounds for the GLD-23K dataset. With the exception of the GLD-23K dataset, where we conduct five local steps, each round of local training encompasses one local epoch. The training regimen employs the SGD optimizer with an initial learning rate of 0.03, cosine decay, and warm-up phase of 100 steps. We set the local training batch size at 32 and apply gradient clipping with unitary norm to stabilize the training process.
The numerical values presented in this paper represent the average results derived from three rep- etitions of the same experiment, each with different initializations, to ensure the robustness and reproducibilityy of our findings. Regarding optimization, model based on the set of values suggested by the respective authors in their original publications. The results reported in this paper reflect the optimal outcomes derived from the hyperparameter search. A more in-depth discussion is given in the supplementary material.
conduct hyperparameter tuning for each
Datasets: We conduct experiments four different datasets, CIFAR-10 31, CelebA [46], Fed- ISIC2019 (68 5 and Google Landmarks Dataset v2 (GLD-23K) 72). Across the experiments, all the images are resized to standard dimension of 224 x 224. The data heterogeneity of the different datasets used in our experiments is characterized in terms of Label Distribution Skew (LDS), Data Distribution Skew (DDS), and Feature Distribution Skew (FDS). The LDS is characterized by heterogeneous number of samples per class for each client. In the DDS, the amount of data per client varies. Lastly, the FDS is defined by varying image features per class across clients due to changes in acquisition sensors, image domains, user preferences, geographical locations, etc. (see supplementary materials for more details).
The CIFAR-10 dataset comprises 60,000 32x32 color images in 10 distinct classes representing common objects. Since CIFAR-10 is not inherently federated dataset, we artificially simulate three federated data partitions across clients following the approach outlined in [57. The original test set is retained as the global test set, and 5,000 images set aside from the training set for validation
'The only exception in terms of model size is MLPMixer-S// S/16. However, we have opted to include this model in its smallest configuration provided by the authors, as it constitutes the inaugural exemplar of MLP- P-Meeer architectures.
5

---

CIFAR-10
FED-ISIC2019
ARCHITECTURE
RESNET-50
CONVNEXI-T
CAFORMER-S18
RESNET-50
CONvNEXI-T
CAFORMER-SI8
FEDAVG 4U
69.9 1.87
93.5 0.36
97.1
67.6 1.59
78.6
83.1 0.49
FEDAVGM 58
80.2 + 0.49
93.4 + 0.48
97.0 + 0.14
66.2 + 2.16
79.0 4 0.52
83.8 + 0.37
FEDPROX 38
79.0 = 0.41
93.4 + 0.16
97.1 + 0.25
68.0 + 1.93
79.4 +: 1.13
83.0 = 0.79
SCAFFOLD
78.1 0.06
94.3 0.30
0.21
62.2 0.26
76.0
77.1 +: 0.61
Table I: Accuracy of ResNet-50, ConvNext-T, and CAFormer 11 and Fed-ISIC2019
across four different optimization methods. The average test accuracy of the model is displayed along with the standard deviation of three repetitions of the experiments.
purposes, resulting in a revised training dataset of 45,000 images. We employ the Kolmogorov- Smirnov statistic (KS) to simulate one Independent and Identically Distributed (IID) data partition, namely Split-1 (KS=0), ensuring balanced labels per client, and two non-IID partitions, Split-2 (KS=0.5) and Split-3 (KS=1), with label distribution skew. In Split-2, each client has access to four classes and does not receive samples from the remaining six classes. In Split-3, each client strictly sees samples from two classes only. Therefore, this is the most challenging partition, featuring pronounced class imbalance. Each data partition has five clients.
The CelebA dataset is large-scale face attributes dataset featuring over 200,000 high-resolution celebrity images. Each image is annotated with 40 binary labels that denote the presence or absence of distinct facial attributes. In this study, we employ the federated version of the CelebA dataset proposed by the LEAF benchmark 33. In line with the methodology outlined in [57], we test the models on a binary classification task, determining whether a celebrity is smiling. The dataset is partitioned across 227 clients, each representing a specific celebrity and collectively accounting for 1,213 images. The dataset presents significant challenges due to the large-scale setting involving numerous clients. Each client has access to only a handful of samples, five per client on average, and in some instances, data from only one class is available.
The Fed-ISIC2019 dataset contains skin lesion images collected from four hospitals. We adopt the partitioning strategy proposed by [63], which is based on the imaging acquisition systems utilized. Given that one hospital employed three distinct imaging technologies over time, the federated dataset is distributed across six clients, encompassing 23,247 images. The task involves classifying the images into eight melanoma classes with high label imbalance across clients. Additionally, the dataset exhibits quantity skew with disparity in the number of images per client. The largest client possesses more than half the data, whereas the smallest has only 439 samples.
The Google Landmark- V2 dataset is a vast collection with over million images of global landmarks. We utilize the federated GLD-23K partition, involving 233 smartphone users as clients contributing 23,080 images across 203 classes. These datasets exhibit a geographically influenced, long-tailed distribution, with some landmarks having many representations while others have very few, mirroring real-world disparities. Also, photographers images present unique distributional characteristics tied their habits and locations, resulting in large diversity and imbalance across clients.
Experimental Results
6.1 Optimizers and Complex Networks in Federated Learning
An effective learning system typically requires both an optimized architecture and an effective optimization technique to shape the learning process and resultant model. The federated learning community has focused considerably on optimization techniques to address challenges related to non-IID data and the need balance statistical diversity and leverage similarities in client data for central model training. However, most recent studies [48] rarely present results incorporating the latest architectures developed by the computer vision community. Many of these publications on vision tasks present their work on simple and shallow networks ranging from few fully connected layers to the ResNet-50 model. This discrepancy highlights significant chasm between the research focus of federated learning and sfate-of-the-a art computer vision architectures.

---

To evaluate the compatibility of advanced architectures with federated optimization methods, we consider the widely used ResNet-50 model in our study and extend it to more recent computer vision architectures. We compare different models with the baseline ResNet-50, combined with diverse optimization techniques across four federated datasets. Alongside ResNet, we employ ConvNext, proposed to narrow the gap of convolutional with transformer architectures, and CAFormer, a recently introduced model in the Metaformer family. We consider four the most popular optimizers in federated learning, namely FedAVG [49] FedAVGM (enhanced with momentum) [58), FedProx (38] and SCAFFOLD [29]. Results reported in Table can be summarized as follows. First, using the standard FedAVG replacing ResNet-50 with the above computer vision architectures significantly increases accuracy across all datasets, especially with the CAFormer architecture, suggesting that architectural modification can significantly boost performance. Secondly, combining optimizers with complex models does not enhance results, with most of the performance maintaining steady status, except for FedAVGM applied with ResNet on the CIFAR-10 dataset. In fact, for the Fed-ISIC2019 dataset, some of the optimizers proved to be detrimental to the performance of the network when compared to the simple FedAVG (e.g., SCAFFOLD for all architectures, and ResNet with FedAVGM and FedProx). The lack of optimization benefits across different datasets and architectures can be attributed to the complexity of modern deep learning architectures, such as Transformer-basee models, which are significantly more complex than models typically used in FL research, leading to a more challenging optimization landscape. Applying these optimization techniques to the tough non-convex optimization problems encountered during the training of modern deep neural networks remains an unresolved issue. Most FI. methods have demonstrated efficacy for certain types of non-convex problems under specific assumptions that might not be applicable to these complex models in practice [T0] 30 Our experiments reveal that the naïve ap- plication of these techniques, combined with advanced models, often leads negligible improvement.
This leaves with the unresolved question of how we can optimize federated learning per- formance for practical applications, bridge the gap between advancements in the computer vision and FL communities, and address the issue of closing the performance gap with centralized settings. Inspired by our findings, we inspect the benefits of architectural design choices. Replacing the architecture is an easy-to-implement strategy, given the wide array of pretrained models available and their proven effectiveness on various vision tasks. Moreover, cutting-edge architectures often include design elements that inherently facilitate learning, such as superior feature extraction, incorporation of prior knowledge (inductive biases), and regularization that could mimic the effects of federated optimization techniques. In the following Section 6.2 we explore the possibility of leveraging the inherent strengths of different computer vision models to enhance performance and naturally bridge the non-IID gap in federated learning.
6.2 Tackling the Heterogeneity Challenge with Architectural Design
We comprchensively compare network architectures from diverse network families, such as CNNs, Vision Transformers, Hybrids, MLP-Mixers, and Metaformers, with a particular focus on how each architecture family performs against each other. Tables 2 and show our results on CIFAR, CelebA Fed-ISIC2019, and GLD-23k datasets.
Architectural Comparison on CIFAR-10: On CIFAR-10, 10, all models show performance degradation compared to the ideal centralized training, with most of the architectures displaying significant reduction in accuracy as we progress to the non-lID splits (see Figure 2 Table 2 shows the performance of each model across all splits and the centralized version. The key observations from this study are summarized below.
Convolutional models exhibit the overall highest performance degradation among all families. ResNet [19] and EfficienNet [2] are found to be non-robust in heterogencous settings, with loss of 36 and 17.1 points, respectively, observed in Split-3. While ConvNeXt [45] shows improvement over its predecessors, it still falls short of achieving top results when compared to other model families.
Convolutional Networks:
Transformers and Metaformers: Transformers consistently demonstrate robust performance, par- ticularly in non-Ill) data partitions, exhibiting only marginal average degradation of 1.7 points. Notably, ViT (60 achieves remarkable resilience, attaining an aceuracy of 96.8% and experiencing minimal drop of merely 0.8%, even in the most challenging split. These findings corroborate the
7

---

MODEL
CENTRAL
SPLIT-1
SPLIT-2
SPLIT-3
RESNET-50 19
95.9 0.21
96.8 2.17
92.7 0.25
59.9 (L 36.0)
OFFICIENTNET-BS
62
97.6
97.8 0.06
95.1 0.06
80.5 0.92 (L
CONVNEXT-T 45
0.15
97.6
96.6 0.21
93.5 (J 3.7)
SWIN-T 144
97.6 0.15
97.9 0.10
97.6 0.06
9..7 ()
1.9)
VIT-S 60
07.6 0.06
0.10
98.2 0.12
9618 (J 0.8)
SWINV2-T
96.4 0.00
97.4
96.4 0.15
93.7 (J
2.7)
DEIT-S 66
96.8 = 0.12
97.8 0.06
97.2 0.12
95.2 (J
1.6)
CONVFORMER [74
97.9 0.12
98.4 0.12
98.0 0.12
4 (L 1.2)
CAFORMER-SS 18 74
98.1 + 0.18
98.5 = 0.06
98.3 + 0.06
97.1 (4
1.0)
RANDFORMER S36
74
9.17
95.8 0.10
94.6 0.06
0.33 (L 4.4)
IDENTITYFORMER-SB (74
95.3 35.3
95.6 0.12
94.6 0.12
90.8 () 4.5)
POOL.OORMER-SA 73
97.1 +
97.4 0.10
96.9 0.06
95.2+ () 1.9)
RIFORMER- 77
95.9 0.08
96.6 0.16
95.5 0.15
93.0 ( 2.9)
MLPMIXER-S/II 6/16 [64
94.8 0.10
96.5 0.12
95.6 = 0.06
92.7 () 2.1)
RESMLP-S24 [65]
96.2
97.2 0.10
96.7 0.06
94.1 ( 2.1)
GMLP-S 41
96.9 0.26
98.0 0.06
97.6 0.12
96.1 (J 0.8)
CONVMIXER- 768/32 67
97.4 0.12
97.3 0.06
94.4 0.15
74.1 1.7 (L 23.3)
COATNET-O 8
97.7 0.15
98.3 0.15
97.5 0.06
94.2 1 0.06 (L 3.4)
MAXVIT-T 69
98.0 1 0.10
98.6 - 0.10
97.3 1 0.12
92.0 1 1.15 (1 6.0)
Table 2: Performance of different types of models across all splits of CIFAR-10. The average test accuracy of the model is displayed along with the standard deviation of the experiments. Split-3 shows the degradation compared to the centralized version of the training.
earlier observations made by Qu et al. [57] regarding the robustness of VIT models against data heterogeneity. However, Qu et al. attribute the superior performance of Transformers, compared to convolutional networks, to the robustness of self-attention in mitigating distribution shifts. In contrast, the emergence of the Metaformer family, employing various token-mixers including convolution operation, suggests that the remarkable proficiency exhibited by Transformers may be primarily attributed to the underlying architectural design of Metaformers, rather than the exclusive reliance on self attentienoo in models like VIT. Indeed, even Metaformers employing basic token mixers, such as RandFormer with a random token mixer, deliver remarkably satisfactory performances. On the challenging Split-3, CAFormer 774 garners the highest result, achieving accuracy of 97.1 % and delivering exemplary performance across all splits, followed by fully convolutional token mixer (without self-attention) based ConvFormer 774]. which is also par with ViTs.
MLP-Mixers: As for MLP-Mixers, on average, their performance on the Split-3 is close to those of Transformers. The best is the gMLP [41 with 0.8% of degradation (second only to ViT) and an accuracy of 96.1% Finally, Hybrid models achieve reasonable performance, but their results are far behind those of the other families. It is worth noticing that most models achieve higher accuracy in Split-1 compared to the centralized setting. We conjuncture that this due to a regularizing effect caused by each client's 's access to the IID data partition (with uniform distribution over 10 classes).
Architectural Comparison on CelebA, Fed-ISIC2019 and GLD-23k: The results for CelebA, Fed-ISIC2019 and GL.D-33k are reported in Table
CelebA: The highest accuracies for CelebA are the ones of Swin 44 (89.0) and ConvFormer 74 188.1), ), followed by MLP-Mixer [64] and CAFormer [74]. Again, all Metaformer-lic architectures (encompassing Transformers and MLP-Mixers) achieve the best results, far surpassing those CNNs and Hybrid models, whose average performance is 84 and 74.9, respectively.
GLD-23K: On the GLD-23K, also observe that ViT remains among the perthrsine
networks. However, Metaformers such as CaFormer and ConvFormer do not yield high performance in previous datasets.
Fed-1SIC-2019: Finally, on the Fed-ISIC-2019 dataset, which naturally federated (due to the differences in the image acquisition systems across clients), we find CAFormer and VIT as the best-

---

MODEL
NORM
MAIN OPERATION
CELEBA
GLD-23K
FED-ISIC2019
RESNET-5O 19
BN
CONY
84.9 1.48
54.3 1 0.48
67.6 1 1.59
BFFSCIENTNET-B5
BN
CONY
79.5 0.50
51.9 1.21
70.0 1: 1.64
CONVNEXI-T 45
LN
CONV
87.5 = 0.87
65.5 + 0.57
78.6 + 0.29
SWIN-T 44
LN
SA
89.0 0.79
72.1
81.9
VIT-S 60
LN
SA
87.3 0.553
76.6 0.69
82.3 0.99 0.95
SWINV2-T
4R
LN
SA
86.8 = 0.62
74.4
81.7 + 0.50
DEIT-S 66
LN
SA
87.4 = 0.60
69.1 + 1.24
82.3 +: 0.59
CONVFORMER-SI8
LN
CONV
85.1 0.42
53.8 1.11
81.1 1.54
CAFORMER S18 74
LN
SA+CONV
87.5
57.8
83.1 + 0.49
RANDFORMER S36
774
LN
CONV
83.9 0.35
56.3 0.10
77.3 1 0.23
IDENTITYORRMERSS
LN
CONV
0.49
56.0 2.07
PO0M--ORM 73
LN
CONV+POOL
0.61
55.8 1.41
0.29
RIFORMER-S36 70
LN
CONV
87.2 0.37
19.40.220
81.9 1.37
MLPMIXER-S/II 66
LN
CONV
87.9 1 0.51
71.3 1: 0.32
80.5 1.62
RESMLP-S24
CONV
87.0 0.35
64.8 0.59
81.1 0.45
GMLPPS 4
LN
CONV
86.6 0.31
67.4 0.16
79.9 1.60
CONVMIXER-768//1 678
BN
CONV
56.5 = 1.07
45.0 = 0.76
58.6 + 2.02
COATNET-O
BN+LN LN
SA+CONY
82.2 1.66
70.4 0.25
66.0 3.40
MAXVIT-T
SA
86.1 0.72
71.7 + 0.90
70.1 1.27
Table Accuracy of the model families on CelebA, GLD-23K, and Fed-ISIC2019, For each model, we report the type of normalization layer, either LayerNorm or BatchNorm (LN/BN), and main operation employed (Convolution/Self- -tteenfion. The mean accuracy of all repetitions is displayed along with the standard deviation.
performing models (83.1 and 82.3 accuracy). In fact, we can see significantly superior performance from all models following the Metaformer structure, indistinctly of their use of convolutions and/or self-attention their basic block structure.
In summary, our comprehensive experiments on 19 state-of-the-art architectures demonstrate that the choice of architecture and its components play noteworthy role in tackling performance degradation caused by data heterogeneity. As matter of fact, the experiments indicate that selecting an appropriate architecture design can have a greater impact on closing the gap with centralized training than the choice of the optimizer. For example, Table[[ shows that CAFormer with simple FedAVG outperforms all other models even when using more advanced optimization methods.
6.2.1 How architectural design improves performances NNorallization Layer
Based the results observed in the previous sections, further investigate the performance reduction by delving into the contribution of specific architectural components. Our objective is to identify some key structural elements to achieve stronger performances in heterogeneous settings.
As shown Table 3 models making use of Batch Normalization (like ResNet, EfficientNet, and ConvMixer) are consistently outperformed by models with Layer Norm on Split-3. This insight sheds light on why older convolutional (such as ResNet EfficientNet) models have such high degradation. On the contrary, ConvNext, which uses Layer Norm, reaches satisfactory performance, contradicting the initial conclusion of (57) that convolutions are inherently problematic in the federated learning scenario under non-III conditions. Moreover, observe how ConyFormer, model without Self- attention based blocks (based primarily on convolutions) but with a MetaFormer-likk structure and Layer Norms, regularly reaches top performance even under the most challenging data partitions.
To further support our claims, we perform experiments with three different models, namely Pool- Former, CoAtNet, and ResNet. Each model trains with a variant using Batch Norm and another using Layer Norm. Lastly, we include training scheme with FedBN 39 (a method developed specifically to deal with the performance degradation caused by the Batch Norm layer). Results (displayed in Table 4 show a trend indicating the superiority of Layer Norm over Batch Norm, even when using a tailored aggregation method for the latter (c.g., FedBN 39

---

VARIANT
CENTRAL
SPLTT-3
Sp112
COATNET-O BN
+ 0.04
20.9 0.45
76.6)
COATNET-O
97.7 0:15
94.2 0.06 3.4)
100
POOLFORMEER FORMER-SRRS BN
0.33
0.32 43.7)
POO.PORMR--S221
0:009
4.1)
RASNET-50 BN
0.21
69.9 + 1.87 [1 36.00
RESNET-50 LN
26.6)
RESNET-50 FEDEN 39
90.1
59.6 1: 1.40 (6.5).
Table 4: Accuracy of ResNet-50, CoAtNext- O, and PoolFormer-S11 with BN and LN in the central and the Split-3 version of CIFAR- 10, The original CoAtNet-O uses BN and LN in convolutional and self-attentie blocks, respectively. i LN layers have no pretraining.
Ppellfommet Pooi-omme--
Coltni
Calrnas der
Expatnam
Figure 3:
Weight divergence with respect to models' centralized version on CIFAR-IO. The of BN leads to larger weight divergence.
Finally, as in [76]. we measure and report the weight divergence of the models with respect to the centralized training. Figure T
shows the increase of divergence for the models using BN their LN counterparts, supporting the selection of Layer Norm over Batch Norm as an architectural choice for federated learning.
7 Limitations
Our research primarily explores visual recognition tasks within the federated learning domain, and we have not explored potential downstream tasks, such as object detection and segmentation. It's worth noting that visual recognition models often serve backbones for extracting input image features in downstream tasks, thereby suggesting that our observed performance trends could likely extend to these downstream tasks. We plan to address this research gap in future studies.
Furthermore, our approach did not design a completely new model from scratch. Instead, we demon- strated our results utilizzig pre-existing models and components. Our study also underscores the limitations employing established optimizers (e.g., FedAVG, FedAVGM, FedProx. SCAFFOLD) along with modern network architectures. However, we abstain from plunging into the creation of innovative optimization techniques. Instead, we focus on evaluating diverse architectural families on many practical federated learning datasets and optimizers. As a result, our investigation leaves an intriguing avenue for future research, integrating our findings to develop dedicated architecture and optimizers tailored specifically for federated learning (FL) scenarios. By exploring this direction, anticipate the potential to further enhance the performance and efficiency of FL systems.
Conclusion
This research expands the architectural domain in federated learning, emphasizing architectural design for computer vision applications under data heterogeneity. Our experiments assess various state-of-the-art models from five architectural families across four federated datasets, demonstrating the potential of addressing FL data heterogeneity architecturally. We show that Metaformer-like architectures exhibit superior robustness non-IID settings. contrast existing studies, we observe that convolution operations are not necessarily sub-optimal choice for architectures in FL applications and highlight the negative impact of Batch Normalization layer on the model performance. Our study also shows that architectural alterations offer an effective and more practical solution to handle data heterogeneity non-IID settings.
Acknowledgements
This work is partially supported by the MBZUAI-WIS Joint Program for Al Research (Project grant number- WIS POO8).
10

---

References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint wtXiv2211.04000 2021.
[2] Daniel Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marquee, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, et al. Flower: A friendly federated learning framework. 2022.
[3] Sehastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv: 1812.01811110099 2018.
[4] Hong- You Chen, Cheng- Hao Tu, Ziwei Li, Han- Wei Shen, and Wei-Lun Chao. On pre-training for federated learning. arXiv preprint arXiv: 14488, 2022.
[5] Noel CF Codella. David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo. Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: challenge the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168-172. IEEE, 2018.
[6] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
[7] Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alícia Barreiro, Allan C Halpern, Susana Puig, et al. Ben20000: Dermoscopic lesions in the wild. arXiv preprint urXiv:1908.022. 2019.
[8] Zihang Dai. Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021.
[9] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- 10 is not imagenet or c0far-10. arXiv preprint arXiv:1810.035005 2018.
[10] Aaron Defazio and Léon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. Advances in Neural Information Processing Systems, 32, 2019.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[12] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13733-13742, 2021.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 19299, 2020.
[14] Haya Elayan, Moayad Alogaily, and Mohsen Guizani. Deep federated learning for iot- based decentralized healtheare systems. In 2021 International Wireless Communications and Mobile Computing (IWCMC), pages 105-109. IEFF, 2021.
[15] Liang Gao, Huazhu Fu, Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Fedde: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10112-10121, 2022.
11

---

[16] Pengfei Guo, Puyang Wang, Jinyuan Zhou, Shanshan Jiang, and Vishal M Patel. Multi- institutional collaborations for improving deep learning- based magnetic resonance image recon- struction using federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2423- 2432, 2021.
[17]
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007. 13518, 2020.
[18) Chaoyang He, Alay Diliphhai Shah, Zhenheng Tang, Di Fanl Adarshan Naiynar Sivashun- mugam, Keerti Bhogaraju, Mita Shimpi, Li Shen, Xiaowen Chu, Mahdi Soltanolkotabi, and Salman Avestimehr. Fedev: a federated learning framework for diverse computer vision tasks. arXiv preprint arXiv:2111.11000 2021.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08411 08411, 3, 2016.
[21] Samuel Horváth, Maziar Sanjabi, Lin Xiao, Peter Richtárik, and Michael Rabbat. Fedshuffle: Recipes for better use of local work in federated learning. arXiv preprint 13169, 2022.
[22] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pages 4387-4398. PMLR, 2020.
[23] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.063: 2019.
[24] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pages 76-92. Springer, 2020.
[25] Ahmed Imteaj. Khandaker Mamun Ahmed, Urmish Thakker, Shiqiang Wang, Jian Li, and M Hadi Amini. Federated learning for resource-constraing constrained iot devices: Panoramas and state of the art. Federated and Transfer Learning, pages 7-27, 2023.
[26] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. arXiv preprint arXiv:2301.09604 2023.
[27] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Ar- jun Nitin Bhagoji. Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and Trends Machine Learning, (4(1-2):1-3 1-100, 2021.
[28] Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebas- tian U Stich, and Ananda Theertha Suresh. Breaking the centralized barrier for cross-device federated learning. Advances in Neural Information Processing Systems, 34:28663-28676. 2021.
[29] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143, PMLR, 2020.
[30] Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of existing momentum schemes for stochastic optimization. In 2018 Information Theory and Applications Workshop (ITA), pages 1-9, IEEE, 2018.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

[32] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
[33] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 36(11)2227822224, 1998.
[34] Hunmin Lee and Dachee Seo. Fedlc: Optimizing federated learning in non-iid data via label- wise clustering. IEEE Access, 2023.
[35] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEÉ international conference on computer vision, pages 5542-5550, 2017.
[36] Qiabin Li, Bingsheng He, and Dawn Song. Model-contrasiive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 10713-10722, 2021.
[37] Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 2021.
[38] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-4500 2020.
[39] Xiaoxiao Li, Meirui Jiang, Xiaofci Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.
[40] Xinle Liang, Yang Liu, Tianjian Chen, Ming Liu, and Qiang Yang. Federated transfer reinforce- ment learning for autonomous driving. In Federated and Transfer Learning, pages 357-371. Springer, 2022.
[41] Hanxiao Liu, Zihang Dai, David So, and Quoe V Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 34:9204-9211 2021.
[42] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng- Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 1013-1023, 2021.
[43] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 12009-12019, 2022.
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[45] Zhuang Liu, Hanzi Mao, Chao-Yuaa Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
[46] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[47] Bing Luo, Xiang Li, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Cost-effective federated learning in mobile edge networks. IEEE Journal on Selected Areas in Communications, 39(12):3606-36) 3621,, 2021.
[48] Xiaodong Ma, Jia Zhu, Zhihao Lin, Shanxuan Chen, and Yangjie Qin. A state-of-the-art survey on solving non-iid data in federated learning. Future Generation Computer Systems, 135:244-250 2022.
13

---

[49] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication cenmumication-cfficient efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[50] Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip:
Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning, pages 15750-15769. PMLR, 2022.
[51] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
[52] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[53] Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, Jun Li, and H Vincent Poor. Federated learning for internet of things: A comprehensive survey, IEEE Communications Surveys & Tutorials, 23(3):1622-1) 1658, 2021.
[54] John Nguyen, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? ex- ploring the impact of pre-training and initialization in federated learning. arXiv preprint arXiv:2206. 15387, 2022.
[55] Sangjoon Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, and Jong Chul Ye. Federated split task-agnostic vision transformer for covid-19 cxr diagnosis. Advances in Neural Information Processing Systems, 34:24617-24630, 2021.
[56] Shiva Raj Pokhrel and Jinho Choi. Federated learning with blockchain for autonomous vehicles: Analysis and design challenges. IEEE Transactions on Communications, 6809)4473-4476. 2020.
[57] Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei- Fei, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10061-10071, 2022.
[58] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
[59] Prasun Roy, Subhankar Ghosh, Saumik Bhattacharya, and Umapada Pal. Effects of degradations on deep neural network architectures. arXiv preprint arXhv:1807.101000 2018.
[60] Gilad Sharir, Asaf Noy, and Lihi Manoik-Manor. An image is worth 16x16 words, what is video worth? arXiv preprint arXiv:2103.13915. 2021.
[61] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310-1321, 2015.
[62] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[63] Jean Ogier du Terrail, Samy-Safwar Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint arXiv:2210.04620, 2022.
[64] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mip architecture for vision. Advances Neural Information Processing Systems, 34:24261-24272, 2021.
14

---

[65] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
[66] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficiees image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[67] Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792. 2022.
[68] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[69] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204.01697 2022.
[70] Jiahao Wang. Songyang Zhang, Yong Liu, Taiqiang Wu, Yujiu Yang. Xihui Liu, Kai Chen, Ping Luo, and Dahua Lin. Riformer: Keep your vision backbone effective while removing token mixer. arXiv preprint 2rXiv:2304.05659, 2023.
[71] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020.
[72] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2575-2584, 2020.
[73] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10819-10829, 2022.
[74] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. arXiv preprint arXiv:2210. 13452, 2022.
[75] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. International Conference on Machine Learning, pages 26311 1-3299. PMLR, 2022.
[76] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv: 00582, 2018.
15

---

Supplementary Material
9 A Re-look into existing Federated Computer Vision Approaches
In this section, we explore the landscape of computer vision techniques in existing literature. We begin by examining foundational works from the literature and the computer vision architectures they adopt. Our comparative assessment reveals that our approach incorporates several state-of-the-an vision architectures, previously unexplored in federated learning contexts. Furthermore, we contrast commonly used datasets with those we have chosen for our study. Notably, while our methodology encompasses a range of architectures, we also feature five datasets, four of which are intrinsically federated.
VINUE
VISION
PEDAVG
AISTATS'1 17
MNIST. CIPAR
PEDAVOM
ARXIV
CIFAR-TO
FROP 13
MLSYS'SR
LOGNTIC REGRESSION
MNIST: FEDMNIST MINIST
SCAFFOLD
ICME 20
2MLP
PEaNoVA 71
NEBRIPA 20
LocisTiC REGaRSSION, VGG
SYNTHETIC CIFAR-IO
MOON 36
CVPR
6UNK, KISNET
CIFAR- Tİ. CIFAR. TONL VINY-RRAGENET
TEDDYN
ICLR
BCNN
MNIST, EMNIST, CIEAR CITAR TW0
TCLRET
RESNET, SMLF
EMIIST
FEARN
SVHN,
MNIST
MIME
NEBRTPS
LOuBVIC RESSIO, JENN MMPP
EMNIST
[EDLC 14
1CME22
RESNET
SVIN, CIFARIO AND CIFARIOO, IMAGENET SUBSET
PROXSKIP 50
ICME/22
LOGISTIC REGRESSION
LIBSVM- "WSA"
PEnIX: LS
CVPR
LOGISTIE REGRESSION, MML
MNIST, FASHIOX MNIST. CIFARID, CIFARTOR EMNIST, TINY IMAGENET, SYNTHETIC
2CNN, RESNET
PEDSHUMLE 21
TMLR
RESNET
TIFAR-10U EMNIST, CIFAR- CIFAR-100 CINIC-10
FRnFXP 26
ICER 23
4CNK, RESNET
TEDML
NEUXIPS WORKSHOP 20
LOGISTIC REGRESSION, 4CNN
PEDMNIST. TEOMNI SENTHETIC
RESNET, MORILENET
CIFAR-10, CIFAR-1O0 CINIC-
FLOWER
ARXIV 20
SUNN, RESNET
FEMNIST MIFAR IMAGENET
FEDCY 15
CVPR 21
EPICIENTNEL MOBILENEL VIT
CIEAR 100, GLD 23k
QU BT AL.
CXPR
RESNET, EMIHENTNET, VII,
SWIN SWIN
RETINA, CIFAR-TII CELIBR
SKEWSCOUT
1CML/20 ECCV
ALEXNET, GooGLENeT, LENET, RESNET
CIFAR-I0 10, PLACK MANMAL, IMAGENET
NETVE SUNN
CIFAR CIFAR
ROSNERSO, EmICINNNUT, CONYNEXT SWIN, VIT. SWIV22, DErT.
RIFORMER-$16 CONY FORMER, CAFORMER, RANDFORMER, FOENIIYYORMMRR POOLFORMER MLPMIXER, RESMLE, OMLP. CONVMIXEE, COÂTNET, MANVIT
OURS
CIFAR 10, CELEBA, PED ISIC2019, GLD 23K. PACS
Table 5: Literature review and comparative analysis of various federated learning methods evaluated on computer vision benchmarks. Each method is classified by the publication venue, the vision architecture utilized, and the vision-related datasets employed for testing. Furthermore, we provide specific details for shallow models on the complexity of the network architectures in terms of the number of convolutional (CNN) and fully connected (MLP) layers.
DATASET
IMAGES
CLASSES
CLIENTS
NATURALLY FEDERATED
CIFAR-10
31
60,000
10
No
CELEBA 46
200,288
2
9,343
YES
FED-ISIC2019 68
23.247
8
YES
GLD-23K 772
23,080
203
233
YES
PACS K5
9,991
7
YES
TINYIMAGENET 33
120,000
203
No
MNIST 33
70,000
10
250
YES
EMNIST 6
814.255
62
500
YES
FEMNIST 3
805,263
62
3500
YES
SVHN [52
600,000
10
No
CINIC 9
270,000
10
No
SYNTHETIC DIGITS 559
12,000
10
No
Table 6: Overview of frequently utilized datasets for visual recognition within the field of federated learning. The number of clients indicated for non-naturally federated datasets are subject to each author's s unique artificial partition.
16

---

9.0.1 Data Heterogeneity Dataset Analysis
In our experiments, we examined data heterogeneity across several datasets, focusing on Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS). The LDS denotes an uneven distribution of samples across classes for each client, while DDS reflects variations in the volume data per client. On other hand, FDS arises from differences image features per class across clients, attributable to factors like varied acquisition sensors, image domains, user preferences, and geographical locations.
Specifically, for LDS, we tested CIFAR- 10 and created three distinct splits with escalating label skew. This skewness was measured using the KS statistic, enabling comprehensive exploration of how label imbalances affect our model's 's efficacy.
For Data Distribution Skew (DDS), primary dataset GLDK, which exhibited the most pronounced variations in the amount of data per client. This helped us delve deep into the challenges and implications of data volume discrepancies across clients in federated setting.
In terms FDS, we incorporated inherently federated datasets like GLD-23K and Fed-ISIC2019 which exhibit this skew. The GLD-23K dataset spans multiple geographical areas, while Fed- ISIC2019 was gathered using diverse medical equipment. offering a broad spectrum of feature distribution skews. Additionally, delve deeper into the PACS dataset in Sec. [10.4] which showcases significant domain variations. For this particular study, our federated divisions allocated unique target domains: Photos, Sketches, Cartoons, and Paintings.
DAIASET
# CLASSES
# CLIENTS
# IMAGES
LDS
FDS
DDS
CIFAR-I0
SPLIT-1
10
60,000
X
X
SPLIT-2
10
60,000
X
X
SPLIT-3
10
5
60,000
CELEBA
2
227
1,213
FED-ISIC2019
8
23,247
GOOGLE LANDMARK- V2
203
223
23,080
V lVV
PACS
9,991
X
Table 7: Summary of datasets and setups. We assess the data heterogeneity of the datasets in terms of three key aspects: Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS). We utilize grading system to quantify the level of skewness, with options ranging from "none" (X), "mild" (V), "moderate" " (Vv), to "severe" (VVV).
10 Additional Experiments
In this section, we delve into supplementary experiments. These encompass optimization strategies applied to complex network results [O.T] further analysis on normalization layers 10.2, the inclusion of the PASCs dataset 10.4. and study of convergence speed 10.5
10.1
Optimization Methods with Complex Architectures
CELEBA
ARCHITECTURE
RESNET-50
CONVNEXT-T
CAFORMER -188
FEDAVG 49
84.9 1
87.5 1 0.87
88.1 1 0.61
FEDAVGM 58
84.1 1 1.51
87.4 1. 1.15
87.4 1 1.70
FEDPROX 38
85.0 + 1.48
87.5 = 0.88
87.5 + 0.37
SCAFFOLD 84.4
86.8 = 1.33
89.0 + 0.40
Table 8: Accuracy of ResNet-50, ConvNext-T, and CAFormer on CelebA across four different
optimization methods. The average accuracy of the model is displayed along with the standard deviation of the experiments. We can appreciate how changing the optimizer achieves virtually negligible improvements, unlike changing the architecture, which yields significant boost.
17

---

10.2
Normalization Layer
In this section, we delve into extended experiments concerning normalization layers. We begin by re- inforcing our earlier findings, showcasing that Layer Normalization outperforms Batch Normalization (BN) when tested on an additional dataset, Fed-ISIC2019, as detailed Table
VARIANT
FED-ISIC2019
COATNET-0 BN COATNET-0 LN
68.5 + 0.70 70.4 + 0.25
POOLFORMER- S12 BN
64.2 1 0.53
POOLFORMER-SI LN
83.2 + 0.63
RESNET-SO BN
54.3 1: 0.48
RESNET-SO LN
62.5 1.50
RESNET-50 FEDBN 39
65.2 + 1.24
Table 9: Accuracy of ResNet-50, CoAtNext- 0, and PoolFormer-S -Snmmer-S112 with Batch (BN) and Layer (LN) Normalization across the Fed-ISIC2019 datasets. Replacing BN with LN improves performance. * The original CoAtNet-O) uses BN and LN in convolutional and self-attention blocks, respectively. f LN layers have no pretraining.
Exploring alternatives BN, and extending beyond LN, we incorporated Group Normalization (GN) in our assessment. We replicated our results on CIFAR-10, encompassing CoAtNet, PoolFormer, and ResNet- in the original manuscript-while augmenting it with GN results (averaging three repetitions per experiment along with standard deviations). In this context, CoatNet and PoolFormer exhibited more pronounced performance drop under GN compared the LN configuration. In- triguingly, ResNet yielded a converse pattern, achieving superior outcomes with GroupNorm as opposed LayerNorm. This trend remained consistent while performing the weight divergence analysis across Split-2 and Split-3. These experiments indicate that LayerNorm is more favorable choice when paired with advanced models. Indeed, the weight divergence in figure 4 serves compelling indicator that the incorporation of I.N offers safeguard against drops in performance in heterogeneous partitions, with LayerNorm having inferior weight divergence wrt BatchNorm in all cases and best results for CoAtNet and PoolFrrmer.
Variant
CIFAR-10 Central
CIFAR-IO Split-3
PoolFormer-S12 BN
95.3 0.35
51.5 + 0.32 (1 43.7)
PoolFormer-S12 GN
94.1 + 0.25
81.5 + 0.29 (1 12.6)
PoolFormer-S12 LN
95.6 + 0.09
91.5 0.13 (1 4.1)
ResNet-50 BN
95.9 + 0.21
59.9 + 1.87 (1 36.0)
ResNet-50 GN
95.8 + 0.19
89.3 + 0.59 (1 6.5)
ResNet-50 LN
92.8 - 0.16
72.2 + 3.06 (L 26.6)
CoAtNet-O BN
97.5 + 0.04
20.9 + 0.45 (1 76.6)
CoAdNet-O GN
94.3 0.39
82.6 + 2.79 (1 11.7)
CoAtNet-O LN
97.7 + 0.15
94.2 + 0.06 (1 3.4)
Table 10: Normalization layer comparison. Accuracy of the models with batch normalization (BN), layer normalization (LN) and group normalization (GN) in the central and the Split-3 version of CIFAR-10. 10. * The original CoAtNet-O uses BN and LN in convolutional and self-attention blocks. respectively. t layers have pretraining.
18

---

140 Spi:2
Spr:3
120
100
Weight
Tedffrrme-tN Posiformnn-M
PoaTormee-0 SN
IOAINALUNN
CoAthel-GN1 Experiment
CoAlher-- SN
Reane:-GNr
Rauuee ON
Figure 4: Weight divergence analysis on CIFAR-IO. Weight divergence with respect to models' centralized version on CIFAR-10. The use of BN leads to larger weight divergence. Complex models tend to pair well with LayerNorm, achieving even smaller divergence than when paired with GroupNorm layers.
10.3 Centralized Results
GLDD-33
FED-ISIC2019
MODEL
CENTRAL
SPLIT
CENTRAL
SPLIT
RESNET-SO 50
19
79.6 + 2.09
54.3 + 0.48
88.2 : 0.15
67.6 + 1.59
EFFICIENTNET-BS 662
85.3 + 0.30
51.9 + 1.21
89.7 0.41
70.0 : 0.64
CONVNEXT-T 455
83.7 + 0.45
65.5 = 0.57
88.1 0.91
78.6 0.29
SWIN-T 44
81.9 + 0.32
72.1 + 0.67
90.8 = 0.31
81.9 = 0.42
VIT-S
60
79.4 0.34
76.6 + 0.69
90.3 = 0.17
82.3 - 0.95
SWINV2-T
43
75.7
74.4 + 0.02
88.1 = 0.58
81.7 = 0.50
DEIT-S 66
78.7 1.04
69.1 + 1.24
89.3 = 0.36
82.3 + 0.59
CONVFORMER-ST S18 [74
83.7 + 0.49
53.8 + 1.11
90.3 = 0.45
81.1 1.54
CAFORMER-SII S18 774
83.6 + 0.31
57.8 0.54
90.7 = 0.22
83.1 + 0.49
RANDFORMER- S36
[74]
81.4 0.27
86.9 = 0.23
77.3 0.23
IDENTITYFORMER S36 [74
81.3 0.95
56.0
87.0 = 0.38
76.9 - 1.43
POOLFORMER-SSO [73
83.4 1.33
55.8 1.41
88.5 - 0.36
79.6 + 0.29
LIFORMER-S34 [70]
80.7
69.4 0.20
89.5 = 0.08
81.9 + 1.37
MLPMIXER- S/11
664
77.5 0.91
71.3 + 0.32
88.6 = 0.50
80.5 1.62
RESMLP-S2 665
73.4 1.03
64.8 + 0.59
89.1 + 0.23
81.1 0.45
GMLP-S 41
76.5 + 0.71
67.4 + 0.16
88.6 + 0.52
79.9 + 1.60
85.6 + 0.40
45.0 + 0.76
87.3 + 0.20
58.6 + 2.02
COATNET-0 8
79.2 + 1.03
70.4 + 0.25
87.6 + 0.17
66.0 + 3.40
MAXVIT-T 69
83.4 + 0.29
71.7 + 0.90
91.0 0.28
70.1 1.27
Table 11: Comparative analysis among different architectural families in both federated and central- ized settings. Notably, each model experiences drop in accuracy when transitioning from centralized to federated learning settings. Importantly, high performance the centralized setting does not necessarily translate comparable success in the federated split. The table particularly emphasizes the robustness and superior performance of Metaformer-like architectures the federated setting. These architectures demonstrate minimal drop in accuracy relative to their centralized counterparts and high overall accuracy results, underlining their adeptness at handling data heterogeneity.
19

---

10.4 PACS: Analysis on Domain Shift
To simulate a domain shift setting with feature distribution skew, where each client has a target domain with distinct characteristics, we implement federated version of PACS [35], 7-class classification dataset commonly used in domain generalization. The total number of images in the dataset is 9991. We partition the data, associating each client with one of the four available domains: Photos, Sketches, Cartoons, and Paintings.
The experiments PACS [3] shown in Table 12 On this task, all models reach competi- tive performances on both the central training and the federated version. The results lead to the aforementioned conclusions, confirming the Metaformer-like models are robust different types of heterogeneity across the datasets. All models have negligible degradation or show an increase the centralized version, except for Resnet [19], EfficientNet [62], and ConvMixer [67]. Our hypothesis for such behavior is the combination of the following elements: the small dimension of the target dataset compared the ImageNet pretraining and the overlapping of classes between both PACS and ImageNet [II].
As hypothesize that the behavior is caused by the solid pretraining ImageNet the following question arises: How will the models behave without pretraining? As shown in Table 12 degradation patterns arise again without pretraining. As expected all models have inferior performances, indicating how crucial pretraining is and making it always preferable weights are available. Moreover, most models with huge drop in the pretrained version still present the same behavior in the non-pretrained. However, the best-performing models are not those observed in the corresponding pretrained version. The best- pest-peroorming model is MLP-Mixer (64, along with EfficientNet [62]. which has the lowest score with pretraining, As one would expect, the worst-performing models are the ones that require large amounts of data. For example, PoolFormer, IdentityFormer, and RandFormer ([74)) perform very poorly already in the centralized setting, showing that a large dataset is crucial to compensate for the simple token mixers. Likewise, ConvNext 45 and ViT 600 reach deficient performance on the centralized training version.
MODEL
KENTRAL-PRETRAINE
SPLIT-PRETRAINED
CENTRAL
SPLIT
RESNET-50
96.1 1 0.26
90.2 1 0.53
(1 5.9)
82.1 0.774
42.5 0.53 (1 39.5)
EPFICIENTNET-BS
97.3 0.05
89.1 0.45
8.3)
3.46
6.64 (1 27.2)
CONvNEX1-T
97.2 0.28
97.6 6 0.21 (1 0.4)
42.6
(L 10. 32.5
SWIN-T
97.2
9.19 (1 0.6)
03.6 6
13.2)
VIT-S
96.9 + 0.31
7.9 (1 1.0)
5
49.0
8.3)
SWINV2-T
96.4 0.25
07.7 (1 1.2)
14.82.00
65.3
0.95 (4 9.5)
DEIT-S
96.5 1 0.14
97.5 1.51100 0.21 (T 1.0)
58.0 2.13
47.7 0.68 (L 10.4)
CONVFORMER-SI8
97.8 0.22
0812 (1 0.4)
70.0 0.24
54.3 0.63 15.7)
CAFORMER-S18
97.9 0.14
98.5 (1 0.6)
68.1 + 1.84
50.2 0.42 17.8)
RANDFORMER $36
95.7 0.27
0..4 96.1 0.7)
47.9 1.77
39.6
(L 8.3)
DERTITYFORMER-S4
95.4 0.26
1.37 (.7) 37.5 4.13
49.7
46.1
P0OLFORMER-S36
96.8 0.20
07.1 0.3)
8.8)
RIFORMER-SSK
96.4 1 0.14
9..7 (1 0.3)
55.9 15.9 0.32
4..8 (L 15.0)
MLPMIXER-S
96.0 1 0.45
9.45 0.44
70.110.11
63.2 0.59 (.
RESMI.P-S24
96.2 0.38
96.4+ (1 0.3)
64.1 0.72 0.72
2.50 (1 29.1)
GMLP-S
96.3 0.41
97.9 1.6)
71.6
59.4 (} 12.2)
CONVMIXER-768/22
97.3 + 0.36
75.7 (1.66
09.8 1.84
0.37 (1 49.1)
COATNET-O
97.0 4 0.29
94.7 (1.33)
79.8
47.7 0.39 (1 32.1)
MAXVIT-T
97.3 4: 0.25
96.7 0.16 (1.0))
83.4 0.74
50.9
32.6)
Table 12: A comprehensive evaluation of the accuracy achieved by various model families on the PACS dataset under different operational configurations. This includes centrally stored data and federated learning scenarios, with and without ImageNet pretraining. The results offer detailed insight into the relative effectiveness of the model families under these varying conditions, providing a clear understanding of the impact of pretraining model performance.
20

---

10.5 Convergence Speed
The efficiency of convergence is crucial in the context of federated learning. The inherent complexity of this learning paradigm imposes communication overhead for each update iteration, reinforeing the need for rapid convergence. Fast convergence reaching the optimal model parameters in fewer iterations minimizes communication demands, reducing bandwidth usage and overall training time. Faster convergence plays significant role real-world federated learning applications where devices may be intermittently available or have unreliable connections. We show that architecture selection (architectural choices) has significant impact on the convergence speed in federated learning scenarios. The rate of convergence among models varies notably. Notably, the best-pert rest-performing models we highlighted. specifically ViT and MetaFormers, also exhibit accelerated convergence in most of the experiments. Surprisingly, within the GLDk-23 set, CAFormer lags with the slowest convergence rate, suggesting a need for more in-depth exploration of complex large-scale settings in future studies.
Tex: Acssney (%0
10-
Tiak (%))
0.4
0..
0.-
Communication Rounds
Communication Rounds
Communication Rounds
Communication Rounds
CIFAR-10
b) ISIC-2019
ColehA
d GLD-2KK
Figure 5: Comparative evaluation of convergence rates for different neural network architectures across diverse datasets. (a) CIFAR-IO, (b) Fed-ISIC2019, (c) CelebA, and (d) GLD-23K. Convergence time is measured in terms of communication rounds. This comparison highlights the significant reduction convergence time enabled by architectural changes.
11 Implementation Details
11.1
Machines
We simulate the federated learning setup (1 server and N devices) on single machine with 32 Intel(R) Xeon(R) Silver 4215 CPU and NVidia Quadro RTX 6000 GPU.
11.2 Hyperparameter Search
Our study only tests a limited set of hyperparameters for the optimization methods, mostly adhering to the respective original authors' recommendations, For FedProx, conduct hyperparameter search across a spectrum of ya values, specifically μ E (10-1, 10-3 10-1,1) keeping the clients learning rate 77 x 10-2
For SCAFFOLD, we perform a hyperparameter search over a grid of client learning rates 7 e {10-4.3 x 10-4 10100- 3.3 x 10-3 3, 10 101210--000 10 10-2,10-1 1,3 1010.,33 10-1, with global learning rate No 1. Hyperparameter tuning is harder for FedAvgM as involves an additional hyperparameter for momentum 3. In our search,
considered B 0.5, 0.5, 0.7, 0.9, 0.99) and server learning Ter E {110-.3 3 x 10-4 10 x 10-3 10-2 2,3 10 1.101.3 10 1). The learning rate of the client optimizer is held constant 17 = x 10
Although additional tuning the regularization parameters may sometimes yield improved empirical performance, we do not anticipate significant impact on the results discussed. This demonstrates the advantage architectural design choices over off the-shelf optimization methods, which can be integrated with minimal modifications to the training process.
21

---

ResNet-S0 50
ConvNext-1
CA
50
5D
20
65
85
418
AB
63
36
81
S0
55
65
56
85
Monner
22
53
72
33
82
68
86
55
59
76
65
86
23
75
50
0.0001
6,00
0.63
0.0001
0.051
0.0003
6,003
Learring Raie
Learning Rate
Learring Rate
Figure 6: Visualization of the results from hyperparameter grid search for the FedAvgM algorithm on the federated version of CelebA dataset. Three different architectures were tested: ResNet, ConvNext, and CAFormer. The explored hyperparameters include momentum and server learning rate.
11.3 Architectures
Ensuring comparable models is indeed crucial for obtaining fair results. All the architectures tested (with the exception of MLP- MLP-Mixer) have similar size, measured in number of parameters (21-31M), and similar computational complexity, measured in FLOPS (4-6G). Other elements that might impact the computational cost, input size, etc., also kept constant.
Regarding the variations in pre-trained accuracy on ImageNet, there are slight differences in perfor- mance among the models. However, it is essential to consider that this diversity is inevitable due the wide range of architectures we incorporated that span across several years of research. To analyzed the absolute numerical result of the experiments
address fairness in our comparisons, and discussed the drop in performance for each model compared its own centralized version the training. Furthermore, all models perform similarly in the centralized setting on CIFAR-10 10, while their performances significantly differ challenging splits. Both analysis allows us to assess the relative impact of each model's modifications while accounting for the individual model's baseline.
MODEL
PARAM
FLOPS
IMAGENET 1K ACC
RESNET-50
26M
4.1G
76.2
EFFICIENTNET-R B5*
2.4G
83.6
CONVNEXT-T
29M
4.5G
82.1
SWIN-T
29M
4.5G
81.3
VrT-S*
22M
4.5G
83.1
SWINV2-T4
28M
4.6G
83.5
DEIT-S
22M
4.6G
79.8
CONYFORMER-SI8
27M
3.9G
83.7
CAFORMER-SII
26M
4.1G
84.1
RANDFORMER $36
31M
5.2G
79.5
SG
79.3
POOLFORMER-S36
31M
5G
81.4
RIFORMER-S36
31M
5G
81.3
MLPMIXER-S/II
59M
13G
76.4
RESMLP-SE S24
30M
6G
79.4
GMIPPS*
20M
6G
79.6
CONVMIXER-768/33
21M
20G
80.16
COAINET-0
25M
4.2G
81.6
MAXVIT-T
31M
5.6G
83.6
Table 13: Comparative analysis of examined architectures based key performance and complexity metrics. The metrics include the number parameters, Floating Point Operations Per Second (FLOPS), and ImageNet- 1K performance results. *FLOPS were computed using the Fvcore library.
22


Reviews:
[{"review_id": "GI0FrqLqu7", "reviewer": "Reviewer_nuGC", "summary": "This work conducted extensive empirical evaluation to study the influence of model structure on federated learning performance. The paper evaluated the performance of 19 models from 5 model families on 4 different federated learning datasets. The results showed that architecture design has a significant impact on federated learning performance. ", "questions": "Please see the weakness section.", "limitations": "There is no discussion about limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe work is well motivated. Studying the impact of architecture design on federated learning is important to improve the performance and inspire new federated learning methods.\n\n2.\tThere are extensive experiments to evaluate the performance of different models.\n\n3.\tThe paper is easy to follow.\n", "weaknesses": "1.\tI appreciate that the author conducted such extensive empirical evaluation. It takes a lot of efforts. However, there are not many insights and takeaways from these evaluations. It seems that the major information this paper tries to convey is that architecture design has critical impact on the federated learning performance. This is not a instructive conclusion and not surprising, as one of the most important thing in deep learning is the model architecture. I suggest the author to conclude more insights and guidelines for future works on designing architectures in federated learning. And it would be better to give some fundamental explanations on why some designs are better rather than just based on empirical evaluations.\n\n2.\tThere should be some introduction to related works. What studies have been done in previous works? How is this work related to them and different from them? What new insights does this work bring to the community?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. **Please find our responses to specific queries below.**\n\n> **1. I appreciate that the author conducted such an extensive empirical evaluation. It takes a lot of effort. ,..., I suggest the authors conclude more insights and guidelines for future works on designing architectures in FL...**.  \n    \nWe thank the reviewer for acknowledging the efforts of our extensive experiments. Although it is true that the importance of architecture design is known in the deep learning field, in the federated learning community, a great number of papers still evaluate the results on very shallow or non-state-of-the-art architectures such as ResNet.  Our study embodies a step toward more efficient, practical, and robust FL systems. Furthermore, most approaches to tackle data heterogeneity by the FL community have been from an FL optimization standpoint, and only limited efforts have been put into the architectural design aspect of the problem.  In this work, we demonstrate that architectural design is important in the FL field and necessary to tackle real-world problems with challenging datasets without increased communication costs.  We also perform fine-grained experiments such as weight divergence analysis (Fig 3 in the paper), convergence speed (Sec 2.5 in the Supplementary material), etc., to better explain the improved performance of certain design choices.  We have provided a comprehensive summary of key insights and practical takeaways from our paper in Sec. A of *common response to all reviewers* provided above.   \n\n> **2. There should be some introduction to related works. What studies have been done?...**   \n\nWe would like to thank the reviewer for the suggestion. In the current draft, we have included a brief discussion about related approaches in the introduction (lines 38-45 in the paper). Moreover, a literature review and comparative analysis of nearly 20 federated learning approaches evaluated on computer vision benchmarks is provided in the supplementary material Sec. 1, Tables 1 and 2.  In our Suppl. Table 1, each method is classified by the publication venue, the vision architecture utilized, and the vision-related datasets employed for testing. Furthermore, we provide specific details for shallow models on the complexity of the network architectures in terms of the number of convolutional (CNN) and fully connected (MLP) layers. This comparative evaluation shows that our method covers various state-of-the-art vision architectures that are never studied in the federated learning literature across five datasets (among which four datasets are naturally federated). As suggested, we will include a related work section along with Tab. 1 suppl. in the final draft. The related work section will highlight the contributions of existing works and explain how our work builds upon and differs from them, as briefly outlined below. \n- The growing focus on data privacy and protection (Li et al., 2021b; Shokri & Shmatikov, 2015) has sparked significant research interest in Federated Learning (FL) as it provides an opportunity for collaborative machine learning in various domains. Despite its potential benefits, FL faces fundamental challenges for real-world applications. The FL environments are complex, as local datasets of individual clients may not accurately represent the global data distribution. Heterogeneity caused by uneven feature distributions, labels, or data amounts across clients presents a significant challenge. FL methods need to accommodate statistical diversity and exploit similarities in client data. The non-IID (Independent and Identically Distributed) nature of clients can affect model performance, resulting in lower accuracy and slower convergence compared to centrally-trained models.\n- The problem of client data heterogeneity has received attention from the optimization community (Lian et al., 2017; Li et al., 2019b; 2020; Karimireddy et al., 2020; Acar et al., 2021; Mishchenko et al., 2022; Gao et al., 2022).\n- An orthogonal approach is investigating how architectural choices can enhance FL training performance. Pretrained models have been shown to alleviate non-IID issues (Chen et al., 2022a; Nguyen et al., 2022).\n- Some previous studies have highlighted the impact of Batch Normalization on performance drops under heterogeneous settings (Li et al.; 2021), and few works (Hsieh et al., 2020; and Hsu et al., 2020) suggested replacing BN with Group normalization. However, these studies often used shallow or non-state-of-the-art architectures. We argue that analyzing more complex and modern models in FL settings is essential.\n- Our work is related to Qu et al.'s (2022) examination of neural architecture robustness across heterogeneous data splits. Unlike their work, we show that convolution-based architectures are not inherently inferior to transformer-based ones in non-IID settings. Our experiments compare 19 state-of-the-art models from different computer vision architectural families on challenging heterogeneous datasets and optimizers. Based on this extensive analysis, we propose architectural modifications like replacing batch normalization with layer normalization to improve FL performance significantly.\n- Our study demonstrates that state-of-the-art FL optimization techniques may not provide optimum performance when combined with state-of-the-art complex networks. In practical scenarios, altering the architecture offers a more effective and simpler choice.\n\n > **3. What new insights does this work bring to the community?**  \n\nKindly refer to our reply to *question 1* and the \"Key Contributions and Insights\" provided in Sec. A of *common response to all reviewers*.\n\n > **4. Discussion about limitations**.  \nPlease see Sec. B  of *common response to all reviewers*  above, under the title \"Discussion on Limitations\". This discussion on limitations will be incorporated into the final draft of our paper. \n"}, {"review_id": "fhfaXdzYiq", "reviewer": "Reviewer_vstz", "summary": "In this comprehensive research study, the authors dive into the realm of federated learning (FL) and its application in visual recognition. They address a crucial aspect often overlooked in FL literature - the impact of architectural design choices on achieving optimal performance. While FL offers collaborative training without exchanging sensitive data, it presents challenges in matching the performance of centrally trained models. The authors conducted an extensive analysis of various cutting-edge architectures including convolutional neural networks, transformers, and MLP-mixers, evaluating their effectiveness in FL systems. By experimenting with 19 visual recognition models from five architectural families across four challenging FL datasets, the authors highlight the significant role of architectural choices in enhancing FL performance, particularly when handling heterogeneous data. ", "questions": "1. Line 162: Can you please describe why you used this cosine weight decay? It should be added to the paper, and can be described in a line or more. \n2. Line 280: “On the challenging Split 3,.....” - You mentioned this Split as challenging, but what special this Split has which makes it challenging? \n", "limitations": "Limitations not added; however, I would suggest adding it. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "For Computer Vision tasks, it is proven that an appropriate architecture design can have a greater impact. However, for Federated Learning, this work fills up such a gap that prior statement can be applicable here too, with centralized training.  ", "weaknesses": "1. Can the figure captions be made small? Instead of writing everything on caption, it is better to write these on the actual paper. \n2. Figure 2: It is better to reposition this adjacent to Table 2. And the discussion of splits and central from here should be reflected after Section 4 (after Dataset). And it should be discussed how uniquely it has been partitioned. \n3. Table 1, third line: “with the standard deviation of the experiments.” - Here, the total number of experiments should be added, in number.\n4. Line 273: “....observations made by Qu et al….”- It is better to add what the observations were made in the paper, at least in short, for better readability. \n5. Table 3 caption: LN/BN - The full form can be added prior/adjacent to this, so that it can be better read. \n6. Referring to #2 of the Questions section, it needs to be mentioned. \n\nMinor:\n1. Naming convention system. If you are naming/specifying/describing something in a specific way, it is better to follow that same thing throughout the paper. For instance, ISIC2019, from line 193, is not written in the same manner in Line 230.   \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. **Please find our responses to specific queries below.**\n\n>**1. Smaller figure captions?....**  \n  \nWe thank the reviewer for the feedback. We will address this in the final draft.\n\n>**2. Reposition  Figure 2 adjacent to Table 2....**    \n\nWe thank the reviewer for the suggestion. We will reposition Fig. 2 adjacent to Table 2 in the final draft and include the discussions in Section 4.\n\n>**3. Table 1, ... the total number of experiments should be added.**    \n\nWe have repeated each experiment three times and reported the standard deviation. We will revise the captions in Table 1 and other tables to include this repetition number. \n\n>**4. Line 273: “....what are the observations made by Qu et al….”.**   \n  \nWe appreciate your suggestion. In line 273, we will enhance the passage by summarizing the key observations made by Qu et al. This addition will provide readers with a clear and concise comprehension of the context. Notably, it will emphasize that the Metaformer and ConvNext experiments exhibit robust performance even with convolutional operations.  This stands in contrast to the insights presented by Qu et al. that convolutional operations relying on local high-frequency patterns are potentially problematic in non-IID FL settings, and self-attention-based architectures are more robust in FL settings. \n\n>**5. Table 3 caption: Adding the full forms of LN/BN**  \n  \nThe full form of Batch normalization (BN) and Layer Norm (LN) is currently defined in lines 326-332. This will be defined in Table 3 caption as well for better readability.\n\n>**6. Referring to #2 of the Questions section, it needs to be mentioned.**    \n\nIn CIFAR-10, the heterogeneity stems from distinct simulated data partitions with different label distribution skew. Split-3, the most demanding, confine each client's labels to only two of the ten classes, introducing a significant label skew (KS=1). We will include this explanation in the manuscript as suggested. For more comprehensive details, please refer to the response provided in *Question 9*.\n\n>**7. Minor: Consistency in naming... For instance, ISIC2019, in line 193 and Line 230.**    \n\nWe thank the reviewer for pointing out this. The dataset will be addressed as \"ISIC2019\"  in the final draft. We will also check for other naming convention mismatches.\n\n>**8. Line 162: describe why you used this cosine weight decay?.**     \n\nApologies for the typo. In line 162, we indeed referred to the cosine learning rate decay. The decision to utilize a learning rate decay was based on its demonstrated effectiveness in achieving optimal solutions within the FL domain, as highlighted in several papers, including (On the convergence of fedavg on non-iid data. - Li et al, 2019). Specifically, the selection of the cosine decay is grounded in the fact that cosine learning rate decay is a common choice in the training recipe across most of the families of architectures selected (Are transformers more robust than cnns? - Bai et al., 2021; Bag of tricks for image classification with convolutional neural networks - He et al., 2019; Early convolutions help transformers see better - Xiao et al., 2021). We will promptly update the final draft to provide a concise description of the cosine learning rate decay for further clarity. \n\n>**9. Line 280: “On the challenging Split 3,.....”  what makes this split challenging?**    \n\nThe reason Split-3 is considered challenging is due to the simulated heterogeneity in the data partition. As CIFAR-10 lacks a natural per-client data partition, we have devised distinct splits to introduce varying degrees of label distribution skew. As mentioned in lines 178-183 of the Experimental Setup - Dataset section, in Split-1, the data is uniformly distributed across client labels. In Split-2, each client has access to only four out of the ten classes, introducing more heterogeneity. Split-3 is the most challenging because each client is limited to samples from only two classes. This restriction creates a significant label skew, making it the most difficult split on the CIFAR-10 dataset. To quantify the heterogeneity, we use the Kolmogorov-Smirnov statistic (KS), and Split-3 has a value of 1, indicating the highest level of label skew compared to other splits. The below table indicates the data heterogeneity of different datasets used in our experiments in terms of  Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS). The LDS is characterized by a heterogeneous number of samples per class for each client. In the DDS, the amount of data per client varies. Lastly, the FDS is defined by varying image features per class across clients due to changes in acquisition sensors, image domains, user preferences, and geographical locations, etc. We will incorporate this discussion in the revised manuscript to enhance the reader's understanding. \n\n|**Dataset**| | **# Classes** | **# Clients** | **# Images** | **LDS** | **FDS** | **DDS** |\n|:--:|:--:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**CIFAR-10**|**Split-1**|10| 5|60,000|✘|✘|✘|\n| |**Split-2**|10|5|60,000|✔✔|✘|✘|\n| |**Split-3**|10 |5|60,000|✔✔✔|✘|✘|\n|**CelebA** | |2|227 |1,213|✔✔|✔|✔|\n|**ISIC2019** | |8|6|23,247|✔✔|✔✔|✔✔|\n|**Google Landmark-V2** | | 203| 223 |23,080|✔✔|✔✔|✔✔✔|\n|**PACS**| | 7|4|9,991|✘|✔✔✔|✘|\n\n*Note*: Summary of datasets and setups. We assess the data heterogeneity of the datasets in terms of three key aspects: Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS).  We utilize a grading system to quantify the level of skewness, with options ranging from \"none\" (✘), \"mild\" (✔), \"moderate\" (✔✔), to  \"severe\" (✔✔✔). \n\n>**10. Limitations not added....**     \n\nPlease see Sec. B in the *common response to all reviewers* above, with the title \"Discussions on Limitations\". This discussion on limitations will be incorporated in the final draft of our paper."}, {"review_id": "H3k5vFM20O", "reviewer": "Reviewer_JN2W", "summary": "This paper focuses on exploring the effects of architectural design choices on the performance of federated learning (FL) systems in visual recognition tasks, particularly when faced with data heterogeneity. To accomplish this, the paper conducts a comprehensive experimental analysis involving 19 state-of-the-art models from five architectural families. The experiments are conducted on four federated datasets, and the results are compared against centralized training and various optimization methods. Additionally, the paper investigates the impact of normalization layers on the performance of FL systems. It provides insights into the role of normalization layers and their influence on FL performance, ultimately proposing guidelines for selecting appropriate architectures in FL scenarios.", "questions": "1, It could be better to provide a discussion on how the findings are related to other key factors such as communication costs in FL. For example, for two given architectures, does the model with lower performance always have better communication efficiency? Can a single model provide good performance and communication efficiency at the same time?", "limitations": "Please refer to the weakness and questions.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1, This paper offers a comprehensive investigation by considering a diverse set of models, datasets, and optimization methods, enabling a thorough evaluation of their performance in the presence of varying levels of data heterogeneity. By conducting this wide-ranging analysis, the paper provides a fair and comprehensive comparison of these approaches.  \n2, This paper introduces several novel architectures, including Metaformers, MLP-Mixers, and Hybrids, which have not been extensively explored in the federated learning (FL) literature. These architectures are shown to achieve comparable or superior performance to more traditional models such as Transformers and CNNs in FL settings.  \n3, The writing is clear and easy to follow.  \n", "weaknesses": "1, While this paper provides a comprehensive study of architectural design for FL, no novel methods are developed in this paper.  \n2, This paper lacks a clear definition or measure of data heterogeneity, which hampers the ability to compare results consistently across different datasets and splits. Without a well-defined metric, it becomes challenging to quantify and understand the extent of data heterogeneity in the experiments conducted.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. **Please find our responses to specific queries below.**  \n\n>**1. While this paper provides a comprehensive study of architectural design for FL, no novel methods are developed.**  \n\nWe acknowledge that our work does not propose new FL methods. Instead, it offers a thorough exploration of architectural design in FL. We believe our study to shed light on the various insights on architectural design that can drive future advancements in FL. We plan to leverage the knowledge gained to pursue future work in creating novel FL-specific architectures that can address challenges and applications in the domain. We have provided a comprehensive summary of key contributions in Sec. A of *common response to all reviewers* given above.\n\n>**2.  Lacks a clear definition or measure of data heterogeneity...**\n\nThank you for the opportunity to clarify this point. The standard approach in the FL community involves evaluating methods on artificially generated datasets, where non-IID splits are created by selecting a heterogenous number of samples per class for each client. The sampling strategy is typically performed either according to a Dirichlet distribution with different $\\alpha$ levels, where the parameter $\\alpha$ controls how much the partition is unbalanced, or Kolmogorov-Smirnov (KS) statistic to quantify the difference data distributions of data across clients as in [Ensemble distillation for robust model fusion in federated learning - Lin et al., 2020], [No fear of heterogeneity: Classifier calibration for federated learning with non-iid data - Luo et al., 2021] and [Rethinking.. - Qu et al., 2022]. In our experiments, we used this second metric for the CIFAR-10 dataset, for which we reported increased KS statistic value for three splits over which we have direct control of the label allocation. \n\nAfter assessing our results on simulated non-IID splits with CIFAR-10, according to what was done by current literature, we evaluated our results on four additional naturally federated datasets (see Appendix for PACS dataset), showcasing the consistency of our findings. This second approach embodies a more challenging setting that aligns closely with real-world scenarios, an aspect we strongly advocate for in FL research.\n\nWe acknowledge the lack of a clear metric to quantify the data heterogeneity in real-world datasets.\nIndeed for datasets that already present a natural federated partition, there is no standard metric in the community to assess the data heterogeneity. In particular, because the data exhibit different heterogeneities apart from the label distribution skew, those are typically intertwined in real-world scenarios and can be highly non-trivial to measure (e.g., the above-mentioned metrics cannot be straightforwardly applied). We considered, in addition to the label distribution skew, feature distribution skew, where the image features per class vary across clients, and data distribution skew, where the amount of data per client varies. We believe this categorization provides a good estimation of the diverse types of heterogeneity that real-world data may exhibit in the wild.\n\nThe below table indicates the data heterogeneity of different datasets used in our experiments in terms of  Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS). The LDS is characterized by a heterogeneous number of samples per class for each client. In the DDS, the amount of data per client varies. Lastly, the FDS is defined by varying image features per class across clients due to changes in acquisition sensors, image domains, user preferences, geographical locations, etc.  Furthermore, apart from the dataset description in Section 4, we refer readers to Image 1 of the manuscript for a visual representation of the client's data and class distribution. We will incorporate this discussion in the revised manuscript.\n\n|**Dataset**| | **# Classes** | **# Clients** | **# Images** | **LDS** | **FDS** | **DDS** |\n|:--:|:--:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**CIFAR-10**|**Split-1**|10| 5|60,000|✘|✘|✘|\n| |**Split-2**|10|5|60,000|✔✔|✘|✘|\n| |**Split-3**|10 |5|60,000|✔✔✔|✘|✘|\n|**CelebA** | |2|227 |1,213|✔✔|✔|✔|\n|**ISIC2019** | |8|6|23,247|✔✔|✔✔|✔✔|\n|**Google Landmark-V2** | | 203| 223 |23,080|✔✔|✔✔|✔✔✔|\n|**PACS**| | 7|4|9,991|✘|✔✔✔|✘|\n\n*Note*: Summary of datasets and setups. We assess the data heterogeneity of the datasets in terms of three key aspects: Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS).  We utilize a grading system to quantify the level of skewness, with options ranging from \"none\" (✘), \"mild\" (✔), \"moderate\" (✔✔), to  \"severe\" (✔✔✔). \n\n> **3.  How the findings are related to other factors, such as communication costs.? ... Can a single model provide good performance and communication efficiency?**   \n\nWe recognize the importance of communication costs within FL settings. We purposefully chose models with approximately the same dimension (parameters and FLOPS ) and upheld consistency in other aspects that affect communication costs (including communication rounds, protocol, and device types) to ensure a fair comparison. Moreover, our architectural modifications led to a negligible impact on communication costs. By design, this approach helps to maintain fair communication complexity across all evaluated methods. For this reason, we did not explicitly analyze communication complexity.\n\nNevertheless, as elaborated in the Supplementary Material (Sec 2.5), the rate of convergence among models varies notably. Models with faster convergence necessitate fewer communication rounds to reach saturation. In practical scenarios, this could translate to a reduction in communication rounds, minimizing communication costs. Notably, the best-performing models we highlighted, specifically ViT and MetaFormers, also exhibit accelerated convergence in our experiments."}, {"review_id": "q2GC4qLz05", "reviewer": "Reviewer_TNv8", "summary": "This paper presents an experimental analysis comparing several state-of-the-art neural networks architectures on four different federated vision datasets in highly heterogeneous settings using multiple different optimization approaches. ", "questions": "Is LayerNorm better than GroupNorm in FL settings?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Thorough experimental study: Four datasets, four optimizers, numerous architectures.\n\nUseful experimental study: \nThis paper conducts experimental study in FL setting with several latest neural network architectures used in the computer vision community. This study takes a step in the direction of bridging the gap between advances in FL and CV communities. This study also provides insights into the roles of architecture and optimizer in FL setting.\n1) Advanced optimizers may not be useful when latest/advanced neural network architectures are used.\n2) Choosing an appropriate architecture design can have a greater impact on closing the gap with centralized training than the choice of the optimizer.\n3) Metaformer-like architectures are more resilient to data heterogeneity. Self-attention is not necessarily the key component.\n4) Convolutions may not be inherently problematic in non-iid FL settings. The type of normalization used matters a lot.\n\n\n\n", "weaknesses": "Various existing works have already emphasized the problem BatchNorm causes in FL settings, and proposed to use GroupNorm instead of BatchNorm For example, \"The Non-IID Data Quagmire of Decentralized Machine Learning\" and \"Federated Visual Classification with Real-World Data Distribution\". This paper proposes to replace BN it with LayerNorm for CNNs, but has not provided any experimental results comparing it with GroupNorm.\n\n\nTypo: \nTable 1 - Caption mentions CelebA but there are no results on CelebA in the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. **Please find our responses to specific queries below.**  \n\n> **1.1 *(Weakness)* Various existing works have already emphasized the problem BatchNorm causes in FL settings, and proposed to use GroupNorm instead of BatchNorm, for example...   \n1.2 *(Question)* Is LayerNorm better than GroupNorm in FL settings?**  \n\nThe papers mentioned by the reviewer (The non-iid data quagmire of decentralized machine learning - Hsieh et al., 2020; Federated visual classification with real-world data distribution - Hsu et al., 2020) acknowledge Batch Normalization's challenges in federated learning scenarios and substitute BatchNorm with Group Normalization for convolutional neural networks.\n\nIn particular, the paper \"The Non-IID Data Quagmire of Decentralized Machine Learning\"  by Hsieh et al. delves into BatchNorm's limitations and explores Group Normalization as a viable alternative. While the research demonstrates an enhanced result for GroupNorm on LeNet over CIFAR-10, it lacks quantitative analysis for other choices, such as LayerNorm, and results on different architectural paradigms, especially non-convolutional architectures.\n\nIndeed, we did not show results on Group Normalization in our manuscript. As we performed experiments to assess the robustness of different architectures, the preliminary results showed that models with BatchNorm had a higher drop in accuracy compared to centrally stored data and inferior performances compared to models with LayerNorm. In particular, the best results were achieved by Transformer and Metaformer models, as LayerNorm was introduced with the advent of Transformer architectures. Replacing BatchNorm with LayerNorm in methods utilizing convolution operations (eg: CoAtNet, ResNet) allowed us to demonstrate that convolutions are not inherently suboptimal in heterogeneous settings (in contrast with the observations of Qu et al., 2022). For this experiment, Layer norm and Barch norm were the natural choices because those two layers are the most commonly used across SOTA architectures, affording a natural Convolution vs. Self Attention evaluation.  As suggested by the reviewer, we have extended our BatchNorm/LayerNorm experiments in this rebuttal by incorporating the GroupNorm layer.\n\nWe replicated our results on CIFAR-10, encompassing  CoAtNet, PoolFormer, and ResNet - as in the original manuscript—while augmenting it with GN results (averaging three repetitions per experiment along with standard deviations). In this context, CoatNet and PoolFormer exhibited a more pronounced performance drop under GN compared to the LN configuration. Intriguingly, ResNet yielded a converse pattern, achieving superior outcomes with GroupNorm as opposed to LayerNorm. This trend remained consistent while performing the weight divergence analysis across Split-2 and Split-3.  These experiments indicate that LayerNorm is a more favorable choice when paired with advanced models. Indeed, the weight divergence serves as a compelling indicator that the incorporation of LN offers a safeguard against drops in performance in heterogeneous partitions, with LayerNorm having inferior weight divergence wrt BatchNorm in all cases and best results for CoAtNet and PoolFormer.\n\nOur findings are presented in the table below, detailing the results, along with an analysis of weight divergence in Image 1 reported in the attached PDF document. \n\n| **Variant**        | **Central** | **Split-3** | \n|:------------------:|:---------------------:|:---------------------:|\n|  PoolFormer-S12 BN |   $95.3 \\pm 0.35$    | $51.5 \\pm 0.32$ (↓43.7) | \n|  PoolFormer-S12 GN† |   $94.1 \\pm 0.25$   | $81.5 \\pm 0.29$ (↓12.6) |  \n|  PoolFormer-S12 LN |   $95.6 \\pm 0.09$   |  $91.5 \\pm 0.13$ (↓4.1) |  \n|  ResNet-50 BN     |   $95.9 \\pm 0.21$    | $59.9 \\pm 1.87$ (↓36.0) |\n|  ResNet-50 GN†   |   $95.8 \\pm 0.19$   |  $89.3 \\pm 0.59$ (↓6.5) | \n|  ResNet-50 LN†    |   $92.8 \\pm 0.16$    | $72.2 \\pm 3.06$ (↓26.6) | \n|  CoAtNet-0 BN      |   $97.5 \\pm 0.04$     | $20.9 \\pm 0.45$ (↓76.6) |  \n|  CoAtNet-0 GN†    |   $94.3 \\pm 0.39$   | $82.6 \\pm 2.79$ (↓11.7) |  \n|  CoAtNet-0 LN *    |   $97.7 \\pm 0.15$    |  $94.2 \\pm 0.06$ (↓3.4) |  \n\n*Note*:  Accuracy of the models with BatchNorm (BN), LayerNorm (LN), and GroupNorm (GN) in the central and the Split-3 version of  CIFAR-10. * The original CoAtNet-0 uses BN and LN in convolutional and self-attention blocks, respectively. † layers have no pretraining.\n\n> **2. Typo: Table 1 - Caption mentions CelebA but there are no results on CelebA in the table.**\n\nThank you for bringing the typo to our attention. CelebA results for Table 1 were indeed reported in Supplementary Material Section 2.1. We acknowledge the oversight in Table 1's caption and will remove the excessive text in the final draft. \nThe paper's conclusions from Table 1 are in synchrony with the results on CelebA. We observed that altering the optimizer along with complex models led to virtually negligible improvements. Conversely, when we modified the architecture, we achieved a substantial boost in performance. Refer to the main paper, Section 5.1 - Optimizers and Complex Networks in Federated Learning, for an in-depth discussion. "}, {"review_id": "YQemBMrR63", "reviewer": "Reviewer_6dpE", "summary": "This paper conducted extensive experiments to show the effect of the neural architecture design on non-iid federated learning (FL) for visual recognition. More specifically, experimental results demonstrated that applying the SOTA neural architecture design to FL can significantly mitigate the performance degradation incurred by non-iid data distribution. Moreover, it also shows that layer normalization performs better than batch normalization in non-iid FL.", "questions": "1. Although it seems to be mentioned, I want to confirm whether the evaluated models with different architectures have similar model size, computational costs and pre-trained accuracy on ImageNet.\n\n2. For the experiments in Table 2 and 3, do they use FedAvg for model aggregation?", "limitations": "The study of the effect of neural architecture on non-iid FL is limited to visual recognition, which limits its impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated. It bridges the gap between the outdated architecture utilized in the FL research community and the SOTA architecture in visual recognition.\n\n2. The authors have some interesting observations, such as the critical roles of layer normalization and the larger effect that the architecture can have over the optimizer.", "weaknesses": "1. The paper lacks technical novelty. It just simply applies existing SOTA visual models to non-iid FL. If the authors could utilize their observation to design an architecture with a customized optimizer, which achieves better performance, then the technical novelty will be improved significantly.\n\n2. It would be better if the author could conduct more fine-grained studies, like the effect of architecture on feature skew setting, label skew setting and convergence rate of the models. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. **Please find our responses to specific queries below.**\n\n>**1. On technical novelty**  \n\nRather than introducing an entirely novel approach to federated learning (FL), our paper focuses on conducting thorough experiments using state-of-the-art (SOTA) visual recognition architectures and network components to provide insights and practical guidelines for effectively tackling FL data heterogeneity within the context of modern computer vision architectures. We hope that  our study will serve as a catalyst for the creation of innovative FL strategies tailored for SOTA computer vision architectures. \n\nWe would like to clarify that our paper includes not only experiments on various SOTA models but also architectures from different families of backbones for visual tasks, along with a component-level analysis. Additionally, we performed an in-depth analysis of common optimization techniques for non-iid FL on SOTA visual recognition models, which is a contribution in itself. Moreover, we perform experiments on five datasets among which four are naturally federated. As suggested by the reviewer, we intend to use this study as a foundation for developing a custom architecture with an optimized approach in the future.\n\nPlease note that we have provided a comprehensive summary of key insights and practical takeaways from our paper in the *common response to all reviewers* provided above.\n\n>**2.  More fine-grained studies, like the effect of architecture on feature skew, label skew, and convergence rate .**  \n\nThank you for your insightful comment. Regarding label distribution skew (LDS), we have conducted experiments on CIFAR-10 and curated three different splits that exhibit increased label skew, quantified using the KS statistic. This approach allows us to thoroughly investigate the impact of label imbalance on our models' performance. \n\nAs for feature distribution skew (FDS), we have included challenging naturally federated datasets, particularly GLD-23K and ISIC2019, that present such heterogeneity. These datasets encompass diverse geographical locations (GLD-23K) and utilize various medical equipment for data acquisition (ISIC2019), providing a rich representation of feature distribution skew. Also, we included in Supplementary Material a dedicated analysis of the PACS dataset, which exhibits notable domain shifts. For this investigation, our federated partitioning involved the allocation of distinct target domains (Photos, Sketches, Cartoons, and Paintings). \n\nConcerning the convergence rate of our models, we have dedicated a section (Sec. 2.5) in the Supplementary Material - Additional Experiments to present the result. We found that Mataformer-like architectures, ViT, and CAFormer exhibit faster convergence rates compared to other methods. As suggested, these fine-grained experiments will be included and highlighted in the final draft.\n\n>**3. Although it seems to be mentioned, confirm whether the evaluated models with different architectures have similar model size, computational costs and pretrained accuracy on ImageNet.**  \n\nEnsuring comparable models is indeed crucial for obtaining fair results. All of the architectures tested (with the exception of MLP-Mixer) have a similar size, measured in number of parameters (21-31M), and similar computational complexity, measured in FLOPS (4-6G). Refer to Table 7 in Supplementary Material - Implementation Details. Other elements that might impact the computational cost, input size, etc., are also kept constant.\n \nRegarding the variations in pre-trained accuracy on ImageNet, there are slight differences in performance among the models. However, it is essential to consider that this diversity is inevitable due to the wide range of architectures we incorporated that span across several years of research. To address fairness in our comparisons, we analyzed the absolute numerical result of the experiments and discussed the drop in performance for each model compared to its own centralized version of the training. \nFurthermore, all models perform similarly in the centralized setting on CIFAR-10, which makes it interesting to analyze the FL setup as their performances significantly differ in challenging splits. \nBoth analysis allows us to assess the relative impact of each model's modifications while accounting for the individual model's baseline.\n\n>**4. For the experiments in Table 2 and 3, do they use FedAvg for model aggregation?**  \n\nYes, Tables 2, 3, and 4 use FedAvg for model aggregation. Experiment results combining optimizers and complex networks are in Table 1. Due to minor gains and inferior performance, subsequent experiments (after Section 5.1) adopted standard FedAvg. This will be better clarified in the final version. Additional results on optimization techniques are in Supplementary Material's Section 2, Table 3.\n\n>**5. The study of the effect of neural architecture on non-iid FL is limited to visual recognition.**\n\nWe acknowledge that our research specifically centered on studying the impact of neural architecture in the context of non-IID FL for visual recognition tasks. While visual recognition significantly influences various vision tasks and real-world scenarios, we agree that FL applies to diverse tasks as well. To overcome this limitation and improve our work's influence, we plan to expand our research in forthcoming projects by investigating other downstream tasks like segmentation and object detection. It's worth noting that visual recognition models often serve as backbones for extracting input image features in downstream tasks, thereby suggesting that our observed performance trends could likely extend to other tasks. We plan to incorporate this point into the limitations section of our revised manuscript, as shown in the \"Sec. B. Discussion on limitations*  listed in the *common response to all reviewers*.\n"}]

Author rebuttal:
We thank the reviewers for the positive comments. In this rebuttal, we have exerted our best efforts to address nearly all of the reviewers' comments, resulting in numerous enhancements to our final draft. *These improvements include*: (i) additional experiments on using Group normalization, (ii) experiments to evaluate convergence speed, and weight divergence, (iii) dedicated sections to discuss limitations and related works, (iv) additional details on data heterogeneity of different datasets.  *Our code and trained models will be made publicly available.*

##**A. The key contributions and insights of our paper are summarized below:** 
1. **Impact of Architecture Design on FL Performance (Acknowledged by Reviewers nuGC, vstz, TNv8, JN2W)**: The paper provides an extensive evaluation of 19 visual recognition models across diverse datasets and optimization methods. It highlights the importance of understanding architecture's impact on federated learning (FL) performance and addressing gaps in FL literature. For example,
given comparable complexity to a ResNet-50, by replacing the architecture, we achieve an increase of more than 37 points in accuracy on the most challenging split-3 of CIFAR-10. Moreover, we perform experiments on 5 challenging datasets, including non-naturally federated datasets such as CIFAR-10, naturally federated datasets such as ISIC2019, Google landmark-V2, CelebA, and Domain generalization dataset PACS (supplementary material).

2. **Introduction of Novel Architectures and Bridging with SOTA Visual Recognition Architecture 
(Acknowledged by Reviewers JN2W, 6dpE, TNv8)**: The paper introduces novel architectures such as Metaformers, MLP-Mixers, and Hybrids to the FL literature. Moreover, this work contributes towards bridging the architecture used in FL research with state-of-the-art visual recognition architecture. It also proposes guidelines and suggestions for selecting appropriate architectures in FL scenarios.

3. **Insights on Normalization Layers (Acknowledged by Reviewers 6dpE, JN2W, TNv8)**: Our paper shows that the use of Batch Normalization hinders the performance in FL non-iid setting, to which the paper proposes an effective off-the-shelf solution: replacing it with layer normalization, typically used in the self-attention mechanism. Furthermore, our experiments in this rebuttal comparing layer norm, batch norm,  and group normalization show that layer norm consistently performs well across various architecture families, including state-of-the-art Metaformers.  Our study modernizes the prior research (SkewScout- ICML20, and Hsu et al. - ECCV20) recommending group normalization in FL setting based on their experiments on convolutional neural networks and shallow networks, highlighting the broader applicability of Layer norm in FL non-IID setting.  

4. **Effect of Architecture over FL Optimizers on Closing Gap with Centralized Training  (Acknowledged by Reviewers TNv8, 6dpE)**: The paper demonstrates that advanced FL optimizers may not provide optimum performance when using the latest neural network architectures, emphasizing architecture's role in closing the gap with centralized training.

5. **Resilience of Metaformer-like Architectures to Data Heterogeneity  (Acknowledged by Reviewer TNv8)**: The paper highlights the resilience of Metaformer-like architectures to data heterogeneity and the insight that self-attention isn’t the sole key component. Moreover, we analyze how the best-performing models, namely ViT, CAFormer, and ConvFormer, also tend to possess a faster convergence speed (See Supplementary material Sec 2.5 and Fig. 1). Once again, vouching for the use of MetaFormer-like architectures for FL.

6. **Validation of Convolutions in Non-IID FL Settings  (Acknowledged by Reviewer TNv8)**: Our study demonstrates that convolutions may not be inherently problematic in non-iid FL settings. This is in contrast with a previous study (Qu et al. - CVPR 2022) suggesting that convolution operations are sensitive to local high-frequency patterns, potentially leading to inferior performance in heterogeneous FL scenarios. Our findings underline the adaptability of convolutional architectures in diverse FL setups.

7. **The work is well motivated** (Acknowledged by Reviewers **nuGC, 6dpE**) and the **writing is clear and easy to follow** (Acknowledged by Reviewers **nuGC, JN2W**).


##**B. Discussions on Limitations**
We thank the reviewers for the suggestions. We will include a dedicated section to discuss the limitations of our paper, as outlined below. 

*Limitations*:  (i) Our research primarily explores visual recognition tasks within the federated learning domain, and we have not explored potential downstream tasks, such as object detection and segmentation. 
It's worth noting that visual recognition models often serve as backbones for extracting input image features in downstream tasks, thereby suggesting that our observed performance trends could likely extend to these downstream tasks. We plan to address this research gap in future studies. 

(ii) Furthermore, our approach did not design a completely new model from scratch. Instead, we demonstrated our results utilizing pre-existing models and components. Our study also underscores the limitations of employing established optimizers (e.g., FedAVG, FedAVGM, FedProx, SCAFFOLD) along with modern network architectures. However, we abstain from plunging into the creation of innovative optimization techniques. Instead, we focus on evaluating diverse architectural families on many practical federated learning datasets and optimizers. As a result, our investigation leaves an intriguing avenue for future research, integrating our findings to develop dedicated architecture and optimizers tailored specifically for federated learning (FL) scenarios. By exploring this direction, we anticipate the potential to further enhance the performance and efficiency of FL systems.

Meta-review: Five experts reviewed the paper, and all recommended the paper for (borderline or weak) acceptance. The reviewers appreciated the careful and extensive experiments and found some results interesting. The study can inspire and benefit future work in federated visual recognition. However, the reviewers also raised concerns and/or suggestions for improving the paper, which the authors are encouraged to consider when revising the paper. The decision to recommend the paper for acceptance.
Decision: Accept (poster)