SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DATASET	# CLASSES	# CLIENTS	# IMAGES	LDS	FDS	DDS
CIFAR-10	SPLIT-1	10	5	60,000	X	X	X
SPLIT-2	10	5	60,000	✓✓	X	X
SPLIT-3	10	5	60,000	✓✓✓	X	X
CELEBA	2	227	1,213	✓✓	✓	✓
FED-ISIC2019	8	6	23,247	✓✓	✓✓	✓✓
GOOGLE LANDMARK-V2	203	223	23,080	✓✓	✓✓	✓✓✓
PACS	7	4	9,991	X	✓✓✓	X
10 Additional Experiments
In this section, we delve into supplementary experiments. These encompass optimization strategies applied to complex network results 10.1, further analysis on normalization layers 10.2, the inclusion of the PASCs dataset 10.4, and a study of convergence speed 10.5.
10.1 Optimization Methods with Complex Architectures
Table 7: Summary of datasets and setups. We assess the data heterogeneity of the datasets in terms of three key aspects: Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS). We utilize a grading system to quantify the level of skewness, with options ranging from "none"  $(\times)$ , "mild"  $(\checkmark)$ , "moderate"  $(\checkmark \checkmark)$ , to "severe"  $(\checkmark \checkmark \checkmark)$ .
CELEBA
ARCHITECTURE	RESNET-50	CONVNEXT-T	CAFORMER-S18
FEDAVG [49]	84.9 ± 1.48	87.5 ± 0.87	88.1 ± 0.61
FEDAVGM [58]	84.1 ± 1.51	87.4 ± 1.15	87.4 ± 1.70
FEDPROX [38]	85.0 ± 1.48	87.5 ± 0.88	87.5 ± 0.37
SCAFFOLD [29]	84.4 ± 1.81	86.8 ± 1.33	89.0 ± 0.40
Table 8: Accuracy of ResNet-50, ConvNext-T, and CAFormer on CelebA across four different optimization methods. The average test accuracy of the model is displayed along with the standard deviation of the experiments. We can appreciate how changing the optimizer achieves virtually negligible improvements, unlike changing the architecture, which yields a significant boost.
10.2 Normalization Layer
In this section, we delve into extended experiments concerning normalization layers. We begin by reinforcing our earlier findings, showcasing that Layer Normalization outperforms Batch Normalization (BN) when tested on an additional dataset, Fed-ISIC2019, as detailed in Table 9.
VARIANT	FED-ISIC2019
COATNET-0 BN	68.5 ± 0.70
COATNET-0 LN *	70.4 ± 0.25
POOLFORMER-S12 BN	64.2 ± 0.53
POOLFORMER-S12 LN	83.2 ± 0.63
RESNET-50 BN	54.3 ± 0.48
RESNET-50 LN†	62.5 ± 1.50
RESNET-50 FEDBN [39]	65.2 ± 1.24
Exploring alternatives to BN, and extending beyond LN, we incorporated Group Normalization (GN) in our assessment. We replicated our results on CIFAR-10, encompassing CoAtNet, PoolFormer, and ResNet - as in the original manuscript--while augmenting it with GN results (averaging three repetitions per experiment along with standard deviations). In this context, CoatNet and PoolFormer exhibited a more pronounced performance drop under GN compared to the LN configuration. Intriguingly, ResNet yielded a converse pattern, achieving superior outcomes with GroupNorm as opposed to LayerNorm. This trend remained consistent while performing the weight divergence analysis across Split-2 and Split-3. These experiments indicate that LayerNorm is a more favorable choice when paired with advanced models. Indeed, the weight divergence in figure 4 serves as a compelling indicator that the incorporation of LN offers a safeguard against drops in performance in heterogeneous partitions, with LayerNorm having inferior weight divergence wrt BatchNorm in all cases and best results for CoAtNet and PoolFormer.
Table 9: Accuracy of ResNet-50, CoAtNext- 0, and PoolFormer-S12 with Batch (BN) and Layer (LN) Normalization across the Fed-ISIC2019 datasets. Replacing BN with LN improves performance. * The original CoAtNet-0 uses BN and LN in convolutional and self-attention blocks, respectively. † LN layers have no pretraining.
Variant	CIFAR-10 Central	CIFAR-10 Split-3
PoolFormer-S12 BN	95.3 ± 0.35	51.5 ± 0.32 (↓43.7)
PoolFormer-S12 GN†	94.1 ± 0.25	81.5 ± 0.29 (↓12.6)
PoolFormer-S12 LN	95.6 ± 0.09	91.5 ± 0.13 (↓4.1)
ResNet-50 BN	95.9 ± 0.21	59.9 ± 1.87 (↓36.0)
ResNet-50 GN†	95.8 ± 0.19	89.3 ± 0.59 (↓6.5)
ResNet-50 LN†	92.8 ± 0.16	72.2 ± 3.06 (↓26.6)
CoAtNet-0 BN	97.5 ± 0.04	20.9 ± 0.45 (↓76.6)
CoAtNet-0 GN†	94.3 ± 0.39	82.6 ± 2.79 (↓11.7)
CoAtNet-0 LN*	97.7 ± 0.15	94.2 ± 0.06 (↓3.4)
Table 10: Normalization layer comparison. Accuracy of the models with batch normalization (BN), layer normalization (LN) and group normalization (GN) in the central and the Split-3 version of CIFAR-10. * The original CoAtNet-0 uses BN and LN in convolutional and self-attention blocks, respectively. † layers have no pretraining.
Figure 4: Weight divergence analysis on CIFAR-10. Weight divergence with respect to models' centralized version on CIFAR-10. The use of BN leads to a larger weight divergence. Complex models tend to pair well with LayerNorm, achieving even smaller divergence than when paired with GroupNorm layers.
10.3 Centralized Results
GLD-23K	FED-ISIC2019
MODEL	CENTRAL	SPLIT	CENTRAL	SPLIT
RESNET-50 [19]	79.6 ± 2.09	54.3 ± 0.48	88.2 ± 0.15	67.6 ± 1.59
EFFICIENTNET-B5 [62]	85.3 ± 0.30	51.9 ± 1.21	89.7 ± 0.41	70.0 ± 0.64
CONVNEXT-T [45]	83.7 ± 0.45	65.5 ± 0.57	88.1 ± 0.91	78.6 ± 0.29
SWIN-T [44]	81.9 ± 0.32	72.1 ± 0.67	90.8 ± 0.31	81.9 ± 0.42
ViT-S [60]	79.4 ± 0.34	76.6 ± 0.69	90.3 ± 0.17	82.3 ± 0.95
SWINV2-T [43]	75.7 ± 3.92	74.4 ± 0.02	88.1 ± 0.58	81.7 ± 0.50
DEIT-S [66]	78.7 ± 1.04	69.1 ± 1.24	89.3 ± 0.36	82.3 ± 0.59
CONVFOMER-S18 [74]	83.7 ± 0.49	53.8 ± 1.11	90.3 ± 0.45	81.1 ± 1.54
CAFORMER-S18 [74]	83.6 ± 0.31	57.8 ± 0.54	90.7 ± 0.22	83.1 ± 0.49
RANDFORMER-S36 [74]	81.4 ± 0.27	56.3 ± 0.10	86.9 ± 0.23	77.3 ± 0.23
IDENTITYFORMER-S36 [74]	81.3 ± 0.95	56.0 ± 2.07	87.0 ± 0.38	76.9 ± 1.43
POOLFORMER-S36 [73]	83.4 ± 1.33	55.8 ± 1.41	88.5 ± 0.36	79.6 ± 0.29
RIFORMER-S36 [70]	80.7 ± 1.17	69.4 ± 0.20	89.5 ± 0.08	81.9 ± 1.37
MLPMIXER-S/16 [64]	77.5 ± 0.91	71.3 ± 0.32	88.6 ± 0.50	80.5 ± 1.62
RESMLP-S24 [65]	73.4 ± 1.03	64.8 ± 0.59	89.1 ± 0.23	81.1 ± 0.45
GMLP-S [41]	76.5 ± 0.71	67.4 ± 0.16	88.6 ± 0.52	79.9 ± 1.60
CONVMIXER-768/32[67]	85.6 ± 0.40	45.0 ± 0.76	87.3 ± 0.20	58.6 ± 2.02
COATNET-0 [8]	79.2 ± 1.03	70.4 ± 0.25	87.6 ± 0.17	66.0 ± 3.40
MAXVIT-T [69]	83.4 ± 0.29	71.7 ± 0.90	91.0 ± 0.28	70.1 ± 1.27
Table 11: Comparative analysis among different architectural families in both federated and centralized settings. Notably, each model experiences a drop in accuracy when transitioning from centralized to federated learning settings. Importantly, high performance in the centralized setting does not necessarily translate to comparable success in the federated split. The table particularly emphasizes the robustness and superior performance of Metaformer-like architectures in the federated setting. These architectures demonstrate a minimal drop in accuracy relative to their centralized counterparts and high overall accuracy results, underlining their adeptness at handling data heterogeneity.
10.4 PACS: Analysis on Domain Shift
To simulate a domain shift setting with feature distribution skew, where each client has a target domain with distinct characteristics, we implement a federated version of PACS [35], a 7-class classification dataset commonly used in domain generalization. The total number of images in the dataset is 9991. We partition the data, associating each client with one of the four available domains: Photos, Sketches, Cartoons, and Paintings.
The experiments on PACS [35] are shown in Table 12. On this task, all models reach competitive performances on both the central training and the federated version. The results lead to the aforementioned conclusions, confirming the Metaformer-like models are robust to different types of heterogeneity across the datasets. All models have negligible degradation or show an increase over the centralized version, except for Resnet [19], EfficientNet [62], and ConvMixer [67]. Our hypothesis for such behavior is the combination of the following elements: the small dimension of the target dataset compared to the ImageNet pretraining and the overlapping of classes between both PACS and ImageNet [11].
As we hypothesize that the behavior is caused by the solid pretraining on ImageNet the following question arises: How will the models behave without pretraining? As shown in Table 12, degradation patterns arise again without pretraining. As expected all models have inferior performances, indicating how crucial pretraining is and making it always preferable if weights are available. Moreover, most models with a huge drop in the pretrained version still present the same behavior in the non-pretrained. However, the best-performing models are not those observed in the corresponding pretrained version. The best-performing model is MLP-Mixer [64], along with EfficientNet [62], which has the lowest score with pretraining. As one would expect, the worst-performing models are the ones that require large amounts of data. For example, PoolFormer, IdentityFormer, and RandFormer ([74]) perform very poorly already in the centralized setting, showing that a large dataset is crucial to compensate for the simple token mixers. Likewise, ConvNext [45], and ViT [60] reach a deficient performance on the centralized training version.
MODEL	CENTRAL-PRETRAINED	SPLIT-PRETRAINED	CENTRAL	SPLIT
RESNET-50	96.1 ± 0.26	90.2 ± 0.53 (↓ 5.9)	82.1 ± 0.74	42.5 ± 0.53 (↓ 39.5)
EFFICIENTNET-B5	97.5 ± 0.05	89.1 ± 0.45 (↓ 8.3)	87.6 ± 0.45	60.4 ± 0.60 (↓ 27.2)
CONVNEXT-T	97.2 ± 0.28	97.6 ± 0.21 (↑ 0.4)	42.6 ± 2.30	32.5 ± 0.34 (↓ 10.1)
SWIN-T	97.2 ± 0.24	97.9 ± 0.12 (↑ 0.6)	66.8 ± 0.41	53.6 ± 0.58 (↓ 13.2)
ViT-S	96.9 ± 0.31	97.9 ± 0.26 (↑ 1.0)	57.3 ± 0.96	49.0 ± 0.53 (↓ 8.3)
SWINV2-T	96.4 ± 0.25	97.7 ± 0.21 (↑ 1.2)	64.8 ± 2.62	55.3 ± 0.95 (↓ 9.5)
DEiT-S	96.5 ± 0.14	97.5 ± 0.21 (↑ 1.0)	58.0 ± 2.13	47.7 ± 0.68 (↓ 10.4)
CONVFOMER-S18	97.8 ± 0.22	98.2 ± 0.16 (↑ 0.4)	70.0 ± 0.24	54.3 ± 0.63 (↓ 15.7)
CAFORMER-S18	97.9 ± 0.14	98.5 ± 0.31 (↑ 0.6)	68.1 ± 1.84	50.2 ± 0.42 (↓ 17.8)
RANDFORMER-S36	95.7 ± 0.27	95.9 ± 0.14 (↑ 0.2)	47.9 ± 1.77	39.6 ± 0.20 (↓ 8.3)
IDENTITYFORMER-S36	95.4 ± 0.26	96.1 ± 0.37 (↑ 0.7)	49.7 ± 3.41	46.1 ± 1.37 (↓ 3.7)
POOLFORMER-S36	96.8 ± 0.20	97.1 ± 0.40 (↑ 0.3)	46.3 ± 2.13	37.5 ± 0.47 (↓ 8.8)
RIFORMER-S36	96.4 ± 0.14	96.7 ± 0.26 (↑ 0.3)	55.9 ± 0.32	40.8 ± 0.41 (↓ 15.0)
MLPMIXER-S/16	96.0 ± 0.45	96.5 ± 0.45 (↑ 0.4)	70.1 ± 0.14	63.2 ± 0.59 (↓ 6.9)
RESMLP-S24	96.2 ± 0.38	96.4 ± 0.03 (↑ 0.3)	64.1 ± 0.72	35.0 ± 2.56 (↓ 29.1)
GMLP-S	96.3 ± 0.41	97.9 ± 0.24 (↑ 1.6)	71.6 ± 0.17	59.4 ± 0.26 (↓ 12.2)
CONVMIXER-768/32	97.3 ± 0.36	75.7 ± 5.31 (↓ 21.6)	69.8 ± 1.84	20.7 ± 0.38 (↓ 49.1)
CoATNET-0	97.0 ± 0.29	94.7 ± 0.29 (↓ 2.3)	79.8 ± 0.71	47.7 ± 0.59 (↓ 32.1)
MAXViT-T	97.3 ± 0.25	96.7 ± 0.16 (↓ 0.6)	83.4 ± 0.74	50.9 ± .041 (↓ 32.6)
Table 12: A comprehensive evaluation of the accuracy achieved by various model families on the PACS dataset under different operational configurations. This includes centrally stored data and federated learning scenarios, with and without ImageNet pretraining. The results offer a detailed insight into the relative effectiveness of the model families under these varying conditions, providing a clear understanding of the impact of pretraining on model performance.
10.5 Convergence Speed
The efficiency of convergence is crucial in the context of federated learning. The inherent complexity of this learning paradigm imposes communication overhead for each update iteration, reinforcing the need for rapid convergence. Fast convergence — reaching the optimal model parameters in fewer iterations — minimizes communication demands, reducing bandwidth usage and overall training time. Faster convergence plays a significant role in real-world federated learning applications where devices may be intermittently available or have unreliable connections. We show that architecture selection (architectural choices) has a significant impact on the convergence speed in federated learning scenarios. The rate of convergence among models varies notably. Notably, the best-performing models we highlighted, specifically ViT and MetaFormers, also exhibit accelerated convergence in most of the experiments. Surprisingly, within the GLDk-23 set, CAFormer lags with the slowest convergence rate, suggesting a need for more in-depth exploration of complex large-scale settings in future studies.
a) CIFAR-10
b) ISIC-2019
c) CelebA
Figure 5: Comparative evaluation of convergence rates for different neural network architectures across diverse datasets. (a) CIFAR-10, (b) Fed-ISIC2019, (c) CelebA, and (d) GLD-23K. Convergence time is measured in terms of communication rounds. This comparison highlights the significant reduction in convergence time enabled by architectural changes.
d) GLD-23K
11 Implementation Details
11.1 Machines
We simulate the federated learning setup (1 server and N devices) on a single machine with 32 Intel(R) Xeon(R) Silver 4215 CPU and 1 NVidia Quadro RTX 6000 GPU.
11.2 Hyperparameter Search
Our study only tests a limited set of hyperparameters for the optimization methods, mostly adhering to the respective original authors' recommendations. For FedProx, we conduct a hyperparameter search across a spectrum of  $\mu$  values, specifically  $\mu \in \{10^{-4}, 10^{-3}, 10^{-1}, 1\}$  keeping the clients learning rate at  $\eta = 3 \times 10^{-2}$ .
For SCAFFOLD, we perform a hyperparameter search over a grid of client learning rates  $\eta \in \{10^{-4}, 3 \times 10^{-4}, 10^{-3}, 3 \times 10^{-3}, 10^{-2}, 3 \times 10^{-2}, 10^{-1}, 3 \times 10^{-1}, 1\}$  with global learning rate  $\eta_{gl} = 1$ . Hyperparameter tuning is harder for FedAvgM as it involves an additional hyperparameter for momentum  $\beta$ . In our search, we considered  $\beta \in \{0, 0.5, 0.7, 0.9, 0.99\}$  and server learning  $\eta_{sr} \in \{10^{-4}, 3 \times 10^{-4}, 10^{-3}, 3 \times 10^{-3}, 10^{-2}, 3 \times 10^{-2}, 10^{-1}, 3 \times 10^{-1}, 1\}$ . The learning rate of the client optimizer is held constant at  $\eta = 3 \times 10^{-2}$ .
Although additional tuning of the regularization parameters may sometimes yield improved empirical performance, we do not anticipate a significant impact on the results discussed. This demonstrates the advantage of architectural design choices over off-the-shelf optimization methods, which can be integrated with minimal modifications to the training process.
Figure 6: Visualization of the results from a hyperparameter grid search for the FedAvgM algorithm on the federated version of CelebA dataset. Three different architectures were tested: ResNet, ConvNext, and CAFormer. The explored hyperparameters include momentum and server learning rate.
11.3 Architectures
Ensuring comparable models is indeed crucial for obtaining fair results. All of the architectures tested (with the exception of MLP-Mixer) have a similar size, measured in number of parameters (21-31M), and similar computational complexity, measured in FLOPS (4-6G). Other elements that might impact the computational cost, input size, etc., are also kept constant.
Regarding the variations in pre-trained accuracy on ImageNet, there are slight differences in performance among the models. However, it is essential to consider that this diversity is inevitable due to the wide range of architectures we incorporated that span across several years of research. To address fairness in our comparisons, we analyzed the absolute numerical result of the experiments and discussed the drop in performance for each model compared to its own centralized version of the training. Furthermore, all models perform similarly in the centralized setting on CIFAR-10, while their performances significantly differ in challenging splits. Both analysis allows us to assess the relative impact of each model's modifications while accounting for the individual model's baseline.
MODEL	PARAM	FLOPS	IMAGENET-1K ACC
RESNET-50	26M	4.1G	76.2
EFFICIENTNET-B5*	30M	2.4G	83.6
CONVNEXT-T	29M	4.5G	82.1
SWIN-T	29M	4.5G	81.3
ViT-S*	22M	4.5G	83.1
SWINV2-T*	28M	4.6G	83.5
DEiT-S	22M	4.6G	79.8
CONVFOMER-S18	27M	3.9G	83.7
CAFORMER-S18	26M	4.1G	84.1
RANDFORMER-S36	31M	5.2G	79.5
IDENTITYFORMER-S36	31M	5G	79.3
POOLFORMER-S36	31M	5G	81.4
RIFOMER-S36	31M	5G	81.3
MLPMIXER-S/16*	59M	13G	76.4
RESMLP-S24	30M	6G	79.4
GMLP-S*	20M	6G	79.6
CONVMIXER-768/32*	21M	20G	80.16
COATNET-0	25M	4.2G	81.6
MAXViT-T	31M	5.6G	83.6
Table 13: Comparative analysis of examined architectures based on key performance and complexity metrics. The metrics include the number of parameters, Floating Point Operations Per Second (FLOPS), and ImageNet-1K performance results. *FLOPS were computed using the Fvcore library.