SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To evaluate the compatibility of advanced architectures with federated optimization methods, we consider the widely used ResNet-50 model in our study and extend it to more recent computer vision architectures. We compare different models with the baseline ResNet-50, combined with diverse optimization techniques across four federated datasets. Alongside ResNet, we employ ConvNext, proposed to narrow the gap of convolutional with transformer architectures, and CAFormer, a recently introduced model in the Metaformer family. We consider four of the most popular optimizers in federated learning, namely FedAVG [49], FedAVGM (enhanced with momentum) [58], FedProx [38], and SCAFFOLD [29].
Results reported in Table 1 can be summarized as follows. First, using the standard FedAVG replacing ResNet-50 with the above computer vision architectures significantly increases accuracy across all datasets, especially with the CAFormer architecture, suggesting that architectural modification can significantly boost performance. Secondly, combining optimizers with complex models does not enhance results, with most of the performance maintaining a steady status, except for FedAVGM applied with ResNet on the CIFAR-10 dataset. In fact, for the Fed-ISIC2019 dataset, some of the optimizers proved to be detrimental to the performance of the network when compared to the simple FedAVG (e.g., SCAFFOLD for all architectures, and ResNet with FedAVGM and FedProx). The lack of optimization benefits across different datasets and architectures can be attributed to the complexity of modern deep learning architectures, such as Transformer-based models, which are significantly more complex than models typically used in FL research, leading to a more challenging optimization landscape. Applying these optimization techniques to the tough non-convex optimization problems encountered during the training of modern deep neural networks remains an unresolved issue. Most FL methods have demonstrated efficacy for certain types of non-convex problems under specific assumptions that might not be applicable to these complex models in practice [10, 30]. Our experiments reveal that the naive application of these techniques, combined with advanced models, often leads to negligible improvement.
This leaves us with the unresolved question of how we can optimize federated learning performance for practical applications, bridge the gap between advancements in the computer vision and FL communities, and address the issue of closing the performance gap with centralized settings. Inspired by our findings, we inspect the benefits of architectural design choices. Replacing the architecture is an easy-to-implement strategy, given the wide array of pretrained models available and their proven effectiveness on various vision tasks. Moreover, cutting-edge architectures often include design elements that inherently facilitate learning, such as superior feature extraction, incorporation of prior knowledge (inductive biases), and regularization that could mimic the effects of federated optimization techniques. In the following Section 6.2, we explore the possibility of leveraging the inherent strengths of different computer vision models to enhance performance and naturally bridge the non-IID gap in federated learning.
6.2 Tackling the Heterogeneity Challenge with Architectural Design
We comprehensively compare network architectures from diverse network families, such as CNNs, Vision Transformers, Hybrids, MLP-Mixers, and Metaformers, with a particular focus on how each architecture family performs against each other. Tables 2 and 3 show our results on CIFAR, CelebA, Fed-ISIC2019, and GLD-23k datasets.
Architectural Comparison on CIFAR-10: On CIFAR-10, all models show performance degradation compared to the ideal centralized training, with most of the architectures displaying a significant reduction in accuracy as we progress to the non-IID splits (see Figure 2). Table 2 shows the performance of each model across all splits and the centralized version. The key observations from this study are summarized below.
Convolutional Networks: Convolutional models exhibit the overall highest performance degradation among all families. ResNet [19] and EfficientNet [62] are found to be non-robust in heterogeneous settings, with a loss of 36 and 17.1 points, respectively, observed in Split-3. While ConvNeXt [45] shows improvement over its predecessors, it still falls short of achieving top results when compared to other model families.
Transformers and Metaformers: Transformers consistently demonstrate robust performance, particularly in non-IID data partitions, exhibiting only a marginal average degradation of 1.7 points. Notably, ViT [60] achieves remarkable resilience, attaining an accuracy of  $96.8\%$  and experiencing a minimal drop of merely  $0.8\%$ , even in the most challenging split. These findings corroborate the
MODEL	CENTRAL	SPLIT-1	SPLIT-2	SPLIT-3
RESNET-50 [19]	95.9 ± 0.21	96.8 ± 0.17	92.7 ± 0.25	59.9 ± 1.87 (↓ 36.0)
EFFICIENTNET-B5 [62]	97.6 ± 0.21	97.8 ± 0.06	95.1 ± 0.06	80.5 ± 0.92 (↓ 17.1)
CONVNEXT-T [45]	97.2 ± 0.15	97.6 ± 0.06	96.6 ± 0.21	93.5 ± 0.36 (↓ 3.7)
SWIN-T [44]	97.6 ± 0.15	97.9 ± 0.10	97.6 ± 0.06	95.7 ± 0.15 (↓ 1.9)
ViT-S [60]	97.6 ± 0.06	98.5 ± 0.10	98.2 ± 0.12	96.8 ± 0.15 (↓ 0.8)
SWINV2-T [43]	96.4 ± 0.00	97.4 ± 0.10	96.4 ± 0.15	93.7 ± 0.38 (↓ 2.7)
DEiT-S [66]	96.8 ± 0.12	97.8 ± 0.06	97.2 ± 0.12	95.2 ± 0.15 (↓ 1.6)
CONVFOMER-S18 [74]	97.9 ± 0.12	98.4 ± 0.12	98.0 ± 0.12	96.8 ± 0.35 (↓ 1.2)
CAFORMER-S18 [74]	98.1 ± 0.18	98.5 ± 0.06	98.3 ± 0.06	97.1 ± 0.14 (↓ 1.0)
RANDFORMER-S36 [74]	95.7 ± 0.12	95.8 ± 0.10	94.6 ± 0.06	91.3 ± 0.35 (↓ 4.4)
IDENTITYFORMER-S36 [74]	95.3 ± 0.26	95.6 ± 0.12	94.6 ± 0.12	90.8 ± 0.45 (↓ 4.5)
POOLFORMER-S36 [73]	97.1 ± 0.25	97.4 ± 0.10	96.9 ± 0.06	95.2 ± 0.12 (↓ 1.9)
RIFOMER-S36 [70]	95.9 ± 0.08	96.6 ± 0.16	95.5 ± 0.15	93.0 ± 0.10 (↓ 2.9)
MLP MIXER-S/16 [64]	94.8 ± 0.10	96.5 ± 0.12	95.6 ± 0.06	92.7 ± 0.06 (↓ 2.1)
RESMLP-S24 [65]	96.2 ± 0.10	97.2 ± 0.10	96.7 ± 0.06	94.1 ± 0.17 (↓ 2.1)
GMLP-S [41]	96.9 ± 0.26	98.0 ± 0.06	97.6 ± 0.12	96.1 ± 0.23 (↓ 0.8)
CONVMIXER-768/32 [67]	97.4 ± 0.12	97.3 ± 0.06	94.4 ± 0.15	74.1 ± 1.07 (↓ 23.3)
COATNET-0 [8]	97.7 ± 0.15	98.3 ± 0.15	97.5 ± 0.06	94.2 ± 0.06 (↓ 3.4)
MAXViT-T [69]	98.0 ± 0.10	98.6 ± 0.10	97.3 ± 0.12	92.0 ± 1.15 (↓ 6.0)
Table 2: Performance of different types of models across all splits of CIFAR-10. The average test accuracy of the model is displayed along with the standard deviation of the experiments. Split-3 shows the degradation compared to the centralized version of the training.
earlier observations made by Qu et al. [57] regarding the robustness of ViT models against data heterogeneity. However, Qu et al. attribute the superior performance of Transformers, compared to convolutional networks, to the robustness of self-attention in mitigating distribution shifts. In contrast, the emergence of the Metaformer family, employing various token-mixers including convolution operation, suggests that the remarkable proficiency exhibited by Transformers may be primarily attributed to the underlying architectural design of Metaformers, rather than the exclusive reliance on self-attention in models like ViT. Indeed, even Metaformers employing basic token mixers, such as RandFormer with a random token mixer, deliver remarkably satisfactory performances. On the challenging Split-3, CAFormer [74] garners the highest result, achieving an accuracy of  $97.1\%$  and delivering exemplary performance across all splits, followed by fully convolutional token mixer (without self-attention) based ConvFormer [74], which is also on par with ViTs.
MLP-Mixers: As for MLP-Mixers, on average, their performance on the Split-3 is close to those of Transformers. The best is the gMLP [41] with  $0.8\%$  of degradation (second only to ViT) and an accuracy of  $96.1\%$ . Finally, Hybrid models achieve reasonable performance, but their results are far behind those of the other families. It is worth noticing that most models achieve higher accuracy in Split-1 compared to the centralized setting. We conjuncture that this is due to a regularizing effect caused by each client's access to the IID data partition (with uniform distribution over 10 classes).
Architectural Comparison on CelebA, Fed-ISIC2019 and GLD-23k: The results for CelebA, Fed-ISIC2019 and GLD-23k are reported in Table 3.
CelebA: The highest accuracies for CelebA are the ones of Swin [44] (89.0) and ConvFormer [74] (88.1), followed by MLP-Mixer [64] and CAFormer [74]. Again, all Metaformer-like architectures (encompassing Transformers and MLP-Mixers) achieve the best results, far surpassing those of CNNs and Hybrid models, whose average performance is 84 and 74.9, respectively.
GLD-23K: On the GLD-23K, we also observe that ViT remains among the highest-performing networks. However, Metaformers such as CaFormer and ConvFormer do not yield high performance as in previous datasets.
Fed-ISIC-2019: Finally, on the Fed-ISIC-2019 dataset, which is naturally federated (due to the differences in the image acquisition systems across clients), we find CAFormer and ViT as the best
MODEL	NORM	MAIN OPERATION	CELEBA	GLD-23K	FED-ISIC2019
RESNET-50 [19]	BN	CONV	84.9 ± 1.48	54.3 ± 0.48	67.6 ± 1.59
EFFICIENTNET-B5 [62]	BN	CONV	79.5 ± 0.50	51.9 ± 1.21	70.0 ± 0.64
CONVNEXT-T [45]	LN	CONV	87.5 ± 0.87	65.5 ± 0.57	78.6 ± 0.29
SWIN-T [44]	LN	SA	89.0 ± 0.79	72.1 ± 0.67	81.9 ± 0.42
ViT-S [60]	LN	SA	87.3 ± 0.53	76.6 ± 0.69	82.3 ± 0.95
SWINV2-T [43]	LN	SA	86.8 ± 0.62	74.4 ± 0.02	81.7 ± 0.50
DEiT-S [66]	LN	SA	87.4 ± 0.60	69.1 ± 1.24	82.3 ± 0.59
CONVFOMER-S18 [74]	LN	CONV	88.1 ± 0.42	53.8 ± 1.11	81.1 ± 1.54
CAFORMER-S18 [74]	LN	SA+CONV	87.5 ± 0.49	57.8 ± 0.54	83.1 ± 0.49
RANDFORMER-S36 [74]	LN	CONV	83.9 ± 0.35	56.3 ± 0.10	77.3 ± 0.23
IDENTITYFORMER-S36 [74]	LN	CONV	85.8 ± 0.49	56.0 ± 2.07	76.9 ± 1.43
POOLFORMER-S36 [73]	LN	CONV+POOL	86.4 ± 0.61	55.8 ± 1.41	79.6 ± 0.29
RIFORMER-S36 [70]	LN	CONV	87.2 ± 0.37	69.4 ± 0.20	81.9 ± 1.37
MLPMIXER-S/16 [64]	LN	CONV	87.9 ± 0.51	71.3 ± 0.32	80.5 ± 1.62
RESMLP-S24 [65]	-	CONV	87.0 ± 0.35	64.8 ± 0.59	81.1 ± 0.45
GMLP-S [41]	LN	CONV	86.6 ± 0.31	67.4 ± 0.16	79.9 ± 1.60
CONVMIXER-768/32[67]	BN	CONV	56.5 ± 1.07	45.0 ± 0.76	58.6 ± 2.02
COATNET-0 [8]	BN+LN	SA+CONV	82.2 ± 1.66	70.4 ± 0.25	66.0 ± 3.40
MAXViT-T [69]	LN	SA	86.1 ± 0.72	71.7 ± 0.90	70.1 ± 1.27
Table 3: Accuracy of the model families on CelebA, GLD-23K, and Fed-ISIC2019. For each model, we report the type of normalization layer, either LayerNorm or BatchNorm (LN/BN), and main operation employed (Convolution/Self-Attention). The mean accuracy of all repetitions is displayed along with the standard deviation.
performing models (83.1 and 82.3 accuracy). In fact, we can see a significantly superior performance from all models following the Metaformer structure, indistinctly of their use of convolutions and/or self-attention in their basic block structure.
In summary, our comprehensive experiments on 19 state-of-the-art architectures demonstrate that the choice of architecture and its components play a noteworthy role in tackling performance degradation caused by data heterogeneity. As a matter of fact, the experiments indicate that selecting an appropriate architecture design can have a greater impact on closing the gap with centralized training than the choice of the optimizer. For example, Table 1 shows that CAFormer with simple FedAVG outperforms all other models even when using more advanced optimization methods.
6.2.1 How architectural design improves performances - Normalization Layer
Based on the results observed in the previous sections, we further investigate the performance reduction by delving into the contribution of specific architectural components. Our objective is to identify some key structural elements to achieve stronger performances in heterogeneous settings.
As shown in Table 3, models making use of Batch Normalization (like ResNet, EfficientNet, and ConvMixer) are consistently outperformed by models with Layer Norm on Split-3. This insight sheds light on why older convolutional (such as ResNet and EfficientNet) models have such high degradation. On the contrary, ConvNext, which uses Layer Norm, reaches satisfactory performance, contradicting the initial conclusion of [57] that convolutions are inherently problematic in the federated learning scenario under non-IID conditions. Moreover, we observe how ConvFormer, a model without Self-attention based blocks (based primarily on convolutions) but with a MetaFormer-like structure and Layer Norms, regularly reaches top performance even under the most challenging data partitions.
To further support our claims, we perform experiments with three different models, namely Pool-Former, CoAtNet, and ResNet. Each model trains with a variant using Batch Norm and another using Layer Norm. Lastly, we include a training scheme with FedBN [39] (a method developed specifically to deal with the performance degradation caused by the Batch Norm layer). Results (displayed in Table 4) show a trend indicating the superiority of Layer Norm over Batch Norm, even when using a tailored aggregation method for the latter (e.g., FedBN [39]).
VARIANT	CENTRAL	SPLIT-3
COATNET-0 BN	97.5 ± 0.04	20.9 ± 0.45 (↓76.6)
COATNET-0 LN *	97.7 ± 0.15	94.2 ± 0.06 (↓3.4)
POOLFORMER-S12 BN	95.3 ± 0.35	51.5 ± 0.32 (↓43.7)
POOLFORMER-S12 LN	95.6 ± 0.09	91.5 ± 0.13 (↓4.1)
RESNET-50 BN	95.9 ± 0.21	59.9 ± 1.87 (↓36.0)
RESNET-50 LN†	92.8 ± 0.16	72.2 ± 3.06 (↓26.6)
RESNET-50 FEDBN [39]	96.1 ± 0.19	59.6 ± 1.40 (↓36.5)
Table 4: Accuracy of ResNet-50, CoAtNext-0, and PoolFormer-S12 with BN and LN in the central and the Split-3 version of CIFAR10. * The original CoAtNet-0 uses BN and LN in convolutional and self-attention blocks, respectively. † LN layers have no pretraining.
Figure 3: Weight divergence with respect to models' centralized version on CIFAR-10. The use of BN leads to a larger weight divergence.
Finally, as in [76], we measure and report the weight divergence of the models with respect to the centralized training. Figure 3 shows the increase of divergence for the models using BN to their LN counterparts, supporting the selection of Layer Norm over Batch Norm as an architectural choice for federated learning.
Our research primarily explores visual recognition tasks within the federated learning domain, and we have not explored potential downstream tasks, such as object detection and segmentation. It's worth noting that visual recognition models often serve as backbones for extracting input image features in downstream tasks, thereby suggesting that our observed performance trends could likely extend to these downstream tasks. We plan to address this research gap in future studies.
Furthermore, our approach did not design a completely new model from scratch. Instead, we demonstrated our results utilizing pre-existing models and components. Our study also underscores the limitations of employing established optimizers (e.g., FedAVG, FedAVGM, FedProx, SCAFFOLD) along with modern network architectures. However, we abstain from plunging into the creation of innovative optimization techniques. Instead, we focus on evaluating diverse architectural families on many practical federated learning datasets and optimizers. As a result, our investigation leaves an intriguing avenue for future research, integrating our findings to develop dedicated architecture and optimizers tailored specifically for federated learning (FL) scenarios. By exploring this direction, we anticipate the potential to further enhance the performance and efficiency of FL systems.
This research expands the architectural domain in federated learning, emphasizing architectural design for computer vision applications under data heterogeneity. Our experiments assess various state-of-the-art models from five architectural families across four federated datasets, demonstrating the potential of addressing FL data heterogeneity architecturally. We show that Metaformer-like architectures exhibit superior robustness in non-IID settings. In contrast to existing studies, we observe that convolution operations are not necessarily a sub-optimal choice for architectures in FL applications and highlight the negative impact of Batch Normalization layer on the model performance. Our study also shows that architectural alterations offer an effective and more practical solution to handle data heterogeneity in non-IID settings.
This work is partially supported by the MBZUAI-WIS Joint Program for AI Research (Project grant number- WIS P008).
[65] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[66] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[67] Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792, 2022.
[68] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[69] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204.01697, 2022.
[70] Jiahao Wang, Songyang Zhang, Yong Liu, Taiqiang Wu, Yujiu Yang, Xihui Liu, Kai Chen, Ping Luo, and Dahua Lin. Riformer: Keep your vision backbone effective while removing token mixer. arXiv preprint arXiv:2304.05659, 2023.
[71] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7623, 2020.
[72] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575-2584, 2020.
[73] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10819-10829, 2022.
[74] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. arXiv preprint arXiv:2210.13452, 2022.
[75] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pages 26311-26329. PMLR, 2022.
[76] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-id data. arXiv preprint arXiv:1806.00582, 2018.
9 A Re-look into existing Federated Computer Vision Approaches
In this section, we explore the landscape of computer vision techniques in existing literature. We begin by examining foundational works from the literature and the computer vision architectures they adopt. Our comparative assessment reveals that our approach incorporates several state-of-the-art vision architectures, previously unexplored in federated learning contexts. Furthermore, we contrast commonly used datasets with those we have chosen for our study. Notably, while our methodology encompasses a range of architectures, we also feature five datasets, four of which are intrinsically federated.
METHOD	VENUE	VISION ARCHITECTURES	VISION DATASETS
FEDAVG [49]	AISTATS'17	2MLP, 2CNN	MNIST, CIFAR-10
FEDAVGM [58]	ARXIV'19	2CNN	CIFAR-10
FEDPROX [38]	MLSYS'20	LOGISTIC REGRESSION	MNIST, FEDMNIST
SCAFFOLD [29]	ICML'20	2MLP	MNIST
FEDNOVA [71]	NEURIPS'20	LOGISTIC REGRESSION, VGG	SYNTHETIC, CIFAR-10
MOON [36]	CVPR'21	6CNN, RESNET	CIFAR-10, CIFAR-100, TINY-IMAGENET
FEDDYN [1]	ICLR'21	6CNN	MNIST, EMNIST, CIFAR-10, CIFAR-100
FEDOPT [58]	ICLR'21	RESNET, 4CNN, 8MLP	CIFAR-10, CIFAR-100, EMNIST
FEDBN [39]	ICLR'21	6CNN	SVHN, USPS HULL, SYNTHETIC DIGITS, MNIST
MIME [28]	NEURIPS'21	LOGISTIC REGRESSION, 4CNN, 2MLP	EMNIST
FEDLC [34]	ICML'22	RESNET	SVHN, CIFAR10 AND CIFAR100, IMAGENET SUBSET
PROSKIP [50]	ICML'22	LOGISTIC REGRESSION	LIBSVM-'w8A'
FEDDC [15]	CVPR'22	LOGISTIC REGRESSION, 2MLP	MNIST, FASHION MNIST, CIFAR10, CIFAR100
2CNN, RESNET	EMNIST, TINY IMAGENET, SYNTHETIC
FEDSHUFFLE [21]	TMLR'22	RESNET	CIFAR-100
FEDEXP [26]	ICLR'23	4CNN, RESNET	EMNIST, CIFAR-10, CIFAR-100, CINIC-10
FEDML [17]	NEURIPS WORKSHOP '20	LOGISTIC REGRESSION, 4CNN	MNIST, FEDMNIST, SYNTHETIC
RESNET, MOBILENET	CIFAR-10, CIFAR-100, CINIC-10
FLOWER [2]	ARXIV'20	4CNN, RESNET	FEMNIST, CIFAR-10, IMAGENET
FEDCV [18]	CVPR'21	EFFICIENTNET, MOBILENET, ViT	CIFAR-100, GLD-23k
QU ET AL. [57]	CVPR'22	RESNET, EFFICIENTNET, ViT, SWIN	RETINA, CIFAR-10, CELEBA
SKEWSCOUT [22]	ICML'20	ALEXNET, GOOGLNET, LENET, RESNET	CIFAR-10, FLICKR-MAMMAL, IMAGENET
HSU ET AL. [24]	ECCV'20	MOBILENETV2, 6CNN	CIFAR-10, CIFAR-100, INATURALIST, GLD-160K
OURS	RESNET-50, EFFICIENTNET, CONVNEXTSWIN, ViT, SWINV2, DEIT,RIFORMER-S36, CONVFORMER, CAFORMER,RANDOMFORMER, IDENTITYFORMER, POOLFORMERMLPMIXER, RESMLP, GMLP,CONVMIXER, COATNET, MAXViT	CIFAR-10, CELEBA, FED-ISIC2019, GLD-23K, PACS
Table 5: Literature review and comparative analysis of various federated learning methods evaluated on computer vision benchmarks. Each method is classified by the publication venue, the vision architecture utilized, and the vision-related datasets employed for testing. Furthermore, we provide specific details for shallow models on the complexity of the network architectures in terms of the number of convolutional (CNN) and fully connected (MLP) layers.
DATASET	IMAGES	CLASSES	CLIENTS	NATURALLY FERERATED
CIFAR-10 [31]	60,000	10	-	No
CELEBA [46]	200,288	2	9,343	YES
FED-ISIC2019 [68, 5, 7]	23,247	8	6	YES
GLD-23K [72]	23,080	203	233	YES
PACS [35]	9,991	7	4	YES
TINYIMAGENET [32]	120,000	203	-	No
MNIST [33]	70,000	10	250	YES
EMNIST [6]	814,255	62	500	YES
FEMNIST [3]	805,263	62	3500	YES
SVHN [52]	600,000	10	-	No
CINIC [9]	270,000	10	-	No
SYNTHETIC DIGITS [59]	12,000	10	-	No
Table 6: Overview of frequently utilized datasets for visual recognition within the field of federated learning. The number of clients indicated for non-naturally federated datasets are subject to each author's unique artificial partition.
9.0.1 Data Heterogeneity - Dataset Analysis
In our experiments, we examined data heterogeneity across several datasets, focusing on Label Distribution Skew (LDS), Feature Distribution Skew (FDS), and Data Distribution Skew (DDS). The LDS denotes an uneven distribution of samples across classes for each client, while DDS reflects variations in the volume of data per client. On the other hand, FDS arises from differences in image features per class across clients, attributable to factors like varied acquisition sensors, image domains, user preferences, and geographical locations.
Specifically, for LDS, we tested CIFAR-10 and created three distinct splits with escalating label skew. This skewness was measured using the KS statistic, enabling a comprehensive exploration of how label imbalances affect our model's efficacy.
For Data Distribution Skew (DDS), our primary dataset was GLDK, which exhibited the most pronounced variations in the amount of data per client. This helped us delve deep into the challenges and implications of data volume discrepancies across clients in a federated setting.
In terms of FDS, we incorporated inherently federated datasets like GLD-23K and Fed-ISIC2019, which exhibit this skew. The GLD-23K dataset spans multiple geographical areas, while Fed-ISIC2019 was gathered using diverse medical equipment, offering a broad spectrum of feature distribution skews. Additionally, we delve deeper into the PACS dataset in Sec. 10.4, which showcases significant domain variations. For this particular study, our federated divisions allocated unique target domains: Photos, Sketches, Cartoons, and Paintings.