SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limitations of the benchmarks. From this table, we can see that SCAFFOLD and FedCM both require double communication costs than the vanilla FedAvg. They adopt the correction term (variance reduction and client-level momentum) to revise each local iteration. Though this achieves good performance, we must indicate that under the millions of edge devices in the FL paradigm, this may introduce a very heavy communication bottleneck. In addition, the FedSAM method considers adopting the local SAM optimizer instead of ERM to approach the flat minimal. However, it requires double gradient calculations per iteration. For the very large model, it brings a large calculation cost that can not be neglected. SCAFFOLD and FedDyn are required to store  $3 \times$  vectors on each local devices. This is also a limitation for the light device, i.e. mobiles.
We also test the practical wall-clock time on real devices. Our experiment environments are stated as follows:
Table 9: Experiment environments.
GPU	CUDA	Driver Version	CUDA Version	Platform
Tesla-V100 (16GB)	NVIDIA-SMI 470.57.02	470.57.02	11.4	Pytorch-1.12.1
In the following table, we test the wall-clock time cost of each method:
Table 10: Wall-clock time cost (s/round).
FedAvg	FedAdam	FedSAM	SCAFFOLD	FedDyn	FedCM	FedInit
10%-100 ratio	19.38	23.22	30.23	28.61	23.84	22.63	20.41
1×	1.19×	1.56×	1.47×	1.23×	1.17×	1.05×
5%-200 ratio	15.87	17.50	22.18	24.49	20.61	18.19	16.14
1×	1.10×	1.40×	1.54×	1.30×	1.15×	1.02×
From this table, due to the different communication costs and calculation costs, the practical wall-clock time is different for each method. Generally, FedAvg adopts the local-SGD updates without any additional calculations. FedAdam adopts similar local-SGD updates and an adaptive optimizer on the global server. FedSAM calculation double gradients, which is the main reason for being slowest among the benchmarks. SCAFFOLD, FedDyn, and FedCM are required to calculate some additional vectors to correct the local updates. Therefore they need some additional time costs. Our proposed FedInit only adopts an additional initialization calculation, which requires the same costs as FedAvg.
B.2.4 Training Efficiency: Communication Rounds and Time Costs
In this part, we mainly show the results of the training efficiency. We set the target accuracy and compare their required communication rounds and training time respectively. We test on the ResNet-18-GN model with the  $10\% - 100$  Dir-0.1 splitting.
Table 11: We train 500 rounds on CIFAR-10 and 800 rounds on CIFAR-100. " -" means the corresponding method can not achieve the target accuracy during the training processes.
Method	CIFAR-10 (≥70%)	CIFAR-100 (≥30%)
Round	Time (s)	Round	Time (s)
Speed Ratio	Speed Ratio	Speed Ratio	Speed Ratio
FedAvg	371	1×	7189	1×	191	1×	3701	1×
FedAdam	489	0.76×	11354	0.63×	256	0.74×	5944	0.62×
FedSAM	377	0.98×	11396	0.63×	204	0.93×	6166	0.60×
SCAFFOLD	248	1.50×	7095	1.01×	211	0.90×	6036	0.61×
FedDyn	192	1.93×	4577	1.57×	122	1.56×	2908	1.27×
FedCM	183	2.02×	4141	1.73×	95	2.01×	2149	1.72×
FedInit	172	2.15×	3510	2.04×	132	1.44×	2694	1.37×
The setups of the test environment are stated in Table 9. According to this table, we clearly see that some advanced methods, i.e. SCAFFOLD and FedDyn, are efficient on the communication round  $T$ . However, due to the additional costs of each training iteration, they must spend more time on the total training. FedInit is a very light and practical method, which only adopts a relaxed initialization on the FedAvg method, which makes it to be better and even achieves SOTA results.