SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Convergence performance of theoretical DELTA on split-FashionMNIST and practical DELTA on FEMNIST. Figure 14(a) illustrates the theoretical DELTA outperforms other methods in convergence speed. Figure 14(b) indicates that cluster-based IS and practical DELTA exhibit rapid initial accuracy improvement, while practical DELTA and practical IS achieve higher accuracy in the end.
Ablation study for DELTA with different sampled numbers. Figure 15 shows the accuracy performance of practical DELTA algorithms on FEMNIST with different sampled numbers of clients. In particular, the larger number of sampled clients, the faster the convergence speed is. This is consistent with our theoretical result (Corollary 4.2).
(a)  $\nu = 10$
(b)  $\nu = 5$
(c)  $\nu = 1$
(d)  $\nu = 0.5$
(e)  $\nu = 0.1$
Figure 12: Performance of different algorithms on the regression model with different (small) noise settings.
(b)  $\nu = 20$
(c)  $\nu = 10$
(a)  $\nu = 30$
(d)  $\nu = 5$
(e)  $\nu = 1$
Figure 13: Performance of different algorithms on synthetic data with different noise settings. Specifically, for testing the large client number setting, in each round, 10 out of 200 clients are selected to participate in training.
Performance on FashionMNIST and CIFAR-10. For CIFAR-10, we report the mean of the best 10 test accuracies on test data. In Table 2, we compare the performance of DELTA, FedIS, and FedAvg on non-IID FashionMNIST and CIFAR-10 datasets. Specifically, we use  $\alpha = 0.1$  for FashionMNIST and  $\alpha = 0.5$  for CIFAR-10 to split the datasets. As for Multinomial Distribution (MD) sampling [29], it samples based on the clients' data ratio and average aggregates. It is symmetric in sampling and aggregation with FedAvg, with similar performance. It can be seen that DELTA has better accuracy than FedIS, while both DELTA and FedIS outperform FedAvg with the same number of communication rounds.
Assessing the Compatibility of FedIS with Other Optimization Methods. In Table 4, we demonstrate that DELTA and FedIS are compatible with other FL optimization algorithms, such as Fedprox [29] and FedMIME [20]. Furthermore, DELTA maintains its superiority in this setting.
(a) Performance FashionMNIST
of algorithms
on split-
(b) Performance of algorithms on FEMNIST
Figure 14: Performance comparison of accuracy using different sampling algorithms.
Figure 15: Ablation study of the number of sampled clients.
Table 4: Performance of sampling algorithms integrated with momentum and prox. We run 500 communication rounds on CIFAR-10 for each algorithm. We report the mean of maximum 5 accuracies for test datasets and the number of communication rounds to reach the threshold accuracy.
Algorithm	Sampling + momentum	Sampling + proximal
Acc (%)	Rounds for 65%	Acc (%)	rounds for 65%
FedAvg (w/ uniform sampling)	0.6567	390	0.6596	283
FedIS	0.6571	252	0.661	266
DELTA	0.6604	283	0.6677	252
In Table 5, we demonstrate that DELTA and FedIS are compatible with other variance reduction algorithms, like FedVARP [18].
It is worth noting that FedVARP utilizes the historic update to approximate the unparticipated clients' updates. However, in this setting, the improvement of the sampling strategy on the results is somewhat reduced. This is because the sampling strategy is slightly redundant when all users are involved. Thus, when VARP and DELTA/FedIS are combined, instead of reassigning weights in the aggregation step, we use (75) or (13) to select the current round update clients and then average aggregate the updates of all clients. One can see that the combination of DELTA/FedIS and VARP can still show the advantages of sampling.
Table 5: Performance of DELTA/FedIS in combination with FedVARP. We run 500 communication rounds on FashionMNIST with  $\alpha = 0.1$  for each algorithm. We report the mean of maximum 5 accuracies for test datasets and the number of communication rounds to reach the threshold accuracy.
Algorithm	FashionMNIST
Acc (%)	Rounds for 73%
FedVARP	73.81 ± 0.18	470
FedIS + FedVARP	73.96 ± 0.14	452
DELTA +FedVARP	74.22± 0.14	436
In addition to the above optimization methods like VARP, we also conduct experiments on LEAF (FEMNIST and CelebA) with other non-vanilla SGD algorithms, namely Adagrad and Adam, to show that the proposed client selection framework be applied to federated learning algorithms other than vanilla SGD. The results are shown in Table 6 and Table 7.
Table 6: Performance of sampling algorithms integrated with Adagrad and Adam on FEMNIST. We run 1000 communication rounds on FEMNIST for each algorithm. In particular, we set the global learning rate  $\eta = 0.01$  to ensure convergence of Adagrad and Adam. We report the mean of the maximum of 5 accuracies for test datasets and the average number of communication rounds that reach the threshold accuracy  $80\%$ .
Algorithm	Sampling + Adagrad	Sampling + Adam
Acc (%)	Rounds for 80%	Acc (%)	rounds for 80%
FedAvg	80.93 ± 0.08	893 (1.0×)	80.04 ± 0.85	882 (1.0×)
Cluster-based IS	80.69±0.42	760 (1.17×)	79.11±0.18	-
FedIS	80.96 ± 0.31	723 (1.24×)	80.10 ±0.25	787 (1.12×)
DELTA	81.79 ± 0.09	612 (1.46×)	80.92 ±0.27	600 (1.47×)
Table 7: Performance of sampling algorithms integrated with Adagrad and Adam on CelebA. We run 1000 communication rounds on CelebA for each algorithm. In particular, we set the global learning rate  $\eta = 0.01$  to ensure convergence of Adagrad and Adam. We report the mean of the maximum of 5 accuracies for test datasets and the average number of communication rounds that reach the threshold accuracy  $80\%$ .
Algorithm	Sampling + Adagrad	Sampling + Adam
Acc (%)	Rounds for 80%	Acc (%)	rounds for 80%
FedAvg	88.92 ± 0.08	329 (1.0×)	89.04 ± 0.22	244 (1.0×)
Cluster-based IS	89.71±0.10	329 (1.0×)	89.26± 0.19	164 (1.49×)
FedIS	90.14 ± 0.01	243 (1.35×)	89.92 ±0.05	140 (1.74×)
DELTA	90.38 ± 0.02	214 (1.54×)	90.58 ±0.07	109 (2.24×)
Table 8 provide results under some common thresholds, including  $50\%$  for CIFAR-10 and  $80\%$  for CelebA, to replace  $54\%$  for CIFAR-10 and  $85\%$  for CelebA in Table 2.
Table 8: Performance of sampling algorithms under the common thresholds. We report the results of algorithms on CIFAR-10 with  $50\%$  threshold and on CelebA with  $80\%$  threshold. We run 500 communication rounds for each algorithm. We report the mean of the maximum of 5 accuracies for test datasets and the average number of communication rounds that reach the threshold accuracy.
Algorithm	CIFAR-10	CelebA
Acc (%)	Rounds for 50%	Acc (%)	rounds for 80%
FedAvg	54.28 ± 0.29	181 (1.0×)	85.92 ± 0.89	339 (1.0×)
Cluster-based IS	54.83± 0.02	187 (0.91×)	86.77± 0.11	303 (1.11×)
FedIS	55.05 ± 0.27	168 (1.07×)	89.12 ±0.71	261 (1.29×)
DELTA	55.20 ± 0.26	151 (1.20×)	89.67 ±0.56	257 (1.32×)
Albation study for  $\alpha$ . In Table 9, we experiment with different choices of heterogeneity  $\alpha$  in the CIFAR-10 dataset. The parameter of heterogeneity  $\alpha$  changes from 0.1 to 0.5 to 1. We observe a consistent improvement of DELTA compared to the other algorithms. This shows that DELTA is robust to changes in the level of heterogeneity in the data distribution.
Table 9: Performance of algorithms under different  $\alpha$ . We run 500 communication rounds on CIFAR10 for each algorithm (with momentum). We report the mean of maximum 5 accuracies for test datasets and the number of communication rounds to reach the threshold accuracy.
Algorithm	α = 0.1	α = 0.5	α = 1.0
Acc (%)	Rounds for 42%	Acc (%)	rounds for 65%	Acc (%)	rounds for 71%
FedAvg (w/ uniform sampling)	0.4209	263	0.6567	283	0.7183	246
FedIS	0.427	305	0.6571	252	0.7218	239
DELTA	0.4311	209	0.6604	283	0.7248	221