SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Setup on MNIST and FMNIST. We consider five additional tasks: 1) training Logistic Regression on MNIST, 2) training Multi-Layer Perceptron (MLP) on MNIST, 3) training LeNet-5 (LeCun et al., 1998) on MNIST, 4) training LeNet-5 on FMNIST, 5) training CNN on FMNIST. We partition the training sets of both MNIST and FMNIST into 500 clients by extended Dirichlet strategy  $C = 1$  and  $C = 2$  (with  $\alpha = 10.0$ ) and spare the test sets for computing test accuracy. We apply gradient clipping with the max norm of 10 for PFL and 50 for SFL. We find the best learning rate by grid search. This is done by running algorithms for 1000 training rounds and choosing the learning rate that achieves the highest test accuracy averaged over the last 40 training rounds. Note that since tasks on MNIST/FMNIST are quite simpler than that on CIFAR-10/CINIC-10, we do not use the coarse, fine-grained search. The results of grid search are given in Table 10. Other hyperparameters without being stated explicitly are identical to that of CIFAR-10/CINIC-10 in the main body.
Results of MNIST and FMNIST. The results of these five tasks are in Figures 13, 14 and Table 11. In the tasks MNIST/MLP, MNIST/LeNet-5, FMNIST/CNN, the performance of SFL is better when  $C = 1$ , which is consistent with the analysis in Subsection 4.2. However, we note that SFL shows worse even when  $C = 1$  in simpler tasks MNIST/Logistic and FMNIST/LeNet-5, especially when  $K$  is large. This may be because the (objective function) heterogeneity on simple models and datasets is limited even with extreme data distributions (i.e.,  $C = 1$ ). Thus, more extensive experiments are still required before drawing conclusions, which is beyond the scope of this paper.
Figure 13: Experimental results on MNIST. For the best viewing experience, we apply moving average over a window length of  $8\%$  of the data points. Note that "Training Loss/Accuracy" are computed over the training data of participating clients, "Training-R Loss/Accuracy" are computed over the training data of random clients and "Test Accuracy" are computed over the original test set.
Figure 14: Experimental results on FMNIST. For the best viewing experience, we apply moving average over a window length of  $8\%$  of the data points. Note that "Training Loss/Accuracy" are computed over the training data of participating clients, "Training-R Loss/Accuracy" are computed over the training data of random clients and "Test Accuracy" are computed over the original test set.
Table 11: Test accuracy results for various settings. We run PFL and SFL for 1000 training rounds for MNIST and FMNIST and 4000 training rounds for CIFAR-10 and CINIC-10 with 3 different random seeds. The results are computed over the random seeds and the last 40 training rounds for MNIST and FMNIST and the last 100 training rounds for CIFAR-10 and CINIC-10. The better results (with more than  $1\%$  advantage for MNIST/FMNIST and  $2\%$  advantage for CIFAR-10/CINIC-10) between PFL and SFL in each setting are marked in color.
Setup	C=1	C=2
Dataset	Model	Method	K=5	K=20	K=50	K=5	K=20
MNIST	Logistic	PFL	91.10±0.35	89.46±1.20	87.82±1.98	91.69±0.17	91.19±0.47
SFL	91.09±0.67	87.11±2.09	80.94±3.70	91.89±0.32	90.16±1.10
MLP	PFL	93.61±1.42	91.84±2.20	90.27±3.17	95.65±0.38	95.46±0.47
SFL	95.91±0.33	95.90±0.44	95.25±0.76	96.26±0.22	96.35±0.22
LeNet-5	PFL	98.21±0.40	98.02±0.63	97.21±1.55	98.94±0.09	98.97±0.10
SFL	98.90±0.18	98.87±0.19	98.79±0.19	98.91±0.11	99.07±0.12
FMNIST	LeNet-5	PFL	82.57±2.03	81.09±3.19	78.22±4.38	85.86±0.87	86.35±1.12
SFL	83.97±2.42	79.39±2.59	76.21±2.95	86.52±1.67	84.69±2.26
CNN	PFL	86.61±1.62	85.40±2.07	84.62±2.18	88.61±0.93	89.16±0.77
SFL	88.03±1.28	86.75±1.39	85.44±1.66	89.91±0.96	89.60±1.01
CIFAR-10	VGG-9	PFL	67.61±4.02	62.00±4.90	45.77±5.91	78.42±1.47	78.88±1.35
SFL	78.43±2.46	72.61±3.27	68.86±4.19	82.56±1.68	82.18±1.97
ResNet-18	PFL	52.12±6.09	44.58±4.79	34.29±4.99	80.27±1.52	82.27±1.55
SFL	83.44±1.83	76.97±4.82	68.91±4.29	87.16±1.34	84.90±3.53
CINIC-10	VGG-9	PFL	52.61±3.19	45.98±4.29	34.08±4.77	55.84±0.55	53.41±0.62
SFL	59.11±0.74	58.71±0.98	56.67±1.18	60.82±0.61	59.78±0.79
ResNet-18	PFL	41.12±4.28	33.19±4.73	24.71±4.89	57.70±1.04	55.59±1.32
SFL	60.36±1.37	51.84±2.15	44.95±2.97	64.17±1.06	58.05±2.54
Figure 15: Experimental results on CIFAR-10. For the best viewing experience, we apply moving average over a window length of  $4\%$  of the data points. Note that "Training Loss/Accuracy" are computed over the training data of participating clients, "Training-R Loss/Accuracy" are computed over the training data of random clients and "Test Accuracy" are computed over the original test set. The shaded areas show the standard deviation with 3 random seeds.
Figure 16: Experimental results on CINIC-10. For the best viewing experience, we apply moving average over a window length of  $4\%$  of the data points. Note that "Training Loss/Accuracy" are computed over the training data of participating clients, "Training-R Loss/Accuracy" are computed over the training data of random clients and "Test Accuracy" are computed over the original test set. The shaded areas show the standard deviation with 3 random seeds.