# Lazy Aggregation for Heterogeneous Federated Learning
## 4 Evaluation
### Effects of the Proposed Algorithm  
#### 4.2.1 Performance  
There are several points to explain: (1) The highest score of testing is taken after training with 300 rounds on Mnist and Emnist or 500 rounds on Cifar10. (2) The learning rates of the experiments are unified 0.002. (3) Other hyperparameters are as follows: FedProx: offset penalty coefficient u = 0.01; FedDyn: regularity coefficient \(\alpha\) = 0.01; CFL: mean-norm threshold eps1 = 0.035, max-norm threshold eps2 = 0.5; FedLA: WDR threshold \(\varepsilon\) = 0.02; and FedLA: historical gradient momentum coefficient \(\mu\) = 0.5. Differences from \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Samples** & **Classes** & **Devices** & **Partitions** & **Models** \\ \hline MNIST & 69,035 & 10 & & IID & MLP & 101,770 \\ CIFAR10 & 60,000 & & 100 & Class 1/3 & CNN & 553,514 \\ EMNIST (letters) & 145,600 & 26 & & Dir-0.1/0.3 & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of federated datasets and models. Figure 2: Schematic diagram of class distribution of data partitions. the above will be noted. We found that the FedLA and FedLAM algorithms perform well in most scenarios, as shown in Table 2, compared with the baseline algorithms. On Mnist, FedLAM is ahead of other algorithms. Compared with FedAvg, it leads by 3.8% in IID, and leads comprehensive by 2.9% in non-IID, especially by 7.3% in class 1. For the more difficult task, Cifar10, it increases by 5.5% in IID and there is an overall increase of 3.2% in non-IID. In most cases, FedProx is very close to FedAvg. The method of adding a proximal term doesn't overcome gradient conflicts caused by heterogeneous data but only reduces excessive gradient offset. As for the straggler devices mentioned in Section 5.2[7], we will not explore this case in this article. FedDyn performs well on Emnist by aligning the local targets with the global target in real-time through an attached dynamic regularizer. As shown in Figure 3, FedDyn has a fast convergence speed in the early stage, but the accuracy of the global model is stagnant in the later stage, and its loss fluctuates significantly. The main reason is that it doesn't adapt to the constant learning rate which needs to be adjusted flexibly. CFL separates the devices with gradient conflicts to build a flesh model and utilizes multiple models to adapt to devices with different distributions to improve comprehensive accuracy. For Cifar10 in class 1, the performances of other algorithms are not satisfactory, and CFL uses recursive dual partition to separate the different classes of devices and bring test accuracies to 100%. We also observed that the pseudo-global model has a poor effect, which verifies the great differences between classes. Although FedLA mitigates the gradient conflict caused by insufficient sampling in a way, it is limited by the aggregation method and it's inability to fuse gradients well in Cifar10 (class 1). Gradient projection [33] technology solves the impact of opposing gradients. In non-IID scenarios, FedLA has a better performance than FedAvg. It reduces redundant aggregation and finds parallel groups of devices which are consistent with global distribution by computing WDR and overcomes the impact of data sparsity on a single device. Furthermore, the collected gradients could be used to correct vertical oscillation and walk out of the steep fall. FedLAM supplemented by CDM has the best performance in most cases, which is an exciting optimization.