# AutoFL: Towards AutoML in a Federated Learning Context
## 5 Evaluation
### Homogeneous Federation: i.i.d. versus Non-i.i.d
training data, each client had about 3000 samples of two classes each and only 2 samples of the other eight classes. In both distributions, each client has a similar amount of training data. Figure 5 compares the distribution of the results for the 100 configurations evaluated by the Bayesian optimization process, and this for a baseline with only one client as well as two configurations where the MNIST or FashionMNIST training data was partitioned across 10 clients in either an i.i.d. or non-i.i.d. manner. Based on the above observations, the accuracy of the 100 single baseline configurations and the i.i.d. configurations are comparable, whereas the non-i.i.d. configurations did not converge to the same accuracy within the 20 federated training and averaging rounds. Note though, that Figure 5 gives a statistical overview of all the 100 individual configurations, including the suboptimal ones. However, we should rather compare the best configuration after the Bayesian optimization process has been completed. Hence, the accuracy on the test set for the best model found is shown in Table 2. The accuracy values for the MNIST single baseline and i.i.d. configurations are comparable to the best ones reported in Table 1. The accuracy values for the CIFAR-10 dataset are reasonably low compared to the state-of-the-art, and we expect this outcome to be due to the limited number of training rounds (i.e., 20 rounds) as well as the low number of configurations (i.e., 100 configurations) for the Bayesian process to evaluate. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Dataset** & **i.i.d.** & **Non-i.i.d.** & **Single** \\ \hline MNIST & 0.9826 & 0.9411 & 0.9813 \\ FashionMNIST & 0.8766 & 0.7589 & 0.8919 \\ CIFAR-10 & 0.6367 & 0.5020 & 0.6839 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Best accuracy of 100 configurations after 20 federated averaging rounds. Figure 5: Benchmarking 100 ML models in i.i.d. and non-i.i.d. dataset configurations on clients with homogeneous resource availability. When comparing the wall clock time for the coordinating server to complete the federated learning, there is hardly any difference between the i.i.d. and non-i.i.d. dataset configurations, neither for the MNIST dataset nor for the FashionMNIST dataset. This is as expected as the same amount of client nodes process their part of the training data. When compared to the single baseline configuration, the overall wall clock time on the coordinating server is higher as now a single client has to learn from the whole training dataset. The fact that the coordinating server now only needs to communicate with 1 client rather than with 10 clients has no significant effect. Similarly, the CPU time on the clients is significantly higher for the single baseline configuration, although not 10 times as high as compared to the i.i.d. dataset configuration. As mentioned earlier, we are exploring other optimization objectives beyond the accuracy of the model. For example, in the case of the i.i.d. MNIST dataset, the best model obtained an accuracy on the test set of 0.9826. However, in the 100 configurations tested, there were 10 with an accuracy higher than 0.98, as depicted in Table 3.  
Of these ten configurations, five are Pareto-optimal (marked in bold) because they have either (a) a higher accuracy, (b) a lower memory usage, or (c) a lower CPU usage compared to the other configurations. Alternatively, for the non-Pareto-optimal configurations, there exists a configuration (last column) that performs equally well or better for all the metrics. The above result is a mere demonstration of how multiple optimization objectives can be considered in parallel. For example, if memory is not a concern, then this trade-off can be dropped or replaced with another one (e.g., network usage at either the coordinating server or the individual clients). Furthermore, the Pareto-optimal configurations were identified with relative comparisons.