# AutoFL: Towards AutoML in a Federated Learning Context
## 5 Evaluation
### Heterogeneous Resource Availability  
In the previous experiments, the clients in the federation had similar resource availability in terms of CPU, memory, and network capacity. In the following experiment, we replaced one of the ten clients with an ARM device, specifically the NVIDIA Jetson TX2 system. The other experimental settings remain the same:  
* Ten clients and one coordinating server in the federation.
* Training data is i.i.d. over all clients.
* One hundred ML model configurations to be selected through Bayesian optimization.
* Twenty federated averaging rounds by the coordinating server.
* One epoch for each local model training round.  
Figure 6 depicts the results of three different experiments with the MNIST dataset that is i.i.d. over ten clients. In the deployment setting (1), all clients have homogeneous  
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Configuration** & **Accuracy** & **Memory (MB)** & **CPU Time (s)** & **Pareto-Optimal Configuration** \\ \hline
**0032** & 0.9824 & 707.252 & 52.782 & \\
**0037** & 0.9826 & 714.510 & 56.720 & \\
0038 & 0.9810 & 714.653 & 56.625 & 0032, 0039 \\
**0039** & 0.9824 & 705.339 & 52.802 & \\
0044 & 0.9822 & 734.507 & 65.311 & 0032, 0037, 0039 \\
**0057** & 0.9808 & 708.911 & 51.312 & \\
0071 & 0.9825 & 744.964 & 76.580 & 0037 \\
0093 & 0.9822 & 747.036 & 77.205 & 0032, 0037, 0039 \\
**0095** & 0.9804 & 708.087 & 51.691 & \\
0100 & 0.9807 & 716.832 & 56.687 & 0032, 0039, 0057 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Configurations with an accuracy \(>=\) 0.98, with Pareto-optimal ones marked in bold.  
resource characteristics, and all ten clients are involved in training and evaluation. In deployment setting (2), one client is a low-end device, i.e., the Jetson TX2 development board. Deployment settings (3) and (4) are similar to deployment setting (2), but now only a random subset of, respectively, three and five clients are used in every federated training and evaluation round.  
From these results, it is clear that the single resource-constrained device is slowing down the overall federated learning process. In the homogeneous deployment setting, the average wall clock time of the coordinating server is 37 s, whereas, for the second configuration, it increases to, on average, 63 s. The accuracy remains unaffected, as expected. However, when only using three random clients in each federation round rather than all ten, the wall clock time drops again to 40 s at the expense of the accuracy dropping from 0.92 down to 0.89. With five random clients, the wall clock time increases to about 47 s while the accuracy increases to 0.91. By using a subset of random clients, there is a chance that the slow client is not included in a federation round, thereby speeding up the overall federated learning process. Additionally, involving fewer clients means that the coordinating server needs less time to compute the federated average of the model updates, although this impact is minimal in our experiments. These four experiments show that there is, again, a trade-off between accuracy and wall clock time that is influenced by federation-specific hyperparameters. In our experiments, the network was never a bottleneck. If that would have been the case, we could have identified further trade-offs between network usage and the number of local training rounds on each client. In practice, however, the heterogeneous resource availability is likely to be much more diverse, even when only using a fraction of the clients, compared to the above deployment settings with one resource-constrained device and nine higher-end systems with similar resource availability.  
Figure 6: Benchmarking MNIST in a federation with homogenous and heterogeneous resource availability and partial client involvement.