# AutoFL: Towards AutoML in a Federated Learning Context
## 5 Evaluation
### Baseline Experiment  
We first trained a selection of classical ML models and neural networks on the MNIST dataset on the various systems and compared their baseline accuracy as well as their resource usage and the time passed. The ML models were trained locally in a centralized manner. The initialization of the ML models and their hyperparameters are listed in Listing 3. The full Python code to prepare the MNIST datasets and train the LogisticRegression() model is listed in Appendix A, respectively, in Listings A1 and A2. The code examples for the other ML models are similar.  
Note that the hyperparameters of these models were manually selected and were not optimal. Hence, they only served to demonstrate resource heterogeneity across the machines. Figure 3 depicts the results of 20 runs of the LR model on the different systems. What we can observe in these results is the fact that the results are more or less consistent (i.e., no big deviations across the different runs). Furthermore, the accuracy and f1 score are the same across all devices, as expected. The overall memory usage of 550 MB is very similar too. It is also not surprising that the CPU usage and wall clock time are the same within a particular device type due to the fact only one CPU core was used. However, there are differences across the different device types.  
Figure 3: \(Cont.\)  
Figure 4 depicts a similar story, but now for the feed-forward neural network implemented with TensorFlow (i.e., clf4). In these benchmarks, a device used its CPU even if a GPU was available to accelerate the training. The accuracy and f1 score are on par across the board. Memory usage is significantly higher and reasonably consistent within the same device type, but less consistent across the devices. Since multiple CPU cores are used, the CPU time is at least twice as high compared to the wall clock time.  
Figure 4: \(Cont\).  
Figure 3: Boxplots of benchmarks of scikit-learn’s logistic regression (LR) on different devices (the green diamond determines the mean value; the ends of the notched box represent the lower and upper quartiles; the orange line inside the notched box indicates the median value).  
In Table 1, we provide a detailed metric overview of different baseline models for MNIST classification on different devices. The reported values are the average of 20 runs. Since the dataset is well-balanced, the accuracy is almost identical to the f1 score. All models except random forest (RF) were trained in multiple rounds. We fixed the number of training rounds to 20 for the logistic regression (LR), multi-layer perceptron (MLP), and feed-forward (FF) models, and set the number of epochs for the convolutional neural network (CNN) to 10. Table A1 in Appendix A reports the results for the same model configurations, but now on the FashionMNIST dataset. The latter has similar characteristics in terms of data format and size but is more challenging to classify compared to MNIST. In Table A2, we report the results for the CIFAR-10 dataset for the same models. This dataset is more sophisticated, and hence the performance results are subpar, as expected. For example, the MLP model has an accuracy of 0.1, which for a dataset with 10 classes is the same as random guessing. From these baseline experiments, it is clear from a computational and accuracy point of view that the MNIST dataset is the least challenging, whereas the CIFAR-10 dataset is the most challenging.  
Figure 4: Benchmarking TensorFlow’s feed-forward neural network (clf4) on different devices.