# AutoFL: Towards AutoML in a Federated Learning Context
## 5 Evaluation
### Homogeneous Federation: i.i.d. versus Non-i.i.d  
The previous experiments demonstrated the computational complexity of a selection of manually configured classical ML models and neural networks. In this particular experiment, we used Flower version 1.4 [31; 32] to implement and configure a federation with ten clients and one coordinating server each having a similar resource availability and compare that against a federation with a coordinating server and only one client. Additionally, we apply our SMAC3-based Bayesian optimization framework to automatically find the best model and corresponding hyperparameters. To simplify the comparison of the results, we constrained the experimental setup with the following settings:  
* The Bayesian optimization procedure was fixed to evaluating 100 different classical ML or deep learning model configurations, during which we not only assessed the accuracy but also measured the CPU, memory, and network usage on the clients as well as the coordinating server. * We excluded all scikit-learn classifiers that could not be aggregated through federated averaging of the model's weights or coefficients (e.g., random forest). Next to TensorFlow-based deep learning models, our framework explored these scikit-learn classifiers: LogisticRegression(), Perceptron(), MLPClassifier() and PassiveAggressiveClassifier(). * For each model configuration, the coordinating server carried out 20 federated averaging rounds, and each client locally trained their model for one epoch before sending the model update to the coordinating server. * All clients were involved in each round of the federated learning (instead of only a random subset), and each client trained on its own partition of the training dataset. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & **Pi 4** & **Jetson TX2** & **Laptop** & **Client** & **Server** \\ \hline
**Accuracy** & & & & & \\ \hline Random Forest & 0.953 & 0.953 & 0.953 & 0.953 & 0.953 \\ Logistic Regression & 0.926 & 0.926 & 0.926 & 0.926 & 0.926 \\ Multi-Layer Perceptron & 0.972 & 0.972 & 0.972 & 0.972 & 0.972 \\ TensorFlow-FF & 0.981 & 0.980 & 0.980 & 0.980 & 0.980 \\ TensorFlow-CNN & 0.986 & 0.986 & 0.986 & 0.986 & 0.986 \\ \hline \hline
**F1 score** & & & & & \\ \hline Random Forest & 0.953 & 0.953 & 0.953 & 0.953 & 0.953 \\ Logistic Regression & 0.926 & 0.926 & 0.926 & 0.926 & 0.926 \\ Multi-Layer Perceptron & 0.972 & 0.972 & 0.972 & 0.972 & 0.972 \\ TensorFlow-FF & 0.981 & 0.980 & 0.980 & 0.980 & 0.980 \\ TensorFlow-CNN & 0.986 & 0.986 & 0.986 & 0.986 & 0.986 \\ \hline \hline
**CPU time (s)** & & & & & \\ \hline Random Forest & 24.013 & 18.996 & 3.468 & 5.401 & 4.768 \\ Logistic Regression & 165.221 & 152.161 & 31.724 & 56.262 & 41.886 \\ Multi-Layer Perceptron & 308.138 & 161.850 & 162.388 & 43.824 & 996.448 \\ TensorFlow-FF & 865.530 & 457.998 & 75.678 & 120.031 & 200.849 \\ TensorFlow-CNN & 3644.037 & 1216.834 & 243.083 & 555.559 & 1441.772 \\ \hline \hline
**Wall clock time (s)** & & & & & \\ \hline Random Forest & 24.019 & 19.149 & 3.470 & 5.426 & 4.773 \\ Logistic Regression & 165.250 & 152.623 & 31.725 & 56.265 & 41.899 \\ Multi-Layer Perceptron & 72.298 & 40.905 & 10.887 & 11.017 & 15.608 \\ TensorFlow-FF & 410.429 & 259.521 & 33.883 & 59.304 & 65.821 \\ TensorFlow-CNN & 1052.609 & 385.810 & 38.621 & 185.502 & 134.152 \\ \hline
**Memory (MB)** & & & & & \\ \hline Random Forest & 742.297 & 743.723 & 759.760 & 731.340 & 737.709 \\ Logistic Regression & 551.831 & 559.388 & 549.855 & 552.725 & 552.236 \\ Multi-Layer Perceptron & 552.848 & 559.618 & 551.349 & 555.092 & 552.052 \\ TensorFlow-FF & 1096.914 & 1098.737 & 1293.157 & 1217.322 & 1324.608 \\ TensorFlow-CNN & 1131.936 & 1140.021 & 1306.701 & 1252.923 & 1572.459 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: MNIST baseline benchmark for classical ML models and neural networks. * For the i.i.d. training data, each client had about the same amount of samples for each class (about 600 for MNIST and FashionMNIST). For the non-i.i.d.