# AutoFL: Towards AutoML in a Federated Learning Context
## 5 Evaluation
### Impact of Secure Aggregation  
Flower v1.14 does not support secure aggregation out of the box. Fortunately, [PERSON] et al. (2015) implemented Salvia (cfr. [[https://hei411.github.io/projects/salvia.html](https://hei411.github.io/projects/salvia.html)]([https://hei411.github.io/projects/salvia.html](https://hei411.github.io/projects/salvia.html)), accessed on 4 May 2023) for a slightly older version of Flower v0.17, adding support for SecAgg(+) protocols under a semi-honest threat model. In the following experiment, we used the i.i.d. MNIST dataset in a federation with ten clients having similar resource availability and compared the traditional federated averaging strategy and this particular secure equivalent implemented in Salvia. For a fair baseline comparison, we used the older vanilla 0.17 version without secure aggregation to compare against. The results in Figure 8 indicate a difference in the baseline between Flower v1.14 and v0.17 in terms of wall clock time at the coordinating server. The mean accuracy for the 100 ML model configurations remains on par. For example, in the case of the i.i.d. dataset configuration, the mean accuracy is 0.918 vs. 0.907 for, respectively, without and with secure aggregation. A slight difference in mean values is to be expected as there is no guarantee that the Bayesian optimization process selects the same sets of 100 configurations. However, there is now a clear difference in the wall clock time caused by the computational impact of secure aggregation. Not only has the mean wall clock time gone up from 25 s to 121 s, but the variation of wall clock times is also more outspoken. The other metrics are indicated in Table 4. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Metric** & **w/o SecAgg (s)** & **w/ SecAgg (s)** \\ \hline Minimum value & 6.43 & 15.92 \\ Lower quartile (25 th percentile) & 12.00 & 26.85 \\ Median value (50 th percentile) & 27.70 & 125.29 \\ Upper quartile (75 th percentile) & 34.30 & 207.99 \\ Maximum value & 50.73 & 250.76 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Quartile and minimum/maximum values of the coordinating server’s wall clock time for secure aggregation configurations of MNIST i.i.d. Figure 8: Benchmarking MNIST in a federation with and without secure aggregation. As the computational impact of secure aggregation is non-negligible, the trade-offs between accuracy on the one hand and CPU time on the client, on the other hand, may now be more meaningful. After exploring the 100 configurations with secure aggregation, we filtered those with an accuracy of 0.98 or higher. The accuracy, memory usage, and CPU time of the clients of the selected eleven configurations are listed in Table 5. The memory usage is comparable except for configuration 0010 (i.e., 914 MB vs. \(\pm\)650 MB). When comparing the different values, we end up with seven Pareto-optimal configurations marked in bold. With a more strict condition that a Pareto-optimal configuration is at least 10% better in terms of memory or CPU usage, the remaining Pareto-optimal solutions are 0016 (low CPU time) and 0021 (higher accuracy). The difference in computational complexity can be explained by the ML model configurations that performed best in terms of accuracy (i.e., all neural networks). Here are some examples:  
* 0002: Scikit-learn MLPClassifier with one hidden layer of size 286
* 0010: TensorFlow neural network with one hidden layer of size 390
* 0016: scikit-learn MLPClassifier with one hidden layer of size 291
* 0021: scikit-learn MLPClassifier with one hidden layer of size 397
* 0043: scikit-learn MLPClassifier with three hidden layers of size 397, 268, 139
* 0096: scikit-learn MLPClassifier with two hidden layers of size 397, 203  
These specifications and benchmark results illustrate that a simple neural network with fully connected layers requires less memory in scikit-learn's MLPClassifier than in TensorFlow.