# AutoFL: Towards AutoML in a Federated Learning Context
## 4 Design and Implementation of the AutoFL Framework
### Conceptual Overview  
In Figure 2, we can observe a high-level representation of the AutoFL framework, showcasing its key components and improvements compared to conventional AutoML frameworks such as auto-sklearn depicted in Figure 1. We will now delve into the similarities and differences between the two frameworks, emphasizing how these variances contribute to addressing the aforementioned limitations. AutoFL has two significant components in common with auto-sklearn and other AutoML frameworks, and those are the _Model Selection_ and the _Hyperparameter Tuning_. AutoFL supports a selection of traditional ML and deep learning models. In practice, multiple ML models and/or different hyperparameters for these models are tested in parallel. Where our framework differs is the search space in which the 'best' model needs to be found:  
* _Model Selection_: Not every traditional ML model has an equivalent collaborative or federated learning implementation, and this reduces the search space for AutoFL. * _Hyperparameter Tuning_: AutoFL must account for the hyperparameters of each ML model (e.g., the max depth of a decision tree), but also the hyperparameters for the federation itself (e.g., local training rounds per epoch). This typically increases the search space. These capabilities of AutoFL effectively tackle limitations 1, 2, 3, and 6. To address limitation 5, AutoFL goes beyond optimizing machine learning models based on a single objective and additionally considers other concurrent optimization objectives if they are relevant to the specific application and deployment environment:  
* _Model Objectives_: In AutoFL, these objectives commonly revolve around established _error metrics_ such as accuracy, F1 score, precision, recall, ROC-AUC, and others. In  
Figure 2: Conceptual block diagram of AutoFL with support for multi-objective optimization and application-specific constraints. addition to these standard metrics, alternative criteria can also be considered, for example, to compare the _interpretability_ of different machine learning models. * _Resource Objectives_: The objectives encompass efficient resource utilization, such as CPU, memory, and network traffic, both during federated training and after model deployment. Two trivial examples are minimizing memory usage and reducing the time required to evaluate a single input sample during deployment. AutoFL implements various runtime monitors and aggregates resource usage statistics across the nodes in the federation. The model error and resource usage metrics then feed the _Multi-Objective Optimization_ component in Figure 2. This component of AutoFL leverages the SMAC3 library [40]. This library, which can be found at [[https://automl.github.io/SMAC3/](https://automl.github.io/SMAC3/)]([https://automl.github.io/SMAC3/](https://automl.github.io/SMAC3/)) (accessed on 3 July 2023) and [[https://github.com/automl/SMAC3](https://github.com/automl/SMAC3)]([https://github.com/automl/SMAC3](https://github.com/automl/SMAC3)) (accessed on 3 July 2023), provides, amongst others, Bayesian optimization capabilities for both single- and multi-objective optimization. For multi-objective optimization, SMAC3 utilizes the ParEGO algorithm [41]. This algorithm combines multiple objectives into a single scalar objective, allowing SMAC3 to optimize it in a manner similar to single-objective optimization. Nonetheless, it retains the ability to identify configurations that lie on the Pareto front. Implementing these optimizations using the SMAC3 library is relatively straightforward. The _Configuration Under Test_ component is responsible for ensuring that any proposed configuration complies with pre-defined deployment, security, and privacy constraints before the model is trained in a federated manner. Additionally, it tracks metrics and other runtime statistics across multiple configurations. For instance, this component integrates techniques to detect class imbalance by analyzing the sample numbers of the minority and majority classes.