# AutoFL: Towards AutoML in a Federated Learning Context
## 1 Introduction  
Federated learning [1, 2] is a distributed approach to machine learning that enables machine learning (ML) models to be trained on decentralized data without the need to first centralize the data. In a classical machine learning approach, the data is collected from different clients and centralized in one location before training the model. With federated learning, however, the data remains decentralized, and the model is trained locally on each client. The model's parameters or coefficients of each client are then sent back to the central coordinating server, where they are aggregated and then used to update the global model. This process is repeated iteratively until the model converges to a satisfactory level. The main advantages of federated learning include better data protection [3, 4], lower communication costs [5] and greater efficiency [6, 7]. It has been applied in various sectors where privacy concerns prevent the centralized collection of data, such as health care [8, 9, 10] and finance [11, 12, 13]. However, federated learning is not without its challenges. Beyond the fact that convergence of the ML model is not straightforward with clients [14] having heterogeneous data distributions, the decentralized training approach is also subject to various security and privacy threats [4, 15]. One particular challenge for developing effective AI-based applications is selecting the type of ML model and its optimal hyperparameters for use. Whenever a single client has plenty of data to train a model, then various AutoML frameworks such as auto-sklearn [16], TPOT [17], and H2O AutoML [18], can help with ML model selection and optimization. For federated learning, however, the related literature typically focuses on how to federate classical machine learning or deep learning methods, how to apply various privacy-enhancing schemes, or how to customize the ML pipelines for specific application areas. The actual ML model configuration is typically defined upfront. As an example, in computer vision tasks, the centralized server might opt for popular neural network architectures such as VGG-16 [19] (with approximately 138 million parameters), ResNet-50 [20] (with around 25.6 million parameters), or Inception-ResNet-V2 [21] (with roughly 56 million parameters). However, these neural networks might be excessively deep and contain a large number of parameters, which can lead to overfitting when the training set is limited. In centralized training scenarios, assessing the quantity and quality of training data is a relatively straightforward process. However, in federated learning, this becomes challenging as the training data is distributed across multiple nodes, possibly with imbalanced classes, making it difficult to evaluate the overall dataset. Certainly, in such scenarios, it is important to note that a model trained on one node may not perform effectively on another node. This discrepancy can arise due to insufficient representation of the training data of the first node, meaning that the test data on the second node falls outside the distribution seen during training. The second node's test data can be considered out-of-distribution. Hence, it is essential for multiple nodes to collaborate and collectively train a model using their respective data. This collaborative learning approach helps ensure that the test data remains within the distribution seen during training, thereby minimizing the occurrence of out-of-distribution scenarios. Last but not least, larger network sizes demand increased computational resources, which may not be readily available on the participating nodes in federated learning. In reality, model selection and optimization in a federated learning context are not straightforward. The challenge that we address in this work is the problem context where (a) a single client's data is not enough or not sufficiently representative to select a model and hyperparameters upfront and (b) centralizing the data is not feasible because of resource or confidentiality constraints.