# AutoFL: Towards AutoML in a Federated Learning Context
## 3 Analysis of a Contemporary AutoML Framework  
Our framework builds upon the design principles of automated machine learning frameworks, including auto-sklearn [16; 37; 38]. The different stages of this optimization pipeline are depicted in Figure 1 for a classifier pipeline. The meta-learning typically predefines the search space for the hyperparameters and preprocessing in the subsequent stages. The idea behind meta-learning is that datasets with similar meta-features perform similarly on the same set of hyperparameters. Auto-sklearn computes about 38 meta-features [37] (e.g., number of classes, number of features, skewness, kurtosis,...) on 140 reference datasets from the OpenML repository ([[https://www.openml.org](https://www.openml.org)]([https://www.openml.org](https://www.openml.org)), accessed on 15 May 2023). The hyperparameters performing the best for a reference dataset with meta-features similar to the new dataset serve as an instantiation for the Bayesian optimizer. After this initialization, the AutoML pipeline will then explore the search space to iteratively start with the data and feature preprocessing before evaluating the classifier against the test data. The results are then evaluated on a test set, and the hyperparameters are further optimized for a given error metric (e.g., accuracy or f1) using Bayesian optimization. In the last step, an ensemble model is constructed based on the top best base models (or only the best model for an ensemble of size 1). Listing 1 illustrates how a classifier is built with the auto-sklearn framework. This particular example constructs an ensemble with only one base model that is constrained to classifiers of type Random Forest (RF). The example also further illustrates how many models are assessed in parallel and how much time and memory are granted to construct the ensemble. The AutoSklearnClassifier() in line 26 constructor implements the whole AutoML pipeline as depicted in Figure 1.  
Figure 1: Simplified representation of auto-sklearn’s optimization pipeline. ```
1#Example of manualfeaturepreprocessing
2 defmake.preprocessor(X.train): numeric.features=[dots] categorical.features=[{dots}] pasthrough.columns=['timestamp']
3
4#Featureselection requires non-negativeinput numeric.transformer=make.pipeline(MinMaxScaler()) categorical.transformer=make.pipeline(OrdianEncoder( handle.unknown='use.encoded.value',unknown.value=1000000))
5
6 Preprocessor=ColumnTransformer( transformers=[ 'passthrough.transformer','passthrough',passthrough.features), {'numeric.transformer',numeric.features'}, {'categorical.transformer',categorical.transformer,categorical.features}
7
8
9 returnPreprocessor
10
11
12
13
14
15
16
17
18 returnPreprocessor
19
20
21
22
23 preprocessor=make.preprocessor(X.train)
24
24
25#ConfiguretheAutoML classifier
26 autom_model=AutoSklearnClassifier{ time.left_for_this.task=3600,#Maxtotaltrainingtime(inseconds) per_run_time.limit=600,#Maxtrainingtimeforsinglemodel(inseconds) memory.limit=6536,#Maxusedmemory(innegatives) tmp.folder=tmp',
27
28 interf1, #Metrictooptimizate:accuracy,f1,recall,dots scoring_functions=[accuracy,f1,precision,recall], n.jobs=20,#Maxtrainingjobsinparallel ensemble.size=1,#Limitensembleto1 basemodelonly initial_configurations_via.metalearning=0, include={ \"classifier\":[\"random.forest\"]#LimitclassifiertypeinensembletoRandomForest }
29
21#ConstructthepipelinewiththemanualpreprocessorandtheAutoMLpipeline
21 model=make.pipeline(preprocessor,autom_model)
23
24#Fitthemodelontrainingset model.fit(X.train,Y.train)
25
26#Evaluateontestset
27V.pred=model.predict(X.test)
28
29 print(\"Accuracy:\",accuracy.score(Y.test,Y.pred)) print(\"F1 score:\",f1.score(Y.test,Y.pred,average='weighted')) print(\"Precision:\",precision.score(Y.test,Y.pred,average='weighted')) print(\"Recall:\",recall.score(Y.test,Y.pred,average='weighted'))
```  
**Listing 1**. Code example of a classifier built as an auto-sklearn ML pipeline.