# Implementing AutoML in Educational Data Mining for Prediction Tasks
## 3 Introduction to Bayesian Optimization for Hyperparameter Optimization
### Bayesian Optimization  
Bayesian optimization is an effective strategy for minimizing (or maximizing) objective functions that are costly to evaluate. The main advantage of this method is that it uses previous results in order to pick the next point to try, while dealing with the dilemma of exploration and exploitation. As such, it reaches the optimal solution with less number of evaluations [44; 53]. Bayesian optimization uses the famous Bayes theorem. The theorem states that the posterior probability of a model \(M\) given data \(D\)\(P(M|D)\) is proportional to the likelihood of \(D\) given \(M\)\(P(D|M)\) multiplied by the prior probability \(P(M)\). As for the hyperparameter optimization problem, model \(M\) should not be mistaken with the output model of machine learning algorithms. On the contrary, \(M\) is actually a regression model that represents our assumptions about \(f\)[54]:  
\[P(f|D_{zt})\propto P(D_{zt}|f)P(f) \tag{2}\]  
where \(D_{zt}=\{x_{zt},f(x_{zt})\}\) defines our accumulated observations of the objective function \(f\) on sequences of data samples \(x_{zt}\)[44]. The prior prescribes our belief (what we think we know) over a space of objective functions. The likelihood captures how likely the data we observed are, given our belief about the prior. These two combined, give us the posterior, which represents our belief about the objective function \(f\). Additionally, the posterior conceptualizes the _surrogate_ function, the function used to estimate \(f\). As it is much easier to optimize the surrogate probability model than the objective function, the Bayesian optimization selects the next set of hyperparameters to evaluate based on its performance on the surrogate. During the execution, the accuracy of the surrogate model is increased by continually incorporating the evaluations on the objective function. The Bayesian optimization is considered as a sequential design strategy (formally Sequential model-based optimization (SMBO)) [10; 55]. At first, a surrogate probabilistic regression model of the objective function is built. Until a budget limit is being reached, new samples of hyperparameters are sequentially selected. These new samples are selected by optimizing an acquisition function S, which uses the surrogate model. Each suggested sample is applied on the true objective function and produce new evaluations. The new observations are used to update the surrogate model (Algorithm 1). There are several variants of the SMBO formalism, which are specified by the selection of the probabilistic model and the criteria (acquisition function) used to select the next hyperparameters. In the model-based optimization literature, the most recognized choices for the surrogate model are the Gaussian Processes [56], Tree Parzen Estimators (TPE) [10; 57], and Random Forests [58]. In this research, we selected Random Forests for our experiments. Some of the advantages of this method is that it can represent the uncertainty of a given prediction, and that it can handle categorical and conditional parameters [59]. [PERSON] et al. [54] suggested that tree-based models (i.e., TPE and Random Forests) work best for large configuration spaces and complicated optimization problems. The same conclusions are marked by the authors of [60]. They further notice that, in their experiment, Random Forests could obtain results of four more configurations at that same time budget compared to TPE and GP. The reason is their support on small number of folds at the cross- validation procedure. Random Forests are used by the Sequential Model-based Algorithm Configuration (SMAC) library ([[https://github.com/automl/SMAC3](https://github.com/automl/SMAC3)]([https://github.com/automl/SMAC3](https://github.com/automl/SMAC3))).