# Implementing AutoML in Educational Data Mining for Prediction Tasks
## 3 Introduction to Bayesian Optimization for Hyperparameter Optimization
### Definitions  
In general, optimization refers to the process of finding the best result under specified circumstances [43]. More formally, it is the (automatic) process of finding the value or a set of values of a function (a real valued function called the objective function) that maximizes (or minimizes) its result. It is consistent with the principle of maximum expected utility or minimum expected loss (risk) [44]. It is well known that there is no single method that can solve every optimization problem efficiently. It can be challenging to choose the best method for a given problem formulation, usually complex and computationally expensive processes are required. However, optimization methods can obtain high quality results with reasonable efforts. In machine learning systems, the targets of automation include mechanisms that optimize machine learning pipelines, such as the feature engineering, model selection, hyperparameter selection, etc. The problem of identifying values for the hyperparameters that optimize the system's performance is called the problem of hyperparameter optimization [45]. Hyperparameters are the parameters whose values are set before the beginning of the learning process, e.g., the number of neighbor's k in the nearest neighbor algorithm, or the depth of the tree in tree-structured algorithms. In contrast, model parameters are parameters that are learned during the training process, e.g., the weights of neurons in \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Paper** & **Prediction Task** & **Metrics** & **Methods Compared** & **Outperformers** \\ \hline \hline \multicolumn{5}{c}{_Related work on predicting student dropout_} \\ \hline
[38] & Classification (dropout) & Accuracy & ANN, SVM DTS, LR, Information fusion, Bagging, Boosting & Information fusion \\ \hline
[39] & Classification (dropout) & Accuracy, Sensitivity, Precision & 3 decision schemes based on NN, SVM, and ARTMAP & Decision scheme 1 \\ \hline
[40] & Classification (dropout) & Accuracy & NNs, DTs, RF, Discriminant Analysis & NNs \\ \hline
[41] & Classification (dropout) & Accuracy & NNs, DTs, LR & DTs \\ \hline
[42] & Classification (dropout) & Accuracy, Sensitivity & Self-Training, Co-Training, Democratic Co-Training, Tri-Training, RASCO, Rel-RASCO, C4.5 & Tri-Training (C4.5) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Cont. a neural network. The central focus of autoML is the hyperparameter optimization (HPO) of machine learning processes. Formally, the general statement of the hyperparameter optimization problem is defined as [44]:  
\[x^{*}=\operatorname*{argmin}_{x\in X}f(x) \tag{1}\]  
where \(f\) is the objective function, given a set of hyperparameters \(x\) from a hyperparameters space \(X\). We are interested in finding \(x^{*}\) set that minimizes the expected loss (the value of \(f(x)\)). One important property of this function \(f\) is that its evaluation is expensive (costly) or even impossible to compute [46]. To make \(f\) more clear, consider the context of machine learning applications, where, for example, the function \(f\) can be a system that predicts student dropout rates (e.g., a set of preprocessing and classification algorithms) with adjustable parameters \(x\) (e.g., the learning algorithm or the depth of the tree when tree structured algorithms are tested), and an observable metric \(y=f(x)\), on data collected from learning systems (e.g., data from a learning management system). As manual tuning is an error-prone process that also requires time and experts in the field [47], various automatic configuration methods have been proposed. A popular approach is to treat the problem as black-box optimization. Grid search has been the traditional and the most basic, yet extremely costly method. A fairly efficient alternative is random search [48]. Other families of methods that have been applied are gradient-based algorithms [49], racing algorithms, e.g., [50], evolutionary optimization algorithms [51], and population-based search, e.g., [52].