## 2 Methods
### Bayesian Optimization Methodology  
Deep learning and more generally machine learning models are driven by a set of input parameters called hyperparameters. In the case of deep learning, this can be, e.g., network architecture, such as the number of convolution blocks, activation functions and training parameters such as epochs number or learning rate. Fine-tuning hyperparameters is essential to leverage the potential of the developed deep learning model. However, manually tuning hyperparameters is time-consuming and nearly impossible. For example, in the simple version of our model, there are more than 8600 hyperparameter combinations. Deep learning hyperparameter optimization (HPO) is an active research area [44; 45; 46; 47; 48; 49]. In such problems, the goal is to maximize/minimize an expensive objective function whose closed form and its gradients are unknown. In this paper, we use a Bayesian optimization (BO)-based approach. BO is a black box (i.e., model-based) optimization algorithm searching for a global optimum of an unknown objective function from which observations can be made. The BO algorithm is composed of two components: a surrogate model for objective function approximation and an activation function for the integration of the regression history. In addition, it is worth noting that one of the main advantages of BO is its ability to learn from past experiments. Thus, the global optimization problem could be formulated as follows:  
\[x^{*}=\underset{x\in X}{\text{argmax}}f(x) \tag{3}\]  
where \(x^{*}\) designs the hyperparameter input in the search space \(X\), i.e., \(f(x)\) is optimal. However, in order to use the BO for hyperparameter optimization, we assume the following constraints:  
* We have a few input control parameters \(x\) (less than 20). * The objective function is relatively \"smooth\" (i.e., not so discontinuous in the discrete meaning). As a black box optimization approach, BO uses a surrogate model for the regression task (i.e., fit the unknown objective function). In this work, we used a statistical Gaussian Process (GP) as a surrogate model. The key ideas behind BO are illustrated in Figure 3.  
From Figure 3, we can describe the underlying ideas behind the BO algorithm as follows:  
1. Based on the initial observation points (i.e., sample points measured) from the objective function \(f(x)\), a first regression attempt is made using an initial version of a Gaussian Process model. In addition, a confidence interval is built due to the probabilistic nature of the GP. Then, an acquisition function \(\alpha(x)\), derived from the GP model, is used to estimate the next computing step. 2. The GP model has been updated from past observation points. Thus, the regression improves and the uncertainty area decreases toward the objective function. The next observation point \(x_{n}\) is computed by taking the global maximum of the associated acquisition function. 3. Finally, the fitting quality is iteratively improved. When the regression becomes acceptable, the best hyperparameter configuration \(x^{*}\) is saved and the optimization problem can be solved by finding a global optimum of the GP model. Several surrogate models have been developed in the field of BO [40; 50]. The most common one is the Gaussian Process because of its acceptable performance and its simplicity of implementation. In addition, using GP as a regression model to fit the unknown objective function is appealing because it considers the uncertainty associated with each prediction.