## 3 Methodology
### Bayesian Hyperparameters Optimization  
Modeling by machine learning (ML) approaches involves the accurate setting of several hyperparameters: these parameters are used to define the topology and size of a neural network (e.g., the number of hidden layers and neurons), and to control the learning process (e.g., the learning rate). Standard procedures involve a grid or random (i.e., based on a sampling method) search for the best combination of hyperparameters within variation intervals accurately defined by the experimenter on the basis of his/her experience [23; 29; 31; 45; 46; 48; 50]. However, to obviate the significant time demands and computational resources of the abovementioned methods, a semi-automatic strategy has been implemented in the current study, namely, the Bayesian Optimization (BO). This method, based on Bayesian statistics, aims to find the maximum (for maximization problems) or the minimum (for minimization problems) of a function: \(f(\mathbf{x})\), \(\mathbf{x}=\left[x_{p}\right]\), \(p\in\{0,\dots,P\}\), \(P\in N\), where \(x_{p}\) is a parameter in a bounded set \(X_{p}\subset R\). This mathematical problem is solved by optimization algorithms that define a probabilistic model over \(f(x)\), to decide at each iteration which is the most likely point in \(X\) to maximize (or minimize) \(f(\cdot)\). In this context, [PERSON] et al. [66] were the first to use the BO for the search of ML model hyperparameters: since the trend of the objective function is unknown, the authors treated \(f(\cdot)\) as a random function and placed a Gaussian process (GP) prior [67] over it, to capture its behavior. During the optimization process, the prior is updated based on ML experimentsresults, produced by different hyperparameters combinations, to form the posterior distribution over the function \(f(\cdot)\). The latter, in turn, is exploited by an acquisition function to determine the next evaluation point. In practice (Figure 6), a set of \(O\) observations of the form \(\left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)}\right\}_{o=1}^{O}\), with \(\mathbf{y}^{(o)}=N\Big{(}f\Big{(}\mathbf{x}^{(o)}\Big{)},\ v\Big{)}\) and where \(\
u\) is the variance of noise in the observation of \(f(\cdot)\), is exploited to determine a multivariate Gaussian distribution over \(R^{O}\) through a mean \(m:\ X\to R\) and covariance \(K:\ X\times X\to R\) functions. The next set of hyperparameters \(\mathbf{x}_{next}\in\ X\) that should be evaluated during the optimization process is determined by solving the equation \(\mathbf{x}_{next}=argmax_{\mathbf{x}}a(\mathbf{x})\). The function \(a:\ X\to R^{+}\) is called acquisition function and generally depends on both the previously observed samples \(\left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)}\right\}_{o=1}^{O}\) and the GP parameters \(\theta\). It is formulated as follows: \(a\Big{(}\mathbf{x};\left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)}\right\},\theta\Big{)}\). In the GP prior setting, \(a(\cdot)\) relies on the predictive mean \(\mu\Big{(}\mathbf{x};\ \left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)}\right\},\ \theta\Big{)}\) and variance \(\sigma^{2}\Big{(}\mathbf{x};\ \left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)}\right\},\ \theta \Big{)}\) functions of the GP model. There are several existing definitions for \(a(\cdot)\)[68; 69], but the GP Upper Confidence Bound (UCB) [70] has been proven to efficiently reduce the number of function evaluations needed to determine the global optimum of several black-box multimodal functions. UCB implements the acquisition function as \(a_{UCB}\Big{(}\mathbf{x};\ \left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)}\right\},\theta \Big{)}=\mu\Big{(}\mathbf{x};\ \left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)}\right\},\ \theta \Big{)}-\kappa\sigma\Big{(}\mathbf{x};\ \left\{\mathbf{x}^{(o)},\ \mathbf{y}^{(o)} \right\},\ \theta\Big{)}\), where \(\kappa\) is a hyperparameter to control the balance between exploitation (i.e., favoring parts that the model predicts as promising) and exploration. Figure 6: Flow chart of the optimization process.