## 2 Methods
### Bayesian Optimization Methodology
Given a set of observations \(\mathbf{D}_{1:\mathbf{t}}=\{(\mathbf{x}_{1},\mathbf{y}_{1}),\ldots,(\mathbf{x}_{t },\mathbf{y}_{t})\}\), a Gaussian Process is a stochastic model defined by Equation (4) as the multivariate normal distribution \(\mathbf{N}\):  
\[\mathrm{P}(y|\mathbf{D}_{1:\mathbf{t}},\mathbf{x})=\mathbf{N}\Big{(}\mu_{ \mathrm{t}},\sigma_{\mathrm{t}}^{2}\Big{)} \tag{4}\]  
where model predictions are represented by the mean term \(\mu_{\mathrm{t}}\) and the uncertainty by the variance \(\sigma_{\mathrm{t}}^{2}\), defined in Equations (5) and (6), respectively. \[\mu_{\mathrm{t}}=\mathbf{k}^{\mathrm{T}}\mathbf{K}^{-1}\mathbf{D}_{1:\mathbf{ t}} \tag{5}\]  
\[\sigma_{t}^{2}=\mathrm{k}(\mathrm{x},\mathrm{x})-\mathbf{k}^{\mathrm{T}} \mathbf{K}^{-1}\mathbf{k} \tag{6}\]  
\[\mathbf{K}= \begin{bmatrix}\mathrm{k}(x_{1},x_{1})&\cdots&\mathrm{k}(x_{1},x_ {t})\\ \vdots&\ddots&\vdots\\ \mathrm{k}(x_{t},x_{1})&\cdots&\mathrm{k}(x_{t},x_{t})\end{bmatrix} +\sigma_{t }^{2}\mathbf{I} \tag{7}\]  
However, since the observation of real systems could be noisy, we introduced a sampling noise \(\sigma_{\mathrm{r}}^{2}\) in Equation (7). In Equations (5)-(7), \(\mathbf{K}\) represents the covariance matrix. It describes how the observations are related to each other. In other words, the covariance matrix measures the correlation strength between observations by using a kernel function \(\mathrm{k}(\mathrm{x}_{1},\mathrm{x}_{1})\) as coefficients. The kernel function has a strong impact on the prediction quality of the GP surrogate model. In fact, it drives the output responses of the GP and the shape of the objective function that we can fit. That is why choosing it carefully according to the optimization problem is of major importance. In this paper, since deep learning metrics such as loss functions or validation IoU are continuous without periodicity, we used the Matern kernel function that is a generalization of radial basis kernel (RBF) [51], defined by Equation (8):  
\[\mathrm{k}\big{(}\mathrm{x}_{i},\mathrm{x}_{j}\big{)}=\frac{1}{\Gamma(\
u) 2^{\
u-1}}\Bigg{[}\frac{\sqrt{2\
u}}{1}\mathrm{d}\big{(}\mathrm{x}_{i}, \mathrm{x}_{j}\big{)}\Bigg{]}^{\
u}\mathrm{K}_{\
u}\Bigg{[}\frac{\sqrt{2\
u}}{ 1}\mathrm{d}\big{(}\mathrm{x}_{i},\mathrm{x}_{j}\big{)}\Bigg{]} \tag{8}\]  
Figure 3: Bayesian optimization methodology: (**a**) Initialization of Gaussian Process model; (**b**) Update of Gaussian Process model; (**c**) Improve Gaussian Process model iteratively. Figure adapted from [36]. where \(\Gamma(\
u)\) is the gamma function, \(\mathrm{d}\left(x_{i},x_{j}\right)\) the Euclidian distance between two observation points and \(\mathrm{K}_{\
u}\) is the modified Bessel function. The parameter \(\
u\) controls the smoothness of the output function. In fact, a smaller \(\
u\) leads to a sharper approximated function. Thus, the first part of the BO algorithm used a Gaussian Process as a surrogate model of the objective function. The remaining part of the algorithm concerns the choice of an activation function. Such a function is used to decide which observation point \(x_{t}\) should be inferred next. In this work, we used the common Expected Improvement (EI) acquisition function defined by Equation (9):  
\[\alpha(x)=\mathbf{EI}(x,D_{t})=\int\limits_{-\infty}^{+\infty}\max(y^{*}-y,0) \mathrm{P}(y\left|D_{1:t},x\right)dy\text{ with }y^{*}=\min\limits_{1<i<n}\mathrm{f}(x_{i}) \tag{9}\]  
Based on the updated GP model \(\mathrm{P}(y\left|D_{1:t},x\right)\) and the observation database \(D_{t}\), the activation function considers the region of high uncertainty with a large confidence interval and those with the minimum observed true objective function score \(y^{*}\) in order to compute the next sampling point \(x_{t}\). Finally, the entire BO algorithm is summarized in Algorithm 1.