## 4 Materials and Methods
### Bayesian Optimization  
Bayesian Optimization (BO) is a strategy utilized in machine learning for hyperparameter tuning. It outperforms random search techniques with fewer iterations by finding the minimum value of an objective function. This is accomplished by creating a surrogate function, a less computationally intensive probabilistic model derived from past evaluations of the objective function. The surrogate guides the selection of the next input values to evaluate, optimizing overall performance. The Gaussian Process (GP), grounded in the Bayesian approach, optimizes its operation by leveraging existing data, exhibiting robust resilience in handling diverse data scenarios. The algorithm primarily relies on input (known information) and output data (predicted information). It aims to fit the posterior distribution of the objective function, continually refined using an increasing number of data points. A significant part of GP's operation involves optimizing model hyperparameters, which are predetermined before the learning process and have a profound impact on the outcome. The process begins with an initial 'prior' distribution, which is updated each time a new sample point is evaluated on the target function. In the final phase, the algorithm analyzes the global posterior distribution to determine the most probable output value. The simplicity of calculating the estimated distribution of the target underpins the widespread usage of GP in Bayesian probabilistic models. When a GP is employed as the foundational model, the objective function \(f(\theta)\) is defined as \(GP(m(x),\,k(x,\,x^{\prime}))\). In this formulation, \(m(x)\) and \(k(x,\,x^{\prime})\) correspond to the mean and covariance functions, respectively. The covariance function, \(k(x,\,x^{\prime})\), plays a pivotal role in determining the intrinsic characteristics of the objective function \(f(\theta)\). These characteristics may include elements such as smoothness, the presence of additive noise, and amplitude. The covariance function's output, interestingly, is the covariance of \(f(\theta)\) and \(f(\theta^{\prime})\). The probabilistic nature of the GP model implies that each feature's probability must be estimated independently before their incorporation into the overarching Gaussian model. This step is crucial in accurately representing the complex interplay of features within the model. Another critical step in developing the GP model involves the establishment of a covariance matrix. The probability values associated with all eigenvectors are employed within this matrix, contributing to the structure of the multivariate Gaussian probability model. To fully appreciate the mechanics of the GP model, one can visualize it as follows:  
\[P(x)=\frac{1}{(2\pi)^{\frac{\theta}{2}}|\text{cov}|^{\frac{1}{2}}}\text{exp} \left(-\frac{1}{2}(x-\mu)^{T}\text{cov}(x-\mu)^{-1}\right) \tag{9}\]  
Here, \(\mu\) signifies the average value, while cov designates the covariance. Acquisition functions hold a critical role within BO, as they are tasked with suggesting potential locations for sampling within the expansive search space. This complex process involves initially drawing a random assortment of candidate samples from the entire domain. Following this step, each of these candidate samples undergoes evaluation through the application of the acquisition function. The next stage of this process involves maximizing the acquisition function or, alternatively, selecting the candidate sample that emerges with the highest score from the evaluation. It is important to note that there is a variety of probabilistic acquisition functions available for use, each presenting its own unique balance between exploitative and explorative tendencies. To break down these tendencies, exploitation typically involves the act of sampling in specific areas where the surrogate model predicts a high target. On the other hand, exploration refers to the act of sampling within regions that are characterized by a significant level of prediction uncertainty.