## 4 Bayesian Optimization of ResNet18
### Optimization Test Analysis  
The most popular hyperparametric optimization algorithms are stochastic search, grid search, and Bayesian optimization. Bayesian optimization methods tend to perform better in parametric optimization problems. Compared to grid search and random search, Bayesian optimization methods are usually able to find better parameter combinations with the same computational resources. [PERSON] et al. (2019) used the random search algorithm to optimize an aircraft engine vibration signal prediction model, and the difference between the test error and the training error was calculated to be more than 1 in the experimental process, which did not meet the accuracy requirements of this experiment, and [PERSON] et al. (2019) optimized the calculation of the random algorithm to meet the computational requirements. The global search algorithm takes too long and costs too much to be applied in this experiment. Bayesian optimization is an optimization method used to find the global optimum of a black box function (optimal objective function). Among them, the prior function and the collection function are the core processes of the Bayesian optimization algorithm. The prior function mainly utilizes Gaussian process regression:  
\[f(x)\sim GP(m(x),k(x,x^{\prime})) \tag{18}\]  
GP denotes a Gaussian process, \(m(x)\) is the mean function, and \(k(x,x^{\prime})\) is the covariance (or kernel) function. A Gaussian process is a probabilistic model that characterizes the distribution of a function by a mean vector and a covariance matrix based on a collection of points in the input space. Bayesian optimization tuning using the Gaussian process can take into account the previous parameter information to continuously update the prior function and guide the next optimal choice, which greatly saves computation time. The collection function is an important part of the Bayesian optimization process used to select the next combination of hyperparameters to evaluate. The construction of the collection function based on the mean and variance calculated by the Gaussian process reflects the degree to which the numerical points are worth searching. And the extreme point of this function is the next search point. Choosing the appropriate collection function can help the algorithm converge to the optimal solution quickly and accurately. Common sampling functions include Thompson Sampling, Expected Improvement, and Upper Confidence Bound. The collection process is represented by Equations (18) and (19):  
\[x_{i+1}=\underset{x\in X}{\text{max}}\lambda(x,D_{1:i}) \tag{19}\]  
\[r_{i}=|y^{*}-y_{i}| \tag{20}\]  
\(r\) is the total loss, \(X\) is the decision space, \(\lambda(x,D_{1:i})\) is the collection function, and \(y^{*}\) is the optimal solution. To summarize, the Delikere process sampling is chosen in this paper. Assume that the Dirichlet sampling of this model is \(G1\), where the underlying distribution of \(G1\) is \(H1\)and the concentration parameter is \(\alpha\). Then, the result of the sampling can be expressed as follows for any segmentation point t:  
\[G1(t)=\sum_{i=1}^{\infty}wi\times\delta(\theta i) \tag{21}\]  
where \(wi\) is a random variable satisfying the Dirichlet distribution, i.e., \(wi\sim Dir(\alpha)\), and \(\theta i\) is a random variable sampled from the underlying distribution \(H1\). The Bayesian optimization process is as follows:  
1. Select initial parameters and evaluate for the objective function. 2. Construct the agent model of the Bayesian optimization algorithm, i.e., the prior function and the learning function. 3. Select the next parameter point xi to be evaluated. 4. Execute the objective function on the selected parameter point and record the results. 5. Add the new parameter objective function evaluation data to the existing data set and retrain the agent model. 6. Determine whether the stopping condition is satisfied; if not, return to step (3) until the optimal value parameter point is output.