## 2 Model Principles and Methods
### Bayesian Optimization Principle  
Bayesian optimization is an approximate method that employs various probabilistic surrogate models such as Gaussian processes, random forests, etc., to model the relationship between hyperparameters and model performance, ultimately identifying the optimal hyperparameter combination [26].  
In Bayesian optimization, the probabilistic surrogate model refers to substituting the objective function with a certain probability model. The update formula for the posterior probability is as follows:  
Figure 2: LSTM network structure.  
\[p(f|D)=\frac{p(D|f)p(f)}{p(D)} \tag{6}\]  
In the equation, \(D=\{(x_{1}f_{1}),(x_{2}f_{2}),(x_{n}f_{n})\}\) represents the collected sample points; \(p(f)\) is the prior distribution, which can be computed using the Bayesian formula to obtain the posterior distribution of \(f\).  
Bayesian optimization surrogate models can be broadly categorized into three types: Tree Parzen Estimator (TPE), Sequential Model-based Algorithm Configuration (SMAC) using random forest regression, and Gaussian Processes (GPs). This paper employs TPE, a non-standard Bayesian optimization algorithm based on a tree-structured Parzen density estimator. In comparison to other models, TPE demonstrates superior performance in high-dimensional spaces, with significantly improved speed.  
The configuration space for TPE parameters is tree-shaped, primarily modeling \(p(x)\) and \(p(y)\). The preceding parameters determine which parameters to select subsequently and the range of values for these parameters.  
TPE defines the following two probability densities:  
\[p(x|y)=\begin{cases}l(x),y<y*\\ l(x),y<y*\end{cases} \tag{7}\]  
In the equation, \(l(x)\) represents the probability density of \(\{x_{i}\}\) corresponding to \(f(x_{i})\) being less than the threshold \(y^{*}\), and \(g(x)\) represents the probability density of \(\{x_{i}\}\) corresponding to \(f(x_{i})\) being greater than the threshold \(y^{*}\).  
By repeatedly measuring the objective function at different locations, more information can be obtained to estimate the distribution of the objective function. This enables the search for the optimal measurement location, aiming to achieve the optimal function value. To assess whether a location is optimal, function evaluations are collected. In the optimal location, the function attains its maximum value. In TPE, the acquisition function for collecting functions is the Expected Improvement (\(EI\)), which represents the expectation of being below the threshold. It performs well in most situations, and the formula is as follows:  
\[\begin{split} EI_{y*}(x)=\int_{-\infty}^{+\infty}\max(y*-y,0)p(y|x )\,dy\\ =\int_{-\infty}^{y*}\max(y*-y,0)\frac{p(x|y)p(y)}{p(x)}dy\end{split} \tag{8}\]  
In the equation, the model \(p\) is the posterior Gaussian distribution over the observation domain.  
In the framework of TPE, let \(\gamma=p(y<y*)\) and \(p(x)=\int p(x|y)p(y)dy=\gamma l(x)\)\(+\)\((1-\gamma)g(x)\); then, the following relationships hold:  
\[EI_{y*}=\frac{l(x)y*\gamma-l(x)\int_{-\infty}^{y*}p(y)dy}{\gamma l(x)+(1- \gamma)g(x)} \tag{9}\]  
\[EI_{y*}=\left(\gamma+\frac{g(x)}{l(x)}(1-\gamma)\right)^{-1} \tag{10}\]  
Equation (9) signifies that \(l(x)\) identifies values with higher probabilities, and \(g(x)\) identifies values with lower probabilities, resulting in a larger Expected Improvement (\(EI\)). Both \(l(x)\) and \(g(x)\) are represented in a tree structure, facilitating the collection of samples to obtain more refined information. In each iteration, the algorithm returns the value \(x^{*}\) with the maximum \(EI\).