## 3 Neural Network-Based Bridge Maintenance Cost Prediction Framework
### Bayesian Optimization  
To improve the prediction performance of the model, this study uses Bayesian optimization to determine some important hyperparameters in random forest, isolation forest, and fully-connected ANN and CNN. The hyperparameters herein are the non-trainable parameters in the model. Those parameters (for example, the number of neurons and the structure of the network layer) are set before the training process starts. Nevertheless, a few hyperparameters, such as the number of trees and the number of predictor variables to sample at each node, are important hyperparameters in random forest and isolation forest. These parameters may have an important impact on the calculation performance of the forest [29]. Other parameters, such as the regularization coefficient and the learning rate,are important hyperparameters in neural networks. These parameters are very important for the neural network model to improve robustness to databases with small sample sizes. Since Bayesian optimization has the advantage of sample efficiency, a relatively good hyperparameter combination can be found with a very small number of cycles. The calculation principle of adopting it to determine hyperparameters is as follows:  
The basic assumption that the objective function \(E_{v}(\overline{\theta})\) is adopted for Bayesian optimization is obtained, a priori, by the Gaussian process (GP), \(E_{v}\big{(}\overline{\theta}\big{)}\sim N(0,K)\); \(\overline{\theta}\) is the hyperparameter vector considered in this study. Since \(E_{v}\big{(}\overline{\theta}\big{)}\) is corrupted by Gaussian noise with a mean value of zero and a standard deviation of \(\sigma_{noise}\), the kernel matrix can be expressed as [30]:  
\[ \begin{array}{cccc}k\big{(}\overline{\theta}_{1},\overline{\theta}_{1})& \cdots&k\big{(}\overline{\theta}_{1},\overline{\theta}_{t}\big{)}\\ \vdots&\ddots&\vdots&+\sigma_{noise}^{2}\mathbf{I}\\ k\big{(}\overline{\theta}_{t},\overline{\theta}_{1})&\cdots&k\big{(} \overline{\theta}_{1},\overline{\theta}_{t}\big{)}\end{array}  \tag{6}\]  
where \(k\big{(}\overline{\theta}_{t},\overline{\theta}_{t}\big{)}\) is the covariance function. The results of previous iterations are expressed as: \(\mathbf{D}_{1:t}=\big{(}\overline{\theta}_{1:t},\mathbf{E}_{1:t}^{V}\big{)}\), where \(\mathbf{E}_{1:t}^{V}=E_{v}\big{(}\overline{\theta}_{1:t}\big{)}\). Assume \(\overline{\theta}_{t+1}\) as the next point to estimate and the value of the function at \(\overline{\theta}_{t+1}\) as \(E_{1:t}^{V}=E_{v}\big{(}\overline{\theta}_{t+1}\big{)}\). \(\mathbf{E}_{1:t}^{V}\) and \(E_{1+1}^{V}\) are joint Gaussian expressions under GP prior. The predictive distribution can be obtained as follows [31]:  
\[E_{t+1}^{V}\Big{|}\mathbf{D}_{1:t}\sim N(\mu\big{(}\overline{\theta}_{t+1} \big{)},\sigma^{2}\big{(}\overline{\theta}_{t+1}\big{)}+\sigma_{noise}^{2} \tag{7}\]  
where  
\[\mu\big{(}\overline{\theta}_{t+1}\big{)}=\mathbf{k}^{T}\Big{(}\mathbf{K}+ \sigma_{noise}^{2}\mathbf{I}\Big{)}^{-1}\mathbf{E}_{1:t}^{V} \tag{8}\]  
\[\sigma^{2}\big{(}\overline{\theta}_{t+1}\big{)}=k\big{(}\overline{\theta}_{t+ 1},\overline{\theta}_{t+1}\big{)}-\mathbf{k}^{T}\Big{(}\mathbf{K}+\sigma_{ noise}^{2}\mathbf{I}\Big{)}^{-1}\mathbf{k} \tag{9}\]  
\[\mathbf{k}=\big{[}k\big{(}\overline{\theta}_{t+1},\overline{\theta}_{1}\big{)} k\big{(}\overline{\theta}_{t+1},\overline{\theta}_{2}\big{)}\cdots k\big{(} \overline{\theta}_{t+1},\overline{\theta}_{t}\big{)}\big{]}^{T} \tag{10}\]  
It can be seen from the above formulas, that both the predicted variance function \(\sigma^{2}\big{(}\overline{\theta}_{t+1}\big{)}\) and the predicted mean function \(\mu\big{(}\overline{\theta}_{t+1}\big{)}\) depend on the choice of the covariance function \(k\big{(}\overline{\theta}_{t},\overline{\theta}_{t}\big{)}\), and they jointly determine the choice of the covariance function \(E_{t+1}^{V}\big{|}\mathbf{D}_{1:t}\). This study uses the automatic relevance determination (ARD) Matern 5/2 kernel [32]. This choice adapts to the problem of the ARD squared index kernel.