# Machine Learning-Based Prediction and Optimisation System for Laser Shock Peening
## 3 [PERSON]'s Algorithm Theory
#### Bayesian Neural Networks  
One of the major drawbacks in the conventional ANN search algorithm is the tendency to get entrapped in local and global minima that can adversely affect the generalisation ability of the network. In a Bayesian framework [29], training the neural network entails minimizing an error function using hyperparameters in order to regularise the weight matrix and control the noise parameters. Such network-pruning algorithms can effectively maximize the likelihood function to obtain the best set of parameters to prevent overfitting. The principle of Occam's razor [30] emphasizes the importance of preferring simpler models to complex ones. This is realised within the Bayesian framework and is especially useful for weight regularisation and marginalisation of the network output [29; 30]. The generalisation ability of the network is determined by the error function \(E(w)\) expressed as:  
\[E(w)=\beta E\text{s}+\alpha E_{R} \tag{9}\]  
\[E_{S}=\frac{1}{2}\sum_{i=1}^{M}\{a-p(x,w)\}^{2} \tag{10}\]  
\[E_{R}=\frac{1}{2}\sum_{i=1}^{R}\lvert w_{i}\rvert^{2} \tag{11}\]  
where \(\beta\) is the hyperparameter controlling the variance in noise, \(\alpha\) is the regularisation coefficient and \(w\) the weight vector. \(E_{S}\) is same as the term \(E(x\),\(w)\) described in Equation (3). The regularisation term (\(E_{R}\)) favours small values of network parameters, thereby eliminating the possibility of overfitting noise in the training data. Figure 5: Schematic representation of [PERSON]’s algorithm showing the different steps involved in the calculation of the relative importance (\(RI\)) of input parameters used in the ANN model. ANNs can perform poorly when the 'weights' are reported to have unrealistically large values to fit the details in the training data. A suitable prior distribution of weights \(P(w)\) was considered before observing the data instead of a single set of weights. Using [PERSON] rule, the posteriori probability distribution of the weights can be expressed as described by the authors of [29], where \(P(D/w)\) is a dataset likelihood function and \(P(D)\) is a normalization factor. \[P(w/D)=\frac{P(D/w)P(w)}{P(D)} \tag{12}\]  
We can express the distribution as an exponential of the form if a Gaussian prior is assumed using [34]:  
\[P(w)=\frac{1}{Q_{R}(\alpha)}exp\bigg{(}\frac{-\alpha}{2}|w|^{2}\bigg{)} \tag{13}\]  
where \(Q_{R}(\alpha)\) is a normalisation factor. Hence, when \(|w|^{2}\) is large, \(E_{R}\) is large, and \(P(w)\) is small. Thus, the choice of prior distribution suggests that smaller weight values lessen the chances of overfitting the data. In this study, a prior distribution of weights following a Gaussian distribution was used with the mean centred around 0 and standard deviation of 0.1. The evidence program [29] that assumes a Gaussian approximation of the Bayesian neural network was used to develop an iterative algorithm for determining optimal weights and hyperparameters. The evidence procedure estimates the posterior density of the hyperparameters as:  
\[P(\alpha,\beta/D)=\frac{P(D/\alpha,\beta)P(\alpha,\beta)}{P(D)} \tag{14}\]  
This was undertaken by evaluating hyperparameters that increased the posterior probability of the weight matrix, and the subsequent calculations were carried out with hyperparameters set to these values. In neural network regression problems where the outputs happen to be very close to the target values, the Hessian matrix (\(H\)), defined as the second-order partial derivatives of the error function (\(E\)), can be calculated using the outer-product approximation [35] given by:  
\[\text{Hessian matrix}\ H=\frac{\partial^{2}E}{\partial w_{ji}w_{kj}}=\sum_{n} \frac{\partial y^{n}}{\partial w_{ji}}\frac{\partial y^{n}}{\partial w_{kj}} \tag{15}\]  
where \(y\) is the predicted output, and \(w_{ji}\) and \(w_{kj}\) denote the first- and second-layer connection weights.