# Application and Evaluation of Surrogate Models for Radiation Source Search
## 3 Surrogate Models
### Neural Networks  
The final surrogate model is based on neural networks, which was originally developed to solve problems in a way that emulates the brain. They are typically organized in layers of their core structures, called neurons or nodes, each of which has an inherent activation function. There are also sets of coefficients that act on the connections between nodes, which are tuned by a learning algorithm and are capable of accurately approximating nonlinear functions [35]. For our model, the input nodes of the neural network are the parameter components \(\{q_{i}\}_{i=1}^{p}\) associated with the location and intensity of a nuclear source. The output node is the neural network surrogate model approximation to the ray-tracing solution defined in Equation (3) for a detectors response \(u(q)\). Here, we develop a feed-forward artificial neural network, meaning that the information is only passed forward through the hidden layer. This is unlike a feedback network, which allows for information transfer in both directions and consequently loops within the network. We note that while we employ supervised machine learning methods, unsupervised learning methods might potentially be employed for future work. The construction of the neural network surrogate model is divided into two steps: choosing a network structure and training the network. To define the network architecture, we must define the number of hidden layers, the number of neurons in each layer, the activation functions associated with each neuron, and the performance function used to evaluate the accuracy of the network during \begin{table}
\begin{tabular}{c c c} \hline \hline
**Kernel Function** & **Computation Time (s)** & **rRMSE** \\ \hline Exponential & 203 & 0.1993 \\ Squared Exponential & 175 & 0.2013 \\ Matern \(3/2\) & 196 & 0.2006 \\ ARD Exponential & 448 & 0.2005 \\ ARD Matern \(3/2\) & 426 & 0.1975 \\ ARD Matern \(5/2\) & 401 & 0.1995 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of Gaussian process kernel functions for surrogate model construction. training. There have been many advances made in the development of deep learning algorithms [36], but for simplicity we consider a single hidden layer for this model. We set the hidden layer of the fitting network to a size of 35, which we obtained from testing hidden layer sizes to obtain an optimal hidden layer size for this problem. We note that the use of a large number of hidden layers or hidden layer neurons can lead to overfitting and increased computational time. Hence, these measures are normally employed only for highly complex problems. Each neuron performs a linear transformation, \(\eta=w_{1}q_{1}+w_{2}q_{2}+...+w_{p}q_{p}\) for the \(p\) components \(\{q_{i}\}_{i=1}^{p}\), where \(w_{j},\ i=1,\ldots,p\) are the neural network coefficients. Therefore, there are \(p\times N+N\times 1\) coefficients to train for this model, where \(N=35\) is the number of neurons in the model and one is the dimension of the output. This transformation is followed by a nonlinear operation defined by the symmetric sigmoid activation function  
\[\Psi(q,w)=\Psi(\eta)=\frac{2}{1+e^{-2\eta}}-1. \tag{54}\]  
We employ the MATLAB neural network toolbox to evaluate the surrogate model. We apply the mean squared error performance function, since this performance function provides a good balance between accuracy and computation time during surrogate model construction when compared with other performance functions provided by the MATLAB neural network toolbox. To train the network, we employ a nonlinear least-squares regression to compute the coefficients using the training data \(\{q^{j},u(q^{j})\}_{j=1}^{M}\). We employ the Levenberg-Marquardt back-propagation training function, since this outperforms other built-in training functions in terms of error. The only exception is Bayesian regulation back-propagation, which requires approximately four times the computational time for an improvement of only 0.01 in the surrogate model \(rRMSE\).