# Application and Evaluation of Surrogate Models for Radiation Source Search
## 3 Surrogate Models
### Legendre Polynomials
}\), where \(p\) is the number of parameters--three in our case--and \(K\) is the maximum degree of the multivariate Legendre polynomials [17]. We construct the multivariate Legendre polynomials as tensor products of univariate Legendre polynomials. As discussed in Ref. [20], the surrogate model error decreases roughly exponentially with polynomial degree, provided that a high enough order quadrature rule is used; e.g., large \(R\). However, results with high degree polynomials--large \(K\)--diverge because of large fluctuations in the function \(u^{N}\) between the quadrature points produced by overfitting; i.e., \(u^{N}\) begins fitting observation errors. To determine the correct degree \(K\), we compute the sum of squares (SSq) error of the surrogate model evaluated at the test points \(q^{s}\)  
\[SSq=\sum_{s=1}^{S}[u(q^{s})-u^{N}(q^{s},w)]^{2}, \tag{24}\]  
and compute the likelihood  
\[\pi(u|q)=\frac{1}{(2\pi\sigma_{0})^{S/2}}e^{-SSq/2\sigma_{0}^{2}}, \tag{25}\]  
for the surrogate model \(u^{N}\) with maximum polynomial degree \(K=1,...,30\). We plot the results in Figure 3, where the results for the \(d=10\) surrogate models are plotted with the mean value overlaid in a thicker line. We see that the mean sum of squares error has a distinct minimum and the mean likelihood has a maximum at \(K=21\). Hence, we use this value when computing the surrogate model and employ \(N=2024\) total basis functions. Akaike information criteria (AIC), Bayesian information criteria (BIC), or cross validation techniques can also be employed to determine the value of \(K\) that balances accuracy versus overfitting of the surrogate model. These cross-validation procedures are discussed further in Ref. [21; 22]. An alternative to using such a high degree polynomial for this surrogate model is to employ splines, which we consider in Section 3.2. We note that these surrogate models, excluding the Gaussian processes surrogate model, are all parametric models. Whereas there are multiple definitions of parametric models, we define a parametric model as one where the class of basis functions is defined prior to construction of the model. Since we employ the class of Legendre polynomials to construct this surrogate model, we classify it as a parametric surrogate model. We classify Gaussian processes as semi-parametric under this definition, as is discussed in Section 3.4. An alternate method to obtain the coefficients \(w_{j}\) is to perform a least absolute shrinkage and selection operator (LASSO) regression [23]. Borrowing from compressive sensing (CS), we bound the \(l_{1}\) norm of the coefficients to enforce sparsity. Therefore, we can formulate the problem as the optimization problem  
\[\min_{c}||\Lambda c-y||_{2}^{2},\quad\text{subject to }||c||_{1}\leq\tau, \tag{26}\]  
where \(\Lambda_{jm}=\Psi_{j}(q^{m})\) is a matrix of the Legendre basis functions, \(y=[y^{1},...,y^{M}]^{T}\) is the vector of observations obtained from the statistical model defined in Equation (12), and \(c=[w_{1},...,w_{N}]^{T}\) is the vector of coefficients. To solve this optimization problem, we use the MATLAB SPGL1 solver, which is detailed in Ref. [23]. We tested multiple values of \(\tau\), but use \(\tau=35\) to obtain the results discussed in Sections 4 and 5. We determine this value of \(\tau\) by computing the 1-norm of the coefficients obtained via discrete projection. We find that these coefficients have a 1-norm between 32 and 38, and, when we decrease \(\tau\) below 32, significant error is introduced. When we increase \(\tau\) to greater than 38, the 1-norm of the coefficients is less than \(\tau\), meaning that the constraint has no effect on the problem. When comparing the coefficients between these two methods, we see that they are similar but not identical, even when \(\tau>38\). Further analysis on determining optimal values of \(\tau\) is discussed in Ref. [23].