# Application and Evaluation of Surrogate Models for Radiation Source Search
## 3 Surrogate Models
### Multivariate Adaptive Regression Splines
The MARS algorithm purposefully overfits the model to the data in its forward simulation and then performs a backward deletion strategy in the second step to remove basis functions that no longer contribute sufficiently to the accuracy of the model fit. This is performed via a model selection procedure employing Generalized Cross-Validation (GCV) to compare models with subsets of the basis functions. The GCV equation,  
\[GCV=\frac{\frac{1}{M}\sum_{m=1}^{M}\left[y^{m}-u^{N}(q^{m},w)\right]^{2}}{ \left[1-\frac{N+\kappa\times(N-1)/2}{M}\right]^{2}}, \tag{30}\]  
is a goodness of fit test that uses the parameter \(\kappa\) to penalize large numbers of basis functions \(N\)[25]. We use a default penalty parameter of \(\kappa=3\) and a threshold value of \(GCV<10^{-3}\), which is used as a stopping criterion for the backward phase. The suggested threshold value is \(10^{-4}\) and should be reduced for noise free data. To better understand the MARS model structure, consider the problem of approximating the data depicted in Figure 4. We employ the MARS model  
\[u^{N}(q)=20-\max(q-12,0)+3\times\max(12-q,0)-2\times\max(22-q,0) \tag{31}\]  
to approximate the data. The knot locations at \(q=12\) and \(q=22\) delimit the regions where different linear relationships are identified. By considering the products of these linear basis functions, we can quantify nonlinear behaviors of the model. To motivate error analysis of the MARS surrogate model, we consider the full MARS model  
\[u^{N}(q,w)=w_{0}+\sum_{j=1}^{N}\,w_{j}\,\prod_{k=1}^{K_{j}}\,[s_{jk}(q_{v(j,k )}-t_{jk})]_{+}, \tag{32}\]  
where \(w_{0}\) is the coefficient of the constant basis function. The sum is over the basis functions  
\[B_{i}(q_{i}) = \sum_{K_{j}=1}w_{j}\Psi_{j}(q_{i})=\sum_{K_{j}=1}w_{j}[s_{j1}(q_{i }-t_{j1})]_{+},\text{ for }i\in V_{j}, \tag{33}\] Figure 3:0 (34) Figure 3:1  
where we now consider higher-order products of basis functions; i.e., \(K_{j}\geq 1\). Here, \(V_{j}=\{v(j,k)\}_{1}^{K_{j}}\) is the set of predictor variable component labels for the \(j\)th basis function \(\Psi_{j}\). Whereas this representation of the model does not provide insight into the model development, it allows us to rearrange the model in a way that reveals the predictive behavior of the model. By collecting basis functions that involve identical predictor variable sets, we obtain the representation  
Figure 3:2  
These sums represent the \(K_{j}\) level interactions, if present, between the variables within the model. By adding the univariate contributions to the bivariate contributions, we obtain the representation  
Figure 3:3  
This provides a bivariate tensor product spline approximation representing the joint bivariate contributions of \(q_{i}\) and \(q_{k}\) to the model [24]. Similar rearranging can be performed by employing this representation combined with the trivariate functions to obtain the joint contributions of \(q_{i}\), \(q_{k}\), and \(q_{\ell}\). Since Equation (35) is similar to analysis of variance decomposition [17], we refer to this as the ANOVA decomposition of the MARS model. The optimal additive approximation corresponds to the first-order terms in the ANOVA expansion, thus, if the higher-order indices are small, then the function can be approximated well by an additive model; i.e., \(K_{j}=1,\ j=1,\ldots,N\). Note that, for our purposes, we consider a model with second-order interactions, thus we truncate Equation (35) at the second-order interactions. Unfortunately, the adaptivity of MARS makes it difficult to bound the error in the manner of spectral approaches that bound the error in terms of the coefficients \(w_{j}\). However, cross validation procedures, such as those detailed in Ref. [21; 22], can be used to provide an error estimate for the MARS model. The MARS algorithm can be cast in a Bayesian framework in which case the number of basis functions \(N\), their coefficients \(w_{j}\), and their form--knot points \(t_{jk}\), sign indicators \(s_{jk}\), and the level of interaction \(K_{j}\)--are considered to be random [26].