# How to Open a Black Box Classifier for Tabular Data
## 2 Materials and Methods
### Methods
#### 2.1.4 Summary of the Method  
The following is the pseudo-code for PRiSM models, taking as a starting point a data set comprising a set of P-dimensional input variables normalised according to (13) and the corresponding black box model predictions. Algorithm Partial Response Models _PRiSM(BB, D)_:  
**Input:** set \(D\) of training examples; predictions \(P(\overline{C}\,|\,x)\) from a pre-trained black box model \(BB\). **1. ANOVA decomposition:** apply the recursion given by Equation (6) to the _logit(P(C\(\,|\,x\))_. This may use either of the suggested measures, Dirac and Lesbesgue. The former leads to an anchored decomposition referenced to the choice of anchor point; the component functions generated by the latter are summations weighted by the density function of the covariates, \(P(x)\). **2. Model selection with the Lasso**: input the set of univariate and bivariate partial responses \(\varphi_{i}(x_{i})\) and \(\varphi_{ij}(x_{i},x_{j})\) from (6) calculated over the training data set \(D\) as new inputs for variable selection with the logistic regression Lasso. The Lasso will also output a linear coefficient for each partial response, \(\beta_{i}\)_and_\(\beta_{ij}\), as well as an intercept \(\beta_{0}\), generally resulting in good calibration. **Output _prBB(BB, D)_:** this is the output of the Lasso in Step 2, which has the form of a GAM, shown in Equation (5), truncated to the selected subset of functions \(\varphi_{i}(x_{i})\) and \(\varphi_{ij}(x_{i},x_{j})\):  
\[logit(prBB(C|x))\equiv\varphi(0)+\sum_{i}\beta_{i}\varphi_{i}(x_{i})+\sum_{i \
eq j}\beta_{ij}\varphi_{ij}(x_{i},x_{j}) \tag{27}\]Each partial response comprises a non-linear function of its arguments. Consequently, the model prediction equals the sum of all partial responses plus the intercept, weighted by the linear coefficients from Step 2, followed by the application of the sigmoid function, which inverts the _logit(\(P(C\,|\,x)\))_. The predictions for anchored decompositions are indexed by the pre-fix _pr_ followed by an abbreviation of the black box algorithm, e.g., prSVM and prGBM. **3. Predictions with _PRiSM models:_** given a test data point, the \(\varphi_{i}(x_{i})\) and \(\varphi_{ij}(x_{i},x_{j})\) are calculated using Equations (14)-(16) or (17)-(20), and the predicted output follows from (27). The input variables are, therefore, directly linked to the predictions through interpretable functions. Anchored decomposition applied to the MLP:  
By denoting the output _prMLP (MLP, D)_ by _MLP-Lasso_, it is then possible to continue training as follows:  
**4. Map the MLP-Lasso into a GANN/SENN**: this has the form of a GANN/SENN, meaning that it is not fully connected, as shown in Figure 2. The adjustments to the weights are explained in Equations (21)-(26) and in Section 2.1.3. **Output Partial Response Network [PRN]:** having initialised a structured neural network in the previous step, so that it exactly replicates the component functions and output of the MLP-Lasso, back-error propagation is applied to continue the training of this network. The PRN is a probabilistic binary classifier, so training will use the log-likelihood cost function. Note that the component functions no longer conform with the requirements of an ANOVA decomposition as they will have been adjusted without the constraint of orthogonality. **Output PRN-Lasso:** Steps 1,2. are then applied to the PRN instead of the original MLP. This generates a new set of partial responses \(\varphi_{i}^{*}(x_{i})\) and \(\varphi_{ij}^{*}(x_{i},x_{j})\) and corresponding coefficients \(\beta_{i}^{*}\)_and_\(\beta_{ij}^{*}\) from which the model predictions follow by inserting these coefficients and partial responses into Equation (27). The second training iteration enables the partial responses to be refined without being adjusted for input variables that were removed from the model by the Lasso in Step 2. Therefore, the PRN-Lasso-BB models will always comprise a subset of the variables in the PRN-BB models, but not necessarily with the same functional form.