# How to Open a Black Box Classifier for Tabular Data
## 3 Results
### Real-World Data  
The benchmarking results for the interpretable models against the original black box classifiers are summarised in Table 5. Values below the CI of the AUC are in italics. All methods use the same data sets, and the AUCs are quoted for test data only. Measuring statistical significance with the McNemar test shows that the performance difference between any pair of models is not significant at the 5% level. While the accuracy of all models is comparable, the PRiSM models use fewer variables and are intuitive to interpret. It is also apparent that the two different measures lead to very similar classification performances. The coefficients of the Lasso used for re-calibration are close to unity for all models. The number of component functions in Table 5 shows the effect of variable selection by the Lasso. The Diabetes data set generates only univariate responses. However, the Credit Card and Shuttle data sets require two-way interactions, as well as univariate effects. Note that the Credit Card data set generates 300 partial responses to choose from. The GAMs, seeded by the SVM and GBM, are calibrated by the LASSO, resulting in the prSVM and prGBM. The univariate and bivariate structure of these models can be used \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**AUC [CI]** & **D** & **Diabetes** & **D** & **Credit Card** & **D** & **Shuttle** \\ \hline MLP & 7 & 0.902 [0.850,0.954] & 24 & 0.815 [0.758,0.872] & 6 & 0.999 [0.998,1.000] \\ SVM & 7 & 0.817 [0.749,0.884] & 24 & 0.793 [0.733,0.852] & 6 & 0.999 [0.999,1.000] \\ _GBM_ & 7 & 0.816 [0.748,0.884] & 24 & 0.784 [0.724,0.845] & 6 & 1.000 [0.999,1.000] \\ \hline _PRiSM models_ & \multicolumn{5}{c}{_Dirac measure_} \\ \hline MLP–Lasso & 5 & 0.902 [0.851,0.954] & 12 & 0.818 [0.762,0.875] & 3 & 0.999 [0.999,1.000] * \\ PRN & 5 & 0.903 [0.851,0.954] & 12 & 0.809 [0.752,0.867] & 3 & 0.999 [0.998,1.000] * \\ PRN–Lasso & 5 & 0.903 [0.851,0.955] & 12 & 0.815 [0.758,0.872] & 2 & 0.998 [0.997,0.999] * \\ prSVM & 5 & 0.884 [0.829,0.940] & 13 & 0.798 [0.739,0.857] & 3 & 0.998 [0.997,0.999] * \\ prGBM & 8 & 0.847 [0.784,0.910] & 10 & 0.763 [0.700,0.825] & 2 & 0.998 [0.997,0.999] \\ \hline _PRiSM models_ & \multicolumn{5}{c}{_Lebesgue measure_} \\ MLP–Lasso & 4 & 0.889 [0.835,0.944] & 12 & 0.819 [0.763,0.876] & 3 & 0.999 [0.998,1.000] * \\ PRN & 4 & 0.903 [0.852,0.955] & 12 & 0.817 [0.760,0.874] & 3 & 0.999 [0.998,1.000] * \\ PRN–Lasso & 4 & 0.905 [0.853,0.956] & 11 & 0.819 [0.762,0.875] & 2 & 0.999 [0.998,1.000] * \\ prSVM & 6 & 0.896 [0.842,0.949] & 12 & 0.803 [0.745,0.861] & 3 & 0.998 [0.997,0.999] * \\ prGBM & 7 & 0.881 [0.824,0.937] & 9 & 0.791 [0.732,0.851] & 2 & 0.997 [0.995,0.998] \\ \hline \hline \end{tabular}  
* Indicates a two-stage model selection process, explained in the text.  
\end{table} 
Table 5: Classification performance for the real-valued data sets. The label ‘D’ indicates the number of input variables for the black boxes and component functions for the PriSM models. to define a PRN model, which is a SENN with MLP components, initialised either with random weights or with univariate and bivariate modules trained to replicate each of the selected partial responses. This will replicate the PRN and, following orthogonalization, the PRN-Lasso. The sparsity of the models and their potential for interpretation are illustrated by the partial responses of two models, the MLP-Lasso and the PRN, shown in Figures 6-11. These functions are derived from the training data and are always used for prediction on out-of-sample data. The corresponding component functions for the other PRiSM models have similar values, although, if derived from random forests, e.g., in the case of the prGBM, they are stepwise constant rather than smooth. This is shown in [20] for a different data set. Figure 6: Contributions to the logit from partial responses to the logit (left axis) for the Diabetes data set, obtained with the Dirac measure, overlapped with the histogram of the training data (right axis).