# How to Open a Black Box Classifier for Tabular Data
###### Abstract  
A lack of transparency in machine learning models can limit their application. We show that analysis of variance (ANOVA) methods extract interpretable predictive models from them. This is possible because ANOVA decompositions represent multivariate functions as sums of functions of fewer variables. Retaining the terms in the ANOVA summation involving functions of only one or two variables provides an efficient method to open black box classifiers. The proposed method builds generalised additive models (GAMs) by application of L1 regularised logistic regression to the component terms retained from the ANOVA decomposition of the logit function. The resulting GAMs are derived using two alternative measures, Dirac and Lebesgue. Both measures produce functions that are smooth and consistent. The term partial responses in structured models (PRISM) describes the family of models that are derived from black box classifiers by application of ANOVA decompositions. We demonstrate their interpretability and performance for the multilayer perceptron, support vector machines and gradient-boosting machines applied to synthetic data and several real-world data sets, namely Pima Diabetes, German Credit Card, and Statlog Shuttle from the UCI repository. The GAMs are shown to be compliant with the basic principles of a formal framework for interpretability.  
ANOVA; Shapley values; self-explaining neural networks; generalised additive models; interpretability +
Footnote †: journal:  
0000-0001-1101-1011-1011-1011-1011-1011-1011-1011-10111-10  
as a combination of independent effects, much in the way that logistic regression does, but allowing for non-linear functions of the input variables. Specifically, for input dimensionality \(d\), the odds ratio of this probability has the form  
\[\frac{P(C|x)}{1-P(C|x)}=e^{\varphi_{1}(x_{1})}e^{\varphi_{1}(x_{2})}\ \ldots\ e^{\varphi_{d(x_{d})}}e^{\varphi_{1,2}(x_{1},x_{2})}\ \ldots\ e^{\varphi_{(d-1),d}(x_{(d-1),x_{d}})}e^{\varphi_{0}} \tag{1}\]  
where the terms \(\varphi_{i}(x_{i})\) are univariate functions, hence easily interpreted, \(\varphi_{ij}(x_{i},x_{j})\) are bivariate functions, which can also be easily plotted, and \(\varphi_{0}\) is the _null model_ for which all of the input variables \(\varphi_{i}(x_{i})\) and \(\varphi_{ij}(x_{i},x_{j})\) are set to 0.  
From a Bayesian perspective, each component \(e^{\varphi}\) models the contribution of an individual variable or pair of variables, which can enhance or suppress the P(C \(\mid\) x) depending on whether the value taken by the argument function \(\varphi\) is positive or negative, acting on the baseline value \(e^{\varphi_{0}}\), which, if \(\varphi_{i}(0)\) is always 0, corresponds to the prior odds ratio in the absence of any of the input variables being present.  
As the variables are entered into Equation (1), they modulate the prediction of P(C \(\mid\) x), much in the same way as a human observer can start with a prior probability, e.g., for the diagnosis of a clinical state, then modulate that diagnosis as the observations of the symptoms are made; each symptom contributing to increasing or reducing the probability of diagnosing the clinical state according to Equation (1). In the medical domain, many risk models are quantified by exactly this model, usually expressed as a risk score, namely  
\[score(x_{1},x_{2},\ldots,x_{d})=\sum_{i=1}^{d}\beta_{i}\cdot x_{i} \tag{2}\]  
with  
\[logit(P(c|x))=log\bigg{(}\frac{P(C|x)}{1-P(C|x)}\bigg{)}=\beta_{0}+score(x_{1 },x_{2},\ldots,x_{d}) \tag{3}\]  
This corresponds to the GAM defined by Equation (1) with only univariate terms given by  
\[\varphi_{i}(x_{i})=\beta_{i}\cdot x_{i} \tag{4}\]