# How to Open a Black Box Classifier for Tabular Data
## 2 Materials and Methods  
### Methods  
#### 2.1.1 ANOVA Decomposition  
The first novelty of the paper is to apply an ANOVA decomposition [10] to pre-trained black box probabilistic binary classifiers in order to extract from the _logit(P(C1x))_, which is a multivariate function, component functions of fewer variables. The ANOVA decomposition is defined as follows  
\[ \begin{split} logit(P(C|x))\equiv\log&\Big{(}\frac{P(C|x )}{1-P(C|x)}\Big{)}\\ &=\varphi_{0}+\sum\limits_{i}\varphi_{i}(x_{i})+\sum\limits_{i \
eq j}\varphi_{ij}\big{(}x_{i},x_{j}\big{)}+\ \ldots\ +\sum\limits_{i_{1}\
eq\ \ldots\ \
eq i_{p}}\varphi_{i_{1}\ \ldots\ i_{p}}\big{(}x_{i_{1}},\ldots,x_{i_{p}}\big{)}\end{split}  \tag{5}\]  
where the general form of the terms in (5) is a recursive function of the nested subsets of the covariate indices \(\{i_{1},\ldots,i_{p}\}\) with the property that the term involving all of the covariates \(x_{i}\): \(i=1\ldots\ P\), where \(P\) is the dimensionality of the input data is given by  
\[ \begin{split}\varphi_{i_{1}\ \ldots\ i_{p}}\big{(}x_{i_{1}},\ldots,x _{i_{p}}\big{)}\\ =&\ logit\big{(}P\big{(}C|x_{i_{1}},\ldots,x_{i_{p}} \big{)}\big{)}-\sum\limits_{\{i_{1}\
eq\ \ldots\ \
eq i_{p-1}\}}\varphi_{i_{1}\ \ldots\ i_{i-1}}\big{(}x_{i_{1}},\ldots,x_{i_{p-1}}\big{)}- \varphi_{0}\end{split}  \tag{6}\]  
Note that Decomposition (5) is an identity that exactly reproduces the values of the \(logit(P(C|x))\), originally predicted by the black box classifier. We call the component functions \(\varphi_{i_{1}\ \ldots i_{p}}(x_{i_{1}},\ldots,x_{i_{n}})\) partial responses, since they involve only a subset of the input variables. The general form of the component terms is given by the following equations, which depend only on the chosen measure \(\mu(x)\)  
\[\varphi_{0}=\int_{[x]^{p}}logit(P(C|x))d\mu(x) \tag{7}\]  
\[\varphi_{S}(x_{s})=\int_{[x]^{p-|S|}}logit(P(C|x))d\mu(x_{-S})-\sum\limits_{T \subset S}\varphi_{T}(x_{T}) \tag{8}\]  
where \(S\in R^{S}\) represents a subset of variables with dimensionality \(|S|\leq P\). The terms \(x_{s}\) and \(x_{-s}\) denote, respectively, the subspace spanned by \(S\): \(|S|=n\) in Equation (6) and its complement \(-S\): \(|-S|=d-n\). It follows from (7) and (8) that the terms \(\varphi_{S}\) are normalised with respect to the chosen measure  
\[\int_{S}\varphi_{S}(x_{s})d\mu\big{(}x_{j}\big{)}=0,\ if\ j\in S \tag{9}\]  
and also orthogonal for disjoint variable sets \(S\) and \(T\)  
\[\int_{S}\varphi_{S}(x_{s})\varphi_{T}(x_{T})d\mu(x)=0,\ if\ S\
eq T. \tag{10}\]  
There are two natural choices of measure, each of which will define the functionality of each of the component terms \(\varphi_{S}\) in response to either one or two arguments:  
* Dirac measure \[d\mu(x)=\delta(x-x_{c})dx\] (11) An arbitrary point \(x_{c}\) that is called anchor point. The partial responses become cuts through the response surface for the _logit(P(C1x)\)_. * [PERSON] measure \[d\mu(x)=\rho(x)dx\] (12) where \(\rho(x)\) is the density function of the variables in the argument of the integral. This measure calculates the weighted mean of the integrand. In both cases, the data matrix \(X\) is first centred using the overall median of the data and scaled by the marginal standard deviation:  
\[X\rightarrow(X-median(X))\Bigg{/}std(X) \tag{13}\]  
The absence of a variable now corresponds to fixing it at the median value, since the median point corresponds to a vector of 0 s. Therefore, the logit value then takes the value \(logit(P(C|0))\). Similarly, if all of the variables except \(x_{i}\) are set to their median values, then the corresponding values of \(logit(P(C|(0,\ldots,x_{i},\ldots,0)))\) represent a function of just that one variable. The same principle applies when only two variables are not 0, then three variables, etc.