Title: LEARNING GLOBAL ADDITIVE EXPLANATIONS FOR NEURAL NETS USING MODEL DISTILLATION Anonymous

Abstract: Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample. These explanations are appealing due to their simplicity and local fidelity. However, they do not provide information about the general behavior of the model. We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions. These global explanations take the form of feature shapes, which are more expressive than feature attributions. Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets. A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ATNcgurNHhc.

1. INTRODUCTION:

Recent research in interpretability has focused on developing local explanations: given an existing model and a sample, explain why the model made a particular prediction for that sample (Ribeiro et al., 2016) . The accuracy and quality of these explanations have rapidly improved, and they are becoming important tools to understand model decisions for individual samples. However, the human cost of examining multiple local explanations can be prohibitive with today's large data sets, and it is unclear whether multiple local explanations can be aggregated without contradicting each other (Ribeiro et al., 2018; Alvarez-Melis & Jaakkola, 2018) .

In this paper, we are interested in global explanations that describe the overall behavior of a model. While usually not as accurate as local explanations on individual samples, global explanations provide a different, complementary view of the model. They allow us to clearly visualize trends in feature space, which is useful for key tasks such as understanding which features are important, detecting unexpected patterns in the training data and debugging errors learned by the model.

We propose to use model distillation techniques (Bucilua et al., 2006; Hinton et al., 2015) to learn global additive explanations of the form

F (x) = h 0 + i h i (x i ) + i =j h ij (x i , x j ) + i =j j =k h ijk (x i , x j , x k ) + • • • (1)

to approximate the prediction function of the model F (x). Figure 1 illustrates our approach. The output of our approach is a set of p feature shapes {h i } p 1 that can be composed to form an explanation model that can be quantitatively evaluated. Through controlled experiments, we empirically validate that feature shapes provide accurate and interesting insights into the behavior of complex models. In this paper, we focus on interpreting F from fully-connected neural nets trained on tabular data.

Our goal is not to replace local explanations nor to explain how the model functions internally. What we claim is that we can complement local explanations with global additive explanations that clearly illustrate the relationship between input features and model predictions. Our contributions are:

• We propose to learn global additive explanations for complex, non-linear models such as neural nets. • We leverage powerful generalized additive models in a model distillation setting to learn feature shapes that are more expressive than feature attributions Figure 1 : Given a black box model and unlabeled samples (new unlabeled data or training data with labels discarded), our approach leverages model distillation to learn feature shapes that describe the relationship between input features and model predictions.

• We perform a quantitative comparison of feature shapes to other global explanation methods in terms of fidelity to the model being explained, accuracy on independent test data, and interpretability through a user study.

2. LEARNING GLOBAL ADDITIVE EXPLANATIONS:

Although our approach of using model distillation with powerful additive models of the form in equation 1 is new, our work is based on two previous research threads: (1) decomposing F into additive F to understand how F is affected by its inputs (e.g. Hooker (2007) ), and ( 2) learning an interpretable model (often some form of decision tree) to mimic F (e.g. Craven & Shavlik (1995) ).

2.1. ADDITIVE F:

Global additive explanations have been used to analyze inputs to complex, nonlinear mathematical models and computer simulations (Sobol, 2001) , analyze how hyperparameters affect the performance of machine learning algorithms (Hutter et al., 2014) , and decompose prediction functions into lower-dimensional components (Hooker, 2004) . They are determined by the choice of metric L between F and its approximation F , degree d of highest order components (d = 3 in equation 1, and type of base learner h. One common theme of these methods is that they decompose F into F using numerical or computational methods (e.g. matrix inversion, quasi Monte Carlo). Rather than approximately decomposing F (which can be prohibitively expensive with large n or p)1 , we propose to learn F using model distillation. This is equivalent to choosing L that minimizes the empirical risk between the prediction function F and our global additive explanation F on the training data.

To minimize ||F -F || L , we select two flexible, nonparametric base learners for h: splines (Wood, 2006) and bagged trees. This gives us two global additive explanation models: Student Bagged Additive Boosted Trees (SAT) and Student Additive Splines (SAS). Other choices of h are possible. We describe our distillation setup to learn these models in Section 2.2. In most of this paper, F consists of main components h i (d = 1 in equation 1). Higher order components h ij and h ijk can increase the accuracy of F , but make interpretation more difficult. When F consists of only main components h i , any pairwise or higher order interactions in F are expressed as a best-fit additive approximation added to main components h i , plus a pure-interaction residual. We show examples of this expression in Section 4.1, and show the utility of adding higher order components h ij and h ijk , when present in F , in Section D.2.

2.2. LEARNING F USING MODEL DISTILLATION:

Neural nets and other black-box models have been approximated by interpretable models such as trees, rule lists, etc., either via model distillation/compression (Craven & Shavlik, 1995; Che et al., 2016; Frosst & Hinton, 2017) or model extraction (Fu, 1994; Sanchez et al., 2015; Lakkaraju et al., 2017; Bastani et al., 2017; Ribeiro et al., 2018) . However, all of them approximated classifier models; there has been less work approximating regression models. Another gap in the literature is rule lists for regression: state-of-the-art rule lists (Letham et al., 2015; Angelino et al., 2017) or rule sets (Lakkaraju et al., 2016) do not have regression implementations. Model distillation requires only that the teacher model label a training set, not repeated probing or access to the teacher's internal structure or derivatives. This, combined with the applicability of generalized additive models to both classification and regression, means that our approach can approximate a broad class of classification and regression models. We also show in Sections 4.2.2 and 4.3, with a user study, that additive explanations have advantages over decision trees when it comes to interpretability.

Training teacher neural nets. Our teacher models are fully-connected neural nets (FNNs) with ReLU nonlinearities. We use the Adam optimizer (Kingma & Ba, 2015) with Xavier initialization (Glorot & Bengio, 2010) and early stopping based on validation loss. At each depth, we search for optimal hyperparameters (number of hidden units, learning rate, weight decay, dropout probability, batch size, enabling batch norm) based on average validation performance on multiple train-val splits and random initializations. The most accurate nets we trained are FNNs with 2-hidden layers and 512 hidden units per layer (2H-512,512); nets with three or more hidden layers had lower training loss, but did not generalize as well. In some experiments we also use a restricted-capacity model with 1 hidden layer of 8 units (1H-8) to compare explanations.

Training student additive explanation models. To train SAT and SAS, we find optimal feature shapes {h i } p 1 that minimize the mean square error between the teacher F and the student F , i.e.

EQUATION

)

where F (x) is the output of the teacher model (scores for regression tasks and logits for classification tasks), T is the number of training samples, x t is the t-th training sample, and x t i is its i-th feature. The exact optimization details depend on the choice of h. For trees we use cyclic gradient boosting (Buhlmann & Yu, 2003; Lou et al., 2012) which learns the feature shapes in a cyclic manner. As trees are high-variance, low-bias learners (Hastie et al., 2001) , when used as base learners in additive models, it is standard to bag multiple trees (Lou et al., 2012; 2013; Caruana et al., 2015) . We follow that approach here. For splines, we use cubic regression splines trained using penalized maximum likelihood in R's mgcv library (Wood, 2011) and cross-validate the splines' smoothing parameters.

2.3. INTERPRETABLE BUILDING BLOCKS OF F : FEATURE SHAPES:

Our global additive explanation models, SAT and SAS, can be visualized as feature shapes (Figure 1 ). These are plots with the x-axis being the domain of input feature x i and the y-axis being the feature's contribution to the prediction h i (x i ). This way of representing the relationship between input features and model predictions has precedence in interpretability, from work that learned monotonic (Gupta et al., 2016) or concave/convex (Pya & Wood, 2015) feature shapes from original data (i.e. without distillation), to post-hoc explanations such as partial dependence (Friedman, 2001) , and Shapley additive explanations dependence plots (Lundberg & Lee, 2017) . The latter two are hence natural baselines for SAT and SAS, and we describe the results from our comparison in Section 4.2.1. In Section 4.3, we also describe the results of a user study to evaluate the interpretability of feature shapes, showing that humans are able to understand and use feature shapes.

How are feature shapes different from feature attribution? A classic way to interpret black-box models is feature attribution/importance measures. Examples include permutation-based measures (Breiman, 2001) , gradients/saliency (see Montavon et al. (2017) or Ancona et al. (2018) for a review), and measures based on variance decomposition (Iooss & Lemaitre, 2015) , game theory (Datta et al., 2016; Lundberg & Lee, 2017) , etc. We highlight that feature shapes are different from and more expressive than feature attributions. Feature attribution is a single number describing the feature's contribution to either the prediction of one sample (local) or the model (global), whereas our feature shapes describe the contribution of a feature, across the entire domain of the feature, to the model. Nonetheless, feature attribution, both global and local, can be automatically derived from feature shapes: global feature attribution by averaging feature shape values at each unique feature value; local feature attribution by simply taking one point on the feature shape. In Section 4.3 we show that humans are able to derive feature attribution from feature shapes.

3. EVALUATING GLOBAL EXPLANATIONS:

Lundberg & Lee (2017) suggested the perspective of viewing an explanation of a model's prediction as a model itself. With this perspective, we propose to quantitatively evaluate explanation models as if they were models. Specifically, we evaluate not just fidelity (how well the explanation matches the teacher's predictions) but also accuracy (how well the explanation predicts the original label). Note that Lundberg & Lee (2017) and Ribeiro et al. (2016) evaluated local fidelity (called local accuracy by Lundberg & Lee (2017)), but not accuracy. A similar evaluation of global accuracy was performed by Kim et al. (2016) who used their explanations (prototypes) to classify test data. In our case, we use the feature shapes generated by our approach to predict on independent test data.

Baselines. We compare to two types of baselines: (1) additive explanations obtained by querying the neural net (i.e. without distillation): partial dependence, Shapley additive explanations (Lundberg & Lee, 2017) and linearization through gradients; (2) interpretable models learned by distilling the neural net: trees, rules, and sparse linear models.

Partial dependence (PD) is a classic global explanation method that estimates how predictions change as feature x j varies over its domain:

P D(x j = z) = 1 T T t=1 F ((x t

1 , . . . , x t j = z, . . . , x t p ) where the neural net is queried with new data samples generated by setting the value of their x j feature to z, a value in the domain of x j . Plotting P D(x j = z) by z returns a feature shape.

Linearization through gradient approximation (GRAD). We construct the additive function G through the Taylor decomposition of F , defining G(x) = F (0) + p i=1 ∂F (x) ∂xi x i , and defining the attribution of feature i of value x i as ∂F (x) ∂xi x i . This formulation is related to the "gradient*input" method (e.g. Shrikumar et al. (2017) ) used to generate saliency maps for images. Shapley additive explanations (SHAP). SHAP is a stateof-the-art local explanation method that satisfies several desirable local explanation properties (Lundberg & Lee, 2017). Given a sample and its prediction, SHAP decomposes the prediction additively between features using a game-theoretic approach. We use the python package by the authors of SHAP.

Both GRAD and SHAP provide local explanations that we adapt to a global setting by averaging the generated local attributions at each unique feature value. For example, the global attribution for feature "Temperature" at value 10 is the average of local attribution "Temperature" for all training samples with "Temperature=10". This is the red line passing through the points in Figure 2 . Applying this procedure to GRAD and SHAP's local attributions, we obtain global attributions gGRAD and gSHAP that we can now plot as feature shapes.

4. EXPERIMENTAL RESULTS:

First, we validate our approach on synthetic data with known ground-truth feature shapes (Section 4.1). Next, we quantitatively evaluate our approach on real data against other non-distilled additive explanations (Section 4.2.1) and distilled, not-necessarily additive, interpretable models (Section 4.2.2). Third, we design a user study to evaluate the interpretability of feature shapes (Section 4.3). Finally, we further validate our approach with controlled experiments on real data (Section 4.4).

4.1. VALIDATION USING SYNTHETIC DATA WITH KNOWN GROUND-TRUTH:

For this experiment, we simulate data from synthetic functions with known ground-truth feature shapes to see if our approach can recover these feature shapes. We are particularly interested in observing how predicted feature shapes differ for neural nets of different capacity trained on the same data. Our expectation is that for neural nets that are accurate, our predicted shapes would match the ground-truth feature shapes, independent of how the features are used internally by the net. On the other hand, predicted shapes of less accurate neural nets should less accurately match ground-truth shapes.

Experimental setup. We designed an additive, highly nonlinear function combining components from synthetic functions proposed by Hooker (2004) , Friedman & Popescu (2008) and Tsang et al. (2018) :

F 1 (x) = 3x 1 + x 3 2 -π x3 + exp(-2x 2 4 ) + 1 2+|x5| + x 6 log(|x 6 |) + 2|x 7 | + max(0, x 7 ) + x 4

8 + 2 cos(πx 8 ). Like Tsang et al. (2018) , we set the domain of all features to be U(-1, 1). Like Friedman & Popescu ( 2008), we add noise features to our samples that have no effect on F 1 (x) via two noise features x 9 and x 10 . We trained two teacher neural nets, 2H-512,512 and 1H-8, as described in Section 2.2 to predict F 1 using all ten features.

Performance of teachers and students. The high-capacity 2H neural net obtained test RMSE of 0.14, while the low-capacity neural net obtained test RMSE of 0.48, more than 3x larger. For each neural net, we used our approach to generate two global additive explanation models, SAT and SAS. These explanation models are faithful: the reconstruction RMSE of SAT is 0.14 for the 1H model and 0.08 for the 2H model, while the reconstruction RMSE of SAS is 0.14 for the 1H model and 0.07 for the 2H model. This suggests that both student methods should accurately represent the teacher, and that they will probably be very similar to each other. Do SAT and SAS' feature shapes match the real behavior of the model? To further validate this we use the feature shapes to predict which samples will be inaccurately predicted by the teacher model. Specifically, we sample testing points from the space regions where the predicted feature shapes agree (or disagree) with the the feature shape ground truth (for example, for the 2H model, x 4 ≈ 0, x 7 ≈ 0, and |x 6 | ≈ 0.3 define a region where the predicted feature shapes and the ground truth feature shapes disagree) and evaluate them using the teacher model. If the learned feature shapes correctly represent the teacher model, we would expect a lower teacher error on the samples drawn from areas of agreement, and a higher teacher error on the samples drawn from areas of disagreement, compared to the RMSE of all samples. Indeed, as shown in Table 1 , points sampled on the agreement regions have lower error than points sampled from the disagreement regions. We performed a two-sample t-test to test if the errors of the samples in the (disjoint) agree and disagree groups are significantly different (p-values 4.3e-21 for 1H, 2.3e-4 for 2H). Additionally, to be robust against potential violation of the t-tests normal distribution assumption, we also performed a nonparametric Mann-Whitney-Wilcoxon rank sum test (p-values 5.4e-14 for 1H, 1.8e-6 for 2H). Hence, the difference between the errors is statistically significant, supporting our conclusion that teacher error is higher for samples where feature shapes do not match ground truth, and vice versa, i.e., feature shapes correctly represent the behavior of the models.

How do interactions between features affect feature shapes? We design an augmented version of F 1 to investigate how interactions in the teacher's predictions are expressed by feature shapes:

F 2 (x) = F 1 (x) + x 1 x 2 + |x 3 | 2|x4| + sec(x 3 x 5 x 6

). We again simulate 50,000 samples. Note that this function is much harder to learn (the 2H model obtained an RMSE of 0.21) and also harder for students that do not model interactions to mimic (SAT and SAS obtain fidelity RMSEs of 0.35). The bottom row of Figure 3 displays features with interactions (x 4 , x 2 ) and a feature without interactions (x 8 ), and compares them with the shapes from F 1 . For x 4 the part of the interactions that can be approximated additively by h i 's has been absorbed into the h i feature shapes, changing their shapes as expected. On the other hand, we were still able to recover perfectly the feature shapes of features without interactions (e.g. x 8

). An interesting case is x 2 , where, despite interacting with x 1 , its feature shape has not changed. This is less surprising if we recall that feature shapes describe the expected importance of the feature, learned in a data-driven fashion. The interaction term is x 1 x 2 , which, for x 1 ∼ U(-1, 1), has an expected value of zero, and therefore does not affect the feature shape. Similarly, the expected value of |x 3 | 2|x4| when x 3 ∼ U(-1, 1) is 1/(2|x 4 | + 1), an upward pointing cusp, which modifies the feature shape as shown in Figure 3 In some cases, such as Magic, SAT (which uses tree base learners) is more accurate, while in some others such as FICO, SAS (which uses spline base learners) has the edge. Trees are locally adaptive smoothers (Breiman et al., 1984) better able to adapt to sudden changes in input-output relationships than splines, but that also gives them more capacity to overfit. We also see this in the feature shapes, where trees tend to be more jagged than splines, particularly in regions with fewer points. Figure 4 displays selected feature shapes for Magic and Loan. The feature shapes produced by PD tend to be much too smooth, which hurts its fidelity and accuracy. Second, in all cases, trees and splines have similar feature shapes and obtain equal or better accuracy and fidelity than the other methods. This is not surprising as the other methods are either local methods adapted to the global setting (gSHAP, gGRAD), or are global explanations that are not optimized to learn the teacher's predictions (PD). For reference, gSHAP when used as a local method (i.e. individual SHAP values, not global feature shapes) achieved a lower RMSE of 0.37 compared to 1.02 on Bikeshare, and a lower RMSE of 1.99 compared to 5.10 on Loan, which is comparable to its 2H teacher's RMSE on test data (Table 2 ). Hence, methods such as gSHAP excel at local explanations and should be used for those, but, to produce global explanations, global model distillation methods optimized to learn the teacher's predictions perform better. Figure 5 presents the fidelity of SAT measured with RMSE (accuracy has similar pattern) compared to two other distilled interpretable models: decision trees (DT) and sparse L1regularized linear model (SPARSE), both trained using scikit-learn. We present results as a function of a model-specific parameter K that controls the complexity of the model. For DT, K represents depth, while for SPARSE it represents the number of features with non-zero coefficients.

4.2.2. COMPARISON WITH OTHER DISTILLED INTERPRETABLE MODELS:

For trees, true model complexity falls between K and 2 K because a binary tree of depth K has 2 K leaves (2 K rules), but the complexity is somewhat less than 2 K because there is overlap in the rules resulting from the tree structure.

SPARSE obtained by far the worst results in terms of accuracy and fidelity: even if it is interpretable, linear models do not have the fidelity necessary to accurately represent most teacher models. Note that two explanation methods that use sparse linear models (Ribeiro et al. Trees start to match the accuracy of SAT on Bikeshare at depth K = 6 (64 leaves) (Figure A7 ). However, the largest tree that is readable on letter-size paper has depth K = 4 (16 leaves). As seen in the user study in Section 4.3, depth hinders the interpretability of trees. Furthermore, they do not always perform as well as powerful additive models such as SAT. For example, on Pneumonia, a depth K = 12 tree (4, 096 leaves) achieved an accuracy of 80.9 AUC and a fidelity of 0.57 RMSE, significantly worse than SAT's 82.24 AUC and 0.35 RMSE.

Under review as a conference paper at ICLR 2019 

4.3. INTERPRETABILITY EVALUATION WITH HUMAN SUBJECTS:

We now describe the results from a user study to see if feature shapes can be understood and used by humans, comparing them to other interpretable models (DT, SPARSE, S-RULES). Table 4 presents quantitative results from the user study.

Study design. 50 subjects were recruited to participate in the study. These subjects -STEM PhD students, or college-educated individuals who had taken a machine learning course -were familiar with concepts such as if-then-else structures (for trees and rule lists), reading scatterplots (for SAT), and interpreting equations (for sparse linear models). Each subject only used one explanation model (between-subject design) to answer a set of questions (see Section C) covering common inferential and comprehension tasks on machine learning models: (1) Rank features by importance; (2) Describe relationship between a feature and the prediction; (3) Determine how the prediction changes when a feature changes value; (4) Detect an error in the data.

The study proceeded in three stages. First, we compared the two most accurate and faithful students of the Bikeshare 2H neural net: trees and SAT. We used the depth 4 tree (16 leaves), the largest tree that is readable on letter-size paper, and which does not lag too far behind the depth 6 tree in accuracy (RMSE: SAT 0.98, DT-6 1, DT-4 1.16). DT-4 used five features: Hour, Year, Temperature, Working Day, Season (Figure A4 ), hence we select the corresponding five feature shapes to display for SAT (Figure A3 ). In the first stage, 24 of 50 subjects were randomly assigned to see output from DT-4 or SAT-5. In the second stage, we experimented with smaller versions of trees and SAT using only the two most important features, Hour and Temperature. 14 of 50 subjects were randomly assigned to see output from SAT-2 or DT-2. In the last stage, the remaining 12 subjects were randomly assigned to see output from one of the two worst performing models (in terms of accuracy and fidelity): sparse linear models and subgroup-rules.

Can humans understand and use feature shapes? From the absolute magnitude of the SAT feature shapes as well as Gini feature importance metrics for the tree, we determined the ground truth feature importance ranking (in decreasing order): Hour, Temperature, Year, Season, Working Day. More SAT-5 than DT-4 subjects were able to rank the top 2 and all features correctly (75% vs. 58%, see Table 4 ). When ranking all 5 features, 0% of the DT-4 subjects were able to predict the right order, while 45% of the SAT-5 subjects correctly predicted the order of the 5 features, showing that ranking feature importance for trees is actually a very hard task. The most common mistake made by DT-4 subjects (42% of subjects) was to invert the ranking of the last two features, Season and Working Day, perhaps because Working Day's first appearance in the tree (in terms of depth) was before Season's first appearance (Figure A4 ).

When asked to describe, in free text, the relationship between the variable Hour and the label, one SAT-5 subject wrote:

There are increases in demand during two periods of commuting hours: morning commute (e.g. 7-9 am) and evening commute (e.g. 4-7 pm). Demand is flat during working hours and predicted to be especially low overnight, whereas DT-4 subjects' answers were not as expressive, e.g.:

Demand is less for early hours, then goes up until afternoon/evening, then goes down again.

75% of SAT-5 subjects detected and described the peak patterns in the mornings and late afternoons, and 42% of them explicitly mentioned commuting or rush hour in their description. On the other hand, none of the DT-4 subjects discovered this pattern on the tree: most (58%) described a concave pattern (low and increasing during the night/morning, high in the afternoon, decreasing in the evening) or a positively correlated relation (42%). Similarly, more SAT-5 subjects were able to precisely compute the change in prediction when temperature changed in value, and detect the error in the data -that spring had lower bike demand whereas winter had high bike demand (bottom right feature shape in Figure A3 ).

How do tree depth and number of feature shapes affect human performance? We also experimented with smaller models, SAT-2 and DT-2, that used only the two most important features, Hour and Temperature. As the models are simpler, some of the tasks become easier. For example, SAT-2 subjects predict the order of the top 2 features 100% of the time (vs 75% for SAT-5), and DT-2 subjects, 85% of the time (vs 58% for DT-4). The most interesting change is in the percentage of subjects able to compute the change in prediction after changing a feature: only 25% for DT-4, compared to 100% for DT-2. Reducing the complexity of the explanation made using it easier, at the price of reducing the fidelity and accuracy of the explanation.

Another important aspect is the time needed to perform the tasks: increasing the number of features from 2 to 5 increases the time needed by the subjects to finish the study by 60% for the SAT model, but increases it by 166% for the DT model, that is, interpreting a tree becomes much more costly as the tree becomes deeper (and more accurate), and, in general, subjects make more mistakes. SAT appears to scale up more gracefully.

Remaining interpretable models: subgroup-rules and sparse linear models. These explanations were the least accurate and faithful. We found that human subjects can easily read the (few) weights of SPARSE, establish feature importance, and compute prediction changes, and do so quickly -at 5.1 minutes on average, this was the fastest explanation to interpret. However, the model is highly constrained and hid interesting patterns. For example, 100% of the subjects described the relation between demand and hour as increasing, and 83% predicted the exact amount of increase, but none were able to provide insights like the ones provided by SAT-5 and DT-4 subjects.

S-RULES was the second hardest explanation to interpret based on mean time required to answer the questions: 14.9 minutes. Understanding non-disjoint rules appears to be hard: none of the subjects correctly predicted the feature importance order, even for just two features; none were able to compute exactly the change in prediction when feature value changes, and none were able to find the data error. The rules in S-RULES are not disjoint because we could not find a regression implementation of disjoint rules. However, 66% of the subjects discovered the peak during rush hour, as that appeared explicitly in some rules, e.g. "If hour=17 and workingday=yes then bike demand is 5".

To summarize, feature shapes, the interpretable representation we focus on in this paper: (1) allowed humans to perform better (than decision trees, sparse linear models, and rules) at ranking feature importance, pointing out patterns between certain feature values and predictions, and catching a data error; (2) Feature shapes were also faster to understand than big decision trees; (3) Very small decision trees and sparse linear models had the edge in calculating how predictions change when feature values change, but were much less faithful and accurate. 

4.4. VALIDATION USING CONTROLLED EXPERIMENTS ON REAL DATA:

In this section we further validate global additive explanations on real data. Although here we do not have an analytic solution for the ground-truth feature shapes, we can still design experiments where we modify data in ways that will lead to expected known changes to the ground-truth feature shapes and then verify that these changes are captured in the learned feature shapes.

Label modification. On Bikeshare, we added 1.0 to the label (the number of rented bikes) for samples where one of the features (humidity) is between 55 and 65. We then retrained a 2H neural net on the modified data, and applied our approach to learn feature shapes from the 2H net. Ideally, the feature shapes of that new neural net should be almost identical to those of the original net except in that particular range of the humidity feature, where we should see an abrupt "bump" that increases its feature shape value by one. Figure 6 (left) displays the feature shapes. Our method was able to recover the change to the label for the neural net in the new feature shape.

Data modification: expert discretization. Sometimes features are transformed before training. For example, in medical data, continuous variables such as body temperature may be discretized by domain experts into bins such as normal, mild fever, moderate fever, high fever, etc. In this experiment we test if our additive explanation models can recover these discretizations from the neural net without access to the discretized features. We train our student additive models using as input features the original un-discretized features, but using as labels the outputs of a neural net that was trained on discretized features. Our expectation is that if the student models are an accurate representation of what the neural net learned from the discretized features, they will detect the discretizations, even if they never have access to the discretized features or to the internal structure of the neural-net teacher. We study the feature shapes of two features in the Pneumonia data (Blood pO 2 and Respiration Rate) in Figure 6 , where we compare the feature shapes learned from teachers trained on the original continuous data (dotted lines) with those from teachers trained on discretized features (solid lines). Recall that in both cases the student models only saw non-discretized features to generate feature shapes. Our approach captures the expected discretization intervals (in yellow) as described in Cooper et al. (1997) .

We discuss extensions & applications of our approach in Section D in the Appendix, including visualizing a neural net as it is trained (https://youtu.be/ATNcgurNHhc).

5. CONCLUSIONS:

We presented a method for "opening up" complex models such as neural nets trained on tabular data. The method, based on distillation with high-accuracy additive models, has clear advantages over other approaches that learn additive explanations but not using distillation, and non-additive explanations using distillation. Our global additive explanations do not aim to compete with local explanations or non-additive explanations such as decision trees. Instead, we show that different interpretable representations work well for different tasks, and global additive explanations are valuable for important tasks that require quick understanding of feature-prediction relationships. Although in this paper we focus on explaining FNNs, the method will work with any classification or regression model including random forests and CNNs, but is not designed to work with raw image inputs such as pixels where providing a global explanation in terms of input pixels is not meaningful. One way to address this is to define more meaningful "features", e.g. the intermediate activations of a CNN, which are known to implicitly represent a hierarchy of concepts. Figure A2 : Feature shapes for features x 1 to x 9 of F 2 from Section 4.1. Notice how x 9 , which is a noise feature that does not affect F 2 , has been assigned an importance of approximately 0 throughout its range. The feature shape of x 10 , another noise feature, is very similar to x 9 and hence not included here. In general, the lower-capacity 1H neural nets are easier to approximate (i.e. better student-teacher fidelity), but their explanations are less accurate on independent test data. Students of simpler teachers tend to be less accurate even if they are faithful to their (simple) teachers. One exception is the FICO data, where the fidelity of the 2H explanations is better. Our interpretation is that many features in the FICO data have almost linear feature shapes (see Figure A5 for a sample of features), and the 2H model may be able to better capture fine details while being simple enough that it can still be faithfully approximated. The accuracy of the SAT and SAS for 1H and 2H neural nets are comparable, taking into account the confidence intervals.

On the Magic data, the fidelity of the gGRAD explanation to the 1H neural net (see * in Table A1 ) is markedly worse than other explanation methods. We investigate the individual gradients of the 1H neural net with respect to each feature ( ∂F (x) ∂xi in GRAD equation in Section 3). 99% of them have reasonable values (between -5.6 and 6). However, 3 are larger than 1,000 (with none between 6 and 1,000) and 13 are lower than -1,000 (with none between -1,000 and -5.6), resulting in the ensuing gGRAD explanation generating extreme predictions for several samples that are not faithful to the teacher's predictions. Because AUC is a ranking loss, accuracy (AUC) is less affected than fidelity (RMSE) by the presence of these extreme values. This shows that gGRAD explanations may be problematic when individual gradients are arbitrarily large, e.g. in overfitted neural nets.

Under review as a conference paper at ICLR 2019 C USER STUDY MATERIALS (FROM SECTION 4.3) All 50 user study subjects answered these questions:

1. What is the most important variable for predicting bike demand? 2. Rank all the variables from most important to least important for predicting bike demand.

3. Describe the relationship between the variable Hour and predicted bike demand.

4. What are variables for which the relationship between the variables and predicted bike demand is positive?

5. The Hour is 11. When Temperature increases from 15 to 20, how does predicted bike demand change?

6. There is one error in the data. Any idea where it might be? Cannot find the error is an ok answer. 

MODEL:



D APPLICATIONS AND EXTENSION:

In this section we discuss applications of our approach and extensions to include higher-order interactions.

D.1 INSIGHTS FROM GLOBAL ADDITIVE EXPLANATIONS:

Checking for monotonicity. Domains such as credit scoring have regulatory requirements that prescribe monotonic relationships between predictions and some features (Federal Reserve Governors, 2007) . For example, the 2018 FICO Explainable ML Challenge encouraged participants to impose monotonicity on 16 features (FICO, 2018) . We use feature shapes to see if the function learned by the neural net is monotone for these features. 15 of 16 features are monotonically increasing/decreasing as required. One feature, however, "Months Since Most Recent Trade Open" was expected to decrease monotonically, but actually increased monotonically. This is true not just in our explanations, but also in PD, gGRAD, and gSHAP (Figure A5 ). Note that testing for monotonicity requires global explanations or checking and aggregating many local explanations.

With the insight from the global explanations that the neural net may not be exhibiting the expected pattern for "Months Since Most Recent Trade Open", we perform a quick experiment to verify this in the neural net. We sample values of this feature across its domain, set all data samples to this value (for this feature), and obtain the neural net's predictions for these modified samples. The majority of samples (70%) had predictions that increased as this feature increased across its domain, confirming that on average, the neural net exhibits a monotonically increasing instead of decreasing pattern for this feature. Note that we could not have checked for a monotonicity pattern (which is by definition a global behavior) without checking and aggregating multiple local explanations. Visualizing neural net training: from underfit to overfit. Using additive models to peek inside a neural net creates many opportunities. For example, we can see what happens in the neural net when it is underfit or overfit; when it is trained with different losses such as squared, log, or rank loss or with different activation functions such as sigmoid or ReLUs; when regularization is performed with dropout or weight decay; when features are coded in different ways; etc. The video at https: //youtu.be/ATNcgurNHhc shows what is learned by a neural net as it trains on a medical dataset. The movie shows feature shapes for five features before, at, and after the early-stopping point as the neural net progresses from underfit to optimally fit to overfit. We had expected that the main cause of overfitting would be increased non-linearity (bumpiness) in the fitting function, but a significant factor in overfitting appears to be unwarranted growth in the confidence of the model as the logits grow more positive or negative than the early-stopping shape suggests is optimal. Functions learned by neural nets cannot always be represented with adequate fidelity by the additive function F in equation 1. We can improve F 's expressive power by adding pairwise and higher-order components h ij , h ijk , and so on to account for interactions between two or more input features. In Bikeshare, RMSE decreases from 0.98 to 0.60 when we add pairwise interactions to the student model. Figure A6 shows an interesting interaction between two features: "Time of Day", and "Working Day". On working days, the highest bike rental demand occurs at 7-9am and 5-7pm, but on weekends there is very low demand at 7-9am (presumably because people are still sleeping) and at 5-7pm, and demand peaks during midday from 10am-4pm. These two features also form a three-way interaction with temperature. Whenever the teacher neural net learned these (and other) interactions, a global explanation method must also incorporate interactions if it is to provide high-fidelity explanations of the teacher model. Our approach is able to do so by adding higher-order components h ij , h ijk , and so on to the global additive explanation F .

:

E.g. Hooker (2007)'s decomposition procedure requires computing X T X, where X is an n by p feature matrix and solving a system of equations involving

p+1 samples.

Figures and Tables:

Figure 2: From SHAP to gSHAP. Blue points are individual SHAP values; red line is gSHAP feature shape.

Figure 3: Three of ten feature shapes learned for F 1 (top row) and F 2 (bottom row) by small (1H-8) and large (2H-512,512) nets. To view shapes learned for all features, see the Appendix.

Figure 4: Example feature shapes from Magic (left), and Loan (right). SAT and SAS tend to agree. Best seen on a screen.

Figure 5: Fidelity (RMSE) of SAT compared to other interpretable models on Bikeshare (left) and Pneumonia (right), as a function of model-specific parameter K.

, 2016) and rules (Ribeiro et al., 2018) use them as local (not global) explanations, and only for classification (not regression).

Figure 6: Feature shapes from controlled experiments on real data. Left: Label modification experiment. Center and right: Data modification experiment. See details in Section 4.4.

The additive model below was trained to predict bike demand. Each plot describes a variable on the x-axis and predicted bike demand on the y-axis. Hour denotes hour-of-day, i.e. 0, 1, 2, … 23. Temperature is in Celsius. Year has 2 possible values: 2011, 2012. Season has 4 possible values: Spring, Summer, Fall, Winter.

Figure A3: Model output shown to SAT-5 subjects.

Figure A4: Model output shown to DT-4 subjects. This visual presentation was simplified from the visual presentation in Figure A7, removing the color and number of samples in each node, to improve readability for the user study.

Figure A5: 3 of 16 features with expected monotonically increasing/decreasing patterns in the FICO data. "Months Since Most Recent Trade Open", the leftmost figure, was expected to decrease monotonically, but actually increased monotonically according to all explanations. The two figures on the right are two related features, "Months Since Oldest Trade Open" and "Number of Trades Open in Last 12 Months", both of which exhibit the expected monotonically decreasing/increasing patterns.

Figure A6: An important pairwise interaction in Bikeshare.

In general, the shapes of the 2H model are very faithful to the ground-truth shapes, but sometimes fall short when there are sharp changes in the ground-truth, highlighting the limitations of a 2-hidden-layer neural net (which achieves 0.14 test RMSE, as noted before). On the other hand, both SAT and SAS' feature shapes for the 1H neural net show a less accurate teacher model that captures the gist of the ground-truth function but not its details, which is consistent with the original teacher RMSE of 0.48. This shows that our methods fit what the teacher model has learned, and not the original data, and that when the teacher model is accurate the learned shapes match the ground-truth shapes.

Table content: <table><tr><td>Model</td><td>All</td><td>Agree</td><td>Disagree</td></tr><tr><td>1H-8</td><td>0.483</td><td>0.407</td><td>0.489</td></tr><tr><td>2H-512,512</td><td>0.142</td><td>0.141</td><td>0.180</td></tr></table>

(bottom left figure).4.2 QUANTITATIVE COMPARISON OF EXPLANATION METHODS ON REAL DATA Performance of neural net teachers. For RMSE, lower is better. For AUC, higher is better.

Table content: <table><tr><td>Performance</td></tr></table>

presents the fidelity and accuracy results for SAT and SAS compared to other additive explanations. SAT and SAS yield similar results in all cases, both in terms of accuracy and fidelity.

Table content: <table/>

Accuracy and fidelity of global additive explanations for 2H neural nets. Accuracy is in terms of RMSE for regression tasks and AUROC for classification tasks; fidelity is always RMSE between the student's predictions and the teacher's scores or logits (equation 2). Results for 1H-8 neural nets in Appendix.

Table content: <table><tr><td>Accuracy</td><td>Bikeshare</td><td>Loan score</td><td>Magic</td><td>Pneumonia</td><td>FICO</td></tr><tr><td>Global Explanation</td><td>RMSE</td><td>RMSE</td><td>AUC</td><td>AUC</td><td>AUC</td></tr><tr><td>SAT</td><td>0.98 ± 0.00</td><td>2.35 ± 0.01</td><td>90.75 ± 0.06</td><td>82.24 ± 0.05</td><td>79.42 ± 0.04</td></tr><tr><td>SAS</td><td>0.98 ± 0.00</td><td>2.34 ± 0.00</td><td>90.58 ± 0.02</td><td>82.12 ± 0.04</td><td>79.51 ± 0.02</td></tr><tr><td>gGRAD</td><td>1.25 ± 0.00</td><td>6.04 ± 0.01</td><td>80.95 ± 0.13</td><td>81.88 ± 0.05</td><td>79.28 ± 0.02</td></tr><tr><td>gSHAP</td><td>1.02 ± 0.00</td><td>5.10 ± 0.01</td><td>88.98 ± 0.05</td><td>82.31 ± 0.03</td><td>79.36 ± 0.01</td></tr><tr><td>PD</td><td>1.00 ± 0.00</td><td>4.31 ± 0.00</td><td>82.78 ± 0.00</td><td>82.15 ± 0.00</td><td>79.47 ± 0.00</td></tr><tr><td>Fidelity</td><td>Bikeshare</td><td>Loan score</td><td>Magic</td><td>Pneumonia</td><td>FICO</td></tr><tr><td>Global Explanation</td><td>RMSE</td><td>RMSE</td><td>RMSE</td><td>RMSE</td><td>RMSE</td></tr><tr><td>SAT</td><td>0.92 ± 0.00</td><td>1.74 ± 0.01</td><td>1.78 ± 0.00</td><td>0.35 ± 0.00</td><td>0.15 ± 0.00</td></tr><tr><td>SAS</td><td>0.92 ± 0.00</td><td>1.71 ± 0.00</td><td>1.75 ± 0.00</td><td>0.35 ± 0.00</td><td>0.14 ± 0.00</td></tr><tr><td>gGRAD</td><td>1.20 ± 0.00</td><td>5.93 ± 0.01</td><td>2.93 ± 0.01</td><td>0.43 ± 0.00</td><td>0.16 ± 0.00</td></tr><tr><td>gSHAP</td><td>0.96 ± 0.00</td><td>4.83 ± 0.00</td><td>2.15 ± 0.00</td><td>0.46 ± 0.00</td><td>0.16 ± 0.00</td></tr><tr><td>PD</td><td>0.94 ± 0.00</td><td>3.85 ± 0.00</td><td>3.17 ± 0.00</td><td>0.47 ± 0.00</td><td>0.16 ± 0.00</td></tr></table>

Quantitative results from user study. Since SAT-2, DT-2, and SPARSE only had two features, the task to rank five features does not apply. Since the data error only appeared in the output of SAT-5, DT-4, and S-RULES, the other subjects could not have caught the error.

Table content: <table><tr><td/><td colspan="2">First stage (n=24)</td><td colspan="2">Second stage (n=14)</td><td colspan="2">Third stage (n=12)</td></tr><tr><td>Task</td><td>SAT-5</td><td>DT-4</td><td>SAT-2</td><td>DT-2</td><td>SPARSE</td><td>S-RULES</td></tr><tr><td>Rank top 2 features</td><td>75%</td><td>58%</td><td>100%</td><td>85.7%</td><td>83.3%</td><td>0%</td></tr><tr><td>Rank all (5) features</td><td>45%</td><td>0%</td><td>N/A</td><td>N/A</td><td>N/A</td><td>0%</td></tr><tr><td>Described increased demand during rush hour</td><td>42%</td><td>0%</td><td>29%</td><td>0%</td><td>0%</td><td>33%</td></tr><tr><td>Described increased demand during mornings and afternoons</td><td>33%</td><td>0%</td><td>29%</td><td>0%</td><td>0%</td><td>33%</td></tr><tr><td>Compute change in prediction when feature changes</td><td>33%</td><td>25%</td><td>14%</td><td>100%</td><td>83%</td><td>0%</td></tr><tr><td>Caught data error</td><td>33%</td><td>8%</td><td>N/A</td><td>N/A</td><td>N/A</td><td>0%</td></tr><tr><td>Time taken (minutes)</td><td>11.7 ± 5.8</td><td>17.5 ± 14.8</td><td>7.2 ± 3.2</td><td>6.2 ± 2.2</td><td>5.2 ± 3.1</td><td>14.9 ± 8.4</td></tr></table>

B QUANTITATIVE COMPARISON OF EXPLANATION METHODS ON REAL DATA: ALL NEURAL NET TEACHERS (CONTINUED FROM SECTION 4.2) Accuracy and fidelity of global explanation models across 1H and 2H teacher neural nets and datasets. Table 3 is a subset of this table with only 2H neural nets.

Table content: <table><tr><td>Accuracy</td><td/><td>Bikeshare</td><td>Loan score</td><td>Magic</td><td>Pneumonia</td><td>FICO</td></tr><tr><td>Teacher</td><td>Global Explanation</td><td>RMSE</td><td>RMSE</td><td>AUC</td><td>AUC</td><td>AUC</td></tr><tr><td/><td>SAT</td><td>1.00 ± 0.00</td><td>2.82 ± 0.00</td><td>90.44 ± 0.05</td><td>82.01 ± 0.05</td><td>79.43 ± 0.02</td></tr><tr><td/><td>SAS</td><td>1.00 ± 0.00</td><td>2.82 ± 0.00</td><td>90.43 ± 0.03</td><td>81.91 ± 0.06</td><td>79.56 ± 0.02</td></tr><tr><td>1H-8</td><td>gGRAD</td><td>1.08 ± 0.00</td><td>2.84 ± 0.00</td><td>84.52 ± 0.67</td><td>81.63 ± 0.06</td><td>79.34 ± 0.05</td></tr><tr><td/><td>gSHAP</td><td>1.04 ± 0.00</td><td>2.87 ± 0.00</td><td>89.94 ± 0.03</td><td>82.02 ± 0.02</td><td>79.49 ± 0.02</td></tr><tr><td/><td>PD</td><td>1.00 ± 0.00</td><td>3.00 ± 0.00</td><td>85.11 ± 0.00</td><td>82.03 ± 0.00</td><td>79.46 ± 0.00</td></tr><tr><td/><td>SAT</td><td>0.98 ± 0.00</td><td>2.35 ± 0.01</td><td>90.75 ± 0.06</td><td>82.24 ± 0.05</td><td>79.42 ± 0.04</td></tr><tr><td/><td>SAS</td><td>0.98 ± 0.00</td><td>2.34 ± 0.00</td><td>90.58 ± 0.02</td><td>82.12 ± 0.04</td><td>79.51 ± 0.02</td></tr><tr><td>2H-512,512</td><td>gGRAD</td><td>1.25 ± 0.00</td><td>6.04 ± 0.01</td><td>80.95 ± 0.13</td><td>81.88 ± 0.05</td><td>79.28 ± 0.02</td></tr><tr><td/><td>gSHAP</td><td>1.02 ± 0.00</td><td>5.10 ± 0.00</td><td>88.98 ± 0.05</td><td>82.31 ± 0.03</td><td>79.36 ± 0.01</td></tr><tr><td/><td>PD</td><td>1.00 ± 0.00</td><td>4.31 ± 0.00</td><td>82.78 ± 0.00</td><td>82.15 ± 0.00</td><td>79.47 ± 0.00</td></tr><tr><td>Fidelity</td><td/><td>Bikeshare</td><td>Loan score</td><td>Magic</td><td>Pneumonia</td><td>FICO</td></tr><tr><td>Teacher</td><td>Global Explanation</td><td>RMSE</td><td>RMSE</td><td>RMSE</td><td>RMSE</td><td>RMSE</td></tr><tr><td/><td>SAT</td><td>0.64 ± 0.00</td><td>1.15 ± 0.00</td><td>1.12 ± 0.00</td><td>0.30 ± 0.00</td><td>0.21 ± 0.00</td></tr><tr><td/><td>SAS</td><td>0.64 ± 0.00</td><td>1.14 ± 0.00</td><td>1.11 ± 0.00</td><td>0.30 ± 0.00</td><td>0.21 ± 0.00</td></tr><tr><td>1H-8</td><td>gGRAD</td><td>0.71 ± 0.00</td><td>1.54 ± 0.00</td><td>35.40 ± 4.47*</td><td>0.36 ± 0.00</td><td>0.24 ± 0.00</td></tr><tr><td/><td>gSHAP</td><td>0.68 ± 0.00</td><td>1.28 ± 0.00</td><td>1.29 ± 0.00</td><td>0.38 ± 0.00</td><td>0.22 ± 0.00</td></tr><tr><td/><td>PD</td><td>0.65 ± 0.00</td><td>1.37 ± 0.00</td><td>1.94 ± 0.00</td><td>0.38 ± 0.00</td><td>0.25 ± 0.00</td></tr><tr><td/><td>SAT</td><td>0.92 ± 0.00</td><td>1.74 ± 0.01</td><td>1.78 ± 0.00</td><td>0.35 ± 0.00</td><td>0.15 ± 0.00</td></tr><tr><td/><td>SAS</td><td>0.92 ± 0.00</td><td>1.71 ± 0.00</td><td>1.75 ± 0.00</td><td>0.35 ± 0.00</td><td>0.14 ± 0.00</td></tr><tr><td>2H-512,512</td><td>gGRAD</td><td>1.20 ± 0.00</td><td>5.93 ± 0.01</td><td>2.93 ± 0.01</td><td>0.43 ± 0.00</td><td>0.16 ± 0.00</td></tr><tr><td/><td>gSHAP</td><td>0.96 ± 0.00</td><td>4.83 ± 0.01</td><td>2.15 ± 0.00</td><td>0.46 ± 0.00</td><td>0.16 ± 0.00</td></tr><tr><td/><td>PD</td><td>0.94 ± 0.00</td><td>3.85 ± 0.00</td><td>3.17 ± 0.00</td><td>0.47 ± 0.00</td><td>0.16 ± 0.00</td></tr></table>

Peer review (iclr2019):
 This paper is of high quality and clarity. I think it*s originality is at least decent. Whether it is significant or not depends on how significant one thinks fully connected neural networks are as these are the models for which this explanation model makes sense. Good things: - It is a very elegant method. It is also very simple (in a good way). - The paper is really well written. - The experiments are carefully conducted and are indeed showing what the authors describe. - I think the method is potentially of practical use. Problems: - I think qualifying this paper as a paper on representation learning is a small stretch. It would be perhaps more suitable to submit it to ICML or NIPS. I think it is close enough though. - The font is too small in many figures. It is impossible to read it. - I am not sure whether model compression is actually necessary here. How good is the additive model if it is trained as a standalone model straight from the training data in comparison to the neural networks and to the additive model when trained with model compression? If the neural network and the additive model were similar in performance when trained from scratch, I would not see the point in explaining the neural network. - Only makes sense to apply this to fully connected networks.