Neural Additive Models: Interpretable Machine Learning with Neural Nets

1 Introduction . While deep neural networks have achieved impressive results on tasks such as computer vision [ 17 ] and language modeling [ 31 ] , it is notoriously difficult to understand how such networks make predictions , and they are often considered as black-box models . This hinders their applicability to highstakes domains such as healthcare , finance and criminal justice . Various efforts have been made to demystify the predictions of neural networks ( NNs ) . For example , one family of methods , represented by LIME [ 33 ] , attempt to explain individual predictions of a neural network by approximating it locally with interpretable models such as linear models and shallow trees2 . However , these approaches often fail to provide a global view of the model and their explanations often are not faithful to what the original model computes or do not provide enough detail to understand the model ’ s behavior [ 35 ] . ∗Correspondence to : Rishabh Agarwal < rishabhagarwal @ google.com > , Levi Melnick < lemeln @ microsoft.com > , and Rich Caruana < rcaruana @ microsoft.com > . 2Linear models , shallow decision trees and GAMs are interpretable only if the features they are trained on are interpretable . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we make restrictions on the structure of neural networks , which yields a family of glass-box models called Neural Additive Models ( NAMs ) , that are inherently interpretable while suffering little loss in prediction accuracy when applied to tabular data . Methodologically , NAMs belong to a model family called Generalized Additive Models ( GAMs ) [ 14 ] . GAMs have the form : g ( E [ y ] ) = β + f1 ( x1 ) + f2 ( x2 ) + · · ·+ fK ( xK ) ( 1 ) where x = ( x1 , x2 , . . . , xK ) is the input with K features , y is the target variable , g ( . ) is the link function ( e.g. , logistic function ) and each fi is a univariate shape function with E [ fi ] = 0 . Generalized linear models , such as logistic regression , are a special form of GAMs where each fi is restricted to be linear . NAMs learn a linear combination of networks that each attend to a single input feature : each fi in ( 1 ) is parametrized by a neural network . These networks are trained jointly using backpropagation and can learn arbitrarily complex shape functions . Interpreting NAMs is easy as the impact of a feature on the prediction does not rely on the other features and can be understood by visualizing its corresponding shape function ( e.g. , plotting fi ( xi ) vs. xi ) . While interpretability of NAMs may seem heuristic , the graphs learned by NAMs are an exact description of how NAMs compute a prediction . Traditionally , GAMs were fitted via iterative backfitting using smooth low-order splines , which reduce overfitting and can be fit analytically . More recently , GAMs [ 5 ] were fitted with boosted decision trees to improve accuracy and to allow GAMs to learn jumps in the feature shaping functions to better match patterns seen in real data that smooth splines could not easily capture . This paper examines using DNNs to fit generalized additive models ( NAMs ) which provides the following advantages : • NAMs introduce an expressive yet intelligible class of models to the deep learning ( DL ) community , a much larger community than the one using tree-based GAMs . • NAMs are likely to be combined with other DL methods in ways we don ’ t foresee . This is important because a key drawback of deep learning is interpretability . For example , NAMs have already been employed for survival analysis [ 46 ] . • NAMs , due to the flexibility of NNs , can be easily extended to various settings problematic for boosted decision trees . For example , extending boosted tree GAMs to multitask , multiclass or multi-label learning requires significant changes to how trees are trained , but is easily accomplished with NAMs without requiring changes to how neural nets are trained due to their composability ( Section 4.2 ) . Futhermore , the differentiability of NAMs allows them to train more complex interpretable models for COVID-19 ( Section 4.1 ) . • Graphs learned by NAMs are not just an explanation but an exact description of how NAMs compute a prediction . As such , a decision-maker can easily interpret NAMs and understand exactly how they make decisions . This would help harness the expressivity of neural nets on high-stakes domains with intelligibility requirements , e.g. , in-hospital mortality prediction [ 22 ] . • NAMs are more scalable as inference and training can be done on GPUs/TPUs or other specialized hardware using the same toolkits developed for deep learning over the past decade – GAMs currently can not . • Accurate GAMs [ 5 ] currently require millions of decision trees to fit each shape function while NAMs only use a small ensemble ( 2 - 100 ) of neural nets . Thus , NAMs are relatively much easier to extend compared to GAMs . 2 Neural Additive Models . Modeling jagged shape functions is required to learn accurate additive models as there are often sharp jumps in real-world datasets , e.g. , see Figure 4 for jumps in graphs for PFRatio and Bilirubin which correspond to real patterns in the MIMIC-II dataset [ 38 ] ( Section A.1 ) . Similarly , Caruana et al . [ 5 ] observe that GAMs fit using splines tend to over regularize and miss genuine details in real data , yielding less accuracy than tree-based GAMs . Therefore , we require that neural networks ( NNs ) are able to learn highly non-linear shape functions , to fit these patterns . Although NNs can approximate arbitrarily complex functions [ 18 ] , we find that standard NNs fail to model highly jumpy 1D functions , and demonstrate this failure empirically using a toy dataset . The toy dataset is constructed as follows : For the input x , we sample 100 evenly spaced points in [ -1 , 1 ] . For each x , we sample p uniformly random in [ 0.1 , 0.9 ) and generate 100 labels from a Bernoulli random variable which takes the value 1 with probability p. This creates a binary classification dataset of ( x , y ) tuples with 10,000 points . Figure 2 shows the log-odds of the empirical probability p ( i.e. , log p1−p ) of classifying the label of x as 1 for each input x . This dataset tests the NN ’ s ability to “ overfit ” the data , rather than its ability to generalize . Over-parameterized NNs with ReLUs [ 25 ] and standard initializations such as Kaiming initialization [ 16 ] and Xavier initialization [ 10 ] struggle to overfit this dataset when trained using mini-batch gradient descent , despite the NN architecture being expressive enough3 ( see Figures 2 ( a ) and A.3 ) . This difficulty of learning large local fluctuations with ReLU networks without affecting their global behavior when fitting jagged functions might be due to their bias towards smoothness [ 2 , 32 ] . We propose exp-centered ( ExU ) hidden units to overcome this neural net failure : we simply learn the weights in the logarithmic space with inputs shifted by a bias . Specifically , for a scalar input x , each hidden unit using an activation function f computes h ( x ) given by h ( x ) = f ( ew ∗ ( x− b ) ) ( 2 ) where w and b are the weight and bias parameters . The intuition behind ExU units is as follows : For modeling jagged functions , a hidden unit should be able to change its output significantly , with a tiny change in input . This requires the unit to have extremely large weight values depending on the sharpness of the jump . The ExU unit computes a linear function of input where the slope can be very steep with small weights , making it easier to modify the output easily during training . ExU units do not improve the expressivity of neural nets , however they do improve their learnability for fitting jumpy functions . While we use ExU units to train accurate NAMs , they are more generally applicable for approximating jumpy functions with neural nets . We noticed that ExU units with standard weight initialization also struggle to learn jagged curves ; instead initializing the weights using a normal distribution N ( x , 0.5 ) with x ∈ [ 3 , 4 ] works well in practice . This initialization simply ensures that the initial network starts with a jagged ( random ) 3This problem doesn ’ t occur with full-batch gradient descent . function which we empirically find to be crucial for fitting any jumpy function . Furthermore , we use ReLU activations capped at n ( ReLU-n ) [ 21 ] to ensure that each ExU unit is active in a small input range , making it easier to model sharp jumps in a function without significantly affecting the global behavior . ExU-units can be combined with any activation function ( i.e. , any f can be used in ( 2 ) ) , but ReLU-n performs well in practice . Figure 2 ( b ) shows that NNs with ExU units are able to fit the toy dataset significantly better than standard NNs . Finally , realistic shape functions typically tend to be smooth with large jumps at only a few points ( Figure 4 ) . To avoid overfitting with ExUs , strong regularization is crucial which can learn such realistic functions ( e.g. , Figure 3 ) . With ReLUs , we can typically fit smooth functions but they might miss some of these jumps . To avoid overfitting when fitting NAMs with ExUs , we employ various regularization methods including dropout , weight decay , output penalty , and feature dropout ( see Section A.3 for an overview ) . 2.1 Intelligibility and Modularity of NAMs . The intelligibility of NAMs results in part from the ease with which they can be visualized . Because each feature is handled independently by a learned shape function parameterized by a neural net , one can get a full view of the model by simply graphing the individual shape functions . For data with a small number of inputs , it is possible to have an accessible explanation of the model ’ s behavior visualized fully on a single page . Please note these shape function plots are not just an explanation but an exact description of how NAMs compute a prediction . A decision-maker can easily interpret such models and understand exactly how they make decisions , for example , we validated the behavior of NAMs on the MIMIC-II dataset [ 38 ] with a doctor ( Appendix A.1 ) . We set the average score for each graph ( i.e. , each feature ) averaged across the entire training dataset to zero by subtracting the mean score . To make individual shape functions identifiable and modular , a single bias term is then added to the model so that the average predictions across all data points matches the observed baseline . This makes interpreting the contribution of each term easier : e.g. , on binary classification tasks , negative scores decrease probability , and positive scores increase probability compared to the baseline probability of observing that class . This property also allows each graph to be removed from the NAM ( zeroed out ) without introducing bias to the predictions . Visualization . We plot each shape function and the corresponding data density on the same graph . Specifically , we plot each learned shape function fk ( xk ) vs. xk for an ensemble of NAMs using a semi transparent blue line , which allows us to see when the models in the ensemble learned the same shape function and when they diverged . This provides a sense of the confidence of the learned shape functions . We also plot on the same graphs the normalized data density , in the form of pink bars . The darker the shade of pink , the more data there is in that region . This allows us to know when the model had adequate training data to learn appropriate shape functions . M ax D el in qu en cy E ve r M on th s Si nc e M os t R ec en t D el in qu en cy M on th s Si nc e M os t R ec en t T ra de # In qu iri es in L as t 6 M on th s # To ta l T ra de s In qu iry N et F ra ct io n In st al lm en t B ur de n M on th s Si nc e M os t R ec en t In qu iry e xc lu di ng 7 d ay s M on th s Si nc e O ld es t T ra de O pe n % In st al lm en t T ra de s N um be r I ns ta llm en t T ra de s w ith B al an ce % T ra de s w ith B al an ce # In qu iri es in L as t 6 M on th s e xc lu di ng 7 d ay s # Tr ad es O pe n in L as t 1 2 M on th s # Tr ad es 9 0+ E ve r # R ev ol vi ng T ra de s w ith B al an ce # Tr ad es 6 0+ E ve r # Sa tis fa ct or y Tr ad es # xB an k/ N at l T ra de s w ith h ig h ut iliz at io n ra tio % T ra de s N ev er D el in qu en t D el in qu en t M ax D el q/ Pu bl ic R ec or ds L as t Y ea r N et F ra ct io n R ev ol vi ng B ur de n Av er ag e M on th s in F ile −8 −6 −4 −2 0 2 4 FI C O S co re C on tri bu tio n Low FICO Score Applicant High FICO Score Applicant Figure 5 : Understanding individual predictions for credit scores . Feature contribution using the learned NAMs for predicting scores of two applicants in the FICO dataset [ 9 ] . For a given input , each feature net in the NAM acts as a lookup table and returns a contribution term . These contributions are combined in a modular way : they are added up , and passed through a link function for prediction . the longer a person ’ s credit history , the better it is for their credit score The high scoring applicant has a long credit history ( Average Months on File ) , which contributes to their credit score better . On the contrary , the low scoring applicant used their credit quite frequently ( Total Number of Trades ) and has a large burden ( Net Fraction Installment Burden ) , thus resulting in a low score . −124 −122 −120 −118 −116 Longitude −10 −5 0 5 10 15 H ou se P ric e C on tri bu tio n 2 4 6 8 10 12 14 Median Income −10 −5 0 5 10 15 H ou se P ric e C on tri bu tio n Figure 6 : California Housing . Graphs learned by NAMs trained to predict house prices [ 27 ] for two most important features . As expected , The house prices increase linearly with median income in high data density regions . Furthermore , the graph for longitude shows sharp jumps in price prediction around the location of San Francisco and Los Angeles .

Summary: In this paper, the authors introduce a new generalized additive model, called neural additive models, which use neural networks to learn a non-linear transformation of each input feature, independently. Along the way, they introduce a new activation function, called ExU, which enables a highly sensitive weights that can fit very large, seemingly noisy, changes of a given function. They demonstrate that NAM performs as well as other GAMs, such as EBMs, as well as black box models, such as DNNs and XGBoost. The major  benefit of NAM is that, similar to EBMs, they are intrinsically interpretable simply by visualizing the learned non-linear function for each input feature.  They compare the performance of these methods across several tasks. The paper is written very clearly with easy to follow examples.