Neural Additive Models: Interpretable Machine Learning with Neural Nets
Authors: Rishabh Agarwal
Levi Melnick
Nicholas Frosst
Xuezhou Zhang
Ben Lengerich
Rich Caruana
Geoffrey Hinton
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Additive Models
Interpretability
Multitask learning
Explainable AI

Abstract: Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. However, their accuracy comes at the cost of intelligibility: it is usually unclear how they make their decisions. This hinders their applicability to high stakes decision-making domains such as healthcare. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature. These networks are trained jointly and can learn arbitrarily complex relationships between their input feature and the output. Our experiments on regression and classification datasets show that NAMs are more accurate than widely used intelligible models such as logistic regression and shallow decision trees. They perform similarly to existing state-of-the-art generalized additive models in accuracy, but are more flexible because they are based on neural nets instead of boosted trees. To demonstrate this, we show how NAMs can be used for multitask learning on synthetic data and on the COMPAS recidivism data due to their composability, and demonstrate that the differentiability of NAMs allows them to train more complex interpretable models for COVID-19. 

Neural Additive Models: Interpretable Machine Learning with Neural Nets
Rishabh Agarwal
Levi Melnick
Google Research, Brain Team
Microsoft Research
Nicholas Frosst Cohere
Xuezhou Zhang
Ben Lengerich MIT
University of Wisconsin-Maddise
Rich Caruana
Geoffrey E. Hinton Google Research, Brain Team
Microsoft Research
Abstract
Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance wide variety of tasks. However, their accuracy comes at the cost of intelligibility: is usually unclear how they make their decisions. This hinders their applicability high stakes decision-makin domains such as healthcare. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn linear combination of neural networks that each attend to a single input feature. These networks are trained jointly and can learn arbitrarily complex relationships between their input feature and the output. Our experiments on regression and classification datasets show that NAMs are more accurate than widely used intelligible models such as logistic regression and shallow decision trees. They perform similarly to existing state-of-the-art generalized additive models in accuracy, but are more flexible because they are based neural nets instead of boosted trees. To demonstrate this, we show how NAMs can be used for multitask learning synthetic data and on the COMPAS recidivism data due to their composability, and demonstrate that the diffcrentiability of NAMs allows them to train more complex interpretable models for COVID-19. Source code is available at heural-addis sedddiiivemmnn models.. giithb..itoo...
Introduction
While deep neural networks have achieved impressive results on tasks such as computer vision [17] and language modeling [31], it is notoriously difficult to understand how such networks make predictions, and they are often considered as black-box models. This hinders their applicability high- stakes domains such as healthcare, finance and criminal justice. Various efforts have been made to demystify the predictions neural networks (NNs). For example, one family of methods, represented by LIME [33], attempt to expluin individual predictions of neural network by approximating it locally with interpretable models such as linear models and shallow trees?. However, these approaches often fail to provide global view of the model and their explanations often not faithful to what the original model computes or do not provide enough detail to understand the model's behavior [35].
"oorrespnddence
Rishabh Agarwal <rishabbaggaraa
comp,
Levi Melnick
<1mmeln0m www.oft..rrooo.ooommoommmmooo... comm) and Rich Caruana <rcaruanatmi crosoft. com-.
"Linaar models, shallow decision trees and GAMs are interpretable only if the features they trained on interpretable.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we make restrictions on the structure of neural networks, which yields family of glass-box models called Neural Additive Models (NAMs), that are inherently interpretable while suffering little loss in prediction accuracy when applied to tabular data. Methodologically, NAMs belong to a model family called Generalized Additive Models (GAMs) [14]. GAMs have the form:
LE (vl) 3 + f1(21) + (2(22) + + fx(*x)
(I)
where = (I1, 22; r is the input with K features, y is the target variable, g(.) is the link function (e.g., logistic function) and each ft a univariate shape function with [[ = O. Generalized linear models, such as logistic regression, are a special form of GAMs where each fi is restricted to be linear.
falmra
NAMs learn linear combination of net- works that each attend to a single input fea- ture: each Si in (1) is parametrized by neural network. These networks are trained jointly using backpropagation and can learn arbitrarily complex shape functions. Inter- preting NAMs is easy the impact of fea- ture on the prediction does not rely on the other features and can be understood by visu- alizing its corresponding shape function (e.g., plotting (1(x)) vs. wt)). While interpretabil- ity of NAMs may seem heuristic, the graphs learned by NAMs are exact description of how NAMs compute a prediction.
Σ
Figure 1: NAM architecture binary classification. Each input variable is handled by a different neural network. This results in easily interpretable yet highly accurate models.
Traditionally, GAMs were fitted via iterative backfitting using smooth low-order splines, which reduce overfitting and can be fit analytically. More recently, GAMs [5] were fitted with boosted decision trees improve accuracy and to allow GAMs to learn jumps in the feature shaping functions to better match patterns seen in real data that smooth splines could not easily capture. This paper examines using DNNs fit generalized additive models (NAMs) which provides the following advantages:
NAMs introduce an expressive yet intelligible class of models to the deep learning (DL) commu- nity, much larger community than the one using tree-based GAMs.
NAMs are likely to be combined with other DL methods in ways we don't foresee, This is important because a key drawback of deep learning is interpretability. For example, NAMs have already been employed for survival analysis [46].
NAMs, due to the flexibility of NNs, can be easily extended various settings problematic for boosted decision trees. For example, extending boosted tree GAMs to multitask, multi- class or multi-label learning requires significant changes how trees are trained, but is easily accomplished with NAMs without requiring changes to how neural nets are trained due to their composability (Section 4.2). Futhermore, the differentiability of NAMs allows them train more complex interpretable models for COVID-19 19 (Section 4.1).
Graphs learned by NAMs are not just an explanation but an exact description of how NAMs compute prediction. As such, a decision- maker can easily interpret NAMs and understand exactly how they make decisions. This would help harness the expressivity neural nets on high-stakes domains with intelligibility requirements, e.g., in-hospital mortality prediction [22].
NAMs are more scalable as inference and training can be done on GPUs/TPUs or other specialized hardware using the same toolkits developed for deep learning over the past decade GAMs currently cannot.
Accurate GAMs [5] currently require millions of decision trees to fit each shape function while NAMs only use small ensemble (2 100) of neural nets. Thus, NAMs are relatively much easier to extend compared to GAMs.

---

True
Prodicted
Log Odd6
Log Oods
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
-0.75
-0.60
-0.25
0.00
0.25
0.50
0.75
1.00
Input
Inpul
(a)
(h)
Figure 2: Accurately Fitting the Toy Dataset: Training predictions Iearned by single hidden layer neural
network with 1024 (a) standard ReLU, and (b) ReLU-n with ExU hidden units trained for 10,000 epochs on the binary classification dataset described in Section 2. We can see that the ReLU network has learned a fairly smooth function while the ExU network has learned a very jumpy function. We find that a DNN with three hidden layers also learned smooth functions (see Figure A.3).
Neural Additive Models
Modeling jagged shape functions is required to learn accurate additive models as there are often sharp jumps in real-world datasets, e.g., see Figure 4 for jumps in graphs for PFRatio and Bilirubin which correspond to real patterns in the MIMIC-II dataset [38] (Section A.1). Similarly, Caruana et al. [5] observe that GAMs fit using splines tend to over regularize and miss genuine details in real data, yielding less accuracy than tree-based GAMs. Therefore, we require that neural networks (NNs) are able to learn highly non-line non-linear shape functions, to fit these patterns.
Although NNs can approximate arbitrarily complex functions [18), we find that standard NNs fail to model highly jumpy 1D functions, and demonstrate this failure empirically using a toy dataset. The toy dataset is constructed as follows: For the input x. we sample 100 evenly spaced points in [-1, 1]. For each x, we sample p uniformly random in [0.1, 0.9) and generate 100 labels from a Bernoulli random variable which takes the value I with probability p. This creates binary classification dataset of (x, 3) tuples with 10,000 points. Figure 2 shows the log-odds of the empirical probability (i.e. for each input x. This dataset tests the NN's ability to
log I-p ) of classifying the label of "overfit" the data, rather than its ability to generalize.
Over-parameterized NNs with ReLUs [25] and standard initializations such as Kaiming initializa- tion [16] and Xavier initialization [10] struggle to overfit this dataset when trained using mini- batch gradient descent, despite the NN architecture being expressive enough (see Figures 2(a) and A.3). This difficulty of learning large local fluctuations with ReLU networks without affecting their global behavior when fitting jagged functions might be due to their bias towards smoothness [2, 32).
We propose exp-centered (ExU) hidden units overcome this neural net failure: simply learn the weights in the logarithmic space with inputs shifted by a bias. Specifically, for a scalar input æ, each hidden unit using an activation function f computes h(x:) given by
h(x)= J (e (x (0-b)) b))
where and b are the weight and bias parameters. The intuition behind ExU units is as follows: For modeling jagged functions, hidden unit should be able change its output significantly, with a tiny change in input. This requires the unit to have extremely large weight values depending on the sharpness of the jump. The ExU unit computes linear function of input where the slope can be very steep with small weights, making it easier to modify the output easily during training. ExU units do not improve the expressivity of neural nets, however they do improve their learnability for fitting jumpy functions. While we use ExU units to train accurate NAMs, they are more generally applicable for approximating jumpy functions with neural nets.
We noticed that ExU units with standard weight initialization also struggle to learn jagged curves; instead initializing the weights using a normal distribution (z 0.5) with [3, works well in practice. This initialization simply ensures that the initial network starts with a jagged (random)
'Tiis problem doesn't 't occur with full- batch gradient descent.
3

---

1.
Sub-network Prediction
1.0
Average Prediction
0.5
6000
(a) Graphs learned by NAMs with ExU units
-1.0
20
.0 60
100
4
Input (XG)
Figure 3: Regularizing ExU networks. Output of ExU feature net trained with dropout 0.2
(b) Graphs learned by NAMs with standard units
for the age feature in the MIMIC-II dataset [38], Predictions from individual subnets (as a result of dropping out hidden units) are much more jagged than the average predictions using the entire fca- ture net. Refer to Section A.3 for overview of regularization approaches used in this work.
Figure 4: ExU vs. standard hidden units. On MIMIC-II,
NAMs trained with ExU units learn jumpier graphs than with standard units while achieving similar AUC ( 0.829), Ensembling them further improves performance (= 0,830), Note that white regions in the plots correspond to regions with low data density (typically few points) and thus we see much higher variance the learned shape functions. We present a detailed case study on the MIMIC-II dataset in Section A.
function which we empirically find to be crucial for fitting any jumpy function. Furthermore, we use ReLU activations capped at n (ReLU-n) [21] to ensure that each ExU unit is active small input range, making it easier to model sharp jumps in a function without significantly affecting the global behavior. ExU-units can be combined with any activation function (i.e., any I can be used in (2)), but ReLUUn performs well in practice. Figure 2(b) shows that NNs with ExU units are able to fit the toy dataset significantly better than standard NNs.
Finally, realistic shape functions typically tend to be smooth with large jumps at only a few points (Fig- ure 4). To avoid overfitting with ExUs, strong regularization is crucial which can learn such realistic functions (e.g.. Figure 3). With ReLUs, we can typically fit smooth functions but they might miss some of these jumps. To avoid overfitting when fitting NAMs with ExUs, we employ various regular- ization methods including dropout, weight decay, output penalty, and feature dropout (see Section A.3 for an overview).
2.1 Intelligibility and Modularity of NAMs
The intelligibility of NAMs results in part from the ease with which they can be visualized. Because each feature is handled independently by a learned shape function parameterized by neural net, one can get full view of the model by simply graphing the individual shape functions. For data with small number of inputs, it is possible to have accessible explanation of the model's behavior visualized fully on a single page. Please note these shape function plots are not just an explanation but an exact description of how NAMs compute prediction. A decision-maker can easily interpret such models and understand exactly how they make decisions, for example, we validated the behavior of NAMs on the MIMIC-II dataset [38] with doctor (Appendix A.
We set the average score for each graph (i.e., each feature) averaged across the entire training dataset to zero by subtracting the mean score. To make individual shape functions identifiable and modular, a single bias term then added to the model so that the average predictions across all data points matches the observed baseline. This makes interpreting the contribution of each term easier: e,g., on binary classification tasks, negative scores decrease probability, and positive scores increase probability compared to the baseline probability of observing that class. This property also allows each graph to be removed from the NAM (zeroed out) without introducing bias to the predictions.
Visualization. We plot each shape function and the corresponding data density on the same graph. Specifically, we plot each learned shape function fax vs. Fy for an ensemble of NAMs using a semi transparent blue line, which allows us to see when the models in the ensemble learned the same shape function and when they diverged. This provides a sense the confidence of the learned shape functions. We also plot on the same graphs the normalized data density, in the form of pink bars. The darker the shade of pink, the more data there that region. This allows us to know when the model had adequate training data to Ieamn appropriate shape functions.

---

Table 1: Single-task learning NAM results. Means and standard deviations reported from 5-fold validation. Higher AUCs and lower RMSEs are better. report results on two widely used regression datasets, namely California Housing [27] for predicting housing prices and FICO [9] for understanding credit score predictions, as well as two classification datasets, namely Credit [7] for financial fraud detection and MIMIC- [38] for predicting mortality in ICUs. We present
study the MIMIC-II II dataset in Section A.1 and discuss interpretations from NAMs on other datasets Section A.2.
Model
MIMIC-II (AUC)
Credit (AUC)
CA Housing (RMSE)
FICO (RMSE)
Log./Lineal Reg.
0.791 + 0.007
0.975 + 0.010
0.728 + 0.015
4.344 + 0.056
CART
0.768 + 0.008
0.956 0.004
0.720 + 0.006
4.900 = 0.1
NAMs
0,830 + 0.008
0,980 + 0.002
0.562 + 0.007
3.490 = 0.081
EBMs
0.835 + 0.007
0.976 + 0.009
0.557 + 0.009
3.512 = 0.095
XGBoost
0.844 +: 0.006
0.981
0.008
0.532 + 0.014
3.345 0.071
DNNs
0.832 + 0.009
0.978 + 0.003
0.492 + 0.009
3.324 + 0.092
House Price Contitation
10
FIGD Score Coro button
-10
Fax Scor
High 710 Scare App lcan
Longitude
Sdee Daliparryy Ever
1o0 Siice Novt
Pocsen Cairparee
Saeant Trods
dayusteee and Shoatte
Troe Tractes
Net Frodion Instatmnne Bunden
Moo the Sirse Noee Soccet
Worth Binos Okder: Trndo Open
lnnal on Traites
Nunbor rodes Froot Trakis win Eclanoo
% Trucko nit Subance
stng LIGA torthh dolkiing "0ye
Truds Ozen Lost 12 Woreta
Trakio doe Ever
#ooowe Toukka with duaacc
tradee ID tver
satu lestrr Tracke
Tresen Nerve Dellvuuert
Delinoent
MecDayyaa da Seccro
Nest recson Deedhingg Durben
Astogg Womber Th
House Price Contriatioon
Losi Tiat
0
10
12 14
Figure 5: Understanding individual predictions for credit scores. Fea- ture contribution using the learned NAMs for predicting scores of two applicants in the FICO dataset [9]. For given input, each feature net in the NAM acts as a lookup table and returns a contribution term. These contributions are combined in a modular way: they are added up, and passed through link function for prediction. the longer person's 's credit history, the better it is for their credit score The high scoring applicant has a long credit history (Average Months on File), which contributes to their credit score better. On the contrary, the low scoring applicant used their credit quite frequently (Total Number of Trades) and has large burden (Net Fraction Instaliment Burden), thus resulting in low score.
Figure 6: California Housing. Graphs learned by NAMs trained predict house prices [27] for two most important features. As expected, The house prices in- crease linearly with median in- in high data density regions. Furthermore, the graph for longi- tude shows sharp jumps in price prediction around the location of San Francisco and Los Angeles.
3 Evaluating the Accuracy of NAMs
In this section, evaluate the single-task learning capacity of NAMs against the following baselines on both regression and classification tasks:
Logistic/ Linear Regression and Decision Trees (CART): Prevalent intelligible models. For both methods above we use the sklearn implementation [28], and tune the hyper-parameters with grid search.
Explainable Boosting Machines (EBMs): Current state-of-the-art GAMs [5, 23] which use gradient boosting of millions of shallow bagged trees that cycle one-at-a-time through the features.
Deep Neural Networks (DNNs): Unrestricted, full-complexiyy models which can model higher- order interaction between the input features. This gives us a sense of how much accuracy we sacrifice in order gain interpretability with NAMs.
Gradient Boosted Trees (XGBoost): Another class of full-complexity models that provides an upper bound on the achievable test accuracy in our experiments. We use the XGBoost implementation [6].
5

---

heurcorE Lyr phooyy
Mextropr 18 Lymahaoyya hate
Westropt// .ymp Teeyte
(a) Architecture
Age seern
Age mearrd
Age Chaars
(b) Anti-Coagulants
(c) NSAIDs
(d) Glucocorticoids
Figure 7: Estimating personalized treatment benefits for Covid-19 patients. NAMs provide unique
combination of intelligibility and differentiability which make them suitable as component in contextual parameter generation (a). By applying NAMs in this way, benefits medical treatments for Covid-19 19 patients (b-d).
able to estimate and interpret personalized
Training and Evaluation. Feature nets in NAMs selected amongst (1) DNNs containing hidden layers with 64, 64 and 32 units and ReLU activation, and (2) single hidden layer NNs with 1024 ExU units and ReLU-1 activation. We perform 5-fold cross validation to evaluate the accuracy of the learned models. To measure performance in Table 1, we use area under the precision-recal curve (AUC) for binary classification and root mean-squared error (RMSE) for regression. More details about training and evaluation protocols can be found in Section A.5 in the appendix.
NAMs achieve comparable performance EBMs both classification and regression datasets, making them a competitive alternative to EBMs. Given this observation, we next look at some additional capabilities of NAMs that are not available to EBMs or any tree-based learning methods.
Unique Capabilities of NAMs
4.1 Intelligible Parameter Generation: Leveraging the Differentiability of NAMs
Medical treatment protocols are designed to deliver treatments to patients who would most benefit from them. To optimize treatment protocols, we would like a model which provides an intelligible map from patient information to estimate of benefit for each potential treatment. To accomplish this, we use a NAM to generate parameters for personalized models of mortality risk given treatment (Fig. 7). By training to match predicted mortality risk with observed mortality, the NAM encodes expected treatment benefits as a function of patient information. NAMs are the only nonlinear GAM suitable for this application because NAMs are differentiable and can be trained via backpropagation.
Figure 7 shows NAM trained to predict treatment benefits for Covid-19 19 patients. We train the model on deidentified data from over 3000 Covid-19 19 patients. The model suggests that the benefits of anti-coagulants and NSAIDs decrease with increased Neutrophil / Lymphocyte Ratio (NLR), while the effectiveness of glucocorticoids slightly increases with increasing NLR. NLR is marker of inflammation and severe Covid- 19; it is thus expected that anti-coagulants (which target a distinct biomedical pathway) and NSAIDs (which weaker) would not be effective for patients with elevated NLR. In contrast, glucocorticoids become more effective for patients with more inflammation. This example shows the utility of differentiable nonlinear additive model such as NAMs.
4.2 Multitask Learning
One advantage of NAMs is that they are easily extended to multitask learning (MTL) [4], whereas MTL is not available in EBMs or in any major boosted-tree package. In NAMs, the composability of neural nets makes it easy train multiple subnets per feature. The model can learn task-specific weights over these subnets to allow sharing of subnets (shape functions) across tasks while also allowing subnets differentiate between tasks as needed. However, it is unclear how implement MTL in EBMs and possibly requires changes to both the backfitting procedure and the information gain rule in decision trees. Figure shows multitask NAM architecture that can jointly learn different feature representations for each task while preserving the intelligibility and modularity of NAMs. As we show, this can benefit both accuracy and interpretability. We first demonstrate multitask NAMs on a synthetic dataset before showing their utility on a multitask formulation of the COMPAS recidivism prediction dataset.

---

Multitask NAM Architecture. The mul- titask architecture identical that of single task NAMs except that each fea- ture is associated with multiple subnets and the model jointly learns task-specific weighted sum over their outputs that deter- mines the shape function for each feature and task. The outputs corresponding to each task are summed and bias is added to obtain the final prediction score. The number of subnets does not need to be the same as the number of tasks - the num- ber of subnets can be less than, equal to, or even more than the number of tasks. Al- though the shape plot for each task is lin- car combination of the shape plots learned by each subnet for that feature, this gen- erates single unique shape plot for each task and there is no need to examine what has been learned by the individual subnets for interpreting multitask NAMs.
Task
Σ Wist
0
Task
(iiefio(az)
Wast fis (ox)
i feature index
a
tusk index.
Figure 8: Multitask NAM architecture for binary classifica- tion. Multiple subnets are trained each input feature and weighted sums are learned over the subnets.
4.2.1 Experiments on Synthetic Multitask Data
Multitask models often show improvement over single task learning when tasks are similar to each other and training data limited. We construct synthetic dataset that showcases the benefit of multitask learning in NAMs and demonstrates their ability to learn task-specific shape plots when needed. We define related tasks, each a function of three variables. All 6 tasks are the same function of variables zo and 11, and differ only in the function applied to ./2
Tasko f(x)) + g(x1) + h(x2)
Taski f(xo) + g(x)) + i(x2)
Task2 = f(xo) + g(z)) - h(12)
Tasks = f(xo) + g({ - i(x22
Task; f(xo) + ((())) + (h(x2) + i(x2))
Tasks = (zo) + g (h(x2) + i(x2))
Functions f(zo), 9(21). (xx2) and i ((x2) are as follows:
1xp
g(xg)
0.0
1.5
1.0
0.0
A NAM with two subnets feature can model every function of x2 by learning two subnets, one for (x22) and one for i(22) and assigning appropriate weights to the output of each. Because we would not know this in advance with real data, we use 6 subnets so that each of the 6 tasks (outputs) could, needed, learn independent shape functions. We train models on 2,500 training examples, evaluate
Task
Task 1
Task
Task
Task 4
Task
AW
Single Task NAM
O
-
Multitask NAM
MMM
M
M
MW
MM
1
X9
X2
Figure 9: Single and Multitask NAM shape plots for from typical (median) run of each task. The learned shape function is blue; the generator function black. See A.8.2 for details of the generator functions.
7

---

Table 2: MSE for STL and MTL NAMs on synthetic data, Average of 20 runs. Lower MSEs are better.
Model
Task 0
Task
Task 2
Task 3
Task
Task 5
Mean
Single Task NAM
0.965 0.710
1.116 0.715
1.347
0.944 0.711
1.058
1.066
1.083
Multitask NAM
0.709
0.717
0.709
0.712
Single Task NAM
Multitask NAM
0.3
1.5
0.2
0.2
1.0
0.1
0.1
0.1
0.0
-0.1
-0.1
0.1
-1.0
-0.3
-0.3
Charge
Charge
0.3
1.5
0.2
1.5
0.2
Recktrin Rikk
1.0
0.1
0.5
0.0
0.5
0.0
0.0
-0.1
0.0
-0.1
-02
-10
Pror Counts
-03
Uaitee
Othe
-1.0
Fomale
Gender
-03
con
Race
Figure 10: Single and Multitask COMPAS Recidivism Prediction. Plots in the left column show the shape functions for each input feature learned by an ensemble of 100 single task NAMs. Thin blue lines represent shape functions for individual members of the ensemble. Pink bars represent the normalized data density for each feature. Plots in the right column show the Race and Charge degree shape plots for ensemble of 100 multitask NAMS, with the Women task shown in green, and the Men task in blue.
them on test set of 10,000 examples, and average the results over 20 trials. Also, we ensured that each subnet has enough parameters to easily learn the necessary feature shape plots. So MTL is not doing better than STL because STL has inadequate capacity and MTL has more capacity.
Table 2 shows that on average across all tasks, multitask NAMs achieve mean squared error 34% lower than single task NAMs, and at least 25% lower on each individual task. In all 120 trials of the 6 tasks combined, MTL achieved a better score than STL on 119 of the 120 trials. Figure 9 shows the shape plots learned by median runs of STL and MTL for the functions of 22 that vary among tasks. Furthermore, we illustrate that a multi-task NAM is as interpretable as a single task NAM by plotting the multi-task NAM predictions on the 3 input features for each of the tasks in Figure A.6.
4.2.2 Single and Multitask COMPAS Recidivism Prediction
COMPAS is a proprictary score developed to predict recidivism risk, which is used to inform bail, sentencing and parole decisions and has been the subject of scrutiny for racial bias [1, 8, 42]. 2016, ProPublica released recidivism data [30] on defendants in Broward County, Florida.
Single Task Recidivism Prediction: First, we ask whether this dataset is biased using the trans- parency of single-task NAMs. Figure 10 shows the learned single-task NAM which is as accurate as black-box models on this dataset (see Table 1). The shape function for race indicates that the learned NAM may be racially biased: Black defendants are predicted to be higher risk for reoffending than white or Asian defendants. This suggests that the recidivism data may be racially- biased. The modularity of NAMs makes it easy to correct this bias by simply removing the contributions learned from the race attribute by zeroing out its mean-centered graph in the learned NAM. Although this would drop the AUC score as we are removing a discriminative feature, it may be more fair model to use for making bail decisions. It is important to keep potentially offending attributes in the model during training so that the bias can be detected and then removed after training. If the offending variables are eliminated before training, makes debiasing the model more difficult: if the offending attributes are correlated with other training attributes, the bias is likely to spread to those attributes [3]. The transparency and modularity of NAMs allows one to detect unanticipated biases in data and makes it casier to correct the bias in the learned model.
8

---

Table 3: ROC AUC for multitask and single task NAMs on COMPAS dataset, broken down by gender. Each cell contains the mean AUC + one standard deviation obtained via 5-fold cross validation. Higher AUCs are better.
Model
COMPAS Women
COMPAS Men
COMPAS Combined
Single Task NAM
0.716 = 0.026 0.723 = 0.019
0.735 + 0.009 0.737 + 0.009
0.737 = 0.010 0.739 = 0.010
Multitask NAM
Multitask Recidivism Prediction: In some settings multitask learning can increase accuracy and intelligibbllity by learning task-specific shape plots that expose task-specific patterns in the data that would not be learned by single task learning. We reformulate COMPAS as a multitask problem where recidivism prediction for men and women are treated as separate tasks NAM with two outputs. Indeed, we find that a multitask NAM reveals different relationships between race, charge degree, and recidivism risk for men and women while achieving slightly higher overall accuracy.
The right column of Figure 10 displays a selection of shape plots learned for a multitask NAM trained on the same data the single task NAM but with Male and Female as separate output tasks. (The remaining MTL shape plots are similar for the two genders, reinforcing that these are strongly related tasks, but we omit them for brevity.) The race shape plot in the multitask NAM shows different pattern of racial bias for each gender. The curve for men looks similar to that of the single task NAM (which is expected because men make up 81% of the data), but the curve for women suggests that recidivism risk is lower for Black women and higher for Caucasian and Hispanic women than for their male counterparts. The multitask shape plots also reveal that charge degree is almost twice as important for women as it is for men. The straightforward extension of NAMs to the multitask setting offers useful modelling technique not currently available with tree-based GAMs.
5 Related Work
Generalized Additive Neural Networks (GANNs) [29] are somewhat similar to the NAMs we propose here. Like NAMs, GANNS used restriction in the neural net architecture to force it to learn additive functions of individual features. GANNs, however, predate deep learning and use a single hidden layer with typically only 1-5 hidden units. Furthermore, GANNS did not use backpropagation [37], required human-in- -n-te--- evaluation and were not successful in training accurate or scalable GAMs with neural nets. See Section A.7 for more detailed overview of GANNs.
In contrast, NAMs in this paper benefit from the advances in deep learning. They use a large number of hidden units and multiple hidden layers per input feature subnet to allow more complex, more accurate shape functions to be learned. Furthermore. NAMs use novel ExU hidden units to allow subnets to learn the more non-linear functions often required for accurate additive models, and then form an ensemble of these nets to provide uncertainty estimates, further improve accuracy and reduce the high-variance that can result from encouraging the model to learn highly non-linear functions.
Prior to NAMs, the state-f-thh-an in high-accuracy, interpretable generalized additive models [12, 14] are the GAM [23] and GA [24] based on regularized boosted decision trees which were successfully applied to healthcare datasets [5]. We compare the accuracy of NAMs to these models in Section 3. We note that pairwise interactions, similar to GA, can be easily added to NAMs GA use a heuristic to compute the importance of each pairwise interaction by fitting residual from first-order terms and select the k (< 10) most important interactions. We don't 't consider such interactions to keep the paper focused on additive modeling with neural nets.
6 Conclusion and Future Work
We present Neural Additive Models (NAMs), which combine the inherent interpretability of GAMs with the expressivity of DNNs, opening the door for other advances in interpretability in deep learning. NAMs are competitive in accuracy to GAMs and accurate alternatives to prevalent interpretable models (e.g., shallow trees) while being more easily extendable than existing GAMs due to their differentiability and composability.
promising direction for future work is improving the expressivity of NAMs by incorporating higher-order feature interactions. While such interactions may result in more expressivity, they might worsen the intelligibility of the learned NAM. Thus, finding a small number of crucial interactions seems important for more expressive yet intelligible NAMs. Another interesting avenue is developing

---

better activation functions or feature representations for easily expressing complex functions using NAMs. For example, fourier features [43] have been shown to be highly effective for learning high frequency functions via neural networks and might be useful for training expressive NAMs.
Extending and applying NAMs beyond tabular data to more complex tasks with high-dimensional inputs, such as computer vision and language understanding, is an exciting avenue for future work. While NAMs only use some of the expressivity of DNNs, one can imagine using NAMs in real- world pipeline where intelligibility is required for decision making from representation [36] (e.g. features learned from images, speech etc). Much of the existing interpretability work in deep learning focuses on making learned representations interpretable. Also. NAMs can be used for interpretability across multiple raw features (e.g. multimodal inputs) where interpretability within a NAM network can utilize existing interpretability methods in ML recently CNN-LSTM based extension of NAMs have already been developed for genomics [40] where the input to each NAM network was a one-hot encoded DNA sequence (passed as an image). Overall, we believe that NAMs are likely to broaden the use of inherently interpretable models in the deep learning community.
Broader Impact
Interpretability in Al systems might be desirable or necessary for various reasons see [44] for an overview; we discuss some of them in the context of NAMs below:
Safeguarding against bias: NAMs can check whether training data is used in ways that result in bias or discriminatory outcomes and can be easily corrected for bias yield possibly more fair models - e.g., Section 4.2.2 demonstrates this utility for recidivism risk prediction.
Improving AI system design: NAMs allow developers to interrogate why it behaved in a certain way (e.g.. tracking system malfunctions), and develop improvements Section A. shows that NAMs can explain seemingly anomalous results in healthcare as well as uncover problems that might put some kinds of patients at risk and need correction before deploying the system.
Adhering to regulatory standards or policy requirements: Interpretability of NAMs can be important in enforcing legal rights surrounding a system - e.g., credit scores in the United States, have a well-established "right to explanation". NAMs can also enable individuals to contest model outputs, e.g., challenging an unsuccessful loan application, based on the interpretations provided by NAMs for specific decision (Figure 5).
This can be particularly important if an AI system is deployed in a new environment, where we cannot be sure of its effectiveness - e.g., NAMs for fraud detection (Section A.2.2) can be analyzed to understand the risks involved or how it might fail before deploying to unseen customers.
Assessing risk, robustness, and vulnerability:
Giving users confidence in the system: Interpretations from NAMs might provide users confi- dence that it works as intended e.g., expensive house prices near metropolitan areas such as San Francisco, as predicted by NAMs (Figure 6), is expected for trustworthy model.
Data-driven scientific discovery: NAMs can be applied in natural sciences e.g., ecology [12], medicine [13], astronomy [15] etc. to obtain novel scientific insights and discoveries from observational or simulated data [34, 45] while remaining scalable to the ever-increasing data.
There are also pitfalls associated with interpretability methods NAMs are no exception. Different contexts give rise to different interpretability needs e.g., public have different expectations of systems used in healthcare vs. recruitment [19]. Furthermore, Al system designs often need to balance competing demands e.g., to optimize the accuracy of system ensure fairness (NAMs for making bail decisions with race feature "removed" may be less accurate but more fair). In many critical decision-making areas e.g., healthcare, justice, and public services complex processes have developed over time to provide safeguards, audit functions, or other forms of accountability. NAMs may therefore be only the first step in creating trustworthy systems. Those developing NAMs must consider how their use fits in the wider socio-technical context of its deployment
Acknowledgments
We would like to thank Kevin Swersky for reviewing early draft of the paper. We also thank Sarah Tan for providing us with pre-processed versions of some of the datasets used in the paper. RA would also like to thank Marlos C. Machado and Mare G. Bellemare for helpful discussions.
10

---

References
[1] Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu. Machine Bias: There's software used across the country to predict future criminals. And it's biased against blacks, 2016. URL https://wwwt www.ppoppbllcc...mmmomm.m.ppa... hwtppalica..og/rrtiiiiiooooooooo -awee--iaa-rii--sseeeeeeeee-eese
org/article/
[Accessed Febru-
ary 1, 2020].
[2] Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. ICML, 2017.
[3] Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods Research, 2018.
[4] Rich Caruana. Multitask learning. Machine Learning, 1997.
[5] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. SIGKDD, 2015.
[6] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. SIGKDD, 2016.
[7] Andrea Dal Pozzolo. Adaptive machine learning for credit card fraud detection. PhD Thesis, Department of Computer Science, Université Libre de Bruxelles, 2015.
[8] Julia Dressel and Hany Farid. The accuracy, fairness, and limits of predicting recidivism. Science advances, 2018.
[9] FICO. FICO Explainable Machine Learning Challenge. https://conmunitg wwc..com.ot......c s/ explainable
www-iaaahhhaeeeeeeeeeeeeeeeeeeee papllainable-mcchine--
-earninn----e---n----ene challenge, 2018.
[10] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010.
[11] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and
D Sculley. Google vizier: A service for black-box optimization. SIGKDD, 2017.
[12] Antoine Guisan, Thomas C Edwards Jr. and Trevor Hastie. Generalized linear and generalized additive models in studies of species distributions: setting the scene. Ecological modelling. 2002.
[13] T Hastie and R Tibshirani. Generalized additive models for medical research. Statistical methods in medical research, 1995.
[14] Trevor Hastie and Robert Tibshirani. Generalized Additive Models. Chapman and Hall/CRC, 1990.
[15] M W Hattab, R S de Souza, B Ciardi, J-P Paardekooper, S Khochfar, and C Dalla Vecchia. A case study of hurdle and generalized additive models in astronomy: the escape of ionizing radiation. Monthly Notices of the Royal Astronomical Society, 2018
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CVPR, 2015.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[18] Kurt Hornik, Maxwell Stincheombe, Halbert White, al. Multilayer feedforward networks are universal approximators. Neural Networks, 1989.
[19] ICO. Project explain: Interim report. 2019, https://ico. org uk/nedda///111003/ project- eess-eepllinn aanl-ain--ia 20190603, pdf.
[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.69888 2014.
[21] Alex Krizhevsky. Convolutional deep belief networks on cifar- 10. 2010.
[22] Christine K Lee, Muntaha Samad, Ira Hofer, Maxime Cannesson, and Pierre Baldi. Development and validation an interpretable neural network for prediction of postoperative in-hospital mortality. NPJ digital medicine, 2021.

---

[23] Yin Lou, Rich Caruana, and Johannes Gehrke Intelligible models for classification and regression. SIGKDD, 2012.
[24] Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. SIGKDD, 2013.
[25] Vinod Nair and Geoffrey E Hinton, Rectified linear units improve restricted boltzmann machines. ICML, 2010.
[26] Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv:1909 1909.0922, 2019.
[27] R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 1997.
[28] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn:
Machine learning in python. JMLR, 2011.
[29]
William JE Potts. Generalized additive neural networks. SIGKDD, 1999.
[30] ProPublica. COMPAS Data and analysis for 'Machine Bias'. https: http://ggttttb.oc
propublica/compas aaaabiiicaa/mmpa-aaaaaaai..a 2016.
[31] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2018.
[32]
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. ICML, 2018.
[33] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. SIGKDD, 2016.
[34] Ribana Roscher, Bastian Bohn, Marco F Duarte, and Jochen Garcke. Explainable machine learning for scientific insights and discoveries. arXiv preprint 1rXiv:1905.00m 08883, 2019.
[35] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 2019.
[36] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong, Interpretable machine learning: Fundamental principles and 10 grand challenges. arXiv preprint 2tfi::2003//25//// 2021.
[37] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 1986.
[38] Mohammed Saeed, Mauricio Villarroel, Andrew T Reisner, Gari Clifford, Wii Lehman, George Moody, Thomas Heldt, Tin Kyaw, Benjamin Moody, and Roger Mark. Multipa- rameter intelligent monitoring in intensive care ii (mimic-ii): a public-access intensive care unit database. Critical care medicine, 2011.
[39] Jasper Snoek. Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. NeurIPS, 2012.
[40] Divyanshi Srivastava, Begüm Aydin, Esteban O Mazzoni, and Shaun Mahony. An interpretable bimodal neural network characterizes the sequence and preexisting chromatin predictors of induced transcription factor binding. Genome Biology, 2021.
[41] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting- JMLR, 2014.
[42] Sarah Tan, Rich Caruana, Giles Hooker, and Yin Lou. Distill-and-compare: Auditing black-box models using transparent model distillation. AAAI/ACM Conference AI, Ethics, and Society, 2018.
[43] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Ragha- van, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006. 10739, 2020.
[44] The Royal Society.
Explainable Al: The Basics Policy Briefing. http:///ee/p/poeeet//////aaaaaaa AI ht-.aliiincppettoolooyoooooyeoae pdf.
2019.
12

---

[45] The Royal Society and The Alan Turing Institute. The Al revolution in scientific research. 2019. https://royalsocietyy ottg://mmaaapppocc/ppoooooooooop -ttp://aaaandddooooe andesoccety/ AI AI-revolution-ieg wwwpoonnii-nciinee .df.
[46] Lev V Utkin, Egor D Satyukov, and Andrei Konstantinov. Survnam: The machine learning survival model explanation. arXiv preprint
2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "bEXNkDvjvZE", "reviewer": "Reviewer_Jr31", "summary": "In this paper, the authors introduce a new generalized additive model, called neural additive models, which use neural networks to learn a non-linear transformation of each input feature, independently. Along the way, they introduce a new activation function, called ExU, which enables a highly sensitive weights that can fit very large, seemingly noisy, changes of a given function. They demonstrate that NAM performs as well as other GAMs, such as EBMs, as well as black box models, such as DNNs and XGBoost. The major  benefit of NAM is that, similar to EBMs, they are intrinsically interpretable simply by visualizing the learned non-linear function for each input feature.  They compare the performance of these methods across several tasks. The paper is written very clearly with easy to follow examples. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the proposed NAM very interesting and a worthwhile contribution to the toolbox of GAMs. The new activation, i.e. ExU, is also quite intriguing. In this regards, I think that the paper has legs to stand on. The work is a simple extension, using neural networks to learn a non-linear transformation of each input feature independently. It is thus learning a suitable non-linearity for each input feature such that its additive contribution is predictive of the task at hand. Instead of using a preset non-linear function, NAMs use an MLP to learn one. It is very clear how this approach is different from previous contributions. A major limitation to this study was that there was no clear example that highlights the benefits of NAM over previous methods, either in terms of performance or interpretability. Below, I highlight my main concerns:\n\n- The argument that a ExU can learn jagged shape functions, whereas the same network with ReLU activations learns something smoother is clearly demonstrated for a given network. However, a deeper network with ReLU activations should be able to fit the noisier function, right? So, the statement is really more about the ability of ExU to express noisier functions using a smaller number of parameters compared to ReLU activations. \n\n- Many of the synthetic examples are a bit contrived. For instance, the example of learning a jagged shape function; It's actually unclear when learning this kind of jagged shape function would be required in practice. On the flip side, how well does ExU do when the underlying function is smooth? Does it over-exaggerate more easily due to its highly sensitive weights? Would a relu or smooth activation perform better in such a circumstance? If there is a big difference between these scenarios, perhaps some guidance would be beneficial. \n\n- Building on this question, how does NAM do when the underlying features are supposed to be additive (i.e. no nonlinearities). In this case, NAMs are overkill, but do their MLPs learn learn a linear function or do they learn complex non-linear functions? If that's the case, how \"interpretable\" is it? I supoose it is still interpretable in a strict sense of the model, but it is very different from being reliable. A comment on this could help an end user think more deeply about these considerations. \n\n- I do appreciate the attempt at comparing different methods across different datasets, but I think the examples don't really highlight NAMs ability to excel at tasks that previous methods struggle. The gains for each model is quite marginal. One question could be -- relative to previous methods, are NAMs robust to a wide setting of hyperparameters? Are there any benefits (beyond interpretability) with training time, hyperparameter search, inference time, scalability? If so, this should be highlighted as the performance alone is right in the pack (and EBMs are also interpretable). \n\n- Since interpretability means so many things these days, I think there should be a clear message what is meant by interpretability here. A comparison of global function vs local function via LIME was made, but it may be nice to contrast other definitions of interpretability, such as post hoc explanations with attribution methods. \n\n- In regards to interpretability, it is a true benefit that NAMs can just plot the learned function and one can get a sense for variations of the function. It's nice that variability is low where data is concentrated and then the uncertainty grows in data deprived regions of function space. it would also be nice to see  interpretations of other models for comparison. \n\n- How do EMB interpretations compare? Do you draw the same conclusions? This may be beyond the scope of the paper but for black box models that were in the comparison, perhaps attribution methods, like Shap, could be sufficient for seeing whether you can draw similar conclusions (or point out weaknesses in their explanations). \n\n- For multi-task learning, it's unclear whether connecting each MLP across tasks leads to lower interpretability  as there are now a linear combination of different non-linear functions for each feature -- not just one. The examples show the prediction of each task, but that doesn't necessarily highlight interpretability. The improved performance may be benefiting from the increased expressivity of the model and not necessarily from the fact that it is in a multi-task learning paradigm. A nice control experiment would be to keep multiple MLP modules for each feature (as if multi-tasking) but train on a single task. I suspect the model would now be able to capture the complex shapes that it was unable to with the original single-task NAM. If this experiment works, I think it would be beneficial to compare the interpretability of the quasi-multi-task vs single-task NAM. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Uh-5wh-565", "reviewer": "Reviewer_yq49", "summary": "This paper propose Neural Additive Model that provides more interpretability with DNNs. It learns a linear combination of neural networks that each attend to a single input feature. Through experiments, NAM is shown to be more accurate than other intelligible models such as logistic regression and explainable boosting machine (EBM), while also provides explanability with respect to the input features. A multitask learning architecture along with the single task architecture is proposed.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed NAM is a DNN version of the generalized additive models (GAM). The proposed NAM allows one to exam the eact description of how NAM compute a prediction since one can sweep the input feature and see the prediction curve. The individual shape function provides the good intelligibility for the input feature. The proposed method contains certain amount of novelty in the NAM and the way to interprete the model, including the multi-task NAM architecture. However, I think the novelty of the proposed NAM is not that significant. Several experiments have been conducted to demonstrate the better accuracy compared with linear regression and the intelligible EBM. The paper also shows several examples on how to interpret the model and explain how the input features affect the performance. The paper is well-written and easy to understand.\n\nWhile it is good to allow each network attend to only one feature, the limitation of the proposed NAM is also evident. The major advantage of deep learning models lie in its representation learning, which learns the non-linear combination and transformation of interleaving features, not individual features. The relationship and interaction of different features empowers the development in many areas where deep learning applies such as computer vision, natural language processing, recommendation, etc. I don't see how the proposed NAM can be more generalized to other areas, where the input feature is of high-dimension and requires more complex understanding.\n\nOther concerns:\n- The proposed exp-centered hidden units (ExU) uses the exponential function for the weight. Will this introduce instability in training? It roughly only allows the weight to be in a much smaller range, and could cause the gradient to go to nan easily.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation of the proposed NAM is that it is a simple linear combination of multiple neural nets which each only attends to one feature. While it provides the intelligibility to the model when input feature is simple, its generalizability and expressiveness to more complex domains is lacking.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "1oqmnoO1aU3", "reviewer": "Reviewer_oFHP", "summary": "The authors propose the neural additive models which have an additive structure and which are more explainable than general neural networks. They apply this model for various tasks and applications and show their interpretability through visualizations. Its a good contribution and a reasonably well-written paper. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose neural additive models, a linear combination of multiple small networks, which are trained jointly and can learn complex functions such as jagged shape functions. The claim is that because the models are modular, the predictions can be propagated back to the individual network functions and easily visualized. They show the explainability of the networks in applications like fraud detection and recidivism risk prediction.\n\nThe authors have explained the proposed structure reasonably well but its readability could be improved with a better presentation of the notations and the model structure. The experimental section is detailed and covers various single-task and multi-task learning applications. \nThey have compared their solution with other explainable ML algorithms and shown comparable prediction accuracy. My only concern is that their analysis results on interpretability is very heuristic and subjective, rather than a formal validation. \n\nOverall, the paper is a good contribution and I recommend acceptance.\n\nUpdate (after author response): The authors have addressed my main concerns. I recommend acceptance with the changes proposed in the author response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper addresses a number of applications like healthcare and recidivism prediction which have a high societal impact. For example, they try to determine the features which contribute to mortality during ICU stays (using the MIMIC dataset) and the features that contribute to bias in the COMPAS dataset. Though the authors do some heuristic analysis, these results should be ideally formally validated. The authors should also address some of the limitations of their current work and potential future avenues of study. ", "review_text": "", "extra_scores": {"ethical_concerns": "I have found no ethical concerns in this paper. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1Rx3qCNraw", "reviewer": "Reviewer_qyF4", "summary": "This paper proposes Neural Additive Models (NAMs), a linear combination of neural networks that can be used as a highly accurate and interpretable model for tabular data. In order to account for the potentially detrimental bias of NNs to be smooth, the authors introduce an exp-centered hidden unit which enables the NN to better fit jagged shaped function, and compare this NAM to other tree-based and DNN based models. NAMs are able to achieve comparable accuracy to tree-based models with the added advantage of being easily adapted to the multitask setting, which can be helpful in increasing accuracy, discovering bias, constructing more helpful models for fields which need interpretability like healthcare. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is incredibly well written, easy to follow, and has some of the best visualizations I have seen. The motivation and core method are clear and the analysis was quite thorough and seems like it could be easily reproduced. Furthermore, the applications to things like covid-19 and COMPAS are well thought out and have impressive results. Overall a very fun read and seems particularly well suited to the NerulIPS audience. \n\nMy only concerns are relatively small: \n\n(1) In section 2.1 the authors mention that they altered the weight initialization to better fit jagged curves; I would love to see a plot showing the difference in curves with this new initialization as well as the traditional weight initialization (similar to figure 2). I looked in A.3 but that section seems to be empty. \n\n(2) While the plots in section 2.1 are convincing that these ExU networks are better for jagged curves, it would be nice to add them as another benchmark in section 3 \n\nOther small nits:\n\n* Units of median income are missing in Fig 6\n* line 327 I believe should be \"the public has different...\"\n* a period needs to be added at the end of the conclusion", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed how this method may not be suitable for certain applications, as they may need a different type of interpretability. One other thing I would think would be an issue is the amount of resources needed/throughput, although I have not benchmarked this myself. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were overall very enthusiastic about this paper, highlighting its novelty, and clarity in terms of writing and visualizations. There are a few changes that I expect you will make based on your responses to the reviewers.
Decision: Accept (Spotlight)