Interpretable Additive Tabular Transformer Networks
Authors: Anton Frederik Thielmann
Arik Reuter
Thomas Kneib
David Rügamer
Benjamin Säfken
Venue: Accepted by TMLR ()

Abstract: Attention based Transformer networks have not only revolutionized Natural Language Processing but have also achieved state-of-the-art results for tabular data modeling. The attention mechanism, in particular, has proven to be highly effective in accurately modeling categorical variables. Although deep learning models recently outperform tree-based models, they often lack a complete comprehension of the individual impact of features because of their opaque nature. In contrast, additive neural network structures have proven to be both predictive and interpretable. Within the context of explainable deep learning, we propose Neural Additive Tabular Transformer Networks (NATT), a modeling framework that combines the intelligibility of additive neural networks with the predictive power of Transformer models. NATT offers inherent intelligibility while achieving similar performance to complex deep learning models. To validate its efficacy, we conduct experiments on multiple datasets and find that NATT performs on par with state-of-the-art methods on tabular data and surpasses other interpretable approaches.

Published in Transactions on Machine Learning Research (05/2024)
Interpretable Additive Tabular Transformer Networks
Anton Frederik Thiclmann
Institute of Mathematics
Clousthal University of Technology
Arik Reuter
arik_ reeter
Instibute of Mathematics
Clausthal University of Technology
Thomas Kneib
thncibb@nni-gocttingn..cc
Chair of Statistics and Campus Institute Dota Science Georg-August Universität Götlingen
David Rügamer
Deportment of Statistics, LMU Munich Munich Center for Machine Learning (MCML)
Benjamin Säfken
Institute of Mathematics
Clausthal University of Technology
Reviewed on OpenReview: https: // openrevew.. net/forune id- TáJTpzAKD
Abstract
Attention based Transformer networks have not only revolutionized Natural Language Processing but have also achieved state-of-the-art results for tabular data modeling. The attention mechanism, in particular, has proven to be highly effective in accurately modeling categorical variables. Although deep learning models recently outperform tree- based models, they often lack a complete comprchension of the individual impact of features bocause of their opaque nature. In contrast, additive neural network structures have proven be both predietive and interpretable. Within the context of explainable deep learning, propose Neural Additive Tabular Transformer Networks (NATT), modeling framework that combines the intelligibility of additive neural networks with the predictive power of Transformer models. NATT offers inherent intelligibility while achieving similar performance to complex deep learning models. To validate its efficacy, we conduct experiments on multiple datasets and find that NATT performs on par with state-of-the-arl methods on tabular data. and surpasses other interpretable approaches.
1 Introduction
Deep neural networks (DNNs) have emerged as one of the most powerful and versatile tools in AI, with remarkable abilities in modeling complex, high-dimensional problems and recognizing intricate patterns in non-tabular data. They have shown exceptional performances on tasks such image classification (Yu et al., 2022; Dosovitskiy et al., 2020), text classification (Huang et al., 2021; Lin et al., 2021), audio classification (Nagrani el al., 2021), Lime-series forecasting (Zhou et al., 2022; Zeng et al., 2022) and many more. However, for the most common data type in real world applications, tabular data, DNNs were still outperformed by tree based methods as XGBoost (Chen et al., 2015) or LightGBM (Ke et al., 2017). In recent years, however, the transformer architecture (Vaswani al., 2017) allowed for an increase in performances of tabular deep learning methods. Huang al. (2020), Gorishniy et al. (2021) and Arik and Pfister (2021)

---

Published in Transactions on Machine Learning Research (05/2024)
introduced architectures that were par with state-of-the-art tree based models and outperformed baseline Multi-layer perceptrons. Gorishniy et al. (2022) introduced a model that even outperformed gradient boosting methods on a majority of popular tabular benchmark datasets. In addition to their increased predictive performance, attention based models for tabular data are cemi-interpretable, by providing further insights through individual feature importances obtained from the attention layer(s). While main feature effects cannot be clearly identified or visually interpreted, the single feature importance can be abstracted from the attention layers in the transformer, allowing for a pseudo-feature significance.
While these models are incredibly powerful in predictive terms, true interpretability is still lost in their black-boz nature. This ultimately limits their applicability as espocially tabular data needs fully interpretable model structures to be applied in domains such as health care, finance or insurance. Explainability in these black-boz boz models is often achieved with post-hoc analyses on the sample level with existing methods resorting to model-agnostic methods. Locally Interpretable Model Explanations (LIME) (Ribeiro et al., 2016), Shapley values (Shapley, 1953) or layer wise relevance propagation (LRP) (Bach et al., 2015) and their extensions (Sundararajan and Najmi, 2020) try to explain model prediction ions via local approximation and feature importance. Sensitivity- approaches (Horel and Giesecke, 2020), exploiting significance statistics, can only be applied to single-layer feed-forward neural networks and can hence not be used to model complex non-linear effects. Although those approaches might be able to indicate how individual predictions are generated, they do not provide a global and complete picture of the underlying decision making process.
Recently, neural networks are making a push towards feature level interpretability by adopting the additive model structures from Generalized Additive Models (GAMs) (Hastie, 2017). GAMs use basis functions that transform features into higher dimensional space that allows feature effects to capture non-linearity. Neural Additive Models (NAMs) (Agarwal et al., 2021) proposed the neural counterpart to GAMs and inspired multiple adaptations which will be introduced in Section 2. One downside of these adaptions is that they treat all features identically, independent of the underlying data type. Numerical features are modeled the same way as categorical fcatures, which leads to parameter-dense networks that can loose their easy interpretability. We rntuoe Neural Additive Tabular Transformer (NATT) Networks, a model class that leverages the additivity constraint from GAMs and NAMs and implements transformer based embeddings for increased predictive performance while also allowing for an interpretable way to model categorical features.
In summary, NATT aims to narrow the performance gap between models that are inherently interpretable (e.g., GAMs (Hastie, 2017), NAMs (Agarwal et al., 2021), EBMs (Nori et al., 2019), NodeGAMs (Chang et al., 2021)) and black-box models. Additionally, NATT presents a strategy for enhancing the interpretability of networks handling categorical features. The issues with commonly used categorical encodings are discussed in detail section 3.1. Crucially, when dealing with numerous categorical variables and particularly a wide array of categorical expressions, NATT offers scalability and maintains ease of interpretability, an area where traditional encodings implemented in models like NAMs struggle. Our proposed approach enables the interpretation of complex interactions among categorical expressions while maintaining minimal number of shape functions, addressing the inherent challenges in scalable and interpretable modeling of categorical data. The contributions of the paper are as follows:
We propose a novel model structure that incorporates categorical feature embeddings into an additive model architecture. This allows for the joint learning of all feature embeddings and shape functions.
We demonstrate that NATTs outperform interpretable baselines as well as state-of-the art neural and tree-based boosting methods without loss of intelligibilitt.
Our experimental results demonstrate that NATTs outperform their bascline counterpart, the Neural Additive Model (NAM), by an average of -5% across multiple datasets and are on par with state-of-the-art Mack-bor models when accounting for feature interactions.
We demonstrate that NATT seamlessly incorporates pairwise or higher order feature interactions and find that 2TT achieves superior performance compared to the interpretable baselines.
2

---

Published in Transactions on Machine Learning Research (05/2024)
Importance
40
fi (x1)
Fa(aaa)
fa(e))
20
Categories
21
222 (cont)
xc(conn)
Feature Importance
Visual Interpretability
Loss
Go + (H + Eiz fo (J)(tom))))
Concatenation
Add & Norm
Feed Forward
Feature Networks
Lx
Add & Norm
Multi-Head Attention
Norualization
[cts]]
Normalization
Categorical Embedding
2. Hcone)
x Li(oon))
Categorical
Continuous features Wcont
Teatures
Dent
Figure 1: NATT model archilecture and visually interpretable output. The continuous feature are visually interpretable thorugh the plotting the output of the independt feature networks. The categorical feature importance can be analyzed the [CLs] token the Transformer architecture.
3

---

Let D = (xx(0 (x),) be the training dataset of size n and y denote the target variable that can be arbitrarily distributed. Each input æ = (x1, ,.,..,, x.) contains J features. Given a link function g()), that connects the linear predictor to the expected value of the response variable, accommodating different types of Published in Transactions on Machine Learning Research (05/2024)
2
Related work
The interpretability of deep neural networks has been long-standing challenge in the field of Artificial Intelligence. While these models have achieved impressive accuracy in varions tasks, their lack of transparency has limited their real-world applicability in some areas. This issue has led researchers to explore different approaches to increase the interpretability of these models, including generating fcaturo-level interprotability.
One promising approach in this ion is the translation of Generalized Additive Models (GAMs) into a neural framework. This idea was first introduced by Potts (1999) and expanded by de Waal and du Toit (2007) in the late 90s and early 2000s. However, the previous approaches didn't use backpropagation and had limilations in their predictive power and interprelability.
In recent years, researchers have made significant progress in this field by introducing more flexible and effective approaches. One of these approaches is Neural Additive Models (NAMs), which were first introduced by Agarwal et al. (2021). NAMs use Deep Neural Networks (DNNs) instead of smooth shape functions from GAMs. This approach has led to several extensions, including adaptations of the additive model structure using different types of shape functions such as gradient boosted trees (Chang et al., 2021) or piecewise polynomials (Dubey et al., 2022).
Several approaches have been proposed to account for feature interactions (Kim et al., 2022; Tsang et al., 2018; Enouen and Liu, 2022). Wang et al. (2021) introduced Pie-GAMs, which are NAMs that incorporate a fully connected black- bow Multi-Layer Perceptron (MLP) trained only on the residuals of a first NAM model not including feature interactions.
Luber et al. (2023) introduced extremely parameter sparse networks, leveraging basis expansion methods. Radenovic al. (2022) switched the basis expansions for neural basis expansions, keeping the overall model structure from Agarwal et al. (2021) intact.
Additionally, researchers have considered the problem of distributional regression, which is essential in many real-world scenarios. Rügamer et al. (2023; 2021) introduced semi-structured distributional regression, combining the advances of deep neural networks with structured regressions. Thielmann et al. (2024) introduced Neural Additive Models for Location Scale and Shape (NAMLSS), the extension of NAMs for distributional regression.
3 Methodology
data distributions, GAM in its fundamental form can be expressed as:
(E = 3o + EB(s)
(1)
j=1
where 30 denotes the global intercept or bias term and ʃj J R + R denote the univariate shape functions corresponding to input feature 2ÿ and capturing the feature main effects. This model structure allows for casy interpretation of the feature effects as the shape functions can be visualized. In classical GAMs, the shape functions f, are smooth functions, c.g. polynomial splines. However, while being parameter sparse and interpretable, classical GAM smoothers lack the predictive power of current deep neural networks. Switching splines for e.g. Multi Layer Perceptrons (Agarwal et al., 2021) or decision trees (Chang et al., 2021) thus has shown to create extremely powerful yet interpretable models.
Independent of the type of shape functions, pairwise feature interactions can be integrated:
E ()) 80 + Σ 5(0)) Σ m))
(2)
j=1
4

---

Published in Transactions on Machine Learning Research (05/2024)
where fyk: :R R denote the feature interactions between input features 2j and Fg. Using neural networks as shape functions not only allows to model pairwise feature interactions (GA2Ms), but higher order features interactions (GA'Ms) These interactions can e.g. be modeled using a fully connected MLP (Kim et al., 2022; Wang et al., 2021). Depending on the shape function and the corresponding optimization method, regularization techniques, e-g. feature dropout as introduced by Agarwal et al. (2021) are used to account for identifiability.
Independent of the type of shape functions, however, all of these models have in common that they fit one shape function per feature main effect for all continuous binary features. Categorical features, however, are either a) one-hot encoded - leading to each category being represented by an independent shape function (see, e.g., Agarwal et al., 2021; Radenovic et al., 2022; Enouen and Liu, 2022; Chang et al., 2021), b) target encoded (Chang et al., 2021; Popov et al., 2019) - such that dependencies between different categorical features cannot be captured and information may be lost if there are different categories with the same mean value of the target variable (Zeng et al., 2022), or c) label encoded (Nori et al., 2019) which problematic when the categories lack natural ordering. Datasets containing lot of categorical variables, or variable with lot of categories can thus lead to increased model sizes and hardly interpretable feature effects due to the excessive amount of shape functions.
Arik and Pfister (2021), Huang et al. (2020), Hollmann et al. (2022) and most recently Gorishniy et al. (2022) demonstrated the advantages that transformer based embeddings can have for modeling categorical features. Let æ = (Ient, Tront) denote the categorical and numerical (continuous) features, respectively with Beont E IR.. Further, let x (cat) denote the j-th categorical feature of the i-th observation. In order to preserve interpretability and allow for complex feature interactions, we adapt (1) to:
(4)
J gB (w)) = Bo + Σ [s(((toon)) + S(cat) (H (E)((()))))
(3)
where H-) denotes a sequence of transformer layers, creating the feature embeddings for the categorical features. The parametric embeddings (xeat))) are the input of the first transformer layer. H()) returns the contextualized embeddings {h1, h,,...,,,,, that are subsequently fed into the shape function, fteatt : [exx)) + R, where e denotes the chosen embedding dimension. The contextual embeddings, (2,, h,,,, are thus jointly learned with the shape function, f(cat)
To embed each categorical feature into a column embedding we follow Huang et al. (2020). Our architecture for the categorical features comprises a column embedding layer, a stack of T. Transformer layers and a Multilayer Perceptron. The Transformer layers follow the classical architecture (Vaswani al., 2017) and are comprised of multi-head self-attention layers and position-wise table es, (+) is created for each categorical feat ure, Tj(ctt)- For the j-th categorical feature with d, classes, the embedding table e,, has (d, + embeddings with the additional dimension accounting for missing values. The set of embeddings for all categorical features is denoted by Es = (eo, (æ1), Gøgy (x2),..,, eo, (xz)]- Thus, the embedding for the encoded value Ij(eat) = l E [0, 1.,.... d,] is equal to es, (Z)((a))) = [co, w Woy Co, E RR denotes a unique identifier that distinguishes the classes from different columns. Wøxs E Rd and the dimension, a, is a tuneable prper-paraaettr just like the number of neurons in fully connected dense layers. To account for interpretability of the categorical variables, multiple adaptations are possible. Appending [CLs) token to the column embodding and fceding the [CLS] into the shape function f(oat) similarly to Gorishniy et al. (2021), or leveraging sequential attention as done by Arik and Pfister (2021) could ensure interpretability and leave the overall model structure intact, without loss of generalizability performance.
layers. An embedding lookup
Note, that depending on the data structures, one could learn different feature embeddings and fit separate shape functions for different categorical features. However, as visually interpreting categorical features is not sufficiently meaningful, fitting one shape function for all categorical variables suffices. Additionally, jointly learning& a single embedding vector for all categorical features, captures all possible interaction effects between all categorical variables while maintaining interpretable. Adapting for pairwise feature interactions between
5

---

Published in Transactions on Machine Learning Research (05/2024)
the continuous features as well the continuous and categorical features represented in (2) would lead to:
( ()) Bo + Σ (F(ttoon)t + (H
E J.b0
(1)(oo)) Fk(cons))
Heatt) ((oon))) (Fea)())))))),
(4)
where felatt accounts for the feature interaction between continuous feature and all categorical features. Note, that the input dimension of f(oor) is increased by 1, as the transformer output and the contimuous feature are concatenated to form the input vector of the shape function.
3.1 Interpretability
One of the key advantages of GAMs is that the learned shape functions can be easily visualized. NATT inherits this feature from GAMs and the shape functions for all numerical variables are interpretable (see Figures 2, 6). For real world data, NATT can accurately detect e.g. jumps in Longitude and Latitude for rental prices in Amsterdam near the city center (sce Figure 3).
Visual interpretation of categorical features, however often not very informative. One-hot encoding can lead an excessive amount of shape functions. Especially for large tabular datasets with lot of categorical features. One-hot encoding would require GAMs to fit cach category with separate shape function (e.g., Radenovic et al., 2022;
0.100
0.075
Agarwal et al., 2021; Enouen and Liu, 2022). That leads to an increased amount of shape functions, that firstly need to be modeled and secondly need to be analyzed for interpretation. Label encoding can lead to ordinal interpretations in categories where no natural ordering is present thus impeding the intelligibility of the model. Target encoding can lead to problematic interpretalions, especially for categories with very few observations, as the encoding for these categories may be unreliable or unstable.
aangibute
0.050
0.050
-0.75
Figure 2: Shape function learned by NATT to predict rental prices in Amsterdan. The rental prices increase the city center, depicted in the longitude graph.
married
47.77%
Married-ci--pouse 58.55%
14.35%
0.130
7.10%
0.125
Divorced
7.60%
22.98%
0.3
native country
marital status
relationship
Separated
2.00%
5.76%
0.115
@ccupati
workclass
gender
Married- seiteedd-ppus---aee
1.85%
0.110
aace
Married-AF-spouse 0.03%
0.19%
0.105
Male
Female
0.100
Figure 3: NATT Attention scores for the features Marital Status and Gender for the Adult dataset. notable disparity exists between genders in the im- portance of the categorical attribute "Never-married" for income prediction in the model.
Figure 4: Categorical feature importances for the Adult dataset retrieved from attention layers. The attention weights for the [CLS] token extracted detailed in Formula 5, See the supplemental material for details.

---

Published in Transactions on Machine Learning Research (05/2024)
NATT can leverage multiple interpretable transformer structures to account for interpretable categorical features and remain fully intelligible across continuous features. Using [CLS] tokens, NATT can use the attention maps to weight the average attention scores of the [CLS] token with the overall importance of the encoded calegorical leatures. The importance scores for the continuous features, as well as for all the combined categorical features, can be obtained due to the additivity constraint following similar method as described in Agarwal al. (2021).
Male
Female
Divorced 15.23%
2.77%
8.25%
7.39%
7.62%
21.46%
13.09%
18.81%
18. 16%
24.15%
0.5
Married-Ar -a-dd-A--ggg AF-spouse
0.24%
0.01%
0.56%
0.22%
Married-civ-spee civus-
44.07%
57.65%
40.72%
46.84%
60.35%
12.85%
23.79%
9.93%
16.57%
14.74%
0.4
Married-ppouee
Married spouse absent
3.41%
5.02%
1.64%
2.34%
0.85%
4.79%
1.89%
1.78%
0.3
Never-married
35.12%
32.30%
42.22%
39.97%
28.43%
47.54%
49.36%
50.73%
49.79%
47.18%
0.2
Separated
1.15%
1.35%
5.30%
3.45%
1.74%
7.97%
1.81%
12.33%
11.38%
4.69%
0.1
Widowed
1.01%
0.93%
1.64%
1.00%
10.18%
6.39%
6.30%
4.11%
7.25%
Amerindian- Eskimo Eskino
Asian- Paan-PPac--a Saian-Pac-lslandee -eltssldderr-
Black-
Other
White
Asian-Pat: Pacllaar--
Black
Other-
White
Amerindian-Eskim
Figure 5: NATT attention scores for the features Marital Status, Race and Gender for the Adult dataset. The artributes "white" and 'divorced' are significantly more important to the model's prediction a person's 's income for females compared to males.
Thus, the importance for the categorical fcatures over n samples and L attention heads is given by:
I = ((oa))
Σ Pi.ha h.l
Hheads x L
where W(eae) E [0, 1] is the importance of all categorical variables, h is the running index of the attention heads, 1 denotes the 1-th layer and denotes the i-th sample. Pe,Aa thus denotes the h-th attention map for the [CLs] token from the forward pass of the 1-th layer for sample Note, that since we are dealing with tabular data, the index or token position of a variable is fixed over all samples. Hence the models attention to specific index of the input is equal to the models attention to that variable. Averaging the attention for every index over all samples and weighting with the overall categorical importance thus returns the average importance scores for every category.
Additionally, feature interactions between the categorical variables are always implicitly modeled. Attention scores for all possible categorical combinations can be extracted by summing not over all samples in Equation (5) but by summing only over the samples where the interaction of interest appears. Pairwise categorical feature interactions can thus easily be visualized (see Figure 3). Even higher order feature interactions can be subtracted by filtering the corresponding samples respectively (see Figure 5).
It is important to note, that attention weights are not to be equated with importance (Jain and Wallace, 2019). However, Brunner el al. (2019) found attention weights to be meaningful and identifiable for shorter sequences. Since the present approach handles tabular data, the sequences are by nature most often comparably short and more specifically shorter than the attention head dimension. While Jain and Wallace (2019) questions the information from attention weights altogether, Wiegreffe and Pinter (2019) provides evidence that challenges the blanket dismissal of attention weights and demonstrates their utility in several contexts. Furthermore,
7

---

Published in Transactions on Machine Learning Research (05/2024)
the work by Gorishniy et al. (2022) is particularly relevant to approach, since it leverages attention weights in tabular setting. By comparing tabular attention scores with established methods like Integrated Gradients (Sundararajan et al., 2017), Gorishmiy et al. (2022) finds consistently high correlation between attention maps and permutation test's feature importances. Our conducted simulation study (section 4.1) additionally showcases the robust of the atlention weights (Figure 6).
4
Experiments
4.1
Simulation study
For small simulation study, we simulate 5000 observations with 2 continuous variables and categorical variables (see Supplemental Material 7.5 for details). We train 100 models for 100 epochs each and visualize the learned shape functions for the continuous features in Figure 2a-2b. Both data generating functions are adequately captured by NATT.
0.45
04-
02-
622
-14
02
Bodban
Degioe
Oby
Multsi_Chees
(a) (X1)
(b) fx(X2)
(c) Categorical
Figure 6: Shape functions and categorical feature importance learned by NATT for the simulation study, 100 models are and each trained model visualized in red. The data generating function visualized black.
To test whether continuous feature interactions adequately recognized, we the same data generating process but add (X1) x f2(X2) to y. We visualize random draw from multiple model fits and find that the complicated interaction patterns are adequately disentangled by NATT.
0.700
3.75
0.700
0.525
330
3.50
0.350
0.350
325
0.175
Pediction
3.25
0.175
Rredictions
2.75
-0.175
2.75
-0.175
-0.350
-0.525
2.25
2.25
-0.700
Figure 7: True feature interaction effect
Figure 8: NATT predicted feature interaction effect.
For categorical feature importance, our simulation consistently yields robust estimations (see Figure 2c). We find an average rank correlation of 0.76 between the true importance and the estimated importance scores over 100 fits which is comparable XGboost with 0.79. Additionally, we compute the Kendall Tau score for demonstrating that NATT accurately detects the correct ordering of feature importance, compared to the

---

Published in Transactions on Machine Learning Research (05/2024)
Mean MSE with Standard Deviations
NAM
1.2
NATT
1.0
0.8
MSE
0.6
0.4
0.2
0.0
2
3
7
8
Number categorical variables
Figure 9: Performance of NATT, NAM and NAM that jointly models all categorical variables in single MLP. With an increasing number of calegorical fealures, NATT starts to outperform both NAM architectures.
overall effect the categorical features have the dependent variable. We find a Kendall Tau score of 0.6 (compared to 0.0 for random ordering) and an accuracy of 65% for the ordering.
4.1.1 Ablation study
To further analyze NATT's capabilities, conduct a small ablation study. We simulate simple regression task dataset with 10 categorical variables and 3 numerical variables. The target variable is dependent on all variables through single effects as well as randomized interaction effects. For comparative analysis, we perform 5-fold cross-validatinn and train a NAM, NATT, and a NAM model that jointly models all categorical features a single MT.P. Note that this architecture loses the interpretability categorical features altogether. For network architectures, (64, 32, 32] neurons, ReLU activation, and 0.1 dropout after each layer. We use the same architecture for all models and employ an embedding size of 64 for NATT well as 4 transformer blocks. We start with a learning rate of 1-03 and implement learning rate decay with a patience of 15 epochs and carly stopping after 25 epochs of no improvement in the validation loss. All results are achieved with the models' best-performing weights on the validation dataset.
orient ourselves on Radenovic al. (2022) and use single feature nets with
Since the NAM and NATT architoctures are identical when there are no categorical fcatures available, g( [y)) 30 Σ (f(oot))))) we analyze the effect of the number of categorical features modeled. Figure 9 shows that the more categorical features NAM structures. Similar to Gorishniy et al. (2022), we find that the Transformer architecture can increase performance in tabular problems. Note that for fewer categorical features, all models perform within their standard deviation bands as expected due to their similarity in architecture.
available, the better NATT performs compared to classical
4.2 Datasets
We validate the effectiveness of our model on 8 machine learning benchmark datasets for both classification and regression. We perform 5-fold cross validation on all datasets and report the average performance well the standard deviations. For the classification tasks we report the Area under the curve (AUC). For

---

Published in Transactions on Machine Learning Research (05/2024)
the regression tasks report the root mean squared error (RMSE). Preprocessing is performed done by Agarwal et al. (2021). We use the Vanilla implementation of NATT as described in Section 3 and thus even leave room for improvement by adapting more refined architectures (Gorishniy et al., 2021; Arik and Pfister, 2021). See the supplemental material for improved performances when leveraging [CLS] tokens during prediction.
Classification datasets. We report performance on the Adult dataset for predicting a persons income (Kohavi et al., 1996), the Titanic dataset retrieved from Kaggle, for predicting the survival of titanic passengers, the Churn dataset retricved from Kaggle, covering whether a customer left bank or not, and the Insurance dataset.
Regression datasets. We report performances on another Insurance dataset (Lantz, 2019), AirBnB datasets with data from the cities of Munich and Amsterdam. Lastly we include the Abalone dataset. retrieved from the UCI (Dua and Graff, 2017) a dataset with only single categorical variable and 3 categories. See the supplemental material for information about the datasets.
4.3 Baselines
We train the following state-of-the-at the-art inter- pretable models as well as state-of-tho-art black- bor models: Gradient Boosted Trees (XG- Boost): Popular decision tree based gradient boosting, often outperforming DNNs on tabu- lar data. We use the implementation provided by Chen and Guestrin (2016).
Table Comparison between NATT and a NAM for 4 classification and regression datasets. The evaluation metrics AUC and RMSE. We express the performance improvement of the NATT over the NAM as percentage. The results are based on 5-fold cross-validation.
Deep Neural Network (DNN): Unre- stricted fully connected deep noural network trained with either a mean squared error loss function (regression) or cross entropy (classifi- cation).
Dataset
NAM
NATT
Gain (%)
Classification
AUC 1
Titanic
84.9
86.5
1.9
Tabular Transformer Networks (Tab- Transformer): A tabular transformer network followed by fully connected MLP as intro- duced by Huang et al. (2020).
Adult
91.0
91.4
0.4
Insurance
91.4
91.6
0.2
Churn
85.1
85.2
0.1
Dataset
Neural Additive Models (NAMs): Linear combination of DNNs as described in Equa- tion (1) and presented by Agarwal et al. (2021). Note that 2s do not scale to account for all feature interactions (Dubey
Baseline NAM
NATT
Gain (%)
Regression
RMSE
Insurance
0.208
0.191
8.2
al., 2022). Pair- wise interactions are thus only implemented between the numerical features.
Munich
0.060
0.052
13.3
Amsterdam
0.042
0.037
11.9
Abalone
2.25
2.24
0.4
Explainable Boosting Machines (EBMs): State-of-the-art Generalized Additive Models leveraging shallow boosted trees (Nori et al., 2019).
Average Increase:
4.9
Neural Generalized Additive Model (NodeGAM). State-of-the-art Generalized Additive Models leveraging Neural Oblivious Decision Trees (Chang et al., 2021).
For neural network approaches, we take the same hyperparameters across all methods to provide a more consistent comparison and orientate on the benchmarks performed by Radenovic et al. (2022). For NodeGAM and EBM we use default values. All chosen hyperparameter settings allow to recover or even exceed performances reported in the literature (Chang al., 2021: Huang al., 2020; Thielmann al., 2024).
4.4 Results
10

---

Published in Transactions on Machine Learning Research (05/2024)
NATT outperforms NAMs all datasets shown in Table 1. We use the baseline model architecture for NATT as well as NAMs and do not account for feature interactions. Additionally, we demonstrate that NATT consistently outperform NAMs while remaining more parameter-sparse when using similar NAM ar- chitectures to Radenovic et al. (2022) and using much parameter sparser architectures than Agarwal et al. (2021). In our experiments, we decrease the amount of parameters in NATT in comparison to NAMs as defined by Radenovic et al. (2022); Dubey et al. (2022); Agarwal et al. (2021) by more than 40% (see Table 7). We use identical architectures for all numerical features for NAMs and NATT. The parameter difference can thus be attributed to the way the categorical features are modeled in the NAM.
Table 2: Average Rank table for interpretable models over all datascts. NATT and NA perform best considering other interpretable methods.
Model
avg.
Rank
NATT
1.1
NAM
3.5
NodeGAM
2.4
EBM
3.0
NA
1.6
NA
3.6
NodeGA
2.6
E
2.1
Table 3: Comparison of 4 regression tasks for different interpretable well as black-box models. The results are achieved with 5-fold cross validation. The average russe values as well standard deviations are reported. Best results per model type are marked in bold. The 2 denote pairwise feature interactions.
RMSE 4
Type Model
Insurance
Munich
Amsterdam
Abalone
XGBoost
0.165 (+0.008)
0.053
(#0.016)
0.039
(=0.014)
2.30 (#0.05)
Blackbox
DNN
0.183 (10.0144)
0.110
(10.045)
0.076 (10.008)
2.13 (10.06)
Tab Transfoforrer
0.212 (#0.013)
0.049
(#0.10)
0.038
(+0.014)
2.56 (#0.10)
NAM
0.208 (40.015)
0.060
(20.016)
0.042
(=0.014)
2.25 (#0.09)
Additive
NodeGAM
0.194 (10.040)
0.054
(10.017)
0.038
(10.015)
2.25 (10.08)
Models
EBM
0.194 (#0.004)
0.053
(#0.017)
0.041
(=0.014)
2.27 (#0.06)
NATT
0.191 (#00001)
0.052
(=0.017)
0.037
(::0014)
2.24
(:0008)
Pairwise
N
0.205 (#0.016)
0.056
(#0.015)
0.042
(=0.013)
2.11 (#0.05)
Feature
NodeGA2M
0.194 (10.040)
0.051
(10.017)
0.037
(10.015)
2.13
(10.05)
Interactions
BºBM
0. 0.145 (100004)
0.048
(I00017)
0.041
(=0.014)
2.22
(10.05)
NA
0.149 (1.0.009)
0.047 (=0.019)
0.036 (10.015)
2.10
(1:0.06
Tables 3 and 4 show the results of all models over all datasets, divided into three calegories: Black-box models, interpretable additive models, and interpretable additive models accounting for pairwise feature interactions. The best model of each category is marked bold. Notably, the baseline NATT version performs comparably black-bor XGBoost model and outperforms fully connected Deep Neural Network on almost all datasets. Moreover, NATT allows for better scaling when in- cluding feature interactions than NAMs. For in- stance, building feature interactions for all possible pairwise feature interactions on the Adult dataset when using one-hot encoding would result more than 4500 shape functions in classical NAM ap- proaches. Thus, fitting NA for such datasets would require additional feature interaction se- lection step (Enouen and Liu, 2022). Even shared basis functions across all shape functions as proposed by Radenovic et al. (2022) may encounter recursion problems for larger datascts. Therefore, the results presented for ? in this section only consider feature interactions between all numerical features.
Table Number of Parameters for NAMs vs. NATT
Classification
Model
Adult
Titanic
Insur.
Churn
NAM NATT
127K 110K
102K
127K
111K
79K
110K
68K
Regression
Model
Insur.
Munich
Amst.
Abalone
NAM NATT
96K
325K
299K
86K
66K
106K
106K
92K
11

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: Comparison of 4 classification tasks for different interpretable as well as black-box models. The results are achieved with 5-fold cross validation. The average AUC values as well as standard deviations are reported. Best results per model type are marked in bold. The denote pairwise feature interactions.
A
Type
Model
Adult
Titanic
Insurance
Churn
XGBoost
92.9 (+0.5)
85.6 (#4.6)
92.8 (+0.3)
84.6 (#1.6)
Blackbox
DNN
90.6 (10.5)
84.1 (1:8.0
90.5 (10.4)
83.7 (+1.3)
TabTransformer
91.0 (10.5)
86.1 (44.0)
91.1 (#0.4)
84.6 (#1.2)
NAM
91.1 (10.3)
84.9 (14.3)
91.4 (10.4)
85.1 (11.1)
Additive
NodeGAM
91.5 (40.4)
82.8 (#8.6)
91.6 (+0.5)
85.1 (#1.4)
Models
EBM
90.9 (10.5)
86.2 (43.5)
91.1 (40.4)
85.0 (+0.9)
NATT
91.4 (+0.3)
86.5 (+3.7)
91.6 (+0.3)
85.2 (+1.4)
Pairwise
N
91.4 (+0.3)
85.8 (#4.0)
91.4 (#0.5)
85.9 (#1.4)
Fealure
NodeGA
91.6 (+0.3)
84.9 (1:2.0
91.7 (#:04)
86.6 (#1.5)
Interactions
E'BM
91.9 (40.3)
86.8 (1:4.6
92.0
(10.4)
86.1 (#144
NATTT
91.5 (10.3)
87.0 (14.0)
91.6 (10.3)
86.9 (11.6)
Overall, we can validate the findings from Huang et al. (2020) and Gorishniy et al. (2022): Incorporating transformer embeddings when analyzing tabular data can improve the overall model prediction. However, we still find that XGBoost slightly outperforms Tab'rrasffomerr by slight margin of 5:3 on the benchmark datasets. Among all interpretable models, i.c., NATTs, NAMs, NodeGAMs and EBMs, NATT performs best in its base version achieving the lowest overall average rank over all datasets (see Table 2). When accounting for urature interactions, our method (NA?TT) and boosting E²BM improve in performance, while others benefit comparably less.
5 Conclusion
In this paper, we present. NATT, a novel model architecture that offers interpretability. Our experiments demonstrate that NATT outperforms other neural interpretable methods across various datasets. While the additivity constraint ensures easy interpretation, this interpretation comes at a price. Throughout all interpretable models, we experience performance trade-off in terms of predictive power. Full Mock-bor models are still more performant than their glass-box counterparts. However, we demonstrate that NATT is a further step in the direction of closing that gap. Additionally, we find that when accounting for feature interactions, NA?TT as well as EºBM and NodeGA are on par with black-bor models. Throughout our benchmarks, we find that NATT only performs around 0.4% worse than the best black-bor benchmark models. It is important to note that NATT's architecture allows further adaptations especially in the realms of modeling interactions. One potential approach is to leverage the trained transformer embeddings and train pairwise tensor-product feature interactions on the residuals using the second-to- lastat output layer of the shape functions. This strategy has the potential to grcatly improve the performance of NATT, as it would allow for the joint learning and pre-training of the embeddings, leveraging the findings from Gorishniy et al. (2022).
6 Limitations and Future Work
A primary limitation of the current work is its focus on tabular data. Extending the presented approach to account for high dimensional input data such as documents or images is a key direction for further integrating interpretability inlo the domain of Deep Learning. By leveraging the transformer architecture for tabular features, NATT demonstrates the possibilities of integrating different network structures and shape functions into single modeling framework. Through multiple experiments the generalizability of the structure is demonstrated. The interpretability of NATT, while being much more interpretable than black-boa models, still lacks the inherent statistical inference of classical GAMs. Adaptations to include significance statistics
12

---

Published in Transactions on Machine Learning Research (05/2024)
could be a further step towards Deep Learning models being deployed in high risk domains. Accounting for the underlying data distributions can already be accounted for, by extending NATT to account for all distributional parameters, as similarly done by Thielmann et al. (2024) (see Supplemental Material 7.1).
Acknowledgements
Funding by the Deutsche Forschungsgemminnccaaft (DFG, German Research Foundation) within project 450330162 is gratefully acknowledged.
References
Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., and Hinton, G. E. (2021).
Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699-4711
Arik, S. Ó. and Pfister, T. (2021). Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 3679-6687.
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R.,, and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 1017)100330
Bouthillier, X., Delaunay, P., Bronzi, M., Trofimov, A., Nichyporuk, B., Szeto, J., Mohammadi Sepahvand, N., Raff, E., Madan, K., Voleti, V., et al. (2021). Accounting for variance machine lcarning benchmarks. Proceedings of Machine Learning and Systems, 3:747 769.
Brumner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., and Wattenhofer, R. (2019). On ident ifiabiiiyy in transformers. arXiv preprint arxiv:1908. 04211.
Chang, C.-H., Caruana, R., and Goldenberg, A. (2021). Node-gam: Neural generalized additive model for interpretable deep learning. arXiv preprint arXiv:2106.0161:
Chen, T. and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16, pages 785 785-794, New York, NY, USA. ACM.
Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, 1., Zhou, T., ct al. (2015). Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4):1-4.
Waal, A. and du Toit, J. V. (2007). Generalized additive models from a neural network perspective. Seventh IEEE International Conference on Data Mining Workshops (ICDMW 2007), pages 263 (60-270. IEEE.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXio preprint arxiv:0010. 11929.
Dua, D. and Graff, C. (2017). UCI machine learning repository.
Duboy, A., Radenovic, F., and Mahajan, D. (2022). Scalable interprotability via polynomials. arXiv preprint arXiv:2205.14108.
Enouen, J. and Liu, Y. (2022). Sparse interaction additive networks via feature interaction detection and sparse selection. arXiv preprint ar Xix:22200 09326.
Gorishniy, Y., Rubachev, I., and Babenko, A. (2022). On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991 25004.
Gorishniy, Y., Rubachev, L. Khrulkov, V., and Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, $4:18932-18844
13

---

Published in Transactions on Machine Learning Research (05/2024)
Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? Advances in nearal information processing systems, 35:507-522 -200
Hastie, T. J. (2017). Generalized additive models. In Statistical models in S, pages 249 307. Routledge.
Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. (2022). Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01844
Horel, E. and Giesecke, K. (2020). Significance tests for neural networks. Journal of Machine Learning Research, (122)):1-29.
Huang, X., Khetan, A., Cvitkovic, M., and Karnin, (2020). Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012. 06678.
Huang, Y., Giledereli, B., Köksal, A., Ozgür, A., and Ozkirimli, E. (2021). Balancing methods for multi-label text classification with long-tailed class distribution. arXiv preprint orXiv:2109.04712.
Jain, S. and Wallace, B. C. (2019). Attention is not explanation. arXie preprint orXiv: 10186.
Ke, G., Meng, Q.: Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. (2017). Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30.
Kim, M., Choi, H.-S., and Kim, J. (2022). Higher-order neural additive models: An interpretable machine learning model with feature interactions. arXiv preprint arXiv:2209.15400
Kohavi, R. et al. (1996). Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pages 202-201 207.
Lantz, B. (2019). Machine leorning with R: expert techniques for predictive modeling. Packt publishing Itd.
Lin, Y., Meng, Y., Sun, X., Han, Q., Kuang, K., Li, J., and Wu, F. (2021). Bertgen: Transductive text classification by combining gen and bert. arXiv preprint (tt::2155.0799
Luber, M., Thielmann, A., and Säfken, B. (2023). Structural neural additive models: Enhanced interpretable machine learning. arXiv preprint ar 230.000009
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., and Sun, C. (2021). Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34:14200 14213.
Nori, H., Jenkins, S., Koch, P., and Caruana, R. (2019). Interpretml: A unified framework for machine lcarning interpretability. arXie preprint arxiv:1099 19020229..
Popov, S., Morozov, S., and Babenko, A. (2019). Neural oblivious decision ensembles for deep learning tabular data. arXiv preprint 1xXix:1099.66
Potts, W. J. (1999). Generalized additive neural networks. In Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and dala miniing, pages 194 200.
Radenovic, F., Dubey, A., and Mahajan, D. (2022). Noural basis modcls for interpretability. arXiv preprint arXiv:2205.14190.
Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
Rügamer, D.,, Kolb, C., Fritz, C., Pfisterer, F., Kopper, P., Bischl, B., Shen, R., Bukas, C., Thalmeier, D., Baumann, P., et al. (2021). Deepregression: flexible neural network framework for semi-structured deep distributional regression. arXiv preprint arXiv:2104.02n0
Rügamer, D., Kolb, C., and Klein, N. (2023). Semi-structured distributional regression. The Americon Statistician, pages 1-12.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Shapley, L. (1953). Quota solutions op n-person gamesl. Edited by Emil Artin and Marston Morse, page 343. Sundararajan, M. and Najmi, A. (2020). The many shapley values for model explanation. In International PMLR.
conference on machine learning, pages
Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International P.M.R.
conference on machine learning, pages
Thielmann, A. F., Kruse, R.-M., Kneib, T., and Säfken, B. (2024). Neural additive models for location scale and shape: A framework for interpretable neural regression beyond the mean. In International Conference on Artificial Intelligence and Statistics, pages 1783 1791. PMLR.
Tsang, M., Liu, H., Purushotham, S., Murali, P., and Liu, Y. (2018). Neural interaction transparency (nit): Disentangling learned interactions for improved interpretability. Advances in Neural Information Processing Systems, 31.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, (2017). Attention all you need. Advances in neural information processing systems, 30.
Wang, T., Yang, J., Li, Y., and Wang. B. (2021). Partially interpretable estimators (pie): black-box-refmeed interpretable machine learning. arXiv preprint arXie:2105.02. 024100.
Wiegreffe, S. and Pinter, Y. (2019). Attention is not not explanation. arXiv preprint arXic:1908.04626
Yu, J., Wang, 7.,, Vasudevan, V., Yeung, L.., Seyedhosseini, M., and Wu, Y. (2022). Coca: Contrastive captioners are image-text foundation models. ar preprint arXi::2205.0199 019917
Zeng, A., Chen, M. Zhang, L., and Xu, Q. (2022). Are transformers effective for time series forecasting? arXiv preprint 45550:200511504
Zhou, T., Ma, Z., Wen, Q, Sun, L., Yao, T., Jin, R., et al. (2022). Film: Frequency improved legendre memory model for long-term time series forecasting, arXiv preprint ar Xrx:2220 08897.
15

---

Published in Transactions on Machine Learning Research (05/2024)
7 Supplemental Material for NATT
7.1 Beyond the mean: Location Scale and Shape
NATT is casily adaptable to account for arbitrarily many distributional parameters. Given a distribution with distributional parameters, a small architecture adaptation lets NATT become Neural Additive Tabular Transformer Networks for Location Scale and Shape (NATTLSS). Similar NAMLSS (Thielmann et al., 2024), we let each shape function have K-dimensional output, with output k 1,2, ...,, K accounting for distributional parameter k.
The loss function gets adapted from e.g. the mean squared error, to the corresponding negative log-likelihood, which is dependent on the distributional parameters. Hence, NATTLSS minimizes log (C(0ly)). Possible parameter interactions between the k- k-parameters are accounted for in the feature dependent shape-functions Each distributional parameter hence modeled by:
((0())= = =b0 + Σ
+ [(ca)) (H(
55)
where the superscript. (k) denotes the k-th distri- butional parameter. h(.) thus denotes a parameter specific output layer activation or link function. E.g. Softplus activation to account for the variance parameter in a normal distribution as the variance must be positive. 3(x) denotes the parameter spe- cific intercept. ,/4) RR + R denote the parameter- feature shape functions for the continuous features and [(aa)) [ddxx++++1) - IR denotes the parameter- feature shape functions for the categorical features. (k)) denotes the transformer layers, which are distri- butional parameter-spocific. Leveraging pre-trained transformer networks, could allow to use global H(-) over all parameters, however, jointly learning the feature embeddings for each distributional param- eter is more straight- forward architecture. Figure 10 demonstrates the advantages that NATTLSS can have over simple NAM approach. While the over- all predictive power might be similar, NATTLSS is much more faithful to the underlying data distribu- tion and also accounts for the aleatoric uncertainty in the data.
Normal Di.iiodion
03
Figure 10: NATTLSS prediction vs. the prediction of a simple MLP for normally distributed respouse variable.
Similar Thielmann al. (2024) we could adapt the network architecture, such that each shape function has a k-dimensional output layer, f(a)) : [ddxx+++11) + R* and 1/4) R + IR. The overall model output would thus also be k-dimensional, with the k-th dimension accounting for the k-th distributional parameter. Thus, the adaptation to location scale and shape can be achieved with the same amount of shape functions as the mere mean prediction.
7.2 Datasets
We evaluated NATT's performance on multiple datasets, all having varying characteristics. NATT also perform well on smaller datasets, a property which could be furthermore increased by leveraging pre-trained networks for the transformer parts and fine-tuning the shape function, S(cat)- For datascts with fewer categorical features, e.g. Abalone and Churn, the performance differences between NAM and NATT are not large for datasets with more categories (Munich, Amsterdam). Surprisingly, the differences for the
16

---

Published in Transactions on Machine Learning Research (05/2024)
Adult dataset with the most categorical features also comparably small, which can be attributed to the overall smaller effect the categorical variables have on the target variable.
Table 6: Statistics of the benchmark datasets.
Dataset
No. Samples
No. Features
No. Categorical
No. Categories
Task
Insurance
1338
6
Regression
Abalone
4177
8
Regression
Mumich
4568
10
Regression
Amsterdam
6998
10
2
26
Regression
Adult
48842
102
Classification
Churn
10000
10
2
Classification
Titanic
627
10
19
Classification
Insurance
32561
13
8
102
Classification
7.2.1 Preprocessing
We implement the samne preprocessing for all used datasets and only adjust to specifically optimize it for different model architectures. We standard normalize the target variables for the regression problems. We closely follow Gorishniy et al. (2021) in their preprocessing steps and use the preprocessing also implemented by Agarwal et al. (2021). All numerical variables are scaled between -1 and 1. In contrast to Gorishniy et al. (2021) we do not implement quantile smoothing, as one of the biggest advantages of neural models is the capability to model jagged shape functions. Where needed, the categorical features are one-hot encoded (e.g. not for Tab Traansfoorer NATT). We use 5-fold cross-validation and report mean results as well as the standard deviations over the folds.
7.3 NATT: Further Results
The reported benchmarks in section 4.3 for NATT are all performed with the Vanilla NATT architecture. Leaving the overall model structure intact, appending the [CI.)] tokens to the column embedding and subsequently only feed the [CLS] tokens into the shape function fleat) leads to very similar, and on average even a little bit better results. Therefore, by obtaining interpretable attention maps from the categorical features,
further enhance performance.
Table 7: NATT performances when appending [CLS] tokens to the column embeddings and using the [CLS] tokens as inputs for the shape function f(oat)-
Classification
Adult
Titanic
Insur.
Churn
NATT
91.4 (10.3
87.3 (13.5)
91.6 (10.3)
85.1 (11.2)
Regression
Insur.
Munich
Amst.
Abalone
NATT 0.191 (40.011) 0.048 (10.017) 0.037
(10.013)
2.21 (10.07)
7.4 Hyperparameters
As outlined in section 4.3, we adopted consistent hyperparameters across all methods to ensure a fair comparison, drawing upon the benchmarks established by Radenovic et al. (2022) and Dubey et al. (2022) for our settings. For NodeGAM, EBM and XGBoost, we relied on default values, only fixing the number of interaction terms for EBM the respective model class (EBM vs. EB² For NAMs, we followed Radenovic al. (2022). Importantly, our chosen hyperparameters allow us to match or surpass performance metrics reported in prior literature (Chang al., 2021; Huang et al., 2020; Thielmann et al., 2024) that involved
17

---

Published in Transactions on Machine Learning Research (05/2024)
extensive hyperparameter optimization. Additionally, echo the sentiments of Bouthillier et al. (2021)
regarding the complexities of comparing models with tuned hyperparameters.
Further supporting our approach, Grinsztajn et al. (2022) conducted extensive hyperparameter tuning across various tree-based and deep learning tabular models. Their findings indicated that while tuning generally benefits Transformer models more, the variability (large standard deviations) performance across different. hyperparameter sets was significant. Crucially, their work suggests that hyperparameter tuning does not consistently alter the relative performance rankings of models compared to evaluations conducted without tuning. This insight aligns with our focus on interpretability and the practicality of models for everyday users, where simpler, less parameter-intensvee models are preferable to avoid the risks of overfitting inherent in tabular datasets with limited sizes.
Over iyporparmmetrr Tining vidalee
Refformance Over ilyperprraeets Tining PMGS
Out dor
Oa typeraamestts
2.4
2.3
01-
Taming Rontor
Taiing Toaes
Figure 11: Optuna Hyperparameter tuning iterations for the NATT architecture for the Abalone dataset. The red line indicates our chosen hyperparameter setting. With more exhaustive hyperparameter tuning for benchmarks, the performance for NATT, due to the Transformer architectures could be further improved. Note, that the large deviations on the left all stem from larger architectures that tend to overfit for limited tabular datasets.
Figure 12: Optuna Hyperparameter tuning iterations for the NAM architecture for the Abalone dataset. The red line indicates chosen hyperparameter set- ting. our findings support the results from Radenovic et al. (2022). Simple architectures in the NAM archi- tecture are often sufficient and perform on par with larger architecture. Note, that the large deviations on the left all stem from larger architectures that tend to overfit for limited tabular datasets.
7.5 Data generating process for simulation study
In this data generation process, aim to create synthetic dataset with both categorical and continuous variables. The dataset consists of a target variable y, two continuous variables X1 and X2, and four categorical variables Categorical,, Categorical,, Categoricals, and Categorical4. We generate dataset with total of 5000 data points. The continuous features are simulated as:
f(() = 5iin(X)) (X1)
5
f2(X1) = 2(X2 - 3)
Categorical features are simulated by assigning categorical expression to each sample. Each expression corresponds to specific numerical value. Consequently, we can assess the importance of each categorical feature based on its average impact on the dependent variable, y. A higher numerical difference between categorical expressions, along with a greater numerical effect on y, indicates a higher overall feature importance.
1.5 if = A
Categorical
=
-155 if = B
0.0
if C
18

---

Published in Transactions on Machine Learning Research (05/2024)
0.0
if Yes
Categoricaly
-0775
if No
0.75
if Maybe
0.0 if Miami
Categoricala
0.2 if NewY ork
0.2 if Chicago
1.0 if Bachelors
Categoricala
1.0 if Masters
0.0 if PhD
Thus, we randomly draw 5000 samples for each categorical variable, draw X1 and X2 from uniform distributions between the values 1 and 10 and and 4.
y = -((i) - f2((2) + Categorical + Calegoricala + Categoricals + Calegoricals
8 Attention scores for tabular transformer networks
In tabular Transformer model that incorporates [CSL] token alongside J variables, the sequence length is equal to the number of variables and due to the [CSL] token extends to J + 1. Consequently, for a single sequence (all variables), the attention weight matrix within a layer and for an atlention head h is of size (J [)) x (J+12 1). This matrix accounts for the attention weights across all variables and the [CSL] token. The attention weight directed towards specific position j in the input sequence, which could represent any of the J variables or the [CSL] token, is denoted as Ph,t." Thus, the importance of this position for the model's decision-making is expressed as AA..j- Note, that position in the input sequence of tabular data references to variable j of J variables. Given that only include the [CSL] token as the input for the downstream tasks, the attention scores of the [CSL] aggregate all necessary information. We can thus quantify the collective atlention it receives from all other tokens and hence variables. This is achieved by aggregating the attention weights directed towards the [CSL] token across various heads and layers. Hence, we can analyze A when the index of the [CLS] token, for all input samples and thus all contexts (feature interactions). Note, that we omitted the last index j in the main body of our paper and used Aq. Ai,ha for the i-th sample for better readability.
19


Reviews:
[{"review_id": "Hh5mChjDQd", "reviewer": "Reviewer_osxC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written and the method simple while outperforming NAMs and other baselines in all benchmarks. \nOf course, generally speaking, the interpretability is limited since still black-boxes are incooportated in the architecture - but this is the same for NAMs. \nFor me, as a kind of outsider of this particular machine learning field, it would be nice to understand why the authors are not simply scale the models to obtain better numbers. Is it because you start overfitting? If this is the case, and actually generally, I would like to see a performance plot showing performance with changes to the number of layers / width / number of paramterers - I think this would especially make sense on the Transformer side, as this is the novel part of the model. \nAlso to fairly examine the baselines / the proposed method, can you provide numbers on sensibility the hyperparameters of the proposed method / baselines. I guess you scan hyperparemters for all methods?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the Neural Additive Tabular Transformer Networks (NATT) which is a model that combines the strengths of additive neural networks for interpretability and Transformer models for handling categorical data. The method achieves high predictive performance.\n\nNATT outperforms other interpretable deep learning models and achieves performance levels similar to complex \"black box\" models. This makes it a valuable tool in areas where explainability is critical alongside accurate predictions. The method can incorporates feature interactions (pairwise or higher-order), further boosting its performance and making it competitive even with non-interpretable models.\nExperiments on diverse datasets demonstrate the superiority of NATT over interpretable baselines and its competitiveness with state-of-the-art tabular data models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Page 7, what is (see Figure 2c)? \n\nPlease provide all training details / architecture details.\nPlease report performance when changing hyperparameters / provide hyperparameter scan ranges for the method / baselines. \nPlease report  ablations: What happens if you exchange the Transformer with an e.g. MLP?\n\nIf you provide these analyses, I think the paper is fit for publication.", "broader_impact_concerns": "I think the paper can be of importance to this particular machine learning community. I can not really judge this.\nThe experiments seem quite toyish, but I can not judge this since I do not know the related work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SHUrf7B0FK", "reviewer": "Reviewer_B5UM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this paper is well written and I enjoyed reading it. \n\n## Strengths\n- **Clear Exposition and Writing**: I found the problem statement clear and well articulated in this work. Each section of the paper also flows from one to another. Figure 1 is quite helpful and clearly demonstrates the problem that this work is trying to solve.\n- **Comprehensive Experiments**: The experiments in this work clearly backup the claim made in the paper. The authors explore several datasets and clearly test across both previous additive models and black-box approaches too.\n- **Simple Solution**: The solution proposed here is simple and easy to implement, which means that it can immediately be used in practice without too much hassle. \n- It is quite nice that in some cases, the number of params for the NATT is smaller than the NAM, yet without substantial performance degradation.\n\n## Weaknesses\n- **Conceptual Clarity**: One challenge I have with the writing is that from the introduction, it still a bit unclear why it is a big deal to deal with categorical variable separately. Perhaps this point could be emphasized more in the current introduction.\n- **Attention Weights Identifiability**: There is a large literature on identifiability of attention weights, see: \"Attention is not explanation\", \"Attention is not not explanations\", and \"On identifiability of transformers\". Since you need the attention weights in your feature importance estimate, how do we know that it is a reliable estimate? \n- **Section 3.1**: Why is 'FIGURE LABEL' in red? \n- Can you justify the feature importance formula somewhere in the appendix? I am missing something simple here I think. Why sum over n samples? Isn't the feature shape a 'global' quantity? Perhaps I am misunderstanding what this formula is computing. Is it the feature importance of a single categorical variable? \n- This is a stylistic one: I think tables 4 and 5 should come before 2 and 3. Tables 2 and 3 are more about specific comparisons among interpretable architectures. While Table 4 and 5 are general.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a Neural additive transformer as an 'interpretable' alternative to transformer models for tabular datasets. This line of work is inspired by generalized additive models, which restrict the shape of the function that is learned per input feature. The key conceptual challenge that this paper tackles is that, in normal GAMs or other Neural additive models, you need a different shape function per feature or feature interaction. However, when one has a categorical variable, the number of features increases and using a single function to represent each category is prohibitive. To sidestep this issue, this paper proposes to use a transformer to embed these categorical embeds. A single shape function is then learned for each group of features. The paper presents several empirical findings that show that the proposed NAT architecture matches the performance of other additive models and even Xgboost on several datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses section above. None of the requested changes are disqualifying, but mostly require additional clarifications.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7o0HWIBDV0", "reviewer": "Reviewer_nMEX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is easy to follow overall. The proposed method seems reasonable.\n\nWeaknesses: Some parts of the paper are unclear. Section 3.1 seems not so interesting to me. For example, why do we want Figures 3 and 5 to be visualized? What can it be used for in your tasks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed Neural Additive Tabular Transformer that incorporates categorical feature embeddings into an additive model architecture. The advantage of this method is better interpretability. The authors also show better performance of this method compared with other baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I failed to find any discussion and illustration in Tables 4 and 5. Why do you divide the models into three parts?", "broader_impact_concerns": "No such concers.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews were quite positive from the beginning of the process, and the authors have successfully addressed the comments of the reviewers. I do not see any barrier for acceptance.
Decision: Accept (poster)