GRAND-SLAMIN’ Interpretable Additive Modeling with Structural Constraints
Authors: Shibal Ibrahim
Gabriel Isaac Afriat
Kayhan Behdin
Rahul Mazumder
Venue: NeurIPS 2023 poster (2023)
Keywords: Generalized additive models
component selection
hierarchy
interpretability
TL;DR: We propose novel scalable approaches for generalized additive models with interactions under structural constraints

Abstract: Generalized Additive Models (GAMs) are a family of flexible and interpretable models with old roots in statistics. GAMs are often used with pairwise interactions to improve model accuracy while still retaining flexibility and interpretability but lead to computational challenges as we are dealing with order of $p^2$ terms. It is desirable to restrict the number of components (i.e., encourage sparsity) for easier interpretability, and better computational and statistical properties. Earlier approaches, considering sparse pairwise interactions, have limited scalability, especially when imposing additional structural interpretability constraints. We propose a flexible GRAND-SLAMIN framework that can learn GAMs with interactions under sparsity and additional structural constraints in a differentiable end-to-end fashion. We customize first-order gradient-based optimization to perform sparse backpropagation to exploit sparsity in additive effects for any differentiable loss function in a GPU-compatible manner. Additionally, we establish novel non-asymptotic prediction bounds for our estimators with tree-based shape functions. Numerical experiments on real-world datasets show that our toolkit performs favorably in terms of performance, variable selection and scalability when compared with popular toolkits to fit GAMs with interactions. Our work expands the landscape of interpretable modeling while maintaining prediction accuracy competitive with non-interpretable black-box models. Our code is available at https://github.com/mazumder-lab/grandslamin.

GRAND-SLAMIN' e Additive Modeling with Structural Constraints
Shibal Ibrahim
Gabriel Isaac Afriat MIT Cambridge, MA afriatgomit.cc edu
Kayhan Behdin MIT Cambridge, MA behdink@mit.cmm edu
Rahul Mazumder MIT Cambridge, MA rahulmazomit t. .de.ua
MIT Cambridge, MA
shibal @mit edu
Abstract
Generalized Additive Models (GAMs) are family of flexible and interpretable models with old roots in statistics. GAMs are often used with pairwise interactions improve model accuracy while still retaining flexibility and interpretability but lead to computational challenges is desirable to restrict the number of components (i.e., encourage sparsity) for easier interpretability, and better computational and statistical properties. Earlier approaches, considering sparse pairwise interactions, have limited scalability, espe- cially when imposing additional structural interpretability constraints. We propose a flexible GRAND-SIAN -SAMIN framework that can learn GAMs with interactions un- der sparsity and additional structural constraints in a differentiable end-to-end fashion. We customize first-order gradient-basee based optimization to perform sparse backpropagation to exploit sparsity in additive effects for any differentiable loss function a GPU-compatible manner. Additionally, we establish novel non- asymptotic prediction bounds for estimators with tree-based shape functions. Numerical experiments on real-world datasets show that our toolkit performs favorably in terms of performance, variable selection and scalability when com- pared with popular toolkits to fit GAMs with interactions. Our work expands the landscape of interpretable modeling while maintaining prediction accuracy competitive with non-interpretable black-box models. Our code is available at https://gi githuu..tt. www/mazumdere.eeee 1ab/ babbgrandsslminn
we are dealing with order of p² terms. It
Introduction
Many state-of-the-an learners e.g., tree ensembles, neural networks, kernel support vector machines, can be difficult to interpret. There have been various efforts derive some post-training explainability
from these models -see [5] for a survey. Post-hoc explainability attempts to explain black-box prediction with interpretable instance-specific approximations e.g. LIME [49] and SHAP [35]. However, such approximations are known to be unstable [12, 28), expensive [51] and inaccurate [32]. Hence, it is desirable consider modeling approaches that are inherently interpretable.
Amongst classical approaches, there are some models that have inherent interpretability e.g., Linear models, CART [4] and Generalized Additive Models (GAMs) 14]. GAMs [15] which have old roots in statistics are considered a front-runner in the context of interpretable modeling. They consider an additive model of the main effects of the form: g (I) ΣÿE(r) fa,), where 2, denotes the jthh feature in input E IR,, fz is a univariate shape function and g denotes the link function that adapts the model to various settings such as regression or classification. GAMs are considered easy to interpret as the impact each feature can be understood function e.g., plotting f, (1z) Tj. However, such models often suffer in prediction performance when compared to black-box methods e.g., deep neural networks (DNNs). This can attributed in part the fact that GAMs do not consider interactions among covariates.
visualizing the corresponding shape
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

There has been some exciting work that aims to reduce this performance gap by considering GAMs with pairwise interactions [see, for example, 34, 60, 6, 46, 9, 10, and the references therein]. GAMs with pairwise interactions consider a model of the following form:
( = Σ f1,))) + JEMD
Σ Sna(I, 25)) (JA)Z
(I)
where f is bivariate shape function and I C (1,,), (p (p- 1,p)} denotes the set of all pairwise interactions. Under this model, fı²Ø)) is the j-main effect and [2,(z 2xx) is the (i, k)-th interaction effect. Pairwise interactions are considered interpretable function f can be visualized as a heatmap on an = #x-plane. Despite their appeal, GAMs with pairwise interactions pose several challenges: (i) Learning all pairwise interaction effects of the order of p² lead to computational and statistical challenges. (ii) Performing component selection such that only few of the components { and {fz,a} are nonzero in an end-to-end fashion (while training) is a hard combinatorial optimization problem. We remind the reader that component selection is needed to aid interpretability. (iii) Imposing structural constraints on the interaction effects, e-g.. hierarchy [42, 13, 7, 39, 3, 60, 10] makes the associated optimization task more complex.
each of the bivariate shape
In this paper, we introduce novel GRAND-SLAMIN framework that allows for flexible way to do component selection in GAMs with interactions under additional structural constraints in an end-to-end fashion. In particular, we introduce an alternative formulation of GAMs with interactions with additional binary variables. Next, we smooth these binary variables so that we can effectively learn these components via first-order methods smooth optimization (c.g, SGD). Our formulation appears to have an edge over existing methods in terms of (i) model flexibility for enhanced structural interpretability and (ii) computational efficiency. First, the binary variables allow us to impose in the model (a) component selection constraints and (b) additional structural constraints (e.g. hierarchy) via a unified optimization formulation. Both of constraints (a), (b) can aid interpretability, model compression, and result in faster inference and better statistical properties. Second. the our smoothing procedure for the binary variables allows us to have customized algorithms that exploit sparsity the forward and backward pass of the backpropagation algorithm.
For structural interpretability, we study two notions: weak and strong hierarchy [42, 13, 3, 3].
Weak Hierarchy:
fiN + 0 = #0 or #0 Vū.k) EI j E [p), & [p]. (2) fy.x # - f and f V(j,k) EI. j je ip), ke [p]. (3)
Strong Hierarchy:
Weak hierarchy allows an interaction effect fy.A to be selected if either main effect f, or fa is selected.
Strong hierarchy allows for an interaction effect f5. to be selected only if both main effects fy and f'x are selected. Such hierarchy constraints are popular in high-dimensional statistics: (i) They lead to more interpretable models [36, 3, 10], (ii) They promote practical sparsity, i.e., reduce the number of features that need to be measured when making new predictions (see Sec. 6.2) - this can reduce future data collection costs [3, 61]. (iii) Additional constraints can also help regularize a model, sometimes resulting in improved AUC (see Sec. 6.1). (iv) They can also reduce variance in estimation of main effects in the presence of interaction effects (see Sec. 6.4), allowing the user to have more "trust" on model interpretability explanations.
Contributions. To summarise, while it's well acknowledged that GAMs with sparse interactions are a useful flexible family of explainable models, learning them pose significant computational challenges due to the combinatorial nature of the associated optimization problem. Our technical contributions in this paper be summarized as:
1. We propose a novel optimization formulation that makes use of indicator (binary) variables. The indicator variables allow us to impose both (a) component selection and (b) structural constraints in an end-to-end fashion. We consider a smooth and continuous parameterization of the binary variables so that the optimization objective is differentiable (for smooth training loss) and hence amenable to first order methods such as SGD.
2. We show the flexibility of our framework by considering two different notions of hierarchy. While these constraints improve interpretability, they make the combinatorial problem more challenging [3, 17]. We propose end-to-end algorithms to train these models, making our approach quite different from existing neural-based toolkits [60, 10].
3. We exploit sparsity in the indicator variables during the course of the training for sparse forward and backward passes in a customized backpropagation algorithm in a GPU-compatible manner. This provides speedups on training times up to a factor of 10x over standard backpropagation.

---

4. We study novel statistical properties of our model, and present non asymmtommtticc prediction error bounds. Different from earlier work, our results apply to learning with tree-based shape functions (for both main and interaction effects).
We introduce a new open-source toolkit GRAND-SLAMEI and perform experiments on collection of 16 real-world datasets to demonstrate the effectiveness of our toolkit in terms of prediction performance, variable selection and scalability.
2 Related Work
GAMs. GAMs have a long history in statistics [14] and have been extensively studied. They 're often studied with smooth spline shape functions [see, e-g. 37, 47, 20, 63, 62, and references therein].
Some works have studied tree-based shape functions [33] and neural basis functions [1, 58).
GAMs with all pairwise interactions. In this thread, [9] study low-rank decomposition with neural network shape functions: [46] fit all pairwise interactions using shared neural bases.
Sparse GAMs with interactions. [31] introduced COSSO, which penalizes the sum of the Sobolev norms of the functional components, producing sparse models. [23] propose ELAAN, which an by- regularized formulation with smooth cubic splines. [23] demonstrate the usefulness of their approach regression setting terms of compact component selection and efficiency on large-scale Census survey response prediction. [34, 40, 6] explore tree-based shape functions for fitting additive models with sparse interactions. [34] consider all main effects and subset of pairwise interactions; the subset of interactions are selected via greedy stage-wise interaction detection heuristics. [40] provide an efficient implementation of the above approach as Explainable Boosting Machines (EBMs). [6] propose NODE-GA M: end-to-end learning approach with differentiable neural oblivious decision (NODE) trees [44]. Component selection in NODE-GAM is achieved by constraining the number of trees, and each tree learns to use one two features via entmax transformation [43].
Structural Constraints. Structural interpretability constraints such as hierarchical interactions, have been studied for both linear settings [3, 30, 59, 17] and nonparametric settings [45, 60, 10, 23]. We briefly review prior work on nonparametric hierarchical interactions as it relates this paper. [60] proposed GAMI-Net, which is a multi-stage neural-based approach that fits all main effects and subset of Top-k interaction effects, selected via fast interaction screening method [34]. Amongst this screened set, interactions that satisfy the weak hierarchy are later used to fit interaction effects. They also prune some main and interaction effects after training based on a variation- based ranking measure. [10] proposed SIAN, which uses Archipelago [54] to measure the strength of cach main and interaction effect from a trained DNN, and then screens (i.c., selects a subset of candidate main and interaction effects) using Archipelago scores to identify main effects and interaction effects that obey strong hierarchy. Then, it fits a GAM model with the screened main and interaction effects. [60] and [10] only support screening of interactions obeying hierarchy before the training for interaction effeets is done. None of these approaches impose hierarchy while training with interactions. [23] with their ELAAN framework also consider strong hierarchy in the presence of f-reeullardeed formulation with splines for the least squares loss (regression). ELÁAN has a two-stage approach: It selects a candidate set of interactions and then applies commercial mixed integer programming solvers to learn sparse interactions under hierarchy constraint. This approach would require customized algorithms to adapt different loss objectives e.g., multiclass classification. To our knowledge, current techniques for sparse hierarchical (nonparametric) interactions are not based on end-to-end differentiable training: they can be limited in flexibility and scalability a gap we intend fill in this work.
Table Relevant work on sparse GAMs with Interactions. Models rows 1-2 have some variable
selection but no hierarchy; models in row 3-5 have screening-based approaches for hierarchy.
Sclection Method
Stuuturrl Constraints
Classif.
Shape
Statistical
Paper
Main
Interactions
Weak Hier.
Strong Hier.
Reg.
Bin.
Multi
Functions
Properties
Scalable
EBM [40]
None
Greedy
X
/
f
trees
x
NODE&GA M
Antmax+Anncal
X
/
trees
x
GAMI-Net [60]
Prune Screening
Screening
X
4
x
neural
x
SIAN [10]
None Screening
Screening
x
neural
/
ELAAN [23]
Group L0
Screening+ Relax
x
splines
ORAND-SLAMTN
Binary Variables
End-to-end
End-to-end
trees
Hierrrey,,, Regressgo,s,,n,,
Clasif.Classsicaation
Bin.=Binary,
Relax.=Relaxattee -Relaxation
3

---

Note [9] and [10] also consider higher-order interactions (beyond two-way ones), which can be hard interpret. For convenience, Table summarizes some relevant work on Sparse GAMs with interactions and possible structural constraints.
3 Problem Formulation
We first present in Sec. 3.1 an alternative formulation of GAMs with interactions using binary variables for imposing sparsity and structural constraints. Next, in Sec. 3.2, we present a smooth reformulation of the objective that can be solved with first-order gradient-basee methods.
3.1 An optimization formulation with binary variables
We first present an alternative formulation of GAMs with interactions under sparsity with/without additional structured hierarchy constraints, Let us consider the paramcterization:
= Σ [555) + Σ [zz(2z,za)g(a) 2k. Zj,k); JE(D)
(4)
(GR)EZ
with main effects (-), interaction effects fz,((+) and binary gates 2j and g(zj, 2k, 2j.x). We consider three different parameterizations for q()) satisfying the following different constraints:
No structural constraint:
((2 Zkr 2j,k)
def Jj,ks
(5)
Weak hierarchy:
(22, 2ki 2j,x))
def
(2) 2 22
(6)
Strong hierarchy:
<(2g, 2k 2z,b))
def
(7)
The binary gates 2j and q(2,, =k; 2).)) play the role of selection. In particular, when 2, 0, the corresponding j-th main effect (-) is excluded from our additive model (4). Similarly, when
q(2z, 2k, 2j.x) 0, the corresponding (ji k))th interaction effect f- is excluded. Then, we can formulate the regularized objective as:
min Ugiha (s:)e(0.1))
El((,
X Σ 2j +a E k2))) JEIP
(8)
(G.A)EZ
where the first term denotes empirical loss over the training data, the penalty term controls model sparsity: À 2 0 is the selection penalty, a E [1, oo) controls the relative selection strength of main and interaction effects. We refer to the framework in (8) under the different constraints (5)-(7) as We discuss extension of this framework to third-order interactions in Supplement Sec. D. However, we do not consider third-order interactions in our experiments third-order interactions are hard to interpret.
The formulation in (8) with binary variables zj, 2j,a and functions fy. fy, with any of the constraint sets (5)-(7 is a challenging discrete optimization problem (with binary variables) and is not amenable to differentiable training via SGD (for example). Sec. 3.2 explores approximate solutions to (8) using a smooth reformulation of the binary variables. Intuitively, we rely on continuous relaxations of the binary variables 2's and parameterize f's with smooth beseb shape functions. The reformulation allows us to use first-order methods.
3.2 A Smooth Reformulation of Problem (8)
We discuss smooth reformulation of the objective in (8). We describe approach to parameterize the continuous relaxation the binary variables 2j. 2j.k with Smooth-Step function [18] and use smooth tree-based shape functions to model (f.) (
3.2.1 Relaxing Binary Variables with Smooth Gates
We present approach to smooth the binary gates z's in (8) using a smooth-step function [18), which we define next.
GGRAND-SLAMIN stands for GeNeRAlizeD Sparse Learning of Additive Models with INteractions.

---

Smooth-Step Function. Smooth-step function is a continuously differentiable function, similar in shape to the logistic function. However, unlike the logistic function, the smooth-step function can output 0) and exactly for sufficiently large magnitudes of the input (see Appendix B for details). This function has been used for smoothing binary representations for conditional computation [18, 19, 21].
We parameterize each of the z's in (8) as S(1), where μ E R learnable parameter and S(.) denotes the Smooth-step function. We parameterize the additive function as: Zje(n)
+ E EgMMe (2 Ix) (( S ((),)))) and optimize the following objective:
min UG (. (My)ERP (ERI))
BB fI] + X Σ S(n)) +a Σ S(p),)).
(9)
jElp
(JAJEZ
Note that S(1)) and S(1,,)) are continuously differentiable, so the formulation in (9) is amenable to first-order gradient- based methods (c.g. SGD).
Achieving binary gates. To encourage each of the S(py)'s and S(py.)) 'ss to achieve binary state
(and, not fractional) by the end of training, we add an entropy regularizer 7 +(E (P(())))
Eu.er (((((1)))))) where ((S(w)) = (p( log S(n) + (1- SS()) log(1 - S(4)) and 7 2
controls how quickly each of the gates Sp)) converges to binary
3.2.2 Soft trees
We use soft trees [25, 24, 53, 11]-based on hyperplane splits (univariate or bivariate) and constant leaf nodes as shape functions to parameterize the main effects (-) and the pair- wise interaction effects f.)) See Figure I for an illustration. Soft trees were introduced as hicrarchical mixture of experts by [25] and fur- ther developed by [24, 53, 11]. They allow for end-to-end learning (27, 18, 22]. They also have efficient implementations when learning tree sembles [22]. A detailed definition of soft tress given in Appendix A.
100 ,th main effect
of (mgg)))
Exeri f((2))z)
Zal-) (J. Ryth interaction
2y 27A
4 Efficient Implementation
a(wet With.)
We discuss fast implementation of our ap- proach. The key elements are: (i) Tensor param- eterization of trees, (ii) Sparse backpropagation, and (iii) Complementary screening heuristics.
Tensor Parameterization of Additive Effects. Typically, in neural-based additive model imple- mentations [1, 60], separate network module constructed for each shape function. The out- puts from each model are sequentially computed and combined additively. This can create bot- tleneck in scaling these models. Some recent approaches c.g., [10] try to work around this ap- proach by constructing large block wise network to compute representations of all shape functions simultancously. However, this comes at cost of large memory footprint. For tree-based shape functions, drawing inspiration from [22], we can implement tensor-based formulation of all shape functions, leveraging the fact that each tree has the same depth. This parameterization can exploit GPU-friendly parallelization in computing all shape functions simultaneously without increasing memory footprints.
Egyrer Syale 2a) 994)
Figure 1: Modeling main and interaction effects with soft trees. & denotes the sigmoid activation function. We omit biases in split nodes for brevity. For interaction effect, W R and W.,.k R denote the weights in i-th node of the (j. k).h)-th tree.
1,2
Sparse Backpropagation. We use first-order optimization methods (e.g.. SGD and its variants) to optimize GRAND-SLAMIN, Typically, a main computational bottleneck in optimizing GAMs with all pairwise interactions via standard gradient-based backpropagation methods is the computation
5

---

of forward pass and gradient computations with respect to all additive components (both main and interaction effects). This can hinder training large GAMs with interactions. We exploit the sparsity in GRAND-SLAMEN via the sparsity in the smooth-step function and its gradient during training.
Recall that (11))'s is and S(pze)'s play role of selection in smoothed fashion. In the early stages of training, S(u,) 's and S(p;))ss are all in the range (0, 1). As the optimization proceeds, due to the entropic regularization and selection regularization, Suy))s and S(pja)'s progressively achieve binary state {0, 1 the gradient with respect to j4j and j4 also reaches 0 because of the nature of smooth-step function S(-). All the additive components corresponding to the selection variables that reached 0 can be removed from both the forward and the backward computational graph. This sparse backpropagation approach can provide large speedups on training times up to a factor of 10x over standard backpropagation. Additionally, there is progressively a reduced memory footprint during the course of training in comparison to standard backpropagation.
The approach outlined above (use of Smooth-Step function for selection) when specialized to additive models allows to implement GPU-friendly sparse backpropagation this makes our work different from [18], which does not support GPU training.
Screening. We describe how screening approaches prior to training, can complement our sparse backpropagation approach when the number of all pairwise interactions is large e.g., of the order 100, 000. Fast screening methods based on shallow-tree like models are proposed in [34] for identify- ing prominent pairwise interactions. These are used by various toolkits e.g., EBM [40], GAMI-Net [60]. These screening approaches are complementary to our selection approach with indicator vari- ables. We used CART [4] for each pairwise interaction and sorted the interaction effects based on AUC performance select an initial screened set of interaction effects. In particular, we can consider I to be a screened subset (e.g., 10, 000) of all pairwise interaction effects of the order 100, 000. Then, we run our end-to-end learning framework under the component selection constraints on the main and screened interaction effects. We observe that such screening can be beneficial for multiple reasons: (i) The training time can be reduced further by 3x- 5x. (ii) The memory footprint of the model reduces by 10x. (iii) There is no loss in accuracy the accuracy can sometimes improve with screening see ablation study in Supplement Sec. F.6. Note that even with screening, our approach different from GAMI-Net as they directly screen to much smaller set of interactions e.g., 500 and there is no component selection when training with these interactions.
5 Statistical Theory
In this section, we explore the statistical properties of our additive model with soft tree shape functions. We assume that observations are noisy versions of some (unknown) underlying noiseless data. We do not assume the noiseless data comes from tree-based model. We show that if our model can approximate the true noiseless data well, the prediction error resulting from the noise converges to zero as the number of observations grows. In particular, for n data observations (æ;, 3/: ))) we consider a sparsit--constrannee version of Problem (8) with the least-squares loss as given by
f(o) (3) E argmin
Σ 1=1
1y
Σ f)(())) - (t 11 Σ fab(Fes, Iz, I1,x)g(2,, 2h, 213, (10)
jEipl
GAAEI
{z E (0. (2 E (0,1)/
Σ
Σ (J,))Z
g(2,, 2k, 2z,x) Ss
jElp
for 1 E (1,2), where Jzs are depth-d soft trees (ef. Section 3.2.2). For t= 1, Problem (10)
simplifies to a main effects model, while i = 2 corresponds to the model with pairwise interactions. We study the case with no hierarchy constraints here, ((2j, Zk. 2j.x) = 2j,0 as in (5). We expect our approach extend to more general cases, but we do not pursue that here.
Model Setup. We assume for i E [n]. the data is bounded, ||0:||a < 1. The noisy observations are given as y:- = /t" (ax)) + E; y + S; where h" is the unknown underlying generative model (need not be a tree ensemble), and e; iid Normal(O, o²) are noise values. Suppose f is feasible solution for Problem (10). Let U4 L denote the set of internal nodes and leaves for a depth d tree, and let (W", o be the set of weights corresponding to internal nodes and leaves of tree j i.c. fy( in this solution, where W E RM o² E RR We also let W of be the weights corresponding to node i U and leaf ( E L in tree j. respectively. We define Witk ofth for fz æx) and i e U, l e L

---

in similar fashion, where Wis. E R x oi* E RIAA, See Figure 1 for an illustration. Define
a(f)= max W V max jelpliec sy=1
1o V
max iuu 3
W l2 V
max Jeleele 3,A==
lof
jelp =j=1
where V denotes maximum. Conceptually, a(f) is the largest weight (in absolute value) that appears in all main effect and interaction soft trees. Let f= f(e)( (1) be the estimator resulting from the noisy data, and f* = ((e)(yy) be the oracle estimator that is the best approximation to the noiseless data among feasible solutions of (10) (i.e., among sparse additive tree models). We assume the following:
(A1) The activation function Φ : + [0, 1] for soft trees is L-Lipschitz for some > 0. (A2) There exists B > O such that u(f) V ū(f*)= S B.
Assumption (A2) ensures the trees resulting from the data are uniformly bounded. This is mild assumption as in practice, the data is bounded and the resulting trees are also bounded as a result.
Main Results. Our first result is for general setup, where p can be much larger than
Theorem 1. Let f. f be defined above and take A = 2 B B2L224+3 and exp(-1). Under the model setup, assume s 2 1 and a Z 1.2 Then.
(1) Fort 1, in 2 0² logp p 5² A?), then with high probability
1 2
Ef(x)))
ut
1 2
Σ (z.)
x2
vt
4/3 3=7
((slog)3//++s44)) (sA)4/3).
71
(2) For t 2, if 2 0² 2+ (s logp + 3 A.), then with high probability
1 1 (f(z))) =
2(1+4)/(24))) π Σ (z,) vt) 17((+a)
(Vslogp + 2/(2+))
(s4)3/2)3//00))
n
Theorem 1 presents non-asymptotic bounds for the performance of our method. Particularly, this theorem shows that under a well-specified model where y/ f=,)), prediction error rates of -2/3 and n 11((++a) x -0.42 are achievable for main effects and interaction models, respectively. Particularly, this implies that the prediction error of our estimator (resulting from the noise in observations) converges to zero the rate from the main effects model is sharper, such models might be too simple to capture the noiseless data, leading to larger oracle error. Therefore, in practice, the interactions model can lead to better performance. Next, we show that under the asymptotic assumption that n + co, the error rate for the interaction models can be further improved.
increase the total number of samples, We note that although
Theorem 2. Under the model setup with t = 2, assume all parameters are fixed except n. For any positive sequence on with limn-+00 On O. there exists a positive sequence bn with limn- bn -1/2(2+an) then with high probability
such that if n Z by
S S (f*(x)) - vi) )2+ + nt7(2+0a]
Theorem 2 shows that essentially when n + Do and other parameters in the problem stay constant, an error rate of -0.5 is achievable for the interactions model which improves upon Theorem
Discussion of previous work: As stated, our results are the first to present prediction bounds special- ized to learning sparse additive models with soft trees. Moreover, as we discuss in Appendix C.5, our upper bounds (rates) generally align with the best known rates available in the literature for learning main effects and interactions, such as [23, 52], or hold under milder assumptions.
6 Experiments
We study the performance of GRAND-SLAMID on 16 real-world datasets and compare against relevant baselines for different cases. We make the following comparisons:
1. Performance comparison of GRAND-SLAMIN without/with structural constraints against existing toolkits for sparse GAMs with interactions.
'The notation S. > depict inequality holds up to universal constant independent of problem data.
7

---

(a) Toolkits that support sparse interactions: EB*M [40] and NODE- GADE-GA'M [6]
(b) Toolkits that support hierarchy constraints: GAMI-Net [60] and SIAN [10]
2. Variable selection comparison against the competing toolkits.
Computational scalability of
toolkit with sparse backpropagation.
4. Variance reduction with structural constraints
Additional results are included in Supplement Sec. that study (i) comparison with full complexity models in F.I, (ii) comparison with GAMs with all pairwise interactions in F.2, (iii) comparison with Group Lasso selection approach in F.3, (iv) choice of shape functions in F.4, (v) effect of entropy on performance and component selection in F.5, and (vi) effect of screening on training times, memory and performance in F.6.
Datasets. We use a collection of 16 open-source classification datasets (8 binary, 6 multiclass and regression) from various domains. We consider datasets with wide range of number of all pairwise interactions 10 200000. A summary of the datasets is in Table E.1 in the Appendix.
Tuning Details. For all the experi- ments, we tune the hyperparameters using Optuna [2] with random search on a held-out validation set. We com- pute statistical averages across multi- ple runs for the optimal hyperparam- eters for all models. In particular, report median test ROC AUC across 10 runs along with the mean absolute deviation (MAD). Additional details in the Appendix E.
Table 2: Test ROC AUC of GRAND -LAMIN, EB and NODE-GA'M. We report median along with mean abso- lute deviation across 10 runs.
Dataset
EB-M
NODE-GA-M
GRAND-SLAMED
Magic
94.27 1 0.13
93.56 1:56:00 0.30
Adult
91.41 + 0.0004
91.75 + 0.14
91.54 0.14
Churn
91.97 + 0.005
89.62 + 5.61
02.40=0.41 (SH)
Satimage
97.65 + 0.0007
98.70 0.07
98.81 0.04
Texture
99.81 0.0001
1.0.00 0.00
MiniBooNE
97.86 + 0.0001
0.02
9.07 (WH)
Covertype
90.08 = 0.0003
95.39 0.39:0..1
98.11
Spambase
98.84 0.01
98.78
98.55 0.07 (SH)
News
73.03 +
73.53 0.06
73.24 (SH)
Optdigits
99.79 0.0003
99.93 + 0.02
99.98 0.0
6.1 Prediction Performance
Bankruptcy
93.85 = 0.01
92.02 - 1.03 1.03
92.51 + 0.54 (WH)
Madelon
88.04 + 0.02
60.07 + 0.82
89.25 1.03 (WH)
Activity
74.96 8.77
99.86 + 0.04
99.24 1.45
Comparison with EB'M and MO We first study the performance of our model in comparison to two tree-based ofate-of- the-art toolkits which support sparse GAMs with interactions without any structural constraints e.g., EB3M and NODE-GAĀM. We report the ROC AUC performance Table 2. Our model outperforms E in 10 out of 14 datasets. Our model is also competitive with NODE-GA PM as can outperform in 50% of the datasets. In summary, our results in Table 2 show that we are at par with state-of-the-arr methods for unstructured component selection. Our key advantage is to do hierarchical interactions, which NODE-GA2M and EB can not support. Additionally, we can achieve faster training times (Sec. 6.3) and improve on variable selection (Sec. 6.2) than NODE-GA 2 and EB²M.
Multiple
99.96 = 0.0002
99.94 = 0.02
99.95 + 0.02
Table 3: Test ROC AUC for GRAND-SLAMIN with structural constraints i.e.. (6) or (7), GAMI-Net and SIAN. We report median across 10 runs along with mean absolute deviation.
GAMI-Net WH
STAN SH
GRAND-8LAND
Model
WH
SH
Magic Adult
91.72 = 0.05
93.02 = 0.06
93.16 = 0.55
93.37
= 0.16
91.01 1 0.04
90.67
1 0.05
91.34
1 0.32
91.46
0 0.15
Chum
90.05 4 0.77
92.98
+: 0.20
92.28
+ 0.75
02.40
+ 0.41
Spanbase
:
98.28
0.04
96.45 0.15
1:
MiniBooNE
96.11 +: 0.41
95.90
97.77 # 0.05
97.62
4 0.30
News
T2.54 0.05
72.28
73.15 - 0.08
73.24
Bankrupicy
92.46
0.12
90.71
92.51
0.54
90.45
1: 1.87
Madelon
84
0.94
83.18
89.25 - 1.03
86.23 + 1.89
WH-=Waak Hierarchy, SH=Strong Hierarchy. For SIAN, for some of the larger datasets (row 5-9), we use the number best trial as SIAN takes 24 hours V100 Texla GPU.
Structural constraints: Weak and Strong Hierarchy. Next, we study our method with structural constraints i.e., (6) for weak hierarchy (7) for strong hierarchy. We compare against two competing neural- based state-of-the-an methods for sparse GAMs with hierarchical interactions: (i) GAMI-Net with support for weak hierarchy, and (ii) SIAN with support for strong hierarchy. We omit multiclass datasets as both GAMI-Net and SIAN do not support them. We report the ROC AUC performance in Table 3. Our models outperform GAMI-Net and SIAN in 7/8 datasets.
Additionally, our models are much more compact in terms of overall number of parameters - our tree-based shape functions have x and 10x smaller number of parameters than the neural-based
8

---

shape functions used by GAMI-Net and SIAN respectively. Moreover, our toolkit is significantly faster than SIAN and GAMI-Net on larger datasets see Sec. 6.3.
Additionally, we compare interpretable modeling toolkits with full complexity models e.g., deep neural network (DNN), in Supplement Sec. F.1. We observed interpretable models outperform full complexity models on these datasets. We also compare our toolkit that fits sparse components with toolkits that fit all pairwise interactions e.g. NA NBAM and SPAM in Supplement Sec. F.2. GRAND-SLAMIN generally outperform these methods with enhanced interpretability due to sparsity and structural constraints. We also study how our toolkit performs when we replace soft tree shape functions with MLP shape functions in Supplement Sec. F.4. Interestingly, we observe that soft trees seem to have an edge when the parameters are matched with MLP.
6.2 Variable Selection
We evaluate the performance of our models in terms of feature selection. We report the num- ber of features selected in Ta- ble 4 by each toolkit for sparse GAMs with interactions. We see that GRAND-SLAMIN with structural constraints, partic- ular strong hierarchy, can sig- nificantly reduce the number of features selected by the GAMs with interactions model. For ex- ample. on Bankruptcy datasets, GRAND-SLAMIN achieves fea- ture compression up to factor of 8x over state-of-the-an GAM toolkits. Having fewer features reinforces the usefulness of addi- tive models being interpretable.
Table 4: Number of features used by GRAND-SLAMIN without/with additional structural constraints and competing approaches. Hy- phen (-) indicates multiclass classification is not supported by GAMI-Net and SIAN.
FB-M
NODE
GAMI
SIAN
GRAND- SLAMIN
Model
GA M
Net
Nome
SH
Magic
10+00
10
Adult
14+0
14+00
14
14
13+1
11 1
11 +
Churn
19::
19
18
19 0
19 0
11 +
Satimage
36
0
36 00
2
Texture
40
4:
40
37 2
17 1-222
MiniBooNE
50
50
+ 12
50
Covertype
54
54 :
34 14:11
54
54 + 0
Spambuse
57
37 2
52 2
55 +
0
56 3
51 51
News
58
58
0
58
Optdigits
641:
6
64 2
61
+1
Bankmptey
95 95-+0
60
69
95 +-0
60
26
7
Madelon
500
61
19
24
Activity
533 +-0
346
182 15
440
+ 22
159 4-21
Multiple
64
649
648
629
9
649
0
WHE Weak Hierurchy, Strnn Hierurchy.
6.3 Computational Scalability
Next, we discuss the scalability of BRAND-SLAKAN
Sparse backpropagation. We highlight the usefulness of our efficient approach with sparse back- propagation Figure on Activity dataset. We show in Figure 2[a] that during the course of training, the number of selected components (selected via binary variables 2's and q(-)'s becomes progressively smaller. This leads to much faster computations at each epoch Figure 2[b] due to
Number of selected effects
Training time, seconds
Sparse
15k
1.5
10k
5k
50
100
150
200
250
50
100
150
200
250
Epoch
Epoch
(a) Number of selected effects at cach epoch.
(b) Training time (scconds) for each epoch.
Figure 2: GRAND-SLAMTE SLAMIN with standard (dense) backpropagation vs sparse backpropagation on Activity dataset. (a) shows the number of nonzero effects: Σ, žj + EGAteez ((2j, =k, 2j,k) and (b) shows the time for each epoch during the course of training.

---

Main effect
Main effect
-0.5
-0.5
-1.5
-1.5
-1.5
10
15
15
20
10
hour
hour
Main effect
0.1
0.3
Main effect
0.3
-0.2
-0.3
-0.3
-0.3
2.5 weether
2.5
weether
weether
Figure 3: Estimated main effects in the presence of interaction effects on bikesharing dataset [Left] without hierarchy, [Middle] weak hierarchy and [Right] strong hierarchy. Strong hierarchy has the smallest error bars.
sparse forward and backward passes. By exploiting sparsity during training, we can get 10x faster training times than with standard backpropagation.
Comparison with other toolkits Our toolkit is highly competitive in terms of training times with all existing tree-based and neural-based toolkits for sparse GAMs with interactions. For example, on Madelon dataset, we are 15x faster than NODE-GA²M, 20x faster than EBM, 1300 faster than SIAN and 25x faster than GAMI-Net. See Supplement Sec. F.7 for more detailed timing comparisons across multiple datasets. Note that, in addition, we can also handle the case of structured interactions - extending the flexibility of existing end-to-end training methods.
6.4 Variance Reduction with Structural Constraints
We provide visualization study further highlight an important contribution of our work. In particular, our framework can support models with structural constraints. Hence, we study the effect of these constraints on the stability of learning main effects (in the presence of interactions) when these structural constraints are imposed or not. For this exercise, we consider bikesharing dataset. We visualize some of the main effects the presence/absence
hierarchy in Figure 3. Note that for visualization, we used the purification strategy [29, 6] post-training that pushes interaction effects into main effects if possible. We can observe Figure that when additional hierarchy constraints imposed, the еттог bars are much more compact across different runs. This can potentially increase the trust you can have on the model for deriving interpretability insights. We show additional visualizations another dataset (American Community Survey from US Census Planning Database 2022 [55]) to show the same behavior in Supplement Section G.
Conclusion
We introduce GRAND-SLAMINE novel and flexible framework for learning sparse GAMs with interactions with additional structural constraints e.g., hierarchy. This is the first approach to do end- to-end training of nonparameteric additive models with hicrarchically structured sparse interactions. Our formulation uses binary variables to encode combinatorial constraints. For computational reasons, we employ smoothing of the indicator variables for end-to-end optimization with first-order methods (e.g., SGD). We propose sparse backpropagation, which exploits sparsity in the nature of smoothing function GPU-compatible manner and results 10x speedups over standard backpropagation. We present non-asymptotic prediction bounds for our estimators with tree-based shape functions. Numerical experiments on a collection of real-world datasets demonstrate the effectiveness of our toolkit in terms of prediction performance, variable selection and scalability.
10

---

Acknowledgments and Disclosure of Funding
This research was supported in part, by grants from the Office of Naval Research (N000142) 42001142110 12841), and Liberty Mutual Insurance. The authors acknowledge the MIT SuperCloud [48] for providing HPC resources that have contributed the research reported within this paper.
References
[1] Rishabh Agarwal. Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey Hinton. Neural additive models: Interpretable machine learning with neural nets. In A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[2] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, et al. Optuna: A next-generation hyperparameter optimization framework. In Proceedings the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
[3] Jacob Bien, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. The Annals of Statistics, 4133)1111-1141, June 2013.
[4] L. Breiman, J. Friedman, CJJ. Stone, and R.A. Olshen. Classification and Regression Trees, Taylor & Francis, 1984.
[5] Nadia Burkart and Marco F. Huber. A survey on the explainability of supervised machine leamning J. Artif. Intell. Res., 70:245-317, 2020.
[6] Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. NODE-GAM: Neural generalized additive model
for interpretable deep learning. In International Conference Learning Representations, 2022.
[7] Hugh Chipman. Bayesian variable selection with related predictors. The Canadian Journal of Statistics / La Revue Canadienne de Statistique, 24(1):17-36, 1996.
[8] Dheeru Dua and Casey Graff. UCI machine learning repository. 2017.
Abhimanyu Dubey, Filip Radenovic, Dhruv Mahajan. Scalable interpretability via polynomials. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[10] James Enouen and Yan Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[11] Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree, 2017.
[12] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):3681-33 3688,, Jul. 2019.
[13] M. Hamada and C. F. J. Wu. Analysis of designed experiments with complex aliasing. Journal of Quality Technology, 24(3):130-1) 137, 1992.
[14] Trevor Hastie and Rohert Tihshirani. Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371-385 1987.
[15] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
[16] Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman & Hall/CRC, 2015.
[17] Hussein Hazimeh and Rahul Mazumder. Learning hierarchical interactions scale: A convex optimization approach. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 Proceedings of Machine Learning Research, pages 1833-1843. PMLR. 26-28 Aug 2020.
[18] Hussein Hazimeh, Natalia Ponomareva, Petros Mol, et al. The tree ensemble layer: Differentiability meets conditional computation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4138 4148. PMLR, 13-18 18 Jul 2020.

---

[19] Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong. and Ed Chi. DSelect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. A. Beygelzimer, Dauphin, P. Liang, and Wortman Vaughan. editors, Advances in Neural Information Processing Systems, 2021.
[20] Huang, J. Horowitz, and F. Wei. Variable selection in nonparametric additive models. The Annals of Statistics, 38:2282-2313, 2010.
[21] Shibal Ibrahim, Wenyu Chen, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Comet: Learning cardinality constrained mixture of experts with trees and local scarch. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 832 844, New York, NY, USA, 2023. Association for Computing Machinery.
[22] Shibal Ibrahim, Hussein Hazimeh, and Rahul Mazumder. Flexible modeling and multitask learning using differentiable tree ensembles. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22. page 666-675, New York, NY, USA, 2022. Association for Computing Machinery.
[23] Shibal Ihrahim, Rahul Mazumder, Peter Radchenko, and Emanuel Ben-David. Predicting census survey response rates with parsimonious additive models and structured interactions. arXiv, abs/2108.1 11328, 2021.
[24] Ozan Irsoy, O. T. Yildiz, and Ethem Alpaydin. Soft decision trees. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pages 1819 1819-182,,00 2012.
[25] Michael I. Jordan and Robert A. Jacobs. Hicrarchical mixtures of experts and the em algorithm. Neural Comput., 6(2):181-214 mar 1994.
[26] Michacl J Kearns and Robert Schapire. Efficient distribution free learning of probabilistic concepts. Journal of Computer and System Sciences, 48(3):444 497, 1994.
[27] Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, et al. Deep neural decision forests. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1467-1475, 2015.
[28] Himabindu Lakkaraju and Osbert Bastani, "how do i fool you?": Manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES 20, page 79 85, New York, NY, USA, 2020. Association for Computing Machinery.
[29] Benjamin J. Lengerich, S. Tan, Chun-Hao Kingsley Chang, Giles Hooker, and Rich Caruana. Purifying interaction effects with the functional anova: An efficient algorithm for recovering identifiable additive models. In International Conference on Artificial Intelligence and Statistics, 2019.
[30] Michael Lim and Trevor Hastie. Learning interactions via hierarchical group-lasso regularization. Journal of Computational and Graphical Statistics, 24(3)x627-654, July 2015.
[31] Y. Lin and H. Zhang. Component selection and smoothing multivariate nonparametric regression. The Annals of Statistics, 34:2272-2297, 2006.
[32] Zachary C. Lipton. The mythos of model interpretabillity In machine learning, the concept of interpretabil- ity is both important and slippery. Queue, 16(3):31-57, jun 2018.
[33] Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD 12, page 150 158, New York, NY, USA, 2012. Association for Computing Machinery.
[34] Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '13, page 623-631, New York, NY, USA, 2013. Association for Computing Machinery.
[35] Scott M. Lundberg and Su-In Lee. A unitied approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 4768 4777, Red Hook, NY, USA, 2017. Curran Associates Inc.
[36] Peter McCullagh and John A Nelder. Generalized linear models, volume 37. CRC press, 1989.
[37] Lukas Meier, Sara van de Geer, and Peter Buhlmann. High-dimensional additive modeling, The Annals of Statistics, 37769) 3821, 2009.
2009.
12

---

[38] Shahar Mendelson and Roman Vershynin. Entropy and the combinatorial dimension. Inventiones mathe- maticue, 152(1):37-55, 2003. 2003.
[39] J. A. Nelder. A reformulation of linear models. Journal of the Royal Statistical Society, Series A (General), 140(1):48 77. 1977.
[40] Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. ArXiv, abs/1909. http/1999.0922,,, 2019.
[41] Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. Pmll: large benchmark suite for machine learning evaluation and comparison. BioData Mining, 10(1):36, 2017.
[42] Julio L. Peixoto. Hierarchical variable selection in polynomial regression models. The American Statistician, 1114333 1987.
[43] Ben Peters, Vlad Niculac, and André F. T. Martins. Sparse sequence-to-sequene models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1504-1519, Florence, Italy, July 2019. Association for Computational Linguistics.
[44] Sergei Popoy, Stanislav Morozov, and Artem Bahenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference on Learning Representaatonss 2020.
[45] Peter Radehenko and Gareth James, Variable selection using adaptive nonlinear interaction structures in high dimensions. Journal of the American Statistical Association, (05(449))1541-1555 December 2010.
[46] Filip Radenovie, Abhimanyu Dubey, and Dhruv Mahajan. Neural basis models for interpretability. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[47] Pradeep Ravikumar, John Lafferty, Han Liu, et al. Sparse additive models. Journal of the Royal Statistical November 2009.
Society: Series B (Statistical Methodology),
[48] Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. Interactive supercomputing 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pages 1-6. IEEE, 2018
[49] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predic- tions any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '1. page 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery.
[50] Mark Rudelson and Roman Vershynin. Combinatorics of random processes and sections of convex bodies. Annals of Mathematics, pages 603-648, 2006.
[51] Dylan Z Slack, Sophie Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc explanations: Modeling uncertainty in explainability. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[52] Zhiqiang Tan and Cun-Hui Zhang. Doubly penalized estimation in additive regression with high- dimensional data. The Annals Statistics, 47(S):2567 2600, 2019.
[53] Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, al. Adaptive neural trees. ArXiv, ahs/1807.0666) 2019.
[54] Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attribution for feature interactions. In Advances in Neural Information Processing Systems, 2020.
[55] US Census Bureau, Planning database, October 2022.
[56] Roman Vershynin. High-ddmensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
[57] Martin J Wainwright. High-dimensional statistics: A non-asymptoric viewpoint, volume 48. Cambridge university press, 2019.
[58] Shibiao Xu, Zhiqi Bu, Pratik Chaudhari, and lan Barnett. Sparse neural additive model: Interpretable deep learning with feature selection via group sparsity. ArXiv, abs/2202, 2022.
13

---

[59] Xiaohan Yan and Jacob Bien. Hierarchical sparse modeling: A choice of two group lasso formulations. Statistical Science, 32(4):531-560, November 2017.
[60] Zebin Yang, Aijun Zhang, and A. Sudjianto. Gami-net: An explainahle neural network based on generalized additive models with structured interactions. Pattern Recognit., 120:108192, 2021.
[61] Guo Yu, Daniela Witten, and Jacob Bien. Controlling costs: Feature selection budget. Stat, 11(1):e427, 2022.
[62] Ming Yuan and Ding-Xuat Xuan Zhou. Minimax optimal rates of estimation in high dimensional additive models. The Annals of Statisticx, 44(6):2564-2593, 2016.
[63] Tuo Zhao and Han Liu. Sparse additive machine. In Neil D. Lawrence and Mark Girolami, editors, Proceedings of the Fifteenth International Conference Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 1435 1443, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.
14

---

Supplementary Material for GRAND-SLAMIN' Interpretable Additive Modeling with Structural Constraints
A Definition of Soft Trees
Formally, an interaction soft tree with the set of (internal) nodes 2 and the set of leaves is function such as f(: W.
W.o) o) IR² R where W E R and E RI The output is then given
as [(Ty, ma)); W. 0) = Eiec P(f(zz, =4) - 1) where P(f xx) - 1)) is the proportion of (Ty, FR) that is routed to leaf I. Particularly, P([(Tj, Ia) + +11) = IlL (i, mx) where A(I) denotes the set of ancestors of I and ri, mx) denotes the proportion of (I), is routed to leaf ( from node i. These values given as ri, (1). = o (WW? (13, (n))
if belongs the left subtree
of i, and (z, m4) o <(W? (æy, m4m)) )) if belongs to the right subtree of where Φ(.) the sigmoid activation function. A soft tree f(; W. o) R + I for main effect is defined similarly.
B Smooth-Step Function
The smooth-step function [18] has been used in soft trees for conditional computation [18] and routing in sparse mixture experts [19, 21]. It can output exact zeros and ones, thus allowing for hard selection. Let y be non-negative scalar parameter. The smooth-stey function takes the form:
if < -7/2 if 5/5
htt) =
3 22
(B.1)
27
2
if 7/2
The smooth-step function is continuously differentiable, similar to the logistic function. Additionally, it performs hard selection, i.e., outside [-7/2, 7/2), the function produces exact zeros and ones.
C Proofs of Main Results
C.1 Preliminaries
Before proceeding with the proof, we define some notation we use throughout this section. Given the data points -1.-.. Tn. let Pa the empirical measure supported " Un. P.. E /n. We define the norm with respect to P, as
lfll..
Σ f(x.)2
(C.1)
Next, we define covering and packing numbers.
Definition C.1 (Covering number [38]). Let F be a functional class and ø1, ... Un be a set of n observations. An e-cover of with respect P, is of functions f1, ...fx such that for any f E F, there exists j E [N] such that
|| fslr,, S.
The covering number of F with respect to P, at scale c, denoted as M(F, Pu, c), is the smallest N such that there exists e-cover of F with respect to P., with size N.
Definition C.2 (Packing number [56]). Let F be a functional class and æ1, Hr be set of observations. An e-packing of F with respect to P, is a set functions fi. .NN such that for any i.j [N].
I fylpe. e.
The packing number of F with respect to P, at scale e, denoted as P(F, Pn. e), is the largest N such that there exists an e-packing of F with respect P, with size N.
The following well-known lemma establishes the equivalence between packing and covering numbers.
15

---

Lemma C.1 (Lemma 4.2.8 [56]). For e > 0,
P(F, Pn, 2e) S WFF, e) S P(F, Pn, E).
Finally, we define the fat-shattering dimension function class. Definition C.3 (Fat-shattering dimension [26]). Let F be a functional class. Given a subset S (#1, ... ]n C IRP, the class F e-shatters S, if there exists = (C1, Cx) E R" such that for every e E {0, 1)" there exists F such that
f(x)) 2 a + c/2 for e; 1, and f(;)) S C: c/2 for e; 0.
The fat-shattering dimension of FF scale e, denoted as fat, (F) the size of the largest subset S that is e-shattered by F.
The following theorem relates the fat-shattering dimension to the covering number discussed above. Theorem C.1. If there exists such that
fatze (F) Vr 2
then
No Pay Cc) Cv.
Proof. The theorem is direct result of Theorems 6.3 of [50] and Lemma C.I.
C.2
Technical Lemmas
Let U, C denote the set of internal nodes and leaves of depth d tree, respectively. Let us define the class of soft trees with depth d and data dimension p as
p(d.p) {s(:; W, o) B R. ((((; W o)) B}
(C.2)
where B3 is the Euclidean ball of dimension p, W puIx and E I [I and
(a: W, o)) max
[ max orl IEC
keu
with wi E Iº being the vector of weights for node & from W. Our first result is characterizing the fat-shattering dimension of F(d, p).
Lemma C.2. Let A = 2V 2d+2 B Bdd2d+3 For > 0,
A
fat, (Fdd p)) <
(C.3)
Proof. First, suppose e 2B2d. We show that fat, (F(d. p)) e F(d,p),
To this end, note that for and
[f(x)
Σ P((x + 1))or
5 Σ lor < B
(C.4)
lEC
lEC
where defined in Appendix A.As a result, for f1, E F(d, p) and x.
[i(()) [2(2) < 2B2 <6
(C.5)
showing no single point can be e-shattered. Therefore, fat. (Fd.
Let < < 2B2d and suppose (æ),.... xNN) is e-shattered by F(d,p) at level c. We show that for i,j [N]. [ - æ 2 e((2d To this end, assume there exists i,j e [N] such that
||o; - æ,112 e/ (B(2d)22) Without loss of generality assume c; 2 oj and choose e {0, 1)P such
that e; 1, e, = 0. Suppose f(: W. o) F(d, p) shatters the data with Particularly,
f(x; W. 2 t/2, f(z;; W. o) S c e/2.
(C.6)
16

---

For h E u,u. if A(l) we have
[ a.) - 20w ax)
(a)
Ww x; w/a,1
LB xz - 23yl2
(C.7)
where Tk, is defined in Appendix A, wy RP the vector of weights for node from W, (a) is by (A1) and (b) is by ū(f(x; W, o)) s B (recall the Assumption (A2)). Next,
1 + - + +1] = II (x,) - II TK,(ay))
(EE((I)
kEA(())
I dLB x; x3y12
(C.8)
where the inequality is by (C.7) and the fact that Tk. E [0, 1] is bounded. Finally,
Is W. 0) - flzy: W. o)1
Σ
P((x; - 11)00 - Σ P(fag + 17)os
IEC
IEC
|oi| [P({2: + +1}) - P(( - +1}1
IEC
2d Bd
dT. - 12yl22 ll
(C.9)
where the last inequality is true as | 2d and |oı| S by ūl ((f(;; W. 0)) s B. As we assumed
||e; - æ;112 < (/(Bd we have from (C.9)
If W. o) fazzi W. o)] <c
while on the other hand, by the shattering property (C.6),
f(a;; W. o) - f(æ;; W, 0) 2 (a ( +2e/2
(C.10)
20
which is a contradiction. Therefore, for every i,j E [N], we must have ||x; æ,l12 2 e/(B2 or in other words, (æ1,'.. ,Ny) a e/ ) of B2. Thus,
N
NBB || e/BB ((22))
IAG
4B2
+1
dB2d
WV VA
1 V 2B2dd
dB2
1 1 2B2d
P
A
S
where (B3 || ||2, e) is the e-covering number B?, (a) is by Lemma C.I, (b) is true by Corol- lary 4.2.13 of [56] and (c) is true as e S 2d+1 B.
Lemma C.3. There exists an absolute constant C > O such that for > 0,
CA
Ngg.) Pxy 2 C
17

---

Proof. From Lemma C.2, for x 2 1
A P
fat (F(,pp)
A
1
a>
A
(C.11)
I
where the last inequality is true as 2 1. The proof is complete by Theorem C.I.
In the next step, consider trees that only subset of variables, such J S [p]. Formally, let
Fo(J. = hfii B + R, alf(Haahhee: W. < B}.
(C.12)
Lemma C.4. For e > 0,
log. ( Pn, cn. SC (CA)
CA 17
Proof. Let Pn.. be the empirical distribution supported on coordinates in J,
Pn.j Σ daz
where øy is the sub-veetor of æ with coordinates in J. Then,
(VFo((J,)) Pn, c) s (VF((d c,
The result follows from Lemma C.3.
Next, let us define the classes of functions with interactions in J C 2
Jld =
B. W3,00) Fo(J,d)
(C.13)
JEJ
and the class of functions with at most interactions of size at most (i.e t-dimensional interactions).
sd,,,s) { f2 J C 2 |]| 5 StAJ E fi (dd fo E FiddJ)))
(C.14)
for fixed J with J < and |J| for J E J
Lemma C.5. Suppose 0 < 2sCA and S t 2. Then.
2CsA
(dFt,,,,,, P.o s 2s logp + 2C's
Proof. Let S(Fo(J,d)) d)) be minimal e/2s-cover of FolJ d). Hence, [S(Fo(J,d))
((FF((,d) Pn,ess) Define
[ 2b p I. $
(C.15)
and
S = U
Σ ((Fo(J,d)) Σ S(Fo(J.d))
(C.16)
JeS
JEJ
JeP
18

---

where A+ B = {a+ b: A, B) denotes the Minkowski sum. Note that from Lemma C.4,
for and J
(c((2)))))) (c(20CA)
131
S exp (c ( (22
Pb, e/2s) I exp
(C.17)
as 2sCA/c2 2 For the rest of the proof, we let
Ne(e) = exp C CA (0(2))
Suppose f E (d,t,,
[(x)=
Σ W Σ
fa(w; W 55,
(C.18)
JElo
JE3
for some Io E S. For any I JoUD', let fs E ((Fo(TT,)) ,d) be such that 115s fJllp, S t/228. Let
f(x) = is(x: (5 + Σ fo(æ; W o,ō))
(C.19)
Jela
FET
Note that E S* Next,
Iff - flle.
E(ts (fo fo) + Σ (fo fz)
JEJ
JEJ
P.
1l - folle, +
Σ
1l 8zllp
JEJa
JEJ
S 2sc/2s=4
(C.20)
|To As result, S* is 6-cover for F',,, t, 8). Moreover,
IS
II ((Fo(( d)) II
S(Fo(J,))
JES JEJ
JEJ-
Σ
II
II JEL
Pns
JES JEJ
Nr(e/25)2z
JEB
(
2 (e(e/2s))
(C.21)
where (a) is by (C.17) and (b) is true [S] S (p') < p2= As result,
log.N(F" (d, s), Paçe) S 2s log + 28 log N.(e/2s)
2slogp 2C's 22CsA)
(C.22)
Lemma C.6. Suppose < 6 < 1A AsC.A. Then, for = exp(-11)
(d,, (u, 222slogp + CsAAVo
62/440
(C.23)
(d
1/2slogp (2C5((()A)) (40/8)a.
82/440
19

---

Proof. Note that if 5 > 40, the lemma is trivial. Therefore, we assume 8 - 4a. From Lemma C.5
200A
(l Pr.c)) P - 2slogp
+0s
2Cs
S v 22slogp-2 + (2C=(
(C.24)
Hence, as 1
10010 (F Pa,e) s)de < / (Log 1,8), Pn, t)de
v 2s 2ologpdde + 10¹ 2CsAc 11/2
$ 2s log + CsA81/2 1/2/2 1/2 + CsA8/12 S v 2s log log
(C.25)
Moreover,
S ((F.((F)(,,2.s))) Pmi c)de <
(")
(F Pn, Pn.
82/40
< /22log 12slogpdl pile +
2CsAv de
82/44
< 28 log + 2v 2/22sCsAlooo(to
(C.26)
Next, consider the function h(x) = x² - log(()) where a = exp(-1). We will show h(x) 2 0 for O. Note that
h'(x)= = az 1
therefore h(a 1/0) 11(aa 1/a)= = 0. Moreover,
h²(x) a(a - 1)x² + x22
so (a = (1 + aaa - 1)a- 2/a) > 0, showing that a 1/a is minimum for h(r), and therefore h(x) 2 for x > Hence, from (C.26),
883/40 (d. sn,
Pa, e)de s 2og + V20s2CsAlog(40//0)
2s + V2C=2CsA(4o/83)
S v 2s log p(40/6) V
(C.27)
as < or 4a/ 2
Lemma C.7. Under the assumptions of Theorem 2, for any sequence On n 2 such that an > and lim On O, there exists a posilive sequence bn such that limn-roo bn and
| S0110 Jogg((F*d,22 2), Pay c)de < (v25 1/2logg- logp V2C:(((AA) (40/8)a0 v8 < bn.
Proof. Consider the function h, (x) pdn log(z). Note that
Drn lim a oo
lim
log
therefore, there exists sequence "/n such that lim Yn and
ha(2) > 0, Vr 'n.
20

---

Therefore, if I 40/ A 40 A =: bng from (C.26),
(o 82/11
(d,2, Pn.c)de
2s ls 22
2o v (40/5)ar.
(C.28)
Lemma C.8. For any e, 6- O,
NIF (d.4,) (Il (.( Pn,e)
(C.29)
Proof. Let (fi,... .fv] minimal e-cover of F Ft,, s) where N = NNFF (d, Pn.e).
Let S" [fi, (NN where for E [N].
[R f (f5//1lf
if [lell,, 5
(F(ttts 5
if lffilp. >
We show S* a c-cover for F (d,t, 8) n (1lflle, 6}..
Let E (d, s) n (Ifflp, I S}.. Let E [M] be such that f fylle, S Let us consider the
following cases:
Case 1: If Il < 6, then f, S showing 8j fg covers f.
Case 2: Suppose [[ll > 5. For any f, let f = (f(x)), f(ra))) V Ra. With this notation,
|r = [[ As a result,
a² 2 If - f,l1s.
[[s - 13)
(fz- 1b)
(J /3)
(17.
5
1)
Sj
8
(r 5)
f3
F1122
5
|[ fi
CF((C -f)
||E 112
8
8
8
|f
ffT 2
||E,
||6,|22
||Eg112
11
(a)
8
2 || 11 2
||0,1120 -
1
811c,
||,
||f/|12
ff - 1 = fl
ff
(C.30)
where (a) is true as
||fy and |f]|a S 8. This shows fÿ covers f. completing the proof.
C.3
Proof of Theorem
Proof. Let
61 (640(V2slup 22loogg-+ CsA)] 2/3
1/3
(v2s + (20s(((sa) (40) 1/(2+a)
(C.31)
16
62
n m/2((2+))
Claim: For / E (1,2).
16
52
(d Pn,
Vn /
46
21

---

We will prove the claim later. Let
s(),t,s) - fi :h:Js 2lel, ||] ss Stvs e fi &
(C.32)
Under this notation, we have
1
E argmin Σ (y/ f(x;))2 1=1
seF(d,t,s) 12
Using the notation from [57][Ch. 13), we have
OF = F(d, (dd,
=(d,t))
with J" being the set of interactions for for f." Moreover, we note that F F* t, s) is star-shaped, that for f E (d, and a E [0, 1]. of E F (d, t, This is as
af(: W o) = f(: W.,o). .xo).
Under the claim and assumptions of the theorem, we have d, Theorem 13.13 of [57] with 7 = 1/2,t t = achieve
invoke Corollary 13.7 and
IP
Evs f(x:))) + +0² 2 1 ai exp(-c2nd) /ond /0²) n Σ lot fc(x:))² x inf
1
(eF(d.t.))
(C.33)
completing the proof.
Proof of Claim: For ( = 1,
644 (v22 22 log + CsA)] 4/3
4
4on2:3
(4o + CsA] 1/3
16
2s logp + CsA)
n/6
16
(V28 logp + CsA)vol
Vn.
16
(F Pu, cide
Vn
85//10
16
(F.
S8 e)de
(C.34)
where the first inequality is by Lemma C.6 and the final inequality is by Lemma C.8. Next, for / = 2,
(4a))+aa 1+a 2/(2+a)
16
(v2slogg logg V2Cs((CsA)
83
40
4on1/(24a)
-o/(2+o)
16 (o ((250og)+((((((())))) (40)
(40)a
16
V 2s +
Vn
"1 a
16
V 2s ( (40/0)a
Vn
16
6
((
(11//l,, 502), Pa, cyde
(C.35)
Vn
53/40
by Lemmas C.6 and C.8.
C.4 Proof of Theorem 2
Proof. Let
[4a) 1/(2+an)
16
228 log + (2Cx(((sA)
82
2/2(2ta...
22

---

By taking 2 bm
2(2+aa))
and invoking Lemmas C.7 and C.8,
82
16
62
S $2),, Page)de
46
vn
82/40
The rest of the proof is similar to the proof of Theorem 1.
C.5 Theory Discussion
the smooth- step function which is differentiable with uniformly bounded derivative, but not twice differentiable. Therefore, our framework operates on L2 Sobolev functions of order 1, i.e. bounded differentiable functions with bounded gradients. For this case, the results of [52] show that an & prediction rate of n -2/3 (see Remark 4 of [52] with & = 0 and Bo = 1). This is the same rate we achieve in Theorem for t = 1. However, we note that our results hold without differentiability assumption and only require the activation function to be Lipschitz, showing stronger results specialized to the soft trees case.
First, we consider the case with only main effects, i. t 1. In our framework,
Next, we consider the interaction case with = 2. In this case, the results of [23] imply an 82 prediction rate of 1/2 for the class of L2 Sobolev functions of order (see Section S2.2 with m leading to r. n 1/4), This is the same rate our method achieves asymptotically (see Theorem 2). However, as we discussed, our results hold without the differentiability assumption.
D Extension to third-order interactions
Our approach can be extended to model third-order interactions. In the case of third-order interactions, the structural constraints for strong and weak hierarchy can be described as follows:
Strong Hierarchy: JJ # -
f 0 and Ja # 0 and St # 0, ft サ 0.
(D.1)
Weak Hierarchy:
f +
f
fx
(D.2)
These constraints can be modeled with binary variables as follows:
Strong Hierarchy: 4(2j, zk. 21, =j,k,)
def
(D.3)
Weak Hierarchy 4(2., zk. 2z,k.1) (z) + zか +
del
2j2k 2 24 + 222k21
(D.4)
Hence, sparse selection and hierarchy constraints can add value in terms of model compactness and feature selection in the settings with third-order interactions as well. However, for third-order interactions, the number of third-order interactions are Opp"). Hence, for scalability, this would also require using pre-training screening approach. Despite the fact that our approach is generalizable to third-order interactions, we do not consider such interactions because these are not considered be easily interpretable.
E Datasets, Computing Setup and Tuning
Datasets We use a collection of 16 open-source classification datasets (binary, multiclass and regression) from various domains, e.g., physics, computing, healthcare, life sciences, finance, and social networks. They are from Penn Machine Learning Benchmarks (PMLB) [41] and UCI databases [8]. For datasets with available training, validation and test splits, used them in their original form. When no test set was available, we treated the original validation set as the test set and split the training set into 80% training and 20% validation. For remaining, we randomly split each of the dataset into 60% training, 20% validation and 20% testing sets. A summary of the 16 datasets considered is in Table E.I.
Computing Sctup. We used a cluster running Ubuntu 7.5.0 and equipped with Intel Xeon Platinum 8260 CPUs and Nvidia Volta V100 GPUs. For all experiments of Sec. 6, each job involving GRAND-SLAMIN, EBM, Node-GAM, SIAN, GAMI-Net and DNN were run on core, 32GB RAM. Jobs involving larger datasets (p > 100) were run Tesla V100 GPUs.
23

---

Table E.1: Summary of Datasets
Dataset
Domain
N
C
No. of interactions ([Z))
Magic
Physies
19,020
10
55
Adult
Socio-economic
48,842
2
14
91
Churn
Business
5,000
19
190
Satimage
Physics
6,435
6
36
640
Texture
Image
5,500
11
40
780
MiniBooNE
Physics
130,065
50
1,225
Covertype
Life Science
581.012
7
54
1,431
Spambase
Computing
4,601
1,596
News
Social networks
39,797
2
1,830
Optdigits
Image
5,620
10
64
2,016
Bankruptcy
Finance
6,819
96
4,560
Madelon
NIPS-2003
2,600
2
500
124,750
Activity
Healthcare
4,480
533
141,778
Multiple
Image
2,000
10
649
210,276
Bike Sharing
Transportation
17,389
16
120
American Community Survey
Demographic 83,059
39
741
The tuning was done in parallel over the competing models and datasets. We tune the hyperparameter using Optuna [2] which optimizes the overall AUC on a validation set. We report results on held-out test set. A list of all the tuning parameters and their distributions is given for GRAND-SLAMIN below:
Tuning.
Learning Rates: Discrete uniform in the set (0.05, 0.01, 0.005) for Adam with multi-step decay rate of 0.9 every 25 epochs.
Batch-size: Discrete uniform in the set (64, 256).
À for selection: Discrete uniform in the set of 11 values {0, 1e 6,. 10-33 33.
Y for Smooth-step: Discrete uniform in the set {0.01, 0.1, 1].
T for Entropy Regularization: Discrete uniform in the set {0.001, 0.01, 0.1}.
for relative penalty interactions: Discrete uniform in the set (1, 10). Epochs: 1000 with early stopping (patience=50) based on validation loss.
For Madelon, Activity and Multiple datasets, we used screening to reduce the initial set of interactions to 5000.
For other toolkits, we considered the tuning protocols outlined in the respective papers such that the parameter controlling the variable selection is tuned. For SIAN, we tuned over the threshold parameter 0 such that the screened set of interactions is upper bounded by {250, 500, ... 1000]. We set such that the model satisfies strong hicrarchy. For GAMI-Net, we tuned over the number of interactions. For small datasets ([Z) 1000), we tuned over the set: [ 0.4 [Z]] For large datasets ([Z) < 1000), we tuned
the set: (250, 500, 1000). For EBM, we tuned over the set of interactions (16, 32, 64, 128} as done by authors in [6]. For NODE-GA M, we tuned the number of trees as this controls the maximum number of interactions selected by the model.
Additional Results
F1 Comparison with full complexity models e.g., DNN
Here, we compare interpretable modeling toolkits based on pairwise interactions with full complexity models. In particular, we consider deep neural network (DNN). We considered the same architecture for the DNN as used by the authors in [10] for similar comparisons. The architecture is 4-yeerd ReL.Uacttivaeed neural network. We show the test AUC performance of different models in Table F.I. Across all datasets, we see that the full complexity DNN model underperforms interpretable models including GRAND-SLAN SLAMIN. However, it is noted that the choice of the architecture for DNN maybe contributing the degradation in performance. There maybe other architecture choices DNN e.g., neural networks with residual connections, that may perform better.

---

Table F.1: Test ROC AUC of GRAND-SLAMIN, EB and NODE-GA'M. We report median across 10 runs along with the mean absolute deviation (MAD).
Interpretable Models
Full Complexity Models
Model
EB EB-M
NODE-GA'M
DNN
Magic
93.12 0.001
94.27 0.13
93.86 = 0.30
93.69 = 0.04
Adult
91.41 0.0004
91.75 = 0.14
91.54 0.14
90.26 + 0.04
Churn
91.97 + 0.005
89.62 5.61
92.40 0.41
90.28 = 0.34
Satimage
97.65 0.001
98.70
98.81 0.04
98.67 = 0.07
Texture
99.81 + 0.0004
=00.00
1.000
99.63 = 0.09
MiniBooNE
97.86 0.0001
98.44
+1.77=0000
97.08 = 0.38
Covertype
90.08 = 0.0003
95.39 0.12
08.11
93.83 = 0.10
Spambase
98.84 0.01
98.78
98.01 = 4.70
98.09 = 0.07
News
73.03 + 0.002
73.53 0.06
T3.24 0.04
72.19 = 0.08
Optdigits
99.79 0.0003
99.93 0.02
99.98 - 0.00
99.77 = 0.04
Bankruptey
93.85 = 0.01
92.02 = 1.03
92.51 = 0.54
88.38 = 0.36
Madelon
88.04 0.02
60.07 = 0.82
89.25 + 1.03
63.60 = 0.47
Activity
74.96 + 8.77
99.86 + 0.04
99.24 + 1.45
96.64 + 1.93
Multiple
99.96 + 0.0002
99.94 + 0.02
99.92 + 0.02
99.94 + 0.01
F.2 GRAND SLAMIN versus GAMs with all pairwise interactions
Next, compare GRAND-SLAMIN i.e., GAMs with sparse pairwise interactions against GAMs with all pairwise interactions toolkit:
1. NA i... Neural Additive Model [1] with pairwise interactions,
2. NB i..., Neural Bases Model [46] with pairwise interactions,
3. SPAM, i.e., Scalable Polynomial Additive Model [9] with pairwise interactions.
For NA NB2M, and SPAM, we tuned over learning rate in the set {0.1, 0.01, 0.001, 0.0001] and number of epochs in the set (50, 100, 5003. We capped the time for each trial for NA²M to 6 hrs.
We show results in Table F.2. We outperform SPAM across many datasets. Notably, GRAND-SLAN improves by 21% over SPAM on Madelon. We are also competitive with NB²M and NA M. For larger datasets e.g., Madelon, Activity and Multiple, NB²M and 2 ran out of memory on compute node with V100 Tesla GPUs, Recall also that the goal of our work is to learn sparse components with/without structural constraints for easier interpretability.
Table F.2: Test ROC AUC for GRAND -SAMIN and GAMs with all pairwise interaction models, e.g.. NA NB²M and SPAM.
Dataset
NA
NB-M (Dense)
SPAM (Dense)
GRAND-SI SLAMIN
(Dense)
(Sparse)
Magic
94.46 + 0.16
94.11 0.08
91.75 0.004
93.86 + 0.30
Adult
90.81 0.10
91.06 0.03
89.65 0.001
91.54 1 0.14
Churn
93.03 + 0.49
02.16
88.41
92.40 + 0.41
Satimage
98.81 0.05
98.89 0.03
97.94 + 0.02
98.81 0.04
Covertype
out of time
98.36 + 0.02
96.29 + 0.02
98.11 + 0.08
Spambase
98.38 0.05
98.37 0.06
97.78 - 0.04
98.55 0.07
News
71.94 + 0.30
72.54 + 0.07
72.43 + 0.06
73.24 0.04
Bankruptcy
87.83 + 0.12
93.01 1.85
89.35 - 0.90
92.51 + 0.54
Madelon
out of memory out of memory
of memory
68.59 + 0.80
89.25 1.03
Activity
out of memory
99.10 + 0.04
99.24 + 1.45
Multiple
out of memory
out of memory
99.98 0.03
99.92 0.02
25

---

F.3 Comparison with Group Lasso
Group Lasso [16] is also a possible choice for sparse selection, which is popularly used in high- dimensional statistics and machine learning. We compare against à version of Group Lasso:
min Othebl
E Myy
Σ f(.))) XE(p]
End i<j
+t
Σ iE(pi
1lfell +
(F.1)
(hakees
where I' denotes the empirical loss on the training dataset and ||.a denotes the regularization imposed via the group regularization on the leaf weights of the effect We compare performance between GRAND-SLAMIN and Group Lasso (with soft tree shape functions) in terms of test AUC and variable selection on a few datasets. The numbers are reported in Table F.3. Overall, our approach significantly outperforms Group Lasso in terms of AUC performance and variable selection.
Table F.3: Comparison of Test ROC AUC of GRAND-SLAMIN with Group Lasso.
Model
Group Lasso
GRAND-SLAMIN
Group Lasso (#features)
GRAND-SLAMIN
Dataset (AUC)
(AUC)
(#features)
Adult
91.19
91.54 + 0.14
14
12
Spambase 98.32
98.81 + 0.04
57
44
Madelon
65,22
90.13 + 1.03
500
F.4 GRAND-SLAMII with different shape functions (Soft Trees vs MLPs)
Next, compare GRAND-SLAMIN with different shape functions. In particular, study the effect of using neural network shape functions instead of soft tree shape functions. We consider multilayer perceptrons (MLPs) as the functional form for each of the shape functions. We compare AUC performance on 5 datasets in Table F.4. Interestingly, it seems to us that soft trees seem to have an edge over MLPs across some datasets, when MLPs matched in number of parameters soft trees.
Table F.4: Soft shape functions MLP shape functions.
Dodaser\Model
BRAND-SLAMIN with MLPs
with Soft Trees
Magic
93.13:0.12
93.36-1..0
Churn
92.3
92.40
MiniBooNE
97.4111.22
97.77
Spambase
98.27=0.13
98.55
News
72.87+00009
73.24+0004
MLPPs with varying complexity We further experimented with MLPs our framework, where we vary the complexity of the individual components. We show training time on News dataset 1800, N = 40k) in Table F.5. The results show that the active set (learnt by learnable indicators) in the model is the primary contributing factor in terms of timing. The functions (1 or 2) > 64 + + have at least 22x more parameters than (1 or 2) > + 1. However, the time only increases by 1.5x. We also show the AUC performance with MLPs with varying complexity in Table F.6. Interestingly, we didn't observe any improvement in AUCs with more complex components. It seems to that since additive modeling framework is fitting dww-dimesiional components - I-dimensional functions for main effects and 2-dimensional functions for pairwise interactions effects the complexity of the function class doesn't 't need to be too large to get good model accuracy.
(p = 61, # #nteraction
F.5 Effect of entropy regularization
Next, we study the impact of entropy regularization performance and variable selection. It can be hypothesized that there might be a trade-off in the speed at which the binary state the gate variables should be reached. On the one hand, suppression of the uninformative terms should happen fast as
26

---

Table F.5: Training times for different choices of MLP shape functions.
% Components Selected
MLPs with varying complexity
Training times
(Ior + -4
252.4 +
(I + 64 64 +
247.6 + 0.8
25% effects
(1 + +64 - 64 + 64
258.8 + 10.7
(1or2 + +128 - 128 +1
268.5 + 19.1
(Ior - 256 256
-
837.0 + 43.6
(Tor -> -
343.3 + 5.3
(1or 64
-
422.7 + 9.9
75% effects
(1or + +64 64 4 64 +1
506.0 + 34.4
(1 1 + 128 128
+1
552.4 + 151.4
or 256
-
1193.8 0.3
Table F6: AUC performance with MLP shape functions.
MLPs with varying complexity
Test AUC
+ 0r
72.90 + 0.09
or +
64
72.48 0.13
(1 or 2) + +44 64
64
+1
72.58 + 0.17
(1 2) + 128 + 128
+1
72.34 + 0.11
(r + 256 + 256
-
71.86 0.37
the early forward and backward passes all p² interaction trees is computationally prohibitive. On the other hand, making this decision prematurely means that the model might not have had enough time to fit terms before suppressing them. We provide an ablation study for model performance function of entropy regularization in Table F.7. We observe that when the entropy regularization is
Table F.7: Effect of entropy regularization test AUC and component selection Spambase. We report median and mean absolute deviation across 50 runs.
0.00001
0.0001
11.01
0.005
0.01
0.05
0.1
1.0
AUC
08.58-0.02
98.58.10.02
08.54
98.37-10.09
98.40.1.12
98.40.0.15
98.43.10.11
08.30.10.08
#Ellects
Sclected
1653 0
1235 + 164
592 + 173
558 328
535 + 352
507 + 451
721 +: 528
1653 +: 26
too high, the model suffers in performance as some informative terms can also get suppressed too quickly. When the entropy regularization is small, the model can produce very dense solutions, which can hurt interpretability as well as computational scalability. In reasonable range of entropy regularization, there is good region where the model produces sparse solution for high AUC. Interestingly, we also observe that we didn't It see huge sensitivity of variable selection for range of entropy values in the middle.
F.6 Effect of screening performance
Here, we highlight that screening can help improve performance of our model. We consider challenging Madelon dataset from the NIPS-2003 feature selection challenge, where only very small subset of features are informative. The number of total features in the dataset are 500. The number of pairwise interac- tions in this dataset are 125k. This large combinatorial space for variable selection in terms of interaction effects. A large portion of these interactions are non-informative. Pre-training screening rules similar to the ones used by [34] can effectively reduce the combinatorial space. Recall that the actual number of interactions selected by the model is much smaller than the screened set interactions through variable selection by our model while training. We can observe from Fig. F.I that for a range of pre-training screening levels, our model can achieve almost 90% AUC performance this performance is better than
100
80-
Test ROC AUI
20
10
Figure F.I: GRAND-SLAMIN with various pre- training screening levels on Madelon.
Performance of
27

---

all existing approaches for sparse GAMs with interactions. In particular, the state-of-the art toolkit NODE-GAM that performs end-to-end variable seclection achieves 60% AUC.
Screening also plays a role in faster runtimes. For example, when the screened set is 10,000 interactions out of total of 125k, the model can be 3x -x faster. Additionally with screening, the overall memory footprint of the model can be much smaller. The overall memory footprint of the model is dictated by the number of interaction effects. With screening, the initial memory footprint of the model can be reduced without a loss in performance.
F.7 Timing comparison
Our toolkit is highly competitive in terms of training times with all existing ree-based and neural- based toolkits for sparse GAMs with interactions. See Table F.8 for timing comparisons.
Table F.8: Training time in seconds of GRAND-SLAMIN, EBAM, NODE-GA-M, GAMI-Net and SIAN. We report median across 10 runs. Hyphen (-) indicates either the toolkit does not support multiclass e.g., GAMI-Net, SIAN does not fit interaction effects for multiclass e.g., EB M.
DatasetUModel
EB*M
NODE-GA'M
GAMI-Net
SIAN
GRAND-SLAMIA
Magic
140
327
1567
608
430
Adult
284
612
3611
2396
1018
Churn
33
699
2340
298
35
Spambase
297
361
1979
2197
133
Miniboone
1181
1523
39334
64200
1662
Online
402
325
9030
3877
432
Bankruptcy
156
240
2630
3410
57
Madelon
403
899
5217
58500
46
Satimage
133
52
Texture
128
98
Optdigits
320
55
Covertype
604
3144
Activity
291
110
Multiple
1417
184
G Additional visualizations for variance reduction in estimation of main effects under structural constraints
Here we consider tract-level American Community Survey dataset from US Census Bureau Plan- ning Database 2022 [55]. Following [23], we consider reduced dataset with ~ 39 covariates (741 possible pairwise interactions) and consider the self-response as the regression target. We fit GRAND-SLAMII with different structural constraints. We visualize the estimated main effects in the presence of interaction effects for these structural constraints for some features in Figure G.I. We can observe that when additional hierarchy constraints are imposed, the error bars are much smaller across different runs.
28

---

No Hierarchy
Weak Hierarchy
Strong Hierarchy
0.05
0.5
0.05
Main effect
Main affect
-03
-0.
-0.
-02
-02
B00k
Average household Income
Average household Incomee
Average household Income
6.3
0.1
Main effect
Main offect
-03
0.1
00.
20
<0
60
80
100
60
80
100
20
40
60
80
100
Percentage
Percentage
Percentage households
Main effecs
0.2
0.2
Main ofters
0.2
0.1
0.1
0.1
Main
-02
-0.
-02
Percentage linited English speaking househokds
Percentage limited English speaking househokds
Percentage limited English speaking househokds
0.3
0.2
Main affect
0.05
0.08
Main offect
0.05
Percenfisg fagge ol people befow poverty level
Percentage people below puvveryy level
Percenlage of people below poverly level
0.2
0.15
0.15
0.15
Main effect
Main offect
0.05
0.00
0.15
001
00.15
20
40 60
80
100
20 60 80
100
20
40
60
80
100
Parcentage singie-family
Percentage singie-famlly
Parcentage singio-tamily
Figure G.1: Estimated main effects in the presence of interaction effects on American Community Survey dataset [Left] without hierarchy, [Middle] with weak hierarchy and [Right] with strong hierarchy. Strong hierarchy has the smallest error bars. We show visualization for 5 features: (a) Row 1: Average household income, (b) Row 2: Percentage of households with internet access, (c) Row 3: Percentage of limited English speaking households, (d) Row 4: Percentage of people below poverty level, (e) Row 5: Percentage of single-fanily homes.
29


Reviews:
[{"review_id": "B3LKpfhP6t", "reviewer": "Reviewer_nF9H", "summary": "This paper deals with interpretable machine learning. Authors propose a new interpretable model based on generalized additive modeling. Namely they propose a scalable solution for sparse interaction modeling. They compare against related work on interpretable benchmarks.", "questions": "See points in weaknesses.", "limitations": "Non adequate limitations discussion in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ Novel contributions: novel optimization of additive models with indicator variables; backward pass customization for indicator functions.\n+ Really good theoretical analysis that is greatly appreciated, especially the level of detail in the appendix\n+ Well and clearly explained approach\n+ SOTA performance on a wide range of datasets\n+ Very scalable as shown with detailed timing results from the appendix", "weaknesses": "- Somewhat weird choice of datasets. Eg, some standard datasets from papers like NAM, NBM, SPAM, NODE-GAM missing. Most of the datasets presented in this work have a very small difference in performance between the methods, more or less they all perform the same.\n- EBM also has EB2M version, unclear why is it not compared with it, or am I wrongly understanding the tables? Some strong related works with open source libraries missing: NAM, NBM, SPAM, all three are available with 2nd and some even with 3rd order interactions.\n- Unclear why would a use of such a model select no hier, weak hier, or strong hier. Depending on the dataset, different hierarchy performs the best. \n- Can this approach model 3rd order interactions, it would be very interesting to see results on that?\n- Is there any interpretability difference between this model and eg NODE-GAM at 2nd order? Is there any benefit in such a sparsity constrains and trainable selection functions over NODE-GAM? Analysis showing some insight into this would improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for his comments and the time spent in reviewing the paper.\n\n1) **Choice of datasets** A central theme of our work is a sparse selection of main/interaction effects. Our selection of datasets was motivated to consider tabular classification datasets which have a sufficient number of possible pairwise interaction effects. Most of the tabular classification datasets e.g., in NODE-GAM paper consider datasets with $p\\leq30$ (\\#interactions $\\leq \\sim 400$). This is the main motivation to consider a different set of datasets so that sparsity can play a more important role when the number of interactions are not too small. Additionally, the differences in performance across methods is statistically significant --- please see the mean absolute deviations in Tables F.1 and F.2 in our Supplemental document.\n\n2) **EB^2M notation** We used EBM with interactions. We will rename it to  EB^2M.\n\n3) **Comparison with NA^2M, NB^2M and SPAM** NA^2M, NB^2M and SPAM appear to be dense approaches that fit all pairwise interactions. Hence, we compared only with methods that learn sparse interactions. However, based on your comment, we have run new experiments to provide comparisons with NA^2M, NB^2M and SPAM. The performance numbers are given below:\n| Dataset      | NA^2M (Dense) | NB^2M (Dense) | SPAM (Dense) | GRAND-SLAMIN (Sparse)|\n|--------------|---------------|---------------|--------------|----------------------|\n| Magic        | 94.27         | 93.76         | 91.54        | **94.48**            |\n| Adult        | 91.33         | 91.28         | 89.89        | **91.81**            |\n| Churn        | 92.81         | 92.52         | 87.10        | **93.88**            |\n| Satimage     | 98.78         | **98.92**     | 97.73        | 98.84                |\n| Covertype    | 98.16         | **98.44**     | 96.21        | 98.40                |\n| Spambase     | 98.41         | 98.43         | 97.81        | **98.64**            |\n| News         | 71.62         | 71.41         | 70.72        | **73.29**            |\n| Bankruptcy   | **94.01**     | 90.36         | 86.95        | 93.01                |\n\n    For larger datasets e.g., Madelon, Activity and Multiple, these approaches ran out-of-memory on a single V100 Tesla GPU. We generally outperform all these 3 approaches in terms of performance. We also support sparsity and hierarchy for easier interpretability.\n\n3) **No hierarchy vs hierarchy** Generally speaking, additional constraints can help regularize a model or reduce variance, sometimes resulting in improved AUC.\n    Since these are real datasets, we are unable to make definitive statements about the structure of a `true’ underlying model. In practice, the model with the best predictive performance or better interpretability can be chosen depending on context.\n\n4) **3rd order interactions** Our approach can be extended to model 3rd order interactions.\n    In the case of third-order interactions, the structural constraints for strong and weak hierarchy can be defined as described next. For strong hierarchy, if we consider the constraint $f_{i,j,k}\\neq 0 \\Longrightarrow f_i \\neq 0\\text{ and }f_j \\neq 0 \\text{ and } f_k \\neq 0$, this can be modeled as:\n    \\begin{align}\n        q(z_i,z_j,z_k,z_{i,j,k})=z_iz_jz_kz_{i,j,k}\n    \\end{align}\n    For weak hierarchy, if we consider the constraint $ f_{i,j,k}\\neq 0 \\Longrightarrow f_i \\neq 0\\text{ or }f_j \\neq 0 \\text{ or } f_k \\neq 0$, we can model it as:\n    \\begin{align}\n        q(z_i,z_j,z_k,z_{i,j,k})=(z_i + z_j + z_k - z_iz_j - z_iz_k - z_jz_k + z_iz_jz_k) z_{i,j,k}\n    \\end{align}\n    Hence, our sparse selection and hierarchy constraints can add value in terms of model compactness and feature selection in the settings with third-order interactions as well.\n    However, for third-order interactions, the number of third-order interactions can be of the order $~p^3$.\n    Hence, for scalability, this would also require using a pre-training screening approach.\n    Despite the fact that our approach is generalizable to third-order interactions, we do not consider such interactions because these are not considered to be easily interpretable.\n\n5) **NODE-GAM vs Our Approach** There are differences in modeling choices by NODE-GAM and our approach. NODE-GAM parameterizes each tree as a NODE tree using all features. Each component learns to use at most 2 features by the end of training (via the entmax transformation). Sparsity in NODE-GAM is achieved by constraining the number of trees. In contrast, our approach explicitly models the pairwise interactions with soft trees. Each tree is explicitly modeled to rely on a pair of features (using lesser number of parameters than NODE trees). The optimization framework uses trainable selection functions $z$ to selects a subset of trees.\n\n    We also show how additional constraints on $z$ can be imposed to achieve hierarchy (weak or strong).\n    It is unclear how NODE-GAM can be generalized to support hierarchy constraints. These constraints can add more meaningful interpretability and improve variable selection without significantly impacting model performance."}, {"review_id": "IG9IaaIi5w", "reviewer": "Reviewer_WskZ", "summary": "The authors proposed an GAM under structure constraints. By incorporating herarical constraint, the proposed method presents better performance than two stage methods. Theoritical analysis are also provided to bound the generalization error. ", "questions": "1, which screening method is used in your method? Does  it same to that in EBM? ", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A GAM method was proposed, which can be learnt in an end to end manner. By introducing several binary variables, herachical constraints are automatically imposed on the model. \nThe theoritical parts are new. ", "weaknesses": "1, Although the authors claim their model can be learnt efficienctly, there still is p^2 models need to be learnt, which is unpractical for large scale datasets. \n2, There is no explanation for why soft trees are used. Actually, the soft tree can be viewed as a NN with special    structure.  So why MLP is not used? \n3, Although same hierarchy structures are also used in GAMI-NET, nevertheless, why it helps to improve the performance of the model is still unclear to me. The motivation to use such a structure is weak in this paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for his comments and the time spent in reviewing the paper.\n\n1) **Sparsity** The goal of our method is to recover a subset of $p^2$ effects. We achieve this goal with two ideas. \nFor small to medium sized datasets, we recover an active subset via an end-to-end learning optimization procedure.\nFor large-scale datasets, we perform a screening to reduce the initial combinatorial search space from $p\\choose2$ to e.g., 5000 or 10000. Later our end-to-end optimization on this screened set further estimates a much sparse set. Hence the model doesn’t estimate $p^2$ trees. It has been highlighted in prior literature (Nori et al. (2019), Ibrahim et al. (2021)) that using all pairwise interactions is unnecessary, can hurt statistical performance and makes interpretability more challenging. In all cases, the training can be efficiently performed using sparse backpropagation. Sparsifying the model during training makes our approach scalable for settings with large number of interactions.\n\n2) **Soft Trees vs MLP shape functions** Our use of soft trees was primarily motivated by their good performance despite being compact structures. To highlight why Soft trees were selected as opposed to MLPs, we performed new experiments where our models were estimated with MLP shape functions. For a fair comparison, we used MLPs, which were matched in parameters to depth-4 soft trees. We observed, soft trees seemed to be more expressive as shown below.\n| Dataset | GRAND-SLAMIN with MLP basis | GRAND-SLAMIN with Soft Trees basis |\n|------------|----------------------------|-----------------------------------|\n| Magic      | 93.18                      | **94.48**                    |\n| Adult      | 90.03                      | **91.81**                   |\n| Churn      | 92.55                      | **93.88**                    |\n| MiniBooNE  | 97.48                      | **97.85**                   |\n| Spambase   | 98.16                      | **98.64**                   |\n| News       | 73.19                      | **73.29**                    |\n\n3) **Motivation of Hierarchy Structure**\n    The motivation to use the structural constraints is multi-fold (i) better interpretability (Bien et al. (2013), Enouen and Liu (2022) ) (ii)  more compact feature selection as shown in Table 4. Lesser number of features also improve interpretability (iii) lesser variance in estimation of main effects (in the presence of interaction effects) --- see visualizations (Figure R1) in the global rebuttal document. This allows the user to have more trust on model interpretability explanations.\n\n    Generally speaking, additional constraints can help regularize a model or reduce variance, sometimes resulting in improved AUC.\n    Since these are real datasets, we are unable to make definitive statements about the structure of a *true* underlying model.\n\n4) **Method of Screening** For screening on large-scale datasets, we used CART for each pairwise interaction and sorted the interaction effects based on AUC performance to select a screened set of interaction effects. Note that this is done only at the start of training. For example, on Madelon dataset. this screening took 30 second to fit CART for all of the 125000 interaction effects separately in a multithreaded environment.\n    The selection method in EBM also used shallow tree-like models to select the interaction effects. However, note that EBM uses their screening approach in their greedy forward stagewise selection to select the best interaction effect in their boosting algorithm.\n\nReferences:\n1) Jacob Bien, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. The Annals of Statistics, 41(3):1111–1141, June 2013.\n2) James Enouen and Yan Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.\n3) Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. ArXiv, abs/1909.09223, 2019.\n4) Shibal Ibrahim, Rahul Mazumder, Peter Radchenko, and Emanuel Ben-David. Predicting census survey response rates via interpretable nonparametric additive models with structured interactions, 2022."}, {"review_id": "l1sKumaT5u", "reviewer": "Reviewer_xnEj", "summary": "The authors introduce GRAND-SLAMIN’, a framework for end-to-end learning of sparse Generalized Additive Models (GAMs). GRAND-SLAMIN’ utilizes soft neural decision trees as base learners and incorporates support for second-order interactions and structural constraints. Sparse variable selection is achieved via a smooth relaxation of binary gate variables which play the role of suppressing unused terms. These gates can also be combined to enable structural constraints, restricting second-order interaction to participating first-order terms. This formulation facilitates a sparse backpropagation strategy in which suppressed terms and interactions are pruned from the model early in training. The experiments demonstrate promising results in terms of prediction accuracy and scalability. The paper also establishes novel non-asymptotic prediction bounds for GAM estimators with soft tree-based shape functions, providing a theoretical foundation for their approach.", "questions": "* I think the inclusion of both local and global visualizations of the recovered shape functions would significantly improve the clarity and impact of the paper. Illustrating the neural soft decision trees would help readers gain a deeper understanding of how the input features contribute to model predictions and encourage its adoption down the line.\n\n* My intuition leads me to believe that there might be a trade-off in the speed at which the binary state of the gate variables should be reached. On the one-hand suppression of the un-informative terms should happen fast as the early forward and backward passes of all $p^2$ interaction trees is computationally prohibitive. On the other, making this decision prematurely means that the model might not have had enough time to fit terms before suppressing them. Maybe you could observe as an ablation model performance as a function of the entropy regularizer hyperparameter $\\tau$?\n\n* Related to the above point, do you find that models with different random initializations agree on the usefulness of features, meaning, are the selected features and selected interactions consistent between runs? Perhaps there is a metric which can quantify this (for ex. Jaccard similarity) and you could numerically compare with the screening approach of the other baselines.\n\n* I am not familiar with neural soft decision trees but they seem like they could be quite sensitive to the choice of hyperparameters, and in particular the choice of learning rate. How difficult are they to train in practice? Can you provide some details on the Optuna hyperparameter search: Is it very time consuming? Is the approximated hyperparameter objective function smooth?\n\n", "limitations": "No, I think the limitations could be better addressed by dedicating a section in the main text.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The end-to-end differentiable parameterization for feature selection is a novel and valuable contribution. It greatly simplifies the implementation of the feature selection and feature interaction detection procedure and integrates well within the model optimization process. It also enables the enforcement of structural constraints, further enhancing the flexibility and interpretability of the framework.\n\n* The inclusion of non-asymptotic bounds provide a theoretical basis for the behavior and performance of the proposed approach, showcasing that robustness to noise improves as more samples are observed.\n\n* The experimental results also support the intuition that incorporating feature selection into the Generalized Additive Model (GAM) can benefit robustness to noise and its generalization. This finding is interesting and can maybe be linked to previous work on feature selection and robustness: L1 regularized regression (Lasso) is a form of robustness to feature-wise disturbance [Xu, 2008] and selecting relevant subsets of features in feature bagging is known to help model performance [Tian, 2021].\n\n[Xu, 2008]: Xu, H., Caramanis, C., & Mannor, S. (2008). Robust Regression and Lasso. Advances in Neural Information Processing Systems, 21. [https://proceedings.neurips.cc/paper_files/paper...](https://proceedings.neurips.cc/paper_files/paper/2008/hash/24681928425f5a9133504de568f5f6df-Abstract.html)\\\n[Tian, 2021]: Tian, Y., & Feng, Y. (2021). RaSE: Random Subspace Ensemble Classification. Journal of Machine Learning Research, 22(45), 1–93.\nhttps://jmlr.org/papers/v22/20-600.html", "weaknesses": "* The lack of visualizations of local and global explanations which can be generated with the proposed framework hinders the assessment of interpretability. Visualizations play an important role in presenting and communicating the decision-making process of interpretable models and shedding light on the contribution of individual features and their interactions.\n\n* The binary gates snap into place reaching a permanent state of 0 or 1 as training progresses. This is understandably a desired property of smoothstep activation, however it means that the model cannot update decisions regarding the usefulness of input features once this permanent state is reached.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for his comments and the time spent in reviewing the paper.\n\n1) **Visualization** We provide a new visualization study to further highlight an important contribution of our paper.\n    In particular, our framework can support models with structural constraints. Hence, we study the effect of these constraints on the stability of learning main effects (in the presence of interactions) when these structural constraints are imposed or not. For this exercise, we consider bikesharing dataset. We visualize some of the main effects in the presence/absence of hierarchy in Figure \n    R1 in rebuttal figure document. We can observe that when additional hierarchy constraints are imposed, the error bars are much more compact across different runs. This can potentially increase the trust you can have on the model for deriving interpretability insights.\n    The figures also highlight that soft trees are smooth basis functions for learning effects.\n\n2) **Speed of reaching binary states** It is correct that reaching the binary state should not be premature. For this reason, our learnable indicator variables are initialized to be around $0.5$. We also do not impose a very large entropy regularization. This gives the algorithm sufficient time to learn the informative terms. Additionally, a smaller learning rate choice can also be useful to give the model more time to find the informative terms.\n\n3) **Effect of entropy on performance and variable selection** We provide an ablation study on Spambase for effect of entropy regularization on model performance and variable selection. Please see Figure R2 in figure rebuttal document. For each entropy regularization, we select the best model corresponding to the optimal selection regularization ($\\lambda$) based on a validation set. Overall, we see small effect of entropy regularization on variable selection and very little effect on test AUC.\n\n4) **Sensitivity to hyperparameters** We observed soft trees to be generally not very sensitive to the choice of typical hyperparameters, e.g., batch size and learning rates. In our tuning, we only considered 3 learning rates $\\{0.05, 0.01, 0.005\\}$ and 2 different batch sizes $\\{64,256\\}$. However, the most important parameter for our modeling framework was $\\lambda$ for sparsity and hence we tuned over a wide range of 11 $\\lambda$ values in the set $(0, 1e-6, \\cdots, 1e-3)$. We can highlight this more in the camera-ready version. More information about the hyperparameter tuning with Optuna are included in the Supplement."}, {"review_id": "7wo84qFrtd", "reviewer": "Reviewer_QfPH", "summary": "The present paper presents a method to incprorate pairwise interaction effects into differentiable generalized additive models while retaining scalability. \n\nThe authors pose the fitting of the GAM with pairwise interactions as a combinatorial optimization problem with indicator variables and hierarchy constraints added to the pairwise interactions. To be able to efficiently solve this optimization problem, the authors relax the indicator variables using the smooth step function, which is exactly zero or one at the tails of the function, allowing them to discard certain functions during the forward pass. As shape functions, the authors use soft trees, as opposed to e.g. NAMs which use neural networks. To ensure that the gates reach binary state by the end of training they add an entropy regularizer.The authors provide prediction bounds for sparse additive models.\n\nIn their experiments, they compare with existing method such as Node-GAM which incorporate pairwise interactions and show favourable results.", "questions": "- I think the contribution of the paper is sound, but currently their experimental evaluation is weak as the comparison between methods appears to be carried out on a single seed (I saw some of the results in the appendix have results over additional seeds, but these results need to be in the main paper.). I would consider increasing my score if this is addressed.\n\n- How sensitive is the variable selection to the entropy regularization strength?", "limitations": "- Limitations are not discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well written and the structure is easy to follow.\n- The methodological contribution is neat and only adds few hyperparameters.\n- The proposed method produces sparser pairwise interactions which are also faster to train than the compared methods.", "weaknesses": "- Table 2, Table 3, Table 4, Figure 2 please provide the number of runs/seeds and either the standard error, mean absolute deviation or bootstrap confidence intervals for the results. The present analysis could be misleading if it was only carried out over one seed.\n- As your method is applicable to any shape function class, it would be interesting to show results for shape functions like in Neural Additive Models modelled via MLPs.\n- I would have expected the models with weak hierarchy to strictly outperform the models with strong hierarchy. Could you comment on this?\n\nTypos:\n- Line 26 'xchallenging'\n- Line 46 'pose'\n- Line 87 'first step'\n- Line 328 'significantly/much faster'\n\n\nImprovements:\n- Consider increasing the font size of Figure 1 and Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for his comments and the time spent in reviewing the paper.\n\n1) **Multiple runs across seeds** Tables F.1 and F.2 in the Supplement correspond to 10 runs/seeds for the optimal trials reported in Table 2 and Table 3. We will use F.1 and F.2 in the main text in the camera-ready version. \n    We will also replace Table 4 with it's multiple seeds version in the camera-ready draft.\n\n2) **Comparison with MLP shape functions** We have performed new experiments for GRAND-SLAMIN with MLP shape functions.\n    For a fair comparison with Soft trees, we used MLPs, which had similar number of parameters as depth-4 soft trees. We observed soft trees  to be more expressive when parameter-matched as shown below.\n| Dataset   | GRAND-SLAMIN with MLP basis | GRAND-SLAMIN with Soft Trees basis |\n|-----------|-----------------------------|------------------------------------|\n| Magic     | 93.18                       | **94.48**                          |\n| Adult     | 90.03                       | **91.81**                          |\n| Churn     | 92.55                       | **93.88**                          |\n| MiniBooNE | 97.48                       | **97.85**                          |\n| Spambase  | 98.16                       | **98.64**                          |\n| News      | 73.19                       | **73.29**                          |\n\n3) **Weak hierarchy vs strong hierarchy** It could be expected that models with hierarchy generally may outperform models with strong hierarchy as the former imposes lesser restrictions. The reviewer can refer to tables F.1 and F.2 in the Supplement (which provide median numbers when the models are run with different seeds). Indeed, typically, models with weak hierarchy tend to outperform models with strong hierarchy. However, there are some 1-2 datasets where strong hierarchy still outperforms in terms of performance in comparison with weak hierarchy.\n    \n    However, strong hierarchy has other advantages. Models with strong hierarchy are considered to be more interpretable. They also lead to more compact feature selection as shown in Table 4. Models with lesser number of features are easier to interpret.\n\n4) **Typos and Figure fonts** Thanks for bringing this to our attention. Typos are addressed and font size of Figures 1 and 2 will be increased for the camera-ready revision.\n\n5) **Sensitivity to Entropy regularization** We provide an ablation study on Spambase for effect of entropy regularization on model performance and variable selection. Please see Figure R2 in figure rebuttal document. For each entropy regularization, we select the best model corresponding to the optimal selection regularization ($\\lambda$) based on a validation set. Overall, we see small effect of entropy regularization on variable selection and very little effect on test AUC.\n"}, {"review_id": "cbatSkz9Pa", "reviewer": "Reviewer_W32k", "summary": "This paper introduces hierarchical structure constraints to improve the sparsity of selected features in interaction-aware GAM. The authors use soft relaxations to enable gradient descent during this procedure. Theoretical analyses are given to show statistical properties when using soft tree shape functions in their method.", "questions": "1.\tWhat do $n$, $\\bar{u}(f)$, and $\\vee $ mean?\n2.\tWhat do $a_n$ and $b_n$ represent in the model?\n3.\tHow does the proposed method improve upon other methods that support hierarchy constraints, such as GAMI-Net and SIAN?", "limitations": "There is no potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tProvides a simple but reasonable design of hierarchical constraints on the sparsity of feature selection in GAM.\n2.\tIntroduces background information and related works in detail.\n3.\tOffers an efficient implementation that drops unused features during training.", "weaknesses": "1.\tThe technical contribution of this paper is limited. Its framework mainly adopts the existing GA2M with tree-shaped functions. The main contribution is introducing hierarchical constraints to restrict the sparsity of selected features. This is accomplished through learnable indicators that consider the hierarchical constraints. However, this approach is somewhat trivial, as learning Boolean indicators for sparsifying inputs is a common practice. Also, many methods, such as based on Lasso or Gumbel-sigmoid, have been proposed to enable gradient descent on such settings.\n2.\tThe theoretical analysis does not appear to be very relevant to the key contribution claimed. Specifically, the paper discusses the statistical properties of an additive model with soft tree-shaped functions, but does not consider any hierarchy constraints. Since GAM with tree-shaped functions is an existing technique and this paper mainly proposes hierarchy constraints, these analyses do not provide many insights into the methodologies proposed in this paper.\n3.\tThe presentation of this paper is poor. In section 5, many symbols appear without specific explanations, making the analysis confusing and difficult to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for his comments and the time spent in reviewing the paper.\n1) **Response to Limited Contribution**\n\n  a) **Discussion on GA^2M** We want to clarify that our approach is very different from GA^2M with tree-shaped functions proposed by Lou et al. (2013) and later packaged as EBM. GA^2M is a two-stage approach for the main and interaction effects. In the first stage, they use gradient boosting to fit the main effects---the boosting procedure cycles through all features in a round-robin fashion to fit *all* main effects. In the second stage, they propose a greedy forward stagewise selection strategy to select a small subset of pairwise interactions -- these interaction effects are fitted with shallow tree-like models via gradient boosting. The GA^2M procedure is not based on a joint penalized optimization criterion and, to our knowledge, no statistical guarantees for it are known.\n\nIn contrast, the main features of our proposal is to introduce learnable indicator variables to be able to model different structural constraints (e.g., sparse interactions, hierarchy, etc.) Our approach is therefore more flexible. Moreover, ours is an end-to-end optimization framework.\n\n  b) **Discussion on Probabalistic Boolean Indicators, e.g., via Gumbel-sigmoid** Techniques like Gumble-sigmoid have been used in other applications to enforce sparsity. However, in prior GAM with interactions literature, to our knowledge, such stochastic parameterization of boolean indicators have not been used for selection. Moreover, such techniques typically lead to fairly dense solutions because of their probabalistic modeling choice. At inference time, such methods use heuristics via thresholding for actual sparsification. With our Smooth-step approach, our selection is such that the model at training and inference are the same. Additionally, sparsity in Smooth-Step can allow us to successively eliminate the non-selected effects during the course of training, so the time/epoch becomes smaller as epochs progresses, making the algorithm more scalable.\n\n  c) **Discussion/Experiments on Lasso** Lasso is also a possible choice for sparse selection. We compare against a version of Lasso:\n$$min~\\hat{E}[l(y, \\sum_{i\\in[p]}f_i(x_i) + \\sum_{i<j}f_{i,j})] + \\lambda G,$$\nwhere $\\hat{E}$ denotes the empirical loss on the training dataset, $G=\\sum_i||f_i|| + \\sum_{i<j} ||f_{i,j}||$ is the group lasso regularization. Overall, our approach consistently outperforms Lasso in terms of AUC performance and variable selection. We provide a subset of results on a few datasets below:\n|       Dataset| Lasso (AUC) | GRAND-SLAMIN    (AUC)         | Lasso   (\\#features)     | GRAND-SLAMIN  (\\#features) |\n|------------|-------|-------------------------|--------------|--------------|\n| Adult      | 91.19 | **91.54**$\\pm0.14$       | 14           | **12**       |\n| Spambase   | 98.32 | **98.81**$\\pm0.04$        | 57           | **44**       |\n| Madelon    | 65.22 | **90.13**$\\pm1.03$        | 500          | **15**       |\n\n2) **Theoretical Contribution** To our knowledge, our work is a first to establish theoretical results for learning with soft trees, specifically in the context of GAMs. To construct our theoretical framework, we chose the model with no structural constraints which is one of the main models we discuss. Extending our theoretical framework to include structural constraints is an interesting avenue of research left for future work.\n\n3)  **Clarification on notations in Section 5** $n$ denotes the number of (training) samples, $\\lor$ is used as a shorthand for maximum. $\\bar{u}(f)$ can be considered the largest weight (in absolute value) appearing in the model. These will be clarified in the revised version. The sequences $a_n,b_n$ do not necessarily represent any parameters in the model, rather they are constructed as a part of proof (see Lemma B.7 for example). Informally, the fact that $a_n\\to 0$ allows us to deduce from Theorem 2 that the error rate $n^{-1/2}$ is achievable as $n\\to\\infty$.\n\n4) **GAMI-Net and SIAN vs Our Approach** GAMI-Net and SIAN both are two-stage learning procedures. In the first stage, they use some screening approach to exclusively rely on pre-training screening to select top k important hierarchical interactions. This is a greedy approach. In the second stage, these models fit on all their screened subset. No further selection is performed in this stage.\n\nOur proposal is to introduce learnable indicator variables to be able to model different hierarchy constraints. In contrast to GAMI-Net and SIAN, our approach can learn under such constraints is an end-to-end fashion. Additionally, our approach can also support sparse backpropagation ."}]

Author rebuttal:
We thank all reviewers for their comments and the time spent in reviewing the paper.

Please see our attached figure rebuttal document.

Meta-review: The reviewers praise the novelty of the approach, the theoretical analysis (esp. non-asymptotic results), the empirical results, the scalability, and the end-to-end trainability. The main criticisms are the choice of benchmark datasets, the lack of some baselines, the applicability to third-order interactions, the motivation for the hierarchical approach, lack of visualizations, and sensitivity to hyperparameter choices. However, most, if not all, of these concerns have been addressed by the rebuttal.
Decision: Accept (poster)