Title: GRAND-SLAMIN’ Interpretable Additive Modeling with Structural Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generalized additive models

component selection

hierarchy

interpretability
Abstract: Generalized Additive Models (GAMs) are a family of flexible and interpretable models with old roots in statistics. GAMs are often used with pairwise interactions to improve model accuracy while still retaining flexibility and interpretability but lead to computational challenges as we are dealing with order of $p^2$ terms. It is desirable to restrict the number of components (i.e., encourage sparsity) for easier interpretability, and better computational and statistical properties. Earlier approaches, considering sparse pairwise interactions, have limited scalability, especially when imposing additional structural interpretability constraints. We propose a flexible GRAND-SLAMIN framework that can learn GAMs with interactions under sparsity and additional structural constraints in a differentiable end-to-end fashion. We customize first-order gradient-based optimization to perform sparse backpropagation to exploit sparsity in additive effects for any differentiable loss function in a GPU-compatible manner. Additionally, we establish novel non-asymptotic prediction bounds for our estimators with tree-based shape functions. Numerical experiments on real-world datasets show that our toolkit performs favorably in terms of performance, variable selection and scalability when compared with popular toolkits to fit GAMs with interactions. Our work expands the landscape of interpretable modeling while maintaining prediction accuracy competitive with non-interpretable black-box models. Our code is available at https://github.com/mazumder-lab/grandslamin.

GRAND-SLAMIN' Interpretable Additive Modeling with Structural Constraints

Abstract

Generalized Additive Models (GAMs) are a family of flexible and interpretable models with old roots in statistics. GAMs are often used with pairwise interactions to improve model accuracy while still retaining flexibility and interpretability but lead to computational challenges as we are dealing with order of p 2 terms. It is desirable to restrict the number of components (i.e., encourage sparsity) for easier interpretability, and better computational and statistical properties. Earlier approaches, considering sparse pairwise interactions, have limited scalability, especially when imposing additional structural interpretability constraints. We propose a flexible GRAND-SLAMIN framework that can learn GAMs with interactions under sparsity and additional structural constraints in a differentiable end-to-end fashion. We customize first-order gradient-based optimization to perform sparse backpropagation to exploit sparsity in additive effects for any differentiable loss function in a GPU-compatible manner. Additionally, we establish novel nonasymptotic prediction bounds for our estimators with tree-based shape functions. Numerical experiments on real-world datasets show that our toolkit performs favorably in terms of performance, variable selection and scalability when compared with popular toolkits to fit GAMs with interactions. Our work expands the landscape of interpretable modeling while maintaining prediction accuracy competitive with non-interpretable black-box models. Our code is available at https://github.com/mazumder-lab/grandslamin .

1 Introduction

Many state-of-the-art learners e.g., tree ensembles, neural networks, kernel support vector machines, can be difficult to interpret. There have been various efforts to derive some post-training explainability from these models -see [5] for a survey. Post-hoc explainability attempts to explain black-box prediction with interpretable instance-specific approximations e.g, LIME [49] and SHAP [35]. However, such approximations are known to be unstable [12, 28], expensive [51] and inaccurate [32]. Hence, it is desirable to consider modeling approaches that are inherently interpretable.

Amongst classical approaches, there are some models that have inherent interpretability e.g., Linear models, CART [4] and Generalized Additive Models (GAMs)[14]. GAMs [15] which have old roots in statistics are considered a front-runner in the context of interpretable modeling. They consider an additive model of the main effects of the form: g (I E[ y ]) = ∑ j ∈ [ p ] f j ( x j ) , where x j denotes the j th feature in input x ∈ R p , f j is a univariate shape function and g denotes the link function that adapts the model to various settings such as regression or classification. GAMs are considered easy to interpret as the impact of each feature can be understood via visualizing the corresponding shape function e.g., plotting f j ( x j ) vs x j . However, such models often suffer in prediction performance when compared to black-box methods e.g., deep neural networks (DNNs). This can be attributed in part to the fact that GAMs do not consider interactions among covariates.

Rahul Mazumder MIT Cambridge, MA

There has been some exciting work that aims to reduce this performance gap by considering GAMs with pairwise interactions [see, for example, 34, 60, 6, 46, 9, 10, and the references therein]. GAMs with pairwise interactions consider a model of the following form:

where f j,k is a bivariate shape function and I ⊆ { (1 , 2) , (1 , 3) , ..., ( p -1 , p ) } denotes the set of all pairwise interactions. Under this model, f j ( x j ) is the j -main effect and f j,k ( x j , x k ) is the ( j, k ) -th interaction effect. Pairwise interactions are considered interpretable as each of the bivariate shape function f j,k can be visualized as a heatmap on an x j , x k -plane. Despite their appeal, GAMs with pairwise interactions pose several challenges: (i) Learning all pairwise interaction effects of the order of p 2 lead to computational and statistical challenges. (ii) Performing component selection such that only a few of the components { f j } and { f j,k } are nonzero in an end-to-end fashion (while training) is a hard combinatorial optimization problem. We remind the reader that component selection is needed to aid interpretability. (iii) Imposing structural constraints on the interaction effects, e.g., hierarchy [42, 13, 7, 39, 3, 60, 10] makes the associated optimization task more complex.

In this paper, we introduce a novel GRAND-SLAMIN framework that allows for a flexible way to do component selection in GAMs with interactions under additional structural constraints in an end-to-end fashion. In particular, we introduce an alternative formulation of GAMs with interactions with additional binary variables. Next, we smooth these binary variables so that we can effectively learn these components via first-order methods in smooth optimization (e.g, SGD). Our formulation appears to have an edge over existing methods in terms of (i) model flexibility for enhanced structural interpretability and (ii) computational efficiency. First, the binary variables allow us to impose in the model (a) component selection constraints and (b) additional structural constraints (e.g, hierarchy) via a unified optimization formulation. Both of constraints (a), (b) can aid interpretability, model compression, and result in faster inference and better statistical properties. Second, the our smoothing procedure for the binary variables allows us to have customized algorithms that exploit sparsity in the forward and backward pass of the backpropagation algorithm.

For structural interpretability, we study two notions: weak and strong hierarchy [42, 13, 7, 39, 3].

̸

̸

̸

̸

̸

̸

Weak hierarchy allows an interaction effect f j,k to be selected if either main effect f j or f k is selected. Strong hierarchy allows for an interaction effect f j,k to be selected only if both main effects f j and f k are selected. Such hierarchy constraints are popular in high-dimensional statistics: (i) They lead to more interpretable models [36, 3, 10], (ii) They promote practical sparsity, i.e., reduce the number of features that need to be measured when making new predictions (see Sec. 6.2) - this can reduce future data collection costs [3, 61], (iii) Additional constraints can also help regularize a model, sometimes resulting in improved AUC (see Sec. 6.1). (iv) They can also reduce variance in estimation of main effects in the presence of interaction effects (see Sec. 6.4), allowing the user to have more 'trust' on model interpretability explanations.

Contributions. To summarise, while it's well acknowledged that GAMs with sparse interactions are a useful flexible family of explainable models, learning them pose significant computational challenges due to the combinatorial nature of the associated optimization problem. Our technical contributions in this paper be summarized as:

We propose a novel optimization formulation that makes use of indicator (binary) variables. The indicator variables allow us to impose both (a) component selection and (b) structural constraints in an end-to-end fashion. We consider a smooth and continuous parameterization of the binary variables so that the optimization objective is differentiable (for a smooth training loss) and hence amenable to first order methods such as SGD.

We show the flexibility of our framework by considering two different notions of hierarchy. While these constraints improve interpretability, they make the combinatorial problem more challenging [3, 17]. We propose end-to-end algorithms to train these models, making our approach quite different from existing neural-based toolkits [60, 10].

We exploit sparsity in the indicator variables during the course of the training for sparse forward and backward passes in a customized backpropagation algorithm in a GPU-compatible manner. This provides speedups on training times up to a factor of 10 × over standard backpropagation.

We study novel statistical properties of our model, and present non-asymptotic prediction error bounds. Different from earlier work, our results apply to learning with tree-based shape functions (for both main and interaction effects).

We introduce a new open-source toolkit GRAND-SLAMIN and perform experiments on a collection of 16 real-world datasets to demonstrate the effectiveness of our toolkit in terms of prediction performance, variable selection and scalability.

2 Related Work

GAMs. GAMs have a long history in statistics [14] and have been extensively studied. They're often studied with smooth spline shape functions [see, e.g, 37, 47, 20, 63, 62, and references therein]. Some works have studied tree-based shape functions [33] and neural basis functions [1, 58].

GAMs with all pairwise interactions. In this thread, [9] study low-rank decomposition with neural network shape functions; [46] fit all pairwise interactions using shared neural bases.

Sparse GAMs with interactions. [31] introduced COSSO, which penalizes the sum of the Sobolev norms of the functional components, producing sparse models. [23] propose ELAAN, which is an ℓ 0 -regularized formulation with smooth cubic splines. [23] demonstrate the usefulness of their approach in a regression setting in terms of compact component selection and efficiency on a large-scale Census survey response prediction. [34, 40, 6] explore tree-based shape functions for fitting additive models with sparse interactions. [34] consider all main effects and a subset of pairwise interactions; the subset of interactions are selected via greedy stage-wise interaction detection heuristics. [40] provide an efficient implementation of the above approach as Explainable Boosting Machines (EBMs). [6] propose NODE-GA 2 M: an end-to-end learning approach with differentiable neural oblivious decision (NODE) trees [44]. Component selection in NODE-GAM is achieved by constraining the number of trees, and each tree learns to use one or two features via entmax transformation [43].

Structural Constraints. Structural interpretability constraints such as hierarchical interactions, have been studied for both linear settings [3, 30, 59, 17] and nonparametric settings [45, 60, 10, 23]. We briefly review prior work on nonparametric hierarchical interactions as it relates to this paper. [60] proposed GAMI-Net, which is a multi-stage neural-based approach that fits all main effects and a subset of Topk interaction effects, selected via a fast interaction screening method [34]. Amongst this screened set, interactions that satisfy the weak hierarchy are later used to fit interaction effects. They also prune some main and interaction effects after training based on a variation-based ranking measure. [10] proposed SIAN, which uses Archipelago [54] to measure the strength of each main and interaction effect from a trained DNN, and then screens (i.e., selects a subset of candidate main and interaction effects) using Archipelago scores to identify main effects and interaction effects that obey strong hierarchy. Then, it fits a GAM model with the screened main and interaction effects. [60] and [10] only support screening of interactions obeying hierarchy before the training for interaction effects is done. None of these approaches impose hierarchy while training with interactions. [23] with their ELAAN framework also consider strong hierarchy in the presence of ℓ 0 -regularized formulation with splines for the least squares loss (regression). ELAAN has a two-stage approach: It selects a candidate set of interactions and then applies commercial mixed integer programming solvers to learn sparse interactions under a hierarchy constraint. This approach would require customized algorithms to adapt to different loss objectives e.g., multiclass classification. To our knowledge, current techniques for sparse hierarchical (nonparametric) interactions are not based on end-to-end differentiable training: they can be limited in flexibility and scalability-a gap we intend to fill in this work.

Hier.=Hierarchy, Reg.=Regression, Classif.=Classification, Bin.=Binary, Relax.=Relaxation

Note [9] and [10] also consider higher-order interactions (beyond two-way ones), which can be hard to interpret. For convenience, Table 1 summarizes some relevant work on Sparse GAMs with interactions and possible structural constraints.

3 Problem Formulation

We first present in Sec. 3.1 an alternative formulation of GAMs with interactions using binary variables for imposing sparsity and structural constraints. Next, in Sec. 3.2, we present a smooth reformulation of the objective that can be solved with first-order gradient-based methods.

3.1 An optimization formulation with binary variables

We first present an alternative formulation of GAMs with interactions under sparsity with/without additional structured hierarchy constraints. Let us consider the parameterization:

with main effects f j ( · ) , interaction effects f j,k ( · ) and binary gates z j and q ( z j , z k , z j,k ) . We consider three different parameterizations for q ( · ) , satisfying the following different constraints:

The binary gates z j and q ( z j , z k , z j,k ) play the role of selection. In particular, when z j = 0 , the corresponding j -th main effect f j ( · ) is excluded from our additive model (4). Similarly, when q ( z j , z k , z j,k ) = 0 , the corresponding ( j, k ) -th interaction effect f j,k ( · ) is excluded. Then, we can formulate the regularized objective as:

where the first term denotes empirical loss over the training data, the penalty term controls model sparsity: λ ≥ 0 is the selection penalty, α ∈ [1 , ∞ ) controls the relative selection strength of main and interaction effects. We refer to the framework in (8) under the different constraints (5)-(7) as GRAND-SLAMIN 1 . We discuss extension of this framework to third-order interactions in Supplement Sec. D. However, we do not consider third-order interactions in our experiments as third-order interactions are hard to interpret.

The formulation in (8) with binary variables z j , z j,k and functions f j , f j,k with any of the constraint sets (5)-(7) is a challenging discrete optimization problem (with binary variables) and is not amenable to differentiable training via SGD (for example). Sec. 3.2 explores approximate solutions to (8) using a smooth reformulation of the binary variables. Intuitively, we rely on continuous relaxations of the binary variables z 's and parameterize f 's with smooth tree-based shape functions. The reformulation allows us to use first-order methods.

3.2 A Smooth Reformulation of Problem (8)

We discuss a smooth reformulation of the objective in (8). We describe an approach to parameterize the continuous relaxation of the binary variables z j , z j,k with a Smooth-Step function [18] and use smooth tree-based shape functions to model { f i } , { f j,k } .

3.2.1 Relaxing Binary Variables with Smooth Gates

We present an approach to smooth the binary gates z 's in (8) using a smooth-step function [18], which we define next.

Smooth-Step Function. Smooth-step function is a continuously differentiable function, similar in shape to the logistic function. However, unlike the logistic function, the smooth-step function can output 0 and 1 exactly for sufficiently large magnitudes of the input (see Appendix B for details). This function has been used for smoothing binary representations for conditional computation [18, 19, 21].

Weparameterize each of the z 's in (8) as S ( µ ) , where µ ∈ R is a learnable parameter and S ( · ) denotes the Smooth-step function. We parameterize the additive function as: f = ∑ j ∈ [ p ] f j ( x j ) S ( µ j ) + ∑ ( j,k ) ∈I f j,k ( x j , x k ) q ( S ( µ j ) , S ( µ k ) , S ( µ j,k ))) and optimize the following objective:

Note that S ( µ j ) and S ( µ j,k ) are continuously differentiable, so the formulation in (9) is amenable to first-order gradient-based methods (e.g, SGD).

Achieving binary gates. To encourage each of the S ( µ j ) 's and S ( µ j,k ) 's to achieve binary state (and, not fractional) by the end of training, we add an entropy regularizer τ ( ∑ j ∈ [ p ] Ω( S ( µ j )) + ∑ ( j,k ) ∈I Ω( S ( µ j,k ))) where Ω( S ( µ )) = -( S ( µ ) log S ( µ ) + (1 -S ( µ )) log(1 -S ( µ ))) and τ ≥ 0 controls how quickly each of the gates S ( µ ) converges to a binary z .

3.2.2 Soft trees

We use soft trees [25, 24, 53, 11]-based on hyperplane splits (univariate or bivariate) and constant leaf nodes - as shape functions to parameterize the main effects f j ( · ) and the pairwise interaction effects f j,k ( · ) . See Figure 1 for an illustration. Soft trees were introduced as hierarchical mixture of experts by [25] and further developed by [24, 53, 11]. They allow for end-to-end learning [27, 18, 22]. They also have efficient implementations when learning tree ensembles [22]. A detailed definition of soft tress is given in Appendix A.

4 Efficient Implementation

We discuss a fast implementation of our approach. The key elements are: (i) Tensor parameterization of trees, (ii) Sparse backpropagation, and (iii) Complementary screening heuristics.

Tensor Parameterization of Additive Effects.

Typically, in neural-based additive model implementations [1, 60], a separate network module is constructed for each shape function. The outputs from each model are sequentially computed and combined additively. This can create bottleneck in scaling these models. Some recent approaches e.g., [10] try to work around this ap-

1

2

3

p

1

2

3

f(): jth main effect

Cj

o(wjaj)

1-9

(wi2)

(Wa

(wa)

(waj)

(wai)

02

Zje网 f(∞j)j

fj,x():(j,k)th interaction

o(wi;+wi

1,2k

i,k

02

03

jk

(j,k)EI

fjk(xjk)q(j,k,j,k)

proach by constructing a large block-wise network to compute representations of all shape functions simultaneously. However, this comes at a cost of large memory footprint. For tree-based shape functions, drawing inspiration from [22], we can implement a tensor-based formulation of all shape functions, leveraging the fact that each tree has the same depth. This parameterization can exploit GPU-friendly parallelization in computing all shape functions simultaneously without increasing memory footprints.

Sparse Backpropagation. We use first-order optimization methods (e.g., SGD and its variants) to optimize GRAND-SLAMIN . Typically, a main computational bottleneck in optimizing GAMs with all pairwise interactions via standard gradient-based backpropagation methods is the computation

of forward pass and gradient computations with respect to all additive components (both main and interaction effects). This can hinder training large GAMs with interactions. We exploit the sparsity in GRAND-SLAMIN via the sparsity in the smooth-step function and its gradient during training.

Recall that S ( µ j ) 's and S ( µ jk ) 's play a role of selection in a smoothed fashion. In the early stages of training, S ( µ j ) 's and S ( µ jk ) 's are all in the range (0 , 1) . As the optimization proceeds, due to the entropic regularization and selection regularization, S ( µ j ) 's and S ( µ jk ) 's progressively achieve binary state { 0 , 1 } -the gradient with respect to µ j and µ jk also reaches 0 because of the nature of smooth-step function S ( · ) . All the additive components corresponding to the selection variables that reached 0 can be removed from both the forward and the backward computational graph. This sparse backpropagation approach can provide large speedups on training times up to a factor of 10 × over standard backpropagation. Additionally, there is progressively a reduced memory footprint during the course of training in comparison to standard backpropagation.

The approach outlined above (use of Smooth-Step function for selection) when specialized to additive models allows us to implement GPU-friendly sparse backpropagation - this makes our work different from [18], which does not support GPU training.

Screening. We describe how screening approaches prior to training, can complement our sparse backpropagation approach when the number of all pairwise interactions is large e.g., of the order 100 , 000 . Fast screening methods based on shallow-tree like models are proposed in [34] for identifying prominent pairwise interactions. These are used by various toolkits e.g., EBM [40], GAMI-Net [60]. These screening approaches are complementary to our selection approach with indicator variables. We used CART [4] for each pairwise interaction and sorted the interaction effects based on AUC performance to select an initial screened set of interaction effects. In particular, we can consider I to be a screened subset (e.g., 10 , 000 ) of all pairwise interaction effects of the order 100 , 000 . Then, we run our end-to-end learning framework under the component selection constraints on the main and screened interaction effects. We observe that such screening can be beneficial for multiple reasons: (i) The training time can be reduced further by 3 × -5 × . (ii) The memory footprint of the model reduces by 10 × . (iii) There is no loss in accuracy-the accuracy can sometimes improve with screening see ablation study in Supplement Sec. F.6. Note that even with screening, our approach is different from GAMI-Net as they directly screen to a much smaller set of interactions e.g., 500 and there is no component selection when training with these interactions.

5 Statistical Theory

In this section, we explore the statistical properties of our additive model with soft tree shape functions. We assume that observations are noisy versions of some (unknown) underlying noiseless data. We do not assume the noiseless data comes from a tree-based model. We show that if our model can approximate the true noiseless data well, the prediction error resulting from the noise converges to zero as the number of observations grows. In particular, for n data observations ( x i , y i ) n i =1 we consider a sparsity-constrained version of Problem (8) with the least-squares loss as given by

for t ∈ { 1 , 2 } , where f j , f j,k are depthd soft trees (cf. Section 3.2.2). For t = 1 , Problem (10) simplifies to a main effects model, while t = 2 corresponds to the model with pairwise interactions. We study the case with no hierarchy constraints here, q ( z j , z k , z j,k ) = z j,k as in (5). We expect our approach to extend to more general cases, but we do not pursue that here.

Model Setup. We assume for i ∈ [ n ] , the data is bounded, ∥ x i ∥ 2 ≤ 1 . The noisy observations are given as y i = h ∗ ( x i ) + ε i = y ∗ i + ε i where h ∗ is the unknown underlying generative model (need not be a tree ensemble), and ε i iid ∼ Normal (0 , σ 2 ) are noise values. Suppose f is a feasible solution for Problem (10). Let U , L denote the set of internal nodes and leaves for a depth d tree, and let ( W j , o j ) be the set of weights corresponding to internal nodes and leaves of tree j i.e. f j ( x j ) in this solution, where W j ∈ R |U| , o j ∈ R |L| . We also let W j i , o j l be the weights corresponding to node i ∈ U and leaf l ∈ L in tree j , respectively. We define W j,k i , o j,k l for f j,k ( x j , x k ) and i ∈ U , l ∈ L

in a similar fashion, where W j,k ∈ R |U|× 2 , o j,k ∈ R |L| . See Figure 1 for an illustration. Define

where ∨ denotes maximum. Conceptually, ¯ u ( f ) is the largest weight (in absolute value) that appears in all main effect and interaction soft trees. Let ˆ f = f ( t ) ( y ) be the estimator resulting from the noisy data, and f ∗ = f ( t ) ( y ∗ ) be the oracle estimator that is the best approximation to the noiseless data among feasible solutions of (10) (i.e., among sparse additive tree models). We assume the following:

(A1 ) The activation function ϕ : R ↦→ [0 , 1] for soft trees is L -Lipschitz for some L > 0 .

(A2 ) There exists B > 0 such that ¯ u ( ˆ f ) ∨ ¯ u ( f ∗ ) ≤ B .

Assumption (A2 ) ensures the trees resulting from the data are uniformly bounded. This is a mild assumption as in practice, the data is bounded and the resulting trees are also bounded as a result.

Main Results. Our first result is for a general setup, where p can be much larger than n .

Theorem 1. Let ˆ f, f ∗ be as defined above and take A = 2 ∨ 2 d +2 B ∨ B 2 dL 2 d +3 and a = exp( -1) . Under the model setup, assume s ≥ 1 and σ ≳ 1 . 2 Then,

(1) For t = 1 , if n ≳ σ 2 ( s log p + s 2 A 2 ) , then with high probability

(2) For t = 2 , if n ≳ σ 2+2 a ( s log p + s 3 A 3 ) , then with high probability

Theorem 1 presents non-asymptotic bounds for the performance of our method. Particularly, this theorem shows that under a well-specified model where y ∗ i = f ∗ ( x i ) , prediction error rates of n -2 / 3 and n -1 / (2+ a ) ≈ n -0 . 42 are achievable for main effects and interaction models, respectively. Particularly, this implies that the prediction error of our estimator (resulting from the noise in observations) converges to zero as we increase the total number of samples, n . We note that although the rate from the main effects model is sharper, such models might be too simple to capture the noiseless data, leading to larger oracle error. Therefore, in practice, the interactions model can lead to better performance. Next, we show that under the asymptotic assumption that n →∞ , the error rate for the interaction models can be further improved.

Theorem 2. Under the model setup with t = 2 , assume all parameters are fixed except n . For any positive sequence a n with lim n →∞ a n = 0 , there exists a positive sequence b n with lim n →∞ b n = 0 such that if n ≳ b -1 / 2(2+ a n ) n then with high probability

Theorem 2 shows that essentially when n →∞ and other parameters in the problem stay constant, an error rate of n -0 . 5 is achievable for the interactions model which improves upon Theorem 1.

Discussion of previous work: As stated, our results are the first to present prediction bounds specialized to learning sparse additive models with soft trees. Moreover, as we discuss in Appendix C.5, our upper bounds (rates) generally align with the best known rates available in the literature for learning main effects and interactions, such as [23, 52], or hold under milder assumptions.

6 Experiments

We study the performance of GRAND-SLAMIN on 16 real-world datasets and compare against relevant baselines for different cases. We make the following comparisons:

Performance comparison of GRAND-SLAMIN without/with structural constraints against existing toolkits for sparse GAMs with interactions.

(a) Toolkits that support sparse interactions: EB 2 M[40] and NODE-GA 2 M [6]

(b) Toolkits that support hierarchy constraints: GAMI-Net [60] and SIAN [10]

Variable selection comparison against the competing toolkits.

Computational scalability of GRAND-SLAMIN toolkit with sparse backpropagation.

Variance reduction with structural constraints

Additional results are included in Supplement Sec. F that study (i) comparison with full complexity models in F.1, (ii) comparison with GAMs with all pairwise interactions in F.2, (iii) comparison with Group Lasso selection approach in F.3, (iv) choice of shape functions in F.4, (v) effect of entropy on performance and component selection in F.5, and (vi) effect of screening on training times, memory and performance in F.6.

Datasets. We use a collection of 16 open-source classification datasets (8 binary, 6 multiclass and 2 regression) from various domains. We consider datasets with a wide range of number of all pairwise interactions 10 -200000 . A summary of the datasets is in Table E.1 in the Appendix.

Tuning Details. For all the experiments, we tune the hyperparameters using Optuna [2] with random search on a held-out validation set. We compute statistical averages across multiple runs for the optimal hyperparameters for all models. In particular, we report median test ROC AUC across 10 runs along with the mean absolute deviation (MAD). Additional details are in the Appendix E.

6.1 Prediction Performance

Comparison with EB 2 M and NODE-GA 2 M. We first study the performance of our model in comparison to two tree-based state-ofthe-art toolkits which support sparse GAMs with interactions without any structural constraints e.g., EB 2 M and NODE-GA 2 M. We report the ROC AUC performance in Table 2. Our model outperforms EB 2 M in 10 out of 14 datasets. Our model is also competitive with NODE-GA 2 M as it can outperform in 50% of the datasets. In summary, our results in Table 2 show that we are at par with state-of-the-art methods for unstructured component selection.

WH=Weak Hierarchy, SH=Strong Hierarchy. For SIAN, for some of the larger datasets (row 5-9), we use the number for best trial as SIAN takes ∼ 24 hours on V100 Tesla GPU.

Our key advantage is to do hierarchical interactions, which NODE-GA 2 Mand EB 2 M can not support. Additionally, we can achieve faster training times (Sec. 6.3) and improve on variable selection (Sec. 6.2) than NODE-GA 2 M and EB 2 M.

Structural constraints: Weak and Strong Hierarchy. Next, we study our method with structural constraints i.e., (6) for weak hierarchy or (7) for strong hierarchy. We compare against two competing neural-based state-of-the-art methods for sparse GAMs with hierarchical interactions: (i) GAMI-Net with support for weak hierarchy, and (ii) SIAN with support for strong hierarchy. We omit multiclass datasets as both GAMI-Net and SIAN do not support them. We report the ROC AUC performance in Table 3. Our models outperform GAMI-Net and SIAN in 7/8 datasets.

Additionally, our models are much more compact in terms of overall number of parameters - our tree-based shape functions have 100 × and 10 × smaller number of parameters than the neural-based

shape functions used by GAMI-Net and SIAN respectively. Moreover, our toolkit is significantly faster than SIAN and GAMI-Net on larger datasets - see Sec. 6.3.

Additionally, we compare interpretable modeling toolkits with full complexity models e.g., deep neural network (DNN), in Supplement Sec. F.1. We observed interpretable models to outperform full complexity models on these datasets. We also compare our toolkit that fits sparse components with toolkits that fit all pairwise interactions e.g., NA 2 M, NB 2 M and SPAM in Supplement Sec. F.2. GRAND-SLAMIN generally outperform these methods with enhanced interpretability due to sparsity and structural constraints. We also study how our toolkit performs when we replace soft tree shape functions with MLP shape functions in Supplement Sec. F.4. Interestingly, we observe that soft trees seem to have an edge when the parameters are matched with MLP.

6.2 Variable Selection

We evaluate the performance of our models in terms of feature selection. We report the number of features selected in Table 4 by each toolkit for sparse GAMs with interactions. We see that GRAND-SLAMIN with structural constraints, in particular strong hierarchy, can significantly reduce the number of features selected by the GAMs with interactions model. For example, on Bankruptcy datasets, GRAND-SLAMIN achieves feature compression up to a factor of 8 × over state-of-the-art GAM toolkits. Having fewer features reinforces the usefulness of additive models as being interpretable.

WH=Weak Hierarchy, SH=Strong Hierarchy.

6.3 Computational Scalability

Next, we discuss the scalability of GRAND-SLAMIN .

Sparse backpropagation. We highlight the usefulness of our efficient approach with sparse backpropagation in Figure 2 on Activity dataset. We show in Figure 2[a] that during the course of training, the number of selected components (selected via binary variables z 's and q ( · ) 's) becomes progressively smaller. This leads to much faster computations at each epoch in Figure 2[b] due to

Numberofselectedeffects

20k

Training time,seconds

Sparse

Dense

2

15k

1.5

10k

1

5k

0.5

0

50

100

150

200

250

Q

50

100

150

200

250

Epoch

Epoch

(a) Number of selected effects at each epoch.

(b) Training time (seconds) for each epoch.

Main effect

0.5

Main effect

0.5

Main effect

0.5

0

0

0

0.5

-0.5

-0.5

-1

-1

1.5

-1.5

-1.5

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

hour

hour

hour

0.3

0.3

0.3

0.2

0.2

0.2

Main effect

0.1

Main effect

0.1

Main effect

0.1

0

0

0

0.1

-0.1

-0.1

0.2

-0.2

-0.2

-0.3

-0.3

-0.3

1

1.5

2

2.5

3

3.5

4

1

1.5

2

2.5

3

3.5

4

1

1.5

2

2.5

3

3.5

weather

weather

weather

sparse forward and backward passes. By exploiting sparsity during training, we can get 10 × faster training times than with standard backpropagation.

Comparison with other toolkits. Our toolkit is highly competitive in terms of training times with all existing tree-based and neural-based toolkits for sparse GAMs with interactions. For example, on Madelon dataset, we are 15 × faster than NODE-GA 2 M, 20 × faster than EBM, 1300 × faster than SIAN and 25 × faster than GAMI-Net. See Supplement Sec. F.7 for more detailed timing comparisons across multiple datasets. Note that, in addition, we can also handle the case of structured interactions - extending the flexibility of existing end-to-end training methods.

6.4 Variance Reduction with Structural Constraints

We provide a visualization study to further highlight an important contribution of our work. In particular, our framework can support models with structural constraints. Hence, we study the effect of these constraints on the stability of learning main effects (in the presence of interactions) when these structural constraints are imposed or not. For this exercise, we consider bikesharing dataset. We visualize some of the main effects in the presence/absence of hierarchy in Figure 3. Note that for visualization, we used the purification strategy [29, 6] post-training that pushes interaction effects into main effects if possible. We can observe in Figure 3 that when additional hierarchy constraints are imposed, the error bars are much more compact across different runs. This can potentially increase the trust you can have on the model for deriving interpretability insights. We show additional visualizations on another dataset (American Community Survey from US Census Planning Database 2022 [55]) to show the same behavior in Supplement Section G.

7 Conclusion

We introduce GRAND-SLAMIN : a novel and flexible framework for learning sparse GAMs with interactions with additional structural constraints e.g., hierarchy. This is the first approach to do endto-end training of nonparameteric additive models with hierarchically structured sparse interactions. Our formulation uses binary variables to encode combinatorial constraints. For computational reasons, we employ smoothing of the indicator variables for end-to-end optimization with first-order methods (e.g., SGD). We propose sparse backpropagation, which exploits sparsity in the nature of the smoothing function in a GPU-compatible manner and results in 10 × speedups over standard backpropagation. We present non-asymptotic prediction bounds for our estimators with tree-based shape functions. Numerical experiments on a collection of 16 real-world datasets demonstrate the effectiveness of our toolkit in terms of prediction performance, variable selection and scalability.

Acknowledgments and Disclosure of Funding

This research was supported in part, by grants from the Office of Naval Research (N000142112841), and Liberty Mutual Insurance. The authors acknowledge the MIT SuperCloud [48] for providing HPC resources that have contributed to the research reported within this paper.

References

Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey Hinton. Neural additive models: Interpretable machine learning with neural nets. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, et al. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2019.

Jacob Bien, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. The Annals of Statistics , 41(3):1111-1141, June 2013.

L. Breiman, J. Friedman, C.J. Stone, and R.A. Olshen. Classification and Regression Trees . Taylor & Francis, 1984.

Nadia Burkart and Marco F. Huber. A survey on the explainability of supervised machine learning. J. Artif. Intell. Res. , 70:245-317, 2020.

Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. NODE-GAM: Neural generalized additive model for interpretable deep learning. In International Conference on Learning Representations , 2022.

Hugh Chipman. Bayesian variable selection with related predictors. The Canadian Journal of Statistics / La Revue Canadienne de Statistique , 24(1):17-36, 1996.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Abhimanyu Dubey, Filip Radenovic, and Dhruv Mahajan. Scalable interpretability via polynomials. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

James Enouen and Yan Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree, 2017.

Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):3681-3688, Jul. 2019.

M. Hamada and C. F. J. Wu. Analysis of designed experiments with complex aliasing. Journal of Quality Technology , 24(3):130-137, 1992.

Trevor Hastie and Robert Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association , 82(398):371-386, 1987.

Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning . Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.

Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations . Chapman & Hall/CRC, 2015.

Hussein Hazimeh and Rahul Mazumder. Learning hierarchical interactions at scale: A convex optimization approach. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 1833-1843. PMLR, 26-28 Aug 2020.

Hussein Hazimeh, Natalia Ponomareva, Petros Mol, et al. The tree ensemble layer: Differentiability meets conditional computation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4138-4148. PMLR, 13-18 Jul 2020.

Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. DSelect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

J. Huang, J.L. Horowitz, and F. Wei. Variable selection in nonparametric additive models. The Annals of Statistics , 38:2282-2313, 2010.

Shibal Ibrahim, Wenyu Chen, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Comet: Learning cardinality constrained mixture of experts with trees and local search. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 832-844, New York, NY, USA, 2023. Association for Computing Machinery.

Shibal Ibrahim, Hussein Hazimeh, and Rahul Mazumder. Flexible modeling and multitask learning using differentiable tree ensembles. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, page 666-675, New York, NY, USA, 2022. Association for Computing Machinery.

Shibal Ibrahim, Rahul Mazumder, Peter Radchenko, and Emanuel Ben-David. Predicting census survey response rates with parsimonious additive models and structured interactions. arXiv , abs/2108.11328, 2021.

Ozan Irsoy, O. T. Yildiz, and Ethem Alpaydin. Soft decision trees. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012) , pages 1819-1822, 2012.

Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Comput. , 6(2):181-214, mar 1994.

Michael J Kearns and Robert E Schapire. Efficient distribution-free learning of probabilistic concepts. Journal of Computer and System Sciences , 48(3):464-497, 1994.

Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, et al. Deep neural decision forests. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 1467-1475, 2015.

Himabindu Lakkaraju and Osbert Bastani. "how do i fool you?": Manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , AIES '20, page 79-85, New York, NY, USA, 2020. Association for Computing Machinery.

Benjamin J. Lengerich, S. Tan, Chun-Hao Kingsley Chang, Giles Hooker, and Rich Caruana. Purifying interaction effects with the functional anova: An efficient algorithm for recovering identifiable additive models. In International Conference on Artificial Intelligence and Statistics , 2019.

Michael Lim and Trevor Hastie. Learning interactions via hierarchical group-lasso regularization. Journal of Computational and Graphical Statistics , 24(3):627-654, July 2015.

Y. Lin and H. H. Zhang. Component selection and smoothing in multivariate nonparametric regression. The Annals of Statistics , 34:2272-2297, 2006.

Zachary C. Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, jun 2018.

Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '12, page 150-158, New York, NY, USA, 2012. Association for Computing Machinery.

Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '13, page 623-631, New York, NY, USA, 2013. Association for Computing Machinery.

Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 4768-4777, Red Hook, NY, USA, 2017. Curran Associates Inc.

Peter McCullagh and John A Nelder. Generalized linear models , volume 37. CRC press, 1989.

Lukas Meier, Sara van de Geer, and Peter Buhlmann. High-dimensional additive modeling. The Annals of Statistics , 37(6B):3779 - 3821, 2009.

Shahar Mendelson and Roman Vershynin. Entropy and the combinatorial dimension. Inventiones mathematicae , 152(1):37-55, 2003.

J. A. Nelder. A reformulation of linear models. Journal of the Royal Statistical Society. Series A (General) , 140(1):48-77, 1977.

Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. ArXiv , abs/1909.09223, 2019.

Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Mining , 10(1):36, 2017.

Julio L. Peixoto. Hierarchical variable selection in polynomial regression models. The American Statistician , 41(4):311-313, 1987.

Ben Peters, Vlad Niculae, and André F. T. Martins. Sparse sequence-to-sequence models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 1504-1519, Florence, Italy, July 2019. Association for Computational Linguistics.

Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference on Learning Representations , 2020.

Peter Radchenko and Gareth M. James. Variable selection using adaptive nonlinear interaction structures in high dimensions. Journal of the American Statistical Association , 105(492):1541-1553, December 2010.

Filip Radenovic, Abhimanyu Dubey, and Dhruv Mahajan. Neural basis models for interpretability. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Pradeep Ravikumar, John Lafferty, Han Liu, et al. Sparse additive models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 71(5):1009-1030, November 2009.

Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC) , pages 1-6. IEEE, 2018.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, page 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery.

Mark Rudelson and Roman Vershynin. Combinatorics of random processes and sections of convex bodies. Annals of Mathematics , pages 603-648, 2006.

Dylan Z Slack, Sophie Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc explanations: Modeling uncertainty in explainability. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Zhiqiang Tan and Cun-Hui Zhang. Doubly penalized estimation in additive regression with highdimensional data. The Annals of Statistics , 47(5):2567 - 2600, 2019.

Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, et al. Adaptive neural trees. ArXiv , abs/1807.06699, 2019.

Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attribution for feature interactions. In Advances in Neural Information Processing Systems , 2020.

US Census Bureau. Planning database, October 2022.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Shibiao Xu, Zhiqi Bu, Pratik Chaudhari, and Ian Barnett. Sparse neural additive model: Interpretable deep learning with feature selection via group sparsity. ArXiv , abs/2202.12482, 2022.

Xiaohan Yan and Jacob Bien. Hierarchical sparse modeling: A choice of two group lasso formulations. Statistical Science , 32(4):531-560, November 2017.

Zebin Yang, Aijun Zhang, and A. Sudjianto. Gami-net: An explainable neural network based on generalized additive models with structured interactions. Pattern Recognit. , 120:108192, 2021.

Guo Yu, Daniela Witten, and Jacob Bien. Controlling costs: Feature selection on a budget. Stat , 11(1):e427, 2022.

Ming Yuan and Ding-Xuan Zhou. Minimax optimal rates of estimation in high dimensional additive models. The Annals of Statistics , 44(6):2564-2593, 2016.

Tuo Zhao and Han Liu. Sparse additive machine. In Neil D. Lawrence and Mark Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics , volume 22 of Proceedings of Machine Learning Research , pages 1435-1443, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.

Supplementary Material for GRAND-SLAMIN' Interpretable Additive Modeling with Structural Constraints

A Definition of Soft Trees

Formally, an interaction soft tree with the set of (internal) nodes U and the set of leaves L is a function such as f ( · ; W , o ) : R 2 ↦→ R where W ∈ R |U|× 2 and o ∈ R |L| . The output is then given as f (( x j , x k ); W , o ) = ∑ l ∈L P ( { ( x j , x k ) → l } ) o l where P ( { ( x j , x k ) → l } ) is the proportion of ( x j , x k ) that is routed to leaf l . Particularly, P ( { ( x j , x k ) → l } ) = ∏ i ∈A ( l ) r i,l ( x j , x k ) where A ( l ) denotes the set of ancestors of l and r i,l ( x j , x k ) denotes the proportion of ( x j , x k ) is routed to leaf l from node i . These values are given as r i,l ( x j , x k ) = ϕ ( W T i ( x j , x k )) if l belongs to the left subtree of i , and r i,l ( x j , x k ) = 1 -ϕ ( W T i ( x j , x k )) if l belongs to the right subtree of i , where ϕ ( · ) is the sigmoid activation function. A soft tree f ( · ; W , o ) : R ↦→ R for a main effect is defined similarly.

B Smooth-Step Function

The smooth-step function [18] has been used in soft trees for conditional computation [18] and routing in sparse mixture of experts [19, 21]. It can output exact zeros and ones, thus allowing for hard selection. Let γ be a non-negative scalar parameter. The smooth-step function takes the form:

The smooth-step function is continuously differentiable, similar to the logistic function. Additionally, it performs hard selection, i.e., outside [ -γ/ 2 , γ/ 2] , the function produces exact zeros and ones.

C Proofs of Main Results

C.1 Preliminaries

Before proceeding with the proof, we define some notation we use throughout this section. Given the data points x 1 , · · · , x n , we let P n the empirical measure supported on x 1 , · · · , x n , P n = ∑ n i =1 δ x i /n . We define the norm with respect to P n as

Next, we define covering and packing numbers.

Definition C.1 (Covering number [38]) . Let F be a functional class and x 1 , · · · , x n be a set of n observations. An ϵ -cover of F with respect to P n is a set of functions f 1 , · · · , f N such that for any f ∈ F , there exists j ∈ [ N ] such that

The covering number of F with respect to P n at scale ϵ , denoted as N ( F , P n , ϵ ) , is the smallest N such that there exists an ϵ -cover of F with respect to P n with size N .

Definition C.2 (Packing number [56]) . Let F be a functional class and x 1 , · · · , x n be a set of n observations. An ϵ -packing of F with respect to P n is a set of functions f 1 , · · · , f N such that for any i, j ∈ [ N ] ,

The packing number of F with respect to P n at scale ϵ , denoted as P ( F , P n , ϵ ) , is the largest N such that there exists an ϵ -packing of F with respect to P n with size N .

The following well-known lemma establishes the equivalence between packing and covering numbers.

Lemma C.1 (Lemma 4.2.8 [56]) . For ϵ > 0 ,

Finally, we define the fat-shattering dimension of a function class.

Definition C.3 (Fat-shattering dimension [26]) . Let F be a functional class. Given a subset S = { x 1 , · · · , x n } ⊆ R p , the class F ϵ -shatters S , if there exists c = ( c 1 , · · · , c n ) ∈ R n , such that for every e ∈ { 0 , 1 } n , there exists f ∈ F such that

The fat-shattering dimension of F at scale ϵ , denoted as fat ϵ ( F ) is the size of the largest subset S that is ϵ -shattered by F .

The following theorem relates the fat-shattering dimension to the covering number discussed above.

Theorem C.1. If there exists v > 0 such that

Proof. The theorem is a direct result of Theorems 6.3 of [50] and Lemma C.1.

C.2 Technical Lemmas

Let U , L denote the set of internal nodes and leaves of a depth d tree, respectively. Let us define the class of soft trees with depth d and data dimension p as

where B 2 p is the Euclidean ball of dimension p , W ∈ R |U|× p and o ∈ R |L| and

with w k ∈ R p being the vector of weights for node k from W . Our first result is characterizing the fat-shattering dimension of F ( d, p ) .

Lemma C.2. Let A = 2 ∨ 2 d +2 B ∨ B 2 dL 2 d +3 . For ϵ > 0 ,

Proof. First, suppose ϵ > 2 B 2 d . We show that fat ϵ ( F ( d, p )) = 0 . To this end, note that for x and f ∈ F ( d, p ) ,

where P is defined in Appendix A.As a result, for f 1 , f 2 ∈ F ( d, p ) and x ,

showing no single point can be ϵ -shattered. Therefore, fat ϵ ( F ( d, p )) = 0 . Let 0 < ϵ ≤ 2 B 2 d and suppose { x 1 , · · · , x N } is ϵ -shattered by F ( d, p ) at level c . We show that for i, j ∈ [ N ] , ∥ x i -x j ∥ 2 ≥ ϵ/ ( B 2 dL 2 d ) . To this end, assume there exists i, j ∈ [ N ] such that ∥ x i -x j ∥ 2 < ϵ/ ( B 2 dL 2 d ) . Without loss of generality assume c i ≥ c j and choose e ∈ { 0 , 1 } p such that e i = 1 , e j = 0 . Suppose f ( · ; W , o ) ∈ F ( d, p ) shatters the data with e . Particularly,

then

For k ∈ U , l ∈ L , if k ∈ A ( l ) we have

where r k,l is defined in Appendix A, w k ∈ R p is the vector of weights for node k from W , ( a ) is by (A1 ) and ( b ) is by ¯ u ( f ( x ; W , o )) ≤ B (recall the Assumption (A2 )). Next,

where the inequality is by (C.7) and the fact that r k,l ∈ [0 , 1] is bounded. Finally,

where the last inequality is true as |L| = 2 d and | o l | ≤ B by ¯ u ( f ( x ; W , o )) ≤ B . As we assumed ∥ x i -x j ∥ 2 < ϵ/ ( B 2 dL 2 d ) , we have from (C.9)

while on the other hand, by the shattering property (C.6),

which is a contradiction. Therefore, for every i, j ∈ [ N ] , we must have ∥ x i -x j ∥ 2 ≥ ϵ/ ( B 2 dL 2 d ) or in other words, { x 1 , · · · , x N } is a ϵ/ ( B 2 dL 2 d ) -packing of B 2 p . Thus,

where N ( B 2 p , ∥ · ∥ 2 , ϵ ) is the ϵ -covering number of B 2 p , ( a ) is by Lemma C.1, ( b ) is true by Corollary 4.2.13 of [56] and ( c ) is true as ϵ ≤ 2 d +1 B .

Lemma C.3. There exists an absolute constant C > 0 such that for ϵ > 0 ,

Proof. From Lemma C.2, for x ≥ 1

where the last inequality is true as p ≥ 1 . The proof is complete by Theorem C.1.

In the next step, we consider trees that only use a subset of variables, such as J ⊆ [ p ] . Formally, we let

and

Proof. Let P n, J be the empirical distribution supported on coordinates in J ,

where x J is the sub-vector of x with coordinates in J . Then,

The result follows from Lemma C.3.

Next, let us define the classes of functions with interactions in J ⊆ 2 [ p ] ,

and the class of functions with at most s interactions of size at most t (i.e t -dimensional interactions),

for a fixed J ∗ with | J ∗ | ≤ s and |J | ≤ t for J ∈ J ∗ .

Lemma C.5. Suppose 0 < ϵ ≤ 2 sCA and 1 ≤ t ≤ 2 . Then,

Proof. Let S ( F 0 ( J , d )) be a minimal ϵ/ 2 s -cover of F 0 ( J , d ) . Hence, |S ( F 0 ( J , d )) | = N ( F 0 ( J , d ) , P n , ϵ/ 2 s ) . Define

where A + B = { a + b : a ∈ A,b ∈ B } denotes the Minkowski sum. Note that from Lemma C.4, for J ∈ S and J ∈ J ,

as 2 sCA/ϵ ≥ 1 . For the rest of the proof, we let

Suppose f ∈ F ∗ ( d, t, s ) ,

for some J 0 ∈ S . For any J ∈ J 0 ∪ J , let f J ∈ S ( F 0 ( J , d )) be such that ∥ f J -f J ∥ P n ≤ ϵ/ 2 s Let

Note that ˜ f ∈ S ∗ . Next,

as | J 0 | , | J ∗ | ≤ s . As a result, S ∗ is a ϵ -cover for F ∗ ( d, t, s ) . Moreover,

where ( a ) is by (C.17) and ( b ) is true as | S | ≤ ( p t ) s ≤ p 2 s . As a result,

Lemma C.6. Suppose 0 < δ ≤ 1 ∧ 2 sCA . Then, for a = exp( -1) ,

Proof. Note that if δ > 4 σ , the lemma is trivial. Therefore, we assume δ ≤ 4 σ . From Lemma C.5

Hence, as δ ≤ 1 ,

Moreover,

Next, consider the function h ( x ) = x a -log( x ) where a = exp( -1) . We will show h ( x ) ≥ 0 for x > 0 . Note that

therefore h ( a -1 /a ) = h ′ ( a -1 /a ) = 0 . Moreover,

so h ′′ ( a -1 /a ) = (1 + a ( a -1) a -1 ) / ( a -2 /a ) > 0 , showing that a -1 /a is a minimum for h ( x ) , and therefore h ( x ) ≥ 0 for x > 0 . Hence, from (C.26),

as δ ≤ 4 σ or 4 σ/δ ≥ 1 .

Lemma C.7. Under the assumptions of Theorem 2, for any sequence a n , n ≥ 1 such that a n > 0 and lim n →∞ a n = 0 , there exists a positive sequence b n such that lim n →∞ b n = 0 and

Proof. Consider the function h n ( x ) = x a n -log( x ) . Note that

therefore, there exists a sequence γ n such that lim n →∞ γ n = ∞ and

Therefore, if δ ≤ 4 σ/γ n ∧ 4 σ ∧ 1 =: b n , from (C.26),

Proof. Let S = { f 1 , · · · , f N } be a minimal ϵ -cover of F ∗ ( d, t, s ) where N = N ( F ∗ ( d, t, s ) , P n , ϵ ) . Let S ∗ = { f ∗ 1 , · · · , f ∗ N } where for i ∈ [ N ] ,

We show S ∗ is a ϵ -cover for F ∗ ( d, t, s ) ∩ {∥ f ∥ P n ≤ δ } . Let f ∈ F ∗ ( d, t, s ) ∩ {∥ f ∥ P n ≤ δ } . Let j ∈ [ N ] be such that ∥ f -f j ∥ P n ≤ ϵ . Let us consider the following cases:

∗ ∗

Case 1: If ∥ f j ∥ P n ≤ δ , then f j ∈ S showing f j = f j covers f . Case 2: Suppose ∥ f j ∥ P n > δ . For any f , let f = ( f ( x 1 ) , · · · , f ( x n )) / √ n ∈ R n . With this notation, ∥ f ∥ P n = ∥ f ∥ 2 . As a result,

where ( a ) is true as ∥ f j ∥ 2 > δ and ∥ f ∥ 2 ≤ δ . This shows f ∗ j covers f , completing the proof.

C.3 Proof of Theorem 1

Proof. Let

Claim: For t ∈ { 1 , 2 } ,

We will prove the claim later. Let

Under this notation, we have

Using the notation from [57][Ch. 13], we have

with J ∗ being the set of interactions for for f ∗ t . Moreover, we note that F ∗ ( d, t, s ) is star-shaped, that is for f ∈ F ∗ ( d, t, s ) and α ∈ [0 , 1] , αf ∈ F ∗ ( d, t, s ) . This is true as

Under the claim and assumptions of the theorem, we have δ t ≤ σ so we invoke Corollary 13.7 and Theorem 13.13 of [57] with γ = 1 / 2 , t = δ to achieve

completing the proof.

Proof of Claim: For t = 1 ,

where the first inequality is by Lemma C.6 and the final inequality is by Lemma C.8. Next, for t = 2 ,

by Lemmas C.6 and C.8.

C.4 Proof of Theorem 2

Proof. Let

By taking n ≳ b -2(2+ a n ) n and invoking Lemmas C.7 and C.8,

The rest of the proof is similar to the proof of Theorem 1.

C.5 Theory Discussion

First, we consider the case with only main effects, i.e t = 1 . In our framework, we use the smoothstep function which is differentiable with uniformly bounded derivative, but not twice differentiable. Therefore, our framework operates on L 2 Sobolev functions of order 1, i.e. bounded differentiable functions with bounded gradients. For this case, the results of [52] show that an ℓ 2 prediction rate of n -2 / 3 (see Remark 4 of [52] with q = 0 and β 0 = 1 ). This is the same rate we achieve in Theorem 1 for t = 1 . However, we note that our results hold without differentiability assumption and only require the activation function to be Lipschitz, showing stronger results specialized to the soft trees case.

Next, we consider the interaction case with t = 2 . In this case, the results of [23] imply an ℓ 2 prediction rate of n -1 / 2 for the class of L 2 Sobolev functions of order 1 (see Section S2.2 with m = 1 , leading to r n = n -1 / 4 ). This is the same rate our method achieves asymptotically (see Theorem 2). However, as we discussed, our results hold without the differentiability assumption.

D Extension to third-order interactions

Our approach can be extended to model third-order interactions. In the case of third-order interactions, the structural constraints for strong and weak hierarchy can be described as follows:

̸

̸

̸

̸

̸

̸

These constraints can be modeled with binary variables as follows:

Hence, our sparse selection and hierarchy constraints can add value in terms of model compactness and feature selection in the settings with third-order interactions as well. However, for third-order interactions, the number of third-order interactions are O ( p 3 ) . Hence, for scalability, this would also require using a pre-training screening approach. Despite the fact that our approach is generalizable to third-order interactions, we do not consider such interactions because these are not considered to be easily interpretable.

E Datasets, Computing Setup and Tuning

Datasets We use a collection of 16 open-source classification datasets (binary, multiclass and regression) from various domains, e.g., physics, computing, healthcare, life sciences, finance, and social networks. They are from Penn Machine Learning Benchmarks (PMLB) [41] and UCI databases [8]. For datasets with available training, validation and test splits, we used them in their original form. When no test set was available, we treated the original validation set as the test set and split the training set into 80% training and 20% validation. For remaining, we randomly split each of the dataset into 60% training, 20% validation and 20% testing sets. A summary of the 16 datasets considered is in Table E.1.

Computing Setup. We used a cluster running Ubuntu 7.5.0 and equipped with Intel Xeon Platinum 8260 CPUs and Nvidia Volta V100 GPUs. For all experiments of Sec. 6, each job involving GRAND-SLAMIN , EBM, Node-GAM, SIAN, GAMI-Net and DNN were run on 8 core, 32GB RAM. Jobs involving larger datasets ( p > 100 ) were run on Tesla V100 GPUs.

̸

̸

Tuning. The tuning was done in parallel over the competing models and datasets. We tune the hyperparameter using Optuna [2] which optimizes the overall AUC on a validation set. We report the results on a held-out test set. A list of all the tuning parameters and their distributions is given for GRAND-SLAMIN below:

Learning Rates: Discrete uniform in the set { 0 . 05 , 0 . 01 , 0 . 005 } for Adam with multi-step decay rate of 0 . 9 every 25 epochs.

Batch-size: Discrete uniform in the set { 64 , 256 } .

λ for selection: Discrete uniform in the set of 11 values { 0 , 1 e -6 , · · · , 1 e -3 } .

γ for Smooth-step: Discrete uniform in the set { 0 . 01 , 0 . 1 , 1 } .

τ for Entropy Regularization: Discrete uniform in the set { 0 . 001 , 0 . 01 , 0 . 1 } .

α for relative penalty on interactions: Discrete uniform in the set { 1 , 10 } .

Epochs: 1000 with early stopping (patience=50) based on validation loss.

For Madelon, Activity and Multiple datasets, we used screening to reduce the initial set of interactions to 5000.

For other toolkits, we considered the tuning protocols outlined in the respective papers such that the parameter controlling the variable selection is tuned. For SIAN, we tuned over the threshold parameter θ such that the screened set of interactions is upper bounded by { 250 , 500 , · · · , 1000 } . We set τ = 1 such that the model satisfies strong hierarchy. For GAMI-Net, we tuned over the number of interactions. For small datasets ( |I| < 1000 ), we tuned over the set: { 0 . 2 |I| , 0 . 4 |I| , · , |I|} . For large datasets ( |I| < 1000 ), we tuned over the set: { 250 , 500 , · · · , 1000 } . For EBM, we tuned over the set of interactions { 16 , 32 , 64 , 128 } as done by authors in [6]. For NODE-GA 2 M, we tuned the number of trees as this controls the maximum number of interactions selected by the model.

F Additional Results

F.1 Comparison with full complexity models e.g., DNN

Here, we compare interpretable modeling toolkits based on pairwise interactions with full complexity models. In particular, we consider a deep neural network (DNN). We considered the same architecture for the DNN as used by the authors in [10] for similar comparisons. The architecture is a 4-layered ReLU-activated neural network. We show the test AUC performance of different models in Table F.1. Across all datasets, we see that the full complexity DNN model underperforms interpretable models including GRAND-SLAMIN . However, it is noted that the choice of the architecture for DNN maybe contributing to the degradation in performance. There maybe other architecture choices for DNN e.g., neural networks with residual connections, that may perform better.

F.2 GRAND-SLAMIN versus GAMs with all pairwise interactions

Next, we compare GRAND-SLAMIN i.e., GAMs with sparse pairwise interactions against GAMs with all pairwise interactions toolkit:

NA 2 M, i.e., Neural Additive Model [1] with pairwise interactions,

NB 2 M, i.e., Neural Bases Model [46] with pairwise interactions,

SPAM, i.e., Scalable Polynomial Additive Model [9] with pairwise interactions.

For NA 2 M, NB 2 M, and SPAM, we tuned over learning rate in the set { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } and number of epochs in the set { 50 , 100 , 500 } . We capped the time for each trial for NA 2 Mto 6 hrs.

We show results in Table F.2. We outperform SPAM across many datasets. Notably, GRAND-SLAMIN improves by 21% over SPAM on Madelon. We are also competitive with NB 2 M and NA 2 M. For larger datasets e.g., Madelon, Activity and Multiple, NB 2 Mand NA 2 M ran out of memory on a compute node with 2 V100 Tesla GPUs. Recall also that the goal of our work is to learn sparse components with/without structural constraints for easier interpretability.

F.3 Comparison with Group Lasso

Group Lasso [16] is also a possible choice for sparse selection, which is popularly used in highdimensional statistics and machine learning. We compare against a version of Group Lasso:

where ˆ I E denotes the empirical loss on the training dataset and ∥ f i ∥ 2 denotes the regularization imposed via the group regularization on the leaf weights of the effect i . We compare performance between GRAND-SLAMIN and Group Lasso (with soft tree shape functions) in terms of test AUC and variable selection on a few datasets. The numbers are reported in Table F.3. Overall, our approach significantly outperforms Group Lasso in terms of AUC performance and variable selection.

F.4 GRAND-SLAMIN with different shape functions (Soft Trees vs MLPs)

Next, we compare GRAND-SLAMIN with different shape functions. In particular, we study the effect of using neural network shape functions instead of soft tree shape functions. We consider multilayer perceptrons (MLPs) as the functional form for each of the shape functions. We compare test AUC performance on 5 datasets in Table F.4. Interestingly, it seems to us that soft trees seem to have an edge over MLPs across some datasets, when MLPs are matched in number of parameters to soft trees.

MLPs with varying complexity We further experimented with MLPs in our framework, where we vary the complexity of the individual components. We show training time on News dataset ( p = 61 , # Interaction = 1800 , N = 40 k ) in Table F.5. The results show that the active set (learnt by learnable indicators) in the model is the primary contributing factor in terms of timing. The functions (1 or 2) → 64 → 64 → 1 have at least 22 × more parameters than (1 or 2) → 64 → 1 . However, the time only increases by ∼ 1 . 5 × . We also show the AUC performance with MLPs with varying complexity in Table F.6. Interestingly, we didn't observe any improvement in AUCs with more complex components. It seems to us that since our additive modeling framework is fitting low-dimensional components - 1-dimensional functions for main effects and 2-dimensional functions for pairwise interactions effects - the complexity of the function class doesn't need to be too large to get good model accuracy.

F.5 Effect of entropy regularization

Next, we study the impact of entropy regularization on performance and variable selection. It can be hypothesized that there might be a trade-off in the speed at which the binary state of the gate variables should be reached. On the one hand, suppression of the uninformative terms should happen fast as

the early forward and backward passes of all p 2 interaction trees is computationally prohibitive. On the other hand, making this decision prematurely means that the model might not have had enough time to fit terms before suppressing them. We provide an ablation study for model performance as a function of entropy regularization in Table F.7. We observe that when the entropy regularization is

too high, the model suffers in performance as some informative terms can also get suppressed too quickly. When the entropy regularization is too small, the model can produce very dense solutions, which can hurt interpretability as well as computational scalability. In a reasonable range of entropy regularization, there is a good region where the model produces a sparse solution for high AUC. Interestingly, we also observe that we didn't see a huge sensitivity of variable selection for a range of entropy values in the middle.

F.6 Effect of screening on performance

Here, we highlight that screening can help improve performance of our model. We consider a challenging Madelon dataset from the NIPS-2003 feature selection challenge, where only a very small subset of features are informative. The number of total features in the dataset are 500. The number of pairwise interactions in this dataset are ∼ 125 k . This is a large combinatorial space for variable selection in terms of interaction effects. A large portion of these interactions are non-informative. Pre-training screening rules similar to the ones used by [34] can effectively reduce the combinatorial space. Recall that the actual number of interactions selected by the model is much smaller than the screened set of interactions through variable selection by our model while training. We can observe from Fig. F.1 that for a range of pre-training screening levels, our model can achieve almost 90% AUC performance - this performance is better than

10 3

10 4

10 5

Size of screened set, |

|

0

20

40

60

80

100

Test ROC AUC (%)

all existing approaches for sparse GAMs with interactions. In particular, the state-of-the art toolkit NODE-GAM that performs end-to-end variable seclection achieves 60% AUC.

Screening also plays a role in faster runtimes. For example, when the screened set is 10,000 interactions out of a total of 125k, the model can be 3 × -5 × faster. Additionally with screening, the overall memory footprint of the model can be much smaller. The overall memory footprint of the model is dictated by the number of interaction effects. With screening, the initial memory footprint of the model can be reduced without a loss in performance.

F.7 Timing comparison

Our toolkit is highly competitive in terms of training times with all existing tree-based and neuralbased toolkits for sparse GAMs with interactions. See Table F.8 for timing comparisons.

G Additional visualizations for variance reduction in estimation of main effects under structural constraints

Here we consider tract-level American Community Survey dataset from US Census Bureau Planning Database 2022 [55]. Following [23], we consider a reduced dataset with ∼ 39 covariates (741 possible pairwise interactions) and consider the self-response as the regression target. We fit GRAND-SLAMIN with different structural constraints. We visualize the estimated main effects in the presence of interaction effects for these structural constraints for some features in Figure G.1. We can observe that when additional hierarchy constraints are imposed, the error bars are much smaller across different runs.

No Hierarchy

Weak Hierarchy

Strong Hierarchy

0.1

0.1

0.1

0.05

0.05

0.05

Main effect

Main effect

0

Main effect

0

-0.05

-0.05

-0.05

-0.1

-0.1

-0.1

-0.15

-0.15

0.15

-0.2

-0.2

-0.2

Averagehousehold Income

Averagehousehold Income

AveragehouseholdIncome

0.15

0.15

0.15

Main effect

0.1

Main effect

0.1

Main effect

0.1

0.05

0.05

0.05

0

0

0.05

-0.05

0.05

-0.1

-0.1

-0.1

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Percentage of households with Internet access

Percentage of households with Internet access

Percentage of households with Internet access

0.3

0.3

0.3

Main effect

0.2

Main effect

0.2

Main effect

0.2

0.1

0.1

0.1

0

0

-0.1

-0.1

-0.1

-0.2

-0.2

-0.2

0

20

40

60

80

0

20

40

60

80

0

20

40

60

80

Percentage of limited English speaking households

Percentage of limited English speaking households

Percentage of limited English speaking households

0.1

0.1

0.1

Main effect

0.05

Main effect

0.05

Main effect

0.05

0

0

0.05

-0.05

-0.05

-0.1

20

40

60

80

100

0

20

40

60

80

100

20

40

60

80

100

Percentage of people below poverty level

Percentage of people below poverty level

Percentage of people below poverty level

0.2

0.2

0.2

0.15

0.15

0.15

Main effect

0.1

Main effect

0.1

Main effect

0.1

0.05

0.05

0.05

0

0

0

-0.05

-0.05

-0.05

-0.1

-0.1

-0.1

-0.15

-0.15

-0.15

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Percentage of single-family homes

Percentage of single-family homes

Percentage of single-family homes

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces hierarchical structure constraints to improve the sparsity of selected features in interaction-aware GAM. The authors use soft relaxations to enable gradient descent during this procedure. Theoretical analyses are given to show statistical properties when using soft tree shape functions in their method.\n\nSTRENGTHS:\n1.\tProvides a simple but reasonable design of hierarchical constraints on the sparsity of feature selection in GAM.\n2.\tIntroduces background information and related works in detail.\n3.\tOffers an efficient implementation that drops unused features during training.\n\nWEAKNESSES:\n1.\tThe technical contribution of this paper is limited. Its framework mainly adopts the existing GA2M with tree-shaped functions. The main contribution is introducing hierarchical constraints to restrict the sparsity of selected features. This is accomplished through learnable indicators that consider the hierarchical constraints. However, this approach is somewhat trivial, as learning Boolean indicators for sparsifying inputs is a common practice. Also, many methods, such as based on Lasso or Gumbel-sigmoid, have been proposed to enable gradient descent on such settings.\n2.\tThe theoretical analysis does not appear to be very relevant to the key contribution claimed. Specifically, the paper discusses the statistical properties of an additive model with soft tree-shaped functions, but does not consider any hierarchy constraints. Since GAM with tree-shaped functions is an existing technique and this paper mainly proposes hierarchy constraints, these analyses do not provide many insights into the methodologies proposed in this paper.\n3.\tThe presentation of this paper is poor. In section 5, many symbols appear without specific explanations, making the analysis confusing and difficult to understand.\n\nQUESTIONS:\n1.\tWhat do $n$, $\\bar{u}(f)$, and $\\vee $ mean?\n2.\tWhat do $a_n$ and $b_n$ represent in the model?\n3.\tHow does the proposed method improve upon other methods that support hierarchy constraints, such as GAMI-Net and SIAN?\n\nLIMITATIONS:\nThere is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present paper presents a method to incprorate pairwise interaction effects into differentiable generalized additive models while retaining scalability. \n\nThe authors pose the fitting of the GAM with pairwise interactions as a combinatorial optimization problem with indicator variables and hierarchy constraints added to the pairwise interactions. To be able to efficiently solve this optimization problem, the authors relax the indicator variables using the smooth step function, which is exactly zero or one at the tails of the function, allowing them to discard certain functions during the forward pass. As shape functions, the authors use soft trees, as opposed to e.g. NAMs which use neural networks. To ensure that the gates reach binary state by the end of training they add an entropy regularizer.The authors provide prediction bounds for sparse additive models.\n\nIn their experiments, they compare with existing method such as Node-GAM which incorporate pairwise interactions and show favourable results.\n\nSTRENGTHS:\n- The paper is overall well written and the structure is easy to follow.\n- The methodological contribution is neat and only adds few hyperparameters.\n- The proposed method produces sparser pairwise interactions which are also faster to train than the compared methods.\n\nWEAKNESSES:\n- Table 2, Table 3, Table 4, Figure 2 please provide the number of runs/seeds and either the standard error, mean absolute deviation or bootstrap confidence intervals for the results. The present analysis could be misleading if it was only carried out over one seed.\n- As your method is applicable to any shape function class, it would be interesting to show results for shape functions like in Neural Additive Models modelled via MLPs.\n- I would have expected the models with weak hierarchy to strictly outperform the models with strong hierarchy. Could you comment on this?\n\nTypos:\n- Line 26 'xchallenging'\n- Line 46 'pose'\n- Line 87 'first step'\n- Line 328 'significantly/much faster'\n\n\nImprovements:\n- Consider increasing the font size of Figure 1 and Figure 2.\n\nQUESTIONS:\n- I think the contribution of the paper is sound, but currently their experimental evaluation is weak as the comparison between methods appears to be carried out on a single seed (I saw some of the results in the appendix have results over additional seeds, but these results need to be in the main paper.). I would consider increasing my score if this is addressed.\n\n- How sensitive is the variable selection to the entropy regularization strength?\n\nLIMITATIONS:\n- Limitations are not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce GRAND-SLAMIN’, a framework for end-to-end learning of sparse Generalized Additive Models (GAMs). GRAND-SLAMIN’ utilizes soft neural decision trees as base learners and incorporates support for second-order interactions and structural constraints. Sparse variable selection is achieved via a smooth relaxation of binary gate variables which play the role of suppressing unused terms. These gates can also be combined to enable structural constraints, restricting second-order interaction to participating first-order terms. This formulation facilitates a sparse backpropagation strategy in which suppressed terms and interactions are pruned from the model early in training. The experiments demonstrate promising results in terms of prediction accuracy and scalability. The paper also establishes novel non-asymptotic prediction bounds for GAM estimators with soft tree-based shape functions, providing a theoretical foundation for their approach.\n\nSTRENGTHS:\n* The end-to-end differentiable parameterization for feature selection is a novel and valuable contribution. It greatly simplifies the implementation of the feature selection and feature interaction detection procedure and integrates well within the model optimization process. It also enables the enforcement of structural constraints, further enhancing the flexibility and interpretability of the framework.\n\n* The inclusion of non-asymptotic bounds provide a theoretical basis for the behavior and performance of the proposed approach, showcasing that robustness to noise improves as more samples are observed.\n\n* The experimental results also support the intuition that incorporating feature selection into the Generalized Additive Model (GAM) can benefit robustness to noise and its generalization. This finding is interesting and can maybe be linked to previous work on feature selection and robustness: L1 regularized regression (Lasso) is a form of robustness to feature-wise disturbance [Xu, 2008] and selecting relevant subsets of features in feature bagging is known to help model performance [Tian, 2021].\n\n[Xu, 2008]: Xu, H., Caramanis, C., & Mannor, S. (2008). Robust Regression and Lasso. Advances in Neural Information Processing Systems, 21. [https://proceedings.neurips.cc/paper_files/paper...](https://proceedings.neurips.cc/paper_files/paper/2008/hash/24681928425f5a9133504de568f5f6df-Abstract.html)\\\n[Tian, 2021]: Tian, Y., & Feng, Y. (2021). RaSE: Random Subspace Ensemble Classification. Journal of Machine Learning Research, 22(45), 1–93.\nhttps://jmlr.org/papers/v22/20-600.html\n\nWEAKNESSES:\n* The lack of visualizations of local and global explanations which can be generated with the proposed framework hinders the assessment of interpretability. Visualizations play an important role in presenting and communicating the decision-making process of interpretable models and shedding light on the contribution of individual features and their interactions.\n\n* The binary gates snap into place reaching a permanent state of 0 or 1 as training progresses. This is understandably a desired property of smoothstep activation, however it means that the model cannot update decisions regarding the usefulness of input features once this permanent state is reached.\n\nQUESTIONS:\n* I think the inclusion of both local and global visualizations of the recovered shape functions would significantly improve the clarity and impact of the paper. Illustrating the neural soft decision trees would help readers gain a deeper understanding of how the input features contribute to model predictions and encourage its adoption down the line.\n\n* My intuition leads me to believe that there might be a trade-off in the speed at which the binary state of the gate variables should be reached. On the one-hand suppression of the un-informative terms should happen fast as the early forward and backward passes of all $p^2$ interaction trees is computationally prohibitive. On the other, making this decision prematurely means that the model might not have had enough time to fit terms before suppressing them. Maybe you could observe as an ablation model performance as a function of the entropy regularizer hyperparameter $\\tau$?\n\n* Related to the above point, do you find that models with different random initializations agree on the usefulness of features, meaning, are the selected features and selected interactions consistent between runs? Perhaps there is a metric which can quantify this (for ex. Jaccard similarity) and you could numerically compare with the screening approach of the other baselines.\n\n* I am not familiar with neural soft decision trees but they seem like they could be quite sensitive to the choice of hyperparameters, and in particular the choice of learning rate. How difficult are they to train in practice? Can you provide some details on the Optuna hyperparameter search: Is it very time consuming? Is the approximated hyperparameter objective function smooth?\n\nLIMITATIONS:\nNo, I think the limitations could be better addressed by dedicating a section in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors proposed an GAM under structure constraints. By incorporating herarical constraint, the proposed method presents better performance than two stage methods. Theoritical analysis are also provided to bound the generalization error.

STRENGTHS:
A GAM method was proposed, which can be learnt in an end to end manner. By introducing several binary variables, herachical constraints are automatically imposed on the model. 
The theoritical parts are new.

WEAKNESSES:
1, Although the authors claim their model can be learnt efficienctly, there still is p^2 models need to be learnt, which is unpractical for large scale datasets. 
2, There is no explanation for why soft trees are used. Actually, the soft tree can be viewed as a NN with special    structure.  So why MLP is not used? 
3, Although same hierarchy structures are also used in GAMI-NET, nevertheless, why it helps to improve the performance of the model is still unclear to me. The motivation to use such a structure is weak in this paper.

QUESTIONS:
1, which screening method is used in your method? Does  it same to that in EBM?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with interpretable machine learning. Authors propose a new interpretable model based on generalized additive modeling. Namely they propose a scalable solution for sparse interaction modeling. They compare against related work on interpretable benchmarks.\n\nSTRENGTHS:\n+ Novel contributions: novel optimization of additive models with indicator variables; backward pass customization for indicator functions.\n+ Really good theoretical analysis that is greatly appreciated, especially the level of detail in the appendix\n+ Well and clearly explained approach\n+ SOTA performance on a wide range of datasets\n+ Very scalable as shown with detailed timing results from the appendix\n\nWEAKNESSES:\n- Somewhat weird choice of datasets. Eg, some standard datasets from papers like NAM, NBM, SPAM, NODE-GAM missing. Most of the datasets presented in this work have a very small difference in performance between the methods, more or less they all perform the same.\n- EBM also has EB2M version, unclear why is it not compared with it, or am I wrongly understanding the tables? Some strong related works with open source libraries missing: NAM, NBM, SPAM, all three are available with 2nd and some even with 3rd order interactions.\n- Unclear why would a use of such a model select no hier, weak hier, or strong hier. Depending on the dataset, different hierarchy performs the best. \n- Can this approach model 3rd order interactions, it would be very interesting to see results on that?\n- Is there any interpretability difference between this model and eg NODE-GAM at 2nd order? Is there any benefit in such a sparsity constrains and trainable selection functions over NODE-GAM? Analysis showing some insight into this would improve the paper.\n\nQUESTIONS:\nSee points in weaknesses.\n\nLIMITATIONS:\nNon adequate limitations discussion in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}