SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1 6}{\sqrt {n}} \int_ {\delta_ {t} ^ {2} / 4 \sigma} ^ {\delta_ {t}} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , t , s) \cap \{\| f \| _ {P _ {n}} \leq \delta_ {t} \} , P _ {n} , \epsilon)} d \epsilon \leq \frac {\delta_ {t} ^ {2}}{4 \sigma}.
$$
We will prove the claim later. Let
$$
\mathcal {F} (d, t, s) = \left\{f _ {1}: \mathbb {J} \subseteq 2 ^ {[ p ]}, | \mathbb {J} | \leq s, | \mathcal {J} | \leq t \forall \mathcal {J} \in \mathbb {J}, f _ {1} \in \mathcal {F} _ {1} (d, \mathbb {J}) \right\}. \tag {C.32}
$$
Under this notation, we have
$$
\hat{f}_{t}\in \operatorname *{argmin}_{f\in \mathcal{F}(d,t,s)}\frac{1}{n}\sum_{i = 1}^{n}(y_{i} - f(\boldsymbol{x}_{i}))^{2}.
$$
Using the notation from [57][Ch. 13], we have
$$
\partial \mathcal {F} (d, t, s) = \mathcal {F} (d, t, s) - \mathcal {F} (d, t, s) = \mathcal {F} ^ {*} (d, t, s)
$$
with  $\mathbb{J}^*$  being the set of interactions for for  $f_{t}^{*}$ . Moreover, we note that  $\mathcal{F}^{*}(d,t,s)$  is star-shaped, that is for  $f\in \mathcal{F}^{*}(d,t,s)$  and  $\alpha \in [0,1]$ ,  $\alpha f\in \mathcal{F}^{*}(d,t,s)$ . This is true as
$$
\alpha f (\cdot ; \boldsymbol {W}, \boldsymbol {o}) = f (\cdot ; \boldsymbol {W}, \alpha \boldsymbol {o}).
$$
Under the claim and assumptions of the theorem, we have  $\delta_t \leq \sigma$  so we invoke Corollary 13.7 and Theorem 13.13 of [57] with  $\gamma = 1/2$ ,  $t = \delta$  to achieve
$$
\mathbb {P} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} ^ {*} - \hat {f} _ {t} (\boldsymbol {x} _ {i})\right) ^ {2} \lesssim \inf  _ {f \in \mathcal {F} (d, t, s)} \sum_ {i = 1} ^ {n} \left(y _ {i} ^ {*} - f (\boldsymbol {x} _ {i})\right) ^ {2} + \delta_ {t} ^ {2}\right) \geq 1 - c _ {1} \exp \left(- c _ {2} n \delta_ {t} ^ {2} / \sigma^ {2}\right) \tag {C.33}
$$
Proof of Claim: For  $t = 1$
$$
\begin{array}{l} \frac {\delta_ {1} ^ {2}}{4 \sigma} = \frac {\left[ 6 4 \sigma (\sqrt {2 s \log p} + C s A) \right] ^ {4 / 3}}{4 \sigma n ^ {2 / 3}} \\ = \frac {1 6}{\sqrt {n}} (\sqrt {2 s \log p} + C s A) \frac {\left[ 6 4 \sigma (\sqrt {2 s \log p} + C s A) \right] ^ {1 / 3}}{n ^ {1 / 6}} \\ = \frac {1 6}{\sqrt {n}} (\sqrt {2 s \log p} + C s A) \sqrt {\delta_ {1}} \\ \geq \frac {1 6}{\sqrt {n}} \int_ {\delta_ {1} ^ {2} / 4 \sigma} ^ {\delta_ {1}} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 1 , s) , P _ {n} , \epsilon)} d \epsilon \\ \geq \frac {1 6}{\sqrt {n}} \int_ {\delta_ {1} ^ {2} / 4 \sigma} ^ {\delta_ {1}} \sqrt {\log \mathcal {N} \left(\mathcal {F} ^ {*} (d , 1 , s) \cap \left\{\| f \| _ {P _ {n}} \leq \delta_ {1} \right\} , P _ {n} , \epsilon\right)} d \epsilon \tag {C.34} \\ \end{array}
$$
where the first inequality is by Lemma C.6 and the final inequality is by Lemma C.8. Next, for  $t = 2$
$$
\begin{array}{l} \frac {\delta_ {2} ^ {2}}{4 \sigma} = \frac {\left[ 1 6 \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) (4 \sigma) ^ {1 + a} \right] ^ {2 / (2 + a)}}{4 \sigma n ^ {1 / (2 + a)}} \\ = \frac {1 6}{\sqrt {n}} \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) \frac {\left[ 1 6 \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) (4 \sigma) ^ {1 + a} \right] ^ {- a / (2 + a)} (4 \sigma) ^ {a}}{n ^ {- a / 2 (2 + a)}} \\ = \frac {1 6}{\sqrt {n}} \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) (4 \sigma / \delta) ^ {a} \\ \geq \frac {1 6}{\sqrt {n}} \int_ {\delta_ {2} ^ {2} / 4 \sigma} ^ {\delta_ {2}} \sqrt {\log \mathcal {N} \left(\mathcal {F} ^ {*} (d , 2 , s) \cap \left\{\| f \| _ {P _ {n}} \leq \delta_ {2} \right\} , P _ {n} , \epsilon\right)} d \epsilon \tag {C.35} \\ \end{array}
$$
by Lemmas C.6 and C.8.
$$
\delta_ {2} = \frac {\left[ 1 6 \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) (4 \sigma) ^ {1 + a _ {n}} \right] ^ {1 / (2 + a _ {n})}}{n ^ {1 / 2 (2 + a _ {n})}}.
$$
By taking  $n \gtrsim b_n^{-2(2 + a_n)}$  and invoking Lemmas C.7 and C.8,
$$
\frac {\delta_ {2} ^ {2}}{4 \sigma} \geq \frac {1 6}{\sqrt {n}} \int_ {\delta_ {2} ^ {2} / 4 \sigma} ^ {\delta_ {2}} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 2 , s) \cap \{\| f \| _ {P _ {n}} \leq \delta_ {2} \} , P _ {n} , \epsilon)} d \epsilon .
$$
The rest of the proof is similar to the proof of Theorem 1.
C.5 Theory Discussion
First, we consider the case with only main effects, i.e  $t = 1$ . In our framework, we use the smooth-step function which is differentiable with uniformly bounded derivative, but not twice differentiable. Therefore, our framework operates on  $L_{2}$  Sobolev functions of order 1, i.e. bounded differentiable functions with bounded gradients. For this case, the results of [52] show that an  $\ell_{2}$  prediction rate of  $n^{-2/3}$  (see Remark 4 of [52] with  $q = 0$  and  $\beta_{0} = 1$ ). This is the same rate we achieve in Theorem 1 for  $t = 1$ . However, we note that our results hold without differentiability assumption and only require the activation function to be Lipschitz, showing stronger results specialized to the soft trees case.
Next, we consider the interaction case with  $t = 2$ . In this case, the results of [23] imply an  $\ell_2$  prediction rate of  $n^{-1/2}$  for the class of  $L_2$  Sobolev functions of order 1 (see Section S2.2 with  $m = 1$ , leading to  $r_n = n^{-1/4}$ ). This is the same rate our method achieves asymptotically (see Theorem 2). However, as we discussed, our results hold without the differentiability assumption.
D Extension to third-order interactions
Our approach can be extended to model third-order interactions. In the case of third-order interactions, the structural constraints for strong and weak hierarchy can be described as follows:
$$
\text {S t r o n g H i e r a c h y}: f _ {j, k, l} \neq 0 \Rightarrow f _ {j} \neq 0 \text {a n d} f _ {k} \neq 0 \text {a n d} f _ {l} \neq 0, \tag {D.1}
$$
$$
\text {W e a k H i e r a c h y}: f _ {j, k, l} \neq 0 \Rightarrow f _ {j} \neq 0 \quad \text {o r} \quad f _ {k} \neq 0 \quad \text {o r} \quad f _ {l} \neq 0. \tag {D.2}
$$
These constraints can be modeled with binary variables as follows:
$$
\text {S t r o n g H i e r a c h y}: q \left(z _ {j}, z _ {k}, z _ {l}, z _ {j, k, l}\right) \stackrel {\text {d e f}} {=} z _ {j} z _ {k} z _ {l} z _ {j, k, l} \tag {D.3}
$$
$$
\text {W e a k H i e r a c h y}: q \left(z _ {j}, z _ {k}, z _ {l}, z _ {j, k, l}\right) \stackrel {\text {d e f}} {=} \left(z _ {j} + z _ {k} + z _ {l} - z _ {j} z _ {k} - z _ {j} z _ {l} - z _ {k} z _ {l} + z _ {j} z _ {k} z _ {l}\right) z _ {j, k, l} \tag {D.4}
$$
Hence, our sparse selection and hierarchy constraints can add value in terms of model compactness and feature selection in the settings with third-order interactions as well. However, for third-order interactions, the number of third-order interactions are  $O(p^3)$ . Hence, for scalability, this would also require using a pre-training screening approach. Despite the fact that our approach is generalizable to third-order interactions, we do not consider such interactions because these are not considered to be easily interpretable.
E Datasets, Computing Setup and Tuning
Datasets We use a collection of 16 open-source classification datasets (binary, multiclass and regression) from various domains, e.g., physics, computing, healthcare, life sciences, finance, and social networks. They are from Penn Machine Learning Benchmarks (PMLB) [41] and UCI databases [8]. For datasets with available training, validation and test splits, we used them in their original form. When no test set was available, we treated the original validation set as the test set and split the training set into  $80\%$  training and  $20\%$  validation. For remaining, we randomly split each of the dataset into  $60\%$  training,  $20\%$  validation and  $20\%$  testing sets. A summary of the 16 datasets considered is in Table E.1.
Computing Setup. We used a cluster running Ubuntu 7.5.0 and equipped with Intel Xeon Platinum 8260 CPUs and Nvidia Volta V100 GPUs. For all experiments of Sec. 6, each job involving GRAND-SLAMIN, EBM, Node-GAM, SIAN, GAMI-Net and DNN were run on 8 core, 32GB RAM. Jobs involving larger datasets  $(p > 100)$  were run on Tesla V100 GPUs.
Table E.1: Summary of Datasets
Dataset	Domain	N	C	p	No. of interactions (|I|)
Magic	Physics	19,020	2	10	55
Adult	Socio-economic	48,842	2	14	91
Churn	Business	5,000	2	19	190
Satimage	Physics	6,435	6	36	640
Texture	Image	5,500	11	40	780
MiniBooNE	Physics	130,065	2	50	1,225
Covertype	Life Science	581,012	7	54	1,431
Spambase	Computing	4,601	2	57	1,596
News	Social networks	39,797	2	61	1,830
Optdigits	Image	5,620	10	64	2,016
Bankruptcy	Finance	6,819	2	96	4,560
Madelon	NIPS-2003	2,600	2	500	124,750
Activity	Healthcare	4,480	4	533	141,778
Multiple	Image	2,000	10	649	210,276
Bike Sharing	Transportation	17,389	-	16	120
American Community Survey	Demographic	83,059	-	39	741
Tuning. The tuning was done in parallel over the competing models and datasets. We tune the hyperparameter using Optuna [2] which optimizes the overall AUC on a validation set. We report the results on a held-out test set. A list of all the tuning parameters and their distributions is given for GRAND-SLAMIN below:
- Learning Rates: Discrete uniform in the set  $\{0.05, 0.01, 0.005\}$  for Adam with multi-step decay rate of 0.9 every 25 epochs.
- Batch-size: Discrete uniform in the set  $\{64, 256\}$ .
-  $\lambda$  for selection: Discrete uniform in the set of 11 values  $\{0, 1e - 6, \dots, 1e - 3\}$ .
-  $\gamma$  for Smooth-step: Discrete uniform in the set  $\{0.01, 0.1, 1\}$ .
-  $\tau$  for Entropy Regularization: Discrete uniform in the set  $\{0.001, 0.01, 0.1\}$ .
-  $\alpha$  for relative penalty on interactions: Discrete uniform in the set  $\{1, 10\}$ .
- Epochs: 1000 with early stopping (patience=50) based on validation loss.
- For Madelon, Activity and Multiple datasets, we used screening to reduce the initial set of interactions to 5000.
For other toolkits, we considered the tuning protocols outlined in the respective papers such that the parameter controlling the variable selection is tuned. For SIAN, we tuned over the threshold parameter  $\theta$  such that the screened set of interactions is upper bounded by  $\{250, 500, \dots, 1000\}$ . We set  $\tau = 1$  such that the model satisfies strong hierarchy. For GAMI-Net, we tuned over the number of interactions. For small datasets  $(|\mathcal{I}| < 1000)$ , we tuned over the set:  $\{0.2|\mathcal{I}|, 0.4|\mathcal{I}|, \dots, |\mathcal{I}|\}$ . For large datasets  $(|\mathcal{I}| < 1000)$ , we tuned over the set:  $\{250, 500, \dots, 1000\}$ . For EBM, we tuned over the set of interactions  $\{16, 32, 64, 128\}$  as done by authors in [6]. For NODE-GA $^2$ M, we tuned the number of trees as this controls the maximum number of interactions selected by the model.
F.1 Comparison with full complexity models e.g., DNN
Here, we compare interpretable modeling toolkits based on pairwise interactions with full complexity models. In particular, we consider a deep neural network (DNN). We considered the same architecture for the DNN as used by the authors in [10] for similar comparisons. The architecture is a 4-layered ReLU-activated neural network. We show the test AUC performance of different models in Table F.1. Across all datasets, we see that the full complexity DNN model underperforms interpretable models including GRAND-SLAMIN. However, it is noted that the choice of the architecture for DNN may be contributing to the degradation in performance. There maybe other architecture choices for DNN e.g., neural networks with residual connections, that may perform better.
Table F.1: Test ROC AUC of GRAND-SLAMIN,  $\mathrm{EB^2M}$  and NODE-  $\mathrm{GA^2M}$ . We report median across 10 runs along with the mean absolute deviation (MAD).
Dataset\Model	Interpretable Models	Full Complexity Models
EB2M	NODE-GA2M	GRAND-SLAMIN	DNN
Magic	93.12 ± 0.001	94.27 ± 0.13	93.86 ± 0.30	93.69 ± 0.04
Adult	91.41 ± 0.0004	91.75 ± 0.14	91.54 ± 0.14	90.26 ± 0.04
Churn	91.97 ± 0.005	89.62 ± 5.61	92.40 ± 0.41	90.28 ± 0.34
Satimage	97.65 ± 0.001	98.70 ± 0.07	98.81 ± 0.04	98.67 ± 0.07
Texture	99.81 ± 0.0004	100.00 ± 0.00	100.00 ± 0.00	99.63 ± 0.09
MiniBooNE	97.86 ± 0.0001	98.44 ± 0.02	97.77 ± 0.05	97.08 ± 0.38
Covertype	90.08 ± 0.0003	95.39 ± 0.12	98.11 ± 0.08	93.83 ± 0.10
Spambase	98.84 ± 0.01	98.78 ± 0.06	98.01 ± 4.70	98.09 ± 0.07
News	73.03 ± 0.002	73.53 ± 0.06	73.24 ± 0.04	72.19 ± 0.08
Optdigits	99.79 ± 0.0003	99.93 ± 0.02	99.98 ± 0.00	99.77 ± 0.04
Bankruptcy	93.85 ± 0.01	92.02 ± 1.03	92.51 ± 0.54	88.38 ± 0.36
Madelon	88.04 ± 0.02	60.07 ± 0.82	89.25 ± 1.03	63.60 ± 0.47
Activity	74.96 ± 8.77	99.86 ± 0.04	99.24 ± 1.45	96.64 ± 1.93
Multiple	99.96 ± 0.0002	99.94 ± 0.02	99.92 ± 0.02	99.94 ± 0.01
F.2 GRAND-SLAMIN versus GAMs with all pairwise interactions
Next, we compare GRAND-SLAMIN i.e., GAMs with sparse pairwise interactions against GAMs with all pairwise interactions toolkit:
1.  $\mathrm{NA}^2\mathrm{M}$ , i.e., Neural Additive Model [1] with pairwise interactions,
2.  $\mathrm{NB}^2\mathrm{M}$ , i.e., Neural Bases Model [46] with pairwise interactions,
3. SPAM, i.e., Scalable Polynomial Additive Model [9] with pairwise interactions.
For  $\mathrm{NA}^2\mathrm{M}$ ,  $\mathrm{NB}^2\mathrm{M}$ , and SPAM, we tuned over learning rate in the set  $\{0.1, 0.01, 0.001, 0.0001\}$  and number of epochs in the set  $\{50, 100, 500\}$ . We capped the time for each trial for  $\mathrm{NA}^2\mathrm{M}$  to 6 hrs.
We show results in Table F.2. We outperform SPAM across many datasets. Notably, GRAND-SLAMIN improves by  $21\%$  over SPAM on Madelon. We are also competitive with  $\mathrm{NB}^2\mathrm{M}$  and  $\mathrm{NA}^2\mathrm{M}$ . For larger datasets e.g., Madelon, Activity and Multiple,  $\mathrm{NB}^2\mathrm{M}$  and  $\mathrm{NA}^2\mathrm{M}$  ran out of memory on a compute node with 2 V100 Tesla GPUs. Recall also that the goal of our work is to learn sparse components with/without structural constraints for easier interpretability.
Table F.2: Test ROC AUC for GRAND-SLAMIN and GAMs with all pairwise interaction models, e.g.,  $\mathrm{NA}^2\mathrm{M}$ ,  $\mathrm{NB}^2\mathrm{M}$  and SPAM.
Dataset	NA2M (Dense)	NB2M (Dense)	SPAM (Dense)	GRAND-SLAMIN (Sparse)
Magic	94.46 ± 0.16	94.11 ± 0.08	91.75 ± 0.004	93.86 ± 0.30
Adult	90.81 ± 0.10	91.06 ± 0.03	89.65 ± 0.001	91.54 ± 0.14
Churn	93.03 ± 0.49	92.16 ± 0.42	88.41 ± 0.05	92.40 ± 0.41
Satimage	98.81 ± 0.05	98.89 ± 0.03	97.94 ± 0.02	98.81 ± 0.04
Covertype	out of time	98.36 ± 0.02	96.29 ± 0.02	98.11 ± 0.08
Spambase	98.38 ± 0.05	98.37 ± 0.06	97.78 ± 0.04	98.55 ± 0.07
News	71.94 ± 0.30	72.54 ± 0.07	72.43 ± 0.06	73.24 ± 0.04
Bankruptcy	87.83 ± 0.12	93.01 ± 1.85	89.35 ± 0.90	92.51 ± 0.54
Madelon	out of memory	out of memory	68.59 ± 0.80	89.25 ± 1.03
Activity	out of memory	out of memory	99.10 ± 0.04	99.24 ± 1.45
Multiple	out of memory	out of memory	99.98 ± 0.03	99.92 ± 0.02
F.3 Comparison with Group Lasso
Group Lasso [16] is also a possible choice for sparse selection, which is popularly used in high-dimensional statistics and machine learning. We compare against a version of Group Lasso:
$$
\min  _ {\left\{f _ {i} \right\} _ {i \in [ p ]}, \left\{f _ {i, j} \right\} _ {i <   j}} \hat {\mathbb {E}} \left[ l \left(y, \sum_ {i \in [ p ]} f _ {i} \left(x _ {i}\right) + \sum_ {i <   j} f _ {i, j}\right) \right] + \lambda \left(\sum_ {i \in [ p ]} \| f _ {i} \| _ {2} + \sum_ {i <   j} \| f _ {i, j} \| _ {2}\right), \tag {F.1}
$$
where  $\hat{\mathbb{E}}$  denotes the empirical loss on the training dataset and  $\| f_i\| _2$  denotes the regularization imposed via the group regularization on the leaf weights of the effect  $i$ . We compare performance between GRAND-SLAMIN and Group Lasso (with soft tree shape functions) in terms of test AUC and variable selection on a few datasets. The numbers are reported in Table F.3. Overall, our approach significantly outperforms Group Lasso in terms of AUC performance and variable selection.
Table F.3: Comparison of Test ROC AUC of GRAND-SLAMIN with Group Lasso.
Model Dataset	Group Lasso (AUC)	GRAND-SLAMIN (AUC)	Group Lasso (#features)	GRAND-SLAMIN (#features)
Adult	91.19	91.54 ± 0.14	14	12
Spambase	98.32	98.81 ± 0.04	57	44
Madelon	65.22	90.13 ± 1.03	500	15
F.4 GRAND-SLAMIN with different shape functions (Soft Trees vs MLPs)
Next, we compare GRAND-SLAMIN with different shape functions. In particular, we study the effect of using neural network shape functions instead of soft tree shape functions. We consider multilayer perceptrons (MLPs) as the functional form for each of the shape functions. We compare test AUC performance on 5 datasets in Table F.4. Interestingly, it seems to us that soft trees seem to have an edge over MLPs across some datasets, when MLPs are matched in number of parameters to soft trees.
Table F.4: Soft tree shape functions vs MLP shape functions.
Dataset\Model	GRAND-SLAMIN with MLPs	GRAND-SLAMIN with Soft Trees
Magic	93.13±0.12	93.86±0.30
Churn	92.33±0.56	92.40±0.41
MiniBooNE	97.41±0.21	97.77±0.05
Spambase	98.27±0.13	98.55±0.07
News	72.87±0.09	73.24±0.04
MLPs with varying complexity We further experimented with MLPs in our framework, where we vary the complexity of the individual components. We show training time on News dataset  $(p = 61, \# Interaction = 1800, N = 40k)$  in Table F.5. The results show that the active set (learnt by learnable indicators) in the model is the primary contributing factor in terms of timing. The functions  $(1$  or  $2) \rightarrow 64 \rightarrow 64 \rightarrow 1$  have at least  $22 \times$  more parameters than  $(1$  or  $2) \rightarrow 64 \rightarrow 1$ . However, the time only increases by  $\sim 1.5 \times$ . We also show the AUC performance with MLPs with varying complexity in Table F.6. Interestingly, we didn't observe any improvement in AUCs with more complex components. It seems to us that since our additive modeling framework is fitting low-dimensional components — 1-dimensional functions for main effects and 2-dimensional functions for pairwise interactions effects — the complexity of the function class doesn't need to be too large to get good model accuracy.
F.5 Effect of entropy regularization
Next, we study the impact of entropy regularization on performance and variable selection. It can be hypothesized that there might be a trade-off in the speed at which the binary state of the gate variables should be reached. On the one hand, suppression of the uninformative terms should happen fast as
Table F.5: Training times for different choices of MLP shape functions.
% Components Selected	MLPs with varying complexity	Training times
25% effects	(1 or 2) → 64 → 1	252.4 ± 0.4
(1 or 2) → 64 → 64 → 1	247.6 ± 0.8
(1 or 2) → 64 → 64 → 64 → 1	258.8 ± 10.7
(1 or 2) → 128 → 128 → 1	268.5 ± 19.1
(1 or 2) → 256 → 256 → 1	837.0 ± 43.6
75% effects	(1 or 2) → 64 → 1	343.3 ± 5.3
(1 or 2) → 64 → 64 → 1	422.7 ± 9.9
(1 or 2) → 64 → 64 → 64 → 1	506.0 ± 34.4
(1 or 2) → 128 → 128 → 1	552.4 ± 151.4
(1 or 2) → 256 → 256 → 1	1193.8 ± 0.3
Table F.6: AUC performance with MLP shape functions.
MLPs with varying complexity	Test AUC
(1 or 2) → 64 → 1	72.90 ± 0.09
(1 or 2) → 64 → 64 → 1	72.48 ± 0.13
(1 or 2) → 64 → 64 → 64 → 1	72.58 ± 0.17
(1 or 2) → 128 → 128 → 1	72.34 ± 0.11
(1 or 2) → 256 → 256 → 1	71.86 ± 0.37
the early forward and backward passes of all  $p^2$  interaction trees is computationally prohibitive. On the other hand, making this decision prematurely means that the model might not have had enough time to fit terms before suppressing them. We provide an ablation study for model performance as a function of entropy regularization in Table F.7. We observe that when the entropy regularization is
Table F.7: Effect of entropy regularization  $\tau$  on test AUC and component selection on Spambase. We report median and mean absolute deviation across 50 runs.
τ	0.00001	0.0001	0.001	0.005	0.01	0.05	0.1	1.0
AUC	98.58±0.02	98.58±0.02	98.54±0.02	98.37±0.09	98.40±0.12	98.40±0.15	98.43±0.11	98.30±0.08
#Effects Selected	1653 ± 0	1235 ± 164	592 ± 173	558 ± 328	535 ± 352	507 ± 451	721 ± 528	1653 ± 26
too high, the model suffers in performance as some informative terms can also get suppressed too quickly. When the entropy regularization is too small, the model can produce very dense solutions, which can hurt interpretability as well as computational scalability. In a reasonable range of entropy regularization, there is a good region where the model produces a sparse solution for high AUC. Interestingly, we also observe that we didn't see a huge sensitivity of variable selection for a range of entropy values in the middle.
F.6 Effect of screening on performance
Here, we highlight that screening can help improve performance of our model. We consider a challenging Madelon dataset from the NIPS-2003 feature selection challenge, where only a very small subset of features are informative. The number of total features in the dataset are 500. The number of pairwise interactions in this dataset are  $\sim 125k$ . This is a large combinatorial space for variable selection in terms of interaction effects. A large portion of these interactions are non-informative. Pre-training screening rules similar to the ones used by [34] can effectively reduce the combinatorial space. Recall that the actual number of interactions selected by the model is much smaller than the screened set of interactions through variable selection by our model while training. We can observe from Fig. F.1 that for a range of pre-training screening levels, our model can achieve almost  $90\%$  AUC performance — this performance is better than
Figure F.1: Performance of GRAND-SLAMIN with various pretraining screening levels on Madelon.
all existing approaches for sparse GAMs with interactions. In particular, the state-of-the-art toolkit NODE-GAM that performs end-to-end variable sclection achieves  $60\%$  AUC.
Screening also plays a role in faster runtimes. For example, when the screened set is 10,000 interactions out of a total of  $125\mathrm{k}$ , the model can be  $3\times -5\times$  faster. Additionally with screening, the overall memory footprint of the model can be much smaller. The overall memory footprint of the model is dictated by the number of interaction effects. With screening, the initial memory footprint of the model can be reduced without a loss in performance.
F.7 Timing comparison
Our toolkit is highly competitive in terms of training times with all existing tree-based and neural-based toolkits for sparse GAMs with interactions. See Table F.8 for timing comparisons.
Table F.8: Training time in seconds of GRAND-SLAMIN,  $\mathrm{EB^2M}$ , NODE-GA $^2$ M, GAMI-Net and SIAN. We report median across 10 runs. Hyphen (-) indicates either the toolkit does not support multiclass e.g., GAMI-Net, SIAN or does not fit interaction effects for multiclass e.g.,  $\mathrm{EB^2M}$ .
Dataset\Model	EB2M	NODE-GA2M	GAMI-Net	SIAN	GRAND-SLAMIN
Magic	140	327	1567	608	430
Adult	284	612	3611	2396	1018
Churn	33	699	2340	298	35
Spambase	297	361	1979	2197	133
Miniboone	1181	1523	39334	64200	1662
Online	402	325	9030	3877	432
Bankruptcy	156	240	2630	3410	57
Madelon	403	899	5217	58500	46
Satimage	—	133	—	—	52
Texture	—	128	—	—	98
Optdigits	—	320	—	—	55
Covertype	—	604	—	—	3144
Activity	—	291	—	—	110
Multiple	—	1417	—	—	184
G Additional visualizations for variance reduction in estimation of main effects under structural constraints
Here we consider tract-level American Community Survey dataset from US Census Bureau Planning Database 2022 [55]. Following [23], we consider a reduced dataset with  $\sim 39$  covariates (741 possible pairwise interactions) and consider the self-response as the regression target. We fit GRAND-SLAMIN with different structural constraints. We visualize the estimated main effects in the presence of interaction effects for these structural constraints for some features in Figure G.1. We can observe that when additional hierarchy constraints are imposed, the error bars are much smaller across different runs.
No Hierarchy
Weak Hierarchy
Strong Hierarchy