SlowGuess/NeurIPS_2023_OCR huggingface dataset:
**Screening.** We describe how screening approaches prior to training, can complement our sparse backpropagation approach when the number of all pairwise interactions is large e.g., of the order 100,000. Fast screening methods based on shallow-tree like models are proposed in [34] for identifying prominent pairwise interactions. These are used by various toolkits e.g., EBM [40], GAMI-Net [60]. These screening approaches are complementary to our selection approach with indicator variables. We used CART [4] for each pairwise interaction and sorted the interaction effects based on AUC performance to select an initial screened set of interaction effects. In particular, we can consider  $\mathcal{Z}$  to be a screened subset (e.g., 10,000) of all pairwise interaction effects of the order 100,000. Then, we run our end-to-end learning framework under the component selection constraints on the main and screened interaction effects. We observe that such screening can be beneficial for multiple reasons: (i) The training time can be reduced further by  $3\times -5\times$ . (ii) The memory footprint of the model reduces by  $10\times$ . (iii) There is no loss in accuracy—the accuracy can sometimes improve with screening — see ablation study in Supplement Sec. F.6. Note that even with screening, our approach is different from GAMI-Net as they directly screen to a much smaller set of interactions e.g., 500 and there is no component selection when training with these interactions.
5 Statistical Theory
In this section, we explore the statistical properties of our additive model with soft tree shape functions. We assume that observations are noisy versions of some (unknown) underlying noiseless data. We do not assume the noiseless data comes from a tree-based model. We show that if our model can approximate the true noiseless data well, the prediction error resulting from the noise converges to zero as the number of observations grows. In particular, for  $n$  data observations  $(x_{i},y_{i})_{i = 1}^{n}$  we consider a sparsity-constrained version of Problem (8) with the least-squares loss as given by
$$
\begin{array}{l} f _ {(t)} (\boldsymbol {y}) \in \operatorname {a r g m i n} \sum_ {i = 1} ^ {n} \left[ y _ {i} - \sum_ {j \in [ p ]} f _ {j} \left(x _ {i, j}\right) z _ {j} - (t - 1) \sum_ {(j, k) \in \mathcal {I}} f _ {j, k} \left(x _ {i, j}, x _ {i, k}\right) q \left(z _ {j}, z _ {k}, z _ {j, k}\right) \right] ^ {2}, \tag {10} \\ \text {s . t .} \{z _ {j} \} \in \{0, 1 \} ^ {p}, \{z _ {j, k} \} \in \{0, 1 \} ^ {| \mathcal {I} |}, \sum_ {j \in [ p ]} z _ {j} + \sum_ {(j, k) \in \mathcal {I}} q (z _ {j}, z _ {k}, z _ {j, k}) \leq s, \\ \end{array}
$$
for  $t \in \{1, 2\}$ , where  $f_j, f_{j,k}$  are depth-  $d$  soft trees (cf. Section 3.2.2). For  $t = 1$ , Problem (10) simplifies to a main effects model, while  $t = 2$  corresponds to the model with pairwise interactions. We study the case with no hierarchy constraints here,  $q(z_j, z_k, z_{j,k}) = z_{j,k}$  as in (5). We expect our approach to extend to more general cases, but we do not pursue that here.
Model Setup. We assume for  $i \in [n]$ , the data is bounded,  $\| \boldsymbol{x}_i\|_2 \leq 1$ . The noisy observations are given as  $y_i = h^*(\boldsymbol{x}_i) + \varepsilon_i = y_i^* + \varepsilon_i$  where  $h^*$  is the unknown underlying generative model (need not be a tree ensemble), and  $\varepsilon_i \stackrel{\mathrm{iid}}{\sim} \mathrm{Normal}(0, \sigma^2)$  are noise values. Suppose  $f$  is a feasible solution for Problem (10). Let  $\mathcal{U}, \mathcal{L}$  denote the set of internal nodes and leaves for a depth  $d$  tree, and let  $(\boldsymbol{W}^j, \boldsymbol{o}^j)$  be the set of weights corresponding to internal nodes and leaves of tree  $j$  i.e.  $f_j(x_j)$  in this solution, where  $\boldsymbol{W}^j \in \mathbb{R}^{|\mathcal{U}|}$ ,  $\boldsymbol{o}^j \in \mathbb{R}^{|\mathcal{L}|}$ . We also let  $W_i^j, o_l^j$  be the weights corresponding to node  $i \in \mathcal{U}$  and leaf  $l \in \mathcal{L}$  in tree  $j$ , respectively. We define  $W_i^{j,k}, o_l^{j,k}$  for  $f_{j,k}(x_j, x_k)$  and  $i \in \mathcal{U}, l \in \mathcal{L}$ .
in a similar fashion, where  $\pmb{W}^{j,k}\in \mathbb{R}^{\lfloor \mathcal{U}\rfloor \times 2}$ ,  $\pmb{o}^{j,k}\in \mathbb{R}^{\lfloor \mathcal{L}\rfloor}$ . See Figure 1 for an illustration. Define
$$
\bar{u} (f) = \max_{\substack{j\in [p],i\in \mathcal{U}\\ z_{j} = 1}}|W_{i}^{j}|\vee \max_{\substack{j\in [p],l\in \mathcal{L}\\ z_{j} = 1}}|o_{l}^{j}|\vee \max_{\substack{j,k\in [p],i\in \mathcal{U}\\ z_{j,k} = 1}}\| \boldsymbol{W}_{i}^{j,k}\|_{2}\vee \max_{\substack{j,k\in [p],l\in \mathcal{L}\\ z_{j,k} = 1}}|o_{l}^{j,k}|
$$
where  $\vee$  denotes maximum. Conceptually,  $\bar{u}(f)$  is the largest weight (in absolute value) that appears in all main effect and interaction soft trees. Let  $\hat{f} = f_{(t)}(\pmb{y})$  be the estimator resulting from the noisy data, and  $f^{*} = f_{(t)}(\pmb{y}^{*})$  be the oracle estimator that is the best approximation to the noiseless data among feasible solutions of (10) (i.e., among sparse additive tree models). We assume the following:
(A1) The activation function  $\phi : \mathbb{R} \mapsto [0, 1]$  for soft trees is  $L$ -Lipschitz for some  $L > 0$ .
(A2) There exists  $B > 0$  such that  $\bar{u} (\hat{f})\vee \bar{u} (f^{*})\leq B$
Assumption (A2) ensures the trees resulting from the data are uniformly bounded. This is a mild assumption as in practice, the data is bounded and the resulting trees are also bounded as a result.
Main Results. Our first result is for a general setup, where  $p$  can be much larger than  $n$ .
Theorem 1. Let  $\hat{f}, f^*$  be as defined above and take  $A = 2 \vee 2^{d+2} B \vee B^2 dL2^{d+3}$  and  $a = \exp(-1)$ . Under the model setup, assume  $s \geq 1$  and  $\sigma \geq 1$ . Then,
(1) For  $t = 1$ , if  $n \gtrsim \hat{\sigma}^2 (s\log p + s^2 A^2)$ , then with high probability
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} (\hat {f} (\boldsymbol {x} _ {i}) - y _ {i} ^ {*}) ^ {2} \lesssim \frac {1}{n} \sum_ {i = 1} ^ {n} (f ^ {*} (\boldsymbol {x} _ {i}) - y _ {i} ^ {*}) ^ {2} + \frac {\sigma^ {4 / 3}}{n ^ {2 / 3}} \left((s \log p) ^ {2 / 3} + (s A) ^ {4 / 3}\right).
$$
(2) For  $t = 2$ , if  $n \gtrsim \sigma^{2 + 2a}(s\log p + s^3 A^3)$ , then with high probability
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} (\hat {f} (\pmb {x} _ {i}) - y _ {i} ^ {*}) ^ {2} \lesssim \frac {1}{n} \sum_ {i = 1} ^ {n} (f ^ {*} (\pmb {x} _ {i}) - y _ {i} ^ {*}) ^ {2} + \frac {\sigma^ {2 (1 + a) / (2 + a)}}{n ^ {1 / (2 + a)}} \left(\sqrt {s \log p} + (s A) ^ {3 / 2}\right) ^ {2 / (2 + a)}.
$$
Theorem 1 presents non-asymptotic bounds for the performance of our method. Particularly, this theorem shows that under a well-specified model where  $y_{i}^{*} = f^{*}(x_{i})$ , prediction error rates of  $n^{-2/3}$  and  $n^{-1/(2+a)} \approx n^{-0.42}$  are achievable for main effects and interaction models, respectively. Particularly, this implies that the prediction error of our estimator (resulting from the noise in observations) converges to zero as we increase the total number of samples,  $n$ . We note that although the rate from the main effects model is sharper, such models might be too simple to capture the noiseless data, leading to larger oracle error. Therefore, in practice, the interactions model can lead to better performance. Next, we show that under the asymptotic assumption that  $n \to \infty$ , the error rate for the interaction models can be further improved.
Theorem 2. Under the model setup with  $t = 2$ , assume all parameters are fixed except  $n$ . For any positive sequence  $a_{n}$  with  $\lim_{n\to \infty}a_n = 0$ , there exists a positive sequence  $b_{n}$  with  $\lim_{n\to \infty}b_n = 0$  such that if  $n\gtrsim b_n^{-1 / 2(2 + a_n)}$  then with high probability
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} (\hat {f} (\pmb {x} _ {i}) - y _ {i}) ^ {2} \lesssim \frac {1}{n} \sum_ {i = 1} ^ {n} (f ^ {*} (\pmb {x} _ {i}) - y _ {i} ^ {*}) ^ {2} + \frac {1}{n ^ {1 / (2 + a _ {n})}}.
$$
Theorem 2 shows that essentially when  $n \to \infty$  and other parameters in the problem stay constant, an error rate of  $n^{-0.5}$  is achievable for the interactions model which improves upon Theorem 1.
Discussion of previous work: As stated, our results are the first to present prediction bounds specialized to learning sparse additive models with soft trees. Moreover, as we discuss in Appendix C.5, our upper bounds (rates) generally align with the best known rates available in the literature for learning main effects and interactions, such as [23, 52], or hold under milder assumptions.
We study the performance of GRAND-SLAMIN on 16 real-world datasets and compare against relevant baselines for different cases. We make the following comparisons:
1. Performance comparison of GRAND-SLAMIN without/with structural constraints against existing toolkits for sparse GAMs with interactions.
2The notation  $\lesssim ,\gtrsim$  depict an inequality holds up to a universal constant independent of problem data.
(a) Toolkits that support sparse interactions:  $\mathrm{EB^2M}$  [40] and NODE-GA $^2$ M [6]
(b) Toolkits that support hierarchy constraints: GAMI-Net [60] and SIAN [10]
2. Variable selection comparison against the competing toolkits.
3. Computational scalability of GRAND-SLAMIN toolkit with sparse backpropagation.
4. Variance reduction with structural constraints
Additional results are included in Supplement Sec. F that study (i) comparison with full complexity models in F.1, (ii) comparison with GAMs with all pairwise interactions in F.2, (iii) comparison with Group Lasso selection approach in F.3, (iv) choice of shape functions in F.4, (v) effect of entropy on performance and component selection in F.5, and (vi) effect of screening on training times, memory and performance in F.6.
Datasets. We use a collection of 16 open-source classification datasets (8 binary, 6 multiclass and 2 regression) from various domains. We consider datasets with a wide range of number of all pairwise interactions  $10 - 200000$ . A summary of the datasets is in Table E.1 in the Appendix.
Tuning Details. For all the experiments, we tune the hyperparameters using Optuna [2] with random search on a held-out validation set. We compute statistical averages across multiple runs for the optimal hyperparameters for all models. In particular, we report median test ROC AUC across 10 runs along with the mean absolute deviation (MAD). Additional details are in the Appendix E.
6.1 Prediction Performance
Comparison with  $\mathbf{EB^2M}$  and NODE-GA $^2$ M. We first study the performance of our model in comparison to two tree-based state-of-the-art toolkits which support sparse GAMs with interactions without any structural constraints e.g.,  $\mathrm{EB}^2\mathrm{M}$  and  $\mathrm{NODE - GA^{2}M}$ . We report the ROC AUC performance in Table 2. Our model outperforms  $\mathrm{EB^2M}$  in 10 out of 14 datasets. Our model is also competitive with  $\mathrm{NODE - GA^2M}$  as it can outperform in  $50\%$  of the datasets. In summary, our results in Table 2 show that we are at par with state-of-the-art methods for unstructured component selection.
Our key advantage is to do hierarchical interactions, which NODE-GA $^2$ M and EB $^2$ M can not support. Additionally, we can achieve faster training times (Sec. 6.3) and improve on variable selection (Sec. 6.2) than NODE-GA $^2$ M and EB $^2$ M.
Structural constraints: Weak and Strong Hierarchy. Next, we study our method with structural constraints i.e., (6) for weak hierarchy or (7) for strong hierarchy. We compare against two competing neural-based state-of-the-art methods for sparse GAMs with hierarchical interactions: (i) GAMI-Net with support for weak hierarchy, and (ii) SIAN with support for strong hierarchy. We omit multiclass datasets as both GAMI-Net and SIAN do not support them. We report the ROC AUC performance in Table 3. Our models outperform GAMI-Net and SIAN in 7/8 datasets.
Additionally, our models are much more compact in terms of overall number of parameters — our tree-based shape functions have  $100 \times$  and  $10 \times$  smaller number of parameters than the neural-based
Table 2: Test ROC AUC of GRAND-SLAMIN,  $\mathrm{EB^2M}$  and NODE-GA $^2$ M. We report median along with mean absolute deviation across 10 runs.
Dataset	EB2M	NODE-GA2M	GRAND-SLAMIN
Magic	93.12 ± 0.001	94.27 ± 0.13	93.86 ± 0.30
Adult	91.41 ± 0.0004	91.75 ± 0.14	91.54 ± 0.14
Churn	91.97 ± 0.005	89.62 ± 5.61	92.40 ± 0.41 (SH)
Satimage	97.65 ± 0.0007	98.70 ± 0.07	98.81 ± 0.04
Texture	99.81 ± 0.0004	100.00 ± 0.00	100.00 ± 0.00
MiniBooNE	97.86 ± 0.0001	98.44 ± 0.02	97.77 ± 0.05 (WH)
Covertype	90.08 ± 0.0003	95.39 ± 0.12	98.11 ± 0.08
Spambase	98.84 ± 0.01	98.78 ± 0.06	98.55 ± 0.07 (SH)
News	73.03 ± 0.002	73.53 ± 0.06	73.24 ± 0.04 (SH)
Optdigits	99.79 ± 0.0003	99.93 ± 0.02	99.98 ± 0.0
Bankruptcy	93.85 ± 0.01	92.02 ± 1.03	92.51 ± 0.54 (WH)
Madelon	88.04 ± 0.02	60.07 ± 0.82	89.25 ± 1.03 (WH)
Activity	74.96 ± 8.77	99.86 ± 0.04	99.24 ± 1.45
Multiple	99.96 ± 0.0002	99.94 ± 0.02	99.95 ± 0.02
Table 3: Test ROC AUC for GRAND-SLAMIN with structural constraints i.e., (6) or (7), GAMI-Net and SIAN. We report median across 10 runs along with mean absolute deviation.
Dataset\Model	GAMI-Net WH	SIAN SH	GRAND-SLAMIN
WH	SH
Magic	91.72 ± 0.05	93.02 ± 0.06	93.16 ± 0.55	93.37 ± 0.16
Adult	91.01 ± 0.04	90.67 ± 0.05	91.34 ± 0.32	91.46 ± 0.15
Churn	90.05 ± 0.77	92.98 ± 0.20	92.28 ± 0.75	92.40 ± 0.41
Spambase	98.67 ± 0.04	98.28 ± 0.04	98.45 ± 0.15	98.55 ± 0.07
MiniBooNE	96.11 ± 0.41	95.90	97.77 ± 0.05	97.62 ± 0.30
News	72.54 ± 0.05	72.28	73.15 ± 0.08	73.24 ± 0.04
Bankruptcy	92.46 ± 0.12	90.71	92.51 ± 0.54	90.45 ± 1.87
Madelon	88.14 ± 0.94	83.18	89.25 ± 1.03	86.23 ± 1.89
WH=Weak Hierarchy, SH=Strong Hierarchy.
For SIAN, for some of the larger datasets (row 5-9),
we use the number for best trial as SIAN takes
$\sim 24$  hours on V100 Tesla GPU.
shape functions used by GAMI-Net and SIAN respectively. Moreover, our toolkit is significantly faster than SIAN and GAMI-Net on larger datasets — see Sec. 6.3.
Additionally, we compare interpretable modeling toolkits with full complexity models e.g., deep neural network (DNN), in Supplement Sec. F.1. We observed interpretable models to outperform full complexity models on these datasets. We also compare our toolkit that fits sparse components with toolkits that fit all pairwise interactions e.g.,  $\mathrm{NA}^2\mathrm{M}$ ,  $\mathrm{NB}^2\mathrm{M}$  and SPAM in Supplement Sec. F.2. GRAND-SLAMIN generally outperform these methods with enhanced interpretability due to sparsity and structural constraints. We also study how our toolkit performs when we replace soft tree shape functions with MLP shape functions in Supplement Sec. F.4. Interestingly, we observe that soft trees seem to have an edge when the parameters are matched with MLP.
6.2 Variable Selection
We evaluate the performance of our models in terms of feature selection. We report the number of features selected in Table 4 by each toolkit for sparse GAMs with interactions. We see that GRAND-SLAMIN with structural constraints, in particular strong hierarchy, can significantly reduce the number of features selected by the GAMs with interactions model. For example, on Bankruptcy datasets, GRAND-SLAMIN achieves feature compression up to a factor of  $8 \times$  over state-of-the-art GAM toolkits. Having fewer features reinforces the usefulness of additive models as being interpretable
Table 4: Number of features used by GRAND-SLAMIN without/with additional structural constraints and competing approaches. Hyphen (-) indicates multiclass classification is not supported by GAMI-Net and SIAN.
Dataset\Model	EB2M	NODE GA2M	GAMI Net	SIAN	GRAND-SLAMIN
None	WH	SH
Magic	10±0	10±0	10±0	10±0	10±0	9±1	7±0
Adult	14±0	14±0	14±1	14±0	13±1	11±1	11±1
Churn	19±0	19±0	18±2	19±0	19±0	11±1	12±2
Satimage	36±0	36±0	-	-	36±0	36±0	22±2
Texture	40±0	40±0	-	-	40±0	37±2	17±2
MiniBooNE	50±0	50±0	16±12	34	50±0	50±0	28±3
Covertype	54±0	54±0	-	-	34±1	54±1	54±0
Spambase	57±0	57±0	52±2	55±1	57±0	56±3	54±2
News	58±0	58±0	47±1	52	58±0	58±0	58±0
Optdigits	64±0	64±0	-	-	64±0	64±0	59±1
Bankruptcy	95±0	95±0	60±15	69	95±0	60±26	7±16
Madelon	500±0	500±0	61±56	490	26±19	19±15	24±9
Activity	533±0	346±6	-	-	182±15	440±22	159±21
Multiple	649±0	649±0	-	-	648±1	629±9	649±0
WH=Weak Hierarchy, SH=Strong Hierarchy.
6.3 Computational Scalability
Next, we discuss the scalability of GRAND-SLAMIN.
Sparse backpropagation. We highlight the usefulness of our efficient approach with sparse backpropagation in Figure 2 on Activity dataset. We show in Figure 2[a] that during the course of training, the number of selected components (selected via binary variables  $z$ 's and  $q(\cdot)$ 's) becomes progressively smaller. This leads to much faster computations at each epoch in Figure 2[b] due to
(a) Number of selected effects at each epoch.
(b) Training time (seconds) for each epoch.
Figure 2: GRAND-SLAMIN with standard (dense) backpropagation vs sparse backpropagation on Activity dataset. (a) shows the number of nonzero effects:  $\sum_{j}z_{j} + \sum_{(j,k)\in \mathcal{I}}q(z_{j},z_{k},z_{j,k})$  and (b) shows the time for each epoch during the course of training.
Figure 3: Estimated main effects in the presence of interaction effects on bikesharing dataset [Left] without hierarchy, [Middle] weak hierarchy and [Right] strong hierarchy. Strong hierarchy has the smallest error bars.
sparse forward and backward passes. By exploiting sparsity during training, we can get  $10 \times$  faster training times than with standard backpropagation.
Comparison with other toolkits. Our toolkit is highly competitive in terms of training times with all existing tree-based and neural-based toolkits for sparse GAMs with interactions. For example, on Madelon dataset, we are  $15 \times$  faster than NODE-  $\mathrm{GA}^2\mathrm{M}$ ,  $20 \times$  faster than EBM,  $1300 \times$  faster than SIAN and  $25 \times$  faster than GAMI-Net. See Supplement Sec. F.7 for more detailed timing comparisons across multiple datasets. Note that, in addition, we can also handle the case of structured interactions — extending the flexibility of existing end-to-end training methods.
6.4 Variance Reduction with Structural Constraints
We provide a visualization study to further highlight an important contribution of our work. In particular, our framework can support models with structural constraints. Hence, we study the effect of these constraints on the stability of learning main effects (in the presence of interactions) when these structural constraints are imposed or not. For this exercise, we consider bikesharing dataset. We visualize some of the main effects in the presence/absence of hierarchy in Figure 3. Note that for visualization, we used the purification strategy [29, 6] post-training that pushes interaction effects into main effects if possible. We can observe in Figure 3 that when additional hierarchy constraints are imposed, the error bars are much more compact across different runs. This can potentially increase the trust you can have on the model for deriving interpretability insights. We show additional visualizations on another dataset (American Community Survey from US Census Planning Database 2022 [55]) to show the same behavior in Supplement Section G.
We introduce GRAND-SLAMIN: a novel and flexible framework for learning sparse GAMs with interactions with additional structural constraints e.g., hierarchy. This is the first approach to do end-to-end training of nonparametric additive models with hierarchically structured sparse interactions. Our formulation uses binary variables to encode combinatorial constraints. For computational reasons, we employ smoothing of the indicator variables for end-to-end optimization with first-order methods (e.g., SGD). We propose sparse backpropagation, which exploits sparsity in the nature of the smoothing function in a GPU-compatible manner and results in  $10 \times$  speedups over standard backpropagation. We present non-asymptotic prediction bounds for our estimators with tree-based shape functions. Numerical experiments on a collection of 16 real-world datasets demonstrate the effectiveness of our toolkit in terms of prediction performance, variable selection and scalability.
This research was supported in part, by grants from the Office of Naval Research (N000142112841), and Liberty Mutual Insurance. The authors acknowledge the MIT SuperCloud [48] for providing HPC resources that have contributed to the research reported within this paper.
[59] Xiaohan Yan and Jacob Bien. Hierarchical sparse modeling: A choice of two group lasso formulations. Statistical Science, 32(4):531-560, November 2017.
[60] Zebin Yang, Aijun Zhang, and A. Sudjianto. Gami-net: An explainable neural network based on generalized additive models with structured interactions. Pattern Recognit., 120:108192, 2021.
[61] Guo Yu, Daniela Witten, and Jacob Bien. Controlling costs: Feature selection on a budget. Stat, 11(1):e427, 2022.
[62] Ming Yuan and Ding-Xuan Zhou. Minimax optimal rates of estimation in high dimensional additive models. The Annals of Statistics, 44(6):2564-2593, 2016.
[63] Tuo Zhao and Han Liu. Sparse additive machine. In Neil D. Lawrence and Mark Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 1435-1443, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.
Supplementary Material for GRAND-SLAMIN' Interpretable Additive Modeling with Structural Constraints
A Definition of Soft Trees
Formally, an interaction soft tree with the set of (internal) nodes  $\mathcal{U}$  and the set of leaves  $\mathcal{L}$  is a function such as  $f(\cdot; \boldsymbol{W}, \boldsymbol{o}) : \mathbb{R}^2 \mapsto \mathbb{R}$  where  $\boldsymbol{W} \in \mathbb{R}^{|\mathcal{U}| \times 2}$  and  $\boldsymbol{o} \in \mathbb{R}^{|\mathcal{L}|}$ . The output is then given as  $f((x_j, x_k); \boldsymbol{W}, \boldsymbol{o}) = \sum_{l \in \mathcal{L}} P(\{(x_j, x_k) \to l\}) o_l$  where  $P(\{(x_j, x_k) \to l\})$  is the proportion of  $(x_j, x_k)$  that is routed to leaf  $l$ . Particularly,  $P(\{(x_j, x_k) \to l\}) = \prod_{i \in \mathcal{A}(l)} r_{i,l}(x_j, x_k)$  where  $\mathcal{A}(l)$  denotes the set of ancestors of  $l$  and  $r_{i,l}(x_j, x_k)$  denotes the proportion of  $(x_j, x_k)$  is routed to leaf  $l$  from node  $i$ . These values are given as  $r_{i,l}(x_j, x_k) = \phi(\boldsymbol{W}_i^T(x_j, x_k))$  if  $l$  belongs to the left subtree of  $i$ , and  $r_{i,l}(x_j, x_k) = 1 - \phi(\boldsymbol{W}_i^T(x_j, x_k))$  if  $l$  belongs to the right subtree of  $i$ , where  $\phi(\cdot)$  is the sigmoid activation function. A soft tree  $f(\cdot; \boldsymbol{W}, \boldsymbol{o}) : \mathbb{R} \mapsto \mathbb{R}$  for a main effect is defined similarly.
B Smooth-Step Function
The smooth-step function [18] has been used in soft trees for conditional computation [18] and routing in sparse mixture of experts [19, 21]. It can output exact zeros and ones, thus allowing for hard selection. Let  $\gamma$  be a non-negative scalar parameter. The smooth-step function takes the form:
$$
h (t) = \left\{ \begin{array}{l l} 0 & \text {i f} t \leq - \gamma / 2 \\ - \frac {2}{\gamma^ {3}} t ^ {3} + \frac {3}{2 \gamma} t + \frac {1}{2} & \text {i f} \gamma / 2 \leq t \leq \gamma / 2 \\ 1 & \text {i f} t \geq \gamma / 2 \end{array} \right. \tag {B.1}
$$
The smooth-step function is continuously differentiable, similar to the logistic function. Additionally, it performs hard selection, i.e., outside  $[- \gamma / 2, \gamma / 2]$ , the function produces exact zeros and ones.
C.1 Preliminaries
Before proceeding with the proof, we define some notation we use throughout this section. Given the data points  $\pmb{x}_1,\dots ,\pmb{x}_n$ , we let  $P_{n}$  the empirical measure supported on  $\pmb{x}_1,\dots ,\pmb{x}_n$ ,  $P_{n} = \sum_{i = 1}^{n}\delta_{\pmb{x}_i} / n$ . We define the norm with respect to  $P_{n}$  as
$$
\| f \| _ {P _ {n}} = \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {x} _ {i}\right) ^ {2}}. \tag {C.1}
$$
Next, we define covering and packing numbers.