GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data
Authors: Sascha Marton
Stefan Lüdtke
Christian Bartelt
Heiner Stuckenschmidt
Venue: ICLR 2024 poster (2024)
Keywords: Tabular Data; Architectures; Ensembles; Decision Trees; Gradient Descent
Primary area: general machine learning (i.e., none of the above)
TL;DR: A novel ensemble method for hard, axis-aligned decision trees learned end-to-end with gradient descent

Abstract: Despite the success of deep learning for text and image data, tree-based ensemble models are still state-of-the-art for machine learning with heterogeneous tabular data. However, there is a significant need for tabular-specific gradient-based methods due to their high flexibility. In this paper, we propose $\text{GRANDE}$, $\text{GRA}$die$\text{N}$t-Based $\text{D}$ecision Tree $\text{E}$nsembles, a novel approach for learning hard, axis-aligned decision tree ensembles using end-to-end gradient descent. GRANDE is based on a dense representation of tree ensembles, which affords to use backpropagation with a straight-through operator to jointly optimize all model parameters. Our method combines axis-aligned splits, which is a useful inductive bias for tabular data, with the flexibility of gradient-based optimization. Furthermore, we introduce an advanced instance-wise weighting that facilitates learning representations for both, simple and complex relations, within a single model. We conducted an extensive evaluation on a predefined benchmark with 19 classification datasets and demonstrate that our method outperforms existing gradient-boosting and deep learning frameworks on most datasets. The method is available under: https://github.com/s-marton/GRANDE

Published as a conference paper at ICLR 2024
GRANDE: GRADIENT-BASED DECISION TREE ENSEMBLES FOR TABULAR DATA
Sascha Marton
Stefan Lüdtke
University of Mannheim, Germany sascha www.rroonnnn.... -mannhei.m.....mmmmmmm..m..... de
University of Rostock, Germany
stefan. luedtke@uni wrwtonckooooocoooooooooo
Christian Bartelt
Heiner Stuckenschmidt
University of Mannheim, Germany christian. www.aaaeeetuuuu tean-eenmeememmmm de
University of Mannheim, Germany
beiner.stuckent @utttunniimmn uniimannheimm.d
ABSTRACT
Despite the success of deep learning for text and image data, tree-based ensemble models still state-of-the-an for machine learning with heterogeneous tabular data. However, there is a significant need for tabular-specific gradient- based meth- ods due to their high flexibility. In this paper, we propose GRANDE, GR Based Decision Tree Ensembles, a novel approach for learning hard, axis-aligned decision tree ensembles using end-to-end gradient descent. GRANDE is based on a dense representation of tree ensembles, which affords to use backpropaga- tion with a straight-through operator to jointly optimize all model parameters. Our method combines axis-aligned splits, which is useful inductive bias for tab- ular data, with the flexibility of gradient-base optimization. Furthermore, we introduce advanced instance-wise weighting that facilitates learning represen- tations for both, simple and complex relations, within single model. We con- ducted an extensive evaluation predefined benchmark with 19 classification datasets and demonstrate that our method outperforms existing condient-boostiing and deep learning frameworks most datasets. The method is available under: btttps://githie com/ -armmmrtoon. GRANDE /RAAND
INTRODUCTION
Heterogeneous tabular data the most frequently used form of data (Chui al. 2018 Shwartz-Ziv & Armon 2022) and is indispensable in wide range of applications such as medical diagnosis (Ul- mer et al. 2020 Somani et al. 202T), estimation of creditworthiness (Clements et al., 2020) and fraud detection Cartella et al. 2021). Therefore, enhancing the predictive performance and robust- ness of models can bring significant advantages to users and companies (Borisov et al., 2022). ). How- ever, tabular data comes with considerable challenges like noise, missing values, class imbalance, and a combination of different feature types, especially categorical and numerical data. Despite the success of deep learning (DL) in various domains, recent studies indicate that tabular data still poses a major challenge and tree-based models like XGBoost and CatBoost outperform them in most cases (Borisov et al. 2022 Grinsztajn al. 2022 Shwartz-Ziv & Armon 2022). At the same time, employing end-to-end gradient- training provides several advantages over traditional machine learning methods (Borisov al., (022). They offer high level of flexibility by allowing an easy integration of arbitrary, differentiable loss functions tailored towards specific problems and support iterative training (Sahoo al., (017). Moreover, gradient-basee methods can be incorporated easily into multimodal learning, with tabular data being of several input types (Lichtenwalter et al. 2021 Pölsterl et al.) 2021). Therefore, creating -peecff, gradient-based methods is a very active field of research and the need for well- well-performing methods is intense (Grinsztajn al. 2022.
Recently, Marton et al. 2023) introduced GradTree. a novel approach that uses gradient descent to leam hard, axis skis-aligned decision trees (DTs). This is achieved by reformulating DTs to dense rep- resentation and jointly optimizing all tree parameters using backpropagation with a straight- through (ST) operator. Learning hard, axis-aligned DT's with gradient descent allows combining the advan-

---

Published as a conference paper at ICLR 2024
tageous inductive bias of tree-based methods with the flexibility of gradient-basee based optimization. In this paper, we propose GRANDE, GRAdieNt-Bal Based Decision Tree Ensembles, a novel approach for learning decision tree ensembles using end-to-end gradient descent. Similar to Marronn et al. (2023), we use a dense representation for split nodes and the ST operator to deal with the non-differentiable nature of DTs.. We build upon their approach, transitioning from individual trees weighted tree ensemble, while maintaining an efficient computation. As a result, GRANDE holds a significant advantage over existing gradient-basee based methods. Typically, DL methods are biased towards smooth solutions (Rahaman et al. (019). As the target function in tabular datasets is usually not smooth, DL methods struggle find these irregular functions. In contrast, models that are based on hard, axis aligned DTs learn piece-wise constant functions and therefore do not show such bias (Grinsz-) tajn et al., 2022). This important advantage is one inherent aspect of GRANDE, as it utilizes hard, axis-aligned D'Tss This is major difference to existing DI. methods for hierarchical representations like NODE, where soft and oblique splits are used (Popov et al. 2019). Furthermore, we introduce instance-wiss weighting in GRANDE. This allows learning approprite representations for simple and complex rules within a single model, which increases the performance of the ensemble. Further- more, we show that our instance-we wise weighting has positive impact the local interpretability relative to other state-of-the-ar methods. More specifically, our contributions are as follows:
We extend GradTree (Marton et al. 2023) from individual trees to end-to-end gradient- based tree ensemble, maintaining efficient computation (Section 3.1
We introduce softsign differentiable split function and show the advantage over com- monly used alternatives (Section 3.2).
We propose a novel weighting technique that emphasizes instance-wise estimator impor- tance (Section 3.3
We conduct an extensive evaluation on 19 binary classification tasks (Section 4 based on the prede- fined tabular benchmark proposed by Bischl et al. (2021). GRANDE outperforms existing methods for both, default and optimized hyperparameters. The performance difference other methods is substantial on several datasets, making GRANDE an important extension to the existing repertoire of tabular data methods.
BACKGROUND: GRADIENT-BASED DECISION TREES
GRANDE builds on gradient-based decision trees (GradTree) the level of individual trees in the ensemble. Hence, we summarize the relevant aspects and notation of GradTree in this section and refer to Marton et al. (2023) for complete overview.
Traditionally, DTs involve nested concatenation of rules. In GradTree, DTs are formulated as arith- metic functions based addition and multiplication to facilitate gradient-based learning. Thereby both, GradTree and GRANDE focus on learning fully-grown (i.e., complete, full) DT's which can be pruned post-hoc. A DT of depth d is formulated with respect to its parameters as:
24_1
(c Σ du (x+l,,,,))
where L is a function that indicates whether a sample E R² belongs to leaf 1, X E C²d denotes class membership for each leaf node, + RR2--1 represents split thresholds and N2 the feature index for each internal node.
To support gradient-based optimization and ensure an efficient computation via matrix operations, a novel dense DT representation is introduced in GradTree. Traditionally, the feature index vector e is one-dimensional, but GradTree expands it into matrix form. Specifically, this representation one-hot encodes the feature index, converting E 2²--1 into a matrix I E 1 Similarly, for split thresholds, instead of a single value for all features, individual values for cach feature stored, leading to a matrix representation T (222-1)x0 By enumerating the internal nodes in breadth- first order, we can redefine the indicator function IL for a leaf l, resulting in
2d
g(*|A.T.1)
L I.I)
2

---

Published as a conference paper at ICLR 2024
where L(
II p(l.j)) (( T(as)) + p(l,) -- (0.,) Ta.s))
(3)
==1
Here, is the index of the internal node preceding leaf node I at certain depth j and p indicates whether the left (p = 0) or the right branch (p 1) was taken.
Typically, DT's the Heaviside step function for splitting, which is non-differentiable. GradTree reformulates the split function to account for reasonable gradients:
S(zlle, ( ---
Where S(2) Itce represents the logistic function, [=] stands for rounding a real number to the nearest integer and b denotes the dot product between two vectors and b. We further need to ensure that & a one-hot encoded vector to account for axis-aligned splits. This is achieved by applying hardmax transformation before calculating S. Both rounding and hardmax operations non-differentiable. To overcome this, GradTreee employs the straight- through (ST) operator during backpropagation. This allows the model use non-differentiable operations in the forward pass while ensuring gradient propagation in the backward pass.
3 GRANDE: GRADIENT-BASER DECISION TREE ENSEMBLES
One core contribution of this paper is the extension of GradTree to tree ensembles (Section 3.1 In Section 3.2 we propose softsign differentiable split function to propagate more reasonable gradients. Furthermore, we introduce an instance-wist weighting in Section 3.3 and regularization techniques Section 3.4 As result, GRANDE can be learned end-to-end with gradient descent, leveraging the potential and flexibility of gradient-based optimization.
3.1 FROM DECISION TREES WEIGHTED TREE ENSEMBLES
One advantage of GRANDE over existing gradient-based methods is the inductive bias of axis- aligned splits for tabular data. Combining this property with an end-to-end gradient-based opti- mization is at the core of GRANDE. This is also major difference to existing DL methods for hierarchical representations like NODE, where soft, oblique splits are used (Popov al.) 2019). Therefore, we can define GRANDE as
E G(ælw,L L, Σ we g( Le, Te,le)
(5)
where E' is the number of estimators in the ensemble and w is weight vector. By extending L to matrix and T. to tensors for the complete ensemble instead of defining them individually for each tree, we can leverage parallel computation for an efficient training.
As GRANDE can be learned end-to-end with gradient descent, we keep an important advantage over existing, non son-gradientt-een induction of the individual trees and the sequential combination of individual trees via boosting are greedy. This results in constraints on the search space and can favor overfitting, highlighted by Marton et al. (2023). In contrast, GRANDE learns parameters of the ensemble jointly and overcomes these limitations.
tree methods like XGBoost and CatBoost. Both, the sequential
3.2 DIFFERENTIARIA SPLIT FUNCTIONS
The Heaviside step function, which is commonly used as split function in DTs, is non-differentiable. To address this challenge, various studies have proposed the employment of differentiable split func- tions. A predominant approach is the adoption of the sigmoid function, which facilitates soft deci- sions [Jordan & Jacobs 1994 Irsoy et al. 2012: Frosst Hinton 2017). A more recent development in this field originated with the introduction of the entmax trans transformation (Peters et al. 2019). Re- searchers utilized a two-class entmax (entmoid) function to turn the decisions more sparse (Popov et al. 2019). Further, Chang et al. (2021) proposed a temperature annealing procedure to gradually turn the decisions hard. Marton et al. (2023) introduced an alternative method for generating hard splits by using straight-through (ST) operator after sigmoid split function generate hard splits.
3

---

Published as a conference paper at ICLR 2024
Sprel
Maxe bollig
(a) Sigmoid
(b) Entmoid
(c) Adjusted Softsign
Figure 1: Differentiable Split Functions. The sigmoid gradient declines smoothly, while entmoid's gradient decays more rapidly but becomes zero for large values. The scaled softsign has high gradi- ents for small values but maintains a responsive gradient for large values, offering greater sensitivity.
While this allows using hard splits for calculating the function values, it also introduces mismatch between the forward and backward pass. However, we can utilize this to incorporate additional information: By using sigmoid function, the distance between a feature value and the threshold is used as additional information during gradient computation. Accordingly, the gradient behavior plays a pivotal role in ensuring effective differentiation, especially in scenarios where input values close to the decision threshold. The traditional sigmoid function be suboptimal due to its smooth gradient decline. Entmoid, although addressing certain limitations of sigmoid, still displays undesirable gradient behavior. Specifically, its gradient drops zero when the difference in val- ues is too pronounced. This can hinder the model's ability to accommodate samples that exhibit substantial variances from the threshold. Therefore, we propose using a softsign function, scaled to (0, 1), as differentiable split function:
Sx(z)) =
++ +1
2
(6)
The distinct gradient characteristics of the softsign, which are pronounced if samples are close to the threshold, reduce sharply but maintain responsive gradients there is a large difference between the feature value and the threshold. These characteristics make it superior for differentiable splitting. This concept is visualized in Figure []] Besides the intuitive advantage of using a softsign split function, also show empirically that this is the superior choice (Table 4
3.3 INSTANCE- WISE ESTIMATOR WEIGHTS
One challenge of ensemble methods is learning good weighting scheme of the individual estima- tors. The flexibility of an end-to-end gradient-basse optimization allows including learnable weight parameters to the optimization. A simple solution would be learning one weight for each estimator and using softmax over all weights, resulting in weighted average. However, this forces a very homogeneous ensemble, in which each tree aims to make equally good predictions for all samples. In contrast, would be beneficial individual trees can account for different areas of the target function, and are not required to make confident predictions for each sample.
To address this, we propose an advanced weighting scheme that allows calculating instance wise weights that can be learned within the gradient- baaed optimization. Instead of using one weight per estimator, we use one weight for each leaf of the estimator as visualized in Figure Q and thus define the weights as W E Ex2" instead of w E E. We define p( | T, I) TR - TR as a function calculate vector comprising the individual prediction of each tree. Further, we define a function w(x W, L. I) R. + R to calculate a weight vector with one weight for each tree based on the leaf which the current sample is assigned to. Subsequently, a softmax applied on these chosen weights for each sample. The process of multiplying the post-softmax weights by the predicted values from each tree equates to computing a weighted average. This results in
C(æ W. L. T, = a (w(x) W, L. I.) p[ T.I1)
Σ
Wo, Lox To,
g( Lo. To Io) g( L1, h1)
Wx h
where w(x|W, L, T,I)
p(a L, T,I)
Σ 3d Wel le)
[xx Te, le)
4

---

Published as a conference paper at ICLR 2024
>
ALI
ML.
412
2,3
414
12.1
A22
003
onn
Lu
dot product
sigmoid function
softmax function
Figure 2: GRANDE Architecture. This figure visualizes the structure and weighting of GRANDE for exemplary ensemble with two trees of depth two. For each tree in the ensemble, and for every sample, we determine the weight of the leaf which the sample is assigned to.
and ar (2) is the softmax function. is important to note that when calculating L (see Equation 3 only the value for the leaf to which the sample is assigned in given tree is non-zero. We want to note that our weighting scheme permits calculating instance-wise weights even for unseen samples.
Our weighting scheme, in addition to its instance-wis nature, is substantially different to existing tree ensemble methods and post-hoc weighting schemes (He et al. 2014 M Cui et al. 2023), as is in- corporated into the training procedure which is necessary capture local interactions. Furthermore, the predictions of individual trees are not separate and changes in the mastance-wise weights one estimator directly impacts the weight of the remaining estimators. In evaluation, we demonstrate that instance-win wise weights significantly enhance the performance of GRANDE and emphasize local interpretability by learning representations for simple and complex rules within model.
3.4 REGULARIZATION: FEATURE SUBSET, DATA SUBSET AND DROPOUT
The combination of tree-based methods with gradient-basse based optimization opens the door for the application of numerous regularization techniques. For each tree in the ensemble, we select a feature subset. Therefore,
regularize model and simultaneously, of Grad'Tree with an increasing number of features. Similarly, we select a subset of the samples for each estimator. Furthermore, implemented dropout by randomly deactivating predefined fraction of the estimators in the ensemble and rescaling the weights accordingly.
solve the poor scalability
4 EXPERIMENTAL EVALUATION
As pointed out by Grinsztajn et al. (2022). most papers presenting new method for tabular data have a highly varying evaluation methodology, with small number of datasets that might be bi- ased towards the authors' model. As result, recent surveys showed that tree boosting methods like XGBoost and CatBoost are still state-of-the-arr and outperform new architectures
tabular data most datasets (Grinsztajn et al.) 2022 Shwartz-Ziv & Armon [022] Borisov et al.) (022).. This highlights the necessity for an extensive and unbiased evaluation, as we will carry out in the following, to accurately assess the performance of new method and draw valid conclusions. We want to emphasize that recent surveys and evaluation on predefined benchmarks indicate that there no mne-size-fits-all" solution for all tabular datasets. Consequently. we should view methods as an extension to the existing repertoire and set our expectations in line with this perspective.
4.1 EXPERIMENTAL SETUP
Datasets and Preprocessing For our evaluation, we used a predefined collection of datasets that was selected based objective criteria from OpenML Benchmark Suites and comprises total of 19 binary classification datasets (see Table [5] details). The selection process was adopted from Bischl al. (202T) and therefore is not biased towards method. A more detailed discussion
5

---

Published as a conference paper at ICLR 2024
Table 2: Performance Comparison. We report the test macro Fi-score (mean = stdev for a 5-fold CV) with optimized parameters. The datasets are sorted based on the data size.
GRANDE
XGB
CatBoost
NODE
dresses-sales
0,612 1 0.049 (1)
0.581 1 0.059 (3)
0,588 = 0.036 (2)
0,564 1 0.051 (4)
dimaae-simulationccaaall
0,853 0.070 (1)
0.763 + 0.064 (4)
0.778 0.050 (3)
0.802 + 0.035 (2)
cylinder-bands
0.819 = 0.032 (1)
0.773 = 0.042 00002 (3)
0.801 = -0.043 (2)
0.754 = 0.040 (4)
wdbe
0.975 0.010 (1)
0.953 0.030
0.963 0.023
0,966 0.016
ilpd
0.657 0.042 (I)
0.632 (3)
0.643 0.053 (2)
0.526 0.069 (4)
tokyol
0.921 + 0.004 (3)
0,915 +0011 (4)
0,927 0.013
0.921 + 0.010 (2)
gsar-biodeg
0.854 = 0.022 (1)
0,853 = 0,020 (2)
0,844 0.023 (3)
0.836 = 0.028 (4)
ozone-level-8hr
0.726 1 0.020 (1)
0,688 1 0.021 (4)
0,721 0.027 (2)
0.703 1 0.029 (3)
madelon
0.803 + 0.010 (3)
0.833 + 0.018 (2)
0.861 0.012 (1)
0.571 + 0.022 (4)
Bioresponse
0.794 = 0.008 (3)
0.799 = 0.011 (2)
0.801 0.014 (1)
0,780 = 0.011 (4)
wilt
0,936 1 0.015 (2)
0.911 1 0.010 (4)
0.919 0.007 (3)
0.937 1 0.017 (1)
churn
0,914 = 0.017 (2)
0,900 + 0.017 (3)
0.869 0.021 (4)
0.930 = 0.011 (I)
phoneme
0.846 = 0.008 (4)
0.722 + 0.007 (2)
0.876 0.005 (1)
0.862 = 0.013 (3)
SpeedDuting
0,723 = 0.013 (I)
0,704 0,015 (4)
0,718 0.014 (2)
0.707 0.015
PhishingWeesstes
Websites
0,969 1 0.006 (1)
0,968 0,006 (2)
0,965 0.003 (4)
0.968 0.006 (3)
amnazon_employee.cm
0.665 = 0.009 (2)
0.621 = 0.008 (4)
0,671 0.011 (1)
0.649 = 0.009 (3)
nomao
0.958 = 0.002 (3)
0,965 = 0,003 (1)
0.964 0.002 (2)
0.956 = 0.001 (4)
adult
0.790 + 0.006 (4)
0.798 1 0.004 (1)
0.796
0.004 (2)
0.794 + 0.004 (3)
numerai28.6
0,519 = 0.003 (1)
0.518 = 0.001 (3)
0.519 0.002 (2)
0.503 = 0.010 (4)
Normalized Mean f
0.776 (1)
0.483 (3)
0.671 (2)
0.327 (4)
Mean Reciprocal Rank (MRR)
0.702 (1)
0.417 (3)
0.570 (2)
0.395 (4)
the selection of the benchmark can be found in Appendix A We one-hot encoded low-cardinality categorical features and used leave-one-out encoding for high-cardinality categorical features (more than 10 categories). To make them suitable for a gradient- -basee optimization, we gaussianized features using quantile transformation, as is common practice (Grinsztajn al. 2022). In line with Borisov et al. (2022), we report the mean and standard deviation of the test performance over a 5-fold cross-validation ensure reliable results.
Methods We compare our approach to XGBoost and CatBoost, which achieved superior results according to recent stud- ies, and NODE, which is most related to our approach. With this setup, we have one state-of-the-art tree-based and one gradient-based based approach for each tree type (see Table DT. In addition, provide extended evaluation including SAINT, RandomForest and ExtraTrec as additional benchmarks in Appendix These additional results are in line with the results presented in the following.
Table I: Categorization of Approaches
Standard DT's
Oblivious DT's
Tree-based
XGBoost
CatBoost
Gradient-based
GRANDE
NODE
Hyperparameters We optimized the hyperparameters using Optuna (Akiba et al. 2019) with 250 trials and selected the search space well as the default parameters for related work in accordance with Borisov et al. (2022). The best parameters were selected based on 5x2 cross-vas validation as sug- gested by Raschha (2018) where the test data of each fold was held out of the HPO to get unbiased results. To deal with class imbalance, we further included class weights. Additional information along with the hyperparameters for each approach are in Appendix E
4.2 RESULTS
GRANDE outperforms existing methods on most datasets We evaluated the performance with optimized hyperparameters based on the macro F1-Score in Table account for class imbalance. Additionally, we report the accuracy and ROC-AUC score in the Appendix B, which are consistent with the results presented in the following. GRANDE outperformed existing methods and achieved the highest mean reciprocal rank (MRR) of 0.702 and the highest normalized mean of 0.776. Cat- Boost yielded the second-bes results (MRR of 0.570 and normalized mean of 0.671) followed by XGBoost (MRR of 0.417 and normalized mean of 0.483) and NODE (MRR of 0.395 and normal- ized mean of 0.327). Yet. our findings are in line with existing work, indicating that there is no universal method for tabular data. However, on several datasets such climate-simulation simusetion-crushes

---

Published as a conference paper at ICLR 2024
Table 4: Ablation Study Summary. Left: Comparison of different options for differentiable split functions (complete results in Table TO). Right: Comparison of our instance- wise weighting based on leaf weights with a single weight for each estimator (complete results in Table
Differentiable Split Function
Weighting Technique
Softsign
Entmoid
Sigmoid
Leaf Weights
Estimator Weights
Normalized Mean f
0.7906 (1)
0.4188 (2)
0.2207 (3)
0.8235 (1)
0.1765 (2)
Mean Reciprocal Rank (MRR)
0.8246 (1)
0.5526 (2)
0.4561 (3)
0.9211 (1)
0,5789 (2)
and cylinder-bands the performance difference other methods was substantial, which highlights the importance of GRANDE as an extension to the existing repertoire. Furthermore, as the datasets are sorted by their size, we can observe that the results of GRANDE are especially good for small datasets, which is an interesting research direction for future work.
GRANDE efficient for large and high-dimensional datasets GRANDE averaged 47 seconds across all datasets, with maximum runtime of 107 seconds. Thereby, the runtime of GRANDE is robust to high-dimensional (37 seconds for Bioresponse with 1,776 features) and larger datasets (39 seconds for numerai28.6 with 96,320 samples). GRANDE achieved significantly lower runtime compared to our gradient-based benchmark NODE, which has an approximately three times higher average runtime of 130 seconds. However, it is important note that GBDT frameworks, especially XGBoost, are highly efficient when executed on the GPU and achieve significantly lower runtimes compared to gradient-based methods. The complete runtimes are listed in the appendix (Table 9
GRANDE outperforms existing methods with default hyperparameters Many meth- ods, especially DL methods, are heavily reliant on proper hyperparameter optimization. Yet, it is desirable property that method achieves good results even with their default setting. GRANDE achieves superior results with de- fault hyperparameters, and significantly outper- forms existing methods on most datasets. More specifically, GRANDE has the highest normal- ized mean performance (0.6371) and the high- est MRR (0.6404) as summarized in Table
Table 3: Default Hyperparameter Performance Summary. The results are based on the test macro F1-score with the default setting. Complete sults are listed in Table 8
Normalized
Mean Reciprocal
Mean f
Rank (MRR) t
GRANDE
0.6371 (1)
0.6404 (1)
XGB
0.5865 (2)
0.5175 (3)
CatBoost
0.5793 (3)
0.5219 (2)
NODE
0.2698 (4)
0.4035 (4)
argue that employing softsign as split index activation propagates informative gradients beneficial for the optimization. In Table 4 support these claims by showing superior performance of GRANDE with softsign activation (before discretizing with the ST operator) compared to sigmoid as the default choice as well as an entmoid function which is commonly used in related work (Popov et al. 2019: Chang et al. 2021).
Softsign improves performance As discussed in Section
3
Instance wise weighting increases model performance GRANDE uses instance-wise weighting assign varying weights estimators for each sample based on selected leaves. This promotes ensemble diversity and encourages estimators to capture unique local interactions. We argue that the ability learn and represent simple, local rules with individual estimators our ensemble can have a positive impact on the overall performance as it simplifies the task that has to be solved by the remaining estimators. As result, GRANDE can efficiently learn compact representations for simple rules, where complex models usually tend to learn overly complex representations.
4.3 CASE STUDY: INSTANCE- WISE WEIGHTING FOR THE PHISHINGWEBSTIE DATASET
In the following case study, we demonstrate the ability of GRANDE learn compact representations for simple rules within a complex ensemble: The Phishing Wehsites dataset is concerned with iden- tifying malicious websites based on metadata and additional observable characteristics, Although the task is challenging (i.e., it is not possible to solve it sufficiently well with a simple model, as shown in Table 12), there exist several clear indicators for phishing websites. Thus, some instances can be categorized using simple rules, while assigning other instances is more difficult. Ideally, if an instance can be easily categorized, the model should follow simple rules to make a prediction.
7

---

Published as a conference paper at ICLR 2024
GRANDE Precision: 1.00
Coverag:::1 0:13
Prefis Suffii
0.5
Plishing
XGBoost Covenge:: 0.0
UfL .Is
SasingSuh, Domain_3 2
tel_traffic_2
Phishing
CatBoost Procision: 1.00
Prefis, Suffiie
of Anehor .12 0.5
having Sul, Damain
Litks tage 0.5
Dunsin ration,legth $0.5
age of domain n.s
Phishing
Coverage: 002
0.5
Coverage: 0.02
Precision:
2
.t's 0.
in
Leiggeg 2
Phishing
Figure 4: Anchors Explanations. This figure shows the local explanations generated by Anchors for the given instance. The explanation for GRANDE only comprises single rule. In contrast, the corresponding explanations for the other methods have significantly higher complexity, which indicates that these methods are not able learn simple representations within complex model.
One example for rule, which holds universally in the given dataset, is that an instance can be classi- fied as phishing if prefix suffix added the domain name. By assessing the weights for an exemplary instance fulfilling this rule, can ob- serve that the DT visualized in 3 accounts for 94% of the prediction. Accordingly, GRANDE has learned very simple representation and the classi- fication is derived by applying an easily comprehen- sible rule. Notably, for the other methods, is not possible to assess the importance of individual esti- mators out-of-the-box similarly, the prediction is either derived by either sequentially summing up the predictions (e.g. XGBoost and CatBoost) or equally weighting all estimators. Furthermore, this has sig- nificant positive impact on the average performance of GRANDE compared to using one single weight for each estimator (see T)ble
Prefix Suffix 0.5
SSLiina 2
= 0199 1145
0372 -5144
0.7% 451
Estimator Index: 252
Normalized Instance-Wia Estimatto Weight: 0.94
Figure
Highest-Weighted Estimator. This figure visualizes the DT from GRANDE (1024 total estimators) which has the highest weight for exemplary instance.
Instance-wise weighting can be beneficial for local interpretability In addition to the perfor- mance increase, our instance-wit wise weighting has notable impact on the local interpretability of GRANDE. For each instance, we can assess the weights of individual estimators and inspect the estimators with the highest importance to understand which rules have the greatest impact on the prediction. For the given example, we only need observe a single tree of depth two (Figure 3 to understand why the given instance was classified as phishing, even though the complete model is very complex. In contrast, existing ensemble methods require a global interpretation of the model and do not provide simple, local explanations
However, similar explanations can be extracted using Anchors (Ribeiro et al. 2018). Anchors, as an extension to LIME (Ribeiro et al.) 2016), provides model-agnostic explanations by identifying conditions (called "ancchos"") which, when satisfied, guarantee a certain prediction with high prob- interpretable, rules-based conditions derived from
ability (noted precision). These anchors input features that consistently lead to the same model prediction. Figure 4 shows the extracted rules for each approach. We clearly see that the anchor extracted for GRANDE matches the rule we have identified based on the instance-wise weights in Figure 3 Furthermore, it is evident that the prediction derived by GRANDE is much simpler compared any other approach, as it only comprises a single rule. Notably, this comes without suffering a loss in the precision, which is 1.00 for all methods. Furthermore, the rule learned by GRANDE has a significantly higher coverage, which means that the rule applied by GRANDE is more broadly representative. The corresponding experiment with additional details can be found in the supplementary material, and a more detailed discussion of the weighting statistics is included in Appendix D
RELATED WORK
Tabular data the most frequently used type of data, and learning methods for tabular data are a field of very active research. Existing work can be divided into tree-based, DL and hybrid methods.
8

---

Published as a conference paper at ICLR 2024
In the following, categorize the most prominent methods based on these three categories and dif- ferentiate our approach from existing work. For a more comprehensive review, we refer to Borisov et al. ).22), Shwartz-Ziv & Armon 2022) and Grinsztaij et al. (2022).
Tree-Based Methods Tree-based methods have been widely used for tabular data due to their interpretability and ability to capture non-linear relationships. While individual trees usually of- fer a higher interpretabillty,, tree ensemble methods [Breiman| (2001): /eeurt et al. (2006), most notably gradient-boosted DTs (GBDT) are commonly used achieve superior performance [Fried- man 200T). The most prominent GBDT methods for tabular data improve the gradient boosting algorithm by for instance introducing advanced regularization (XGBoost (Chen & Guestrin 2016)) )
a special handling for categorical variables (CatBoost (Prokhorenkova et al. (018)) or a leaf-wise growth strategy (LightGBM (Ke et al. 2017)) Regarding the structure, GRANDE is similar to ex- isting tree-based models. The main difference is the end-to-end gradient-based training procedure, which offers additional flexibility, and the instance-wise weighting.
Deep Learning Methods With the success of DL in various domains, researchers have started to adjust DL architectures, mostly transformers, to tabular data (Gorishniy et al. 2021 Arik & Pfister 2021 Huang et al. 2020; Cai et al.] 202 Kossen et al., (021). According to recent studies, Self- Attention and Intersample Attention Transformer (SAINT) is the superior DL method for tabular data using attention over both, rows and columns (Somepalli et al. (02I). Although GRANDE, similar to DL methods, uses gradient descent for training, it has a shallow, hierarchical structure comprising hard, axis-aligned splits.
Hybrid Methods
Hybrid methods aim to combine the strengths of a gradient- optimization
with other algorithms, most commonly tree-based methods (Yang et al., 2018 Abutbul al. 2020
Hehn et al. 2020: Chen 2020; Ke et al. 2019: 2018 Katzir et al. 2020 One prominent way to achieve this using soft DTs to apply gradient descent by replacing hard decisions with soft ones, and uxis-aligned with oblique splits (Frosst & Hinton 2017, Kontschieder et al. 2015 Luo et al. 202T Hazimeh et al. 2020, Yu et al. [202]]. Neural Oblivious Decision Ensembles (NODE) is one prominent hybrid method which learns ensembles of oblivious DT's with gradient descent and therefore closely related to our work (Popov et al. 2019). Oblivious DT's use the same splitting feature and threshold for each internal node at the same depth, which allows an efficient, parallel computation and makes them suitable as weak learners. In contrast, GRANDE uses standard DTs as weak learners. GRANDE to existing methods is the use of hard, axis-aligned splits, which prevents overly smooth solution typically inherent in soft, oblique trees.
also be categorized as hybrid method. The main difference
While some works demonstrate strong results of DL. methods (adra et al. 2021), recent studies indicate that, despite huge effort in finding nigh-performant DL methods, tree-based models still outperform DL for tabular data (Grinsztajn et al. [022] Borisov et al. 2022 Shwartz-Ziv & Armon 2022), even though the gap is diminishing (McElfresh et al. (023). One main reason for the superior performance of tree-based methods lies in the use of axis-aligned splits that are not biased towards overly smooth solutions (Grinsztajn et al., 2022). Therefore. GRANDE aligns with this argument and uses hard, axis- axis-aligned splits combined with the flexibility of a gradient- based optimization.
CONCLUSION AND FUTURE WORK
In this paper, we introduced GRANDE, new method for learning hard, axis-aligned tree ensembles with gradient-descene descent. GRANDE combines the advantageous inductive bias of axis-aligned splits with the flexibility offered by gradient descent optimization. In an extensive evaluation on prede- fined benchmark, we demonstrated that GRANDE achieved superior results. Both with optimized and default parameters, it outperformed existing state-of-the-ee methods on most datasets. Further- more, we showed that the instance- wise weighting of GRANDE emphasizes learning representations for simple and complex relations within a single model, which increases the local interpretability compared to existing methods.
Currently, the proposed architecture is a shallow ensemble and already achieves state-of-the-an art performance. However, the flexibility of a gradient-based based optimization holds potential e.g., by in- cluding categorical embeddings, stacking of tree layers and an incorporation of tree layers to DL frameworks, which is subject to future work.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported in part by the German Federal Ministry for Economic Affairs and Climate Action of Germany (BMWK), and in part by the German Federal Ministry for Environment, Nature Conservation and Nuclear Safety (BMUV).
REFERENCES
Ami Abutbul, Gal Elidan, Liran Katzir, and Ran El-Yaniv, Dnf-net: neural architecture for tabular data. arXiv preprint arXiv:2006. 06465, 2020.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631, 2019.
Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6679-6687, 2021.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang. Rafael Gomes Mantovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Net- works and Learning Systems, 2022.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Shaofeng Cai, Kaiping Zheng, Gang Chen, HV Jagadish, Beng Chin Ooi, and Meihui Zhang, Arm- net: Adaptive relation modeling network for structured data. In Proceedings of the 2021 Interna- tional Conference on Management of Data, pp. 207-220, 2021.
Francesco Cartella, Orlando Anunciacao, Yuki Funabiki, Daisuke Yamaguchi, Toru Akishita, and Olivier Elshocht. Adversarial attacks for tabular data: Application to fraud detection and imbal- anced data. arXiv preprint arXiv: 1xXi:22010000000 2021.
Chun-Hao Chang. Rich Caruana, and Anna Goldenberg. Node-gam: model for interpretable deep learning. In International Conference on Leurning Representations, 2021.
Neural generalized additive
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Yingshi Chen. Attention augmented differentiable forest for tabular data. arXiv preprint arXiv22010.09221, 2020.
Michael Chui, James Manyika, Mehdi Miremadi, Nicolaus Henke, Rita Chung, Picter Nel, and Sankalp Malhotra. Notes from the ai frontier: Insights from hundreds of use cases. McKinsey Global Institute, 2, 2018.
Jillian M Clements, Di Xu. Nooshin Yousefi, and Dmitry Efimov. Sequential deep learning for credit risk monitoring with tabular financial data. arXiv preprint 20t.:2222..533300 2020.
Shijie Cui, Agus Sudjianto, Aijun Zhang, and Runze Li. Enhancing robustness of madienntbbossted decision trees through one-hot encoding and regularization. arXiv preprint arXiv:2304.13761, 2023.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into soft decision tree. arXiv preprint arXiv: /71 1,99844 2017.
10

---

Published as a conference paper at ICLR 2024
Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine learning, 13:3-42,4 2006.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances Neural Information Processing Systems, 34:1891 1441199321 2021.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outper- form deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pp. 4138-4148. PMLR, 2020.
Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the eighth international workshop data mining for online advertising, pp. 1-9, 2014.
Thomas M Hehn, Julian FP Kooij, and Fred A Hamprecht. End-to-end learning of decision trees and forests. International Journal of Computer Vision, 128(4):997-1011, 2020.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
Ozan Irsoy, Olcay Taner Yıldız, and Ethem Alpaydın. Soft decision trees. In Proceedings of the 21 international conference pattern recognition (ICPR2012), pp. 1819-1822. IEEE, 2012.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint 10X 2018.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214 1994.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928-2333 2021.
Liran Katzir, Gal Flidan, and Ran EI- Yaniv. Net-dnf: Effective deep modeling of tabular data, In International conference on learning representationss 2020.
Guolin Ke. Qi Meng. Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tic- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie-Yan Liu. Tabnn: A universal neural network solution for tabular data. openreview preprint, 2018.
Guolin Ke, Zhenhui Xu, Zhang, Jiang Bian, and Tie-Yan Liu. Deepgbm: A deep learning framework distilled by gbdt for online prediction tasks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 384-394, 2019.
Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE international conference on computer vision, pp- 1467-1475, 2015.
Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning. Ad- vances Neural Information Processing Systems, 34:28742-28756. 2021.
David Lichtenwalter, Peter Burggräf, Johannes Wagner, and Tim Weißer. Deep multimodal learning for manufacturing problem solving. Procedia CIRP, 99:615-620, 2021.

---

Published as a conference paper at ICLR 2024
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.088 033980, 2016.
Haoran Luo, Fan Cheng, Heng Yu, and Yuqi Yi. Sdtr: Soft decision tree regressor for tabular data. IEEE Access, 9:55999-56011 2021.
Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. Gradtree: Learning axis-aligned decision trees with gradient descent. In NeurIPS 2023 Second Table Representation Learning Workshop, 2023.
Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrish- nan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabu- lar data? In Thirty-sevnnth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
CiVdXey44
Ben Peters, Vlad Niculae, and André FT Martins. Sparse sequence-to-ssequence models. arXiv preprint 100700 2019.
Sebastian Pölsterl, Tom Nuno Wolf, and Christian Wachinger. Combining 3d image and tabular data via the dynamic affine feature map transform. In Medical Image Computing and Com- puter Assisted Enervention-MIICAL 2021: 24th International Conference, Strasbourg, France, September 27-October 2021, Proceedings, Part 24, pp. 388.698. Springer, 2021.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv: /909,06399,, 2019.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Sebastian Raschka. Model evaluation, model selection, and algorithm selection machine learning. arXiv preprint arXiv: 1111. 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Fruncisco, CA, USA, August 13-17, 2016, pp. 1135-1144 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In Proceedings the AAAI conference artificial intelligence, volume 32, 2018.
Doyen Sahoo, Quang Pham, Jing Lu, and Steven CH Hoi. Online deep learning: Learning deep neural networks the fly. arXiv preprint arXhv:1711.03705. 2017.
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84 90, 2022.
Sulaiman Somani, Adam J Russak, Felix Richter, Shan Zhao, Akhil Vaid, Fayzan Chaudhry, Jes- sica K De Freitas, Nidhi Naik, Riccardo Miotto, Girish N Nadkami. al. Deep learning and the clectrocardiogram: review of the current sate-off-he-ane EP Europace, 23(8):1179-199) 2021.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint 106.v:216600133 2021.
12

---

Published as a conference paper at ICLR 2024
Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data. In Machine Learning for Health, pp. 341- 354. PMLR, 2020.
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. arXiv preprint arXiv:/806.066 06988, 2018.
Gyeong-In Yu, Saeed Amizadeh, Sehoon Kim, Artidoro Pagnoni, Ce Zhang, Byung-Gon Chun, Markus Weimer, and Matteo Interlandi. Windtumnel: towards differentiable ml pipelines beyond single model. Proceedings of the VLDB Endowment, 15(1):11-20, 2021.
A BENCHMARK DATASET SELCTION
We decided against using the original CC18 benchmark, as the number datasets (72) is extremely high and reported by Grinsztajn et al. (2022), the selection process not strict enough, as there are many simple datasets contained. Similarly, we decided not to use the tabular benchmark presented by Grinsztajj et al. (2022), the datasets
adjusted be extremely homogenous by removing all side-aspects (e.g. class imbalance, high--limessinnaityy dataset size). We argue that this also removes some main challenges when dealing with tabular data. As result, decided to use the benchmark proposed by [Bischl et al. (2021 which has more strict selection process than CC18. The benchmark originally includes both, binary and multi-class tasks. For this paper, due to the limited scope, we focused only on binary classification tasks. Yet, our benchmark has large overlap with CC18 as 16/19 datasets are also contained in CC18. The overlap with Grinsztajn et al. (2022) in contrast is rather small. This is mainly caused by the fact that most datasets in their tabular benchmark are binarized versions of multi-class or regression datasets, which was not allowed during the selection of our benchmark. Table lists the used datasets, along with relevant statistics and the source based on the OpenML-II.
Table 5: Datasets
Categorical Features
Features
Minority Class
Samples
Features
OpenML ID
(preprocessed)
dresses-sales
500
42.00%
23381
slimatesiimlation-eraaeee
540
8.52%
40994
cylinder-bands
540
42.22%
6332
wdbe
37.26%
1510
ilpd
28.64%
1480
tokyol
36.08%
40705
qsar-biodeg
1.055
41
33.74%
1494
ozone-level-8hr
2,534
72
72
6.31%
1487
madelon
2,600
500
500
50.00%
1485
Bioresponse
3,751
1,776
1,776
45.77%
4134
wilt
4,839
5
5.39% 5
40983
churn
5,000
20
22
14.14%
40701
phoneme
5,404
5
5
29.35%
1489
SpeedDating
8,378
120
61
241
16.47%
40536
Phishing Websites
11,055
30
30
46
44.31%
4534
Amazon Amazon.empluyee.eo www..eeeellllclcc.......eeeee
32,769
9
5.79%
4135
34,465
118
172
28.56%
1486
adult
48,842
23.93%
1590
numerai28.6
96,320
21
49.43%
23517
B ADDITIONAL RESULTS
'The notehook for dataset selection can be accessed under https://gi http:ttt...oooooooeeoeootttttttt com/openm1) benchmark http:///ww..aaeeeoooo/////////ea
1pymb
13

---

Published as a conference paper at ICLR 2024
1.0
0.8
0.4
LaNROOST
MODE
1.0
1.1
1.2
1.3
1.4
15
0
Figure 5: Performance Profile (HPO 250 Trials). The performance profile is based on the macro Fl-Score with optimized hyperparameters (complete grids, 250 trials). The x-axis represents a tol- erance factor, and the y-axis proportion of the evaluated datasets.
Table 6: ROC-AUC Performance Comparison (HPO 250 Trials), We report the test ROC-AUC (mean + stdev for 5-fold CV) with optimized parameters (complete grids, 250 trials) and the ranking of each approach in parentheses. The datasets are sorted based on the data size.
GRANDE
XGB
CatBoost
NODE
dresses-sales
0.623 + 0.049 (2)
0.589
0.060 (4) 0.607 0.040 (3)
0.632 0.041 (1)
0,958 0.031 (1)
0.923
0.031 (4)
0.938
0.036 (2)
0.933
0.030
cylinder-bands
1.019 (3)
0.046 (2)
0.837
wdbe
0.993
0.006
0.990 0.010 (4)
0,990
0.010
0.993 0.007
ilpd
0,748
0.046 (1)
0,721 0,047 (4)
0,728
0.055 (3)
0.745 0.048
tokyol
0.983 = 0.005 (2)
0,982 = 0.006 (3)
0.984 0.005 (1)
0.980 = 0.005 (4)
gsar-biodeg
0.934 1 0.008 (1)
0,925 1 0.008 (3)
0.933 0.011 (2)
0.920 0.009 (4)
azone-level-Shr
0.925 = 0.013 (1)
0.879 + 0.012 (4)
0,910
0.011 (2)
0.906 = 0.021 (3)
madelon
0.875 = 0.008 (3)
0,944 = 0.014 (2)
0.928 0.012 (1)
0.612 = 0.016 (4)
Bioresponse
0,872 + 0.003 (3)
0,873 1: 0.007 (1)
0,733 0.002 (2)
0.859 0.008 (4)
wilt
0,994 = 0.007 (2)
0,680 + 0.015 (4)
0.991
0.009 (3)
0.996 0.003
churn
0,928 = 0.014 (1)
0,119 0,018 (4)
0,920
0.013 (3)
0.927 = 0.014
phoneme
0.939 = 0,859 1 0.012 (I)
0,955
(2)
0.959 0.055
0.934 0.010 (4)
SpeedDating
0.827 1 0,017 (4)
0.856 0.014 (2)
0.853 0.014 (3)
Phishing Websites
0.996 = 0.001 (2)
0.996 = 0,001 (I)
0.996
0.001 (4)
0.9% = 0.001 (3)
Amazon employee access
0.830 = 0.010 (3)
0.778 1 0,015 (4)
0.842
0.014 (1)
0.841 = 0.009 (2)
nomao
0.994 + 0.001 (3)
0.996 + 0.001 (1)
0.995
0.001 (2)
0.993 = 0.001 (4)
adult
0.910 = 0.005 (4)
0.927 = 0.002 (1)
0.925
0.003 (2)
0.915 = 0.003 (3)
numerai28.6 6
0.529 1 0.003 (3)
0.529
0.002 (2)
0.529
0.002 (1)
0.529 1 0.003 (4)
Normalized Mean f
0.769 (1)
0.419 (3)
0.684 (2)
0.395 (4)
Mean Reciprocal Rank (MRR)
0.645 (1)
0.461 (3)
0.575 (2)
0.404 (4)
14

---

Published as a conference paper at ICLR 2024
Table 7: Accuracy Performance Comparison (HPO 250 Trials). We report the test balanced accuracy (mean + stdev for a 5-fold CV) with optimized parameters (complete grids, 250 trials) and the ranking of each approach in parentheses. The datasets are sorted based on the data size.
GRANDE
XGB
CatBoost
NODE
dresscs-sales
0.613 = 0.048 (I)
0.585 0.060 (3) 0.589 0.037 (2)
0,581 0.037
dimmat-simmulationccaaahh
0.875 0.079 (1)
0.762 + 0.060 (4)
0.832 0.035 (2)
0.773 + 0.052 (3)
cylinder-bands
0,817 = 0.032 (1)
0.777 = 0,045 (3)
0.798 0.042 (2)
0.751 = 0.040 (4)
wdbe
0.973 1 0.010 (1)
0,952 1 0.029 (4)
0.963 0.023 (3)
0.963 1 0.017 (2)
ilpd
0.709 = 0.046 (1)
0.673 = 0.042 (3)
0.689 0.062 (2)
0.548 = 0.054 (4)
tokyol
0,925 = 0.002 (2)
0.918 0.012 (4)
0.932 0.010
0,920 = 0.007 (3)
gsar-biodeg
0,853 + 0.024 (2)
0.856 + 0.022 (1)
0.847 0.028 (3)
0.831 + 0.032 (4)
ozone-level-Shr
0,774 = 0.016 (1)
0.733 = 0.021 (3)
0.735
0.034 (2)
0.669 = 0.033 (4)
madelon
0.803 1 0.010 (3)
0,833 1 0.018 (2)
0.861
0.012 (1)
0.571 1 0.022 (4)
Bioresponse
0.795 = 0.009 (3)
0,799 0,011 (2)
0,801 0.014 (1)
0.780 0.011
wilt
0.962 1 0.026 (I)
0,941 0,012 (4)
0,955
0.021 (2)
0.948 0.024 (3)
churn
0,909 = 0.014 (I)
0,895 = 0.022 (3)
0.894
0.014 (4)
0.904 = 0.021 (2)
phoneme
0,859 = 0.011 (4)
0.882 1 0.005 (2)
0,886
0.006 (1)
0.859 = 0.013 (3)
SpeedDating
0.752 + 0.020 (2)
0.740 + 0.017 (3)
0.758
0.012 (1)
0.694 + 0.015 (4)
Phishing Websites
0,969 = 0.006 (1)
0.968 = 0.006
0.965
0.003 (4)
0.968 = 0.006 (3)
Amazon_empp wwwizoonemployee..ccoo
0.707 1 0.010 (2)
0.701 1 0.015 (3)
0.775
0.009 (1)
0.617 0.007 (4)
nomao
0,961 = 0.001 (3)
0.969 0.002 (2)
0.969
0.001 (1)
0,956 0.001 (4)
adult
0.817 + 0.008 (3)
0.841 0.004 (2)
0.841
0.004 (1)
0.778
0.008
numerai28.6
0.520 0.004 (2)
0.520 0.000 (3)
0.521
0.002
0.519 0.003
Normalized Mean f
0.793 (1)
0.523 (3)
0.730 (2)
0.126 (4)
Mean Reciprocal Rank (MRR)
0.689 (2)
0.404 (3)
0.693 (1)
0.298 (4)
Table 8: Default Parameter Performance Comparison. We report the test macro Fi-score (mean + stdev over 10 trials) with default parameters and the ranking of each approach in parentheses. The datasets are sorted based on the data size.
GRANDE
XGB
CatBoost
NODE
dresses-sales
0.5%6 + 0.014 (1)
0.570 + 0.056 (3) 0.573
0.031 (2)
0.559 + 0.045 (4)
commatt-simulation..m
0.758 = 0.065 (4)
0.781 0.060 (1)
0.781
0.050
0.766 - 0.088 (3)
cylinder-bands
0.813 0.023 (1)
0.770 1.0.00 (3)
0.795
0.051 (2)
0.696 0.028 (4)
wdbc
0.962 - 0.008 (3)
0.966 0.023 (1)
0.555
0.029 (4)
0.964 0.017
ilpd
0.646 0.021 (1)
0.629 + 0.052 (3)
0.643 0.042
0.501 = 0.085 (4)
tokyol
0.922 = 0.014 (1)
0.917 = 0.016 (4)
0.917 0.013
0.921 = 0.011 (2)
gsar-biodeg
0.851 1 0.032 (1)
0,844 1 0.021 (2)
0,843 0.017 (3)
0.838 1 0.027 (4)
azone-level-Shr
0.735 = 0.011 (1)
0.686 + 0.034 (3)
0.702 0.029 (2)
0.662 = 0.019 (4)
madelon
0.768 = 0.022 (3)
08811 = 0.016 (2)
0.851 0.015 (1)
0.650 = 0.017 (4)
Bioresponse
0.789 1 0.014 (2)
0.789 1 0.013 (3)
0.792 0.004 (1)
0.786 1 0.010 (4)
wilt
0,933 = 0.021 (I)
0,903 = 0.011 (3)
0.898
0.011 (4)
0.904 = 0.026 (2)
churn
0.896 + 0.007 (3)
0.897 0.022 (2)
0.862 0.015 (4)
0.925 0.025
phoneme
0.860 = 0.008 (3)
0,864 0,003 (I)
0,861 0.008 (2)
0.842 0.005
SpeedDating
0.725 0.007 (I)
0.686 0,010 (4)
0.693 0.013 (3)
0.703 0.013 (2)
Phishing Websites aanazon.employee.uce
0,969 = 0.006 (1)
0,969 = 0.007 (2)
0.963 0.005 (3)
0.961 = 0.004 (4)
0,602 = 0.006 (4)
0.608 = 0,016 (3)
0.652 0.006 (1)
0.621 = 0.010 (2)
nomao adult
0.955 : 0.004 (3)
0,965 0.003 (1)
0.962 0.003 (2)
0.955 + 0.002 (4)
0.785 = 0.008 (4)
0.796 = 0.003 (2)
0.796 0.005 (3)
0.799 = 0.003 (I)
numerai28.6
0.503 + 0.003 (4)
0.516 + 0.002 (2)
0.519
0.001 (1)
0.506 0.009 (3)
Normalized Mean f
0.637 (1)
0.587 (3)
0.579 (1)
0,270 (4)
Mean Reciprocal Rank (MRR)
0.640 (1)
0.518 (3)
0.522 (2)
0.404 (4)
15

---

Published as a conference paper at ICLR 2024
Table 9: Runtime Performance Comparison. We report the runtime (mean + stdev for 5-fold CV) with optimized parameters (complete grids, 250 trials) and the ranking of each approach in parentheses. The datasets are sorted based on the data size. For all methods, we used a single NVIDIA RTX A6000.
GRANDE
XGB
CatBnost
NODE
dresses-sales
11.121 + (3)
0.052 + 0.0 (1)
4.340 2.0 (2)
31,371 + 17.0
16.838 + (30
0.077 : 0.0 (1)
8.700
97.693 + (4)
sylinder-bande
20.554 2.0
0.093 0.0 (1)
4.887 2.0
38.983 + 12.0
wdhc
29.704 (.0
0.10
1.46
83.548
16.0
ilpd tokyol
11.424 (.0(3)
0.049 (1)
3.48
59.085 + 24.0
21.483 + (30
0.078 0.0
1.485
84,895
5.0
gsar-biodeg
21.565 2.0(3)
0.087
1.09
96.204 + 20.0
ozone-Ievel
56.889 (.0
0.092
0.80
137.910 = 27.0
madelon
44.783 (4.0
0.360 1.0
1.247
90.329 + 13,0
Bioresponse
37.224 2.0(3)
0.865 0.0
2.136110.0
309.178 = $4.0
wilt
44.476 7.0
0.127 (1)
0.00
199.653 = 20.0
churn
49.096 (.0
0.099 0.0
4.117
150.088 = 34,0
phoneme
59.286 (3)
0.201
1.793
240.607 + (4)
SpeedDating
83.458 + (4)
0.207
1.0
66.560 + 22.0()
Phishing Websites
107.101 (3.0
0.271
2.0
340.660 + 102.0
Amazon_employyee employee.access
37.190 = (40
0.047
0.0
30.309 = 4.0
nomao
95.775 (1)0
0.268 (1)
2..911
208.682 + (4.0
adult
96.737 = (3)
0.125 0.0
171.783 = 34.0
numerai28.6
39.031 + (3)
0.083 = 0.0
1.323 = (2)
47.520 + 34)
Mean
46.512 (3)
0.175 (I)
3.66 (2)
130.80 (4)
Table 10: Ablation Study Split Activation. We report the test macro Fl-Score (mean + stdev for 5-fold CV) with optimized parameters (complete grids, 250 trials) and the ranking of each approach in parentheses. The datasets are sorted based on the data size.
Softsign
Entmoid
Sigmoid
dresses-sales
0.612 + 0.049 (1)
0.580 + 0.047 (2)
0.568 0.052
climate-simulation- crasses
0.853 0.070 (1)
0.840 + 0.038 (2)
0.838 0.041 (3)
cylinder-bands
0,819 1 0.032 (1)
0.802 1 0.028 (3)
0.807 + 0.020 (2)
wdbe
0,975 1 0.010 (1)
0.970 1 0.007 (3)
0.972 + 0.008 (2)
ilpd
0.657 1 0.042 (2)
0.652 1 0.047 (3)
0.663 1 0.047
tokyol
0.921 0.004 (1)
0.010
0.921 0.008
qsar-biodeg
0.854 + 0.022 (3)
0.855 = 0.018
0.845 0.018 (1)
ozone-level-She
0.726 + 0.020 (1)
0.710 0.024
0.707 = 0.031 (2)
madelon
0.803 +: 0.010 (1)
0.773 0.009
0.747 0.009 (3)
Bioresponse
0.794 + 0.008 (2)
0.795 + 0.012 (1)
0.792 0.010
wilt
0.936 + 0.015 (1)
0.932 + 0.014
0.929 + 0.012
churn
0.914 = 0.017 (L)
+ 0.010 (2)
0.887 + 0,015 (3)
phoneme
0,846 + 0.008 (1)
0.829 0.002 (2)
0.828 + 0.010 (3)
SpeedDating
0.723 - 0.013
0.725 - 0.012 (1)
0.725 = 0.012
PhishingWebsiine
0.969 + 0.006 (1)
0.968 = 0.006 (2)
0.967 0.006
Amazon employee access
0.665 + 0.009 (1)
0.663 + (3)
0.664 0.016
nomao
0.958 + 0.002 (1)
0.956 - 0.002 (2)
0.954 0.003 (3)
adult
0.790 1 0.006 (2)
0.793 1 0.005 (1)
0.790 T 0.006 (3)
numerai28.6
0.519 1 0.003 (2)
0.520 1 0.004 (1)
0.519
0.003
Normalized Mean f
0.791 (1)
0.419 (2)
0.221 (3)
Mean Reciprocal Rank (MRR) t
0.825 (1)
0.553 (2)
0.456 (3)
16

---

Published as a conference paper at ICLR 2024
Table 11: Ablation Study Weighting. We report the macro FI-Score (mean + stdev for a 5- fold CV) with optimized parameters (complete grids, 250 trials) and the ranking of each approach in parentheses. The datasets are sorted based on the data size.
Leaf Weights
Estimator Weights
dresses-sales
0.612 + 0.049 (1)
0.605 + 0.050 (2)
siinateesiiuulatoo-crraassss
0.853 + 0.070 (1)
0.801 + 0.040 (2)
cylinder-
0,819 0.032 (1)
0.787 + 0.051 (2)
wdbe
0.975 1 0.010 (1)
0.970 + 0.009 (2)
ilpd
0.657 + 0.042 (1)
0.612 + 0.064 (2)
tokyol
0.921 + 0.004 (2)
0.922 + 0.016 (1)
qsar-biodeg
0.854 + 0.022
0.850 0.019
ozone-level-8hh
0.726 0.020 (1)
0.711 0.020 (2)
madelon
0.803 + 0.010 (1)
0.606 + 0.028 (2)
Bioresponse
0.794 + 0.008 (1)
0.784 + 0.010
wilt
0.936 0.015
0.930 0.019
churn
0,914 0.017
0,873 + 0.018
phoneme
0,846 : 0.008 (1)
0.845 + 0.011 (2)
SpeedDating
0.723 + 0.013 (2)
0.728 + 0.009 (1)
PhishingWebsites
Websites
0,969 + 0.006 (1)
0.965 + 0.006 (2)
Amazon employee access
0.665 + 0.009
0.675 0.008
nomao
0.958 + 0.002 (1)
0.954 + 0.002 (2)
adult
0.790 1 0.006 (1)
0.790 1 0.005 (2)
numerai28.6
0,519 1 0.003 (1)
0.519 T 0.003 (2)
Normalized Mean f
0.824 (1)
0.177 (2)
Mean Reciprocal Rank (MRR)
0.921 (1)
0.579 (2)
Table 12: Pairwise Confusion Matrix Phishing Websites We compare the predictions of each ap-
proach with the predictions of CART DT. It becomes evident, that a simple model is not sufficient to solve the task well, as CART makes more than twice as many mistakes as state-of-the-ee models.
Correct DT
Incorrect DT
Total
Correct GRANDE
2012
128
2140
Incorrect GRANDE
15
56
71
Total
2027
184
Correct DT
Incorrect DT
Total
Correct XGBoost
2018
115
2133
Incorrect XGBoost
69
78
Total
2027
Correct DT
Incorrect DT
Total
Correct CatBoost
2019
105
2124
Incorrect CatBoost
79
87
Total
2027
184
Correct DT
Incorrect DT
Total
Correct NODE
2012
112
2124
Incorrect NODE
15
72
87
Total
2027
184
C ADDITIONAL BENCHMARKS
In addition to the representative evaluation in the main paper, we provide an extended comparison with additional benchmarks, including tree ensemble methods Random Forest (Breiman) 2001) and ExtraTrees (Geurts et al. 2006), as well as SAINT Somepalli et al. (2021) as an additional DL benchmark, which is the superior gradient-based method according to Borisov et al. (2022). In the following, we present results using default hyperparameters for all methods (Table 15) and optimized hyperparameters (Table 1 and Table 14). The results for HPO presented twofold. In Table [3 we report the results of an extensive search comprising 250 trials. Due to the very long runtime of SAINT, were not able to optimize 250 trials for SAINT. Therefore, present results from
17

---

Published as a conference paper at ICLR 2024
Extended Performance Comparison (HPO 250 trials). We report the test macro F1- score (mean for 5-fold CV) with optimized parameters based on HPO with 250 trials. The datasets are sorted based on the data size.
Table 13:
GRANDE
XGB
CatBoost
NODE
ExtraTTree
RandomForest
dresses-sales
0.612
0.581 (4)
0.588 (3)
0.564 (5)
0.589 (2)
0.559
climate-modelt sidilllioooon
0.853 (1)
0.763 (4)
0.778 (3)
0.802 (2)
0.754 (5)
0.751 (6)
cylinder-hands
0.819
0.773 (4)
0.801
0.754 (6)
0.762
0.782
wdbc
0.975 (1)
0.953 (4)
0.963
0,966 (2)
0.951
0.949
ilpd
0.657
0.632 (5)
0.643
0.526 (6)
0.645
0.640
tokyol
0.921 (3)
0.915
0.927 (1)
0,921 (2)
0.916 (4)
0.909 (6)
qsar-biodeg
0.854 (2)
0.853 (3)
0.844 (5)
0.836 (6)
0.851 (4)
0.855 (1)
ozone-level-Shr
0,726 (1)
0.688 (6)
0.721 (2)
0.703 (3)
0.695 (5)
0.697 (4)
madelon
0.803 (3)
0.833 (2)
0.861 (1)
0.571 (6)
0.766 (4)
0.764 (5)
Bioresponse
0.794 (3)
0.799 (2)
0.801 (1)
0,780 (6)
0.794 (5)
0.794 (4)
wilt
0.936 (2)
0.911 (6)
0.919 (3)
0.937 (1)
0.917 (5)
0.918 (4)
churn
0.914 (2)
0,900 (3)
0.869 (6)
0,930 (1)
0.899 (4)
0.899 (5)
phoneme
0.846 (6)
0.872 (2)
0.876 (1)
0.862 (5)
0.868 (4)
0.868 (3)
SpeedDating
0.723 (1)
0.704 (6)
0.718 (2)
0.707 (5)
0.708 (3)
0.707 (4)
PhishingWWbsites
0,969 (1)
0.968 (2)
0.965 (4)
0.968 (3)
0.963 (5)
0.963 (5)
Amazon employee access
0.665 (2)
0.621 (6)
0.671 (1)
0.649 (3)
0.642 (5)
0.648 (4)
nomao
0.958 (3)
0.965 (1)
0.964 (2)
0.956 (4)
0.951 (6)
0.955 (5)
adult
0.790 (5)
0.798 (1)
0.796 (2)
0.794 (3)
0.785 (6)
0.790 (4)
numerai28.6
0,519 (1)
0,518 (5)
0,519 (2)
0.503 (6)
0.519 (4)
0.519 (3)
Normalized Mean
0.817 (1)
0.518 (3)
0.705 (2)
0.382 (6)
0.385
0.404 (4)
Mean Reciprocal Rank (MRR) f
0.668 (1)
0.365 (3)
0.541 (2)
0.352 (4)
0.251 (6)
0.275 (5)
additional HPO consisting of 30 trials for each method (similar to McElfresh et al. (2023)) including SAINT (Table 3 Details on the HPO can be found in Appendix E The implementation for SAINT, similar to NODE, was adopted from Borisov et al. (002)).
In all experiments, GRANDE demonstrated superior results over existing methods, achieving the highest normalized accuracy, mean reciprocal rank (MRR) and number of wins in almost all scenar- ios. In line with the experiments from the paper, CatBoost achieved the second-best and XGBoost the third-best results. The only exception is using default parameters, where XGBoost achieved slightly higher normalized mean, but GRANDE still an significant higher MRR. While SAINT provided competitive results for some datasets (1 wins with HPO). it also achieved a very perfor- mance in some cases (e.g. for the madelon dataset, the prediction of SAINT was only predicting the majority class). Similarly, ExtraTreee and RandomForest, especially with optimized hyperparame- ters achieved competitive results, but were almost consistently outperformed by gradient-boosting methods and GRANDE.
18

---

Published as a conference paper at ICLR 2024
Table 14: Extended Performance Comparison (HPO 30 trials). We report the test macro Fl-score (mean for a 5-fold CV) with optimized parameters (30 trials). The datasets are sorted based on the data size.
GRANDE
XGB
CatBoost
NODE
ExtralTree
RandomfForest
SAINT
dresses-sales
0.596 (1)
0.567 (5)
0.592 (2)
0.564 (6)
0.583 (3) 0.570 (4)
0,516 (7)
climate-model-sinee momoll-iimmllaiioncrssseee
0.841 (1)
0.764 (6)
0.782 (5)
0.802 (3)
0.802 (2)
0.793 (4)
0.744 (7)
sylinder-badde
0,799 (1)
0.769 (4)
0.787 (3)
0.754 (6)
0.766 (5)
0.753 (7)
0.791 (2)
wdbe
0.966 (2)
0.959 (4)
0.964 (3)
0,966 (1)
0,955 (5)
0.951 (7)
0,952 (6)
ilpd
0.656 (1)
0.625 (5)
0.650 (2)
0.526 (6)
0.650 (3)
0.642
0.425 (7)
tokyol
0.933 (1)
0.919
0.925 (3)
0.921 (4)
0.929 (2)
0,920 (5)
0.915
suar-biddeg
0.857 (1)
0.841 (6)
0,850 (5)
0.836 (7)
0.850 (4)
0.855 (2)
0.850 (3)
prone-level--oe
0.716 (2)
(6)
0.736 (1)
0.703 (3)
0.695 (5)
0.697 (4)
0.659 (7)
madelon
0.787 (3)
0.833 (2)
0.861 (I)
0.571 (6)
0.656 (5)
0.774 (4)
0.333 (7)
Bioresponse
0.794 (5)
0.799 (1)
0.797 (2)
0.780 (6)
0.796 (3)
0.795 (4)
0.780 (6)
wilt
0,934 (3)
0.913 (6)
0,919 (4)
0,937 (2)
(1)
0,916
0.574 (7)
churn
0.914 (2)
0.899 (4)
0.869 (7)
0.930 (I) 0.903 (3)
0.899 (5)
0.873 (6)
phoneme
0.855 (6)
0.870 (2)
0.881 (I)
0.862 (4)
0.852 (7)
0.868 (3)
0.858 (5)
SpeedDDating
0,728 (1)
0.704 (6)
0.715 (2)
0.707 (5)
0.709 (4)
0.709
0.678 (7)
Phishing Wehsites
0,966 (4)
0.968 (1)
0,964 (5)
0,968 (2)
0.953 (7)
0,962 (6)
0.966 (3)
Amazon employee access
0.622 (6)
0.620
0.671 (2)
0.649
0.633 (5)
0.647 (4)
0.696 (1)
nomao
0.955 (5)
0.963 (2)
0,964 (1)
0,956 (3)
0.944 (7)
0.954
0,956 (4)
adult
0.790
0.798 (1)
0.796 (2)
0.794
0.784 (6)
0.790
0.774
mumerai28.6 6
0,521 (1)
0,518 (5)
0,519 (3)
0.503 (7)
0.518 (4)
0,520 (2)
015009 (6)
Normalized Mean f
0.804 (1)
0.609 (3)
0.787 (2)
0,510 (6)
0.523 (5)
0.585 (4)
0,244 (7)
Mean Reciprocal Rank (MRR) 1
(1)
0.368 (3)
0.491 (2)
0.341 (4) 0.299 (5)
0.257 (6)
0,240 (7)
Table 15: Extended Performance Comparison (Default Parameters). We report the test macro F1-score (mean for a f-fold CV) with default parameters. The datasets are sorted based on the data size.
GRANDE
XGB
CatBonst
NODE
Extra/Tree
RandomForest
SAINT
seessss-sales
0.596 (1)
0.570 (3) 0.573 (2)
0.559 (4) 0.545 (6)
0.554 (5)
0.422 (7)
cimoat-modd--simolaaoooo-a-
0.758 (5)
0.781 (2)
0.781 (3)
0.766 (4)
0.728 (6)
0.714
0,850 (I)
eylinder-bands
0.813 (1)
0.770 (5)
0.795 (2)
0.696 (7)
0.780 (4)
0.787 (3)
0.733 (6)
wdibe
0,962 (3)
0.966 (1)
0,955 (4)
0,964 (2)
0,940 (7)
0,942 (6)
0,945 (S)
ilpd
0.646 (1)
0.629 (3)
0.643 (2)
0.501 (6)
0.582 (5)
0.592
0,416 (7)
tokyol
0.922 (1)
0.917
0.917
0.921 (2)
0.917 (3)
0,917 (4)
0,907 (7)
qsar-biodeg
0.851 (1)
0.844
0.843 (3)
0.838 (4)
0.836 (7)
0.838 (5)
0.836 (6)
ozone-leeve--Sfe
0.735 (1)
0.686 (3)
0.702 (2)
0.662 (4)
0.601 (7)
0.605 (6)
0.651 (5)
madelon
0.768 (5)
0.811 (2)
0.851 (1)
0.650 (6)
0.773 (4)
0.773 (3)
0.333 (7)
Bioresponse
0.789 (4)
0.789
0.792 (3)
0.786 (6)
0.799 (2)
0,801 (1)
0,786 (6)
0,933 (1)
0.903
0.898 (6)
0.904 (4)
0.916
0.911 (3)
0,486 (7)
chur
0.896 (3)
0.897 (2)
0.862 (7)
0.925
0.892
0.893 (4)
0.867 (6)
phoneme
0.860 (6)
0.864
0.861 (5)
0.842
0.875 (2)
0,880 (1)
0.862 (4)
SpeedDating
0.725 (1)
0.969 (4686(4)
0.693 (3)
0.703 (2)
0.642 (6)
0.640 (7)
0.643 (5)
PhishingWehhar Wehsites
0.969 (1)
0.963 (6)
0.961 (7)
0.968 (3)
0.968 (3)
0.964 (5)
Amazon, Amazor_cmployee wwwpeype.pccooocccccc
0.602 (7)
0.608 (6)
0.652 (4)
0.621 (5) 0.662 (3)
0.682 (1)
0.682 (2)
nomao
0.955 (6)
0.965 (1)
0.962 (2)
0.955 (7)
0.956 (4)
0.961 (3)
0.956 (5)
adult
0.785 (5)
0.796 (2)
0.796 (3)
0.799 (1)
0.784
0.793 (4)
0.768
mumerai28.6
0.503 (7)
0.516 (2)
0.519 (1)
0.506 (6)
0.514 (3)
0,512 (4)
0.509 (5)
Normalized Mcan 1
0.673 (2)
0.691 (1)
0.666 (3)
0.460 (6)
0.518 (5)
0.596 (4)
0,228 (7)
Mean Reciprocal Rank (MRR) f
0.586 (1)
0.422 (2)
0.397 (3)
0.326 (5)
0.267 (6)
0.365 (4)
0.235 (7)
D DISCUSSION WEIGHTING STATISTICS
Within our paper, we showed in an ablation study that our instance wise weighting has a positive impact on the performance of GRANDE. In addition, we showcased real-world scenario where the weighting significantly increases local interpretability by enabling the model to learn represen- tations for simple and complex rules. In the following, we provide more detailed evaluation of the instance-wise weighting, including statistics on the highest weighted estimators and the weight distributions.
The following statistics were obtained by calculating the post-softmax weights for each sample and averaging the values over all samples. In addition, calculated the same statistics for the top 5% of samples with the highest weights. This provides additional insights since we agrue that unique local interactions might exist only for subset of the samples.
19

---

Published as a conference paper at ICLR 2024
Table 16: Weighting Statistics Estimator
Number Internal Nodes
Number Leaf Nodes
Percentage Highest Estimator
Percentage Highest Estimator
Count Highest Estimator
Count Highest
Estimator
Top 5%
Top 5%
dresses-sales
1.12
2.12
0.470
0.667
10.20
2.40
clmate-model-simllate
3.12
4.12
0.241
0.467
15.80
4,00
cylinder-bands
5.44
6.44
0.124
0.400
31.20
4,00
wdbc
1.16
2.16
0.587
0.733
4,20
1.60
ilpd
3.96
4.96
0,319
0.600
15.00
3.40
tokyol
3.20
4.20
0.273
0.780
19.80
2.00
qsar- biar-bideeg
2.80
3.80
0.191
0.745
29,80
2,60
ozone-level-8hr
2.00
3.00
0.198
0.662
27.80
5.40
madelon
3.92
4,92
0.322
0.741
19.60
3.20
Bioresponse
3.96
4.96
0.099
0.311
82.00
11.80
wilt
1.76
2.76
0.274
0.604
19.20
5.00
churn
2.28
3.28
0.111
0.408
72.00
11.40
phoneme
3.28
4.28
0.163
0.636
40.80
4.40
SpeedDating
6.96
7.96
0.168
0.640
90.20
5.00
PhishingWebsst Websites
5.60
6.60
0.178
0,594
116,80
12.20
Amazon wwt.eeeeplllccccccceooo
0.24
1.24
0,882
0,999
2.80
1.20
nomao
3.20
4,20
0.113
0,260
218.20
15.80
adult
0.36
1.36
0,905
0.912
5.20
1.60
numerai28.6
1.08
2.08
0,559
0.824
19.60
4.40
Mean
2.92
3.92
0.325
0.631
44.22
5.35
We can observe that on average, the highest weighted estimator comprises a moderate number of Rx3 internal and 244 leaf nodes, allowing an easy interpretation (see Table 16). Furthermore, on average, the highest weighted estimator is the same for =33% of the data instances, with total of xe44 different estimators having the highest weight for at least one instance. If we further inspect the top 5% of instances with the highest weight for single estimator, we can additionally observe that the highest weighted estimator is the same for =63% of these instances and only different estimators have the highest weight for at least one instance. This suggests the presence of a limited number of local experts in most datasets, which have high weights for specific instances where local rules are applicable.
In addition, Table 17 summarizes the skewness kurtosis as measure of the tailedness of a distribution, for all datasets. In general, it stands out that both values increase significantly when considering only the top 5% of samples with the highest weight for a single estimator. This again indicates the presence of local expert estimators for a subset of the data, where unique local interactions were identified. In addition, there are major differences in the values depending on the datasets, which we will discuss more detailled in the following:
of the asymmetry of distribution, and
Skewness In total, for 16/19 datasets the weights have skewed (10) or very skewed (6) distribu- tion, i.c., are long-tailed (see Table 18). Generally, left-skewed distributions suggest few estimators with very low weights, thereby contributing significantly less and resulting in a more compact en- semble. However, right-skewed distributions are more desired as they indicate a small number of trees having high weights (which we consider as local experts), while the majority of estimators have small weights. Considering
top 5% of samples with the highest weights of a single estimator, we can see that the number of very skewed distributions increases from 6 to 13 (see T9ble indicating samples that can be assigned to class more easily based on a small number of trees (similar to the most interested in sigwe distributions this
example in the case study). In general, indicates small number of estimators with high weights, which we can consider as local experts.
(Exeess) Kurtosis We can observe that 11/19 distributions are leptokurtic, i.c., have heavy tails, with 4 considered as very leptokurtic (see Table 20 In general, distributions with heavy tails are interesting, as this indicates outliers (estimators with significantly higher lower weights). Trees comprising significantly higher weights be considered local experts that have learned unique local interactions. Again, when considering the top 5% of samples with the highest weights of a single estimator, the number of leptokurtic distributions increases from 11 to 15 with the number of
20

---

Published as a conference paper at ICLR 2024
Weighting Statisties Distribution. We provide the skewness, as measure of the asym- metry of distribution, and kurtosis, as measure of the tailedness of distribution, for all datasets. The measures are split in an average over all samples and an average over the top 5% of the samples with the highest maximum weight.
Table 17:
Skewness
Skewness Top 5%
Kurtosis
Kurtosis Top 5%
dresses-sales
0.946
1.187
0.617
1.597
climate-model-simulee
1.259
1.909
0.742
3.095
cylinder-bands
0.853
1.199
-0.562
0.230
wdbe
0,082
0.251
-1084
-0,980
ilpd
0.180
0,385
-0.649
-0.528
tokyol
0.975
1,839
1.322
4,919
qsar- bir-bbodeg
0.875
1.194
0.200
1.302
ozone-level-Shr
0.752
1,462
-0.048
1.972
madelon
1,901
2,208
2.797
4,465
Bioresponse
0.777
1.139
-0.496
0.317
wilt churn
0.508
1,150
-0.499
1,100
2.369
3,488
5.912
13,419
phoneme
0.563
0.789
-0,447
0.119
SpeedDating
1.069
1,180
1.148
1.627
PhishingWebsites
1.263
1.978
1.136
4.343
Amazon_employee
jsccesssee_cccsss _access
-0.029
-0.119
3.348
3.081
nomao
2.411
4.190
6.920
24.791
adult
0.621
0.679
3.345
3,280
numerai28.6
-0.297
-0,269
0.833
0.829
Mean
0.899
1.360
1.291
3.630
Table 18: Skewness
Very skewed
Skewed
Slight skew
Sum
(-00,-1.0) (1.0, oo)
110,-0.33 -0.5) or (0.5, 1.0
0.5, 0.5
Left skew
Right skew
6
9
17
Sum
very leptokurtic distributions increasing from 4 to 8. Again, this indicates that there exists samples that can be assigned a class more easily with simple rules (from local experts) in many datasets.
Overall, the additional statistics are in line with the discussion in the paper. For several datasets, the distribution of weight is long- and/or heavy-tailed, indicating the presence of local expert estima- tors. However, the benefit of learning local expert estimators is dataset-dependee scenarios. This also confirmed by our results, as there are datasets where the distribution is symmetric and mesokurtie. This is valid for instance for numerai28.6. which is very complex dataset without many simple rules (as indicated by the poor performance all methods). addi- tion. it should be noted that more comprehensive analysis is necessary to confirm these additional findings.
and not feasible
Table 19: Skewness Top 5%
Very skewed
Skewed
Slight skew
Sum
(-00--1.00 (1.0, oo)
1.00, -055) (0.5, 1.0]
[- 0.5, 0.5]
Left skew Right skew Sum
2
13
3
17
13
2
4
21

---

Published as a conference paper at ICLR 2024
Table 20: Kurtosis
Excess Kurtosis
Count
Count Top 5%
Very leptokurtic (3.0, oo)
8
Moderately leptokurtic 30.5
Mcsokurtic [0.55, 0.5)
Platykurtic (-00, -0.5)
2
E HYPERPARAMETERS
We optimized the hyperparameters using Optuna (Akiba et al. 2019) with 250 trials and selected the search space and default parameters for related work in accordance with Borisov et al. (2022).
The best parameters were selected based on 5x2 cross-validation as suggested by Raschka (2018) where the test data of each fold was held out of the HPO to get unbiased results. To deal with class imbalance, we further included class weights. In line with Borisov et al. (2022), we did not tune the number of estimators for XGBoost and CatBoost, but used early stopping. To validate our approach of not tuning the estimators, we conducted an additional HPO with tuning the estimators for XGBoost and CatBoost (see Table 21 and T2ble[22) and show that this results in similar (slightly worse) results.
While using 250 trials can be considered as very exhaustive search, we include additional results for HPO with only 30 trials, similar to McHlfresh et al. (2023). Based on the results in Table 23 we can verify that even with small number trials, GRANDE achieves strong results. Yet, we can also observe that searching for more trials is especially beneficial for GRANDE.
Furthermore, GRANDE has total of 11 hyperparameters that were optimized during the HPO. In contrast, for XGBoost and CatBoost we only optimized 4 and 3 hyperparameters, respectively. The choice to optimize only a small number of hyperparameters was accordance with Borisov et al.) (2022) based the assumption that exhaustive search the most relevant parameters benefi- cial for the performance. To account for this, we performed an additional HPO for GRANDE, where similarly only optimized parameters (one overall learning_rate, focal factor, cosine decay steps and dropout). The results are displayed in Table 24 and show that GRANDE still achieves SOTA results. However, becomes evident, that GRANDE benefits from using larger grid, especially since using different learning rates for the different components (split values, split indices, leaf prob- abilities and leaf weights) has a positive impact on the performance.
For GRANDE, we used a batch size of 64 and early stopping after 25 epochs. Similar to NODE Popov et al. (2019). GRANDE uses an Adam optimizer with stochastic weight averaging over 5 checkpoints (Izmailov et al. 2018) and a learning rate schedule that uses a cosine decay with Hutter 2016). Furthermore, GRANDE allows using a focal fac- tor (Lin al. (017), similar to GradTree Marton et al. (2023). In the supplementary material, we provide the notebook used for the optimization along with the search space for each approach.
optional warmup (Loshchilov
22

---

Published as a conference paper at ICLR 2024
Table 21: Comparison XGB HPO (250 Trials). We compare a HPO for XGBoost with and without tuning the number of estimators based on 250 trials. Tuning the estimators does not improve the performance, and using early stopping in addition setting the number of estimators high number is sufficient.
Without
With
Difference
tuning estimators
tuning estimators
dresses-sales
0.5755
0.5813
-0.0058
0.7932
0.7626
0.0306
eylinder-bands
0.7670
0.7734
-0.0064
wdbe
0.9568
0,9531
0.0037
ilpd
0.6316
0.6321
-0,0005
tokyol
0,9147
0.9147
0,0000
qsar-biodeg
0.8486
0.8526
-0.039
ozone-level-Shr
0.7044
0.6885
0.0159
madelon
0.8226
0.8334
-0.0108
Bioresponse
0.7924
0.7988
-0,0064
wilt
0.9102
0.9114
-0,0012
churn
0.9003
0,8998
0.0005
phoneme
0.8725
0.8718
0.0007
SpeedDating
0.7071
0.7043
0,0028
Phishing Wchsites
0,9674
0,9683
-0,0009
Amazon employee access
0.6225
0.6209
0,0016
nomao
0.9642
0.9650
-0,0008
adult
0.7977
0.7980
-0.0003
numerai28.6
0.5187
0,5180
0,0007
Mcan
0.7930
0.7920
0,0010
Table 22: Comparison CatBoost HPO (250 Trials). We compare a HPO for CatBoost with and without tuning the number of estimators based on 250 trials. Tuning the estimators does not improve the performance, and using early stopping in addition to setting the number of estimators a high number is sufficient.
Without
With tuning estimators
Difference
tuning estimators
dresses-sales
0.5611
0.5880
-0.0270
diinatemoadel-immaltionno-omr cylinder-bands
0.7902
0.7775
0.0127
0,8080
0.8014
0.0066
wdbe
0.9551
0.9625
-0.0074
ilpd
0.6379
0.6428
+0.0049
tokyol
0,9249
0.9274
-0.0225
qsar-hiedeg
0.8467
0.8444
0.0023
ozone-level-Shr
0.7330
0.7206
0.0124
madelon
0.8650
0.8607
0.0042
Bioresponse
0,7992
0,8013
-0,0211
wilt
0.9227
0,9187
0,0040
chur
0,8734
0.8693
0.0042
phoneme
0,8828
0.8763
0.0065
SpeedDating
0.7182
0.7184
00.0011
Phishing Websites
0.9657
0.9650
0.0007
Amazon_em wtt.eeeeepeeececee.....
0.6704
0.6709
-0.0044
0.9633
0.9644
-0.0011
adult
0.7957
0.7960
-0.0003
numerai28.6
0.5172
0.5193
-0.0021
Mean
0.8016
0.8013
0.0003
23

---

Published as a conference paper at ICLR 2024
Table 23: HPO 30 Trials Performance Comparison. We report the test macro Fl-score (mean + stdev over 10 trials) based on reduced HPO with only 30 trials. The datasets are sorted based on the data size.
GRANDE
XGB
CatBoost
NODE
dresses-sales
0.596 = 0.047 (1)
0.567 + 0.055 (3)
0.592 0.046 (2)
0.564 + 0.051 (4)
Mlmmte-model-simulatte
0,841 = 0.058 (1)
0.764 = 0.073 (4)
0,782 0.057 (3)
0.802 = 0.035 (2)
cy sylinder-bands
0,799 1 0.037 (1)
0.769 1 0.040 (3)
0.787
0.044 (2)
0.754 1 0.040 (4)
wdbe
0,966 = 0.019 (2)
0.959 0.026 (4)
0.964
0.022 (3)
0.966 0.016 (I)
ilpd
0.656 0.044 (1)
0.625 0.052 (3)
0.650 0.059 (2)
0.526 0.069
tokyo1
0,933 = 0,011 (I)
0,919 0,012 (4)
0,925 0.011 (2)
0.921 0.010
gsar-biodeg
0.857 1 0.025 (I)
0,841 0,015 (3)
0,850 0.030 (2)
0.836 0.028 (4)
azone-level-Shr
0.716 = 0.007 (2)
0.676 = 0.028 (4)
0.736 0.027 (I)
0.703 = 0.029 (3)
madelon
0.787 = 0.014 (3)
0,833 = 0,015 (2)
0,861 0.012 (1)
0.571 = 0.022 (4)
Bioresponse
0.794 + 0.010 (3)
0.799 + 0.011 (1)
0.797 0.008 (2)
0.780 + 0.011 (4)
wilt
0.934 = 0.01 (2)
0.913 = 0.011 (4)
0.919 0.009 (3)
0.937 = 0.017 (I)
churn
0,914 = 0.013 (2)
0.899 + 0.020 (3)
0.869 0.014 (4)
0.930 0.011
phoneme
0,855 = 0.008 (4)
0.870 = 0.008 (2)
0.881
0.007 (1)
0.862 = 0.013
SpeedDating
0.728 1 0.007 (I)
0.704 0.012 (4)
0.715
0.015 (2)
0.707 0.015
PhishingWeheee Websites
0.966 = 0.004 (3)
0,968 0,008 (I)
0,964
0.004 (4)
0.968 0.006
Amaazon.emloyye..oo
0.622 = 0.028 (3)
0.620 0,008 (4)
0,671
0.013 (1)
0.649 0.009
nomao
0.955 + 0.001 (4)
0,963 0.004 (2)
0.964 0.004 (1)
0.956 = 0.001 (3)
adult
0.790 = 0.006 (4)
0.798 = 0.004 (I)
0.796
0.005 (2)
0.794 = 0.004 (3)
numerai28.6
0.521 1 0.004 (1)
0,518
0.002 (3)
0.519
0.002 (2)
0.503 1 (4)
Normalized Mean f
0.694 (1)
0.430
0.676
0,336 (4)
Mean Reciprocal Rank (MRR)
0.636 (1)
0.434 (3)
0.579 (2)
0.434 (3)
Table 24: HPO Reduced Grid Performance Comparison (250 Trials). We report the test macro F1-score (mean + stdev over 10 trials) based on a HPO with a reduced parameter grid for GRANDE (only one overall Dearning_rate, focal factor, cosine deeeeedeeayyyaee sstpss and dropout) and 250 trials for all methods. The datasets are sorted based on the data size.
GRANDE
XGB
CatBoost
NODE
dresscs-sales
0,590 = 0.042 (1)
0.581 0.059 (3) 0.588 0.036 (2)
0.564 = 0.051 (4)
semmate-model-smmllle
0.800 1 0.055 (2)
0.763 0.066 (4)
0.778
0.050 (3)
0.802 0.035
cylinder-hands
0.783 + 0.040 (2)
0.773
0.801
0.043 (1)
0.754 0.040
wdbc
0.964 = 0.007 (2)
0 (4)
0.963 0.023 (3)
0.966 0.016
ilpd
0,645 0.030 (1)
(3)
0.643 0.053 (2)
0.526 0.069 (4)
tokyo
0.928 = 0.017 (1)
0,915 0.011 (4)
0.927 0.013 (2)
0.921 = 0.010 (3)
gsar-biodeg
0.859 + 0.024 (1)
0.853 + 0.020 (2)
0.844 0.023 (3)
0.836 + 0.028 (4)
ozone-level-She
0,733 = 0.016 (1)
0.688 - 0.021 (4)
0.721
0.027 (2)
0.703 = 0.029 (3)
madelon
0.809 1 0.010 (3)
0.833 12331000 0.018 (2)
0.861 0.012 (1)
0,571 1 0.022 (4)
Bioresponse
0.783 = 0.009 (3)
0.799 + 0.011 (2)
0.801 0.014 (1)
0.780 - 0.011 (4)
wilt
0,921 = 0.014 (2)
0.911 = 0.010 (4)
0.919
0.007 (3)
0.937 - 0.017
churn
0.888 1 0.017 (3)
0,900 0.017 (2)
0.869 0.021 (4)
0.930 1 0.011
phoneme
0.859 = 0.010 (4)
0.872 0.007 (2)
0,876 0.005
0.862 0.013
SpeedDating
0.725 - 0.016 (1)
0.704 0,015 (4)
0,718
0.014 (2)
0.707 0.015
PhishingWebsites
0,970 = 0.006 (1)
0,968 = 0,006 (2)
0,965 0.003 (4)
0.968 = 0.006 (3)
Amazon_emppa http:oonmmmloyyeeecmo accesss
0,643 1 0.024 (3)
0.621 T 0.008 (4)
0.671 0.011 (1)
0.649 1 0.009 (2)
nomao
0.956 = 0.004 (4)
0.965 - 0.003 (1)
0.964
0.002 (2)
0.956 = 0.001 (3)
adult
0.791 = 0.005 (4)
0.798 = 0.004 (1)
0.796
0.004 (2)
0.794 = 0.004 (3)
numerai28.6
0,519 + 0.005 (2)
0,518 1 0.001 (3)
0.519 0.002 (1)
0.503 1 0.010 (4)
Normalized Mean f
0,659 1 0.509 (2)
0.488 1 0,497 (3)
0.721
0.373 (1)
0.346 1 0.561 (4)
Mean Recipmeal Rank (MRR) 1
0.610 + 0.531 (1)
0.425 + 0.469 (4)
0.596
0.456 (2)
0.452 + 0.627 (3)
24

---

Published as a conference paper at ICLR 2024
Table 25: Hyperparameters GRANDE (Part 1), We report the hyperparameters for GRANDE
based on the extensive HPO with 250 trials.
depth
n_estimators
Ir_weights
Ir_iddex
Ir_values
Ir.leff
dresses-sales
4
512
0,0015
0,0278
0.1966
0,0111
cummate-simulation-craaeee
4
2048
0.0007
0.0243
0.0156
0.0134
cylinder-bande
6
2048
0.0009
0.0084
0.0086
0.0474
wdbe
4
1024
0.0151
0.0140
0.1127
0.1758
ilpd
4
512
0.0007
0.0059
0.0532
0.0094
tokyol
6
1024
0.0029
0.1254
0.0056
0.0734
qsar-biodeg
6
2048
0.0595
0.0074
0.0263
0.0414
ozone-level-Shr
4
2048
0.0022
0.0465
0.0342
0.0503
madelon
4 2048
0.0003
0.0575
0.0177
0.0065
Bioresponse
6 2048
0.0304
0.0253
0,0073
0,0784
wilt
6 2048
0.0377
0.1471
0.0396
0.1718
churn
6 2048
0.0293
0.0716
0.0179
0.0225
phoneme
6 2048
0.0472
0.0166
0.0445
0.1107
SpeedDating
6 2048
0.0148
0.0130
0.0095
0.0647
WhishingWebsites
6
2048
0.0040
0.0118
0.0104
0.1850
Amazon employce access
6
2048
0.0036
0.0056
0.1959
0.1992
nomao
6
2048
0.0059
0.0224
0.0072
0.0402
adult
6
1024
0.0015
0.0087
0,0553
0,1482
numerai28.6
4
512
0.0001
0.0737
0,0513
0.0371
Table 26: Hyperparameters GRANDE (Part 2).
We report the hyperparameters for GRANDE
based the extensive HPO with 250 trials.
dropout
sclected_variablles variables
data_fraction
focal factor
cosine cosine_decay _steps
dresses-sales
0.25
(1.9966
0.9779
0.0
climate-simulation simulation-crashes
0.00
0.6103
0.8956
1000.0
cylinder-bands
0.25
0,5309
0.8825
1000.0
wdbc
0.50
0.8941
0,8480
0.0
ilpd
0.50
0.6839
0.9315
1000.0
tokyol
0.50
0.5849
0.9009
1000.0
qsar-biodeg
0.00
0.5892
0.8098
0.0
ozone-level-Shr
0.25
0.7373
0.8531
1000.0
madelon
0.25
0.9865
0,9885
100.0
Bioresponse
0.50
0,5646
0,8398
0.0
wilt
0.25
0.9234
0,8299
0
0.0
churn
0.00
0.6920
0,8174
0
1000.0
phoneme
0.00
0.7665
0.8694
1000.0
SpeedDating
0.00
0.8746
0.8229
3
0.1
Phishing Websites
0.00
0.9792
0,9588
0
0.1
Amazon.cmm employee access
0.50
0.9614
0,9196
3
0.0
nomao
0.00
0.8659
0,8136
0
100.0
adult
0.50
0.5149
0.8448
3
100.0
numerai28.6
0.50
0.7355
0.8998
0
0.1
25

---

Published as a conference paper at ICLR 2024
Table 27: Hyperparameters XGBoost. We report the hyperparameters for GRANDE based on the extensive HPO with 250 trials.
learning rate
max-depth
reg rgg-alpha
reg. lambda
dresses-sales
0.1032
3
0.0000
0.0000
dimaatesiimulationcrrsssss
0.0356
11
0.5605
0.0000
cylinder-bands
0,2172
11
0.0002
0.0057
wdbe
0.2640
0.0007 2
0.0000
ilpd
0,0251
0.3198 4
0.0000
tokyol
0,0293
0.2910 3
0.3194
qsar-biodeg
0.0965
0.0000 5
0.0000
ozone-level-Shr
0.0262
0.0000
0.6151
madelon
0.0259
6 0.0000
0,9635
Bioresponse
0,0468
0.9185 5
0.0000
wilt
0.1305
8
0.0000
0.0003
churn
0,0473
0.0000
0.3132
phoneme
0.0737
0.9459 11
0,2236
SpeedDating
0,0277
9 0.0000
0,9637
PhishingWebsites
0.1243
11
0.0017
0.3710
0.0758
0.9785 11
0.0042
nomao
0.1230
0.0000
0.0008
adult
0,0502
11
0.0000
0.7464
numerai28.6
0.1179
0.0001 2
0.0262
Table 28: Hyperparameters CatBoost. We report the hyperparameters for GRANDE based on the extensive HPO with 250 trials.
learning-rate
max_depth
12.leaf
dresses-sales saimatessiullatio-crrshhee
0,0675
3
19,8219
0.0141
19,6955
cylinder-bande wdbe
0.0716
19.6932 3
0,1339
0.7173
ilpd tokyol
0,0351
5,0922 4
0.0228
0.5016 5
qsar-biodeg
0.0152
0.7771
ozone-level-Shr
0.0118
3,0447 10
madelon
0,0102
9,0338
Bioresponse
0.0195
11
8. 1005
wilt
0.0192
1.95 11
churn
0.0248
7,0362
phoneme
0.0564
0,6744 11
SpeedDating
0.0169
1,5494 11
PhishingWebsiter
0.0239
1.6860 8
Amazon Amazon.cmployac..em wwwwemmw.ppccccccecooe
0.0123
1.6544
nomao
0.0392
2,6583
adult
0.1518
11
29.3098
numerai28.6
0,0272
18.6675 4
26

---

Published as a conference paper at ICLR 2024
Table 29: Hyperparameters NODE. We report the hyperparameters for GRANDE based on the grid suggested by the authors Popov et 2019).
num layers
total tree count
tree depth
dresses-sales
2048
6
dlimatesiimllation-craahe
1024
8
cylinder-baud -ands
1024
wdbe
2048
6
ilpd
4
2048
8
tokyol
4
1024
8
qsar-biodeg
2048
ozone-level-8ie
1024
madelon
2048
6
Bioresponse
1024
8
wilt
1024
churn
1024
phoneme
2048
8
SpeedDating
4
1024
6
PhishingWebsster
1024
8
Amazon maaazoo_employce_cccom
2048
nomao
2048
adult
1024
8
numerai28.6
2
1024
8
27


Reviews:
[{"review_id": "jEJBBZy0Tg", "reviewer": "Reviewer_Duhp", "summary": "The paper extends the GradTree support of Marton to weighted ensembles of trees, and using a different splitting function based on the soft-sign.\nThe paper compares their results to XGBoost, CatBoost and NODE, and show that they outperform these based on average rank of F1 score.", "questions": "How do the results in Table 4 compare to using the hard split function?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper gives a detailed and clear description of the approach. The experimental evaluation and the evaluation protocol are well-defined and sound, and the results look promising.", "weaknesses": "Given the popularity of gradient-based tree models in recent years, I feel like a more thorough comparison with competing methods would be warranted. In particular relating this work to work on learning weighting for fixed tree structures would be interesting, as first discussed in \"Practical Lessons from Predicting Clicks on Ads at Facebook\" by He et.al.\n\"Deep Neural Decision Trees\" by Yang et al also seems relevant, as well as \"Deep Neural Decision Forests\" by Kontschieder et al, \"SDTR: Soft Decision Tree Regressor for Tabular Data\" by Luo and \". The tree ensemble layer: Differentiability meets conditional computation.\" by Hazimeh et al.\n\n\n\"WindTunnel: Towards Differentiable ML Pipelines Beyond a Single Model\" also seems closely related, though they only fine-tune existing tree models with gradient descent.\n\nI find it also somewhat confusing that it is claimed that gradient boosted models are the defacto state-of-the-art on tabular data, when there has been a lot of recent work on neural networks for tabular data, often outperforming gradient boosting models, see \"Well-tuned Simple Nets Excel on Tabular Datasets\" by Kadra for example. It would be great to include at least one other neural approach in the comparison.\n\n\nFor the evaluation, I would have expected performance profiles or critical difference diagrams based on AUC or AP, instead of the mean and rank (or in addition to it, though the unnormalized mean is not a good way to aggregate performance across datasets).\n\nOverall, I find the contribution beyond Borisov somewhat slim, though that work is not published at a conference. If these are the same authors, I would recommend combining the two into a single submission.\n \nMinor:\nThe cylinder-bands dataset is called out, it would be interesting to see if this is the version with target leakage via the job_number column, or without that column.\nThe \"straight-through operator\" is simply the subgradient of the max, and I think the standard reference for that is \"Evaluation of pooling operations in convolutional architectures for object recognition\" by Scherer et.al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2RHyFdVHRR", "reviewer": "Reviewer_JdQZ", "summary": "This paper proposes a novel approach for learning hard, axis-aligned decision tree ensembles using end-to-end gradient descent based on the straight-through operator. The core contributions are two folds: 1) using an alternative differentiable split function (softsign); 2) introducing an advanced instance-wise weighting mechanism for tree ensemble. Experiments on tabular benchmark show that this method outperforms existing gradient-boosting and deep learning frameworks.", "questions": "See \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is one of the few deep learning based works which beat XGB on tabular benchmark. \n2. The contributions (alternative differentiable split function and instance-wise weighting) are supported by ablation experiments.\n3. It provides all the hyperparameters in appendix, which helps reproduction.", "weaknesses": "1. This paper lacks further analysis for instance-wise weighting. Because the final results are weighted by Softmax, the prediction of each tree is not separate now. If we cut off one tree, the contributions of the other tree are also changed. This is different from XGB and NODE, but the authors did not point out it. Moreover, it is better to analysis the distribution of instance weights. For example:\n\na) Is it long-tailed?\n\nb) Are some trees very important for most of the samples?\n\n2. Too many hyperparameters, i.e. 11 are tuned for the proposed method. While the number of hyperparameters is no more than 4 for compared methods. We have reason to doubt that the performance gain is obtained by finer hyperparameter tuning. Could you reduce your number of hyperparameters to 4 and retrain your model for comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LY0Tz31ZUq", "reviewer": "Reviewer_Tbch", "summary": "This article present \"GRANDE\" a gradient-based extension of GradTree to weighed ensembles of trees.\n\"GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent\" is a (yet-unpuplished to the best of my knowledge) work by (Marton et al. 2023) that proposes to learn by gradient descent on a relaxation of decision trees.\nThis relaxation works by reformulating the decision tree as a sum of indicative functions (one for each leaf), and relaxing the decision thresholds with a \"straight-through\" (ST) operator that replaces the hard decision by a soft decision during the backward pass of gradient descent.\nThe article first summarizes GradTree, then introduces the specificities of GRANDE:\n- a novel variant soft-thresholding called \"softsign\" that is used with the straigh-through operator\n- a novel instance-wise weighing scheme for ensemble of trees\n\nThen a thorough experimental study is provided to evaluate the model and other models on several datasets. The appendix provides several experiments and ablation studies.", "questions": "- Your weighing scheme seems interesting but does it really require gradient descent ? Could you take a fixed forest, add weights to the leaves and optimize these weights afterwards ? I am thinking of a paper like (Cui et al. 2023) https://arxiv.org/pdf/2304.13761.pdf\n- Did you consider hybridizing greedy and gradient approaches ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Dealing with tabular data, as efficiently as gradient-boosted trees do, though neural networks and gradient descent is yet an open challenge. For this very reason, proposing new, or even slightly new models that are able to train tree ensembles in a reasonable time through gradient descent is an interesting contribution.\n\n- The paper is clear, well written, and illustrated with several illustrating Figures. I liked reading it.\n- I could not manage to run the supplementary material code, but the provided experiments seems serious and convincing with several datasets and models tested and some ablation studies\n- This method, although very slow at training time when compared to gradient-boosting, seems promising for the many neural/tree hybridizations its suggests\n- The instance-based weighing is interesting (especially for explainability)", "weaknesses": "- My major concern is about hyper-parameters tuning (section C of appendix): I understand that compute resources should be spared, but it seems unfair to optimize the number of trees for GRANDE but not for XGBoost and CatBoost, especially given the fact that XGBoost and CatBoost are the cheapest algorithms to train.\n- The results of GRANDE are good on several datasets, but become less impressive when the number of features is high\n- The 2^d term in the sums suggests that the depth is a real limitation of the method\n- Fact is that gradient descent is much slower than greedy optimization\n- The related-works part seems a bit short given the huge literature on ensemble trees\n\nMinor Remarks:\npage 4 (section 3.3) : the W matrix is in R^{E\\times 2^d}, not R^E \\times R^{2^d}\nRegarding the code I tried unsuccessfully to run the notebooks: two packages were missing in the installation script: \"chardet\" and \"cchardet\", even with this that fixed there is a remaining issue with the \"TabSurvey\" sub-package which seems to require a specific environment to run (It fails on \"from models import\" statements).\npage 7:  the Phishing Website study should appear as a subsection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yYoUMo5vIN", "reviewer": "Reviewer_SMN1", "summary": "This paper presents GRANDE, a novel method for learning hard, axis-aligned decision tree ensembles using end-to-end gradient descent. The paper extends GradTree to a weighted tree ensemble, introduces softsign as a differentiable split function, and proposes a novel instance-wise weighting technique. The paper evaluates GRANDE on a predefined benchmark of 19 binary classification datasets and shows that it outperforms existing gradient-boosting and deep learning frameworks on most datasets The paper also demonstrates that GRANDE can learn simple and complex rules within a single model and provide local explanations based on instance-wise weights.", "questions": "1. Instead of using exclusively binary class benchmark datasets, could the authors also present comparative results for multi-label classification problems?\n    \n2. It would be valuable if the authors could include comparisons with other tree ensemble methods, such as random forest and extra trees.\n    \n3. Regarding the interpretability advantages of instance-wise weighting, could the authors provide a more comprehensive analysis? For instance, could they share statistics like the average node count of the highest-weighted estimators across all datasets? Furthermore, could the authors offer any theoretical insights into the benefits of instance-wise weighting?\n    \n4. Instance-wise weighting introduces a larger number of weights compared to estimator-wise weighting. Particularly when trees have greater depth (e.g., depth=10, 2^10 vs. 1 for one estimator), is there a significant impact on the training time due to the increased number of weights?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The transition from GradTree to a Tree ensemble is a natural and straightforward progression, and the reported performance is commendable.\n    \n2. The choice of an approximation function for the Heaviside step function is well-justified. The concept of instance-wise weighting is innovative and logically sound.", "weaknesses": "My main concern is about the performance evaluation, please find the specific info in the Questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)